2024-09-18T12:15:06.209 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T12:15:06.227 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T12:15:06.307 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} email: skanta@redhat.com first_in_suite: false job_id: '7911038' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfREDwErBvlw5jI9vDF3S4DF4e8CF4WAQ6TQsntqI3Zod1NpCed6cD2EIhPwwuhZSSYOHWA7nEvZeALyp2g19Y= smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIrgMxfgkfdCf7hmaE8/o8bMZQXWVHtXJf6pMogy8/2+kNmmsUaXy5ea4zPeOQg4M5z1i4Tg5HBeiUfqxR4yTzg= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T12:15:06.307 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T12:15:06.308 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T12:15:06.308 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T12:15:06.310 INFO:teuthology.task.internal:Checking packages... 2024-09-18T12:15:06.342 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T12:15:06.342 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T12:15:06.343 INFO:teuthology.packaging:ref: None 2024-09-18T12:15:06.343 INFO:teuthology.packaging:tag: None 2024-09-18T12:15:06.343 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T12:15:06.343 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T12:15:06.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T12:15:06.486 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645.g7061ea6c 2024-09-18T12:15:06.488 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T12:15:06.494 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T12:15:06.495 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T12:15:06.558 INFO:teuthology.task.internal:Saving configuration 2024-09-18T12:15:06.572 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T12:15:06.579 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T12:15:06.604 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 12:10:09.244490', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfREDwErBvlw5jI9vDF3S4DF4e8CF4WAQ6TQsntqI3Zod1NpCed6cD2EIhPwwuhZSSYOHWA7nEvZeALyp2g19Y='} 2024-09-18T12:15:06.627 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 12:10:09.245644', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIrgMxfgkfdCf7hmaE8/o8bMZQXWVHtXJf6pMogy8/2+kNmmsUaXy5ea4zPeOQg4M5z1i4Tg5HBeiUfqxR4yTzg='} 2024-09-18T12:15:06.627 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T12:15:06.633 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-18T12:15:06.633 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-18T12:15:06.633 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T12:15:06.695 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb7238e7ac0>, signals=[15]) 2024-09-18T12:15:06.696 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T12:15:06.701 INFO:teuthology.task.internal:Opening connections... 2024-09-18T12:15:06.701 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:15:06.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:15:06.783 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:15:06.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:15:06.871 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T12:15:06.878 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-09-18T12:15:06.896 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-09-18T12:15:06.896 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-09-18T12:15:06.953 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-09-18T12:15:06.953 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-09-18T12:15:06.954 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T12:15:06.955 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T12:15:06.955 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T12:15:06.955 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-09-18T12:15:06.974 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2024-09-18T12:15:06.993 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2024-09-18T12:15:06.993 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-09-18T12:15:07.051 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="9" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="9" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el9" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:LOGO="fedora-logo-icon" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2024-09-18T12:15:07.052 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T12:15:07.053 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T12:15:07.053 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T12:15:07.053 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-09-18T12:15:07.072 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T12:15:07.080 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T12:15:07.086 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T12:15:07.086 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-09-18T12:15:07.089 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2024-09-18T12:15:07.109 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T12:15:07.115 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T12:15:07.115 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-09-18T12:15:07.147 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2024-09-18T12:15:07.358 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T12:15:07.407 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T12:15:07.434 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T12:15:07.435 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T12:15:07.435 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T12:15:07.435 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T12:15:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:15:07.453 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-09-18T12:15:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:15:07.454 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-18T12:15:07.512 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-503.el9.x86_64 2024-09-18T12:15:07.512 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-503.el9.x86_64 2024-09-18T12:15:07.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-09-18T12:15:07.514 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-503.el9.x86_64 2024-09-18T12:15:07.514 INFO:teuthology.task.kernel:Running kernel on smithi144: 5.14.0-503.el9.x86_64 2024-09-18T12:15:07.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum install -y kernel 2024-09-18T12:15:09.727 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-18T12:15:11.386 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-18T12:15:11.713 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.3 MB 00:02 2024-09-18T12:15:13.514 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-18T12:15:15.653 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-18T12:15:17.312 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-18T12:15:18.700 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-18T12:15:19.106 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-18T12:15:20.615 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-18T12:15:21.861 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-18T12:15:24.123 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-18T12:15:26.020 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T12:15:26.021 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T12:15:26.119 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:15:26.123 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:15:26.123 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-09-18T12:15:26.123 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:15:26.123 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:15:26.123 INFO:teuthology.orchestra.run.smithi144.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T12:15:26.123 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout:Install 4 Packages 2024-09-18T12:15:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:15:26.125 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 79 M 2024-09-18T12:15:26.126 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 126 M 2024-09-18T12:15:26.126 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:15:26.307 INFO:teuthology.orchestra.run.smithi144.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 462 kB/s | 36 kB 00:00 2024-09-18T12:15:26.907 INFO:teuthology.orchestra.run.smithi144.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-18T12:15:27.225 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-18T12:15:27.349 INFO:teuthology.orchestra.run.smithi144.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 27 MB/s | 29 MB 00:01 2024-09-18T12:15:27.766 INFO:teuthology.orchestra.run.smithi144.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 22 MB/s | 35 MB 00:01 2024-09-18T12:15:27.766 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:15:27.767 INFO:teuthology.orchestra.run.smithi144.stdout:Total 48 MB/s | 79 MB 00:01 2024-09-18T12:15:28.451 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:15:28.823 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:15:28.823 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:15:29.132 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T12:15:29.132 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T12:15:29.225 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:15:29.233 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:15:29.234 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-09-18T12:15:29.235 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:15:29.235 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 79 M 2024-09-18T12:15:29.235 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 126 M 2024-09-18T12:15:29.235 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:15:29.485 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 235 kB/s | 36 kB 00:00 2024-09-18T12:15:29.801 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:15:29.802 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:15:30.152 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-09-18T12:15:30.677 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-18T12:15:31.036 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 20 MB/s | 35 MB 00:01 2024-09-18T12:15:31.036 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:15:31.036 INFO:teuthology.orchestra.run.smithi116.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-18T12:15:31.717 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:15:32.066 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:15:32.066 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:15:32.231 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:15:32.946 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T12:15:33.016 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:15:33.016 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:15:33.027 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T12:15:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:15:35.807 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T12:15:35.954 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T12:15:36.088 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T12:15:36.169 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T12:15:38.898 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T12:15:39.023 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T12:15:40.783 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T12:15:40.828 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:15:43.834 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T12:15:43.882 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:15:45.648 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:15:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:23.992 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:24.037 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:24.347 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:24.347 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T12:16:24.347 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T12:16:24.347 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T12:16:25.165 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:25.165 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:16:25.165 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:16:25.165 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T12:16:25.166 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T12:16:25.166 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T12:16:25.166 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T12:16:25.166 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:16:25.166 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:16:25.525 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T12:16:25.611 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T12:16:25.611 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T12:16:25.611 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T12:16:25.612 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T12:16:25.612 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T12:16:25.612 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-18T12:16:25.612 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T12:16:25.612 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T12:16:25.671 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T12:16:25.672 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T12:16:25.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -qi grub2-tools 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Name : grub2-tools 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Epoch : 1 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Version : 2.06 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Release : 82.el9 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: x86_64 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Group : Unspecified 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Size : 8274670 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:License : GPLv3+ 2024-09-18T12:16:25.711 INFO:teuthology.orchestra.run.smithi144.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:Packager : builder@centos.org 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:Vendor : CentOS 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:Summary : Support tools for GRUB. 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:Description : 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T12:16:25.712 INFO:teuthology.orchestra.run.smithi144.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T12:16:25.713 INFO:teuthology.orchestra.run.smithi144.stdout:hardware devices. 2024-09-18T12:16:25.713 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:16:25.713 INFO:teuthology.orchestra.run.smithi144.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T12:16:25.714 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T12:16:25.714 INFO:teuthology.task.kernel:Updating grub on smithi144 to boot 5.14.0-508.el9.x86_64 2024-09-18T12:16:25.714 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T12:16:26.406 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2024-09-18T12:16:26.881 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:26.921 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:27.305 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:27.305 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T12:16:27.306 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T12:16:27.306 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T12:16:27.772 INFO:teuthology.orchestra.run.smithi144.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T12:16:27.801 INFO:teuthology.orchestra.run.smithi144.stderr:done 2024-09-18T12:16:27.803 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-09-18T12:16:27.819 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.tvvogXkBIQ 2024-09-18T12:16:27.819 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tvvogXkBIQ 2024-09-18T12:16:27.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.tvvogXkBIQ 2024-09-18T12:16:28.029 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.tvvogXkBIQ is 6KB 2024-09-18T12:16:28.052 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.tvvogXkBIQ 2024-09-18T12:16:28.067 DEBUG:teuthology.orchestra.run.smithi144:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T12:16:28.131 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T12:16:28.131 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T12:16:28.131 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T12:16:28.132 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T12:16:28.133 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T12:16:28.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:16:28.180 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:16:28.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2024-09-18T12:16:28.409 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-09-18T12:16:28.510 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T12:16:28.598 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T12:16:28.599 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T12:16:28.599 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T12:16:28.599 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T12:16:28.599 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T12:16:28.599 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T12:16:28.599 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T12:16:28.599 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T12:16:28.659 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T12:16:28.659 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T12:16:28.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 82.el9 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8274670 2024-09-18T12:16:28.694 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T12:16:28.695 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-09-18T12:16:28.696 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:16:28.696 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T12:16:28.697 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T12:16:28.697 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-508.el9.x86_64 2024-09-18T12:16:28.697 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T12:16:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-09-18T12:16:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T12:16:30.749 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-09-18T12:16:30.752 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-09-18T12:16:30.767 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.wsIB602OAu 2024-09-18T12:16:30.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wsIB602OAu 2024-09-18T12:16:30.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.wsIB602OAu 2024-09-18T12:16:31.511 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.wsIB602OAu is 6KB 2024-09-18T12:16:31.560 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.wsIB602OAu 2024-09-18T12:16:31.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T12:16:31.640 INFO:teuthology.orchestra.run.smithi116.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T12:16:31.640 INFO:teuthology.orchestra.run.smithi116.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T12:16:31.640 INFO:teuthology.orchestra.run.smithi116.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T12:16:31.640 INFO:teuthology.orchestra.run.smithi116.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T12:16:31.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T12:16:31.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-09-18T12:16:58.329 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-18T12:16:58.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-18T12:16:58.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:17:01.797 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T12:17:01.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-09-18T12:17:01.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:17:16.861 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-09-18T12:17:25.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-18T12:17:25.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:18:01.834 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T12:18:10.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-09-18T12:18:10.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:18:11.219 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-09-18T12:18:11.870 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-09-18T12:18:11.871 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T12:18:11.871 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-09-18T12:18:11.886 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-508.el9.x86_64 2024-09-18T12:18:11.886 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T12:18:11.886 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T12:18:11.886 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T12:18:12.887 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T12:18:12.888 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T12:18:12.963 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-09-18T12:18:12.994 DEBUG:teuthology.parallel:result is None 2024-09-18T12:18:25.873 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T12:18:37.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-18T12:18:37.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:18:38.239 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-18T12:18:38.633 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-18T12:18:38.633 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-18T12:18:38.633 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-18T12:18:38.649 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-508.el9.x86_64 2024-09-18T12:18:38.649 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T12:18:38.649 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T12:18:38.649 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T12:18:39.650 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-18T12:18:39.651 DEBUG:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T12:18:39.708 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2024-09-18T12:18:39.730 DEBUG:teuthology.parallel:result is None 2024-09-18T12:18:39.730 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T12:18:39.737 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T12:18:39.738 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T12:18:39.741 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T12:18:39.758 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T12:18:39.764 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T12:18:39.774 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T12:18:39.775 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T12:18:39.801 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T12:18:39.832 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T12:18:39.838 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T12:18:39.838 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T12:18:39.871 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T12:18:39.904 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:18:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:18:39.915 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:18:39.926 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T12:18:39.927 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T12:18:39.934 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T12:18:39.935 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T12:18:39.960 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T12:18:39.995 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T12:18:40.004 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T12:18:40.004 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T12:18:40.026 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T12:18:40.051 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T12:18:40.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T12:18:40.175 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T12:18:40.256 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T12:18:40.298 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:18:40.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T12:18:40.363 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T12:18:40.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T12:18:40.460 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T12:18:40.543 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T12:18:40.586 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:18:40.586 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T12:18:40.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-09-18T12:18:40.656 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2024-09-18T12:18:40.704 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T12:18:40.738 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T12:18:41.135 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T12:18:41.143 INFO:teuthology.task.internal:Starting timer... 2024-09-18T12:18:41.143 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T12:18:41.155 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T12:18:41.162 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-09-18T12:18:41.252 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-09-18T12:18:41.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2024-09-18T12:18:41.314 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2024-09-18T12:18:41.316 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T12:18:41.316 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-09-18T12:18:41.346 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-09-18T12:18:41.346 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2024-09-18T12:18:41.370 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2024-09-18T12:18:41.370 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi144.front.sepia.ceph.com': 'permissive'} 2024-09-18T12:18:41.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T12:18:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:18:41.411 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T12:18:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:18:41.440 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T12:18:41.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-09-18T12:18:41.483 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2024-09-18T12:18:41.519 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T12:18:41.528 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T12:18:41.540 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T12:18:41.541 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T12:22:27.676 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2024-09-18T12:22:27.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-09-18T12:22:27.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:22:27.758 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-09-18T12:22:27.836 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-09-18T12:22:27.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-18T12:22:27.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T12:22:27.912 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-18T12:22:27.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-18T12:22:27.994 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T12:22:28.006 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T12:22:28.007 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T12:22:28.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:22:28.010 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T12:22:28.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:22:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T12:22:28.051 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T12:22:28.064 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T12:22:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-09-18T12:22:28.078 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T12:22:28.085 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-09-18T12:22:28.100 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T12:22:28.103 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2024-09-18T12:22:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T12:22:28.114 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2024-09-18T12:22:28.129 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T12:22:28.142 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T12:22:28.170 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-09-18T12:22:28.173 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T12:22:28.173 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-09-18T12:22:28.173 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.173 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.173 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.173 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.199 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-09-18T12:22:28.203 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T12:22:28.203 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-09-18T12:22:28.203 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.203 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.203 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.203 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:22:28.203 INFO:teuthology.run_tasks:Running task install... 2024-09-18T12:22:28.211 DEBUG:teuthology.task.install:project ceph 2024-09-18T12:22:28.211 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T12:22:28.211 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T12:22:28.211 INFO:teuthology.task.install:Using flavor: default 2024-09-18T12:22:28.217 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T12:22:28.217 INFO:teuthology.task.install:extra packages: [] 2024-09-18T12:22:28.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo dnf -y install 'dnf-command(copr)' 2024-09-18T12:22:28.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo dnf -y install 'dnf-command(copr)' 2024-09-18T12:22:28.788 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:48 ago on Wed 18 Sep 2024 12:21:40 PM UTC. 2024-09-18T12:22:28.817 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:48 ago on Wed 18 Sep 2024 12:21:40 PM UTC. 2024-09-18T12:22:28.874 INFO:teuthology.orchestra.run.smithi116.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-18T12:22:28.903 INFO:teuthology.orchestra.run.smithi144.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-18T12:22:28.916 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:22:28.917 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:22:28.917 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:22:28.945 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:22:28.946 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:22:28.946 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:22:28.986 DEBUG:teuthology.orchestra.run.smithi116:> sudo dnf -y copr enable ceph/el9 2024-09-18T12:22:29.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo dnf -y copr enable ceph/el9 2024-09-18T12:22:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-18T12:22:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:of the main distribution, and quality may vary. 2024-09-18T12:22:29.155 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:22:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-18T12:22:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-18T12:22:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:, 2024-09-18T12:22:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:and packages are not held to any quality or security level. 2024-09-18T12:22:29.156 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:22:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:Please do not file bug reports about these packages in Fedora 2024-09-18T12:22:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:of the main distribution, and quality may vary. 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:, 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:and packages are not held to any quality or security level. 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:Please do not file bug reports about these packages in Fedora 2024-09-18T12:22:29.180 INFO:teuthology.orchestra.run.smithi144.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-18T12:22:29.207 INFO:teuthology.orchestra.run.smithi116.stdout:Repository successfully enabled. 2024-09-18T12:22:29.232 INFO:teuthology.orchestra.run.smithi144.stdout:Repository successfully enabled. 2024-09-18T12:22:29.242 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'tag': None, 'wait_for_package': False} 2024-09-18T12:22:29.242 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T12:22:29.267 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'tag': None, 'wait_for_package': False} 2024-09-18T12:22:29.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T12:22:29.395 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/ 2024-09-18T12:22:29.396 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1645.g7061ea6c 2024-09-18T12:22:29.417 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/ 2024-09-18T12:22:29.418 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1645.g7061ea6c 2024-09-18T12:22:29.505 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T12:22:29.506 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:22:29.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T12:22:29.524 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T12:22:29.525 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:22:29.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T12:22:29.530 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T12:22:29.531 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7061ea6ca50908d104e9f3979dc615e118d9e153/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T12:22:29.550 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T12:22:29.550 DEBUG:teuthology.orchestra.run.smithi144:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7061ea6ca50908d104e9f3979dc615e118d9e153/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T12:22:29.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T12:22:29.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T12:22:29.671 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T12:22:29.691 DEBUG:teuthology.orchestra.run.smithi144:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T12:22:29.736 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-09-18T12:22:29.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-09-18T12:22:29.755 INFO:teuthology.orchestra.run.smithi144.stdout:check_obsoletes = 1 2024-09-18T12:22:29.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum clean all 2024-09-18T12:22:29.972 INFO:teuthology.orchestra.run.smithi116.stdout:45 files removed 2024-09-18T12:22:29.984 INFO:teuthology.orchestra.run.smithi144.stdout:45 files removed 2024-09-18T12:22:29.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw 2024-09-18T12:22:30.016 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw 2024-09-18T12:22:30.551 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 289 kB/s | 77 kB 00:00 2024-09-18T12:22:30.561 INFO:teuthology.orchestra.run.smithi144.stdout:ceph packages for x86_64 301 kB/s | 77 kB 00:00 2024-09-18T12:22:30.835 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-18T12:22:30.861 INFO:teuthology.orchestra.run.smithi144.stdout:ceph noarch packages 42 kB/s | 11 kB 00:00 2024-09-18T12:22:31.111 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-18T12:22:31.137 INFO:teuthology.orchestra.run.smithi144.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-18T12:22:31.269 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for el9 owned by ceph 62 kB/s | 7.6 kB 00:00 2024-09-18T12:22:31.288 INFO:teuthology.orchestra.run.smithi144.stdout:Copr repo for el9 owned by ceph 66 kB/s | 7.6 kB 00:00 2024-09-18T12:22:33.086 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-18T12:22:33.112 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-18T12:22:36.414 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 20 MB 00:02 2024-09-18T12:22:42.137 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-18T12:22:42.447 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-18T12:22:44.248 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-18T12:22:45.552 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-18T12:22:46.396 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-18T12:22:47.822 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 19 kB 00:00 2024-09-18T12:22:49.326 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-18T12:22:50.966 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-18T12:22:53.258 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:22:53.269 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 14 M 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.1 M 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 5.0 M 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 22 M 2024-09-18T12:22:53.272 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 637 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 458 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 4.9 M 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 45 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 120 k 2024-09-18T12:22:53.273 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 156 k 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 320 k 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 296 k 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 99 k 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout:Install 22 Packages 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 2 Packages 2024-09-18T12:22:53.274 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:22:53.275 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 58 M 2024-09-18T12:22:53.275 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:22:54.126 INFO:teuthology.orchestra.run.smithi144.stdout:(1/24): ceph-base-17.2.7-1645.g7061ea6c.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-18T12:22:54.192 INFO:teuthology.orchestra.run.smithi144.stdout:(2/24): ceph-selinux-17.2.7-1645.g7061ea6c.el9. 378 kB/s | 25 kB 00:00 2024-09-18T12:22:54.284 INFO:teuthology.orchestra.run.smithi144.stdout:(3/24): libcephfs2-17.2.7-1645.g7061ea6c.el9.x8 6.8 MB/s | 637 kB 00:00 2024-09-18T12:22:54.377 INFO:teuthology.orchestra.run.smithi144.stdout:(4/24): libradosstriper1-17.2.7-1645.g7061ea6c. 4.9 MB/s | 458 kB 00:00 2024-09-18T12:22:54.535 INFO:teuthology.orchestra.run.smithi144.stdout:(5/24): ceph-radosgw-17.2.7-1645.g7061ea6c.el9. 17 MB/s | 14 MB 00:00 2024-09-18T12:22:54.577 INFO:teuthology.orchestra.run.smithi144.stdout:(6/24): python3-ceph-argparse-17.2.7-1645.g7061 1.1 MB/s | 45 kB 00:00 2024-09-18T12:22:54.652 INFO:teuthology.orchestra.run.smithi144.stdout:(7/24): librgw2-17.2.7-1645.g7061ea6c.el9.x86_6 18 MB/s | 4.9 MB 00:00 2024-09-18T12:22:54.677 INFO:teuthology.orchestra.run.smithi144.stdout:(8/24): python3-ceph-common-17.2.7-1645.g7061ea 1.2 MB/s | 120 kB 00:00 2024-09-18T12:22:54.703 INFO:teuthology.orchestra.run.smithi144.stdout:(9/24): python3-cephfs-17.2.7-1645.g7061ea6c.el 3.0 MB/s | 156 kB 00:00 2024-09-18T12:22:54.728 INFO:teuthology.orchestra.run.smithi144.stdout:(10/24): python3-rados-17.2.7-1645.g7061ea6c.el 6.3 MB/s | 320 kB 00:00 2024-09-18T12:22:54.756 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-18T12:22:54.770 INFO:teuthology.orchestra.run.smithi144.stdout:(11/24): python3-rbd-17.2.7-1645.g7061ea6c.el9. 4.3 MB/s | 296 kB 00:00 2024-09-18T12:22:54.795 INFO:teuthology.orchestra.run.smithi144.stdout:(12/24): python3-rgw-17.2.7-1645.g7061ea6c.el9. 1.4 MB/s | 99 kB 00:00 2024-09-18T12:22:55.071 INFO:teuthology.orchestra.run.smithi144.stdout:(13/24): ceph-common-17.2.7-1645.g7061ea6c.el9. 16 MB/s | 22 MB 00:01 2024-09-18T12:22:55.096 INFO:teuthology.orchestra.run.smithi144.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 151 kB/s | 45 kB 00:00 2024-09-18T12:22:55.129 INFO:teuthology.orchestra.run.smithi144.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-18T12:22:55.171 INFO:teuthology.orchestra.run.smithi144.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-18T12:22:55.196 INFO:teuthology.orchestra.run.smithi144.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-18T12:22:55.247 INFO:teuthology.orchestra.run.smithi144.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-18T12:22:55.272 INFO:teuthology.orchestra.run.smithi144.stdout:(19/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-18T12:22:55.297 INFO:teuthology.orchestra.run.smithi144.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 519 kB/s | 49 kB 00:00 2024-09-18T12:22:55.356 INFO:teuthology.orchestra.run.smithi144.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 618 kB/s | 67 kB 00:00 2024-09-18T12:22:55.431 INFO:teuthology.orchestra.run.smithi144.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-18T12:22:55.490 INFO:teuthology.orchestra.run.smithi144.stdout:(23/24): librados2-17.2.7-1645.g7061ea6c.el9.x8 17 MB/s | 3.1 MB 00:00 2024-09-18T12:22:55.549 INFO:teuthology.orchestra.run.smithi144.stdout:(24/24): librbd1-17.2.7-1645.g7061ea6c.el9.x86_ 15 MB/s | 2.9 MB 00:00 2024-09-18T12:22:55.553 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:22:55.553 INFO:teuthology.orchestra.run.smithi144.stdout:Total 25 MB/s | 58 MB 00:02 2024-09-18T12:22:55.612 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:22:55.644 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:22:55.644 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:22:56.308 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:22:56.308 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:22:57.100 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:22:57.116 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 14 M 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.1 M 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 5.0 M 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 22 M 2024-09-18T12:22:57.117 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 637 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 458 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T12:22:57.118 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 4.9 M 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 45 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 120 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 156 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 320 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 296 k 2024-09-18T12:22:57.119 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 99 k 2024-09-18T12:22:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T12:22:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:22:57.120 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:22:57.120 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:22:57.120 INFO:teuthology.orchestra.run.smithi116.stdout:Install 22 Packages 2024-09-18T12:22:57.120 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2024-09-18T12:22:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:22:57.121 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 58 M 2024-09-18T12:22:57.121 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:22:57.524 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:22:57.891 INFO:teuthology.orchestra.run.smithi116.stdout:(1/24): ceph-base-17.2.7-1645.g7061ea6c.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-18T12:22:57.897 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-18T12:22:57.941 INFO:teuthology.orchestra.run.smithi116.stdout:(2/24): ceph-selinux-17.2.7-1645.g7061ea6c.el9. 503 kB/s | 25 kB 00:00 2024-09-18T12:22:57.959 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T12:22:58.025 INFO:teuthology.orchestra.run.smithi116.stdout:(3/24): libcephfs2-17.2.7-1645.g7061ea6c.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-18T12:22:58.034 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T12:22:58.117 INFO:teuthology.orchestra.run.smithi116.stdout:(4/24): libradosstriper1-17.2.7-1645.g7061ea6c. 4.9 MB/s | 458 kB 00:00 2024-09-18T12:22:58.120 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T12:22:58.144 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T12:22:58.215 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T12:22:58.261 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-18T12:22:58.290 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-18T12:22:58.467 INFO:teuthology.orchestra.run.smithi116.stdout:(5/24): librgw2-17.2.7-1645.g7061ea6c.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-18T12:22:58.617 INFO:teuthology.orchestra.run.smithi116.stdout:(6/24): ceph-radosgw-17.2.7-1645.g7061ea6c.el9. 12 MB/s | 14 MB 00:01 2024-09-18T12:22:58.642 INFO:teuthology.orchestra.run.smithi116.stdout:(7/24): python3-ceph-argparse-17.2.7-1645.g7061 257 kB/s | 45 kB 00:00 2024-09-18T12:22:58.668 INFO:teuthology.orchestra.run.smithi116.stdout:(8/24): python3-ceph-common-17.2.7-1645.g7061ea 2.4 MB/s | 120 kB 00:00 2024-09-18T12:22:58.693 INFO:teuthology.orchestra.run.smithi116.stdout:(9/24): python3-cephfs-17.2.7-1645.g7061ea6c.el 3.0 MB/s | 156 kB 00:00 2024-09-18T12:22:58.727 INFO:teuthology.orchestra.run.smithi116.stdout:(10/24): python3-rados-17.2.7-1645.g7061ea6c.el 5.3 MB/s | 320 kB 00:00 2024-09-18T12:22:58.731 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-18T12:22:58.771 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T12:22:58.834 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T12:22:58.880 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-18T12:22:58.922 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 10/26 2024-09-18T12:22:58.936 INFO:teuthology.orchestra.run.smithi116.stdout:(11/24): ceph-common-17.2.7-1645.g7061ea6c.el9. 15 MB/s | 22 MB 00:01 2024-09-18T12:22:58.959 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T12:22:58.961 INFO:teuthology.orchestra.run.smithi116.stdout:(12/24): python3-rbd-17.2.7-1645.g7061ea6c.el9. 1.1 MB/s | 296 kB 00:00 2024-09-18T12:22:58.986 INFO:teuthology.orchestra.run.smithi116.stdout:(13/24): python3-rgw-17.2.7-1645.g7061ea6c.el9. 384 kB/s | 99 kB 00:00 2024-09-18T12:22:59.023 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T12:22:59.049 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T12:22:59.114 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T12:22:59.163 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-18T12:22:59.209 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-18T12:22:59.246 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-18T12:22:59.336 INFO:teuthology.orchestra.run.smithi116.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 121 kB/s | 45 kB 00:00 2024-09-18T12:22:59.512 INFO:teuthology.orchestra.run.smithi116.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 278 kB/s | 160 kB 00:00 2024-09-18T12:22:59.543 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-18T12:22:59.586 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T12:22:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-09-18T12:22:59.660 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T12:22:59.670 INFO:teuthology.orchestra.run.smithi116.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-18T12:22:59.696 INFO:teuthology.orchestra.run.smithi116.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-18T12:22:59.721 INFO:teuthology.orchestra.run.smithi116.stdout:(19/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 760 kB/s | 292 kB 00:00 2024-09-18T12:22:59.732 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 19/26 2024-09-18T12:22:59.746 INFO:teuthology.orchestra.run.smithi116.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-18T12:22:59.772 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 20/26 2024-09-18T12:22:59.822 INFO:teuthology.orchestra.run.smithi116.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 793 kB/s | 662 kB 00:00 2024-09-18T12:22:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-18T12:23:00.006 INFO:teuthology.orchestra.run.smithi116.stdout:(23/24): librados2-17.2.7-1645.g7061ea6c.el9.x8 12 MB/s | 3.1 MB 00:00 2024-09-18T12:23:00.073 INFO:teuthology.orchestra.run.smithi116.stdout:(24/24): librbd1-17.2.7-1645.g7061ea6c.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-09-18T12:23:00.077 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:00.077 INFO:teuthology.orchestra.run.smithi116.stdout:Total 20 MB/s | 58 MB 00:02 2024-09-18T12:23:00.143 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:23:00.177 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:23:00.177 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:23:00.814 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:23:00.815 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:23:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:23:02.154 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T12:23:02.194 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T12:23:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-18T12:23:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T12:23:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T12:23:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T12:23:02.709 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T12:23:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T12:23:02.747 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T12:23:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T12:23:02.850 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T12:23:02.850 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T12:23:02.850 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T12:23:02.850 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-18T12:23:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-18T12:23:02.898 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T12:23:03.336 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-18T12:23:03.362 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T12:23:03.425 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T12:23:03.462 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-18T12:23:03.504 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 10/26 2024-09-18T12:23:03.551 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T12:23:03.619 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T12:23:03.648 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T12:23:03.705 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T12:23:03.746 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-18T12:23:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-18T12:23:03.828 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-18T12:23:04.117 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-18T12:23:04.169 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T12:23:04.258 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T12:23:04.331 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 19/26 2024-09-18T12:23:04.362 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 20/26 2024-09-18T12:23:06.710 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T12:23:06.750 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T12:23:07.274 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T12:23:07.312 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T12:23:07.410 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T12:23:07.410 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T12:23:07.411 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T12:23:07.411 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:07.455 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T12:23:12.894 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T12:23:12.894 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /sys 2024-09-18T12:23:12.895 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /proc 2024-09-18T12:23:12.895 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /mnt 2024-09-18T12:23:12.895 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /var/tmp 2024-09-18T12:23:12.895 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /home 2024-09-18T12:23:12.895 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /root 2024-09-18T12:23:12.895 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /tmp 2024-09-18T12:23:12.895 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:14.217 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T12:23:14.296 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T12:23:14.296 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:14.296 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T12:23:14.296 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T12:23:14.296 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T12:23:14.296 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:14.298 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T12:23:14.387 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T12:23:14.387 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T12:23:15.682 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 6/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 7/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 8/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 9/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 10/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T12:23:15.683 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 13/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-18T12:23:15.684 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 25/26 2024-09-18T12:23:16.594 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T12:23:16.594 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T12:23:16.595 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.596 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:16.597 INFO:teuthology.orchestra.run.smithi144.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T12:23:16.597 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:16.597 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:16.958 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-test 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-09-18T12:23:17.301 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:17.538 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:27 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:17.669 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:17.669 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 48 M 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 163 k 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:17.670 INFO:teuthology.orchestra.run.smithi144.stdout:Install 4 Packages 2024-09-18T12:23:17.671 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:17.671 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 48 M 2024-09-18T12:23:17.671 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 193 M 2024-09-18T12:23:17.671 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:23:17.993 INFO:teuthology.orchestra.run.smithi144.stdout:(1/4): libcephsqlite-17.2.7-1645.g7061ea6c.el9. 737 kB/s | 163 kB 00:00 2024-09-18T12:23:18.334 INFO:teuthology.orchestra.run.smithi144.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 541 kB/s | 304 kB 00:00 2024-09-18T12:23:18.359 INFO:teuthology.orchestra.run.smithi144.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 174 kB/s | 64 kB 00:00 2024-09-18T12:23:18.639 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T12:23:18.701 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T12:23:18.701 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:18.701 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T12:23:18.701 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T12:23:18.701 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T12:23:18.701 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:18.702 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T12:23:18.800 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T12:23:18.800 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T12:23:19.293 INFO:teuthology.orchestra.run.smithi144.stdout:(4/4): ceph-test-17.2.7-1645.g7061ea6c.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-18T12:23:19.295 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:19.295 INFO:teuthology.orchestra.run.smithi144.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-18T12:23:19.323 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:23:19.342 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:23:19.342 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:23:19.707 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:23:19.708 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:23:20.019 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T12:23:20.019 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/26 2024-09-18T12:23:20.019 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 6/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 7/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 8/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 9/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 10/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 13/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-18T12:23:20.020 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-18T12:23:20.021 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 25/26 2024-09-18T12:23:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:23:20.331 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T12:23:20.475 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T12:23:20.512 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T12:23:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T12:23:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:21.065 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-09-18T12:23:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.065 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T12:23:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:21.068 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:21.420 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-test 2024-09-18T12:23:22.004 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:28 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:22.146 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:22.146 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:22.146 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:23:22.146 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 48 M 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 163 k 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-09-18T12:23:22.147 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:22.148 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 48 M 2024-09-18T12:23:22.148 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 193 M 2024-09-18T12:23:22.148 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:23:22.413 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): libcephsqlite-17.2.7-1645.g7061ea6c.el9. 759 kB/s | 163 kB 00:00 2024-09-18T12:23:22.438 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-18T12:23:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 449 kB/s | 64 kB 00:00 2024-09-18T12:23:24.030 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): ceph-test-17.2.7-1645.g7061ea6c.el9.x86_ 26 MB/s | 48 MB 00:01 2024-09-18T12:23:24.032 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:24.033 INFO:teuthology.orchestra.run.smithi116.stdout:Total 26 MB/s | 48 MB 00:01 2024-09-18T12:23:24.060 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:23:24.080 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:23:24.080 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:23:24.444 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:23:24.445 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:23:24.924 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T12:23:25.001 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T12:23:25.008 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:23:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T12:23:25.129 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T12:23:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T12:23:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T12:23:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/4 2024-09-18T12:23:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/4 2024-09-18T12:23:27.042 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T12:23:27.338 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T12:23:27.338 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:27.338 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:23:27.338 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:27.338 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:27.338 INFO:teuthology.orchestra.run.smithi144.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T12:23:27.338 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T12:23:27.339 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:27.339 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:27.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph 2024-09-18T12:23:28.203 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:38 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:28.344 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:28.346 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:28.346 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: ceph x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 6.5 k 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.1 M 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 1.4 M 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 240 k 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.9 M 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 15 M 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T12:23:28.347 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T12:23:28.348 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T12:23:28.349 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T12:23:28.350 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T12:23:28.350 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:28.350 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:23:28.350 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:28.350 INFO:teuthology.orchestra.run.smithi144.stdout:Install 34 Packages 2024-09-18T12:23:28.350 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:28.351 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 27 M 2024-09-18T12:23:28.351 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 96 M 2024-09-18T12:23:28.351 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:23:29.085 INFO:teuthology.orchestra.run.smithi144.stdout:(1/34): ceph-17.2.7-1645.g7061ea6c.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-18T12:23:29.286 INFO:teuthology.orchestra.run.smithi144.stdout:(2/34): ceph-mds-17.2.7-1645.g7061ea6c.el9.x86_ 6.4 MB/s | 2.1 MB 00:00 2024-09-18T12:23:29.319 INFO:teuthology.orchestra.run.smithi144.stdout:(3/34): ceph-mgr-17.2.7-1645.g7061ea6c.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-09-18T12:23:29.436 INFO:teuthology.orchestra.run.smithi144.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1645.g7061 2.0 MB/s | 240 kB 00:00 2024-09-18T12:23:29.562 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T12:23:29.637 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T12:23:29.645 INFO:teuthology.orchestra.run.smithi144.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-18T12:23:29.704 INFO:teuthology.orchestra.run.smithi144.stdout:(6/34): ceph-mon-17.2.7-1645.g7061ea6c.el9.x86_ 6.4 MB/s | 3.9 MB 00:00 2024-09-18T12:23:29.837 INFO:teuthology.orchestra.run.smithi144.stdout:(7/34): python3-cryptography-36.0.1-4.el9.x86_6 6.4 MB/s | 1.2 MB 00:00 2024-09-18T12:23:29.996 INFO:teuthology.orchestra.run.smithi144.stdout:(8/34): ceph-osd-17.2.7-1645.g7061ea6c.el9.x86_ 22 MB/s | 15 MB 00:00 2024-09-18T12:23:30.021 INFO:teuthology.orchestra.run.smithi144.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 336 kB/s | 106 kB 00:00 2024-09-18T12:23:30.047 INFO:teuthology.orchestra.run.smithi144.stdout:(10/34): python3-pycparser-2.20-6.el9.noarch.rp 646 kB/s | 135 kB 00:00 2024-09-18T12:23:30.163 INFO:teuthology.orchestra.run.smithi144.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-09-18T12:23:30.189 INFO:teuthology.orchestra.run.smithi144.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 649 kB/s | 125 kB 00:00 2024-09-18T12:23:30.214 INFO:teuthology.orchestra.run.smithi144.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-18T12:23:30.256 INFO:teuthology.orchestra.run.smithi144.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 376 kB/s | 35 kB 00:00 2024-09-18T12:23:30.281 INFO:teuthology.orchestra.run.smithi144.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 763 kB/s | 51 kB 00:00 2024-09-18T12:23:30.315 INFO:teuthology.orchestra.run.smithi144.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 614 kB/s | 77 kB 00:00 2024-09-18T12:23:30.340 INFO:teuthology.orchestra.run.smithi144.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 498 kB/s | 42 kB 00:00 2024-09-18T12:23:30.374 INFO:teuthology.orchestra.run.smithi144.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-18T12:23:30.407 INFO:teuthology.orchestra.run.smithi144.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 445 kB/s | 11 kB 00:00 2024-09-18T12:23:30.433 INFO:teuthology.orchestra.run.smithi144.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-18T12:23:30.458 INFO:teuthology.orchestra.run.smithi144.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 366 kB/s | 18 kB 00:00 2024-09-18T12:23:30.483 INFO:teuthology.orchestra.run.smithi144.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-18T12:23:30.508 INFO:teuthology.orchestra.run.smithi144.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-18T12:23:30.534 INFO:teuthology.orchestra.run.smithi144.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-18T12:23:30.559 INFO:teuthology.orchestra.run.smithi144.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-18T12:23:30.584 INFO:teuthology.orchestra.run.smithi144.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-18T12:23:30.618 INFO:teuthology.orchestra.run.smithi144.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-18T12:23:30.643 INFO:teuthology.orchestra.run.smithi144.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-09-18T12:23:30.677 INFO:teuthology.orchestra.run.smithi144.stdout:(29/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-18T12:23:30.702 INFO:teuthology.orchestra.run.smithi144.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-09-18T12:23:30.727 INFO:teuthology.orchestra.run.smithi144.stdout:(31/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-09-18T12:23:30.753 INFO:teuthology.orchestra.run.smithi144.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-09-18T12:23:30.778 INFO:teuthology.orchestra.run.smithi144.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-18T12:23:30.803 INFO:teuthology.orchestra.run.smithi144.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-18T12:23:30.808 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:30.808 INFO:teuthology.orchestra.run.smithi144.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-18T12:23:30.943 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:23:30.981 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:23:30.981 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:23:31.393 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:23:31.394 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:23:31.814 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T12:23:31.814 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/4 2024-09-18T12:23:31.814 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/4 2024-09-18T12:23:31.814 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T12:23:32.113 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T12:23:32.113 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:32.113 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:23:32.113 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:32.113 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:32.114 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T12:23:32.114 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T12:23:32.114 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:32.114 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:32.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph 2024-09-18T12:23:32.635 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:23:32.706 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-18T12:23:32.755 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-18T12:23:32.796 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-18T12:23:32.888 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-18T12:23:32.943 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:38 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:32.983 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-18T12:23:33.024 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-18T12:23:33.059 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-18T12:23:33.079 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 6.5 k 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.1 M 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 1.4 M 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 240 k 2024-09-18T12:23:33.082 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.9 M 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 15 M 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T12:23:33.083 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T12:23:33.084 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout:Install 34 Packages 2024-09-18T12:23:33.085 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:33.086 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 27 M 2024-09-18T12:23:33.086 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 96 M 2024-09-18T12:23:33.086 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:23:33.089 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-18T12:23:33.127 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-18T12:23:33.161 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-18T12:23:33.201 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-18T12:23:33.236 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-18T12:23:33.270 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-18T12:23:33.319 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-18T12:23:33.355 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T12:23:33.432 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-18T12:23:33.489 INFO:teuthology.orchestra.run.smithi116.stdout:(1/34): ceph-17.2.7-1645.g7061ea6c.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-18T12:23:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-18T12:23:33.651 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-18T12:23:33.698 INFO:teuthology.orchestra.run.smithi116.stdout:(2/34): ceph-mgr-17.2.7-1645.g7061ea6c.el9.x86_ 4.3 MB/s | 1.4 MB 00:00 2024-09-18T12:23:33.704 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-18T12:23:33.748 INFO:teuthology.orchestra.run.smithi116.stdout:(3/34): ceph-mds-17.2.7-1645.g7061ea6c.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-09-18T12:23:33.751 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-18T12:23:33.800 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-18T12:23:33.823 INFO:teuthology.orchestra.run.smithi116.stdout:(4/34): ceph-mon-17.2.7-1645.g7061ea6c.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-18T12:23:33.848 INFO:teuthology.orchestra.run.smithi116.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1645.g7061 2.3 MB/s | 240 kB 00:00 2024-09-18T12:23:33.855 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-18T12:23:34.041 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-18T12:23:34.105 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-18T12:23:34.183 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-18T12:23:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:(6/34): ceph-osd-17.2.7-1645.g7061ea6c.el9.x86_ 31 MB/s | 15 MB 00:00 2024-09-18T12:23:34.271 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-18T12:23:34.315 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-18T12:23:34.436 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-18T12:23:34.592 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 29/34 2024-09-18T12:23:34.631 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T12:23:34.657 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T12:23:34.657 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:34.657 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T12:23:34.657 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T12:23:34.657 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T12:23:34.658 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:34.665 INFO:teuthology.orchestra.run.smithi116.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 301 kB/s | 253 kB 00:00 2024-09-18T12:23:34.874 INFO:teuthology.orchestra.run.smithi116.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 160 kB/s | 106 kB 00:00 2024-09-18T12:23:34.966 INFO:teuthology.orchestra.run.smithi116.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-09-18T12:23:35.366 INFO:teuthology.orchestra.run.smithi116.stdout:(10/34): python3-pycparser-2.20-6.el9.noarch.rp 193 kB/s | 135 kB 00:00 2024-09-18T12:23:35.483 INFO:teuthology.orchestra.run.smithi116.stdout:(11/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-18T12:23:35.525 INFO:teuthology.orchestra.run.smithi116.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 834 kB/s | 35 kB 00:00 2024-09-18T12:23:35.567 INFO:teuthology.orchestra.run.smithi116.stdout:(13/34): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-18T12:23:35.592 INFO:teuthology.orchestra.run.smithi116.stdout:(14/34): python3-requests-2.25.1-8.el9.noarch.r 174 kB/s | 125 kB 00:00 2024-09-18T12:23:35.626 INFO:teuthology.orchestra.run.smithi116.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 884 kB/s | 51 kB 00:00 2024-09-18T12:23:35.676 INFO:teuthology.orchestra.run.smithi116.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 870 kB/s | 43 kB 00:00 2024-09-18T12:23:35.701 INFO:teuthology.orchestra.run.smithi116.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-09-18T12:23:35.735 INFO:teuthology.orchestra.run.smithi116.stdout:(18/34): python3-urllib3-1.26.5-6.el9.noarch.rp 279 kB/s | 215 kB 00:00 2024-09-18T12:23:35.769 INFO:teuthology.orchestra.run.smithi116.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-18T12:23:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 771 kB/s | 18 kB 00:00 2024-09-18T12:23:35.819 INFO:teuthology.orchestra.run.smithi116.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 131 kB/s | 11 kB 00:00 2024-09-18T12:23:35.844 INFO:teuthology.orchestra.run.smithi116.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-18T12:23:35.870 INFO:teuthology.orchestra.run.smithi116.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-18T12:23:35.895 INFO:teuthology.orchestra.run.smithi116.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-18T12:23:35.920 INFO:teuthology.orchestra.run.smithi116.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-09-18T12:23:35.946 INFO:teuthology.orchestra.run.smithi116.stdout:(26/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-18T12:23:35.971 INFO:teuthology.orchestra.run.smithi116.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-18T12:23:35.984 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T12:23:35.987 INFO:teuthology.orchestra.run.smithi116.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-09-18T12:23:36.013 INFO:teuthology.orchestra.run.smithi116.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-18T12:23:36.015 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T12:23:36.015 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:36.015 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T12:23:36.015 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T12:23:36.015 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T12:23:36.015 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:36.030 INFO:teuthology.orchestra.run.smithi116.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-09-18T12:23:36.055 INFO:teuthology.orchestra.run.smithi116.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-18T12:23:36.089 INFO:teuthology.orchestra.run.smithi116.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-18T12:23:36.114 INFO:teuthology.orchestra.run.smithi116.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-09-18T12:23:36.139 INFO:teuthology.orchestra.run.smithi116.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-18T12:23:36.145 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:36.145 INFO:teuthology.orchestra.run.smithi116.stdout:Total 8.9 MB/s | 27 MB 00:03 2024-09-18T12:23:36.275 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:23:36.314 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:23:36.314 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:23:36.411 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T12:23:36.437 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T12:23:36.437 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:36.437 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T12:23:36.437 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T12:23:36.437 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T12:23:36.437 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:36.653 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T12:23:36.678 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T12:23:36.678 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:36.678 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T12:23:36.678 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T12:23:36.678 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T12:23:36.678 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:36.704 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:23:36.704 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:23:36.743 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T12:23:37.998 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:23:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-18T12:23:38.100 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-18T12:23:38.132 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-18T12:23:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-18T12:23:38.302 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-18T12:23:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-18T12:23:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-18T12:23:38.417 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-18T12:23:38.451 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-18T12:23:38.462 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T12:23:38.462 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/34 2024-09-18T12:23:38.462 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 6/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-18T12:23:38.463 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-18T12:23:38.464 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-18T12:23:38.465 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-18T12:23:38.484 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-18T12:23:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-18T12:23:38.560 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-18T12:23:38.603 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-18T12:23:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-18T12:23:38.697 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T12:23:38.775 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-18T12:23:38.921 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-18T12:23:39.002 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-18T12:23:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-18T12:23:39.094 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-18T12:23:39.142 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-18T12:23:39.198 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-18T12:23:39.322 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-18T12:23:39.322 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:39.322 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:23:39.322 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:39.322 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:39.322 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T12:23:39.323 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T12:23:39.324 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T12:23:39.325 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T12:23:39.325 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T12:23:39.325 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T12:23:39.325 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T12:23:39.325 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:39.325 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:39.391 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-18T12:23:39.440 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-18T12:23:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-18T12:23:39.597 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-18T12:23:39.641 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-18T12:23:39.718 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-base 2024-09-18T12:23:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-18T12:23:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 29/34 2024-09-18T12:23:39.949 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T12:23:39.975 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T12:23:39.975 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:39.975 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T12:23:39.975 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T12:23:39.976 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T12:23:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:40.298 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:50 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:40.388 INFO:teuthology.orchestra.run.smithi144.stdout:Package ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:23:40.430 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:40.431 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:23:40.431 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:40.506 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install cephadm 2024-09-18T12:23:41.082 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:51 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:41.211 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 83 k 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:23:41.212 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:41.213 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 83 k 2024-09-18T12:23:41.213 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 375 k 2024-09-18T12:23:41.213 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:23:41.351 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T12:23:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T12:23:41.381 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:41.381 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T12:23:41.381 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T12:23:41.381 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T12:23:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:41.404 INFO:teuthology.orchestra.run.smithi144.stdout:cephadm-17.2.7-1645.g7061ea6c.el9.noarch.rpm 433 kB/s | 83 kB 00:00 2024-09-18T12:23:41.405 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:41.405 INFO:teuthology.orchestra.run.smithi144.stdout:Total 430 kB/s | 83 kB 00:00 2024-09-18T12:23:41.405 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:23:41.410 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:23:41.410 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:23:41.411 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:23:41.412 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:23:41.486 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:23:41.760 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T12:23:41.784 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T12:23:41.784 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:41.784 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T12:23:41.784 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T12:23:41.785 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T12:23:41.785 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:41.894 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:41.986 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T12:23:41.990 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:42.010 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T12:23:42.010 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:42.010 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T12:23:42.010 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T12:23:42.010 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T12:23:42.010 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:42.074 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T12:23:42.334 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:42.584 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:42.584 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:42.584 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:23:42.584 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:42.585 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:42.585 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:42.783 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T12:23:43.373 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:53 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 137 k 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:43.506 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:23:43.507 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:43.507 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 137 k 2024-09-18T12:23:43.507 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 379 k 2024-09-18T12:23:43.507 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:23:43.727 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-immutable-object-cache-17.2.7-1645.g7061ea 626 kB/s | 137 kB 00:00 2024-09-18T12:23:43.728 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:43.728 INFO:teuthology.orchestra.run.smithi144.stdout:Total 621 kB/s | 137 kB 00:00 2024-09-18T12:23:43.728 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:23:43.735 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T12:23:43.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/34 2024-09-18T12:23:43.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/34 2024-09-18T12:23:43.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/34 2024-09-18T12:23:43.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 6/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-18T12:23:43.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-18T12:23:43.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-18T12:23:43.738 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-18T12:23:43.738 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-18T12:23:43.738 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:23:43.738 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:23:43.792 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:23:43.792 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:23:43.948 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:23:44.037 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T12:23:44.062 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T12:23:44.062 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:44.062 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T12:23:44.062 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:44.696 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-18T12:23:44.696 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:44.696 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:23:44.696 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:44.696 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:44.696 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:44.696 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T12:23:44.697 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T12:23:44.698 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T12:23:44.699 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T12:23:44.699 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T12:23:44.699 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T12:23:44.699 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:44.699 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:44.932 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T12:23:44.932 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:44.932 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:23:44.932 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:44.932 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:44.932 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:45.041 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-base 2024-09-18T12:23:45.188 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr 2024-09-18T12:23:45.619 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:51 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:45.707 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:23:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:23:45.750 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:45.780 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:55 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:45.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install cephadm 2024-09-18T12:23:45.872 INFO:teuthology.orchestra.run.smithi144.stdout:Package ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:23:45.913 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:45.914 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:23:45.914 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:45.980 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T12:23:46.390 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:52 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:46.520 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:46.521 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:23:46.521 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 83 k 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 83 k 2024-09-18T12:23:46.522 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 375 k 2024-09-18T12:23:46.523 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:23:46.557 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:56 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:46.693 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 3.4 M 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 24 k 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 15 k 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T12:23:46.694 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout:Installing weak dependencies: 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout:Install 7 Packages 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:46.695 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 3.7 M 2024-09-18T12:23:46.696 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 80 M 2024-09-18T12:23:46.696 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:23:46.717 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm-17.2.7-1645.g7061ea6c.el9.noarch.rpm 424 kB/s | 83 kB 00:00 2024-09-18T12:23:46.718 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:46.718 INFO:teuthology.orchestra.run.smithi116.stdout:Total 420 kB/s | 83 kB 00:00 2024-09-18T12:23:46.718 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:23:46.723 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:23:46.724 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:23:46.725 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:23:46.726 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:23:46.806 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:23:47.181 INFO:teuthology.orchestra.run.smithi144.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1645.g7061 100 kB/s | 15 kB 00:00 2024-09-18T12:23:47.206 INFO:teuthology.orchestra.run.smithi144.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1645.g706 142 kB/s | 24 kB 00:00 2024-09-18T12:23:47.234 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:47.248 INFO:teuthology.orchestra.run.smithi144.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-18T12:23:47.282 INFO:teuthology.orchestra.run.smithi144.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 964 kB/s | 31 kB 00:00 2024-09-18T12:23:47.307 INFO:teuthology.orchestra.run.smithi144.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 405 kB/s | 41 kB 00:00 2024-09-18T12:23:47.333 INFO:teuthology.orchestra.run.smithi144.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-18T12:23:47.339 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:47.416 INFO:teuthology.orchestra.run.smithi144.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1645.g7061ea6c 8.8 MB/s | 3.4 MB 00:00 2024-09-18T12:23:47.420 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:47.420 INFO:teuthology.orchestra.run.smithi144.stdout:Total 5.0 MB/s | 3.7 MB 00:00 2024-09-18T12:23:47.420 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:23:47.428 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:23:47.428 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:23:47.518 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:23:47.519 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:23:47.685 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:47.822 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:23:47.892 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-18T12:23:47.927 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-18T12:23:47.927 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:23:47.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:47.927 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:23:47.928 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:47.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:47.928 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:47.981 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-18T12:23:48.013 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-18T12:23:48.042 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 5/7 2024-09-18T12:23:48.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T12:23:48.683 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:48.815 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:48.816 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:48.816 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-09-18T12:23:48.816 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:48.816 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:23:48.816 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 137 k 2024-09-18T12:23:48.816 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:48.816 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:23:48.817 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:48.817 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:23:48.817 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:48.817 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 137 k 2024-09-18T12:23:48.817 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 379 k 2024-09-18T12:23:48.817 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:23:49.025 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-immutable-object-cache-17.2.7-1645.g7061ea 661 kB/s | 137 kB 00:00 2024-09-18T12:23:49.026 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:49.026 INFO:teuthology.orchestra.run.smithi116.stdout:Total 657 kB/s | 137 kB 00:00 2024-09-18T12:23:49.026 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:23:49.033 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:23:49.033 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:23:49.051 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 6/7 2024-09-18T12:23:49.087 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T12:23:49.089 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:23:49.089 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:23:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:23:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T12:23:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T12:23:49.325 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:23:49.325 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T12:23:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T12:23:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 1/7 2024-09-18T12:23:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 2/7 2024-09-18T12:23:50.106 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 3/7 2024-09-18T12:23:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-18T12:23:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-18T12:23:50.107 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-18T12:23:50.317 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T12:23:50.317 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:50.317 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:23:50.317 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:23:50.317 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:50.317 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:50.389 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-18T12:23:50.389 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:50.390 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:23:50.535 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr 2024-09-18T12:23:50.621 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T12:23:51.127 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:51.205 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:01 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:23:51.217 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:23:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:23:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:51.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T12:23:51.337 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 7.4 M 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T12:23:51.338 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:23:51.339 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-09-18T12:23:51.340 INFO:teuthology.orchestra.run.smithi144.stdout:Install 12 Packages 2024-09-18T12:23:51.340 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:23:51.340 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 38 M 2024-09-18T12:23:51.340 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 212 M 2024-09-18T12:23:51.340 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:23:51.716 INFO:teuthology.orchestra.run.smithi144.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 977 kB/s | 191 kB 00:00 2024-09-18T12:23:51.749 INFO:teuthology.orchestra.run.smithi144.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.4 MB/s | 801 kB 00:00 2024-09-18T12:23:51.783 INFO:teuthology.orchestra.run.smithi144.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-09-18T12:23:51.905 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:51.916 INFO:teuthology.orchestra.run.smithi144.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 18 MB/s | 3.0 MB 00:00 2024-09-18T12:23:51.933 INFO:teuthology.orchestra.run.smithi144.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 121 kB/s | 18 kB 00:00 2024-09-18T12:23:52.026 INFO:teuthology.orchestra.run.smithi144.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-18T12:23:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:52.036 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:23:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:23:52.036 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 3.4 M 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 24 k 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 15 k 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:23:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:23:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:Install 7 Packages 2024-09-18T12:23:52.038 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.7 M 2024-09-18T12:23:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 80 M 2024-09-18T12:23:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:23:52.042 INFO:teuthology.orchestra.run.smithi144.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 307 kB/s | 38 kB 00:00 2024-09-18T12:23:52.193 INFO:teuthology.orchestra.run.smithi144.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2024-09-18T12:23:52.268 INFO:teuthology.orchestra.run.smithi144.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-18T12:23:52.293 INFO:teuthology.orchestra.run.smithi144.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 916 kB/s | 245 kB 00:00 2024-09-18T12:23:52.327 INFO:teuthology.orchestra.run.smithi144.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-18T12:23:52.581 INFO:teuthology.orchestra.run.smithi116.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1645.g7061 87 kB/s | 15 kB 00:00 2024-09-18T12:23:52.595 INFO:teuthology.orchestra.run.smithi116.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1645.g706 131 kB/s | 24 kB 00:00 2024-09-18T12:23:52.610 INFO:teuthology.orchestra.run.smithi144.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-18T12:23:52.611 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:52.611 INFO:teuthology.orchestra.run.smithi144.stdout:Total 30 MB/s | 38 MB 00:01 2024-09-18T12:23:52.645 INFO:teuthology.orchestra.run.smithi116.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-18T12:23:52.662 INFO:teuthology.orchestra.run.smithi116.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-18T12:23:52.679 INFO:teuthology.orchestra.run.smithi116.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 488 kB/s | 41 kB 00:00 2024-09-18T12:23:52.705 INFO:teuthology.orchestra.run.smithi116.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-18T12:23:52.780 INFO:teuthology.orchestra.run.smithi116.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1645.g7061ea6c 9.1 MB/s | 3.4 MB 00:00 2024-09-18T12:23:52.785 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:23:52.785 INFO:teuthology.orchestra.run.smithi116.stdout:Total 4.9 MB/s | 3.7 MB 00:00 2024-09-18T12:23:52.785 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:23:52.791 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:23:52.792 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:23:52.875 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:23:52.876 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:23:52.966 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:23:52.992 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:23:52.992 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:23:53.197 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:23:53.264 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-18T12:23:53.306 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-18T12:23:53.362 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-18T12:23:53.396 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-18T12:23:53.431 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 5/7 2024-09-18T12:23:53.495 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:23:53.495 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:23:54.297 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:23:54.362 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T12:23:54.400 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T12:23:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 6/7 2024-09-18T12:23:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T12:23:54.508 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T12:23:54.543 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T12:23:54.730 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T12:23:54.842 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T12:23:54.918 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T12:23:55.574 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T12:23:55.582 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T12:23:55.582 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 1/7 2024-09-18T12:23:55.582 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 2/7 2024-09-18T12:23:55.582 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 3/7 2024-09-18T12:23:55.582 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-18T12:23:55.582 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-18T12:23:55.582 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-18T12:23:55.689 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T12:23:55.865 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-18T12:23:55.865 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:55.865 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:55.866 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:23:56.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T12:23:56.669 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:02 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:23:56.801 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 7.4 M 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T12:23:56.803 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-09-18T12:23:56.804 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:23:56.805 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 38 M 2024-09-18T12:23:56.805 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 212 M 2024-09-18T12:23:56.805 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:23:56.956 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T12:23:57.554 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 669 kB/s | 191 kB 00:00 2024-09-18T12:23:57.646 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.1 MB/s | 801 kB 00:00 2024-09-18T12:23:57.671 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-18T12:23:57.764 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-18T12:23:57.822 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 121 kB/s | 18 kB 00:00 2024-09-18T12:23:57.914 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 257 kB/s | 38 kB 00:00 2024-09-18T12:23:57.982 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T12:23:58.017 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T12:23:58.064 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-18T12:23:58.656 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:01 2024-09-18T12:23:58.765 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-18T12:23:59.332 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.3 MB/s | 4.9 MB 00:01 2024-09-18T12:23:59.615 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.3 MB/s | 5.1 MB 00:01 2024-09-18T12:24:00.473 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 1/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T12:24:00.474 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T12:24:00.900 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T12:24:00.900 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:00.900 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:00.900 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:00.900 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T12:24:00.900 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:00.901 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:01.226 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-rook 2024-09-18T12:24:01.818 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:11 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:01.951 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:01.952 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 47 k 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T12:24:01.953 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout:Install 13 Packages 2024-09-18T12:24:01.954 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:01.955 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 2.2 M 2024-09-18T12:24:01.955 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 26 M 2024-09-18T12:24:01.955 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:02.175 INFO:teuthology.orchestra.run.smithi144.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 285 kB/s | 26 kB 00:00 2024-09-18T12:24:02.200 INFO:teuthology.orchestra.run.smithi144.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 167 kB/s | 19 kB 00:00 2024-09-18T12:24:02.292 INFO:teuthology.orchestra.run.smithi144.stdout:(3/13): ceph-mgr-rook-17.2.7-1645.g7061ea6c.el9 229 kB/s | 47 kB 00:00 2024-09-18T12:24:02.317 INFO:teuthology.orchestra.run.smithi144.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-18T12:24:02.343 INFO:teuthology.orchestra.run.smithi144.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-18T12:24:02.368 INFO:teuthology.orchestra.run.smithi144.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-18T12:24:02.410 INFO:teuthology.orchestra.run.smithi144.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-18T12:24:02.435 INFO:teuthology.orchestra.run.smithi144.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 234 kB/s | 14 kB 00:00 2024-09-18T12:24:02.477 INFO:teuthology.orchestra.run.smithi144.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.3 MB/s | 220 kB 00:00 2024-09-18T12:24:02.502 INFO:teuthology.orchestra.run.smithi144.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-18T12:24:02.536 INFO:teuthology.orchestra.run.smithi144.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 10 MB/s | 1.0 MB 00:00 2024-09-18T12:24:02.561 INFO:teuthology.orchestra.run.smithi144.stdout:(12/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.0 MB/s | 279 kB 00:00 2024-09-18T12:24:02.586 INFO:teuthology.orchestra.run.smithi144.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-09-18T12:24:02.588 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:02.588 INFO:teuthology.orchestra.run.smithi144.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-18T12:24:02.615 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.2 MB/s | 16 MB 00:03 2024-09-18T12:24:02.616 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:02.616 INFO:teuthology.orchestra.run.smithi116.stdout:Total 6.6 MB/s | 38 MB 00:05 2024-09-18T12:24:02.652 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:02.667 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:02.667 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:02.757 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:02.977 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:03.003 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:03.003 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:03.274 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:03.454 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T12:24:03.509 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:03.510 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:03.521 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T12:24:03.585 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T12:24:03.647 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T12:24:03.706 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T12:24:03.817 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T12:24:03.926 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T12:24:03.983 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T12:24:04.265 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:04.331 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T12:24:04.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T12:24:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T12:24:04.506 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T12:24:04.700 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T12:24:04.713 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T12:24:04.729 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T12:24:04.763 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T12:24:04.796 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T12:24:04.805 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T12:24:04.842 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T12:24:04.866 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T12:24:05.471 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T12:24:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T12:24:05.630 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T12:24:05.630 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 1/13 2024-09-18T12:24:05.630 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T12:24:05.630 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T12:24:05.630 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T12:24:05.631 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T12:24:05.966 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T12:24:05.967 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T12:24:05.967 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T12:24:05.967 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T12:24:05.967 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T12:24:05.967 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T12:24:05.967 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:05.967 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:06.231 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T12:24:06.830 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:16 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:06.841 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T12:24:06.963 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 124 k 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T12:24:06.964 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T12:24:06.965 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T12:24:06.965 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T12:24:06.965 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:06.965 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:06.965 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:06.965 INFO:teuthology.orchestra.run.smithi144.stdout:Install 6 Packages 2024-09-18T12:24:06.965 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:06.966 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 7.0 M 2024-09-18T12:24:06.966 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 33 M 2024-09-18T12:24:06.966 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:07.398 INFO:teuthology.orchestra.run.smithi144.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-09-18T12:24:07.440 INFO:teuthology.orchestra.run.smithi144.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1645.g7061ea6c.e 623 kB/s | 124 kB 00:00 2024-09-18T12:24:07.474 INFO:teuthology.orchestra.run.smithi144.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-18T12:24:07.503 INFO:teuthology.orchestra.run.smithi144.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-18T12:24:07.525 INFO:teuthology.orchestra.run.smithi144.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 688 kB/s | 58 kB 00:00 2024-09-18T12:24:07.885 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T12:24:07.921 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T12:24:08.725 INFO:teuthology.orchestra.run.smithi144.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.0 MB/s | 6.0 MB 00:01 2024-09-18T12:24:08.728 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:08.728 INFO:teuthology.orchestra.run.smithi144.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-09-18T12:24:08.802 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:08.814 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:08.814 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:08.911 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:08.912 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:09.219 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:09.374 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T12:24:09.426 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T12:24:09.947 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T12:24:10.045 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T12:24:10.101 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T12:24:10.129 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 1/12 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T12:24:10.554 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T12:24:10.555 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T12:24:10.555 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T12:24:10.555 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T12:24:10.555 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T12:24:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T12:24:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/6 2024-09-18T12:24:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T12:24:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T12:24:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T12:24:10.921 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T12:24:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T12:24:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T12:24:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T12:24:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T12:24:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:10.983 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T12:24:11.185 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T12:24:11.189 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T12:24:11.189 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:11.189 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:11.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-rook 2024-09-18T12:24:11.447 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-fuse 2024-09-18T12:24:11.959 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:17 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:12.028 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:22 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:12.097 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:12.098 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:12.098 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:24:12.098 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 47 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T12:24:12.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:12.100 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.2 M 2024-09-18T12:24:12.101 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2024-09-18T12:24:12.101 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:12.161 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:12.161 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:24:12.161 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 813 k 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:Install 2 Packages 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 893 k 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 2.6 M 2024-09-18T12:24:12.162 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:12.372 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 267 kB/s | 19 kB 00:00 2024-09-18T12:24:12.397 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 264 kB/s | 26 kB 00:00 2024-09-18T12:24:12.447 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-18T12:24:12.468 INFO:teuthology.orchestra.run.smithi144.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 447 kB/s | 80 kB 00:00 2024-09-18T12:24:12.481 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): ceph-mgr-rook-17.2.7-1645.g7061ea6c.el9 262 kB/s | 47 kB 00:00 2024-09-18T12:24:12.506 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-18T12:24:12.531 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-09-18T12:24:12.534 INFO:teuthology.orchestra.run.smithi144.stdout:(2/2): ceph-fuse-17.2.7-1645.g7061ea6c.el9.x86_ 3.2 MB/s | 813 kB 00:00 2024-09-18T12:24:12.535 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:12.535 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.3 MB/s | 893 kB 00:00 2024-09-18T12:24:12.547 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:12.570 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:12.570 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 581 kB/s | 54 kB 00:00 2024-09-18T12:24:12.598 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-18T12:24:12.632 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 150 kB/s | 14 kB 00:00 2024-09-18T12:24:12.650 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:12.651 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:12.665 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-18T12:24:12.699 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-18T12:24:12.725 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-09-18T12:24:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.3 MB/s | 1.0 MB 00:00 2024-09-18T12:24:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:Total 3.3 MB/s | 2.2 MB 00:00 2024-09-18T12:24:12.835 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:12.844 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:12.850 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:12.850 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:12.939 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:12.941 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:12.979 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T12:24:13.041 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T12:24:13.442 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:13.604 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T12:24:13.667 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T12:24:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T12:24:13.706 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/2 2024-09-18T12:24:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T12:24:13.769 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T12:24:13.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T12:24:13.911 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T12:24:13.979 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T12:24:13.979 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:13.979 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:13.979 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T12:24:13.979 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:13.979 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T12:24:14.070 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T12:24:14.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-volume 2024-09-18T12:24:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T12:24:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T12:24:14.847 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:24 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T12:24:14.922 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T12:24:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T12:24:14.979 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 270 k 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:14.980 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:14.981 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:24:14.981 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:14.981 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 270 k 2024-09-18T12:24:14.981 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 1.3 M 2024-09-18T12:24:14.981 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:15.189 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-volume-17.2.7-1645.g7061ea6c.el9.noarch.rp 1.3 MB/s | 270 kB 00:00 2024-09-18T12:24:15.189 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:15.190 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.3 MB/s | 270 kB 00:00 2024-09-18T12:24:15.190 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:15.198 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:15.198 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:15.228 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:15.228 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:15.437 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:15.508 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:24:15.524 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:24:15.524 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:24:15.524 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T12:24:15.524 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T12:24:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 1/13 2024-09-18T12:24:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T12:24:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T12:24:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T12:24:16.079 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T12:24:16.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:16.079 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:16.079 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T12:24:16.080 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:16.081 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:16.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T12:24:16.527 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:24:16.528 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:16.528 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:16.528 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:16.528 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:16.528 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:16.858 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados-devel 2024-09-18T12:24:16.964 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:22 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:17.097 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 124 k 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T12:24:17.098 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T12:24:17.099 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T12:24:17.099 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:17.099 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:17.099 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:17.099 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-09-18T12:24:17.099 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:17.099 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 7.0 M 2024-09-18T12:24:17.100 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 33 M 2024-09-18T12:24:17.100 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:17.368 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-18T12:24:17.394 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1645.g7061ea6c.e 564 kB/s | 124 kB 00:00 2024-09-18T12:24:17.449 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:27 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:17.461 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 864 kB/s | 58 kB 00:00 2024-09-18T12:24:17.495 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-18T12:24:17.528 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-09-18T12:24:17.584 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:17.584 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 126 k 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 126 k 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 448 k 2024-09-18T12:24:17.585 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:17.777 INFO:teuthology.orchestra.run.smithi144.stdout:librados-devel-17.2.7-1645.g7061ea6c.el9.x86_64 658 kB/s | 126 kB 00:00 2024-09-18T12:24:17.778 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:17.778 INFO:teuthology.orchestra.run.smithi144.stdout:Total 653 kB/s | 126 kB 00:00 2024-09-18T12:24:17.778 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:17.783 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:17.783 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:17.847 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:17.847 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:18.000 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:18.092 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:18.554 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:18.645 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.1 MB/s | 6.0 MB 00:01 2024-09-18T12:24:18.648 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:18.648 INFO:teuthology.orchestra.run.smithi116.stdout:Total 4.5 MB/s | 7.0 MB 00:01 2024-09-18T12:24:18.720 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:18.731 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:18.732 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:18.794 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:18.794 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:18.794 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:18.794 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:18.794 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:18.794 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:18.830 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:18.831 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:19.055 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install libcephfs2 2024-09-18T12:24:19.140 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:19.303 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T12:24:19.365 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T12:24:19.642 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:29 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:19.731 INFO:teuthology.orchestra.run.smithi144.stdout:Package libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:19.774 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:19.775 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:24:19.776 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:19.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install libcephfs-devel 2024-09-18T12:24:19.885 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T12:24:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T12:24:20.030 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T12:24:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T12:24:20.455 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:30 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:20.587 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:24:20.588 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:20.589 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 25 k 2024-09-18T12:24:20.589 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 110 k 2024-09-18T12:24:20.589 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:20.731 INFO:teuthology.orchestra.run.smithi144.stdout:libcephfs-devel-17.2.7-1645.g7061ea6c.el9.x86_6 175 kB/s | 25 kB 00:00 2024-09-18T12:24:20.733 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:20.733 INFO:teuthology.orchestra.run.smithi144.stdout:Total 173 kB/s | 25 kB 00:00 2024-09-18T12:24:20.733 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:20.736 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:20.736 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:20.737 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:20.737 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:20.813 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T12:24:20.813 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/6 2024-09-18T12:24:20.813 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T12:24:20.813 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T12:24:20.813 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T12:24:20.813 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T12:24:20.821 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:20.919 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T12:24:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:21.086 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T12:24:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T12:24:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T12:24:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T12:24:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T12:24:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:21.087 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:21.293 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:21.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-fuse 2024-09-18T12:24:21.524 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:21.524 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:21.524 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:21.524 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:21.524 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:21.524 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:21.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados2 2024-09-18T12:24:21.916 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:27 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 813 k 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-18T12:24:22.052 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 893 k 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 2.6 M 2024-09-18T12:24:22.053 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:22.221 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 535 kB/s | 80 kB 00:00 2024-09-18T12:24:22.287 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:32 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:22.363 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): ceph-fuse-17.2.7-1645.g7061ea6c.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-18T12:24:22.364 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:22.364 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.8 MB/s | 893 kB 00:00 2024-09-18T12:24:22.375 INFO:teuthology.orchestra.run.smithi144.stdout:Package librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:22.376 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:22.397 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:22.397 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:22.418 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:22.418 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:24:22.419 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:22.480 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:22.481 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:22.498 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librbd1 2024-09-18T12:24:22.664 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:22.813 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T12:24:22.879 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T12:24:23.077 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:23.165 INFO:teuthology.orchestra.run.smithi144.stdout:Package librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:23.207 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:23.208 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:24:23.208 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:23.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rados 2024-09-18T12:24:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T12:24:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/2 2024-09-18T12:24:23.864 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:23.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T12:24:23.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:23.873 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:23.873 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T12:24:23.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:23.873 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:23.955 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:23.997 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:23.997 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:24:23.998 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:24.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rgw 2024-09-18T12:24:24.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-volume 2024-09-18T12:24:24.651 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:24.738 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:30 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:24.741 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:24.784 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:24.785 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:24:24.785 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:24.856 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-cephfs 2024-09-18T12:24:24.875 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:24.876 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:24.876 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-18T12:24:24.876 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:24.876 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:24.876 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 270 k 2024-09-18T12:24:24.876 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:24.877 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:24.877 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:24.877 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:24:24.877 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:24.877 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 270 k 2024-09-18T12:24:24.877 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 1.3 M 2024-09-18T12:24:24.877 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:25.110 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-volume-17.2.7-1645.g7061ea6c.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-18T12:24:25.111 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:25.111 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-18T12:24:25.111 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:25.120 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:25.120 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:25.150 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:25.150 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:25.359 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:25.420 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:24:25.436 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:35 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:25.437 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:24:25.437 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:24:25.437 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T12:24:25.437 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:25.527 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:25.569 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:25.570 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:24:25.570 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:25.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rbd 2024-09-18T12:24:26.223 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:26.312 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:26.354 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:26.355 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-09-18T12:24:26.355 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:26.422 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-fuse 2024-09-18T12:24:26.469 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T12:24:26.470 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:26.470 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:26.470 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T12:24:26.470 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:26.470 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:26.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados-devel 2024-09-18T12:24:27.006 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:27.139 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:27.140 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-09-18T12:24:27.140 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:27.140 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 82 k 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 82 k 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 222 k 2024-09-18T12:24:27.141 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:27.343 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:27.476 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 126 k 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:24:27.477 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:27.478 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 126 k 2024-09-18T12:24:27.478 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 448 k 2024-09-18T12:24:27.478 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:27.526 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-fuse-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 213 kB/s | 82 kB 00:00 2024-09-18T12:24:27.527 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:27.527 INFO:teuthology.orchestra.run.smithi144.stdout:Total 212 kB/s | 82 kB 00:00 2024-09-18T12:24:27.527 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:27.536 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:27.536 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:27.596 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:27.596 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:27.657 INFO:teuthology.orchestra.run.smithi116.stdout:librados-devel-17.2.7-1645.g7061ea6c.el9.x86_64 704 kB/s | 126 kB 00:00 2024-09-18T12:24:27.658 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:27.658 INFO:teuthology.orchestra.run.smithi116.stdout:Total 699 kB/s | 126 kB 00:00 2024-09-18T12:24:27.659 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:27.663 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:27.663 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:27.726 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:27.726 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:27.743 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:27.839 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:27.880 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:27.979 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:28.217 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:28.403 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:28.484 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:28.484 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:28.484 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:28.484 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:28.484 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:28.484 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:28.669 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:28.669 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:28.670 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:28.750 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-mirror 2024-09-18T12:24:28.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs2 2024-09-18T12:24:29.331 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:29.466 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:29.467 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:29.468 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:24:29.468 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:29.468 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 2.9 M 2024-09-18T12:24:29.468 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 12 M 2024-09-18T12:24:29.468 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:29.485 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:35 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:29.575 INFO:teuthology.orchestra.run.smithi116.stdout:Package libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:29.620 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:29.621 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:24:29.621 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:29.690 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs-devel 2024-09-18T12:24:29.780 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-mirror-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 9.4 MB/s | 2.9 MB 00:00 2024-09-18T12:24:29.780 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:29.781 INFO:teuthology.orchestra.run.smithi144.stdout:Total 9.4 MB/s | 2.9 MB 00:00 2024-09-18T12:24:29.781 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:29.790 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:29.790 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:29.870 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:29.870 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:30.260 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:30.280 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:30.347 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:30.372 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:30.372 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:24:30.372 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T12:24:30.372 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T12:24:30.372 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T12:24:30.372 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:30.390 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:24:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 25 k 2024-09-18T12:24:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 110 k 2024-09-18T12:24:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:30.565 INFO:teuthology.orchestra.run.smithi116.stdout:libcephfs-devel-17.2.7-1645.g7061ea6c.el9.x86_6 145 kB/s | 25 kB 00:00 2024-09-18T12:24:30.565 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:30.565 INFO:teuthology.orchestra.run.smithi116.stdout:Total 144 kB/s | 25 kB 00:00 2024-09-18T12:24:30.566 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:30.568 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:30.568 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:30.570 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:30.570 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:30.756 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:31.107 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:31.350 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:31.350 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:31.476 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:31.477 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:31.477 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:31.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados2 2024-09-18T12:24:31.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-nbd 2024-09-18T12:24:32.101 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:32.189 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:32.233 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:32.234 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:24:32.234 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:32.305 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librbd1 2024-09-18T12:24:32.346 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 12:22:50 PM UTC. 2024-09-18T12:24:32.478 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 165 k 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-18T12:24:32.479 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-09-18T12:24:32.480 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:32.480 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 165 k 2024-09-18T12:24:32.480 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 477 k 2024-09-18T12:24:32.480 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-18T12:24:32.692 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-nbd-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 779 kB/s | 165 kB 00:00 2024-09-18T12:24:32.692 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:32.693 INFO:teuthology.orchestra.run.smithi144.stdout:Total 774 kB/s | 165 kB 00:00 2024-09-18T12:24:32.693 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-18T12:24:32.700 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-18T12:24:32.700 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-18T12:24:32.763 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-18T12:24:32.763 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-18T12:24:32.882 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:32.911 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-18T12:24:32.971 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:33.014 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:33.015 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:24:33.015 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:33.023 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:33.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rados 2024-09-18T12:24:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:33.673 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:33.726 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:33.726 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:33.726 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-18T12:24:33.726 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:33.726 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:33.727 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-18T12:24:33.759 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:33.801 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:33.802 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:24:33.802 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:33.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rgw 2024-09-18T12:24:33.975 DEBUG:teuthology.parallel:result is None 2024-09-18T12:24:34.447 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:40 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:34.537 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:34.580 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:34.581 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:24:34.581 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:34.648 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-cephfs 2024-09-18T12:24:35.228 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:41 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:35.319 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:35.361 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:35.362 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:24:35.362 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:35.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rbd 2024-09-18T12:24:36.007 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:36.096 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T12:24:36.139 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:36.140 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-09-18T12:24:36.140 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:36.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-fuse 2024-09-18T12:24:36.777 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:36.908 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 82 k 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 82 k 2024-09-18T12:24:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 222 k 2024-09-18T12:24:36.910 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:37.096 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-fuse-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 438 kB/s | 82 kB 00:00 2024-09-18T12:24:37.097 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:37.097 INFO:teuthology.orchestra.run.smithi116.stdout:Total 435 kB/s | 82 kB 00:00 2024-09-18T12:24:37.097 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:37.166 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:37.166 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:37.318 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:37.410 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:37.795 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:38.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:38.068 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:38.069 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:38.069 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:38.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2024-09-18T12:24:38.908 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:44 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:39.044 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:39.045 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:39.047 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.9 M 2024-09-18T12:24:39.047 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 12 M 2024-09-18T12:24:39.047 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-mirror-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-09-18T12:24:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:39.384 INFO:teuthology.orchestra.run.smithi116.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-18T12:24:39.384 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:39.392 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:39.393 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:39.473 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:39.473 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:39.893 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:39.950 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:39.976 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T12:24:39.976 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T12:24:39.976 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T12:24:39.976 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T12:24:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:41.041 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:41.041 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:41.041 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:41.042 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:41.042 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:41.042 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:41.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-nbd 2024-09-18T12:24:41.884 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:47 ago on Wed 18 Sep 2024 12:22:54 PM UTC. 2024-09-18T12:24:42.016 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 165 k 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 165 k 2024-09-18T12:24:42.017 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 477 k 2024-09-18T12:24:42.018 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-18T12:24:42.236 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-nbd-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 756 kB/s | 165 kB 00:00 2024-09-18T12:24:42.237 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-18T12:24:42.237 INFO:teuthology.orchestra.run.smithi116.stdout:Total 749 kB/s | 165 kB 00:00 2024-09-18T12:24:42.237 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-18T12:24:42.245 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-18T12:24:42.245 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-18T12:24:42.308 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-18T12:24:42.309 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-18T12:24:42.459 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-18T12:24:42.558 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:43.081 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:43.394 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T12:24:43.394 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:43.394 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-18T12:24:43.394 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T12:24:43.394 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:43.394 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-18T12:24:43.636 DEBUG:teuthology.parallel:result is None 2024-09-18T12:24:43.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T12:24:43.784 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T12:24:43.805 INFO:teuthology.orchestra.run.smithi116.stdout:17.2.7-1645.g7061ea6c.el9 2024-09-18T12:24:43.806 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645.g7061ea6c.el9 2024-09-18T12:24:43.806 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645.g7061ea6c is installed. 2024-09-18T12:24:43.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T12:24:43.953 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T12:24:43.979 INFO:teuthology.orchestra.run.smithi144.stdout:17.2.7-1645.g7061ea6c.el9 2024-09-18T12:24:43.980 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645.g7061ea6c.el9 2024-09-18T12:24:43.980 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645.g7061ea6c is installed. 2024-09-18T12:24:43.982 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T12:24:43.982 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:43.982 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T12:24:44.008 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:24:44.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T12:24:44.050 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T12:24:44.050 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:44.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T12:24:44.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T12:24:44.139 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:24:44.139 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T12:24:44.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T12:24:44.237 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T12:24:44.238 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:44.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T12:24:44.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T12:24:44.329 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:24:44.329 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T12:24:44.358 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T12:24:44.426 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T12:24:44.426 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:44.426 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T12:24:44.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T12:24:44.517 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:24:44.517 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T12:24:44.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T12:24:44.612 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T12:24:44.693 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T12:24:44.693 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-09-18T12:24:44.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /var/log/ceph 2024-09-18T12:24:44.722 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T12:24:44.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T12:24:44.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T12:24:44.789 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T12:24:44.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T12:24:44.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T12:24:44.865 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T12:24:44.865 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T12:24:44.865 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038', 'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7911038', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 4631, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfREDwErBvlw5jI9vDF3S4DF4e8CF4WAQ6TQsntqI3Zod1NpCed6cD2EIhPwwuhZSSYOHWA7nEvZeALyp2g19Y=', 'smithi144.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIrgMxfgkfdCf7hmaE8/o8bMZQXWVHtXJf6pMogy8/2+kNmmsUaXy5ea4zPeOQg4M5z1i4Tg5HBeiUfqxR4yTzg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_09:13:00', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T12:24:44.866 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T12:24:44.891 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T12:24:44.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T12:24:44.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T12:24:44.976 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:44.976 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T12:24:45.025 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T12:24:45.026 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-18 12:24:40.225542792 +0000 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-18 12:22:06.001738465 +0000 2024-09-18T12:24:45.081 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-18 12:22:06.001738465 +0000 2024-09-18T12:24:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-18 12:22:06.001738465 +0000 2024-09-18T12:24:45.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T12:24:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-18T12:24:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-18T12:24:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00020941 s, 2.4 MB/s 2024-09-18T12:24:45.145 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T12:24:45.202 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-09-18T12:24:45.257 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T12:24:45.257 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:45.257 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-18T12:24:45.258 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:45.258 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:45.258 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-18 12:24:40.225542792 +0000 2024-09-18T12:24:45.258 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-18 12:22:06.333733167 +0000 2024-09-18T12:24:45.258 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-18 12:22:06.333733167 +0000 2024-09-18T12:24:45.258 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-18 12:22:06.333733167 +0000 2024-09-18T12:24:45.258 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T12:24:45.321 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-18T12:24:45.321 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-18T12:24:45.321 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000270483 s, 1.9 MB/s 2024-09-18T12:24:45.322 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T12:24:45.379 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-09-18T12:24:45.434 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T12:24:45.434 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-18T12:24:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-18 12:24:40.226542776 +0000 2024-09-18T12:24:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-18 12:22:06.654728046 +0000 2024-09-18T12:24:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-18 12:22:06.654728046 +0000 2024-09-18T12:24:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-18 12:22:06.654728046 +0000 2024-09-18T12:24:45.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T12:24:45.497 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-18T12:24:45.498 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-18T12:24:45.498 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000197731 s, 2.6 MB/s 2024-09-18T12:24:45.499 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T12:24:45.556 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-09-18T12:24:45.611 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T12:24:45.611 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-18T12:24:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-18 12:24:40.226542776 +0000 2024-09-18T12:24:45.611 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-18 12:22:06.990722685 +0000 2024-09-18T12:24:45.612 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-18 12:22:06.990722685 +0000 2024-09-18T12:24:45.612 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-18 12:22:06.990722685 +0000 2024-09-18T12:24:45.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T12:24:45.674 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-18T12:24:45.675 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-18T12:24:45.675 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000203456 s, 2.5 MB/s 2024-09-18T12:24:45.676 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T12:24:45.732 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T12:24:45.732 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:24:45.733 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T12:24:45.750 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T12:24:45.750 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-18 12:24:30.616897536 +0000 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-18 12:22:06.006373571 +0000 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-18 12:22:06.006373571 +0000 2024-09-18T12:24:45.809 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-18 12:22:06.006373571 +0000 2024-09-18T12:24:45.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T12:24:45.874 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-18T12:24:45.874 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-18T12:24:45.874 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000190314 s, 2.7 MB/s 2024-09-18T12:24:45.875 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T12:24:45.932 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2024-09-18T12:24:45.988 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T12:24:45.988 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:45.988 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-18T12:24:45.988 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:45.988 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:45.988 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-18 12:24:30.616897536 +0000 2024-09-18T12:24:45.989 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-18 12:22:06.351367899 +0000 2024-09-18T12:24:45.989 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-18 12:22:06.351367899 +0000 2024-09-18T12:24:45.989 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-18 12:22:06.351367899 +0000 2024-09-18T12:24:45.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T12:24:46.052 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-18T12:24:46.052 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-18T12:24:46.052 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000240409 s, 2.1 MB/s 2024-09-18T12:24:46.053 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T12:24:46.110 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2024-09-18T12:24:46.166 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-18 12:24:30.616897536 +0000 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-18 12:22:06.671362639 +0000 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-18 12:22:06.671362639 +0000 2024-09-18T12:24:46.167 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-18 12:22:06.671362639 +0000 2024-09-18T12:24:46.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T12:24:46.230 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-18T12:24:46.230 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-18T12:24:46.230 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00020555 s, 2.5 MB/s 2024-09-18T12:24:46.231 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T12:24:46.288 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2024-09-18T12:24:46.344 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T12:24:46.344 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T12:24:46.345 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-18T12:24:46.345 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T12:24:46.345 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T12:24:46.345 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-18 12:24:30.617897519 +0000 2024-09-18T12:24:46.345 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-18 12:22:06.998357263 +0000 2024-09-18T12:24:46.345 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-18 12:22:06.998357263 +0000 2024-09-18T12:24:46.345 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-18 12:22:06.998357263 +0000 2024-09-18T12:24:46.345 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T12:24:46.409 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-18T12:24:46.409 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-18T12:24:46.409 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000208857 s, 2.5 MB/s 2024-09-18T12:24:46.411 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T12:24:46.467 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T12:24:46.467 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T12:24:46.467 INFO:tasks.ceph:Generating config... 2024-09-18T12:24:46.468 INFO:tasks.ceph:[client] debug monc = 20 2024-09-18T12:24:46.468 INFO:tasks.ceph:[client] debug ms = 1 2024-09-18T12:24:46.468 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T12:24:46.469 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-18T12:24:46.469 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-18T12:24:46.469 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-18T12:24:46.469 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-18T12:24:46.469 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-18T12:24:46.469 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-18T12:24:46.469 INFO:tasks.ceph:[global] ms type = async 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-18T12:24:46.469 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-18T12:24:46.470 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-18T12:24:46.470 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-18T12:24:46.470 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-18T12:24:46.470 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-18T12:24:46.470 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-18T12:24:46.470 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-18T12:24:46.470 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-18T12:24:46.470 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-18T12:24:46.470 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T12:24:46.470 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T12:24:46.470 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-18T12:24:46.471 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T12:24:46.472 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-18T12:24:46.472 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-18T12:24:46.472 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-18T12:24:46.472 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T12:24:46.472 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-18T12:24:46.472 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-18T12:24:46.472 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T12:24:46.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T12:24:46.519 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T12:24:46.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T12:24:46.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T12:24:46.625 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.116'), ('mon.c', '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]'), ('mon.b', '172.21.15.144')] 2024-09-18T12:24:46.625 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790],172.21.15.144', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-18T12:24:46.626 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:46.626 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T12:24:46.681 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --addv c '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]' --add b 172.21.15.144 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:24:46.764 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:24:46.764 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:24:46.764 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = octopus 2024-09-18T12:24:46.764 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2024-09-18T12:24:46.764 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 15 (octopus) 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:24:46.765 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T12:24:46.767 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T12:24:46.822 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2f704457-11dd-41ce-9561-691d67058c00... 2024-09-18T12:24:46.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T12:24:46.864 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T12:24:46.904 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-09-18T12:24:46.905 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-09-18T12:24:46.906 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.907 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790],172.21.15.144" 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: mon client directed command retry = 5 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: mon election default strategy = 1 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject delay max = 1 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject delay probability = 0.005 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject delay type = mon 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject internal delays = 0.002 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject socket failures = 2500 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: ms type = async 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:24:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-09-18T12:24:46.909 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.910 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore zero block detection = True 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 20 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 20 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 10 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-09-18T12:24:46.911 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify cached snaps = True 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify missing on start = True 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock override recovery settings = True 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd memory target = 1610612736 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd op queue = debug_random 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: osd op queue cut off = debug_random 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.912 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug monc = 10 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-09-18T12:24:46.913 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T12:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon min osdmap epochs = 25 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune interval = 2 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune min = 15 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune txsize = 2 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon scrub inject missing keys = 0.05 2024-09-18T12:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: mon scrub interval = 300 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: paxos service trim min = 5 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: debug monc = 20 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-09-18T12:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-09-18T12:24:46.917 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-09-18T12:24:46.917 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2024-09-18T12:24:46.917 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2024-09-18T12:24:46.917 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T12:24:46.917 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2024-09-18T12:24:46.917 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.917 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2024-09-18T12:24:46.918 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2024-09-18T12:24:46.919 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2024-09-18T12:24:46.920 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: mon health detail to clog = false 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = "172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790],172.21.15.144" 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: mon client directed command retry = 5 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: mon election default strategy = 1 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: ms inject delay max = 1 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: ms inject delay probability = 0.005 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: ms inject delay type = mon 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: ms inject internal delays = 0.002 2024-09-18T12:24:46.921 INFO:teuthology.orchestra.run.smithi144.stdout: ms inject socket failures = 2500 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: ms type = async 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 600 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock profile = high_recovery_ops 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2024-09-18T12:24:46.922 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2024-09-18T12:24:46.923 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: bdev async discard = True 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: bdev enable discard = True 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore block size = 96636764160 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore fsck on mount = True 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore zero block detection = True 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluefs = 20 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluestore = 20 2024-09-18T12:24:46.924 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 20 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: debug rocksdb = 10 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd full ratio = 0.9 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd nearfull ratio = 0.8 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify cached snaps = True 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify missing on start = True 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd failsafe full ratio = 0.95 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock override recovery settings = True 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd memory target = 1610612736 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd objectstore = bluestore 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd op queue = debug_random 2024-09-18T12:24:46.925 INFO:teuthology.orchestra.run.smithi144.stdout: osd op queue cut off = debug_random 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: debug monc = 10 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-18T12:24:46.926 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T12:24:46.927 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: # 1m isn't quite enough 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon_down_mkfs_grace = 2m 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_filestore_osds = false 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon min osdmap epochs = 25 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon osdmap full prune interval = 2 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon osdmap full prune min = 15 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon osdmap full prune txsize = 2 2024-09-18T12:24:46.928 INFO:teuthology.orchestra.run.smithi144.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: mon scrub inject missing keys = 0.05 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: mon scrub interval = 300 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: paxos service trim min = 5 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: debug monc = 20 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.c] 2024-09-18T12:24:46.929 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.b] 2024-09-18T12:24:46.931 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T12:24:46.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T12:24:47.020 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T12:24:47.021 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:47.021 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T12:24:47.036 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:47.036 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T12:24:47.091 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:24:47.091 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:47.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T12:24:47.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T12:24:47.165 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:24:47.165 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:24:47.221 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:24:47.221 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:24:47.221 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T12:24:47.221 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T12:24:47.256 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:24:47.257 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:24:47.312 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T12:24:47.313 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T12:24:47.313 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T12:24:47.435 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T12:24:47.437 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T12:24:47.437 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T12:24:47.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-18T12:24:47.475 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-18T12:24:47.487 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T12:24:47.487 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T12:24:47.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T12:24:47.512 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T12:24:47.512 INFO:tasks.ceph:role: osd.0 2024-09-18T12:24:47.512 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:24:47.513 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T12:24:47.584 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T12:24:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T12:24:47.585 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T12:24:48.061 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-09-18T12:24:48.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-09-18T12:24:48.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T12:24:48.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T12:24:48.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T12:24:48.283 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T12:24:48.284 INFO:tasks.ceph:role: osd.1 2024-09-18T12:24:48.284 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:24:48.284 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T12:24:48.356 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T12:24:48.356 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T12:24:48.356 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T12:24:48.356 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T12:24:48.356 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T12:24:48.356 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-09-18T12:24:48.356 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T12:24:48.357 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T12:24:48.357 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T12:24:48.357 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T12:24:48.813 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-09-18T12:24:48.815 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-09-18T12:24:48.815 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T12:24:48.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T12:24:48.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T12:24:49.024 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T12:24:49.024 INFO:tasks.ceph:role: osd.2 2024-09-18T12:24:49.024 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:24:49.024 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T12:24:49.096 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T12:24:49.097 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T12:24:49.586 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-09-18T12:24:49.588 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-09-18T12:24:49.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T12:24:49.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T12:24:49.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:24:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:49.821+0000 7f4dce71d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T12:24:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:49.826+0000 7f4dce71d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T12:24:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:49.826+0000 7f4dce71d440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T12:24:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:49.826+0000 7f4dce71d440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T12:24:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:49.826+0000 7f4dce71d440 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T12:24:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:49.828+0000 7f4dce71d440 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-18T12:24:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:50.360+0000 7f4dce71d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:51.121+0000 7f4dce71d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:51.884+0000 7f4dce71d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:52.681 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T12:24:52.710 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:24:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:52.795+0000 7f09dba10440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T12:24:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:52.795+0000 7f09dba10440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T12:24:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:52.795+0000 7f09dba10440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T12:24:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:52.795+0000 7f09dba10440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T12:24:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:52.795+0000 7f09dba10440 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T12:24:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:52.796+0000 7f09dba10440 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-18T12:24:53.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:53.321+0000 7f09dba10440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:54.089+0000 7f09dba10440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:54.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:54.855+0000 7f09dba10440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:55.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T12:24:55.679 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:24:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:55.764+0000 7fb101c12440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T12:24:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:55.764+0000 7fb101c12440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T12:24:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:55.764+0000 7fb101c12440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T12:24:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:55.764+0000 7fb101c12440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T12:24:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:55.764+0000 7fb101c12440 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T12:24:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:55.764+0000 7fb101c12440 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-18T12:24:56.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:56.289+0000 7fb101c12440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:57.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:57.055+0000 7fb101c12440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:57.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:24:57.821+0000 7fb101c12440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:24:58.621 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T12:24:58.648 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-18T12:24:58.678 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T12:24:58.678 INFO:tasks.ceph:role: osd.3 2024-09-18T12:24:58.678 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:24:58.678 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T12:24:58.754 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T12:24:58.754 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T12:24:58.755 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T12:24:59.273 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-18T12:24:59.275 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-18T12:24:59.275 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-18T12:24:59.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-18T12:24:59.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-18T12:24:59.524 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T12:24:59.524 INFO:tasks.ceph:role: osd.4 2024-09-18T12:24:59.524 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:24:59.524 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T12:24:59.597 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T12:24:59.598 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T12:25:00.049 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-18T12:25:00.051 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-18T12:25:00.052 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-18T12:25:00.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-18T12:25:00.204 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-18T12:25:00.269 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T12:25:00.269 INFO:tasks.ceph:role: osd.5 2024-09-18T12:25:00.269 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:25:00.269 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T12:25:00.343 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T12:25:00.825 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-18T12:25:00.827 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-18T12:25:00.828 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-18T12:25:00.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-18T12:25:01.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:25:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:01.094+0000 7fda1a91d440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-18T12:25:01.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:01.096+0000 7fda1a91d440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-18T12:25:01.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:01.096+0000 7fda1a91d440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T12:25:01.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:01.096+0000 7fda1a91d440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T12:25:01.098 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:01.096+0000 7fda1a91d440 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T12:25:01.105 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:01.104+0000 7fda1a91d440 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-18T12:25:01.634 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:01.633+0000 7fda1a91d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:02.395 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:02.394+0000 7fda1a91d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:03.159 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:03.157+0000 7fda1a91d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:03.955 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-18T12:25:03.984 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:25:04.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:04.068+0000 7fcbc4449440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-18T12:25:04.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:04.069+0000 7fcbc4449440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-18T12:25:04.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:04.069+0000 7fcbc4449440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T12:25:04.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:04.069+0000 7fcbc4449440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T12:25:04.070 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:04.069+0000 7fcbc4449440 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T12:25:04.071 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:04.069+0000 7fcbc4449440 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-18T12:25:04.594 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:04.592+0000 7fcbc4449440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:05.361 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:05.356+0000 7fcbc4449440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:06.125 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:06.124+0000 7fcbc4449440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:06.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T12:25:06.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:25:07.033 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:07.032+0000 7f4e96612440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-18T12:25:07.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:07.032+0000 7f4e96612440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-18T12:25:07.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:07.032+0000 7f4e96612440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T12:25:07.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:07.032+0000 7f4e96612440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T12:25:07.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:07.032+0000 7f4e96612440 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T12:25:07.034 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:07.033+0000 7f4e96612440 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-18T12:25:07.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:07.557+0000 7f4e96612440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:08.328 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:08.326+0000 7f4e96612440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:09.091 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:09.090+0000 7f4e96612440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T12:25:09.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T12:25:09.951 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T12:25:09.951 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:25:09.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T12:25:09.981 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:25:09.982 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T12:25:10.051 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:25:10.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T12:25:10.118 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:10.118 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T12:25:10.149 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:10.149 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-18T12:25:10.217 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:10.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-18T12:25:10.288 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:10.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-18T12:25:10.355 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:10.355 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-18T12:25:10.411 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T12:25:10.411 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T12:25:10.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T12:25:10.454 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCRxupmkFABMRAA7/kAXaW4LNjkG/AjVCcS0w== 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCUxupmjK5xLxAAJRUIN73c4DvTXIOBmXRMXw== 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCXxupm2/SULRAAIGPQQUiPmahd3eIDogcoig== 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCPxupmroXsGRAA6ZP/flhygPZWz0q6CYvWRg== 2024-09-18T12:25:10.455 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2024-09-18T12:25:10.456 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCdxupmR4q1BRAAuOUrZtaghj/8qNrzpHeJbw== 2024-09-18T12:25:10.456 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2024-09-18T12:25:10.456 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCgxupmpG8nBBAAw4q0JsY9Ggqz5EW1I5J+ag== 2024-09-18T12:25:10.456 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2024-09-18T12:25:10.456 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjxupmEGj6ARAAruF1zwnaIGQYiPwVLg6PVg== 2024-09-18T12:25:10.456 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-09-18T12:25:10.456 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCPxupmo1NSHBAARJ/Vw+jHHUtAF5WNglgShA== 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCRxupmkFABMRAA7/kAXaW4LNjkG/AjVCcS0w== 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCUxupmjK5xLxAAJRUIN73c4DvTXIOBmXRMXw== 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCXxupm2/SULRAAIGPQQUiPmahd3eIDogcoig== 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCPxupmroXsGRAA6ZP/flhygPZWz0q6CYvWRg== 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.3] 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCdxupmR4q1BRAAuOUrZtaghj/8qNrzpHeJbw== 2024-09-18T12:25:10.475 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.4] 2024-09-18T12:25:10.476 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCgxupmpG8nBBAAw4q0JsY9Ggqz5EW1I5J+ag== 2024-09-18T12:25:10.476 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.5] 2024-09-18T12:25:10.476 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCjxupmEGj6ARAAruF1zwnaIGQYiPwVLg6PVg== 2024-09-18T12:25:10.476 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2024-09-18T12:25:10.476 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQCPxupmo1NSHBAARJ/Vw+jHHUtAF5WNglgShA== 2024-09-18T12:25:10.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.565 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.661 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T12:25:10.663 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T12:25:10.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.711 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.809 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T12:25:10.857 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T12:25:10.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T12:25:10.907 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T12:25:10.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T12:25:10.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T12:25:11.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T12:25:11.049 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T12:25:11.113 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T12:25:11.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T12:25:11.229 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T12:25:11.255 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T12:25:11.346 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T12:25:11.371 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:25:11.374 DEBUG:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T12:25:11.427 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T12:25:11.428 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:25:11.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:25:11.430 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:25:11.430 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:25:11.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:25:11.433 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:25:11.433 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T12:25:11.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T12:25:11.469 INFO:tasks.ceph.mon.b:Started 2024-09-18T12:25:11.470 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T12:25:11.470 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T12:25:11.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T12:25:11.472 INFO:tasks.ceph.mgr.x:Started 2024-09-18T12:25:11.472 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T12:25:11.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T12:25:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.567+0000 7f0b8aa4f640 10 monclient: get_monmap_and_config 2024-09-18T12:25:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.567+0000 7f0b8aa4f640 10 monclient: build_initial_monmap 2024-09-18T12:25:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.567+0000 7f0b8aa4f640 10 monclient: monmap: 2024-09-18T12:25:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:11.568085+0000 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:11.568085+0000 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:11.569 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 Processor -- start 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 -- start start 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient: init 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): picked mon.noname-c con 0x7f0b840a0b90 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8409ef00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): picked mon.noname-a con 0x7f0b84131b70 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8406a430 con 0x7f0b84131b70 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8409f810 con 0x7f0b840a0b90 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840a6220 con 0x7f0b8409ef00 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b8aa4f640 10 monclient(hunting): authenticate will time out at 738.889823s 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b88fec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b837fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b88fec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52618/0 (socket says 172.21.15.116:52618) 2024-09-18T12:25:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b88fec640 1 -- 172.21.15.116:0/614980328 learned_addr learned my addr 172.21.15.116:0/614980328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b88fec640 10 monclient(hunting): get_auth_request con 0x7f0b84131b70 auth_method 0 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b88fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b88fec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b88fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b837fe640 10 monclient(hunting): get_auth_request con 0x7f0b8409ef00 auth_method 0 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.568+0000 7f0b837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b88fec640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 msgr2=0x7f0b84132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b88fec640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 msgr2=0x7f0b84132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 10 monclient(hunting): get_auth_request con 0x7f0b840a0b90 auth_method 0 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 msgr2=0x7f0b840694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 msgr2=0x7f0b840694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.569+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 10 monclient(hunting): get_auth_request con 0x7f0b84131b70 auth_method 0 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 10 monclient(hunting): get_auth_request con 0x7f0b8409ef00 auth_method 0 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 msgr2=0x7f0b84132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 msgr2=0x7f0b84132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 10 monclient(hunting): get_auth_request con 0x7f0b840a0b90 auth_method 0 2024-09-18T12:25:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 msgr2=0x7f0b840694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:25:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 msgr2=0x7f0b840694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:11.770+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:25:12.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 10 monclient(hunting): get_auth_request con 0x7f0b8409ef00 auth_method 0 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 10 monclient(hunting): get_auth_request con 0x7f0b84131b70 auth_method 0 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 msgr2=0x7f0b84132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 msgr2=0x7f0b84132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 10 monclient(hunting): get_auth_request con 0x7f0b840a0b90 auth_method 0 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 msgr2=0x7f0b840694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 msgr2=0x7f0b840694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.171+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:25:12.685 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:12.684+0000 7f40fdbbfec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T12:25:12.724 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:12.723+0000 7f40fdbbfec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T12:25:12.772 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:12.771+0000 7f40fdbbfec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T12:25:12.907 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:12.906+0000 7f40fdbbfec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T12:25:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 10 monclient(hunting): get_auth_request con 0x7f0b84131b70 auth_method 0 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): get_auth_request con 0x7f0b8409ef00 auth_method 0 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 10 monclient(hunting): get_auth_request con 0x7f0b840a0b90 auth_method 0 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient(hunting): handle_auth_done global_id 4103 payload 274 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 msgr2=0x7f0b840694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 msgr2=0x7f0b84132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 1 monclient: found mon.noname-b 2024-09-18T12:25:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8409ca30 con 0x7f0b8409ef00 2024-09-18T12:25:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.972+0000 7f0b88fec640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.975+0000 7f0b837fe640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.975+0000 7f0b837fe640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0b8406a690 tx=0x7f0b78008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ee5a6c072ebea2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.975+0000 7f0b82ffd640 1 -- 172.21.15.116:0/614980328 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78030480 con 0x7f0b8409ef00 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: dump: 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 1 -- 172.21.15.116:0/614980328 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7802d030 con 0x7f0b8409ef00 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 5 monclient: authenticate success, global_id 4103 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 4 set_mon_vals no callback set 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 1 -- 172.21.15.116:0/614980328 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78030830 con 0x7f0b8409ef00 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b82ffd640 10 monclient: dump: 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 10 monclient: shutdown 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0b8406a690 tx=0x7f0b78008040 comp rx=0 tx=0).stop 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/614980328 shutdown_connections 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.116:3300/0 conn(0x7f0b84131b70 0x7f0b84132010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/614980328 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b8409ef00 0x7f0b8409f2d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/614980328 >> v2:172.21.15.144:3300/0 conn(0x7f0b840a0b90 0x7f0b840694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.976+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/614980328 >> 172.21.15.116:0/614980328 conn(0x7f0b840679c0 msgr2=0x7f0b840a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.977+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/614980328 shutdown_connections 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.977+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/614980328 wait complete. 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.977+0000 7f0b8aa4f640 10 monclient: build_initial_monmap 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.977+0000 7f0b8aa4f640 10 monclient: monmap: 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.977+0000 7f0b8aa4f640 1 Processor -- start 2024-09-18T12:25:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 1 -- start start 2024-09-18T12:25:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient: init 2024-09-18T12:25:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:25:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b8409ef00 0x7f0b8409a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8409ef00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 0x7f0b84130f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): picked mon.noname-c con 0x7f0b840a0b90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b84131b70 0x7f0b8411b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): picked mon.noname-a con 0x7f0b84131b70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840a12a0 con 0x7f0b84131b70 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840a4e00 con 0x7f0b8409ef00 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b841423c0 con 0x7f0b840a0b90 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b8aa4f640 10 monclient(hunting): authenticate will time out at 740.299882s 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b837fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 0x7f0b84130f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.978+0000 7f0b837fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 0x7f0b84130f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33234/0 (socket says 172.21.15.116:33234) 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 1 -- 172.21.15.116:0/2895699768 learned_addr learned my addr 172.21.15.116:0/2895699768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): get_auth_request con 0x7f0b840a0b90 auth_method 0 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b84131b70 0x7f0b8411b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b8409ef00 0x7f0b8409a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 10 monclient(hunting): get_auth_request con 0x7f0b84131b70 auth_method 0 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 10 monclient(hunting): get_auth_request con 0x7f0b8409ef00 auth_method 0 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:12.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient(hunting): handle_auth_done global_id 4106 payload 824 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 1 -- 172.21.15.116:0/2895699768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b8409ef00 msgr2=0x7f0b8409a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b8409ef00 0x7f0b8409a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 1 -- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b84131b70 msgr2=0x7f0b8411b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b84131b70 0x7f0b8411b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 1 monclient: found mon.noname-c 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b837fe640 1 -- 172.21.15.116:0/2895699768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c009780 con 0x7f0b840a0b90 2024-09-18T12:25:12.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.979+0000 7f0b88fec640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b84131b70 0x7f0b8411b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b837fe640 1 -- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 msgr2=0x7f0b84130f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b837fe640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 0x7f0b84130f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b78030500 tx=0x7f0b78038000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 1 -- 172.21.15.116:0/2895699768 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b780307b0 con 0x7f0b840a0b90 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 10 monclient: dump: 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 10 monclient: _finish_auth 0 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 1 -- 172.21.15.116:0/2895699768 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7803d620 con 0x7f0b840a0b90 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b8aa4f640 5 monclient: authenticate success, global_id 4106 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b8aa4f640 10 monclient: _renew_subs 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b8aa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7800e040 con 0x7f0b840a0b90 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.981+0000 7f0b6bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:12.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b6bfff640 1 -- 172.21.15.116:0/2895699768 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7803d9e0 con 0x7f0b840a0b90 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b6bfff640 10 monclient: dump: 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:12.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b8aa4f640 10 monclient: _renew_subs 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b8aa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8409b090 con 0x7f0b840a0b90 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b827fc640 4 set_mon_vals no callback set 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b6bfff640 1 -- 172.21.15.116:0/2895699768 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f0b7803a070 con 0x7f0b840a0b90 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.982+0000 7f0b6bfff640 1 -- 172.21.15.116:0/2895699768 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0b78046a70 con 0x7f0b840a0b90 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.983+0000 7f0b8aa4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.983+0000 7f0b8aa4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.983+0000 7f0b8aa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.983+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8405ed70 con 0x7f0b840a0b90 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.985+0000 7f0b6bfff640 1 -- 172.21.15.116:0/2895699768 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f0b7804b060 con 0x7f0b840a0b90 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.985+0000 7f0b6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:12.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:12.985+0000 7f0b6bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:13.013 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:13.012+0000 7f40fdbbfec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T12:25:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.046+0000 7f0b8aa4f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T12:25:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.046+0000 7f0b8aa4f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T12:25:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.046+0000 7f0b8aa4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.046+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0b8409d1d0 con 0x7f0b840a0b90 2024-09-18T12:25:13.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.050+0000 7f0b6bfff640 1 -- 172.21.15.116:0/2895699768 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0b78041070 con 0x7f0b840a0b90 2024-09-18T12:25:13.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.050+0000 7f0b6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T12:25:13.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.050+0000 7f0b6bfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-18T12:25:13.051 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-09-18T12:25:13.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 10 monclient: shutdown 2024-09-18T12:25:13.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 msgr2=0x7f0b84130f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 0x7f0b84130f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b78030500 tx=0x7f0b78038000 comp rx=0 tx=0).stop 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 shutdown_connections 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b84131b70 0x7f0b8411b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b840a0b90 0x7f0b84130f40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 --2- 172.21.15.116:0/2895699768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b8409ef00 0x7f0b8409a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 >> 172.21.15.116:0/2895699768 conn(0x7f0b840679c0 msgr2=0x7f0b840a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 shutdown_connections 2024-09-18T12:25:13.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:13.051+0000 7f0b8aa4f640 1 -- 172.21.15.116:0/2895699768 wait complete. 2024-09-18T12:25:13.063 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T12:25:13.063 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:25:13.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T12:25:13.089 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:25:13.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T12:25:13.155 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-18T12:25:13.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T12:25:13.220 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:13.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-18T12:25:13.245 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:13.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-18T12:25:13.310 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:25:13.310 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-18T12:25:13.317 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:13.316+0000 7f40fdbbfec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T12:25:13.341 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new d06ed7ed-3035-4edf-bcb6-d5e47db40d9c 0 2024-09-18T12:25:13.371 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:13.370+0000 7f40fdbbfec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T12:25:13.464 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.463+0000 7f865bd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.463+0000 7f865bd9d640 10 monclient: build_initial_monmap 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.463+0000 7f865bd9d640 10 monclient: monmap: 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:25:13.464255+0000 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:25:13.464255+0000 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:13.465 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.466 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.463+0000 7f865bd9d640 1 Processor -- start 2024-09-18T12:25:13.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 1 -- start start 2024-09-18T12:25:13.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient: init 2024-09-18T12:25:13.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:13.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:25:13.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86540a0b90 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:13.476 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f865409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f865409ef00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8654136ea0 0x7f8654139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8654136ea0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865409f810 con 0x7f86540a0b90 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865406a430 con 0x7f8654136ea0 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86540a6220 con 0x7f865409ef00 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865bd9d640 10 monclient(hunting): authenticate will time out at 749.033703s 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f8659b39640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f86540a0b90 0x7f86540694d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:44344/0 (socket says 172.21.15.144:44344) 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f8659b39640 1 -- 172.21.15.144:0/3169683479 learned_addr learned my addr 172.21.15.144:0/3169683479 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:25:13.477 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 1 --2- 172.21.15.144:0/3169683479 >> v2:172.21.15.144:3300/0 conn(0x7f8654136ea0 0x7f8654139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 10 monclient(hunting): get_auth_request con 0x7f8654136ea0 auth_method 0 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f865a33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f8659338640 1 --2- 172.21.15.144:0/3169683479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f865409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f8659338640 10 monclient(hunting): get_auth_request con 0x7f865409ef00 auth_method 0 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f8659338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f8659338640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.464+0000 7f8659338640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-18T12:25:13.478 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 1 -- 172.21.15.144:0/3169683479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 msgr2=0x7f865409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 1 --2- 172.21.15.144:0/3169683479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f865409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 1 -- 172.21.15.144:0/3169683479 >> v1:172.21.15.144:6789/0 conn(0x7f86540a0b90 legacy=0x7f86540694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 10 monclient: _finish_hunting 0 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 1 monclient: found mon.noname-c 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f865a33a640 1 -- 172.21.15.144:0/3169683479 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865409ca30 con 0x7f8654136ea0 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.465+0000 7f8659338640 1 --2- 172.21.15.144:0/3169683479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f865409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f865a33a640 1 -- 172.21.15.144:0/3169683479 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 msgr2=0x7f8654139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f865a33a640 1 --2- 172.21.15.144:0/3169683479 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654139280 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f864c009e60 tx=0x7f864c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59826aba9c51ed78 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 1 -- 172.21.15.144:0/3169683479 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c007760 con 0x7f8654136ea0 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: dump: 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: _finish_auth 0 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f865bd9d640 5 monclient: authenticate success, global_id 4108 2024-09-18T12:25:13.480 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 1 -- 172.21.15.144:0/3169683479 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c02d050 con 0x7f8654136ea0 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f865bd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f865bd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f865bd9d640 4 set_mon_vals no callback set 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 1 -- 172.21.15.144:0/3169683479 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c007af0 con 0x7f8654136ea0 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.468+0000 7f8658b37640 10 monclient: dump: 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.481 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 10 monclient: shutdown 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 -- 172.21.15.144:0/3169683479 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 msgr2=0x7f8654139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 --2- 172.21.15.144:0/3169683479 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654139280 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f864c009e60 tx=0x7f864c009780 comp rx=0 tx=0).stop 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 -- 172.21.15.144:0/3169683479 shutdown_connections 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 --2- 172.21.15.144:0/3169683479 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654139280 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 --2- 172.21.15.144:0/3169683479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f865409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.482 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 -- 172.21.15.144:0/3169683479 >> 172.21.15.144:0/3169683479 conn(0x7f86540679c0 msgr2=0x7f86540a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 -- 172.21.15.144:0/3169683479 shutdown_connections 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 1 -- 172.21.15.144:0/3169683479 wait complete. 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 10 monclient: build_initial_monmap 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.469+0000 7f865bd9d640 10 monclient: monmap: 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:13.483 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 Processor -- start 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 -- start start 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient: init 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f8654140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f865409ef00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 0x7f865409b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.484 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f86540a0b90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654135150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8654136ea0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86540970b0 con 0x7f86540a0b90 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86540a12a0 con 0x7f8654136ea0 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86540a47b0 con 0x7f865409ef00 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865bd9d640 10 monclient(hunting): authenticate will time out at 749.039576s 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654135150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654135150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44164/0 (socket says 172.21.15.144:44164) 2024-09-18T12:25:13.485 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 1 -- 172.21.15.144:0/3247615938 learned_addr learned my addr 172.21.15.144:0/3247615938 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): get_auth_request con 0x7f8654136ea0 auth_method 0 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f8654140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 10 monclient(hunting): get_auth_request con 0x7f865409ef00 auth_method 0 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659338640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 0x7f865409b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.486 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659338640 1 -- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 msgr2=0x7f865409b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659338640 1 -- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 msgr2=0x7f865409b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659338640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 0x7f865409b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659338640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 0x7f865409b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f8659b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.470+0000 7f865a33a640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 1 -- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 msgr2=0x7f8654140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f8654140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 1 -- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 msgr2=0x7f865409b150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 0x7f865409b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 10 monclient: _finish_hunting 0 2024-09-18T12:25:13.487 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 1 monclient: found mon.noname-b 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.471+0000 7f865a33a640 1 -- 172.21.15.144:0/3247615938 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c009a20 con 0x7f8654136ea0 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865a33a640 1 -- 172.21.15.144:0/3247615938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 msgr2=0x7f8654135150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865a33a640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654135150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f864c0099f0 tx=0x7f864c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 1 -- 172.21.15.144:0/3247615938 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c031c10 con 0x7f8654136ea0 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: dump: 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.488 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 1 -- 172.21.15.144:0/3247615938 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c040070 con 0x7f8654136ea0 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865bd9d640 5 monclient: authenticate success, global_id 4111 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 1 -- 172.21.15.144:0/3247615938 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c045430 con 0x7f8654136ea0 2024-09-18T12:25:13.489 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 10 monclient: dump: 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865bd9d640 10 monclient: _renew_subs 2024-09-18T12:25:13.490 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8654096d60 con 0x7f8654136ea0 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f864bfff640 4 set_mon_vals no callback set 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865bd9d640 10 monclient: _renew_subs 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8654097480 con 0x7f8654136ea0 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 1 -- 172.21.15.144:0/3247615938 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f864c042030 con 0x7f8654136ea0 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.473+0000 7f8649ffb640 1 -- 172.21.15.144:0/3247615938 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f864c045b60 con 0x7f8654136ea0 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.474+0000 7f865bd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.474+0000 7f865bd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.474+0000 7f865bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.491 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.474+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8654097670 con 0x7f8654136ea0 2024-09-18T12:25:13.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.477+0000 7f8649ffb640 1 -- 172.21.15.144:0/3247615938 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f864c04e440 con 0x7f8654136ea0 2024-09-18T12:25:13.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.477+0000 7f8649ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:13.492 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.477+0000 7f8649ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:13.511 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:13.510+0000 7f40fdbbfec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T12:25:13.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.543+0000 7f865bd9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d06ed7ed-3035-4edf-bcb6-d5e47db40d9c", "id": 0}] 2024-09-18T12:25:13.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.543+0000 7f865bd9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d06ed7ed-3035-4edf-bcb6-d5e47db40d9c", "id": 0}] 2024-09-18T12:25:13.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.543+0000 7f865bd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.545 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.543+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d06ed7ed-3035-4edf-bcb6-d5e47db40d9c", "id": 0} v 0) v1 -- 0x7f865405ed70 con 0x7f8654136ea0 2024-09-18T12:25:13.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.553+0000 7f8649ffb640 1 -- 172.21.15.144:0/3247615938 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d06ed7ed-3035-4edf-bcb6-d5e47db40d9c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f864c04e5e0 con 0x7f8654136ea0 2024-09-18T12:25:13.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.553+0000 7f8649ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d06ed7ed-3035-4edf-bcb6-d5e47db40d9c", "id": 0}] 2024-09-18T12:25:13.554 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.553+0000 7f8649ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:13.554 INFO:teuthology.orchestra.run.smithi144.stdout:0 2024-09-18T12:25:13.555 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 10 monclient: shutdown 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 msgr2=0x7f8654135150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654135150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f864c0099f0 tx=0x7f864c037000 comp rx=0 tx=0).stop 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 shutdown_connections 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8654136ea0 0x7f8654135150 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86540a0b90 0x7f865409b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 --2- 172.21.15.144:0/3247615938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f865409ef00 0x7f8654140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 >> 172.21.15.144:0/3247615938 conn(0x7f86540679c0 msgr2=0x7f865409dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 shutdown_connections 2024-09-18T12:25:13.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.554+0000 7f865bd9d640 1 -- 172.21.15.144:0/3247615938 wait complete. 2024-09-18T12:25:13.569 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 93e7e90b-2e87-4214-be44-eee121567700 1 2024-09-18T12:25:13.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.661+0000 7fb84175d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:13.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.661+0000 7fb84175d640 10 monclient: build_initial_monmap 2024-09-18T12:25:13.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.661+0000 7fb84175d640 10 monclient: monmap: 2024-09-18T12:25:13.662 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:13.662 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.662 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:25:13.662087+0000 2024-09-18T12:25:13.662 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:25:13.662087+0000 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:13.663 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.661+0000 7fb84175d640 1 Processor -- start 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.661+0000 7fb84175d640 1 -- start start 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.661+0000 7fb84175d640 10 monclient: init 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb83c0971a0 0x7fb83c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient(hunting): picked mon.noname-a con 0x7fb83c0971a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb83c098c20 0x7fb83c06f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient(hunting): picked mon.noname-c con 0x7fb83c098c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb83c09c820 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:13.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c097570 con 0x7fb83c09c820 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c0977e0 con 0x7fb83c0971a0 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c097ef0 con 0x7fb83c098c20 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb84175d640 10 monclient(hunting): authenticate will time out at 749.231321s 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb83c098c20 0x7fb83c06f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb83c098c20 0x7fb83c06f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44166/0 (socket says 172.21.15.144:44166) 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 -- 172.21.15.144:0/3058870183 learned_addr learned my addr 172.21.15.144:0/3058870183 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): get_auth_request con 0x7fb83c098c20 auth_method 0 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83affd640 1 --2- 172.21.15.144:0/3058870183 >> v2:172.21.15.116:3300/0 conn(0x7fb83c0971a0 0x7fb83c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 -- 172.21.15.144:0/3058870183 >> v2:172.21.15.116:3300/0 conn(0x7fb83c0971a0 msgr2=0x7fb83c06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 --2- 172.21.15.144:0/3058870183 >> v2:172.21.15.116:3300/0 conn(0x7fb83c0971a0 0x7fb83c06ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 -- 172.21.15.144:0/3058870183 >> v1:172.21.15.116:6789/0 conn(0x7fb83c09c820 legacy=0x7fb83c09cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 monclient: found mon.noname-c 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83a7fc640 1 -- 172.21.15.144:0/3058870183 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0981d0 con 0x7fb83c098c20 2024-09-18T12:25:13.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83affd640 10 monclient: get_auth_request con 0x7fb83c0971a0 auth_method 0 2024-09-18T12:25:13.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.662+0000 7fb83affd640 1 --2- 172.21.15.144:0/3058870183 >> v2:172.21.15.116:3300/0 conn(0x7fb83c0971a0 0x7fb83c06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.665+0000 7fb83a7fc640 1 -- 172.21.15.144:0/3058870183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c098c20 msgr2=0x7fb83c06f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.665+0000 7fb83a7fc640 1 --2- 172.21.15.144:0/3058870183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c098c20 0x7fb83c06f170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb83c076530 tx=0x7fb82c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a320e4067c33f1e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 1 -- 172.21.15.144:0/3058870183 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c032440 con 0x7fb83c098c20 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: dump: 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.674 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 1 -- 172.21.15.144:0/3058870183 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c0325a0 con 0x7fb83c098c20 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 5 monclient: authenticate success, global_id 4114 2024-09-18T12:25:13.675 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 1 -- 172.21.15.144:0/3058870183 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c032940 con 0x7fb83c098c20 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 4 set_mon_vals no callback set 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb839ffb640 10 monclient: dump: 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.676 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 10 monclient: shutdown 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 -- 172.21.15.144:0/3058870183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c098c20 msgr2=0x7fb83c06f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 --2- 172.21.15.144:0/3058870183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c098c20 0x7fb83c06f170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb83c076530 tx=0x7fb82c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 -- 172.21.15.144:0/3058870183 shutdown_connections 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 --2- 172.21.15.144:0/3058870183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c098c20 0x7fb83c06f170 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 --2- 172.21.15.144:0/3058870183 >> v2:172.21.15.116:3300/0 conn(0x7fb83c0971a0 0x7fb83c06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 -- 172.21.15.144:0/3058870183 >> 172.21.15.144:0/3058870183 conn(0x7fb83c0694c0 msgr2=0x7fb83c0a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 -- 172.21.15.144:0/3058870183 shutdown_connections 2024-09-18T12:25:13.677 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.666+0000 7fb84175d640 1 -- 172.21.15.144:0/3058870183 wait complete. 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.667+0000 7fb84175d640 10 monclient: build_initial_monmap 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.667+0000 7fb84175d640 10 monclient: monmap: 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.667+0000 7fb84175d640 1 Processor -- start 2024-09-18T12:25:13.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.667+0000 7fb84175d640 1 -- start start 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.667+0000 7fb84175d640 10 monclient: init 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83c0971a0 0x7fb83c067110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): picked mon.noname-c con 0x7fb83c0971a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb83c098c20 0x7fb83c12ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): picked mon.noname-a con 0x7fb83c098c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 0x7fb83c12ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): picked mon.noname-b con 0x7fb83c09c820 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c11f110 con 0x7fb83c098c20 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c06e980 con 0x7fb83c09c820 2024-09-18T12:25:13.679 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c1417a0 con 0x7fb83c0971a0 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb84175d640 10 monclient(hunting): authenticate will time out at 749.237392s 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 0x7fb83c12ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 0x7fb83c12ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44176/0 (socket says 172.21.15.144:44176) 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 -- 172.21.15.144:0/4057150663 learned_addr learned my addr 172.21.15.144:0/4057150663 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): get_auth_request con 0x7fb83c09c820 auth_method 0 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:13.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb83c098c20 0x7fb83c12ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83c0971a0 0x7fb83c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 10 monclient(hunting): get_auth_request con 0x7fb83c098c20 auth_method 0 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 10 monclient(hunting): get_auth_request con 0x7fb83c0971a0 auth_method 0 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:13.681 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 -- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83c0971a0 msgr2=0x7fb83c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83c0971a0 0x7fb83c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 -- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb83c098c20 msgr2=0x7fb83c12ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb83c098c20 0x7fb83c12ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.668+0000 7fb83b7fe640 1 -- 172.21.15.144:0/4057150663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c009450 con 0x7fb83c09c820 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb83b7fe640 1 -- 172.21.15.144:0/4057150663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 msgr2=0x7fb83c12ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb83b7fe640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 0x7fb83c12ef40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c890 tx=0x7fb834002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 1 -- 172.21.15.144:0/4057150663 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83400ce80 con 0x7fb83c09c820 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: dump: 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.683 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 1 -- 172.21.15.144:0/4057150663 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83400aa00 con 0x7fb83c09c820 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb84175d640 5 monclient: authenticate success, global_id 4117 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb84175d640 10 monclient: _renew_subs 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb84175d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c0679a0 con 0x7fb83c09c820 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 1 -- 172.21.15.144:0/4057150663 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83400adc0 con 0x7fb83c09c820 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:13.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 10 monclient: dump: 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb84175d640 10 monclient: _renew_subs 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb84175d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.685 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c11ec00 con 0x7fb83c09c820 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb8397fa640 4 set_mon_vals no callback set 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 1 -- 172.21.15.144:0/4057150663 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fb834012630 con 0x7fb83c09c820 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.671+0000 7fb822ffd640 1 -- 172.21.15.144:0/4057150663 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fb83401b7c0 con 0x7fb83c09c820 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.672+0000 7fb84175d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.672+0000 7fb84175d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.672+0000 7fb84175d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.672+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c11f2c0 con 0x7fb83c09c820 2024-09-18T12:25:13.686 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.674+0000 7fb822ffd640 1 -- 172.21.15.144:0/4057150663 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7fb83403a990 con 0x7fb83c09c820 2024-09-18T12:25:13.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.674+0000 7fb822ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:13.687 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.674+0000 7fb822ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:13.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.741+0000 7fb84175d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "93e7e90b-2e87-4214-be44-eee121567700", "id": 1}] 2024-09-18T12:25:13.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.741+0000 7fb84175d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "93e7e90b-2e87-4214-be44-eee121567700", "id": 1}] 2024-09-18T12:25:13.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.741+0000 7fb84175d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:13.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:13.741+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "93e7e90b-2e87-4214-be44-eee121567700", "id": 1} v 0) v1 -- 0x7fb83c05ed70 con 0x7fb83c09c820 2024-09-18T12:25:14.432 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:14.430+0000 7f40fdbbfec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T12:25:14.481 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:14.480+0000 7f40fdbbfec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T12:25:14.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.538+0000 7fb822ffd640 1 -- 172.21.15.144:0/4057150663 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93e7e90b-2e87-4214-be44-eee121567700", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb834029070 con 0x7fb83c09c820 2024-09-18T12:25:14.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.538+0000 7fb822ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "93e7e90b-2e87-4214-be44-eee121567700", "id": 1}] 2024-09-18T12:25:14.540 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.538+0000 7fb822ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:14.540 INFO:teuthology.orchestra.run.smithi144.stdout:1 2024-09-18T12:25:14.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 10 monclient: shutdown 2024-09-18T12:25:14.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:14.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 msgr2=0x7fb83c12ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:14.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 0x7fb83c12ef40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c890 tx=0x7fb834002640 comp rx=0 tx=0).stop 2024-09-18T12:25:14.541 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 shutdown_connections 2024-09-18T12:25:14.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb83c09c820 0x7fb83c12ef40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb83c098c20 0x7fb83c12ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 1 --2- 172.21.15.144:0/4057150663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb83c0971a0 0x7fb83c067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.539+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 >> 172.21.15.144:0/4057150663 conn(0x7fb83c0694c0 msgr2=0x7fb83c0a1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:14.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.540+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 shutdown_connections 2024-09-18T12:25:14.542 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.540+0000 7fb84175d640 1 -- 172.21.15.144:0/4057150663 wait complete. 2024-09-18T12:25:14.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 34d020ff-733c-4528-a9ff-b1ef60f8f74d 2 2024-09-18T12:25:14.645 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.644+0000 7fd123446640 10 monclient: get_monmap_and_config 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.644+0000 7fd123446640 10 monclient: build_initial_monmap 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.644+0000 7fd123446640 10 monclient: monmap: 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:25:14.645247+0000 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:25:14.645247+0000 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:14.646 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:14.647 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:14.647 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:14.647 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:14.647 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:14.647 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.644+0000 7fd123446640 1 Processor -- start 2024-09-18T12:25:14.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 1 -- start start 2024-09-18T12:25:14.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient: init 2024-09-18T12:25:14.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:14.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T12:25:14.649 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c0694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient(hunting): picked mon.noname-b con 0x7fd11c0a0b90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd11c09ef00 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd11c09f2d0 0x7fd11c132280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient(hunting): picked mon.noname-a con 0x7fd11c09f2d0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11c0a6220 con 0x7fd11c09ef00 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11c06a430 con 0x7fd11c09f2d0 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c09ca30 con 0x7fd11c0a0b90 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd123446640 10 monclient(hunting): authenticate will time out at 750.214591s 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1219e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd11c09f2d0 0x7fd11c132280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c0694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:14.650 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c0694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:57130/0 (socket says 172.21.15.144:57130) 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1219e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd11c09f2d0 0x7fd11c132280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:50962/0 (socket says 172.21.15.144:50962) 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 1 -- 172.21.15.144:0/2534177257 learned_addr learned my addr 172.21.15.144:0/2534177257 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1209e1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd11c09ef00 0x7fd11c131b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:58706/0 (socket says 172.21.15.144:58706) 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 10 monclient(hunting): get_auth_request con 0x7fd11c0a0b90 auth_method 0 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1219e3640 10 monclient(hunting): get_auth_request con 0x7fd11c09f2d0 auth_method 0 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1219e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1219e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1219e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 1 -- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 msgr2=0x7fd11c0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:25:14.651 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 1 -- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 msgr2=0x7fd11c0694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.645+0000 7fd1211e2640 1 --2- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c0694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1211e2640 1 --2- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c0694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 1 -- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 msgr2=0x7fd11c0694d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 1 --2- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c0694d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 1 -- 172.21.15.144:0/2534177257 >> v1:172.21.15.116:6789/0 conn(0x7fd11c09ef00 legacy=0x7fd11c131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 10 monclient: _finish_hunting 0 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 1 monclient: found mon.noname-a 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:14.652 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.646+0000 7fd1219e3640 1 -- 172.21.15.144:0/2534177257 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c0a0f60 con 0x7fd11c09f2d0 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd1219e3640 1 -- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 msgr2=0x7fd11c132280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd1219e3640 1 --2- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 0x7fd11c132280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd114009e60 tx=0x7fd114009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11bc0ae65a1c0cd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 1 -- 172.21.15.144:0/2534177257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114007760 con 0x7fd11c09f2d0 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: dump: 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:14.659 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 1 -- 172.21.15.144:0/2534177257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11402d050 con 0x7fd11c09f2d0 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd123446640 5 monclient: authenticate success, global_id 4113 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd123446640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd123446640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:14.660 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd123446640 4 set_mon_vals no callback set 2024-09-18T12:25:14.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 1 -- 172.21.15.144:0/2534177257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114007af0 con 0x7fd11c09f2d0 2024-09-18T12:25:14.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:14.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:14.661 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.649+0000 7fd113fff640 10 monclient: dump: 2024-09-18T12:25:14.661 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:14.661 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 10 monclient: shutdown 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 -- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 msgr2=0x7fd11c132280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 --2- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 0x7fd11c132280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd114009e60 tx=0x7fd114009780 comp rx=0 tx=0).stop 2024-09-18T12:25:14.662 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 -- 172.21.15.144:0/2534177257 shutdown_connections 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 --2- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 0x7fd11c132280 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 --2- 172.21.15.144:0/2534177257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c0694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 -- 172.21.15.144:0/2534177257 >> 172.21.15.144:0/2534177257 conn(0x7fd11c0679c0 msgr2=0x7fd11c0a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 -- 172.21.15.144:0/2534177257 shutdown_connections 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 1 -- 172.21.15.144:0/2534177257 wait complete. 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 10 monclient: build_initial_monmap 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.650+0000 7fd123446640 10 monclient: monmap: 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:14.663 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 Processor -- start 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 -- start start 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient: init 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 0x7fd11c141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): picked mon.noname-b con 0x7fd11c09ef00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 0x7fd11c133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): picked mon.noname-a con 0x7fd11c09f2d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:14.664 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c11d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): picked mon.noname-c con 0x7fd11c0a0b90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c0a12a0 con 0x7fd11c09f2d0 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c121360 con 0x7fd11c09ef00 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c121080 con 0x7fd11c0a0b90 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd123446640 10 monclient(hunting): authenticate will time out at 750.220702s 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 0x7fd11c141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 0x7fd11c141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44186/0 (socket says 172.21.15.144:44186) 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 1 -- 172.21.15.144:0/1042760653 learned_addr learned my addr 172.21.15.144:0/1042760653 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:14.665 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 10 monclient(hunting): get_auth_request con 0x7fd11c09ef00 auth_method 0 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1211e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1219e3640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c11d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1209e1640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 0x7fd11c133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1219e3640 10 monclient(hunting): get_auth_request con 0x7fd11c0a0b90 auth_method 0 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1219e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1219e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1219e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1209e1640 10 monclient(hunting): get_auth_request con 0x7fd11c09f2d0 auth_method 0 2024-09-18T12:25:14.666 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1209e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1209e1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.651+0000 7fd1209e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1219e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1219e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1219e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1209e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1209e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1209e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 1 -- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 msgr2=0x7fd11c11d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c11d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 1 -- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 msgr2=0x7fd11c133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 0x7fd11c133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 10 monclient: _finish_hunting 0 2024-09-18T12:25:14.667 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 1 monclient: found mon.noname-b 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.652+0000 7fd1211e2640 1 -- 172.21.15.144:0/1042760653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd114009a20 con 0x7fd11c09ef00 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1211e2640 1 -- 172.21.15.144:0/1042760653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 msgr2=0x7fd11c141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1211e2640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 0x7fd11c141080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd11800e870 tx=0x7fd118008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 1 -- 172.21.15.144:0/1042760653 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11800c6d0 con 0x7fd11c09ef00 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: dump: 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:14.668 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd123446640 5 monclient: authenticate success, global_id 4120 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 1 -- 172.21.15.144:0/1042760653 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11800c830 con 0x7fd11c09ef00 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd123446640 10 monclient: _renew_subs 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd123446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:14.669 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c141f30 con 0x7fd11c09ef00 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 1 -- 172.21.15.144:0/1042760653 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11800cbf0 con 0x7fd11c09ef00 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 10 monclient: dump: 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:14.670 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd123446640 10 monclient: _renew_subs 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd123446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c11f2e0 con 0x7fd11c09ef00 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1137fe640 4 set_mon_vals no callback set 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 1 -- 172.21.15.144:0/1042760653 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd11801c450 con 0x7fd11c09ef00 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.654+0000 7fd1117fa640 1 -- 172.21.15.144:0/1042760653 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fd118013840 con 0x7fd11c09ef00 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.655+0000 7fd123446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.655+0000 7fd123446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.655+0000 7fd123446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.655+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c05ed70 con 0x7fd11c09ef00 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.657+0000 7fd1117fa640 1 -- 172.21.15.144:0/1042760653 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7fd118021060 con 0x7fd11c09ef00 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.657+0000 7fd1117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:14.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.657+0000 7fd1117fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:14.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.724+0000 7fd123446640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "34d020ff-733c-4528-a9ff-b1ef60f8f74d", "id": 2}] 2024-09-18T12:25:14.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.724+0000 7fd123446640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "34d020ff-733c-4528-a9ff-b1ef60f8f74d", "id": 2}] 2024-09-18T12:25:14.727 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.724+0000 7fd123446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:14.728 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.724+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "34d020ff-733c-4528-a9ff-b1ef60f8f74d", "id": 2} v 0) v1 -- 0x7fd11c0723d0 con 0x7fd11c09ef00 2024-09-18T12:25:14.856 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:14.855+0000 7f40fdbbfec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-18T12:25:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.890+0000 7fd1117fa640 1 -- 172.21.15.144:0/1042760653 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "34d020ff-733c-4528-a9ff-b1ef60f8f74d", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd118017070 con 0x7fd11c09ef00 2024-09-18T12:25:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.890+0000 7fd1117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "34d020ff-733c-4528-a9ff-b1ef60f8f74d", "id": 2}] 2024-09-18T12:25:14.892 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.890+0000 7fd1117fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:14.892 INFO:teuthology.orchestra.run.smithi144.stdout:2 2024-09-18T12:25:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 10 monclient: shutdown 2024-09-18T12:25:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 msgr2=0x7fd11c141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 0x7fd11c141080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd11800e870 tx=0x7fd118008040 comp rx=0 tx=0).stop 2024-09-18T12:25:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 shutdown_connections 2024-09-18T12:25:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd11c0a0b90 0x7fd11c11d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.893 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd11c09f2d0 0x7fd11c133c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 --2- 172.21.15.144:0/1042760653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd11c09ef00 0x7fd11c141080 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 >> 172.21.15.144:0/1042760653 conn(0x7fd11c0679c0 msgr2=0x7fd11c134d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.891+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 shutdown_connections 2024-09-18T12:25:14.894 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.892+0000 7fd123446640 1 -- 172.21.15.144:0/1042760653 wait complete. 2024-09-18T12:25:14.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 26e02223-6416-401c-a325-01fad4cf7ce6 3 2024-09-18T12:25:14.944 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:14.943+0000 7f40fdbbfec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T12:25:14.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.995+0000 7f5836d5d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:14.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.995+0000 7f5836d5d640 10 monclient: build_initial_monmap 2024-09-18T12:25:14.997 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.995+0000 7f5836d5d640 10 monclient: monmap: 2024-09-18T12:25:14.997 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:14.997 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:14.997 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:25:14.996692+0000 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:25:14.996692+0000 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:14.998 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 1 Processor -- start 2024-09-18T12:25:15.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 1 -- start start 2024-09-18T12:25:15.000 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 10 monclient: init 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58300a0b90 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f583009ef00 0x7f583009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f583009ef00 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.996+0000 7f5836d5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5830136ea0 0x7f5830139280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f5830136ea0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583009f810 con 0x7f58300a0b90 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583006a430 con 0x7f5830136ea0 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.001 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58300a6220 con 0x7f583009ef00 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f5836d5d640 10 monclient(hunting): authenticate will time out at 750.566034s 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f583009ef00 0x7f583009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f583009ef00 0x7f583009f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44192/0 (socket says 172.21.15.144:44192) 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 -- 172.21.15.144:0/1045960998 learned_addr learned my addr 172.21.15.144:0/1045960998 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): get_auth_request con 0x7f583009ef00 auth_method 0 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.002 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f58352fa640 1 --2- 172.21.15.144:0/1045960998 >> v2:172.21.15.116:3300/0 conn(0x7f5830136ea0 0x7f5830139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f58352fa640 10 monclient(hunting): get_auth_request con 0x7f5830136ea0 auth_method 0 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f58352fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f58352fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f58352fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 -- 172.21.15.144:0/1045960998 >> v2:172.21.15.116:3300/0 conn(0x7f5830136ea0 msgr2=0x7f5830139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 --2- 172.21.15.144:0/1045960998 >> v2:172.21.15.116:3300/0 conn(0x7f5830136ea0 0x7f5830139280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 -- 172.21.15.144:0/1045960998 >> v1:172.21.15.116:6789/0 conn(0x7f58300a0b90 legacy=0x7f58300694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.003 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 monclient: found mon.noname-c 2024-09-18T12:25:15.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.004 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:14.997+0000 7f582ffff640 1 -- 172.21.15.144:0/1045960998 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583009ca30 con 0x7f583009ef00 2024-09-18T12:25:15.009 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582ffff640 1 -- 172.21.15.144:0/1045960998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f583009ef00 msgr2=0x7f583009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582ffff640 1 --2- 172.21.15.144:0/1045960998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f583009ef00 0x7f583009f2d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58300680d0 tx=0x7f581c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52f615e068559017 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 1 -- 172.21.15.144:0/1045960998 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c032440 con 0x7f583009ef00 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: dump: 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.010 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 1 -- 172.21.15.144:0/1045960998 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c0325a0 con 0x7f583009ef00 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 5 monclient: authenticate success, global_id 4123 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.011 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 1 -- 172.21.15.144:0/1045960998 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c032980 con 0x7f583009ef00 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f582f7fe640 10 monclient: dump: 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.012 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 4 set_mon_vals no callback set 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 10 monclient: shutdown 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 1 -- 172.21.15.144:0/1045960998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f583009ef00 msgr2=0x7f583009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.000+0000 7f5836d5d640 1 --2- 172.21.15.144:0/1045960998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f583009ef00 0x7f583009f2d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58300680d0 tx=0x7f581c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.013 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 1 -- 172.21.15.144:0/1045960998 shutdown_connections 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 1 --2- 172.21.15.144:0/1045960998 >> v2:172.21.15.116:3300/0 conn(0x7f5830136ea0 0x7f5830139280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 1 --2- 172.21.15.144:0/1045960998 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f583009ef00 0x7f583009f2d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 1 -- 172.21.15.144:0/1045960998 >> 172.21.15.144:0/1045960998 conn(0x7f58300679c0 msgr2=0x7f58300a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 1 -- 172.21.15.144:0/1045960998 shutdown_connections 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 1 -- 172.21.15.144:0/1045960998 wait complete. 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 10 monclient: build_initial_monmap 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 10 monclient: monmap: 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.014 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.001+0000 7f5836d5d640 1 Processor -- start 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 1 -- start start 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient: init 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583009ef00 0x7f583011c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f583009ef00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58300a0b90 0x7f583011c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f58300a0b90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 0x7f583011cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.015 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f5830136ea0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:15.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58300a12a0 con 0x7f583009ef00 2024-09-18T12:25:15.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58300a4b70 con 0x7f5830136ea0 2024-09-18T12:25:15.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.016 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5830136a60 con 0x7f58300a0b90 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5836d5d640 10 monclient(hunting): authenticate will time out at 750.571489s 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 0x7f583011cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 0x7f583011cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44198/0 (socket says 172.21.15.144:44198) 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 -- 172.21.15.144:0/732668355 learned_addr learned my addr 172.21.15.144:0/732668355 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): get_auth_request con 0x7f5830136ea0 auth_method 0 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.017 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58300a0b90 0x7f583011c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583009ef00 0x7f583011c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 10 monclient(hunting): get_auth_request con 0x7f58300a0b90 auth_method 0 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 10 monclient(hunting): get_auth_request con 0x7f583009ef00 auth_method 0 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f582ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.018 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f5834af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 -- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58300a0b90 msgr2=0x7f583011c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58300a0b90 0x7f583011c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 -- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583009ef00 msgr2=0x7f583011c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583009ef00 0x7f583011c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 monclient: found mon.noname-b 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.002+0000 7f58352fa640 1 -- 172.21.15.144:0/732668355 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c009450 con 0x7f5830136ea0 2024-09-18T12:25:15.019 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f58352fa640 1 -- 172.21.15.144:0/732668355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 msgr2=0x7f583011cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f58352fa640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 0x7f583011cde0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5828009fa0 tx=0x7f5828009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 1 -- 172.21.15.144:0/732668355 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582800ee60 con 0x7f5830136ea0 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: dump: 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.020 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 1 -- 172.21.15.144:0/732668355 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5828013590 con 0x7f5830136ea0 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 1 -- 172.21.15.144:0/732668355 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828013950 con 0x7f5830136ea0 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f5836d5d640 5 monclient: authenticate success, global_id 4126 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.021 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582effd640 4 set_mon_vals no callback set 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 10 monclient: dump: 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f5836d5d640 10 monclient: _renew_subs 2024-09-18T12:25:15.022 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f5836d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583011dea0 con 0x7f5830136ea0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f5836d5d640 10 monclient: _renew_subs 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f5836d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583011ea70 con 0x7f5830136ea0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 1 -- 172.21.15.144:0/732668355 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f5828013cc0 con 0x7f5830136ea0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.005+0000 7f582cff9640 1 -- 172.21.15.144:0/732668355 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f5828014420 con 0x7f5830136ea0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.006+0000 7f5836d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.006+0000 7f5836d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.006+0000 7f5836d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.006+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583011fe50 con 0x7f5830136ea0 2024-09-18T12:25:15.023 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.008+0000 7f582cff9640 1 -- 172.21.15.144:0/732668355 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f582801e460 con 0x7f5830136ea0 2024-09-18T12:25:15.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.008+0000 7f582cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.008+0000 7f582cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:15.037 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.036+0000 7f40fdbbfec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T12:25:15.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.075+0000 7f5836d5d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26e02223-6416-401c-a325-01fad4cf7ce6", "id": 3}] 2024-09-18T12:25:15.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.075+0000 7f5836d5d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26e02223-6416-401c-a325-01fad4cf7ce6", "id": 3}] 2024-09-18T12:25:15.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.075+0000 7f5836d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.075+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "26e02223-6416-401c-a325-01fad4cf7ce6", "id": 3} v 0) v1 -- 0x7f583011ec90 con 0x7f5830136ea0 2024-09-18T12:25:15.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.079+0000 7f582cff9640 1 -- 172.21.15.144:0/732668355 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26e02223-6416-401c-a325-01fad4cf7ce6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f582801ea50 con 0x7f5830136ea0 2024-09-18T12:25:15.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.079+0000 7f582cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26e02223-6416-401c-a325-01fad4cf7ce6", "id": 3}] 2024-09-18T12:25:15.080 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.079+0000 7f582cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:15.081 INFO:teuthology.orchestra.run.smithi144.stdout:3 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 10 monclient: shutdown 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 msgr2=0x7f583011cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 0x7f583011cde0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5828009fa0 tx=0x7f5828009900 comp rx=0 tx=0).stop 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 shutdown_connections 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5830136ea0 0x7f583011cde0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58300a0b90 0x7f583011c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 --2- 172.21.15.144:0/732668355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f583009ef00 0x7f583011c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 >> 172.21.15.144:0/732668355 conn(0x7f58300679c0 msgr2=0x7f583009ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 shutdown_connections 2024-09-18T12:25:15.082 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.080+0000 7f5836d5d640 1 -- 172.21.15.144:0/732668355 wait complete. 2024-09-18T12:25:15.083 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.082+0000 7f40fdbbfec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T12:25:15.091 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 9263cabb-9a65-4cec-a9ab-e25999ca2ad7 4 2024-09-18T12:25:15.182 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.181+0000 7f40fdbbfec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-18T12:25:15.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.224+0000 7f85d5217640 10 monclient: get_monmap_and_config 2024-09-18T12:25:15.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.224+0000 7f85d5217640 10 monclient: build_initial_monmap 2024-09-18T12:25:15.225 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.224+0000 7f85d5217640 10 monclient: monmap: 2024-09-18T12:25:15.225 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:15.225 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:25:15.225054+0000 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:25:15.225054+0000 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:15.226 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.224+0000 7f85d5217640 1 Processor -- start 2024-09-18T12:25:15.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.224+0000 7f85d5217640 1 -- start start 2024-09-18T12:25:15.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.224+0000 7f85d5217640 10 monclient: init 2024-09-18T12:25:15.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T12:25:15.231 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85d00a0b90 0x7f85d00694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): picked mon.noname-a con 0x7f85d00a0b90 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d009ef00 0x7f85d009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): picked mon.noname-b con 0x7f85d009ef00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85d0131b70 0x7f85d0132010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): picked mon.noname-c con 0x7f85d0131b70 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d006a430 con 0x7f85d00a0b90 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d009f810 con 0x7f85d0131b70 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00a6220 con 0x7f85d009ef00 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.232 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85d5217640 10 monclient(hunting): authenticate will time out at 750.794268s 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85d0131b70 0x7f85d0132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85d0131b70 0x7f85d0132010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44202/0 (socket says 172.21.15.144:44202) 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 -- 172.21.15.144:0/1805876726 learned_addr learned my addr 172.21.15.144:0/1805876726 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): get_auth_request con 0x7f85d0131b70 auth_method 0 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ce59c640 1 --2- 172.21.15.144:0/1805876726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d009ef00 0x7f85d009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ced9d640 1 --2- 172.21.15.144:0/1805876726 >> v2:172.21.15.116:3300/0 conn(0x7f85d00a0b90 0x7f85d00694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.233 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ce59c640 10 monclient(hunting): get_auth_request con 0x7f85d009ef00 auth_method 0 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ced9d640 10 monclient(hunting): get_auth_request con 0x7f85d00a0b90 auth_method 0 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 -- 172.21.15.144:0/1805876726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d009ef00 msgr2=0x7f85d009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 --2- 172.21.15.144:0/1805876726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d009ef00 0x7f85d009f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.234 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 -- 172.21.15.144:0/1805876726 >> v2:172.21.15.116:3300/0 conn(0x7f85d00a0b90 msgr2=0x7f85d00694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 --2- 172.21.15.144:0/1805876726 >> v2:172.21.15.116:3300/0 conn(0x7f85d00a0b90 0x7f85d00694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 monclient: found mon.noname-c 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85cf59e640 1 -- 172.21.15.144:0/1805876726 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d009ca30 con 0x7f85d0131b70 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.225+0000 7f85ced9d640 1 --2- 172.21.15.144:0/1805876726 >> v2:172.21.15.116:3300/0 conn(0x7f85d00a0b90 0x7f85d00694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cf59e640 1 -- 172.21.15.144:0/1805876726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d0131b70 msgr2=0x7f85d0132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cf59e640 1 --2- 172.21.15.144:0/1805876726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d0131b70 0x7f85d0132010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f85d0063c10 tx=0x7f85c8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=978f663230c1c91b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 1 -- 172.21.15.144:0/1805876726 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8007610 con 0x7f85d0131b70 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: dump: 2024-09-18T12:25:15.235 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85d5217640 5 monclient: authenticate success, global_id 4129 2024-09-18T12:25:15.236 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 1 -- 172.21.15.144:0/1805876726 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c802d050 con 0x7f85d0131b70 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85d5217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85d5217640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85d5217640 4 set_mon_vals no callback set 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 1 -- 172.21.15.144:0/1805876726 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c80079e0 con 0x7f85d0131b70 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.228+0000 7f85cdd9b640 10 monclient: dump: 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.237 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 10 monclient: shutdown 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 -- 172.21.15.144:0/1805876726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d0131b70 msgr2=0x7f85d0132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 --2- 172.21.15.144:0/1805876726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d0131b70 0x7f85d0132010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f85d0063c10 tx=0x7f85c8009840 comp rx=0 tx=0).stop 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 -- 172.21.15.144:0/1805876726 shutdown_connections 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 --2- 172.21.15.144:0/1805876726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d0131b70 0x7f85d0132010 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 --2- 172.21.15.144:0/1805876726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d009ef00 0x7f85d009f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.238 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 --2- 172.21.15.144:0/1805876726 >> v2:172.21.15.116:3300/0 conn(0x7f85d00a0b90 0x7f85d00694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 -- 172.21.15.144:0/1805876726 >> 172.21.15.144:0/1805876726 conn(0x7f85d00679c0 msgr2=0x7f85d00a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 -- 172.21.15.144:0/1805876726 shutdown_connections 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 1 -- 172.21.15.144:0/1805876726 wait complete. 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 10 monclient: build_initial_monmap 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.229+0000 7f85d5217640 10 monclient: monmap: 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.239 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:15.240 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:15.240 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:15.240 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 Processor -- start 2024-09-18T12:25:15.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 -- start start 2024-09-18T12:25:15.240 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient: init 2024-09-18T12:25:15.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:25:15.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 0x7f85d0141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): picked mon.noname-b con 0x7f85d009ef00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:15.244 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d00a0b90 0x7f85d01417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): picked mon.noname-c con 0x7f85d00a0b90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0131b70 0x7f85d011e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): picked mon.noname-a con 0x7f85d0131b70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00a4bc0 con 0x7f85d0131b70 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d0122080 con 0x7f85d009ef00 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d0120870 con 0x7f85d00a0b90 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85d5217640 10 monclient(hunting): authenticate will time out at 750.799753s 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 0x7f85d0141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.245 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 0x7f85d0141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44204/0 (socket says 172.21.15.144:44204) 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 1 -- 172.21.15.144:0/2371905329 learned_addr learned my addr 172.21.15.144:0/2371905329 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 10 monclient(hunting): get_auth_request con 0x7f85d009ef00 auth_method 0 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85cf59e640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0131b70 0x7f85d011e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.230+0000 7f85ce59c640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d00a0b90 0x7f85d01417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85cf59e640 10 monclient(hunting): get_auth_request con 0x7f85d0131b70 auth_method 0 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.246 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85cf59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ce59c640 10 monclient(hunting): get_auth_request con 0x7f85d00a0b90 auth_method 0 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 1 -- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d00a0b90 msgr2=0x7f85d01417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d00a0b90 0x7f85d01417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 1 -- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0131b70 msgr2=0x7f85d011e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0131b70 0x7f85d011e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 1 monclient: found mon.noname-b 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.247 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.231+0000 7f85ced9d640 1 -- 172.21.15.144:0/2371905329 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8009ae0 con 0x7f85d009ef00 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85ced9d640 1 -- 172.21.15.144:0/2371905329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 msgr2=0x7f85d0141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85ced9d640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 0x7f85d0141290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f85c000d890 tx=0x7f85c0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 1 -- 172.21.15.144:0/2371905329 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c000de80 con 0x7f85d009ef00 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: dump: 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.248 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85d5217640 5 monclient: authenticate success, global_id 4132 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85d5217640 10 monclient: _renew_subs 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 1 -- 172.21.15.144:0/2371905329 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c000ba60 con 0x7f85d009ef00 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85d5217640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d0121a50 con 0x7f85d009ef00 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 1 -- 172.21.15.144:0/2371905329 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c000be20 con 0x7f85d009ef00 2024-09-18T12:25:15.249 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 10 monclient: dump: 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85cd59a640 4 set_mon_vals no callback set 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85d5217640 10 monclient: _renew_subs 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85d5217640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d0141d10 con 0x7f85d009ef00 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 1 -- 172.21.15.144:0/2371905329 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f85c00148d0 con 0x7f85d009ef00 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.233+0000 7f85c6ffd640 1 -- 172.21.15.144:0/2371905329 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f85c001eba0 con 0x7f85d009ef00 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.234+0000 7f85d5217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.234+0000 7f85d5217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.234+0000 7f85d5217640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.234+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d005ed70 con 0x7f85d009ef00 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.236+0000 7f85c6ffd640 1 -- 172.21.15.144:0/2371905329 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f85c0014d60 con 0x7f85d009ef00 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.236+0000 7f85c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.251 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.236+0000 7f85c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:15.252 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.237+0000 7f40fdbbfec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T12:25:15.283 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.282+0000 7f40fdbbfec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T12:25:15.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.302+0000 7f85d5217640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9263cabb-9a65-4cec-a9ab-e25999ca2ad7", "id": 4}] 2024-09-18T12:25:15.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.302+0000 7f85d5217640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9263cabb-9a65-4cec-a9ab-e25999ca2ad7", "id": 4}] 2024-09-18T12:25:15.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.302+0000 7f85d5217640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.304 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.302+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9263cabb-9a65-4cec-a9ab-e25999ca2ad7", "id": 4} v 0) v1 -- 0x7f85d01318e0 con 0x7f85d009ef00 2024-09-18T12:25:15.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.307+0000 7f85c6ffd640 1 -- 172.21.15.144:0/2371905329 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9263cabb-9a65-4cec-a9ab-e25999ca2ad7", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f85c002d0b0 con 0x7f85d009ef00 2024-09-18T12:25:15.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.307+0000 7f85c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9263cabb-9a65-4cec-a9ab-e25999ca2ad7", "id": 4}] 2024-09-18T12:25:15.308 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.307+0000 7f85c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:15.308 INFO:teuthology.orchestra.run.smithi144.stdout:4 2024-09-18T12:25:15.309 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 10 monclient: shutdown 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 msgr2=0x7f85d0141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 0x7f85d0141290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f85c000d890 tx=0x7f85c0008040 comp rx=0 tx=0).stop 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 shutdown_connections 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0131b70 0x7f85d011e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d00a0b90 0x7f85d01417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 --2- 172.21.15.144:0/2371905329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d009ef00 0x7f85d0141290 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 >> 172.21.15.144:0/2371905329 conn(0x7f85d00679c0 msgr2=0x7f85d00706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 shutdown_connections 2024-09-18T12:25:15.310 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.308+0000 7f85d5217640 1 -- 172.21.15.144:0/2371905329 wait complete. 2024-09-18T12:25:15.320 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 174ff7a9-21fc-4aa3-8801-81ebe54449b1 5 2024-09-18T12:25:15.382 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.380+0000 7f40fdbbfec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T12:25:15.412 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.411+0000 7fc8e92dd640 10 monclient: get_monmap_and_config 2024-09-18T12:25:15.412 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.411+0000 7fc8e92dd640 10 monclient: build_initial_monmap 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.411+0000 7fc8e92dd640 10 monclient: monmap: 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:25:15.412421+0000 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:25:15.412421+0000 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:15.413 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 1 Processor -- start 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 1 -- start start 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient: init 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8e40a0b90 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8e409ef00 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8e413a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient(hunting): picked mon.noname-b con 0x7fc8e409f2d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e40a6220 con 0x7fc8e409ef00 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e409ca30 con 0x7fc8e40a0b90 2024-09-18T12:25:15.424 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e406a430 con 0x7fc8e409f2d0 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e92dd640 10 monclient(hunting): authenticate will time out at 750.981741s 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e2ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc8e40a0b90 0x7fc8e40694d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:44348/0 (socket says 172.21.15.144:44348) 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e2ffd640 1 -- 172.21.15.144:0/2228135446 learned_addr learned my addr 172.21.15.144:0/2228135446 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.412+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8e413a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request con 0x7fc8e409f2d0 auth_method 0 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.425 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 msgr2=0x7fc8e413a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 msgr2=0x7fc8e413a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8e413a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.413+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8e413a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.414+0000 7fc8e2ffd640 1 -- 172.21.15.144:0/2228135446 >> v1:172.21.15.144:6789/0 conn(0x7fc8e40a0b90 legacy=0x7fc8e40694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 -- 172.21.15.144:0/2228135446 >> v1:172.21.15.116:6789/0 conn(0x7fc8e409ef00 legacy=0x7fc8e4136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782231435 0 0) 0x7fc8e409ca30 con 0x7fc8e40a0b90 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.426 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): my global_id is 4135 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d00034e0 con 0x7fc8e40a0b90 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565005906 0 0) 0x7fc8e40a6220 con 0x7fc8e409ef00 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): my global_id is 4131 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e409ca30 con 0x7fc8e409ef00 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186515575 0 0) 0x7fc8d00034e0 con 0x7fc8e40a0b90 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 msgr2=0x7fc8e413a5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8e413a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 >> v1:172.21.15.116:6789/0 conn(0x7fc8e409ef00 legacy=0x7fc8e4136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:25:15.427 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e40a0f60 con 0x7fc8e40a0b90 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e92dd640 5 monclient: authenticate success, global_id 4135 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fc8d4001b20 con 0x7fc8e40a0b90 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e92dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: dump: 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.428 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 >> v1:172.21.15.144:6789/0 conn(0x7fc8e40a0b90 legacy=0x7fc8e40694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.429 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0003cd0 0x7fc8d00040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): picked mon.a con 0x7fc8d0003cd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8d0004780 0x7fc8d0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): picked mon.b con 0x7fc8d0004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8d000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): picked mon.c con 0x7fc8e409f2d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0005530 con 0x7fc8d0003cd0 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d000bf10 con 0x7fc8d0004780 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d000c050 con 0x7fc8e409f2d0 2024-09-18T12:25:15.430 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e92dd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8d0004780 0x7fc8d0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0003cd0 0x7fc8d00040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): get_auth_request con 0x7fc8d0004780 auth_method 0 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc8d0003cd0 auth_method 0 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:15.431 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8d000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request con 0x7fc8e409f2d0 auth_method 0 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient(hunting): handle_auth_done global_id 4135 payload 293 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 -- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 msgr2=0x7fc8d000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.432 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8d000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 -- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0003cd0 msgr2=0x7fc8d00040c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0003cd0 0x7fc8d00040c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 monclient: found mon.b 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 1 -- 172.21.15.144:0/2228135446 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d00034e0 con 0x7fc8d0004780 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.415+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e409f2d0 0x7fc8d000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8da7fc640 1 -- 172.21.15.144:0/2228135446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8d0004780 msgr2=0x7fc8d0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8da7fc640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8d0004780 0x7fc8d0004bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40a63b0 tx=0x7fc8cc001b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=762cba348c387223 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc02adb0 con 0x7fc8d0004780 2024-09-18T12:25:15.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 10 monclient: dump: 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.434 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e92dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8cc0070f0 con 0x7fc8d0004780 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 1 -- 172.21.15.144:0/2228135446 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0074b0 con 0x7fc8d0004780 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e92dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e92dd640 4 set_mon_vals no callback set 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.418+0000 7fc8e27fc640 10 monclient: dump: 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.435 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 10 monclient: shutdown 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2228135446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8d0004780 msgr2=0x7fc8d0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 1 --2- 172.21.15.144:0/2228135446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8d0004780 0x7fc8d0004bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40a63b0 tx=0x7fc8cc001b80 comp rx=0 tx=0).stop 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2228135446 reap_dead start 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2228135446 shutdown_connections 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2228135446 >> 172.21.15.144:0/2228135446 conn(0x7fc8e40679c0 msgr2=0x7fc8e40a5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2228135446 shutdown_connections 2024-09-18T12:25:15.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2228135446 wait complete. 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.419+0000 7fc8e92dd640 10 monclient: build_initial_monmap 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient: monmap: 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 Processor -- start 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 -- start start 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient: init 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0004780 0x7fc8e4136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): picked mon.noname-a con 0x7fc8d0004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40977f0 0x7fc8e4137100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): picked mon.noname-c con 0x7fc8e40977f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 0x7fc8e4137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): picked mon.noname-b con 0x7fc8e4137640 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e4138020 con 0x7fc8d0004780 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e4097170 con 0x7fc8e4137640 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e4134b30 con 0x7fc8e40977f0 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.420+0000 7fc8e92dd640 10 monclient(hunting): authenticate will time out at 750.989959s 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 0x7fc8e4137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 0x7fc8e4137ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44222/0 (socket says 172.21.15.144:44222) 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2175295367 learned_addr learned my addr 172.21.15.144:0/2175295367 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request con 0x7fc8e4137640 auth_method 0 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0004780 0x7fc8e4136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40977f0 0x7fc8e4137100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc8d0004780 auth_method 0 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 10 monclient(hunting): get_auth_request con 0x7fc8e40977f0 auth_method 0 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40977f0 msgr2=0x7fc8e4137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40977f0 0x7fc8e4137100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0004780 msgr2=0x7fc8e4136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0004780 0x7fc8e4136bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 monclient: found mon.noname-b 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.421+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2175295367 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e40a12a0 con 0x7fc8e4137640 2024-09-18T12:25:15.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e37fe640 1 -- 172.21.15.144:0/2175295367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 msgr2=0x7fc8e4137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e37fe640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 0x7fc8e4137ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8e406a860 tx=0x7fc8dc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 1 -- 172.21.15.144:0/2175295367 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc009480 con 0x7fc8e4137640 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: dump: 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.444 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 1 -- 172.21.15.144:0/2175295367 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc011c40 con 0x7fc8e4137640 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e92dd640 5 monclient: authenticate success, global_id 4138 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e92dd640 10 monclient: _renew_subs 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 1 -- 172.21.15.144:0/2175295367 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc015460 con 0x7fc8e4137640 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e92dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e40986e0 con 0x7fc8e4137640 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8db7fe640 10 monclient: dump: 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e1ffb640 4 set_mon_vals no callback set 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e92dd640 10 monclient: _renew_subs 2024-09-18T12:25:15.446 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e92dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.423+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e4147780 con 0x7fc8e4137640 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.424+0000 7fc8db7fe640 1 -- 172.21.15.144:0/2175295367 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fc8dc015890 con 0x7fc8e4137640 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.424+0000 7fc8db7fe640 1 -- 172.21.15.144:0/2175295367 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fc8dc01ec50 con 0x7fc8e4137640 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.424+0000 7fc8e92dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.424+0000 7fc8e92dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.424+0000 7fc8e92dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.424+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e405ed70 con 0x7fc8e4137640 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.426+0000 7fc8db7fe640 1 -- 172.21.15.144:0/2175295367 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7fc8dc03d990 con 0x7fc8e4137640 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.426+0000 7fc8db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.447 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.426+0000 7fc8db7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:15.447 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.432+0000 7f40fdbbfec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T12:25:15.488 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.484+0000 7f40fdbbfec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T12:25:15.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.494+0000 7fc8e92dd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "174ff7a9-21fc-4aa3-8801-81ebe54449b1", "id": 5}] 2024-09-18T12:25:15.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.494+0000 7fc8e92dd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "174ff7a9-21fc-4aa3-8801-81ebe54449b1", "id": 5}] 2024-09-18T12:25:15.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.494+0000 7fc8e92dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:15.495 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.494+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd new", "uuid": "174ff7a9-21fc-4aa3-8801-81ebe54449b1", "id": 5} v 0) v1 -- 0x7fc8e40723d0 con 0x7fc8e4137640 2024-09-18T12:25:15.551 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.550+0000 7f40fdbbfec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T12:25:15.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.555+0000 7fc8db7fe640 1 -- 172.21.15.144:0/2175295367 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "174ff7a9-21fc-4aa3-8801-81ebe54449b1", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc8dc02c070 con 0x7fc8e4137640 2024-09-18T12:25:15.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.555+0000 7fc8db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "174ff7a9-21fc-4aa3-8801-81ebe54449b1", "id": 5}] 2024-09-18T12:25:15.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.555+0000 7fc8db7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:15.557 INFO:teuthology.orchestra.run.smithi144.stdout:5 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 10 monclient: shutdown 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 msgr2=0x7fc8e4137ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 0x7fc8e4137ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8e406a860 tx=0x7fc8dc008040 comp rx=0 tx=0).stop 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 shutdown_connections 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8e4137640 0x7fc8e4137ae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8e40977f0 0x7fc8e4137100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.556+0000 7fc8e92dd640 1 --2- 172.21.15.144:0/2175295367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d0004780 0x7fc8e4136bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.557+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 >> 172.21.15.144:0/2175295367 conn(0x7fc8e40679c0 msgr2=0x7fc8e409dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:15.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.557+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 shutdown_connections 2024-09-18T12:25:15.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-18T12:25:15.557+0000 7fc8e92dd640 1 -- 172.21.15.144:0/2175295367 wait complete. 2024-09-18T12:25:15.568 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T12:25:15.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T12:25:15.571 INFO:tasks.ceph.osd.0:Started 2024-09-18T12:25:15.571 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T12:25:15.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T12:25:15.574 INFO:tasks.ceph.osd.1:Started 2024-09-18T12:25:15.574 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T12:25:15.575 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T12:25:15.577 INFO:tasks.ceph.osd.2:Started 2024-09-18T12:25:15.577 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T12:25:15.577 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T12:25:15.580 INFO:tasks.ceph.osd.3:Started 2024-09-18T12:25:15.580 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-18T12:25:15.580 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-18T12:25:15.582 INFO:tasks.ceph.osd.4:Started 2024-09-18T12:25:15.582 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-18T12:25:15.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-18T12:25:15.585 INFO:tasks.ceph.osd.5:Started 2024-09-18T12:25:15.585 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:25:15.601 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.600+0000 7f40fdbbfec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T12:25:15.650 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:15.648+0000 7f40fdbbfec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T12:25:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.690+0000 7f2447f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.690+0000 7f2447f9d640 10 monclient: build_initial_monmap 2024-09-18T12:25:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.690+0000 7f2447f9d640 10 monclient: monmap: 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:15.692344+0000 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:15.692344+0000 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:15.694 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:15.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 1 Processor -- start 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 1 -- start start 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient: init 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f244006eb80 0x7f244006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f244006eb80 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2440068a20 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f244006f490 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2440069b80 con 0x7f244006f490 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2440069e00 con 0x7f2440068a20 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2440099f90 con 0x7f244006eb80 2024-09-18T12:25:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.691+0000 7f2447f9d640 10 monclient(hunting): authenticate will time out at 743.013988s 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f244006eb80 0x7f244006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f244653a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f244006f490 0x7f2440138ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59958/0 (socket says 172.21.15.116:59958) 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f244006eb80 0x7f244006ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53778/0 (socket says 172.21.15.116:53778) 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f244653a640 1 -- 172.21.15.116:0/3051495603 learned_addr learned my addr 172.21.15.116:0/3051495603 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): get_auth_request con 0x7f244006eb80 auth_method 0 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient(hunting): handle_auth_done global_id 4152 payload 274 2024-09-18T12:25:15.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 1 -- 172.21.15.116:0/3051495603 >> v1:172.21.15.144:6789/0 conn(0x7f2440068a20 legacy=0x7f244009c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 1 -- 172.21.15.116:0/3051495603 >> v1:172.21.15.116:6789/0 conn(0x7f244006f490 legacy=0x7f2440138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:15.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 1 monclient: found mon.noname-a 2024-09-18T12:25:15.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:15.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.692+0000 7f2445d39640 1 -- 172.21.15.116:0/3051495603 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244009cc50 con 0x7f244006eb80 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2445d39640 1 -- 172.21.15.116:0/3051495603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 msgr2=0x7f244006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2445d39640 1 --2- 172.21.15.116:0/3051495603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244006ef50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f243c009850 tx=0x7f243c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5020e89180870759 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 1 -- 172.21.15.116:0/3051495603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c032440 con 0x7f244006eb80 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: dump: 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 5 monclient: authenticate success, global_id 4152 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 1 -- 172.21.15.116:0/3051495603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243c0325a0 con 0x7f244006eb80 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 4 set_mon_vals no callback set 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 1 -- 172.21.15.116:0/3051495603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c032940 con 0x7f244006eb80 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2444d37640 10 monclient: dump: 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.705 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 10 monclient: shutdown 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 1 -- 172.21.15.116:0/3051495603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 msgr2=0x7f244006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 1 --2- 172.21.15.116:0/3051495603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244006ef50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f243c009850 tx=0x7f243c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 1 -- 172.21.15.116:0/3051495603 shutdown_connections 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 1 --2- 172.21.15.116:0/3051495603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244006ef50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.695+0000 7f2447f9d640 1 -- 172.21.15.116:0/3051495603 >> 172.21.15.116:0/3051495603 conn(0x7f2440067cd0 msgr2=0x7f24400a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.696+0000 7f2447f9d640 1 -- 172.21.15.116:0/3051495603 shutdown_connections 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.696+0000 7f2447f9d640 1 -- 172.21.15.116:0/3051495603 wait complete. 2024-09-18T12:25:15.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.696+0000 7f2447f9d640 10 monclient: build_initial_monmap 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.696+0000 7f2447f9d640 10 monclient: monmap: 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.696+0000 7f2447f9d640 1 Processor -- start 2024-09-18T12:25:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.696+0000 7f2447f9d640 1 -- start start 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.696+0000 7f2447f9d640 10 monclient: init 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2440068a20 0x7f244011ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2440068a20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244011ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f244006eb80 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f244006f490 0x7f244011f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f244006f490 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24401394f0 con 0x7f244006eb80 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440120900 con 0x7f244006f490 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244009cf40 con 0x7f2440068a20 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2447f9d640 10 monclient(hunting): authenticate will time out at 743.019319s 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2440068a20 0x7f244011ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2440068a20 0x7f244011ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45856/0 (socket says 172.21.15.116:45856) 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 1 -- 172.21.15.116:0/3705459381 learned_addr learned my addr 172.21.15.116:0/3705459381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 10 monclient(hunting): get_auth_request con 0x7f2440068a20 auth_method 0 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244011ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f244006f490 0x7f244011f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): get_auth_request con 0x7f244006eb80 auth_method 0 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 10 monclient(hunting): get_auth_request con 0x7f244006f490 auth_method 0 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 1 -- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2440068a20 msgr2=0x7f244011ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2440068a20 0x7f244011ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 1 -- 172.21.15.116:0/3705459381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f244006f490 msgr2=0x7f244011f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f244006f490 0x7f244011f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient: _finish_hunting 0 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 1 monclient: found mon.noname-a 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f2445538640 1 -- 172.21.15.116:0/3705459381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c009390 con 0x7f244006eb80 2024-09-18T12:25:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.697+0000 7f244653a640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f244006f490 0x7f244011f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2445538640 1 -- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 msgr2=0x7f244011ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2445538640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244011ef60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f243000c4e0 tx=0x7f2430007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24300117a0 con 0x7f244006eb80 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: dump: 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2430011920 con 0x7f244006eb80 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2447f9d640 5 monclient: authenticate success, global_id 4155 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2447f9d640 10 monclient: _renew_subs 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2447f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24400a2770 con 0x7f244006eb80 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2430011ce0 con 0x7f244006eb80 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2435ffb640 10 monclient: dump: 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2437fff640 4 set_mon_vals no callback set 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2447f9d640 10 monclient: _renew_subs 2024-09-18T12:25:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2447f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.700+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2440139f50 con 0x7f244006eb80 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.701+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f2430012630 con 0x7f244006eb80 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.701+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f243001a960 con 0x7f244006eb80 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.701+0000 7f2447f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.701+0000 7f2447f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.701+0000 7f2447f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.701+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2440120410 con 0x7f244006eb80 2024-09-18T12:25:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:15.703+0000 7f2445538640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244011ef60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f243000c4e0 tx=0x7f2430007e10 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f2440120410 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:25:16.172 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-09-18T12:25:16.170+0000 7f65d1a0e440 -1 Falling back to public interface 2024-09-18T12:25:16.173 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-09-18T12:25:16.171+0000 7f025192f440 -1 Falling back to public interface 2024-09-18T12:25:16.176 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-09-18T12:25:16.174+0000 7f3a4de69440 -1 Falling back to public interface 2024-09-18T12:25:16.178 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-09-18T12:25:16.176+0000 7f613280a440 -1 Falling back to public interface 2024-09-18T12:25:16.178 INFO:tasks.ceph.osd.3.smithi144.stderr:2024-09-18T12:25:16.177+0000 7fa576729440 -1 Falling back to public interface 2024-09-18T12:25:16.178 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-09-18T12:25:16.177+0000 7f4d90f23440 -1 Falling back to public interface 2024-09-18T12:25:16.266 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:16.265+0000 7f40fdbbfec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T12:25:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.532+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f2440120410 con 0x7f244006eb80 2024-09-18T12:25:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.532+0000 7f2435ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.532+0000 7f2435ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:16.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.553+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 2) v1 ==== 44392+0+0 (secure 0 0 0) 0x7f243001acd0 con 0x7f244006eb80 2024-09-18T12:25:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.593+0000 7f2447f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.593+0000 7f2447f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.593+0000 7f2447f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.593+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f244004bf60 con 0x7f244006eb80 2024-09-18T12:25:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.593+0000 7f2435ffb640 1 -- 172.21.15.116:0/3705459381 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f2430024540 con 0x7f244006eb80 2024-09-18T12:25:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.593+0000 7f2435ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.593+0000 7f2435ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:16.597 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:16.597 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:25:15.496296+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T12:25:15.496296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:25:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 10 monclient: shutdown 2024-09-18T12:25:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 msgr2=0x7f244011ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244011ef60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f243000c4e0 tx=0x7f2430007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 shutdown_connections 2024-09-18T12:25:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f244006f490 0x7f244011f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244006eb80 0x7f244011ef60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 --2- 172.21.15.116:0/3705459381 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2440068a20 0x7f244011ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 >> 172.21.15.116:0/3705459381 conn(0x7f2440067cd0 msgr2=0x7f244009af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 shutdown_connections 2024-09-18T12:25:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:16.594+0000 7f2447f9d640 1 -- 172.21.15.116:0/3705459381 wait complete. 2024-09-18T12:25:16.607 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T12:25:16.608 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T12:25:17.213 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T12:25:17.309 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.307+0000 7f6a1d99d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:17.309 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.307+0000 7f6a1d99d640 10 monclient: build_initial_monmap 2024-09-18T12:25:17.309 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.307+0000 7f6a1d99d640 10 monclient: monmap: 2024-09-18T12:25:17.309 INFO:teuthology.misc.health.smithi116.stderr:epoch 0 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:25:17.309393+0000 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:25:17.309393+0000 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:17.310 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 1 Processor -- start 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 1 -- start start 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient: init 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a18097fa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a18098370 0x7f6a18098770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:17.320 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a18098370 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a18136c00 0x7f6a18138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a18136c00 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a18097b00 con 0x7f6a18097fa0 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a180a1470 con 0x7f6a18098370 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1809fdb0 con 0x7f6a18136c00 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.308+0000 7f6a1d99d640 10 monclient(hunting): authenticate will time out at 744.630905s 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a18098370 0x7f6a18098770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a18098370 0x7f6a18098770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53792/0 (socket says 172.21.15.116:53792) 2024-09-18T12:25:17.321 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 -- 172.21.15.116:0/3471127135 learned_addr learned my addr 172.21.15.116:0/3471127135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): get_auth_request con 0x7f6a18098370 auth_method 0 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a177fe640 1 --2- 172.21.15.116:0/3471127135 >> v2:172.21.15.144:3300/0 conn(0x7f6a18136c00 0x7f6a18138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a177fe640 10 monclient(hunting): get_auth_request con 0x7f6a18136c00 auth_method 0 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:17.322 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient(hunting): handle_auth_done global_id 4176 payload 274 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 -- 172.21.15.116:0/3471127135 >> v2:172.21.15.144:3300/0 conn(0x7f6a18136c00 msgr2=0x7f6a18138fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 --2- 172.21.15.116:0/3471127135 >> v2:172.21.15.144:3300/0 conn(0x7f6a18136c00 0x7f6a18138fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 -- 172.21.15.116:0/3471127135 >> v1:172.21.15.144:6789/0 conn(0x7f6a18097fa0 legacy=0x7f6a180973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 monclient: found mon.noname-a 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.309+0000 7f6a0e7fc640 1 -- 172.21.15.116:0/3471127135 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a180a1190 con 0x7f6a18098370 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a0e7fc640 1 -- 172.21.15.116:0/3471127135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18098370 msgr2=0x7f6a18098770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a0e7fc640 1 --2- 172.21.15.116:0/3471127135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18098370 0x7f6a18098770 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6a180a0160 tx=0x7f6a000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b4bfac6e5809ad server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:17.323 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 1 -- 172.21.15.116:0/3471127135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00032440 con 0x7f6a18098370 2024-09-18T12:25:17.324 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:17.324 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:17.324 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: dump: 2024-09-18T12:25:17.324 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:17.324 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:17.324 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.325 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.325 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:17.325 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 1 -- 172.21.15.116:0/3471127135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a000325a0 con 0x7f6a18098370 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 5 monclient: authenticate success, global_id 4176 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 1 -- 172.21.15.116:0/3471127135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00032980 con 0x7f6a18098370 2024-09-18T12:25:17.326 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 4 set_mon_vals no callback set 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a167fc640 10 monclient: dump: 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:17.327 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 10 monclient: shutdown 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 1 -- 172.21.15.116:0/3471127135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18098370 msgr2=0x7f6a18098770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.312+0000 7f6a1d99d640 1 --2- 172.21.15.116:0/3471127135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18098370 0x7f6a18098770 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6a180a0160 tx=0x7f6a000091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 1 -- 172.21.15.116:0/3471127135 shutdown_connections 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 1 --2- 172.21.15.116:0/3471127135 >> v2:172.21.15.144:3300/0 conn(0x7f6a18136c00 0x7f6a18138fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 1 --2- 172.21.15.116:0/3471127135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18098370 0x7f6a18098770 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 1 -- 172.21.15.116:0/3471127135 >> 172.21.15.116:0/3471127135 conn(0x7f6a180676c0 msgr2=0x7f6a180a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 1 -- 172.21.15.116:0/3471127135 shutdown_connections 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 1 -- 172.21.15.116:0/3471127135 wait complete. 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 10 monclient: build_initial_monmap 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.313+0000 7f6a1d99d640 10 monclient: monmap: 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:epoch 0 2024-09-18T12:25:17.328 INFO:teuthology.misc.health.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:created 0.000000 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 Processor -- start 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 -- start start 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient: init 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:17.329 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 0x7f6a1811c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a18097fa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a18098370 0x7f6a1811c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a18098370 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a18136c00 0x7f6a1811cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a18136c00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1811f9a0 con 0x7f6a18097fa0 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18136750 con 0x7f6a18098370 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18098cd0 con 0x7f6a18136c00 2024-09-18T12:25:17.330 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a1d99d640 10 monclient(hunting): authenticate will time out at 744.636589s 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 0x7f6a1811c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 0x7f6a1811c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53798/0 (socket says 172.21.15.116:53798) 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a18136c00 0x7f6a1811cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 1 -- 172.21.15.116:0/1905929888 learned_addr learned my addr 172.21.15.116:0/1905929888 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): get_auth_request con 0x7f6a18097fa0 auth_method 0 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 10 monclient(hunting): get_auth_request con 0x7f6a18136c00 auth_method 0 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:17.331 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a18098370 0x7f6a1811c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 10 monclient(hunting): get_auth_request con 0x7f6a18098370 auth_method 0 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:17.332 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.314+0000 7f6a16ffd640 10 monclient(hunting): handle_auth_done global_id 4179 payload 824 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 1 -- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a18136c00 msgr2=0x7f6a1811cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a18136c00 0x7f6a1811cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 1 -- 172.21.15.116:0/1905929888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a18098370 msgr2=0x7f6a1811c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a18098370 0x7f6a1811c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 1 monclient: found mon.noname-a 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a16ffd640 1 -- 172.21.15.116:0/1905929888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a00009450 con 0x7f6a18097fa0 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.315+0000 7f6a177fe640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a18136c00 0x7f6a1811cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a16ffd640 1 -- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 msgr2=0x7f6a1811c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:17.333 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a16ffd640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 0x7f6a1811c0e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c5e0 tx=0x7f6a08007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 1 -- 172.21.15.116:0/1905929888 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08012840 con 0x7f6a18097fa0 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: dump: 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:17.334 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a1d99d640 5 monclient: authenticate success, global_id 4179 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 1 -- 172.21.15.116:0/1905929888 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a080129a0 con 0x7f6a18097fa0 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a1d99d640 10 monclient: _renew_subs 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a1d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a18135d70 con 0x7f6a18097fa0 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 1 -- 172.21.15.116:0/1905929888 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08012da0 con 0x7f6a18097fa0 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a0f7fe640 10 monclient: dump: 2024-09-18T12:25:17.335 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a1d99d640 10 monclient: _renew_subs 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a1d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a15ffb640 4 set_mon_vals no callback set 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.317+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1811e760 con 0x7f6a18097fa0 2024-09-18T12:25:17.336 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.318+0000 7f6a0f7fe640 1 -- 172.21.15.116:0/1905929888 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 2) v1 ==== 44392+0+0 (secure 0 0 0) 0x7f6a08011590 con 0x7f6a18097fa0 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.318+0000 7f6a1d99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.318+0000 7f6a1d99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.318+0000 7f6a1d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.318+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1811e9e0 con 0x7f6a18097fa0 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.318+0000 7f6a0f7fe640 1 -- 172.21.15.116:0/1905929888 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f6a0804a6d0 con 0x7f6a18097fa0 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.320+0000 7f6a0f7fe640 1 -- 172.21.15.116:0/1905929888 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f6a1811e9e0 con 0x7f6a18097fa0 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.320+0000 7f6a0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:17.337 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.320+0000 7f6a0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:17.377 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.375+0000 7f6a1d99d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:17.377 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.375+0000 7f6a1d99d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:17.377 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.375+0000 7f6a1d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:17.377 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.375+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a1811fbc0 con 0x7f6a18097fa0 2024-09-18T12:25:17.377 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.375+0000 7f6a0f7fe640 1 -- 172.21.15.116:0/1905929888 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f6a1811fbc0 con 0x7f6a18097fa0 2024-09-18T12:25:17.377 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.375+0000 7f6a0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:17.378 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.375+0000 7f6a0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:17.378 INFO:teuthology.misc.health.smithi116.stdout: 2024-09-18T12:25:17.378 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":8,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:25:15.496296+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T12:25:15.496296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:25:17.379 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 10 monclient: shutdown 2024-09-18T12:25:17.379 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:17.379 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 msgr2=0x7f6a1811c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:17.379 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 0x7f6a1811c0e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c5e0 tx=0x7f6a08007ed0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.379 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 shutdown_connections 2024-09-18T12:25:17.379 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a18136c00 0x7f6a1811cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.380 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a18098370 0x7f6a1811c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.380 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 --2- 172.21.15.116:0/1905929888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18097fa0 0x7f6a1811c0e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:17.380 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 >> 172.21.15.116:0/1905929888 conn(0x7f6a180676c0 msgr2=0x7f6a18137830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:17.380 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 shutdown_connections 2024-09-18T12:25:17.380 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:17.376+0000 7f6a1d99d640 1 -- 172.21.15.116:0/1905929888 wait complete. 2024-09-18T12:25:17.387 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-18T12:25:17.564 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:17.563+0000 7f40e2150640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T12:25:17.570 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-18T12:25:17.569+0000 7f40e2150640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T12:25:19.643 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-09-18T12:25:19.641+0000 7f65d1a0e440 -1 osd.1 0 log_to_monitors true 2024-09-18T12:25:19.673 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-09-18T12:25:19.671+0000 7f025192f440 -1 osd.2 0 log_to_monitors true 2024-09-18T12:25:19.677 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-09-18T12:25:19.675+0000 7f3a4de69440 -1 osd.0 0 log_to_monitors true 2024-09-18T12:25:19.678 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-09-18T12:25:19.677+0000 7f613280a440 -1 osd.4 0 log_to_monitors true 2024-09-18T12:25:19.683 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-09-18T12:25:19.682+0000 7f4d90f23440 -1 osd.5 0 log_to_monitors true 2024-09-18T12:25:19.696 INFO:tasks.ceph.osd.3.smithi144.stderr:2024-09-18T12:25:19.695+0000 7fa576729440 -1 osd.3 0 log_to_monitors true 2024-09-18T12:25:20.880 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-09-18T12:25:20.878+0000 7f65cc1f6640 -1 osd.1 0 waiting for initial osdmap 2024-09-18T12:25:20.881 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-09-18T12:25:20.878+0000 7f612cff6640 -1 osd.4 0 waiting for initial osdmap 2024-09-18T12:25:20.881 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-09-18T12:25:20.878+0000 7f4d8b9f6640 -1 osd.5 0 waiting for initial osdmap 2024-09-18T12:25:20.883 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-09-18T12:25:20.882+0000 7f6125fe8640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T12:25:20.883 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-09-18T12:25:20.881+0000 7f65c51e8640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T12:25:20.884 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-09-18T12:25:20.882+0000 7f4d849e8640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T12:25:20.889 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-09-18T12:25:20.887+0000 7f3a4a7f9640 -1 osd.0 0 waiting for initial osdmap 2024-09-18T12:25:20.890 INFO:tasks.ceph.osd.3.smithi144.stderr:2024-09-18T12:25:20.888+0000 7fa5729f9640 -1 osd.3 0 waiting for initial osdmap 2024-09-18T12:25:20.893 INFO:tasks.ceph.osd.3.smithi144.stderr:2024-09-18T12:25:20.892+0000 7fa56a084640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T12:25:20.893 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-09-18T12:25:20.891+0000 7f3a41fe8640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T12:25:21.113 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-09-18T12:25:21.111+0000 7f024dbf9640 -1 osd.2 0 waiting for initial osdmap 2024-09-18T12:25:21.116 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-09-18T12:25:21.114+0000 7f02453e8640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T12:25:23.993 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T12:25:24.082 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.080+0000 7f8e7355d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:24.082 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.080+0000 7f8e7355d640 10 monclient: build_initial_monmap 2024-09-18T12:25:24.082 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.080+0000 7f8e7355d640 10 monclient: monmap: 2024-09-18T12:25:24.082 INFO:teuthology.misc.health.smithi116.stderr:epoch 0 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:25:24.082176+0000 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:25:24.082176+0000 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:24.083 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:24.085 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 1 Processor -- start 2024-09-18T12:25:24.085 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 1 -- start start 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient: init 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8e6c09e610 0x7f8e6c09e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e6c09e610 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c09c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e6c099c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e6c09c540 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c09b050 con 0x7f8e6c09c540 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c09b360 con 0x7f8e6c09e610 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.086 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c09b890 con 0x7f8e6c099c60 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e7355d640 10 monclient(hunting): authenticate will time out at 751.403850s 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.081+0000 7f8e70af8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c09c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c09c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45914/0 (socket says 172.21.15.116:45914) 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 1 -- 172.21.15.116:0/4088869443 learned_addr learned my addr 172.21.15.116:0/4088869443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): get_auth_request con 0x7f8e6c099c60 auth_method 0 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.087 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e712f9640 1 --2- 172.21.15.116:0/4088869443 >> v2:172.21.15.144:3300/0 conn(0x7f8e6c09e610 0x7f8e6c09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e712f9640 10 monclient(hunting): get_auth_request con 0x7f8e6c09e610 auth_method 0 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e712f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e712f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e712f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 1 -- 172.21.15.116:0/4088869443 >> v2:172.21.15.144:3300/0 conn(0x7f8e6c09e610 msgr2=0x7f8e6c09e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 1 --2- 172.21.15.116:0/4088869443 >> v2:172.21.15.144:3300/0 conn(0x7f8e6c09e610 0x7f8e6c09e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 1 -- 172.21.15.116:0/4088869443 >> v1:172.21.15.144:6789/0 conn(0x7f8e6c09c540 legacy=0x7f8e6c09c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient: _finish_hunting 0 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 1 monclient: found mon.noname-b 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:24.088 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.082+0000 7f8e70af8640 1 -- 172.21.15.116:0/4088869443 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c09f250 con 0x7f8e6c099c60 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e70af8640 1 -- 172.21.15.116:0/4088869443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 msgr2=0x7f8e6c09c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e70af8640 1 --2- 172.21.15.116:0/4088869443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c09c000 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009910 tx=0x7f8e600091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7456600ed42130ea server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 1 -- 172.21.15.116:0/4088869443 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60032440 con 0x7f8e6c099c60 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: dump: 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.094 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: _finish_auth 0 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 1 -- 172.21.15.116:0/4088869443 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e600325a0 con 0x7f8e6c099c60 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 5 monclient: authenticate success, global_id 4184 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:24.095 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 1 -- 172.21.15.116:0/4088869443 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60032940 con 0x7f8e6c099c60 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 4 set_mon_vals no callback set 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e5ffff640 10 monclient: dump: 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.096 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 10 monclient: shutdown 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 -- 172.21.15.116:0/4088869443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 msgr2=0x7f8e6c09c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 --2- 172.21.15.116:0/4088869443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c09c000 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009910 tx=0x7f8e600091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 -- 172.21.15.116:0/4088869443 shutdown_connections 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 --2- 172.21.15.116:0/4088869443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c09c000 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 --2- 172.21.15.116:0/4088869443 >> v2:172.21.15.144:3300/0 conn(0x7f8e6c09e610 0x7f8e6c09e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 -- 172.21.15.116:0/4088869443 >> 172.21.15.116:0/4088869443 conn(0x7f8e6c067b40 msgr2=0x7f8e6c0a2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 -- 172.21.15.116:0/4088869443 shutdown_connections 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.085+0000 7f8e7355d640 1 -- 172.21.15.116:0/4088869443 wait complete. 2024-09-18T12:25:24.097 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.086+0000 7f8e7355d640 10 monclient: build_initial_monmap 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.086+0000 7f8e7355d640 10 monclient: monmap: 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:epoch 0 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:created 0.000000 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.086+0000 7f8e7355d640 1 Processor -- start 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.086+0000 7f8e7355d640 1 -- start start 2024-09-18T12:25:24.098 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.086+0000 7f8e7355d640 10 monclient: init 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c13ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e6c099c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c09c540 0x7f8e6c131a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e6c09c540 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e6c09e610 0x7f8e6c11b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e6c09e610 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c11ee80 con 0x7f8e6c09c540 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c098ce0 con 0x7f8e6c09e610 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.099 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c11eba0 con 0x7f8e6c099c60 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e7355d640 10 monclient(hunting): authenticate will time out at 751.409397s 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c13ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c09c540 0x7f8e6c131a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c13ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45922/0 (socket says 172.21.15.116:45922) 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 -- 172.21.15.116:0/1818056378 learned_addr learned my addr 172.21.15.116:0/1818056378 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): get_auth_request con 0x7f8e6c099c60 auth_method 0 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 10 monclient(hunting): get_auth_request con 0x7f8e6c09c540 auth_method 0 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.100 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e6c09e610 0x7f8e6c11b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 10 monclient(hunting): get_auth_request con 0x7f8e6c09e610 auth_method 0 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e70af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.101 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e71afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 -- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e6c09e610 msgr2=0x7f8e6c11b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e6c09e610 0x7f8e6c11b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 -- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c09c540 msgr2=0x7f8e6c131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c09c540 0x7f8e6c131a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient: _finish_hunting 0 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 monclient: found mon.noname-c 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.087+0000 7f8e712f9640 1 -- 172.21.15.116:0/1818056378 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e60009450 con 0x7f8e6c099c60 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e712f9640 1 -- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 msgr2=0x7f8e6c13ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:24.102 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e712f9640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c13ed60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800d890 tx=0x7f8e68008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/1818056378 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6800de20 con 0x7f8e6c099c60 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: dump: 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.103 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/1818056378 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6800bae0 con 0x7f8e6c099c60 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e7355d640 5 monclient: authenticate success, global_id 4187 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e7355d640 10 monclient: _renew_subs 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/1818056378 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6800be80 con 0x7f8e6c099c60 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e7355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c13f2a0 con 0x7f8e6c099c60 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5d7fa640 10 monclient: dump: 2024-09-18T12:25:24.104 INFO:teuthology.misc.health.smithi116.stderr:epoch 1 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr: 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e5effd640 4 set_mon_vals no callback set 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e7355d640 10 monclient: _renew_subs 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e7355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:24.105 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.090+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c11dde0 con 0x7f8e6c099c60 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/1818056378 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 4) v1 ==== 44625+0+0 (secure 0 0 0) 0x7f8e68014640 con 0x7f8e6c099c60 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e7355d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e7355d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e7355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e5d7fa640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 0x7f8e5003a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c06ec30 con 0x7f8e6c099c60 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/1818056378 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3989+0+0 (secure 0 0 0) 0x7f8e6804ba00 con 0x7f8e6c099c60 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e70af8640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 0x7f8e5003a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e70af8640 10 monclient: get_auth_request con 0x7f8e50038430 auth_method 0 2024-09-18T12:25:24.106 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e70af8640 1 -- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 msgr2=0x7f8e5003a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:25:24.107 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e70af8640 1 -- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 msgr2=0x7f8e5003a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:25:24.107 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e70af8640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 0x7f8e5003a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:24.107 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.091+0000 7f8e70af8640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 0x7f8e5003a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:24.107 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.096+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/1818056378 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8e6c06ec30 con 0x7f8e6c099c60 2024-09-18T12:25:24.107 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.096+0000 7f8e5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:24.107 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.096+0000 7f8e5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:24.252 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.249+0000 7f8e7355d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:24.252 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.249+0000 7f8e7355d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:24.252 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.249+0000 7f8e7355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:24.252 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.249+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e6c06ec30 con 0x7f8e6c099c60 2024-09-18T12:25:24.253 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.250+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/1818056378 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+9838 (secure 0 0 0) 0x7f8e6c06ec30 con 0x7f8e6c099c60 2024-09-18T12:25:24.253 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.250+0000 7f8e5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:24.253 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.250+0000 7f8e5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:24.253 INFO:teuthology.misc.health.smithi116.stdout: 2024-09-18T12:25:24.253 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":13,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:25:23.905991+0000","last_up_change":"2024-09-18T12:25:21.884529+0000","last_in_change":"2024-09-18T12:25:15.496296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T12:25:22.596367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":31578},{"type":"v1","addr":"172.21.15.116:6817","nonce":31578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":31578},{"type":"v1","addr":"172.21.15.116:6819","nonce":31578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":31578},{"type":"v1","addr":"172.21.15.116:6823","nonce":31578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":31578},{"type":"v1","addr":"172.21.15.116:6821","nonce":31578}]},"public_addr":"172.21.15.116:6817/31578","cluster_addr":"172.21.15.116:6819/31578","heartbeat_back_addr":"172.21.15.116:6823/31578","heartbeat_front_addr":"172.21.15.116:6821/31578","state":["exists","up"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":31579},{"type":"v1","addr":"172.21.15.116:6801","nonce":31579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":31579},{"type":"v1","addr":"172.21.15.116:6803","nonce":31579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":31579},{"type":"v1","addr":"172.21.15.116:6807","nonce":31579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":31579},{"type":"v1","addr":"172.21.15.116:6805","nonce":31579}]},"public_addr":"172.21.15.116:6801/31579","cluster_addr":"172.21.15.116:6803/31579","heartbeat_back_addr":"172.21.15.116:6807/31579","heartbeat_front_addr":"172.21.15.116:6805/31579","state":["exists","up"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":31580},{"type":"v1","addr":"172.21.15.116:6809","nonce":31580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":31580},{"type":"v1","addr":"172.21.15.116:6811","nonce":31580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":31580},{"type":"v1","addr":"172.21.15.116:6815","nonce":31580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":31580},{"type":"v1","addr":"172.21.15.116:6813","nonce":31580}]},"public_addr":"172.21.15.116:6809/31580","cluster_addr":"172.21.15.116:6811/31580","heartbeat_back_addr":"172.21.15.116:6815/31580","heartbeat_front_addr":"172.21.15.116:6813/31580","state":["exists","up"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":32112},{"type":"v1","addr":"172.21.15.144:6806","nonce":32112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":32112},{"type":"v1","addr":"172.21.15.144:6811","nonce":32112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":32112},{"type":"v1","addr":"172.21.15.144:6818","nonce":32112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":32112},{"type":"v1","addr":"172.21.15.144:6815","nonce":32112}]},"public_addr":"172.21.15.144:6806/32112","cluster_addr":"172.21.15.144:6811/32112","heartbeat_back_addr":"172.21.15.144:6818/32112","heartbeat_front_addr":"172.21.15.144:6815/32112","state":["exists","up"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":32113},{"type":"v1","addr":"172.21.15.144:6801","nonce":32113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":32113},{"type":"v1","addr":"172.21.15.144:6803","nonce":32113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":32113},{"type":"v1","addr":"172.21.15.144:6810","nonce":32113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6805","nonce":32113},{"type":"v1","addr":"172.21.15.144:6807","nonce":32113}]},"public_addr":"172.21.15.144:6801/32113","cluster_addr":"172.21.15.144:6803/32113","heartbeat_back_addr":"172.21.15.144:6810/32113","heartbeat_front_addr":"172.21.15.144:6807/32113","state":["exists","up"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":32114},{"type":"v1","addr":"172.21.15.144:6814","nonce":32114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":32114},{"type":"v1","addr":"172.21.15.144:6819","nonce":32114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":32114},{"type":"v1","addr":"172.21.15.144:6823","nonce":32114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":32114},{"type":"v1","addr":"172.21.15.144:6821","nonce":32114}]},"public_addr":"172.21.15.144:6814/32114","cluster_addr":"172.21.15.144:6819/32114","heartbeat_back_addr":"172.21.15.144:6823/32114","heartbeat_front_addr":"172.21.15.144:6821/32114","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.664684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.608350+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.689049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.706015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.672781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.692341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:25:24.254 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 msgr2=0x7f8e5003a8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:24.254 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 0x7f8e5003a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.255 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 10 monclient: shutdown 2024-09-18T12:25:24.255 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:24.255 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 msgr2=0x7f8e6c13ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.255 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c13ed60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800d890 tx=0x7f8e68008040 comp rx=0 tx=0).stop 2024-09-18T12:25:24.255 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 shutdown_connections 2024-09-18T12:25:24.255 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8e50038430 0x7f8e5003a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.256 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e6c09e610 0x7f8e6c11b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.256 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c09c540 0x7f8e6c131a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.256 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 --2- 172.21.15.116:0/1818056378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e6c099c60 0x7f8e6c13ed60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.256 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 >> 172.21.15.116:0/1818056378 conn(0x7f8e6c067b40 msgr2=0x7f8e6c0973d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:24.256 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 shutdown_connections 2024-09-18T12:25:24.256 INFO:teuthology.misc.health.smithi116.stderr:2024-09-18T12:25:24.252+0000 7f8e7355d640 1 -- 172.21.15.116:0/1818056378 wait complete. 2024-09-18T12:25:24.263 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-18T12:25:24.263 INFO:tasks.ceph:Creating RBD pool 2024-09-18T12:25:24.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-18T12:25:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.353+0000 7faf0ef5d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.353+0000 7faf0ef5d640 10 monclient: build_initial_monmap 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.353+0000 7faf0ef5d640 10 monclient: monmap: 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:24.355578+0000 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:24.355578+0000 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:24.356 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.354+0000 7faf0ef5d640 1 Processor -- start 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.354+0000 7faf0ef5d640 1 -- start start 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.354+0000 7faf0ef5d640 10 monclient: init 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faf0809c440 0x7faf0809c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7faf0809c440 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf0806ed40 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faf0809dfd0 0x7faf08139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7faf0809dfd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf08069fe0 con 0x7faf0806ed40 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0806a610 con 0x7faf0809c440 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf08068da0 con 0x7faf0809dfd0 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ef5d640 10 monclient(hunting): authenticate will time out at 751.677183s 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faf0809c440 0x7faf0809c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faf0809c440 0x7faf0809c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53856/0 (socket says 172.21.15.116:53856) 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faefffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf0806ed40 0x7faf0809d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59970/0 (socket says 172.21.15.116:59970) 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 -- 172.21.15.116:0/3941481896 learned_addr learned my addr 172.21.15.116:0/3941481896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): get_auth_request con 0x7faf0809c440 auth_method 0 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 -- 172.21.15.116:0/3941481896 >> v2:172.21.15.144:3300/0 conn(0x7faf0809dfd0 msgr2=0x7faf08139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 --2- 172.21.15.116:0/3941481896 >> v2:172.21.15.144:3300/0 conn(0x7faf0809dfd0 0x7faf08139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 -- 172.21.15.116:0/3941481896 >> v1:172.21.15.116:6789/0 conn(0x7faf0806ed40 legacy=0x7faf0809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient: _finish_hunting 0 2024-09-18T12:25:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 monclient: found mon.noname-a 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.355+0000 7faf0ccf9640 1 -- 172.21.15.116:0/3941481896 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08069d00 con 0x7faf0809c440 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ccf9640 1 -- 172.21.15.116:0/3941481896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0809c440 msgr2=0x7faf0809c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ccf9640 1 --2- 172.21.15.116:0/3941481896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0809c440 0x7faf0809c810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf080a3f20 tx=0x7faf040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168148eb29afef50 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 1 -- 172.21.15.116:0/3941481896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04032440 con 0x7faf0809c440 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: dump: 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 1 -- 172.21.15.116:0/3941481896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf040325a0 con 0x7faf0809c440 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 5 monclient: authenticate success, global_id 4203 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 1 -- 172.21.15.116:0/3941481896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04032980 con 0x7faf0809c440 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 4 set_mon_vals no callback set 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faeff7fe640 10 monclient: dump: 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.372 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 10 monclient: shutdown 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 1 -- 172.21.15.116:0/3941481896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0809c440 msgr2=0x7faf0809c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.358+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/3941481896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0809c440 0x7faf0809c810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf080a3f20 tx=0x7faf040091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 1 -- 172.21.15.116:0/3941481896 shutdown_connections 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/3941481896 >> v2:172.21.15.144:3300/0 conn(0x7faf0809dfd0 0x7faf08139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/3941481896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0809c440 0x7faf0809c810 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 1 -- 172.21.15.116:0/3941481896 >> 172.21.15.116:0/3941481896 conn(0x7faf08067d30 msgr2=0x7faf080a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 1 -- 172.21.15.116:0/3941481896 shutdown_connections 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 1 -- 172.21.15.116:0/3941481896 wait complete. 2024-09-18T12:25:24.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 10 monclient: build_initial_monmap 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 10 monclient: monmap: 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.359+0000 7faf0ef5d640 1 Processor -- start 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 1 -- start start 2024-09-18T12:25:24.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient: init 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 0x7faf08141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7faf0806ed40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf0809c440 0x7faf08131f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7faf0809c440 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf0809dfd0 0x7faf0806cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7faf0809dfd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08142270 con 0x7faf0806ed40 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0806e710 con 0x7faf0809dfd0 2024-09-18T12:25:24.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0809b5d0 con 0x7faf0809c440 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ef5d640 10 monclient(hunting): authenticate will time out at 751.682529s 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 0x7faf08141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf0809c440 0x7faf08131f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 0x7faf08141370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53862/0 (socket says 172.21.15.116:53862) 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 -- 172.21.15.116:0/2255910780 learned_addr learned my addr 172.21.15.116:0/2255910780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): get_auth_request con 0x7faf0806ed40 auth_method 0 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 10 monclient(hunting): get_auth_request con 0x7faf0809c440 auth_method 0 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0d4fa640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf0809dfd0 0x7faf0806cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient(hunting): handle_auth_done global_id 4206 payload 824 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 -- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf0809c440 msgr2=0x7faf08131f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf0809c440 0x7faf08131f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 -- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf0809dfd0 msgr2=0x7faf0806cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf0809dfd0 0x7faf0806cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient: _finish_hunting 0 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 monclient: found mon.noname-a 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0ccf9640 1 -- 172.21.15.116:0/2255910780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf04009450 con 0x7faf0806ed40 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0d4fa640 10 monclient: get_auth_request con 0x7faf0809dfd0 auth_method 0 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faf0d4fa640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf0809dfd0 0x7faf0806cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.360+0000 7faefffff640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf0809c440 0x7faf08131f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ccf9640 1 -- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 msgr2=0x7faf08141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ccf9640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 0x7faf08141370 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf04009910 tx=0x7faf04032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:24.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 1 -- 172.21.15.116:0/2255910780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf040318c0 con 0x7faf0806ed40 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: dump: 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.379 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: _finish_auth 0 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ef5d640 5 monclient: authenticate success, global_id 4206 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 1 -- 172.21.15.116:0/2255910780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf04031a20 con 0x7faf0806ed40 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 1 -- 172.21.15.116:0/2255910780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04031de0 con 0x7faf0806ed40 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefcff9640 10 monclient: dump: 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.380 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ef5d640 10 monclient: _renew_subs 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0811f320 con 0x7faf0806ed40 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faefeffd640 4 set_mon_vals no callback set 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ef5d640 10 monclient: _renew_subs 2024-09-18T12:25:24.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.363+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0811e0e0 con 0x7faf0806ed40 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faefcff9640 1 -- 172.21.15.116:0/2255910780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 44625+0+0 (secure 0 0 0) 0x7faf0403c610 con 0x7faf0806ed40 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faefcff9640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faef4038430 0x7faef403a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faefcff9640 1 -- 172.21.15.116:0/2255910780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3989+0+0 (secure 0 0 0) 0x7faf0406bc70 con 0x7faf0806ed40 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faf0ef5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faf0ef5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faf0ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0805ed70 con 0x7faf0806ed40 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faefffff640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faef4038430 0x7faef403a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.364+0000 7faefffff640 10 monclient: get_auth_request con 0x7faef4038430 auth_method 0 2024-09-18T12:25:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.367+0000 7faefffff640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faef4038430 0x7faef403a8e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf0809d100 tx=0x7faef8007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.369+0000 7faefcff9640 1 -- 172.21.15.116:0/2255910780 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7faf04037080 con 0x7faf0806ed40 2024-09-18T12:25:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.369+0000 7faefcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.369+0000 7faefcff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.522+0000 7faf0ef5d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T12:25:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.522+0000 7faf0ef5d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T12:25:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.522+0000 7faf0ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:24.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.522+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7faf0806f430 con 0x7faf0806ed40 2024-09-18T12:25:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.922+0000 7faefcff9640 1 -- 172.21.15.116:0/2255910780 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7faf04049020 con 0x7faf0806ed40 2024-09-18T12:25:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.922+0000 7faefcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T12:25:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.922+0000 7faefcff9640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-18T12:25:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'rbd' created 2024-09-18T12:25:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faef4038430 msgr2=0x7faef403a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faef4038430 0x7faef403a8e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf0809d100 tx=0x7faef8007e40 comp rx=0 tx=0).stop 2024-09-18T12:25:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 10 monclient: shutdown 2024-09-18T12:25:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 msgr2=0x7faf08141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 0x7faf08141370 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf04009910 tx=0x7faf04032000 comp rx=0 tx=0).stop 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 shutdown_connections 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faef4038430 0x7faef403a8e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faf0809dfd0 0x7faf0806cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faf0809c440 0x7faf08131f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 --2- 172.21.15.116:0/2255910780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf0806ed40 0x7faf08141370 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 >> 172.21.15.116:0/2255910780 conn(0x7faf08067d30 msgr2=0x7faf0809bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 shutdown_connections 2024-09-18T12:25:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.925+0000 7faf0ef5d640 1 -- 172.21.15.116:0/2255910780 wait complete. 2024-09-18T12:25:24.937 DEBUG:teuthology.orchestra.run.smithi116:> rbd --cluster ceph pool init rbd 2024-09-18T12:25:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.985+0000 7f41d26cbb80 10 monclient: get_monmap_and_config 2024-09-18T12:25:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.985+0000 7f41d26cbb80 10 monclient: build_initial_monmap 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.985+0000 7f41d26cbb80 10 monclient: monmap: 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:24.987405+0000 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:24.987405+0000 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:24.988 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.986+0000 7f41d26cbb80 1 Processor -- start 2024-09-18T12:25:24.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.986+0000 7f41d26cbb80 1 -- start start 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.986+0000 7f41d26cbb80 10 monclient: init 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.986+0000 7f41d26cbb80 10 monclient: _reopen_session rank -1 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.986+0000 7f41d26cbb80 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 10 monclient(hunting): picked mon.noname-a-legacy con 0x560beb9ee9d0 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 10 monclient(hunting): picked mon.noname-c-legacy con 0x560beb9f0b30 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beba887c0 0x560beba8aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 10 monclient(hunting): picked mon.noname-b con 0x560beba887c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x560beb9110e0 con 0x560beb9ee9d0 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x560beb8ea270 con 0x560beb9f0b30 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x560beb94a260 con 0x560beba887c0 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 10 monclient(hunting): _renew_subs 2024-09-18T12:25:24.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41d26cbb80 10 monclient(hunting): authenticate will time out at 752.309086s 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cf44b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x560beb9ee9d0 0x560beb9f0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59984/0 (socket says 172.21.15.116:59984) 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cf44b640 1 -- 172.21.15.116:0/3451871774 learned_addr learned my addr 172.21.15.116:0/3451871774 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 1 --2- 172.21.15.116:0/3451871774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beba887c0 0x560beba8aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): get_auth_request con 0x560beba887c0 auth_method 0 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-09-18T12:25:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3451871774 >> v1:172.21.15.144:6789/0 conn(0x560beb9f0b30 legacy=0x560beba870a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3451871774 >> v1:172.21.15.116:6789/0 conn(0x560beb9ee9d0 legacy=0x560beb9f0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient: _finish_hunting 0 2024-09-18T12:25:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 1 monclient: found mon.noname-b 2024-09-18T12:25:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.987+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3451871774 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560beb8f70e0 con 0x560beba887c0 2024-09-18T12:25:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3451871774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beba887c0 msgr2=0x560beba8aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41cfc4c640 1 --2- 172.21.15.116:0/3451871774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beba887c0 0x560beba8aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41c000a5d0 tx=0x7f41c0045b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=628e1559b200a5b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 1 -- 172.21.15.116:0/3451871774 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0050460 con 0x560beba887c0 2024-09-18T12:25:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: dump: 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: _finish_auth 0 2024-09-18T12:25:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 5 monclient: authenticate success, global_id 4199 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 1 -- 172.21.15.116:0/3451871774 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c00505c0 con 0x560beba887c0 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 10 monclient: get_monmap_and_config success 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 1 -- 172.21.15.116:0/3451871774 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0050770 con 0x560beba887c0 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 4 set_mon_vals no callback set 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41ce449640 10 monclient: dump: 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 10 monclient: shutdown 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3451871774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beba887c0 msgr2=0x560beba8aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.991+0000 7f41d26cbb80 1 --2- 172.21.15.116:0/3451871774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beba887c0 0x560beba8aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41c000a5d0 tx=0x7f41c0045b90 comp rx=0 tx=0).stop 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3451871774 shutdown_connections 2024-09-18T12:25:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 1 --2- 172.21.15.116:0/3451871774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beba887c0 0x560beba8aba0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3451871774 >> 172.21.15.116:0/3451871774 conn(0x560beb9f1cb0 msgr2=0x560beb9f20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3451871774 shutdown_connections 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3451871774 wait complete. 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 10 monclient: build_initial_monmap 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 10 monclient: monmap: 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 1 Processor -- start 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 1 -- start start 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.992+0000 7f41d26cbb80 10 monclient: init 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient: _reopen_session rank -1 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560beb9ee9d0 0x560beba6d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): picked mon.noname-a con 0x560beb9ee9d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beb9f0b30 0x560beba6d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): picked mon.noname-c con 0x560beb9f0b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560beba887c0 0x560beba6dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): picked mon.noname-b con 0x560beba887c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560beb945f40 con 0x560beb9ee9d0 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x560beb908ba0 con 0x560beba887c0 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x560beb908840 con 0x560beb9f0b30 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): _renew_subs 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41d26cbb80 10 monclient(hunting): authenticate will time out at 752.315394s 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560beb9ee9d0 0x560beba6d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beb9f0b30 0x560beba6d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beb9f0b30 0x560beba6d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45936/0 (socket says 172.21.15.116:45936) 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560beb9ee9d0 0x560beba6d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53878/0 (socket says 172.21.15.116:53878) 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 -- 172.21.15.116:0/3511417273 learned_addr learned my addr 172.21.15.116:0/3511417273 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): get_auth_request con 0x560beb9f0b30 auth_method 0 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 10 monclient(hunting): get_auth_request con 0x560beb9ee9d0 auth_method 0 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560beba887c0 0x560beba6dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 10 monclient(hunting): get_auth_request con 0x560beba887c0 auth_method 0 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cfc4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 -- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560beba887c0 msgr2=0x560beba6dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x560beba887c0 0x560beba6dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:25.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 -- 172.21.15.116:0/3511417273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560beb9ee9d0 msgr2=0x560beba6d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560beb9ee9d0 0x560beba6d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient: _finish_hunting 0 2024-09-18T12:25:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 monclient: found mon.noname-c 2024-09-18T12:25:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cec4a640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c000a0e0 con 0x560beb9f0b30 2024-09-18T12:25:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.993+0000 7f41cf44b640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560beb9ee9d0 0x560beba6d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41cec4a640 1 -- 172.21.15.116:0/3511417273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beb9f0b30 msgr2=0x560beba6d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41cec4a640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beb9f0b30 0x560beba6d580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41b8002ba0 tx=0x7f41b80028b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8009de0 con 0x560beb9f0b30 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: dump: 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41d26cbb80 5 monclient: authenticate success, global_id 4202 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b8008030 con 0x560beb9f0b30 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b800f650 con 0x560beb9f0b30 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41d26cbb80 10 monclient: _renew_subs 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41d26cbb80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560beb8ea600 con 0x560beb9f0b30 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41cdc48640 4 set_mon_vals no callback set 2024-09-18T12:25:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41b77fe640 10 monclient: dump: 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:25.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41d26cbb80 10 monclient: _renew_subs 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41d26cbb80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.996+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560beb8ea060 con 0x560beb9f0b30 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 4) v1 ==== 44625+0+0 (secure 0 0 0) 0x7f41b800fac0 con 0x560beb9f0b30 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41b77fe640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41b0036d30 0x7f41b00391e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4331+0+0 (secure 0 0 0) 0x7f41b8049c30 con 0x560beb9f0b30 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41d26cbb80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41d26cbb80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41cf44b640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41b0036d30 0x7f41b00391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41d26cbb80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x560beba6ebe0 con 0x560beb9f0b30 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:24.997+0000 7f41cf44b640 10 monclient: get_auth_request con 0x7f41b0036d30 auth_method 0 2024-09-18T12:25:25.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.000+0000 7f41cf44b640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41b0036d30 0x7f41b00391e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41c4005fd0 tx=0x7f41c4005910 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.921+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 141+0+0 (secure 0 0 0) 0x7f41b801fd20 con 0x560beb9f0b30 2024-09-18T12:25:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.921+0000 7f41b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T12:25:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.921+0000 7f41b77fe640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-18T12:25:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.921+0000 7f41d26cbb80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.922+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x560beb908630 con 0x560beb9f0b30 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.927+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41b801fd20 con 0x560beb9f0b30 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.927+0000 7f41b77fe640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.927+0000 7f41cdc48640 10 monclient: _renew_subs 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.927+0000 7f41cdc48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.927+0000 7f41cdc48640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=15}) v3 -- 0x7f41a8002990 con 0x560beb9f0b30 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.928+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 650+0+0 (secure 0 0 0) 0x7f41b801fac0 con 0x560beb9f0b30 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.928+0000 7f41d26cbb80 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x560beba97150 0x560beba99530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.928+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x560beba99a70 con 0x560beba97150 2024-09-18T12:25:25.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.928+0000 7f41cfc4c640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x560beba97150 0x560beba99530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:25.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.928+0000 7f41cfc4c640 10 monclient: get_auth_request con 0x560beba97150 auth_method 0 2024-09-18T12:25:25.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.933+0000 7f41cfc4c640 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x560beba97150 0x560beba99530 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.939+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3511417273 <== osd.5 v2:172.21.15.144:6812/32114 1 ==== osd_op_reply(1 rbd_trash [create] v15'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f41c0050860 con 0x560beba97150 2024-09-18T12:25:25.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.941+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e15) v8 -- 0x560bebae6fb0 con 0x560beba97150 2024-09-18T12:25:25.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.942+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3511417273 <== osd.5 v2:172.21.15.144:6812/32114 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f41c0050860 con 0x560beba97150 2024-09-18T12:25:25.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.942+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e15) v8 -- 0x560bebae6090 con 0x560beba97150 2024-09-18T12:25:25.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.942+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3511417273 <== osd.5 v2:172.21.15.144:6812/32114 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f41c0050860 con 0x560beba97150 2024-09-18T12:25:25.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.942+0000 7f41cd447640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:25.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:25.942+0000 7f41cd447640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f41ac001ee0 con 0x560beb9f0b30 2024-09-18T12:25:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.100+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 9 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f41b800eb80 con 0x560beb9f0b30 2024-09-18T12:25:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.554+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 10 ==== pool_op_reply(tid 4 (0) Success v16) v1 ==== 55+0+0 (secure 0 0 0) 0x7f41b802c5f0 con 0x560beb9f0b30 2024-09-18T12:25:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.554+0000 7f41b77fe640 10 monclient: _renew_subs 2024-09-18T12:25:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.554+0000 7f41b77fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.554+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=16}) v3 -- 0x7f41b003d020 con 0x560beb9f0b30 2024-09-18T12:25:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.554+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 11 ==== osd_map(16..16 src has 1..16) v4 ==== 618+0+0 (secure 0 0 0) 0x7f41b800ee30 con 0x560beb9f0b30 2024-09-18T12:25:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.554+0000 7f41cdc48640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f41a8005450 con 0x560beba97150 2024-09-18T12:25:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.558+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3511417273 <== osd.5 v2:172.21.15.144:6812/32114 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v16'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f41c0050860 con 0x560beba97150 2024-09-18T12:25:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.558+0000 7f41cd447640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.558+0000 7f41cd447640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v16) v4 -- 0x7f41ac003360 con 0x560beb9f0b30 2024-09-18T12:25:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.994+0000 7f41b67fc640 10 monclient: tick 2024-09-18T12:25:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.994+0000 7f41b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:27.994+0000 7f41b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.108+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 12 ==== pool_op_reply(tid 6 (0) Success v17) v1 ==== 43+0+0 (secure 0 0 0) 0x7f41b802c7d0 con 0x560beb9f0b30 2024-09-18T12:25:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.108+0000 7f41b77fe640 10 monclient: _renew_subs 2024-09-18T12:25:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.108+0000 7f41b77fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.108+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=17}) v3 -- 0x7f41b003d020 con 0x560beb9f0b30 2024-09-18T12:25:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.108+0000 7f41b77fe640 1 -- 172.21.15.116:0/3511417273 <== mon.2 v2:172.21.15.116:3301/0 13 ==== osd_map(17..17 src has 1..17) v4 ==== 646+0+0 (secure 0 0 0) 0x7f41b8017a30 con 0x560beb9f0b30 2024-09-18T12:25:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.108+0000 7f41cdc48640 1 -- 172.21.15.116:0/3511417273 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) v8 -- 0x7f41a8005820 con 0x560beba97150 2024-09-18T12:25:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3511417273 <== osd.5 v2:172.21.15.144:6812/32114 5 ==== osd_op_reply(7 rbd_info [write 0~19] v17'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f41c0050860 con 0x560beba97150 2024-09-18T12:25:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x560beba97150 msgr2=0x560beba99530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x560beba97150 0x560beba99530 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41b0036d30 msgr2=0x7f41b00391e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41b0036d30 0x7f41b00391e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41c4005fd0 tx=0x7f41c4005910 comp rx=0 tx=0).stop 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 10 monclient: shutdown 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beb9f0b30 msgr2=0x560beba6d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 --2- 172.21.15.116:0/3511417273 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x560beb9f0b30 0x560beba6d580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41b8002ba0 tx=0x7f41b80028b0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41cfc4c640 1 -- 172.21.15.116:0/3511417273 reap_dead start 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 shutdown_connections 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 >> 172.21.15.116:0/3511417273 conn(0x560beb9f1cb0 msgr2=0x560beb9eb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 shutdown_connections 2024-09-18T12:25:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.112+0000 7f41d26cbb80 1 -- 172.21.15.116:0/3511417273 wait complete. 2024-09-18T12:25:28.119 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T12:25:28.119 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T12:25:28.119 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T12:25:28.119 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T12:25:28.120 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T12:25:28.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.218+0000 7f5efc8a8640 10 monclient: get_monmap_and_config 2024-09-18T12:25:28.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.218+0000 7f5efc8a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:28.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.219+0000 7f5efc8a8640 10 monclient: monmap: 2024-09-18T12:25:28.221 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:28.220659+0000 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:28.220659+0000 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:28.222 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.219+0000 7f5efc8a8640 1 Processor -- start 2024-09-18T12:25:28.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.219+0000 7f5efc8a8640 1 -- start start 2024-09-18T12:25:28.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.219+0000 7f5efc8a8640 10 monclient: init 2024-09-18T12:25:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T12:25:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5ef80974f0 0x7f5ef812ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef80974f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ef809a8a0 0x7f5ef809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef809a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ef812dd40 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef80978c0 con 0x7f5ef812dd40 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef809b220 con 0x7f5ef809a8a0 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef809ca20 con 0x7f5ef80974f0 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5efc8a8640 10 monclient(hunting): authenticate will time out at 755.542364s 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ef809a8a0 0x7f5ef809ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef6d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5ef812dd40 0x7f5ef812e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57032/0 (socket says 172.21.15.116:57032) 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef6d9d640 1 -- 172.21.15.116:0/1281450068 learned_addr learned my addr 172.21.15.116:0/1281450068 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): get_auth_request con 0x7f5ef809a8a0 auth_method 0 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 1 --2- 172.21.15.116:0/1281450068 >> v2:172.21.15.144:3300/0 conn(0x7f5ef80974f0 0x7f5ef812ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 10 monclient(hunting): get_auth_request con 0x7f5ef80974f0 auth_method 0 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-09-18T12:25:28.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/1281450068 >> v2:172.21.15.144:3300/0 conn(0x7f5ef80974f0 msgr2=0x7f5ef812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:28.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 1 --2- 172.21.15.116:0/1281450068 >> v2:172.21.15.144:3300/0 conn(0x7f5ef80974f0 0x7f5ef812ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/1281450068 >> v1:172.21.15.116:6789/0 conn(0x7f5ef812dd40 legacy=0x7f5ef812e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:28.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:28.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 1 monclient: found mon.noname-a 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.220+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/1281450068 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef8098ad0 con 0x7f5ef809a8a0 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/1281450068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 msgr2=0x7f5ef809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef5d9b640 1 --2- 172.21.15.116:0/1281450068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef809ace0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80768c0 tx=0x7f5eec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9a8cf98ed43381 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 1 -- 172.21.15.116:0/1281450068 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec02adf0 con 0x7f5ef809a8a0 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: dump: 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.237 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: _finish_auth 0 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 1 -- 172.21.15.116:0/1281450068 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec032440 con 0x7f5ef809a8a0 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5efc8a8640 5 monclient: authenticate success, global_id 4221 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5efc8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:28.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5efc8a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 1 -- 172.21.15.116:0/1281450068 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec032820 con 0x7f5ef809a8a0 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5efc8a8640 4 set_mon_vals no callback set 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.223+0000 7f5ef559a640 10 monclient: dump: 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.239 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 10 monclient: shutdown 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 -- 172.21.15.116:0/1281450068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 msgr2=0x7f5ef809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/1281450068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef809ace0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80768c0 tx=0x7f5eec0090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 -- 172.21.15.116:0/1281450068 shutdown_connections 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/1281450068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef809ace0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/1281450068 >> v2:172.21.15.144:3300/0 conn(0x7f5ef80974f0 0x7f5ef812ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 -- 172.21.15.116:0/1281450068 >> 172.21.15.116:0/1281450068 conn(0x7f5ef80672b0 msgr2=0x7f5ef80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 -- 172.21.15.116:0/1281450068 shutdown_connections 2024-09-18T12:25:28.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.224+0000 7f5efc8a8640 1 -- 172.21.15.116:0/1281450068 wait complete. 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient: monmap: 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:28.241 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 Processor -- start 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 -- start start 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient: init 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5ef80974f0 0x7f5ef806d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef80974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef806d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef809a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef812dd40 0x7f5ef806dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef812dd40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:28.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef8127ab0 con 0x7f5ef809a8a0 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef809e0b0 con 0x7f5ef812dd40 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81189c0 con 0x7f5ef80974f0 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5efc8a8640 10 monclient(hunting): authenticate will time out at 755.547892s 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5ef5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef806d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.225+0000 7f5ef659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5ef80974f0 0x7f5ef806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef806d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51878/0 (socket says 172.21.15.116:51878) 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/3616410376 learned_addr learned my addr 172.21.15.116:0/3616410376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:28.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): get_auth_request con 0x7f5ef809a8a0 auth_method 0 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef659c640 10 monclient(hunting): get_auth_request con 0x7f5ef80974f0 auth_method 0 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef6d9d640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef812dd40 0x7f5ef806dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef6d9d640 10 monclient(hunting): get_auth_request con 0x7f5ef812dd40 auth_method 0 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5ef80974f0 msgr2=0x7f5ef806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5ef80974f0 0x7f5ef806d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef812dd40 msgr2=0x7f5ef806dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef812dd40 0x7f5ef806dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 monclient: found mon.noname-a 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.226+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/3616410376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec009390 con 0x7f5ef809a8a0 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ef5d9b640 1 -- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 msgr2=0x7f5ef806d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ef5d9b640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef806d860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009880 tx=0x7f5eec032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 1 -- 172.21.15.116:0/3616410376 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec031a10 con 0x7f5ef809a8a0 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: dump: 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.247 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 1 -- 172.21.15.116:0/3616410376 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec031b70 con 0x7f5ef809a8a0 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 1 -- 172.21.15.116:0/3616410376 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec03d440 con 0x7f5ef809a8a0 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5efc8a8640 5 monclient: authenticate success, global_id 4224 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ee67fc640 10 monclient: dump: 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.248 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5efc8a8640 10 monclient: _renew_subs 2024-09-18T12:25:28.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5efc8a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef8118400 con 0x7f5ef809a8a0 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5ef4d99640 4 set_mon_vals no callback set 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5efc8a8640 10 monclient: _renew_subs 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5efc8a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.228+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef81285c0 con 0x7f5ef809a8a0 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.229+0000 7f5ee67fc640 1 -- 172.21.15.116:0/3616410376 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5eec039040 con 0x7f5ef809a8a0 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.229+0000 7f5efc8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.229+0000 7f5efc8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.229+0000 7f5efc8a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.229+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef804bf60 con 0x7f5ef809a8a0 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.230+0000 7f5ee67fc640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ed80384e0 0x7f5ed803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.230+0000 7f5ee67fc640 1 -- 172.21.15.116:0/3616410376 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f5eec06b280 con 0x7f5ef809a8a0 2024-09-18T12:25:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.230+0000 7f5ef659c640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ed80384e0 0x7f5ed803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.230+0000 7f5ef659c640 10 monclient: get_auth_request con 0x7f5ed80384e0 auth_method 0 2024-09-18T12:25:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.232+0000 7f5ef659c640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ed80384e0 0x7f5ed803a990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d20 tx=0x7f5ee80090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.234+0000 7f5ee67fc640 1 -- 172.21.15.116:0/3616410376 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5eec02f2a0 con 0x7f5ef809a8a0 2024-09-18T12:25:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.234+0000 7f5ee67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.234+0000 7f5ee67fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.429+0000 7f5efc8a8640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-18T12:25:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.429+0000 7f5efc8a8640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T12:25:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.429+0000 7f5efc8a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.429+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5ef806f580 con 0x7f5ef809a8a0 2024-09-18T12:25:28.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.431+0000 7f5ee67fc640 1 -- 172.21.15.116:0/3616410376 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+87224 (secure 0 0 0) 0x7f5eec044310 con 0x7f5ef809a8a0 2024-09-18T12:25:28.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.431+0000 7f5ee67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T12:25:28.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.431+0000 7f5ee67fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:28.434 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:28.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ed80384e0 msgr2=0x7f5ed803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ed80384e0 0x7f5ed803a990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d20 tx=0x7f5ee80090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 10 monclient: shutdown 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 msgr2=0x7f5ef806d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef806d860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009880 tx=0x7f5eec032000 comp rx=0 tx=0).stop 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 shutdown_connections 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ed80384e0 0x7f5ed803a990 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef812dd40 0x7f5ef806dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ef809a8a0 0x7f5ef806d860 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 --2- 172.21.15.116:0/3616410376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5ef80974f0 0x7f5ef806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 >> 172.21.15.116:0/3616410376 conn(0x7f5ef80672b0 msgr2=0x7f5ef812d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 shutdown_connections 2024-09-18T12:25:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.433+0000 7f5efc8a8640 1 -- 172.21.15.116:0/3616410376 wait complete. 2024-09-18T12:25:28.445 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":31687},{"type":"v1","addr":"172.21.15.144:6825","nonce":31687}]},"active_addr":"172.21.15.144:6825/31687","active_change":"2024-09-18T12:25:16.268452+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":95086345}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3164666590}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":4058829443}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":876043238}]}]} 2024-09-18T12:25:28.448 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T12:25:28.448 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T12:25:28.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.539+0000 7f0dd9ec0640 10 monclient: get_monmap_and_config 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.539+0000 7f0dd9ec0640 10 monclient: build_initial_monmap 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.539+0000 7f0dd9ec0640 10 monclient: monmap: 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:28.540881+0000 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:28.540881+0000 2024-09-18T12:25:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:28.542 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.539+0000 7f0dd9ec0640 1 Processor -- start 2024-09-18T12:25:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 1 -- start start 2024-09-18T12:25:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient: init 2024-09-18T12:25:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0dd41318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 0x7f0dd409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f0dd409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0dd4136c20 0x7f0dd4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f0dd4136c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd4132640 con 0x7f0dd41318f0 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd4131d00 con 0x7f0dd4136c20 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd409fbf0 con 0x7f0dd409ea50 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd9ec0640 10 monclient(hunting): authenticate will time out at 755.862563s 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 0x7f0dd409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 0x7f0dd409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53790/0 (socket says 172.21.15.116:53790) 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 -- 172.21.15.116:0/1183512128 learned_addr learned my addr 172.21.15.116:0/1183512128 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): get_auth_request con 0x7f0dd409ea50 auth_method 0 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd3fff640 1 --2- 172.21.15.116:0/1183512128 >> v2:172.21.15.144:3300/0 conn(0x7f0dd4136c20 0x7f0dd4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-09-18T12:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 -- 172.21.15.116:0/1183512128 >> v2:172.21.15.144:3300/0 conn(0x7f0dd4136c20 msgr2=0x7f0dd4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 --2- 172.21.15.116:0/1183512128 >> v2:172.21.15.144:3300/0 conn(0x7f0dd4136c20 0x7f0dd4139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 -- 172.21.15.116:0/1183512128 >> v1:172.21.15.144:6789/0 conn(0x7f0dd41318f0 legacy=0x7f0dd4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 monclient: found mon.noname-b 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dcaffd640 1 -- 172.21.15.116:0/1183512128 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd409fe30 con 0x7f0dd409ea50 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd3fff640 10 monclient: get_auth_request con 0x7f0dd4136c20 auth_method 0 2024-09-18T12:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.540+0000 7f0dd3fff640 1 --2- 172.21.15.116:0/1183512128 >> v2:172.21.15.144:3300/0 conn(0x7f0dd4136c20 0x7f0dd4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dcaffd640 1 -- 172.21.15.116:0/1183512128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 msgr2=0x7f0dd409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dcaffd640 1 --2- 172.21.15.116:0/1183512128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 0x7f0dd409ee20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009910 tx=0x7f0dbc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d51e12677e1fb383 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 1 -- 172.21.15.116:0/1183512128 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc032440 con 0x7f0dd409ea50 2024-09-18T12:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: dump: 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 1 -- 172.21.15.116:0/1183512128 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc0325a0 con 0x7f0dd409ea50 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd9ec0640 5 monclient: authenticate success, global_id 4211 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd9ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 1 -- 172.21.15.116:0/1183512128 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc032980 con 0x7f0dd409ea50 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd9ec0640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd9ec0640 4 set_mon_vals no callback set 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.543+0000 7f0dd2ffd640 10 monclient: dump: 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 10 monclient: shutdown 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/1183512128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 msgr2=0x7f0dd409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/1183512128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 0x7f0dd409ee20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009910 tx=0x7f0dbc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/1183512128 shutdown_connections 2024-09-18T12:25:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/1183512128 >> v2:172.21.15.144:3300/0 conn(0x7f0dd4136c20 0x7f0dd4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/1183512128 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd409ea50 0x7f0dd409ee20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/1183512128 >> 172.21.15.116:0/1183512128 conn(0x7f0dd4067110 msgr2=0x7f0dd4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/1183512128 shutdown_connections 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/1183512128 wait complete. 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 10 monclient: build_initial_monmap 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.544+0000 7f0dd9ec0640 10 monclient: monmap: 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 Processor -- start 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 -- start start 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient: init 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dd409ea50 0x7f0dd4136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f0dd409ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd41318f0 0x7f0dd412ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f0dd41318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd4136c20 0x7f0dd412f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f0dd4136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd409bf10 con 0x7f0dd41318f0 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd409db00 con 0x7f0dd409ea50 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4135990 con 0x7f0dd4136c20 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.545+0000 7f0dd9ec0640 10 monclient(hunting): authenticate will time out at 755.867972s 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd41318f0 0x7f0dd412ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd4136c20 0x7f0dd412f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd41318f0 0x7f0dd412ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51888/0 (socket says 172.21.15.116:51888) 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 1 -- 172.21.15.116:0/522088031 learned_addr learned my addr 172.21.15.116:0/522088031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 10 monclient(hunting): get_auth_request con 0x7f0dd41318f0 auth_method 0 2024-09-18T12:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): get_auth_request con 0x7f0dd4136c20 auth_method 0 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd37fe640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dd409ea50 0x7f0dd4136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 1 -- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dd409ea50 msgr2=0x7f0dd4136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dd409ea50 0x7f0dd4136360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 1 -- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd41318f0 msgr2=0x7f0dd412ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd41318f0 0x7f0dd412ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 1 monclient: found mon.noname-c 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd3fff640 1 -- 172.21.15.116:0/522088031 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc009450 con 0x7f0dd4136c20 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd37fe640 10 monclient: get_auth_request con 0x7f0dd409ea50 auth_method 0 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.546+0000 7f0dd37fe640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dd409ea50 0x7f0dd4136360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd3fff640 1 -- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd4136c20 msgr2=0x7f0dd412f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd3fff640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd4136c20 0x7f0dd412f4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc007fa0 tx=0x7f0dcc007860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 1 -- 172.21.15.116:0/522088031 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00d860 con 0x7f0dd4136c20 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: dump: 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: _finish_auth 0 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 1 -- 172.21.15.116:0/522088031 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc00d9e0 con 0x7f0dd4136c20 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd9ec0640 5 monclient: authenticate success, global_id 4214 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 1 -- 172.21.15.116:0/522088031 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00dda0 con 0x7f0dd4136c20 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd9ec0640 10 monclient: _renew_subs 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd9ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd4130a70 con 0x7f0dd4136c20 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd27fc640 4 set_mon_vals no callback set 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dcbfff640 10 monclient: dump: 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd9ec0640 10 monclient: _renew_subs 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd9ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:28.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.548+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd41340d0 con 0x7f0dd4136c20 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.549+0000 7f0dcbfff640 1 -- 172.21.15.116:0/522088031 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0dcc01e3b0 con 0x7f0dd4136c20 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.549+0000 7f0dd9ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.549+0000 7f0dd9ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.549+0000 7f0dd9ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.549+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd404bf60 con 0x7f0dd4136c20 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.549+0000 7f0dcbfff640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0db4038490 0x7f0db403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.549+0000 7f0dcbfff640 1 -- 172.21.15.116:0/522088031 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f0dcc04bf60 con 0x7f0dd4136c20 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.550+0000 7f0dd37fe640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0db4038490 0x7f0db403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.550+0000 7f0dd37fe640 10 monclient: get_auth_request con 0x7f0db4038490 auth_method 0 2024-09-18T12:25:28.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.552+0000 7f0dd37fe640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0db4038490 0x7f0db403a940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40027e0 tx=0x7f0dc4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.554+0000 7f0dcbfff640 1 -- 172.21.15.116:0/522088031 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0dcc0128a0 con 0x7f0dd4136c20 2024-09-18T12:25:28.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.554+0000 7f0dcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.554+0000 7f0dcbfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.709+0000 7f0dd9ec0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:28.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.709+0000 7f0dd9ec0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:28.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.709+0000 7f0dd9ec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:28.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.709+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dd4133af0 con 0x7f0dd4136c20 2024-09-18T12:25:28.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.710+0000 7f0dcbfff640 1 -- 172.21.15.116:0/522088031 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11452 (secure 0 0 0) 0x7f0dcc01a070 con 0x7f0dd4136c20 2024-09-18T12:25:28.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.710+0000 7f0dcbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:28.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.710+0000 7f0dcbfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:28.712 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:28.713 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:25:28.097673+0000","last_up_change":"2024-09-18T12:25:21.884529+0000","last_in_change":"2024-09-18T12:25:15.496296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T12:25:22.596367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T12:25:24.541264+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":31578},{"type":"v1","addr":"172.21.15.116:6817","nonce":31578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":31578},{"type":"v1","addr":"172.21.15.116:6819","nonce":31578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":31578},{"type":"v1","addr":"172.21.15.116:6823","nonce":31578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":31578},{"type":"v1","addr":"172.21.15.116:6821","nonce":31578}]},"public_addr":"172.21.15.116:6817/31578","cluster_addr":"172.21.15.116:6819/31578","heartbeat_back_addr":"172.21.15.116:6823/31578","heartbeat_front_addr":"172.21.15.116:6821/31578","state":["exists","up"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":31579},{"type":"v1","addr":"172.21.15.116:6801","nonce":31579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":31579},{"type":"v1","addr":"172.21.15.116:6803","nonce":31579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":31579},{"type":"v1","addr":"172.21.15.116:6807","nonce":31579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":31579},{"type":"v1","addr":"172.21.15.116:6805","nonce":31579}]},"public_addr":"172.21.15.116:6801/31579","cluster_addr":"172.21.15.116:6803/31579","heartbeat_back_addr":"172.21.15.116:6807/31579","heartbeat_front_addr":"172.21.15.116:6805/31579","state":["exists","up"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":31580},{"type":"v1","addr":"172.21.15.116:6809","nonce":31580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":31580},{"type":"v1","addr":"172.21.15.116:6811","nonce":31580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":31580},{"type":"v1","addr":"172.21.15.116:6815","nonce":31580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":31580},{"type":"v1","addr":"172.21.15.116:6813","nonce":31580}]},"public_addr":"172.21.15.116:6809/31580","cluster_addr":"172.21.15.116:6811/31580","heartbeat_back_addr":"172.21.15.116:6815/31580","heartbeat_front_addr":"172.21.15.116:6813/31580","state":["exists","up"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":32112},{"type":"v1","addr":"172.21.15.144:6806","nonce":32112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":32112},{"type":"v1","addr":"172.21.15.144:6811","nonce":32112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":32112},{"type":"v1","addr":"172.21.15.144:6818","nonce":32112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":32112},{"type":"v1","addr":"172.21.15.144:6815","nonce":32112}]},"public_addr":"172.21.15.144:6806/32112","cluster_addr":"172.21.15.144:6811/32112","heartbeat_back_addr":"172.21.15.144:6818/32112","heartbeat_front_addr":"172.21.15.144:6815/32112","state":["exists","up"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":32113},{"type":"v1","addr":"172.21.15.144:6801","nonce":32113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":32113},{"type":"v1","addr":"172.21.15.144:6803","nonce":32113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":32113},{"type":"v1","addr":"172.21.15.144:6810","nonce":32113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6805","nonce":32113},{"type":"v1","addr":"172.21.15.144:6807","nonce":32113}]},"public_addr":"172.21.15.144:6801/32113","cluster_addr":"172.21.15.144:6803/32113","heartbeat_back_addr":"172.21.15.144:6810/32113","heartbeat_front_addr":"172.21.15.144:6807/32113","state":["exists","up"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":32114},{"type":"v1","addr":"172.21.15.144:6814","nonce":32114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":32114},{"type":"v1","addr":"172.21.15.144:6819","nonce":32114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":32114},{"type":"v1","addr":"172.21.15.144:6823","nonce":32114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":32114},{"type":"v1","addr":"172.21.15.144:6821","nonce":32114}]},"public_addr":"172.21.15.144:6814/32114","cluster_addr":"172.21.15.144:6819/32114","heartbeat_back_addr":"172.21.15.144:6823/32114","heartbeat_front_addr":"172.21.15.144:6821/32114","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.664684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.608350+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.689049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.706015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.672781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.692341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:25:28.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0db4038490 msgr2=0x7f0db403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0db4038490 0x7f0db403a940 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40027e0 tx=0x7f0dc4009040 comp rx=0 tx=0).stop 2024-09-18T12:25:28.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 10 monclient: shutdown 2024-09-18T12:25:28.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:28.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd4136c20 msgr2=0x7f0dd412f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd4136c20 0x7f0dd412f4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc007fa0 tx=0x7f0dcc007860 comp rx=0 tx=0).stop 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 shutdown_connections 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0db4038490 0x7f0db403a940 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0dd4136c20 0x7f0dd412f4c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dd41318f0 0x7f0dd412ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 --2- 172.21.15.116:0/522088031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0dd409ea50 0x7f0dd4136360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 >> 172.21.15.116:0/522088031 conn(0x7f0dd4067110 msgr2=0x7f0dd412cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 shutdown_connections 2024-09-18T12:25:28.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.712+0000 7f0dd9ec0640 1 -- 172.21.15.116:0/522088031 wait complete. 2024-09-18T12:25:28.724 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T12:25:28.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:25:28.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.815+0000 7fb940694640 10 monclient: get_monmap_and_config 2024-09-18T12:25:28.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.815+0000 7fb940694640 10 monclient: build_initial_monmap 2024-09-18T12:25:28.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.815+0000 7fb940694640 10 monclient: monmap: 2024-09-18T12:25:28.817 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:28.817 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:28.816718+0000 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:28.816718+0000 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.815+0000 7fb940694640 1 Processor -- start 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.815+0000 7fb940694640 1 -- start start 2024-09-18T12:25:28.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.815+0000 7fb940694640 10 monclient: init 2024-09-18T12:25:28.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:28.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9381318f0 0x7fb938131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient(hunting): picked mon.noname-c con 0x7fb9381318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb93809ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb938136c20 0x7fb938139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient(hunting): picked mon.noname-a con 0x7fb938136c20 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93809f530 con 0x7fb93809ea50 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb938131d50 con 0x7fb938136c20 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb93809fbf0 con 0x7fb9381318f0 2024-09-18T12:25:28.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb940694640 10 monclient(hunting): authenticate will time out at 756.138271s 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb938136c20 0x7fb938139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb938136c20 0x7fb938139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51898/0 (socket says 172.21.15.116:51898) 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 -- 172.21.15.116:0/2072224336 learned_addr learned my addr 172.21.15.116:0/2072224336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): get_auth_request con 0x7fb938136c20 auth_method 0 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93e430640 1 --2- 172.21.15.116:0/2072224336 >> v2:172.21.15.144:3300/0 conn(0x7fb9381318f0 0x7fb938131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93e430640 10 monclient(hunting): get_auth_request con 0x7fb9381318f0 auth_method 0 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93e430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93e430640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93e430640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 -- 172.21.15.116:0/2072224336 >> v2:172.21.15.144:3300/0 conn(0x7fb9381318f0 msgr2=0x7fb938131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 --2- 172.21.15.116:0/2072224336 >> v2:172.21.15.144:3300/0 conn(0x7fb9381318f0 0x7fb938131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 -- 172.21.15.116:0/2072224336 >> v1:172.21.15.144:6789/0 conn(0x7fb93809ea50 legacy=0x7fb93809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient: _finish_hunting 0 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 monclient: found mon.noname-a 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.816+0000 7fb93ec31640 1 -- 172.21.15.116:0/2072224336 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93809fe30 con 0x7fb938136c20 2024-09-18T12:25:28.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93ec31640 1 -- 172.21.15.116:0/2072224336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb938136c20 msgr2=0x7fb938139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93ec31640 1 --2- 172.21.15.116:0/2072224336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb938136c20 0x7fb938139000 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb938076600 tx=0x7fb9340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202f605c8c01d48d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 1 -- 172.21.15.116:0/2072224336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934032440 con 0x7fb938136c20 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: dump: 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.826 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: _finish_auth 0 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 1 -- 172.21.15.116:0/2072224336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9340325a0 con 0x7fb938136c20 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 1 -- 172.21.15.116:0/2072224336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934032940 con 0x7fb938136c20 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb93d42e640 10 monclient: dump: 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.827 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb940694640 5 monclient: authenticate success, global_id 4230 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb940694640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.819+0000 7fb940694640 4 set_mon_vals no callback set 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 10 monclient: shutdown 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 -- 172.21.15.116:0/2072224336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb938136c20 msgr2=0x7fb938139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 --2- 172.21.15.116:0/2072224336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb938136c20 0x7fb938139000 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb938076600 tx=0x7fb9340091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 -- 172.21.15.116:0/2072224336 shutdown_connections 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 --2- 172.21.15.116:0/2072224336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb938136c20 0x7fb938139000 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 --2- 172.21.15.116:0/2072224336 >> v2:172.21.15.144:3300/0 conn(0x7fb9381318f0 0x7fb938131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 -- 172.21.15.116:0/2072224336 >> 172.21.15.116:0/2072224336 conn(0x7fb938067110 msgr2=0x7fb938067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 -- 172.21.15.116:0/2072224336 shutdown_connections 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 -- 172.21.15.116:0/2072224336 wait complete. 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 10 monclient: build_initial_monmap 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 10 monclient: monmap: 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:28.829 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:28.830 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:28.830 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.820+0000 7fb940694640 1 Processor -- start 2024-09-18T12:25:28.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 1 -- start start 2024-09-18T12:25:28.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient: init 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 0x7fb938099700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): picked mon.noname-a con 0x7fb93809ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9381318f0 0x7fb938099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): picked mon.noname-c con 0x7fb9381318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb938136c20 0x7fb93809a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): picked mon.noname-b con 0x7fb938136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:28.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93809ddd0 con 0x7fb93809ea50 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93811d9d0 con 0x7fb938136c20 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93811ccf0 con 0x7fb9381318f0 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb940694640 10 monclient(hunting): authenticate will time out at 756.143483s 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 0x7fb938099700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 0x7fb938099700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51902/0 (socket says 172.21.15.116:51902) 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 -- 172.21.15.116:0/3714264009 learned_addr learned my addr 172.21.15.116:0/3714264009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): get_auth_request con 0x7fb93809ea50 auth_method 0 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9381318f0 0x7fb938099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 10 monclient(hunting): get_auth_request con 0x7fb9381318f0 auth_method 0 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb938136c20 0x7fb93809a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 10 monclient(hunting): get_auth_request con 0x7fb938136c20 auth_method 0 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93ec31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 -- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9381318f0 msgr2=0x7fb938099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9381318f0 0x7fb938099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 -- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb938136c20 msgr2=0x7fb93809a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb938136c20 0x7fb93809a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient: _finish_hunting 0 2024-09-18T12:25:28.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 monclient: found mon.noname-a 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93e430640 1 -- 172.21.15.116:0/3714264009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934009450 con 0x7fb93809ea50 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.821+0000 7fb93dc2f640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9381318f0 0x7fb938099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb93e430640 1 -- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 msgr2=0x7fb938099700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb93e430640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 0x7fb938099700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb93000e190 tx=0x7fb930009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 1 -- 172.21.15.116:0/3714264009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930013590 con 0x7fb93809ea50 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: dump: 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.839 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb940694640 5 monclient: authenticate success, global_id 4233 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 1 -- 172.21.15.116:0/3714264009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9300136f0 con 0x7fb93809ea50 2024-09-18T12:25:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 1 -- 172.21.15.116:0/3714264009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930013a90 con 0x7fb93809ea50 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb92a7fc640 10 monclient: dump: 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:28.841 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb940694640 10 monclient: _renew_subs 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb940694640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb93cc2d640 4 set_mon_vals no callback set 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93811b840 con 0x7fb93809ea50 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb940694640 10 monclient: _renew_subs 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb940694640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.824+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93811c730 con 0x7fb93809ea50 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb92a7fc640 1 -- 172.21.15.116:0/3714264009 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb93001e460 con 0x7fb93809ea50 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb940694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb940694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb940694640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93804bf60 con 0x7fb93809ea50 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb92a7fc640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb91c038490 0x7fb91c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb92a7fc640 1 -- 172.21.15.116:0/3714264009 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fb93004be60 con 0x7fb93809ea50 2024-09-18T12:25:28.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb93dc2f640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb91c038490 0x7fb91c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:28.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.825+0000 7fb93dc2f640 10 monclient: get_auth_request con 0x7fb91c038490 auth_method 0 2024-09-18T12:25:28.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.827+0000 7fb93dc2f640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb91c038490 0x7fb91c03a940 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb92c007fd0 tx=0x7fb92c007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:28.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.829+0000 7fb92a7fc640 1 -- 172.21.15.116:0/3714264009 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb930013e40 con 0x7fb93809ea50 2024-09-18T12:25:28.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.829+0000 7fb92a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:28.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.829+0000 7fb92a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:28.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.986+0000 7fb940694640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:28.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.986+0000 7fb940694640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:28.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.986+0000 7fb940694640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:28.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.986+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb938135e60 con 0x7fb93809ea50 2024-09-18T12:25:28.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.987+0000 7fb92a7fc640 1 -- 172.21.15.116:0/3714264009 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11452 (secure 0 0 0) 0x7fb93001b310 con 0x7fb93809ea50 2024-09-18T12:25:28.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.987+0000 7fb92a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:28.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.987+0000 7fb92a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:28.989 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:28.989 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:25:28.097673+0000","last_up_change":"2024-09-18T12:25:21.884529+0000","last_in_change":"2024-09-18T12:25:15.496296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T12:25:22.596367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T12:25:24.541264+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":31578},{"type":"v1","addr":"172.21.15.116:6817","nonce":31578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":31578},{"type":"v1","addr":"172.21.15.116:6819","nonce":31578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":31578},{"type":"v1","addr":"172.21.15.116:6823","nonce":31578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":31578},{"type":"v1","addr":"172.21.15.116:6821","nonce":31578}]},"public_addr":"172.21.15.116:6817/31578","cluster_addr":"172.21.15.116:6819/31578","heartbeat_back_addr":"172.21.15.116:6823/31578","heartbeat_front_addr":"172.21.15.116:6821/31578","state":["exists","up"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":31579},{"type":"v1","addr":"172.21.15.116:6801","nonce":31579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":31579},{"type":"v1","addr":"172.21.15.116:6803","nonce":31579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":31579},{"type":"v1","addr":"172.21.15.116:6807","nonce":31579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":31579},{"type":"v1","addr":"172.21.15.116:6805","nonce":31579}]},"public_addr":"172.21.15.116:6801/31579","cluster_addr":"172.21.15.116:6803/31579","heartbeat_back_addr":"172.21.15.116:6807/31579","heartbeat_front_addr":"172.21.15.116:6805/31579","state":["exists","up"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":31580},{"type":"v1","addr":"172.21.15.116:6809","nonce":31580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":31580},{"type":"v1","addr":"172.21.15.116:6811","nonce":31580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":31580},{"type":"v1","addr":"172.21.15.116:6815","nonce":31580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":31580},{"type":"v1","addr":"172.21.15.116:6813","nonce":31580}]},"public_addr":"172.21.15.116:6809/31580","cluster_addr":"172.21.15.116:6811/31580","heartbeat_back_addr":"172.21.15.116:6815/31580","heartbeat_front_addr":"172.21.15.116:6813/31580","state":["exists","up"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":32112},{"type":"v1","addr":"172.21.15.144:6806","nonce":32112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":32112},{"type":"v1","addr":"172.21.15.144:6811","nonce":32112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":32112},{"type":"v1","addr":"172.21.15.144:6818","nonce":32112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":32112},{"type":"v1","addr":"172.21.15.144:6815","nonce":32112}]},"public_addr":"172.21.15.144:6806/32112","cluster_addr":"172.21.15.144:6811/32112","heartbeat_back_addr":"172.21.15.144:6818/32112","heartbeat_front_addr":"172.21.15.144:6815/32112","state":["exists","up"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":32113},{"type":"v1","addr":"172.21.15.144:6801","nonce":32113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":32113},{"type":"v1","addr":"172.21.15.144:6803","nonce":32113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":32113},{"type":"v1","addr":"172.21.15.144:6810","nonce":32113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6805","nonce":32113},{"type":"v1","addr":"172.21.15.144:6807","nonce":32113}]},"public_addr":"172.21.15.144:6801/32113","cluster_addr":"172.21.15.144:6803/32113","heartbeat_back_addr":"172.21.15.144:6810/32113","heartbeat_front_addr":"172.21.15.144:6807/32113","state":["exists","up"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":32114},{"type":"v1","addr":"172.21.15.144:6814","nonce":32114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":32114},{"type":"v1","addr":"172.21.15.144:6819","nonce":32114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":32114},{"type":"v1","addr":"172.21.15.144:6823","nonce":32114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":32114},{"type":"v1","addr":"172.21.15.144:6821","nonce":32114}]},"public_addr":"172.21.15.144:6814/32114","cluster_addr":"172.21.15.144:6819/32114","heartbeat_back_addr":"172.21.15.144:6823/32114","heartbeat_front_addr":"172.21.15.144:6821/32114","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.664684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.608350+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.689049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.706015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.672781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.692341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:25:28.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb91c038490 msgr2=0x7fb91c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb91c038490 0x7fb91c03a940 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb92c007fd0 tx=0x7fb92c007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:28.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 10 monclient: shutdown 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 msgr2=0x7fb938099700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 0x7fb938099700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb93000e190 tx=0x7fb930009920 comp rx=0 tx=0).stop 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 shutdown_connections 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb91c038490 0x7fb91c03a940 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb938136c20 0x7fb93809a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9381318f0 0x7fb938099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 --2- 172.21.15.116:0/3714264009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb93809ea50 0x7fb938099700 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 >> 172.21.15.116:0/3714264009 conn(0x7fb938067110 msgr2=0x7fb9380a40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 shutdown_connections 2024-09-18T12:25:28.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:28.988+0000 7fb940694640 1 -- 172.21.15.116:0/3714264009 wait complete. 2024-09-18T12:25:29.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T12:25:29.002 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T12:25:29.002 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T12:25:29.003 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T12:25:29.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-18T12:25:29.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.109+0000 7f532e53b640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.109+0000 7f532e53b640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.109+0000 7f532e53b640 10 monclient: monmap: 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.111561+0000 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.111561+0000 2024-09-18T12:25:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.113 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.110+0000 7f532e53b640 1 Processor -- start 2024-09-18T12:25:29.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 1 -- start start 2024-09-18T12:25:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient: init 2024-09-18T12:25:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T12:25:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f532809e6b0 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5328069b50 0x7f532809ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient(hunting): picked mon.noname-b con 0x7f5328069b50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f532809efc0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5328069f50 con 0x7f532809e6b0 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532809d4d0 con 0x7f532809efc0 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532809f920 con 0x7f5328069b50 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.111+0000 7f532e53b640 10 monclient(hunting): authenticate will time out at 756.433802s 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532d539640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f532809e6b0 0x7f532809cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57046/0 (socket says 172.21.15.116:57046) 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532d539640 1 -- 172.21.15.116:0/1597920565 learned_addr learned my addr 172.21.15.116:0/1597920565 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 1 --2- 172.21.15.116:0/1597920565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5328069b50 0x7f532809ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): get_auth_request con 0x7f5328069b50 auth_method 0 2024-09-18T12:25:29.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 1 -- 172.21.15.116:0/1597920565 >> v1:172.21.15.144:6789/0 conn(0x7f532809efc0 legacy=0x7f532813d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.112+0000 7f532cd38640 1 -- 172.21.15.116:0/1597920565 >> v1:172.21.15.116:6789/0 conn(0x7f532809e6b0 legacy=0x7f532809cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f532cd38640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f532cd38640 1 monclient: found mon.noname-b 2024-09-18T12:25:29.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f532cd38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f532cd38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f532cd38640 1 -- 172.21.15.116:0/1597920565 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532809fb60 con 0x7f5328069b50 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f25676c9640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f25676c9640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.113+0000 7f25676c9640 10 monclient: monmap: 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.115264+0000 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.115264+0000 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.114+0000 7f25676c9640 1 Processor -- start 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.114+0000 7f25676c9640 1 -- start start 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.114+0000 7f25676c9640 10 monclient: init 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f25676c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f25676c9640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f25676c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25601318f0 0x7f2560131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f25676c9640 10 monclient(hunting): picked mon.noname-a con 0x7f25601318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f25676c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f256009ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f2565465640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25601318f0 0x7f2560131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f2565465640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25601318f0 0x7f2560131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51914/0 (socket says 172.21.15.116:51914) 2024-09-18T12:25:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f2564c64640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f256009ea50 0x7f256009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57056/0 (socket says 172.21.15.116:57056) 2024-09-18T12:25:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.115+0000 7f25676c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2560136c20 0x7f2560139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f532cd38640 1 -- 172.21.15.116:0/1597920565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5328069b50 msgr2=0x7f532809ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f532cd38640 1 --2- 172.21.15.116:0/1597920565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5328069b50 0x7f532809ea80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5318009880 tx=0x7f53180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f37d1c6f67bb12a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565c66640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2560136c20 0x7f2560139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565465640 1 -- 172.21.15.116:0/1289673201 learned_addr learned my addr 172.21.15.116:0/1289673201 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 1 -- 172.21.15.116:0/1597920565 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531802aea0 con 0x7f5328069b50 2024-09-18T12:25:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: dump: 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 1 -- 172.21.15.116:0/1597920565 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318032440 con 0x7f5328069b50 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f532e53b640 5 monclient: authenticate success, global_id 4220 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f532e53b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f532e53b640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f532e53b640 4 set_mon_vals no callback set 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 1 -- 172.21.15.116:0/1597920565 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53180327e0 con 0x7f5328069b50 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f531ffff640 10 monclient: dump: 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.127 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 10 monclient(hunting): picked mon.noname-b con 0x7f2560136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2560131cc0 con 0x7f256009ea50 2024-09-18T12:25:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256009f530 con 0x7f25601318f0 2024-09-18T12:25:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256009fbf0 con 0x7f2560136c20 2024-09-18T12:25:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f25676c9640 10 monclient(hunting): authenticate will time out at 756.438905s 2024-09-18T12:25:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565c66640 10 monclient(hunting): get_auth_request con 0x7f2560136c20 auth_method 0 2024-09-18T12:25:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565c66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 10 monclient: shutdown 2024-09-18T12:25:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565c66640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565c66640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565465640 10 monclient(hunting): get_auth_request con 0x7f25601318f0 auth_method 0 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565465640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565465640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.116+0000 7f2565465640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 1 -- 172.21.15.116:0/1597920565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5328069b50 msgr2=0x7f532809ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565465640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565465640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565465640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-18T12:25:29.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 1 --2- 172.21.15.116:0/1597920565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5328069b50 0x7f532809ea80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5318009880 tx=0x7f53180090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 1 -- 172.21.15.116:0/1597920565 shutdown_connections 2024-09-18T12:25:29.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 1 --2- 172.21.15.116:0/1597920565 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5328069b50 0x7f532809ea80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 1 -- 172.21.15.116:0/1597920565 >> 172.21.15.116:0/1597920565 conn(0x7f53280a4e70 msgr2=0x7f53280a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 1 -- 172.21.15.116:0/1289673201 >> v2:172.21.15.116:3300/0 conn(0x7f25601318f0 msgr2=0x7f2560131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 1 -- 172.21.15.116:0/1597920565 shutdown_connections 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f532e53b640 1 -- 172.21.15.116:0/1597920565 wait complete. 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 1 --2- 172.21.15.116:0/1289673201 >> v2:172.21.15.116:3300/0 conn(0x7f25601318f0 0x7f2560131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 1 -- 172.21.15.116:0/1289673201 >> v1:172.21.15.116:6789/0 conn(0x7f256009ea50 legacy=0x7f256009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 1 monclient: found mon.noname-b 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565c66640 1 -- 172.21.15.116:0/1289673201 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256009fe30 con 0x7f2560136c20 2024-09-18T12:25:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.117+0000 7f2565465640 1 --2- 172.21.15.116:0/1289673201 >> v2:172.21.15.116:3300/0 conn(0x7f25601318f0 0x7f2560131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.118+0000 7f532e53b640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.118+0000 7f532e53b640 10 monclient: monmap: 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.138 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.118+0000 7f532e53b640 1 Processor -- start 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.118+0000 7f532e53b640 1 -- start start 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.118+0000 7f532e53b640 10 monclient: init 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5328069b50 0x7f532812dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): picked mon.noname-a con 0x7f5328069b50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532809e6b0 0x7f532812a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): picked mon.noname-b con 0x7f532809e6b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532809efc0 0x7f532813dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): picked mon.noname-c con 0x7f532809efc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53280993c0 con 0x7f5328069b50 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532809aef0 con 0x7f532809e6b0 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532809b1d0 con 0x7f532809efc0 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532e53b640 10 monclient(hunting): authenticate will time out at 756.441118s 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5328069b50 0x7f532812dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5328069b50 0x7f532812dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51916/0 (socket says 172.21.15.116:51916) 2024-09-18T12:25:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 -- 172.21.15.116:0/2910498319 learned_addr learned my addr 172.21.15.116:0/2910498319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): get_auth_request con 0x7f5328069b50 auth_method 0 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532809e6b0 0x7f532812a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 10 monclient(hunting): get_auth_request con 0x7f532809e6b0 auth_method 0 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 -- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532809efc0 msgr2=0x7f532813dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532809efc0 0x7f532813dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 -- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532809e6b0 msgr2=0x7f532812a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532809e6b0 0x7f532812a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532d539640 1 -- 172.21.15.116:0/2910498319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318009390 con 0x7f5328069b50 2024-09-18T12:25:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.119+0000 7f532cd38640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532809e6b0 0x7f532812a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.120+0000 7f2565c66640 1 -- 172.21.15.116:0/1289673201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2560136c20 msgr2=0x7f2560139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.120+0000 7f2565c66640 1 --2- 172.21.15.116:0/1289673201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2560136c20 0x7f2560139000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2558009e60 tx=0x7f2558009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db17e0f78d091f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 1 -- 172.21.15.116:0/1289673201 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558007760 con 0x7f2560136c20 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: dump: 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f25676c9640 5 monclient: authenticate success, global_id 4223 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f25676c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 1 -- 172.21.15.116:0/1289673201 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255802d050 con 0x7f2560136c20 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 1 -- 172.21.15.116:0/1289673201 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558007af0 con 0x7f2560136c20 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f2557fff640 10 monclient: dump: 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f25676c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.121+0000 7f25676c9640 4 set_mon_vals no callback set 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 10 monclient: shutdown 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2560136c20 msgr2=0x7f2560139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 --2- 172.21.15.116:0/1289673201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2560136c20 0x7f2560139000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2558009e60 tx=0x7f2558009780 comp rx=0 tx=0).stop 2024-09-18T12:25:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532d539640 1 -- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5328069b50 msgr2=0x7f532812dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532d539640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5328069b50 0x7f532812dcf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f532000cca0 tx=0x7f53200053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 shutdown_connections 2024-09-18T12:25:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 --2- 172.21.15.116:0/1289673201 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2560136c20 0x7f2560139000 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 --2- 172.21.15.116:0/1289673201 >> v2:172.21.15.116:3300/0 conn(0x7f25601318f0 0x7f2560131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 >> 172.21.15.116:0/1289673201 conn(0x7f2560067110 msgr2=0x7f2560067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532000aa60 con 0x7f5328069b50 2024-09-18T12:25:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f531d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 shutdown_connections 2024-09-18T12:25:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f25676c9640 1 -- 172.21.15.116:0/1289673201 wait complete. 2024-09-18T12:25:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f531d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f531d7fa640 10 monclient: dump: 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f531d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f531d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532e53b640 5 monclient: authenticate success, global_id 4239 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532e53b640 10 monclient: _renew_subs 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532e53b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532809b4b0 con 0x7f5328069b50 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532e53b640 10 monclient: _renew_subs 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532e53b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.122+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5328099f90 con 0x7f5328069b50 2024-09-18T12:25:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532000abe0 con 0x7f5328069b50 2024-09-18T12:25:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f531d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f25676c9640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f25676c9640 10 monclient: monmap: 2024-09-18T12:25:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.154 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f531f7fe640 4 set_mon_vals no callback set 2024-09-18T12:25:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200070f0 con 0x7f5328069b50 2024-09-18T12:25:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.124+0000 7f25676c9640 1 Processor -- start 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f531d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f531d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f531d7fa640 10 monclient: dump: 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.157 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f532e53b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.123+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5308000f20 con 0x7f5328069b50 2024-09-18T12:25:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.124+0000 7f25676c9640 1 -- start start 2024-09-18T12:25:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.124+0000 7f25676c9640 10 monclient: init 2024-09-18T12:25:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.124+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5320007520 con 0x7f5328069b50 2024-09-18T12:25:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.124+0000 7f531d7fa640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384e0 0x7f530c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.124+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f532004b910 con 0x7f5328069b50 2024-09-18T12:25:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.124+0000 7f531d7fa640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f530c03e3e0 0x7f530c040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 --> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f530c040ee0 con 0x7f530c03e3e0 2024-09-18T12:25:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53200502a0 con 0x7f5328069b50 2024-09-18T12:25:29.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f531d7fa640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T12:25:29.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f25676c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f532cd38640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384e0 0x7f530c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f532cd38640 10 monclient: get_auth_request con 0x7f530c0384e0 auth_method 0 2024-09-18T12:25:29.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f532dd3a640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f530c03e3e0 0x7f530c040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f532dd3a640 10 monclient: get_auth_request con 0x7f530c03e3e0 auth_method 0 2024-09-18T12:25:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f25676c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:25:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f25676c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256009ea50 0x7f25601412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f25676c9640 10 monclient(hunting): picked mon.noname-b con 0x7f256009ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f25676c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25601318f0 0x7f2560141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f2565465640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256009ea50 0x7f25601412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f2565465640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256009ea50 0x7f25601412c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:44346/0 (socket says 172.21.15.116:44346) 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.125+0000 7f2565465640 1 -- 172.21.15.116:0/1696250608 learned_addr learned my addr 172.21.15.116:0/1696250608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f2564c64640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25601318f0 0x7f2560141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 10 monclient(hunting): picked mon.noname-c con 0x7f25601318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2560136c20 0x7f2560099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 10 monclient(hunting): picked mon.noname-a con 0x7f2560136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f2565c66640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2560136c20 0x7f2560099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256009dac0 con 0x7f2560136c20 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256011cad0 con 0x7f256009ea50 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256009bf10 con 0x7f25601318f0 2024-09-18T12:25:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.126+0000 7f25676c9640 10 monclient(hunting): authenticate will time out at 756.448977s 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565465640 10 monclient(hunting): get_auth_request con 0x7f256009ea50 auth_method 0 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565465640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565465640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565465640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): get_auth_request con 0x7f25601318f0 auth_method 0 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f532cd38640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384e0 0x7f530c03a990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5318031000 tx=0x7f5318032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565c66640 10 monclient(hunting): get_auth_request con 0x7f2560136c20 auth_method 0 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565c66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565c66640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565c66640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565465640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565465640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565465640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565c66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565c66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2565c66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 1 -- 172.21.15.116:0/1696250608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256009ea50 msgr2=0x7f25601412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256009ea50 0x7f25601412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 1 -- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2560136c20 msgr2=0x7f2560099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2560136c20 0x7f2560099510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 1 monclient: found mon.noname-c 2024-09-18T12:25:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.127+0000 7f2564c64640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c009450 con 0x7f25601318f0 2024-09-18T12:25:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f532dd3a640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f530c03e3e0 0x7f530c040810 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f2564c64640 1 -- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25601318f0 msgr2=0x7f2560141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f2564c64640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25601318f0 0x7f2560141800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f255000c4c0 tx=0x7f2550007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25500074f0 con 0x7f25601318f0 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 10 monclient: dump: 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2550007670 con 0x7f25601318f0 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25676c9640 5 monclient: authenticate success, global_id 4226 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25676c9640 10 monclient: _renew_subs 2024-09-18T12:25:29.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25676c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2558009a20 con 0x7f25601318f0 2024-09-18T12:25:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25577fe640 4 set_mon_vals no callback set 2024-09-18T12:25:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25676c9640 10 monclient: _renew_subs 2024-09-18T12:25:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25676c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.130+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256011b720 con 0x7f25601318f0 2024-09-18T12:25:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.131+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== osd.2 v2:172.21.15.116:6808/31580 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f530c040ee0 con 0x7f530c03e3e0 2024-09-18T12:25:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.131+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550005a20 con 0x7f25601318f0 2024-09-18T12:25:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.131+0000 7f25557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.131+0000 7f25557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.131+0000 7f25557fa640 10 monclient: dump: 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.131+0000 7f25676c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.131+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f256004bf60 con 0x7f25601318f0 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f255001b3b0 con 0x7f25601318f0 2024-09-18T12:25:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f25557fa640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2544038490 0x7f254403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f255004ab60 con 0x7f25601318f0 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f25557fa640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f254403e290 0x7f25440406c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2544040d90 con 0x7f254403e290 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f255004f320 con 0x7f25601318f0 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f25557fa640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f2565465640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2544038490 0x7f254403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f2565c66640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f254403e290 0x7f25440406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.132+0000 7f2565c66640 10 monclient: get_auth_request con 0x7f254403e290 auth_method 0 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.133+0000 7f2565465640 10 monclient: get_auth_request con 0x7f2544038490 auth_method 0 2024-09-18T12:25:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.135+0000 7f2565465640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2544038490 0x7f254403a940 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f255c009f70 tx=0x7f255c0091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.137+0000 7f2565c66640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f254403e290 0x7f25440406c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.138+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== osd.0 v2:172.21.15.116:6816/31578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2544040d90 con 0x7f254403e290 2024-09-18T12:25:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.141+0000 7fd88f04f640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.141+0000 7fd88f04f640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.141+0000 7fd88f04f640 10 monclient: monmap: 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.143268+0000 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.143268+0000 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.142+0000 7fd88f04f640 1 Processor -- start 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.142+0000 7fd88f04f640 1 -- start start 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.142+0000 7fd88f04f640 10 monclient: init 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.143+0000 7fd88f04f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.143+0000 7fd88f04f640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.143+0000 7fd88f04f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8881318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.143+0000 7fd88f04f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd88809ea50 0x7fd88809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7fd887fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd88809ea50 0x7fd88809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7fd88cdeb640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd8881318f0 0x7fd888131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:48928/0 (socket says 172.21.15.116:48928) 2024-09-18T12:25:29.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7fd887fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd88809ea50 0x7fd88809ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51930/0 (socket says 172.21.15.116:51930) 2024-09-18T12:25:29.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7fd88f04f640 10 monclient(hunting): picked mon.noname-a con 0x7fd88809ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7f9b782dd640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7f9b782dd640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7f9b782dd640 10 monclient: monmap: 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.146159+0000 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.146159+0000 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.184 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7fd88cdeb640 1 -- 172.21.15.116:0/2807942725 learned_addr learned my addr 172.21.15.116:0/2807942725 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7fd88f04f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd888136c20 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.144+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd888131cc0 con 0x7fd888136c20 2024-09-18T12:25:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88809fbf0 con 0x7fd8881318f0 2024-09-18T12:25:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd88f04f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7f9b782dd640 1 Processor -- start 2024-09-18T12:25:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88809fe30 con 0x7fd88809ea50 2024-09-18T12:25:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7f9b782dd640 1 -- start start 2024-09-18T12:25:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7f9b782dd640 10 monclient: init 2024-09-18T12:25:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd88f04f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd88f04f640 10 monclient(hunting): authenticate will time out at 756.467326s 2024-09-18T12:25:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): get_auth_request con 0x7fd88809ea50 auth_method 0 2024-09-18T12:25:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7f9b782dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7f9b782dd640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T12:25:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7f9b782dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b701318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2024-09-18T12:25:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b7009ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9b70137c30 0x7f9b7013a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 10 monclient(hunting): picked mon.noname-a con 0x7f9b70137c30 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 1 -- 172.21.15.116:0/2807942725 >> v1:172.21.15.144:6789/0 conn(0x7fd8881318f0 legacy=0x7fd888131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 1 -- 172.21.15.116:0/2807942725 >> v1:172.21.15.116:6789/0 conn(0x7fd888136c20 legacy=0x7fd888139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.145+0000 7fd887fff640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b75878640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9b7009ea50 0x7f9b7009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57066/0 (socket says 172.21.15.116:57066) 2024-09-18T12:25:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b75878640 1 -- 172.21.15.116:0/3313461569 learned_addr learned my addr 172.21.15.116:0/3313461569 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b70131cc0 con 0x7f9b7009ea50 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7009f530 con 0x7f9b701318f0 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b7009fbf0 con 0x7f9b70137c30 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b782dd640 10 monclient(hunting): authenticate will time out at 756.468186s 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 1 --2- 172.21.15.116:0/3313461569 >> v2:172.21.15.116:3300/0 conn(0x7f9b70137c30 0x7f9b7013a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): get_auth_request con 0x7f9b70137c30 auth_method 0 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 1 -- 172.21.15.116:0/3313461569 >> v1:172.21.15.144:6789/0 conn(0x7f9b701318f0 legacy=0x7f9b70131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 1 -- 172.21.15.116:0/3313461569 >> v1:172.21.15.116:6789/0 conn(0x7f9b7009ea50 legacy=0x7f9b7009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7f9b7687a640 1 -- 172.21.15.116:0/3313461569 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7009fe30 con 0x7f9b70137c30 2024-09-18T12:25:29.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7fd887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7fd887fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.146+0000 7fd887fff640 1 -- 172.21.15.116:0/2807942725 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880a1210 con 0x7fd88809ea50 2024-09-18T12:25:29.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.149+0000 7f9b7687a640 1 -- 172.21.15.116:0/3313461569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b70137c30 msgr2=0x7f9b7013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.149+0000 7f9b7687a640 1 --2- 172.21.15.116:0/3313461569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b70137c30 0x7f9b7013a010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009e60 tx=0x7f9b68009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d846a7482c09e2cb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.150+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 --> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5308002b40 con 0x7f530c03e3e0 2024-09-18T12:25:29.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.150+0000 7f531d7fa640 1 -- 172.21.15.116:0/2910498319 <== osd.2 v2:172.21.15.116:6808/31580 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5308002b40 con 0x7f530c03e3e0 2024-09-18T12:25:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f530c03e3e0 msgr2=0x7f530c040810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f530c03e3e0 0x7f530c040810 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384e0 msgr2=0x7f530c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384e0 0x7f530c03a990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5318031000 tx=0x7f5318032000 comp rx=0 tx=0).stop 2024-09-18T12:25:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 10 monclient: shutdown 2024-09-18T12:25:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5328069b50 msgr2=0x7f532812dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 --2- 172.21.15.116:0/2910498319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5328069b50 0x7f532812dcf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f532000cca0 tx=0x7f53200053f0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532dd3a640 1 -- 172.21.15.116:0/2910498319 reap_dead start 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 shutdown_connections 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 >> 172.21.15.116:0/2910498319 conn(0x7f53280a4e70 msgr2=0x7f53280a42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 shutdown_connections 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f532e53b640 1 -- 172.21.15.116:0/2910498319 wait complete. 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7fd887fff640 1 -- 172.21.15.116:0/2807942725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd88809ea50 msgr2=0x7fd88809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7fd887fff640 1 --2- 172.21.15.116:0/2807942725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd88809ea50 0x7fd88809ee20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd874009ab0 tx=0x7fd8740090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ea2caa568de7ee server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 1 -- 172.21.15.116:0/3313461569 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68007840 con 0x7f9b70137c30 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: dump: 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 1 -- 172.21.15.116:0/3313461569 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6802d050 con 0x7f9b70137c30 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 1 -- 172.21.15.116:0/3313461569 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68007bd0 con 0x7f9b70137c30 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b75077640 10 monclient: dump: 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b782dd640 5 monclient: authenticate success, global_id 4248 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b782dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7f9b782dd640 4 set_mon_vals no callback set 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.151+0000 7fd8877fe640 1 -- 172.21.15.116:0/2807942725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874032440 con 0x7fd88809ea50 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: dump: 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 1 -- 172.21.15.116:0/2807942725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8740325a0 con 0x7fd88809ea50 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd88f04f640 5 monclient: authenticate success, global_id 4245 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd88f04f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 1 -- 172.21.15.116:0/2807942725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874032940 con 0x7fd88809ea50 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd88f04f640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd88f04f640 4 set_mon_vals no callback set 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7fd8877fe640 10 monclient: dump: 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7f91ed19d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7f91ed19d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7f9b782dd640 10 monclient: shutdown 2024-09-18T12:25:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7f9b782dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b70137c30 msgr2=0x7f9b7013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7f9b782dd640 1 --2- 172.21.15.116:0/3313461569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b70137c30 0x7f9b7013a010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009e60 tx=0x7f9b68009780 comp rx=0 tx=0).stop 2024-09-18T12:25:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.152+0000 7f91ed19d640 10 monclient: monmap: 2024-09-18T12:25:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.154422+0000 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.154422+0000 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 shutdown_connections 2024-09-18T12:25:29.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f9b782dd640 1 --2- 172.21.15.116:0/3313461569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b70137c30 0x7f9b7013a010 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 >> 172.21.15.116:0/3313461569 conn(0x7f9b70067110 msgr2=0x7f9b70067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 10 monclient: shutdown 2024-09-18T12:25:29.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd88809ea50 msgr2=0x7fd88809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 1 --2- 172.21.15.116:0/2807942725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd88809ea50 0x7fd88809ee20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd874009ab0 tx=0x7fd8740090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 shutdown_connections 2024-09-18T12:25:29.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f9b782dd640 1 -- 172.21.15.116:0/3313461569 wait complete. 2024-09-18T12:25:29.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 shutdown_connections 2024-09-18T12:25:29.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 1 --2- 172.21.15.116:0/2807942725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd88809ea50 0x7fd88809ee20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 >> 172.21.15.116:0/2807942725 conn(0x7fd888067110 msgr2=0x7fd888067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 shutdown_connections 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd88f04f640 1 -- 172.21.15.116:0/2807942725 wait complete. 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f9b782dd640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f9b782dd640 10 monclient: monmap: 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.211 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.212 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.212 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.212 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f91ed19d640 1 Processor -- start 2024-09-18T12:25:29.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f91ed19d640 1 -- start start 2024-09-18T12:25:29.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7f91ed19d640 10 monclient: init 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.153+0000 7fd5060a8640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7fd5060a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7fd5060a8640 10 monclient: monmap: 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.155682+0000 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.155682+0000 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.213 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.214 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.214 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.214 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.214 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.214 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 1 Processor -- start 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7fd88f04f640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7fd88f04f640 10 monclient: monmap: 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.216 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.216 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.216 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.216 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 1 -- start start 2024-09-18T12:25:29.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 10 monclient: init 2024-09-18T12:25:29.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f91ed19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f91ed19d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:25:29.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:29.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b7009ea50 0x7f9b7009b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b701318f0 0x7f9b701327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 10 monclient(hunting): picked mon.noname-a con 0x7f9b701318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b70137c30 0x7f9b70132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7fd88f04f640 1 Processor -- start 2024-09-18T12:25:29.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 10 monclient(hunting): picked mon.noname-b con 0x7f9b70137c30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91e81318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.154+0000 7f9b782dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7009bf10 con 0x7f9b701318f0 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b782dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b782dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7009d800 con 0x7f9b70137c30 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b782dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b782dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b701359f0 con 0x7f9b7009ea50 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b782dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b782dd640 10 monclient(hunting): authenticate will time out at 756.477022s 2024-09-18T12:25:29.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b701318f0 0x7f9b701327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91e809ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b701318f0 0x7f9b701327d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51944/0 (socket says 172.21.15.116:51944) 2024-09-18T12:25:29.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 -- 172.21.15.116:0/4013758834 learned_addr learned my addr 172.21.15.116:0/4013758834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): get_auth_request con 0x7f9b701318f0 auth_method 0 2024-09-18T12:25:29.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f91e8137c30 0x7f91e813a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 10 monclient(hunting): picked mon.noname-c con 0x7f91e8137c30 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7fd88f04f640 1 -- start start 2024-09-18T12:25:29.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e8131cc0 con 0x7f91e81318f0 2024-09-18T12:25:29.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7fd5060a8640 1 Processor -- start 2024-09-18T12:25:29.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e809f530 con 0x7f91e809ea50 2024-09-18T12:25:29.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b76079640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b7009ea50 0x7f9b7009b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e809fbf0 con 0x7f91e8137c30 2024-09-18T12:25:29.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f91e8137c30 0x7f91e813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b76079640 10 monclient(hunting): get_auth_request con 0x7f9b7009ea50 auth_method 0 2024-09-18T12:25:29.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b76079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b76079640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e759e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f91e8137c30 0x7f91e813a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:44364/0 (socket says 172.21.15.116:44364) 2024-09-18T12:25:29.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e759e640 1 -- 172.21.15.116:0/2012280114 learned_addr learned my addr 172.21.15.116:0/2012280114 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b76079640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7fd5060a8640 1 -- start start 2024-09-18T12:25:29.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7fd5060a8640 10 monclient: init 2024-09-18T12:25:29.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91ed19d640 10 monclient(hunting): authenticate will time out at 756.477472s 2024-09-18T12:25:29.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e759e640 10 monclient(hunting): get_auth_request con 0x7f91e8137c30 auth_method 0 2024-09-18T12:25:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-09-18T12:25:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 -- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b7009ea50 msgr2=0x7f9b7009b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7fd88f04f640 10 monclient: init 2024-09-18T12:25:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b7009ea50 0x7f9b7009b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 -- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b70137c30 msgr2=0x7f9b70132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9b70137c30 0x7f9b70132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b75878640 1 -- 172.21.15.116:0/4013758834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b68009a20 con 0x7f9b701318f0 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f9b76079640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9b7009ea50 0x7f9b7009b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.155+0000 7f91e6d9d640 1 --1- 172.21.15.116:0/2012280114 >> v1:172.21.15.116:6789/0 conn(0x7f91e81318f0 0x7f91e8131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57070/0 (socket says 172.21.15.116:57070) 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T12:25:29.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd50009e680 0x7fd50009f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd50009e680 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 0x7fd5000a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd50009ff10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 10 monclient(hunting): handle_auth_done global_id 4237 payload 274 2024-09-18T12:25:29.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 1 -- 172.21.15.116:0/2012280114 >> v1:172.21.15.144:6789/0 conn(0x7f91e809ea50 legacy=0x7f91e809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 1 -- 172.21.15.116:0/2012280114 >> v1:172.21.15.116:6789/0 conn(0x7f91e81318f0 legacy=0x7f91e8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 1 monclient: found mon.noname-c 2024-09-18T12:25:29.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd500135f70 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50009dba0 con 0x7fd500135f70 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd50009d9c0 con 0x7fd50009e680 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5000a0820 con 0x7fd50009ff10 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7f91e759e640 1 -- 172.21.15.116:0/2012280114 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e809fe30 con 0x7f91e8137c30 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd5060a8640 10 monclient(hunting): authenticate will time out at 756.478226s 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 0x7fd5000a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4ff7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd50009e680 0x7fd50009f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 0x7fd5000a02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53852/0 (socket says 172.21.15.116:53852) 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 -- 172.21.15.116:0/3726458553 learned_addr learned my addr 172.21.15.116:0/3726458553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): get_auth_request con 0x7fd50009ff10 auth_method 0 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88f04f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4ff7fe640 10 monclient(hunting): get_auth_request con 0x7fd50009e680 auth_method 0 2024-09-18T12:25:29.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88f04f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:25:29.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88f04f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88809ea50 0x7fd888132b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-18T12:25:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 -- 172.21.15.116:0/3726458553 >> v2:172.21.15.144:3300/0 conn(0x7fd50009e680 msgr2=0x7fd50009f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 --2- 172.21.15.116:0/3726458553 >> v2:172.21.15.144:3300/0 conn(0x7fd50009e680 0x7fd50009f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88f04f640 10 monclient(hunting): picked mon.noname-b con 0x7fd88809ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 -- 172.21.15.116:0/3726458553 >> v1:172.21.15.144:6789/0 conn(0x7fd500135f70 legacy=0x7fd500136380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 monclient: found mon.noname-b 2024-09-18T12:25:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd4feffd640 1 -- 172.21.15.116:0/3726458553 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50009ea50 con 0x7fd50009ff10 2024-09-18T12:25:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88f04f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881318f0 0x7fd8881330d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88cdeb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88809ea50 0x7fd888132b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88cdeb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88809ea50 0x7fd888132b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:44386/0 (socket says 172.21.15.116:44386) 2024-09-18T12:25:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88cdeb640 1 -- 172.21.15.116:0/2605423733 learned_addr learned my addr 172.21.15.116:0/2605423733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88f04f640 10 monclient(hunting): picked mon.noname-a con 0x7fd8881318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.156+0000 7fd88f04f640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd888136c20 0x7fd888133610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 10 monclient(hunting): picked mon.noname-c con 0x7fd888136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd888136c20 0x7fd888133610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88813a480 con 0x7fd8881318f0 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd888136300 con 0x7fd88809ea50 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8881309a0 con 0x7fd888136c20 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88f04f640 10 monclient(hunting): authenticate will time out at 756.479300s 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88cdeb640 10 monclient(hunting): get_auth_request con 0x7fd88809ea50 auth_method 0 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88cdeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88cdeb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88cdeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): get_auth_request con 0x7fd888136c20 auth_method 0 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 1 -- 172.21.15.116:0/2605423733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88809ea50 msgr2=0x7fd888132b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88809ea50 0x7fd888132b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 1 -- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881318f0 msgr2=0x7fd8881330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881318f0 0x7fd8881330d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 1 monclient: found mon.noname-c 2024-09-18T12:25:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.157+0000 7fd88d5ec640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd874009390 con 0x7fd888136c20 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b75878640 1 -- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b701318f0 msgr2=0x7f9b701327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b75878640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b701318f0 0x7f9b701327d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c5e0 tx=0x7f9b60004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b60005cb0 con 0x7f9b701318f0 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: dump: 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.245 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b60005e30 con 0x7f9b701318f0 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b782dd640 5 monclient: authenticate success, global_id 4251 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b782dd640 10 monclient: _renew_subs 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b782dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b70134990 con 0x7f9b701318f0 2024-09-18T12:25:29.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b60007340 con 0x7f9b701318f0 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b667fc640 10 monclient: dump: 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.247 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b74876640 4 set_mon_vals no callback set 2024-09-18T12:25:29.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b782dd640 10 monclient: _renew_subs 2024-09-18T12:25:29.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b782dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b70137650 con 0x7f9b701318f0 2024-09-18T12:25:29.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.158+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 --> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f256011d960 con 0x7f254403e290 2024-09-18T12:25:29.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e759e640 1 -- 172.21.15.116:0/2012280114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e8137c30 msgr2=0x7f91e813a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e759e640 1 --2- 172.21.15.116:0/2012280114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e8137c30 0x7f91e813a010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009910 tx=0x7f91e00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f06877daf38b3d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25557fa640 1 -- 172.21.15.116:0/1696250608 <== osd.0 v2:172.21.15.116:6816/31578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f256011d960 con 0x7f254403e290 2024-09-18T12:25:29.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9b6001b400 con 0x7f9b701318f0 2024-09-18T12:25:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b782dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b7004bf60 con 0x7f9b701318f0 2024-09-18T12:25:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b667fc640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9b540384e0 0x7f9b5403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f9b6004b2c0 con 0x7f9b701318f0 2024-09-18T12:25:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f254403e290 msgr2=0x7f25440406c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f254403e290 0x7f25440406c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 1 -- 172.21.15.116:0/2012280114 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0032440 con 0x7f91e8137c30 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 10 monclient: dump: 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.252 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2544038490 msgr2=0x7f254403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2544038490 0x7f254403a940 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f255c009f70 tx=0x7f255c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 10 monclient: shutdown 2024-09-18T12:25:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25601318f0 msgr2=0x7f2560141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f25676c9640 1 --2- 172.21.15.116:0/1696250608 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25601318f0 0x7f2560141800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f255000c4c0 tx=0x7f2550007e40 comp rx=0 tx=0).stop 2024-09-18T12:25:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 1 -- 172.21.15.116:0/2012280114 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e00325a0 con 0x7f91e8137c30 2024-09-18T12:25:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91ed19d640 5 monclient: authenticate success, global_id 4237 2024-09-18T12:25:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91ed19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f2565c66640 1 -- 172.21.15.116:0/1696250608 reap_dead start 2024-09-18T12:25:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91e659c640 1 -- 172.21.15.116:0/2012280114 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0032940 con 0x7f91e8137c30 2024-09-18T12:25:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91ed19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f91ed19d640 4 set_mon_vals no callback set 2024-09-18T12:25:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7fd4feffd640 1 -- 172.21.15.116:0/3726458553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 msgr2=0x7fd5000a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4feffd640 1 --2- 172.21.15.116:0/3726458553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 0x7fd5000a02e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd5000a09b0 tx=0x7fd4e80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a2014cd77eb885a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f91e659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f91e659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f91e659c640 10 monclient: dump: 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.257 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88d5ec640 1 -- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd888136c20 msgr2=0x7fd888133610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88d5ec640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd888136c20 0x7fd888133610 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd88000c7e0 tx=0x7fd8800025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b667fc640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f9b5403e360 0x7f9b54040790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 --> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b54040e60 con 0x7f9b5403e360 2024-09-18T12:25:29.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b6001b890 con 0x7f9b701318f0 2024-09-18T12:25:29.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.159+0000 7f9b667fc640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T12:25:29.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f9b7687a640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f9b5403e360 0x7f9b54040790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f9b7687a640 10 monclient: get_auth_request con 0x7f9b5403e360 auth_method 0 2024-09-18T12:25:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 shutdown_connections 2024-09-18T12:25:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 >> 172.21.15.116:0/1696250608 conn(0x7f2560067110 msgr2=0x7f25600a3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 shutdown_connections 2024-09-18T12:25:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f25676c9640 1 -- 172.21.15.116:0/1696250608 wait complete. 2024-09-18T12:25:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f9b76079640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9b540384e0 0x7f9b5403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f9b76079640 10 monclient: get_auth_request con 0x7f9b540384e0 auth_method 0 2024-09-18T12:25:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 1 -- 172.21.15.116:0/3726458553 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8032440 con 0x7fd50009ff10 2024-09-18T12:25:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: dump: 2024-09-18T12:25:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd5060a8640 5 monclient: authenticate success, global_id 4232 2024-09-18T12:25:29.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88000a850 con 0x7fd888136c20 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: dump: 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88f04f640 5 monclient: authenticate success, global_id 4235 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88f04f640 10 monclient: _renew_subs 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88000a9d0 con 0x7fd888136c20 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88f04f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd888134530 con 0x7fd888136c20 2024-09-18T12:25:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd5060a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88f04f640 10 monclient: _renew_subs 2024-09-18T12:25:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88f04f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8881350c0 con 0x7fd888136c20 2024-09-18T12:25:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 1 -- 172.21.15.116:0/3726458553 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e80325a0 con 0x7fd50009ff10 2024-09-18T12:25:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88000ad90 con 0x7fd888136c20 2024-09-18T12:25:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 1 -- 172.21.15.116:0/3726458553 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8032440 con 0x7fd50009ff10 2024-09-18T12:25:29.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd4fe7fc640 10 monclient: dump: 2024-09-18T12:25:29.267 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd5060a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f91ed19d640 10 monclient: shutdown 2024-09-18T12:25:29.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd5060a8640 4 set_mon_vals no callback set 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f91ed19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f91ed19d640 1 -- 172.21.15.116:0/2012280114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e8137c30 msgr2=0x7f91e813a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7f91ed19d640 1 --2- 172.21.15.116:0/2012280114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e8137c30 0x7f91e813a010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009910 tx=0x7f91e00091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7f91ed19d640 1 -- 172.21.15.116:0/2012280114 shutdown_connections 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7f91ed19d640 1 --2- 172.21.15.116:0/2012280114 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e8137c30 0x7f91e813a010 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7f91ed19d640 1 -- 172.21.15.116:0/2012280114 >> 172.21.15.116:0/2012280114 conn(0x7f91e8067110 msgr2=0x7f91e8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7f91ed19d640 1 -- 172.21.15.116:0/2012280114 shutdown_connections 2024-09-18T12:25:29.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7f91ed19d640 1 -- 172.21.15.116:0/2012280114 wait complete. 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd886ffd640 4 set_mon_vals no callback set 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.160+0000 7fd884ff9640 10 monclient: dump: 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.271 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.272 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 10 monclient: shutdown 2024-09-18T12:25:29.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 -- 172.21.15.116:0/3726458553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 msgr2=0x7fd5000a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 --2- 172.21.15.116:0/3726458553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 0x7fd5000a02e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd5000a09b0 tx=0x7fd4e80091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 -- 172.21.15.116:0/3726458553 shutdown_connections 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 --2- 172.21.15.116:0/3726458553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009ff10 0x7fd5000a02e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 --2- 172.21.15.116:0/3726458553 >> v2:172.21.15.144:3300/0 conn(0x7fd50009e680 0x7fd50009f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 -- 172.21.15.116:0/3726458553 >> 172.21.15.116:0/3726458553 conn(0x7fd500067d30 msgr2=0x7fd5000a4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 -- 172.21.15.116:0/3726458553 shutdown_connections 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7fd5060a8640 1 -- 172.21.15.116:0/3726458553 wait complete. 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd88f04f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd868000f20 con 0x7fd888136c20 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7f91ed19d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.161+0000 7f91ed19d640 10 monclient: monmap: 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.274 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.275 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd880012630 con 0x7fd888136c20 2024-09-18T12:25:29.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 Processor -- start 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 -- start start 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient: init 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd5060a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd5060a8640 10 monclient: monmap: 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.276 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.277 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.277 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.277 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.277 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.277 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.277 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd884ff9640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd86c0384e0 0x7fd86c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd88004b9f0 con 0x7fd888136c20 2024-09-18T12:25:29.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd884ff9640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7fd86c03e3e0 0x7fd86c040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd86c040ee0 con 0x7fd86c03e3e0 2024-09-18T12:25:29.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:29.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8800502a0 con 0x7fd888136c20 2024-09-18T12:25:29.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e809ea50 0x7f91e8136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd884ff9640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T12:25:29.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): picked mon.noname-c con 0x7f91e809ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e81318f0 0x7f91e8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): picked mon.noname-b con 0x7f91e81318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e8137c30 0x7f91e8096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): picked mon.noname-a con 0x7f91e8137c30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd88cdeb640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd86c0384e0 0x7fd86c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7fd88cdeb640 10 monclient: get_auth_request con 0x7fd86c0384e0 auth_method 0 2024-09-18T12:25:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e8098150 con 0x7f91e8137c30 2024-09-18T12:25:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e809bf10 con 0x7f91e81318f0 2024-09-18T12:25:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e809d7d0 con 0x7f91e809ea50 2024-09-18T12:25:29.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.162+0000 7f91ed19d640 10 monclient(hunting): authenticate will time out at 756.484971s 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e8137c30 0x7f91e8096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e8137c30 0x7f91e8096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51958/0 (socket says 172.21.15.116:51958) 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 1 -- 172.21.15.116:0/2951539212 learned_addr learned my addr 172.21.15.116:0/2951539212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 10 monclient(hunting): get_auth_request con 0x7f91e8137c30 auth_method 0 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd887fff640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7fd86c03e3e0 0x7fd86c040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd887fff640 10 monclient: get_auth_request con 0x7fd86c03e3e0 auth_method 0 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e809ea50 0x7f91e8136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e81318f0 0x7f91e8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 Processor -- start 2024-09-18T12:25:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): get_auth_request con 0x7f91e809ea50 auth_method 0 2024-09-18T12:25:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 10 monclient(hunting): get_auth_request con 0x7f91e81318f0 auth_method 0 2024-09-18T12:25:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 -- start start 2024-09-18T12:25:29.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient: init 2024-09-18T12:25:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f9b76079640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9b540384e0 0x7f9b5403a990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c002860 tx=0x7f9b6c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 1 -- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e81318f0 msgr2=0x7f91e8096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e81318f0 0x7f91e8096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 1 -- 172.21.15.116:0/2951539212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e8137c30 msgr2=0x7f91e8096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e8137c30 0x7f91e8096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 1 monclient: found mon.noname-c 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91e6d9d640 1 -- 172.21.15.116:0/2951539212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0009450 con 0x7f91e809ea50 2024-09-18T12:25:29.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7f91dffff640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91e81318f0 0x7f91e8096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009e680 0x7fd50006dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd50009e680 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd50009ff10 0x7fd500135190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd50009ff10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500135f70 0x7fd5001455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd500135f70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50006e8d0 con 0x7fd500135f70 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5001265d0 con 0x7fd50009ff10 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5000a26a0 con 0x7fd50009e680 2024-09-18T12:25:29.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.163+0000 7fd5060a8640 10 monclient(hunting): authenticate will time out at 756.485978s 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500135f70 0x7fd5001455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500135f70 0x7fd5001455a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51960/0 (socket says 172.21.15.116:51960) 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 -- 172.21.15.116:0/685447449 learned_addr learned my addr 172.21.15.116:0/685447449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): get_auth_request con 0x7fd500135f70 auth_method 0 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd50009ff10 0x7fd500135190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 10 monclient(hunting): get_auth_request con 0x7fd50009ff10 auth_method 0 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 -- 172.21.15.116:0/685447449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009e680 msgr2=0x7fd50006dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd50009e680 0x7fd50006dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 -- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd50009ff10 msgr2=0x7fd500135190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd50009ff10 0x7fd500135190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7fd4fffff640 1 -- 172.21.15.116:0/685447449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8009450 con 0x7fd500135f70 2024-09-18T12:25:29.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.164+0000 7f9b7687a640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f9b5403e360 0x7f9b54040790 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.165+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== osd.4 v2:172.21.15.144:6800/32113 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9b54040e60 con 0x7f9b5403e360 2024-09-18T12:25:29.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.165+0000 7fd88cdeb640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd86c0384e0 0x7fd86c03a990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd87c007fd0 tx=0x7fd87c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.165+0000 7f91e6d9d640 1 -- 172.21.15.116:0/2951539212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e809ea50 msgr2=0x7f91e8136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91e6d9d640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e809ea50 0x7f91e8136e70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f91d800c840 tx=0x7f91d8004ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.297 INFO:teuthology.orchestra.run.smithi116.stdout:47244640260 2024-09-18T12:25:29.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d800ce30 con 0x7f91e809ea50 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: dump: 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.299 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d800a640 con 0x7f91e809ea50 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d800aa00 con 0x7f91e809ea50 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91deffd640 10 monclient: dump: 2024-09-18T12:25:29.300 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91e559a640 4 set_mon_vals no callback set 2024-09-18T12:25:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91ed19d640 5 monclient: authenticate success, global_id 4238 2024-09-18T12:25:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91ed19d640 10 monclient: _renew_subs 2024-09-18T12:25:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91ed19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7fd4fffff640 1 -- 172.21.15.116:0/685447449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500135f70 msgr2=0x7fd5001455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7fd4fffff640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500135f70 0x7fd5001455a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c5e0 tx=0x7fd4f0007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.166+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e8099690 con 0x7f91e809ea50 2024-09-18T12:25:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7f91ed19d640 10 monclient: _renew_subs 2024-09-18T12:25:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7f91ed19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e809ab30 con 0x7f91e809ea50 2024-09-18T12:25:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0012840 con 0x7fd500135f70 2024-09-18T12:25:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: dump: 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f00129a0 con 0x7fd500135f70 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0012d60 con 0x7fd500135f70 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4f77fe640 10 monclient: dump: 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd4fdffb640 4 set_mon_vals no callback set 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd5060a8640 5 monclient: authenticate success, global_id 4257 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd5060a8640 10 monclient: _renew_subs 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd5060a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5001262f0 con 0x7fd500135f70 2024-09-18T12:25:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd5060a8640 10 monclient: _renew_subs 2024-09-18T12:25:29.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd5060a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.167+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd500124f00 con 0x7fd500135f70 2024-09-18T12:25:29.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7fd887fff640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7fd86c03e3e0 0x7fd86c040810 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7f91ed19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f91e804bf60 con 0x7f91e809ea50 2024-09-18T12:25:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7fd5060a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd50004bf60 con 0x7fd500135f70 2024-09-18T12:25:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f91d80070f0 con 0x7f91e809ea50 2024-09-18T12:25:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd4f0011590 con 0x7fd500135f70 2024-09-18T12:25:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== osd.1 v2:172.21.15.116:6800/31579 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd86c040ee0 con 0x7fd86c03e3e0 2024-09-18T12:25:29.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.168+0000 7f91deffd640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91c80384e0 0x7f91c803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f91d804cd50 con 0x7f91e809ea50 2024-09-18T12:25:29.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91deffd640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7f91c803e360 0x7f91c8040790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 --> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91c8040e60 con 0x7f91c803e360 2024-09-18T12:25:29.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91d80543a0 con 0x7f91e809ea50 2024-09-18T12:25:29.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91deffd640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T12:25:29.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4f77fe640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd4e0038490 0x7fd4e003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd4f004b760 con 0x7fd500135f70 2024-09-18T12:25:29.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4f77fe640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fd4e003e310 0x7fd4e0040740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4e0040e10 con 0x7fd4e003e310 2024-09-18T12:25:29.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91e759e640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7f91c803e360 0x7f91c8040790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91dffff640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91c80384e0 0x7f91c803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4f004dcf0 con 0x7fd500135f70 2024-09-18T12:25:29.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4f77fe640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-18T12:25:29.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91e759e640 10 monclient: get_auth_request con 0x7f91c803e360 auth_method 0 2024-09-18T12:25:29.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7f91dffff640 10 monclient: get_auth_request con 0x7f91c80384e0 auth_method 0 2024-09-18T12:25:29.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4feffd640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fd4e003e310 0x7fd4e0040740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.169+0000 7fd4feffd640 10 monclient: get_auth_request con 0x7fd4e003e310 auth_method 0 2024-09-18T12:25:29.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.172+0000 7f91dffff640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91c80384e0 0x7f91c803a990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f91e81375f0 tx=0x7f91d0007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.314 INFO:teuthology.orchestra.run.smithi116.stdout:47244640260 2024-09-18T12:25:29.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T12:25:29.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.174+0000 7f91e759e640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7f91c803e360 0x7f91c8040790 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.174+0000 7fd4feffd640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fd4e003e310 0x7fd4e0040740 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.174+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== osd.3 v2:172.21.15.144:6804/32112 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f91c8040e60 con 0x7f91c803e360 2024-09-18T12:25:29.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.175+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== osd.5 v2:172.21.15.144:6812/32114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd4e0040e10 con 0x7fd4e003e310 2024-09-18T12:25:29.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.180+0000 7fd4ff7fe640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd4e0038490 0x7fd4e003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.180+0000 7fd4ff7fe640 10 monclient: get_auth_request con 0x7fd4e0038490 auth_method 0 2024-09-18T12:25:29.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.181+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 --> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd868002b40 con 0x7fd86c03e3e0 2024-09-18T12:25:29.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd884ff9640 1 -- 172.21.15.116:0/2605423733 <== osd.1 v2:172.21.15.116:6800/31579 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd868002b40 con 0x7fd86c03e3e0 2024-09-18T12:25:29.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7fd86c03e3e0 msgr2=0x7fd86c040810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7fd86c03e3e0 0x7fd86c040810 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd86c0384e0 msgr2=0x7fd86c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd86c0384e0 0x7fd86c03a990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd87c007fd0 tx=0x7fd87c009040 comp rx=0 tx=0).stop 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 10 monclient: shutdown 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd888136c20 msgr2=0x7fd888133610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 --2- 172.21.15.116:0/2605423733 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd888136c20 0x7fd888133610 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd88000c7e0 tx=0x7fd8800025b0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88d5ec640 1 -- 172.21.15.116:0/2605423733 reap_dead start 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 shutdown_connections 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 >> 172.21.15.116:0/2605423733 conn(0x7fd888067110 msgr2=0x7fd888139000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 shutdown_connections 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.182+0000 7fd88f04f640 1 -- 172.21.15.116:0/2605423733 wait complete. 2024-09-18T12:25:29.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.183+0000 7fd4ff7fe640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd4e0038490 0x7fd4e003a940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009fd0 tx=0x7fd4f8009970 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.186+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 --> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9b70133490 con 0x7f9b5403e360 2024-09-18T12:25:29.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.187+0000 7f9b667fc640 1 -- 172.21.15.116:0/4013758834 <== osd.4 v2:172.21.15.144:6800/32113 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9b70133490 con 0x7f9b5403e360 2024-09-18T12:25:29.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.187+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f9b5403e360 msgr2=0x7f9b54040790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.187+0000 7f9b782dd640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f9b5403e360 0x7f9b54040790 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd500125640 con 0x7fd4e003e310 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9b540384e0 msgr2=0x7f9b5403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9b540384e0 0x7f9b5403a990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c002860 tx=0x7f9b6c009040 comp rx=0 tx=0).stop 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 10 monclient: shutdown 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b701318f0 msgr2=0x7f9b701327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 --2- 172.21.15.116:0/4013758834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b701318f0 0x7f9b701327d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c5e0 tx=0x7f9b60004960 comp rx=0 tx=0).stop 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b7687a640 1 -- 172.21.15.116:0/4013758834 reap_dead start 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 shutdown_connections 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 >> 172.21.15.116:0/4013758834 conn(0x7f9b70067110 msgr2=0x7f9b700a3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 shutdown_connections 2024-09-18T12:25:29.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7f9b782dd640 1 -- 172.21.15.116:0/4013758834 wait complete. 2024-09-18T12:25:29.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.188+0000 7fd4f77fe640 1 -- 172.21.15.116:0/685447449 <== osd.5 v2:172.21.15.144:6812/32114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd500125640 con 0x7fd4e003e310 2024-09-18T12:25:29.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fd4e003e310 msgr2=0x7fd4e0040740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fd4e003e310 0x7fd4e0040740 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd4e0038490 msgr2=0x7fd4e003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd4e0038490 0x7fd4e003a940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009fd0 tx=0x7fd4f8009970 comp rx=0 tx=0).stop 2024-09-18T12:25:29.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 10 monclient: shutdown 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500135f70 msgr2=0x7fd5001455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 --2- 172.21.15.116:0/685447449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd500135f70 0x7fd5001455a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c5e0 tx=0x7fd4f0007ed0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd4fffff640 1 -- 172.21.15.116:0/685447449 reap_dead start 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 shutdown_connections 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 >> 172.21.15.116:0/685447449 conn(0x7fd500067d30 msgr2=0x7fd500097530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 shutdown_connections 2024-09-18T12:25:29.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.189+0000 7fd5060a8640 1 -- 172.21.15.116:0/685447449 wait complete. 2024-09-18T12:25:29.325 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:29.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T12:25:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.195+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 --> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f91e80989e0 con 0x7f91c803e360 2024-09-18T12:25:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91deffd640 1 -- 172.21.15.116:0/2951539212 <== osd.3 v2:172.21.15.144:6804/32112 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f91e80989e0 con 0x7f91c803e360 2024-09-18T12:25:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7f91c803e360 msgr2=0x7f91c8040790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7f91c803e360 0x7f91c8040790 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91c80384e0 msgr2=0x7f91c803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91c80384e0 0x7f91c803a990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f91e81375f0 tx=0x7f91d0007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 10 monclient: shutdown 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e809ea50 msgr2=0x7f91e8136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 --2- 172.21.15.116:0/2951539212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f91e809ea50 0x7f91e8136e70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f91d800c840 tx=0x7f91d8004ea0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91e759e640 1 -- 172.21.15.116:0/2951539212 reap_dead start 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 shutdown_connections 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 >> 172.21.15.116:0/2951539212 conn(0x7f91e8067110 msgr2=0x7f91e80a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 shutdown_connections 2024-09-18T12:25:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.196+0000 7f91ed19d640 1 -- 172.21.15.116:0/2951539212 wait complete. 2024-09-18T12:25:29.329 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:29.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T12:25:29.330 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:29.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T12:25:29.332 INFO:teuthology.orchestra.run.smithi116.stdout:47244640260 2024-09-18T12:25:29.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T12:25:29.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.460+0000 7f2d0671d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.460+0000 7f2d0671d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.460+0000 7f2d0671d640 10 monclient: monmap: 2024-09-18T12:25:29.462 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.462 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.462 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.461741+0000 2024-09-18T12:25:29.462 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.461741+0000 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.463 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.460+0000 7f2d0671d640 1 Processor -- start 2024-09-18T12:25:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 1 -- start start 2024-09-18T12:25:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 10 monclient: init 2024-09-18T12:25:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T12:25:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d001318f0 0x7f2d00131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d001318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 0x7f2d0009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f0e9c034640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f0e9c034640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d0009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f0e9c034640 10 monclient: monmap: 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.463380+0000 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.463380+0000 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.466 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.467 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.467 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.467 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.467 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.467 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2d0671d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d00068d90 0x7f2d000691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.461+0000 7f2cff7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 0x7f2d0009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 0x7f2d0009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53878/0 (socket says 172.21.15.116:53878) 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d001318f0 0x7f2d00131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d001318f0 0x7f2d00131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51970/0 (socket says 172.21.15.116:51970) 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d00068d90 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 -- 172.21.15.116:0/2589284034 learned_addr learned my addr 172.21.15.116:0/2589284034 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d00131cc0 con 0x7f2d001318f0 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0009fbf0 con 0x7f2d00068d90 2024-09-18T12:25:29.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0009fe30 con 0x7f2d0009ea50 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d0671d640 10 monclient(hunting): authenticate will time out at 756.784186s 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): get_auth_request con 0x7f2d0009ea50 auth_method 0 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 10 monclient(hunting): get_auth_request con 0x7f2d001318f0 auth_method 0 2024-09-18T12:25:29.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f0e9c034640 1 Processor -- start 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f0e9c034640 1 -- start start 2024-09-18T12:25:29.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f0e9c034640 10 monclient: init 2024-09-18T12:25:29.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d04cba640 1 --2- 172.21.15.116:0/2589284034 >> v2:172.21.15.144:3300/0 conn(0x7f2d00068d90 0x7f2d000691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 -- 172.21.15.116:0/2589284034 >> v2:172.21.15.144:3300/0 conn(0x7f2d00068d90 msgr2=0x7f2d000691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 --2- 172.21.15.116:0/2589284034 >> v2:172.21.15.144:3300/0 conn(0x7f2d00068d90 0x7f2d000691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 -- 172.21.15.116:0/2589284034 >> v2:172.21.15.116:3300/0 conn(0x7f2d001318f0 msgr2=0x7f2d00131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 --2- 172.21.15.116:0/2589284034 >> v2:172.21.15.116:3300/0 conn(0x7f2d001318f0 0x7f2d00131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e941318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e9409ea50 0x7f0e9409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9409ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0e94136c20 0x7f0e94139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient(hunting): picked mon.noname-a con 0x7f0e94136c20 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e94131cc0 con 0x7f0e941318f0 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9409fbf0 con 0x7f0e94136c20 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9409fe30 con 0x7f0e9409ea50 2024-09-18T12:25:29.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9c034640 10 monclient(hunting): authenticate will time out at 756.785302s 2024-09-18T12:25:29.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 monclient: found mon.noname-b 2024-09-18T12:25:29.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2cff7fe640 1 -- 172.21.15.116:0/2589284034 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d000a1210 con 0x7f2d0009ea50 2024-09-18T12:25:29.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d04cba640 10 monclient: get_auth_request con 0x7f2d00068d90 auth_method 0 2024-09-18T12:25:29.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.462+0000 7f2d04cba640 1 --2- 172.21.15.116:0/2589284034 >> v2:172.21.15.144:3300/0 conn(0x7f2d00068d90 0x7f2d000691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:29.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0e94136c20 0x7f0e94139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e9409ea50 0x7f0e9409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e9409ea50 0x7f0e9409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:44404/0 (socket says 172.21.15.116:44404) 2024-09-18T12:25:29.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e99dd0640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f0e941318f0 0x7f0e94131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:48962/0 (socket says 172.21.15.116:48962) 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e99dd0640 1 -- 172.21.15.116:0/3268528175 learned_addr learned my addr 172.21.15.116:0/3268528175 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 10 monclient(hunting): get_auth_request con 0x7f0e9409ea50 auth_method 0 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): get_auth_request con 0x7f0e94136c20 auth_method 0 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f2cfffff640 1 --2- 172.21.15.116:0/2589284034 >> v2:172.21.15.116:3300/0 conn(0x7f2d001318f0 0x7f2d00131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-09-18T12:25:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 1 -- 172.21.15.116:0/3268528175 >> v2:172.21.15.144:3300/0 conn(0x7f0e9409ea50 msgr2=0x7f0e9409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 1 --2- 172.21.15.116:0/3268528175 >> v2:172.21.15.144:3300/0 conn(0x7f0e9409ea50 0x7f0e9409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 1 -- 172.21.15.116:0/3268528175 >> v1:172.21.15.144:6789/0 conn(0x7f0e941318f0 legacy=0x7f0e94131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e9a5d1640 1 -- 172.21.15.116:0/3268528175 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e940a1210 con 0x7f0e94136c20 2024-09-18T12:25:29.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.463+0000 7f0e995cf640 1 --2- 172.21.15.116:0/3268528175 >> v2:172.21.15.144:3300/0 conn(0x7f0e9409ea50 0x7f0e9409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cff7fe640 1 -- 172.21.15.116:0/2589284034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 msgr2=0x7f2d0009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cff7fe640 1 --2- 172.21.15.116:0/2589284034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 0x7f2d0009ee20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d00064050 tx=0x7f2cf40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b023811f169044 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 1 -- 172.21.15.116:0/2589284034 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4032440 con 0x7f2d0009ea50 2024-09-18T12:25:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: dump: 2024-09-18T12:25:29.484 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 1 -- 172.21.15.116:0/2589284034 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf40325a0 con 0x7f2d0009ea50 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2d0671d640 5 monclient: authenticate success, global_id 4241 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2d0671d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2d0671d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2d0671d640 4 set_mon_vals no callback set 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 1 -- 172.21.15.116:0/2589284034 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4032940 con 0x7f2d0009ea50 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f2cfeffd640 10 monclient: dump: 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.486 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f0e9a5d1640 1 -- 172.21.15.116:0/3268528175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e94136c20 msgr2=0x7f0e94139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.466+0000 7f0e9a5d1640 1 --2- 172.21.15.116:0/3268528175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e94136c20 0x7f0e94139000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e9406f600 tx=0x7f0e8c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deebc6eb16f5362 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 1 -- 172.21.15.116:0/3268528175 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c007760 con 0x7f0e94136c20 2024-09-18T12:25:29.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 10 monclient: shutdown 2024-09-18T12:25:29.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 msgr2=0x7f2d0009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 --2- 172.21.15.116:0/2589284034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 0x7f2d0009ee20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d00064050 tx=0x7f2cf40091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: dump: 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.489 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 1 -- 172.21.15.116:0/3268528175 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c02d050 con 0x7f0e94136c20 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 1 -- 172.21.15.116:0/3268528175 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c007af0 con 0x7f0e94136c20 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e98dce640 10 monclient: dump: 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.490 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.491 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 shutdown_connections 2024-09-18T12:25:29.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 --2- 172.21.15.116:0/2589284034 >> v2:172.21.15.144:3300/0 conn(0x7f2d00068d90 0x7f2d000691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 --2- 172.21.15.116:0/2589284034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d0009ea50 0x7f2d0009ee20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 --2- 172.21.15.116:0/2589284034 >> v2:172.21.15.116:3300/0 conn(0x7f2d001318f0 0x7f2d00131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e9c034640 5 monclient: authenticate success, global_id 4263 2024-09-18T12:25:29.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e9c034640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f0e9c034640 4 set_mon_vals no callback set 2024-09-18T12:25:29.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 >> 172.21.15.116:0/2589284034 conn(0x7f2d00067110 msgr2=0x7f2d00067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 shutdown_connections 2024-09-18T12:25:29.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.467+0000 7f2d0671d640 1 -- 172.21.15.116:0/2589284034 wait complete. 2024-09-18T12:25:29.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 10 monclient: shutdown 2024-09-18T12:25:29.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 -- 172.21.15.116:0/3268528175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e94136c20 msgr2=0x7f0e94139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 --2- 172.21.15.116:0/3268528175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e94136c20 0x7f0e94139000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e9406f600 tx=0x7f0e8c009780 comp rx=0 tx=0).stop 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f2d0671d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f2d0671d640 10 monclient: monmap: 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.494 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.495 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.495 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.495 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.495 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.495 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 -- 172.21.15.116:0/3268528175 shutdown_connections 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 --2- 172.21.15.116:0/3268528175 >> v2:172.21.15.144:3300/0 conn(0x7f0e9409ea50 0x7f0e9409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 --2- 172.21.15.116:0/3268528175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e94136c20 0x7f0e94139000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 -- 172.21.15.116:0/3268528175 >> 172.21.15.116:0/3268528175 conn(0x7f0e94067110 msgr2=0x7f0e94067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 -- 172.21.15.116:0/3268528175 shutdown_connections 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f0e9c034640 1 -- 172.21.15.116:0/3268528175 wait complete. 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.468+0000 7f2d0671d640 1 Processor -- start 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f2d0671d640 1 -- start start 2024-09-18T12:25:29.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f2d0671d640 10 monclient: init 2024-09-18T12:25:29.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f0e9c034640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f0e9c034640 10 monclient: monmap: 2024-09-18T12:25:29.497 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.497 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.497 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.497 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.497 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f0e9c034640 1 Processor -- start 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f0e9c034640 1 -- start start 2024-09-18T12:25:29.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f0e9c034640 10 monclient: init 2024-09-18T12:25:29.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f2d0671d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f2d0671d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:29.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.469+0000 7f2d0671d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 0x7f2d0009ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d00068d90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 0x7f2d0009ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 0x7f2d0009ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51984/0 (socket says 172.21.15.116:51984) 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d0009ea50 0x7f2d00130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 1 -- 172.21.15.116:0/973597852 learned_addr learned my addr 172.21.15.116:0/973597852 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d0009ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d001318f0 0x7f2d00140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d001318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0009bf80 con 0x7f2d00068d90 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0011e0e0 con 0x7f2d0009ea50 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cff7fe640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d0009ea50 0x7f2d00130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d04cba640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d001318f0 0x7f2d00140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d000a4390 con 0x7f2d001318f0 2024-09-18T12:25:29.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:29.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 0x7f0e94130860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d0671d640 10 monclient(hunting): authenticate will time out at 756.792740s 2024-09-18T12:25:29.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9409ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e941318f0 0x7f0e940996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 10 monclient(hunting): get_auth_request con 0x7f2d00068d90 auth_method 0 2024-09-18T12:25:29.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient(hunting): picked mon.noname-b con 0x7f0e941318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e94136c20 0x7f0e94099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient(hunting): picked mon.noname-c con 0x7f0e94136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9409bf10 con 0x7f0e9409ea50 2024-09-18T12:25:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9409dcc0 con 0x7f0e941318f0 2024-09-18T12:25:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e99dd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 0x7f0e94130860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e99dd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 0x7f0e94130860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51988/0 (socket says 172.21.15.116:51988) 2024-09-18T12:25:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e99dd0640 1 -- 172.21.15.116:0/1090069104 learned_addr learned my addr 172.21.15.116:0/1090069104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e995cf640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e941318f0 0x7f0e940996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d04cba640 10 monclient(hunting): get_auth_request con 0x7f2d001318f0 auth_method 0 2024-09-18T12:25:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d04cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d04cba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2d04cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f2cfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2d04cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.470+0000 7f0e9c034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e941374c0 con 0x7f0e94136c20 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9c034640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9c034640 10 monclient(hunting): authenticate will time out at 756.793461s 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): get_auth_request con 0x7f0e9409ea50 auth_method 0 2024-09-18T12:25:29.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2d04cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e995cf640 10 monclient(hunting): get_auth_request con 0x7f0e941318f0 auth_method 0 2024-09-18T12:25:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e995cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e995cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e995cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2d04cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cff7fe640 10 monclient(hunting): get_auth_request con 0x7f2d0009ea50 auth_method 0 2024-09-18T12:25:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-09-18T12:25:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 1 -- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d001318f0 msgr2=0x7f2d00140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d001318f0 0x7f2d00140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 1 -- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d0009ea50 msgr2=0x7f2d00130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d0009ea50 0x7f2d00130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f2cfffff640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0009390 con 0x7f2d00068d90 2024-09-18T12:25:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e94136c20 0x7f0e94099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 10 monclient(hunting): get_auth_request con 0x7f0e94136c20 auth_method 0 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e9a5d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 1 -- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e94136c20 msgr2=0x7f0e94099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e94136c20 0x7f0e94099c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 1 -- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e941318f0 msgr2=0x7f0e940996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e941318f0 0x7f0e940996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.471+0000 7f0e99dd0640 1 -- 172.21.15.116:0/1090069104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e90009390 con 0x7f0e9409ea50 2024-09-18T12:25:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.472+0000 7f0e995cf640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e941318f0 0x7f0e940996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2cfffff640 1 -- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 msgr2=0x7f2d0009ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2cfffff640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 0x7f2d0009ab40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009ec0 tx=0x7f2cf00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2ce7fff640 1 -- 172.21.15.116:0/973597852 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf000cdd0 con 0x7f2d00068d90 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2ce7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2ce7fff640 10 monclient: dump: 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2ce7fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2d0671d640 5 monclient: authenticate success, global_id 4266 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2d0671d640 10 monclient: _renew_subs 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2d0671d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.473+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf4009450 con 0x7f2d00068d90 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2ce7fff640 1 -- 172.21.15.116:0/973597852 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0017c60 con 0x7f2d00068d90 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2ce7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2ce7fff640 1 -- 172.21.15.116:0/973597852 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0017dd0 con 0x7f2d00068d90 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2ce7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2ce7fff640 10 monclient: dump: 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2cfe7fc640 4 set_mon_vals no callback set 2024-09-18T12:25:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2d0671d640 10 monclient: _renew_subs 2024-09-18T12:25:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2d0671d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0011c790 con 0x7f2d00068d90 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f0e99dd0640 1 -- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 msgr2=0x7f0e94130860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.474+0000 7f0e99dd0640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 0x7f0e94130860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400c590 tx=0x7f0e84004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 1 -- 172.21.15.116:0/1090069104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84005cb0 con 0x7f0e9409ea50 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: dump: 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 1 -- 172.21.15.116:0/1090069104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e84005e30 con 0x7f0e9409ea50 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 1 -- 172.21.15.116:0/1090069104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84007320 con 0x7f0e9409ea50 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e89ffb640 10 monclient: dump: 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e9c034640 5 monclient: authenticate success, global_id 4269 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e9c034640 10 monclient: _renew_subs 2024-09-18T12:25:29.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e9c034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e9c034640 1 -- 172.21.15.116:0/1090069104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c009a20 con 0x7f0e9409ea50 2024-09-18T12:25:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e9c034640 10 monclient: _renew_subs 2024-09-18T12:25:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e9c034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f2ce7fff640 1 -- 172.21.15.116:0/973597852 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2cf00283b0 con 0x7f2d00068d90 2024-09-18T12:25:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f2ce7fff640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cec038490 0x7f2cec03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f2ce7fff640 1 -- 172.21.15.116:0/973597852 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f2cf0053d00 con 0x7f2d00068d90 2024-09-18T12:25:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e9c034640 1 -- 172.21.15.116:0/1090069104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9411ba80 con 0x7f0e9409ea50 2024-09-18T12:25:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.475+0000 7f0e8bfff640 4 set_mon_vals no callback set 2024-09-18T12:25:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f2d0671d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f2d0671d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f2d0671d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0004bf60 con 0x7f2d00068d90 2024-09-18T12:25:29.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f0e7f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f0e7f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f2cff7fe640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cec038490 0x7f2cec03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f2cff7fe640 10 monclient: get_auth_request con 0x7f2cec038490 auth_method 0 2024-09-18T12:25:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f0e89ffb640 1 -- 172.21.15.116:0/1090069104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0e8401b3b0 con 0x7f0e9409ea50 2024-09-18T12:25:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f0e7f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.476+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e94136630 con 0x7f0e9409ea50 2024-09-18T12:25:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.477+0000 7f0e89ffb640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e780384e0 0x7f0e7803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.477+0000 7f0e89ffb640 1 -- 172.21.15.116:0/1090069104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f0e8404ac00 con 0x7f0e9409ea50 2024-09-18T12:25:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.477+0000 7f0e995cf640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e780384e0 0x7f0e7803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.477+0000 7f0e995cf640 10 monclient: get_auth_request con 0x7f0e780384e0 auth_method 0 2024-09-18T12:25:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.479+0000 7f2cff7fe640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cec038490 0x7f2cec03a940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4031000 tx=0x7f2cf4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.479+0000 7f0e995cf640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e780384e0 0x7f0e7803a990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009360 tx=0x7f0e90008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.481+0000 7f2ce7fff640 1 -- 172.21.15.116:0/973597852 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2cf0024080 con 0x7f2d00068d90 2024-09-18T12:25:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.481+0000 7f2ce7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.481+0000 7f2ce7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.485+0000 7f0e89ffb640 1 -- 172.21.15.116:0/1090069104 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0e94136630 con 0x7f0e9409ea50 2024-09-18T12:25:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.485+0000 7f0e89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.485+0000 7f0e89ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.495+0000 7fd8336a8640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.495+0000 7fd8336a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.495+0000 7fd8336a8640 10 monclient: monmap: 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.497309+0000 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.497309+0000 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.534 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.495+0000 7fc63b51d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.495+0000 7fc63b51d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.495+0000 7fc63b51d640 10 monclient: monmap: 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.497581+0000 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.497581+0000 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.536 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.496+0000 7fd8336a8640 1 Processor -- start 2024-09-18T12:25:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.496+0000 7fd8336a8640 1 -- start start 2024-09-18T12:25:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.496+0000 7fd8336a8640 10 monclient: init 2024-09-18T12:25:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.496+0000 7fc63b51d640 1 Processor -- start 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd82c131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 0x7fd82c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd82c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd82c09f220 0x7fd82c138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd82c09f220 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82c09faf0 con 0x7fd82c131ab0 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82c132590 con 0x7fd82c09f220 2024-09-18T12:25:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c09fd30 con 0x7fd82c09e910 2024-09-18T12:25:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd8336a8640 10 monclient(hunting): authenticate will time out at 756.819316s 2024-09-18T12:25:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fc63b51d640 1 -- start start 2024-09-18T12:25:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fc63b51d640 10 monclient: init 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 0x7fd82c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 0x7fd82c09ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53916/0 (socket says 172.21.15.116:53916) 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 1 -- 172.21.15.116:0/128597400 learned_addr learned my addr 172.21.15.116:0/128597400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): get_auth_request con 0x7fd82c09e910 auth_method 0 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 1 --2- 172.21.15.116:0/128597400 >> v2:172.21.15.144:3300/0 conn(0x7fd82c09f220 0x7fd82c138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 10 monclient(hunting): get_auth_request con 0x7fd82c09f220 auth_method 0 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd831c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-09-18T12:25:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fc63b51d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fc63b51d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:25:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 1 -- 172.21.15.116:0/128597400 >> v2:172.21.15.144:3300/0 conn(0x7fd82c09f220 msgr2=0x7fd82c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fd830c43640 1 --2- 172.21.15.116:0/128597400 >> v2:172.21.15.144:3300/0 conn(0x7fd82c09f220 0x7fd82c138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fc63b51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc63406eb90 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.497+0000 7fc63b51d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc63406ef60 0x7fc63413ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fd830c43640 1 -- 172.21.15.116:0/128597400 >> v1:172.21.15.144:6789/0 conn(0x7fd82c131ab0 legacy=0x7fd82c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fd830c43640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 10 monclient(hunting): picked mon.noname-c con 0x7fc63406ef60 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc63406f360 0x7fc63413d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 10 monclient(hunting): picked mon.noname-a con 0x7fc63406f360 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fd830c43640 1 monclient: found mon.noname-b 2024-09-18T12:25:29.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fd830c43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fd830c43640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fd830c43640 1 -- 172.21.15.116:0/128597400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c0a1110 con 0x7fd82c09e910 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc6392b9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc63406eb90 0x7fc634097a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:48966/0 (socket says 172.21.15.116:48966) 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc6392b9640 1 -- 172.21.15.116:0/865387859 learned_addr learned my addr 172.21.15.116:0/865387859 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc634097320 con 0x7fc63406eb90 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc634097040 con 0x7fc63406f360 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc634096a60 con 0x7fc63406ef60 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc63b51d640 10 monclient(hunting): authenticate will time out at 756.820324s 2024-09-18T12:25:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 1 --2- 172.21.15.116:0/865387859 >> v2:172.21.15.116:3300/0 conn(0x7fc63406f360 0x7fc63413d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): get_auth_request con 0x7fc63406f360 auth_method 0 2024-09-18T12:25:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fdee579d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fdee579d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fdee579d640 10 monclient: monmap: 2024-09-18T12:25:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.499999+0000 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.499999+0000 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.549 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 1 -- 172.21.15.116:0/865387859 >> v2:172.21.15.144:3300/0 conn(0x7fc63406ef60 msgr2=0x7fc63413ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 1 --2- 172.21.15.116:0/865387859 >> v2:172.21.15.144:3300/0 conn(0x7fc63406ef60 0x7fc63413ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 1 -- 172.21.15.116:0/865387859 >> v1:172.21.15.144:6789/0 conn(0x7fc63406eb90 legacy=0x7fc634097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.498+0000 7fc639aba640 1 -- 172.21.15.116:0/865387859 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634096d60 con 0x7fc63406f360 2024-09-18T12:25:29.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 1 Processor -- start 2024-09-18T12:25:29.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 1 -- start start 2024-09-18T12:25:29.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient: init 2024-09-18T12:25:29.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T12:25:29.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdee0096ca0 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdee0135df0 0x7fdee01361f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient(hunting): picked mon.noname-a con 0x7fdee0135df0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdee0136730 0x7fdee013d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient(hunting): picked mon.noname-c con 0x7fdee0136730 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee0097070 con 0x7fdee0096ca0 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee0097a30 con 0x7fdee0135df0 2024-09-18T12:25:29.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee0099620 con 0x7fdee0136730 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.499+0000 7fdee579d640 10 monclient(hunting): authenticate will time out at 756.821920s 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdedeffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdee0096ca0 0x7fdee0098e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57082/0 (socket says 172.21.15.116:57082) 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdedeffd640 1 -- 172.21.15.116:0/3066471020 learned_addr learned my addr 172.21.15.116:0/3066471020 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 1 --2- 172.21.15.116:0/3066471020 >> v2:172.21.15.116:3300/0 conn(0x7fdee0135df0 0x7fdee01361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): get_auth_request con 0x7fdee0135df0 auth_method 0 2024-09-18T12:25:29.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 1 -- 172.21.15.116:0/3066471020 >> v2:172.21.15.144:3300/0 conn(0x7fdee0136730 msgr2=0x7fdee013d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 1 --2- 172.21.15.116:0/3066471020 >> v2:172.21.15.144:3300/0 conn(0x7fdee0136730 0x7fdee013d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 1 -- 172.21.15.116:0/3066471020 >> v1:172.21.15.116:6789/0 conn(0x7fdee0096ca0 legacy=0x7fdee0098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.500+0000 7fdede7fc640 1 -- 172.21.15.116:0/3066471020 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0097e70 con 0x7fdee0135df0 2024-09-18T12:25:29.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd830c43640 1 -- 172.21.15.116:0/128597400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 msgr2=0x7fd82c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd830c43640 1 --2- 172.21.15.116:0/128597400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 0x7fd82c09ece0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009910 tx=0x7fd81c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe7494f535f0869c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc639aba640 1 -- 172.21.15.116:0/865387859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406f360 msgr2=0x7fc63413d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc639aba640 1 --2- 172.21.15.116:0/865387859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406f360 0x7fc63413d370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc6340a2f00 tx=0x7fc62c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390f65eddf9cfea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 1 -- 172.21.15.116:0/865387859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c007840 con 0x7fc63406f360 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: dump: 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 1 -- 172.21.15.116:0/865387859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c02d050 con 0x7fc63406f360 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 1 -- 172.21.15.116:0/865387859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c007bd0 con 0x7fc63406f360 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc62bfff640 10 monclient: dump: 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc63b51d640 5 monclient: authenticate success, global_id 4272 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc63b51d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fc63b51d640 4 set_mon_vals no callback set 2024-09-18T12:25:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd823fff640 1 -- 172.21.15.116:0/128597400 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c032440 con 0x7fd82c09e910 2024-09-18T12:25:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd823fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd823fff640 10 monclient: dump: 2024-09-18T12:25:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd823fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd823fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd8336a8640 5 monclient: authenticate success, global_id 4250 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.501+0000 7fd8336a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd823fff640 1 -- 172.21.15.116:0/128597400 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c0325a0 con 0x7fd82c09e910 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd823fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 4 set_mon_vals no callback set 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd823fff640 1 -- 172.21.15.116:0/128597400 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c032940 con 0x7fd82c09e910 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd823fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd823fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd823fff640 10 monclient: dump: 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 10 monclient: shutdown 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 1 -- 172.21.15.116:0/128597400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 msgr2=0x7fd82c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 1 --2- 172.21.15.116:0/128597400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 0x7fd82c09ece0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009910 tx=0x7fd81c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 1 -- 172.21.15.116:0/128597400 shutdown_connections 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 1 --2- 172.21.15.116:0/128597400 >> v2:172.21.15.144:3300/0 conn(0x7fd82c09f220 0x7fd82c138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 1 --2- 172.21.15.116:0/128597400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c09e910 0x7fd82c09ece0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009910 tx=0x7fd81c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fc63b51d640 10 monclient: shutdown 2024-09-18T12:25:29.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fd8336a8640 1 -- 172.21.15.116:0/128597400 >> 172.21.15.116:0/128597400 conn(0x7fd82c067d30 msgr2=0x7fd82c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fc63b51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406f360 msgr2=0x7fc63413d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.502+0000 7fc63b51d640 1 --2- 172.21.15.116:0/865387859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406f360 0x7fc63413d370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc6340a2f00 tx=0x7fc62c009780 comp rx=0 tx=0).stop 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 shutdown_connections 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 1 --2- 172.21.15.116:0/865387859 >> v2:172.21.15.144:3300/0 conn(0x7fc63406ef60 0x7fc63413ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 1 --2- 172.21.15.116:0/865387859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406f360 0x7fc63413d370 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 >> 172.21.15.116:0/865387859 conn(0x7fc634069350 msgr2=0x7fc6340a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 shutdown_connections 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7f3a87657640 10 monclient: get_monmap_and_config 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7f3a87657640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7f3a87657640 10 monclient: monmap: 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:29.504886+0000 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:29.504886+0000 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:29.569 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fdede7fc640 1 -- 172.21.15.116:0/3066471020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0135df0 msgr2=0x7fdee01361f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fdede7fc640 1 --2- 172.21.15.116:0/3066471020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0135df0 0x7fdee01361f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdee0098850 tx=0x7fded40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590f932db6d27740 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fd8336a8640 1 -- 172.21.15.116:0/128597400 shutdown_connections 2024-09-18T12:25:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fd8336a8640 1 -- 172.21.15.116:0/128597400 wait complete. 2024-09-18T12:25:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fd8336a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fd8336a8640 10 monclient: monmap: 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 1 -- 172.21.15.116:0/865387859 wait complete. 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7fc63b51d640 10 monclient: monmap: 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.572 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.503+0000 7f3a87657640 1 Processor -- start 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 1 -- 172.21.15.116:0/3066471020 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4032440 con 0x7fdee0135df0 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: dump: 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7f3a87657640 1 -- start start 2024-09-18T12:25:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7f3a87657640 10 monclient: init 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fd8336a8640 1 Processor -- start 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 1 -- 172.21.15.116:0/3066471020 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded40325a0 con 0x7fdee0135df0 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 1 -- 172.21.15.116:0/3066471020 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4032940 con 0x7fdee0135df0 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdeddffb640 10 monclient: dump: 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.575 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.576 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fd8336a8640 1 -- start start 2024-09-18T12:25:29.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fd8336a8640 10 monclient: init 2024-09-18T12:25:29.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fc63b51d640 1 Processor -- start 2024-09-18T12:25:29.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdee579d640 5 monclient: authenticate success, global_id 4275 2024-09-18T12:25:29.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdee579d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fdee579d640 4 set_mon_vals no callback set 2024-09-18T12:25:29.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7f3a87657640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7f3a87657640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T12:25:29.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7f3a87657640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3a801318f0 0x7f3a80131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fd8336a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7f3a87657640 10 monclient(hunting): picked mon.noname-c con 0x7f3a801318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:29.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a8009ea50 0x7f3a8009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fd8336a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:29.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 10 monclient(hunting): picked mon.noname-a con 0x7f3a8009ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fd8336a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd82c09f220 0x7fd82c137af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd82c09f220 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c131ab0 0x7fd82c137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd82c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd82c099170 0x7fd82c0995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd82c099170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c09b540 con 0x7fd82c09f220 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c09bde0 con 0x7fd82c099170 2024-09-18T12:25:29.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c1387d0 con 0x7fd82c131ab0 2024-09-18T12:25:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd8336a8640 10 monclient(hunting): authenticate will time out at 756.827090s 2024-09-18T12:25:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a80068d90 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:29.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a8009ea50 0x7f3a8009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fc63b51d640 1 -- start start 2024-09-18T12:25:29.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.504+0000 7fc63b51d640 10 monclient: init 2024-09-18T12:25:29.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd82c09f220 0x7fd82c137af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd82c09f220 0x7fd82c137af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52014/0 (socket says 172.21.15.116:52014) 2024-09-18T12:25:29.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 1 -- 172.21.15.116:0/225138419 learned_addr learned my addr 172.21.15.116:0/225138419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 10 monclient(hunting): get_auth_request con 0x7fd82c09f220 auth_method 0 2024-09-18T12:25:29.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 10 monclient: shutdown 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 -- 172.21.15.116:0/3066471020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0135df0 msgr2=0x7fdee01361f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 --2- 172.21.15.116:0/3066471020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0135df0 0x7fdee01361f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdee0098850 tx=0x7fded40090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 -- 172.21.15.116:0/3066471020 shutdown_connections 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 --2- 172.21.15.116:0/3066471020 >> v2:172.21.15.144:3300/0 conn(0x7fdee0136730 0x7fdee013d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 --2- 172.21.15.116:0/3066471020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0135df0 0x7fdee01361f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 -- 172.21.15.116:0/3066471020 >> 172.21.15.116:0/3066471020 conn(0x7fdee00672b0 msgr2=0x7fdee00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd82c099170 0x7fd82c0995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): get_auth_request con 0x7fd82c099170 auth_method 0 2024-09-18T12:25:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 -- 172.21.15.116:0/3066471020 shutdown_connections 2024-09-18T12:25:29.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fdee579d640 1 -- 172.21.15.116:0/3066471020 wait complete. 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8009fbf0 con 0x7f3a80068d90 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a80131d50 con 0x7f3a8009ea50 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8009fe30 con 0x7f3a801318f0 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a87657640 10 monclient(hunting): authenticate will time out at 756.827389s 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a8009ea50 0x7f3a8009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52012/0 (socket says 172.21.15.116:52012) 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 1 -- 172.21.15.116:0/3801322423 learned_addr learned my addr 172.21.15.116:0/3801322423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 10 monclient(hunting): get_auth_request con 0x7f3a8009ea50 auth_method 0 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 1 --2- 172.21.15.116:0/3801322423 >> v2:172.21.15.144:3300/0 conn(0x7f3a801318f0 0x7f3a80131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): get_auth_request con 0x7f3a801318f0 auth_method 0 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a84bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 1 -- 172.21.15.116:0/3801322423 >> v2:172.21.15.116:3300/0 conn(0x7f3a8009ea50 msgr2=0x7f3a8009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 1 --2- 172.21.15.116:0/3801322423 >> v2:172.21.15.116:3300/0 conn(0x7f3a8009ea50 0x7f3a8009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 1 -- 172.21.15.116:0/3801322423 >> v1:172.21.15.116:6789/0 conn(0x7f3a80068d90 legacy=0x7f3a800691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 1 monclient: found mon.noname-c 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7f3a853f3640 1 -- 172.21.15.116:0/3801322423 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a800a1210 con 0x7f3a801318f0 2024-09-18T12:25:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd830c43640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c131ab0 0x7fd82c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.505+0000 7fd831c45640 1 -- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c131ab0 msgr2=0x7fd82c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c131ab0 0x7fd82c137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 1 -- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd82c09f220 msgr2=0x7fd82c137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd82c09f220 0x7fd82c137af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 1 monclient: found mon.noname-b 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd831c45640 1 -- 172.21.15.116:0/225138419 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c009450 con 0x7fd82c099170 2024-09-18T12:25:29.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd830c43640 10 monclient: get_auth_request con 0x7fd82c131ab0 auth_method 0 2024-09-18T12:25:29.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fd830c43640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c131ab0 0x7fd82c137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:29.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 0x7fc63406cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): picked mon.noname-a con 0x7fc63406eb90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc63406ef60 0x7fc63406d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): picked mon.noname-b con 0x7fc63406ef60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc63406f360 0x7fc63406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): picked mon.noname-c con 0x7fc63406f360 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6341234c0 con 0x7fc63406eb90 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63409e590 con 0x7fc63406ef60 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6340a1510 con 0x7fc63406f360 2024-09-18T12:25:29.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc63b51d640 10 monclient(hunting): authenticate will time out at 756.828222s 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fdee579d640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fdee579d640 10 monclient: monmap: 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.596 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.597 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 0x7fc63406cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 0x7fc63406cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52026/0 (socket says 172.21.15.116:52026) 2024-09-18T12:25:29.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 -- 172.21.15.116:0/1099843084 learned_addr learned my addr 172.21.15.116:0/1099843084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): get_auth_request con 0x7fc63406eb90 auth_method 0 2024-09-18T12:25:29.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc638ab8640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc63406ef60 0x7fc63406d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc63406f360 0x7fc63406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fdee579d640 1 Processor -- start 2024-09-18T12:25:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fdee579d640 1 -- start start 2024-09-18T12:25:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fdee579d640 10 monclient: init 2024-09-18T12:25:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 10 monclient(hunting): get_auth_request con 0x7fc63406f360 auth_method 0 2024-09-18T12:25:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc638ab8640 10 monclient(hunting): get_auth_request con 0x7fc63406ef60 auth_method 0 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc638ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc638ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc638ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 -- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc63406f360 msgr2=0x7fc63406d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc63406f360 0x7fc63406d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 -- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc63406ef60 msgr2=0x7fc63406d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc63406ef60 0x7fc63406d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 monclient: found mon.noname-a 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc6392b9640 1 -- 172.21.15.116:0/1099843084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c009a20 con 0x7fc63406eb90 2024-09-18T12:25:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.506+0000 7fc639aba640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc63406f360 0x7fc63406d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0096ca0 0x7fdee01344d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): picked mon.noname-a con 0x7fdee0096ca0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee0135df0 0x7fdee0134a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): picked mon.noname-b con 0x7fdee0135df0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 0x7fdee006b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): picked mon.noname-c con 0x7fdee0136730 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee006f0b0 con 0x7fdee0096ca0 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee009d1d0 con 0x7fdee0135df0 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee009ed70 con 0x7fdee0136730 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdee579d640 10 monclient(hunting): authenticate will time out at 756.829357s 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 0x7fdee006b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0096ca0 0x7fdee01344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 0x7fdee006b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53940/0 (socket says 172.21.15.116:53940) 2024-09-18T12:25:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 -- 172.21.15.116:0/3367939 learned_addr learned my addr 172.21.15.116:0/3367939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): get_auth_request con 0x7fdee0136730 auth_method 0 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 10 monclient(hunting): get_auth_request con 0x7fdee0096ca0 auth_method 0 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 -- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee0135df0 msgr2=0x7fdee0134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee0135df0 0x7fdee0134a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 -- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0096ca0 msgr2=0x7fdee01344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0096ca0 0x7fdee01344d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 monclient: found mon.noname-c 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.507+0000 7fdedf7fe640 1 -- 172.21.15.116:0/3367939 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded4009390 con 0x7fdee0136730 2024-09-18T12:25:29.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fdedeffd640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0096ca0 0x7fdee01344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd831c45640 1 -- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd82c099170 msgr2=0x7fd82c0995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd831c45640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd82c099170 0x7fd82c0995e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd824009e80 tx=0x7fd82400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 1 -- 172.21.15.116:0/225138419 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824014480 con 0x7fd82c099170 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: dump: 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.609 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 1 -- 172.21.15.116:0/225138419 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82400f040 con 0x7fd82c099170 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 1 -- 172.21.15.116:0/225138419 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824014850 con 0x7fd82c099170 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8217fa640 10 monclient: dump: 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.508+0000 7fd8237fe640 4 set_mon_vals no callback set 2024-09-18T12:25:29.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a853f3640 1 -- 172.21.15.116:0/3801322423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a801318f0 msgr2=0x7f3a80131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fd8336a8640 5 monclient: authenticate success, global_id 4264 2024-09-18T12:25:29.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fd8336a8640 10 monclient: _renew_subs 2024-09-18T12:25:29.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fd8336a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:29.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82c133210 con 0x7fd82c099170 2024-09-18T12:25:29.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a853f3640 1 --2- 172.21.15.116:0/3801322423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a801318f0 0x7f3a80131f30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3a80076600 tx=0x7f3a7c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1037fcb2d67ef708 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fd8336a8640 10 monclient: _renew_subs 2024-09-18T12:25:29.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fd8336a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:29.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82c132ed0 con 0x7fd82c099170 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 1 -- 172.21.15.116:0/3801322423 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c02adf0 con 0x7f3a801318f0 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: dump: 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a87657640 5 monclient: authenticate success, global_id 4267 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a87657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 1 -- 172.21.15.116:0/3801322423 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7c032440 con 0x7f3a801318f0 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 1 -- 172.21.15.116:0/3801322423 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c0327e0 con 0x7f3a801318f0 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a77fff640 10 monclient: dump: 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a87657640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7f3a87657640 4 set_mon_vals no callback set 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6392b9640 1 -- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 msgr2=0x7fc63406cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6392b9640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 0x7fc63406cdd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c5e0 tx=0x7fc630004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6297fa640 1 -- 172.21.15.116:0/1099843084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630005cb0 con 0x7fc63406eb90 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6297fa640 10 monclient: dump: 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6297fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.509+0000 7fc6297fa640 1 -- 172.21.15.116:0/1099843084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630005e10 con 0x7fc63406eb90 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc6297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc6297fa640 1 -- 172.21.15.116:0/1099843084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630007340 con 0x7fc63406eb90 2024-09-18T12:25:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc6297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc6297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc6297fa640 10 monclient: dump: 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc62b7fe640 4 set_mon_vals no callback set 2024-09-18T12:25:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdedf7fe640 1 -- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 msgr2=0x7fdee006b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdedf7fe640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 0x7fdee006b400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fded800bc90 tx=0x7fded8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fd8217fa640 1 -- 172.21.15.116:0/225138419 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd824014bc0 con 0x7fd82c099170 2024-09-18T12:25:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 1 -- 172.21.15.116:0/3367939 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded80092e0 con 0x7fdee0136730 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: dump: 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 1 -- 172.21.15.116:0/3367939 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded800cb40 con 0x7fdee0136730 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 1 -- 172.21.15.116:0/3367939 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded8015470 con 0x7fdee0136730 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdec6ffd640 10 monclient: dump: 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdee579d640 5 monclient: authenticate success, global_id 4256 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdee579d640 10 monclient: _renew_subs 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdee579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee006e760 con 0x7fdee0136730 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 10 monclient: shutdown 2024-09-18T12:25:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 1 -- 172.21.15.116:0/3801322423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a801318f0 msgr2=0x7f3a80131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 1 --2- 172.21.15.116:0/3801322423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a801318f0 0x7f3a80131f30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3a80076600 tx=0x7f3a7c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc63b51d640 5 monclient: authenticate success, global_id 4284 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc63b51d640 10 monclient: _renew_subs 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc63b51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc634121250 con 0x7fc63406eb90 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc63b51d640 10 monclient: _renew_subs 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc63b51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc634120d40 con 0x7fc63406eb90 2024-09-18T12:25:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fd8217fa640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd810038490 0x7fd81003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fd8217fa640 1 -- 172.21.15.116:0/225138419 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd82404c4c0 con 0x7fd82c099170 2024-09-18T12:25:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdee579d640 10 monclient: _renew_subs 2024-09-18T12:25:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdee579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee006cf30 con 0x7fdee0136730 2024-09-18T12:25:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 1 -- 172.21.15.116:0/3801322423 shutdown_connections 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 1 --2- 172.21.15.116:0/3801322423 >> v2:172.21.15.116:3300/0 conn(0x7f3a8009ea50 0x7f3a8009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 1 --2- 172.21.15.116:0/3801322423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a801318f0 0x7f3a80131f30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7f3a87657640 1 -- 172.21.15.116:0/3801322423 >> 172.21.15.116:0/3801322423 conn(0x7f3a80067110 msgr2=0x7f3a80067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fd8336a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fd8336a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fd8336a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82c133550 con 0x7fd82c099170 2024-09-18T12:25:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.510+0000 7fdedd7fa640 4 set_mon_vals no callback set 2024-09-18T12:25:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7f3a87657640 1 -- 172.21.15.116:0/3801322423 shutdown_connections 2024-09-18T12:25:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fdee579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fdee579d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fdee579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee004bf60 con 0x7fdee0136730 2024-09-18T12:25:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.511+0000 7f3a87657640 1 -- 172.21.15.116:0/3801322423 wait complete. 2024-09-18T12:25:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc63b51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc63b51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc63b51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63404bf60 con 0x7fc63406eb90 2024-09-18T12:25:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fd831444640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd810038490 0x7fd81003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc6297fa640 1 -- 172.21.15.116:0/1099843084 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc63001b400 con 0x7fc63406eb90 2024-09-18T12:25:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fd831444640 10 monclient: get_auth_request con 0x7fd810038490 auth_method 0 2024-09-18T12:25:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc6297fa640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc618038490 0x7fc61803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc6297fa640 1 -- 172.21.15.116:0/1099843084 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc63004b150 con 0x7fc63406eb90 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7f3a87657640 10 monclient: build_initial_monmap 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7f3a87657640 10 monclient: monmap: 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:29.631 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc638ab8640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc618038490 0x7fc61803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fc638ab8640 10 monclient: get_auth_request con 0x7fc618038490 auth_method 0 2024-09-18T12:25:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.512+0000 7fdec6ffd640 1 -- 172.21.15.116:0/3367939 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fded8012070 con 0x7fdee0136730 2024-09-18T12:25:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.513+0000 7f3a87657640 1 Processor -- start 2024-09-18T12:25:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.513+0000 7f3a87657640 1 -- start start 2024-09-18T12:25:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.513+0000 7f3a87657640 10 monclient: init 2024-09-18T12:25:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.513+0000 7fdec6ffd640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdecc0384e0 0x7fdecc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.513+0000 7fdec6ffd640 1 -- 172.21.15.116:0/3367939 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fded804b210 con 0x7fdee0136730 2024-09-18T12:25:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.513+0000 7fdedeffd640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdecc0384e0 0x7fdecc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.513+0000 7fdedeffd640 10 monclient: get_auth_request con 0x7fdecc0384e0 auth_method 0 2024-09-18T12:25:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:25:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a80068d90 0x7f3a8009acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): picked mon.noname-b con 0x7f3a80068d90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 0x7f3a80130d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): picked mon.noname-c con 0x7f3a8009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a801318f0 0x7f3a8011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): picked mon.noname-a con 0x7f3a801318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8009e500 con 0x7f3a801318f0 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8011e460 con 0x7f3a80068d90 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a80141cc0 con 0x7f3a8009ea50 2024-09-18T12:25:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a87657640 10 monclient(hunting): authenticate will time out at 756.836394s 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 0x7f3a80130d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a80068d90 0x7f3a8009acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 0x7f3a80130d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53942/0 (socket says 172.21.15.116:53942) 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 -- 172.21.15.116:0/4099293163 learned_addr learned my addr 172.21.15.116:0/4099293163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): get_auth_request con 0x7f3a8009ea50 auth_method 0 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 10 monclient(hunting): get_auth_request con 0x7f3a80068d90 auth_method 0 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.515+0000 7fd831444640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd810038490 0x7fd81003a940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd828003b00 tx=0x7fd828007ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 -- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a80068d90 msgr2=0x7f3a8009acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a80068d90 0x7f3a8009acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 -- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a801318f0 msgr2=0x7f3a8011b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a801318f0 0x7f3a8011b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient: _finish_hunting 0 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 monclient: found mon.noname-c 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a84bf2640 1 -- 172.21.15.116:0/4099293163 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c009390 con 0x7f3a8009ea50 2024-09-18T12:25:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.514+0000 7f3a853f3640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a80068d90 0x7f3a8009acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.515+0000 7fc638ab8640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc618038490 0x7fc61803a940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.515+0000 7fdedeffd640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdecc0384e0 0x7fdecc03a990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fded0009e40 tx=0x7fded0006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.516+0000 7fd8217fa640 1 -- 172.21.15.116:0/225138419 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd82c133550 con 0x7fd82c099170 2024-09-18T12:25:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.516+0000 7fd8217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.516+0000 7fd8217fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:29.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.517+0000 7f3a84bf2640 1 -- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 msgr2=0x7f3a80130d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:29.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.517+0000 7f3a84bf2640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 0x7f3a80130d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000ea20 tx=0x7f3a700091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.517+0000 7fc6297fa640 1 -- 172.21.15.116:0/1099843084 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc63001b6b0 con 0x7fc63406eb90 2024-09-18T12:25:29.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.517+0000 7fc6297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.517+0000 7fc6297fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:29.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7fdec6ffd640 1 -- 172.21.15.116:0/3367939 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fded8007400 con 0x7fdee0136730 2024-09-18T12:25:29.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7fdec6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7fdec6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:29.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 1 -- 172.21.15.116:0/4099293163 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7000cb50 con 0x7f3a8009ea50 2024-09-18T12:25:29.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: dump: 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 1 -- 172.21.15.116:0/4099293163 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7000ccb0 con 0x7f3a8009ea50 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 1 -- 172.21.15.116:0/4099293163 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70013470 con 0x7f3a8009ea50 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a757fa640 10 monclient: dump: 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:29.647 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a777fe640 4 set_mon_vals no callback set 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a87657640 5 monclient: authenticate success, global_id 4259 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a87657640 10 monclient: _renew_subs 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a87657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8011dea0 con 0x7f3a8009ea50 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a87657640 10 monclient: _renew_subs 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a87657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.522+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8011cf40 con 0x7f3a8009ea50 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.523+0000 7f3a87657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.524+0000 7f3a87657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.524+0000 7f3a87657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.524+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8004bf60 con 0x7f3a8009ea50 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.524+0000 7f3a757fa640 1 -- 172.21.15.116:0/4099293163 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3a70013820 con 0x7f3a8009ea50 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.524+0000 7f3a757fa640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a640384f0 0x7f3a6403a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.525+0000 7f3a757fa640 1 -- 172.21.15.116:0/4099293163 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f3a7004b9c0 con 0x7f3a8009ea50 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.525+0000 7f3a853f3640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a640384f0 0x7f3a6403a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.525+0000 7f3a853f3640 10 monclient: get_auth_request con 0x7f3a640384f0 auth_method 0 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.528+0000 7f3a853f3640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a640384f0 0x7f3a6403a9a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3a8009c990 tx=0x7f3a7c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.533+0000 7f3a757fa640 1 -- 172.21.15.116:0/4099293163 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3a70023ad0 con 0x7f3a8009ea50 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.533+0000 7f3a757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.533+0000 7f3a757fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.671+0000 7f0e7f7fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.671+0000 7f0e7f7fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.671+0000 7f0e7f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.671+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0e94136630 con 0x7f0e9409ea50 2024-09-18T12:25:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.671+0000 7f0e89ffb640 1 -- 172.21.15.116:0/1090069104 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0e94136630 con 0x7f0e9409ea50 2024-09-18T12:25:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.671+0000 7f0e89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.671+0000 7f0e89ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:29.674 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.673+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e780384e0 msgr2=0x7f0e7803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.673+0000 7f0e7f7fe640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e780384e0 0x7f0e7803a990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009360 tx=0x7f0e90008040 comp rx=0 tx=0).stop 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.673+0000 7f0e7f7fe640 10 monclient: shutdown 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.673+0000 7f0e7f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.673+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 msgr2=0x7f0e94130860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.673+0000 7f0e7f7fe640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 0x7f0e94130860 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400c590 tx=0x7f0e84004960 comp rx=0 tx=0).stop 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 shutdown_connections 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e780384e0 0x7f0e7803a990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e94136c20 0x7f0e94099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e941318f0 0x7f0e940996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 --2- 172.21.15.116:0/1090069104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e9409ea50 0x7f0e94130860 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 >> 172.21.15.116:0/1090069104 conn(0x7f0e94067110 msgr2=0x7f0e940a3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 shutdown_connections 2024-09-18T12:25:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.674+0000 7f0e7f7fe640 1 -- 172.21.15.116:0/1090069104 wait complete. 2024-09-18T12:25:29.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.3 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.685+0000 7fc63b51d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.685+0000 7fc63b51d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.685+0000 7fc63b51d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.685+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc63413ce30 con 0x7fc63406eb90 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.685+0000 7fc6297fa640 1 -- 172.21.15.116:0/1099843084 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc63001a920 con 0x7fc63406eb90 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.685+0000 7fc6297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.685+0000 7fc6297fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:29.688 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc618038490 msgr2=0x7fc61803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc618038490 0x7fc61803a940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).stop 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 10 monclient: shutdown 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 msgr2=0x7fc63406cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 0x7fc63406cdd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c5e0 tx=0x7fc630004960 comp rx=0 tx=0).stop 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 shutdown_connections 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc618038490 0x7fc61803a940 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc63406f360 0x7fc63406d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc63406ef60 0x7fc63406d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 --2- 172.21.15.116:0/1099843084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63406eb90 0x7fc63406cdd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 >> 172.21.15.116:0/1099843084 conn(0x7fc634069350 msgr2=0x7fc63413a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 shutdown_connections 2024-09-18T12:25:29.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.687+0000 7fc63b51d640 1 -- 172.21.15.116:0/1099843084 wait complete. 2024-09-18T12:25:29.700 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.778+0000 7f2d0671d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.778+0000 7f2d0671d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.778+0000 7f2d0671d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.778+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2d0011cea0 con 0x7f2d00068d90 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.779+0000 7f2ce7fff640 1 -- 172.21.15.116:0/973597852 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2cf0025800 con 0x7f2d00068d90 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.779+0000 7f2ce7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.779+0000 7f2ce7fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:29.781 INFO:teuthology.orchestra.run.smithi116.stdout:47244640258 2024-09-18T12:25:29.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cec038490 msgr2=0x7f2cec03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cec038490 0x7f2cec03a940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4031000 tx=0x7f2cf4032000 comp rx=0 tx=0).stop 2024-09-18T12:25:29.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 10 monclient: shutdown 2024-09-18T12:25:29.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 msgr2=0x7f2d0009ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 0x7f2d0009ab40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009ec0 tx=0x7f2cf00090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 shutdown_connections 2024-09-18T12:25:29.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cec038490 0x7f2cec03a940 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d001318f0 0x7f2d00140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d0009ea50 0x7f2d00130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 --2- 172.21.15.116:0/973597852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d00068d90 0x7f2d0009ab40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 >> 172.21.15.116:0/973597852 conn(0x7f2d00067110 msgr2=0x7f2d000a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 shutdown_connections 2024-09-18T12:25:29.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.781+0000 7f2d0671d640 1 -- 172.21.15.116:0/973597852 wait complete. 2024-09-18T12:25:29.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-18T12:25:29.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.800+0000 7fd8336a8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T12:25:29.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.800+0000 7fd8336a8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T12:25:29.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.800+0000 7fd8336a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:29.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.800+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd82c133550 con 0x7fd82c099170 2024-09-18T12:25:29.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.800+0000 7fd8217fa640 1 -- 172.21.15.116:0/225138419 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd82c133550 con 0x7fd82c099170 2024-09-18T12:25:29.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.800+0000 7fd8217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T12:25:29.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.800+0000 7fd8217fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:29.802 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:29.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd810038490 msgr2=0x7fd81003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd810038490 0x7fd81003a940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd828003b00 tx=0x7fd828007ed0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 10 monclient: shutdown 2024-09-18T12:25:29.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd82c099170 msgr2=0x7fd82c0995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd82c099170 0x7fd82c0995e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd824009e80 tx=0x7fd82400b040 comp rx=0 tx=0).stop 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 shutdown_connections 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd810038490 0x7fd81003a940 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd82c099170 0x7fd82c0995e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd82c131ab0 0x7fd82c137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 --2- 172.21.15.116:0/225138419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd82c09f220 0x7fd82c137af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 >> 172.21.15.116:0/225138419 conn(0x7fd82c067d30 msgr2=0x7fd82c09ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 shutdown_connections 2024-09-18T12:25:29.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.803+0000 7fd8336a8640 1 -- 172.21.15.116:0/225138419 wait complete. 2024-09-18T12:25:29.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-09-18T12:25:29.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.824+0000 7fdee579d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T12:25:29.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.824+0000 7fdee579d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T12:25:29.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.824+0000 7fdee579d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.824+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdee009c5e0 con 0x7fdee0136730 2024-09-18T12:25:29.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.825+0000 7fdec6ffd640 1 -- 172.21.15.116:0/3367939 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fded8012320 con 0x7fdee0136730 2024-09-18T12:25:29.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.825+0000 7fdec6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T12:25:29.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.825+0000 7fdec6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:29.827 INFO:teuthology.orchestra.run.smithi116.stdout:47244640258 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdecc0384e0 msgr2=0x7fdecc03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdecc0384e0 0x7fdecc03a990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fded0009e40 tx=0x7fded0006f50 comp rx=0 tx=0).stop 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 10 monclient: shutdown 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 msgr2=0x7fdee006b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 0x7fdee006b400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fded800bc90 tx=0x7fded8009840 comp rx=0 tx=0).stop 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 shutdown_connections 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdecc0384e0 0x7fdecc03a990 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdee0136730 0x7fdee006b400 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdee0135df0 0x7fdee0134a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.827+0000 7fdee579d640 1 --2- 172.21.15.116:0/3367939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0096ca0 0x7fdee01344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.828+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 >> 172.21.15.116:0/3367939 conn(0x7fdee00672b0 msgr2=0x7fdee009b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.828+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 shutdown_connections 2024-09-18T12:25:29.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.828+0000 7fdee579d640 1 -- 172.21.15.116:0/3367939 wait complete. 2024-09-18T12:25:29.843 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2024-09-18T12:25:29.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.842+0000 7f3a87657640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T12:25:29.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.842+0000 7f3a87657640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T12:25:29.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.842+0000 7f3a87657640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:29.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.842+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3a80062f40 con 0x7f3a8009ea50 2024-09-18T12:25:29.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.843+0000 7f3a757fa640 1 -- 172.21.15.116:0/4099293163 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a700232f0 con 0x7f3a8009ea50 2024-09-18T12:25:29.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.843+0000 7f3a757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T12:25:29.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.843+0000 7f3a757fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:29.845 INFO:teuthology.orchestra.run.smithi116.stdout:47244640258 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a640384f0 msgr2=0x7f3a6403a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a640384f0 0x7f3a6403a9a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3a8009c990 tx=0x7f3a7c032000 comp rx=0 tx=0).stop 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 10 monclient: shutdown 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 msgr2=0x7f3a80130d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 0x7f3a80130d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000ea20 tx=0x7f3a700091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 shutdown_connections 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a640384f0 0x7f3a6403a9a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a801318f0 0x7f3a8011b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a8009ea50 0x7f3a80130d80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 --2- 172.21.15.116:0/4099293163 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a80068d90 0x7f3a8009acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:29.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 >> 172.21.15.116:0/4099293163 conn(0x7f3a80067110 msgr2=0x7f3a800a3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:29.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 shutdown_connections 2024-09-18T12:25:29.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:29.845+0000 7f3a87657640 1 -- 172.21.15.116:0/4099293163 wait complete. 2024-09-18T12:25:29.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-09-18T12:25:30.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T12:25:30.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T12:25:30.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.786+0000 7f31b119d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:30.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.786+0000 7f31b119d640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.786+0000 7f31b119d640 10 monclient: monmap: 2024-09-18T12:25:30.788 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:30.788172+0000 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:30.788172+0000 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:30.789 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 Processor -- start 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 -- start start 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient: init 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f31ac1318f0 0x7f31ac131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): picked mon.noname-a con 0x7f31ac1318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:30.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 0x7f31ac09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): picked mon.noname-b con 0x7f31ac09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f31ac068d90 0x7f31ac0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): picked mon.noname-c con 0x7f31ac068d90 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac131cc0 con 0x7f31ac1318f0 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac09fbf0 con 0x7f31ac068d90 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac09fe30 con 0x7f31ac09ea50 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31b119d640 10 monclient(hunting): authenticate will time out at 758.109685s 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f31ac1318f0 0x7f31ac131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 0x7f31ac09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 0x7f31ac09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53956/0 (socket says 172.21.15.116:53956) 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f31ac1318f0 0x7f31ac131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52064/0 (socket says 172.21.15.116:52064) 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 1 -- 172.21.15.116:0/2385543359 learned_addr learned my addr 172.21.15.116:0/2385543359 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 10 monclient(hunting): get_auth_request con 0x7f31ac09ea50 auth_method 0 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 10 monclient(hunting): get_auth_request con 0x7f31ac1318f0 auth_method 0 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.787+0000 7f31aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 1 -- 172.21.15.116:0/2385543359 >> v2:172.21.15.144:3300/0 conn(0x7f31ac068d90 msgr2=0x7f31ac0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31ab59e640 1 --2- 172.21.15.116:0/2385543359 >> v2:172.21.15.144:3300/0 conn(0x7f31ac068d90 0x7f31ac0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 1 --2- 172.21.15.116:0/2385543359 >> v2:172.21.15.144:3300/0 conn(0x7f31ac068d90 0x7f31ac0691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 1 -- 172.21.15.116:0/2385543359 >> v2:172.21.15.116:3300/0 conn(0x7f31ac1318f0 msgr2=0x7f31ac131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 1 --2- 172.21.15.116:0/2385543359 >> v2:172.21.15.116:3300/0 conn(0x7f31ac1318f0 0x7f31ac131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 10 monclient: _finish_hunting 0 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 1 monclient: found mon.noname-b 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:30.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aa59c640 1 -- 172.21.15.116:0/2385543359 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac0a1210 con 0x7f31ac09ea50 2024-09-18T12:25:30.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31ab59e640 10 monclient: get_auth_request con 0x7f31ac068d90 auth_method 0 2024-09-18T12:25:30.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31aad9d640 1 --2- 172.21.15.116:0/2385543359 >> v2:172.21.15.116:3300/0 conn(0x7f31ac1318f0 0x7f31ac131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:30.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.788+0000 7f31ab59e640 1 --2- 172.21.15.116:0/2385543359 >> v2:172.21.15.144:3300/0 conn(0x7f31ac068d90 0x7f31ac0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:30.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T12:25:30.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.790+0000 7f31aa59c640 1 -- 172.21.15.116:0/2385543359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 msgr2=0x7f31ac09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:30.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.790+0000 7f31aa59c640 1 --2- 172.21.15.116:0/2385543359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 0x7f31ac09ee20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31ac064050 tx=0x7f31a00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f670530cb7a78a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 1 -- 172.21.15.116:0/2385543359 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0032440 con 0x7f31ac09ea50 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: dump: 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: _finish_auth 0 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 1 -- 172.21.15.116:0/2385543359 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a00325a0 con 0x7f31ac09ea50 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 1 -- 172.21.15.116:0/2385543359 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0032940 con 0x7f31ac09ea50 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31a9d9b640 10 monclient: dump: 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 5 monclient: authenticate success, global_id 4262 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 4 set_mon_vals no callback set 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 10 monclient: shutdown 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 -- 172.21.15.116:0/2385543359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 msgr2=0x7f31ac09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 --2- 172.21.15.116:0/2385543359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 0x7f31ac09ee20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31ac064050 tx=0x7f31a00091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 -- 172.21.15.116:0/2385543359 shutdown_connections 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 --2- 172.21.15.116:0/2385543359 >> v2:172.21.15.144:3300/0 conn(0x7f31ac068d90 0x7f31ac0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 --2- 172.21.15.116:0/2385543359 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ea50 0x7f31ac09ee20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 --2- 172.21.15.116:0/2385543359 >> v2:172.21.15.116:3300/0 conn(0x7f31ac1318f0 0x7f31ac131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 -- 172.21.15.116:0/2385543359 >> 172.21.15.116:0/2385543359 conn(0x7f31ac067110 msgr2=0x7f31ac067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 -- 172.21.15.116:0/2385543359 shutdown_connections 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.791+0000 7f31b119d640 1 -- 172.21.15.116:0/2385543359 wait complete. 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.792+0000 7f31b119d640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.792+0000 7f31b119d640 10 monclient: monmap: 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.792+0000 7f31b119d640 1 Processor -- start 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.792+0000 7f31b119d640 1 -- start start 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.792+0000 7f31b119d640 10 monclient: init 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 0x7f31ac09ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): picked mon.noname-c con 0x7f31ac068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac09ea50 0x7f31ac130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): picked mon.noname-b con 0x7f31ac09ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac1318f0 0x7f31ac140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): picked mon.noname-a con 0x7f31ac1318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac09bf80 con 0x7f31ac1318f0 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac11e0e0 con 0x7f31ac09ea50 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac0a4390 con 0x7f31ac068d90 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31b119d640 10 monclient(hunting): authenticate will time out at 758.115213s 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 0x7f31ac09ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac1318f0 0x7f31ac140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 0x7f31ac09ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53960/0 (socket says 172.21.15.116:53960) 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 -- 172.21.15.116:0/276051456 learned_addr learned my addr 172.21.15.116:0/276051456 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac1318f0 0x7f31ac140ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52076/0 (socket says 172.21.15.116:52076) 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): get_auth_request con 0x7f31ac068d90 auth_method 0 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 10 monclient(hunting): get_auth_request con 0x7f31ac1318f0 auth_method 0 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac09ea50 0x7f31ac130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 10 monclient(hunting): get_auth_request con 0x7f31ac09ea50 auth_method 0 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 -- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac09ea50 msgr2=0x7f31ac130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac09ea50 0x7f31ac130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 -- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac1318f0 msgr2=0x7f31ac140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac1318f0 0x7f31ac140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 monclient: found mon.noname-c 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31aad9d640 1 -- 172.21.15.116:0/276051456 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c009390 con 0x7f31ac068d90 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.793+0000 7f31ab59e640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac1318f0 0x7f31ac140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.795+0000 7f31aad9d640 1 -- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 msgr2=0x7f31ac09ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.795+0000 7f31aad9d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 0x7f31ac09ab40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f319c00ecb0 tx=0x7f319c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 1 -- 172.21.15.116:0/276051456 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c00cdd0 con 0x7f31ac068d90 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: dump: 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 1 -- 172.21.15.116:0/276051456 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c013470 con 0x7f31ac068d90 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 1 -- 172.21.15.116:0/276051456 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c013810 con 0x7f31ac068d90 2024-09-18T12:25:30.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31b119d640 5 monclient: authenticate success, global_id 4265 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31b119d640 10 monclient: _renew_subs 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31b119d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a0009450 con 0x7f31ac068d90 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31a959a640 4 set_mon_vals no callback set 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f3192ffd640 10 monclient: dump: 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31b119d640 10 monclient: _renew_subs 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31b119d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.796+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac11c790 con 0x7f31ac068d90 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f3192ffd640 1 -- 172.21.15.116:0/276051456 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f319c013b80 con 0x7f31ac068d90 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f31b119d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f31b119d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f31b119d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ac04bf60 con 0x7f31ac068d90 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f3192ffd640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31980384e0 0x7f319803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f3192ffd640 1 -- 172.21.15.116:0/276051456 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f319c04bb00 con 0x7f31ac068d90 2024-09-18T12:25:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f31aa59c640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31980384e0 0x7f319803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.797+0000 7f31aa59c640 10 monclient: get_auth_request con 0x7f31980384e0 auth_method 0 2024-09-18T12:25:30.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.800+0000 7f31aa59c640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31980384e0 0x7f319803a990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f31ac076600 tx=0x7f31a0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.801+0000 7f3192ffd640 1 -- 172.21.15.116:0/276051456 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f319c0123d0 con 0x7f31ac068d90 2024-09-18T12:25:30.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.801+0000 7f3192ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.801+0000 7f3192ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:30.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T12:25:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.831+0000 7fd0e36c0640 10 monclient: get_monmap_and_config 2024-09-18T12:25:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.831+0000 7fd0e36c0640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.831+0000 7fd0e36c0640 10 monclient: monmap: 2024-09-18T12:25:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:30.833206+0000 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:30.833206+0000 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:30.834 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 1 Processor -- start 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 1 -- start start 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient: init 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0dc131ab0 0x7fd0dc131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient(hunting): picked mon.noname-a con 0x7fd0dc131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0dc09e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd0dc136bb0 0x7fd0dc138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient(hunting): picked mon.noname-c con 0x7fd0dc136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc09faf0 con 0x7fd0dc09e910 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0dc09fd30 con 0x7fd0dc131ab0 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0dc0a1110 con 0x7fd0dc136bb0 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e36c0640 10 monclient(hunting): authenticate will time out at 758.154847s 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.832+0000 7fd0e145c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0dc131ab0 0x7fd0dc131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0dc131ab0 0x7fd0dc131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52092/0 (socket says 172.21.15.116:52092) 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 1 -- 172.21.15.116:0/3949826488 learned_addr learned my addr 172.21.15.116:0/3949826488 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): get_auth_request con 0x7fd0dc131ab0 auth_method 0 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 1 --2- 172.21.15.116:0/3949826488 >> v2:172.21.15.144:3300/0 conn(0x7fd0dc136bb0 0x7fd0dc138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 10 monclient(hunting): get_auth_request con 0x7fd0dc136bb0 auth_method 0 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e1c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 1 -- 172.21.15.116:0/3949826488 >> v2:172.21.15.144:3300/0 conn(0x7fd0dc136bb0 msgr2=0x7fd0dc138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 1 --2- 172.21.15.116:0/3949826488 >> v2:172.21.15.144:3300/0 conn(0x7fd0dc136bb0 0x7fd0dc138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 1 -- 172.21.15.116:0/3949826488 >> v1:172.21.15.144:6789/0 conn(0x7fd0dc09e910 legacy=0x7fd0dc09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient: _finish_hunting 0 2024-09-18T12:25:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 1 monclient: found mon.noname-a 2024-09-18T12:25:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.833+0000 7fd0e145c640 1 -- 172.21.15.116:0/3949826488 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0a13f0 con 0x7fd0dc131ab0 2024-09-18T12:25:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0e145c640 1 -- 172.21.15.116:0/3949826488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 msgr2=0x7fd0dc131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0e145c640 1 --2- 172.21.15.116:0/3949826488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc131e80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009910 tx=0x7fd0d80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1daccbb380301c7a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 1 -- 172.21.15.116:0/3949826488 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d802aea0 con 0x7fd0dc131ab0 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: dump: 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 1 -- 172.21.15.116:0/3949826488 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d8032440 con 0x7fd0dc131ab0 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0e36c0640 5 monclient: authenticate success, global_id 4296 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0e36c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0e36c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 1 -- 172.21.15.116:0/3949826488 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80327e0 con 0x7fd0dc131ab0 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0e36c0640 4 set_mon_vals no callback set 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.836+0000 7fd0d3fff640 10 monclient: dump: 2024-09-18T12:25:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 10 monclient: shutdown 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 -- 172.21.15.116:0/3949826488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 msgr2=0x7fd0dc131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/3949826488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc131e80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009910 tx=0x7fd0d80091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 -- 172.21.15.116:0/3949826488 shutdown_connections 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/3949826488 >> v2:172.21.15.144:3300/0 conn(0x7fd0dc136bb0 0x7fd0dc138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/3949826488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc131e80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 -- 172.21.15.116:0/3949826488 >> 172.21.15.116:0/3949826488 conn(0x7fd0dc067d30 msgr2=0x7fd0dc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 -- 172.21.15.116:0/3949826488 shutdown_connections 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 1 -- 172.21.15.116:0/3949826488 wait complete. 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.837+0000 7fd0e36c0640 10 monclient: monmap: 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 Processor -- start 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 -- start start 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient: init 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0dc09e910 0x7fd0dc0975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): picked mon.noname-b con 0x7fd0dc09e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:30.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc097b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): picked mon.noname-a con 0x7fd0dc131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 0x7fd0dc098060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): picked mon.noname-c con 0x7fd0dc136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc09ac20 con 0x7fd0dc131ab0 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc09bde0 con 0x7fd0dc09e910 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc09d900 con 0x7fd0dc136bb0 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.838+0000 7fd0e36c0640 10 monclient(hunting): authenticate will time out at 758.160963s 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 0x7fd0dc098060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 0x7fd0dc098060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53968/0 (socket says 172.21.15.116:53968) 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 10 monclient(hunting): get_auth_request con 0x7fd0dc131ab0 auth_method 0 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 -- 172.21.15.116:0/1697541346 learned_addr learned my addr 172.21.15.116:0/1697541346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): get_auth_request con 0x7fd0dc136bb0 auth_method 0 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 -- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0dc09e910 msgr2=0x7fd0dc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0dc09e910 0x7fd0dc0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 -- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 msgr2=0x7fd0dc097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient: _finish_hunting 0 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 monclient: found mon.noname-c 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:30.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e1c5d640 1 -- 172.21.15.116:0/1697541346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8009450 con 0x7fd0dc136bb0 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.839+0000 7fd0e0c5b640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc097b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.841+0000 7fd0e1c5d640 1 -- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 msgr2=0x7fd0dc098060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.841+0000 7fd0e1c5d640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 0x7fd0dc098060 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009ec0 tx=0x7fd0d400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 1 -- 172.21.15.116:0/1697541346 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4014480 con 0x7fd0dc136bb0 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 10 monclient: dump: 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 1 -- 172.21.15.116:0/1697541346 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d400f040 con 0x7fd0dc136bb0 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d17fa640 1 -- 172.21.15.116:0/1697541346 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4014810 con 0x7fd0dc136bb0 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0e36c0640 5 monclient: authenticate success, global_id 4268 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0e36c0640 10 monclient: _renew_subs 2024-09-18T12:25:30.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0e36c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc098e10 con 0x7fd0dc136bb0 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0d37fe640 4 set_mon_vals no callback set 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0e36c0640 10 monclient: _renew_subs 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0e36c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.842+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc131560 con 0x7fd0dc136bb0 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0d17fa640 10 monclient: dump: 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.855 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.856 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.856 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T12:25:30.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0d17fa640 1 -- 172.21.15.116:0/1697541346 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd0d4014bc0 con 0x7fd0dc136bb0 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0d17fa640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0c00384e0 0x7fd0c003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0e36c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0e36c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0e36c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc04bf60 con 0x7fd0dc136bb0 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.843+0000 7fd0d17fa640 1 -- 172.21.15.116:0/1697541346 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd0d404c0b0 con 0x7fd0dc136bb0 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.844+0000 7fd0e145c640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0c00384e0 0x7fd0c003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.844+0000 7fd0e145c640 10 monclient: get_auth_request con 0x7fd0c00384e0 auth_method 0 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.846+0000 7fd0e145c640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0c00384e0 0x7fd0c003a990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc09bdb0 tx=0x7fd0d8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.848+0000 7fd0d17fa640 1 -- 172.21.15.116:0/1697541346 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd0d4018070 con 0x7fd0dc136bb0 2024-09-18T12:25:30.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.848+0000 7fd0d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.848+0000 7fd0d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:30.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T12:25:30.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.911+0000 7fd28971d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.911+0000 7fd28971d640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.911+0000 7fd28971d640 10 monclient: monmap: 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:30.913187+0000 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:30.913187+0000 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:30.915 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:30.915 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.912+0000 7fd28971d640 1 Processor -- start 2024-09-18T12:25:30.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.912+0000 7fd28971d640 1 -- start start 2024-09-18T12:25:30.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.912+0000 7fd28971d640 10 monclient: init 2024-09-18T12:25:30.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T12:25:30.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2841318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd28409ea50 0x7fd28409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient(hunting): picked mon.noname-b con 0x7fd28409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd28409ea50 0x7fd28409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2837fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd284136c20 0x7fd284139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57102/0 (socket says 172.21.15.116:57102) 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2837fe640 1 -- 172.21.15.116:0/1641067901 learned_addr learned my addr 172.21.15.116:0/1641067901 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd284136c20 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd284132640 con 0x7fd284136c20 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd284131d00 con 0x7fd2841318f0 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28409fc60 con 0x7fd28409ea50 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd28971d640 10 monclient(hunting): authenticate will time out at 758.235656s 2024-09-18T12:25:30.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 10 monclient(hunting): get_auth_request con 0x7fd28409ea50 auth_method 0 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.913+0000 7fd2827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 1 -- 172.21.15.116:0/1641067901 >> v1:172.21.15.144:6789/0 conn(0x7fd2841318f0 legacy=0x7fd284131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 1 -- 172.21.15.116:0/1641067901 >> v1:172.21.15.116:6789/0 conn(0x7fd284136c20 legacy=0x7fd284139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 1 monclient: found mon.noname-b 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:30.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.914+0000 7fd2827fc640 1 -- 172.21.15.116:0/1641067901 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28409fe30 con 0x7fd28409ea50 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd2827fc640 1 -- 172.21.15.116:0/1641067901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd28409ea50 msgr2=0x7fd28409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd2827fc640 1 --2- 172.21.15.116:0/1641067901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd28409ea50 0x7fd28409ee20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd278009880 tx=0x7fd2780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7c1ec0339d7402 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 1 -- 172.21.15.116:0/1641067901 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27802aea0 con 0x7fd28409ea50 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: dump: 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.927 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 1 -- 172.21.15.116:0/1641067901 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd278032440 con 0x7fd28409ea50 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd28971d640 5 monclient: authenticate success, global_id 4271 2024-09-18T12:25:30.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd28971d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd28971d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 1 -- 172.21.15.116:0/1641067901 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278032820 con 0x7fd28409ea50 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd28971d640 4 set_mon_vals no callback set 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.917+0000 7fd281ffb640 10 monclient: dump: 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.929 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 10 monclient: shutdown 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd28409ea50 msgr2=0x7fd28409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 1 --2- 172.21.15.116:0/1641067901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd28409ea50 0x7fd28409ee20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd278009880 tx=0x7fd2780090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 shutdown_connections 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 1 --2- 172.21.15.116:0/1641067901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd28409ea50 0x7fd28409ee20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 >> 172.21.15.116:0/1641067901 conn(0x7fd284067110 msgr2=0x7fd284067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.918+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 shutdown_connections 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.919+0000 7fd28971d640 1 -- 172.21.15.116:0/1641067901 wait complete. 2024-09-18T12:25:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.919+0000 7fd28971d640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.919+0000 7fd28971d640 10 monclient: monmap: 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.920+0000 7fd28971d640 1 Processor -- start 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.920+0000 7fd28971d640 1 -- start start 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.920+0000 7fd28971d640 10 monclient: init 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2841318f0 0x7fd284135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2841318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284136c20 0x7fd284096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): picked mon.noname-a con 0x7fd284136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 0x7fd2840973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284136c20 0x7fd284096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284136c20 0x7fd284096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52104/0 (socket says 172.21.15.116:52104) 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2841318f0 0x7fd284135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2841318f0 0x7fd284135f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:44530/0 (socket says 172.21.15.116:44530) 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 0x7fd2840973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 0x7fd2840973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53990/0 (socket says 172.21.15.116:53990) 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): picked mon.noname-c con 0x7fd284096fa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840978f0 con 0x7fd284136c20 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28409bf10 con 0x7fd2841318f0 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28409dd00 con 0x7fd284096fa0 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 1 -- 172.21.15.116:0/3747256024 learned_addr learned my addr 172.21.15.116:0/3747256024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd28971d640 10 monclient(hunting): authenticate will time out at 758.243672s 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 10 monclient(hunting): get_auth_request con 0x7fd284136c20 auth_method 0 2024-09-18T12:25:30.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 10 monclient(hunting): get_auth_request con 0x7fd2841318f0 auth_method 0 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 10 monclient(hunting): get_auth_request con 0x7fd284096fa0 auth_method 0 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd282ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.921+0000 7fd2837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 1 -- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2841318f0 msgr2=0x7fd284135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2841318f0 0x7fd284135f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 1 -- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284136c20 msgr2=0x7fd284096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284136c20 0x7fd284096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 1 monclient: found mon.noname-c 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2837fe640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd278009390 con 0x7fd284096fa0 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.922+0000 7fd2827fc640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284136c20 0x7fd284096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.924+0000 7fd2837fe640 1 -- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 msgr2=0x7fd2840973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.924+0000 7fd2837fe640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 0x7fd2840973b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00cb60 tx=0x7fd27c0052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd26affd640 1 -- 172.21.15.116:0/3747256024 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c00a850 con 0x7fd284096fa0 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd26affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd26affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd26affd640 10 monclient: dump: 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.936 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd26affd640 10 monclient: _finish_auth 0 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd26affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd26affd640 1 -- 172.21.15.116:0/3747256024 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c00a9d0 con 0x7fd284096fa0 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd28971d640 5 monclient: authenticate success, global_id 4274 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd28971d640 10 monclient: _renew_subs 2024-09-18T12:25:30.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd28971d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.925+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2840997d0 con 0x7fd284096fa0 2024-09-18T12:25:30.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.926+0000 7fd26affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:30.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.926+0000 7fd2817fa640 4 set_mon_vals no callback set 2024-09-18T12:25:30.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd28971d640 10 monclient: _renew_subs 2024-09-18T12:25:30.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd26affd640 1 -- 172.21.15.116:0/3747256024 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c00ad90 con 0x7fd284096fa0 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd28971d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2840988e0 con 0x7fd284096fa0 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd26affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd26affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd26affd640 10 monclient: dump: 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd26affd640 1 -- 172.21.15.116:0/3747256024 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd27c007300 con 0x7fd284096fa0 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd26affd640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd270038590 0x7fd27003aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.927+0000 7fd26affd640 1 -- 172.21.15.116:0/3747256024 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd27c04e1e0 con 0x7fd284096fa0 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.928+0000 7fd282ffd640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd270038590 0x7fd27003aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.928+0000 7fd282ffd640 10 monclient: get_auth_request con 0x7fd270038590 auth_method 0 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.928+0000 7fd28971d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.928+0000 7fd28971d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.928+0000 7fd28971d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.928+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd284099c20 con 0x7fd284096fa0 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.930+0000 7fd282ffd640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd270038590 0x7fd27003aa40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd274003300 tx=0x7fd274014040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.937+0000 7fd26affd640 1 -- 172.21.15.116:0/3747256024 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd27c01bae0 con 0x7fd284096fa0 2024-09-18T12:25:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.937+0000 7fd26affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.937+0000 7fd26affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.971+0000 7f8c5935d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.971+0000 7f8c5935d640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.972+0000 7f8c5935d640 10 monclient: monmap: 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:30.973679+0000 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:30.973679+0000 2024-09-18T12:25:30.974 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.975 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.975 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:30.975 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:30.975 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:30.975 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:30.975 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:30.975 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 1 Processor -- start 2024-09-18T12:25:30.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 1 -- start start 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 10 monclient: init 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c541318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c5409ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.973+0000 7f8c5935d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c5413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c52ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8c541318f0 0x7f8c54131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57108/0 (socket says 172.21.15.116:57108) 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c5413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c4bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f8c5409ea50 0x7f8c5409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:49008/0 (socket says 172.21.15.116:49008) 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c5413a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54006/0 (socket says 172.21.15.116:54006) 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c5935d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c54137c30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c52ffd640 1 -- 172.21.15.116:0/2031724079 learned_addr learned my addr 172.21.15.116:0/2031724079 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c54132640 con 0x7f8c541318f0 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c54131cc0 con 0x7f8c5409ea50 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c5935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5409f530 con 0x7f8c54137c30 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c5935d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c5935d640 10 monclient(hunting): authenticate will time out at 758.296512s 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): get_auth_request con 0x7f8c54137c30 auth_method 0 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 1 -- 172.21.15.116:0/2031724079 >> v1:172.21.15.144:6789/0 conn(0x7f8c5409ea50 legacy=0x7f8c5409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 1 -- 172.21.15.116:0/2031724079 >> v1:172.21.15.116:6789/0 conn(0x7f8c541318f0 legacy=0x7f8c54131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 1 monclient: found mon.noname-b 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.974+0000 7f8c537fe640 1 -- 172.21.15.116:0/2031724079 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5409fbf0 con 0x7f8c54137c30 2024-09-18T12:25:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c537fe640 1 -- 172.21.15.116:0/2031724079 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 msgr2=0x7f8c5413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c537fe640 1 --2- 172.21.15.116:0/2031724079 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c5413a010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00b180 tx=0x7f8c4c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3de418739ca0d5c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 1 -- 172.21.15.116:0/2031724079 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c007630 con 0x7f8c54137c30 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: dump: 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 1 -- 172.21.15.116:0/2031724079 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c02d050 con 0x7f8c54137c30 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 1 -- 172.21.15.116:0/2031724079 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c0079c0 con 0x7f8c54137c30 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c527fc640 10 monclient: dump: 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c5935d640 5 monclient: authenticate success, global_id 4277 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c5935d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.978+0000 7f8c5935d640 4 set_mon_vals no callback set 2024-09-18T12:25:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.979+0000 7f8c5935d640 10 monclient: shutdown 2024-09-18T12:25:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.979+0000 7f8c5935d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.979+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 msgr2=0x7f8c5413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.979+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2031724079 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c5413a010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00b180 tx=0x7f8c4c009780 comp rx=0 tx=0).stop 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 shutdown_connections 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2031724079 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c5413a010 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 >> 172.21.15.116:0/2031724079 conn(0x7f8c54067110 msgr2=0x7f8c54067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 shutdown_connections 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f8c5935d640 1 -- 172.21.15.116:0/2031724079 wait complete. 2024-09-18T12:25:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f68474b8640 10 monclient: get_monmap_and_config 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f68474b8640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.980+0000 7f68474b8640 10 monclient: monmap: 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:30.982504+0000 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:30.982504+0000 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f8c5935d640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f8c5935d640 10 monclient: monmap: 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.988 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:30.988 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:30.988 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:30.988 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f68474b8640 1 Processor -- start 2024-09-18T12:25:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f8c5935d640 1 Processor -- start 2024-09-18T12:25:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f8c5935d640 1 -- start start 2024-09-18T12:25:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f8c5935d640 10 monclient: init 2024-09-18T12:25:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f68474b8640 1 -- start start 2024-09-18T12:25:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.981+0000 7f68474b8640 10 monclient: init 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f4f56a99640 10 monclient: get_monmap_and_config 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f4f56a99640 10 monclient: build_initial_monmap 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f4f56a99640 10 monclient: monmap: 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:30.984259+0000 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:30.984259+0000 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:30.991 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:30.991 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:30.991 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:30.991 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:30.991 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:30.991 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:30.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f8c5935d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f8c5935d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:30.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f8c5935d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 0x7f8c54137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f68474b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f68474b8640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T12:25:30.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f68474b8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6840067820 0x7f684009cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.982+0000 7f68474b8640 10 monclient(hunting): picked mon.noname-c con 0x7f6840067820 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:30.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 0x7f8c54137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 0x7f8c54137270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52118/0 (socket says 172.21.15.116:52118) 2024-09-18T12:25:30.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f684009d060 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:30.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f684009e0c0 0x7f68401377d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:30.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c52ffd640 1 -- 172.21.15.116:0/2222570180 learned_addr learned my addr 172.21.15.116:0/2222570180 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:30.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f4f56a99640 1 Processor -- start 2024-09-18T12:25:30.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f4f56a99640 1 -- start start 2024-09-18T12:25:30.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 10 monclient(hunting): picked mon.noname-b con 0x7f684009e0c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:30.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6840067820 0x7f684009cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6844a53640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f684009d060 0x7f684009d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:49018/0 (socket says 172.21.15.116:49018) 2024-09-18T12:25:30.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6840067820 0x7f684009cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:44538/0 (socket says 172.21.15.116:44538) 2024-09-18T12:25:30.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6844a53640 1 -- 172.21.15.116:0/813216536 learned_addr learned my addr 172.21.15.116:0/813216536 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:30.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f4f56a99640 10 monclient: init 2024-09-18T12:25:30.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684009d460 con 0x7f684009d060 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 1 --2- 172.21.15.116:0/813216536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f684009e0c0 0x7f68401377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684009f420 con 0x7f6840067820 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6840067bf0 con 0x7f684009e0c0 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f68474b8640 10 monclient(hunting): authenticate will time out at 758.305625s 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): get_auth_request con 0x7f6840067820 auth_method 0 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 10 monclient(hunting): get_auth_request con 0x7f684009e0c0 auth_method 0 2024-09-18T12:25:30.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:30.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:30.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:30.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): picked mon.noname-a con 0x7f8c5409ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:30.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:30.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:30.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:30.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:30.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:25:30.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f501318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:31.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f5009ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:31.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f50137c30 0x7f4f5013a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient(hunting): picked mon.noname-b con 0x7f4f50137c30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:31.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f50132640 con 0x7f4f5009ea50 2024-09-18T12:25:31.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f50131cc0 con 0x7f4f501318f0 2024-09-18T12:25:31.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5009f530 con 0x7f4f50137c30 2024-09-18T12:25:31.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2024-09-18T12:25:31.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 1 -- 172.21.15.116:0/813216536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f684009e0c0 msgr2=0x7f68401377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:31.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f6845254640 1 --2- 172.21.15.116:0/813216536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f684009e0c0 0x7f68401377d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f6845254640 1 -- 172.21.15.116:0/813216536 >> v1:172.21.15.144:6789/0 conn(0x7f684009d060 legacy=0x7f684009d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:31.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f6845254640 10 monclient: _finish_hunting 0 2024-09-18T12:25:31.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f6845254640 1 monclient: found mon.noname-c 2024-09-18T12:25:31.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f6845254640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:31.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f6845254640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:31.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f6845254640 1 -- 172.21.15.116:0/813216536 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684009af80 con 0x7f6840067820 2024-09-18T12:25:31.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f4ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4f5009ea50 0x7f4f5009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57124/0 (socket says 172.21.15.116:57124) 2024-09-18T12:25:31.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f4ffff640 1 -- 172.21.15.116:0/43943195 learned_addr learned my addr 172.21.15.116:0/43943195 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:31.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:31.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f56a99640 10 monclient(hunting): authenticate will time out at 758.306417s 2024-09-18T12:25:31.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c541318f0 0x7f8c54140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c541318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c54141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): picked mon.noname-c con 0x7f8c54137c30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5412fca0 con 0x7f8c5409ea50 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5413a2e0 con 0x7f8c541318f0 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5412f990 con 0x7f8c54137c30 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:31.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.983+0000 7f8c5935d640 10 monclient(hunting): authenticate will time out at 758.305893s 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): get_auth_request con 0x7f8c5409ea50 auth_method 0 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 1 --2- 172.21.15.116:0/43943195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f50137c30 0x7f4f5013a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 10 monclient(hunting): get_auth_request con 0x7f4f50137c30 auth_method 0 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 1 -- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 msgr2=0x7f8c54141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c54141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 1 -- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c541318f0 msgr2=0x7f8c54140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c541318f0 0x7f8c54140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:31.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 1 monclient: found mon.noname-a 2024-09-18T12:25:31.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:31.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f8c52ffd640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c009a20 con 0x7f8c5409ea50 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.984+0000 7f4f55036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 1 -- 172.21.15.116:0/43943195 >> v1:172.21.15.144:6789/0 conn(0x7f4f501318f0 legacy=0x7f4f50131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 1 -- 172.21.15.116:0/43943195 >> v1:172.21.15.116:6789/0 conn(0x7f4f5009ea50 legacy=0x7f4f5009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 10 monclient: _finish_hunting 0 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 1 monclient: found mon.noname-b 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:31.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:31.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.985+0000 7f4f55036640 1 -- 172.21.15.116:0/43943195 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5009fbf0 con 0x7f4f50137c30 2024-09-18T12:25:31.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c52ffd640 1 -- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 msgr2=0x7f8c54137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:31.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c52ffd640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 0x7f8c54137270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c440055d0 tx=0x7f8c44004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 1 -- 172.21.15.116:0/2222570180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c440138e0 con 0x7f8c5409ea50 2024-09-18T12:25:31.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6845254640 1 -- 172.21.15.116:0/813216536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6840067820 msgr2=0x7f684009cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:31.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:31.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: dump: 2024-09-18T12:25:31.010 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.010 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.010 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: _finish_auth 0 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 1 -- 172.21.15.116:0/2222570180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c44013a60 con 0x7f8c5409ea50 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 1 -- 172.21.15.116:0/2222570180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c44013e20 con 0x7f8c5409ea50 2024-09-18T12:25:31.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c4affd640 10 monclient: dump: 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c51ffb640 4 set_mon_vals no callback set 2024-09-18T12:25:31.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6845254640 1 --2- 172.21.15.116:0/813216536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6840067820 0x7f684009cb20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f683c009850 tx=0x7f683c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7566edab9b8e58e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c5935d640 5 monclient: authenticate success, global_id 4305 2024-09-18T12:25:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c5935d640 10 monclient: _renew_subs 2024-09-18T12:25:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c5935d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c541314e0 con 0x7f8c5409ea50 2024-09-18T12:25:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c5935d640 10 monclient: _renew_subs 2024-09-18T12:25:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c5935d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c54130300 con 0x7f8c5409ea50 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 1 -- 172.21.15.116:0/813216536 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c02add0 con 0x7f6840067820 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 10 monclient: dump: 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f68474b8640 5 monclient: authenticate success, global_id 4285 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f68474b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 1 -- 172.21.15.116:0/813216536 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c032440 con 0x7f6840067820 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f6837fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f68474b8640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.987+0000 7f68474b8640 4 set_mon_vals no callback set 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f6837fff640 1 -- 172.21.15.116:0/813216536 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c02add0 con 0x7f6840067820 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f6837fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f6837fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f6837fff640 10 monclient: dump: 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.018 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f55036640 1 -- 172.21.15.116:0/43943195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f50137c30 msgr2=0x7f4f5013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f55036640 1 --2- 172.21.15.116:0/43943195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f50137c30 0x7f4f5013a010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009e90 tx=0x7f4f48009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c775f7e105f52720 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 10 monclient: shutdown 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6840067820 msgr2=0x7f684009cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 --2- 172.21.15.116:0/813216536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6840067820 0x7f684009cb20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f683c009850 tx=0x7f683c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 shutdown_connections 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 --2- 172.21.15.116:0/813216536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f684009e0c0 0x7f68401377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 --2- 172.21.15.116:0/813216536 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6840067820 0x7f684009cb20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 >> 172.21.15.116:0/813216536 conn(0x7f68400a3a40 msgr2=0x7f6840067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 shutdown_connections 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 1 -- 172.21.15.116:0/43943195 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48007840 con 0x7f4f50137c30 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: dump: 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 1 -- 172.21.15.116:0/43943195 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4802d050 con 0x7f4f50137c30 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 1 -- 172.21.15.116:0/43943195 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48007bd0 con 0x7f4f50137c30 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f4f7fe640 10 monclient: dump: 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f4f56a99640 5 monclient: authenticate success, global_id 4283 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f4f56a99640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f4f56a99640 4 set_mon_vals no callback set 2024-09-18T12:25:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f68474b8640 1 -- 172.21.15.116:0/813216536 wait complete. 2024-09-18T12:25:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.988+0000 7f8c4affd640 1 -- 172.21.15.116:0/2222570180 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8c44005aa0 con 0x7f8c5409ea50 2024-09-18T12:25:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c4affd640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8c340384e0 0x7f8c3403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c4affd640 1 -- 172.21.15.116:0/2222570180 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f8c44053740 con 0x7f8c5409ea50 2024-09-18T12:25:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c5935d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c4bfff640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8c340384e0 0x7f8c3403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f68474b8640 10 monclient: build_initial_monmap 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f68474b8640 10 monclient: monmap: 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:31.026 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c5935d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c5935d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f4f56a99640 10 monclient: shutdown 2024-09-18T12:25:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f4f56a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f4f56a99640 1 -- 172.21.15.116:0/43943195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f50137c30 msgr2=0x7f4f5013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f4f56a99640 1 --2- 172.21.15.116:0/43943195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f50137c30 0x7f4f5013a010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009e90 tx=0x7f4f48009780 comp rx=0 tx=0).stop 2024-09-18T12:25:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c54131310 con 0x7f8c5409ea50 2024-09-18T12:25:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.989+0000 7f8c4bfff640 10 monclient: get_auth_request con 0x7f8c340384e0 auth_method 0 2024-09-18T12:25:31.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f4f56a99640 1 -- 172.21.15.116:0/43943195 shutdown_connections 2024-09-18T12:25:31.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f4f56a99640 1 --2- 172.21.15.116:0/43943195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f50137c30 0x7f4f5013a010 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f4f56a99640 1 -- 172.21.15.116:0/43943195 >> 172.21.15.116:0/43943195 conn(0x7f4f50067110 msgr2=0x7f4f50067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f68474b8640 1 Processor -- start 2024-09-18T12:25:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f4f56a99640 1 -- 172.21.15.116:0/43943195 shutdown_connections 2024-09-18T12:25:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f4f56a99640 1 -- 172.21.15.116:0/43943195 wait complete. 2024-09-18T12:25:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f68474b8640 1 -- start start 2024-09-18T12:25:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.990+0000 7f68474b8640 10 monclient: init 2024-09-18T12:25:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f4f56a99640 10 monclient: build_initial_monmap 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f4f56a99640 10 monclient: monmap: 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.031 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:31.031 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:31.031 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:31.031 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6840067820 0x7f6840141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): picked mon.noname-c con 0x7f6840067820 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 0x7f6840135300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): picked mon.noname-b con 0x7f684009d060 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f684009e0c0 0x7f6840145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): picked mon.noname-a con 0x7f684009e0c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6840098f40 con 0x7f684009e0c0 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684011cc90 con 0x7f684009d060 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684011cf70 con 0x7f6840067820 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:31.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f68474b8640 10 monclient(hunting): authenticate will time out at 758.313329s 2024-09-18T12:25:31.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 0x7f6840135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f4f56a99640 1 Processor -- start 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6845a55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f684009e0c0 0x7f6840145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 0x7f6840135300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:44554/0 (socket says 172.21.15.116:44554) 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 1 -- 172.21.15.116:0/1371764616 learned_addr learned my addr 172.21.15.116:0/1371764616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 10 monclient(hunting): get_auth_request con 0x7f684009d060 auth_method 0 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6845254640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6840067820 0x7f6840141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f4f56a99640 1 -- start start 2024-09-18T12:25:31.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f4f56a99640 10 monclient: init 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6845a55640 10 monclient(hunting): get_auth_request con 0x7f684009e0c0 auth_method 0 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6845a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6845a55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6845a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.991+0000 7f6844a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 10 monclient(hunting): get_auth_request con 0x7f6840067820 auth_method 0 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 1 -- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6840067820 msgr2=0x7f6840141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6840067820 0x7f6840141fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 1 -- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f684009e0c0 msgr2=0x7f6840145510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f684009e0c0 0x7f6840145510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 10 monclient: _finish_hunting 0 2024-09-18T12:25:31.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 1 monclient: found mon.noname-b 2024-09-18T12:25:31.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:31.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:31.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f8c4bfff640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8c340384e0 0x7f8c3403a990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c5412f5f0 tx=0x7f8c3c007ac0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6844a53640 1 -- 172.21.15.116:0/1371764616 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c009390 con 0x7f684009d060 2024-09-18T12:25:31.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845a55640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f684009e0c0 0x7f6840145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:31.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.992+0000 7f6845254640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6840067820 0x7f6840141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5009ea50 0x7f4f5009b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): picked mon.noname-c con 0x7f4f5009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f501318f0 0x7f4f50096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f54835640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5009ea50 0x7f4f5009b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f54835640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5009ea50 0x7f4f5009b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54062/0 (socket says 172.21.15.116:54062) 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): picked mon.noname-b con 0x7f4f501318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f54835640 1 -- 172.21.15.116:0/988274019 learned_addr learned my addr 172.21.15.116:0/988274019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:31.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50137c30 0x7f4f50096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): picked mon.noname-a con 0x7f4f50137c30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f4ffff640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f501318f0 0x7f4f50096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f55036640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50137c30 0x7f4f50096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5009bf10 con 0x7f4f50137c30 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5009d0a0 con 0x7f4f501318f0 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50099780 con 0x7f4f5009ea50 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f56a99640 10 monclient(hunting): authenticate will time out at 758.315907s 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f54835640 10 monclient(hunting): get_auth_request con 0x7f4f5009ea50 auth_method 0 2024-09-18T12:25:31.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f54835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f54835640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f54835640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.993+0000 7f4f4ffff640 10 monclient(hunting): get_auth_request con 0x7f4f501318f0 auth_method 0 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): get_auth_request con 0x7f4f50137c30 auth_method 0 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f54835640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f54835640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f54835640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 1 -- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5009ea50 msgr2=0x7f4f5009b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5009ea50 0x7f4f5009b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 1 -- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f501318f0 msgr2=0x7f4f50096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f501318f0 0x7f4f50096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient: _finish_hunting 0 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 1 monclient: found mon.noname-a 2024-09-18T12:25:31.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:31.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f4f55036640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f48009a20 con 0x7f4f50137c30 2024-09-18T12:25:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f6844a53640 1 -- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 msgr2=0x7f6840135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f6844a53640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 0x7f6840135300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f683000c610 tx=0x7f6830004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f68357fa640 1 -- 172.21.15.116:0/1371764616 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830005cb0 con 0x7f684009d060 2024-09-18T12:25:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.994+0000 7f68357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:31.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 10 monclient: dump: 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:31.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 1 -- 172.21.15.116:0/1371764616 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6830005e10 con 0x7f684009d060 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68474b8640 5 monclient: authenticate success, global_id 4288 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68474b8640 10 monclient: _renew_subs 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68474b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684011d250 con 0x7f684009d060 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68474b8640 10 monclient: _renew_subs 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f8c4affd640 1 -- 172.21.15.116:0/2222570180 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8c4401bd60 con 0x7f8c5409ea50 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f8c4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f8c4affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68474b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684011a860 con 0x7f684009d060 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 1 -- 172.21.15.116:0/1371764616 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830007340 con 0x7f684009d060 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68357fa640 10 monclient: dump: 2024-09-18T12:25:31.052 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.995+0000 7f68377fe640 4 set_mon_vals no callback set 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f68357fa640 1 -- 172.21.15.116:0/1371764616 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f683001b400 con 0x7f684009d060 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f68474b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f68357fa640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6824038490 0x7f682403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f68474b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f68474b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:31.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6820005010 con 0x7f684009d060 2024-09-18T12:25:31.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f68357fa640 1 -- 172.21.15.116:0/1371764616 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f683004b0e0 con 0x7f684009d060 2024-09-18T12:25:31.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f6845254640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6824038490 0x7f682403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f4f55036640 1 -- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50137c30 msgr2=0x7f4f50096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:31.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f4f55036640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50137c30 0x7f4f50096fa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f48034000 tx=0x7f4f48037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.996+0000 7f6845254640 10 monclient: get_auth_request con 0x7f6824038490 auth_method 0 2024-09-18T12:25:31.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 1 -- 172.21.15.116:0/988274019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48031cd0 con 0x7f4f50137c30 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: dump: 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: _finish_auth 0 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 1 -- 172.21.15.116:0/988274019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f48040070 con 0x7f4f50137c30 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f56a99640 5 monclient: authenticate success, global_id 4311 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f56a99640 10 monclient: _renew_subs 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f56a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f50099a60 con 0x7f4f50137c30 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 1 -- 172.21.15.116:0/988274019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f480456d0 con 0x7f4f50137c30 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4cff9640 10 monclient: dump: 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f4effd640 4 set_mon_vals no callback set 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f56a99640 10 monclient: _renew_subs 2024-09-18T12:25:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f56a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.997+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5009a440 con 0x7f4f50137c30 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.998+0000 7f4f4cff9640 1 -- 172.21.15.116:0/988274019 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4f48003020 con 0x7f4f50137c30 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.998+0000 7f4f56a99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.998+0000 7f4f56a99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.998+0000 7f4f56a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.998+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5004bf60 con 0x7f4f50137c30 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.998+0000 7f4f4cff9640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f34038510 0x7f4f3403a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:30.999+0000 7f4f4cff9640 1 -- 172.21.15.116:0/988274019 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f4f4806ce90 con 0x7f4f50137c30 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.000+0000 7f4f54835640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f34038510 0x7f4f3403a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.000+0000 7f6845254640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6824038490 0x7f682403a940 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f684009bfc0 tx=0x7f683c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.000+0000 7f4f54835640 10 monclient: get_auth_request con 0x7f4f34038510 auth_method 0 2024-09-18T12:25:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.001+0000 7f68357fa640 1 -- 172.21.15.116:0/1371764616 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6830012760 con 0x7f684009d060 2024-09-18T12:25:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.001+0000 7f68357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.001+0000 7f68357fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.003+0000 7f4f54835640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f34038510 0x7f4f3403a9c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009af0 tx=0x7f4f440090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.007+0000 7f4f4cff9640 1 -- 172.21.15.116:0/988274019 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4f4803bb00 con 0x7f4f50137c30 2024-09-18T12:25:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.007+0000 7f4f4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.007+0000 7f4f4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.047+0000 7fd0e36c0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.047+0000 7fd0e36c0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.047+0000 7fd0e36c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.047+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd0dc066280 con 0x7fd0dc136bb0 2024-09-18T12:25:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.048+0000 7fd0d17fa640 1 -- 172.21.15.116:0/1697541346 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd0d401c3b0 con 0x7fd0dc136bb0 2024-09-18T12:25:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.048+0000 7fd0d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.048+0000 7fd0d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:31.063 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:31.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.050+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0c00384e0 msgr2=0x7fd0c003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.050+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0c00384e0 0x7fd0c003a990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc09bdb0 tx=0x7fd0d8032000 comp rx=0 tx=0).stop 2024-09-18T12:25:31.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.050+0000 7fd0e36c0640 10 monclient: shutdown 2024-09-18T12:25:31.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.050+0000 7fd0e36c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.050+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 msgr2=0x7fd0dc098060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.050+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 0x7fd0dc098060 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009ec0 tx=0x7fd0d400b040 comp rx=0 tx=0).stop 2024-09-18T12:25:31.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 shutdown_connections 2024-09-18T12:25:31.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0c00384e0 0x7fd0c003a990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0dc136bb0 0x7fd0dc098060 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0dc131ab0 0x7fd0dc097b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 --2- 172.21.15.116:0/1697541346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0dc09e910 0x7fd0dc0975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 >> 172.21.15.116:0/1697541346 conn(0x7fd0dc067d30 msgr2=0x7fd0dc09c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 shutdown_connections 2024-09-18T12:25:31.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.051+0000 7fd0e36c0640 1 -- 172.21.15.116:0/1697541346 wait complete. 2024-09-18T12:25:31.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-09-18T12:25:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.070+0000 7f31b119d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.070+0000 7f31b119d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.070+0000 7f31b119d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.070+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f31ac11e550 con 0x7f31ac068d90 2024-09-18T12:25:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.071+0000 7f3192ffd640 1 -- 172.21.15.116:0/276051456 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f319c04be60 con 0x7f31ac068d90 2024-09-18T12:25:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.071+0000 7f3192ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.071+0000 7f3192ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:31.073 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.073+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31980384e0 msgr2=0x7f319803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.073+0000 7f31b119d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31980384e0 0x7f319803a990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f31ac076600 tx=0x7f31a0032000 comp rx=0 tx=0).stop 2024-09-18T12:25:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.073+0000 7f31b119d640 10 monclient: shutdown 2024-09-18T12:25:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.073+0000 7f31b119d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.073+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 msgr2=0x7f31ac09ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.073+0000 7f31b119d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 0x7f31ac09ab40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f319c00ecb0 tx=0x7f319c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 shutdown_connections 2024-09-18T12:25:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31980384e0 0x7f319803a990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac1318f0 0x7f31ac140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac09ea50 0x7f31ac130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 --2- 172.21.15.116:0/276051456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac068d90 0x7f31ac09ab40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 >> 172.21.15.116:0/276051456 conn(0x7f31ac067110 msgr2=0x7f31ac0a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 shutdown_connections 2024-09-18T12:25:31.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.074+0000 7f31b119d640 1 -- 172.21.15.116:0/276051456 wait complete. 2024-09-18T12:25:31.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.3 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.163+0000 7f8c5935d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.163+0000 7f8c5935d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.163+0000 7f8c5935d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.163+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8c5404bf60 con 0x7f8c5409ea50 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.164+0000 7f8c4affd640 1 -- 172.21.15.116:0/2222570180 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c4401b760 con 0x7f8c5409ea50 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.164+0000 7f8c4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.164+0000 7f8c4affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:31.166 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8c340384e0 msgr2=0x7f8c3403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8c340384e0 0x7f8c3403a990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c5412f5f0 tx=0x7f8c3c007ac0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 10 monclient: shutdown 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 msgr2=0x7f8c54137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 0x7f8c54137270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c440055d0 tx=0x7f8c44004960 comp rx=0 tx=0).stop 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 shutdown_connections 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8c340384e0 0x7f8c3403a990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8c54137c30 0x7f8c54141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c541318f0 0x7f8c54140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 --2- 172.21.15.116:0/2222570180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c5409ea50 0x7f8c54137270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 >> 172.21.15.116:0/2222570180 conn(0x7f8c54067110 msgr2=0x7f8c5409d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 shutdown_connections 2024-09-18T12:25:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.166+0000 7f8c5935d640 1 -- 172.21.15.116:0/2222570180 wait complete. 2024-09-18T12:25:31.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2024-09-18T12:25:31.179 DEBUG:teuthology.parallel:result is None 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.198+0000 7fd28971d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.198+0000 7fd28971d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.198+0000 7fd28971d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.198+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd28404bf60 con 0x7fd284096fa0 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.199+0000 7fd26affd640 1 -- 172.21.15.116:0/3747256024 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd27c010300 con 0x7fd284096fa0 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.199+0000 7fd26affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.199+0000 7fd26affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:31.201 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:31.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd270038590 msgr2=0x7fd27003aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd270038590 0x7fd27003aa40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd274003300 tx=0x7fd274014040 comp rx=0 tx=0).stop 2024-09-18T12:25:31.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 10 monclient: shutdown 2024-09-18T12:25:31.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 msgr2=0x7fd2840973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 0x7fd2840973b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00cb60 tx=0x7fd27c0052b0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 shutdown_connections 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd270038590 0x7fd27003aa40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd284096fa0 0x7fd2840973b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd284136c20 0x7fd284096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 --2- 172.21.15.116:0/3747256024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2841318f0 0x7fd284135f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 >> 172.21.15.116:0/3747256024 conn(0x7fd284067110 msgr2=0x7fd28409df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 shutdown_connections 2024-09-18T12:25:31.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.202+0000 7fd28971d640 1 -- 172.21.15.116:0/3747256024 wait complete. 2024-09-18T12:25:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.213+0000 7f68474b8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T12:25:31.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.213+0000 7f68474b8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T12:25:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.213+0000 7f68474b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.213+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f68200052c0 con 0x7f684009d060 2024-09-18T12:25:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.214+0000 7f68357fa640 1 -- 172.21.15.116:0/1371764616 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6830019300 con 0x7f684009d060 2024-09-18T12:25:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.214+0000 7f68357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T12:25:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.214+0000 7f68357fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:31.217 INFO:teuthology.orchestra.run.smithi116.stdout:47244640259 2024-09-18T12:25:31.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-18T12:25:31.218 DEBUG:teuthology.parallel:result is None 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6824038490 msgr2=0x7f682403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6824038490 0x7f682403a940 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f684009bfc0 tx=0x7f683c032000 comp rx=0 tx=0).stop 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 10 monclient: shutdown 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 msgr2=0x7f6840135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 0x7f6840135300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f683000c610 tx=0x7f6830004960 comp rx=0 tx=0).stop 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 shutdown_connections 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6824038490 0x7f682403a940 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f684009e0c0 0x7f6840145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f684009d060 0x7f6840135300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 --2- 172.21.15.116:0/1371764616 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6840067820 0x7f6840141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 >> 172.21.15.116:0/1371764616 conn(0x7f68400a3a40 msgr2=0x7f68400a0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 shutdown_connections 2024-09-18T12:25:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.216+0000 7f68474b8640 1 -- 172.21.15.116:0/1371764616 wait complete. 2024-09-18T12:25:31.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-09-18T12:25:31.230 DEBUG:teuthology.parallel:result is None 2024-09-18T12:25:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.241+0000 7f4f56a99640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T12:25:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.242+0000 7f4f56a99640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T12:25:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.242+0000 7f4f56a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.242+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4f5004bf60 con 0x7f4f50137c30 2024-09-18T12:25:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.242+0000 7f4f4cff9640 1 -- 172.21.15.116:0/988274019 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4f48044bd0 con 0x7f4f50137c30 2024-09-18T12:25:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.242+0000 7f4f4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T12:25:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.242+0000 7f4f4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:31.245 INFO:teuthology.orchestra.run.smithi116.stdout:47244640260 2024-09-18T12:25:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f34038510 msgr2=0x7f4f3403a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f34038510 0x7f4f3403a9c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009af0 tx=0x7f4f440090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 10 monclient: shutdown 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50137c30 msgr2=0x7f4f50096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50137c30 0x7f4f50096fa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f48034000 tx=0x7f4f48037000 comp rx=0 tx=0).stop 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 shutdown_connections 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f34038510 0x7f4f3403a9c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f50137c30 0x7f4f50096fa0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f501318f0 0x7f4f50096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 --2- 172.21.15.116:0/988274019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5009ea50 0x7f4f5009b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 >> 172.21.15.116:0/988274019 conn(0x7f4f50067110 msgr2=0x7f4f500a34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 shutdown_connections 2024-09-18T12:25:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:31.244+0000 7f4f56a99640 1 -- 172.21.15.116:0/988274019 wait complete. 2024-09-18T12:25:31.256 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-18T12:25:31.257 DEBUG:teuthology.parallel:result is None 2024-09-18T12:25:32.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T12:25:32.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.166+0000 7f5af54c9640 10 monclient: get_monmap_and_config 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.166+0000 7f5af54c9640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.166+0000 7f5af54c9640 10 monclient: monmap: 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:32.167778+0000 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:32.167778+0000 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:32.168 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:32.169 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.166+0000 7f5af54c9640 1 Processor -- start 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.166+0000 7f5af54c9640 1 -- start start 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.166+0000 7f5af54c9640 10 monclient: init 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af009ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient(hunting): picked mon.noname-b con 0x7f5af00a5760 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5af009e900 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5af0136b60 0x7f5af0138f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient(hunting): picked mon.noname-c con 0x7f5af0136b60 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af009ffc0 con 0x7f5af009e900 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af00a10a0 con 0x7f5af0136b60 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00a5b30 con 0x7f5af00a5760 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5af54c9640 10 monclient(hunting): authenticate will time out at 759.489221s 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af009ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af009ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54074/0 (socket says 172.21.15.116:54074) 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 -- 172.21.15.116:0/1006535584 learned_addr learned my addr 172.21.15.116:0/1006535584 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): get_auth_request con 0x7f5af00a5760 auth_method 0 2024-09-18T12:25:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aef7fe640 1 --2- 172.21.15.116:0/1006535584 >> v2:172.21.15.144:3300/0 conn(0x7f5af0136b60 0x7f5af0138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aef7fe640 10 monclient(hunting): get_auth_request con 0x7f5af0136b60 auth_method 0 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 -- 172.21.15.116:0/1006535584 >> v2:172.21.15.144:3300/0 conn(0x7f5af0136b60 msgr2=0x7f5af0138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 --2- 172.21.15.116:0/1006535584 >> v2:172.21.15.144:3300/0 conn(0x7f5af0136b60 0x7f5af0138f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 -- 172.21.15.116:0/1006535584 >> v1:172.21.15.144:6789/0 conn(0x7f5af009e900 legacy=0x7f5af009ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 monclient: found mon.noname-b 2024-09-18T12:25:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.167+0000 7f5aeeffd640 1 -- 172.21.15.116:0/1006535584 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af009d4c0 con 0x7f5af00a5760 2024-09-18T12:25:32.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aeeffd640 1 -- 172.21.15.116:0/1006535584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 msgr2=0x7f5af009ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aeeffd640 1 --2- 172.21.15.116:0/1006535584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af009ce40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009910 tx=0x7f5ae80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6581ef74d519858a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 1 -- 172.21.15.116:0/1006535584 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae802ae90 con 0x7f5af00a5760 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: dump: 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 1 -- 172.21.15.116:0/1006535584 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8032440 con 0x7f5af00a5760 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5af54c9640 5 monclient: authenticate success, global_id 4292 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5af54c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 1 -- 172.21.15.116:0/1006535584 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8032820 con 0x7f5af00a5760 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5af54c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5af54c9640 4 set_mon_vals no callback set 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.170+0000 7f5aedffb640 10 monclient: dump: 2024-09-18T12:25:32.181 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 10 monclient: shutdown 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 -- 172.21.15.116:0/1006535584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 msgr2=0x7f5af009ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 --2- 172.21.15.116:0/1006535584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af009ce40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009910 tx=0x7f5ae80091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 -- 172.21.15.116:0/1006535584 shutdown_connections 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 --2- 172.21.15.116:0/1006535584 >> v2:172.21.15.144:3300/0 conn(0x7f5af0136b60 0x7f5af0138f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 --2- 172.21.15.116:0/1006535584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af009ce40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 -- 172.21.15.116:0/1006535584 >> 172.21.15.116:0/1006535584 conn(0x7f5af0067680 msgr2=0x7f5af00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 -- 172.21.15.116:0/1006535584 shutdown_connections 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 1 -- 172.21.15.116:0/1006535584 wait complete. 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.171+0000 7f5af54c9640 10 monclient: monmap: 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 Processor -- start 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 -- start start 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient: init 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 0x7f5af00983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): picked mon.noname-a con 0x7f5af009e900 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af00988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): picked mon.noname-c con 0x7f5af00a5760 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:32.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5af0136b60 0x7f5af0098e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): picked mon.noname-b con 0x7f5af0136b60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af009bd70 con 0x7f5af009e900 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af011ba30 con 0x7f5af0136b60 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0137420 con 0x7f5af00a5760 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5af54c9640 10 monclient(hunting): authenticate will time out at 759.494823s 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aeeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 0x7f5af00983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aee7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af00988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aeeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 0x7f5af00983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52152/0 (socket says 172.21.15.116:52152) 2024-09-18T12:25:32.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aeeffd640 1 -- 172.21.15.116:0/4160593639 learned_addr learned my addr 172.21.15.116:0/4160593639 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aee7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af00988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54082/0 (socket says 172.21.15.116:54082) 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aeeffd640 10 monclient(hunting): get_auth_request con 0x7f5af009e900 auth_method 0 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aee7fc640 10 monclient(hunting): get_auth_request con 0x7f5af00a5760 auth_method 0 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.172+0000 7f5aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aef7fe640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5af0136b60 0x7f5af0098e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 1 -- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 msgr2=0x7f5af00988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af00988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 1 -- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5af0136b60 msgr2=0x7f5af0098e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5af0136b60 0x7f5af0098e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 1 monclient: found mon.noname-a 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aeeffd640 1 -- 172.21.15.116:0/4160593639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae8009450 con 0x7f5af009e900 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aef7fe640 10 monclient: get_auth_request con 0x7f5af0136b60 auth_method 0 2024-09-18T12:25:32.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aee7fc640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af00988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.173+0000 7f5aef7fe640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5af0136b60 0x7f5af0098e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5aeeffd640 1 -- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 msgr2=0x7f5af00983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5aeeffd640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 0x7f5af00983a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009210 tx=0x7f5ae8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 1 -- 172.21.15.116:0/4160593639 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8006b30 con 0x7f5af009e900 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: dump: 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.188 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 1 -- 172.21.15.116:0/4160593639 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae802ae90 con 0x7f5af009e900 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5af54c9640 5 monclient: authenticate success, global_id 4314 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5af54c9640 10 monclient: _renew_subs 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5af54c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af011b750 con 0x7f5af009e900 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 1 -- 172.21.15.116:0/4160593639 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae803c400 con 0x7f5af009e900 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5ad6ffd640 10 monclient: dump: 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5af54c9640 10 monclient: _renew_subs 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5af54c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5aed7fa640 4 set_mon_vals no callback set 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.175+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af009a920 con 0x7f5af009e900 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.176+0000 7f5ad6ffd640 1 -- 172.21.15.116:0/4160593639 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5ae803c7b0 con 0x7f5af009e900 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.176+0000 7f5ad6ffd640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5adc038490 0x7f5adc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.176+0000 7f5ad6ffd640 1 -- 172.21.15.116:0/4160593639 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f5ae806bf70 con 0x7f5af009e900 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.176+0000 7f5af54c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.176+0000 7f5af54c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.176+0000 7f5af54c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.176+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af004bf60 con 0x7f5af009e900 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.177+0000 7f5aee7fc640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5adc038490 0x7f5adc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.177+0000 7f5aee7fc640 10 monclient: get_auth_request con 0x7f5adc038490 auth_method 0 2024-09-18T12:25:32.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.179+0000 7f5aee7fc640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5adc038490 0x7f5adc03a940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4007950 tx=0x7f5ae4009760 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.185+0000 7f5ad6ffd640 1 -- 172.21.15.116:0/4160593639 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5ae8037080 con 0x7f5af009e900 2024-09-18T12:25:32.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.185+0000 7f5ad6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.185+0000 7f5ad6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.209+0000 7f71f6e6f640 10 monclient: get_monmap_and_config 2024-09-18T12:25:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.209+0000 7f71f6e6f640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.209+0000 7f71f6e6f640 10 monclient: monmap: 2024-09-18T12:25:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:32.211097+0000 2024-09-18T12:25:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:32.211097+0000 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:32.212 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 1 Processor -- start 2024-09-18T12:25:32.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 1 -- start start 2024-09-18T12:25:32.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient: init 2024-09-18T12:25:32.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f01318f0 0x7f71f0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f71f01318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71f009ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71f0136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f0131cc0 con 0x7f71f009ea50 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f009f530 con 0x7f71f0136c20 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f009fc40 con 0x7f71f01318f0 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f6e6f640 10 monclient(hunting): authenticate will time out at 759.532729s 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71effff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f71f009ea50 0x7f71f009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57138/0 (socket says 172.21.15.116:57138) 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f01318f0 0x7f71f0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71effff640 1 -- 172.21.15.116:0/1096792801 learned_addr learned my addr 172.21.15.116:0/1096792801 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:32.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 10 monclient(hunting): get_auth_request con 0x7f71f01318f0 auth_method 0 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.210+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 1 -- 172.21.15.116:0/1096792801 >> v1:172.21.15.144:6789/0 conn(0x7f71f0136c20 legacy=0x7f71f0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 1 -- 172.21.15.116:0/1096792801 >> v1:172.21.15.116:6789/0 conn(0x7f71f009ea50 legacy=0x7f71f009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 1 monclient: found mon.noname-b 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.211+0000 7f71f4c0b640 1 -- 172.21.15.116:0/1096792801 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f009fe30 con 0x7f71f01318f0 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.213+0000 7f71f4c0b640 1 -- 172.21.15.116:0/1096792801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f01318f0 msgr2=0x7f71f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.213+0000 7f71f4c0b640 1 --2- 172.21.15.116:0/1096792801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f01318f0 0x7f71f0131f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009880 tx=0x7f71e40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83580f3a076ecad server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 1 -- 172.21.15.116:0/1096792801 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e402aea0 con 0x7f71f01318f0 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: dump: 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.222 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 5 monclient: authenticate success, global_id 4298 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 1 -- 172.21.15.116:0/1096792801 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4032440 con 0x7f71f01318f0 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 4 set_mon_vals no callback set 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 1 -- 172.21.15.116:0/1096792801 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e40327e0 con 0x7f71f01318f0 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71ef7fe640 10 monclient: dump: 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 10 monclient: shutdown 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 1 -- 172.21.15.116:0/1096792801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f01318f0 msgr2=0x7f71f0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/1096792801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f01318f0 0x7f71f0131f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009880 tx=0x7f71e40090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 1 -- 172.21.15.116:0/1096792801 shutdown_connections 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/1096792801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f01318f0 0x7f71f0131f30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 1 -- 172.21.15.116:0/1096792801 >> 172.21.15.116:0/1096792801 conn(0x7f71f0067110 msgr2=0x7f71f0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 1 -- 172.21.15.116:0/1096792801 shutdown_connections 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.214+0000 7f71f6e6f640 1 -- 172.21.15.116:0/1096792801 wait complete. 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.215+0000 7f71f6e6f640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.215+0000 7f71f6e6f640 10 monclient: monmap: 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.215+0000 7f71f6e6f640 1 Processor -- start 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.215+0000 7f71f6e6f640 1 -- start start 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.215+0000 7f71f6e6f640 10 monclient: init 2024-09-18T12:25:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 0x7f71f009b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): picked mon.noname-c con 0x7f71f009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71f01318f0 0x7f71f0096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f71f01318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71f0136c20 0x7f71f0096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): picked mon.noname-a con 0x7f71f0136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f013a4f0 con 0x7f71f0136c20 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0099a90 con 0x7f71f01318f0 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0097c80 con 0x7f71f009ea50 2024-09-18T12:25:32.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f6e6f640 10 monclient(hunting): authenticate will time out at 759.538140s 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 0x7f71f009b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 0x7f71f009b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54110/0 (socket says 172.21.15.116:54110) 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 -- 172.21.15.116:0/578934365 learned_addr learned my addr 172.21.15.116:0/578934365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): get_auth_request con 0x7f71f009ea50 auth_method 0 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71f0136c20 0x7f71f0096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 10 monclient(hunting): get_auth_request con 0x7f71f0136c20 auth_method 0 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71effff640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71f01318f0 0x7f71f0096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71effff640 10 monclient(hunting): get_auth_request con 0x7f71f01318f0 auth_method 0 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 -- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71f01318f0 msgr2=0x7f71f0096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71f01318f0 0x7f71f0096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 -- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71f0136c20 msgr2=0x7f71f0096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71f0136c20 0x7f71f0096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 monclient: found mon.noname-c 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f4c0b640 1 -- 172.21.15.116:0/578934365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4009390 con 0x7f71f009ea50 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.216+0000 7f71f540c640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71f0136c20 0x7f71f0096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71f4c0b640 1 -- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 msgr2=0x7f71f009b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71f4c0b640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 0x7f71f009b660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009880 tx=0x7f71e4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 1 -- 172.21.15.116:0/578934365 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e402aea0 con 0x7f71f009ea50 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:32.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: dump: 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 1 -- 172.21.15.116:0/578934365 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4031990 con 0x7f71f009ea50 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71f6e6f640 5 monclient: authenticate success, global_id 4301 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71f6e6f640 10 monclient: _renew_subs 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71f6e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f00997b0 con 0x7f71f009ea50 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 1 -- 172.21.15.116:0/578934365 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4031d50 con 0x7f71f009ea50 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71ecff9640 10 monclient: dump: 2024-09-18T12:25:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.218+0000 7f71eeffd640 4 set_mon_vals no callback set 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.219+0000 7f71f6e6f640 10 monclient: _renew_subs 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.219+0000 7f71f6e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.219+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f009a610 con 0x7f71f009ea50 2024-09-18T12:25:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.219+0000 7f71ecff9640 1 -- 172.21.15.116:0/578934365 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f71e4039040 con 0x7f71f009ea50 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.219+0000 7f71ecff9640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f71d4038490 0x7f71d403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.219+0000 7f71ecff9640 1 -- 172.21.15.116:0/578934365 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f71e406b990 con 0x7f71f009ea50 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.220+0000 7f71f6e6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.220+0000 7f71f6e6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.220+0000 7f71f6e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.220+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f0099c20 con 0x7f71f009ea50 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.220+0000 7f71effff640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f71d4038490 0x7f71d403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.220+0000 7f71effff640 10 monclient: get_auth_request con 0x7f71d4038490 auth_method 0 2024-09-18T12:25:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.222+0000 7f71effff640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f71d4038490 0x7f71d403a940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71dc007fd0 tx=0x7f71dc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.224+0000 7f71ecff9640 1 -- 172.21.15.116:0/578934365 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f71e403f020 con 0x7f71f009ea50 2024-09-18T12:25:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.224+0000 7f71ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.224+0000 7f71ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:32.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.348+0000 7f5af54c9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:32.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.348+0000 7f5af54c9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:32.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.348+0000 7f5af54c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.348+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5af00638d0 con 0x7f5af009e900 2024-09-18T12:25:32.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.349+0000 7f5ad6ffd640 1 -- 172.21.15.116:0/4160593639 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ae803b3a0 con 0x7f5af009e900 2024-09-18T12:25:32.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.349+0000 7f5ad6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T12:25:32.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.349+0000 7f5ad6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:32.351 INFO:teuthology.orchestra.run.smithi116.stdout:47244640260 2024-09-18T12:25:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5adc038490 msgr2=0x7f5adc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5adc038490 0x7f5adc03a940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4007950 tx=0x7f5ae4009760 comp rx=0 tx=0).stop 2024-09-18T12:25:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 10 monclient: shutdown 2024-09-18T12:25:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 msgr2=0x7f5af00983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 0x7f5af00983a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009210 tx=0x7f5ae8032000 comp rx=0 tx=0).stop 2024-09-18T12:25:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 shutdown_connections 2024-09-18T12:25:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5adc038490 0x7f5adc03a940 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5af0136b60 0x7f5af0098e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5af00a5760 0x7f5af00988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 --2- 172.21.15.116:0/4160593639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5af009e900 0x7f5af00983a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 >> 172.21.15.116:0/4160593639 conn(0x7f5af0067680 msgr2=0x7f5af009d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 shutdown_connections 2024-09-18T12:25:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.351+0000 7f5af54c9640 1 -- 172.21.15.116:0/4160593639 wait complete. 2024-09-18T12:25:32.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-18T12:25:32.363 DEBUG:teuthology.parallel:result is None 2024-09-18T12:25:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.385+0000 7f71f6e6f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.385+0000 7f71f6e6f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.385+0000 7f71f6e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.385+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f71f0099c20 con 0x7f71f009ea50 2024-09-18T12:25:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.385+0000 7f71ecff9640 1 -- 172.21.15.116:0/578934365 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f71e403b610 con 0x7f71f009ea50 2024-09-18T12:25:32.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.385+0000 7f71ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T12:25:32.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.385+0000 7f71ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:32.388 INFO:teuthology.orchestra.run.smithi116.stdout:47244640260 2024-09-18T12:25:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f71d4038490 msgr2=0x7f71d403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f71d4038490 0x7f71d403a940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71dc007fd0 tx=0x7f71dc009040 comp rx=0 tx=0).stop 2024-09-18T12:25:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 10 monclient: shutdown 2024-09-18T12:25:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 msgr2=0x7f71f009b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 0x7f71f009b660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009880 tx=0x7f71e4032000 comp rx=0 tx=0).stop 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 shutdown_connections 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f71d4038490 0x7f71d403a940 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71f0136c20 0x7f71f0096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f71f01318f0 0x7f71f0096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 --2- 172.21.15.116:0/578934365 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71f009ea50 0x7f71f009b660 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 >> 172.21.15.116:0/578934365 conn(0x7f71f0067110 msgr2=0x7f71f009e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 shutdown_connections 2024-09-18T12:25:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.387+0000 7f71f6e6f640 1 -- 172.21.15.116:0/578934365 wait complete. 2024-09-18T12:25:32.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-18T12:25:32.399 DEBUG:teuthology.parallel:result is None 2024-09-18T12:25:32.400 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T12:25:32.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:25:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.531+0000 7f9a3ccdd640 10 monclient: get_monmap_and_config 2024-09-18T12:25:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.531+0000 7f9a3ccdd640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.531+0000 7f9a3ccdd640 10 monclient: monmap: 2024-09-18T12:25:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:32.533194+0000 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:32.533194+0000 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:32.534 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 1 Processor -- start 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 1 -- start start 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient: init 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a38096c60 0x7f9a3809a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient(hunting): picked mon.noname-a con 0x7f9a38096c60 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a38097030 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:32.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38133b40 0x7f9a38135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient(hunting): picked mon.noname-b con 0x7f9a38133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380696a0 con 0x7f9a38097030 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a3809ac00 con 0x7f9a38096c60 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3809d6b0 con 0x7f9a38133b40 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3ccdd640 10 monclient(hunting): authenticate will time out at 759.854744s 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a38096c60 0x7f9a3809a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38133b40 0x7f9a38135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a38096c60 0x7f9a3809a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52160/0 (socket says 172.21.15.116:52160) 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 1 -- 172.21.15.116:0/408672632 learned_addr learned my addr 172.21.15.116:0/408672632 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 10 monclient(hunting): get_auth_request con 0x7f9a38096c60 auth_method 0 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 10 monclient(hunting): get_auth_request con 0x7f9a38133b40 auth_method 0 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.532+0000 7f9a3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 1 -- 172.21.15.116:0/408672632 >> v2:172.21.15.116:3300/0 conn(0x7f9a38096c60 msgr2=0x7f9a3809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 1 --2- 172.21.15.116:0/408672632 >> v2:172.21.15.116:3300/0 conn(0x7f9a38096c60 0x7f9a3809a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 1 -- 172.21.15.116:0/408672632 >> v1:172.21.15.144:6789/0 conn(0x7f9a38097030 legacy=0x7f9a38097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 1 monclient: found mon.noname-b 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a36d9d640 1 -- 172.21.15.116:0/408672632 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3809d2e0 con 0x7f9a38133b40 2024-09-18T12:25:32.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.533+0000 7f9a3659c640 1 --2- 172.21.15.116:0/408672632 >> v2:172.21.15.116:3300/0 conn(0x7f9a38096c60 0x7f9a3809a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:32.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.535+0000 7f9a36d9d640 1 -- 172.21.15.116:0/408672632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38133b40 msgr2=0x7f9a38135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.535+0000 7f9a36d9d640 1 --2- 172.21.15.116:0/408672632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38133b40 0x7f9a38135f20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009e60 tx=0x7f9a30009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b9baac0636a5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 1 -- 172.21.15.116:0/408672632 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30007760 con 0x7f9a38133b40 2024-09-18T12:25:32.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: dump: 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 5 monclient: authenticate success, global_id 4304 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 1 -- 172.21.15.116:0/408672632 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3002d050 con 0x7f9a38133b40 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 4 set_mon_vals no callback set 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 1 -- 172.21.15.116:0/408672632 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30007af0 con 0x7f9a38133b40 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3559a640 10 monclient: dump: 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.548 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 10 monclient: shutdown 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/408672632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38133b40 msgr2=0x7f9a38135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/408672632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38133b40 0x7f9a38135f20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009e60 tx=0x7f9a30009780 comp rx=0 tx=0).stop 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/408672632 shutdown_connections 2024-09-18T12:25:32.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/408672632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38133b40 0x7f9a38135f20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/408672632 >> v2:172.21.15.116:3300/0 conn(0x7f9a38096c60 0x7f9a3809a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/408672632 >> 172.21.15.116:0/408672632 conn(0x7f9a380672b0 msgr2=0x7f9a380676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/408672632 shutdown_connections 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.536+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/408672632 wait complete. 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 10 monclient: monmap: 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.550 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 1 Processor -- start 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 1 -- start start 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 10 monclient: init 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38096c60 0x7f9a3806e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 10 monclient(hunting): picked mon.noname-c con 0x7f9a38096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a38097030 0x7f9a38117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 10 monclient(hunting): picked mon.noname-a con 0x7f9a38097030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.537+0000 7f9a3ccdd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a38133b40 0x7f9a381179c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 10 monclient(hunting): picked mon.noname-b con 0x7f9a38133b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a38133390 con 0x7f9a38097030 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3809add0 con 0x7f9a38133b40 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3806ede0 con 0x7f9a38096c60 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3ccdd640 10 monclient(hunting): authenticate will time out at 759.860035s 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38096c60 0x7f9a3806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38096c60 0x7f9a3806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54142/0 (socket says 172.21.15.116:54142) 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 1 -- 172.21.15.116:0/565736545 learned_addr learned my addr 172.21.15.116:0/565736545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 10 monclient(hunting): get_auth_request con 0x7f9a38096c60 auth_method 0 2024-09-18T12:25:32.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a38097030 0x7f9a38117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): get_auth_request con 0x7f9a38097030 auth_method 0 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a36d9d640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a38133b40 0x7f9a381179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a36d9d640 10 monclient(hunting): get_auth_request con 0x7f9a38133b40 auth_method 0 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a36d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 1 -- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38096c60 msgr2=0x7f9a3806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38096c60 0x7f9a3806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 1 -- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a38133b40 msgr2=0x7f9a381179c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a38133b40 0x7f9a381179c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 1 monclient: found mon.noname-a 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a35d9b640 1 -- 172.21.15.116:0/565736545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2800b700 con 0x7f9a38097030 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.538+0000 7f9a3659c640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38096c60 0x7f9a3806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a35d9b640 1 -- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a38097030 msgr2=0x7f9a38117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a35d9b640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a38097030 0x7f9a38117480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00c4e0 tx=0x7f9a2c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 1 -- 172.21.15.116:0/565736545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0117a0 con 0x7f9a38097030 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: dump: 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.555 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 1 -- 172.21.15.116:0/565736545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2c011920 con 0x7f9a38097030 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a3ccdd640 5 monclient: authenticate success, global_id 4323 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a3ccdd640 10 monclient: _renew_subs 2024-09-18T12:25:32.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a3ccdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a30009a20 con 0x7f9a38097030 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 1 -- 172.21.15.116:0/565736545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c011ce0 con 0x7f9a38097030 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a267fc640 10 monclient: dump: 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a34d99640 4 set_mon_vals no callback set 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a3ccdd640 10 monclient: _renew_subs 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a3ccdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.540+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3811a730 con 0x7f9a38097030 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.541+0000 7f9a267fc640 1 -- 172.21.15.116:0/565736545 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9a2c012630 con 0x7f9a38097030 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.541+0000 7f9a3ccdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.541+0000 7f9a3ccdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.541+0000 7f9a3ccdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.541+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3804bf60 con 0x7f9a38097030 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.541+0000 7f9a267fc640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a180384c0 0x7f9a1803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.541+0000 7f9a267fc640 1 -- 172.21.15.116:0/565736545 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f9a2c04b240 con 0x7f9a38097030 2024-09-18T12:25:32.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.542+0000 7f9a3659c640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a180384c0 0x7f9a1803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.542+0000 7f9a3659c640 10 monclient: get_auth_request con 0x7f9a180384c0 auth_method 0 2024-09-18T12:25:32.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.544+0000 7f9a3659c640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a180384c0 0x7f9a1803a970 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a28011b00 tx=0x7f9a2800b460 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.546+0000 7f9a267fc640 1 -- 172.21.15.116:0/565736545 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9a2c0128e0 con 0x7f9a38097030 2024-09-18T12:25:32.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.546+0000 7f9a267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.546+0000 7f9a267fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.699+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a380983b0 con 0x7f9a180384c0 2024-09-18T12:25:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.701+0000 7f9a267fc640 1 -- 172.21.15.116:0/565736545 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41049 (secure 0 0 0) 0x7f9a380983b0 con 0x7f9a180384c0 2024-09-18T12:25:32.703 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a180384c0 msgr2=0x7f9a1803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a180384c0 0x7f9a1803a970 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a28011b00 tx=0x7f9a2800b460 comp rx=0 tx=0).stop 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 10 monclient: shutdown 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a38097030 msgr2=0x7f9a38117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a38097030 0x7f9a38117480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00c4e0 tx=0x7f9a2c007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 shutdown_connections 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.702+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a180384c0 0x7f9a1803a970 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.703+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a38133b40 0x7f9a381179c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.703+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a38097030 0x7f9a38117480 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.703+0000 7f9a3ccdd640 1 --2- 172.21.15.116:0/565736545 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a38096c60 0x7f9a3806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.703+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 >> 172.21.15.116:0/565736545 conn(0x7f9a380672b0 msgr2=0x7f9a38131a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.703+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 shutdown_connections 2024-09-18T12:25:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.703+0000 7f9a3ccdd640 1 -- 172.21.15.116:0/565736545 wait complete. 2024-09-18T12:25:32.715 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-18T12:25:32.562663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786384,"kb_used_data":1960,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444656,"statfs":{"total":579820584960,"available":577991327744,"internally_reserved":0,"allocated":2007040,"data_stored":1352350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.458062"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106005+0000","last_change":"2024-09-18T12:25:28.106207+0000","last_active":"2024-09-18T12:25:28.106005+0000","last_peered":"2024-09-18T12:25:28.106005+0000","last_clean":"2024-09-18T12:25:28.106005+0000","last_became_active":"2024-09-18T12:25:25.932573+0000","last_became_peered":"2024-09-18T12:25:25.932573+0000","last_unstale":"2024-09-18T12:25:28.106005+0000","last_undegraded":"2024-09-18T12:25:28.106005+0000","last_fullsized":"2024-09-18T12:25:28.106005+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:11:31.670093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.114271+0000","last_change":"2024-09-18T12:25:28.114567+0000","last_active":"2024-09-18T12:25:28.114271+0000","last_peered":"2024-09-18T12:25:28.114271+0000","last_clean":"2024-09-18T12:25:28.114271+0000","last_became_active":"2024-09-18T12:25:25.938144+0000","last_became_peered":"2024-09-18T12:25:25.938144+0000","last_unstale":"2024-09-18T12:25:28.114271+0000","last_undegraded":"2024-09-18T12:25:28.114271+0000","last_fullsized":"2024-09-18T12:25:28.114271+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:02:14.591426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106056+0000","last_change":"2024-09-18T12:25:28.106209+0000","last_active":"2024-09-18T12:25:28.106056+0000","last_peered":"2024-09-18T12:25:28.106056+0000","last_clean":"2024-09-18T12:25:28.106056+0000","last_became_active":"2024-09-18T12:25:26.082919+0000","last_became_peered":"2024-09-18T12:25:26.082919+0000","last_unstale":"2024-09-18T12:25:28.106056+0000","last_undegraded":"2024-09-18T12:25:28.106056+0000","last_fullsized":"2024-09-18T12:25:28.106056+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:55:28.708584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033820399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.114218+0000","last_change":"2024-09-18T12:25:28.114468+0000","last_active":"2024-09-18T12:25:28.114218+0000","last_peered":"2024-09-18T12:25:28.114218+0000","last_clean":"2024-09-18T12:25:28.114218+0000","last_became_active":"2024-09-18T12:25:26.082963+0000","last_became_peered":"2024-09-18T12:25:26.082963+0000","last_unstale":"2024-09-18T12:25:28.114218+0000","last_undegraded":"2024-09-18T12:25:28.114218+0000","last_fullsized":"2024-09-18T12:25:28.114218+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:44:20.609265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.112853+0000","last_change":"2024-09-18T12:25:28.112853+0000","last_active":"2024-09-18T12:25:28.112853+0000","last_peered":"2024-09-18T12:25:28.112853+0000","last_clean":"2024-09-18T12:25:28.112853+0000","last_became_active":"2024-09-18T12:25:25.932570+0000","last_became_peered":"2024-09-18T12:25:25.932570+0000","last_unstale":"2024-09-18T12:25:28.112853+0000","last_undegraded":"2024-09-18T12:25:28.112853+0000","last_fullsized":"2024-09-18T12:25:28.112853+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:03.513828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.742178+0000","last_change":"2024-09-18T12:25:28.742381+0000","last_active":"2024-09-18T12:25:28.742178+0000","last_peered":"2024-09-18T12:25:28.742178+0000","last_clean":"2024-09-18T12:25:28.742178+0000","last_became_active":"2024-09-18T12:25:25.930894+0000","last_became_peered":"2024-09-18T12:25:25.930894+0000","last_unstale":"2024-09-18T12:25:28.742178+0000","last_undegraded":"2024-09-18T12:25:28.742178+0000","last_fullsized":"2024-09-18T12:25:28.742178+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:09:49.928338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106258+0000","last_change":"2024-09-18T12:25:28.106419+0000","last_active":"2024-09-18T12:25:28.106258+0000","last_peered":"2024-09-18T12:25:28.106258+0000","last_clean":"2024-09-18T12:25:28.106258+0000","last_became_active":"2024-09-18T12:25:25.930456+0000","last_became_peered":"2024-09-18T12:25:25.930456+0000","last_unstale":"2024-09-18T12:25:28.106258+0000","last_undegraded":"2024-09-18T12:25:28.106258+0000","last_fullsized":"2024-09-18T12:25:28.106258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:26:39.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.112214+0000","last_change":"2024-09-18T12:25:28.112432+0000","last_active":"2024-09-18T12:25:28.112214+0000","last_peered":"2024-09-18T12:25:28.112214+0000","last_clean":"2024-09-18T12:25:28.112214+0000","last_became_active":"2024-09-18T12:25:25.932281+0000","last_became_peered":"2024-09-18T12:25:25.932281+0000","last_unstale":"2024-09-18T12:25:28.112214+0000","last_undegraded":"2024-09-18T12:25:28.112214+0000","last_fullsized":"2024-09-18T12:25:28.112214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:40.111669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":102,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T12:25:27.101718+0000","last_change":"2024-09-18T12:25:24.722024+0000","last_active":"2024-09-18T12:25:27.101718+0000","last_peered":"2024-09-18T12:25:27.101718+0000","last_clean":"2024-09-18T12:25:27.101718+0000","last_became_active":"2024-09-18T12:25:24.721108+0000","last_became_peered":"2024-09-18T12:25:24.721108+0000","last_unstale":"2024-09-18T12:25:27.101718+0000","last_undegraded":"2024-09-18T12:25:27.101718+0000","last_fullsized":"2024-09-18T12:25:27.101718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:41:42.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":167936,"data_stored":52371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2640000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1210000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2839999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2320000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":163840,"data_stored":52352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298172,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073668,"statfs":{"total":96636764160,"available":96331436032,"internally_reserved":0,"allocated":696320,"data_stored":581280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":139264,"data_stored":42524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":167936,"data_stored":52371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298140,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073700,"statfs":{"total":96636764160,"available":96331468800,"internally_reserved":0,"allocated":671744,"data_stored":571452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:25:32.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.808+0000 7feaa5834640 10 monclient: get_monmap_and_config 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.808+0000 7feaa5834640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.808+0000 7feaa5834640 10 monclient: monmap: 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:32.809686+0000 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:32.809686+0000 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.808+0000 7feaa5834640 1 Processor -- start 2024-09-18T12:25:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.808+0000 7feaa5834640 1 -- start start 2024-09-18T12:25:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.808+0000 7feaa5834640 10 monclient: init 2024-09-18T12:25:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feaa0131980 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7feaa009fda0 0x7feaa009e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient(hunting): picked mon.noname-c con 0x7feaa009fda0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feaa009ef00 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa00a1180 con 0x7feaa0131980 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa00a1460 con 0x7feaa009ef00 2024-09-18T12:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa0132460 con 0x7feaa009fda0 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7feaa5834640 10 monclient(hunting): authenticate will time out at 760.131280s 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feaa0131980 0x7feaa0131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57154/0 (socket says 172.21.15.116:57154) 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9effd640 1 -- 172.21.15.116:0/729911692 learned_addr learned my addr 172.21.15.116:0/729911692 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 1 --2- 172.21.15.116:0/729911692 >> v2:172.21.15.144:3300/0 conn(0x7feaa009fda0 0x7feaa009e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): get_auth_request con 0x7feaa009fda0 auth_method 0 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 1 -- 172.21.15.116:0/729911692 >> v1:172.21.15.144:6789/0 conn(0x7feaa009ef00 legacy=0x7feaa0138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 1 -- 172.21.15.116:0/729911692 >> v1:172.21.15.116:6789/0 conn(0x7feaa0131980 legacy=0x7feaa0131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 1 monclient: found mon.noname-c 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:32.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.809+0000 7fea9e7fc640 1 -- 172.21.15.116:0/729911692 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa01325a0 con 0x7feaa009fda0 2024-09-18T12:25:32.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.812+0000 7fea9e7fc640 1 -- 172.21.15.116:0/729911692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 msgr2=0x7feaa009e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.812+0000 7fea9e7fc640 1 --2- 172.21.15.116:0/729911692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 0x7feaa009e9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fea94009850 tx=0x7fea940090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afd06657f69f5ef8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 1 -- 172.21.15.116:0/729911692 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea94032440 con 0x7feaa009fda0 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: dump: 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.823 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 1 -- 172.21.15.116:0/729911692 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea940325a0 con 0x7feaa009fda0 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7feaa5834640 5 monclient: authenticate success, global_id 4303 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7feaa5834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 1 -- 172.21.15.116:0/729911692 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea94032940 con 0x7feaa009fda0 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7feaa5834640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7feaa5834640 4 set_mon_vals no callback set 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:32.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.813+0000 7fea9dffb640 10 monclient: dump: 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 10 monclient: shutdown 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 1 -- 172.21.15.116:0/729911692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 msgr2=0x7feaa009e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 1 --2- 172.21.15.116:0/729911692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 0x7feaa009e9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fea94009850 tx=0x7fea940090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 1 -- 172.21.15.116:0/729911692 shutdown_connections 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 1 --2- 172.21.15.116:0/729911692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 0x7feaa009e9c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 1 -- 172.21.15.116:0/729911692 >> 172.21.15.116:0/729911692 conn(0x7feaa0067bd0 msgr2=0x7feaa00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 1 -- 172.21.15.116:0/729911692 shutdown_connections 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 1 -- 172.21.15.116:0/729911692 wait complete. 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 10 monclient: build_initial_monmap 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.814+0000 7feaa5834640 10 monclient: monmap: 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:32.826 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 Processor -- start 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 -- start start 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient: init 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 0x7feaa0097220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): picked mon.noname-a con 0x7feaa009ef00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 0x7feaa0130240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): picked mon.noname-b con 0x7feaa009fda0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaa0131980 0x7feaa0130780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): picked mon.noname-c con 0x7feaa0131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0097760 con 0x7feaa009ef00 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0099010 con 0x7feaa009fda0 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0098d30 con 0x7feaa0131980 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7feaa5834640 10 monclient(hunting): authenticate will time out at 760.137736s 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 0x7feaa0097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 0x7feaa0097220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52180/0 (socket says 172.21.15.116:52180) 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 1 -- 172.21.15.116:0/4167763023 learned_addr learned my addr 172.21.15.116:0/4167763023 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:32.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 10 monclient(hunting): get_auth_request con 0x7feaa009ef00 auth_method 0 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaa0131980 0x7feaa0130780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 10 monclient(hunting): get_auth_request con 0x7feaa0131980 auth_method 0 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9e7fc640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 0x7feaa0130240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9e7fc640 10 monclient(hunting): get_auth_request con 0x7feaa009fda0 auth_method 0 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.815+0000 7fea9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 1 -- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaa0131980 msgr2=0x7feaa0130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaa0131980 0x7feaa0130780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 1 -- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 msgr2=0x7feaa0130240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 0x7feaa0130240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 1 monclient: found mon.noname-a 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9effd640 1 -- 172.21.15.116:0/4167763023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea94009390 con 0x7feaa009ef00 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.816+0000 7fea9f7fe640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaa0131980 0x7feaa0130780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea9effd640 1 -- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 msgr2=0x7feaa0097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea9effd640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 0x7feaa0097220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fea9000cca0 tx=0x7fea900053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 1 -- 172.21.15.116:0/4167763023 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9000aa60 con 0x7feaa009ef00 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: dump: 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.831 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 1 -- 172.21.15.116:0/4167763023 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9000abe0 con 0x7feaa009ef00 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7feaa5834640 5 monclient: authenticate success, global_id 4326 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 1 -- 172.21.15.116:0/4167763023 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea900070f0 con 0x7feaa009ef00 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7feaa5834640 10 monclient: _renew_subs 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7feaa5834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa0096e00 con 0x7feaa009ef00 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea86ffd640 10 monclient: dump: 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7fea9d7fa640 4 set_mon_vals no callback set 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7feaa5834640 10 monclient: _renew_subs 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7feaa5834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.818+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa0130cc0 con 0x7feaa009ef00 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.819+0000 7fea86ffd640 1 -- 172.21.15.116:0/4167763023 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fea90007520 con 0x7feaa009ef00 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.819+0000 7feaa5834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.819+0000 7feaa5834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.819+0000 7feaa5834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.819+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa004bf60 con 0x7feaa009ef00 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.819+0000 7fea86ffd640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea8c038490 0x7fea8c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.819+0000 7fea86ffd640 1 -- 172.21.15.116:0/4167763023 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fea9004b9c0 con 0x7feaa009ef00 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.820+0000 7fea9e7fc640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea8c038490 0x7fea8c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.820+0000 7fea9e7fc640 10 monclient: get_auth_request con 0x7fea8c038490 auth_method 0 2024-09-18T12:25:32.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.822+0000 7fea9e7fc640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea8c038490 0x7fea8c03a940 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea94031000 tx=0x7fea94032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:32.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.824+0000 7fea86ffd640 1 -- 172.21.15.116:0/4167763023 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fea900077d0 con 0x7feaa009ef00 2024-09-18T12:25:32.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.824+0000 7fea86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:32.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.824+0000 7fea86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:32.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.976+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feaa0098ea0 con 0x7fea8c038490 2024-09-18T12:25:32.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.979+0000 7fea86ffd640 1 -- 172.21.15.116:0/4167763023 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41049 (secure 0 0 0) 0x7feaa0098ea0 con 0x7fea8c038490 2024-09-18T12:25:32.980 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:32.981 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea8c038490 msgr2=0x7fea8c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea8c038490 0x7fea8c03a940 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea94031000 tx=0x7fea94032000 comp rx=0 tx=0).stop 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 10 monclient: shutdown 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 msgr2=0x7feaa0097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 0x7feaa0097220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fea9000cca0 tx=0x7fea900053f0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 shutdown_connections 2024-09-18T12:25:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea8c038490 0x7fea8c03a940 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feaa0131980 0x7feaa0130780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feaa009fda0 0x7feaa0130240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 --2- 172.21.15.116:0/4167763023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa009ef00 0x7feaa0097220 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 >> 172.21.15.116:0/4167763023 conn(0x7feaa0067bd0 msgr2=0x7feaa00a1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 shutdown_connections 2024-09-18T12:25:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:32.980+0000 7feaa5834640 1 -- 172.21.15.116:0/4167763023 wait complete. 2024-09-18T12:25:32.992 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-18T12:25:32.562663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786384,"kb_used_data":1960,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444656,"statfs":{"total":579820584960,"available":577991327744,"internally_reserved":0,"allocated":2007040,"data_stored":1352350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.458062"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106005+0000","last_change":"2024-09-18T12:25:28.106207+0000","last_active":"2024-09-18T12:25:28.106005+0000","last_peered":"2024-09-18T12:25:28.106005+0000","last_clean":"2024-09-18T12:25:28.106005+0000","last_became_active":"2024-09-18T12:25:25.932573+0000","last_became_peered":"2024-09-18T12:25:25.932573+0000","last_unstale":"2024-09-18T12:25:28.106005+0000","last_undegraded":"2024-09-18T12:25:28.106005+0000","last_fullsized":"2024-09-18T12:25:28.106005+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:11:31.670093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.114271+0000","last_change":"2024-09-18T12:25:28.114567+0000","last_active":"2024-09-18T12:25:28.114271+0000","last_peered":"2024-09-18T12:25:28.114271+0000","last_clean":"2024-09-18T12:25:28.114271+0000","last_became_active":"2024-09-18T12:25:25.938144+0000","last_became_peered":"2024-09-18T12:25:25.938144+0000","last_unstale":"2024-09-18T12:25:28.114271+0000","last_undegraded":"2024-09-18T12:25:28.114271+0000","last_fullsized":"2024-09-18T12:25:28.114271+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:02:14.591426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106056+0000","last_change":"2024-09-18T12:25:28.106209+0000","last_active":"2024-09-18T12:25:28.106056+0000","last_peered":"2024-09-18T12:25:28.106056+0000","last_clean":"2024-09-18T12:25:28.106056+0000","last_became_active":"2024-09-18T12:25:26.082919+0000","last_became_peered":"2024-09-18T12:25:26.082919+0000","last_unstale":"2024-09-18T12:25:28.106056+0000","last_undegraded":"2024-09-18T12:25:28.106056+0000","last_fullsized":"2024-09-18T12:25:28.106056+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:55:28.708584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033820399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.114218+0000","last_change":"2024-09-18T12:25:28.114468+0000","last_active":"2024-09-18T12:25:28.114218+0000","last_peered":"2024-09-18T12:25:28.114218+0000","last_clean":"2024-09-18T12:25:28.114218+0000","last_became_active":"2024-09-18T12:25:26.082963+0000","last_became_peered":"2024-09-18T12:25:26.082963+0000","last_unstale":"2024-09-18T12:25:28.114218+0000","last_undegraded":"2024-09-18T12:25:28.114218+0000","last_fullsized":"2024-09-18T12:25:28.114218+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:44:20.609265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.112853+0000","last_change":"2024-09-18T12:25:28.112853+0000","last_active":"2024-09-18T12:25:28.112853+0000","last_peered":"2024-09-18T12:25:28.112853+0000","last_clean":"2024-09-18T12:25:28.112853+0000","last_became_active":"2024-09-18T12:25:25.932570+0000","last_became_peered":"2024-09-18T12:25:25.932570+0000","last_unstale":"2024-09-18T12:25:28.112853+0000","last_undegraded":"2024-09-18T12:25:28.112853+0000","last_fullsized":"2024-09-18T12:25:28.112853+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:03.513828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.742178+0000","last_change":"2024-09-18T12:25:28.742381+0000","last_active":"2024-09-18T12:25:28.742178+0000","last_peered":"2024-09-18T12:25:28.742178+0000","last_clean":"2024-09-18T12:25:28.742178+0000","last_became_active":"2024-09-18T12:25:25.930894+0000","last_became_peered":"2024-09-18T12:25:25.930894+0000","last_unstale":"2024-09-18T12:25:28.742178+0000","last_undegraded":"2024-09-18T12:25:28.742178+0000","last_fullsized":"2024-09-18T12:25:28.742178+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:09:49.928338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106258+0000","last_change":"2024-09-18T12:25:28.106419+0000","last_active":"2024-09-18T12:25:28.106258+0000","last_peered":"2024-09-18T12:25:28.106258+0000","last_clean":"2024-09-18T12:25:28.106258+0000","last_became_active":"2024-09-18T12:25:25.930456+0000","last_became_peered":"2024-09-18T12:25:25.930456+0000","last_unstale":"2024-09-18T12:25:28.106258+0000","last_undegraded":"2024-09-18T12:25:28.106258+0000","last_fullsized":"2024-09-18T12:25:28.106258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:26:39.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.112214+0000","last_change":"2024-09-18T12:25:28.112432+0000","last_active":"2024-09-18T12:25:28.112214+0000","last_peered":"2024-09-18T12:25:28.112214+0000","last_clean":"2024-09-18T12:25:28.112214+0000","last_became_active":"2024-09-18T12:25:25.932281+0000","last_became_peered":"2024-09-18T12:25:25.932281+0000","last_unstale":"2024-09-18T12:25:28.112214+0000","last_undegraded":"2024-09-18T12:25:28.112214+0000","last_fullsized":"2024-09-18T12:25:28.112214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:40.111669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":102,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T12:25:27.101718+0000","last_change":"2024-09-18T12:25:24.722024+0000","last_active":"2024-09-18T12:25:27.101718+0000","last_peered":"2024-09-18T12:25:27.101718+0000","last_clean":"2024-09-18T12:25:27.101718+0000","last_became_active":"2024-09-18T12:25:24.721108+0000","last_became_peered":"2024-09-18T12:25:24.721108+0000","last_unstale":"2024-09-18T12:25:27.101718+0000","last_undegraded":"2024-09-18T12:25:27.101718+0000","last_fullsized":"2024-09-18T12:25:27.101718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:41:42.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":167936,"data_stored":52371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2640000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1210000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2839999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2320000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":163840,"data_stored":52352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298172,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073668,"statfs":{"total":96636764160,"available":96331436032,"internally_reserved":0,"allocated":696320,"data_stored":581280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":139264,"data_stored":42524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":167936,"data_stored":52371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298140,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073700,"statfs":{"total":96636764160,"available":96331468800,"internally_reserved":0,"allocated":671744,"data_stored":571452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:25:32.993 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T12:25:32.993 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T12:25:32.994 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T12:25:32.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T12:25:33.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.083+0000 7f3465090640 10 monclient: get_monmap_and_config 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.083+0000 7f3465090640 10 monclient: build_initial_monmap 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.083+0000 7f3465090640 10 monclient: monmap: 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:33.085240+0000 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:33.085240+0000 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:33.087 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:33.087 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:33.087 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 1 Processor -- start 2024-09-18T12:25:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 1 -- start start 2024-09-18T12:25:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient: init 2024-09-18T12:25:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:25:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f346009c9c0 0x7f346009d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient(hunting): picked mon.noname-c con 0x7f346009c9c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f346009e340 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f346009e750 0x7f346012fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient(hunting): picked mon.noname-a con 0x7f346009e750 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346009bdd0 con 0x7f346009e340 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346009a880 con 0x7f346009e750 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346009a440 con 0x7f346009c9c0 2024-09-18T12:25:33.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:33.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.084+0000 7f3465090640 10 monclient(hunting): authenticate will time out at 760.406923s 2024-09-18T12:25:33.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f346009e750 0x7f346012fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f346009e750 0x7f346012fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52192/0 (socket says 172.21.15.116:52192) 2024-09-18T12:25:33.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f3457fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f346009e340 0x7f346012f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57158/0 (socket says 172.21.15.116:57158) 2024-09-18T12:25:33.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 -- 172.21.15.116:0/1274290774 learned_addr learned my addr 172.21.15.116:0/1274290774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): get_auth_request con 0x7f346009e750 auth_method 0 2024-09-18T12:25:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345ed9d640 1 --2- 172.21.15.116:0/1274290774 >> v2:172.21.15.144:3300/0 conn(0x7f346009c9c0 0x7f346009d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 -- 172.21.15.116:0/1274290774 >> v2:172.21.15.144:3300/0 conn(0x7f346009c9c0 msgr2=0x7f346009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 --2- 172.21.15.116:0/1274290774 >> v2:172.21.15.144:3300/0 conn(0x7f346009c9c0 0x7f346009d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 -- 172.21.15.116:0/1274290774 >> v1:172.21.15.116:6789/0 conn(0x7f346009e340 legacy=0x7f346012f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient: _finish_hunting 0 2024-09-18T12:25:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 monclient: found mon.noname-a 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.085+0000 7f345f59e640 1 -- 172.21.15.116:0/1274290774 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346009ba60 con 0x7f346009e750 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345f59e640 1 -- 172.21.15.116:0/1274290774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009e750 msgr2=0x7f346012fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345f59e640 1 --2- 172.21.15.116:0/1274290774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009e750 0x7f346012fe80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f346004ca20 tx=0x7f3458009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdec98c378b97a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 1 -- 172.21.15.116:0/1274290774 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458007610 con 0x7f346009e750 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: dump: 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: _finish_auth 0 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 1 -- 172.21.15.116:0/1274290774 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345802d050 con 0x7f346009e750 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 1 -- 172.21.15.116:0/1274290774 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580079a0 con 0x7f346009e750 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f3465090640 5 monclient: authenticate success, global_id 4329 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f3465090640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f3465090640 4 set_mon_vals no callback set 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f345e59c640 10 monclient: dump: 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f3465090640 10 monclient: shutdown 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f3465090640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f3465090640 1 -- 172.21.15.116:0/1274290774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009e750 msgr2=0x7f346012fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.088+0000 7f3465090640 1 --2- 172.21.15.116:0/1274290774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009e750 0x7f346012fe80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f346004ca20 tx=0x7f3458009840 comp rx=0 tx=0).stop 2024-09-18T12:25:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 -- 172.21.15.116:0/1274290774 shutdown_connections 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 --2- 172.21.15.116:0/1274290774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009e750 0x7f346012fe80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 --2- 172.21.15.116:0/1274290774 >> v2:172.21.15.144:3300/0 conn(0x7f346009c9c0 0x7f346009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 -- 172.21.15.116:0/1274290774 >> 172.21.15.116:0/1274290774 conn(0x7f34600a2e40 msgr2=0x7f34600a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 -- 172.21.15.116:0/1274290774 shutdown_connections 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 -- 172.21.15.116:0/1274290774 wait complete. 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 10 monclient: build_initial_monmap 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 10 monclient: monmap: 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 Processor -- start 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 1 -- start start 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.089+0000 7f3465090640 10 monclient: init 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 0x7f3460140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): picked mon.noname-a con 0x7f346009c9c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346009e340 0x7f34601319f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): picked mon.noname-c con 0x7f346009e340 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346009e750 0x7f3460131f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): picked mon.noname-b con 0x7f346009e750 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460141ce0 con 0x7f346009c9c0 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460099cc0 con 0x7f346009e750 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34601327b0 con 0x7f346009e340 2024-09-18T12:25:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3465090640 10 monclient(hunting): authenticate will time out at 760.412451s 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 0x7f3460140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 0x7f3460140d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52196/0 (socket says 172.21.15.116:52196) 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 -- 172.21.15.116:0/3283265713 learned_addr learned my addr 172.21.15.116:0/3283265713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): get_auth_request con 0x7f346009c9c0 auth_method 0 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346009e340 0x7f34601319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 10 monclient(hunting): get_auth_request con 0x7f346009e340 auth_method 0 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f3457fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 -- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346009e340 msgr2=0x7f34601319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346009e340 0x7f34601319f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345f59e640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346009e750 0x7f3460131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 -- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346009e750 msgr2=0x7f3460131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346009e750 0x7f3460131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 monclient: found mon.noname-a 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.090+0000 7f345ed9d640 1 -- 172.21.15.116:0/3283265713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458009ae0 con 0x7f346009c9c0 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f345ed9d640 1 -- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 msgr2=0x7f3460140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f345ed9d640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 0x7f3460140d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f345000c880 tx=0x7f3450008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 1 -- 172.21.15.116:0/3283265713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345000ce10 con 0x7f346009c9c0 2024-09-18T12:25:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: dump: 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.114 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 1 -- 172.21.15.116:0/3283265713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3450007a40 con 0x7f346009c9c0 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3465090640 5 monclient: authenticate success, global_id 4332 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3465090640 10 monclient: _renew_subs 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3465090640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346006a400 con 0x7f346009c9c0 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 1 -- 172.21.15.116:0/3283265713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450007de0 con 0x7f346009c9c0 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3456ffd640 10 monclient: dump: 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f345dd9b640 4 set_mon_vals no callback set 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3465090640 10 monclient: _renew_subs 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3465090640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.093+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460120be0 con 0x7f346009c9c0 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3456ffd640 1 -- 172.21.15.116:0/3283265713 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3450011600 con 0x7f346009c9c0 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3465090640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3465090640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3465090640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346004bf60 con 0x7f346009c9c0 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3456ffd640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3440038490 0x7f344003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3456ffd640 1 -- 172.21.15.116:0/3283265713 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f345004b7b0 con 0x7f346009c9c0 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3457fff640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3440038490 0x7f344003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.094+0000 7f3457fff640 10 monclient: get_auth_request con 0x7f3440038490 auth_method 0 2024-09-18T12:25:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.096+0000 7f3457fff640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3440038490 0x7f344003a940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3448007310 tx=0x7f3448007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.098+0000 7f3456ffd640 1 -- 172.21.15.116:0/3283265713 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3450017af0 con 0x7f346009c9c0 2024-09-18T12:25:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.098+0000 7f3456ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.098+0000 7f3456ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:33.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.286+0000 7f3465090640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-18T12:25:33.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.286+0000 7f3465090640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-18T12:25:33.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.286+0000 7f3465090640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.286+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3460099ef0 con 0x7f346009c9c0 2024-09-18T12:25:33.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.286+0000 7f3456ffd640 1 -- 172.21.15.116:0/3283265713 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f345001e020 con 0x7f346009c9c0 2024-09-18T12:25:33.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.286+0000 7f3456ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-18T12:25:33.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.286+0000 7f3456ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:33.290 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:33.291 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T12:25:33.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3440038490 msgr2=0x7f344003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3440038490 0x7f344003a940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3448007310 tx=0x7f3448007e40 comp rx=0 tx=0).stop 2024-09-18T12:25:33.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 10 monclient: shutdown 2024-09-18T12:25:33.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:33.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 msgr2=0x7f3460140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 0x7f3460140d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f345000c880 tx=0x7f3450008040 comp rx=0 tx=0).stop 2024-09-18T12:25:33.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 shutdown_connections 2024-09-18T12:25:33.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3440038490 0x7f344003a940 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346009e750 0x7f3460131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346009e340 0x7f34601319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 --2- 172.21.15.116:0/3283265713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f346009c9c0 0x7f3460140d90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.288+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 >> 172.21.15.116:0/3283265713 conn(0x7f34600a2e40 msgr2=0x7f34600a2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:33.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.289+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 shutdown_connections 2024-09-18T12:25:33.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.289+0000 7f3465090640 1 -- 172.21.15.116:0/3283265713 wait complete. 2024-09-18T12:25:33.301 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T12:25:33.301 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-18T12:25:33.369 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-18T12:25:33.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:25:33.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.460+0000 7f70ecef8640 10 monclient: get_monmap_and_config 2024-09-18T12:25:33.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.460+0000 7f70ecef8640 10 monclient: build_initial_monmap 2024-09-18T12:25:33.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.460+0000 7f70ecef8640 10 monclient: monmap: 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:33.461869+0000 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:33.461869+0000 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.460+0000 7f70ecef8640 1 Processor -- start 2024-09-18T12:25:33.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.460+0000 7f70ecef8640 1 -- start start 2024-09-18T12:25:33.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.460+0000 7f70ecef8640 10 monclient: init 2024-09-18T12:25:33.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f70e81318f0 0x7f70e8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient(hunting): picked mon.noname-a con 0x7f70e81318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70e809ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f70e8136c20 0x7f70e8139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient(hunting): picked mon.noname-c con 0x7f70e8136c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e809f530 con 0x7f70e809ea50 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e8131d50 con 0x7f70e81318f0 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e809fbf0 con 0x7f70e8136c20 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70ecef8640 10 monclient(hunting): authenticate will time out at 760.783489s 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f70e81318f0 0x7f70e8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f70e81318f0 0x7f70e8131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52216/0 (socket says 172.21.15.116:52216) 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 -- 172.21.15.116:0/472978610 learned_addr learned my addr 172.21.15.116:0/472978610 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): get_auth_request con 0x7f70e81318f0 auth_method 0 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 -- 172.21.15.116:0/472978610 >> v2:172.21.15.144:3300/0 conn(0x7f70e8136c20 msgr2=0x7f70e8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 --2- 172.21.15.116:0/472978610 >> v2:172.21.15.144:3300/0 conn(0x7f70e8136c20 0x7f70e8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 -- 172.21.15.116:0/472978610 >> v1:172.21.15.144:6789/0 conn(0x7f70e809ea50 legacy=0x7f70e809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient: _finish_hunting 0 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 monclient: found mon.noname-a 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.461+0000 7f70e659c640 1 -- 172.21.15.116:0/472978610 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e809fe30 con 0x7f70e81318f0 2024-09-18T12:25:33.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e659c640 1 -- 172.21.15.116:0/472978610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e81318f0 msgr2=0x7f70e8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e659c640 1 --2- 172.21.15.116:0/472978610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e81318f0 0x7f70e8131f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70e809c9e0 tx=0x7f70e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12dcdcae78fc4bc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 1 -- 172.21.15.116:0/472978610 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0032440 con 0x7f70e81318f0 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: dump: 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.474 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: _finish_auth 0 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 1 -- 172.21.15.116:0/472978610 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e00325a0 con 0x7f70e81318f0 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70ecef8640 5 monclient: authenticate success, global_id 4335 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70ecef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70ecef8640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70ecef8640 4 set_mon_vals no callback set 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 1 -- 172.21.15.116:0/472978610 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0032940 con 0x7f70e81318f0 2024-09-18T12:25:33.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.464+0000 7f70e559a640 10 monclient: dump: 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.476 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 10 monclient: shutdown 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 -- 172.21.15.116:0/472978610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e81318f0 msgr2=0x7f70e8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 --2- 172.21.15.116:0/472978610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e81318f0 0x7f70e8131f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70e809c9e0 tx=0x7f70e00091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 -- 172.21.15.116:0/472978610 shutdown_connections 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 --2- 172.21.15.116:0/472978610 >> v2:172.21.15.144:3300/0 conn(0x7f70e8136c20 0x7f70e8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 --2- 172.21.15.116:0/472978610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e81318f0 0x7f70e8131f30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 -- 172.21.15.116:0/472978610 >> 172.21.15.116:0/472978610 conn(0x7f70e8067110 msgr2=0x7f70e8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 -- 172.21.15.116:0/472978610 shutdown_connections 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.465+0000 7f70ecef8640 1 -- 172.21.15.116:0/472978610 wait complete. 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient: build_initial_monmap 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient: monmap: 2024-09-18T12:25:33.477 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 Processor -- start 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 -- start start 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient: init 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:33.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e809ea50 0x7f70e8130990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): picked mon.noname-a con 0x7f70e809ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70e81318f0 0x7f70e8130ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): picked mon.noname-c con 0x7f70e81318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70e8136c20 0x7f70e8099750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): picked mon.noname-b con 0x7f70e8136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e8141ef0 con 0x7f70e809ea50 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e811d110 con 0x7f70e8136c20 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e811cda0 con 0x7f70e81318f0 2024-09-18T12:25:33.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70ecef8640 10 monclient(hunting): authenticate will time out at 760.788926s 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.466+0000 7f70e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e809ea50 0x7f70e8130990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70e81318f0 0x7f70e8130ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e809ea50 0x7f70e8130990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52224/0 (socket says 172.21.15.116:52224) 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): get_auth_request con 0x7f70e81318f0 auth_method 0 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 1 -- 172.21.15.116:0/3101730345 learned_addr learned my addr 172.21.15.116:0/3101730345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 10 monclient(hunting): get_auth_request con 0x7f70e809ea50 auth_method 0 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e6d9d640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70e8136c20 0x7f70e8099750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e6d9d640 10 monclient(hunting): get_auth_request con 0x7f70e8136c20 auth_method 0 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 1 -- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70e8136c20 msgr2=0x7f70e8099750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70e8136c20 0x7f70e8099750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 1 -- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e809ea50 msgr2=0x7f70e8130990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e809ea50 0x7f70e8130990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 1 monclient: found mon.noname-c 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e5d9b640 1 -- 172.21.15.116:0/3101730345 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e0009450 con 0x7f70e81318f0 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e659c640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e809ea50 0x7f70e8130990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.467+0000 7f70e6d9d640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70e8136c20 0x7f70e8099750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:33.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70e5d9b640 1 -- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70e81318f0 msgr2=0x7f70e8130ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70e5d9b640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70e81318f0 0x7f70e8130ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00bb30 tx=0x7f70dc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 1 -- 172.21.15.116:0/3101730345 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc00cbf0 con 0x7f70e81318f0 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: dump: 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.483 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 1 -- 172.21.15.116:0/3101730345 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc00cd70 con 0x7f70e81318f0 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70ecef8640 5 monclient: authenticate success, global_id 4316 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70ecef8640 10 monclient: _renew_subs 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70ecef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e809b2c0 con 0x7f70e81318f0 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 1 -- 172.21.15.116:0/3101730345 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc014660 con 0x7f70e81318f0 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:33.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70d67fc640 10 monclient: dump: 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70e4d99640 4 set_mon_vals no callback set 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70ecef8640 10 monclient: _renew_subs 2024-09-18T12:25:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70ecef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.469+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e81374d0 con 0x7f70e81318f0 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70d67fc640 1 -- 172.21.15.116:0/3101730345 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f70dc014a50 con 0x7f70e81318f0 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70d67fc640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70c8038490 0x7f70c803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70d67fc640 1 -- 172.21.15.116:0/3101730345 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f70dc04b740 con 0x7f70e81318f0 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70ecef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70ecef8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70ecef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e804bf60 con 0x7f70e81318f0 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70e659c640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70c8038490 0x7f70c803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.470+0000 7f70e659c640 10 monclient: get_auth_request con 0x7f70c8038490 auth_method 0 2024-09-18T12:25:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.473+0000 7f70e659c640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70c8038490 0x7f70c803a940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009420 tx=0x7f70e0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.475+0000 7f70d67fc640 1 -- 172.21.15.116:0/3101730345 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f70dc001520 con 0x7f70e81318f0 2024-09-18T12:25:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.475+0000 7f70d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.475+0000 7f70d67fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.629+0000 7f70ecef8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.629+0000 7f70ecef8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.629+0000 7f70ecef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.629+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70e81376f0 con 0x7f70e81318f0 2024-09-18T12:25:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.630+0000 7f70d67fc640 1 -- 172.21.15.116:0/3101730345 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+11409 (secure 0 0 0) 0x7f70dc01bab0 con 0x7f70e81318f0 2024-09-18T12:25:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.630+0000 7f70d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:25:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.630+0000 7f70d67fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:33.636 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:33.636 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":18,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:25:31.508205+0000","last_up_change":"2024-09-18T12:25:21.884529+0000","last_in_change":"2024-09-18T12:25:15.496296+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T12:25:22.596367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T12:25:24.541264+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":31578},{"type":"v1","addr":"172.21.15.116:6817","nonce":31578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":31578},{"type":"v1","addr":"172.21.15.116:6819","nonce":31578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":31578},{"type":"v1","addr":"172.21.15.116:6823","nonce":31578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":31578},{"type":"v1","addr":"172.21.15.116:6821","nonce":31578}]},"public_addr":"172.21.15.116:6817/31578","cluster_addr":"172.21.15.116:6819/31578","heartbeat_back_addr":"172.21.15.116:6823/31578","heartbeat_front_addr":"172.21.15.116:6821/31578","state":["exists","up"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":31579},{"type":"v1","addr":"172.21.15.116:6801","nonce":31579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":31579},{"type":"v1","addr":"172.21.15.116:6803","nonce":31579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":31579},{"type":"v1","addr":"172.21.15.116:6807","nonce":31579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":31579},{"type":"v1","addr":"172.21.15.116:6805","nonce":31579}]},"public_addr":"172.21.15.116:6801/31579","cluster_addr":"172.21.15.116:6803/31579","heartbeat_back_addr":"172.21.15.116:6807/31579","heartbeat_front_addr":"172.21.15.116:6805/31579","state":["exists","up"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":31580},{"type":"v1","addr":"172.21.15.116:6809","nonce":31580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":31580},{"type":"v1","addr":"172.21.15.116:6811","nonce":31580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":31580},{"type":"v1","addr":"172.21.15.116:6815","nonce":31580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":31580},{"type":"v1","addr":"172.21.15.116:6813","nonce":31580}]},"public_addr":"172.21.15.116:6809/31580","cluster_addr":"172.21.15.116:6811/31580","heartbeat_back_addr":"172.21.15.116:6815/31580","heartbeat_front_addr":"172.21.15.116:6813/31580","state":["exists","up"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":32112},{"type":"v1","addr":"172.21.15.144:6806","nonce":32112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":32112},{"type":"v1","addr":"172.21.15.144:6811","nonce":32112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":32112},{"type":"v1","addr":"172.21.15.144:6818","nonce":32112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":32112},{"type":"v1","addr":"172.21.15.144:6815","nonce":32112}]},"public_addr":"172.21.15.144:6806/32112","cluster_addr":"172.21.15.144:6811/32112","heartbeat_back_addr":"172.21.15.144:6818/32112","heartbeat_front_addr":"172.21.15.144:6815/32112","state":["exists","up"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":32113},{"type":"v1","addr":"172.21.15.144:6801","nonce":32113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":32113},{"type":"v1","addr":"172.21.15.144:6803","nonce":32113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":32113},{"type":"v1","addr":"172.21.15.144:6810","nonce":32113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6805","nonce":32113},{"type":"v1","addr":"172.21.15.144:6807","nonce":32113}]},"public_addr":"172.21.15.144:6801/32113","cluster_addr":"172.21.15.144:6803/32113","heartbeat_back_addr":"172.21.15.144:6810/32113","heartbeat_front_addr":"172.21.15.144:6807/32113","state":["exists","up"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":32114},{"type":"v1","addr":"172.21.15.144:6814","nonce":32114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":32114},{"type":"v1","addr":"172.21.15.144:6819","nonce":32114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":32114},{"type":"v1","addr":"172.21.15.144:6823","nonce":32114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":32114},{"type":"v1","addr":"172.21.15.144:6821","nonce":32114}]},"public_addr":"172.21.15.144:6814/32114","cluster_addr":"172.21.15.144:6819/32114","heartbeat_back_addr":"172.21.15.144:6823/32114","heartbeat_front_addr":"172.21.15.144:6821/32114","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.664684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.608350+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.689049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.706015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.672781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.692341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70c8038490 msgr2=0x7f70c803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70c8038490 0x7f70c803a940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009420 tx=0x7f70e0032000 comp rx=0 tx=0).stop 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 10 monclient: shutdown 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70e81318f0 msgr2=0x7f70e8130ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70e81318f0 0x7f70e8130ed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00bb30 tx=0x7f70dc009780 comp rx=0 tx=0).stop 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 shutdown_connections 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70c8038490 0x7f70c803a940 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70e8136c20 0x7f70e8099750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70e81318f0 0x7f70e8130ed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 --2- 172.21.15.116:0/3101730345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e809ea50 0x7f70e8130990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 >> 172.21.15.116:0/3101730345 conn(0x7f70e8067110 msgr2=0x7f70e80a2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 shutdown_connections 2024-09-18T12:25:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.632+0000 7f70ecef8640 1 -- 172.21.15.116:0/3101730345 wait complete. 2024-09-18T12:25:33.645 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T12:25:22.596367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T12:25:24.541264+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 17, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-18T12:25:33.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-18T12:25:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.738+0000 7fc7728c9640 10 monclient: get_monmap_and_config 2024-09-18T12:25:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.738+0000 7fc7728c9640 10 monclient: build_initial_monmap 2024-09-18T12:25:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.738+0000 7fc7728c9640 10 monclient: monmap: 2024-09-18T12:25:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:33.740120+0000 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:33.740120+0000 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:33.741 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 1 Processor -- start 2024-09-18T12:25:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 1 -- start start 2024-09-18T12:25:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient: init 2024-09-18T12:25:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc76c09eb00 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc76c069cf0 0x7fc76c09d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc76c069cf0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc76c136bb0 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76c09cd90 con 0x7fc76c136bb0 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76c0a00c0 con 0x7fc76c09eb00 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc76c06a150 con 0x7fc76c069cf0 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc7728c9640 10 monclient(hunting): authenticate will time out at 761.061714s 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc770e66640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc76c136bb0 0x7fc76c138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57164/0 (socket says 172.21.15.116:57164) 2024-09-18T12:25:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.739+0000 7fc770e66640 1 -- 172.21.15.116:0/3420820889 learned_addr learned my addr 172.21.15.116:0/3420820889 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/3420820889 >> v2:172.21.15.144:3300/0 conn(0x7fc76c069cf0 0x7fc76c09d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): get_auth_request con 0x7fc76c069cf0 auth_method 0 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 1 -- 172.21.15.116:0/3420820889 >> v1:172.21.15.144:6789/0 conn(0x7fc76c09eb00 legacy=0x7fc76c09eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 1 -- 172.21.15.116:0/3420820889 >> v1:172.21.15.116:6789/0 conn(0x7fc76c136bb0 legacy=0x7fc76c138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:25:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.740+0000 7fc76b7fe640 1 -- 172.21.15.116:0/3420820889 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c09fe10 con 0x7fc76c069cf0 2024-09-18T12:25:33.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.743+0000 7fc76b7fe640 1 -- 172.21.15.116:0/3420820889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c069cf0 msgr2=0x7fc76c09d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.743+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/3420820889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c069cf0 0x7fc76c09d000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc760009850 tx=0x7fc7600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db3f6288cfc207fc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.743+0000 7fc76affd640 1 -- 172.21.15.116:0/3420820889 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760032440 con 0x7fc76c069cf0 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.743+0000 7fc76affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.743+0000 7fc76affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.743+0000 7fc76affd640 10 monclient: dump: 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.750 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 10 monclient: _finish_auth 0 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 1 -- 172.21.15.116:0/3420820889 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7600325a0 con 0x7fc76c069cf0 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 5 monclient: authenticate success, global_id 4312 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 1 -- 172.21.15.116:0/3420820889 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760032940 con 0x7fc76c069cf0 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 4 set_mon_vals no callback set 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:33.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc76affd640 10 monclient: dump: 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 10 monclient: shutdown 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 1 -- 172.21.15.116:0/3420820889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c069cf0 msgr2=0x7fc76c09d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 1 --2- 172.21.15.116:0/3420820889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c069cf0 0x7fc76c09d000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc760009850 tx=0x7fc7600090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 1 -- 172.21.15.116:0/3420820889 shutdown_connections 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 1 --2- 172.21.15.116:0/3420820889 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c069cf0 0x7fc76c09d000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 1 -- 172.21.15.116:0/3420820889 >> 172.21.15.116:0/3420820889 conn(0x7fc76c067d30 msgr2=0x7fc76c0a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 1 -- 172.21.15.116:0/3420820889 shutdown_connections 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.744+0000 7fc7728c9640 1 -- 172.21.15.116:0/3420820889 wait complete. 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.745+0000 7fc7728c9640 10 monclient: build_initial_monmap 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.745+0000 7fc7728c9640 10 monclient: monmap: 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:33.756 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.756 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:33.756 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:33.756 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:33.756 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.745+0000 7fc7728c9640 1 Processor -- start 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.745+0000 7fc7728c9640 1 -- start start 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.745+0000 7fc7728c9640 10 monclient: init 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc76c069cf0 0x7fc76c11b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc76c069cf0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c09eb00 0x7fc76c11bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc76c09eb00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 0x7fc76c11c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc76c136bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c135d80 con 0x7fc76c136bb0 2024-09-18T12:25:33.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c09bf10 con 0x7fc76c09eb00 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0a30f0 con 0x7fc76c069cf0 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc7728c9640 10 monclient(hunting): authenticate will time out at 761.068129s 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 0x7fc76c11c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc76c069cf0 0x7fc76c11b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 0x7fc76c11c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52228/0 (socket says 172.21.15.116:52228) 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 10 monclient(hunting): get_auth_request con 0x7fc76c069cf0 auth_method 0 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 -- 172.21.15.116:0/2108665109 learned_addr learned my addr 172.21.15.116:0/2108665109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): get_auth_request con 0x7fc76c136bb0 auth_method 0 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c09eb00 0x7fc76c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 10 monclient(hunting): get_auth_request con 0x7fc76c09eb00 auth_method 0 2024-09-18T12:25:33.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 -- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc76c069cf0 msgr2=0x7fc76c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc76c069cf0 0x7fc76c11b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 -- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c09eb00 msgr2=0x7fc76c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c09eb00 0x7fc76c11bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient: _finish_hunting 0 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 monclient: found mon.noname-a 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:33.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc770e66640 1 -- 172.21.15.116:0/2108665109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760009390 con 0x7fc76c136bb0 2024-09-18T12:25:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.746+0000 7fc76bfff640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc76c069cf0 0x7fc76c11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc770e66640 1 -- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 msgr2=0x7fc76c11c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc770e66640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 0x7fc76c11c1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc76400cc20 tx=0x7fc764005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 1 -- 172.21.15.116:0/2108665109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76400ce40 con 0x7fc76c136bb0 2024-09-18T12:25:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: dump: 2024-09-18T12:25:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.765 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.765 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.765 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.765 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.765 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.765 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.766 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.766 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.766 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:33.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:33.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc7728c9640 5 monclient: authenticate success, global_id 4341 2024-09-18T12:25:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc7728c9640 10 monclient: _renew_subs 2024-09-18T12:25:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc7728c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76c11e630 con 0x7fc76c136bb0 2024-09-18T12:25:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 1 -- 172.21.15.116:0/2108665109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76400aa60 con 0x7fc76c136bb0 2024-09-18T12:25:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 1 -- 172.21.15.116:0/2108665109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76400ae20 con 0x7fc76c136bb0 2024-09-18T12:25:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc753fff640 10 monclient: dump: 2024-09-18T12:25:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:33.769 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc7728c9640 10 monclient: _renew_subs 2024-09-18T12:25:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc7728c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c09b580 con 0x7fc76c136bb0 2024-09-18T12:25:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.748+0000 7fc76a7fc640 4 set_mon_vals no callback set 2024-09-18T12:25:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.749+0000 7fc753fff640 1 -- 172.21.15.116:0/2108665109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc764007300 con 0x7fc76c136bb0 2024-09-18T12:25:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.749+0000 7fc753fff640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc758038490 0x7fc75803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.749+0000 7fc7728c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.749+0000 7fc7728c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.749+0000 7fc7728c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.749+0000 7fc753fff640 1 -- 172.21.15.116:0/2108665109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fc76404b740 con 0x7fc76c136bb0 2024-09-18T12:25:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.749+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76c05ed70 con 0x7fc76c136bb0 2024-09-18T12:25:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.750+0000 7fc76bfff640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc758038490 0x7fc75803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.750+0000 7fc76bfff640 10 monclient: get_auth_request con 0x7fc758038490 auth_method 0 2024-09-18T12:25:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.752+0000 7fc76bfff640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc758038490 0x7fc75803a940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc75c007fd0 tx=0x7fc75c007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.754+0000 7fc753fff640 1 -- 172.21.15.116:0/2108665109 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc7640075b0 con 0x7fc76c136bb0 2024-09-18T12:25:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.754+0000 7fc753fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.754+0000 7fc753fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:33.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.906+0000 7fc7728c9640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T12:25:33.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.907+0000 7fc7728c9640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T12:25:33.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.907+0000 7fc7728c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:33.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.907+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc76c11ea50 con 0x7fc76c136bb0 2024-09-18T12:25:33.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.907+0000 7fc753fff640 1 -- 172.21.15.116:0/2108665109 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc764010af0 con 0x7fc76c136bb0 2024-09-18T12:25:33.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.907+0000 7fc753fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T12:25:33.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.907+0000 7fc753fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:33.910 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-09-18T12:25:33.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc758038490 msgr2=0x7fc75803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc758038490 0x7fc75803a940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc75c007fd0 tx=0x7fc75c007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:33.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 10 monclient: shutdown 2024-09-18T12:25:33.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:33.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 msgr2=0x7fc76c11c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 0x7fc76c11c1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc76400cc20 tx=0x7fc764005340 comp rx=0 tx=0).stop 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 shutdown_connections 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc758038490 0x7fc75803a940 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c136bb0 0x7fc76c11c1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc76c09eb00 0x7fc76c11bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 --2- 172.21.15.116:0/2108665109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc76c069cf0 0x7fc76c11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 >> 172.21.15.116:0/2108665109 conn(0x7fc76c067d30 msgr2=0x7fc76c139000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 shutdown_connections 2024-09-18T12:25:33.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:33.909+0000 7fc7728c9640 1 -- 172.21.15.116:0/2108665109 wait complete. 2024-09-18T12:25:33.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient: get_monmap_and_config 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient: monmap: 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:34.017332+0000 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:34.017332+0000 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 1 Processor -- start 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 1 -- start start 2024-09-18T12:25:34.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient: init 2024-09-18T12:25:34.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T12:25:34.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64a009c440 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:34.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f64a009c810 0x7f64a006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient(hunting): picked mon.noname-a con 0x7f64a009c810 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64a009dfd0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a0069fe0 con 0x7f64a009c440 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a006f280 con 0x7f64a009dfd0 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a006a610 con 0x7f64a009c810 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.016+0000 7f64a7ce8640 10 monclient(hunting): authenticate will time out at 761.338925s 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.017+0000 7f64a5283640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f64a009c810 0x7f64a006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.017+0000 7f64a5283640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f64a009c810 0x7f64a006ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52234/0 (socket says 172.21.15.116:52234) 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.017+0000 7f64a5283640 1 -- 172.21.15.116:0/3415139876 learned_addr learned my addr 172.21.15.116:0/3415139876 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.017+0000 7f64a5283640 10 monclient(hunting): get_auth_request con 0x7f64a009c810 auth_method 0 2024-09-18T12:25:34.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.017+0000 7f64a5283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.017+0000 7f64a5283640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.017+0000 7f64a5283640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5a84640 1 -- 172.21.15.116:0/3415139876 >> v1:172.21.15.116:6789/0 conn(0x7f64a009c440 legacy=0x7f64a009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a6285640 1 -- 172.21.15.116:0/3415139876 >> v1:172.21.15.144:6789/0 conn(0x7f64a009dfd0 legacy=0x7f64a0139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 1 -- 172.21.15.116:0/3415139876 >> v1:172.21.15.144:6789/0 conn(0x7f64a009dfd0 legacy=0x7f64a0139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 1 -- 172.21.15.116:0/3415139876 >> v1:172.21.15.116:6789/0 conn(0x7f64a009c440 legacy=0x7f64a009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 1 monclient: found mon.noname-a 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.019+0000 7f64a5283640 1 -- 172.21.15.116:0/3415139876 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0068da0 con 0x7f64a009c810 2024-09-18T12:25:34.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a5283640 1 -- 172.21.15.116:0/3415139876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009c810 msgr2=0x7f64a006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a5283640 1 --2- 172.21.15.116:0/3415139876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009c810 0x7f64a006ed40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6490009a10 tx=0x7f64900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11402c475335b684 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 1 -- 172.21.15.116:0/3415139876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649002f4b0 con 0x7f64a009c810 2024-09-18T12:25:34.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:34.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: dump: 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 1 -- 172.21.15.116:0/3415139876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649002f610 con 0x7f64a009c810 2024-09-18T12:25:34.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 5 monclient: authenticate success, global_id 4344 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 1 -- 172.21.15.116:0/3415139876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649002f9f0 con 0x7f64a009c810 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 4 set_mon_vals no callback set 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a4a82640 10 monclient: dump: 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.029 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 10 monclient: shutdown 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3415139876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009c810 msgr2=0x7f64a006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3415139876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009c810 0x7f64a006ed40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6490009a10 tx=0x7f64900091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3415139876 shutdown_connections 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3415139876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009c810 0x7f64a006ed40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3415139876 >> 172.21.15.116:0/3415139876 conn(0x7f64a0067d30 msgr2=0x7f64a00a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3415139876 shutdown_connections 2024-09-18T12:25:34.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3415139876 wait complete. 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 10 monclient: monmap: 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 Processor -- start 2024-09-18T12:25:34.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 1 -- start start 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.023+0000 7f64a7ce8640 10 monclient: init 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64a009c440 0x7f64a0141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): picked mon.noname-b con 0x7f64a009c440 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64a009c810 0x7f64a0131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): picked mon.noname-c con 0x7f64a009c810 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 0x7f64a0132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): picked mon.noname-a con 0x7f64a009dfd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a0142370 con 0x7f64a009dfd0 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00997d0 con 0x7f64a009c440 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a0099e50 con 0x7f64a009c810 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a7ce8640 10 monclient(hunting): authenticate will time out at 761.346194s 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 0x7f64a0132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64a009c810 0x7f64a0131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 0x7f64a0132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52240/0 (socket says 172.21.15.116:52240) 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 -- 172.21.15.116:0/3252055695 learned_addr learned my addr 172.21.15.116:0/3252055695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): get_auth_request con 0x7f64a009dfd0 auth_method 0 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 10 monclient(hunting): get_auth_request con 0x7f64a009c810 auth_method 0 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 -- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64a009c810 msgr2=0x7f64a0131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64a009c810 0x7f64a0131fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a5a84640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64a009c440 0x7f64a0141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 -- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64a009c440 msgr2=0x7f64a0141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64a009c440 0x7f64a0141470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 monclient: found mon.noname-a 2024-09-18T12:25:34.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.024+0000 7f64a6285640 1 -- 172.21.15.116:0/3252055695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490009450 con 0x7f64a009dfd0 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f64a6285640 1 -- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 msgr2=0x7f64a0132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f64a6285640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 0x7f64a0132510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f649800c910 tx=0x7f64980018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f6495ffb640 1 -- 172.21.15.116:0/3252055695 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649800cea0 con 0x7f64a009dfd0 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f6495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f6495ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f6495ffb640 10 monclient: dump: 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.037 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f6495ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f6495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f6495ffb640 1 -- 172.21.15.116:0/3252055695 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649800a640 con 0x7f64a009dfd0 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f64a7ce8640 5 monclient: authenticate success, global_id 4350 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.026+0000 7f64a7ce8640 10 monclient: _renew_subs 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f64a7ce8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a009b6e0 con 0x7f64a009dfd0 2024-09-18T12:25:34.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f6495ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f6495ffb640 1 -- 172.21.15.116:0/3252055695 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649800aa20 con 0x7f64a009dfd0 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f6495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f6495ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f6495ffb640 10 monclient: dump: 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.039 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f64a7ce8640 10 monclient: _renew_subs 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f6497fff640 4 set_mon_vals no callback set 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f64a7ce8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a011e190 con 0x7f64a009dfd0 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.027+0000 7f6495ffb640 1 -- 172.21.15.116:0/3252055695 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f64980070f0 con 0x7f64a009dfd0 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f64a7ce8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f64a7ce8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f64a7ce8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f6495ffb640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f64840384e0 0x7f648403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a005ed70 con 0x7f64a009dfd0 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f6495ffb640 1 -- 172.21.15.116:0/3252055695 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f649804ad30 con 0x7f64a009dfd0 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f64a5a84640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f64840384e0 0x7f648403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.028+0000 7f64a5a84640 10 monclient: get_auth_request con 0x7f64840384e0 auth_method 0 2024-09-18T12:25:34.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.030+0000 7f64a5a84640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f64840384e0 0x7f648403a990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f649c007fd0 tx=0x7f649c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.032+0000 7f6495ffb640 1 -- 172.21.15.116:0/3252055695 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6498018620 con 0x7f64a009dfd0 2024-09-18T12:25:34.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.032+0000 7f6495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.032+0000 7f6495ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:34.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.189+0000 7f64a7ce8640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T12:25:34.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.189+0000 7f64a7ce8640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T12:25:34.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.189+0000 7f64a7ce8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.189+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f64a0068b40 con 0x7f64a009dfd0 2024-09-18T12:25:34.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.189+0000 7f6495ffb640 1 -- 172.21.15.116:0/3252055695 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6498018260 con 0x7f64a009dfd0 2024-09-18T12:25:34.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.189+0000 7f6495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T12:25:34.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.189+0000 7f6495ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:34.192 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 8 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f64840384e0 msgr2=0x7f648403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f64840384e0 0x7f648403a990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f649c007fd0 tx=0x7f649c009040 comp rx=0 tx=0).stop 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 10 monclient: shutdown 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 msgr2=0x7f64a0132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 0x7f64a0132510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f649800c910 tx=0x7f64980018c0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 shutdown_connections 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f64840384e0 0x7f648403a990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a009dfd0 0x7f64a0132510 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f64a009c810 0x7f64a0131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 --2- 172.21.15.116:0/3252055695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64a009c440 0x7f64a0141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 >> 172.21.15.116:0/3252055695 conn(0x7f64a0067d30 msgr2=0x7f64a0134d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 shutdown_connections 2024-09-18T12:25:34.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.191+0000 7f64a7ce8640 1 -- 172.21.15.116:0/3252055695 wait complete. 2024-09-18T12:25:34.204 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-18T12:25:34.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:25:34.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.296+0000 7f18b80ef640 10 monclient: get_monmap_and_config 2024-09-18T12:25:34.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.296+0000 7f18b80ef640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.296+0000 7f18b80ef640 10 monclient: monmap: 2024-09-18T12:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:34.297751+0000 2024-09-18T12:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:34.297751+0000 2024-09-18T12:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.296+0000 7f18b80ef640 1 Processor -- start 2024-09-18T12:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.296+0000 7f18b80ef640 1 -- start start 2024-09-18T12:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.296+0000 7f18b80ef640 10 monclient: init 2024-09-18T12:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18b01318f0 0x7f18b0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient(hunting): picked mon.noname-a con 0x7f18b01318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient(hunting): picked mon.noname-b con 0x7f18b009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18b0068d90 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b0131cc0 con 0x7f18b0068d90 2024-09-18T12:25:34.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b009fbf0 con 0x7f18b01318f0 2024-09-18T12:25:34.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b009fe30 con 0x7f18b009ea50 2024-09-18T12:25:34.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b80ef640 10 monclient(hunting): authenticate will time out at 761.619391s 2024-09-18T12:25:34.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b5e8b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18b01318f0 0x7f18b0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b5e8b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18b01318f0 0x7f18b0131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52252/0 (socket says 172.21.15.116:52252) 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b5e8b640 1 -- 172.21.15.116:0/3280475239 learned_addr learned my addr 172.21.15.116:0/3280475239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b5e8b640 10 monclient(hunting): get_auth_request con 0x7f18b01318f0 auth_method 0 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b5e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b5e8b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b5e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): get_auth_request con 0x7f18b009ea50 auth_method 0 2024-09-18T12:25:34.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 1 -- 172.21.15.116:0/3280475239 >> v2:172.21.15.116:3300/0 conn(0x7f18b01318f0 msgr2=0x7f18b0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 1 --2- 172.21.15.116:0/3280475239 >> v2:172.21.15.116:3300/0 conn(0x7f18b01318f0 0x7f18b0131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 1 -- 172.21.15.116:0/3280475239 >> v1:172.21.15.116:6789/0 conn(0x7f18b0068d90 legacy=0x7f18b00691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 1 monclient: found mon.noname-b 2024-09-18T12:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.297+0000 7f18b568a640 1 -- 172.21.15.116:0/3280475239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b00a1210 con 0x7f18b009ea50 2024-09-18T12:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b568a640 1 -- 172.21.15.116:0/3280475239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 msgr2=0x7f18b009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b568a640 1 --2- 172.21.15.116:0/3280475239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b009ee20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009850 tx=0x7f18a40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b7c44f98939010f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 1 -- 172.21.15.116:0/3280475239 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a402add0 con 0x7f18b009ea50 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: dump: 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.323 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 5 monclient: authenticate success, global_id 4325 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 1 -- 172.21.15.116:0/3280475239 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a4032440 con 0x7f18b009ea50 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 4 set_mon_vals no callback set 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 1 -- 172.21.15.116:0/3280475239 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a40327e0 con 0x7f18b009ea50 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b4e89640 10 monclient: dump: 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.325 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 10 monclient: shutdown 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 -- 172.21.15.116:0/3280475239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 msgr2=0x7f18b009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 --2- 172.21.15.116:0/3280475239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b009ee20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009850 tx=0x7f18a40090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 -- 172.21.15.116:0/3280475239 shutdown_connections 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 --2- 172.21.15.116:0/3280475239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b009ee20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 --2- 172.21.15.116:0/3280475239 >> v2:172.21.15.116:3300/0 conn(0x7f18b01318f0 0x7f18b0131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 -- 172.21.15.116:0/3280475239 >> 172.21.15.116:0/3280475239 conn(0x7f18b0067110 msgr2=0x7f18b0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 -- 172.21.15.116:0/3280475239 shutdown_connections 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.301+0000 7f18b80ef640 1 -- 172.21.15.116:0/3280475239 wait complete. 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient: monmap: 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:34.327 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 Processor -- start 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 -- start start 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient: init 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 0x7f18b009a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): picked mon.noname-a con 0x7f18b0068d90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b0130b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): picked mon.noname-c con 0x7f18b009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18b01318f0 0x7f18b01310a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): picked mon.noname-b con 0x7f18b01318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:34.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b009b5d0 con 0x7f18b0068d90 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b009bf10 con 0x7f18b01318f0 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b00a44b0 con 0x7f18b009ea50 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b80ef640 10 monclient(hunting): authenticate will time out at 761.624837s 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b5e8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 0x7f18b009a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b568a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b0130b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b5e8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 0x7f18b009a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52262/0 (socket says 172.21.15.116:52262) 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b5e8b640 1 -- 172.21.15.116:0/727887694 learned_addr learned my addr 172.21.15.116:0/727887694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b5e8b640 10 monclient(hunting): get_auth_request con 0x7f18b0068d90 auth_method 0 2024-09-18T12:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b5e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b5e8b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b5e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b568a640 10 monclient(hunting): get_auth_request con 0x7f18b009ea50 auth_method 0 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b568a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b568a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.302+0000 7f18b568a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b568a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b568a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b568a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 1 -- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 msgr2=0x7f18b0130b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b0130b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 1 -- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18b01318f0 msgr2=0x7f18b01310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b668c640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18b01318f0 0x7f18b01310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18b01318f0 0x7f18b01310a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 1 monclient: found mon.noname-a 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b5e8b640 1 -- 172.21.15.116:0/727887694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4009390 con 0x7f18b0068d90 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.303+0000 7f18b568a640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b0130b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b5e8b640 1 -- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 msgr2=0x7f18b009a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b5e8b640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 0x7f18b009a8f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f18a0007e80 tx=0x7f18a0009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 1 -- 172.21.15.116:0/727887694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0009e90 con 0x7f18b0068d90 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: dump: 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 1 -- 172.21.15.116:0/727887694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a000f040 con 0x7f18b0068d90 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 1 -- 172.21.15.116:0/727887694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0015630 con 0x7f18b0068d90 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b80ef640 5 monclient: authenticate success, global_id 4356 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b80ef640 10 monclient: _renew_subs 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b80ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b011dc80 con 0x7f18b0068d90 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18adffb640 10 monclient: dump: 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18affff640 4 set_mon_vals no callback set 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b80ef640 10 monclient: _renew_subs 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b80ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.305+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b011c630 con 0x7f18b0068d90 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18adffb640 1 -- 172.21.15.116:0/727887694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f18a00159a0 con 0x7f18b0068d90 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18adffb640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f189c0384c0 0x7f189c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18adffb640 1 -- 172.21.15.116:0/727887694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f18a004bb60 con 0x7f18b0068d90 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18b80ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18b80ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18b80ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b011e360 con 0x7f18b0068d90 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18b568a640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f189c0384c0 0x7f189c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.306+0000 7f18b568a640 10 monclient: get_auth_request con 0x7f189c0384c0 auth_method 0 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.309+0000 7f18b568a640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f189c0384c0 0x7f189c03a970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f18b0064050 tx=0x7f18a4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.311+0000 7f18adffb640 1 -- 172.21.15.116:0/727887694 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f18a001d590 con 0x7f18b0068d90 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.311+0000 7f18adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.311+0000 7f18adffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:34.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.461+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18b004bf60 con 0x7f189c0384c0 2024-09-18T12:25:34.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.464+0000 7f18adffb640 1 -- 172.21.15.116:0/727887694 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41049 (secure 0 0 0) 0x7f18b004bf60 con 0x7f189c0384c0 2024-09-18T12:25:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:34.466 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:25:34.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f189c0384c0 msgr2=0x7f189c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f189c0384c0 0x7f189c03a970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f18b0064050 tx=0x7f18a4032000 comp rx=0 tx=0).stop 2024-09-18T12:25:34.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 10 monclient: shutdown 2024-09-18T12:25:34.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:34.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 msgr2=0x7f18b009a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 0x7f18b009a8f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f18a0007e80 tx=0x7f18a0009330 comp rx=0 tx=0).stop 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 shutdown_connections 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f189c0384c0 0x7f189c03a970 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18b01318f0 0x7f18b01310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18b009ea50 0x7f18b0130b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 --2- 172.21.15.116:0/727887694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b0068d90 0x7f18b009a8f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 >> 172.21.15.116:0/727887694 conn(0x7f18b0067110 msgr2=0x7f18b009ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 shutdown_connections 2024-09-18T12:25:34.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.465+0000 7f18b80ef640 1 -- 172.21.15.116:0/727887694 wait complete. 2024-09-18T12:25:34.477 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-18T12:25:32.562663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786384,"kb_used_data":1960,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444656,"statfs":{"total":579820584960,"available":577991327744,"internally_reserved":0,"allocated":2007040,"data_stored":1352350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.458062"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106005+0000","last_change":"2024-09-18T12:25:28.106207+0000","last_active":"2024-09-18T12:25:28.106005+0000","last_peered":"2024-09-18T12:25:28.106005+0000","last_clean":"2024-09-18T12:25:28.106005+0000","last_became_active":"2024-09-18T12:25:25.932573+0000","last_became_peered":"2024-09-18T12:25:25.932573+0000","last_unstale":"2024-09-18T12:25:28.106005+0000","last_undegraded":"2024-09-18T12:25:28.106005+0000","last_fullsized":"2024-09-18T12:25:28.106005+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:11:31.670093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.114271+0000","last_change":"2024-09-18T12:25:28.114567+0000","last_active":"2024-09-18T12:25:28.114271+0000","last_peered":"2024-09-18T12:25:28.114271+0000","last_clean":"2024-09-18T12:25:28.114271+0000","last_became_active":"2024-09-18T12:25:25.938144+0000","last_became_peered":"2024-09-18T12:25:25.938144+0000","last_unstale":"2024-09-18T12:25:28.114271+0000","last_undegraded":"2024-09-18T12:25:28.114271+0000","last_fullsized":"2024-09-18T12:25:28.114271+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:02:14.591426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106056+0000","last_change":"2024-09-18T12:25:28.106209+0000","last_active":"2024-09-18T12:25:28.106056+0000","last_peered":"2024-09-18T12:25:28.106056+0000","last_clean":"2024-09-18T12:25:28.106056+0000","last_became_active":"2024-09-18T12:25:26.082919+0000","last_became_peered":"2024-09-18T12:25:26.082919+0000","last_unstale":"2024-09-18T12:25:28.106056+0000","last_undegraded":"2024-09-18T12:25:28.106056+0000","last_fullsized":"2024-09-18T12:25:28.106056+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:55:28.708584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033820399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.114218+0000","last_change":"2024-09-18T12:25:28.114468+0000","last_active":"2024-09-18T12:25:28.114218+0000","last_peered":"2024-09-18T12:25:28.114218+0000","last_clean":"2024-09-18T12:25:28.114218+0000","last_became_active":"2024-09-18T12:25:26.082963+0000","last_became_peered":"2024-09-18T12:25:26.082963+0000","last_unstale":"2024-09-18T12:25:28.114218+0000","last_undegraded":"2024-09-18T12:25:28.114218+0000","last_fullsized":"2024-09-18T12:25:28.114218+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:44:20.609265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.112853+0000","last_change":"2024-09-18T12:25:28.112853+0000","last_active":"2024-09-18T12:25:28.112853+0000","last_peered":"2024-09-18T12:25:28.112853+0000","last_clean":"2024-09-18T12:25:28.112853+0000","last_became_active":"2024-09-18T12:25:25.932570+0000","last_became_peered":"2024-09-18T12:25:25.932570+0000","last_unstale":"2024-09-18T12:25:28.112853+0000","last_undegraded":"2024-09-18T12:25:28.112853+0000","last_fullsized":"2024-09-18T12:25:28.112853+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:03.513828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.742178+0000","last_change":"2024-09-18T12:25:28.742381+0000","last_active":"2024-09-18T12:25:28.742178+0000","last_peered":"2024-09-18T12:25:28.742178+0000","last_clean":"2024-09-18T12:25:28.742178+0000","last_became_active":"2024-09-18T12:25:25.930894+0000","last_became_peered":"2024-09-18T12:25:25.930894+0000","last_unstale":"2024-09-18T12:25:28.742178+0000","last_undegraded":"2024-09-18T12:25:28.742178+0000","last_fullsized":"2024-09-18T12:25:28.742178+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:09:49.928338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.106258+0000","last_change":"2024-09-18T12:25:28.106419+0000","last_active":"2024-09-18T12:25:28.106258+0000","last_peered":"2024-09-18T12:25:28.106258+0000","last_clean":"2024-09-18T12:25:28.106258+0000","last_became_active":"2024-09-18T12:25:25.930456+0000","last_became_peered":"2024-09-18T12:25:25.930456+0000","last_unstale":"2024-09-18T12:25:28.106258+0000","last_undegraded":"2024-09-18T12:25:28.106258+0000","last_fullsized":"2024-09-18T12:25:28.106258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:26:39.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.112214+0000","last_change":"2024-09-18T12:25:28.112432+0000","last_active":"2024-09-18T12:25:28.112214+0000","last_peered":"2024-09-18T12:25:28.112214+0000","last_clean":"2024-09-18T12:25:28.112214+0000","last_became_active":"2024-09-18T12:25:25.932281+0000","last_became_peered":"2024-09-18T12:25:25.932281+0000","last_unstale":"2024-09-18T12:25:28.112214+0000","last_undegraded":"2024-09-18T12:25:28.112214+0000","last_fullsized":"2024-09-18T12:25:28.112214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:40.111669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":102,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T12:25:27.101718+0000","last_change":"2024-09-18T12:25:24.722024+0000","last_active":"2024-09-18T12:25:27.101718+0000","last_peered":"2024-09-18T12:25:27.101718+0000","last_clean":"2024-09-18T12:25:27.101718+0000","last_became_active":"2024-09-18T12:25:24.721108+0000","last_became_peered":"2024-09-18T12:25:24.721108+0000","last_unstale":"2024-09-18T12:25:27.101718+0000","last_undegraded":"2024-09-18T12:25:27.101718+0000","last_fullsized":"2024-09-18T12:25:27.101718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:41:42.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":167936,"data_stored":52371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2640000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1210000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2839999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2320000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":163840,"data_stored":52352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298172,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073668,"statfs":{"total":96636764160,"available":96331436032,"internally_reserved":0,"allocated":696320,"data_stored":581280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":139264,"data_stored":42524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":167936,"data_stored":52371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298140,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073700,"statfs":{"total":96636764160,"available":96331468800,"internally_reserved":0,"allocated":671744,"data_stored":571452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:25:34.478 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:25:34.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.569+0000 7f3a2d60f640 10 monclient: get_monmap_and_config 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.569+0000 7f3a2d60f640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.569+0000 7f3a2d60f640 10 monclient: monmap: 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:34.570683+0000 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:34.570683+0000 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:34.577 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.569+0000 7f3a2d60f640 1 Processor -- start 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.569+0000 7f3a2d60f640 1 -- start start 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.569+0000 7f3a2d60f640 10 monclient: init 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a28131980 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a2809fda0 0x7f3a2809e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2809fda0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a2809ef00 0x7f3a28138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2809ef00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:34.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a280a1180 con 0x7f3a28131980 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a280a1460 con 0x7f3a2809fda0 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a28132460 con 0x7f3a2809ef00 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a2d60f640 10 monclient(hunting): authenticate will time out at 761.892089s 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a2809fda0 0x7f3a2809e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a2809ef00 0x7f3a28138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a2809fda0 0x7f3a2809e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52274/0 (socket says 172.21.15.116:52274) 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 1 -- 172.21.15.116:0/2127785431 learned_addr learned my addr 172.21.15.116:0/2127785431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 10 monclient(hunting): get_auth_request con 0x7f3a2809fda0 auth_method 0 2024-09-18T12:25:34.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): get_auth_request con 0x7f3a2809ef00 auth_method 0 2024-09-18T12:25:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-09-18T12:25:34.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 1 -- 172.21.15.116:0/2127785431 >> v2:172.21.15.116:3300/0 conn(0x7f3a2809fda0 msgr2=0x7f3a2809e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 1 --2- 172.21.15.116:0/2127785431 >> v2:172.21.15.116:3300/0 conn(0x7f3a2809fda0 0x7f3a2809e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 1 -- 172.21.15.116:0/2127785431 >> v1:172.21.15.144:6789/0 conn(0x7f3a28131980 legacy=0x7f3a28131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 1 monclient: found mon.noname-b 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a277fe640 1 -- 172.21.15.116:0/2127785431 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a28132620 con 0x7f3a2809ef00 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.570+0000 7f3a267fc640 1 --2- 172.21.15.116:0/2127785431 >> v2:172.21.15.116:3300/0 conn(0x7f3a2809fda0 0x7f3a2809e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a277fe640 1 -- 172.21.15.116:0/2127785431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a2809ef00 msgr2=0x7f3a28138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a277fe640 1 --2- 172.21.15.116:0/2127785431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a2809ef00 0x7f3a28138cc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009e60 tx=0x7f3a20009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c8e2744df3204aa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 1 -- 172.21.15.116:0/2127785431 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20007760 con 0x7f3a2809ef00 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: dump: 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 1 -- 172.21.15.116:0/2127785431 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2002d050 con 0x7f3a2809ef00 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a2d60f640 5 monclient: authenticate success, global_id 4331 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a2d60f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a2d60f640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 1 -- 172.21.15.116:0/2127785431 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20007b30 con 0x7f3a2809ef00 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a2d60f640 4 set_mon_vals no callback set 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.573+0000 7f3a25ffb640 10 monclient: dump: 2024-09-18T12:25:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.585 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.585 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.585 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.585 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.585 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.586 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 10 monclient: shutdown 2024-09-18T12:25:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 -- 172.21.15.116:0/2127785431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a2809ef00 msgr2=0x7f3a28138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/2127785431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a2809ef00 0x7f3a28138cc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009e60 tx=0x7f3a20009780 comp rx=0 tx=0).stop 2024-09-18T12:25:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 -- 172.21.15.116:0/2127785431 shutdown_connections 2024-09-18T12:25:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/2127785431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a2809ef00 0x7f3a28138cc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/2127785431 >> v2:172.21.15.116:3300/0 conn(0x7f3a2809fda0 0x7f3a2809e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 -- 172.21.15.116:0/2127785431 >> 172.21.15.116:0/2127785431 conn(0x7f3a28067bd0 msgr2=0x7f3a280a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 -- 172.21.15.116:0/2127785431 shutdown_connections 2024-09-18T12:25:34.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 1 -- 172.21.15.116:0/2127785431 wait complete. 2024-09-18T12:25:34.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.574+0000 7f3a2d60f640 10 monclient: monmap: 2024-09-18T12:25:34.589 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.589 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:34.589 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:34.589 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:34.590 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 Processor -- start 2024-09-18T12:25:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 -- start start 2024-09-18T12:25:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient: init 2024-09-18T12:25:34.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a2809ef00 0x7f3a28138070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2809ef00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a2809fda0 0x7f3a28133dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2809fda0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 0x7f3a28134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): picked mon.noname-c con 0x7f3a28131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2809bf10 con 0x7f3a2809ef00 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a280a4160 con 0x7f3a2809fda0 2024-09-18T12:25:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a281379e0 con 0x7f3a28131980 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a2d60f640 10 monclient(hunting): authenticate will time out at 761.897561s 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 0x7f3a28134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a2809ef00 0x7f3a28138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 0x7f3a28134310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54212/0 (socket says 172.21.15.116:54212) 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 -- 172.21.15.116:0/1233038151 learned_addr learned my addr 172.21.15.116:0/1233038151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a2809ef00 0x7f3a28138070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52290/0 (socket says 172.21.15.116:52290) 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): get_auth_request con 0x7f3a28131980 auth_method 0 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 10 monclient(hunting): get_auth_request con 0x7f3a2809ef00 auth_method 0 2024-09-18T12:25:34.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-18T12:25:34.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a267fc640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a2809fda0 0x7f3a28133dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 -- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a2809fda0 msgr2=0x7f3a28133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a2809fda0 0x7f3a28133dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 -- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a2809ef00 msgr2=0x7f3a28138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a2809ef00 0x7f3a28138070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 monclient: found mon.noname-c 2024-09-18T12:25:34.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a277fe640 1 -- 172.21.15.116:0/1233038151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c009450 con 0x7f3a28131980 2024-09-18T12:25:34.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.575+0000 7f3a26ffd640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a2809ef00 0x7f3a28138070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:34.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a277fe640 1 -- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 msgr2=0x7f3a28134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a277fe640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 0x7f3a28134310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009ec0 tx=0x7f3a20037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 1 -- 172.21.15.116:0/1233038151 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20031b90 con 0x7f3a28131980 2024-09-18T12:25:34.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:34.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: dump: 2024-09-18T12:25:34.603 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.603 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.603 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.604 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.604 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.604 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.604 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.604 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.605 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.605 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a2d60f640 5 monclient: authenticate success, global_id 4334 2024-09-18T12:25:34.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a2d60f640 10 monclient: _renew_subs 2024-09-18T12:25:34.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 1 -- 172.21.15.116:0/1233038151 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a20040070 con 0x7f3a28131980 2024-09-18T12:25:34.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a2d60f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a20009a20 con 0x7f3a28131980 2024-09-18T12:25:34.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 1 -- 172.21.15.116:0/1233038151 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20045430 con 0x7f3a28131980 2024-09-18T12:25:34.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:34.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a0effd640 10 monclient: dump: 2024-09-18T12:25:34.607 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.608 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.608 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.608 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.608 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.609 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.609 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.609 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.609 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.610 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a2d60f640 10 monclient: _renew_subs 2024-09-18T12:25:34.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a2d60f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a281373b0 con 0x7f3a28131980 2024-09-18T12:25:34.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.578+0000 7f3a257fa640 4 set_mon_vals no callback set 2024-09-18T12:25:34.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a0effd640 1 -- 172.21.15.116:0/1233038151 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3a20042030 con 0x7f3a28131980 2024-09-18T12:25:34.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a0effd640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a14038490 0x7f3a1403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a2d60f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a2d60f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a2d60f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a28137790 con 0x7f3a28131980 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a0effd640 1 -- 172.21.15.116:0/1233038151 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f3a2006cea0 con 0x7f3a28131980 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a26ffd640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a14038490 0x7f3a1403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.579+0000 7f3a26ffd640 10 monclient: get_auth_request con 0x7f3a14038490 auth_method 0 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.582+0000 7f3a26ffd640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a14038490 0x7f3a1403a940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a18007fd0 tx=0x7f3a18007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.583+0000 7f3a0effd640 1 -- 172.21.15.116:0/1233038151 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3a20030b80 con 0x7f3a28131980 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.583+0000 7f3a0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.583+0000 7f3a0effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:34.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.737+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a2804bf60 con 0x7f3a14038490 2024-09-18T12:25:34.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.740+0000 7f3a0effd640 1 -- 172.21.15.116:0/1233038151 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40768 (secure 0 0 0) 0x7f3a2804bf60 con 0x7f3a14038490 2024-09-18T12:25:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:25:34.742 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:25:34.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a14038490 msgr2=0x7f3a1403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a14038490 0x7f3a1403a940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a18007fd0 tx=0x7f3a18007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:34.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 10 monclient: shutdown 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 msgr2=0x7f3a28134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 0x7f3a28134310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009ec0 tx=0x7f3a20037000 comp rx=0 tx=0).stop 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 shutdown_connections 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3a14038490 0x7f3a1403a940 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3a28131980 0x7f3a28134310 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3a2809fda0 0x7f3a28133dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 --2- 172.21.15.116:0/1233038151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a2809ef00 0x7f3a28138070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 >> 172.21.15.116:0/1233038151 conn(0x7f3a28067bd0 msgr2=0x7f3a2809e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 shutdown_connections 2024-09-18T12:25:34.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.741+0000 7f3a2d60f640 1 -- 172.21.15.116:0/1233038151 wait complete. 2024-09-18T12:25:34.754 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-18T12:25:34.563005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786216,"kb_used_data":2080,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444824,"statfs":{"total":579820584960,"available":577991499776,"internally_reserved":0,"allocated":2129920,"data_stored":1399546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.458404"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.513079+0000","last_change":"2024-09-18T12:25:28.106207+0000","last_active":"2024-09-18T12:25:31.513079+0000","last_peered":"2024-09-18T12:25:31.513079+0000","last_clean":"2024-09-18T12:25:31.513079+0000","last_became_active":"2024-09-18T12:25:25.932573+0000","last_became_peered":"2024-09-18T12:25:25.932573+0000","last_unstale":"2024-09-18T12:25:31.513079+0000","last_undegraded":"2024-09-18T12:25:31.513079+0000","last_fullsized":"2024-09-18T12:25:31.513079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:11:31.670093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.569797+0000","last_change":"2024-09-18T12:25:28.114567+0000","last_active":"2024-09-18T12:25:31.569797+0000","last_peered":"2024-09-18T12:25:31.569797+0000","last_clean":"2024-09-18T12:25:31.569797+0000","last_became_active":"2024-09-18T12:25:25.938144+0000","last_became_peered":"2024-09-18T12:25:25.938144+0000","last_unstale":"2024-09-18T12:25:31.569797+0000","last_undegraded":"2024-09-18T12:25:31.569797+0000","last_fullsized":"2024-09-18T12:25:31.569797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:02:14.591426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.513026+0000","last_change":"2024-09-18T12:25:28.106209+0000","last_active":"2024-09-18T12:25:31.513026+0000","last_peered":"2024-09-18T12:25:31.513026+0000","last_clean":"2024-09-18T12:25:31.513026+0000","last_became_active":"2024-09-18T12:25:26.082919+0000","last_became_peered":"2024-09-18T12:25:26.082919+0000","last_unstale":"2024-09-18T12:25:31.513026+0000","last_undegraded":"2024-09-18T12:25:31.513026+0000","last_fullsized":"2024-09-18T12:25:31.513026+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:55:28.708584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033820399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.569782+0000","last_change":"2024-09-18T12:25:28.114468+0000","last_active":"2024-09-18T12:25:31.569782+0000","last_peered":"2024-09-18T12:25:31.569782+0000","last_clean":"2024-09-18T12:25:31.569782+0000","last_became_active":"2024-09-18T12:25:26.082963+0000","last_became_peered":"2024-09-18T12:25:26.082963+0000","last_unstale":"2024-09-18T12:25:31.569782+0000","last_undegraded":"2024-09-18T12:25:31.569782+0000","last_fullsized":"2024-09-18T12:25:31.569782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:44:20.609265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.512566+0000","last_change":"2024-09-18T12:25:28.112853+0000","last_active":"2024-09-18T12:25:31.512566+0000","last_peered":"2024-09-18T12:25:31.512566+0000","last_clean":"2024-09-18T12:25:31.512566+0000","last_became_active":"2024-09-18T12:25:25.932570+0000","last_became_peered":"2024-09-18T12:25:25.932570+0000","last_unstale":"2024-09-18T12:25:31.512566+0000","last_undegraded":"2024-09-18T12:25:31.512566+0000","last_fullsized":"2024-09-18T12:25:31.512566+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:03.513828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.753531+0000","last_change":"2024-09-18T12:25:28.742381+0000","last_active":"2024-09-18T12:25:31.753531+0000","last_peered":"2024-09-18T12:25:31.753531+0000","last_clean":"2024-09-18T12:25:31.753531+0000","last_became_active":"2024-09-18T12:25:25.930894+0000","last_became_peered":"2024-09-18T12:25:25.930894+0000","last_unstale":"2024-09-18T12:25:31.753531+0000","last_undegraded":"2024-09-18T12:25:31.753531+0000","last_fullsized":"2024-09-18T12:25:31.753531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:09:49.928338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.513115+0000","last_change":"2024-09-18T12:25:28.106419+0000","last_active":"2024-09-18T12:25:31.513115+0000","last_peered":"2024-09-18T12:25:31.513115+0000","last_clean":"2024-09-18T12:25:31.513115+0000","last_became_active":"2024-09-18T12:25:25.930456+0000","last_became_peered":"2024-09-18T12:25:25.930456+0000","last_unstale":"2024-09-18T12:25:31.513115+0000","last_undegraded":"2024-09-18T12:25:31.513115+0000","last_fullsized":"2024-09-18T12:25:31.513115+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:26:39.281521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T12:25:31.512572+0000","last_change":"2024-09-18T12:25:28.112432+0000","last_active":"2024-09-18T12:25:31.512572+0000","last_peered":"2024-09-18T12:25:31.512572+0000","last_clean":"2024-09-18T12:25:31.512572+0000","last_became_active":"2024-09-18T12:25:25.932281+0000","last_became_peered":"2024-09-18T12:25:25.932281+0000","last_unstale":"2024-09-18T12:25:31.512572+0000","last_undegraded":"2024-09-18T12:25:31.512572+0000","last_fullsized":"2024-09-18T12:25:31.512572+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:40.111669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":103,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T12:25:28.105889+0000","last_change":"2024-09-18T12:25:24.722024+0000","last_active":"2024-09-18T12:25:28.105889+0000","last_peered":"2024-09-18T12:25:28.105889+0000","last_clean":"2024-09-18T12:25:28.105889+0000","last_became_active":"2024-09-18T12:25:24.721108+0000","last_became_peered":"2024-09-18T12:25:24.721108+0000","last_unstale":"2024-09-18T12:25:28.105889+0000","last_undegraded":"2024-09-18T12:25:28.105889+0000","last_fullsized":"2024-09-18T12:25:28.105889+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:41:42.913931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":180224,"data_stored":56961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":176128,"data_stored":56942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298048,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073792,"statfs":{"total":96636764160,"available":96331563008,"internally_reserved":0,"allocated":708608,"data_stored":585870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297528,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074312,"statfs":{"total":96636764160,"available":96332095488,"internally_reserved":0,"allocated":176128,"data_stored":56942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":180224,"data_stored":56961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298048,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073792,"statfs":{"total":96636764160,"available":96331563008,"internally_reserved":0,"allocated":708608,"data_stored":585870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:25:34.755 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-18T12:25:34.755 DEBUG:tasks.mon_thrash:Yielding 2024-09-18T12:25:34.755 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T12:25:34.769 INFO:tasks.workunit:Pulling workunits from ref 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T12:25:34.769 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T12:25:34.769 DEBUG:teuthology.orchestra.run.smithi144:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T12:25:34.770 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-18T12:25:34.770 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726662334, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-18T12:25:34.770 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:25:34.770 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:25:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:25:34.789 INFO:teuthology.orchestra.run.smithi144.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T12:25:34.790 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T12:25:34.848 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T12:25:34.849 DEBUG:teuthology.orchestra.run.smithi144:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient: monmap: 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:34.861837+0000 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:34.861837+0000 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:34.863 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 1 Processor -- start 2024-09-18T12:25:34.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 1 -- start start 2024-09-18T12:25:34.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient: init 2024-09-18T12:25:34.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 0x7f2f0c098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient(hunting): picked mon.noname-b con 0x7f2f0c0981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f0c131da0 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f0c132170 0x7f2f0c067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0c132170 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c098ad0 con 0x7f2f0c131da0 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c131bd0 con 0x7f2f0c132170 2024-09-18T12:25:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c09ca60 con 0x7f2f0c0981c0 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1239d640 10 monclient(hunting): authenticate will time out at 762.183477s 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1093a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f0c132170 0x7f2f0c067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 0x7f2f0c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1093a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f0c132170 0x7f2f0c067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52292/0 (socket says 172.21.15.116:52292) 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1093a640 1 -- 172.21.15.116:0/1593878495 learned_addr learned my addr 172.21.15.116:0/1593878495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1093a640 10 monclient(hunting): get_auth_request con 0x7f2f0c132170 auth_method 0 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 1 --2- 172.21.15.116:0/1593878495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 0x7f2f0c098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54226/0 (socket says 172.21.15.116:54226) 2024-09-18T12:25:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1093a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f1093a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): get_auth_request con 0x7f2f0c0981c0 auth_method 0 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 1 -- 172.21.15.116:0/1593878495 >> v2:172.21.15.116:3300/0 conn(0x7f2f0c132170 msgr2=0x7f2f0c067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 1 --2- 172.21.15.116:0/1593878495 >> v2:172.21.15.116:3300/0 conn(0x7f2f0c132170 0x7f2f0c067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 1 -- 172.21.15.116:0/1593878495 >> v1:172.21.15.116:6789/0 conn(0x7f2f0c131da0 legacy=0x7f2f0c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 1 monclient: found mon.noname-b 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.861+0000 7f2f0bfff640 1 -- 172.21.15.116:0/1593878495 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c09cca0 con 0x7f2f0c0981c0 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0bfff640 1 -- 172.21.15.116:0/1593878495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 msgr2=0x7f2f0c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0bfff640 1 --2- 172.21.15.116:0/1593878495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 0x7f2f0c098590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009850 tx=0x7f2efc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e52629c3c46ada32 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 1 -- 172.21.15.116:0/1593878495 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc02add0 con 0x7f2f0c0981c0 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: dump: 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 1 -- 172.21.15.116:0/1593878495 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc032440 con 0x7f2f0c0981c0 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f1239d640 5 monclient: authenticate success, global_id 4337 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f1239d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f1239d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 1 -- 172.21.15.116:0/1593878495 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc032820 con 0x7f2f0c0981c0 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f1239d640 4 set_mon_vals no callback set 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.864+0000 7f2f0affd640 10 monclient: dump: 2024-09-18T12:25:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 10 monclient: shutdown 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 -- 172.21.15.116:0/1593878495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 msgr2=0x7f2f0c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 --2- 172.21.15.116:0/1593878495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 0x7f2f0c098590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009850 tx=0x7f2efc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 -- 172.21.15.116:0/1593878495 shutdown_connections 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 --2- 172.21.15.116:0/1593878495 >> v2:172.21.15.116:3300/0 conn(0x7f2f0c132170 0x7f2f0c067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 --2- 172.21.15.116:0/1593878495 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c0981c0 0x7f2f0c098590 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 -- 172.21.15.116:0/1593878495 >> 172.21.15.116:0/1593878495 conn(0x7f2f0c0a1e30 msgr2=0x7f2f0c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 -- 172.21.15.116:0/1593878495 shutdown_connections 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 1 -- 172.21.15.116:0/1593878495 wait complete. 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 10 monclient: build_initial_monmap 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.865+0000 7f2f1239d640 10 monclient: monmap: 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 Processor -- start 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 -- start start 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient: init 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c0981c0 0x7f2f0c1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): picked mon.noname-b con 0x7f2f0c0981c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 0x7f2f0c141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): picked mon.noname-c con 0x7f2f0c131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f0c132170 0x7f2f0c11aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0c132170 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c098cb0 con 0x7f2f0c132170 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0a0ae0 con 0x7f2f0c0981c0 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c11eb10 con 0x7f2f0c131da0 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1239d640 10 monclient(hunting): authenticate will time out at 762.188922s 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f0b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 0x7f2f0c141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.866+0000 7f2f1093a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f0c132170 0x7f2f0c11aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f0c132170 0x7f2f0c11aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52300/0 (socket says 172.21.15.116:52300) 2024-09-18T12:25:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 0x7f2f0c141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54238/0 (socket says 172.21.15.116:54238) 2024-09-18T12:25:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 1 -- 172.21.15.116:0/3049705588 learned_addr learned my addr 172.21.15.116:0/3049705588 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 10 monclient(hunting): get_auth_request con 0x7f2f0c132170 auth_method 0 2024-09-18T12:25:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): get_auth_request con 0x7f2f0c131da0 auth_method 0 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 1 -- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c0981c0 msgr2=0x7f2f0c1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c0981c0 0x7f2f0c1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 1 -- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f0c132170 msgr2=0x7f2f0c11aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f0c132170 0x7f2f0c11aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f0b7fe640 1 -- 172.21.15.116:0/3049705588 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc009390 con 0x7f2f0c131da0 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.867+0000 7f2f1093a640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f0c132170 0x7f2f0c11aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f0b7fe640 1 -- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 msgr2=0x7f2f0c141e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f0b7fe640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 0x7f2f0c141e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c510 tx=0x7f2ef4007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 1 -- 172.21.15.116:0/3049705588 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40117a0 con 0x7f2f0c131da0 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: dump: 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: _finish_auth 0 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 1 -- 172.21.15.116:0/3049705588 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4011900 con 0x7f2f0c131da0 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f1239d640 5 monclient: authenticate success, global_id 4340 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f1239d640 10 monclient: _renew_subs 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f1239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c11d5f0 con 0x7f2f0c131da0 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 1 -- 172.21.15.116:0/3049705588 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4011ca0 con 0x7f2f0c131da0 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f03fff640 10 monclient: dump: 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f0a7fc640 4 set_mon_vals no callback set 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f1239d640 10 monclient: _renew_subs 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f1239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.869+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c11f6e0 con 0x7f2f0c131da0 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.870+0000 7f2f03fff640 1 -- 172.21.15.116:0/3049705588 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2ef4012630 con 0x7f2f0c131da0 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.870+0000 7f2f1239d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.870+0000 7f2f1239d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.870+0000 7f2f1239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.870+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c04bf60 con 0x7f2f0c131da0 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.870+0000 7f2f03fff640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2eec038490 0x7f2eec03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.870+0000 7f2f03fff640 1 -- 172.21.15.116:0/3049705588 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f2ef404b310 con 0x7f2f0c131da0 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.871+0000 7f2f0bfff640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2eec038490 0x7f2eec03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.871+0000 7f2f0bfff640 10 monclient: get_auth_request con 0x7f2eec038490 auth_method 0 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.873+0000 7f2f0bfff640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2eec038490 0x7f2eec03a940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2efc034000 tx=0x7f2efc032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.876+0000 7f2f03fff640 1 -- 172.21.15.116:0/3049705588 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2ef40128e0 con 0x7f2f0c131da0 2024-09-18T12:25:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.876+0000 7f2f03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:34.876+0000 7f2f03fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:34.906 INFO:tasks.workunit:timeout=3h 2024-09-18T12:25:34.906 INFO:tasks.workunit:cleanup=True 2024-09-18T12:25:34.907 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T12:25:34.964 INFO:tasks.workunit.client.0.smithi144.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T12:25:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.087+0000 7f2f1239d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:25:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.087+0000 7f2f1239d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.087+0000 7f2f1239d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.087+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2f0c067860 con 0x7f2f0c131da0 2024-09-18T12:25:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.087+0000 7f2f03fff640 1 -- 172.21.15.116:0/3049705588 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2ef40102e0 con 0x7f2f0c131da0 2024-09-18T12:25:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.087+0000 7f2f03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.087+0000 7f2f03fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:35.090 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:25:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2eec038490 msgr2=0x7f2eec03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2eec038490 0x7f2eec03a940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2efc034000 tx=0x7f2efc032000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 10 monclient: shutdown 2024-09-18T12:25:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 msgr2=0x7f2f0c141e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 0x7f2f0c141e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c510 tx=0x7f2ef4007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 shutdown_connections 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2eec038490 0x7f2eec03a940 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f0c132170 0x7f2f0c11aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f0c131da0 0x7f2f0c141e30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 --2- 172.21.15.116:0/3049705588 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f0c0981c0 0x7f2f0c1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 >> 172.21.15.116:0/3049705588 conn(0x7f2f0c0a1e30 msgr2=0x7f2f0c096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 shutdown_connections 2024-09-18T12:25:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.089+0000 7f2f1239d640 1 -- 172.21.15.116:0/3049705588 wait complete. 2024-09-18T12:25:35.102 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:25:35.102 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:25:35.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:25:35.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:25:35.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.194+0000 7f4347154640 10 monclient: get_monmap_and_config 2024-09-18T12:25:35.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.194+0000 7f4347154640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.194+0000 7f4347154640 10 monclient: monmap: 2024-09-18T12:25:35.196 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.196 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:35.195801+0000 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:35.195801+0000 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:35.197 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.194+0000 7f4347154640 1 Processor -- start 2024-09-18T12:25:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.194+0000 7f4347154640 1 -- start start 2024-09-18T12:25:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.194+0000 7f4347154640 10 monclient: init 2024-09-18T12:25:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T12:25:35.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4340131ab0 0x7f4340131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient(hunting): picked mon.noname-a con 0x7f4340131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient(hunting): picked mon.noname-b con 0x7f434009e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f434009f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434009faf0 con 0x7f434009f220 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434009fd30 con 0x7f4340131ab0 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43400a1110 con 0x7f434009e910 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4347154640 10 monclient(hunting): authenticate will time out at 762.517447s 2024-09-18T12:25:35.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4344ef0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4340131ab0 0x7f4340131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4344ef0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4340131ab0 0x7f4340131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52308/0 (socket says 172.21.15.116:52308) 2024-09-18T12:25:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4344ef0640 1 -- 172.21.15.116:0/2669747950 learned_addr learned my addr 172.21.15.116:0/2669747950 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4344ef0640 10 monclient(hunting): get_auth_request con 0x7f4340131ab0 auth_method 0 2024-09-18T12:25:35.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4344ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4344ef0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f4344ef0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): get_auth_request con 0x7f434009e910 auth_method 0 2024-09-18T12:25:35.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2024-09-18T12:25:35.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 1 -- 172.21.15.116:0/2669747950 >> v2:172.21.15.116:3300/0 conn(0x7f4340131ab0 msgr2=0x7f4340131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 1 --2- 172.21.15.116:0/2669747950 >> v2:172.21.15.116:3300/0 conn(0x7f4340131ab0 0x7f4340131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 1 -- 172.21.15.116:0/2669747950 >> v1:172.21.15.116:6789/0 conn(0x7f434009f220 legacy=0x7f4340068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 1 monclient: found mon.noname-b 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.195+0000 7f433ffff640 1 -- 172.21.15.116:0/2669747950 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400a13f0 con 0x7f434009e910 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433ffff640 1 -- 172.21.15.116:0/2669747950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 msgr2=0x7f434009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433ffff640 1 --2- 172.21.15.116:0/2669747950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4330009e60 tx=0x7f4330009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcac8cfc16096956 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 1 -- 172.21.15.116:0/2669747950 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330007760 con 0x7f434009e910 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: dump: 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.216 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 5 monclient: authenticate success, global_id 4343 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 1 -- 172.21.15.116:0/2669747950 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433002d050 con 0x7f434009e910 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 4 set_mon_vals no callback set 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 1 -- 172.21.15.116:0/2669747950 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330007b30 con 0x7f434009e910 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:35.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f433f7fe640 10 monclient: dump: 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 10 monclient: shutdown 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 -- 172.21.15.116:0/2669747950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 msgr2=0x7f434009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 --2- 172.21.15.116:0/2669747950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4330009e60 tx=0x7f4330009780 comp rx=0 tx=0).stop 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 -- 172.21.15.116:0/2669747950 shutdown_connections 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 --2- 172.21.15.116:0/2669747950 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009ece0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 --2- 172.21.15.116:0/2669747950 >> v2:172.21.15.116:3300/0 conn(0x7f4340131ab0 0x7f4340131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 -- 172.21.15.116:0/2669747950 >> 172.21.15.116:0/2669747950 conn(0x7f4340067d30 msgr2=0x7f43400a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 -- 172.21.15.116:0/2669747950 shutdown_connections 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 1 -- 172.21.15.116:0/2669747950 wait complete. 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.199+0000 7f4347154640 10 monclient: monmap: 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:35.219 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 1 Processor -- start 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 1 -- start start 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 10 monclient: init 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 10 monclient(hunting): picked mon.noname-c con 0x7f434009e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434009f220 0x7f4340130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 10 monclient(hunting): picked mon.noname-b con 0x7f434009f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131ab0 0x7f4340140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.200+0000 7f4347154640 10 monclient(hunting): picked mon.noname-a con 0x7f4340131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434009b640 con 0x7f4340131ab0 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434011c6a0 con 0x7f434009f220 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434009dd70 con 0x7f434009e910 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4347154640 10 monclient(hunting): authenticate will time out at 762.523035s 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54268/0 (socket says 172.21.15.116:54268) 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 -- 172.21.15.116:0/1246892816 learned_addr learned my addr 172.21.15.116:0/1246892816 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): get_auth_request con 0x7f434009e910 auth_method 0 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131ab0 0x7f4340140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 10 monclient(hunting): get_auth_request con 0x7f4340131ab0 auth_method 0 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f433ffff640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434009f220 0x7f4340130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f433ffff640 10 monclient(hunting): get_auth_request con 0x7f434009f220 auth_method 0 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f433ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f433ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f433ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434009f220 msgr2=0x7f4340130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f434009f220 0x7f4340130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131ab0 msgr2=0x7f4340140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131ab0 0x7f4340140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 monclient: found mon.noname-c 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f4344ef0640 1 -- 172.21.15.116:0/1246892816 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330009a20 con 0x7f434009e910 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.201+0000 7f43456f1640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131ab0 0x7f4340140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4344ef0640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 msgr2=0x7f434009a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4344ef0640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009a9f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f433800d870 tx=0x7f4338008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433800de60 con 0x7f434009e910 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: dump: 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.224 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4338009ac0 con 0x7f434009e910 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4347154640 5 monclient: authenticate success, global_id 4346 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338009e80 con 0x7f434009e910 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433cff9640 10 monclient: dump: 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4347154640 10 monclient: _renew_subs 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4347154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f433effd640 4 set_mon_vals no callback set 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434011dc30 con 0x7f434009e910 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4347154640 10 monclient: _renew_subs 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4347154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.203+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434011cd40 con 0x7f434009e910 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f4347154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f4347154640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 0x7f4340141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f4347154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434011e0b0 con 0x7f434011baf0 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4338014620 con 0x7f434009e910 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 0x7f4340141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 10 monclient: get_auth_request con 0x7f434011baf0 auth_method 0 2024-09-18T12:25:35.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.204+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.205+0000 7f433cff9640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43200384e0 0x7f432003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.205+0000 7f433ffff640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.205+0000 7f433ffff640 1 -- 172.21.15.116:0/1246892816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f434011e5f0 con 0x7f434011baf0 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.205+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f433804bdf0 con 0x7f434009e910 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.205+0000 7f43456f1640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43200384e0 0x7f432003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.205+0000 7f43456f1640 10 monclient: get_auth_request con 0x7f43200384e0 auth_method 0 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.207+0000 7f433ffff640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 msgr2=0x7f4340141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.207+0000 7f433ffff640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 0x7f4340141460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f434009bd80 tx=0x7f4330037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.207+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433003db30 con 0x7f434011baf0 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.207+0000 7f433cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.207+0000 7f433cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.207+0000 7f433cff9640 10 monclient: dump: 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.207+0000 7f43456f1640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43200384e0 0x7f432003a990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f434009af70 tx=0x7f4328007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.208+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f433003dc90 con 0x7f434011baf0 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.208+0000 7f433cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.208+0000 7f433cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.208+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 msgr2=0x7f4340141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.208+0000 7f433cff9640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 0x7f4340141460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f434009bd80 tx=0x7f4330037000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.208+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 msgr2=0x7f4340141460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:35.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.208+0000 7f433cff9640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f434011baf0 0x7f4340141460 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f434009bd80 tx=0x7f4330037000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f4347154640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:25:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f4347154640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 0x7f43401419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f4347154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4340063740 con 0x7f4340131050 2024-09-18T12:25:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 0x7f43401419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient: get_auth_request con 0x7f4340131050 auth_method 0 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.214+0000 7f433ffff640 1 -- 172.21.15.116:0/1246892816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f434004bf60 con 0x7f4340131050 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433ffff640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 msgr2=0x7f43401419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433ffff640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 0x7f43401419a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43300097c0 tx=0x7f433003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300319d0 con 0x7f4340131050 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 10 monclient: dump: 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f4330031b30 con 0x7f4340131050 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:35.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 msgr2=0x7f43401419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 0x7f43401419a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43300097c0 tx=0x7f433003f040 comp rx=0 tx=0).stop 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 msgr2=0x7f43401419a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f433cff9640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4340131050 0x7f43401419a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 4, 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:25:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 22, 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:25:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:25:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:25:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:25:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:25:35.246 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:25:35.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43200384e0 msgr2=0x7f432003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43200384e0 0x7f432003a990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f434009af70 tx=0x7f4328007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 10 monclient: shutdown 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 msgr2=0x7f434009a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 --2- 172.21.15.116:0/1246892816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f434009e910 0x7f434009a9f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f433800d870 tx=0x7f4338008040 comp rx=0 tx=0).stop 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f43456f1640 1 -- 172.21.15.116:0/1246892816 reap_dead start 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f43456f1640 1 -- 172.21.15.116:0/1246892816 reap_dead start 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 shutdown_connections 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 >> 172.21.15.116:0/1246892816 conn(0x7f4340067d30 msgr2=0x7f43401349f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 shutdown_connections 2024-09-18T12:25:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.217+0000 7f4347154640 1 -- 172.21.15.116:0/1246892816 wait complete. 2024-09-18T12:25:35.248 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:25:35.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:25:35.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:25:35.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.341+0000 7f7663a94640 10 monclient: get_monmap_and_config 2024-09-18T12:25:35.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.341+0000 7f7663a94640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.341+0000 7f7663a94640 10 monclient: monmap: 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:35.342399+0000 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:35.342399+0000 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:35.343 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 1 Processor -- start 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 1 -- start start 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient: init 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f765c06eb80 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f765c068a20 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f765c137ac0 0x7f765c139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient(hunting): picked mon.noname-c con 0x7f765c137ac0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765c069b80 con 0x7f765c06eb80 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765c069e00 con 0x7f765c068a20 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765c099f90 con 0x7f765c137ac0 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.342+0000 7f7663a94640 10 monclient(hunting): authenticate will time out at 762.663992s 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7661830640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f765c06eb80 0x7f765c06ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57196/0 (socket says 172.21.15.116:57196) 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7661830640 1 -- 172.21.15.116:0/1742950675 learned_addr learned my addr 172.21.15.116:0/1742950675 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 1 --2- 172.21.15.116:0/1742950675 >> v2:172.21.15.144:3300/0 conn(0x7f765c137ac0 0x7f765c139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): get_auth_request con 0x7f765c137ac0 auth_method 0 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 1 -- 172.21.15.116:0/1742950675 >> v1:172.21.15.144:6789/0 conn(0x7f765c068a20 legacy=0x7f765c09c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 1 -- 172.21.15.116:0/1742950675 >> v1:172.21.15.116:6789/0 conn(0x7f765c06eb80 legacy=0x7f765c06ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 1 monclient: found mon.noname-c 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:35.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.343+0000 7f7662031640 1 -- 172.21.15.116:0/1742950675 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c06f660 con 0x7f765c137ac0 2024-09-18T12:25:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f7662031640 1 -- 172.21.15.116:0/1742950675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c137ac0 msgr2=0x7f765c139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f7662031640 1 --2- 172.21.15.116:0/1742950675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c137ac0 0x7f765c139ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7654009e60 tx=0x7f7654009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=719f254848169d40 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 1 -- 172.21.15.116:0/1742950675 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654007840 con 0x7f765c137ac0 2024-09-18T12:25:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: dump: 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f7663a94640 5 monclient: authenticate success, global_id 4324 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f7663a94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 1 -- 172.21.15.116:0/1742950675 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765402d050 con 0x7f765c137ac0 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f7663a94640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 1 -- 172.21.15.116:0/1742950675 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654007bd0 con 0x7f765c137ac0 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f7663a94640 4 set_mon_vals no callback set 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.346+0000 7f766082e640 10 monclient: dump: 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.356 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 10 monclient: shutdown 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 1 -- 172.21.15.116:0/1742950675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c137ac0 msgr2=0x7f765c139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 1 --2- 172.21.15.116:0/1742950675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c137ac0 0x7f765c139ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7654009e60 tx=0x7f7654009780 comp rx=0 tx=0).stop 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 1 -- 172.21.15.116:0/1742950675 shutdown_connections 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 1 --2- 172.21.15.116:0/1742950675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c137ac0 0x7f765c139ea0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 1 -- 172.21.15.116:0/1742950675 >> 172.21.15.116:0/1742950675 conn(0x7f765c067cd0 msgr2=0x7f765c0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 1 -- 172.21.15.116:0/1742950675 shutdown_connections 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 1 -- 172.21.15.116:0/1742950675 wait complete. 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.347+0000 7f7663a94640 10 monclient: monmap: 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 Processor -- start 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 -- start start 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient: init 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 0x7f765c097840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): picked mon.noname-a con 0x7f765c068a20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c06eb80 0x7f765c099010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): picked mon.noname-b con 0x7f765c06eb80 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c137ac0 0x7f765c06c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): picked mon.noname-c con 0x7f765c137ac0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c098b40 con 0x7f765c068a20 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c09c930 con 0x7f765c06eb80 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c098160 con 0x7f765c137ac0 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7663a94640 10 monclient(hunting): authenticate will time out at 762.669773s 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 0x7f765c097840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 0x7f765c097840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52338/0 (socket says 172.21.15.116:52338) 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 1 -- 172.21.15.116:0/299469838 learned_addr learned my addr 172.21.15.116:0/299469838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 10 monclient(hunting): get_auth_request con 0x7f765c068a20 auth_method 0 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7662031640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c137ac0 0x7f765c06c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7662031640 10 monclient(hunting): get_auth_request con 0x7f765c137ac0 auth_method 0 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7662031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7662031640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7662031640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7661830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7662031640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.348+0000 7f7662031640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7662031640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f766102f640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c06eb80 0x7f765c099010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c137ac0 msgr2=0x7f765c06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c137ac0 0x7f765c06c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c06eb80 msgr2=0x7f765c099010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c06eb80 0x7f765c099010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 1 monclient: found mon.noname-a 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7661830640 1 -- 172.21.15.116:0/299469838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654009a20 con 0x7f765c068a20 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f766102f640 10 monclient: get_auth_request con 0x7f765c06eb80 auth_method 0 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f7662031640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c137ac0 0x7f765c06c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.349+0000 7f766102f640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f765c06eb80 0x7f765c099010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7661830640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 msgr2=0x7f765c097840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7661830640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 0x7f765c097840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f765800c5e0 tx=0x7f7658004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658005cb0 con 0x7f765c068a20 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: dump: 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.363 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658005e30 con 0x7f765c068a20 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7663a94640 5 monclient: authenticate success, global_id 4383 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7663a94640 10 monclient: _renew_subs 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7663a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c097e80 con 0x7f765c068a20 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658007340 con 0x7f765c068a20 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7651ffb640 10 monclient: dump: 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7663a94640 10 monclient: _renew_subs 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7663a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7653fff640 4 set_mon_vals no callback set 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.351+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c096f20 con 0x7f765c068a20 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f7663a94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f7663a94640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 0x7f765c06cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f7663a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c065e00 con 0x7f765c06caf0 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f765801b400 con 0x7f765c068a20 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 0x7f765c06cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient: get_auth_request con 0x7f765c06caf0 auth_method 0 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f7651ffb640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7640038490 0x7f764003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f766102f640 1 -- 172.21.15.116:0/299469838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f765c06d400 con 0x7f765c06caf0 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.352+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f765804b210 con 0x7f765c068a20 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.353+0000 7f7662031640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7640038490 0x7f764003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.353+0000 7f7662031640 10 monclient: get_auth_request con 0x7f7640038490 auth_method 0 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f766102f640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 msgr2=0x7f765c06cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f766102f640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 0x7f765c06cec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f764c00d4d0 tx=0x7f764c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c012810 con 0x7f765c06caf0 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 10 monclient: dump: 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f764c012970 con 0x7f765c06caf0 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 msgr2=0x7f765c06cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 0x7f765c06cec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f764c00d4d0 tx=0x7f764c009040 comp rx=0 tx=0).stop 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 msgr2=0x7f765c06cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7651ffb640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c06caf0 0x7f765c06cec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.355+0000 7f7662031640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7640038490 0x7f764003a940 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f765c06aae0 tx=0x7f7654037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f7663a94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f7663a94640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 0x7f765c138850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f7663a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c04bf60 con 0x7f765c138450 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 0x7f765c138850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient: get_auth_request con 0x7f765c138450 auth_method 0 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-18T12:25:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.361+0000 7f766102f640 1 -- 172.21.15.116:0/299469838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f765c0987b0 con 0x7f765c138450 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.363+0000 7f766102f640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 msgr2=0x7f765c138850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.363+0000 7f766102f640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 0x7f765c138850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f764c018000 tx=0x7f764c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.363+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c021070 con 0x7f765c138450 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.363+0000 7f7651ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.363+0000 7f7651ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.363+0000 7f7651ffb640 10 monclient: dump: 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f764c0076a0 con 0x7f765c138450 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7651ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7651ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 msgr2=0x7f765c138850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7651ffb640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 0x7f765c138850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f764c018000 tx=0x7f764c01b000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7651ffb640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 msgr2=0x7f765c138850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7651ffb640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f765c138450 0x7f765c138850 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f764c018000 tx=0x7f764c01b000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:25:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 4, 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 22, 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:25:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:25:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:25:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:25:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:25:35.386 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7640038490 msgr2=0x7f764003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7640038490 0x7f764003a940 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f765c06aae0 tx=0x7f7654037000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 10 monclient: shutdown 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 msgr2=0x7f765c097840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 0x7f765c097840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f765800c5e0 tx=0x7f7658004960 comp rx=0 tx=0).stop 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7662031640 1 -- 172.21.15.116:0/299469838 reap_dead start 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 shutdown_connections 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 --2- 172.21.15.116:0/299469838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f765c068a20 0x7f765c097840 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 >> 172.21.15.116:0/299469838 conn(0x7f765c067cd0 msgr2=0x7f765c131d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 shutdown_connections 2024-09-18T12:25:35.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.364+0000 7f7663a94640 1 -- 172.21.15.116:0/299469838 wait complete. 2024-09-18T12:25:35.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:25:35.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:25:35.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.482+0000 7f25dd35d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.482+0000 7f25dd35d640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.482+0000 7f25dd35d640 10 monclient: monmap: 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:35.482793+0000 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:35.482793+0000 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:35.484 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.482+0000 7f25dd35d640 1 Processor -- start 2024-09-18T12:25:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.482+0000 7f25dd35d640 1 -- start start 2024-09-18T12:25:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.482+0000 7f25dd35d640 10 monclient: init 2024-09-18T12:25:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f25d81318f0 0x7f25d8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient(hunting): picked mon.noname-c con 0x7f25d81318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25d809ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d8139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient(hunting): picked mon.noname-b con 0x7f25d8136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d8131cc0 con 0x7f25d809ea50 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d809f530 con 0x7f25d81318f0 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d809fbf0 con 0x7f25d8136c20 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25dd35d640 10 monclient(hunting): authenticate will time out at 762.804233s 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d8139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:49058/0 (socket says 172.21.15.116:49058) 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 -- 172.21.15.116:0/4213325697 learned_addr learned my addr 172.21.15.116:0/4213325697 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): get_auth_request con 0x7f25d8136c20 auth_method 0 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d6ffd640 1 --2- 172.21.15.116:0/4213325697 >> v2:172.21.15.144:3300/0 conn(0x7f25d81318f0 0x7f25d8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 -- 172.21.15.116:0/4213325697 >> v2:172.21.15.144:3300/0 conn(0x7f25d81318f0 msgr2=0x7f25d8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 --2- 172.21.15.116:0/4213325697 >> v2:172.21.15.144:3300/0 conn(0x7f25d81318f0 0x7f25d8131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 -- 172.21.15.116:0/4213325697 >> v1:172.21.15.144:6789/0 conn(0x7f25d809ea50 legacy=0x7f25d809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 monclient: found mon.noname-b 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.483+0000 7f25d77fe640 1 -- 172.21.15.116:0/4213325697 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d809fe30 con 0x7f25d8136c20 2024-09-18T12:25:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d77fe640 1 -- 172.21.15.116:0/4213325697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 msgr2=0x7f25d8139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d77fe640 1 --2- 172.21.15.116:0/4213325697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d8139000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009910 tx=0x7f25d00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eb47b6abb55dc44 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 1 -- 172.21.15.116:0/4213325697 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0032440 con 0x7f25d8136c20 2024-09-18T12:25:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: dump: 2024-09-18T12:25:35.496 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 1 -- 172.21.15.116:0/4213325697 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d00325a0 con 0x7f25d8136c20 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25dd35d640 5 monclient: authenticate success, global_id 4358 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25dd35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25dd35d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 1 -- 172.21.15.116:0/4213325697 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0032940 con 0x7f25d8136c20 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25dd35d640 4 set_mon_vals no callback set 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.486+0000 7f25d5ffb640 10 monclient: dump: 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 10 monclient: shutdown 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 -- 172.21.15.116:0/4213325697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 msgr2=0x7f25d8139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 --2- 172.21.15.116:0/4213325697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d8139000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009910 tx=0x7f25d00091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 -- 172.21.15.116:0/4213325697 shutdown_connections 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 --2- 172.21.15.116:0/4213325697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d8139000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 --2- 172.21.15.116:0/4213325697 >> v2:172.21.15.144:3300/0 conn(0x7f25d81318f0 0x7f25d8131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 -- 172.21.15.116:0/4213325697 >> 172.21.15.116:0/4213325697 conn(0x7f25d8067110 msgr2=0x7f25d8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 -- 172.21.15.116:0/4213325697 shutdown_connections 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 1 -- 172.21.15.116:0/4213325697 wait complete. 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.487+0000 7f25dd35d640 10 monclient: monmap: 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:35.500 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 Processor -- start 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 -- start start 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient: init 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:25:35.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809ea50 0x7f25d8099640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): picked mon.noname-b con 0x7f25d809ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d81318f0 0x7f25d8099b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): picked mon.noname-a con 0x7f25d81318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d809a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): picked mon.noname-c con 0x7f25d8136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d809b5a0 con 0x7f25d81318f0 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d809dd00 con 0x7f25d809ea50 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d809bf10 con 0x7f25d8136c20 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25dd35d640 10 monclient(hunting): authenticate will time out at 762.809626s 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d809a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d81318f0 0x7f25d8099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d809a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:49062/0 (socket says 172.21.15.116:49062) 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 1 -- 172.21.15.116:0/1561293790 learned_addr learned my addr 172.21.15.116:0/1561293790 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): get_auth_request con 0x7f25d8136c20 auth_method 0 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 10 monclient(hunting): get_auth_request con 0x7f25d81318f0 auth_method 0 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d6ffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809ea50 0x7f25d8099640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d6ffd640 10 monclient(hunting): get_auth_request con 0x7f25d809ea50 auth_method 0 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.488+0000 7f25d77fe640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809ea50 msgr2=0x7f25d8099640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809ea50 0x7f25d8099640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d81318f0 msgr2=0x7f25d8099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d81318f0 0x7f25d8099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 1 monclient: found mon.noname-c 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d77fe640 1 -- 172.21.15.116:0/1561293790 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0009450 con 0x7f25d8136c20 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.489+0000 7f25d67fc640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d81318f0 0x7f25d8099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.491+0000 7f25d77fe640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 msgr2=0x7f25d809a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.491+0000 7f25d77fe640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d809a0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25d0035040 tx=0x7f25d0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0031c70 con 0x7f25d8136c20 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: dump: 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d0031df0 con 0x7f25d8136c20 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25dd35d640 5 monclient: authenticate success, global_id 4361 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25dd35d640 10 monclient: _renew_subs 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d003d690 con 0x7f25d8136c20 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25dd35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d811d040 con 0x7f25d8136c20 2024-09-18T12:25:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25beffd640 10 monclient: dump: 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25d57fa640 4 set_mon_vals no callback set 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25dd35d640 10 monclient: _renew_subs 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25dd35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.493+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d811b8b0 con 0x7f25d8136c20 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25dd35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f25d0039040 con 0x7f25d8136c20 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25dd35d640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 0x7f25d802aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25dd35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d811c940 con 0x7f25d809a600 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25beffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c40384e0 0x7f25c403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f25d006b7b0 con 0x7f25d8136c20 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 0x7f25d802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d67fc640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c40384e0 0x7f25c403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient: get_auth_request con 0x7f25d809a600 auth_method 0 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d67fc640 10 monclient: get_auth_request con 0x7f25c40384e0 auth_method 0 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-09-18T12:25:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.494+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1561293790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25d804bf60 con 0x7f25d809a600 2024-09-18T12:25:35.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25d67fc640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c40384e0 0x7f25c403a990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25d809ac20 tx=0x7f25cc007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 msgr2=0x7f25d802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25d6ffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 0x7f25d802aa40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007f10 tx=0x7f25c80079a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c800cea0 con 0x7f25d809a600 2024-09-18T12:25:35.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 10 monclient: dump: 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f25c8013070 con 0x7f25d809a600 2024-09-18T12:25:35.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 msgr2=0x7f25d802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 0x7f25d802aa40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007f10 tx=0x7f25c80079a0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 msgr2=0x7f25d802aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.497+0000 7f25beffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d809a600 0x7f25d802aa40 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007f10 tx=0x7f25c80079a0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25dd35d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25dd35d640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 0x7f25d805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25dd35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8065e00 con 0x7f25d805e310 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 0x7f25d805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient: get_auth_request con 0x7f25d805e310 auth_method 0 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.503+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1561293790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f25d809b020 con 0x7f25d805e310 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.505+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 msgr2=0x7f25d805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.505+0000 7f25d6ffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 0x7f25d805e710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007f10 tx=0x7f25c8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c800fe40 con 0x7f25d805e310 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 10 monclient: dump: 2024-09-18T12:25:35.516 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f25c8018380 con 0x7f25d805e310 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:35.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 msgr2=0x7f25d805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 0x7f25d805e710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007f10 tx=0x7f25c8019000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 msgr2=0x7f25d805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25beffd640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d805e310 0x7f25d805e710 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f25c8007f10 tx=0x7f25c8019000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 4, 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:25:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 22, 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:25:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:25:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:25:35.531 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:25:35.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c40384e0 msgr2=0x7f25c403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c40384e0 0x7f25c403a990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25d809ac20 tx=0x7f25cc007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 10 monclient: shutdown 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 msgr2=0x7f25d809a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 --2- 172.21.15.116:0/1561293790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d8136c20 0x7f25d809a0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25d0035040 tx=0x7f25d0032000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25d77fe640 1 -- 172.21.15.116:0/1561293790 reap_dead start 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25d77fe640 1 -- 172.21.15.116:0/1561293790 reap_dead start 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 shutdown_connections 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 >> 172.21.15.116:0/1561293790 conn(0x7f25d8067110 msgr2=0x7f25d80a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 shutdown_connections 2024-09-18T12:25:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.506+0000 7f25dd35d640 1 -- 172.21.15.116:0/1561293790 wait complete. 2024-09-18T12:25:35.534 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:25:35.534 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:25:35.534 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:25:35.534 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:25:35.534 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:25:35.534 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:25:35.535 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:25:35.545 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:25:35.545 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:25:35.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:25:35.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.677+0000 7f2780e60640 10 monclient: get_monmap_and_config 2024-09-18T12:25:35.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.677+0000 7f2780e60640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.677+0000 7f2780e60640 10 monclient: monmap: 2024-09-18T12:25:35.678 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.678 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:35.678166+0000 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:35.678166+0000 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:35.679 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 1 Processor -- start 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 1 -- start start 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient: init 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f277c0981c0 0x7f277c098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient(hunting): picked mon.noname-a con 0x7f277c0981c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:35.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f277c131da0 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f277c132170 0x7f277c067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient(hunting): picked mon.noname-c con 0x7f277c132170 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f277c098ad0 con 0x7f277c131da0 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f277c131bd0 con 0x7f277c0981c0 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f277c09ca60 con 0x7f277c132170 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f2780e60640 10 monclient(hunting): authenticate will time out at 762.999814s 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f277a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f277c0981c0 0x7f277c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f277a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f277c0981c0 0x7f277c098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45524/0 (socket says 172.21.15.116:45524) 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f277a59c640 1 -- 172.21.15.116:0/1112383785 learned_addr learned my addr 172.21.15.116:0/1112383785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:35.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f277a59c640 10 monclient(hunting): get_auth_request con 0x7f277c0981c0 auth_method 0 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f277a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f277a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.678+0000 7f277a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277ad9d640 1 --2- 172.21.15.116:0/1112383785 >> v2:172.21.15.144:3300/0 conn(0x7f277c132170 0x7f277c067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277ad9d640 10 monclient(hunting): get_auth_request con 0x7f277c132170 auth_method 0 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 10 monclient(hunting): handle_auth_done global_id 4389 payload 274 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 1 -- 172.21.15.116:0/1112383785 >> v2:172.21.15.144:3300/0 conn(0x7f277c132170 msgr2=0x7f277c067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 1 --2- 172.21.15.116:0/1112383785 >> v2:172.21.15.144:3300/0 conn(0x7f277c132170 0x7f277c067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 1 -- 172.21.15.116:0/1112383785 >> v1:172.21.15.144:6789/0 conn(0x7f277c131da0 legacy=0x7f277c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:35.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 1 monclient: found mon.noname-a 2024-09-18T12:25:35.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277a59c640 1 -- 172.21.15.116:0/1112383785 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277c09cca0 con 0x7f277c0981c0 2024-09-18T12:25:35.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.679+0000 7f277ad9d640 1 --2- 172.21.15.116:0/1112383785 >> v2:172.21.15.144:3300/0 conn(0x7f277c132170 0x7f277c067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:25:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.681+0000 7f277a59c640 1 -- 172.21.15.116:0/1112383785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 msgr2=0x7f277c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.681+0000 7f277a59c640 1 --2- 172.21.15.116:0/1112383785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c098590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f277c06f820 tx=0x7f276c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74193b7a78ef66b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 1 -- 172.21.15.116:0/1112383785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c032440 con 0x7f277c0981c0 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: dump: 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 1 -- 172.21.15.116:0/1112383785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c0325a0 con 0x7f277c0981c0 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 5 monclient: authenticate success, global_id 4389 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 1 -- 172.21.15.116:0/1112383785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c032980 con 0x7f277c0981c0 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 4 set_mon_vals no callback set 2024-09-18T12:25:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2779d9b640 10 monclient: dump: 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.693 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 10 monclient: shutdown 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 -- 172.21.15.116:0/1112383785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 msgr2=0x7f277c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 --2- 172.21.15.116:0/1112383785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c098590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f277c06f820 tx=0x7f276c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 -- 172.21.15.116:0/1112383785 shutdown_connections 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 --2- 172.21.15.116:0/1112383785 >> v2:172.21.15.144:3300/0 conn(0x7f277c132170 0x7f277c067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 --2- 172.21.15.116:0/1112383785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c098590 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 -- 172.21.15.116:0/1112383785 >> 172.21.15.116:0/1112383785 conn(0x7f277c0a1e30 msgr2=0x7f277c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 -- 172.21.15.116:0/1112383785 shutdown_connections 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.682+0000 7f2780e60640 1 -- 172.21.15.116:0/1112383785 wait complete. 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.683+0000 7f2780e60640 10 monclient: build_initial_monmap 2024-09-18T12:25:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.683+0000 7f2780e60640 10 monclient: monmap: 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.683+0000 7f2780e60640 1 Processor -- start 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.683+0000 7f2780e60640 1 -- start start 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.683+0000 7f2780e60640 10 monclient: init 2024-09-18T12:25:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.683+0000 7f2780e60640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c12ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): picked mon.noname-a con 0x7f277c0981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f277c131da0 0x7f277c11cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): picked mon.noname-b con 0x7f277c131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f277c132170 0x7f277c11d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): picked mon.noname-c con 0x7f277c132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c098cb0 con 0x7f277c0981c0 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c09ba50 con 0x7f277c131da0 2024-09-18T12:25:35.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277c120c70 con 0x7f277c132170 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2780e60640 10 monclient(hunting): authenticate will time out at 763.005095s 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277ad9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f277c132170 msgr2=0x7f277c11d100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f277c132170 0x7f277c11d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c12ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c12ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45532/0 (socket says 172.21.15.116:45532) 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 -- 172.21.15.116:0/3154669964 learned_addr learned my addr 172.21.15.116:0/3154669964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): get_auth_request con 0x7f277c0981c0 auth_method 0 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2773fff640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f277c131da0 0x7f277c11cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2773fff640 10 monclient(hunting): get_auth_request con 0x7f277c131da0 auth_method 0 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2773fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2773fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f2773fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 -- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f277c132170 msgr2=0x7f277c11d100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f277c132170 0x7f277c11d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 -- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f277c131da0 msgr2=0x7f277c11cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f277c131da0 0x7f277c11cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 monclient: found mon.noname-a 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.684+0000 7f277a59c640 1 -- 172.21.15.116:0/3154669964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c009450 con 0x7f277c0981c0 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f277a59c640 1 -- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 msgr2=0x7f277c12ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f277a59c640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c12ffd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f276c035040 tx=0x7f276c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 1 -- 172.21.15.116:0/3154669964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c031940 con 0x7f277c0981c0 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: dump: 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.699 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.700 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f2780e60640 5 monclient: authenticate success, global_id 4392 2024-09-18T12:25:35.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 1 -- 172.21.15.116:0/3154669964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c031aa0 con 0x7f277c0981c0 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f2780e60640 10 monclient: _renew_subs 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f2780e60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277c12f5c0 con 0x7f277c0981c0 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 1 -- 172.21.15.116:0/3154669964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c031e60 con 0x7f277c0981c0 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f27727fc640 10 monclient: dump: 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f2780e60640 10 monclient: _renew_subs 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f277959a640 4 set_mon_vals no callback set 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f2780e60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.686+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277c11e480 con 0x7f277c0981c0 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.687+0000 7f27727fc640 1 -- 172.21.15.116:0/3154669964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f276c039040 con 0x7f277c0981c0 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.687+0000 7f2780e60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.687+0000 7f2780e60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.687+0000 7f2780e60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.687+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277c04bf60 con 0x7f277c0981c0 2024-09-18T12:25:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.687+0000 7f27727fc640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f275c038470 0x7f275c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.687+0000 7f27727fc640 1 -- 172.21.15.116:0/3154669964 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f276c06af90 con 0x7f277c0981c0 2024-09-18T12:25:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.688+0000 7f2773fff640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f275c038470 0x7f275c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.688+0000 7f2773fff640 10 monclient: get_auth_request con 0x7f275c038470 auth_method 0 2024-09-18T12:25:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.690+0000 7f2773fff640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f275c038470 0x7f275c03a920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27640037d0 tx=0x7f2764009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:35.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.692+0000 7f27727fc640 1 -- 172.21.15.116:0/3154669964 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f276c044b40 con 0x7f277c0981c0 2024-09-18T12:25:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.692+0000 7f27727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.692+0000 7f27727fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:35.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.906+0000 7f2780e60640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:25:35.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.906+0000 7f2780e60640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:35.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.906+0000 7f2780e60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:35.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.906+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f277c098ee0 con 0x7f277c0981c0 2024-09-18T12:25:35.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.906+0000 7f27727fc640 1 -- 172.21.15.116:0/3154669964 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f276c037020 con 0x7f277c0981c0 2024-09-18T12:25:35.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.906+0000 7f27727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:35.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.906+0000 7f27727fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:35.908 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f275c038470 msgr2=0x7f275c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f275c038470 0x7f275c03a920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27640037d0 tx=0x7f2764009040 comp rx=0 tx=0).stop 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 10 monclient: shutdown 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 msgr2=0x7f277c12ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c12ffd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f276c035040 tx=0x7f276c032000 comp rx=0 tx=0).stop 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 shutdown_connections 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f275c038470 0x7f275c03a920 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f277c132170 0x7f277c11d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f277c131da0 0x7f277c11cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 --2- 172.21.15.116:0/3154669964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f277c0981c0 0x7f277c12ffd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 >> 172.21.15.116:0/3154669964 conn(0x7f277c0a1e30 msgr2=0x7f277c09b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 shutdown_connections 2024-09-18T12:25:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:35.908+0000 7f2780e60640 1 -- 172.21.15.116:0/3154669964 wait complete. 2024-09-18T12:25:38.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:25:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.024+0000 7f025831d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.024+0000 7f025831d640 10 monclient: build_initial_monmap 2024-09-18T12:25:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.024+0000 7f025831d640 10 monclient: monmap: 2024-09-18T12:25:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:39.024746+0000 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:39.024746+0000 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.024+0000 7f025831d640 1 Processor -- start 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.024+0000 7f025831d640 1 -- start start 2024-09-18T12:25:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.024+0000 7f025831d640 10 monclient: init 2024-09-18T12:25:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02501318f0 0x7f0250131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): picked mon.noname-c con 0x7f02501318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f025009ea50 0x7f025009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): picked mon.noname-b con 0x7f025009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0250068d90 0x7f02500691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): picked mon.noname-a con 0x7f0250068d90 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0250131cc0 con 0x7f0250068d90 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f025009fbf0 con 0x7f02501318f0 2024-09-18T12:25:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025009fe30 con 0x7f025009ea50 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f025831d640 10 monclient(hunting): authenticate will time out at 766.346523s 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02558b8640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f025009ea50 msgr2=0x7f025009ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02558b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f025009ea50 0x7f025009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0250068d90 0x7f02500691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0250068d90 0x7f02500691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45538/0 (socket says 172.21.15.116:45538) 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 -- 172.21.15.116:0/1964818284 learned_addr learned my addr 172.21.15.116:0/1964818284 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): get_auth_request con 0x7f0250068d90 auth_method 0 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02560b9640 1 --2- 172.21.15.116:0/1964818284 >> v2:172.21.15.144:3300/0 conn(0x7f02501318f0 0x7f0250131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient(hunting): handle_auth_done global_id 4395 payload 274 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 -- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f025009ea50 msgr2=0x7f025009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 --2- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f025009ea50 0x7f025009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 -- 172.21.15.116:0/1964818284 >> v2:172.21.15.144:3300/0 conn(0x7f02501318f0 msgr2=0x7f0250131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 --2- 172.21.15.116:0/1964818284 >> v2:172.21.15.144:3300/0 conn(0x7f02501318f0 0x7f0250131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient: _finish_hunting 0 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 monclient: found mon.noname-a 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02568ba640 1 -- 172.21.15.116:0/1964818284 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02500a1210 con 0x7f0250068d90 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02560b9640 10 monclient: get_auth_request con 0x7f02501318f0 auth_method 0 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.025+0000 7f02560b9640 1 --2- 172.21.15.116:0/1964818284 >> v2:172.21.15.144:3300/0 conn(0x7f02501318f0 0x7f0250131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02568ba640 1 -- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0250068d90 msgr2=0x7f02500691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02568ba640 1 --2- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0250068d90 0x7f02500691f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0250076600 tx=0x7f02480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c28f6ca3a2164fc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 1 -- 172.21.15.116:0/1964818284 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248032440 con 0x7f0250068d90 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: dump: 2024-09-18T12:25:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: _finish_auth 0 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f025831d640 5 monclient: authenticate success, global_id 4395 2024-09-18T12:25:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 1 -- 172.21.15.116:0/1964818284 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02480325a0 con 0x7f0250068d90 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f025831d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f025831d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f025831d640 4 set_mon_vals no callback set 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 1 -- 172.21.15.116:0/1964818284 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248032940 con 0x7f0250068d90 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.028+0000 7f02550b7640 10 monclient: dump: 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 10 monclient: shutdown 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 -- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0250068d90 msgr2=0x7f02500691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 --2- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0250068d90 0x7f02500691f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0250076600 tx=0x7f02480091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 -- 172.21.15.116:0/1964818284 shutdown_connections 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 --2- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0250068d90 0x7f02500691f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 --2- 172.21.15.116:0/1964818284 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f025009ea50 0x7f025009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 --2- 172.21.15.116:0/1964818284 >> v2:172.21.15.144:3300/0 conn(0x7f02501318f0 0x7f0250131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 -- 172.21.15.116:0/1964818284 >> 172.21.15.116:0/1964818284 conn(0x7f0250067110 msgr2=0x7f0250067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 -- 172.21.15.116:0/1964818284 shutdown_connections 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.029+0000 7f025831d640 1 -- 172.21.15.116:0/1964818284 wait complete. 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 10 monclient: build_initial_monmap 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 10 monclient: monmap: 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 1 Processor -- start 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 1 -- start start 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 10 monclient: init 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.030+0000 7f025831d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0250068d90 0x7f025009a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): picked mon.noname-c con 0x7f0250068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 0x7f0250130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): picked mon.noname-a con 0x7f025009ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02501318f0 0x7f025011b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): picked mon.noname-b con 0x7f02501318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025011e0b0 con 0x7f025009ea50 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025009bf10 con 0x7f02501318f0 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02560b9640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0250068d90 msgr2=0x7f025009a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02500a41e0 con 0x7f0250068d90 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f025831d640 10 monclient(hunting): authenticate will time out at 766.352054s 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02560b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0250068d90 0x7f025009a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 0x7f0250130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 0x7f0250130940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45552/0 (socket says 172.21.15.116:45552) 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 -- 172.21.15.116:0/531019126 learned_addr learned my addr 172.21.15.116:0/531019126 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): get_auth_request con 0x7f025009ea50 auth_method 0 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-18T12:25:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02568ba640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02501318f0 0x7f025011b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 -- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0250068d90 msgr2=0x7f025009a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0250068d90 0x7f025009a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 -- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02501318f0 msgr2=0x7f025011b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02501318f0 0x7f025011b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient: _finish_hunting 0 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 monclient: found mon.noname-a 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.031+0000 7f02558b8640 1 -- 172.21.15.116:0/531019126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0248009450 con 0x7f025009ea50 2024-09-18T12:25:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02558b8640 1 -- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 msgr2=0x7f0250130940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02558b8640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 0x7f0250130940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f024000b630 tx=0x7f02400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 1 -- 172.21.15.116:0/531019126 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240015070 con 0x7f025009ea50 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: dump: 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:39.048 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 1 -- 172.21.15.116:0/531019126 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240003d90 con 0x7f025009ea50 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f025831d640 5 monclient: authenticate success, global_id 4398 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f025831d640 10 monclient: _renew_subs 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f025831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025009b2f0 con 0x7f025009ea50 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 1 -- 172.21.15.116:0/531019126 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240006380 con 0x7f025009ea50 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02467fc640 10 monclient: dump: 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f025831d640 10 monclient: _renew_subs 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f02548b6640 4 set_mon_vals no callback set 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f025831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.033+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025011da60 con 0x7f025009ea50 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.034+0000 7f02467fc640 1 -- 172.21.15.116:0/531019126 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0240010440 con 0x7f025009ea50 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.034+0000 7f025831d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.034+0000 7f025831d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.034+0000 7f025831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.034+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025011cbc0 con 0x7f025009ea50 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.034+0000 7f02467fc640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0234038440 0x7f023403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.035+0000 7f02467fc640 1 -- 172.21.15.116:0/531019126 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f024004b1e0 con 0x7f025009ea50 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.035+0000 7f02560b9640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0234038440 0x7f023403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.035+0000 7f02560b9640 10 monclient: get_auth_request con 0x7f0234038440 auth_method 0 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.037+0000 7f02560b9640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0234038440 0x7f023403a8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f024c0012a0 tx=0x7f024c0084e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.039+0000 7f02467fc640 1 -- 172.21.15.116:0/531019126 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0240018a70 con 0x7f025009ea50 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.039+0000 7f02467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.039+0000 7f02467fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.253+0000 7f025831d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.253+0000 7f025831d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.253+0000 7f025831d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.253+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f025004bf60 con 0x7f025009ea50 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.253+0000 7f02467fc640 1 -- 172.21.15.116:0/531019126 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f0240018a70 con 0x7f025009ea50 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.253+0000 7f02467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.253+0000 7f02467fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:39.254 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0234038440 msgr2=0x7f023403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0234038440 0x7f023403a8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f024c0012a0 tx=0x7f024c0084e0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 10 monclient: shutdown 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 msgr2=0x7f0250130940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 0x7f0250130940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f024000b630 tx=0x7f02400091a0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 shutdown_connections 2024-09-18T12:25:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0234038440 0x7f023403a8f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02501318f0 0x7f025011b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f025009ea50 0x7f0250130940 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 --2- 172.21.15.116:0/531019126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0250068d90 0x7f025009a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 >> 172.21.15.116:0/531019126 conn(0x7f0250067110 msgr2=0x7f02500a2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 shutdown_connections 2024-09-18T12:25:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:39.255+0000 7f025831d640 1 -- 172.21.15.116:0/531019126 wait complete. 2024-09-18T12:25:42.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.369+0000 7f43fe2a8640 10 monclient: get_monmap_and_config 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.369+0000 7f43fe2a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.369+0000 7f43fe2a8640 10 monclient: monmap: 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:42.369890+0000 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:42.369890+0000 2024-09-18T12:25:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:42.371 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.369+0000 7f43fe2a8640 1 Processor -- start 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 1 -- start start 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient: init 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f43f8096c60 0x7f43f809a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f43f8096c60 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43f8097030 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43f8133b40 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f80696a0 con 0x7f43f8133b40 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f809d2e0 con 0x7f43f8097030 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f809ac60 con 0x7f43f8096c60 2024-09-18T12:25:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fe2a8640 10 monclient(hunting): authenticate will time out at 769.691686s 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fc845640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f43f8133b40 0x7f43f8135f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53634/0 (socket says 172.21.15.116:53634) 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43fc845640 1 -- 172.21.15.116:0/1903957655 learned_addr learned my addr 172.21.15.116:0/1903957655 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43f7fff640 1 --2- 172.21.15.116:0/1903957655 >> v2:172.21.15.144:3300/0 conn(0x7f43f8096c60 0x7f43f809a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43f7fff640 1 -- 172.21.15.116:0/1903957655 >> v2:172.21.15.144:3300/0 conn(0x7f43f8096c60 msgr2=0x7f43f809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43f7fff640 1 -- 172.21.15.116:0/1903957655 >> v2:172.21.15.144:3300/0 conn(0x7f43f8096c60 msgr2=0x7f43f809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43f7fff640 1 --2- 172.21.15.116:0/1903957655 >> v2:172.21.15.144:3300/0 conn(0x7f43f8096c60 0x7f43f809a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.370+0000 7f43f7fff640 1 --2- 172.21.15.116:0/1903957655 >> v2:172.21.15.144:3300/0 conn(0x7f43f8096c60 0x7f43f809a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.372+0000 7f43fc845640 1 -- 172.21.15.116:0/1903957655 >> v1:172.21.15.116:6789/0 conn(0x7f43f8133b40 legacy=0x7f43f8135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.373+0000 7f43f77fe640 1 -- 172.21.15.116:0/1903957655 >> v1:172.21.15.144:6789/0 conn(0x7f43f8097030 legacy=0x7f43f8097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 1 --2- 172.21.15.116:0/1903957655 >> v2:172.21.15.144:3300/0 conn(0x7f43f8096c60 0x7f43f809a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): get_auth_request con 0x7f43f8096c60 auth_method 0 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 1 -- 172.21.15.116:0/1903957655 >> v1:172.21.15.144:6789/0 conn(0x7f43f8097030 legacy=0x7f43f8097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 1 -- 172.21.15.116:0/1903957655 >> v1:172.21.15.116:6789/0 conn(0x7f43f8133b40 legacy=0x7f43f8135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 1 monclient: found mon.noname-c 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:25:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.571+0000 7f43f7fff640 1 -- 172.21.15.116:0/1903957655 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f809d6b0 con 0x7f43f8096c60 2024-09-18T12:25:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f7fff640 1 -- 172.21.15.116:0/1903957655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8096c60 msgr2=0x7f43f809a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f7fff640 1 --2- 172.21.15.116:0/1903957655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8096c60 0x7f43f809a6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f43e8006b30 tx=0x7f43e8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1efa616a943a4b5c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 1 -- 172.21.15.116:0/1903957655 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80308f0 con 0x7f43f8096c60 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: dump: 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 1 -- 172.21.15.116:0/1903957655 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8030a70 con 0x7f43f8096c60 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43fe2a8640 5 monclient: authenticate success, global_id 4342 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43fe2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43fe2a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43fe2a8640 4 set_mon_vals no callback set 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 1 -- 172.21.15.116:0/1903957655 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8030e30 con 0x7f43f8096c60 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.574+0000 7f43f6ffd640 10 monclient: dump: 2024-09-18T12:25:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:42.578 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 10 monclient: shutdown 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 1 -- 172.21.15.116:0/1903957655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8096c60 msgr2=0x7f43f809a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/1903957655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8096c60 0x7f43f809a6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f43e8006b30 tx=0x7f43e8008040 comp rx=0 tx=0).stop 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 1 -- 172.21.15.116:0/1903957655 shutdown_connections 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/1903957655 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8096c60 0x7f43f809a6c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 1 -- 172.21.15.116:0/1903957655 >> 172.21.15.116:0/1903957655 conn(0x7f43f80672b0 msgr2=0x7f43f80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 1 -- 172.21.15.116:0/1903957655 shutdown_connections 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.575+0000 7f43fe2a8640 1 -- 172.21.15.116:0/1903957655 wait complete. 2024-09-18T12:25:42.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.576+0000 7f43fe2a8640 10 monclient: build_initial_monmap 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.576+0000 7f43fe2a8640 10 monclient: monmap: 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:42.586 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.576+0000 7f43fe2a8640 1 Processor -- start 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.576+0000 7f43fe2a8640 1 -- start start 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.576+0000 7f43fe2a8640 10 monclient: init 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43f8096c60 0x7f43f806e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f43f8096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43f8097030 0x7f43f812ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f43f8097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 0x7f43f8141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:42.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f43f8133b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f806f160 con 0x7f43f8096c60 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f811c8a0 con 0x7f43f8133b40 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f8133670 con 0x7f43f8097030 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fe2a8640 10 monclient(hunting): authenticate will time out at 769.898329s 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f77fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43f8097030 msgr2=0x7f43f812ea90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43f8097030 0x7f43f812ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:42.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43f8096c60 0x7f43f806e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43f8096c60 0x7f43f806e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45564/0 (socket says 172.21.15.116:45564) 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 1 -- 172.21.15.116:0/3856712477 learned_addr learned my addr 172.21.15.116:0/3856712477 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 10 monclient(hunting): get_auth_request con 0x7f43f8096c60 auth_method 0 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 0x7f43f8141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): get_auth_request con 0x7f43f8133b40 auth_method 0 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:42.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 1 -- 172.21.15.116:0/3856712477 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43f8097030 msgr2=0x7f43f812ea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43f8097030 0x7f43f812ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 1 -- 172.21.15.116:0/3856712477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43f8096c60 msgr2=0x7f43f806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43f8096c60 0x7f43f806e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:42.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient: _finish_hunting 0 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 1 monclient: found mon.noname-b 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43fc845640 1 -- 172.21.15.116:0/3856712477 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e800c700 con 0x7f43f8133b40 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.577+0000 7f43f7fff640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43f8096c60 0x7f43f806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fc845640 1 -- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 msgr2=0x7f43f8141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fc845640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 0x7f43f8141260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43f000cbc0 tx=0x7f43f0001bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 1 -- 172.21.15.116:0/3856712477 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f000a850 con 0x7f43f8133b40 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:42.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: dump: 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:42.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: _finish_auth 0 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 1 -- 172.21.15.116:0/3856712477 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f000a9d0 con 0x7f43f8133b40 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fe2a8640 5 monclient: authenticate success, global_id 4345 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fe2a8640 10 monclient: _renew_subs 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fe2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f811dbd0 con 0x7f43f8133b40 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 1 -- 172.21.15.116:0/3856712477 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f000ad90 con 0x7f43f8133b40 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43effff640 10 monclient: dump: 2024-09-18T12:25:42.593 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fe2a8640 10 monclient: _renew_subs 2024-09-18T12:25:42.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fe2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f811cce0 con 0x7f43f8133b40 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.580+0000 7f43f67fc640 4 set_mon_vals no callback set 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43fc845640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 0x7f43f8141260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43f000cbc0 tx=0x7f43f0001bc0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f43f0007300 mgrmap(e 5) v1 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43effff640 1 -- 172.21.15.116:0/3856712477 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f43f0007300 con 0x7f43f8133b40 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43fe2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43fe2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43fe2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f811e2b0 con 0x7f43f8133b40 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43effff640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43d8038440 0x7f43d803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43f7fff640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43d8038440 0x7f43d803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.581+0000 7f43f7fff640 10 monclient: get_auth_request con 0x7f43d8038440 auth_method 0 2024-09-18T12:25:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.584+0000 7f43f7fff640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43d8038440 0x7f43d803a8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f43e8006da0 tx=0x7f43e8039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:42.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:42.586+0000 7f43effff640 1 -- 172.21.15.116:0/3856712477 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f43f004b620 con 0x7f43f8133b40 2024-09-18T12:25:43.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.409+0000 7f43effff640 1 -- 172.21.15.116:0/3856712477 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f43f001b400 con 0x7f43f8133b40 2024-09-18T12:25:43.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.409+0000 7f43effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:43.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.409+0000 7f43effff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:43.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.626+0000 7f43fe2a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:25:43.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.626+0000 7f43fe2a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:43.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.626+0000 7f43fe2a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:43.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.626+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f43f804bf60 con 0x7f43f8133b40 2024-09-18T12:25:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.627+0000 7f43effff640 1 -- 172.21.15.116:0/3856712477 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f43f0010300 con 0x7f43f8133b40 2024-09-18T12:25:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.627+0000 7f43effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:43.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.627+0000 7f43effff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:43.628 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43d8038440 msgr2=0x7f43d803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43d8038440 0x7f43d803a8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f43e8006da0 tx=0x7f43e8039000 comp rx=0 tx=0).stop 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 10 monclient: shutdown 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 msgr2=0x7f43f8141260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 0x7f43f8141260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f43f000cbc0 tx=0x7f43f0001bc0 comp rx=0 tx=0).stop 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 shutdown_connections 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43d8038440 0x7f43d803a8f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43f8133b40 0x7f43f8141260 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43f8097030 0x7f43f812ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 --2- 172.21.15.116:0/3856712477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43f8096c60 0x7f43f806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 >> 172.21.15.116:0/3856712477 conn(0x7f43f80672b0 msgr2=0x7f43f809f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.628+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 shutdown_connections 2024-09-18T12:25:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:43.629+0000 7f43fe2a8640 1 -- 172.21.15.116:0/3856712477 wait complete. 2024-09-18T12:25:46.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:25:46.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.743+0000 7f0f15e34640 10 monclient: get_monmap_and_config 2024-09-18T12:25:46.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.743+0000 7f0f15e34640 10 monclient: build_initial_monmap 2024-09-18T12:25:46.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.743+0000 7f0f15e34640 10 monclient: monmap: 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:46.744246+0000 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:46.744246+0000 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:46.745 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 1 Processor -- start 2024-09-18T12:25:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 1 -- start start 2024-09-18T12:25:46.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient: init 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f1009a510 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 0x7f0f101349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient(hunting): picked mon.noname-b con 0x7f0f100973b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient(hunting): picked mon.noname-c con 0x7f0f10097780 addr v2:172.21.15.144:3300/0 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1006ee10 con 0x7f0f1009a510 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1009cb00 con 0x7f0f10097780 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1009a8e0 con 0x7f0f100973b0 2024-09-18T12:25:46.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f15e34640 10 monclient(hunting): authenticate will time out at 774.065950s 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.744+0000 7f0f0effd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 msgr2=0x7f0f101349a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 0x7f0f101349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0f1009a510 0x7f0f10096ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37574/0 (socket says 172.21.15.116:37574) 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/1593357831 learned_addr learned my addr 172.21.15.116:0/1593357831 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request con 0x7f0f10097780 auth_method 0 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:46.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.745+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:46.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.747+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/1593357831 >> v1:172.21.15.116:6789/0 conn(0x7f0f1009a510 legacy=0x7f0f10096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:46.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 msgr2=0x7f0f101349a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:46.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 0x7f0f101349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request con 0x7f0f10097780 auth_method 0 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:46.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:46.946+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:25:47.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 msgr2=0x7f0f101349a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:47.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 0x7f0f101349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:25:47.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:47.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request con 0x7f0f10097780 auth_method 0 2024-09-18T12:25:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:47.347+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:25:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 msgr2=0x7f0f101349a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 0x7f0f101349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:25:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request con 0x7f0f10097780 auth_method 0 2024-09-18T12:25:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:48.148+0000 7f0f0ffff640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 10 monclient(hunting): tick 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 monclient(hunting): continuing hunt 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 msgr2=0x7f0f101349a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f100973b0 0x7f0f101349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 msgr2=0x7f0f10136ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.144:3300/0 conn(0x7f0f10097780 0x7f0f10136ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 -- 172.21.15.116:0/1593357831 >> v1:172.21.15.116:6789/0 conn(0x7f0f1009a510 legacy=0x7f0f10096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 10 monclient: _add_conns ranks=[1,3] 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0efc003750 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:49.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0efc0073d0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 -- 172.21.15.116:0/1593357831 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1009d590 con 0x7f0efc003750 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 1 -- 172.21.15.116:0/1593357831 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1009cb00 con 0x7f0efc0073d0 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0d7fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 10 monclient(hunting): get_auth_request con 0x7f0efc0073d0 auth_method 0 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 msgr2=0x7f0efc009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 msgr2=0x7f0efc009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:49.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:49.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.745+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:49.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.747+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/1593357831 >> v1:172.21.15.144:6789/0 conn(0x7f0efc003750 legacy=0x7f0efc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 10 monclient(hunting): get_auth_request con 0x7f0efc0073d0 auth_method 0 2024-09-18T12:25:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 msgr2=0x7f0efc009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:25:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 msgr2=0x7f0efc009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:25:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:25:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:49.946+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:25:50.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871700909 0 0) 0x7f0f1009d590 con 0x7f0efc003750 2024-09-18T12:25:50.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): my global_id is 4348 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef80034e0 con 0x7f0efc003750 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3142717266 0 0) 0x7f0ef80034e0 con 0x7f0efc003750 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 msgr2=0x7f0efc009880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 --2- 172.21.15.116:0/1593357831 >> v2:172.21.15.116:3300/0 conn(0x7f0efc0073d0 0x7f0efc009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc00af50 con 0x7f0efc003750 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f0f00001760 con 0x7f0efc003750 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f15e34640 5 monclient: authenticate success, global_id 4348 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: dump: 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.151 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 >> v1:172.21.15.144:6789/0 conn(0x7f0efc003750 legacy=0x7f0efc005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0ffff640 1 -- 172.21.15.116:0/1593357831 reap_dead start 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef80039d0 0x7f0ef8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): picked mon.b con 0x7f0ef80039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8004780 0x7f0ef8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): picked mon.c con 0x7f0ef8004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8005250 0x7f0ef800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): picked mon.a con 0x7f0ef8005250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1009cb00 con 0x7f0ef8005250 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef800c1c0 con 0x7f0ef80039d0 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef800c380 con 0x7f0ef8004780 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f15e34640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0effd640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8004780 msgr2=0x7f0ef8004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.146+0000 7f0f0effd640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8004780 0x7f0ef8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef80039d0 0x7f0ef8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): get_auth_request con 0x7f0ef80039d0 auth_method 0 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient(hunting): handle_auth_done global_id 4348 payload 293 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8004780 msgr2=0x7f0ef8004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8004780 0x7f0ef8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8005250 msgr2=0x7f0ef800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8005250 0x7f0ef800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:25:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 1 monclient: found mon.b 2024-09-18T12:25:50.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:50.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:50.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/1593357831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef80034e0 con 0x7f0ef80039d0 2024-09-18T12:25:50.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:50.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.147+0000 7f0f0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:50.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef80039d0 msgr2=0x7f0ef8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0f7fe640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef80039d0 0x7f0ef8003dc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0f00001940 tx=0x7f0f000079c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79f7be4207fa8a6e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00005890 con 0x7f0ef80039d0 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 10 monclient: dump: 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.161 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f000059f0 con 0x7f0ef80039d0 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f15e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f15e34640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f15e34640 4 set_mon_vals no callback set 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 1 -- 172.21.15.116:0/1593357831 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00005db0 con 0x7f0ef80039d0 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:50.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.150+0000 7f0f0e7fc640 10 monclient: dump: 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 10 monclient: shutdown 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 -- 172.21.15.116:0/1593357831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef80039d0 msgr2=0x7f0ef8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef80039d0 0x7f0ef8003dc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0f00001940 tx=0x7f0f000079c0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 -- 172.21.15.116:0/1593357831 shutdown_connections 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8005250 0x7f0ef800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8004780 0x7f0ef8004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 --2- 172.21.15.116:0/1593357831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef80039d0 0x7f0ef8003dc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 -- 172.21.15.116:0/1593357831 >> 172.21.15.116:0/1593357831 conn(0x7f0f100672b0 msgr2=0x7f0f100676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 -- 172.21.15.116:0/1593357831 shutdown_connections 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.151+0000 7f0f15e34640 1 -- 172.21.15.116:0/1593357831 wait complete. 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.152+0000 7f0f15e34640 10 monclient: build_initial_monmap 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.152+0000 7f0f15e34640 10 monclient: monmap: 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:50.164 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.152+0000 7f0f15e34640 1 Processor -- start 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 1 -- start start 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient: init 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef80039d0 0x7f0f1012f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef80039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 0x7f0f1011c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef8004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8005250 0x7f0f1011c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef8005250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1013f660 con 0x7f0ef80039d0 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1009e440 con 0x7f0ef8004780 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1009c370 con 0x7f0ef8005250 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f15e34640 10 monclient(hunting): authenticate will time out at 777.474600s 2024-09-18T12:25:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0ffff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8005250 msgr2=0x7f0f1011c930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8005250 0x7f0f1011c930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef80039d0 0x7f0f1012f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 0x7f0f1011c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef80039d0 0x7f0f1012f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43224/0 (socket says 172.21.15.116:43224) 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 0x7f0f1011c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:42876/0 (socket says 172.21.15.116:42876) 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 1 -- 172.21.15.116:0/2946722690 learned_addr learned my addr 172.21.15.116:0/2946722690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 10 monclient(hunting): get_auth_request con 0x7f0ef80039d0 auth_method 0 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 10 monclient(hunting): get_auth_request con 0x7f0ef8004780 auth_method 0 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.153+0000 7f0f0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 1 -- 172.21.15.116:0/2946722690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8005250 msgr2=0x7f0f1011c930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8005250 0x7f0f1011c930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 1 -- 172.21.15.116:0/2946722690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef80039d0 msgr2=0x7f0f1012f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef80039d0 0x7f0f1012f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 10 monclient: _finish_hunting 0 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 1 monclient: found mon.noname-b 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0effd640 1 -- 172.21.15.116:0/2946722690 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1011f900 con 0x7f0ef8004780 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.154+0000 7f0f0f7fe640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef80039d0 0x7f0f1012f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f0effd640 1 -- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 msgr2=0x7f0f1011c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f0effd640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 0x7f0f1011c3f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f04000ea0 tx=0x7f0f0400b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 1 -- 172.21.15.116:0/2946722690 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0400e050 con 0x7f0ef8004780 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: dump: 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.169 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 1 -- 172.21.15.116:0/2946722690 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0400f040 con 0x7f0ef8004780 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f15e34640 5 monclient: authenticate success, global_id 4351 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f15e34640 10 monclient: _renew_subs 2024-09-18T12:25:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f15e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1011fc40 con 0x7f0ef8004780 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 1 -- 172.21.15.116:0/2946722690 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04008b80 con 0x7f0ef8004780 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0ef77fe640 10 monclient: dump: 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f15e34640 10 monclient: _renew_subs 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f0dffb640 4 set_mon_vals no callback set 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f15e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.156+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f10130bc0 con 0x7f0ef8004780 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0ef77fe640 1 -- 172.21.15.116:0/2946722690 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0f04009660 con 0x7f0ef8004780 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0ef77fe640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0efc038d70 0x7f0efc03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0ef77fe640 1 -- 172.21.15.116:0/2946722690 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f0f0404b7d0 con 0x7f0ef8004780 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0f15e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0f15e34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0f15e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0f0f7fe640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0efc038d70 0x7f0efc03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.157+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1011de30 con 0x7f0ef8004780 2024-09-18T12:25:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.158+0000 7f0f0f7fe640 10 monclient: get_auth_request con 0x7f0efc038d70 auth_method 0 2024-09-18T12:25:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.160+0000 7f0f0f7fe640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0efc038d70 0x7f0efc03b220 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007a00 tx=0x7f0f00034000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.163+0000 7f0ef77fe640 1 -- 172.21.15.116:0/2946722690 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0f0401d460 con 0x7f0ef8004780 2024-09-18T12:25:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.163+0000 7f0ef77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.163+0000 7f0ef77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.380+0000 7f0f15e34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:25:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.380+0000 7f0f15e34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.380+0000 7f0f15e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:25:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.380+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0f1004bf60 con 0x7f0ef8004780 2024-09-18T12:25:50.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.380+0000 7f0ef77fe640 1 -- 172.21.15.116:0/2946722690 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1532 (secure 0 0 0) 0x7f0f0402b610 con 0x7f0ef8004780 2024-09-18T12:25:50.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.381+0000 7f0ef77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:25:50.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.381+0000 7f0ef77fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:50.382 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.382+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0efc038d70 msgr2=0x7f0efc03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.382+0000 7f0f15e34640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0efc038d70 0x7f0efc03b220 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007a00 tx=0x7f0f00034000 comp rx=0 tx=0).stop 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 10 monclient: shutdown 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 msgr2=0x7f0f1011c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 0x7f0f1011c3f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f04000ea0 tx=0x7f0f0400b680 comp rx=0 tx=0).stop 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 shutdown_connections 2024-09-18T12:25:50.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0efc038d70 0x7f0efc03b220 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0ef8005250 0x7f0f1011c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ef8004780 0x7f0f1011c3f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 --2- 172.21.15.116:0/2946722690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef80039d0 0x7f0f1012f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 >> 172.21.15.116:0/2946722690 conn(0x7f0f100672b0 msgr2=0x7f0f1006c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:50.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 shutdown_connections 2024-09-18T12:25:50.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.383+0000 7f0f15e34640 1 -- 172.21.15.116:0/2946722690 wait complete. 2024-09-18T12:25:50.394 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:25:50.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:25:50.395 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:25:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.488+0000 7ff6df46f640 10 monclient: get_monmap_and_config 2024-09-18T12:25:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.488+0000 7ff6df46f640 10 monclient: build_initial_monmap 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.488+0000 7ff6df46f640 10 monclient: monmap: 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:50.489444+0000 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:50.489444+0000 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:50.490 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 1 Processor -- start 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 1 -- start start 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient: init 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6d80975f0 0x7ff6d8095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d80975f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80979c0 0x7ff6d8097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d80979c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6d809bf50 addr v1:172.21.15.116:6789/0 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d809d700 con 0x7ff6d809bf50 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d809d1b0 con 0x7ff6d80975f0 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.489+0000 7ff6df46f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d809d420 con 0x7ff6d80979c0 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6df46f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6df46f640 10 monclient(hunting): authenticate will time out at 777.811012s 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dca0a640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80979c0 msgr2=0x7ff6d8097dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dca0a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80979c0 0x7ff6d8097dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6d80975f0 0x7ff6d8095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6d80975f0 0x7ff6d8095ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43236/0 (socket says 172.21.15.116:43236) 2024-09-18T12:25:50.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 -- 172.21.15.116:0/4240186465 learned_addr learned my addr 172.21.15.116:0/4240186465 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): get_auth_request con 0x7ff6d80975f0 auth_method 0 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dda0c640 1 --1- 172.21.15.116:0/4240186465 >> v1:172.21.15.116:6789/0 conn(0x7ff6d809bf50 0x7ff6d809c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37582/0 (socket says 172.21.15.116:37582) 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 -- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80979c0 msgr2=0x7ff6d8097dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:50.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 --2- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80979c0 0x7ff6d8097dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 -- 172.21.15.116:0/4240186465 >> v1:172.21.15.116:6789/0 conn(0x7ff6d809bf50 legacy=0x7ff6d809c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 monclient: found mon.noname-a 2024-09-18T12:25:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.490+0000 7ff6dd20b640 1 -- 172.21.15.116:0/4240186465 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d809d9e0 con 0x7ff6d80975f0 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6dd20b640 1 -- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 msgr2=0x7ff6d8095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6dd20b640 1 --2- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d8095ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009910 tx=0x7ff6d40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b7290a34f3ad85 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 1 -- 172.21.15.116:0/4240186465 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4032440 con 0x7ff6d80975f0 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: dump: 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: _finish_auth 0 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 1 -- 172.21.15.116:0/4240186465 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d40325a0 con 0x7ff6d80975f0 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 5 monclient: authenticate success, global_id 4407 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 4 set_mon_vals no callback set 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 1 -- 172.21.15.116:0/4240186465 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4032980 con 0x7ff6d80975f0 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6cffff640 10 monclient: dump: 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 10 monclient: shutdown 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 -- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 msgr2=0x7ff6d8095ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 --2- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d8095ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009910 tx=0x7ff6d40091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 -- 172.21.15.116:0/4240186465 shutdown_connections 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 --2- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d80979c0 0x7ff6d8097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 --2- 172.21.15.116:0/4240186465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d8095ad0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 -- 172.21.15.116:0/4240186465 >> 172.21.15.116:0/4240186465 conn(0x7ff6d80678a0 msgr2=0x7ff6d80a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 -- 172.21.15.116:0/4240186465 shutdown_connections 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.493+0000 7ff6df46f640 1 -- 172.21.15.116:0/4240186465 wait complete. 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.494+0000 7ff6df46f640 10 monclient: build_initial_monmap 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.494+0000 7ff6df46f640 10 monclient: monmap: 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.494+0000 7ff6df46f640 1 Processor -- start 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.494+0000 7ff6df46f640 1 -- start start 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.494+0000 7ff6df46f640 10 monclient: init 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d80975f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6d80979c0 0x7ff6d812fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d80979c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d809bf50 0x7ff6d8126850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d809bf50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8129f10 con 0x7ff6d80975f0 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d806eb80 con 0x7ff6d80979c0 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d81281d0 con 0x7ff6d809bf50 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6df46f640 10 monclient(hunting): authenticate will time out at 777.816363s 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dda0c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d809bf50 msgr2=0x7ff6d8126850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dda0c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d809bf50 0x7ff6d8126850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d813d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43242/0 (socket says 172.21.15.116:43242) 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 -- 172.21.15.116:0/1014084400 learned_addr learned my addr 172.21.15.116:0/1014084400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): get_auth_request con 0x7ff6d80975f0 auth_method 0 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dca0a640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6d80979c0 0x7ff6d812fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient(hunting): handle_auth_done global_id 4410 payload 824 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d809bf50 msgr2=0x7ff6d8126850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6d809bf50 0x7ff6d8126850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6d80979c0 msgr2=0x7ff6d812fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6d80979c0 0x7ff6d812fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient: _finish_hunting 0 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 monclient: found mon.noname-a 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dd20b640 1 -- 172.21.15.116:0/1014084400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009450 con 0x7ff6d80975f0 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dca0a640 10 monclient: get_auth_request con 0x7ff6d80979c0 auth_method 0 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.495+0000 7ff6dca0a640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6d80979c0 0x7ff6d812fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.497+0000 7ff6dd20b640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 msgr2=0x7ff6d813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.497+0000 7ff6dd20b640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d813d3f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009910 tx=0x7ff6d4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6dd20b640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d813d3f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009910 tx=0x7ff6d4032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7ff6d403ca40 mon_map magic: 0 v1 2024-09-18T12:25:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d403ca40 con 0x7ff6d80975f0 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 10 monclient: dump: 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.518 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 10 monclient: _finish_auth 0 2024-09-18T12:25:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d403cba0 con 0x7ff6d80975f0 2024-09-18T12:25:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 5 monclient: authenticate success, global_id 4410 2024-09-18T12:25:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 10 monclient: _renew_subs 2024-09-18T12:25:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d812a4d0 con 0x7ff6d80975f0 2024-09-18T12:25:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6cf7fe640 4 set_mon_vals no callback set 2024-09-18T12:25:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 10 monclient: _renew_subs 2024-09-18T12:25:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d812ff80 con 0x7ff6d80975f0 2024-09-18T12:25:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.498+0000 7ff6df46f640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 0x7ff6d8126d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6df46f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8065e00 con 0x7ff6d8130200 2024-09-18T12:25:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40313a0 con 0x7ff6d80975f0 2024-09-18T12:25:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6cd7fa640 10 monclient: dump: 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.525 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff6d4039040 con 0x7ff6d80975f0 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 0x7ff6d8126d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient: get_auth_request con 0x7ff6d8130200 auth_method 0 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6cd7fa640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6bc038490 0x7ff6bc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dca0a640 1 -- 172.21.15.116:0/1014084400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6d812a150 con 0x7ff6d8130200 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dda0c640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6bc038490 0x7ff6bc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.499+0000 7ff6dda0c640 10 monclient: get_auth_request con 0x7ff6bc038490 auth_method 0 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6dca0a640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 msgr2=0x7ff6d8126d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6dca0a640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 0x7ff6d8126d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c700 tx=0x7ff6c8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8003d90 con 0x7ff6d8130200 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 10 monclient: dump: 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6dda0c640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6bc038490 0x7ff6bc03a940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0004c50 tx=0x7ff6d0007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff6c8006150 con 0x7ff6d8130200 2024-09-18T12:25:50.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 msgr2=0x7ff6d8126d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 0x7ff6d8126d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c700 tx=0x7ff6c8008040 comp rx=0 tx=0).stop 2024-09-18T12:25:50.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 msgr2=0x7ff6d8126d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:50.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.501+0000 7ff6cd7fa640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d8130200 0x7ff6d8126d90 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c700 tx=0x7ff6c8008040 comp rx=0 tx=0).stop 2024-09-18T12:25:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6df46f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:25:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6df46f640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 0x7ff6d81276d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6df46f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8127c10 con 0x7ff6d81272d0 2024-09-18T12:25:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 0x7ff6d81276d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient: get_auth_request con 0x7ff6d81272d0 auth_method 0 2024-09-18T12:25:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.507+0000 7ff6dca0a640 1 -- 172.21.15.116:0/1014084400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6d804bf60 con 0x7ff6d81272d0 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6dca0a640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 msgr2=0x7ff6d81276d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6dca0a640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 0x7ff6d81276d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c700 tx=0x7ff6c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8024070 con 0x7ff6d81272d0 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 10 monclient: dump: 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.534 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7ff6c8013e70 con 0x7ff6d81272d0 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 msgr2=0x7ff6d81276d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 0x7ff6d81276d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c700 tx=0x7ff6c8017000 comp rx=0 tx=0).stop 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 msgr2=0x7ff6d81276d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d81272d0 0x7ff6d81276d0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c700 tx=0x7ff6c8017000 comp rx=0 tx=0).stop 2024-09-18T12:25:50.535 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 8, 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:25:50.536 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:50.540 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:25:50.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.542 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:25:50.543 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:50.544 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:25:50.545 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:50.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:25:50.548 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.548 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:50.548 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:25:50.548 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:50.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:50.550 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:25:50.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.553 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:25:50.553 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:25:50.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6bc038490 msgr2=0x7ff6bc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6bc038490 0x7ff6bc03a940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0004c50 tx=0x7ff6d0007e40 comp rx=0 tx=0).stop 2024-09-18T12:25:50.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 10 monclient: shutdown 2024-09-18T12:25:50.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:50.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 msgr2=0x7ff6d813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 --2- 172.21.15.116:0/1014084400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6d80975f0 0x7ff6d813d3f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009910 tx=0x7ff6d4032000 comp rx=0 tx=0).stop 2024-09-18T12:25:50.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6dda0c640 1 -- 172.21.15.116:0/1014084400 reap_dead start 2024-09-18T12:25:50.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6cd7fa640 1 -- 172.21.15.116:0/1014084400 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (unknown 0 0 0) 0x7ff6d406b7f0 con 0x7ff6d80975f0 2024-09-18T12:25:50.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6dda0c640 1 -- 172.21.15.116:0/1014084400 reap_dead start 2024-09-18T12:25:50.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 shutdown_connections 2024-09-18T12:25:50.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 >> 172.21.15.116:0/1014084400 conn(0x7ff6d80678a0 msgr2=0x7ff6d8096340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:50.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 shutdown_connections 2024-09-18T12:25:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.510+0000 7ff6df46f640 1 -- 172.21.15.116:0/1014084400 wait complete. 2024-09-18T12:25:50.557 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:25:50.557 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:25:50.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:25:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.651+0000 7fd62619d640 10 monclient: get_monmap_and_config 2024-09-18T12:25:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.651+0000 7fd62619d640 10 monclient: build_initial_monmap 2024-09-18T12:25:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.651+0000 7fd62619d640 10 monclient: monmap: 2024-09-18T12:25:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:25:50.652189+0000 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:25:50.652189+0000 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:25:50.653 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 1 Processor -- start 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 1 -- start start 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient: init 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd62006eb80 addr v1:172.21.15.144:6789/0 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd620068a20 0x7fd62009c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient(hunting): picked mon.noname-a con 0x7fd620068a20 addr v2:172.21.15.116:3300/0 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 0x7fd620138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient(hunting): picked mon.noname-b con 0x7fd62009ca80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd620069b80 con 0x7fd62006eb80 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd620099f90 con 0x7fd620068a20 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd620069e00 con 0x7fd62009ca80 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd62619d640 10 monclient(hunting): authenticate will time out at 777.973907s 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd61ffff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 msgr2=0x7fd620138ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.652+0000 7fd61ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 0x7fd620138ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd620068a20 0x7fd62009c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd620068a20 0x7fd62009c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43252/0 (socket says 172.21.15.116:43252) 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 -- 172.21.15.116:0/578264678 learned_addr learned my addr 172.21.15.116:0/578264678 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): get_auth_request con 0x7fd620068a20 auth_method 0 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient(hunting): handle_auth_done global_id 4419 payload 274 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 -- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 msgr2=0x7fd620138ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 --2- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 0x7fd620138ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 -- 172.21.15.116:0/578264678 >> v1:172.21.15.144:6789/0 conn(0x7fd62006eb80 legacy=0x7fd62006ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 monclient: found mon.noname-a 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.653+0000 7fd617fff640 1 -- 172.21.15.116:0/578264678 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62006f660 con 0x7fd620068a20 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd617fff640 1 -- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620068a20 msgr2=0x7fd62009c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd617fff640 1 --2- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620068a20 0x7fd62009c540 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd608009910 tx=0x7fd6080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4966c972d236fb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 1 -- 172.21.15.116:0/578264678 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608032440 con 0x7fd620068a20 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: dump: 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: _finish_auth 0 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 1 -- 172.21.15.116:0/578264678 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6080325a0 con 0x7fd620068a20 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 5 monclient: authenticate success, global_id 4419 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:25:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 10 monclient: get_monmap_and_config success 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 1 -- 172.21.15.116:0/578264678 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608032980 con 0x7fd620068a20 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 4 set_mon_vals no callback set 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd61effd640 10 monclient: dump: 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 10 monclient: shutdown 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 -- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620068a20 msgr2=0x7fd62009c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 --2- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620068a20 0x7fd62009c540 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd608009910 tx=0x7fd6080091d0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 -- 172.21.15.116:0/578264678 shutdown_connections 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 --2- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 0x7fd620138ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 --2- 172.21.15.116:0/578264678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620068a20 0x7fd62009c540 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 -- 172.21.15.116:0/578264678 >> 172.21.15.116:0/578264678 conn(0x7fd620067cd0 msgr2=0x7fd6200a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 -- 172.21.15.116:0/578264678 shutdown_connections 2024-09-18T12:25:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.656+0000 7fd62619d640 1 -- 172.21.15.116:0/578264678 wait complete. 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.657+0000 7fd62619d640 10 monclient: build_initial_monmap 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.657+0000 7fd62619d640 10 monclient: monmap: 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.657+0000 7fd62619d640 1 Processor -- start 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.657+0000 7fd62619d640 1 -- start start 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.657+0000 7fd62619d640 10 monclient: init 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd620068a20 0x7fd62011e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): picked mon.noname-b con 0x7fd620068a20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62006eb80 0x7fd62011ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): picked mon.noname-a con 0x7fd62006eb80 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 0x7fd62011f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): picked mon.noname-c con 0x7fd62009ca80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62009cf40 con 0x7fd62006eb80 2024-09-18T12:25:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62009b6e0 con 0x7fd620068a20 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd620121f50 con 0x7fd62009ca80 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd62619d640 10 monclient(hunting): authenticate will time out at 777.979091s 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61ffff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 msgr2=0x7fd62011f170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 0x7fd62011f170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62006eb80 0x7fd62011ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62006eb80 0x7fd62011ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43258/0 (socket says 172.21.15.116:43258) 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 -- 172.21.15.116:0/1194367131 learned_addr learned my addr 172.21.15.116:0/1194367131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): get_auth_request con 0x7fd62006eb80 auth_method 0 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd620068a20 0x7fd62011e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 10 monclient(hunting): get_auth_request con 0x7fd620068a20 auth_method 0 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 msgr2=0x7fd62011f170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd62009ca80 0x7fd62011f170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd620068a20 msgr2=0x7fd62011e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd620068a20 0x7fd62011e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient: _finish_hunting 0 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 monclient: found mon.noname-a 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.658+0000 7fd617fff640 1 -- 172.21.15.116:0/1194367131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd608009450 con 0x7fd62006eb80 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd617fff640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62006eb80 msgr2=0x7fd62011ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd617fff640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62006eb80 0x7fd62011ec30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd608035040 tx=0x7fd608032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60803c960 con 0x7fd62006eb80 2024-09-18T12:25:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: dump: 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: _finish_auth 0 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd62619d640 5 monclient: authenticate success, global_id 4422 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60803cac0 con 0x7fd62006eb80 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd62619d640 10 monclient: _renew_subs 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd62619d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd620120e20 con 0x7fd62006eb80 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60803ce60 con 0x7fd62006eb80 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd6177fe640 10 monclient: dump: 2024-09-18T12:25:50.676 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd62619d640 10 monclient: _renew_subs 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd62619d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd620138d00 con 0x7fd62006eb80 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.660+0000 7fd61e7fc640 4 set_mon_vals no callback set 2024-09-18T12:25:50.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd62619d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd62619d640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 0x7fd62006b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd62619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201223c0 con 0x7fd62011f6b0 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd608039040 con 0x7fd62006eb80 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 0x7fd62006b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 10 monclient: get_auth_request con 0x7fd62011f6b0 auth_method 0 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd6177fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd600038440 0x7fd60003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fd60806b6f0 con 0x7fd62006eb80 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.661+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.662+0000 7fd61ffff640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd600038440 0x7fd60003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.662+0000 7fd61ffff640 10 monclient: get_auth_request con 0x7fd600038440 auth_method 0 2024-09-18T12:25:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.662+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2024-09-18T12:25:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.662+0000 7fd61f7fe640 1 -- 172.21.15.116:0/1194367131 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd620121d80 con 0x7fd62011f6b0 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd61f7fe640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 msgr2=0x7fd62006b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 0x7fd62006b840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd61000f780 tx=0x7fd610007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61000d8b0 con 0x7fd62011f6b0 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 10 monclient: dump: 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd61ffff640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd600038440 0x7fd60003a8f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd618003cc0 tx=0x7fd618009370 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd61000da10 con 0x7fd62011f6b0 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:25:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 msgr2=0x7fd62006b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 0x7fd62006b840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd61000f780 tx=0x7fd610007e10 comp rx=0 tx=0).stop 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 msgr2=0x7fd62006b840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.664+0000 7fd6177fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62011f6b0 0x7fd62006b840 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd62619d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd62619d640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 0x7fd62006c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd62619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd620065e00 con 0x7fd62006bd80 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 0x7fd62006c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd61f7fe640 10 monclient: get_auth_request con 0x7fd62006bd80 auth_method 0 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd61f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:25:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd61f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.670+0000 7fd61f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.671+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.671+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.671+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.671+0000 7fd61f7fe640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.671+0000 7fd61f7fe640 1 -- 172.21.15.116:0/1194367131 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd62004bf60 con 0x7fd62006bd80 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd61f7fe640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 msgr2=0x7fd62006c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 0x7fd62006c180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd61000f780 tx=0x7fd610019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610014ea0 con 0x7fd62006bd80 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 10 monclient: dump: 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:25:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4735 (secure 0 0 0) 0x7fd61000fd20 con 0x7fd62006bd80 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 msgr2=0x7fd62006c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 0x7fd62006c180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd61000f780 tx=0x7fd610019000 comp rx=0 tx=0).stop 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 msgr2=0x7fd62006c180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.673+0000 7fd6177fe640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd62006bd80 0x7fd62006c180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 8, 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:25:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:25:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:25:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:25:50.693 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:50.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:25:50.696 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:25:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:25:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:25:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:25:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:25:50.700 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:25:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd600038440 msgr2=0x7fd60003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd600038440 0x7fd60003a8f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd618003cc0 tx=0x7fd618009370 comp rx=0 tx=0).stop 2024-09-18T12:25:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 10 monclient: shutdown 2024-09-18T12:25:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:25:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62006eb80 msgr2=0x7fd62011ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:25:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 --2- 172.21.15.116:0/1194367131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62006eb80 0x7fd62011ec30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd608035040 tx=0x7fd608032000 comp rx=0 tx=0).stop 2024-09-18T12:25:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd61ffff640 1 -- 172.21.15.116:0/1194367131 reap_dead start 2024-09-18T12:25:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd61ffff640 1 -- 172.21.15.116:0/1194367131 reap_dead start 2024-09-18T12:25:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 shutdown_connections 2024-09-18T12:25:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 >> 172.21.15.116:0/1194367131 conn(0x7fd620067cd0 msgr2=0x7fd6200a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:25:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 shutdown_connections 2024-09-18T12:25:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:25:50.674+0000 7fd62619d640 1 -- 172.21.15.116:0/1194367131 wait complete. 2024-09-18T12:25:50.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:25:50.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:26:10.703 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:26:10.703 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:26:10.703 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:26:10.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:26:10.707 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:26:10.707 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:26:10.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.811+0000 7fe987457640 10 monclient: get_monmap_and_config 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.811+0000 7fe987457640 10 monclient: build_initial_monmap 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.811+0000 7fe987457640 10 monclient: monmap: 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:10.812110+0000 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:10.812110+0000 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:10.813 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:10.814 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 1 Processor -- start 2024-09-18T12:26:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 1 -- start start 2024-09-18T12:26:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient: init 2024-09-18T12:26:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe98009ec60 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe980069da0 0x7fe98009f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient(hunting): picked mon.noname-c con 0x7fe980069da0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 0x7fe98013d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient(hunting): picked mon.noname-b con 0x7fe98013b080 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9800a14b0 con 0x7fe98009ec60 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe98009c820 con 0x7fe980069da0 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98009d8b0 con 0x7fe98013b080 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe987457640 10 monclient(hunting): authenticate will time out at 798.133866s 2024-09-18T12:26:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.812+0000 7fe9859f4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 0x7fe98013d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 0x7fe98013d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39410/0 (socket says 172.21.15.116:39410) 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe980069da0 0x7fe98009f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 1 -- 172.21.15.116:0/3106870785 learned_addr learned my addr 172.21.15.116:0/3106870785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 10 monclient(hunting): get_auth_request con 0x7fe98013b080 auth_method 0 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 10 monclient(hunting): get_auth_request con 0x7fe980069da0 auth_method 0 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 1 -- 172.21.15.116:0/3106870785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 msgr2=0x7fe98013d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 1 -- 172.21.15.116:0/3106870785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 msgr2=0x7fe98013d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 1 --2- 172.21.15.116:0/3106870785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 0x7fe98013d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9859f4640 1 --2- 172.21.15.116:0/3106870785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 0x7fe98013d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.813+0000 7fe9849f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9851f3640 1 -- 172.21.15.116:0/3106870785 >> v1:172.21.15.144:6789/0 conn(0x7fe98009ec60 legacy=0x7fe98009d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 1 -- 172.21.15.116:0/3106870785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 msgr2=0x7fe98013d460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 1 --2- 172.21.15.116:0/3106870785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 0x7fe98013d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 1 -- 172.21.15.116:0/3106870785 >> v1:172.21.15.144:6789/0 conn(0x7fe98009ec60 legacy=0x7fe98009d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 10 monclient: _finish_hunting 0 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 1 monclient: found mon.noname-c 2024-09-18T12:26:10.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:10.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:10.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.815+0000 7fe9849f2640 1 -- 172.21.15.116:0/3106870785 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98009f570 con 0x7fe980069da0 2024-09-18T12:26:10.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe9849f2640 1 -- 172.21.15.116:0/3106870785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe980069da0 msgr2=0x7fe98009f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:10.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe9849f2640 1 --2- 172.21.15.116:0/3106870785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe980069da0 0x7fe98009f030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe970009900 tx=0x7fe9700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24aa98a7e56e7fd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:10.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 1 -- 172.21.15.116:0/3106870785 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe970032440 con 0x7fe980069da0 2024-09-18T12:26:10.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:10.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: dump: 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: _finish_auth 0 2024-09-18T12:26:10.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 5 monclient: authenticate success, global_id 4363 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 1 -- 172.21.15.116:0/3106870785 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9700325a0 con 0x7fe980069da0 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 4 set_mon_vals no callback set 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 1 -- 172.21.15.116:0/3106870785 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe970032980 con 0x7fe980069da0 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:10.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe977fff640 10 monclient: dump: 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 10 monclient: shutdown 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:10.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 1 -- 172.21.15.116:0/3106870785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe980069da0 msgr2=0x7fe98009f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.818+0000 7fe987457640 1 --2- 172.21.15.116:0/3106870785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe980069da0 0x7fe98009f030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe970009900 tx=0x7fe9700090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 -- 172.21.15.116:0/3106870785 shutdown_connections 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 --2- 172.21.15.116:0/3106870785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98013b080 0x7fe98013d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 --2- 172.21.15.116:0/3106870785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe980069da0 0x7fe98009f030 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 -- 172.21.15.116:0/3106870785 >> 172.21.15.116:0/3106870785 conn(0x7fe980067110 msgr2=0x7fe980067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 -- 172.21.15.116:0/3106870785 shutdown_connections 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 -- 172.21.15.116:0/3106870785 wait complete. 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 10 monclient: build_initial_monmap 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 10 monclient: monmap: 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:10.829 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 Processor -- start 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 1 -- start start 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.819+0000 7fe987457640 10 monclient: init 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:10.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 0x7fe9800986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): picked mon.noname-a con 0x7fe980069da0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 0x7fe980098c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): picked mon.noname-c con 0x7fe98009ec60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe98013b080 0x7fe980099170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): picked mon.noname-b con 0x7fe98013b080 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe980124390 con 0x7fe980069da0 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98009aaa0 con 0x7fe98013b080 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98009a5b0 con 0x7fe98009ec60 2024-09-18T12:26:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe987457640 10 monclient(hunting): authenticate will time out at 798.141525s 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 0x7fe9800986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 0x7fe980098c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 0x7fe980098c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39412/0 (socket says 172.21.15.116:39412) 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 1 -- 172.21.15.116:0/2639307978 learned_addr learned my addr 172.21.15.116:0/2639307978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 0x7fe9800986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35834/0 (socket says 172.21.15.116:35834) 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 10 monclient(hunting): get_auth_request con 0x7fe98009ec60 auth_method 0 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): get_auth_request con 0x7fe980069da0 auth_method 0 2024-09-18T12:26:10.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe98013b080 0x7fe980099170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 10 monclient(hunting): get_auth_request con 0x7fe98013b080 auth_method 0 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 1 -- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 msgr2=0x7fe980098c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 1 -- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 msgr2=0x7fe980098c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 0x7fe980098c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9849f2640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 0x7fe980098c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:10.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9859f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 1 -- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 msgr2=0x7fe980098c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 0x7fe980098c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 1 -- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe98013b080 msgr2=0x7fe980099170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe98013b080 0x7fe980099170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 10 monclient: _finish_hunting 0 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.820+0000 7fe9851f3640 1 monclient: found mon.noname-a 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.821+0000 7fe9851f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.821+0000 7fe9851f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:10.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.821+0000 7fe9851f3640 1 -- 172.21.15.116:0/2639307978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe970009390 con 0x7fe980069da0 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9851f3640 1 -- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 msgr2=0x7fe9800986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9851f3640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 0x7fe9800986f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c590 tx=0x7fe97c007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 1 -- 172.21.15.116:0/2639307978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0074f0 con 0x7fe980069da0 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: dump: 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: _finish_auth 0 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 1 -- 172.21.15.116:0/2639307978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c005780 con 0x7fe980069da0 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe987457640 5 monclient: authenticate success, global_id 4425 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe987457640 10 monclient: _renew_subs 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe987457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9801240b0 con 0x7fe980069da0 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 1 -- 172.21.15.116:0/2639307978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c005b40 con 0x7fe980069da0 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:10.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9757fa640 10 monclient: dump: 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe9777fe640 4 set_mon_vals no callback set 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe987457640 10 monclient: _renew_subs 2024-09-18T12:26:10.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe987457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.823+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9800996b0 con 0x7fe980069da0 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe9757fa640 1 -- 172.21.15.116:0/2639307978 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe97c01b3b0 con 0x7fe980069da0 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe987457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe987457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe987457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98004bf60 con 0x7fe980069da0 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe9757fa640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe964038490 0x7fe96403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe9757fa640 1 -- 172.21.15.116:0/2639307978 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe97c04aa20 con 0x7fe980069da0 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe9849f2640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe964038490 0x7fe96403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:10.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.824+0000 7fe9849f2640 10 monclient: get_auth_request con 0x7fe964038490 auth_method 0 2024-09-18T12:26:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.827+0000 7fe9849f2640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe964038490 0x7fe96403a940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe970035040 tx=0x7fe970032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.829+0000 7fe9757fa640 1 -- 172.21.15.116:0/2639307978 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe97c018820 con 0x7fe980069da0 2024-09-18T12:26:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.829+0000 7fe9757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:10.829+0000 7fe9757fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:11.046+0000 7fe987457640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:11.046+0000 7fe987457640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:11.046+0000 7fe987457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:11.046+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe9800a1120 con 0x7fe980069da0 2024-09-18T12:26:13.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:13.820+0000 7fe96bfff640 10 monclient: tick 2024-09-18T12:26:13.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:13.820+0000 7fe96bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:13.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:13.820+0000 7fe96bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:14.819+0000 7fe96bfff640 10 monclient: tick 2024-09-18T12:26:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:14.819+0000 7fe96bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:15.820+0000 7fe96bfff640 10 monclient: tick 2024-09-18T12:26:15.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:15.820+0000 7fe96bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:16.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.027+0000 7fe9757fa640 1 -- 172.21.15.116:0/2639307978 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fe97c017020 con 0x7fe980069da0 2024-09-18T12:26:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.027+0000 7fe9757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:16.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.027+0000 7fe9757fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:16.030 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":12,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:26:16.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe964038490 msgr2=0x7fe96403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:16.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe964038490 0x7fe96403a940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe970035040 tx=0x7fe970032000 comp rx=0 tx=0).stop 2024-09-18T12:26:16.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 10 monclient: shutdown 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 msgr2=0x7fe9800986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 0x7fe9800986f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c590 tx=0x7fe97c007cb0 comp rx=0 tx=0).stop 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 shutdown_connections 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe964038490 0x7fe96403a940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe98013b080 0x7fe980099170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe98009ec60 0x7fe980098c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 --2- 172.21.15.116:0/2639307978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe980069da0 0x7fe9800986f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 >> 172.21.15.116:0/2639307978 conn(0x7fe980067110 msgr2=0x7fe98013ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 shutdown_connections 2024-09-18T12:26:16.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:16.030+0000 7fe987457640 1 -- 172.21.15.116:0/2639307978 wait complete. 2024-09-18T12:26:19.047 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:19.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.149+0000 7fbc9520f640 10 monclient: get_monmap_and_config 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.149+0000 7fbc9520f640 10 monclient: build_initial_monmap 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.149+0000 7fbc9520f640 10 monclient: monmap: 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:19.151619+0000 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:19.151619+0000 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:19.152 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:19.153 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:19.153 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:19.153 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.150+0000 7fbc9520f640 1 Processor -- start 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.150+0000 7fbc9520f640 1 -- start start 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.150+0000 7fbc9520f640 10 monclient: init 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc900a1ef0 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc9009c6a0 0x7fbc9009b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9009c6a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc90136d00 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9006f010 con 0x7fbc90136d00 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9009dcc0 con 0x7fbc900a1ef0 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9009da00 con 0x7fbc9009c6a0 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc9520f640 10 monclient(hunting): authenticate will time out at 806.473201s 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8f59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbc90136d00 0x7fbc901390e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47694/0 (socket says 172.21.15.116:47694) 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8f59e640 1 -- 172.21.15.116:0/201433547 learned_addr learned my addr 172.21.15.116:0/201433547 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 1 --2- 172.21.15.116:0/201433547 >> v2:172.21.15.144:3300/0 conn(0x7fbc9009c6a0 0x7fbc9009b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request con 0x7fbc9009c6a0 auth_method 0 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 1 -- 172.21.15.116:0/201433547 >> v1:172.21.15.144:6789/0 conn(0x7fbc900a1ef0 legacy=0x7fbc9009abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 1 -- 172.21.15.116:0/201433547 >> v1:172.21.15.116:6789/0 conn(0x7fbc90136d00 legacy=0x7fbc901390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:26:19.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 1 monclient: found mon.noname-c 2024-09-18T12:26:19.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:19.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:19.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.151+0000 7fbc8e59c640 1 -- 172.21.15.116:0/201433547 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc900a22c0 con 0x7fbc9009c6a0 2024-09-18T12:26:19.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.154+0000 7fbc8e59c640 1 -- 172.21.15.116:0/201433547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc9009c6a0 msgr2=0x7fbc9009b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.154+0000 7fbc8e59c640 1 --2- 172.21.15.116:0/201433547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc9009c6a0 0x7fbc9009b2e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc900645f0 tx=0x7fbc780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc9ada9702caef3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.154+0000 7fbc8dd9b640 1 -- 172.21.15.116:0/201433547 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78032440 con 0x7fbc9009c6a0 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.154+0000 7fbc8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.154+0000 7fbc8dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.154+0000 7fbc8dd9b640 10 monclient: dump: 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.165 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 1 -- 172.21.15.116:0/201433547 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc780325a0 con 0x7fbc9009c6a0 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 5 monclient: authenticate success, global_id 4369 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 1 -- 172.21.15.116:0/201433547 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78032940 con 0x7fbc9009c6a0 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 4 set_mon_vals no callback set 2024-09-18T12:26:19.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc8dd9b640 10 monclient: dump: 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 10 monclient: shutdown 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 1 -- 172.21.15.116:0/201433547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc9009c6a0 msgr2=0x7fbc9009b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 1 --2- 172.21.15.116:0/201433547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc9009c6a0 0x7fbc9009b2e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc900645f0 tx=0x7fbc780090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 1 -- 172.21.15.116:0/201433547 shutdown_connections 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 1 --2- 172.21.15.116:0/201433547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc9009c6a0 0x7fbc9009b2e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 1 -- 172.21.15.116:0/201433547 >> 172.21.15.116:0/201433547 conn(0x7fbc900678a0 msgr2=0x7fbc900a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 1 -- 172.21.15.116:0/201433547 shutdown_connections 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.155+0000 7fbc9520f640 1 -- 172.21.15.116:0/201433547 wait complete. 2024-09-18T12:26:19.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.156+0000 7fbc9520f640 10 monclient: build_initial_monmap 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.156+0000 7fbc9520f640 10 monclient: monmap: 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.156+0000 7fbc9520f640 1 Processor -- start 2024-09-18T12:26:19.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.156+0000 7fbc9520f640 1 -- start start 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.156+0000 7fbc9520f640 10 monclient: init 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 0x7fbc901357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9009c6a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc900a1ef0 0x7fbc901318d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): picked mon.noname-b con 0x7fbc900a1ef0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc90136d00 0x7fbc90131e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): picked mon.noname-a con 0x7fbc90136d00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc90133730 con 0x7fbc90136d00 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9009b910 con 0x7fbc900a1ef0 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9013a2d0 con 0x7fbc9009c6a0 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc9520f640 10 monclient(hunting): authenticate will time out at 806.479183s 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 0x7fbc901357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc90136d00 0x7fbc90131e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 0x7fbc901357a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36646/0 (socket says 172.21.15.116:36646) 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 1 -- 172.21.15.116:0/3539464161 learned_addr learned my addr 172.21.15.116:0/3539464161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 10 monclient(hunting): get_auth_request con 0x7fbc9009c6a0 auth_method 0 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): get_auth_request con 0x7fbc90136d00 auth_method 0 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc900a1ef0 0x7fbc901318d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request con 0x7fbc900a1ef0 auth_method 0 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 1 -- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 msgr2=0x7fbc901357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 1 -- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 msgr2=0x7fbc901357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 0x7fbc901357a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8ed9d640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 0x7fbc901357a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 1 -- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 msgr2=0x7fbc901357a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 0x7fbc901357a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 1 -- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc900a1ef0 msgr2=0x7fbc901318d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc900a1ef0 0x7fbc901318d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient: _finish_hunting 0 2024-09-18T12:26:19.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 1 monclient: found mon.noname-a 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.157+0000 7fbc8f59e640 1 -- 172.21.15.116:0/3539464161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78009390 con 0x7fbc90136d00 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc8f59e640 1 -- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc90136d00 msgr2=0x7fbc90131e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc8f59e640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc90136d00 0x7fbc90131e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c490 tx=0x7fbc80007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 1 -- 172.21.15.116:0/3539464161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80005cb0 con 0x7fbc90136d00 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 10 monclient: dump: 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.174 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 1 -- 172.21.15.116:0/3539464161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc80005e10 con 0x7fbc90136d00 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc9520f640 5 monclient: authenticate success, global_id 4428 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc9520f640 10 monclient: _renew_subs 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc9520f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc90134a30 con 0x7fbc90136d00 2024-09-18T12:26:19.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.159+0000 7fbc86ffd640 1 -- 172.21.15.116:0/3539464161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80007320 con 0x7fbc90136d00 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc86ffd640 10 monclient: dump: 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc8d59a640 4 set_mon_vals no callback set 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc9520f640 10 monclient: _renew_subs 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc9520f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc90135ce0 con 0x7fbc90136d00 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc86ffd640 1 -- 172.21.15.116:0/3539464161 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbc8001b3b0 con 0x7fbc90136d00 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc9520f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc9520f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc9520f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.160+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc90135f30 con 0x7fbc90136d00 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.161+0000 7fbc86ffd640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc70038490 0x7fbc7003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.161+0000 7fbc86ffd640 1 -- 172.21.15.116:0/3539464161 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fbc8004aaa0 con 0x7fbc90136d00 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.161+0000 7fbc8ed9d640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc70038490 0x7fbc7003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.161+0000 7fbc8ed9d640 10 monclient: get_auth_request con 0x7fbc70038490 auth_method 0 2024-09-18T12:26:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.163+0000 7fbc8ed9d640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc70038490 0x7fbc7003a940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc880024d0 tx=0x7fbc88009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.165+0000 7fbc86ffd640 1 -- 172.21.15.116:0/3539464161 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbc90135f30 con 0x7fbc90136d00 2024-09-18T12:26:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.165+0000 7fbc86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.165+0000 7fbc86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.376+0000 7fbc9520f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.376+0000 7fbc9520f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.376+0000 7fbc9520f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.376+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbc90135f30 con 0x7fbc90136d00 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.377+0000 7fbc86ffd640 1 -- 172.21.15.116:0/3539464161 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fbc90135f30 con 0x7fbc90136d00 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.377+0000 7fbc86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.377+0000 7fbc86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:19.379 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:26:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc70038490 msgr2=0x7fbc7003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc70038490 0x7fbc7003a940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc880024d0 tx=0x7fbc88009040 comp rx=0 tx=0).stop 2024-09-18T12:26:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 10 monclient: shutdown 2024-09-18T12:26:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc90136d00 msgr2=0x7fbc90131e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc90136d00 0x7fbc90131e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c490 tx=0x7fbc80007ee0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 shutdown_connections 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc70038490 0x7fbc7003a940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc90136d00 0x7fbc90131e10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc900a1ef0 0x7fbc901318d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 --2- 172.21.15.116:0/3539464161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc9009c6a0 0x7fbc901357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 >> 172.21.15.116:0/3539464161 conn(0x7fbc900678a0 msgr2=0x7fbc9009f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 shutdown_connections 2024-09-18T12:26:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.379+0000 7fbc9520f640 1 -- 172.21.15.116:0/3539464161 wait complete. 2024-09-18T12:26:19.391 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:26:19.391 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:26:19.391 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:26:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.482+0000 7f4499d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.482+0000 7f4499d1d640 10 monclient: build_initial_monmap 2024-09-18T12:26:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.482+0000 7f4499d1d640 10 monclient: monmap: 2024-09-18T12:26:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:19.484228+0000 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:19.484228+0000 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:19.485 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 1 Processor -- start 2024-09-18T12:26:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 1 -- start start 2024-09-18T12:26:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient: init 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44941318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f449409ea50 0x7f449409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f449409ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 0x7f4494139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f4494136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4494132640 con 0x7f44941318f0 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4494131cc0 con 0x7f449409ea50 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449409fbf0 con 0x7f4494136c20 2024-09-18T12:26:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4499d1d640 10 monclient(hunting): authenticate will time out at 806.805795s 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4492ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f449409ea50 0x7f449409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4493fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 0x7f4494139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4492ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f449409ea50 0x7f449409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42004/0 (socket says 172.21.15.116:42004) 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4493fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 0x7f4494139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36660/0 (socket says 172.21.15.116:36660) 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4492ffd640 1 -- 172.21.15.116:0/1265827771 learned_addr learned my addr 172.21.15.116:0/1265827771 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4492ffd640 10 monclient(hunting): get_auth_request con 0x7f449409ea50 auth_method 0 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4493fff640 10 monclient(hunting): get_auth_request con 0x7f4494136c20 auth_method 0 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4493fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4493fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.483+0000 7f4493fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4493fff640 1 -- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 msgr2=0x7f4494139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4493fff640 1 -- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 msgr2=0x7f4494139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4493fff640 1 --2- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 0x7f4494139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4493fff640 1 --2- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 0x7f4494139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 10 monclient(hunting): handle_auth_done global_id 4431 payload 274 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 1 -- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 msgr2=0x7f4494139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 1 --2- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 0x7f4494139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 1 -- 172.21.15.116:0/1265827771 >> v1:172.21.15.144:6789/0 conn(0x7f44941318f0 legacy=0x7f4494131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:26:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 1 monclient: found mon.noname-a 2024-09-18T12:26:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.484+0000 7f4492ffd640 1 -- 172.21.15.116:0/1265827771 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449409fe30 con 0x7f449409ea50 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.486+0000 7f4492ffd640 1 -- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449409ea50 msgr2=0x7f449409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.486+0000 7f4492ffd640 1 --2- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449409ea50 0x7f449409ee20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f447c009910 tx=0x7f447c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c93b6f9795541a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 1 -- 172.21.15.116:0/1265827771 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c02ae90 con 0x7f449409ea50 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: dump: 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 1 -- 172.21.15.116:0/1265827771 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c032440 con 0x7f449409ea50 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 5 monclient: authenticate success, global_id 4431 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 4 set_mon_vals no callback set 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 1 -- 172.21.15.116:0/1265827771 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c032820 con 0x7f449409ea50 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f44927fc640 10 monclient: dump: 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 10 monclient: shutdown 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 -- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449409ea50 msgr2=0x7f449409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 --2- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449409ea50 0x7f449409ee20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f447c009910 tx=0x7f447c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 -- 172.21.15.116:0/1265827771 shutdown_connections 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 --2- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4494136c20 0x7f4494139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 --2- 172.21.15.116:0/1265827771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f449409ea50 0x7f449409ee20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 -- 172.21.15.116:0/1265827771 >> 172.21.15.116:0/1265827771 conn(0x7f4494067110 msgr2=0x7f4494067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 -- 172.21.15.116:0/1265827771 shutdown_connections 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.487+0000 7f4499d1d640 1 -- 172.21.15.116:0/1265827771 wait complete. 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.488+0000 7f4499d1d640 10 monclient: build_initial_monmap 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.488+0000 7f4499d1d640 10 monclient: monmap: 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.488+0000 7f4499d1d640 1 Processor -- start 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.488+0000 7f4499d1d640 1 -- start start 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.488+0000 7f4499d1d640 10 monclient: init 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 0x7f449409b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f449409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44941318f0 0x7f4494097250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f44941318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494136c20 0x7f4494097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f4494136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449409bf10 con 0x7f4494136c20 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449409db90 con 0x7f44941318f0 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449409abd0 con 0x7f449409ea50 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4499d1d640 10 monclient(hunting): authenticate will time out at 806.811152s 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 0x7f449409b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494136c20 0x7f4494097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 0x7f449409b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36666/0 (socket says 172.21.15.116:36666) 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 1 -- 172.21.15.116:0/368081340 learned_addr learned my addr 172.21.15.116:0/368081340 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): get_auth_request con 0x7f4494136c20 auth_method 0 2024-09-18T12:26:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 10 monclient(hunting): get_auth_request con 0x7f449409ea50 auth_method 0 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44941318f0 0x7f4494097250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 msgr2=0x7f449409b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 msgr2=0x7f449409b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 10 monclient(hunting): get_auth_request con 0x7f44941318f0 auth_method 0 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 0x7f449409b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f44937fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 0x7f449409b520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4492ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 msgr2=0x7f449409b520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f449409ea50 0x7f449409b520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44941318f0 msgr2=0x7f4494097250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44941318f0 0x7f4494097250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 1 monclient: found mon.noname-a 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.489+0000 7f4493fff640 1 -- 172.21.15.116:0/368081340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c009450 con 0x7f4494136c20 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4493fff640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494136c20 msgr2=0x7f4494097790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4493fff640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494136c20 0x7f4494097790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4484006ea0 tx=0x7f4484008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484009850 con 0x7f4494136c20 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: dump: 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4499d1d640 5 monclient: authenticate success, global_id 4434 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4499d1d640 10 monclient: _renew_subs 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44840099b0 con 0x7f4494136c20 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4499d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449409a860 con 0x7f4494136c20 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484009d70 con 0x7f4494136c20 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f448b7fe640 10 monclient: dump: 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4491ffb640 4 set_mon_vals no callback set 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4499d1d640 10 monclient: _renew_subs 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.491+0000 7f4499d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4494099970 con 0x7f4494136c20 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4484014620 con 0x7f4494136c20 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f4499d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f4499d1d640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 0x7f44941327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f4499d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4494065e00 con 0x7f4494097cd0 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f448b7fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4474038490 0x7f447403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.492+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f448404bd60 con 0x7f4494136c20 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 0x7f44941327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient: get_auth_request con 0x7f4494097cd0 auth_method 0 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f4492ffd640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4474038490 0x7f447403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f4492ffd640 10 monclient: get_auth_request con 0x7f4474038490 auth_method 0 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.493+0000 7f44937fe640 1 -- 172.21.15.116:0/368081340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4494131750 con 0x7f4494097cd0 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f44937fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 msgr2=0x7f44941327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f44937fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 0x7f44941327d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f448c00ea90 tx=0x7f448c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c019070 con 0x7f4494097cd0 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 10 monclient: dump: 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f4492ffd640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4474038490 0x7f447403a940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f447c034000 tx=0x7f447c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f448c00cc60 con 0x7f4494097cd0 2024-09-18T12:26:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 msgr2=0x7f44941327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 0x7f44941327d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f448c00ea90 tx=0x7f448c009040 comp rx=0 tx=0).stop 2024-09-18T12:26:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 msgr2=0x7f44941327d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.495+0000 7f448b7fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494097cd0 0x7f44941327d0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f448c00ea90 tx=0x7f448c009040 comp rx=0 tx=0).stop 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f4499d1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f4499d1d640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 0x7f4494133110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f4499d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4494099b60 con 0x7f4494132d10 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 0x7f4494133110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient: get_auth_request con 0x7f4494132d10 auth_method 0 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.501+0000 7f44937fe640 1 -- 172.21.15.116:0/368081340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f449404bf60 con 0x7f4494132d10 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f44937fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 msgr2=0x7f4494133110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f44937fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 0x7f4494133110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f448c00f010 tx=0x7f448c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c015ea0 con 0x7f4494132d10 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 10 monclient: dump: 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f448c022650 con 0x7f4494132d10 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 msgr2=0x7f4494133110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 0x7f4494133110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f448c00f010 tx=0x7f448c01b000 comp rx=0 tx=0).stop 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 msgr2=0x7f4494133110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f448b7fe640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494132d10 0x7f4494133110 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 14, 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:26:19.518 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:19.519 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:19.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:19.521 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:19.522 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:19.525 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:19.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:19.528 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:19.529 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:19.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:19.530 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:19.530 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4474038490 msgr2=0x7f447403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4474038490 0x7f447403a940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f447c034000 tx=0x7f447c032000 comp rx=0 tx=0).stop 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 10 monclient: shutdown 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494136c20 msgr2=0x7f4494097790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4493fff640 1 -- 172.21.15.116:0/368081340 reap_dead start 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494136c20 0x7f4494097790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4484006ea0 tx=0x7f4484008040 comp rx=0 tx=0).stop 2024-09-18T12:26:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 shutdown_connections 2024-09-18T12:26:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 --2- 172.21.15.116:0/368081340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4494136c20 0x7f4494097790 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 >> 172.21.15.116:0/368081340 conn(0x7f4494067110 msgr2=0x7f44940a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 shutdown_connections 2024-09-18T12:26:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.504+0000 7f4499d1d640 1 -- 172.21.15.116:0/368081340 wait complete. 2024-09-18T12:26:19.532 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:26:19.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:26:19.532 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:26:19.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.623+0000 7ff06075d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:19.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.623+0000 7ff06075d640 10 monclient: build_initial_monmap 2024-09-18T12:26:19.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.623+0000 7ff06075d640 10 monclient: monmap: 2024-09-18T12:26:19.625 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:19.625 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:19.625279+0000 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:19.625279+0000 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:19.626 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 1 Processor -- start 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 1 -- start start 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient: init 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff05809c860 0x7ff05809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient(hunting): picked mon.noname-a con 0x7ff05809c860 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff05809aba0 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 0x7ff05809df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient(hunting): picked mon.noname-b con 0x7ff05809afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05806a320 con 0x7ff05809aba0 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05809a9d0 con 0x7ff05809c860 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff058068a20 con 0x7ff05809afa0 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff06075d640 10 monclient(hunting): authenticate will time out at 806.946858s 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff05ecfa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 0x7ff05809df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff05e4f9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff05809c860 0x7ff05809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.624+0000 7ff05dcf8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff05809aba0 0x7ff05809d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47698/0 (socket says 172.21.15.116:47698) 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05dcf8640 1 -- 172.21.15.116:0/4121558358 learned_addr learned my addr 172.21.15.116:0/4121558358 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 10 monclient(hunting): get_auth_request con 0x7ff05809afa0 auth_method 0 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request con 0x7ff05809c860 auth_method 0 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 1 -- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 msgr2=0x7ff05809df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 1 -- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 msgr2=0x7ff05809df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 1 --2- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 0x7ff05809df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05ecfa640 1 --2- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 0x7ff05809df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_done global_id 4443 payload 274 2024-09-18T12:26:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 1 -- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 msgr2=0x7ff05809df40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 0x7ff05809df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 1 -- 172.21.15.116:0/4121558358 >> v1:172.21.15.116:6789/0 conn(0x7ff05809aba0 legacy=0x7ff05809d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient: _finish_hunting 0 2024-09-18T12:26:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 1 monclient: found mon.noname-a 2024-09-18T12:26:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.625+0000 7ff05e4f9640 1 -- 172.21.15.116:0/4121558358 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058068e60 con 0x7ff05809c860 2024-09-18T12:26:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.627+0000 7ff05e4f9640 1 -- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809c860 msgr2=0x7ff05809cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.627+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809c860 0x7ff05809cc30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff054009850 tx=0x7ff0540090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c92abbcb06acd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 1 -- 172.21.15.116:0/4121558358 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05402ae60 con 0x7ff05809c860 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: dump: 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: _finish_auth 0 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 5 monclient: authenticate success, global_id 4443 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 1 -- 172.21.15.116:0/4121558358 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff054032440 con 0x7ff05809c860 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 4 set_mon_vals no callback set 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 1 -- 172.21.15.116:0/4121558358 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0540327e0 con 0x7ff05809c860 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff05d4f7640 10 monclient: dump: 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 10 monclient: shutdown 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 -- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809c860 msgr2=0x7ff05809cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 --2- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809c860 0x7ff05809cc30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff054009850 tx=0x7ff0540090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 -- 172.21.15.116:0/4121558358 shutdown_connections 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 --2- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809afa0 0x7ff05809df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 --2- 172.21.15.116:0/4121558358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809c860 0x7ff05809cc30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 -- 172.21.15.116:0/4121558358 >> 172.21.15.116:0/4121558358 conn(0x7ff0580a2e40 msgr2=0x7ff0580a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 -- 172.21.15.116:0/4121558358 shutdown_connections 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.628+0000 7ff06075d640 1 -- 172.21.15.116:0/4121558358 wait complete. 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient: build_initial_monmap 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient: monmap: 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 Processor -- start 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 -- start start 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient: init 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff05809aba0 0x7ff05806e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient(hunting): picked mon.noname-b con 0x7ff05809aba0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809afa0 0x7ff058132a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient(hunting): picked mon.noname-a con 0x7ff05809afa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 0x7ff05811d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient(hunting): picked mon.noname-c con 0x7ff05809c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05811fb80 con 0x7ff05809afa0 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058133870 con 0x7ff05809aba0 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff058099cc0 con 0x7ff05809c860 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.629+0000 7ff06075d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff06075d640 10 monclient(hunting): authenticate will time out at 806.952006s 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809afa0 0x7ff058132a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 0x7ff05811d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809afa0 0x7ff058132a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42060/0 (socket says 172.21.15.116:42060) 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 -- 172.21.15.116:0/1520723953 learned_addr learned my addr 172.21.15.116:0/1520723953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): get_auth_request con 0x7ff05809afa0 auth_method 0 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 10 monclient(hunting): get_auth_request con 0x7ff05809c860 auth_method 0 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff05809aba0 0x7ff05806e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request con 0x7ff05809aba0 auth_method 0 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 msgr2=0x7ff05811d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 msgr2=0x7ff05811d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 0x7ff05811d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05ecfa640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 0x7ff05811d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 msgr2=0x7ff05811d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05809c860 0x7ff05811d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff05809aba0 msgr2=0x7ff05806e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff05809aba0 0x7ff05806e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient: _finish_hunting 0 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 monclient: found mon.noname-a 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.630+0000 7ff05dcf8640 1 -- 172.21.15.116:0/1520723953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff054009390 con 0x7ff05809afa0 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff05dcf8640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809afa0 msgr2=0x7ff058132a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff05dcf8640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809afa0 0x7ff058132a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c610 tx=0x7ff048004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048005cb0 con 0x7ff05809afa0 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: dump: 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.647 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff048005e30 con 0x7ff05809afa0 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff06075d640 5 monclient: authenticate success, global_id 4446 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048007340 con 0x7ff05809afa0 2024-09-18T12:26:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff06075d640 10 monclient: _renew_subs 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff06075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff058134c00 con 0x7ff05809afa0 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff05ccf6640 4 set_mon_vals no callback set 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff04e7fc640 10 monclient: dump: 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff06075d640 10 monclient: _renew_subs 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff06075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.632+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05811ec90 con 0x7ff05809afa0 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff06075d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff06075d640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff06075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05806eb80 con 0x7ff058132fa0 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff04801b400 con 0x7ff05809afa0 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 10 monclient: get_auth_request con 0x7ff058132fa0 auth_method 0 2024-09-18T12:26:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff04e7fc640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff03c0384c0 0x7ff03c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7ff04804b230 con 0x7ff05809afa0 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.633+0000 7ff05ecfa640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff03c0384c0 0x7ff03c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.634+0000 7ff05ecfa640 10 monclient: get_auth_request con 0x7ff03c0384c0 auth_method 0 2024-09-18T12:26:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.636+0000 7ff05ecfa640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff03c0384c0 0x7ff03c03a970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0581207d0 tx=0x7ff05000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:19.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:19.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 10 monclient: get_auth_request con 0x7ff058132fa0 auth_method 0 2024-09-18T12:26:19.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:19.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:19.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:19.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:26:19.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:19.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:19.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:19.834+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:26:20.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:20.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 10 monclient: get_auth_request con 0x7ff058132fa0 auth_method 0 2024-09-18T12:26:20.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:20.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:20.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:20.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:26:20.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:20.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:20.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:20.235+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:26:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient: get_auth_request con 0x7ff058132fa0 auth_method 0 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2024-09-18T12:26:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.036+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff05804bf60 con 0x7ff058132fa0 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff054034000 tx=0x7ff054032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054046410 con 0x7ff058132fa0 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 10 monclient: dump: 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff0540313a0 con 0x7ff058132fa0 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff054034000 tx=0x7ff054032000 comp rx=0 tx=0).stop 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 msgr2=0x7ff05811ddf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:21.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.039+0000 7ff04e7fc640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132fa0 0x7ff05811ddf0 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff054034000 tx=0x7ff054032000 comp rx=0 tx=0).stop 2024-09-18T12:26:21.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff06075d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:21.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff06075d640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 0x7ff05806b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff06075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff058065e00 con 0x7ff05811e330 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 0x7ff05806b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient: get_auth_request con 0x7ff05811e330 auth_method 0 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-09-18T12:26:21.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.046+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff05811e730 con 0x7ff05811e330 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.048+0000 7ff05e4f9640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 msgr2=0x7ff05806b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.048+0000 7ff05e4f9640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 0x7ff05806b8b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff058068680 tx=0x7ff054040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054051450 con 0x7ff05811e330 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 10 monclient: dump: 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.051 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.052 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.052 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.052 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.052 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7ff0540515b0 con 0x7ff05811e330 2024-09-18T12:26:21.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 msgr2=0x7ff05806b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 0x7ff05806b8b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff058068680 tx=0x7ff054040040 comp rx=0 tx=0).stop 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 msgr2=0x7ff05806b8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.049+0000 7ff04e7fc640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff05811e330 0x7ff05806b8b0 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff058068680 tx=0x7ff054040040 comp rx=0 tx=0).stop 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:26:21.053 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 14, 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:21.054 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:21.055 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:21.056 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:21.057 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:21.058 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:21.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:21.061 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:21.065 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff03c0384c0 msgr2=0x7ff03c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff03c0384c0 0x7ff03c03a970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0581207d0 tx=0x7ff05000b040 comp rx=0 tx=0).stop 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 10 monclient: shutdown 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809afa0 msgr2=0x7ff058132a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 --2- 172.21.15.116:0/1520723953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff05809afa0 0x7ff058132a60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff04800c610 tx=0x7ff048004960 comp rx=0 tx=0).stop 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff05ecfa640 1 -- 172.21.15.116:0/1520723953 reap_dead start 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff05ecfa640 1 -- 172.21.15.116:0/1520723953 reap_dead start 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 shutdown_connections 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 >> 172.21.15.116:0/1520723953 conn(0x7ff0580a2e40 msgr2=0x7ff058134e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:21.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 shutdown_connections 2024-09-18T12:26:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.050+0000 7ff06075d640 1 -- 172.21.15.116:0/1520723953 wait complete. 2024-09-18T12:26:21.068 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:26:21.068 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:26:21.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:26:21.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.168+0000 7f905303f640 10 monclient: get_monmap_and_config 2024-09-18T12:26:21.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.168+0000 7f905303f640 10 monclient: build_initial_monmap 2024-09-18T12:26:21.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.168+0000 7f905303f640 10 monclient: monmap: 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:21.170397+0000 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:21.170397+0000 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:21.171 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 1 Processor -- start 2024-09-18T12:26:21.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 1 -- start start 2024-09-18T12:26:21.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 10 monclient: init 2024-09-18T12:26:21.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:21.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T12:26:21.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f904c09c9c0 0x7f904c09d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 10 monclient(hunting): picked mon.noname-a con 0x7f904c09c9c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.169+0000 7f905303f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f904c09e340 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f905303f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f904c09e750 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f905303f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c09bdd0 con 0x7f904c09e750 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f905303f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c09a880 con 0x7f904c09e340 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f905303f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f905303f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c130590 con 0x7f904c09c9c0 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f905303f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f905303f640 10 monclient(hunting): authenticate will time out at 808.492058s 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f904c09c9c0 0x7f904c09d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f90515dc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f904c09e750 0x7f904c12fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47714/0 (socket says 172.21.15.116:47714) 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f904c09c9c0 0x7f904c09d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42070/0 (socket says 172.21.15.116:42070) 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f90515dc640 1 -- 172.21.15.116:0/4235818934 learned_addr learned my addr 172.21.15.116:0/4235818934 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): get_auth_request con 0x7f904c09c9c0 auth_method 0 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 1 -- 172.21.15.116:0/4235818934 >> v1:172.21.15.144:6789/0 conn(0x7f904c09e340 legacy=0x7f904c12f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 1 -- 172.21.15.116:0/4235818934 >> v1:172.21.15.116:6789/0 conn(0x7f904c09e750 legacy=0x7f904c12fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient: _finish_hunting 0 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 1 monclient: found mon.noname-a 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:21.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.170+0000 7f9050ddb640 1 -- 172.21.15.116:0/4235818934 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c09a440 con 0x7f904c09c9c0 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f9050ddb640 1 -- 172.21.15.116:0/4235818934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 msgr2=0x7f904c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f9050ddb640 1 --2- 172.21.15.116:0/4235818934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c09d830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9040009850 tx=0x7f90400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70165603547a62bb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 1 -- 172.21.15.116:0/4235818934 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040032440 con 0x7f904c09c9c0 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: dump: 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.183 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 1 -- 172.21.15.116:0/4235818934 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90400325a0 con 0x7f904c09c9c0 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 5 monclient: authenticate success, global_id 4449 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 1 -- 172.21.15.116:0/4235818934 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040032980 con 0x7f904c09c9c0 2024-09-18T12:26:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 4 set_mon_vals no callback set 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f904b7fe640 10 monclient: dump: 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.185 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 10 monclient: shutdown 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 1 -- 172.21.15.116:0/4235818934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 msgr2=0x7f904c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.173+0000 7f905303f640 1 --2- 172.21.15.116:0/4235818934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c09d830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9040009850 tx=0x7f90400090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 1 -- 172.21.15.116:0/4235818934 shutdown_connections 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 1 --2- 172.21.15.116:0/4235818934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c09d830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 1 -- 172.21.15.116:0/4235818934 >> 172.21.15.116:0/4235818934 conn(0x7f904c0a2e40 msgr2=0x7f904c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 1 -- 172.21.15.116:0/4235818934 shutdown_connections 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 1 -- 172.21.15.116:0/4235818934 wait complete. 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 10 monclient: build_initial_monmap 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 10 monclient: monmap: 2024-09-18T12:26:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.174+0000 7f905303f640 1 Processor -- start 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 1 -- start start 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient: init 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:26:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c11ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): picked mon.noname-a con 0x7f904c09c9c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 0x7f904c11bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): picked mon.noname-c con 0x7f904c09e340 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c09e750 0x7f904c11c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): picked mon.noname-b con 0x7f904c09e750 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c12f190 con 0x7f904c09c9c0 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c099cc0 con 0x7f904c09e750 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c11f6c0 con 0x7f904c09e340 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:21.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f905303f640 10 monclient(hunting): authenticate will time out at 808.497602s 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c11ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c11ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42080/0 (socket says 172.21.15.116:42080) 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 -- 172.21.15.116:0/157268046 learned_addr learned my addr 172.21.15.116:0/157268046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): get_auth_request con 0x7f904c09c9c0 auth_method 0 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f904bfff640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 0x7f904c11bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f904bfff640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 msgr2=0x7f904c11bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f904bfff640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 msgr2=0x7f904c11bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c09e750 0x7f904c11c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f904bfff640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 0x7f904c11bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:21.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f904bfff640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 0x7f904c11bfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 10 monclient(hunting): get_auth_request con 0x7f904c09e750 auth_method 0 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f90515dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient(hunting): handle_auth_done global_id 4452 payload 824 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 msgr2=0x7f904c11bfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f904c09e340 0x7f904c11bfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c09e750 msgr2=0x7f904c11c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c09e750 0x7f904c11c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient: _finish_hunting 0 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 monclient: found mon.noname-a 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.175+0000 7f9050ddb640 1 -- 172.21.15.116:0/157268046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9040009390 con 0x7f904c09c9c0 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9050ddb640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 msgr2=0x7f904c11ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9050ddb640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c11ba60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9040009360 tx=0x7f9040032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040031990 con 0x7f904c09c9c0 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:21.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: dump: 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: _finish_auth 0 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9040031af0 con 0x7f904c09c9c0 2024-09-18T12:26:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f905303f640 5 monclient: authenticate success, global_id 4452 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f905303f640 10 monclient: _renew_subs 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040031e90 con 0x7f904c09c9c0 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f905303f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c11f100 con 0x7f904c09c9c0 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f904affd640 4 set_mon_vals no callback set 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f9048ff9640 10 monclient: dump: 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f905303f640 10 monclient: _renew_subs 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f905303f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.178+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c11f490 con 0x7f904c09c9c0 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f905303f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f905303f640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 0x7f904c11ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f905303f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c065e00 con 0x7f904c11e210 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9040039040 con 0x7f904c09c9c0 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 0x7f904c11ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 10 monclient: get_auth_request con 0x7f904c11e210 auth_method 0 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f9048ff9640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9030038490 0x7f903003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f904bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.179+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f904006ba70 con 0x7f904c09c9c0 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.180+0000 7f904bfff640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.180+0000 7f90515dc640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9030038490 0x7f903003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.180+0000 7f904bfff640 1 -- 172.21.15.116:0/157268046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f904c04bf60 con 0x7f904c11e210 2024-09-18T12:26:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.180+0000 7f90515dc640 10 monclient: get_auth_request con 0x7f9030038490 auth_method 0 2024-09-18T12:26:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f904bfff640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 msgr2=0x7f904c11ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f904bfff640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 0x7f904c11ca20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f904c11d240 tx=0x7f9038009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f90515dc640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9030038490 0x7f903003a940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9044006d70 tx=0x7f90440052b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9038007450 con 0x7f904c11e210 2024-09-18T12:26:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 10 monclient: dump: 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f90380075b0 con 0x7f904c11e210 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 msgr2=0x7f904c11ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 0x7f904c11ca20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f904c11d240 tx=0x7f9038009040 comp rx=0 tx=0).stop 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 msgr2=0x7f904c11ca20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.182+0000 7f9048ff9640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c11e210 0x7f904c11ca20 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f904c11d240 tx=0x7f9038009040 comp rx=0 tx=0).stop 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f905303f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f905303f640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 0x7f904c06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f905303f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c099ef0 con 0x7f904c06d8c0 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 0x7f904c06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 10 monclient: get_auth_request con 0x7f904c06d8c0 auth_method 0 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.188+0000 7f904bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.189+0000 7f904bfff640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.189+0000 7f904bfff640 1 -- 172.21.15.116:0/157268046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f904c0638d0 con 0x7f904c06d8c0 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f904bfff640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 msgr2=0x7f904c06dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f904bfff640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 0x7f904c06dcc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9038002220 tx=0x7f9038019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9038012e40 con 0x7f904c06d8c0 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f9048ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f9048ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f9048ff9640 10 monclient: dump: 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.201 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f903801b620 con 0x7f904c06d8c0 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.191+0000 7f9048ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f9048ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 msgr2=0x7f904c06dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f9048ff9640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 0x7f904c06dcc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9038002220 tx=0x7f9038019000 comp rx=0 tx=0).stop 2024-09-18T12:26:21.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f9048ff9640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 msgr2=0x7f904c06dcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f9048ff9640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f904c06d8c0 0x7f904c06dcc0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9038002220 tx=0x7f9038019000 comp rx=0 tx=0).stop 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 14, 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:21.203 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:21.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:21.206 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:21.207 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:21.208 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:21.210 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:21.211 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.213 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.214 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:21.216 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:21.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:21.216 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:21.216 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:21.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9030038490 msgr2=0x7f903003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9030038490 0x7f903003a940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9044006d70 tx=0x7f90440052b0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 10 monclient: shutdown 2024-09-18T12:26:21.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:21.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 msgr2=0x7f904c11ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c11ba60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9040009360 tx=0x7f9040032000 comp rx=0 tx=0).stop 2024-09-18T12:26:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f90515dc640 1 -- 172.21.15.116:0/157268046 reap_dead start 2024-09-18T12:26:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 shutdown_connections 2024-09-18T12:26:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 --2- 172.21.15.116:0/157268046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904c09c9c0 0x7f904c11ba60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 >> 172.21.15.116:0/157268046 conn(0x7f904c0a2e40 msgr2=0x7f904c12fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 shutdown_connections 2024-09-18T12:26:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.192+0000 7f905303f640 1 -- 172.21.15.116:0/157268046 wait complete. 2024-09-18T12:26:21.218 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:26:21.218 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:26:21.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:26:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.309+0000 7f1d4f8c0640 10 monclient: get_monmap_and_config 2024-09-18T12:26:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.309+0000 7f1d4f8c0640 10 monclient: build_initial_monmap 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.309+0000 7f1d4f8c0640 10 monclient: monmap: 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:21.310909+0000 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:21.310909+0000 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:21.313 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.309+0000 7f1d4f8c0640 1 Processor -- start 2024-09-18T12:26:21.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 1 -- start start 2024-09-18T12:26:21.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient: init 2024-09-18T12:26:21.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:21.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T12:26:21.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1d480981c0 0x7f1d48098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1d480981c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:21.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d48131da0 0x7f1d48067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1d48131da0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d48132170 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d48098ad0 con 0x7f1d48132170 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d48131bd0 con 0x7f1d48131da0 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4809ca60 con 0x7f1d480981c0 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4f8c0640 10 monclient(hunting): authenticate will time out at 808.632529s 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d48131da0 0x7f1d48067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4de5d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1d48132170 0x7f1d48067650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47726/0 (socket says 172.21.15.116:47726) 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4de5d640 1 -- 172.21.15.116:0/212645917 learned_addr learned my addr 172.21.15.116:0/212645917 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): get_auth_request con 0x7f1d48131da0 auth_method 0 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 1 --2- 172.21.15.116:0/212645917 >> v2:172.21.15.144:3300/0 conn(0x7f1d480981c0 0x7f1d48098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 10 monclient(hunting): get_auth_request con 0x7f1d480981c0 auth_method 0 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4d65c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient(hunting): handle_auth_done global_id 4455 payload 274 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 1 -- 172.21.15.116:0/212645917 >> v2:172.21.15.144:3300/0 conn(0x7f1d480981c0 msgr2=0x7f1d48098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 1 --2- 172.21.15.116:0/212645917 >> v2:172.21.15.144:3300/0 conn(0x7f1d480981c0 0x7f1d48098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 1 -- 172.21.15.116:0/212645917 >> v1:172.21.15.116:6789/0 conn(0x7f1d48132170 legacy=0x7f1d48067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 1 monclient: found mon.noname-a 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.310+0000 7f1d4ce5b640 1 -- 172.21.15.116:0/212645917 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4809cca0 con 0x7f1d48131da0 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.313+0000 7f1d4ce5b640 1 -- 172.21.15.116:0/212645917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d48131da0 msgr2=0x7f1d48067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.313+0000 7f1d4ce5b640 1 --2- 172.21.15.116:0/212645917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d48131da0 0x7f1d48067110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1d48066560 tx=0x7f1d440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40055449bbad4203 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 1 -- 172.21.15.116:0/212645917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4402adf0 con 0x7f1d48131da0 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: dump: 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.324 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: _finish_auth 0 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 1 -- 172.21.15.116:0/212645917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d44032440 con 0x7f1d48131da0 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 5 monclient: authenticate success, global_id 4455 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 1 -- 172.21.15.116:0/212645917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d440327e0 con 0x7f1d48131da0 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 4 set_mon_vals no callback set 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d3ffff640 10 monclient: dump: 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 10 monclient: shutdown 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/212645917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d48131da0 msgr2=0x7f1d48067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/212645917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d48131da0 0x7f1d48067110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1d48066560 tx=0x7f1d440090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/212645917 shutdown_connections 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/212645917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d48131da0 0x7f1d48067110 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/212645917 >> v2:172.21.15.144:3300/0 conn(0x7f1d480981c0 0x7f1d48098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/212645917 >> 172.21.15.116:0/212645917 conn(0x7f1d480a1e30 msgr2=0x7f1d480a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/212645917 shutdown_connections 2024-09-18T12:26:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.314+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/212645917 wait complete. 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.315+0000 7f1d4f8c0640 10 monclient: build_initial_monmap 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.315+0000 7f1d4f8c0640 10 monclient: monmap: 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.315+0000 7f1d4f8c0640 1 Processor -- start 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.315+0000 7f1d4f8c0640 1 -- start start 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.315+0000 7f1d4f8c0640 10 monclient: init 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 0x7f1d4811db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1d480981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d48131da0 0x7f1d4811e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1d48131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d48132170 0x7f1d4811e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1d48132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d48120130 con 0x7f1d480981c0 2024-09-18T12:26:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d48098cb0 con 0x7f1d48131da0 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d480a08c0 con 0x7f1d48132170 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4f8c0640 10 monclient(hunting): authenticate will time out at 808.638307s 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 0x7f1d4811db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 0x7f1d4811db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42096/0 (socket says 172.21.15.116:42096) 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 -- 172.21.15.116:0/203176362 learned_addr learned my addr 172.21.15.116:0/203176362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): get_auth_request con 0x7f1d480981c0 auth_method 0 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d48132170 0x7f1d4811e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d48131da0 0x7f1d4811e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 10 monclient(hunting): get_auth_request con 0x7f1d48132170 auth_method 0 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 10 monclient(hunting): get_auth_request con 0x7f1d48131da0 auth_method 0 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4de5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4ce5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 -- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d48132170 msgr2=0x7f1d4811e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d48132170 0x7f1d4811e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 -- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d48131da0 msgr2=0x7f1d4811e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d48131da0 0x7f1d4811e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient: _finish_hunting 0 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 monclient: found mon.noname-a 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.316+0000 7f1d4d65c640 1 -- 172.21.15.116:0/203176362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44009390 con 0x7f1d480981c0 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d4d65c640 1 -- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 msgr2=0x7f1d4811db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d4d65c640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 0x7f1d4811db50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800ea20 tx=0x7f1d380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d3d7fa640 1 -- 172.21.15.116:0/203176362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800cc20 con 0x7f1d480981c0 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d3d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d3d7fa640 10 monclient: dump: 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d3d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d3d7fa640 1 -- 172.21.15.116:0/203176362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3800cd80 con 0x7f1d480981c0 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.318+0000 7f1d4f8c0640 5 monclient: authenticate success, global_id 4458 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d4f8c0640 10 monclient: _renew_subs 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d4f8c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4811f730 con 0x7f1d480981c0 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d3d7fa640 1 -- 172.21.15.116:0/203176362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38013680 con 0x7f1d480981c0 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d3d7fa640 10 monclient: dump: 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d4f8c0640 10 monclient: _renew_subs 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d4f8c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d3f7fe640 4 set_mon_vals no callback set 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d481211f0 con 0x7f1d480981c0 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.319+0000 7f1d3d7fa640 1 -- 172.21.15.116:0/203176362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1d38013a30 con 0x7f1d480981c0 2024-09-18T12:26:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d4f8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d4f8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d4f8c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4804bf60 con 0x7f1d480981c0 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d3d7fa640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d2c038490 0x7f1d2c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d3d7fa640 1 -- 172.21.15.116:0/203176362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f1d3804be10 con 0x7f1d480981c0 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d4ce5b640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d2c038490 0x7f1d2c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.320+0000 7f1d4ce5b640 10 monclient: get_auth_request con 0x7f1d2c038490 auth_method 0 2024-09-18T12:26:21.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.322+0000 7f1d4ce5b640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d2c038490 0x7f1d2c03a940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d4806f820 tx=0x7f1d44032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.324+0000 7f1d3d7fa640 1 -- 172.21.15.116:0/203176362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1d3801bdf0 con 0x7f1d480981c0 2024-09-18T12:26:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.324+0000 7f1d3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.324+0000 7f1d3d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.540+0000 7f1d4f8c0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:26:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.540+0000 7f1d4f8c0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:26:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.540+0000 7f1d4f8c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.540+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1d48098ee0 con 0x7f1d480981c0 2024-09-18T12:26:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.541+0000 7f1d3d7fa640 1 -- 172.21.15.116:0/203176362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f1d3804d030 con 0x7f1d480981c0 2024-09-18T12:26:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.541+0000 7f1d3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:26:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.541+0000 7f1d3d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.542+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d2c038490 msgr2=0x7f1d2c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.542+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d2c038490 0x7f1d2c03a940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d4806f820 tx=0x7f1d44032000 comp rx=0 tx=0).stop 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.542+0000 7f1d4f8c0640 10 monclient: shutdown 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.542+0000 7f1d4f8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.542+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 msgr2=0x7f1d4811db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.542+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 0x7f1d4811db50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800ea20 tx=0x7f1d380090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 shutdown_connections 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d2c038490 0x7f1d2c03a940 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d48132170 0x7f1d4811e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d48131da0 0x7f1d4811e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 --2- 172.21.15.116:0/203176362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d480981c0 0x7f1d4811db50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 >> 172.21.15.116:0/203176362 conn(0x7f1d480a1e30 msgr2=0x7f1d48096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 shutdown_connections 2024-09-18T12:26:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:21.543+0000 7f1d4f8c0640 1 -- 172.21.15.116:0/203176362 wait complete. 2024-09-18T12:26:21.555 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:26:21.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.804+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.804+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=7,logm=70} crc {auth=4048894511,config=3140514009,health=305525474,kv=2350409610,logm=2808165430}) 2024-09-18T12:26:21.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.804+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=9,logm=68} crc {auth=4048894511,config=3140514009,health=305525474,kv=2187923472,logm=705796269}) 2024-09-18T12:26:21.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.804+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.804+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=7,logm=70} crc {auth=4048894511,config=3140514009,health=305525474,kv=2350409610,logm=2808165430}) 2024-09-18T12:26:21.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.804+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=11,kv=9,logm=68} crc {auth=4048894511,config=3140514009,health=305525474,kv=2187923472,logm=2948060682}) 2024-09-18T12:26:21.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.807+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.807+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148903749}) 2024-09-18T12:26:21.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.807+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742865813}) 2024-09-18T12:26:21.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.807+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.807+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1148903749}) 2024-09-18T12:26:21.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.807+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1896880987}) 2024-09-18T12:26:21.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.870+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.870+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764244612}) 2024-09-18T12:26:21.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.870+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655555996}) 2024-09-18T12:26:21.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.870+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.870+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764244612}) 2024-09-18T12:26:21.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.870+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1313738879}) 2024-09-18T12:26:21.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.872+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.872+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=2315270002,mdsmap=3585761405,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2353631573,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2468821820,osd_snap=2033388227,osdmap=2044857618}) 2024-09-18T12:26:21.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.872+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=60,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=4} crc {logm=3480928827,mdsmap=1531151339,mgr=3280793501,mgr_metadata=3239289319,mgrstat=2353631573,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2468821820,osd_snap=2033388227,osdmap=3165621792}) 2024-09-18T12:26:21.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.872+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.872+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=2315270002,mdsmap=3585761405,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2353631573,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2468821820,osd_snap=2033388227,osdmap=2044857618}) 2024-09-18T12:26:21.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.872+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=2945340985,mdsmap=1531151339,mgr=3340183362,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1113119468,mon_config_key=3655569570,monmap=148614106,osd_metadata=3127564625,osd_pg_creating=2468821820,osd_snap=553062916,osdmap=2044857618}) 2024-09-18T12:26:21.875 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.873+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.875 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.873+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=2895050742}) 2024-09-18T12:26:21.875 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.873+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=34} crc {osdmap=3919491261}) 2024-09-18T12:26:21.875 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.873+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:26:21.875 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.873+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=2895050742}) 2024-09-18T12:26:21.875 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:26:21.873+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=2356436522}) 2024-09-18T12:26:22.555 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:26:22.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.657+0000 7f7d4b35d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.657+0000 7f7d4b35d640 10 monclient: build_initial_monmap 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.657+0000 7f7d4b35d640 10 monclient: monmap: 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:22.659558+0000 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:22.659558+0000 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:22.661 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:22.661 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:22.661 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:22.661 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:22.661 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:22.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.658+0000 7f7d4b35d640 1 Processor -- start 2024-09-18T12:26:22.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.658+0000 7f7d4b35d640 1 -- start start 2024-09-18T12:26:22.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.658+0000 7f7d4b35d640 10 monclient: init 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d440a1ef0 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d4409c6a0 0x7f7d4409b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d4409c6a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7d44136d00 0x7f7d441390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d44136d00 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4406f010 con 0x7f7d440a1ef0 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4409da00 con 0x7f7d4409c6a0 2024-09-18T12:26:22.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4409dcc0 con 0x7f7d44136d00 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d4b35d640 10 monclient(hunting): authenticate will time out at 809.981247s 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d4409c6a0 0x7f7d4409b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d4409c6a0 0x7f7d4409b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42098/0 (socket says 172.21.15.116:42098) 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 -- 172.21.15.116:0/4189754299 learned_addr learned my addr 172.21.15.116:0/4189754299 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): get_auth_request con 0x7f7d4409c6a0 auth_method 0 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d498fa640 1 --2- 172.21.15.116:0/4189754299 >> v2:172.21.15.144:3300/0 conn(0x7f7d44136d00 0x7f7d441390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d498fa640 10 monclient(hunting): get_auth_request con 0x7f7d44136d00 auth_method 0 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d498fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d498fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d498fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient(hunting): handle_auth_done global_id 4461 payload 274 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 -- 172.21.15.116:0/4189754299 >> v2:172.21.15.144:3300/0 conn(0x7f7d44136d00 msgr2=0x7f7d441390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 --2- 172.21.15.116:0/4189754299 >> v2:172.21.15.144:3300/0 conn(0x7f7d44136d00 0x7f7d441390e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 -- 172.21.15.116:0/4189754299 >> v1:172.21.15.144:6789/0 conn(0x7f7d440a1ef0 legacy=0x7f7d4409abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient: _finish_hunting 0 2024-09-18T12:26:22.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 monclient: found mon.noname-a 2024-09-18T12:26:22.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:22.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:22.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d488f8640 1 -- 172.21.15.116:0/4189754299 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440a22c0 con 0x7f7d4409c6a0 2024-09-18T12:26:22.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.659+0000 7f7d498fa640 1 --2- 172.21.15.116:0/4189754299 >> v2:172.21.15.144:3300/0 conn(0x7f7d44136d00 0x7f7d441390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:26:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d488f8640 1 -- 172.21.15.116:0/4189754299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 msgr2=0x7f7d4409b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d488f8640 1 --2- 172.21.15.116:0/4189754299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d4409b2e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d440951c0 tx=0x7f7d400091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a64c44b11bcf61 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 1 -- 172.21.15.116:0/4189754299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40032440 con 0x7f7d4409c6a0 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: dump: 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:22.674 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: _finish_auth 0 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 1 -- 172.21.15.116:0/4189754299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d400325a0 con 0x7f7d4409c6a0 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d4b35d640 5 monclient: authenticate success, global_id 4461 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d4b35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d4b35d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 1 -- 172.21.15.116:0/4189754299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40032980 con 0x7f7d4409c6a0 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d4b35d640 4 set_mon_vals no callback set 2024-09-18T12:26:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.662+0000 7f7d37fff640 10 monclient: dump: 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 10 monclient: shutdown 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 -- 172.21.15.116:0/4189754299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 msgr2=0x7f7d4409b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/4189754299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d4409b2e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d440951c0 tx=0x7f7d400091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 -- 172.21.15.116:0/4189754299 shutdown_connections 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/4189754299 >> v2:172.21.15.144:3300/0 conn(0x7f7d44136d00 0x7f7d441390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/4189754299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d4409b2e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 -- 172.21.15.116:0/4189754299 >> 172.21.15.116:0/4189754299 conn(0x7f7d440678a0 msgr2=0x7f7d440a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 -- 172.21.15.116:0/4189754299 shutdown_connections 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.663+0000 7f7d4b35d640 1 -- 172.21.15.116:0/4189754299 wait complete. 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.664+0000 7f7d4b35d640 10 monclient: build_initial_monmap 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.664+0000 7f7d4b35d640 10 monclient: monmap: 2024-09-18T12:26:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.664+0000 7f7d4b35d640 1 Processor -- start 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.664+0000 7f7d4b35d640 1 -- start start 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.664+0000 7f7d4b35d640 10 monclient: init 2024-09-18T12:26:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d44141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d4409c6a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d440a1ef0 0x7f7d4412eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d440a1ef0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d44136d00 0x7f7d441349b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d44136d00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d44136520 con 0x7f7d4409c6a0 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4412fa00 con 0x7f7d44136d00 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d44128030 con 0x7f7d440a1ef0 2024-09-18T12:26:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d4b35d640 10 monclient(hunting): authenticate will time out at 809.987214s 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d44141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d44141250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42108/0 (socket says 172.21.15.116:42108) 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 -- 172.21.15.116:0/3843381186 learned_addr learned my addr 172.21.15.116:0/3843381186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): get_auth_request con 0x7f7d4409c6a0 auth_method 0 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d440a1ef0 0x7f7d4412eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 10 monclient(hunting): get_auth_request con 0x7f7d440a1ef0 auth_method 0 2024-09-18T12:26:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d498fa640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d44136d00 0x7f7d441349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d488f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d498fa640 10 monclient(hunting): get_auth_request con 0x7f7d44136d00 auth_method 0 2024-09-18T12:26:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d498fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d498fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d498fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 -- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d440a1ef0 msgr2=0x7f7d4412eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d440a1ef0 0x7f7d4412eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 -- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d44136d00 msgr2=0x7f7d441349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d44136d00 0x7f7d441349b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient: _finish_hunting 0 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 monclient: found mon.noname-a 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d490f9640 1 -- 172.21.15.116:0/3843381186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40009450 con 0x7f7d4409c6a0 2024-09-18T12:26:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.665+0000 7f7d498fa640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d44136d00 0x7f7d441349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d490f9640 1 -- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 msgr2=0x7f7d44141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d490f9640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d44141250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800b4f0 tx=0x7f7d380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d357fa640 1 -- 172.21.15.116:0/3843381186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3800f720 con 0x7f7d4409c6a0 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d357fa640 10 monclient: dump: 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d357fa640 10 monclient: _finish_auth 0 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d357fa640 1 -- 172.21.15.116:0/3843381186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3800f8a0 con 0x7f7d4409c6a0 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.667+0000 7f7d4b35d640 5 monclient: authenticate success, global_id 4464 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d357fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d357fa640 1 -- 172.21.15.116:0/3843381186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3800fc60 con 0x7f7d4409c6a0 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d4b35d640 10 monclient: _renew_subs 2024-09-18T12:26:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d4b35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d441420c0 con 0x7f7d4409c6a0 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d377fe640 4 set_mon_vals no callback set 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d357fa640 10 monclient: dump: 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d4b35d640 10 monclient: _renew_subs 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d4b35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d44141930 con 0x7f7d4409c6a0 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.668+0000 7f7d357fa640 1 -- 172.21.15.116:0/3843381186 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7d38011610 con 0x7f7d4409c6a0 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d357fa640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d28038490 0x7f7d2803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d357fa640 1 -- 172.21.15.116:0/3843381186 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f7d3804b130 con 0x7f7d4409c6a0 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d4b35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:22.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d4b35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d4b35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4404bf60 con 0x7f7d4409c6a0 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d488f8640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d28038490 0x7f7d2803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.669+0000 7f7d488f8640 10 monclient: get_auth_request con 0x7f7d28038490 auth_method 0 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.671+0000 7f7d488f8640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d28038490 0x7f7d2803a940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d40034000 tx=0x7f7d40032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.673+0000 7f7d357fa640 1 -- 172.21.15.116:0/3843381186 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7d38017670 con 0x7f7d4409c6a0 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.673+0000 7f7d357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:22.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.673+0000 7f7d357fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.891+0000 7f7d4b35d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.891+0000 7f7d4b35d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.891+0000 7f7d4b35d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.891+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7d441304a0 con 0x7f7d4409c6a0 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.892+0000 7f7d357fa640 1 -- 172.21.15.116:0/3843381186 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7d380172b0 con 0x7f7d4409c6a0 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.892+0000 7f7d357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.892+0000 7f7d357fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:22.894 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:26:22.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d28038490 msgr2=0x7f7d2803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:22.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d28038490 0x7f7d2803a940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d40034000 tx=0x7f7d40032000 comp rx=0 tx=0).stop 2024-09-18T12:26:22.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 10 monclient: shutdown 2024-09-18T12:26:22.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:22.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 msgr2=0x7f7d44141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d44141250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800b4f0 tx=0x7f7d380090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 shutdown_connections 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d28038490 0x7f7d2803a940 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d44136d00 0x7f7d441349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d440a1ef0 0x7f7d4412eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 --2- 172.21.15.116:0/3843381186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4409c6a0 0x7f7d44141250 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 >> 172.21.15.116:0/3843381186 conn(0x7f7d440678a0 msgr2=0x7f7d44097440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 shutdown_connections 2024-09-18T12:26:22.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.894+0000 7f7d4b35d640 1 -- 172.21.15.116:0/3843381186 wait complete. 2024-09-18T12:26:22.906 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:26:22.906 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:26:22.907 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:26:22.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:26:23.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.998+0000 7f21e8d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.998+0000 7f21e8d1d640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.998+0000 7f21e8d1d640 10 monclient: monmap: 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:23.000241+0000 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:23.000241+0000 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.001 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:23.002 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 1 Processor -- start 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 1 -- start start 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient: init 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21e4096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21e40971e0 0x7f21e40975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f21e40971e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f21e4133b30 0x7f21e4135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f21e4133b30 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e4099620 con 0x7f21e4096e10 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e4099340 con 0x7f21e40971e0 2024-09-18T12:26:23.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e4099900 con 0x7f21e4133b30 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e8d1d640 10 monclient(hunting): authenticate will time out at 810.321864s 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21dbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21e40971e0 0x7f21e40975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21dbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21e40971e0 0x7f21e40975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42116/0 (socket says 172.21.15.116:42116) 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21e259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f21e4096e10 0x7f21e409a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47742/0 (socket says 172.21.15.116:47742) 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21dbfff640 1 -- 172.21.15.116:0/1408907423 learned_addr learned my addr 172.21.15.116:0/1408907423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21dbfff640 10 monclient(hunting): get_auth_request con 0x7f21e40971e0 auth_method 0 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:22.999+0000 7f21dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21e2d9d640 1 --2- 172.21.15.116:0/1408907423 >> v2:172.21.15.144:3300/0 conn(0x7f21e4133b30 0x7f21e4135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request con 0x7f21e4133b30 auth_method 0 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 10 monclient(hunting): handle_auth_done global_id 4467 payload 274 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 1 -- 172.21.15.116:0/1408907423 >> v2:172.21.15.144:3300/0 conn(0x7f21e4133b30 msgr2=0x7f21e4135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 1 --2- 172.21.15.116:0/1408907423 >> v2:172.21.15.144:3300/0 conn(0x7f21e4133b30 0x7f21e4135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 1 -- 172.21.15.116:0/1408907423 >> v1:172.21.15.116:6789/0 conn(0x7f21e4096e10 legacy=0x7f21e409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:23.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 1 monclient: found mon.noname-a 2024-09-18T12:26:23.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.000+0000 7f21dbfff640 1 -- 172.21.15.116:0/1408907423 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e409ac70 con 0x7f21e40971e0 2024-09-18T12:26:23.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.002+0000 7f21dbfff640 1 -- 172.21.15.116:0/1408907423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 msgr2=0x7f21e40975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.002+0000 7f21dbfff640 1 --2- 172.21.15.116:0/1408907423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e40975e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f21e409b020 tx=0x7f21cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557d88d7f35174ba server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 1 -- 172.21.15.116:0/1408907423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc02ae80 con 0x7f21e40971e0 2024-09-18T12:26:23.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:23.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: dump: 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 1 -- 172.21.15.116:0/1408907423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc032440 con 0x7f21e40971e0 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e8d1d640 5 monclient: authenticate success, global_id 4467 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e8d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e8d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e8d1d640 4 set_mon_vals no callback set 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 1 -- 172.21.15.116:0/1408907423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc0327c0 con 0x7f21e40971e0 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.004+0000 7f21e1d9b640 10 monclient: dump: 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 10 monclient: shutdown 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 -- 172.21.15.116:0/1408907423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 msgr2=0x7f21e40975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 --2- 172.21.15.116:0/1408907423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e40975e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f21e409b020 tx=0x7f21cc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 -- 172.21.15.116:0/1408907423 shutdown_connections 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 --2- 172.21.15.116:0/1408907423 >> v2:172.21.15.144:3300/0 conn(0x7f21e4133b30 0x7f21e4135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 --2- 172.21.15.116:0/1408907423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e40975e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 -- 172.21.15.116:0/1408907423 >> 172.21.15.116:0/1408907423 conn(0x7f21e40672b0 msgr2=0x7f21e40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 -- 172.21.15.116:0/1408907423 shutdown_connections 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 -- 172.21.15.116:0/1408907423 wait complete. 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 10 monclient: monmap: 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:23.019 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 Processor -- start 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 1 -- start start 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.005+0000 7f21e8d1d640 10 monclient: init 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e4096e10 0x7f21e406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f21e4096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e412e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f21e40971e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21e4133b30 0x7f21e412ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f21e4133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e411de70 con 0x7f21e40971e0 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e4133350 con 0x7f21e4096e10 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e41347c0 con 0x7f21e4133b30 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e8d1d640 10 monclient(hunting): authenticate will time out at 810.328409s 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e412e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21e4133b30 0x7f21e412ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e412e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42130/0 (socket says 172.21.15.116:42130) 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 -- 172.21.15.116:0/4267544841 learned_addr learned my addr 172.21.15.116:0/4267544841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): get_auth_request con 0x7f21e40971e0 auth_method 0 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request con 0x7f21e4133b30 auth_method 0 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e4096e10 0x7f21e406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 10 monclient(hunting): get_auth_request con 0x7f21e4096e10 auth_method 0 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient(hunting): handle_auth_done global_id 4470 payload 824 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21e4133b30 msgr2=0x7f21e412ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21e4133b30 0x7f21e412ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e4096e10 msgr2=0x7f21e406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21e4096e10 0x7f21e406e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 monclient: found mon.noname-a 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21dbfff640 1 -- 172.21.15.116:0/4267544841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc009450 con 0x7f21e40971e0 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.006+0000 7f21e2d9d640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21e4133b30 0x7f21e412ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21dbfff640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 msgr2=0x7f21e412e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21dbfff640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e412e750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009940 tx=0x7f21cc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc006b30 con 0x7f21e40971e0 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: dump: 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.024 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.025 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.025 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.025 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.025 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.025 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.025 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.025 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc031c70 con 0x7f21e40971e0 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 5 monclient: authenticate success, global_id 4470 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 10 monclient: _renew_subs 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e411d8b0 con 0x7f21e40971e0 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc03e450 con 0x7f21e40971e0 2024-09-18T12:26:23.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21da7fc640 10 monclient: dump: 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 10 monclient: _renew_subs 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e159a640 4 set_mon_vals no callback set 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e406edb0 con 0x7f21e40971e0 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.009+0000 7f21e8d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e8d1d640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 0x7f21e40298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e8d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e411dd30 con 0x7f21e411c050 2024-09-18T12:26:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f21cc03e8c0 con 0x7f21e40971e0 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 0x7f21e40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient: get_auth_request con 0x7f21e411c050 auth_method 0 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21da7fc640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f21c40384e0 0x7f21c403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f21cc06bca0 con 0x7f21e40971e0 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 10 monclient(hunting): handle_auth_done global_id 4473 payload 824 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e259c640 1 -- 172.21.15.116:0/4267544841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21e404bf60 con 0x7f21e411c050 2024-09-18T12:26:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e2d9d640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f21c40384e0 0x7f21c403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.010+0000 7f21e2d9d640 10 monclient: get_auth_request con 0x7f21c40384e0 auth_method 0 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21e259c640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 msgr2=0x7f21e40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21e259c640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 0x7f21e40298c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f21d400e740 tx=0x7f21d4007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4005050 con 0x7f21e411c050 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 10 monclient: dump: 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.030 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f21d40051b0 con 0x7f21e411c050 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 msgr2=0x7f21e40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 0x7f21e40298c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f21d400e740 tx=0x7f21d4007e40 comp rx=0 tx=0).stop 2024-09-18T12:26:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21e2d9d640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f21c40384e0 0x7f21c403a990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21e406e860 tx=0x7f21dc009850 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 msgr2=0x7f21e40298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.012+0000 7f21da7fc640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c050 0x7f21e40298c0 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f21d400e740 tx=0x7f21d4007e40 comp rx=0 tx=0).stop 2024-09-18T12:26:23.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e8d1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e8d1d640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 0x7f21e4029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e8d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e4065f20 con 0x7f21e411c420 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e259c640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 0x7f21e4029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e259c640 10 monclient: get_auth_request con 0x7f21e411c420 auth_method 0 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.018+0000 7f21e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.019+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.019+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.019+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.019+0000 7f21e259c640 10 monclient(hunting): handle_auth_done global_id 4476 payload 824 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.019+0000 7f21e259c640 1 -- 172.21.15.116:0/4267544841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f21e411c9e0 con 0x7f21e411c420 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21e259c640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 msgr2=0x7f21e4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21e259c640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 0x7f21e4029e00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f21d400e740 tx=0x7f21d4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4014e60 con 0x7f21e411c420 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 10 monclient: dump: 2024-09-18T12:26:23.035 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f21d4025070 con 0x7f21e411c420 2024-09-18T12:26:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:23.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:23.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 msgr2=0x7f21e4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 0x7f21e4029e00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f21d400e740 tx=0x7f21d4019000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 msgr2=0x7f21e4029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:23.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21da7fc640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e411c420 0x7f21e4029e00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.037 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:23.037 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 14, 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:26:23.038 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.039 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.040 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:23.041 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:23.042 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.043 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.044 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.045 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.047 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:23.048 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:23.049 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:23.051 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.021+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f21c40384e0 msgr2=0x7f21c403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f21c40384e0 0x7f21c403a990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21e406e860 tx=0x7f21dc009850 comp rx=0 tx=0).stop 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 10 monclient: shutdown 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 msgr2=0x7f21e412e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 1 --2- 172.21.15.116:0/4267544841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21e40971e0 0x7f21e412e750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009940 tx=0x7f21cc032000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e2d9d640 1 -- 172.21.15.116:0/4267544841 reap_dead start 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e2d9d640 1 -- 172.21.15.116:0/4267544841 reap_dead start 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 shutdown_connections 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 >> 172.21.15.116:0/4267544841 conn(0x7f21e40672b0 msgr2=0x7f21e409f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 shutdown_connections 2024-09-18T12:26:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.022+0000 7f21e8d1d640 1 -- 172.21.15.116:0/4267544841 wait complete. 2024-09-18T12:26:23.054 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:26:23.054 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:26:23.054 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:26:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.145+0000 7f832c6ef640 10 monclient: get_monmap_and_config 2024-09-18T12:26:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.145+0000 7f832c6ef640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.145+0000 7f832c6ef640 10 monclient: monmap: 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:23.146952+0000 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:23.146952+0000 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:23.149 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.145+0000 7f832c6ef640 1 Processor -- start 2024-09-18T12:26:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.145+0000 7f832c6ef640 1 -- start start 2024-09-18T12:26:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.145+0000 7f832c6ef640 10 monclient: init 2024-09-18T12:26:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 0x7f8324131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient(hunting): picked mon.noname-b con 0x7f83241318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f832409ea50 0x7f832409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient(hunting): picked mon.noname-a con 0x7f832409ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8324068d90 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8324131cc0 con 0x7f8324068d90 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832409fbf0 con 0x7f832409ea50 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832409fe30 con 0x7f83241318f0 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832c6ef640 10 monclient(hunting): authenticate will time out at 810.468587s 2024-09-18T12:26:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 0x7f8324131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 0x7f8324131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36760/0 (socket says 172.21.15.116:36760) 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 1 -- 172.21.15.116:0/2996917280 learned_addr learned my addr 172.21.15.116:0/2996917280 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): get_auth_request con 0x7f83241318f0 auth_method 0 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 1 --2- 172.21.15.116:0/2996917280 >> v2:172.21.15.116:3300/0 conn(0x7f832409ea50 0x7f832409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 10 monclient(hunting): get_auth_request con 0x7f832409ea50 auth_method 0 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f8329c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient(hunting): handle_auth_done global_id 34115 payload 274 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 1 -- 172.21.15.116:0/2996917280 >> v2:172.21.15.116:3300/0 conn(0x7f832409ea50 msgr2=0x7f832409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 1 --2- 172.21.15.116:0/2996917280 >> v2:172.21.15.116:3300/0 conn(0x7f832409ea50 0x7f832409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 1 -- 172.21.15.116:0/2996917280 >> v1:172.21.15.144:6789/0 conn(0x7f8324068d90 legacy=0x7f83240691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 1 monclient: found mon.noname-b 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.146+0000 7f832a48b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.147+0000 7f832a48b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:26:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.147+0000 7f832a48b640 1 -- 172.21.15.116:0/2996917280 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240a1210 con 0x7f83241318f0 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.147+0000 7f8329c8a640 1 --2- 172.21.15.116:0/2996917280 >> v2:172.21.15.116:3300/0 conn(0x7f832409ea50 0x7f832409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f832a48b640 1 -- 172.21.15.116:0/2996917280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 msgr2=0x7f8324131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f832a48b640 1 --2- 172.21.15.116:0/2996917280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 0x7f8324131f30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f831c009910 tx=0x7f831c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea9216568fe0cb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 1 -- 172.21.15.116:0/2996917280 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c02aea0 con 0x7f83241318f0 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: dump: 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 1 -- 172.21.15.116:0/2996917280 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c032440 con 0x7f83241318f0 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f832c6ef640 5 monclient: authenticate success, global_id 34115 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f832c6ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 1 -- 172.21.15.116:0/2996917280 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c032820 con 0x7f83241318f0 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f832c6ef640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f832c6ef640 4 set_mon_vals no callback set 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.149+0000 7f8329489640 10 monclient: dump: 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.162 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 10 monclient: shutdown 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 -- 172.21.15.116:0/2996917280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 msgr2=0x7f8324131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 --2- 172.21.15.116:0/2996917280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 0x7f8324131f30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f831c009910 tx=0x7f831c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 -- 172.21.15.116:0/2996917280 shutdown_connections 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 --2- 172.21.15.116:0/2996917280 >> v2:172.21.15.116:3300/0 conn(0x7f832409ea50 0x7f832409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 --2- 172.21.15.116:0/2996917280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241318f0 0x7f8324131f30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 -- 172.21.15.116:0/2996917280 >> 172.21.15.116:0/2996917280 conn(0x7f8324067110 msgr2=0x7f8324067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 -- 172.21.15.116:0/2996917280 shutdown_connections 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 1 -- 172.21.15.116:0/2996917280 wait complete. 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.150+0000 7f832c6ef640 10 monclient: monmap: 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 Processor -- start 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 -- start start 2024-09-18T12:26:23.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient: init 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8324068d90 0x7f83241413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): picked mon.noname-b con 0x7f8324068d90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 0x7f832409abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): picked mon.noname-c con 0x7f832409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83241318f0 0x7f8324130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): picked mon.noname-a con 0x7f83241318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832409bf10 con 0x7f83241318f0 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832411e350 con 0x7f8324068d90 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83240a4d80 con 0x7f832409ea50 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832c6ef640 10 monclient(hunting): authenticate will time out at 810.473774s 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 0x7f832409abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 0x7f832409abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36766/0 (socket says 172.21.15.116:36766) 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832ac8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83241318f0 0x7f8324130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 1 -- 172.21.15.116:0/1362801854 learned_addr learned my addr 172.21.15.116:0/1362801854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 10 monclient(hunting): get_auth_request con 0x7f832409ea50 auth_method 0 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832ac8c640 10 monclient(hunting): get_auth_request con 0x7f83241318f0 auth_method 0 2024-09-18T12:26:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832ac8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832ac8c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832ac8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832a48b640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8324068d90 0x7f83241413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832a48b640 10 monclient(hunting): get_auth_request con 0x7f8324068d90 auth_method 0 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832a48b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832a48b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832a48b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f8329c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832ac8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.151+0000 7f832ac8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f832ac8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 10 monclient(hunting): handle_auth_done global_id 34118 payload 824 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8324068d90 msgr2=0x7f83241413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8324068d90 0x7f83241413a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83241318f0 msgr2=0x7f8324130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83241318f0 0x7f8324130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 1 monclient: found mon.noname-c 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.152+0000 7f8329c8a640 1 -- 172.21.15.116:0/1362801854 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318009390 con 0x7f832409ea50 2024-09-18T12:26:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f8329c8a640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 msgr2=0x7f832409abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f8329c8a640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 0x7f832409abc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f831800ec80 tx=0x7f83180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831800eec0 con 0x7f832409ea50 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: dump: 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.169 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f832c6ef640 5 monclient: authenticate success, global_id 34118 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831800cdd0 con 0x7f832409ea50 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318013680 con 0x7f832409ea50 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f83227fc640 10 monclient: dump: 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f832c6ef640 10 monclient: _renew_subs 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f832c6ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c009450 con 0x7f832409ea50 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f8328c88640 4 set_mon_vals no callback set 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f832c6ef640 10 monclient: _renew_subs 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f832c6ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:26:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.154+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832411dd20 con 0x7f832409ea50 2024-09-18T12:26:23.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832c6ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832c6ef640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 0x7f832411b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832c6ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8324065e00 con 0x7f83241311a0 2024-09-18T12:26:23.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f8329c8a640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 0x7f832409abc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f831800ec80 tx=0x7f83180090f0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f831804ba80 osd_map(18..18 src has 1..18) v4 2024-09-18T12:26:23.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 0x7f832411b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f83180139f0 con 0x7f832409ea50 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 10 monclient: get_auth_request con 0x7f83241311a0 auth_method 0 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.155+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.156+0000 7f83227fc640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f83100384c0 0x7f831003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.156+0000 7f832a48b640 10 monclient(hunting): handle_auth_done global_id 34121 payload 824 2024-09-18T12:26:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.156+0000 7f832a48b640 1 -- 172.21.15.116:0/1362801854 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f832404bf60 con 0x7f83241311a0 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.156+0000 7f832ac8c640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f83100384c0 0x7f831003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.156+0000 7f832ac8c640 10 monclient: get_auth_request con 0x7f83100384c0 auth_method 0 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f832a48b640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 msgr2=0x7f832411b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f832a48b640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 0x7f832411b720 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8324064050 tx=0x7f831c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c02aea0 con 0x7f83241311a0 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 10 monclient: dump: 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f832ac8c640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f83100384c0 0x7f831003a970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8314007fd0 tx=0x7f83140025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f831c031d30 con 0x7f83241311a0 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 msgr2=0x7f832411b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 0x7f832411b720 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8324064050 tx=0x7f831c032000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 msgr2=0x7f832411b720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.158+0000 7f83227fc640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83241311a0 0x7f832411b720 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8324064050 tx=0x7f831c032000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832c6ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832c6ef640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 0x7f832411c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832c6ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832411e170 con 0x7f832411bc60 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 0x7f832411c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient: get_auth_request con 0x7f832411bc60 auth_method 0 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 10 monclient(hunting): handle_auth_done global_id 34124 payload 824 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.164+0000 7f832a48b640 1 -- 172.21.15.116:0/1362801854 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f832411c5a0 con 0x7f832411bc60 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.166+0000 7f832a48b640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 msgr2=0x7f832411c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.166+0000 7f832a48b640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 0x7f832411c060 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83240686d0 tx=0x7f831c03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c031ea0 con 0x7f832411bc60 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:26:23.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 10 monclient: dump: 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f831c03ca50 con 0x7f832411bc60 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 msgr2=0x7f832411c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 0x7f832411c060 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83240686d0 tx=0x7f831c03d000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 msgr2=0x7f832411c060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832411bc60 0x7f832411c060 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83240686d0 tx=0x7f831c03d000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:23.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 14, 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.183 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.184 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:23.185 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.186 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.187 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.189 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:23.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:23.190 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.190 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.190 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:23.191 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.194 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.195 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:23.195 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f83100384c0 msgr2=0x7f831003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f83100384c0 0x7f831003a970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8314007fd0 tx=0x7f83140025b0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 10 monclient: shutdown 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 msgr2=0x7f832409abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 --2- 172.21.15.116:0/1362801854 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f832409ea50 0x7f832409abc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f831800ec80 tx=0x7f83180090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832ac8c640 1 -- 172.21.15.116:0/1362801854 reap_dead start 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832ac8c640 1 -- 172.21.15.116:0/1362801854 reap_dead start 2024-09-18T12:26:23.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f83227fc640 1 -- 172.21.15.116:0/1362801854 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (unknown 0 0 0) 0x7f831804ba80 con 0x7f832409ea50 2024-09-18T12:26:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 shutdown_connections 2024-09-18T12:26:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 >> 172.21.15.116:0/1362801854 conn(0x7f8324067110 msgr2=0x7f832409dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 shutdown_connections 2024-09-18T12:26:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.167+0000 7f832c6ef640 1 -- 172.21.15.116:0/1362801854 wait complete. 2024-09-18T12:26:23.197 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:26:23.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:26:23.197 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:26:23.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.291+0000 7f8b0c39d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.291+0000 7f8b0c39d640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.291+0000 7f8b0c39d640 10 monclient: monmap: 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:23.293126+0000 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:23.293126+0000 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:23.295 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:23.295 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:23.295 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.291+0000 7f8b0c39d640 1 Processor -- start 2024-09-18T12:26:23.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 1 -- start start 2024-09-18T12:26:23.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient: init 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b04131980 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8b0409fe10 0x7f8b0409ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b0409fe10 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b0409ef70 0x7f8b04138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f8b0409ef70 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b040a14d0 con 0x7f8b04131980 2024-09-18T12:26:23.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b041325b0 con 0x7f8b0409ef70 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b040a11f0 con 0x7f8b0409fe10 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0c39d640 10 monclient(hunting): authenticate will time out at 810.614658s 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b0409ef70 0x7f8b04138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a139640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8b04131980 0x7f8b04131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47754/0 (socket says 172.21.15.116:47754) 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a139640 1 -- 172.21.15.116:0/45091911 learned_addr learned my addr 172.21.15.116:0/45091911 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 10 monclient(hunting): get_auth_request con 0x7f8b0409ef70 auth_method 0 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b09938640 1 --2- 172.21.15.116:0/45091911 >> v2:172.21.15.144:3300/0 conn(0x7f8b0409fe10 0x7f8b0409ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b0a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b09938640 10 monclient(hunting): get_auth_request con 0x7f8b0409fe10 auth_method 0 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b09938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b09938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b09938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b09938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.292+0000 7f8b09938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b09938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 10 monclient(hunting): handle_auth_done global_id 4485 payload 274 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 1 -- 172.21.15.116:0/45091911 >> v2:172.21.15.144:3300/0 conn(0x7f8b0409fe10 msgr2=0x7f8b0409ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 1 --2- 172.21.15.116:0/45091911 >> v2:172.21.15.144:3300/0 conn(0x7f8b0409fe10 0x7f8b0409ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 1 -- 172.21.15.116:0/45091911 >> v1:172.21.15.116:6789/0 conn(0x7f8b04131980 legacy=0x7f8b04131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:23.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 1 monclient: found mon.noname-a 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.293+0000 7f8b0a93a640 1 -- 172.21.15.116:0/45091911 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0409f390 con 0x7f8b0409ef70 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.295+0000 7f8b0a93a640 1 -- 172.21.15.116:0/45091911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409ef70 msgr2=0x7f8b04138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.295+0000 7f8b0a93a640 1 --2- 172.21.15.116:0/45091911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409ef70 0x7f8b04138cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b0409c9c0 tx=0x7f8af40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef9f83b5e4aa250 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.295+0000 7f8b09137640 1 -- 172.21.15.116:0/45091911 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af402adf0 con 0x7f8b0409ef70 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.295+0000 7f8b09137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.295+0000 7f8b09137640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.295+0000 7f8b09137640 10 monclient: dump: 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 1 -- 172.21.15.116:0/45091911 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af4032440 con 0x7f8b0409ef70 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 5 monclient: authenticate success, global_id 4485 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 1 -- 172.21.15.116:0/45091911 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af40327e0 con 0x7f8b0409ef70 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 4 set_mon_vals no callback set 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b09137640 10 monclient: dump: 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.312 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 10 monclient: shutdown 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 -- 172.21.15.116:0/45091911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409ef70 msgr2=0x7f8b04138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/45091911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409ef70 0x7f8b04138cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b0409c9c0 tx=0x7f8af40090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 -- 172.21.15.116:0/45091911 shutdown_connections 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/45091911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409ef70 0x7f8b04138cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/45091911 >> v2:172.21.15.144:3300/0 conn(0x7f8b0409fe10 0x7f8b0409ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 -- 172.21.15.116:0/45091911 >> 172.21.15.116:0/45091911 conn(0x7f8b040a5050 msgr2=0x7f8b040a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 -- 172.21.15.116:0/45091911 shutdown_connections 2024-09-18T12:26:23.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.296+0000 7f8b0c39d640 1 -- 172.21.15.116:0/45091911 wait complete. 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient: monmap: 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 Processor -- start 2024-09-18T12:26:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 -- start start 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient: init 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b0409ef70 0x7f8b04134610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f8b0409ef70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 0x7f8b04134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f8b0409fe10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04131980 0x7f8b04135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f8b04131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b04069eb0 con 0x7f8b0409fe10 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b040a4200 con 0x7f8b04131980 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b04137f60 con 0x7f8b0409ef70 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.297+0000 7f8b0c39d640 10 monclient(hunting): authenticate will time out at 810.619938s 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 0x7f8b04134b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b0409ef70 0x7f8b04134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 0x7f8b04134b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42182/0 (socket says 172.21.15.116:42182) 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b0409ef70 0x7f8b04134610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36786/0 (socket says 172.21.15.116:36786) 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 -- 172.21.15.116:0/2398042550 learned_addr learned my addr 172.21.15.116:0/2398042550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): get_auth_request con 0x7f8b0409fe10 auth_method 0 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 10 monclient(hunting): get_auth_request con 0x7f8b0409ef70 auth_method 0 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04131980 0x7f8b04135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 10 monclient(hunting): get_auth_request con 0x7f8b04131980 auth_method 0 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient(hunting): handle_auth_done global_id 4488 payload 824 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b0409ef70 msgr2=0x7f8b04134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b0409ef70 0x7f8b04134610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04131980 msgr2=0x7f8b04135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b04131980 0x7f8b04135090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 monclient: found mon.noname-a 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b09938640 1 -- 172.21.15.116:0/2398042550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af4009390 con 0x7f8b0409fe10 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.298+0000 7f8b0a139640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b0409ef70 0x7f8b04134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b09938640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 msgr2=0x7f8b04134b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b09938640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 0x7f8b04134b50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8af800ea20 tx=0x7f8af80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af8009600 con 0x7f8b0409fe10 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: dump: 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b0c39d640 5 monclient: authenticate success, global_id 4488 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af800cc20 con 0x7f8b0409fe10 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b0c39d640 10 monclient: _renew_subs 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b0c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b041363f0 con 0x7f8b0409fe10 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af8013470 con 0x7f8b0409fe10 2024-09-18T12:26:23.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b027fc640 10 monclient: dump: 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b0c39d640 10 monclient: _renew_subs 2024-09-18T12:26:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b08936640 4 set_mon_vals no callback set 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b0c39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.300+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b04137c80 con 0x7f8b0409fe10 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 0x7f8b0402aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b04065e00 con 0x7f8b041355d0 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8af8013820 con 0x7f8b0409fe10 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 0x7f8b0402aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient: get_auth_request con 0x7f8b041355d0 auth_method 0 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b027fc640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8af0038490 0x7f8af003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 10 monclient(hunting): handle_auth_done global_id 4417 payload 824 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f8af804bd70 con 0x7f8b0409fe10 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.301+0000 7f8b0a139640 1 -- 172.21.15.116:0/2398042550 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b0404bf60 con 0x7f8b041355d0 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.302+0000 7f8b0a93a640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8af0038490 0x7f8af003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.302+0000 7f8b0a93a640 10 monclient: get_auth_request con 0x7f8af0038490 auth_method 0 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b0a139640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 msgr2=0x7f8b0402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b0a139640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 0x7f8b0402aa40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00d3b0 tx=0x7f8afc009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc007320 con 0x7f8b041355d0 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 10 monclient: dump: 2024-09-18T12:26:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b0a93a640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8af0038490 0x7f8af003a940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8b0406f930 tx=0x7f8af4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8afc007480 con 0x7f8b041355d0 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 msgr2=0x7f8b0402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 0x7f8b0402aa40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00d3b0 tx=0x7f8afc009040 comp rx=0 tx=0).stop 2024-09-18T12:26:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 msgr2=0x7f8b0402aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:23.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.304+0000 7f8b027fc640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b041355d0 0x7f8b0402aa40 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00d3b0 tx=0x7f8afc009040 comp rx=0 tx=0).stop 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0c39d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 0x7f8b0405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b04136ab0 con 0x7f8b0405e310 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 0x7f8b0405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient: get_auth_request con 0x7f8b0405e310 auth_method 0 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.310+0000 7f8b0a139640 1 -- 172.21.15.116:0/2398042550 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8b040a4d40 con 0x7f8b0405e310 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.312+0000 7f8b0a139640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 msgr2=0x7f8b0405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.312+0000 7f8b0a139640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 0x7f8b0405e710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00d3b0 tx=0x7f8afc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc023600 con 0x7f8b0405e310 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:23.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 10 monclient: dump: 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f8afc023760 con 0x7f8b0405e310 2024-09-18T12:26:23.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:23.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:23.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 msgr2=0x7f8b0405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 0x7f8b0405e710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00d3b0 tx=0x7f8afc017000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 msgr2=0x7f8b0405e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:23.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b027fc640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b0405e310 0x7f8b0405e710 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00d3b0 tx=0x7f8afc017000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.330 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:23.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 14, 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:23.331 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.332 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.333 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:23.334 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:23.335 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:23.336 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.337 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.338 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.339 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:23.340 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:23.342 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:26:23.343 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:23.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:23.345 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:23.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:23.345 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:23.345 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:23.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8af0038490 msgr2=0x7f8af003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8af0038490 0x7f8af003a940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8b0406f930 tx=0x7f8af4032000 comp rx=0 tx=0).stop 2024-09-18T12:26:23.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 10 monclient: shutdown 2024-09-18T12:26:23.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 msgr2=0x7f8b04134b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 0x7f8b04134b50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8af800ea20 tx=0x7f8af80090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0a93a640 1 -- 172.21.15.116:0/2398042550 reap_dead start 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 shutdown_connections 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 --2- 172.21.15.116:0/2398042550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b0409fe10 0x7f8b04134b50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 >> 172.21.15.116:0/2398042550 conn(0x7f8b040a5050 msgr2=0x7f8b0409d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 shutdown_connections 2024-09-18T12:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.313+0000 7f8b0c39d640 1 -- 172.21.15.116:0/2398042550 wait complete. 2024-09-18T12:26:23.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:26:23.348 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:26:23.348 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:26:23.348 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:26:23.348 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:26:23.348 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:26:23.348 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:26:23.357 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:26:23.357 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:26:23.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:23.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.487+0000 7fede4eef640 10 monclient: get_monmap_and_config 2024-09-18T12:26:23.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.487+0000 7fede4eef640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.487+0000 7fede4eef640 10 monclient: monmap: 2024-09-18T12:26:23.490 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.490 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.490 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:23.489477+0000 2024-09-18T12:26:23.490 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:23.489477+0000 2024-09-18T12:26:23.490 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.491 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.491 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:23.491 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:23.491 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:23.491 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:23.491 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:23.491 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.488+0000 7fede4eef640 1 Processor -- start 2024-09-18T12:26:23.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.488+0000 7fede4eef640 1 -- start start 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.488+0000 7fede4eef640 10 monclient: init 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 0x7fede012f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient(hunting): picked mon.noname-b con 0x7fede0067110 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fede009b010 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fede009b400 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede0130540 con 0x7fede009b010 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede0099800 con 0x7fede009b400 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede0067ed0 con 0x7fede0067110 2024-09-18T12:26:23.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fedde59c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 msgr2=0x7fede012f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:23.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fede4eef640 10 monclient(hunting): authenticate will time out at 810.811246s 2024-09-18T12:26:23.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fedde59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 0x7fede012f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:23.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fedddd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fede009b010 0x7fede009c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47768/0 (socket says 172.21.15.116:47768) 2024-09-18T12:26:23.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.489+0000 7fedddd9b640 1 -- 172.21.15.116:0/673982820 learned_addr learned my addr 172.21.15.116:0/673982820 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7fedddd9b640 1 -- 172.21.15.116:0/673982820 >> v1:172.21.15.116:6789/0 conn(0x7fede009b010 legacy=0x7fede009c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7fedded9d640 1 -- 172.21.15.116:0/673982820 >> v1:172.21.15.144:6789/0 conn(0x7fede009b400 legacy=0x7fede009cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5380065 0 0) 0x7fede0130540 con 0x7fede009b010 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient(hunting): my global_id is 4491 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc80034e0 con 0x7fede009b010 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919098613 0 0) 0x7fede0099800 con 0x7fede009b400 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient(hunting): my global_id is 4423 2024-09-18T12:26:23.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede0130540 con 0x7fede009b400 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597172644 0 0) 0x7fedc80034e0 con 0x7fede009b010 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 msgr2=0x7fede012f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 0x7fede012f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 >> v1:172.21.15.144:6789/0 conn(0x7fede009b400 legacy=0x7fede009cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00674e0 con 0x7fede009b010 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7fede4eef640 5 monclient: authenticate success, global_id 4491 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7fede4eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fedd4001860 con 0x7fede009b010 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: dump: 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.505 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 >> v1:172.21.15.116:6789/0 conn(0x7fede009b010 legacy=0x7fede009c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.491+0000 7feddd59a640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 0x7fedc8003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): picked mon.c con 0x7fede0067110 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedc8004200 0x7fedc8004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): picked mon.a con 0x7fedc8004200 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedc8004ec0 0x7fedc800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): picked mon.b con 0x7fedc8004ec0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:23.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc800bf10 con 0x7fedc8004200 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedde59c640 1 -- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 msgr2=0x7fedc8003c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc800c050 con 0x7fedc8004ec0 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc800c190 con 0x7fede0067110 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedde59c640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 0x7fedc8003c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7feddd59a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fede4eef640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedc8004200 0x7fedc8004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): get_auth_request con 0x7fedc8004200 auth_method 0 2024-09-18T12:26:23.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedc8004ec0 0x7fedc800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 10 monclient(hunting): get_auth_request con 0x7fedc8004ec0 auth_method 0 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedded9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient(hunting): handle_auth_done global_id 4491 payload 293 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 1 -- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 msgr2=0x7fedc8003c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede0067110 0x7fedc8003c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 1 -- 172.21.15.116:0/673982820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedc8004ec0 msgr2=0x7fedc800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedc8004ec0 0x7fedc800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 1 monclient: found mon.a 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 1 -- 172.21.15.116:0/673982820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc80034e0 con 0x7fedc8004200 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.492+0000 7fedddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fedddd9b640 1 -- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedc8004200 msgr2=0x7fedc8004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fedddd9b640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedc8004200 0x7fedc8004780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fedd400c5f0 tx=0x7fedd4007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65066581f9cc8d73 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4005890 con 0x7fedc8004200 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 10 monclient: dump: 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.510 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd4005a10 con 0x7fedc8004200 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 1 -- 172.21.15.116:0/673982820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4005dd0 con 0x7fedc8004200 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 4 set_mon_vals no callback set 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7feddd59a640 10 monclient: dump: 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.511 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 10 monclient: shutdown 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 1 -- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedc8004200 msgr2=0x7fedc8004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 1 --2- 172.21.15.116:0/673982820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedc8004200 0x7fedc8004780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fedd400c5f0 tx=0x7fedd4007a30 comp rx=0 tx=0).stop 2024-09-18T12:26:23.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fedded9d640 1 -- 172.21.15.116:0/673982820 reap_dead start 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 1 -- 172.21.15.116:0/673982820 shutdown_connections 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 1 -- 172.21.15.116:0/673982820 >> 172.21.15.116:0/673982820 conn(0x7fede00a1e30 msgr2=0x7fede00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 1 -- 172.21.15.116:0/673982820 shutdown_connections 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.495+0000 7fede4eef640 1 -- 172.21.15.116:0/673982820 wait complete. 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.496+0000 7fede4eef640 10 monclient: build_initial_monmap 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.496+0000 7fede4eef640 10 monclient: monmap: 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:23.513 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.496+0000 7fede4eef640 1 Processor -- start 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.496+0000 7fede4eef640 1 -- start start 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.496+0000 7fede4eef640 10 monclient: init 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedc8004200 0x7fede012e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): picked mon.noname-c con 0x7fedc8004200 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede012ed00 0x7fede0118490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): picked mon.noname-b con 0x7fede012ed00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:23.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 0x7fede0118e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): picked mon.noname-a con 0x7fede01189d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede012f0d0 con 0x7fede01189d0 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede009e330 con 0x7fede012ed00 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede012f7a0 con 0x7fedc8004200 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fede4eef640 10 monclient(hunting): authenticate will time out at 810.819105s 2024-09-18T12:26:23.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedde59c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedc8004200 msgr2=0x7fede012e7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedde59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedc8004200 0x7fede012e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 0x7fede0118e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 0x7fede0118e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42194/0 (socket says 172.21.15.116:42194) 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 -- 172.21.15.116:0/176073211 learned_addr learned my addr 172.21.15.116:0/176073211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): get_auth_request con 0x7fede01189d0 auth_method 0 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:23.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedddd9b640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede012ed00 0x7fede0118490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient(hunting): handle_auth_done global_id 4494 payload 824 2024-09-18T12:26:23.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 -- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedc8004200 msgr2=0x7fede012e7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:23.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedc8004200 0x7fede012e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 -- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede012ed00 msgr2=0x7fede0118490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede012ed00 0x7fede0118490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 monclient: found mon.noname-a 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedded9d640 1 -- 172.21.15.116:0/176073211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede0131e00 con 0x7fede01189d0 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedddd9b640 10 monclient: get_auth_request con 0x7fede012ed00 auth_method 0 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.497+0000 7fedddd9b640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede012ed00 0x7fede0118490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedded9d640 1 -- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 msgr2=0x7fede0118e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedded9d640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 0x7fede0118e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fedd8001490 tx=0x7fedd8001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 1 -- 172.21.15.116:0/176073211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8005d10 con 0x7fede01189d0 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: dump: 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:23.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 1 -- 172.21.15.116:0/176073211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd8005e70 con 0x7fede01189d0 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fede4eef640 5 monclient: authenticate success, global_id 4494 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fede4eef640 10 monclient: _renew_subs 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fede4eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede011b940 con 0x7fede01189d0 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 1 -- 172.21.15.116:0/176073211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8006260 con 0x7fede01189d0 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fedce7fc640 10 monclient: dump: 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fede4eef640 10 monclient: _renew_subs 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fede4eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede011a980 con 0x7fede01189d0 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.499+0000 7feddcd99640 4 set_mon_vals no callback set 2024-09-18T12:26:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.500+0000 7fedce7fc640 1 -- 172.21.15.116:0/176073211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fedd8006650 con 0x7fede01189d0 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.500+0000 7fede4eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.500+0000 7fede4eef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.500+0000 7fede4eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.500+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede011bd60 con 0x7fede01189d0 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.500+0000 7fedce7fc640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedc0038470 0x7fedc003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.501+0000 7fedce7fc640 1 -- 172.21.15.116:0/176073211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fedd804a4c0 con 0x7fede01189d0 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.501+0000 7fedde59c640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedc0038470 0x7fedc003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.501+0000 7fedde59c640 10 monclient: get_auth_request con 0x7fedc0038470 auth_method 0 2024-09-18T12:26:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.503+0000 7fedde59c640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedc0038470 0x7fedc003a920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fedd0007d50 tx=0x7fedd000d2c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.505+0000 7fedce7fc640 1 -- 172.21.15.116:0/176073211 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fedd800f260 con 0x7fede01189d0 2024-09-18T12:26:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.505+0000 7fedce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.505+0000 7fedce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.717+0000 7fede4eef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.717+0000 7fede4eef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.717+0000 7fede4eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.717+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fede0119e60 con 0x7fede01189d0 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.718+0000 7fedce7fc640 1 -- 172.21.15.116:0/176073211 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fedd8019080 con 0x7fede01189d0 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.718+0000 7fedce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.718+0000 7fedce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:23.720 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:26:23.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedc0038470 msgr2=0x7fedc003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedc0038470 0x7fedc003a920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fedd0007d50 tx=0x7fedd000d2c0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 10 monclient: shutdown 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 msgr2=0x7fede0118e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 0x7fede0118e10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fedd8001490 tx=0x7fedd8001a30 comp rx=0 tx=0).stop 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 shutdown_connections 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedc0038470 0x7fedc003a920 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede01189d0 0x7fede0118e10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede012ed00 0x7fede0118490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 --2- 172.21.15.116:0/176073211 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedc8004200 0x7fede012e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.719+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 >> 172.21.15.116:0/176073211 conn(0x7fede00a1e30 msgr2=0x7fede01306d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.720+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 shutdown_connections 2024-09-18T12:26:23.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:23.720+0000 7fede4eef640 1 -- 172.21.15.116:0/176073211 wait complete. 2024-09-18T12:26:25.890 INFO:tasks.workunit.client.0.smithi144.stderr:Note: switching to '7061ea6ca50908d104e9f3979dc615e118d9e153'. 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr: git switch -c 2024-09-18T12:26:25.891 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:25.892 INFO:tasks.workunit.client.0.smithi144.stderr:Or undo this operation with: 2024-09-18T12:26:25.892 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:25.892 INFO:tasks.workunit.client.0.smithi144.stderr: git switch - 2024-09-18T12:26:25.892 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:25.892 INFO:tasks.workunit.client.0.smithi144.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T12:26:25.892 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:25.892 INFO:tasks.workunit.client.0.smithi144.stderr:HEAD is now at 7061ea6ca50 Merge branch 'wip-65946-quincy' of https://github.com/amathuria/ceph into wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T12:26:25.895 DEBUG:teuthology.orchestra.run.smithi144:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T12:26:25.981 INFO:tasks.workunit.client.0.smithi144.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-18T12:26:25.983 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T12:26:25.983 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-18T12:26:26.608 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-18T12:26:26.678 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-18T12:26:26.718 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T12:26:26.720 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T12:26:26.720 INFO:tasks.workunit.client.0.smithi144.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-18T12:26:26.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:26.758 INFO:tasks.workunit.client.0.smithi144.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T12:26:26.761 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-18T12:26:26.761 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T12:26:26.777 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-18T12:26:26.778 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-18T12:26:26.778 DEBUG:teuthology.orchestra.run.smithi144:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-18T12:26:26.836 INFO:tasks.workunit.client.0.smithi144.stderr:+ TEST_POOL=testpool1234 2024-09-18T12:26:26.836 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-18T12:26:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.835+0000 7f2fb52c9640 10 monclient: get_monmap_and_config 2024-09-18T12:26:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.835+0000 7f2fb52c9640 10 monclient: build_initial_monmap 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.836+0000 7f2fb52c9640 10 monclient: monmap: 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:26.837663+0000 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:26.837663+0000 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:26.839 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:26.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.836+0000 7f2fb52c9640 1 Processor -- start 2024-09-18T12:26:26.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.836+0000 7f2fb52c9640 1 -- start start 2024-09-18T12:26:26.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.836+0000 7f2fb52c9640 10 monclient: init 2024-09-18T12:26:26.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:26.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2fb0131980 0x7f2fb0131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb0131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb0132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb009fda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fb009e9c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb009f510 con 0x7f2fb009e9c0 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb00a1180 con 0x7f2fb0131980 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00a1460 con 0x7f2fb009fda0 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:26.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fb52c9640 10 monclient(hunting): authenticate will time out at 814.159126s 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fae7fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 msgr2=0x7f2fb0132290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2fae7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb0132290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2fb0131980 0x7f2fb0131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2fb0131980 0x7f2fb0131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41522/0 (socket says 172.21.15.116:41522) 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 -- 172.21.15.116:0/3502712249 learned_addr learned my addr 172.21.15.116:0/3502712249 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): get_auth_request con 0x7f2fb0131980 auth_method 0 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faf7fe640 1 --1- 172.21.15.116:0/3502712249 >> v1:172.21.15.116:6789/0 conn(0x7f2fb009e9c0 0x7f2fb009ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59724/0 (socket says 172.21.15.116:59724) 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:26.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient(hunting): handle_auth_done global_id 4497 payload 274 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 -- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 msgr2=0x7f2fb0132290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 --2- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb0132290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 -- 172.21.15.116:0/3502712249 >> v1:172.21.15.116:6789/0 conn(0x7f2fb009e9c0 legacy=0x7f2fb009ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 monclient: found mon.noname-a 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:26.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.837+0000 7f2faeffd640 1 -- 172.21.15.116:0/3502712249 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0139cd0 con 0x7f2fb0131980 2024-09-18T12:26:26.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2faeffd640 1 -- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 msgr2=0x7f2fb0131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:26.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2faeffd640 1 --2- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0131d50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009f20 tx=0x7f2fa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f2f079fd4bc1bc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:26.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 1 -- 172.21.15.116:0/3502712249 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8007610 con 0x7f2fb0131980 2024-09-18T12:26:26.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:26.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: dump: 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: _finish_auth 0 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:26.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 1 -- 172.21.15.116:0/3502712249 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa802d050 con 0x7f2fb0131980 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fb52c9640 5 monclient: authenticate success, global_id 4497 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fb52c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fb52c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 1 -- 172.21.15.116:0/3502712249 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa80079e0 con 0x7f2fb0131980 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fb52c9640 4 set_mon_vals no callback set 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.840+0000 7f2fadffb640 10 monclient: dump: 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 10 monclient: shutdown 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 -- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 msgr2=0x7f2fb0131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0131d50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009f20 tx=0x7f2fa8009840 comp rx=0 tx=0).stop 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 -- 172.21.15.116:0/3502712249 shutdown_connections 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb0132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/3502712249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0131d50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 -- 172.21.15.116:0/3502712249 >> 172.21.15.116:0/3502712249 conn(0x7f2fb0067bd0 msgr2=0x7f2fb00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 -- 172.21.15.116:0/3502712249 shutdown_connections 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 1 -- 172.21.15.116:0/3502712249 wait complete. 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 10 monclient: build_initial_monmap 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.841+0000 7f2fb52c9640 10 monclient: monmap: 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 Processor -- start 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 -- start start 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient: init 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb009e9c0 0x7f2fb0137540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb009e9c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb01411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb009fda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb0131980 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0136e80 con 0x7f2fb0131980 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0136ba0 con 0x7f2fb009e9c0 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb01365e0 con 0x7f2fb009fda0 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fb52c9640 10 monclient(hunting): authenticate will time out at 814.164708s 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fae7fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 msgr2=0x7f2fb01411c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2fae7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb01411c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0141700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41524/0 (socket says 172.21.15.116:41524) 2024-09-18T12:26:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 1 -- 172.21.15.116:0/274995213 learned_addr learned my addr 172.21.15.116:0/274995213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 10 monclient(hunting): get_auth_request con 0x7f2fb0131980 auth_method 0 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faeffd640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb009e9c0 0x7f2fb0137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faeffd640 10 monclient(hunting): get_auth_request con 0x7f2fb009e9c0 auth_method 0 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.842+0000 7f2faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 10 monclient(hunting): handle_auth_done global_id 4500 payload 824 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 1 -- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 msgr2=0x7f2fb01411c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb01411c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 1 -- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb009e9c0 msgr2=0x7f2fb0137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb009e9c0 0x7f2fb0137540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 1 monclient: found mon.noname-a 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.843+0000 7f2faf7fe640 1 -- 172.21.15.116:0/274995213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa8009ae0 con 0x7f2fb0131980 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2faf7fe640 1 -- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 msgr2=0x7f2fb0141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2faf7fe640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0141700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0007fd0 tx=0x7f2fa0007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 1 -- 172.21.15.116:0/274995213 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00074f0 con 0x7f2fb0131980 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: dump: 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fb52c9640 5 monclient: authenticate success, global_id 4500 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 1 -- 172.21.15.116:0/274995213 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa0007670 con 0x7f2fb0131980 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 1 -- 172.21.15.116:0/274995213 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0005a00 con 0x7f2fb0131980 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fb52c9640 10 monclient: _renew_subs 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fb52c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb01368c0 con 0x7f2fb0131980 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fad7fa640 4 set_mon_vals no callback set 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fa6ffd640 10 monclient: dump: 2024-09-18T12:26:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fb52c9640 10 monclient: _renew_subs 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fb52c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.845+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb01356f0 con 0x7f2fb0131980 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2fa6ffd640 1 -- 172.21.15.116:0/274995213 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2fa001b3b0 con 0x7f2fb0131980 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2fb52c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2fb52c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2fb52c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb004bf60 con 0x7f2fb0131980 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2fa6ffd640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f90038440 0x7f2f9003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2fa6ffd640 1 -- 172.21.15.116:0/274995213 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f2fa004ab60 con 0x7f2fb0131980 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2faeffd640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f90038440 0x7f2f9003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.846+0000 7f2faeffd640 10 monclient: get_auth_request con 0x7f2f90038440 auth_method 0 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.849+0000 7f2faeffd640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f90038440 0x7f2f9003a8f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00a0170 tx=0x7f2fa8036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.851+0000 7f2fa6ffd640 1 -- 172.21.15.116:0/274995213 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2fa001be60 con 0x7f2fb0131980 2024-09-18T12:26:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.851+0000 7f2fa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:26.851+0000 7f2fa6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:26.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient: get_monmap_and_config 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient: build_initial_monmap 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient: monmap: 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:26.912566+0000 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:26.912566+0000 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:26.913 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:26.914 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:26.914 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:26.914 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:26.914 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:26.914 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:26.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 Processor -- start 2024-09-18T12:26:26.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 -- start start 2024-09-18T12:26:26.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient: init 2024-09-18T12:26:26.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:26.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:26:26.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed1c096a30 0x7fed1c09d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c096a30 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fed1c096e00 0x7fed1c097200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): picked mon.noname-a con 0x7fed1c096e00 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c12c6a0 0x7fed1c12cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): picked mon.noname-b con 0x7fed1c12c6a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1c09b970 con 0x7fed1c096e00 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1c09c400 con 0x7fed1c096a30 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c09c120 con 0x7fed1c12c6a0 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed2164f640 10 monclient(hunting): authenticate will time out at 822.481879s 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed1affd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed1c096a30 0x7fed1c09d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.911+0000 7fed1b7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c12c6a0 msgr2=0x7fed1c12cb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c12c6a0 0x7fed1c12cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed1c096a30 0x7fed1c09d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54386/0 (socket says 172.21.15.144:54386) 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 -- 172.21.15.144:0/4286083357 learned_addr learned my addr 172.21.15.144:0/4286083357 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): get_auth_request con 0x7fed1c096a30 auth_method 0 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:26.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 1 --2- 172.21.15.144:0/4286083357 >> v2:172.21.15.116:3300/0 conn(0x7fed1c096e00 0x7fed1c097200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 10 monclient(hunting): get_auth_request con 0x7fed1c096e00 auth_method 0 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient(hunting): handle_auth_done global_id 4429 payload 274 2024-09-18T12:26:26.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 -- 172.21.15.144:0/4286083357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c12c6a0 msgr2=0x7fed1c12cb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 --2- 172.21.15.144:0/4286083357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c12c6a0 0x7fed1c12cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 -- 172.21.15.144:0/4286083357 >> v2:172.21.15.116:3300/0 conn(0x7fed1c096e00 msgr2=0x7fed1c097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 --2- 172.21.15.144:0/4286083357 >> v2:172.21.15.116:3300/0 conn(0x7fed1c096e00 0x7fed1c097200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient: _finish_hunting 0 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 monclient: found mon.noname-c 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.912+0000 7fed1affd640 1 -- 172.21.15.144:0/4286083357 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c09c6e0 con 0x7fed1c096a30 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1affd640 1 -- 172.21.15.144:0/4286083357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 msgr2=0x7fed1c09d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1affd640 1 --2- 172.21.15.144:0/4286083357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c09d090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009910 tx=0x7fed0c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70ef7b0e31337822 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 1 -- 172.21.15.144:0/4286083357 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c032440 con 0x7fed1c096a30 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:26.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: dump: 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:26.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 1 -- 172.21.15.144:0/4286083357 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c0325a0 con 0x7fed1c096a30 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 5 monclient: authenticate success, global_id 4429 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 4 set_mon_vals no callback set 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 1 -- 172.21.15.144:0/4286083357 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c032940 con 0x7fed1c096a30 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed1a7fc640 10 monclient: dump: 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.924 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 10 monclient: shutdown 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 -- 172.21.15.144:0/4286083357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 msgr2=0x7fed1c09d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 --2- 172.21.15.144:0/4286083357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c09d090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009910 tx=0x7fed0c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 -- 172.21.15.144:0/4286083357 shutdown_connections 2024-09-18T12:26:26.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 --2- 172.21.15.144:0/4286083357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c12c6a0 0x7fed1c12cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 --2- 172.21.15.144:0/4286083357 >> v2:172.21.15.116:3300/0 conn(0x7fed1c096e00 0x7fed1c097200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 --2- 172.21.15.144:0/4286083357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c09d090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 -- 172.21.15.144:0/4286083357 >> 172.21.15.144:0/4286083357 conn(0x7fed1c0711a0 msgr2=0x7fed1c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 -- 172.21.15.144:0/4286083357 shutdown_connections 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 -- 172.21.15.144:0/4286083357 wait complete. 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 10 monclient: build_initial_monmap 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 10 monclient: monmap: 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:26.926 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:26.927 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:26.927 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:26.927 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:26.927 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:26.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 Processor -- start 2024-09-18T12:26:26.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 1 -- start start 2024-09-18T12:26:26.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.916+0000 7fed2164f640 10 monclient: init 2024-09-18T12:26:26.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:26.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:26.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c13ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): picked mon.noname-b con 0x7fed1c096a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c096e00 0x7fed1c12b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c096e00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed1c12c6a0 0x7fed1c130750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): picked mon.noname-a con 0x7fed1c12c6a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c132310 con 0x7fed1c12c6a0 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c118cc0 con 0x7fed1c096a30 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c117c30 con 0x7fed1c096e00 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:26.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed2164f640 10 monclient(hunting): authenticate will time out at 822.487341s 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed127fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c096e00 msgr2=0x7fed1c12b8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed127fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c096e00 0x7fed1c12b8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c13ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c13ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54390/0 (socket says 172.21.15.144:54390) 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 -- 172.21.15.144:0/1356173230 learned_addr learned my addr 172.21.15.144:0/1356173230 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): get_auth_request con 0x7fed1c096a30 auth_method 0 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed1c12c6a0 0x7fed1c130750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request con 0x7fed1c12c6a0 auth_method 0 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:26.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient(hunting): handle_auth_done global_id 4432 payload 824 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 -- 172.21.15.144:0/1356173230 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c096e00 msgr2=0x7fed1c12b8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c096e00 0x7fed1c12b8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 -- 172.21.15.144:0/1356173230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed1c12c6a0 msgr2=0x7fed1c130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:26.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed1c12c6a0 0x7fed1c130750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient: _finish_hunting 0 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 monclient: found mon.noname-b 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.917+0000 7fed1affd640 1 -- 172.21.15.144:0/1356173230 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c009450 con 0x7fed1c096a30 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed1affd640 1 -- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 msgr2=0x7fed1c13ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed1affd640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c13ef50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed0c035040 tx=0x7fed0c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c031a90 con 0x7fed1c096a30 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: dump: 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:26.935 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c031bf0 con 0x7fed1c096a30 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed2164f640 5 monclient: authenticate success, global_id 4432 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed2164f640 10 monclient: _renew_subs 2024-09-18T12:26:26.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed2164f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c119000 con 0x7fed1c096a30 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c03d440 con 0x7fed1c096a30 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed137fe640 10 monclient: dump: 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:26.937 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed19ffb640 4 set_mon_vals no callback set 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed2164f640 10 monclient: _renew_subs 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed2164f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.920+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c13fe40 con 0x7fed1c096a30 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fed0c039040 con 0x7fed1c096a30 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed2164f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed2164f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:26.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed2164f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c0008d0 con 0x7fed1c096a30 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed137fe640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fecfc038470 0x7fecfc03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fed0c06b0f0 con 0x7fed1c096a30 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed127fc640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fecfc038470 0x7fecfc03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.921+0000 7fed127fc640 10 monclient: get_auth_request con 0x7fecfc038470 auth_method 0 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.924+0000 7fed127fc640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fecfc038470 0x7fecfc03a920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed1c131fb0 tx=0x7fed040090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.926+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fed0c044af0 con 0x7fed1c096a30 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.926+0000 7fed137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:26.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:26.926+0000 7fed137fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.063+0000 7f2fb52c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.063+0000 7f2fb52c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.063+0000 7f2fb52c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.063+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2fb00638d0 con 0x7f2fb0131980 2024-09-18T12:26:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.064+0000 7f2fa6ffd640 1 -- 172.21.15.116:0/274995213 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2fa0017020 con 0x7f2fb0131980 2024-09-18T12:26:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.064+0000 7f2fa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.064+0000 7f2fa6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:27.067 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f90038440 msgr2=0x7f2f9003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f90038440 0x7f2f9003a8f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00a0170 tx=0x7f2fa8036000 comp rx=0 tx=0).stop 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 10 monclient: shutdown 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 msgr2=0x7f2fb0141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0141700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0007fd0 tx=0x7f2fa0007f00 comp rx=0 tx=0).stop 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 shutdown_connections 2024-09-18T12:26:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f90038440 0x7f2f9003a8f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fb0131980 0x7f2fb0141700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fb009fda0 0x7f2fb01411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 --2- 172.21.15.116:0/274995213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fb009e9c0 0x7f2fb0137540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 >> 172.21.15.116:0/274995213 conn(0x7f2fb0067bd0 msgr2=0x7f2fb009d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 shutdown_connections 2024-09-18T12:26:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:27.066+0000 7f2fb52c9640 1 -- 172.21.15.116:0/274995213 wait complete. 2024-09-18T12:26:27.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:27.080+0000 7fed2164f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T12:26:27.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:27.080+0000 7fed2164f640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T12:26:27.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:27.080+0000 7fed2164f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:27.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:27.080+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7fed1c119190 con 0x7fed1c096a30 2024-09-18T12:26:29.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:29.918+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:29.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:29.918+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:29.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:29.918+0000 7fed11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:30.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.182+0000 7f13c235d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.182+0000 7f13c235d640 10 monclient: build_initial_monmap 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.183+0000 7f13c235d640 10 monclient: monmap: 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:30.184672+0000 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:30.184672+0000 2024-09-18T12:26:30.185 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:30.186 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:30.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.183+0000 7f13c235d640 1 Processor -- start 2024-09-18T12:26:30.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.183+0000 7f13c235d640 1 -- start start 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.183+0000 7f13c235d640 10 monclient: init 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13bc06eb80 0x7f13bc06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc06eb80 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc068a20 0x7f13bc09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc068a20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f13bc09caf0 0x7f13bc09d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc09caf0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:30.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc06f7f0 con 0x7f13bc06eb80 2024-09-18T12:26:30.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:30.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc06a7a0 con 0x7f13bc09caf0 2024-09-18T12:26:30.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:30.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc06a0e0 con 0x7f13bc068a20 2024-09-18T12:26:30.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:30.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c235d640 10 monclient(hunting): authenticate will time out at 817.506453s 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bb7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc068a20 msgr2=0x7f13bc09c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bb7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc068a20 0x7f13bc09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13bc06eb80 0x7f13bc06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13bc06eb80 0x7f13bc06ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41526/0 (socket says 172.21.15.116:41526) 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 -- 172.21.15.116:0/228924399 learned_addr learned my addr 172.21.15.116:0/228924399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): get_auth_request con 0x7f13bc06eb80 auth_method 0 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient(hunting): handle_auth_done global_id 4509 payload 274 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 -- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc068a20 msgr2=0x7f13bc09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 --2- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc068a20 0x7f13bc09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13c08fa640 1 --2- 172.21.15.116:0/228924399 >> v2:172.21.15.144:3300/0 conn(0x7f13bc09caf0 0x7f13bc09d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 -- 172.21.15.116:0/228924399 >> v2:172.21.15.144:3300/0 conn(0x7f13bc09caf0 msgr2=0x7f13bc09d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 --2- 172.21.15.116:0/228924399 >> v2:172.21.15.144:3300/0 conn(0x7f13bc09caf0 0x7f13bc09d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 monclient: found mon.noname-a 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.184+0000 7f13bbfff640 1 -- 172.21.15.116:0/228924399 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc069e00 con 0x7f13bc06eb80 2024-09-18T12:26:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13bbfff640 1 -- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc06eb80 msgr2=0x7f13bc06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13bbfff640 1 --2- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc06eb80 0x7f13bc06ef50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13bc09f590 tx=0x7f13ac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd66bbfc51ec3b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 1 -- 172.21.15.116:0/228924399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac032440 con 0x7f13bc06eb80 2024-09-18T12:26:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: dump: 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:30.200 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 1 -- 172.21.15.116:0/228924399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac0325a0 con 0x7f13bc06eb80 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 1 -- 172.21.15.116:0/228924399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac032940 con 0x7f13bc06eb80 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13baffd640 10 monclient: dump: 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:30.201 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13c235d640 5 monclient: authenticate success, global_id 4509 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13c235d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.187+0000 7f13c235d640 4 set_mon_vals no callback set 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 10 monclient: shutdown 2024-09-18T12:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 -- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc06eb80 msgr2=0x7f13bc06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 --2- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc06eb80 0x7f13bc06ef50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13bc09f590 tx=0x7f13ac0091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 -- 172.21.15.116:0/228924399 shutdown_connections 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 --2- 172.21.15.116:0/228924399 >> v2:172.21.15.144:3300/0 conn(0x7f13bc09caf0 0x7f13bc09d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 --2- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc068a20 0x7f13bc09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 --2- 172.21.15.116:0/228924399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc06eb80 0x7f13bc06ef50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 -- 172.21.15.116:0/228924399 >> 172.21.15.116:0/228924399 conn(0x7f13bc067cd0 msgr2=0x7f13bc0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 -- 172.21.15.116:0/228924399 shutdown_connections 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.188+0000 7f13c235d640 1 -- 172.21.15.116:0/228924399 wait complete. 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.189+0000 7f13c235d640 10 monclient: build_initial_monmap 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.189+0000 7f13c235d640 10 monclient: monmap: 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.189+0000 7f13c235d640 1 Processor -- start 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.189+0000 7f13c235d640 1 -- start start 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.189+0000 7f13c235d640 10 monclient: init 2024-09-18T12:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13bc068a20 0x7f13bc140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc068a20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc06eb80 0x7f13bc134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc06eb80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 0x7f13bc11e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc09caf0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc09d050 con 0x7f13bc09caf0 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0a0880 con 0x7f13bc068a20 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1220d0 con 0x7f13bc06eb80 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bb7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc06eb80 msgr2=0x7f13bc134cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c235d640 10 monclient(hunting): authenticate will time out at 817.512196s 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bb7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc06eb80 0x7f13bc134cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 0x7f13bc11e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 0x7f13bc11e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41534/0 (socket says 172.21.15.116:41534) 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 -- 172.21.15.116:0/3943995501 learned_addr learned my addr 172.21.15.116:0/3943995501 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): get_auth_request con 0x7f13bc09caf0 auth_method 0 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bbfff640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13bc068a20 0x7f13bc140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bbfff640 10 monclient(hunting): get_auth_request con 0x7f13bc068a20 auth_method 0 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient(hunting): handle_auth_done global_id 4512 payload 824 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 -- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc06eb80 msgr2=0x7f13bc134cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc06eb80 0x7f13bc134cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 -- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13bc068a20 msgr2=0x7f13bc140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13bc068a20 0x7f13bc140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient: _finish_hunting 0 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 monclient: found mon.noname-a 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13c08fa640 1 -- 172.21.15.116:0/3943995501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac009450 con 0x7f13bc09caf0 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.190+0000 7f13bbfff640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13bc068a20 0x7f13bc140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13c08fa640 1 -- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 msgr2=0x7f13bc11e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13c08fa640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 0x7f13bc11e7d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f13b400ceb0 tx=0x7f13b4007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:30.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13a3fff640 1 -- 172.21.15.116:0/3943995501 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400e610 con 0x7f13bc09caf0 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13a3fff640 10 monclient: dump: 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13a3fff640 10 monclient: _finish_auth 0 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13a3fff640 1 -- 172.21.15.116:0/3943995501 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b400e770 con 0x7f13bc09caf0 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.192+0000 7f13c235d640 5 monclient: authenticate success, global_id 4512 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13a3fff640 1 -- 172.21.15.116:0/3943995501 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400eb30 con 0x7f13bc09caf0 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13c235d640 10 monclient: _renew_subs 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13c235d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:30.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc121df0 con 0x7f13bc09caf0 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13ba7fc640 4 set_mon_vals no callback set 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13a3fff640 10 monclient: dump: 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13c235d640 10 monclient: _renew_subs 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13c235d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.193+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc1413b0 con 0x7f13bc09caf0 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13a3fff640 1 -- 172.21.15.116:0/3943995501 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f13b4009030 con 0x7f13bc09caf0 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13c235d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13c235d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13c235d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc120320 con 0x7f13bc09caf0 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13a3fff640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a8038440 0x7f13a803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13a3fff640 1 -- 172.21.15.116:0/3943995501 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f13b4012070 con 0x7f13bc09caf0 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13bbfff640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a8038440 0x7f13a803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.194+0000 7f13bbfff640 10 monclient: get_auth_request con 0x7f13a8038440 auth_method 0 2024-09-18T12:26:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.197+0000 7f13bbfff640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a8038440 0x7f13a803a8f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f13bc064eb0 tx=0x7f13ac032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.198+0000 7f13a3fff640 1 -- 172.21.15.116:0/3943995501 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f13b40153d0 con 0x7f13bc09caf0 2024-09-18T12:26:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.198+0000 7f13a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.198+0000 7f13a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.413+0000 7f13c235d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.413+0000 7f13c235d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.413+0000 7f13c235d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.413+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f13bc04bf60 con 0x7f13bc09caf0 2024-09-18T12:26:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.413+0000 7f13a3fff640 1 -- 172.21.15.116:0/3943995501 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f13b4014330 con 0x7f13bc09caf0 2024-09-18T12:26:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.413+0000 7f13a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.413+0000 7f13a3fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:30.416 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:26:30.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a8038440 msgr2=0x7f13a803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a8038440 0x7f13a803a8f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f13bc064eb0 tx=0x7f13ac032000 comp rx=0 tx=0).stop 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 10 monclient: shutdown 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 msgr2=0x7f13bc11e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 0x7f13bc11e7d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f13b400ceb0 tx=0x7f13b4007870 comp rx=0 tx=0).stop 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 shutdown_connections 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a8038440 0x7f13a803a8f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13bc09caf0 0x7f13bc11e7d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13bc06eb80 0x7f13bc134cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 --2- 172.21.15.116:0/3943995501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13bc068a20 0x7f13bc140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 >> 172.21.15.116:0/3943995501 conn(0x7f13bc067cd0 msgr2=0x7f13bc0a1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 shutdown_connections 2024-09-18T12:26:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:30.415+0000 7f13c235d640 1 -- 172.21.15.116:0/3943995501 wait complete. 2024-09-18T12:26:30.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:30.918+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:30.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:30.918+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:31.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:31.918+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:31.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:31.918+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:32.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:32.918+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:32.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:32.918+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:33.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.529+0000 7f8a60c34640 10 monclient: get_monmap_and_config 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.529+0000 7f8a60c34640 10 monclient: build_initial_monmap 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.529+0000 7f8a60c34640 10 monclient: monmap: 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:33.531539+0000 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:33.531539+0000 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:33.532 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:33.533 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.530+0000 7f8a60c34640 1 Processor -- start 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.530+0000 7f8a60c34640 1 -- start start 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.530+0000 7f8a60c34640 10 monclient: init 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.530+0000 7f8a60c34640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a5c096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient(hunting): picked mon.noname-c con 0x7f8a5c0971e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 0x7f8a5c135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient(hunting): picked mon.noname-b con 0x7f8a5c133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c099620 con 0x7f8a5c096e10 2024-09-18T12:26:33.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5c099340 con 0x7f8a5c0971e0 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c099900 con 0x7f8a5c133b30 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a5ad9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 msgr2=0x7f8a5c135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a60c34640 10 monclient(hunting): authenticate will time out at 820.853157s 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a5ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 0x7f8a5c135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a5a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8a5c096e10 0x7f8a5c09a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59736/0 (socket says 172.21.15.116:59736) 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 learned_addr learned my addr 172.21.15.116:0/2582533340 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request con 0x7f8a5c0971e0 auth_method 0 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:33.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:33.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:33.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:33.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:33.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:33.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.531+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:33.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.533+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> v1:172.21.15.116:6789/0 conn(0x7f8a5c096e10 legacy=0x7f8a5c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:33.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a5ad9d640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 msgr2=0x7f8a5c135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:33.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a5ad9d640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 0x7f8a5c135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request con 0x7f8a5c0971e0 auth_method 0 2024-09-18T12:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:33.732+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:26:33.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:33.919+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:33.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:33.919+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:34.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a5ad9d640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 msgr2=0x7f8a5c135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:34.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a5ad9d640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 0x7f8a5c135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request con 0x7f8a5c0971e0 auth_method 0 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:34.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.133+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:26:34.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:34.919+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:34.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:34.919+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:34.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a5ad9d640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 msgr2=0x7f8a5c135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a5ad9d640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 0x7f8a5c135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:26:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request con 0x7f8a5c0971e0 auth_method 0 2024-09-18T12:26:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:34.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:34.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:34.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:34.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:34.934+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:26:35.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:35.919+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:35.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:35.919+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 10 monclient(hunting): tick 2024-09-18T12:26:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 monclient(hunting): continuing hunt 2024-09-18T12:26:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:26:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 msgr2=0x7f8a5c135f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a5c133b30 0x7f8a5c135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 msgr2=0x7f8a5c0975e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.144:3300/0 conn(0x7f8a5c0971e0 0x7f8a5c0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 -- 172.21.15.116:0/2582533340 >> v1:172.21.15.116:6789/0 conn(0x7f8a5c096e10 legacy=0x7f8a5c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 10 monclient: _add_conns ranks=[3,1] 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3c003720 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a3c006280 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 -- 172.21.15.116:0/2582533340 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c09ac70 con 0x7f8a3c006280 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 1 -- 172.21.15.116:0/2582533340 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5c099340 con 0x7f8a3c003720 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a58d99640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request con 0x7f8a3c003720 auth_method 0 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 msgr2=0x7f8a3c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 msgr2=0x7f8a3c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.532+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.535+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> v1:172.21.15.144:6789/0 conn(0x7f8a3c006280 legacy=0x7f8a3c008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request con 0x7f8a3c003720 auth_method 0 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 msgr2=0x7f8a3c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 msgr2=0x7f8a3c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:36.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:36.733+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:26:36.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:36.919+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:36.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:36.919+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request con 0x7f8a3c003720 auth_method 0 2024-09-18T12:26:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 msgr2=0x7f8a3c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 msgr2=0x7f8a3c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.134+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:26:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.829+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928132033 0 0) 0x7f8a5c09ac70 con 0x7f8a3c006280 2024-09-18T12:26:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.830+0000 7f8a59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.830+0000 7f8a59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.830+0000 7f8a59d9b640 10 monclient(hunting): my global_id is 4438 2024-09-18T12:26:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.830+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a480034e0 con 0x7f8a3c006280 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3119416406 0 0) 0x7f8a480034e0 con 0x7f8a3c006280 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 msgr2=0x7f8a3c005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 --2- 172.21.15.116:0/2582533340 >> v2:172.21.15.116:3300/0 conn(0x7f8a3c003720 0x7f8a3c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c00afc0 con 0x7f8a3c006280 2024-09-18T12:26:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: _finish_auth 0 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f8a4400d480 con 0x7f8a3c006280 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a60c34640 5 monclient: authenticate success, global_id 4438 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a60c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: dump: 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 >> v1:172.21.15.144:6789/0 conn(0x7f8a3c006280 legacy=0x7f8a3c008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a5ad9d640 1 -- 172.21.15.116:0/2582533340 reap_dead start 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a480039d0 0x7f8a48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): picked mon.b con 0x7f8a480039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48004780 0x7f8a48004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): picked mon.c con 0x7f8a48004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48005250 0x7f8a4800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): picked mon.a con 0x7f8a48005250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c099340 con 0x7f8a48005250 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4800c1c0 con 0x7f8a480039d0 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4800e590 con 0x7f8a48004780 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a59d9b640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:26:37.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a60c34640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48004780 msgr2=0x7f8a48004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.834+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48004780 0x7f8a48004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a480039d0 0x7f8a48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request con 0x7f8a480039d0 auth_method 0 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient(hunting): handle_auth_done global_id 4438 payload 293 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48004780 msgr2=0x7f8a48004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:37.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48004780 0x7f8a48004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48005250 msgr2=0x7f8a4800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48005250 0x7f8a4800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 1 monclient: found mon.b 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a480034e0 con 0x7f8a480039d0 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 10 monclient: _finish_auth 0 2024-09-18T12:26:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.835+0000 7f8a5a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a480039d0 msgr2=0x7f8a48003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a480039d0 0x7f8a48003dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0997b0 tx=0x7f8a4c009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=363323cb561a3638 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c004860 con 0x7f8a480039d0 2024-09-18T12:26:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 10 monclient: dump: 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a60c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c0049e0 con 0x7f8a480039d0 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a60c34640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a60c34640 4 set_mon_vals no callback set 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 1 -- 172.21.15.116:0/2582533340 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c004b90 con 0x7f8a480039d0 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.838+0000 7f8a59d9b640 10 monclient: dump: 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 10 monclient: shutdown 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 -- 172.21.15.116:0/2582533340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a480039d0 msgr2=0x7f8a48003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a480039d0 0x7f8a48003dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0997b0 tx=0x7f8a4c009040 comp rx=0 tx=0).stop 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 -- 172.21.15.116:0/2582533340 shutdown_connections 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a48005250 0x7f8a4800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48004780 0x7f8a48004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2582533340 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a480039d0 0x7f8a48003dc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 -- 172.21.15.116:0/2582533340 >> 172.21.15.116:0/2582533340 conn(0x7f8a5c0672b0 msgr2=0x7f8a5c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 -- 172.21.15.116:0/2582533340 shutdown_connections 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.839+0000 7f8a60c34640 1 -- 172.21.15.116:0/2582533340 wait complete. 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.840+0000 7f8a60c34640 10 monclient: build_initial_monmap 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.840+0000 7f8a60c34640 10 monclient: monmap: 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 Processor -- start 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 -- start start 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient: init 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480039d0 0x7f8a5c132670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): picked mon.noname-a con 0x7f8a480039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a48004780 0x7f8a5c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): picked mon.noname-b con 0x7f8a48004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48005250 0x7f8a5c06e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): picked mon.noname-c con 0x7f8a48005250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c131d40 con 0x7f8a480039d0 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c06f040 con 0x7f8a48004780 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c131680 con 0x7f8a48005250 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a60c34640 10 monclient(hunting): authenticate will time out at 825.162681s 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5ad9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48005250 msgr2=0x7f8a5c06e0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48005250 0x7f8a5c06e0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480039d0 0x7f8a5c132670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480039d0 0x7f8a5c132670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55568/0 (socket says 172.21.15.116:55568) 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5a59c640 1 -- 172.21.15.116:0/2492966609 learned_addr learned my addr 172.21.15.116:0/2492966609 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request con 0x7f8a480039d0 auth_method 0 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a48004780 0x7f8a5c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request con 0x7f8a48004780 auth_method 0 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.841+0000 7f8a51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:37.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 10 monclient(hunting): handle_auth_done global_id 4441 payload 824 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2492966609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48005250 msgr2=0x7f8a5c06e0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48005250 0x7f8a5c06e0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2492966609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480039d0 msgr2=0x7f8a5c132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480039d0 0x7f8a5c132670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 1 monclient: found mon.noname-b 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.842+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2492966609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c098830 con 0x7f8a48004780 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.844+0000 7f8a51d9b640 1 -- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a48004780 msgr2=0x7f8a5c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.844+0000 7f8a51d9b640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a48004780 0x7f8a5c06db60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8a44006ea0 tx=0x7f8a44008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 1 -- 172.21.15.116:0/2492966609 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44011ca0 con 0x7f8a48004780 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: dump: 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 1 -- 172.21.15.116:0/2492966609 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a44011e20 con 0x7f8a48004780 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a60c34640 5 monclient: authenticate success, global_id 4441 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a60c34640 10 monclient: _renew_subs 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a60c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c1300b0 con 0x7f8a48004780 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 1 -- 172.21.15.116:0/2492966609 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440166b0 con 0x7f8a48004780 2024-09-18T12:26:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a52ffd640 10 monclient: dump: 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a60c34640 10 monclient: _renew_subs 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a5959a640 4 set_mon_vals no callback set 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a60c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.845+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c130790 con 0x7f8a48004780 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a52ffd640 1 -- 172.21.15.116:0/2492966609 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8a44016a60 con 0x7f8a48004780 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a60c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a52ffd640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8a3c038cd0 0x7f8a3c03b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a60c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a60c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c04bf60 con 0x7f8a48004780 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a52ffd640 1 -- 172.21.15.116:0/2492966609 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f8a4404bf10 con 0x7f8a48004780 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.846+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8a3c038cd0 0x7f8a3c03b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.847+0000 7f8a5a59c640 10 monclient: get_auth_request con 0x7f8a3c038cd0 auth_method 0 2024-09-18T12:26:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.849+0000 7f8a5a59c640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8a3c038cd0 0x7f8a3c03b180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c02b100 tx=0x7f8a4c038000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.851+0000 7f8a52ffd640 1 -- 172.21.15.116:0/2492966609 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8a4401e4f0 con 0x7f8a48004780 2024-09-18T12:26:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.851+0000 7f8a52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:37.851+0000 7f8a52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:37.919+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:37.919+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:38.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.069+0000 7f8a60c34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:38.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.069+0000 7f8a60c34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:38.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.069+0000 7f8a60c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:38.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.069+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8a5c0a0290 con 0x7f8a48004780 2024-09-18T12:26:38.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.070+0000 7f8a52ffd640 1 -- 172.21.15.116:0/2492966609 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f8a4402e080 con 0x7f8a48004780 2024-09-18T12:26:38.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.070+0000 7f8a52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:38.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.070+0000 7f8a52ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:38.071 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:26:38.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8a3c038cd0 msgr2=0x7f8a3c03b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8a3c038cd0 0x7f8a3c03b180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c02b100 tx=0x7f8a4c038000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 10 monclient: shutdown 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a48004780 msgr2=0x7f8a5c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a48004780 0x7f8a5c06db60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8a44006ea0 tx=0x7f8a44008040 comp rx=0 tx=0).stop 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 shutdown_connections 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8a3c038cd0 0x7f8a3c03b180 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8a48005250 0x7f8a5c06e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a48004780 0x7f8a5c06db60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 --2- 172.21.15.116:0/2492966609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480039d0 0x7f8a5c132670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 >> 172.21.15.116:0/2492966609 conn(0x7f8a5c0672b0 msgr2=0x7f8a5c12a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 shutdown_connections 2024-09-18T12:26:38.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.072+0000 7f8a60c34640 1 -- 172.21.15.116:0/2492966609 wait complete. 2024-09-18T12:26:38.084 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:26:38.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:26:38.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:26:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.176+0000 7f9d7c95d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.176+0000 7f9d7c95d640 10 monclient: build_initial_monmap 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.176+0000 7f9d7c95d640 10 monclient: monmap: 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:38.177312+0000 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:38.177312+0000 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:38.178 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 1 Processor -- start 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 1 -- start start 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient: init 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d78067110 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7809b010 0x7f9d7809b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d7809b010 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9d781349a0 0x7f9d78136d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d781349a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d78130540 con 0x7f9d78067110 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7809b940 con 0x7f9d781349a0 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78067ed0 con 0x7f9d7809b010 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d7c95d640 10 monclient(hunting): authenticate will time out at 825.498980s 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.177+0000 7f9d6ffff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7809b010 msgr2=0x7f9d7809b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7809b010 0x7f9d7809b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9d781349a0 0x7f9d78136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9d781349a0 0x7f9d78136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55576/0 (socket says 172.21.15.116:55576) 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 -- 172.21.15.116:0/2376920038 learned_addr learned my addr 172.21.15.116:0/2376920038 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): get_auth_request con 0x7f9d781349a0 auth_method 0 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient(hunting): handle_auth_done global_id 4518 payload 274 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 -- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7809b010 msgr2=0x7f9d7809b400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7809b010 0x7f9d7809b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 -- 172.21.15.116:0/2376920038 >> v1:172.21.15.144:6789/0 conn(0x7f9d78067110 legacy=0x7f9d7812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 monclient: found mon.noname-a 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.178+0000 7f9d76d9d640 1 -- 172.21.15.116:0/2376920038 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78099800 con 0x7f9d781349a0 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d76d9d640 1 -- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d781349a0 msgr2=0x7f9d78136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d781349a0 0x7f9d78136d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009f20 tx=0x7f9d70009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a427e55cdde957cf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 1 -- 172.21.15.116:0/2376920038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70007610 con 0x7f9d781349a0 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: dump: 2024-09-18T12:26:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: _finish_auth 0 2024-09-18T12:26:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 1 -- 172.21.15.116:0/2376920038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7002d050 con 0x7f9d781349a0 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 5 monclient: authenticate success, global_id 4518 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 1 -- 172.21.15.116:0/2376920038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d700079a0 con 0x7f9d781349a0 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 4 set_mon_vals no callback set 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d75d9b640 10 monclient: dump: 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 10 monclient: shutdown 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 -- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d781349a0 msgr2=0x7f9d78136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d781349a0 0x7f9d78136d80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009f20 tx=0x7f9d70009840 comp rx=0 tx=0).stop 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 -- 172.21.15.116:0/2376920038 shutdown_connections 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d7809b010 0x7f9d7809b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/2376920038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d781349a0 0x7f9d78136d80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 -- 172.21.15.116:0/2376920038 >> 172.21.15.116:0/2376920038 conn(0x7f9d780a1e30 msgr2=0x7f9d780a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 -- 172.21.15.116:0/2376920038 shutdown_connections 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.181+0000 7f9d7c95d640 1 -- 172.21.15.116:0/2376920038 wait complete. 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.182+0000 7f9d7c95d640 10 monclient: build_initial_monmap 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.182+0000 7f9d7c95d640 10 monclient: monmap: 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:38.190 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:38.191 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.182+0000 7f9d7c95d640 1 Processor -- start 2024-09-18T12:26:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.182+0000 7f9d7c95d640 1 -- start start 2024-09-18T12:26:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.182+0000 7f9d7c95d640 10 monclient: init 2024-09-18T12:26:38.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 0x7f9d7806e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d78067110 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d7809b010 0x7f9d78096a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d7809b010 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d781349a0 0x7f9d78096fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d781349a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7809e3a0 con 0x7f9d78067110 2024-09-18T12:26:38.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780a05f0 con 0x7f9d7809b010 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7806cae0 con 0x7f9d781349a0 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7c95d640 10 monclient(hunting): authenticate will time out at 825.504113s 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d76d9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d781349a0 msgr2=0x7f9d78096fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d76d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d781349a0 0x7f9d78096fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 0x7f9d7806e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 0x7f9d7806e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55586/0 (socket says 172.21.15.116:55586) 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 -- 172.21.15.116:0/277785199 learned_addr learned my addr 172.21.15.116:0/277785199 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): get_auth_request con 0x7f9d78067110 auth_method 0 2024-09-18T12:26:38.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d7809b010 0x7f9d78096a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d6ffff640 10 monclient(hunting): get_auth_request con 0x7f9d7809b010 auth_method 0 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient(hunting): handle_auth_done global_id 4521 payload 824 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d781349a0 msgr2=0x7f9d78096fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d781349a0 0x7f9d78096fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d7809b010 msgr2=0x7f9d78096a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d7809b010 0x7f9d78096a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient: _finish_hunting 0 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 monclient: found mon.noname-a 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:38.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.183+0000 7f9d7659c640 1 -- 172.21.15.116:0/277785199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d70009ae0 con 0x7f9d78067110 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7659c640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 msgr2=0x7f9d7806e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7659c640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 0x7f9d7806e610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800c5e0 tx=0x7f9d68007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68012840 con 0x7f9d78067110 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: dump: 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d680129a0 con 0x7f9d78067110 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7c95d640 5 monclient: authenticate success, global_id 4521 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:38.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7c95d640 10 monclient: _renew_subs 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d68012da0 con 0x7f9d78067110 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7c95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7806dc30 con 0x7f9d78067110 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d6e7fc640 10 monclient: dump: 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.204 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7559a640 4 set_mon_vals no callback set 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7c95d640 10 monclient: _renew_subs 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7c95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.185+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7806eb50 con 0x7f9d78067110 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9d68011590 con 0x7f9d78067110 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d7c95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 0x7f9d78132700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d7c95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78065f90 con 0x7f9d78097510 2024-09-18T12:26:38.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6e7fc640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d58038440 0x7f9d5803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 0x7f9d78132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f9d6804b7b0 con 0x7f9d78067110 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 10 monclient: get_auth_request con 0x7f9d78097510 auth_method 0 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.186+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.187+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d58038440 0x7f9d5803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.187+0000 7f9d76d9d640 10 monclient: get_auth_request con 0x7f9d58038440 auth_method 0 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.187+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_done global_id 4524 payload 824 2024-09-18T12:26:38.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.187+0000 7f9d6ffff640 1 -- 172.21.15.116:0/277785199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7806edd0 con 0x7f9d78097510 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6ffff640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 msgr2=0x7f9d78132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 0x7f9d78132700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9d60003940 tx=0x7f9d60004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60012820 con 0x7f9d78097510 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 10 monclient: dump: 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.207 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d58038440 0x7f9d5803a8f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d780681f0 tx=0x7f9d70036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9d60012980 con 0x7f9d78097510 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 msgr2=0x7f9d78132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 0x7f9d78132700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9d60003940 tx=0x7f9d60004ba0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 msgr2=0x7f9d78132700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.189+0000 7f9d6e7fc640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78097510 0x7f9d78132700 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d7c95d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:38.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 0x7f9d78133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d7c95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7806e0a0 con 0x7f9d78132c40 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 0x7f9d78133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient: get_auth_request con 0x7f9d78132c40 auth_method 0 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 10 monclient(hunting): handle_auth_done global_id 4527 payload 824 2024-09-18T12:26:38.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.195+0000 7f9d6ffff640 1 -- 172.21.15.116:0/277785199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d780978e0 con 0x7f9d78132c40 2024-09-18T12:26:38.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6ffff640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 msgr2=0x7f9d78133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 0x7f9d78133040 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9d60003940 tx=0x7f9d60018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60012e90 con 0x7f9d78132c40 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 10 monclient: dump: 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f9d60023070 con 0x7f9d78132c40 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 msgr2=0x7f9d78133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 0x7f9d78133040 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9d60003940 tx=0x7f9d60018000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 msgr2=0x7f9d78133040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d6e7fc640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78132c40 0x7f9d78133040 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9d60003940 tx=0x7f9d60018000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 18, 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.214 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:38.215 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:38.222 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.223 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.224 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:38.226 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d58038440 msgr2=0x7f9d5803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d58038440 0x7f9d5803a8f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d780681f0 tx=0x7f9d70036000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d7c95d640 10 monclient: shutdown 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d7c95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 msgr2=0x7f9d7806e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 0x7f9d7806e610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800c5e0 tx=0x7f9d68007ed0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.198+0000 7f9d76d9d640 1 -- 172.21.15.116:0/277785199 reap_dead start 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.199+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 shutdown_connections 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.199+0000 7f9d7c95d640 1 --2- 172.21.15.116:0/277785199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78067110 0x7f9d7806e610 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.199+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 >> 172.21.15.116:0/277785199 conn(0x7f9d780a1e30 msgr2=0x7f9d7809c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.199+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 shutdown_connections 2024-09-18T12:26:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.199+0000 7f9d7c95d640 1 -- 172.21.15.116:0/277785199 wait complete. 2024-09-18T12:26:38.229 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:26:38.229 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:26:38.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:26:38.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.321+0000 7f70ad24f640 10 monclient: get_monmap_and_config 2024-09-18T12:26:38.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.321+0000 7f70ad24f640 10 monclient: build_initial_monmap 2024-09-18T12:26:38.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.321+0000 7f70ad24f640 10 monclient: monmap: 2024-09-18T12:26:38.322 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:38.322 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:38.322187+0000 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:38.322187+0000 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:38.323 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 1 Processor -- start 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 1 -- start start 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient: init 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70a80981c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70a8131da0 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f70a8132170 0x7f70a813a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient(hunting): picked mon.noname-c con 0x7f70a8132170 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a809ca60 con 0x7f70a80981c0 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a809cca0 con 0x7f70a8131da0 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a8131bd0 con 0x7f70a8132170 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70ad24f640 10 monclient(hunting): authenticate will time out at 825.643697s 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70a6d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f70a80981c0 0x7f70a8098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47344/0 (socket says 172.21.15.116:47344) 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70a6d9d640 1 -- 172.21.15.116:0/631665915 learned_addr learned my addr 172.21.15.116:0/631665915 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70a759e640 1 --2- 172.21.15.116:0/631665915 >> v2:172.21.15.144:3300/0 conn(0x7f70a8132170 0x7f70a813a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70a759e640 10 monclient(hunting): get_auth_request con 0x7f70a8132170 auth_method 0 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70a759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.322+0000 7f70a759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 10 monclient(hunting): handle_auth_done global_id 4447 payload 274 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 1 -- 172.21.15.116:0/631665915 >> v1:172.21.15.144:6789/0 conn(0x7f70a8131da0 legacy=0x7f70a8136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 1 -- 172.21.15.116:0/631665915 >> v1:172.21.15.116:6789/0 conn(0x7f70a80981c0 legacy=0x7f70a8098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 10 monclient: _finish_hunting 0 2024-09-18T12:26:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 1 monclient: found mon.noname-c 2024-09-18T12:26:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.323+0000 7f70a759e640 1 -- 172.21.15.116:0/631665915 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a809e010 con 0x7f70a8132170 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a759e640 1 -- 172.21.15.116:0/631665915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8132170 msgr2=0x7f70a813a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a759e640 1 --2- 172.21.15.116:0/631665915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8132170 0x7f70a813a2a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009e60 tx=0x7f70a0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c2e02d6b0c23c4d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 1 -- 172.21.15.116:0/631665915 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a0007840 con 0x7f70a8132170 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: dump: 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.334 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: _finish_auth 0 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 5 monclient: authenticate success, global_id 4447 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 1 -- 172.21.15.116:0/631665915 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a002d050 con 0x7f70a8132170 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:38.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 1 -- 172.21.15.116:0/631665915 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a0007bd0 con 0x7f70a8132170 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 4 set_mon_vals no callback set 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70a5d9b640 10 monclient: dump: 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.336 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 10 monclient: shutdown 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 1 -- 172.21.15.116:0/631665915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8132170 msgr2=0x7f70a813a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.326+0000 7f70ad24f640 1 --2- 172.21.15.116:0/631665915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8132170 0x7f70a813a2a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009e60 tx=0x7f70a0009780 comp rx=0 tx=0).stop 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 1 -- 172.21.15.116:0/631665915 shutdown_connections 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 1 --2- 172.21.15.116:0/631665915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8132170 0x7f70a813a2a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 1 -- 172.21.15.116:0/631665915 >> 172.21.15.116:0/631665915 conn(0x7f70a80a1e30 msgr2=0x7f70a80a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 1 -- 172.21.15.116:0/631665915 shutdown_connections 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 1 -- 172.21.15.116:0/631665915 wait complete. 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 10 monclient: build_initial_monmap 2024-09-18T12:26:38.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 10 monclient: monmap: 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.327+0000 7f70ad24f640 1 Processor -- start 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 1 -- start start 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient: init 2024-09-18T12:26:38.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 0x7f70a8140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): picked mon.noname-a con 0x7f70a80981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131da0 0x7f70a8131120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a8132170 0x7f70a8137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): picked mon.noname-c con 0x7f70a8132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a81191f0 con 0x7f70a80981c0 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a809ba60 con 0x7f70a8131da0 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8098cb0 con 0x7f70a8132170 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70ad24f640 10 monclient(hunting): authenticate will time out at 825.649543s 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a759e640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a8132170 msgr2=0x7f70a8137e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a759e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a8132170 0x7f70a8137e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 0x7f70a8140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 0x7f70a8140ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55618/0 (socket says 172.21.15.116:55618) 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 -- 172.21.15.116:0/1528975776 learned_addr learned my addr 172.21.15.116:0/1528975776 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request con 0x7f70a80981c0 auth_method 0 2024-09-18T12:26:38.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a659c640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131da0 0x7f70a8131120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a659c640 10 monclient(hunting): get_auth_request con 0x7f70a8131da0 auth_method 0 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_done global_id 4530 payload 824 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a8132170 msgr2=0x7f70a8137e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a8132170 0x7f70a8137e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131da0 msgr2=0x7f70a8131120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131da0 0x7f70a8131120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 monclient: found mon.noname-a 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.328+0000 7f70a6d9d640 1 -- 172.21.15.116:0/1528975776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a0009a20 con 0x7f70a80981c0 2024-09-18T12:26:38.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70a6d9d640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 msgr2=0x7f70a8140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70a6d9d640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 0x7f70a8140ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f709800c640 tx=0x7f7098004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098005cb0 con 0x7f70a80981c0 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: dump: 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.343 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: _finish_auth 0 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098005e30 con 0x7f70a80981c0 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70ad24f640 5 monclient: authenticate success, global_id 4530 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098007340 con 0x7f70a80981c0 2024-09-18T12:26:38.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f708effd640 10 monclient: dump: 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.345 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70ad24f640 10 monclient: _renew_subs 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70ad24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a813a120 con 0x7f70a80981c0 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70a559a640 4 set_mon_vals no callback set 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70ad24f640 10 monclient: _renew_subs 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70ad24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.331+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a8141d00 con 0x7f70a80981c0 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70ad24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f709801b400 con 0x7f70a80981c0 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70ad24f640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 0x7f70a8138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70ad24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8065f90 con 0x7f70a8131660 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f708effd640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7094038440 0x7f709403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f709804b2d0 con 0x7f70a80981c0 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 0x7f70a8138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient: get_auth_request con 0x7f70a8131660 auth_method 0 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a759e640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7094038440 0x7f709403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a759e640 10 monclient: get_auth_request con 0x7f7094038440 auth_method 0 2024-09-18T12:26:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 10 monclient(hunting): handle_auth_done global_id 4453 payload 824 2024-09-18T12:26:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.332+0000 7f70a659c640 1 -- 172.21.15.116:0/1528975776 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70a804bf60 con 0x7f70a8131660 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.334+0000 7f70a659c640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 msgr2=0x7f70a8138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.334+0000 7f70a659c640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 0x7f70a8138350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a02c0 tx=0x7f709c009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f70a759e640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7094038440 0x7f709403a8f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70a8066560 tx=0x7f70a0037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c007450 con 0x7f70a8131660 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 10 monclient: dump: 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.352 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f709c0075b0 con 0x7f70a8131660 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 msgr2=0x7f70a8138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 0x7f70a8138350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a02c0 tx=0x7f709c009040 comp rx=0 tx=0).stop 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 msgr2=0x7f70a8138350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.335+0000 7f708effd640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8131660 0x7f70a8138350 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a02c0 tx=0x7f709c009040 comp rx=0 tx=0).stop 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.340+0000 7f70ad24f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70ad24f640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 0x7f70a8134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70ad24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8138c90 con 0x7f70a8138890 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 0x7f70a8134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient: get_auth_request con 0x7f70a8138890 auth_method 0 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 10 monclient(hunting): handle_auth_done global_id 4456 payload 824 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.341+0000 7f70a659c640 1 -- 172.21.15.116:0/1528975776 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f70a8098ee0 con 0x7f70a8138890 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.343+0000 7f70a659c640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 msgr2=0x7f70a8134650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70a659c640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 0x7f70a8134650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a02c0 tx=0x7f709c01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c013c80 con 0x7f70a8138890 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 10 monclient: dump: 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f709c013de0 con 0x7f70a8138890 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:38.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 msgr2=0x7f70a8134650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 0x7f70a8134650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a02c0 tx=0x7f709c01b000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 msgr2=0x7f70a8134650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f708effd640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a8138890 0x7f70a8134650 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f70a80a02c0 tx=0x7f709c01b000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 18, 2024-09-18T12:26:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:26:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:26:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:26:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:26:38.369 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7094038440 msgr2=0x7f709403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7094038440 0x7f709403a8f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70a8066560 tx=0x7f70a0037000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 10 monclient: shutdown 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 msgr2=0x7f70a8140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 0x7f70a8140ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f709800c640 tx=0x7f7098004960 comp rx=0 tx=0).stop 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70a759e640 1 -- 172.21.15.116:0/1528975776 reap_dead start 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 shutdown_connections 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 --2- 172.21.15.116:0/1528975776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a80981c0 0x7f70a8140ef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 >> 172.21.15.116:0/1528975776 conn(0x7f70a80a1e30 msgr2=0x7f70a8096f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 shutdown_connections 2024-09-18T12:26:38.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:38.344+0000 7f70ad24f640 1 -- 172.21.15.116:0/1528975776 wait complete. 2024-09-18T12:26:38.372 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:26:38.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:26:38.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.840+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v19) v1 ==== 152+0+0 (secure 0 0 0) 0x7fed0c042070 con 0x7fed1c096a30 2024-09-18T12:26:38.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.840+0000 7fed137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T12:26:38.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.840+0000 7fed137fe640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-18T12:26:38.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.919+0000 7fed11ffb640 10 monclient: tick 2024-09-18T12:26:38.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.919+0000 7fed11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:38.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.936+0000 7fed2164f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T12:26:38.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.936+0000 7fed2164f640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T12:26:38.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.936+0000 7fed2164f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:38.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.936+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7fed1c097740 con 0x7fed1c096a30 2024-09-18T12:26:38.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.937+0000 7fed137fe640 1 -- 172.21.15.144:0/1356173230 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v19) v1 ==== 159+0+0 (secure 0 0 0) 0x7fed0c042030 con 0x7fed1c096a30 2024-09-18T12:26:38.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.937+0000 7fed137fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T12:26:38.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.937+0000 7fed137fe640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-18T12:26:38.939 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'testpool1234' already exists 2024-09-18T12:26:38.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fecfc038470 msgr2=0x7fecfc03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fecfc038470 0x7fecfc03a920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed1c131fb0 tx=0x7fed040090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 10 monclient: shutdown 2024-09-18T12:26:38.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:38.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 msgr2=0x7fed1c13ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c13ef50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed0c035040 tx=0x7fed0c032000 comp rx=0 tx=0).stop 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 shutdown_connections 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fecfc038470 0x7fecfc03a920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.939+0000 7fed2164f640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed1c12c6a0 0x7fed1c130750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.940+0000 7fed2164f640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed1c096e00 0x7fed1c12b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.940+0000 7fed2164f640 1 --2- 172.21.15.144:0/1356173230 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed1c096a30 0x7fed1c13ef50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.940+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 >> 172.21.15.144:0/1356173230 conn(0x7fed1c0711a0 msgr2=0x7fed1c132c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:38.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.940+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 shutdown_connections 2024-09-18T12:26:38.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:38.940+0000 7fed2164f640 1 -- 172.21.15.144:0/1356173230 wait complete. 2024-09-18T12:26:38.950 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.024+0000 7fb50471d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.024+0000 7fb50471d640 10 monclient: build_initial_monmap 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.024+0000 7fb50471d640 10 monclient: monmap: 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:39.025759+0000 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:39.025759+0000 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:39.026 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:39.027 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:39.027 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:39.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.025+0000 7fb50471d640 1 Processor -- start 2024-09-18T12:26:39.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.025+0000 7fb50471d640 1 -- start start 2024-09-18T12:26:39.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.025+0000 7fb50471d640 10 monclient: init 2024-09-18T12:26:39.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:39.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:26:39.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4fc0980c0 0x7fb4fc132080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:39.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4fc0980c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb4fc098490 0x7fb4fc098890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4fc098490 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4fc068dc0 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc0a1860 con 0x7fb4fc068dc0 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4fc0a1580 con 0x7fb4fc0980c0 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4fc0a12a0 con 0x7fb4fc098490 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb50471d640 10 monclient(hunting): authenticate will time out at 834.595169s 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb4fc098490 0x7fb4fc098890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb4fc098490 0x7fb4fc098890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39586/0 (socket says 172.21.15.144:39586) 2024-09-18T12:26:39.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 -- 172.21.15.144:0/3516384698 learned_addr learned my addr 172.21.15.144:0/3516384698 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): get_auth_request con 0x7fb4fc098490 auth_method 0 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb5024b9640 1 --2- 172.21.15.144:0/3516384698 >> v2:172.21.15.116:3300/0 conn(0x7fb4fc0980c0 0x7fb4fc132080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb5024b9640 10 monclient(hunting): get_auth_request con 0x7fb4fc0980c0 auth_method 0 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb5024b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb5024b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:39.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb5024b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient(hunting): handle_auth_done global_id 4459 payload 274 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 -- 172.21.15.144:0/3516384698 >> v2:172.21.15.116:3300/0 conn(0x7fb4fc0980c0 msgr2=0x7fb4fc132080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 --2- 172.21.15.144:0/3516384698 >> v2:172.21.15.116:3300/0 conn(0x7fb4fc0980c0 0x7fb4fc132080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 -- 172.21.15.144:0/3516384698 >> v1:172.21.15.116:6789/0 conn(0x7fb4fc068dc0 legacy=0x7fb4fc069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient: _finish_hunting 0 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 monclient: found mon.noname-c 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:39.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.026+0000 7fb501cb8640 1 -- 172.21.15.144:0/3516384698 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc09fe90 con 0x7fb4fc098490 2024-09-18T12:26:39.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb501cb8640 1 -- 172.21.15.144:0/3516384698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc098490 msgr2=0x7fb4fc098890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:39.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb501cb8640 1 --2- 172.21.15.144:0/3516384698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc098490 0x7fb4fc098890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc06f0f0 tx=0x7fb4ec0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=964c6e9f8eb5fe51 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:39.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 1 -- 172.21.15.144:0/3516384698 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec032440 con 0x7fb4fc098490 2024-09-18T12:26:39.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:39.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:39.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: dump: 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:39.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: _finish_auth 0 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 5 monclient: authenticate success, global_id 4459 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 1 -- 172.21.15.144:0/3516384698 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec0325a0 con 0x7fb4fc098490 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 4 set_mon_vals no callback set 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 1 -- 172.21.15.144:0/3516384698 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec032980 con 0x7fb4fc098490 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:39.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb5014b7640 10 monclient: dump: 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 10 monclient: shutdown 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:39.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 1 -- 172.21.15.144:0/3516384698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc098490 msgr2=0x7fb4fc098890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.029+0000 7fb50471d640 1 --2- 172.21.15.144:0/3516384698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc098490 0x7fb4fc098890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc06f0f0 tx=0x7fb4ec0091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 -- 172.21.15.144:0/3516384698 shutdown_connections 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 --2- 172.21.15.144:0/3516384698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc098490 0x7fb4fc098890 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 --2- 172.21.15.144:0/3516384698 >> v2:172.21.15.116:3300/0 conn(0x7fb4fc0980c0 0x7fb4fc132080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 -- 172.21.15.144:0/3516384698 >> 172.21.15.144:0/3516384698 conn(0x7fb4fc070f00 msgr2=0x7fb4fc0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 -- 172.21.15.144:0/3516384698 shutdown_connections 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 -- 172.21.15.144:0/3516384698 wait complete. 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 10 monclient: build_initial_monmap 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 10 monclient: monmap: 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:39.043 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 Processor -- start 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 1 -- start start 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.030+0000 7fb50471d640 10 monclient: init 2024-09-18T12:26:39.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 0x7fb4fc09b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4fc068dc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4fc0980c0 0x7fb4fc130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4fc0980c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4fc098490 0x7fb4fc141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4fc098490 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:39.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc11dd00 con 0x7fb4fc098490 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc098de0 con 0x7fb4fc068dc0 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc09e8c0 con 0x7fb4fc0980c0 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb50471d640 10 monclient(hunting): authenticate will time out at 834.600325s 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 0x7fb4fc09b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 0x7fb4fc09b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39598/0 (socket says 172.21.15.144:39598) 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb501cb8640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4fc0980c0 msgr2=0x7fb4fc130be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 -- 172.21.15.144:0/3016650507 learned_addr learned my addr 172.21.15.144:0/3016650507 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:39.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): get_auth_request con 0x7fb4fc068dc0 auth_method 0 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb501cb8640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4fc0980c0 0x7fb4fc130be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb502cba640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4fc098490 0x7fb4fc141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb502cba640 10 monclient(hunting): get_auth_request con 0x7fb4fc098490 auth_method 0 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb502cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb502cba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:39.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb502cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient(hunting): handle_auth_done global_id 4462 payload 824 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 -- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4fc0980c0 msgr2=0x7fb4fc130be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4fc0980c0 0x7fb4fc130be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 -- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4fc098490 msgr2=0x7fb4fc141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4fc098490 0x7fb4fc141050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient: _finish_hunting 0 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 monclient: found mon.noname-b 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb5024b9640 1 -- 172.21.15.144:0/3016650507 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec009450 con 0x7fb4fc068dc0 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.031+0000 7fb502cba640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4fc098490 0x7fb4fc141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:26:39.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.033+0000 7fb5024b9640 1 -- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 msgr2=0x7fb4fc09b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.033+0000 7fb5024b9640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 0x7fb4fc09b0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800d890 tx=0x7fb4f8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.033+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800de20 con 0x7fb4fc068dc0 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.033+0000 7fb4f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.033+0000 7fb4f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.033+0000 7fb4f27fc640 10 monclient: dump: 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:39.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f800bae0 con 0x7fb4fc068dc0 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 5 monclient: authenticate success, global_id 4462 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 10 monclient: _renew_subs 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc11cda0 con 0x7fb4fc068dc0 2024-09-18T12:26:39.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800be80 con 0x7fb4fc068dc0 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 10 monclient: dump: 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:39.051 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 10 monclient: _renew_subs 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb500cb6640 4 set_mon_vals no callback set 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc11bcc0 con 0x7fb4fc068dc0 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb4f8014640 con 0x7fb4fc068dc0 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:39.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.034+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc04bf60 con 0x7fb4fc068dc0 2024-09-18T12:26:39.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.035+0000 7fb4f27fc640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4e0038440 0x7fb4e003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:39.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.035+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4693+0+0 (secure 0 0 0) 0x7fb4f804be80 con 0x7fb4fc068dc0 2024-09-18T12:26:39.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.035+0000 7fb501cb8640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4e0038440 0x7fb4e003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:39.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.035+0000 7fb501cb8640 10 monclient: get_auth_request con 0x7fb4e0038440 auth_method 0 2024-09-18T12:26:39.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.037+0000 7fb501cb8640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4e0038440 0x7fb4e003a8f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec034000 tx=0x7fb4ec032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:39.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.039+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb4f801e460 con 0x7fb4fc068dc0 2024-09-18T12:26:39.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.039+0000 7fb4f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:39.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.039+0000 7fb4f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:39.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.194+0000 7fb50471d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T12:26:39.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.194+0000 7fb50471d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T12:26:39.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.194+0000 7fb50471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:39.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.194+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7fb4fc0649b0 con 0x7fb4fc068dc0 2024-09-18T12:26:39.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.841+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fb4f801dbb0 con 0x7fb4fc068dc0 2024-09-18T12:26:39.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.841+0000 7fb4f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T12:26:39.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.841+0000 7fb4f27fc640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-18T12:26:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.940+0000 7fb50471d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T12:26:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.941+0000 7fb50471d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T12:26:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.941+0000 7fb50471d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:39.941+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7fb4fc11c7f0 con 0x7fb4fc068dc0 2024-09-18T12:26:40.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.847+0000 7fb4f27fc640 1 -- 172.21.15.144:0/3016650507 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v21) v1 ==== 143+0+0 (secure 0 0 0) 0x7fb4f802d0b0 con 0x7fb4fc068dc0 2024-09-18T12:26:40.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.847+0000 7fb4f27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T12:26:40.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.847+0000 7fb4f27fc640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-18T12:26:40.849 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 3 pg_num_min to 2 2024-09-18T12:26:40.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4e0038440 msgr2=0x7fb4e003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:40.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4e0038440 0x7fb4e003a8f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec034000 tx=0x7fb4ec032000 comp rx=0 tx=0).stop 2024-09-18T12:26:40.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 10 monclient: shutdown 2024-09-18T12:26:40.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 msgr2=0x7fb4fc09b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 0x7fb4fc09b0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800d890 tx=0x7fb4f8008040 comp rx=0 tx=0).stop 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 shutdown_connections 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4e0038440 0x7fb4e003a8f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4fc098490 0x7fb4fc141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4fc0980c0 0x7fb4fc130be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 --2- 172.21.15.144:0/3016650507 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4fc068dc0 0x7fb4fc09b0d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 >> 172.21.15.144:0/3016650507 conn(0x7fb4fc070f00 msgr2=0x7fb4fc09ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 shutdown_connections 2024-09-18T12:26:40.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.850+0000 7fb50471d640 1 -- 172.21.15.144:0/3016650507 wait complete. 2024-09-18T12:26:40.862 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-18T12:26:40.862 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 2 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.938+0000 7f4316f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.938+0000 7f4316f9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.939+0000 7f4316f9d640 10 monclient: monmap: 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:40.939910+0000 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:40.939910+0000 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:40.941 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:40.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.939+0000 7f4316f9d640 1 Processor -- start 2024-09-18T12:26:40.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.939+0000 7f4316f9d640 1 -- start start 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.939+0000 7f4316f9d640 10 monclient: init 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4310098c70 0x7f4310099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4310098c70 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4310099580 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431009e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4310099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431009f680 con 0x7f4310099580 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:40.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431009f420 con 0x7f4310098c70 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4310068a20 con 0x7f4310099c70 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4316f9d640 10 monclient(hunting): authenticate will time out at 836.509385s 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f430ffff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f4310099580 0x7f431009dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59716/0 (socket says 172.21.15.144:59716) 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f430ffff640 1 -- 172.21.15.144:0/548925719 learned_addr learned my addr 172.21.15.144:0/548925719 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f431553a640 1 -- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 msgr2=0x7f431009e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f431553a640 1 --2- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431009e1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 1 --2- 172.21.15.144:0/548925719 >> v2:172.21.15.116:3300/0 conn(0x7f4310098c70 0x7f4310099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 10 monclient(hunting): get_auth_request con 0x7f4310098c70 auth_method 0 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:40.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.940+0000 7f4314d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 10 monclient(hunting): handle_auth_done global_id 4539 payload 274 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 1 -- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 msgr2=0x7f431009e1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 1 --2- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431009e1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 1 -- 172.21.15.144:0/548925719 >> v1:172.21.15.144:6789/0 conn(0x7f4310099580 legacy=0x7f431009dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 10 monclient: _finish_hunting 0 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 1 monclient: found mon.noname-a 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:40.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.941+0000 7f4314d39640 1 -- 172.21.15.144:0/548925719 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4310096a30 con 0x7f4310098c70 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.943+0000 7f4314d39640 1 -- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 msgr2=0x7f4310099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4314d39640 1 --2- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310099040 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4300009880 tx=0x7f43000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09777e06e22f8aa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 1 -- 172.21.15.144:0/548925719 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300032440 con 0x7f4310098c70 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: dump: 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.952 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 5 monclient: authenticate success, global_id 4539 2024-09-18T12:26:40.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 1 -- 172.21.15.144:0/548925719 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43000325a0 con 0x7f4310098c70 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 4 set_mon_vals no callback set 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 1 -- 172.21.15.144:0/548925719 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300032940 con 0x7f4310098c70 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f430f7fe640 10 monclient: dump: 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:40.954 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 10 monclient: shutdown 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 1 -- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 msgr2=0x7f4310099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 1 --2- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310099040 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4300009880 tx=0x7f43000090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 1 -- 172.21.15.144:0/548925719 shutdown_connections 2024-09-18T12:26:40.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 1 --2- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431009e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 1 --2- 172.21.15.144:0/548925719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310099040 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 1 -- 172.21.15.144:0/548925719 >> 172.21.15.144:0/548925719 conn(0x7f4310070e90 msgr2=0x7f43100a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.944+0000 7f4316f9d640 1 -- 172.21.15.144:0/548925719 shutdown_connections 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.945+0000 7f4316f9d640 1 -- 172.21.15.144:0/548925719 wait complete. 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.945+0000 7f4316f9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.945+0000 7f4316f9d640 10 monclient: monmap: 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:40.956 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.945+0000 7f4316f9d640 1 Processor -- start 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.945+0000 7f4316f9d640 1 -- start start 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.945+0000 7f4316f9d640 10 monclient: init 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4310098c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 0x7f4310135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4310099580 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:40.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431011ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4310099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4310122ae0 con 0x7f4310098c70 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431006eba0 con 0x7f4310099580 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4310097ca0 con 0x7f4310099c70 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4316f9d640 10 monclient(hunting): authenticate will time out at 836.515417s 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 0x7f4310135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 0x7f4310135090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39614/0 (socket says 172.21.15.144:39614) 2024-09-18T12:26:40.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 -- 172.21.15.144:0/2278938752 learned_addr learned my addr 172.21.15.144:0/2278938752 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): get_auth_request con 0x7f4310099580 auth_method 0 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f431553a640 1 -- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 msgr2=0x7f431011ec00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f431553a640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431011ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4314d39640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4314d39640 10 monclient(hunting): get_auth_request con 0x7f4310098c70 auth_method 0 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4314d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:40.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4314d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4314d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient(hunting): handle_auth_done global_id 4465 payload 824 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 -- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 msgr2=0x7f431011ec00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431011ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 -- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 msgr2=0x7f4310141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 monclient: found mon.noname-b 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f430ffff640 1 -- 172.21.15.144:0/2278938752 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4300009390 con 0x7f4310099580 2024-09-18T12:26:40.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.946+0000 7f4314d39640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430ffff640 1 -- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 msgr2=0x7f4310135090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430ffff640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 0x7f4310135090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f430400c610 tx=0x7f4304004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304005cb0 con 0x7f4310099580 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: dump: 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:40.961 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: _finish_auth 0 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f4316f9d640 5 monclient: authenticate success, global_id 4465 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304005e30 con 0x7f4310099580 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f4316f9d640 10 monclient: _renew_subs 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f4316f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4310120680 con 0x7f4310099580 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304007340 con 0x7f4310099580 2024-09-18T12:26:40.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430cff9640 10 monclient: dump: 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:40.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f4316f9d640 10 monclient: _renew_subs 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f430effd640 4 set_mon_vals no callback set 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f4316f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.949+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43101420c0 con 0x7f4310099580 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f430401b400 con 0x7f4310099580 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f4316f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f4316f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f4316f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43101225b0 con 0x7f4310099580 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f430cff9640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f42fc038440 0x7f42fc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f430404b400 con 0x7f4310099580 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f4314d39640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f42fc038440 0x7f42fc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:40.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.950+0000 7f4314d39640 10 monclient: get_auth_request con 0x7f42fc038440 auth_method 0 2024-09-18T12:26:40.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.952+0000 7f4314d39640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f42fc038440 0x7f42fc03a8f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4300031000 tx=0x7f4300032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:40.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.954+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f430401a720 con 0x7f4310099580 2024-09-18T12:26:40.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.954+0000 7f430cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:40.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:40.954+0000 7f430cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:41.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.108+0000 7f4316f9d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:41.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.108+0000 7f4316f9d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:41.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.108+0000 7f4316f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:41.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.108+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f43100008d0 con 0x7f4310099580 2024-09-18T12:26:41.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.108+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v21) v1 ==== 105+0+14 (secure 0 0 0) 0x7f430401a360 con 0x7f4310099580 2024-09-18T12:26:41.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.108+0000 7f430cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:41.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.108+0000 7f430cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:41.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.200+0000 7f4316f9d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:41.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.200+0000 7f4316f9d640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:41.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.200+0000 7f4316f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:41.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.200+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f43101212b0 con 0x7f4310099580 2024-09-18T12:26:41.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.200+0000 7f430cff9640 1 -- 172.21.15.144:0/2278938752 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v21) v1 ==== 105+0+14 (secure 0 0 0) 0x7f430401a540 con 0x7f4310099580 2024-09-18T12:26:41.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.200+0000 7f430cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:41.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.200+0000 7f430cff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:26:41.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f42fc038440 msgr2=0x7f42fc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:41.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f42fc038440 0x7f42fc03a8f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4300031000 tx=0x7f4300032000 comp rx=0 tx=0).stop 2024-09-18T12:26:41.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 10 monclient: shutdown 2024-09-18T12:26:41.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:41.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 msgr2=0x7f4310135090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:41.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 0x7f4310135090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f430400c610 tx=0x7f4304004960 comp rx=0 tx=0).stop 2024-09-18T12:26:41.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 shutdown_connections 2024-09-18T12:26:41.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f42fc038440 0x7f42fc03a8f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4310099c70 0x7f431011ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4310099580 0x7f4310135090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 --2- 172.21.15.144:0/2278938752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4310098c70 0x7f4310141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 >> 172.21.15.144:0/2278938752 conn(0x7f4310070e90 msgr2=0x7f43100980f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:41.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 shutdown_connections 2024-09-18T12:26:41.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.202+0000 7f4316f9d640 1 -- 172.21.15.144:0/2278938752 wait complete. 2024-09-18T12:26:41.212 INFO:tasks.workunit.client.0.smithi144.stdout:pg_num_min: 2 2024-09-18T12:26:41.213 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-18T12:26:41.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.286+0000 7f008c99d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.286+0000 7f008c99d640 10 monclient: build_initial_monmap 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.286+0000 7f008c99d640 10 monclient: monmap: 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:41.287582+0000 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:41.287582+0000 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:41.288 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:41.289 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:41.289 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:41.289 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:41.289 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:41.289 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:41.289 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:41.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 1 Processor -- start 2024-09-18T12:26:41.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 1 -- start start 2024-09-18T12:26:41.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient: init 2024-09-18T12:26:41.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0088098050 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f008809fe90 0x7f0088098420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient(hunting): picked mon.noname-c con 0x7f008809fe90 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098960 0x7f0088138d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient(hunting): picked mon.noname-b con 0x7f0088098960 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0088132710 con 0x7f0088098050 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008806a230 con 0x7f008809fe90 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:41.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00880a1230 con 0x7f0088098960 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.287+0000 7f008c99d640 10 monclient(hunting): authenticate will time out at 836.856878s 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0086d9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098960 msgr2=0x7f0088138d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f008809fe90 0x7f0088098420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0086d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098960 0x7f0088138d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f008809fe90 0x7f0088098420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39628/0 (socket says 172.21.15.144:39628) 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 -- 172.21.15.144:0/4208895587 learned_addr learned my addr 172.21.15.144:0/4208895587 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): get_auth_request con 0x7f008809fe90 auth_method 0 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:41.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient(hunting): handle_auth_done global_id 4468 payload 274 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 -- 172.21.15.144:0/4208895587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098960 msgr2=0x7f0088138d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 --2- 172.21.15.144:0/4208895587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098960 0x7f0088138d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 -- 172.21.15.144:0/4208895587 >> v1:172.21.15.116:6789/0 conn(0x7f0088098050 legacy=0x7f0088132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 monclient: found mon.noname-c 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:41.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.288+0000 7f0085d9b640 1 -- 172.21.15.144:0/4208895587 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00880a1510 con 0x7f008809fe90 2024-09-18T12:26:41.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f0085d9b640 1 -- 172.21.15.144:0/4208895587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 msgr2=0x7f0088098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:41.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f0085d9b640 1 --2- 172.21.15.144:0/4208895587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088098420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f007c009910 tx=0x7f007c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c20707c5af1f0a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:41.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 1 -- 172.21.15.144:0/4208895587 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c032440 con 0x7f008809fe90 2024-09-18T12:26:41.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:41.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: dump: 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: _finish_auth 0 2024-09-18T12:26:41.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 5 monclient: authenticate success, global_id 4468 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 1 -- 172.21.15.144:0/4208895587 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c0325a0 con 0x7f008809fe90 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 4 set_mon_vals no callback set 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 1 -- 172.21.15.144:0/4208895587 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c032940 con 0x7f008809fe90 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008559a640 10 monclient: dump: 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:41.302 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 10 monclient: shutdown 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 1 -- 172.21.15.144:0/4208895587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 msgr2=0x7f0088098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.291+0000 7f008c99d640 1 --2- 172.21.15.144:0/4208895587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088098420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f007c009910 tx=0x7f007c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 -- 172.21.15.144:0/4208895587 shutdown_connections 2024-09-18T12:26:41.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 --2- 172.21.15.144:0/4208895587 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098960 0x7f0088138d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 --2- 172.21.15.144:0/4208895587 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088098420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 -- 172.21.15.144:0/4208895587 >> 172.21.15.144:0/4208895587 conn(0x7f00880a5190 msgr2=0x7f00880a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 -- 172.21.15.144:0/4208895587 shutdown_connections 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 -- 172.21.15.144:0/4208895587 wait complete. 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 10 monclient: build_initial_monmap 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 10 monclient: monmap: 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:41.304 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 Processor -- start 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 1 -- start start 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.292+0000 7f008c99d640 10 monclient: init 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098050 0x7f0088134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): picked mon.noname-c con 0x7f0088098050 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088098960 0x7f0088134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:41.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): picked mon.noname-a con 0x7f0088098960 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088134fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): picked mon.noname-b con 0x7f008809fe90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00880a3100 con 0x7f0088098960 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881363c0 con 0x7f008809fe90 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0088098d70 con 0x7f0088098050 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008c99d640 10 monclient(hunting): authenticate will time out at 836.862238s 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088134fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:41.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088134fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39644/0 (socket says 172.21.15.144:39644) 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 -- 172.21.15.144:0/119283976 learned_addr learned my addr 172.21.15.144:0/119283976 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): get_auth_request con 0x7f008809fe90 auth_method 0 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008659c640 1 -- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098050 msgr2=0x7f0088134520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f008659c640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098050 0x7f0088134520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0085d9b640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088098960 0x7f0088134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0085d9b640 10 monclient(hunting): get_auth_request con 0x7f0088098960 auth_method 0 2024-09-18T12:26:41.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0085d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0085d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0085d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient(hunting): handle_auth_done global_id 4471 payload 824 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 -- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098050 msgr2=0x7f0088134520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098050 0x7f0088134520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 -- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088098960 msgr2=0x7f0088134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088098960 0x7f0088134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 monclient: found mon.noname-b 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.293+0000 7f0086d9d640 1 -- 172.21.15.144:0/119283976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c009450 con 0x7f008809fe90 2024-09-18T12:26:41.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f0086d9d640 1 -- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 msgr2=0x7f0088134fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f0086d9d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088134fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f008000bb40 tx=0x7f00800097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008000cbf0 con 0x7f008809fe90 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: dump: 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:41.309 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f008c99d640 5 monclient: authenticate success, global_id 4471 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008000cd70 con 0x7f008809fe90 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080006150 con 0x7f008809fe90 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f00767fc640 10 monclient: dump: 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:41.310 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f008c99d640 10 monclient: _renew_subs 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f0084d99640 4 set_mon_vals no callback set 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f008c99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.295+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0088137900 con 0x7f008809fe90 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f008c99d640 10 monclient: _renew_subs 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f008c99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:41.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00881316a0 con 0x7f008809fe90 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0080012020 con 0x7f008809fe90 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f008c99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f008c99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f008c99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.296+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0088135ea0 con 0x7f008809fe90 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.297+0000 7f00767fc640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0068038440 0x7f006803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.297+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f008004b6e0 con 0x7f008809fe90 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.297+0000 7f008659c640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0068038440 0x7f006803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.297+0000 7f008659c640 10 monclient: get_auth_request con 0x7f0068038440 auth_method 0 2024-09-18T12:26:41.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.299+0000 7f008659c640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0068038440 0x7f006803a8f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0078007fd0 tx=0x7f0078007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:41.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:41.301+0000 7f0086d9d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088134fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f008000bb40 tx=0x7f00800097d0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f008001d900 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:26:42.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.129+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f008001d900 con 0x7f008809fe90 2024-09-18T12:26:42.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.129+0000 7f00767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:42.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.129+0000 7f00767fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:42.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.286+0000 7f008c99d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T12:26:42.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.286+0000 7f008c99d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T12:26:42.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.286+0000 7f008c99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:42.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.286+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f008804bf60 con 0x7f008809fe90 2024-09-18T12:26:42.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.853+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v22) v1 ==== 145+0+0 (secure 0 0 0) 0x7f008001d300 con 0x7f008809fe90 2024-09-18T12:26:42.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.853+0000 7f00767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T12:26:42.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.853+0000 7f00767fc640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-18T12:26:42.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.947+0000 7f008c99d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T12:26:42.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.947+0000 7f008c99d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T12:26:42.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.947+0000 7f008c99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:42.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:42.947+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f0088131cf0 con 0x7f008809fe90 2024-09-18T12:26:44.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.093+0000 7f00767fc640 1 -- 172.21.15.144:0/119283976 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v23) v1 ==== 145+0+0 (secure 0 0 0) 0x7f008001d4e0 con 0x7f008809fe90 2024-09-18T12:26:44.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.093+0000 7f00767fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T12:26:44.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.093+0000 7f00767fc640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-18T12:26:44.293 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 3 pg_num_max to 33 2024-09-18T12:26:44.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0068038440 msgr2=0x7f006803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0068038440 0x7f006803a8f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0078007fd0 tx=0x7f0078007a70 comp rx=0 tx=0).stop 2024-09-18T12:26:44.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 10 monclient: shutdown 2024-09-18T12:26:44.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 msgr2=0x7f0088134fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088134fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f008000bb40 tx=0x7f00800097d0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 shutdown_connections 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0068038440 0x7f006803a8f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f008809fe90 0x7f0088134fa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088098960 0x7f0088134a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 --2- 172.21.15.144:0/119283976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0088098050 0x7f0088134520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 >> 172.21.15.144:0/119283976 conn(0x7f00880a5190 msgr2=0x7f008809afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 shutdown_connections 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.096+0000 7f008c99d640 1 -- 172.21.15.144:0/119283976 wait complete. 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-18T12:26:44.303 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 33 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.186+0000 7f9aed35d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.186+0000 7f9aed35d640 10 monclient: build_initial_monmap 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.186+0000 7f9aed35d640 10 monclient: monmap: 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:44.187419+0000 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:44.187419+0000 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.304 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 1 Processor -- start 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 1 -- start start 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient: init 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ae809ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:44.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9ae809f020 0x7f9ae8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae809f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae8136d60 0x7f9ae8139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae8136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae8067420 con 0x7f9ae809ec50 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae809fdc0 con 0x7f9ae809f020 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae80a0510 con 0x7f9ae8136d60 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9aed35d640 10 monclient(hunting): authenticate will time out at 839.756677s 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9ae809f020 0x7f9ae8069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae77fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae8136d60 msgr2=0x7f9ae8139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:44.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9ae809f020 0x7f9ae8069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39658/0 (socket says 172.21.15.144:39658) 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 1 -- 172.21.15.144:0/1366029591 learned_addr learned my addr 172.21.15.144:0/1366029591 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 10 monclient(hunting): get_auth_request con 0x7f9ae809f020 auth_method 0 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae77fe640 1 --2- 172.21.15.144:0/1366029591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae8136d60 0x7f9ae8139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.187+0000 7f9ae67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 10 monclient(hunting): handle_auth_done global_id 4474 payload 274 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 1 -- 172.21.15.144:0/1366029591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae8136d60 msgr2=0x7f9ae8139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 1 --2- 172.21.15.144:0/1366029591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae8136d60 0x7f9ae8139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 1 -- 172.21.15.144:0/1366029591 >> v1:172.21.15.116:6789/0 conn(0x7f9ae809ec50 legacy=0x7f9ae8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:44.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 10 monclient: _finish_hunting 0 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 1 monclient: found mon.noname-c 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.188+0000 7f9ae67fc640 1 -- 172.21.15.144:0/1366029591 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae80a0200 con 0x7f9ae809f020 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.190+0000 7f9ae67fc640 1 -- 172.21.15.144:0/1366029591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809f020 msgr2=0x7f9ae8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.190+0000 7f9ae67fc640 1 --2- 172.21.15.144:0/1366029591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809f020 0x7f9ae8069d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009910 tx=0x7f9ad00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f802c3071a9cc58e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 1 -- 172.21.15.144:0/1366029591 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0032440 con 0x7f9ae809f020 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: dump: 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.308 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 1 -- 172.21.15.144:0/1366029591 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad00325a0 con 0x7f9ae809f020 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 5 monclient: authenticate success, global_id 4474 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:44.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 1 -- 172.21.15.144:0/1366029591 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0032940 con 0x7f9ae809f020 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 4 set_mon_vals no callback set 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9ae5ffb640 10 monclient: dump: 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.310 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 10 monclient: shutdown 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 -- 172.21.15.144:0/1366029591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809f020 msgr2=0x7f9ae8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 --2- 172.21.15.144:0/1366029591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809f020 0x7f9ae8069d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0009910 tx=0x7f9ad00091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 -- 172.21.15.144:0/1366029591 shutdown_connections 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 --2- 172.21.15.144:0/1366029591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae8136d60 0x7f9ae8139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 --2- 172.21.15.144:0/1366029591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809f020 0x7f9ae8069d10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 -- 172.21.15.144:0/1366029591 >> 172.21.15.144:0/1366029591 conn(0x7f9ae80a5d40 msgr2=0x7f9ae80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 -- 172.21.15.144:0/1366029591 shutdown_connections 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.191+0000 7f9aed35d640 1 -- 172.21.15.144:0/1366029591 wait complete. 2024-09-18T12:26:44.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.192+0000 7f9aed35d640 10 monclient: build_initial_monmap 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.192+0000 7f9aed35d640 10 monclient: monmap: 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.192+0000 7f9aed35d640 1 Processor -- start 2024-09-18T12:26:44.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.192+0000 7f9aed35d640 1 -- start start 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.192+0000 7f9aed35d640 10 monclient: init 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 0x7f9ae80972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae809ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae809f020 0x7f9ae8097800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8136d60 0x7f9ae8097d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae8136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae8098d50 con 0x7f9ae8136d60 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae809c000 con 0x7f9ae809ec50 2024-09-18T12:26:44.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae80a3b80 con 0x7f9ae809f020 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae67fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae809f020 msgr2=0x7f9ae8097800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9aed35d640 10 monclient(hunting): authenticate will time out at 839.762419s 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae67fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae809f020 0x7f9ae8097800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 0x7f9ae80972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 0x7f9ae80972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39674/0 (socket says 172.21.15.144:39674) 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 -- 172.21.15.144:0/3129909861 learned_addr learned my addr 172.21.15.144:0/3129909861 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): get_auth_request con 0x7f9ae809ec50 auth_method 0 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8136d60 0x7f9ae8097d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 10 monclient(hunting): get_auth_request con 0x7f9ae8136d60 auth_method 0 2024-09-18T12:26:44.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:44.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:44.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2024-09-18T12:26:44.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 -- 172.21.15.144:0/3129909861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae809f020 msgr2=0x7f9ae8097800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:44.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae809f020 0x7f9ae8097800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 -- 172.21.15.144:0/3129909861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8136d60 msgr2=0x7f9ae8097d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8136d60 0x7f9ae8097d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 monclient: found mon.noname-b 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.193+0000 7f9ae6ffd640 1 -- 172.21.15.144:0/3129909861 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0009450 con 0x7f9ae809ec50 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9ae6ffd640 1 -- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 msgr2=0x7f9ae80972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9ae6ffd640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 0x7f9ae80972c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c610 tx=0x7f9ad8007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8012840 con 0x7f9ae809ec50 2024-09-18T12:26:44.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:44.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: dump: 2024-09-18T12:26:44.318 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.318 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.318 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.319 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.319 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.319 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.319 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.319 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.319 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.319 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9aed35d640 5 monclient: authenticate success, global_id 4477 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad80129e0 con 0x7f9ae809ec50 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8012da0 con 0x7f9ae809ec50 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:44.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9adeffd640 10 monclient: dump: 2024-09-18T12:26:44.321 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.321 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.321 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.321 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.321 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.321 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9aed35d640 10 monclient: _renew_subs 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9ae57fa640 4 set_mon_vals no callback set 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9aed35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae809a7f0 con 0x7f9ae809ec50 2024-09-18T12:26:44.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9aed35d640 10 monclient: _renew_subs 2024-09-18T12:26:44.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9aed35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.196+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae809b650 con 0x7f9ae809ec50 2024-09-18T12:26:44.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9ad8011590 con 0x7f9ae809ec50 2024-09-18T12:26:44.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9aed35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9aed35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9aed35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae809abf0 con 0x7f9ae809ec50 2024-09-18T12:26:44.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9adeffd640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9ac8038440 0x7f9ac803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f9ad804b2d0 con 0x7f9ae809ec50 2024-09-18T12:26:44.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9ae67fc640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9ac8038440 0x7f9ac803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.197+0000 7f9ae67fc640 10 monclient: get_auth_request con 0x7f9ac8038440 auth_method 0 2024-09-18T12:26:44.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.200+0000 7f9ae67fc640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9ac8038440 0x7f9ac803a8f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80719d0 tx=0x7f9ad0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.202+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9ad8019910 con 0x7f9ae809ec50 2024-09-18T12:26:44.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.202+0000 7f9adeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.202+0000 7f9adeffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:44.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.356+0000 7f9aed35d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:44.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.356+0000 7f9aed35d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:44.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.356+0000 7f9aed35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.356+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f9ae80008d0 con 0x7f9ae809ec50 2024-09-18T12:26:44.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.356+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v23) v1 ==== 105+0+15 (secure 0 0 0) 0x7f9ad8019310 con 0x7f9ae809ec50 2024-09-18T12:26:44.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.356+0000 7f9adeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:44.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.356+0000 7f9adeffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:26:44.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.447+0000 7f9aed35d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:44.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.447+0000 7f9aed35d640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:44.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.447+0000 7f9aed35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.447+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f9ae8064900 con 0x7f9ae809ec50 2024-09-18T12:26:44.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.447+0000 7f9adeffd640 1 -- 172.21.15.144:0/3129909861 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v23) v1 ==== 105+0+15 (secure 0 0 0) 0x7f9ad80194f0 con 0x7f9ae809ec50 2024-09-18T12:26:44.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.447+0000 7f9adeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:44.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.447+0000 7f9adeffd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9ac8038440 msgr2=0x7f9ac803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9ac8038440 0x7f9ac803a8f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80719d0 tx=0x7f9ad0032000 comp rx=0 tx=0).stop 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 10 monclient: shutdown 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 msgr2=0x7f9ae80972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 0x7f9ae80972c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c610 tx=0x7f9ad8007ed0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 shutdown_connections 2024-09-18T12:26:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9ac8038440 0x7f9ac803a8f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8136d60 0x7f9ae8097d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9ae809f020 0x7f9ae8097800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 --2- 172.21.15.144:0/3129909861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9ae809ec50 0x7f9ae80972c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 >> 172.21.15.144:0/3129909861 conn(0x7f9ae80a5d40 msgr2=0x7f9ae809d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 shutdown_connections 2024-09-18T12:26:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.449+0000 7f9aed35d640 1 -- 172.21.15.144:0/3129909861 wait complete. 2024-09-18T12:26:44.459 INFO:tasks.workunit.client.0.smithi144.stdout:pg_num_max: 33 2024-09-18T12:26:44.460 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-18T12:26:44.460 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:26:44.460 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.533+0000 7fe2c499d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.533+0000 7fe2c499d640 10 monclient: build_initial_monmap 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.533+0000 7fe2c499d640 10 monclient: monmap: 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:44.534716+0000 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:44.534716+0000 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:44.535 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:44.536 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:44.536 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:44.536 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.534+0000 7fe2c499d640 1 Processor -- start 2024-09-18T12:26:44.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.534+0000 7fe2c499d640 1 -- start start 2024-09-18T12:26:44.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.534+0000 7fe2c499d640 10 monclient: init 2024-09-18T12:26:44.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:44.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T12:26:44.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 0x7fe2c0132000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c0098050 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2c009fe90 0x7fe2c0098420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c009fe90 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2c0098960 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c00a1230 con 0x7fe2c0098960 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c006a230 con 0x7fe2c009fe90 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00a1510 con 0x7fe2c0098050 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2c499d640 10 monclient(hunting): authenticate will time out at 840.104194s 2024-09-18T12:26:44.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2be59c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 msgr2=0x7fe2c0132000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2be59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 0x7fe2c0132000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2c009fe90 0x7fe2c0098420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2c009fe90 0x7fe2c0098420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:50034/0 (socket says 172.21.15.144:50034) 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 -- 172.21.15.144:0/4036693061 learned_addr learned my addr 172.21.15.144:0/4036693061 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): get_auth_request con 0x7fe2c009fe90 auth_method 0 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe2c0098960 0x7fe2c0068dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:37734/0 (socket says 172.21.15.144:37734) 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient(hunting): handle_auth_done global_id 4551 payload 274 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 -- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 msgr2=0x7fe2c0132000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 --2- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 0x7fe2c0132000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 -- 172.21.15.144:0/4036693061 >> v1:172.21.15.116:6789/0 conn(0x7fe2c0098960 legacy=0x7fe2c0068dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 monclient: found mon.noname-a 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.535+0000 7fe2bdd9b640 1 -- 172.21.15.144:0/4036693061 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0132540 con 0x7fe2c009fe90 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.538+0000 7fe2bdd9b640 1 -- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c009fe90 msgr2=0x7fe2c0098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.538+0000 7fe2bdd9b640 1 --2- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c009fe90 0x7fe2c0098420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009910 tx=0x7fe2b40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178e5458244e04e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 1 -- 172.21.15.144:0/4036693061 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4032440 con 0x7fe2c009fe90 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: dump: 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.547 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: _finish_auth 0 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 1 -- 172.21.15.144:0/4036693061 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b40325a0 con 0x7fe2c009fe90 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 1 -- 172.21.15.144:0/4036693061 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4032940 con 0x7fe2c009fe90 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2c499d640 5 monclient: authenticate success, global_id 4551 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2c499d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2c499d640 4 set_mon_vals no callback set 2024-09-18T12:26:44.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.539+0000 7fe2bd59a640 10 monclient: dump: 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 10 monclient: shutdown 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 -- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c009fe90 msgr2=0x7fe2c0098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 --2- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c009fe90 0x7fe2c0098420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009910 tx=0x7fe2b40091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 -- 172.21.15.144:0/4036693061 shutdown_connections 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 --2- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c009fe90 0x7fe2c0098420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 --2- 172.21.15.144:0/4036693061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 0x7fe2c0132000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 -- 172.21.15.144:0/4036693061 >> 172.21.15.144:0/4036693061 conn(0x7fe2c00a5190 msgr2=0x7fe2c00a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 -- 172.21.15.144:0/4036693061 shutdown_connections 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 1 -- 172.21.15.144:0/4036693061 wait complete. 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 10 monclient: build_initial_monmap 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.540+0000 7fe2c499d640 10 monclient: monmap: 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 Processor -- start 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 -- start start 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient: init 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 0x7fe2c009e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c0098050 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c0098960 0x7fe2c0130d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c0098960 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c009fe90 0x7fe2c0140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c009fe90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c009b360 con 0x7fe2c0098960 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c011cec0 con 0x7fe2c009fe90 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c011de20 con 0x7fe2c0098050 2024-09-18T12:26:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2c499d640 10 monclient(hunting): authenticate will time out at 840.110647s 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c009fe90 0x7fe2c0140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2be59c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 msgr2=0x7fe2c009e690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2be59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 0x7fe2c009e690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c009fe90 0x7fe2c0140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39682/0 (socket says 172.21.15.144:39682) 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 1 -- 172.21.15.144:0/3586561116 learned_addr learned my addr 172.21.15.144:0/3586561116 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): get_auth_request con 0x7fe2c009fe90 auth_method 0 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bdd9b640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c0098960 0x7fe2c0130d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.541+0000 7fe2bed9d640 10 monclient(hunting): handle_auth_done global_id 4480 payload 824 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 1 -- 172.21.15.144:0/3586561116 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 msgr2=0x7fe2c009e690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2c0098050 0x7fe2c009e690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 1 -- 172.21.15.144:0/3586561116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c0098960 msgr2=0x7fe2c0130d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c0098960 0x7fe2c0130d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 1 monclient: found mon.noname-b 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bed9d640 1 -- 172.21.15.144:0/3586561116 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b4009450 con 0x7fe2c009fe90 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bdd9b640 10 monclient: get_auth_request con 0x7fe2c0098960 auth_method 0 2024-09-18T12:26:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.542+0000 7fe2bdd9b640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2c0098960 0x7fe2c0130d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2bed9d640 1 -- 172.21.15.144:0/3586561116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c009fe90 msgr2=0x7fe2c0140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2bed9d640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2c009fe90 0x7fe2c0140f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8007fd0 tx=0x7fe2b8004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 1 -- 172.21.15.144:0/3586561116 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b800ce40 con 0x7fe2c009fe90 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: dump: 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 1 -- 172.21.15.144:0/3586561116 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b800a640 con 0x7fe2c009fe90 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2c499d640 5 monclient: authenticate success, global_id 4480 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2c499d640 10 monclient: _renew_subs 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2c499d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2c499d640 1 -- 172.21.15.144:0/3586561116 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c011e100 con 0x7fe2c009fe90 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 1 -- 172.21.15.144:0/3586561116 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b800aa00 con 0x7fe2c009fe90 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2ae7fc640 10 monclient: dump: 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2bcd99640 4 set_mon_vals no callback set 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2c499d640 10 monclient: _renew_subs 2024-09-18T12:26:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2c499d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.544+0000 7fe2c499d640 1 -- 172.21.15.144:0/3586561116 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c011bcc0 con 0x7fe2c009fe90 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2ae7fc640 1 -- 172.21.15.144:0/3586561116 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe2b80070f0 con 0x7fe2c009fe90 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2c499d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2c499d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2c499d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2c499d640 1 -- 172.21.15.144:0/3586561116 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c0098de0 con 0x7fe2c009fe90 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2ae7fc640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe2a0038490 0x7fe2a003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2ae7fc640 1 -- 172.21.15.144:0/3586561116 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fe2b804b680 con 0x7fe2c009fe90 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2be59c640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe2a0038490 0x7fe2a003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.545+0000 7fe2be59c640 10 monclient: get_auth_request con 0x7fe2a0038490 auth_method 0 2024-09-18T12:26:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.547+0000 7fe2be59c640 1 --2- 172.21.15.144:0/3586561116 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe2a0038490 0x7fe2a003a940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00038f0 tx=0x7fe2b0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.549+0000 7fe2ae7fc640 1 -- 172.21.15.144:0/3586561116 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe2c0098de0 con 0x7fe2c009fe90 2024-09-18T12:26:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.549+0000 7fe2ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.549+0000 7fe2ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.704+0000 7fe2c499d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-18T12:26:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.704+0000 7fe2c499d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-18T12:26:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.704+0000 7fe2c499d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.704+0000 7fe2c499d640 1 -- 172.21.15.144:0/3586561116 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) v1 -- 0x7fe2c0098de0 con 0x7fe2c009fe90 2024-09-18T12:26:44.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.705+0000 7fe2ae7fc640 1 -- 172.21.15.144:0/3586561116 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v23) v1 ==== 150+0+0 (secure 0 0 0) 0x7fe2c0098de0 con 0x7fe2c009fe90 2024-09-18T12:26:44.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.705+0000 7fe2ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-18T12:26:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.705+0000 7fe2ae7fc640 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-18T12:26:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-18T12:26:44.710 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:26:44.710 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-18T12:26:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:26:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-18T12:26:44.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.784+0000 7f14a5017640 10 monclient: get_monmap_and_config 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.784+0000 7f14a5017640 10 monclient: build_initial_monmap 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.784+0000 7f14a5017640 10 monclient: monmap: 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:44.785567+0000 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:44.785567+0000 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:44.786 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:44.787 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 1 Processor -- start 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 1 -- start start 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient: init 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14a0098050 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 0x7f14a0098420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient(hunting): picked mon.noname-b con 0x7f14a009fe90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14a0098960 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:44.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a0132710 con 0x7f14a0098050 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a00a1230 con 0x7f14a0098960 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a006a230 con 0x7f14a009fe90 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f14a5017640 10 monclient(hunting): authenticate will time out at 840.354876s 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.785+0000 7f1497fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 msgr2=0x7f14a0098420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.786+0000 7f149f59e640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f14a0098960 0x7f14a0138d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59726/0 (socket says 172.21.15.144:59726) 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.786+0000 7f149f59e640 1 -- 172.21.15.144:0/1929673790 learned_addr learned my addr 172.21.15.144:0/1929673790 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:44.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.786+0000 7f1497fff640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 0x7f14a0098420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:44.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149f59e640 1 -- 172.21.15.144:0/1929673790 >> v1:172.21.15.144:6789/0 conn(0x7f14a0098960 legacy=0x7f14a0138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149ed9d640 1 -- 172.21.15.144:0/1929673790 >> v1:172.21.15.116:6789/0 conn(0x7f14a0098050 legacy=0x7f14a0132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491932847 0 0) 0x7f14a00a1230 con 0x7f14a0098960 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): my global_id is 4483 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148c0034e0 con 0x7f14a0098960 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923465435 0 0) 0x7f14a0132710 con 0x7f14a0098050 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): my global_id is 4554 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a00a1230 con 0x7f14a0098050 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3884734168 0 0) 0x7f148c0034e0 con 0x7f14a0098960 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 msgr2=0x7f14a0098420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 0x7f14a0098420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 >> v1:172.21.15.116:6789/0 conn(0x7f14a0098050 legacy=0x7f14a0132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a00a1510 con 0x7f14a0098960 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: _finish_auth 0 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f1498001b20 con 0x7f14a0098960 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: dump: 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.798 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 >> v1:172.21.15.144:6789/0 conn(0x7f14a0098960 legacy=0x7f14a0138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 0x7f148c003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): picked mon.c con 0x7f14a009fe90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:44.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f148c004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): picked mon.b con 0x7f148c0041d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f148c004ec0 0x7f148c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): picked mon.a con 0x7f148c004ec0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a006a230 con 0x7f148c004ec0 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c00bf10 con 0x7f148c0041d0 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c00c050 con 0x7f14a009fe90 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149e59c640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149ed9d640 1 -- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 msgr2=0x7f148c003c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:44.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.788+0000 7f149ed9d640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 0x7f148c003c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f148c004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): get_auth_request con 0x7f148c0041d0 auth_method 0 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f148c004ec0 0x7f148c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 10 monclient(hunting): get_auth_request con 0x7f148c004ec0 auth_method 0 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f149f59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient(hunting): handle_auth_done global_id 4483 payload 293 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 1 -- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 msgr2=0x7f148c003c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a009fe90 0x7f148c003c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 1 -- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f148c004ec0 msgr2=0x7f148c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f148c004ec0 0x7f148c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 1 monclient: found mon.b 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 1 -- 172.21.15.144:0/1929673790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0034e0 con 0x7f148c0041d0 2024-09-18T12:26:44.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 10 monclient: _finish_auth 0 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f1497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f14a5017640 5 monclient: authenticate success, global_id 4483 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.789+0000 7f14a5017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.791+0000 7f1497fff640 1 -- 172.21.15.144:0/1929673790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 msgr2=0x7f148c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.791+0000 7f1497fff640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f148c004780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14880099c0 tx=0x7f1488009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2162c1922f9a4fbf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488003b80 con 0x7f148c0041d0 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 10 monclient: dump: 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.803 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488003d20 con 0x7f148c0041d0 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 1 -- 172.21.15.144:0/1929673790 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148802ab80 con 0x7f148c0041d0 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 4 set_mon_vals no callback set 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149e59c640 10 monclient: dump: 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 10 monclient: shutdown 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:44.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 1 -- 172.21.15.144:0/1929673790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 msgr2=0x7f148c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 1 --2- 172.21.15.144:0/1929673790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f148c004780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14880099c0 tx=0x7f1488009060 comp rx=0 tx=0).stop 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f149f59e640 1 -- 172.21.15.144:0/1929673790 reap_dead start 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 1 -- 172.21.15.144:0/1929673790 shutdown_connections 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 1 -- 172.21.15.144:0/1929673790 >> 172.21.15.144:0/1929673790 conn(0x7f14a00a5190 msgr2=0x7f14a00a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 1 -- 172.21.15.144:0/1929673790 shutdown_connections 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.792+0000 7f14a5017640 1 -- 172.21.15.144:0/1929673790 wait complete. 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 10 monclient: build_initial_monmap 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 10 monclient: monmap: 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:44.806 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 1 Processor -- start 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 1 -- start start 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 10 monclient: init 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f14a009e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 10 monclient(hunting): picked mon.noname-b con 0x7f148c0041d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14a01363d0 0x7f14a0130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 10 monclient(hunting): picked mon.noname-a con 0x7f14a01363d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:44.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.793+0000 7f14a5017640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a0130fc0 0x7f14a011b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 10 monclient(hunting): picked mon.noname-c con 0x7f14a0130fc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a009ef50 con 0x7f14a01363d0 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0098d70 con 0x7f148c0041d0 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0134ae0 con 0x7f14a0130fc0 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f14a5017640 10 monclient(hunting): authenticate will time out at 840.363029s 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f14a009e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f14a009e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39700/0 (socket says 172.21.15.144:39700) 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 -- 172.21.15.144:0/2771604579 learned_addr learned my addr 172.21.15.144:0/2771604579 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:44.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): get_auth_request con 0x7f148c0041d0 auth_method 0 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149f59e640 1 -- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a0130fc0 msgr2=0x7f14a011b7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149f59e640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a0130fc0 0x7f14a011b7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient(hunting): handle_auth_done global_id 4486 payload 824 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f1497fff640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14a01363d0 0x7f14a0130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 -- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a0130fc0 msgr2=0x7f14a011b7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f14a0130fc0 0x7f14a011b7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 -- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14a01363d0 msgr2=0x7f14a0130a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14a01363d0 0x7f14a0130a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 monclient: found mon.noname-b 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f149ed9d640 1 -- 172.21.15.144:0/2771604579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a0141e00 con 0x7f148c0041d0 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f1497fff640 10 monclient: get_auth_request con 0x7f14a01363d0 auth_method 0 2024-09-18T12:26:44.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.794+0000 7f1497fff640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14a01363d0 0x7f14a0130a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:26:44.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f149ed9d640 1 -- 172.21.15.144:0/2771604579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 msgr2=0x7f14a009e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:44.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f149ed9d640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f148c0041d0 0x7f14a009e400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f149000c7f0 tx=0x7f1490001ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 1 -- 172.21.15.144:0/2771604579 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490007510 con 0x7f148c0041d0 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: dump: 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.813 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 1 -- 172.21.15.144:0/2771604579 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1490005780 con 0x7f148c0041d0 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f14a5017640 5 monclient: authenticate success, global_id 4486 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f14a5017640 10 monclient: _renew_subs 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f14a5017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f14a5017640 1 -- 172.21.15.144:0/2771604579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a011dba0 con 0x7f148c0041d0 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 1 -- 172.21.15.144:0/2771604579 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490005b40 con 0x7f148c0041d0 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f1496ffd640 10 monclient: dump: 2024-09-18T12:26:44.814 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f149dd9b640 4 set_mon_vals no callback set 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f14a5017640 10 monclient: _renew_subs 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f14a5017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.796+0000 7f14a5017640 1 -- 172.21.15.144:0/2771604579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a011bd30 con 0x7f148c0041d0 2024-09-18T12:26:44.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f1496ffd640 1 -- 172.21.15.144:0/2771604579 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f149001b3b0 con 0x7f148c0041d0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f1496ffd640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14800384c0 0x7f148003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f14a5017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f1496ffd640 1 -- 172.21.15.144:0/2771604579 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f149004ace0 con 0x7f148c0041d0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f14a5017640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f14a5017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f1497fff640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14800384c0 0x7f148003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f14a5017640 1 -- 172.21.15.144:0/2771604579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a011dde0 con 0x7f148c0041d0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.797+0000 7f1497fff640 10 monclient: get_auth_request con 0x7f14800384c0 auth_method 0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.800+0000 7f1497fff640 1 --2- 172.21.15.144:0/2771604579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14800384c0 0x7f148003a970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f148802f000 tx=0x7f1488030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.802+0000 7f1496ffd640 1 -- 172.21.15.144:0/2771604579 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f14a011dde0 con 0x7f148c0041d0 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.802+0000 7f1496ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:44.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.802+0000 7f1496ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:44.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.955+0000 7f14a5017640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-18T12:26:44.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.955+0000 7f14a5017640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-18T12:26:44.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.955+0000 7f14a5017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:44.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.955+0000 7f14a5017640 1 -- 172.21.15.144:0/2771604579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) v1 -- 0x7f14a011dde0 con 0x7f148c0041d0 2024-09-18T12:26:44.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.956+0000 7f1496ffd640 1 -- 172.21.15.144:0/2771604579 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v23) v1 ==== 150+0+0 (secure 0 0 0) 0x7f14a011dde0 con 0x7f148c0041d0 2024-09-18T12:26:44.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.956+0000 7f1496ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-18T12:26:44.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:44.956+0000 7f1496ffd640 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-18T12:26:44.958 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-18T12:26:44.961 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:26:44.961 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-18T12:26:44.961 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:26:44.961 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.036+0000 7fae2e034640 10 monclient: get_monmap_and_config 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.036+0000 7fae2e034640 10 monclient: build_initial_monmap 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.036+0000 7fae2e034640 10 monclient: monmap: 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:45.037045+0000 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:45.037045+0000 2024-09-18T12:26:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:45.038 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 Processor -- start 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 -- start start 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient: init 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fae2809ec50 0x7fae28131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): picked mon.noname-c con 0x7fae2809ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fae281321a0 0x7fae28068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): picked mon.noname-a con 0x7fae281321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809f020 0x7fae28069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): picked mon.noname-b con 0x7fae2809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae28067420 con 0x7fae281321a0 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2809fdc0 con 0x7fae2809ec50 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae280a0510 con 0x7fae2809f020 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae2e034640 10 monclient(hunting): authenticate will time out at 840.606675s 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fae2809ec50 0x7fae28131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fae2809ec50 0x7fae28131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39712/0 (socket says 172.21.15.144:39712) 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 1 -- 172.21.15.144:0/1305104590 learned_addr learned my addr 172.21.15.144:0/1305104590 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): get_auth_request con 0x7fae2809ec50 auth_method 0 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae27fff640 1 -- 172.21.15.144:0/1305104590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809f020 msgr2=0x7fae28069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae27fff640 1 --2- 172.21.15.144:0/1305104590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809f020 0x7fae28069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae1ffff640 1 --2- 172.21.15.144:0/1305104590 >> v2:172.21.15.116:3300/0 conn(0x7fae281321a0 0x7fae28068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae1ffff640 10 monclient(hunting): get_auth_request con 0x7fae281321a0 auth_method 0 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.037+0000 7fae277fe640 10 monclient(hunting): handle_auth_done global_id 4489 payload 274 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 1 -- 172.21.15.144:0/1305104590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809f020 msgr2=0x7fae28069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 1 --2- 172.21.15.144:0/1305104590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809f020 0x7fae28069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 1 -- 172.21.15.144:0/1305104590 >> v2:172.21.15.116:3300/0 conn(0x7fae281321a0 msgr2=0x7fae28068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 1 --2- 172.21.15.144:0/1305104590 >> v2:172.21.15.116:3300/0 conn(0x7fae281321a0 0x7fae28068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 10 monclient: _finish_hunting 0 2024-09-18T12:26:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 1 monclient: found mon.noname-c 2024-09-18T12:26:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.038+0000 7fae277fe640 1 -- 172.21.15.144:0/1305104590 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280a0200 con 0x7fae2809ec50 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae277fe640 1 -- 172.21.15.144:0/1305104590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809ec50 msgr2=0x7fae28131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae277fe640 1 --2- 172.21.15.144:0/1305104590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809ec50 0x7fae28131c60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae280719d0 tx=0x7fae180091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ba61337d9d88a62 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 1 -- 172.21.15.144:0/1305104590 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18032440 con 0x7fae2809ec50 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: dump: 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae2e034640 5 monclient: authenticate success, global_id 4489 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae2e034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 1 -- 172.21.15.144:0/1305104590 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae180325a0 con 0x7fae2809ec50 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae2e034640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae2e034640 4 set_mon_vals no callback set 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 1 -- 172.21.15.144:0/1305104590 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18032940 con 0x7fae2809ec50 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.040+0000 7fae26ffd640 10 monclient: dump: 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 10 monclient: shutdown 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 -- 172.21.15.144:0/1305104590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809ec50 msgr2=0x7fae28131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 --2- 172.21.15.144:0/1305104590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809ec50 0x7fae28131c60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae280719d0 tx=0x7fae180091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 -- 172.21.15.144:0/1305104590 shutdown_connections 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 --2- 172.21.15.144:0/1305104590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809f020 0x7fae28069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 --2- 172.21.15.144:0/1305104590 >> v2:172.21.15.116:3300/0 conn(0x7fae281321a0 0x7fae28068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 --2- 172.21.15.144:0/1305104590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809ec50 0x7fae28131c60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 -- 172.21.15.144:0/1305104590 >> 172.21.15.144:0/1305104590 conn(0x7fae280a5d40 msgr2=0x7fae280a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 -- 172.21.15.144:0/1305104590 shutdown_connections 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.041+0000 7fae2e034640 1 -- 172.21.15.144:0/1305104590 wait complete. 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient: build_initial_monmap 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient: monmap: 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 Processor -- start 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 -- start start 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient: init 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809ec50 0x7fae2809a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): picked mon.noname-c con 0x7fae2809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809f020 0x7fae28130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): picked mon.noname-b con 0x7fae2809f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae281321a0 0x7fae28141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): picked mon.noname-a con 0x7fae281321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2809c000 con 0x7fae281321a0 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2809de10 con 0x7fae2809f020 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2811df30 con 0x7fae2809ec50 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.042+0000 7fae2e034640 10 monclient(hunting): authenticate will time out at 840.611975s 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae277fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809ec50 msgr2=0x7fae2809a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae277fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809ec50 0x7fae2809a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809f020 0x7fae28130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809f020 0x7fae28130b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39720/0 (socket says 172.21.15.144:39720) 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 -- 172.21.15.144:0/2058042624 learned_addr learned my addr 172.21.15.144:0/2058042624 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): get_auth_request con 0x7fae2809f020 auth_method 0 2024-09-18T12:26:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 1 --2- 172.21.15.144:0/2058042624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae281321a0 0x7fae28141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 10 monclient(hunting): get_auth_request con 0x7fae281321a0 auth_method 0 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient(hunting): handle_auth_done global_id 4492 payload 824 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 -- 172.21.15.144:0/2058042624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809ec50 msgr2=0x7fae2809a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 --2- 172.21.15.144:0/2058042624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fae2809ec50 0x7fae2809a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 -- 172.21.15.144:0/2058042624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae281321a0 msgr2=0x7fae28141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 --2- 172.21.15.144:0/2058042624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae281321a0 0x7fae28141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 monclient: found mon.noname-b 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.043+0000 7fae1ffff640 1 -- 172.21.15.144:0/2058042624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae18009450 con 0x7fae2809f020 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1ffff640 1 -- 172.21.15.144:0/2058042624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809f020 msgr2=0x7fae28130b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:45.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1ffff640 1 --2- 172.21.15.144:0/2058042624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae2809f020 0x7fae28130b30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae1000d890 tx=0x7fae10008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 1 -- 172.21.15.144:0/2058042624 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000de20 con 0x7fae2809f020 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: dump: 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.063 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 1 -- 172.21.15.144:0/2058042624 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1000bae0 con 0x7fae2809f020 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae2e034640 5 monclient: authenticate success, global_id 4492 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae2e034640 10 monclient: _renew_subs 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 1 -- 172.21.15.144:0/2058042624 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000be80 con 0x7fae2809f020 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae2e034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae2e034640 1 -- 172.21.15.144:0/2058042624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2811dc50 con 0x7fae2809f020 2024-09-18T12:26:45.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae1f7fe640 10 monclient: dump: 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.065 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae267fc640 4 set_mon_vals no callback set 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae2e034640 10 monclient: _renew_subs 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae2e034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.045+0000 7fae2e034640 1 -- 172.21.15.144:0/2058042624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2811cd60 con 0x7fae2809f020 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae1f7fe640 1 -- 172.21.15.144:0/2058042624 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fae10014640 con 0x7fae2809f020 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae2e034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae2e034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae2e034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae2e034640 1 -- 172.21.15.144:0/2058042624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2811e0d0 con 0x7fae2809f020 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae1f7fe640 1 --2- 172.21.15.144:0/2058042624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fae08038490 0x7fae0803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae1f7fe640 1 -- 172.21.15.144:0/2058042624 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fae1004bd00 con 0x7fae2809f020 2024-09-18T12:26:45.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae277fe640 1 --2- 172.21.15.144:0/2058042624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fae08038490 0x7fae0803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.046+0000 7fae277fe640 10 monclient: get_auth_request con 0x7fae08038490 auth_method 0 2024-09-18T12:26:45.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.049+0000 7fae277fe640 1 --2- 172.21.15.144:0/2058042624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fae08038490 0x7fae0803a940 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fae280a2b00 tx=0x7fae18032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.051+0000 7fae1f7fe640 1 -- 172.21.15.144:0/2058042624 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fae10011570 con 0x7fae2809f020 2024-09-18T12:26:45.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.051+0000 7fae1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.051+0000 7fae1f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:45.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.206+0000 7fae2e034640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-18T12:26:45.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.206+0000 7fae2e034640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-18T12:26:45.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.206+0000 7fae2e034640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.206+0000 7fae2e034640 1 -- 172.21.15.144:0/2058042624 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) v1 -- 0x7fae2809b500 con 0x7fae2809f020 2024-09-18T12:26:45.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.207+0000 7fae1f7fe640 1 -- 172.21.15.144:0/2058042624 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v23) v1 ==== 146+0+0 (secure 0 0 0) 0x7fae1001dbb0 con 0x7fae2809f020 2024-09-18T12:26:45.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.207+0000 7fae1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-18T12:26:45.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.207+0000 7fae1f7fe640 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-18T12:26:45.210 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-18T12:26:45.212 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:26:45.212 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-18T12:26:45.213 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:26:45.213 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-18T12:26:45.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.286+0000 7f5b0c478640 10 monclient: get_monmap_and_config 2024-09-18T12:26:45.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.286+0000 7f5b0c478640 10 monclient: build_initial_monmap 2024-09-18T12:26:45.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.286+0000 7f5b0c478640 10 monclient: monmap: 2024-09-18T12:26:45.288 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:45.288 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.288 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:45.287663+0000 2024-09-18T12:26:45.288 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:45.287663+0000 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 1 Processor -- start 2024-09-18T12:26:45.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 1 -- start start 2024-09-18T12:26:45.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient: init 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04098c70 0x7f5b04099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient(hunting): picked mon.noname-b con 0x7f5b04098c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b04099580 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b04099c70 0x7f5b0409e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient(hunting): picked mon.noname-a con 0x7f5b04099c70 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:45.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0409f680 con 0x7f5b04099580 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0409f420 con 0x7f5b04099c70 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04068a20 con 0x7f5b04098c70 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.287+0000 7f5b0c478640 10 monclient(hunting): authenticate will time out at 840.856948s 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0a214640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04098c70 msgr2=0x7f5b04099040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0a214640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04098c70 0x7f5b04099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b04099c70 0x7f5b0409e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b09a13640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5b04099580 0x7f5b0409dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:37758/0 (socket says 172.21.15.144:37758) 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b09a13640 1 -- 172.21.15.144:0/1018544508 learned_addr learned my addr 172.21.15.144:0/1018544508 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:45.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): get_auth_request con 0x7f5b04099c70 auth_method 0 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient(hunting): handle_auth_done global_id 4563 payload 274 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 1 -- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04098c70 msgr2=0x7f5b04099040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 1 --2- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04098c70 0x7f5b04099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 1 -- 172.21.15.144:0/1018544508 >> v1:172.21.15.116:6789/0 conn(0x7f5b04099580 legacy=0x7f5b0409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient: _finish_hunting 0 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 1 monclient: found mon.noname-a 2024-09-18T12:26:45.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.288+0000 7f5b0aa15640 1 -- 172.21.15.144:0/1018544508 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b04096a30 con 0x7f5b04099c70 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b0aa15640 1 -- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 msgr2=0x7f5b0409e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b0aa15640 1 --2- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 0x7f5b0409e1e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009e60 tx=0x7f5afc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5748655b4949c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 1 -- 172.21.15.144:0/1018544508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc007840 con 0x7f5b04099c70 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 10 monclient: dump: 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.303 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 10 monclient: _finish_auth 0 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 1 -- 172.21.15.144:0/1018544508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc02d050 con 0x7f5b04099c70 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 1 -- 172.21.15.144:0/1018544508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc007bd0 con 0x7f5b04099c70 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b0c478640 5 monclient: authenticate success, global_id 4563 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b0c478640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b0c478640 4 set_mon_vals no callback set 2024-09-18T12:26:45.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.291+0000 7f5b09212640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b09212640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b09212640 10 monclient: dump: 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 10 monclient: shutdown 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 -- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 msgr2=0x7f5b0409e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 --2- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 0x7f5b0409e1e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009e60 tx=0x7f5afc009780 comp rx=0 tx=0).stop 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 -- 172.21.15.144:0/1018544508 shutdown_connections 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 --2- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 0x7f5b0409e1e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 --2- 172.21.15.144:0/1018544508 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04098c70 0x7f5b04099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 -- 172.21.15.144:0/1018544508 >> 172.21.15.144:0/1018544508 conn(0x7f5b04070e90 msgr2=0x7f5b040a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 -- 172.21.15.144:0/1018544508 shutdown_connections 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.292+0000 7f5b0c478640 1 -- 172.21.15.144:0/1018544508 wait complete. 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient: build_initial_monmap 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient: monmap: 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:45.306 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 Processor -- start 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 -- start start 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient: init 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:45.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b04098c70 0x7f5b04134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): picked mon.noname-b con 0x7f5b04098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04099580 0x7f5b0411e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): picked mon.noname-c con 0x7f5b04099580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 0x7f5b0411ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): picked mon.noname-a con 0x7f5b04099c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0406eba0 con 0x7f5b04099c70 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040a22b0 con 0x7f5b04098c70 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04120790 con 0x7f5b04099580 2024-09-18T12:26:45.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:45.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.293+0000 7f5b0c478640 10 monclient(hunting): authenticate will time out at 840.862961s 2024-09-18T12:26:45.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b09a13640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04099580 msgr2=0x7f5b0411e7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:45.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b09a13640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04099580 0x7f5b0411e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:45.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b04098c70 0x7f5b04134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b04098c70 0x7f5b04134c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39736/0 (socket says 172.21.15.144:39736) 2024-09-18T12:26:45.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 -- 172.21.15.144:0/4109407434 learned_addr learned my addr 172.21.15.144:0/4109407434 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:45.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): get_auth_request con 0x7f5b04098c70 auth_method 0 2024-09-18T12:26:45.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 1 --2- 172.21.15.144:0/4109407434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 0x7f5b0411ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 10 monclient(hunting): get_auth_request con 0x7f5b04099c70 auth_method 0 2024-09-18T12:26:45.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0aa15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 -- 172.21.15.144:0/4109407434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04099580 msgr2=0x7f5b0411e7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 --2- 172.21.15.144:0/4109407434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b04099580 0x7f5b0411e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 -- 172.21.15.144:0/4109407434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 msgr2=0x7f5b0411ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 --2- 172.21.15.144:0/4109407434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b04099c70 0x7f5b0411ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient: _finish_hunting 0 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 monclient: found mon.noname-b 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.294+0000 7f5b0a214640 1 -- 172.21.15.144:0/4109407434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc009a20 con 0x7f5b04098c70 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0a214640 1 -- 172.21.15.144:0/4109407434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b04098c70 msgr2=0x7f5b04134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0a214640 1 --2- 172.21.15.144:0/4109407434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b04098c70 0x7f5b04134c90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000b630 tx=0x7f5b000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 1 -- 172.21.15.144:0/4109407434 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00015070 con 0x7f5b04098c70 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: dump: 2024-09-18T12:26:45.312 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 1 -- 172.21.15.144:0/4109407434 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b00003d90 con 0x7f5b04098c70 2024-09-18T12:26:45.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0c478640 5 monclient: authenticate success, global_id 4495 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0c478640 10 monclient: _renew_subs 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0c478640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0c478640 1 -- 172.21.15.144:0/4109407434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b041223d0 con 0x7f5b04098c70 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 1 -- 172.21.15.144:0/4109407434 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00006380 con 0x7f5b04098c70 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5afa7fc640 10 monclient: dump: 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.314 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0c478640 10 monclient: _renew_subs 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b08a11640 4 set_mon_vals no callback set 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0c478640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.296+0000 7f5b0c478640 1 -- 172.21.15.144:0/4109407434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b04120e50 con 0x7f5b04098c70 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5afa7fc640 1 -- 172.21.15.144:0/4109407434 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5b00010440 con 0x7f5b04098c70 2024-09-18T12:26:45.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5afa7fc640 1 --2- 172.21.15.144:0/4109407434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ae8038440 0x7f5ae803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5afa7fc640 1 -- 172.21.15.144:0/4109407434 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f5b0004b170 con 0x7f5b04098c70 2024-09-18T12:26:45.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5b0c478640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5b0c478640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5b0c478640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5b0c478640 1 -- 172.21.15.144:0/4109407434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b040008d0 con 0x7f5b04098c70 2024-09-18T12:26:45.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5b09a13640 1 --2- 172.21.15.144:0/4109407434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ae8038440 0x7f5ae803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.297+0000 7f5b09a13640 10 monclient: get_auth_request con 0x7f5ae8038440 auth_method 0 2024-09-18T12:26:45.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.300+0000 7f5b09a13640 1 --2- 172.21.15.144:0/4109407434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ae8038440 0x7f5ae803a8f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5af40047d0 tx=0x7f5af4007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.302+0000 7f5afa7fc640 1 -- 172.21.15.144:0/4109407434 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5b00006770 con 0x7f5b04098c70 2024-09-18T12:26:45.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.302+0000 7f5afa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.302+0000 7f5afa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:45.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.455+0000 7f5b0c478640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-18T12:26:45.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.455+0000 7f5b0c478640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-18T12:26:45.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.455+0000 7f5b0c478640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.455+0000 7f5b0c478640 1 -- 172.21.15.144:0/4109407434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) v1 -- 0x7f5b04064a20 con 0x7f5b04098c70 2024-09-18T12:26:45.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.456+0000 7f5afa7fc640 1 -- 172.21.15.144:0/4109407434 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v23) v1 ==== 149+0+0 (secure 0 0 0) 0x7f5b00018a70 con 0x7f5b04098c70 2024-09-18T12:26:45.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.456+0000 7f5afa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-18T12:26:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.456+0000 7f5afa7fc640 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-18T12:26:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-18T12:26:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:26:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-18T12:26:45.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.534+0000 7f79ecd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:45.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.534+0000 7f79ecd9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.534+0000 7f79ecd9d640 10 monclient: monmap: 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:45.535403+0000 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:45.535403+0000 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:45.536 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:45.537 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:45.537 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:45.537 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:45.537 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 1 Processor -- start 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 1 -- start start 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient: init 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79e8069a30 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79e80983e0 0x7f79e8098840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f79e80983e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8136e90 0x7f79e8139270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f79e8136e90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e809cd00 con 0x7f79e8069a30 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e81327f0 con 0x7f79e80983e0 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8069e00 con 0x7f79e8136e90 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79ecd9d640 10 monclient(hunting): authenticate will time out at 841.104739s 2024-09-18T12:26:45.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e6d9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8136e90 msgr2=0x7f79e8139270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:45.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e5d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79e80983e0 0x7f79e8098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8136e90 0x7f79e8139270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:45.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e5d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79e80983e0 0x7f79e8098840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39744/0 (socket says 172.21.15.144:39744) 2024-09-18T12:26:45.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e5d9b640 1 -- 172.21.15.144:0/960557731 learned_addr learned my addr 172.21.15.144:0/960557731 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:45.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e5d9b640 10 monclient(hunting): get_auth_request con 0x7f79e80983e0 auth_method 0 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.535+0000 7f79e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 10 monclient(hunting): handle_auth_done global_id 4498 payload 274 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 1 -- 172.21.15.144:0/960557731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8136e90 msgr2=0x7f79e8139270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 1 --2- 172.21.15.144:0/960557731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8136e90 0x7f79e8139270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 1 -- 172.21.15.144:0/960557731 >> v1:172.21.15.116:6789/0 conn(0x7f79e8069a30 legacy=0x7f79e81320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 1 monclient: found mon.noname-c 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.536+0000 7f79e5d9b640 1 -- 172.21.15.144:0/960557731 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e809d140 con 0x7f79e80983e0 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.538+0000 7f79e5d9b640 1 -- 172.21.15.144:0/960557731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80983e0 msgr2=0x7f79e8098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e5d9b640 1 --2- 172.21.15.144:0/960557731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80983e0 0x7f79e8098840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009970 tx=0x7f79d80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c6d826d81678407 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 1 -- 172.21.15.144:0/960557731 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8032440 con 0x7f79e80983e0 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: dump: 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.553 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: _finish_auth 0 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 5 monclient: authenticate success, global_id 4498 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 1 -- 172.21.15.144:0/960557731 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d80325a0 con 0x7f79e80983e0 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 4 set_mon_vals no callback set 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 1 -- 172.21.15.144:0/960557731 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8032940 con 0x7f79e80983e0 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79e559a640 10 monclient: dump: 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 10 monclient: shutdown 2024-09-18T12:26:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 -- 172.21.15.144:0/960557731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80983e0 msgr2=0x7f79e8098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/960557731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80983e0 0x7f79e8098840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009970 tx=0x7f79d80091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 -- 172.21.15.144:0/960557731 shutdown_connections 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/960557731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8136e90 0x7f79e8139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/960557731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80983e0 0x7f79e8098840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 -- 172.21.15.144:0/960557731 >> 172.21.15.144:0/960557731 conn(0x7f79e809e9f0 msgr2=0x7f79e809edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 -- 172.21.15.144:0/960557731 shutdown_connections 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.539+0000 7f79ecd9d640 1 -- 172.21.15.144:0/960557731 wait complete. 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.540+0000 7f79ecd9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.540+0000 7f79ecd9d640 10 monclient: monmap: 2024-09-18T12:26:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.540+0000 7f79ecd9d640 1 Processor -- start 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.540+0000 7f79ecd9d640 1 -- start start 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.540+0000 7f79ecd9d640 10 monclient: init 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8069a30 0x7f79e81413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f79e8069a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80983e0 0x7f79e812d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f79e80983e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 0x7f79e8134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f79e8136e90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81422a0 con 0x7f79e80983e0 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8119fe0 con 0x7f79e8136e90 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8119d70 con 0x7f79e8069a30 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79ecd9d640 10 monclient(hunting): authenticate will time out at 841.110148s 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 0x7f79e8134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e659c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8069a30 msgr2=0x7f79e81413e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8069a30 0x7f79e81413e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 0x7f79e8134ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39750/0 (socket says 172.21.15.144:39750) 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 -- 172.21.15.144:0/613220672 learned_addr learned my addr 172.21.15.144:0/613220672 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): get_auth_request con 0x7f79e8136e90 auth_method 0 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80983e0 0x7f79e812d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 10 monclient(hunting): get_auth_request con 0x7f79e80983e0 auth_method 0 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient(hunting): handle_auth_done global_id 4501 payload 824 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 -- 172.21.15.144:0/613220672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8069a30 msgr2=0x7f79e81413e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8069a30 0x7f79e81413e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 -- 172.21.15.144:0/613220672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80983e0 msgr2=0x7f79e812d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80983e0 0x7f79e812d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 monclient: found mon.noname-b 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.541+0000 7f79e6d9d640 1 -- 172.21.15.144:0/613220672 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8009450 con 0x7f79e8136e90 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79e6d9d640 1 -- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 msgr2=0x7f79e8134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79e6d9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 0x7f79e8134ab0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f79e000bb10 tx=0x7f79e00097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000cbc0 con 0x7f79e8136e90 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: dump: 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e000cd40 con 0x7f79e8136e90 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79ecd9d640 5 monclient: authenticate success, global_id 4501 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79ecd9d640 10 monclient: _renew_subs 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79ecd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e80a08f0 con 0x7f79e8136e90 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0006150 con 0x7f79e8136e90 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79de7fc640 10 monclient: dump: 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:45.565 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79e4d99640 4 set_mon_vals no callback set 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79ecd9d640 10 monclient: _renew_subs 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79ecd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.543+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e8141ac0 con 0x7f79e8136e90 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.544+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f79e0012020 con 0x7f79e8136e90 2024-09-18T12:26:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.544+0000 7f79ecd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.544+0000 7f79ecd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.544+0000 7f79ecd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.544+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e804bf60 con 0x7f79e8136e90 2024-09-18T12:26:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.545+0000 7f79de7fc640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79c8038440 0x7f79c803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.545+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f79e004b510 con 0x7f79e8136e90 2024-09-18T12:26:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.545+0000 7f79e659c640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79c8038440 0x7f79c803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.545+0000 7f79e659c640 10 monclient: get_auth_request con 0x7f79c8038440 auth_method 0 2024-09-18T12:26:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.547+0000 7f79e659c640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79c8038440 0x7f79c803a8f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f79d0007fd0 tx=0x7f79d0007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.547+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:26:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.549+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f79e001dae0 con 0x7f79e8136e90 2024-09-18T12:26:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.549+0000 7f79de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.549+0000 7f79de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:45.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.705+0000 7f79ecd9d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T12:26:45.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.705+0000 7f79ecd9d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T12:26:45.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.705+0000 7f79ecd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:45.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:45.705+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f79e81329c0 con 0x7f79e8136e90 2024-09-18T12:26:46.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:46.101+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v24) v1 ==== 143+0+0 (secure 0 0 0) 0x7f79e001d300 con 0x7f79e8136e90 2024-09-18T12:26:46.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:46.101+0000 7f79de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T12:26:46.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:46.101+0000 7f79de7fc640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-18T12:26:46.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:46.192+0000 7f79ecd9d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T12:26:46.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:46.192+0000 7f79ecd9d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T12:26:46.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:46.192+0000 7f79ecd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:46.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:46.192+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f79e80a0890 con 0x7f79e8136e90 2024-09-18T12:26:47.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.104+0000 7f79de7fc640 1 -- 172.21.15.144:0/613220672 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v25) v1 ==== 143+0+0 (secure 0 0 0) 0x7f79e001d4e0 con 0x7f79e8136e90 2024-09-18T12:26:47.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.104+0000 7f79de7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T12:26:47.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.104+0000 7f79de7fc640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-18T12:26:47.106 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 3 pg_num_min to 0 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79c8038440 msgr2=0x7f79c803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79c8038440 0x7f79c803a8f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f79d0007fd0 tx=0x7f79d0007a70 comp rx=0 tx=0).stop 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 10 monclient: shutdown 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 msgr2=0x7f79e8134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 0x7f79e8134ab0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f79e000bb10 tx=0x7f79e00097d0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 shutdown_connections 2024-09-18T12:26:47.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79c8038440 0x7f79c803a8f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e8136e90 0x7f79e8134ab0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80983e0 0x7f79e812d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 --2- 172.21.15.144:0/613220672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8069a30 0x7f79e81413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 >> 172.21.15.144:0/613220672 conn(0x7f79e809e9f0 msgr2=0x7f79e80971a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:47.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 shutdown_connections 2024-09-18T12:26:47.110 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.107+0000 7f79ecd9d640 1 -- 172.21.15.144:0/613220672 wait complete. 2024-09-18T12:26:47.119 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-18T12:26:47.119 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:26:47.120 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.196+0000 7f8dc4d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.196+0000 7f8dc4d1d640 10 monclient: build_initial_monmap 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.197+0000 7f8dc4d1d640 10 monclient: monmap: 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:47.197899+0000 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:47.197899+0000 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:47.199 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.197+0000 7f8dc4d1d640 1 Processor -- start 2024-09-18T12:26:47.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.197+0000 7f8dc4d1d640 1 -- start start 2024-09-18T12:26:47.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.197+0000 7f8dc4d1d640 10 monclient: init 2024-09-18T12:26:47.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8dc00a0ad0 0x7f8dc00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc00a0ad0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8dc009ee60 0x7f8dc009f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc009ee60 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dc0131ad0 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc006a5b0 con 0x7f8dc0131ad0 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc009f7a0 con 0x7f8dc009ee60 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc00a0ea0 con 0x7f8dc00a0ad0 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dc4d1d640 10 monclient(hunting): authenticate will time out at 842.767136s 2024-09-18T12:26:47.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8dc00a0ad0 0x7f8dc00691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8dc00a0ad0 0x7f8dc00691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57678/0 (socket says 172.21.15.144:57678) 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 -- 172.21.15.144:0/1026509546 learned_addr learned my addr 172.21.15.144:0/1026509546 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): get_auth_request con 0x7f8dc00a0ad0 auth_method 0 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 1 --2- 172.21.15.144:0/1026509546 >> v2:172.21.15.116:3300/0 conn(0x7f8dc009ee60 0x7f8dc009f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 10 monclient(hunting): get_auth_request con 0x7f8dc009ee60 auth_method 0 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8db5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_done global_id 4504 payload 274 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 -- 172.21.15.144:0/1026509546 >> v2:172.21.15.116:3300/0 conn(0x7f8dc009ee60 msgr2=0x7f8dc009f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 --2- 172.21.15.144:0/1026509546 >> v2:172.21.15.116:3300/0 conn(0x7f8dc009ee60 0x7f8dc009f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 -- 172.21.15.144:0/1026509546 >> v1:172.21.15.116:6789/0 conn(0x7f8dc0131ad0 legacy=0x7f8dc0131ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient: _finish_hunting 0 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 monclient: found mon.noname-c 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.198+0000 7f8dbe59c640 1 -- 172.21.15.144:0/1026509546 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00a0460 con 0x7f8dc00a0ad0 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbe59c640 1 -- 172.21.15.144:0/1026509546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc00a0ad0 msgr2=0x7f8dc00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbe59c640 1 --2- 172.21.15.144:0/1026509546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc00a0ad0 0x7f8dc00691d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8dc006abc0 tx=0x7f8db00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=631eb75cdeb70427 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 1 -- 172.21.15.144:0/1026509546 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0032440 con 0x7f8dc00a0ad0 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: dump: 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.211 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: _finish_auth 0 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 1 -- 172.21.15.144:0/1026509546 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db00325a0 con 0x7f8dc00a0ad0 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dc4d1d640 5 monclient: authenticate success, global_id 4504 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dc4d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dc4d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:47.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dc4d1d640 4 set_mon_vals no callback set 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 1 -- 172.21.15.144:0/1026509546 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0032940 con 0x7f8dc00a0ad0 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.201+0000 7f8dbdd9b640 10 monclient: dump: 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.213 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 10 monclient: shutdown 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/1026509546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc00a0ad0 msgr2=0x7f8dc00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 --2- 172.21.15.144:0/1026509546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc00a0ad0 0x7f8dc00691d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8dc006abc0 tx=0x7f8db00091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/1026509546 shutdown_connections 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 --2- 172.21.15.144:0/1026509546 >> v2:172.21.15.116:3300/0 conn(0x7f8dc009ee60 0x7f8dc009f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 --2- 172.21.15.144:0/1026509546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc00a0ad0 0x7f8dc00691d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/1026509546 >> 172.21.15.144:0/1026509546 conn(0x7f8dc0071370 msgr2=0x7f8dc00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/1026509546 shutdown_connections 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/1026509546 wait complete. 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 10 monclient: build_initial_monmap 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.202+0000 7f8dc4d1d640 10 monclient: monmap: 2024-09-18T12:26:47.214 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 Processor -- start 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 -- start start 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient: init 2024-09-18T12:26:47.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc009ee60 0x7f8dc009a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc009ee60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc00a0ad0 0x7f8dc0130b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f8dc00a0ad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dc0131ad0 0x7f8dc011b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f8dc0131ad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00a11e0 con 0x7f8dc00a0ad0 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc011df90 con 0x7f8dc009ee60 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0141e70 con 0x7f8dc0131ad0 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dc4d1d640 10 monclient(hunting): authenticate will time out at 842.772737s 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc009ee60 0x7f8dc009a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc009ee60 0x7f8dc009a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57680/0 (socket says 172.21.15.144:57680) 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 1 -- 172.21.15.144:0/3667849138 learned_addr learned my addr 172.21.15.144:0/3667849138 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 10 monclient(hunting): get_auth_request con 0x7f8dc009ee60 auth_method 0 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbed9d640 1 -- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dc0131ad0 msgr2=0x7f8dc011b900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:47.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8dbed9d640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dc0131ad0 0x7f8dc011b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8db5d9b640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc00a0ad0 0x7f8dc0130b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8db5d9b640 10 monclient(hunting): get_auth_request con 0x7f8dc00a0ad0 auth_method 0 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8db5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8db5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.203+0000 7f8db5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 10 monclient(hunting): handle_auth_done global_id 4507 payload 824 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 1 -- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dc0131ad0 msgr2=0x7f8dc011b900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8dc0131ad0 0x7f8dc011b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 1 -- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc00a0ad0 msgr2=0x7f8dc0130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc00a0ad0 0x7f8dc0130b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 10 monclient: _finish_hunting 0 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 1 monclient: found mon.noname-b 2024-09-18T12:26:47.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8dbe59c640 1 -- 172.21.15.144:0/3667849138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0009450 con 0x7f8dc009ee60 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.204+0000 7f8db5d9b640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc00a0ad0 0x7f8dc0130b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dbe59c640 1 -- 172.21.15.144:0/3667849138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc009ee60 msgr2=0x7f8dc009a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dbe59c640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8dc009ee60 0x7f8dc009a8e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8db0035040 tx=0x7f8db0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 1 -- 172.21.15.144:0/3667849138 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db00319c0 con 0x7f8dc009ee60 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: dump: 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.219 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 1 -- 172.21.15.144:0/3667849138 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db0031b40 con 0x7f8dc009ee60 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dc4d1d640 5 monclient: authenticate success, global_id 4507 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dc4d1d640 10 monclient: _renew_subs 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 1 -- 172.21.15.144:0/3667849138 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db003d440 con 0x7f8dc009ee60 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dc4d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/3667849138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc009b330 con 0x7f8dc009ee60 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dbd59a640 4 set_mon_vals no callback set 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8db6ffd640 10 monclient: dump: 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.221 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dc4d1d640 10 monclient: _renew_subs 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dc4d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.206+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/3667849138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc011dd60 con 0x7f8dc009ee60 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8db6ffd640 1 -- 172.21.15.144:0/3667849138 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8db0039040 con 0x7f8dc009ee60 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8dc4d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8dc4d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8dc4d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/3667849138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc00008d0 con 0x7f8dc009ee60 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8db6ffd640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8da0038490 0x7f8da003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8db6ffd640 1 -- 172.21.15.144:0/3667849138 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f8db006b5b0 con 0x7f8dc009ee60 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8db5d9b640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8da0038490 0x7f8da003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.207+0000 7f8db5d9b640 10 monclient: get_auth_request con 0x7f8da0038490 auth_method 0 2024-09-18T12:26:47.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.210+0000 7f8db5d9b640 1 --2- 172.21.15.144:0/3667849138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8da0038490 0x7f8da003a940 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0131aa0 tx=0x7f8da8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:47.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.211+0000 7f8db6ffd640 1 -- 172.21.15.144:0/3667849138 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8db0030b70 con 0x7f8dc009ee60 2024-09-18T12:26:47.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.211+0000 7f8db6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:47.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.211+0000 7f8db6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:47.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.366+0000 7f8dc4d1d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:47.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.366+0000 7f8dc4d1d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:47.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.366+0000 7f8dc4d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.366+0000 7f8dc4d1d640 1 -- 172.21.15.144:0/3667849138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7f8dc0064a20 con 0x7f8dc009ee60 2024-09-18T12:26:47.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.366+0000 7f8db6ffd640 1 -- 172.21.15.144:0/3667849138 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v25) v1 ==== 158+0+0 (secure 0 0 0) 0x7f8db0034030 con 0x7f8dc009ee60 2024-09-18T12:26:47.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.366+0000 7f8db6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T12:26:47.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.366+0000 7f8db6ffd640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-18T12:26:47.368 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-18T12:26:47.371 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:26:47.371 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-18T12:26:47.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.445+0000 7f07daf9d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:47.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.445+0000 7f07daf9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:47.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.445+0000 7f07daf9d640 10 monclient: monmap: 2024-09-18T12:26:47.446 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:47.446 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.446 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:47.446080+0000 2024-09-18T12:26:47.446 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:47.446080+0000 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:47.447 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.445+0000 7f07daf9d640 1 Processor -- start 2024-09-18T12:26:47.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.445+0000 7f07daf9d640 1 -- start start 2024-09-18T12:26:47.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.445+0000 7f07daf9d640 10 monclient: init 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07d4131bd0 0x7f07d4131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f07d4131bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07d409ebe0 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07d409efe0 0x7f07d4069510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f07d409efe0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d40a06d0 con 0x7f07d409ebe0 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d4071780 con 0x7f07d409efe0 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d40a0010 con 0x7f07d4131bd0 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07daf9d640 10 monclient(hunting): authenticate will time out at 843.015449s 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07d4131bd0 0x7f07d4131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07d4131bd0 0x7f07d4131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57686/0 (socket says 172.21.15.144:57686) 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 -- 172.21.15.144:0/2611912391 learned_addr learned my addr 172.21.15.144:0/2611912391 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): get_auth_request con 0x7f07d4131bd0 auth_method 0 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d953a640 1 --2- 172.21.15.144:0/2611912391 >> v2:172.21.15.116:3300/0 conn(0x7f07d409efe0 0x7f07d4069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d953a640 10 monclient(hunting): get_auth_request con 0x7f07d409efe0 auth_method 0 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d953a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d953a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d953a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient(hunting): handle_auth_done global_id 4510 payload 274 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 -- 172.21.15.144:0/2611912391 >> v2:172.21.15.116:3300/0 conn(0x7f07d409efe0 msgr2=0x7f07d4069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 --2- 172.21.15.144:0/2611912391 >> v2:172.21.15.116:3300/0 conn(0x7f07d409efe0 0x7f07d4069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 -- 172.21.15.144:0/2611912391 >> v1:172.21.15.116:6789/0 conn(0x7f07d409ebe0 legacy=0x7f07d4068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient: _finish_hunting 0 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 monclient: found mon.noname-c 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d8d39640 1 -- 172.21.15.144:0/2611912391 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d40a1910 con 0x7f07d4131bd0 2024-09-18T12:26:47.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.446+0000 7f07d953a640 1 --2- 172.21.15.144:0/2611912391 >> v2:172.21.15.116:3300/0 conn(0x7f07d409efe0 0x7f07d4069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:26:47.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d8d39640 1 -- 172.21.15.144:0/2611912391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d4131bd0 msgr2=0x7f07d4131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:47.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d8d39640 1 --2- 172.21.15.144:0/2611912391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d4131bd0 0x7f07d4131fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07d4068370 tx=0x7f07c80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7828d3c1a02b3f94 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:47.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 1 -- 172.21.15.144:0/2611912391 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8032440 con 0x7f07d4131bd0 2024-09-18T12:26:47.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:47.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: dump: 2024-09-18T12:26:47.459 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.460 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07daf9d640 5 monclient: authenticate success, global_id 4510 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 1 -- 172.21.15.144:0/2611912391 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c80325a0 con 0x7f07d4131bd0 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07daf9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07daf9d640 4 set_mon_vals no callback set 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 1 -- 172.21.15.144:0/2611912391 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8032980 con 0x7f07d4131bd0 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:47.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.449+0000 7f07d37fe640 10 monclient: dump: 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 10 monclient: shutdown 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 -- 172.21.15.144:0/2611912391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d4131bd0 msgr2=0x7f07d4131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 --2- 172.21.15.144:0/2611912391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d4131bd0 0x7f07d4131fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07d4068370 tx=0x7f07c80091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 -- 172.21.15.144:0/2611912391 shutdown_connections 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 --2- 172.21.15.144:0/2611912391 >> v2:172.21.15.116:3300/0 conn(0x7f07d409efe0 0x7f07d4069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 --2- 172.21.15.144:0/2611912391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d4131bd0 0x7f07d4131fa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 -- 172.21.15.144:0/2611912391 >> 172.21.15.144:0/2611912391 conn(0x7f07d40713b0 msgr2=0x7f07d40a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 -- 172.21.15.144:0/2611912391 shutdown_connections 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.450+0000 7f07daf9d640 1 -- 172.21.15.144:0/2611912391 wait complete. 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 10 monclient: monmap: 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 1 Processor -- start 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 1 -- start start 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 10 monclient: init 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07d409ebe0 0x7f07d409a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.451+0000 7f07daf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f07d409ebe0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 0x7f07d411bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f07d409efe0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d4131bd0 0x7f07d4130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f07d4131bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d411c8d0 con 0x7f07d4131bd0 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d411e110 con 0x7f07d409efe0 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d411dbc0 con 0x7f07d409ebe0 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07daf9d640 10 monclient(hunting): authenticate will time out at 843.021064s 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d8d39640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07d409ebe0 msgr2=0x7f07d409a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:47.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 0x7f07d411bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d8d39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07d409ebe0 0x7f07d409a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 0x7f07d411bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57696/0 (socket says 172.21.15.144:57696) 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 -- 172.21.15.144:0/3129994675 learned_addr learned my addr 172.21.15.144:0/3129994675 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): get_auth_request con 0x7f07d409efe0 auth_method 0 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:47.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d4131bd0 0x7f07d4130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 10 monclient(hunting): get_auth_request con 0x7f07d4131bd0 auth_method 0 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d953a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient(hunting): handle_auth_done global_id 4513 payload 824 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 -- 172.21.15.144:0/3129994675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07d409ebe0 msgr2=0x7f07d409a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07d409ebe0 0x7f07d409a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 -- 172.21.15.144:0/3129994675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d4131bd0 msgr2=0x7f07d4130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d4131bd0 0x7f07d4130a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:47.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 monclient: found mon.noname-b 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.452+0000 7f07d3fff640 1 -- 172.21.15.144:0/3129994675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c8009450 con 0x7f07d409efe0 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d3fff640 1 -- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 msgr2=0x7f07d411bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d3fff640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 0x7f07d411bbc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f07c000b4f0 tx=0x7f07c00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000f720 con 0x7f07d409efe0 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: dump: 2024-09-18T12:26:47.468 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:26:47.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000f8a0 con 0x7f07d409efe0 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07daf9d640 5 monclient: authenticate success, global_id 4513 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07daf9d640 10 monclient: _renew_subs 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07daf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d411ce60 con 0x7f07d409efe0 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000fc60 con 0x7f07d409efe0 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d0ff9640 10 monclient: dump: 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:47.470 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07daf9d640 10 monclient: _renew_subs 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07d2ffd640 4 set_mon_vals no callback set 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07daf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.454+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d4130f70 con 0x7f07d409efe0 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.455+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f07c0011610 con 0x7f07d409efe0 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.455+0000 7f07daf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:47.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.455+0000 7f07daf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:47.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.455+0000 7f07daf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.455+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d411cc60 con 0x7f07d409efe0 2024-09-18T12:26:47.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.455+0000 7f07d0ff9640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b8038440 0x7f07b803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:47.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.456+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f07c004b2f0 con 0x7f07d409efe0 2024-09-18T12:26:47.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.456+0000 7f07d8d39640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b8038440 0x7f07b803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:47.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.456+0000 7f07d8d39640 10 monclient: get_auth_request con 0x7f07b8038440 auth_method 0 2024-09-18T12:26:47.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.458+0000 7f07d8d39640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b8038440 0x7f07b803a8f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f07c8034000 tx=0x7f07c8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:47.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.460+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f07c00118c0 con 0x7f07d409efe0 2024-09-18T12:26:47.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.460+0000 7f07d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:47.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.460+0000 7f07d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:47.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.615+0000 7f07daf9d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T12:26:47.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.615+0000 7f07daf9d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T12:26:47.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.615+0000 7f07daf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:47.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:47.615+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f07d411df70 con 0x7f07d409efe0 2024-09-18T12:26:48.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:48.522+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v26) v1 ==== 143+0+0 (secure 0 0 0) 0x7f07d411df70 con 0x7f07d409efe0 2024-09-18T12:26:48.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:48.522+0000 7f07d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T12:26:48.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:48.522+0000 7f07d0ff9640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-18T12:26:48.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:48.615+0000 7f07daf9d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T12:26:48.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:48.615+0000 7f07daf9d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T12:26:48.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:48.615+0000 7f07daf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:48.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:48.615+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f07d411df70 con 0x7f07d409efe0 2024-09-18T12:26:49.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.612+0000 7f07d0ff9640 1 -- 172.21.15.144:0/3129994675 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v27) v1 ==== 143+0+0 (secure 0 0 0) 0x7f07d411df70 con 0x7f07d409efe0 2024-09-18T12:26:49.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.612+0000 7f07d0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T12:26:49.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.612+0000 7f07d0ff9640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-18T12:26:49.614 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 3 pg_num_max to 0 2024-09-18T12:26:49.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b8038440 msgr2=0x7f07b803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:49.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b8038440 0x7f07b803a8f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f07c8034000 tx=0x7f07c8032000 comp rx=0 tx=0).stop 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 10 monclient: shutdown 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 msgr2=0x7f07d411bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 0x7f07d411bbc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f07c000b4f0 tx=0x7f07c00090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 shutdown_connections 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b8038440 0x7f07b803a8f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07d4131bd0 0x7f07d4130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07d409efe0 0x7f07d411bbc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 --2- 172.21.15.144:0/3129994675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07d409ebe0 0x7f07d409a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 >> 172.21.15.144:0/3129994675 conn(0x7f07d40713b0 msgr2=0x7f07d409cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:49.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 shutdown_connections 2024-09-18T12:26:49.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.614+0000 7f07daf9d640 1 -- 172.21.15.144:0/3129994675 wait complete. 2024-09-18T12:26:49.626 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-18T12:26:49.626 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:26:49.626 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.701+0000 7fa0a739d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.701+0000 7fa0a739d640 10 monclient: build_initial_monmap 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.701+0000 7fa0a739d640 10 monclient: monmap: 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:49.702860+0000 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:49.702860+0000 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:49.703 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.704 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:49.704 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:49.704 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:49.704 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:49.704 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:49.704 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.702+0000 7fa0a739d640 1 Processor -- start 2024-09-18T12:26:49.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.702+0000 7fa0a739d640 1 -- start start 2024-09-18T12:26:49.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.702+0000 7fa0a739d640 10 monclient: init 2024-09-18T12:26:49.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa0a00a0ad0 0x7fa0a00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0a00a0ad0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0a009ee60 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 0x7fa0a01321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0a009f260 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a006a5b0 con 0x7fa0a009ee60 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a00a0ea0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00a1bb0 con 0x7fa0a009f260 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:49.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a739d640 10 monclient(hunting): authenticate will time out at 845.272223s 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa0a00a0ad0 0x7fa0a00691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a593a640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 msgr2=0x7fa0a01321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa0a00a0ad0 0x7fa0a00691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57708/0 (socket says 172.21.15.144:57708) 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 -- 172.21.15.144:0/1557857498 learned_addr learned my addr 172.21.15.144:0/1557857498 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a593a640 1 --2- 172.21.15.144:0/1557857498 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 0x7fa0a01321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): get_auth_request con 0x7fa0a00a0ad0 auth_method 0 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:49.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient(hunting): handle_auth_done global_id 4516 payload 274 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 -- 172.21.15.144:0/1557857498 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 msgr2=0x7fa0a01321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 --2- 172.21.15.144:0/1557857498 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 0x7fa0a01321e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 -- 172.21.15.144:0/1557857498 >> v1:172.21.15.116:6789/0 conn(0x7fa0a009ee60 legacy=0x7fa0a0131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient: _finish_hunting 0 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 monclient: found mon.noname-c 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.703+0000 7fa0a5139640 1 -- 172.21.15.144:0/1557857498 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a00a0460 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa0a5139640 1 -- 172.21.15.144:0/1557857498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 msgr2=0x7fa0a00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:49.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa0a5139640 1 --2- 172.21.15.144:0/1557857498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 0x7fa0a00691d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa090009f20 tx=0x7fa090009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a66ce983d722b81 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:49.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 1 -- 172.21.15.144:0/1557857498 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090007610 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: dump: 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: _finish_auth 0 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa0a739d640 5 monclient: authenticate success, global_id 4516 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 1 -- 172.21.15.144:0/1557857498 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09002d050 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa0a739d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa0a739d640 4 set_mon_vals no callback set 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 1 -- 172.21.15.144:0/1557857498 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900079a0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.706+0000 7fa09ffff640 10 monclient: dump: 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.717 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 10 monclient: shutdown 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 -- 172.21.15.144:0/1557857498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 msgr2=0x7fa0a00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 --2- 172.21.15.144:0/1557857498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 0x7fa0a00691d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa090009f20 tx=0x7fa090009840 comp rx=0 tx=0).stop 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 -- 172.21.15.144:0/1557857498 shutdown_connections 2024-09-18T12:26:49.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 --2- 172.21.15.144:0/1557857498 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 0x7fa0a01321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 --2- 172.21.15.144:0/1557857498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 0x7fa0a00691d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 -- 172.21.15.144:0/1557857498 >> 172.21.15.144:0/1557857498 conn(0x7fa0a0071370 msgr2=0x7fa0a00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 -- 172.21.15.144:0/1557857498 shutdown_connections 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 -- 172.21.15.144:0/1557857498 wait complete. 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 10 monclient: build_initial_monmap 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 10 monclient: monmap: 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:49.719 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 Processor -- start 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 1 -- start start 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.707+0000 7fa0a739d640 10 monclient: init 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0a009ee60 0x7fa0a0131ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0a009ee60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 0x7fa0a011b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0a009f260 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 0x7fa0a011be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0a00a0ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a01366c0 con 0x7fa0a009ee60 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a011eec0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a0134cd0 con 0x7fa0a009f260 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a739d640 10 monclient(hunting): authenticate will time out at 845.277576s 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 0x7fa0a011be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:49.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 0x7fa0a011be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57718/0 (socket says 172.21.15.144:57718) 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 -- 172.21.15.144:0/904092939 learned_addr learned my addr 172.21.15.144:0/904092939 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): get_auth_request con 0x7fa0a00a0ad0 auth_method 0 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a4938640 1 -- 172.21.15.144:0/904092939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 msgr2=0x7fa0a011b900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a4938640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 0x7fa0a011b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a5139640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0a009ee60 0x7fa0a0131ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a5139640 10 monclient(hunting): get_auth_request con 0x7fa0a009ee60 auth_method 0 2024-09-18T12:26:49.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a5139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a5139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a5139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient(hunting): handle_auth_done global_id 4519 payload 824 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 -- 172.21.15.144:0/904092939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 msgr2=0x7fa0a011b900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0a009f260 0x7fa0a011b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 -- 172.21.15.144:0/904092939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0a009ee60 msgr2=0x7fa0a0131ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0a009ee60 0x7fa0a0131ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient: _finish_hunting 0 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 monclient: found mon.noname-b 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.708+0000 7fa0a593a640 1 -- 172.21.15.144:0/904092939 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa090009ae0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a593a640 1 -- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 msgr2=0x7fa0a011be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a593a640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0a00a0ad0 0x7fa0a011be40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa09800b510 tx=0x7fa0980090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 1 -- 172.21.15.144:0/904092939 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098003d40 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: dump: 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.724 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 1 -- 172.21.15.144:0/904092939 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098006150 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a739d640 5 monclient: authenticate success, global_id 4519 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a739d640 10 monclient: _renew_subs 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a739d640 1 -- 172.21.15.144:0/904092939 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a011ebe0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 1 -- 172.21.15.144:0/904092939 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098003ea0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09d7fa640 10 monclient: dump: 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a739d640 10 monclient: _renew_subs 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa0a739d640 1 -- 172.21.15.144:0/904092939 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a0141ff0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.711+0000 7fa09f7fe640 4 set_mon_vals no callback set 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa09d7fa640 1 -- 172.21.15.144:0/904092939 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa09801b3b0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a739d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a739d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a739d640 1 -- 172.21.15.144:0/904092939 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a011dce0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa09d7fa640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa08c038440 0x7fa08c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa09d7fa640 1 -- 172.21.15.144:0/904092939 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4693+0+0 (secure 0 0 0) 0x7fa09804ad00 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a5139640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa08c038440 0x7fa08c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a5139640 10 monclient: get_auth_request con 0x7fa08c038440 auth_method 0 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a5139640 1 -- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa08c038440 msgr2=0x7fa08c03a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:26:49.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a5139640 1 -- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa08c038440 msgr2=0x7fa08c03a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:49.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a5139640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa08c038440 0x7fa08c03a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:49.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.712+0000 7fa0a5139640 1 --2- 172.21.15.144:0/904092939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa08c038440 0x7fa08c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:49.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.716+0000 7fa09d7fa640 1 -- 172.21.15.144:0/904092939 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa098018830 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.716+0000 7fa09d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:49.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.716+0000 7fa09d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.869+0000 7fa0a739d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.869+0000 7fa0a739d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.869+0000 7fa0a739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.869+0000 7fa0a739d640 1 -- 172.21.15.144:0/904092939 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7fa0a00008d0 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.870+0000 7fa09d7fa640 1 -- 172.21.15.144:0/904092939 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v27) v1 ==== 158+0+0 (secure 0 0 0) 0x7fa098017030 con 0x7fa0a00a0ad0 2024-09-18T12:26:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.870+0000 7fa09d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T12:26:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.870+0000 7fa09d7fa640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-18T12:26:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-18T12:26:49.875 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:26:49.875 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.948+0000 7f572031d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.948+0000 7f572031d640 10 monclient: build_initial_monmap 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.949+0000 7f572031d640 10 monclient: monmap: 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:49.949915+0000 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:49.949915+0000 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.950 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:49.951 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:49.951 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:49.951 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:49.951 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:49.951 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.949+0000 7f572031d640 1 Processor -- start 2024-09-18T12:26:49.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.949+0000 7f572031d640 1 -- start start 2024-09-18T12:26:49.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.949+0000 7f572031d640 10 monclient: init 2024-09-18T12:26:49.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:49.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:26:49.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5718131ac0 0x7f5718131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient(hunting): picked mon.noname-c con 0x7f5718131ac0 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f571809ff50 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57181323d0 0x7f5718098860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient(hunting): picked mon.noname-b con 0x7f57181323d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180a1610 con 0x7f571809ff50 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57181327d0 con 0x7f5718131ac0 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180a1330 con 0x7f57181323d0 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f572031d640 10 monclient(hunting): authenticate will time out at 845.519368s 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5718131ac0 0x7f5718131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:49.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571d8b8640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f571809ff50 0x7f5718098150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33240/0 (socket says 172.21.15.144:33240) 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571d8b8640 1 -- 172.21.15.144:0/2697430917 learned_addr learned my addr 172.21.15.144:0/2697430917 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): get_auth_request con 0x7f5718131ac0 auth_method 0 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e8ba640 1 -- 172.21.15.144:0/2697430917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57181323d0 msgr2=0x7f5718098860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e8ba640 1 --2- 172.21.15.144:0/2697430917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57181323d0 0x7f5718098860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient(hunting): handle_auth_done global_id 4522 payload 274 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 1 -- 172.21.15.144:0/2697430917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57181323d0 msgr2=0x7f5718098860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 1 --2- 172.21.15.144:0/2697430917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57181323d0 0x7f5718098860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 1 -- 172.21.15.144:0/2697430917 >> v1:172.21.15.144:6789/0 conn(0x7f571809ff50 legacy=0x7f5718098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:49.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient: _finish_hunting 0 2024-09-18T12:26:49.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 1 monclient: found mon.noname-c 2024-09-18T12:26:49.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:49.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.950+0000 7f571e0b9640 1 -- 172.21.15.144:0/2697430917 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718139e10 con 0x7f5718131ac0 2024-09-18T12:26:49.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571e0b9640 1 -- 172.21.15.144:0/2697430917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5718131ac0 msgr2=0x7f5718131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571e0b9640 1 --2- 172.21.15.144:0/2697430917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5718131ac0 0x7f5718131e90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5714009850 tx=0x7f57140090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fdf992502018c39 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 1 -- 172.21.15.144:0/2697430917 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714032440 con 0x7f5718131ac0 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: dump: 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.962 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: _finish_auth 0 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 1 -- 172.21.15.144:0/2697430917 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57140325a0 con 0x7f5718131ac0 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f572031d640 5 monclient: authenticate success, global_id 4522 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f572031d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f572031d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 1 -- 172.21.15.144:0/2697430917 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714032980 con 0x7f5718131ac0 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f572031d640 4 set_mon_vals no callback set 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.953+0000 7f571d0b7640 10 monclient: dump: 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.964 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 10 monclient: shutdown 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 -- 172.21.15.144:0/2697430917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5718131ac0 msgr2=0x7f5718131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 --2- 172.21.15.144:0/2697430917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5718131ac0 0x7f5718131e90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5714009850 tx=0x7f57140090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 -- 172.21.15.144:0/2697430917 shutdown_connections 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 --2- 172.21.15.144:0/2697430917 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57181323d0 0x7f5718098860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 --2- 172.21.15.144:0/2697430917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5718131ac0 0x7f5718131e90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 -- 172.21.15.144:0/2697430917 >> 172.21.15.144:0/2697430917 conn(0x7f57180a5190 msgr2=0x7f57180a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 -- 172.21.15.144:0/2697430917 shutdown_connections 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 1 -- 172.21.15.144:0/2697430917 wait complete. 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 10 monclient: build_initial_monmap 2024-09-18T12:26:49.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.954+0000 7f572031d640 10 monclient: monmap: 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 Processor -- start 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 -- start start 2024-09-18T12:26:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient: init 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 0x7f5718141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): picked mon.noname-b con 0x7f571809ff50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5718131ac0 0x7f5718137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): picked mon.noname-c con 0x7f5718131ac0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57181323d0 0x7f571812eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): picked mon.noname-a con 0x7f57181323d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181337d0 con 0x7f57181323d0 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718098e70 con 0x7f571809ff50 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180985d0 con 0x7f5718131ac0 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f572031d640 10 monclient(hunting): authenticate will time out at 845.524859s 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 0x7f5718141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 0x7f5718141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57742/0 (socket says 172.21.15.144:57742) 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 1 -- 172.21.15.144:0/1251071810 learned_addr learned my addr 172.21.15.144:0/1251071810 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:49.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 10 monclient(hunting): get_auth_request con 0x7f571809ff50 auth_method 0 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571d8b8640 1 -- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5718131ac0 msgr2=0x7f5718137f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571e0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.955+0000 7f571d8b8640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5718131ac0 0x7f5718137f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 10 monclient(hunting): handle_auth_done global_id 4525 payload 824 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 1 -- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5718131ac0 msgr2=0x7f5718137f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5718131ac0 0x7f5718137f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 1 -- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57181323d0 msgr2=0x7f571812eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57181323d0 0x7f571812eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 10 monclient: _finish_hunting 0 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 1 monclient: found mon.noname-b 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.956+0000 7f571e0b9640 1 -- 172.21.15.144:0/1251071810 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714009390 con 0x7f571809ff50 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f571e0b9640 1 -- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 msgr2=0x7f5718141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f571e0b9640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 0x7f5718141290 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5714009360 tx=0x7f5714032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714031a70 con 0x7f571809ff50 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: dump: 2024-09-18T12:26:49.970 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:49.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f572031d640 5 monclient: authenticate success, global_id 4525 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5714031bd0 con 0x7f571809ff50 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f572031d640 10 monclient: _renew_subs 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f572031d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5718130ba0 con 0x7f571809ff50 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714044410 con 0x7f571809ff50 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f570e7fc640 10 monclient: dump: 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.972 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f571c8b6640 4 set_mon_vals no callback set 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f572031d640 10 monclient: _renew_subs 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f572031d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.958+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5718133180 con 0x7f571809ff50 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5714039040 con 0x7f571809ff50 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f570e7fc640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56fc038440 0x7f56fc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:49.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f571406b440 con 0x7f571809ff50 2024-09-18T12:26:49.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f572031d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:49.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f572031d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:49.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f572031d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:49.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5718130b40 con 0x7f571809ff50 2024-09-18T12:26:49.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f571d8b8640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56fc038440 0x7f56fc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:49.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.959+0000 7f571d8b8640 10 monclient: get_auth_request con 0x7f56fc038440 auth_method 0 2024-09-18T12:26:49.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.962+0000 7f571d8b8640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56fc038440 0x7f56fc03a8f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f57180a4c60 tx=0x7f5708007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:49.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.964+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f571403b400 con 0x7f571809ff50 2024-09-18T12:26:49.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.964+0000 7f570e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:49.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:49.964+0000 7f570e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:50.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.118+0000 7f572031d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T12:26:50.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.118+0000 7f572031d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T12:26:50.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.118+0000 7f572031d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:50.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.118+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f571804bf60 con 0x7f571809ff50 2024-09-18T12:26:50.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.615+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v28) v1 ==== 176+0+0 (secure 0 0 0) 0x7f5714037030 con 0x7f571809ff50 2024-09-18T12:26:50.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.615+0000 7f570e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T12:26:50.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.615+0000 7f570e7fc640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-18T12:26:50.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.708+0000 7f572031d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T12:26:50.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.708+0000 7f572031d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T12:26:50.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.708+0000 7f572031d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:50.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.708+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57180a3170 con 0x7f571809ff50 2024-09-18T12:26:50.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.709+0000 7f570e7fc640 1 -- 172.21.15.144:0/1251071810 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v28) v1 ==== 183+0+0 (secure 0 0 0) 0x7f5714037030 con 0x7f571809ff50 2024-09-18T12:26:50.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.709+0000 7f570e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T12:26:50.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.709+0000 7f570e7fc640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-18T12:26:50.710 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'testpool1234' does not exist 2024-09-18T12:26:50.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56fc038440 msgr2=0x7f56fc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:50.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56fc038440 0x7f56fc03a8f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f57180a4c60 tx=0x7f5708007b80 comp rx=0 tx=0).stop 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 10 monclient: shutdown 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 msgr2=0x7f5718141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 0x7f5718141290 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5714009360 tx=0x7f5714032000 comp rx=0 tx=0).stop 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 shutdown_connections 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56fc038440 0x7f56fc03a8f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57181323d0 0x7f571812eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5718131ac0 0x7f5718137f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 --2- 172.21.15.144:0/1251071810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f571809ff50 0x7f5718141290 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 >> 172.21.15.144:0/1251071810 conn(0x7f57180a5190 msgr2=0x7f5718069690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 shutdown_connections 2024-09-18T12:26:50.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.711+0000 7f572031d640 1 -- 172.21.15.144:0/1251071810 wait complete. 2024-09-18T12:26:50.721 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-18T12:26:50.721 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:26:50.721 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.798+0000 7f543635d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.798+0000 7f543635d640 10 monclient: build_initial_monmap 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.798+0000 7f543635d640 10 monclient: monmap: 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:50.799754+0000 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:50.799754+0000 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:50.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:50.801 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:50.801 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:50.801 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:50.801 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.799+0000 7f543635d640 1 Processor -- start 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.799+0000 7f543635d640 1 -- start start 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.799+0000 7f543635d640 10 monclient: init 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5430096a30 0x7f543009d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient(hunting): picked mon.noname-c con 0x7f5430096a30 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5430096e00 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5430134e20 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543009c400 con 0x7f5430096e00 2024-09-18T12:26:50.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543009c120 con 0x7f5430134e20 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543009b410 con 0x7f5430096a30 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f543635d640 10 monclient(hunting): authenticate will time out at 846.369434s 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5430096a30 0x7f543009d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f54348fa640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f5430134e20 0x7f5430137200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33242/0 (socket says 172.21.15.144:33242) 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f54348fa640 1 -- 172.21.15.144:0/3335647138 learned_addr learned my addr 172.21.15.144:0/3335647138 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): get_auth_request con 0x7f5430096a30 auth_method 0 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:50.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient(hunting): handle_auth_done global_id 4528 payload 274 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 1 -- 172.21.15.144:0/3335647138 >> v1:172.21.15.144:6789/0 conn(0x7f5430134e20 legacy=0x7f5430137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 1 -- 172.21.15.144:0/3335647138 >> v1:172.21.15.116:6789/0 conn(0x7f5430096e00 legacy=0x7f5430097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 1 monclient: found mon.noname-c 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:50.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.800+0000 7f542ffff640 1 -- 172.21.15.144:0/3335647138 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543009c6e0 con 0x7f5430096a30 2024-09-18T12:26:50.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542ffff640 1 -- 172.21.15.144:0/3335647138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 msgr2=0x7f543009d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542ffff640 1 --2- 172.21.15.144:0/3335647138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f543009d090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5420009880 tx=0x7f54200090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=323dabb014af67b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 1 -- 172.21.15.144:0/3335647138 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420032440 con 0x7f5430096a30 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: dump: 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:50.810 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 1 -- 172.21.15.144:0/3335647138 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54200325a0 con 0x7f5430096a30 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f543635d640 5 monclient: authenticate success, global_id 4528 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f543635d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 1 -- 172.21.15.144:0/3335647138 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420032940 con 0x7f5430096a30 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f543635d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f543635d640 4 set_mon_vals no callback set 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:50.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.803+0000 7f542f7fe640 10 monclient: dump: 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 10 monclient: shutdown 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 1 -- 172.21.15.144:0/3335647138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 msgr2=0x7f543009d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:50.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 1 --2- 172.21.15.144:0/3335647138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f543009d090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5420009880 tx=0x7f54200090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 1 -- 172.21.15.144:0/3335647138 shutdown_connections 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 1 --2- 172.21.15.144:0/3335647138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f543009d090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 1 -- 172.21.15.144:0/3335647138 >> 172.21.15.144:0/3335647138 conn(0x7f54300711a0 msgr2=0x7f54300a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 1 -- 172.21.15.144:0/3335647138 shutdown_connections 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.804+0000 7f543635d640 1 -- 172.21.15.144:0/3335647138 wait complete. 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient: build_initial_monmap 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient: monmap: 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:50.813 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 Processor -- start 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 -- start start 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient: init 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f5430133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): picked mon.noname-b con 0x7f5430096a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:50.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430096e00 0x7f543011c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): picked mon.noname-a con 0x7f5430096e00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5430134e20 0x7f543011c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): picked mon.noname-c con 0x7f5430134e20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543009bbd0 con 0x7f5430096e00 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5430069570 con 0x7f5430096a30 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5430135740 con 0x7f5430134e20 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f543635d640 10 monclient(hunting): authenticate will time out at 846.374919s 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f542ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f5430133780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f542ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f5430133780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57762/0 (socket says 172.21.15.144:57762) 2024-09-18T12:26:50.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f542ffff640 1 -- 172.21.15.144:0/205658236 learned_addr learned my addr 172.21.15.144:0/205658236 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f542ffff640 10 monclient(hunting): get_auth_request con 0x7f5430096a30 auth_method 0 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f542ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f542ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f542ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.805+0000 7f54348fa640 1 -- 172.21.15.144:0/205658236 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5430134e20 msgr2=0x7f543011c780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f54348fa640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5430134e20 0x7f543011c780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:50.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:50.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 10 monclient(hunting): handle_auth_done global_id 4531 payload 824 2024-09-18T12:26:50.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 1 -- 172.21.15.144:0/205658236 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5430134e20 msgr2=0x7f543011c780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:50.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5430134e20 0x7f543011c780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 1 -- 172.21.15.144:0/205658236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430096e00 msgr2=0x7f543011c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:50.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f5427fff640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430096e00 0x7f543011c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:50.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430096e00 0x7f543011c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:50.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 1 monclient: found mon.noname-b 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.806+0000 7f542ffff640 1 -- 172.21.15.144:0/205658236 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5420009390 con 0x7f5430096a30 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542ffff640 1 -- 172.21.15.144:0/205658236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 msgr2=0x7f5430133780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542ffff640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f5430133780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5420009360 tx=0x7f5420032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542ffff640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5430096a30 0x7f5430133780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5420009360 tx=0x7f5420032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f5420031a70 mon_map magic: 0 v1 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 1 -- 172.21.15.144:0/205658236 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420031a70 con 0x7f5430096a30 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 10 monclient: dump: 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:50.818 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 10 monclient: _finish_auth 0 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 1 -- 172.21.15.144:0/205658236 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5420031bd0 con 0x7f5430096a30 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f543635d640 5 monclient: authenticate success, global_id 4531 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f543635d640 10 monclient: _renew_subs 2024-09-18T12:26:50.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f543635d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:50.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f543635d640 1 -- 172.21.15.144:0/205658236 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543011ddd0 con 0x7f5430096a30 2024-09-18T12:26:50.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:50.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f542effd640 4 set_mon_vals no callback set 2024-09-18T12:26:50.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f543635d640 10 monclient: _renew_subs 2024-09-18T12:26:50.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f543635d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:50.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.808+0000 7f543635d640 1 -- 172.21.15.144:0/205658236 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543011f890 con 0x7f5430096a30 2024-09-18T12:26:50.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f542cff9640 1 -- 172.21.15.144:0/205658236 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542003b400 con 0x7f5430096a30 2024-09-18T12:26:50.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f542cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:50.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f542cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:50.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f542cff9640 10 monclient: dump: 2024-09-18T12:26:50.821 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:50.821 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f542cff9640 1 -- 172.21.15.144:0/205658236 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5420039040 con 0x7f5430096a30 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f543635d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f542cff9640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5410038490 0x7f541003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f543635d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:50.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f543635d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:50.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f543635d640 1 -- 172.21.15.144:0/205658236 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54300008d0 con 0x7f5430096a30 2024-09-18T12:26:50.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f5427fff640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5410038490 0x7f541003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:50.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.809+0000 7f5427fff640 10 monclient: get_auth_request con 0x7f5410038490 auth_method 0 2024-09-18T12:26:50.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.812+0000 7f5427fff640 1 --2- 172.21.15.144:0/205658236 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5410038490 0x7f541003a940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5418002750 tx=0x7f5418009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:50.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:50.814+0000 7f542cff9640 1 -- 172.21.15.144:0/205658236 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f542006bb10 con 0x7f5430096a30 2024-09-18T12:26:51.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.431+0000 7f542cff9640 1 -- 172.21.15.144:0/205658236 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f542003f020 con 0x7f5430096a30 2024-09-18T12:26:51.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.431+0000 7f542cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:51.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.431+0000 7f542cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:51.590 INFO:tasks.workunit.client.0.smithi144.stderr:foo-rule not valid: foo-rule doesn't represent a float 2024-09-18T12:26:51.591 INFO:tasks.workunit.client.0.smithi144.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-18T12:26:51.591 INFO:tasks.workunit.client.0.smithi144.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] : create pool 2024-09-18T12:26:51.591 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: invalid command 2024-09-18T12:26:51.592 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:26:51.592 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.667+0000 7fb5e26b8640 10 monclient: get_monmap_and_config 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.667+0000 7fb5e26b8640 10 monclient: build_initial_monmap 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.667+0000 7fb5e26b8640 10 monclient: monmap: 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:51.668717+0000 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:51.668717+0000 2024-09-18T12:26:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:51.670 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.668+0000 7fb5e26b8640 1 Processor -- start 2024-09-18T12:26:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.668+0000 7fb5e26b8640 1 -- start start 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.668+0000 7fb5e26b8640 10 monclient: init 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb5dc0a0ad0 0x7fb5dc0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient(hunting): picked mon.noname-a con 0x7fb5dc0a0ad0 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc09ee60 0x7fb5dc09f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient(hunting): picked mon.noname-b con 0x7fb5dc09ee60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5dc131ad0 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5dc06a5b0 con 0x7fb5dc131ad0 2024-09-18T12:26:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc09f7a0 con 0x7fb5dc0a0ad0 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc0a0ea0 con 0x7fb5dc09ee60 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e26b8640 10 monclient(hunting): authenticate will time out at 847.238112s 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e0c55640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb5dc131ad0 0x7fb5dc131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33258/0 (socket says 172.21.15.144:33258) 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5e0c55640 1 -- 172.21.15.144:0/691238216 learned_addr learned my addr 172.21.15.144:0/691238216 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5db7fe640 1 -- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc09ee60 msgr2=0x7fb5dc09f260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5db7fe640 1 --2- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc09ee60 0x7fb5dc09f260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5daffd640 1 -- 172.21.15.144:0/691238216 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:26:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 1 --2- 172.21.15.144:0/691238216 >> v2:172.21.15.116:3300/0 conn(0x7fb5dc0a0ad0 0x7fb5dc0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request con 0x7fb5dc0a0ad0 auth_method 0 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_done global_id 4587 payload 274 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 1 -- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc09ee60 msgr2=0x7fb5dc09f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 1 --2- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc09ee60 0x7fb5dc09f260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 1 -- 172.21.15.144:0/691238216 >> v1:172.21.15.144:6789/0 conn(0x7fb5dc131ad0 legacy=0x7fb5dc131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 1 monclient: found mon.noname-a 2024-09-18T12:26:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.669+0000 7fb5dbfff640 1 -- 172.21.15.144:0/691238216 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc0a0460 con 0x7fb5dc0a0ad0 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5dbfff640 1 -- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc0a0ad0 msgr2=0x7fb5dc0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5dbfff640 1 --2- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc0a0ad0 0x7fb5dc0691d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009850 tx=0x7fb5cc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e74163a3528b1e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 1 -- 172.21.15.144:0/691238216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc032440 con 0x7fb5dc0a0ad0 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: dump: 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: _finish_auth 0 2024-09-18T12:26:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5e26b8640 5 monclient: authenticate success, global_id 4587 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5e26b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 1 -- 172.21.15.144:0/691238216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc0325a0 con 0x7fb5dc0a0ad0 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5e26b8640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 1 -- 172.21.15.144:0/691238216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc032940 con 0x7fb5dc0a0ad0 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5e26b8640 4 set_mon_vals no callback set 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.672+0000 7fb5daffd640 10 monclient: dump: 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:51.684 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 10 monclient: shutdown 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 -- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc0a0ad0 msgr2=0x7fb5dc0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc0a0ad0 0x7fb5dc0691d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009850 tx=0x7fb5cc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:51.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 -- 172.21.15.144:0/691238216 shutdown_connections 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc09ee60 0x7fb5dc09f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/691238216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc0a0ad0 0x7fb5dc0691d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 -- 172.21.15.144:0/691238216 >> 172.21.15.144:0/691238216 conn(0x7fb5dc071370 msgr2=0x7fb5dc0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 -- 172.21.15.144:0/691238216 shutdown_connections 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 1 -- 172.21.15.144:0/691238216 wait complete. 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.673+0000 7fb5e26b8640 10 monclient: build_initial_monmap 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient: monmap: 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:51.686 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 Processor -- start 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 -- start start 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient: init 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 0x7fb5dc09a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): picked mon.noname-b con 0x7fb5dc09ee60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:51.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc0a0ad0 0x7fb5dc130b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): picked mon.noname-c con 0x7fb5dc0a0ad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc131ad0 0x7fb5dc11b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): picked mon.noname-a con 0x7fb5dc131ad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc09b390 con 0x7fb5dc131ad0 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc0a11e0 con 0x7fb5dc09ee60 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc134b70 con 0x7fb5dc0a0ad0 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.674+0000 7fb5e26b8640 10 monclient(hunting): authenticate will time out at 847.243959s 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 0x7fb5dc09a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:51.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5db7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc0a0ad0 msgr2=0x7fb5dc130b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5db7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc0a0ad0 0x7fb5dc130b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 0x7fb5dc09a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57774/0 (socket says 172.21.15.144:57774) 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 -- 172.21.15.144:0/2488063071 learned_addr learned my addr 172.21.15.144:0/2488063071 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request con 0x7fb5dc09ee60 auth_method 0 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc131ad0 0x7fb5dc11b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 10 monclient(hunting): get_auth_request con 0x7fb5dc131ad0 auth_method 0 2024-09-18T12:26:51.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5e0c55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient(hunting): handle_auth_done global_id 4534 payload 824 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 -- 172.21.15.144:0/2488063071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc0a0ad0 msgr2=0x7fb5dc130b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc0a0ad0 0x7fb5dc130b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 -- 172.21.15.144:0/2488063071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc131ad0 msgr2=0x7fb5dc11b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc131ad0 0x7fb5dc11b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 monclient: found mon.noname-b 2024-09-18T12:26:51.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.675+0000 7fb5dbfff640 1 -- 172.21.15.144:0/2488063071 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc009390 con 0x7fb5dc09ee60 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5dbfff640 1 -- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 msgr2=0x7fb5dc09a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5dbfff640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 0x7fb5dc09a6b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009850 tx=0x7fb5cc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc031940 con 0x7fb5dc09ee60 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: dump: 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:51.691 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: _finish_auth 0 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc031aa0 con 0x7fb5dc09ee60 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5e26b8640 5 monclient: authenticate success, global_id 4534 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5e26b8640 10 monclient: _renew_subs 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5e26b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc11dc80 con 0x7fb5dc09ee60 2024-09-18T12:26:51.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc031e60 con 0x7fb5dc09ee60 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5c3fff640 10 monclient: dump: 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:51.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5e26b8640 10 monclient: _renew_subs 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5e26b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc11cd90 con 0x7fb5dc09ee60 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.677+0000 7fb5da7fc640 4 set_mon_vals no callback set 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.678+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb5cc039040 con 0x7fb5dc09ee60 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.678+0000 7fb5e26b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.678+0000 7fb5e26b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.678+0000 7fb5e26b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.678+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc0008d0 con 0x7fb5dc09ee60 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.678+0000 7fb5c3fff640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5c8038470 0x7fb5c803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.678+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fb5cc06b750 con 0x7fb5dc09ee60 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.679+0000 7fb5db7fe640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5c8038470 0x7fb5c803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.679+0000 7fb5db7fe640 10 monclient: get_auth_request con 0x7fb5c8038470 auth_method 0 2024-09-18T12:26:51.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.681+0000 7fb5db7fe640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5c8038470 0x7fb5c803a920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc11d500 tx=0x7fb5d0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:51.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.683+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb5cc0398f0 con 0x7fb5dc09ee60 2024-09-18T12:26:51.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.683+0000 7fb5c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:51.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.683+0000 7fb5c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.841+0000 7fb5e26b8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T12:26:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.841+0000 7fb5e26b8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T12:26:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.841+0000 7fb5e26b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:51.841+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7fb5dc064a20 con 0x7fb5dc09ee60 2024-09-18T12:26:52.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.632+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v29) v1 ==== 154+0+0 (secure 0 0 0) 0x7fb5cc03a360 con 0x7fb5dc09ee60 2024-09-18T12:26:52.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.632+0000 7fb5c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T12:26:52.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.632+0000 7fb5c3fff640 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-18T12:26:52.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.728+0000 7fb5e26b8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T12:26:52.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.728+0000 7fb5e26b8640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T12:26:52.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.728+0000 7fb5e26b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:52.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.728+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7fb5dc064a20 con 0x7fb5dc09ee60 2024-09-18T12:26:52.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.730+0000 7fb5c3fff640 1 -- 172.21.15.144:0/2488063071 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v29) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb5cc03f0c0 con 0x7fb5dc09ee60 2024-09-18T12:26:52.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.730+0000 7fb5c3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T12:26:52.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.730+0000 7fb5c3fff640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-18T12:26:52.731 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'foo' already exists 2024-09-18T12:26:52.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.731+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5c8038470 msgr2=0x7fb5c803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:52.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.731+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5c8038470 0x7fb5c803a920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc11d500 tx=0x7fb5d0006d80 comp rx=0 tx=0).stop 2024-09-18T12:26:52.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.731+0000 7fb5e26b8640 10 monclient: shutdown 2024-09-18T12:26:52.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.731+0000 7fb5e26b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:52.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.731+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 msgr2=0x7fb5dc09a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:52.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.731+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 0x7fb5dc09a6b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009850 tx=0x7fb5cc032000 comp rx=0 tx=0).stop 2024-09-18T12:26:52.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 shutdown_connections 2024-09-18T12:26:52.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5c8038470 0x7fb5c803a920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5dc131ad0 0x7fb5dc11b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5dc0a0ad0 0x7fb5dc130b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 --2- 172.21.15.144:0/2488063071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5dc09ee60 0x7fb5dc09a6b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 >> 172.21.15.144:0/2488063071 conn(0x7fb5dc071370 msgr2=0x7fb5dc0a4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:52.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 shutdown_connections 2024-09-18T12:26:52.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.732+0000 7fb5e26b8640 1 -- 172.21.15.144:0/2488063071 wait complete. 2024-09-18T12:26:52.741 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-18T12:26:52.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.817+0000 7f0b13660640 10 monclient: get_monmap_and_config 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.817+0000 7f0b13660640 10 monclient: build_initial_monmap 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.817+0000 7f0b13660640 10 monclient: monmap: 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:52.818092+0000 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:52.818092+0000 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:52.819 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:52.820 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:52.820 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:52.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.817+0000 7f0b13660640 1 Processor -- start 2024-09-18T12:26:52.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.817+0000 7f0b13660640 1 -- start start 2024-09-18T12:26:52.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.817+0000 7f0b13660640 10 monclient: init 2024-09-18T12:26:52.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:52.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:26:52.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b0c09eb50 0x7f0b0c09d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient(hunting): picked mon.noname-c con 0x7f0b0c09eb50 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b0c09ef20 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b0c09f320 0x7f0b0c069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient(hunting): picked mon.noname-a con 0x7f0b0c09f320 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c09c8c0 con 0x7f0b0c09ef20 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c0a17e0 con 0x7f0b0c09f320 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c09d920 con 0x7f0b0c09eb50 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:52.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b13660640 10 monclient(hunting): authenticate will time out at 848.387489s 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b0c09eb50 0x7f0b0c09d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b0c09eb50 0x7f0b0c09d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57778/0 (socket says 172.21.15.144:57778) 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 -- 172.21.15.144:0/1920327491 learned_addr learned my addr 172.21.15.144:0/1920327491 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): get_auth_request con 0x7f0b0c09eb50 auth_method 0 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:52.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient(hunting): handle_auth_done global_id 4537 payload 274 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 -- 172.21.15.144:0/1920327491 >> v2:172.21.15.116:3300/0 conn(0x7f0b0c09f320 msgr2=0x7f0b0c069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b11bfd640 1 --2- 172.21.15.144:0/1920327491 >> v2:172.21.15.116:3300/0 conn(0x7f0b0c09f320 0x7f0b0c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 --2- 172.21.15.144:0/1920327491 >> v2:172.21.15.116:3300/0 conn(0x7f0b0c09f320 0x7f0b0c069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 -- 172.21.15.144:0/1920327491 >> v1:172.21.15.116:6789/0 conn(0x7f0b0c09ef20 legacy=0x7f0b0c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient: _finish_hunting 0 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 monclient: found mon.noname-c 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b113fc640 1 -- 172.21.15.144:0/1920327491 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0a02e0 con 0x7f0b0c09eb50 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b11bfd640 10 monclient: get_auth_request con 0x7f0b0c09f320 auth_method 0 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.818+0000 7f0b11bfd640 1 --2- 172.21.15.144:0/1920327491 >> v2:172.21.15.116:3300/0 conn(0x7f0b0c09f320 0x7f0b0c069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.821+0000 7f0b113fc640 1 -- 172.21.15.144:0/1920327491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09eb50 msgr2=0x7f0b0c09d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:52.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.821+0000 7f0b113fc640 1 --2- 172.21.15.144:0/1920327491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09eb50 0x7f0b0c09d2c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c071950 tx=0x7f0b000091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b96d6dc298d4f31 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:52.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 1 -- 172.21.15.144:0/1920327491 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00032440 con 0x7f0b0c09eb50 2024-09-18T12:26:52.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:52.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:52.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: dump: 2024-09-18T12:26:52.831 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: _finish_auth 0 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:52.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 1 -- 172.21.15.144:0/1920327491 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b000325a0 con 0x7f0b0c09eb50 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 5 monclient: authenticate success, global_id 4537 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 1 -- 172.21.15.144:0/1920327491 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00032940 con 0x7f0b0c09eb50 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 4 set_mon_vals no callback set 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0afffff640 10 monclient: dump: 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.833 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 10 monclient: shutdown 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 -- 172.21.15.144:0/1920327491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09eb50 msgr2=0x7f0b0c09d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 --2- 172.21.15.144:0/1920327491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09eb50 0x7f0b0c09d2c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c071950 tx=0x7f0b000091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 -- 172.21.15.144:0/1920327491 shutdown_connections 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 --2- 172.21.15.144:0/1920327491 >> v2:172.21.15.116:3300/0 conn(0x7f0b0c09f320 0x7f0b0c069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 --2- 172.21.15.144:0/1920327491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09eb50 0x7f0b0c09d2c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 -- 172.21.15.144:0/1920327491 >> 172.21.15.144:0/1920327491 conn(0x7f0b0c0a5d40 msgr2=0x7f0b0c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 -- 172.21.15.144:0/1920327491 shutdown_connections 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.822+0000 7f0b13660640 1 -- 172.21.15.144:0/1920327491 wait complete. 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.823+0000 7f0b13660640 10 monclient: build_initial_monmap 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.823+0000 7f0b13660640 10 monclient: monmap: 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:52.835 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.823+0000 7f0b13660640 1 Processor -- start 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.823+0000 7f0b13660640 1 -- start start 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.823+0000 7f0b13660640 10 monclient: init 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b0c09eb50 0x7f0b0c09a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): picked mon.noname-c con 0x7f0b0c09eb50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 0x7f0b0c139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): picked mon.noname-b con 0x7f0b0c09ef20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b0c09f320 0x7f0b0c123f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:52.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): picked mon.noname-a con 0x7f0b0c09f320 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c09bcb0 con 0x7f0b0c09f320 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c1268e0 con 0x7f0b0c09ef20 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c146250 con 0x7f0b0c09eb50 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b13660640 10 monclient(hunting): authenticate will time out at 848.393340s 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 0x7f0b0c139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 0x7f0b0c139200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57794/0 (socket says 172.21.15.144:57794) 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 -- 172.21.15.144:0/246000311 learned_addr learned my addr 172.21.15.144:0/246000311 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): get_auth_request con 0x7f0b0c09ef20 auth_method 0 2024-09-18T12:26:52.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b113fc640 1 -- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b0c09eb50 msgr2=0x7f0b0c09a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b113fc640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b0c09eb50 0x7f0b0c09a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b11bfd640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b0c09f320 0x7f0b0c123f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient(hunting): handle_auth_done global_id 4540 payload 824 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 -- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b0c09eb50 msgr2=0x7f0b0c09a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b0c09eb50 0x7f0b0c09a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 -- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b0c09f320 msgr2=0x7f0b0c123f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b0c09f320 0x7f0b0c123f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient: _finish_hunting 0 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 monclient: found mon.noname-b 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b10bfb640 1 -- 172.21.15.144:0/246000311 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b00009450 con 0x7f0b0c09ef20 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b11bfd640 10 monclient: get_auth_request con 0x7f0b0c09f320 auth_method 0 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.824+0000 7f0b11bfd640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b0c09f320 0x7f0b0c123f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.826+0000 7f0b10bfb640 1 -- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 msgr2=0x7f0b0c139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.826+0000 7f0b10bfb640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 0x7f0b0c139200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800c610 tx=0x7f0b08007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b08012920 con 0x7f0b0c09ef20 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:52.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: dump: 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: _finish_auth 0 2024-09-18T12:26:52.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b08012aa0 con 0x7f0b0c09ef20 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0b13660640 5 monclient: authenticate success, global_id 4540 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0b13660640 10 monclient: _renew_subs 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0b13660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c126320 con 0x7f0b0c09ef20 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b08012e60 con 0x7f0b0c09ef20 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 10 monclient: dump: 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:52.841 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0aff7fe640 4 set_mon_vals no callback set 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0b13660640 10 monclient: _renew_subs 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0b13660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c13a830 con 0x7f0b0c09ef20 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.827+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0b08011590 con 0x7f0b0c09ef20 2024-09-18T12:26:52.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0afd7fa640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0af0038490 0x7f0af003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4684+0+0 (secure 0 0 0) 0x7f0b0804ba60 con 0x7f0b0c09ef20 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0b113fc640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0af0038490 0x7f0af003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0b13660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0b13660640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0b13660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c126a20 con 0x7f0b0c09ef20 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.828+0000 7f0b113fc640 10 monclient: get_auth_request con 0x7f0af0038490 auth_method 0 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.830+0000 7f0b113fc640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0af0038490 0x7f0af003a940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0a2c40 tx=0x7f0b00032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:52.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.832+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0b0c126a20 con 0x7f0b0c09ef20 2024-09-18T12:26:52.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.832+0000 7f0afd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:52.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.832+0000 7f0afd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:52.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.987+0000 7f0b13660640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T12:26:52.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.987+0000 7f0b13660640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T12:26:52.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.987+0000 7f0b13660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:52.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:52.987+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f0b0c126a20 con 0x7f0b0c09ef20 2024-09-18T12:26:54.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.640+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v31) v1 ==== 188+0+0 (secure 0 0 0) 0x7f0b0c126a20 con 0x7f0b0c09ef20 2024-09-18T12:26:54.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.640+0000 7f0afd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T12:26:54.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.640+0000 7f0afd7fa640 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-18T12:26:54.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.748+0000 7f0b13660640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T12:26:54.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.749+0000 7f0b13660640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T12:26:54.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.749+0000 7f0b13660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:54.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.749+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f0b0c0008d0 con 0x7f0b0c09ef20 2024-09-18T12:26:54.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.750+0000 7f0afd7fa640 1 -- 172.21.15.144:0/246000311 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v31) v1 ==== 195+0+0 (secure 0 0 0) 0x7f0b08016020 con 0x7f0b0c09ef20 2024-09-18T12:26:54.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.750+0000 7f0afd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T12:26:54.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.750+0000 7f0afd7fa640 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-18T12:26:54.751 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'fooo' already exists 2024-09-18T12:26:54.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0af0038490 msgr2=0x7f0af003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:54.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0af0038490 0x7f0af003a940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0a2c40 tx=0x7f0b00032000 comp rx=0 tx=0).stop 2024-09-18T12:26:54.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 10 monclient: shutdown 2024-09-18T12:26:54.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 msgr2=0x7f0b0c139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 0x7f0b0c139200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800c610 tx=0x7f0b08007ed0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 shutdown_connections 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0af0038490 0x7f0af003a940 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b0c09f320 0x7f0b0c123f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b0c09ef20 0x7f0b0c139200 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 --2- 172.21.15.144:0/246000311 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b0c09eb50 0x7f0b0c09a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 >> 172.21.15.144:0/246000311 conn(0x7f0b0c0a5d40 msgr2=0x7f0b0c06cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 shutdown_connections 2024-09-18T12:26:54.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.752+0000 7f0b13660640 1 -- 172.21.15.144:0/246000311 wait complete. 2024-09-18T12:26:54.762 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create foooo 123 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.835+0000 7fcb12b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.835+0000 7fcb12b9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.835+0000 7fcb12b9d640 10 monclient: monmap: 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:54.836750+0000 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:54.836750+0000 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:54.837 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:54.838 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:54.838 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:54.838 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:54.838 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:54.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.836+0000 7fcb12b9d640 1 Processor -- start 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.836+0000 7fcb12b9d640 1 -- start start 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.836+0000 7fcb12b9d640 10 monclient: init 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb0c131c90 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb0c09d210 0x7fcb0c069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0c09d210 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c0a0ab0 0x7fcb0c138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0c0a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0a1800 con 0x7fcb0c131c90 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:54.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c09fea0 con 0x7fcb0c09d210 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0a1520 con 0x7fcb0c0a0ab0 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb12b9d640 10 monclient(hunting): authenticate will time out at 850.406253s 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb10939640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fcb0c131c90 0x7fcb0c132060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33270/0 (socket says 172.21.15.144:33270) 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb10939640 1 -- 172.21.15.144:0/1637401409 learned_addr learned my addr 172.21.15.144:0/1637401409 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb1113a640 1 -- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c0a0ab0 msgr2=0x7fcb0c138fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb1113a640 1 --2- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c0a0ab0 0x7fcb0c138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 1 --2- 172.21.15.144:0/1637401409 >> v2:172.21.15.116:3300/0 conn(0x7fcb0c09d210 0x7fcb0c069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): get_auth_request con 0x7fcb0c09d210 auth_method 0 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:54.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 1 -- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c0a0ab0 msgr2=0x7fcb0c138fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 1 --2- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c0a0ab0 0x7fcb0c138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 1 -- 172.21.15.144:0/1637401409 >> v1:172.21.15.144:6789/0 conn(0x7fcb0c131c90 legacy=0x7fcb0c132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 1 monclient: found mon.noname-a 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:54.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.837+0000 7fcb0bfff640 1 -- 172.21.15.144:0/1637401409 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c132770 con 0x7fcb0c09d210 2024-09-18T12:26:54.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.840+0000 7fcb0bfff640 1 -- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c09d210 msgr2=0x7fcb0c069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.840+0000 7fcb0bfff640 1 --2- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c09d210 0x7fcb0c069e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0769f0 tx=0x7fcb000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4b006e6ce813c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 1 -- 172.21.15.144:0/1637401409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0002aea0 con 0x7fcb0c09d210 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: dump: 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:54.850 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 5 monclient: authenticate success, global_id 4593 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 1 -- 172.21.15.144:0/1637401409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00032440 con 0x7fcb0c09d210 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 1 -- 172.21.15.144:0/1637401409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb000327e0 con 0x7fcb0c09d210 2024-09-18T12:26:54.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 4 set_mon_vals no callback set 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb0b7fe640 10 monclient: dump: 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:54.852 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 10 monclient: shutdown 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c09d210 msgr2=0x7fcb0c069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c09d210 0x7fcb0c069e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0769f0 tx=0x7fcb000090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1637401409 shutdown_connections 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c0a0ab0 0x7fcb0c138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1637401409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c09d210 0x7fcb0c069e10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1637401409 >> 172.21.15.144:0/1637401409 conn(0x7fcb0c071370 msgr2=0x7fcb0c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1637401409 shutdown_connections 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.841+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1637401409 wait complete. 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.842+0000 7fcb12b9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:54.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.842+0000 7fcb12b9d640 10 monclient: monmap: 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:54.854 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.842+0000 7fcb12b9d640 1 Processor -- start 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.842+0000 7fcb12b9d640 1 -- start start 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.842+0000 7fcb12b9d640 10 monclient: init 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c0a0ab0 0x7fcb0c1313c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0c0a0ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c131c90 0x7fcb0c096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb0c131c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 0x7fcb0c0973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0c096f70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c097900 con 0x7fcb0c0a0ab0 2024-09-18T12:26:54.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c099c20 con 0x7fcb0c096f70 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c097f50 con 0x7fcb0c131c90 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb12b9d640 10 monclient(hunting): authenticate will time out at 850.412145s 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb0bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c131c90 msgr2=0x7fcb0c096a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb0bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c131c90 0x7fcb0c096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 0x7fcb0c0973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:54.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 0x7fcb0c0973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57798/0 (socket says 172.21.15.144:57798) 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 -- 172.21.15.144:0/1404439613 learned_addr learned my addr 172.21.15.144:0/1404439613 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): get_auth_request con 0x7fcb0c096f70 auth_method 0 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c0a0ab0 0x7fcb0c1313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 10 monclient(hunting): get_auth_request con 0x7fcb0c0a0ab0 auth_method 0 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:54.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb10939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient(hunting): handle_auth_done global_id 4543 payload 824 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 -- 172.21.15.144:0/1404439613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c131c90 msgr2=0x7fcb0c096a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c131c90 0x7fcb0c096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 -- 172.21.15.144:0/1404439613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c0a0ab0 msgr2=0x7fcb0c1313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c0a0ab0 0x7fcb0c1313c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient: _finish_hunting 0 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 monclient: found mon.noname-b 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:54.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.843+0000 7fcb1113a640 1 -- 172.21.15.144:0/1404439613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb00009390 con 0x7fcb0c096f70 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb1113a640 1 -- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 msgr2=0x7fcb0c0973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb1113a640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 0x7fcb0c0973c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400bb40 tx=0x7fcb040097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400cbf0 con 0x7fcb0c096f70 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: dump: 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.859 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: _finish_auth 0 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:54.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0400cd70 con 0x7fcb0c096f70 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb12b9d640 5 monclient: authenticate success, global_id 4543 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb12b9d640 10 monclient: _renew_subs 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb12b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c09ad20 con 0x7fcb0c096f70 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04006150 con 0x7fcb0c096f70 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb08ff9640 10 monclient: dump: 2024-09-18T12:26:54.861 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb0affd640 4 set_mon_vals no callback set 2024-09-18T12:26:54.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb12b9d640 10 monclient: _renew_subs 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb12b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.845+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c099940 con 0x7fcb0c096f70 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.846+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fcb04012020 con 0x7fcb0c096f70 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.846+0000 7fcb12b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.846+0000 7fcb12b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.846+0000 7fcb12b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.846+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c0008d0 con 0x7fcb0c096f70 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.847+0000 7fcb08ff9640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcaf8038440 0x7fcaf803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.847+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5126+0+0 (secure 0 0 0) 0x7fcb0404b880 con 0x7fcb0c096f70 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.847+0000 7fcb10939640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcaf8038440 0x7fcaf803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:54.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.847+0000 7fcb10939640 10 monclient: get_auth_request con 0x7fcaf8038440 auth_method 0 2024-09-18T12:26:54.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.849+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:26:54.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.849+0000 7fcb10939640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcaf8038440 0x7fcaf803a8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcafc007fd0 tx=0x7fcafc007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:54.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.851+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fcb0401da60 con 0x7fcb0c096f70 2024-09-18T12:26:54.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.851+0000 7fcb08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:54.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:54.851+0000 7fcb08ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:55.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.015+0000 7fcb12b9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T12:26:55.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.015+0000 7fcb12b9d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T12:26:55.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.015+0000 7fcb12b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:55.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.015+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7fcb0c1394f0 con 0x7fcb0c096f70 2024-09-18T12:26:55.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.640+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v32) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcb0401da60 con 0x7fcb0c096f70 2024-09-18T12:26:55.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.640+0000 7fcb08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T12:26:55.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.640+0000 7fcb08ff9640 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-18T12:26:55.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.778+0000 7fcb12b9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T12:26:55.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.778+0000 7fcb12b9d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T12:26:55.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.778+0000 7fcb12b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:55.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.778+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7fcb0c064a20 con 0x7fcb0c096f70 2024-09-18T12:26:55.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.779+0000 7fcb08ff9640 1 -- 172.21.15.144:0/1404439613 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcb0401d800 con 0x7fcb0c096f70 2024-09-18T12:26:55.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.779+0000 7fcb08ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T12:26:55.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.779+0000 7fcb08ff9640 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-18T12:26:55.780 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'foooo' already exists 2024-09-18T12:26:55.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcaf8038440 msgr2=0x7fcaf803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcaf8038440 0x7fcaf803a8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcafc007fd0 tx=0x7fcafc007f00 comp rx=0 tx=0).stop 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 10 monclient: shutdown 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 msgr2=0x7fcb0c0973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 0x7fcb0c0973c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400bb40 tx=0x7fcb040097d0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 shutdown_connections 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcaf8038440 0x7fcaf803a8f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb0c096f70 0x7fcb0c0973c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb0c131c90 0x7fcb0c096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 --2- 172.21.15.144:0/1404439613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb0c0a0ab0 0x7fcb0c1313c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 >> 172.21.15.144:0/1404439613 conn(0x7fcb0c071370 msgr2=0x7fcb0c0a3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 shutdown_connections 2024-09-18T12:26:55.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.781+0000 7fcb12b9d640 1 -- 172.21.15.144:0/1404439613 wait complete. 2024-09-18T12:26:55.791 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create foo 123 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.865+0000 7f85dc91d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.865+0000 7f85dc91d640 10 monclient: build_initial_monmap 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.865+0000 7f85dc91d640 10 monclient: monmap: 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:55.865940+0000 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:55.865940+0000 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:55.866 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:55.867 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:55.867 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:55.867 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:55.867 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:55.867 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:55.867 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:55.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.865+0000 7f85dc91d640 1 Processor -- start 2024-09-18T12:26:55.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.865+0000 7f85dc91d640 1 -- start start 2024-09-18T12:26:55.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.865+0000 7f85dc91d640 10 monclient: init 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85d809ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d809f020 0x7f85d8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient(hunting): picked mon.noname-b con 0x7f85d809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85d8136d60 0x7f85d8139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient(hunting): picked mon.noname-c con 0x7f85d8136d60 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d8067420 con 0x7f85d809ec50 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d809fdc0 con 0x7f85d8136d60 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80a0510 con 0x7f85d809f020 2024-09-18T12:26:55.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85dc91d640 10 monclient(hunting): authenticate will time out at 851.435293s 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d659c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f85d809ec50 0x7f85d8131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33272/0 (socket says 172.21.15.144:33272) 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d659c640 1 -- 172.21.15.144:0/2965851500 learned_addr learned my addr 172.21.15.144:0/2965851500 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d5d9b640 1 -- 172.21.15.144:0/2965851500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d809f020 msgr2=0x7f85d8069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d5d9b640 1 --2- 172.21.15.144:0/2965851500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d809f020 0x7f85d8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d559a640 1 -- 172.21.15.144:0/2965851500 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 1 --2- 172.21.15.144:0/2965851500 >> v2:172.21.15.144:3300/0 conn(0x7f85d8136d60 0x7f85d8139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): get_auth_request con 0x7f85d8136d60 auth_method 0 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:55.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient(hunting): handle_auth_done global_id 4546 payload 274 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 1 -- 172.21.15.144:0/2965851500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d809f020 msgr2=0x7f85d8069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 1 --2- 172.21.15.144:0/2965851500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d809f020 0x7f85d8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 1 -- 172.21.15.144:0/2965851500 >> v1:172.21.15.144:6789/0 conn(0x7f85d809ec50 legacy=0x7f85d8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 1 monclient: found mon.noname-c 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:55.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.866+0000 7f85d6d9d640 1 -- 172.21.15.144:0/2965851500 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80a0200 con 0x7f85d8136d60 2024-09-18T12:26:55.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d6d9d640 1 -- 172.21.15.144:0/2965851500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d8136d60 msgr2=0x7f85d8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:55.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d6d9d640 1 --2- 172.21.15.144:0/2965851500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d8136d60 0x7f85d8139140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009850 tx=0x7f85c80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b76d9d0c1d24dd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 1 -- 172.21.15.144:0/2965851500 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8032440 con 0x7f85d8136d60 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: dump: 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:55.878 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: _finish_auth 0 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 1 -- 172.21.15.144:0/2965851500 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c80325a0 con 0x7f85d8136d60 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85dc91d640 5 monclient: authenticate success, global_id 4546 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85dc91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85dc91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85dc91d640 4 set_mon_vals no callback set 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 1 -- 172.21.15.144:0/2965851500 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8032940 con 0x7f85d8136d60 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.869+0000 7f85d559a640 10 monclient: dump: 2024-09-18T12:26:55.879 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 10 monclient: shutdown 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 -- 172.21.15.144:0/2965851500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d8136d60 msgr2=0x7f85d8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:55.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 --2- 172.21.15.144:0/2965851500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d8136d60 0x7f85d8139140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009850 tx=0x7f85c80090f0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 -- 172.21.15.144:0/2965851500 shutdown_connections 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 --2- 172.21.15.144:0/2965851500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d8136d60 0x7f85d8139140 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 --2- 172.21.15.144:0/2965851500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d809f020 0x7f85d8069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 -- 172.21.15.144:0/2965851500 >> 172.21.15.144:0/2965851500 conn(0x7f85d80a5d40 msgr2=0x7f85d80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 -- 172.21.15.144:0/2965851500 shutdown_connections 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 1 -- 172.21.15.144:0/2965851500 wait complete. 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.870+0000 7f85dc91d640 10 monclient: build_initial_monmap 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient: monmap: 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:55.881 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 Processor -- start 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 -- start start 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient: init 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d809ec50 0x7f85d81363f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): picked mon.noname-a con 0x7f85d809ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:55.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 0x7f85d812efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): picked mon.noname-b con 0x7f85d809f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d8136d60 0x7f85d812f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): picked mon.noname-c con 0x7f85d8136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d8133b80 con 0x7f85d809ec50 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80a4d30 con 0x7f85d809f020 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81335b0 con 0x7f85d8136d60 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85dc91d640 10 monclient(hunting): authenticate will time out at 851.440784s 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 0x7f85d812efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 0x7f85d812efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57810/0 (socket says 172.21.15.144:57810) 2024-09-18T12:26:55.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 1 -- 172.21.15.144:0/4225389480 learned_addr learned my addr 172.21.15.144:0/4225389480 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 10 monclient(hunting): get_auth_request con 0x7f85d809f020 auth_method 0 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d6d9d640 1 -- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d8136d60 msgr2=0x7f85d812f4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d6d9d640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d8136d60 0x7f85d812f4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.871+0000 7f85d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 10 monclient(hunting): handle_auth_done global_id 4549 payload 824 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d659c640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d809ec50 0x7f85d81363f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 1 -- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d8136d60 msgr2=0x7f85d812f4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:55.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d8136d60 0x7f85d812f4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 1 -- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d809ec50 msgr2=0x7f85d81363f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d809ec50 0x7f85d81363f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 1 monclient: found mon.noname-b 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d5d9b640 1 -- 172.21.15.144:0/4225389480 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8009390 con 0x7f85d809f020 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d659c640 10 monclient: get_auth_request con 0x7f85d809ec50 auth_method 0 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.872+0000 7f85d659c640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d809ec50 0x7f85d81363f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85d5d9b640 1 -- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 msgr2=0x7f85d812efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85d5d9b640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 0x7f85d812efb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f85c000b600 tx=0x7f85c00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0015070 con 0x7f85d809f020 2024-09-18T12:26:55.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: dump: 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: _finish_auth 0 2024-09-18T12:26:55.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c0003e70 con 0x7f85d809f020 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85dc91d640 5 monclient: authenticate success, global_id 4549 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85dc91d640 10 monclient: _renew_subs 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85dc91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d8133870 con 0x7f85d809f020 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0006360 con 0x7f85d809f020 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85ce7fc640 10 monclient: dump: 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:55.887 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85d4d99640 4 set_mon_vals no callback set 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85dc91d640 10 monclient: _renew_subs 2024-09-18T12:26:55.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85dc91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.874+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d8131520 con 0x7f85d809f020 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f85c0010440 con 0x7f85d809f020 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85dc91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85dc91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85dc91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d80008d0 con 0x7f85d809f020 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85ce7fc640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f85b8038440 0x7f85b803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f85c004b6e0 con 0x7f85d809f020 2024-09-18T12:26:55.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85d659c640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f85b8038440 0x7f85b803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:55.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.875+0000 7f85d659c640 10 monclient: get_auth_request con 0x7f85b8038440 auth_method 0 2024-09-18T12:26:55.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.878+0000 7f85d659c640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f85b8038440 0x7f85b803a8f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85d0005bc0 tx=0x7f85d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:55.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:55.880+0000 7f85d5d9b640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 0x7f85d812efb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f85c000b600 tx=0x7f85c00091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f85c0018ab0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:26:56.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.708+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f85c0018ab0 con 0x7f85d809f020 2024-09-18T12:26:56.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.708+0000 7f85ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:56.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.708+0000 7f85ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:56.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.864+0000 7f85dc91d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T12:26:56.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.864+0000 7f85dc91d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T12:26:56.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.864+0000 7f85dc91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:56.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.864+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7f85d8133cf0 con 0x7f85d809f020 2024-09-18T12:26:56.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.865+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f85c0018690 con 0x7f85d809f020 2024-09-18T12:26:56.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.865+0000 7f85ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T12:26:56.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.865+0000 7f85ce7fc640 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-18T12:26:56.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.959+0000 7f85dc91d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T12:26:56.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.959+0000 7f85dc91d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T12:26:56.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.959+0000 7f85dc91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:56.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.959+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7f85d8072640 con 0x7f85d809f020 2024-09-18T12:26:56.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.960+0000 7f85ce7fc640 1 -- 172.21.15.144:0/4225389480 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f85c0018850 con 0x7f85d809f020 2024-09-18T12:26:56.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.960+0000 7f85ce7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T12:26:56.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.960+0000 7f85ce7fc640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-18T12:26:56.961 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'foo' already exists 2024-09-18T12:26:56.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f85b8038440 msgr2=0x7f85b803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:56.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f85b8038440 0x7f85b803a8f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85d0005bc0 tx=0x7f85d0009040 comp rx=0 tx=0).stop 2024-09-18T12:26:56.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 10 monclient: shutdown 2024-09-18T12:26:56.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 msgr2=0x7f85d812efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 0x7f85d812efb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f85c000b600 tx=0x7f85c00091a0 comp rx=0 tx=0).stop 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 shutdown_connections 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f85b8038440 0x7f85b803a8f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85d8136d60 0x7f85d812f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85d809f020 0x7f85d812efb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 --2- 172.21.15.144:0/4225389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d809ec50 0x7f85d81363f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 >> 172.21.15.144:0/4225389480 conn(0x7f85d80a5d40 msgr2=0x7f85d812ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 shutdown_connections 2024-09-18T12:26:56.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:56.962+0000 7f85dc91d640 1 -- 172.21.15.144:0/4225389480 wait complete. 2024-09-18T12:26:56.972 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.046+0000 7fc9a20a8640 10 monclient: get_monmap_and_config 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.046+0000 7fc9a20a8640 10 monclient: build_initial_monmap 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.046+0000 7fc9a20a8640 10 monclient: monmap: 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:26:57.047772+0000 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:26:57.047772+0000 2024-09-18T12:26:57.048 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:57.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:57.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:57.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:57.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:57.049 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:57.049 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:57.049 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.047+0000 7fc9a20a8640 1 Processor -- start 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.047+0000 7fc9a20a8640 1 -- start start 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.047+0000 7fc9a20a8640 10 monclient: init 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc99c098c70 0x7fc99c099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc99c098c70 addr v2:172.21.15.116:3300/0 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc99c099580 0x7fc99c09dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc99c099580 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc99c099c70 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c09f680 con 0x7fc99c099c70 2024-09-18T12:26:57.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c09f420 con 0x7fc99c098c70 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c068a20 con 0x7fc99c099580 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc9a20a8640 10 monclient(hunting): authenticate will time out at 852.617293s 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc99c099580 0x7fc99c09dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc99c099580 0x7fc99c09dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57820/0 (socket says 172.21.15.144:57820) 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 -- 172.21.15.144:0/2991126826 learned_addr learned my addr 172.21.15.144:0/2991126826 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): get_auth_request con 0x7fc99c099580 auth_method 0 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:57.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc99b7fe640 1 --2- 172.21.15.144:0/2991126826 >> v2:172.21.15.116:3300/0 conn(0x7fc99c098c70 0x7fc99c099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient(hunting): handle_auth_done global_id 4552 payload 274 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 -- 172.21.15.144:0/2991126826 >> v2:172.21.15.116:3300/0 conn(0x7fc99c098c70 msgr2=0x7fc99c099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 --2- 172.21.15.144:0/2991126826 >> v2:172.21.15.116:3300/0 conn(0x7fc99c098c70 0x7fc99c099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 -- 172.21.15.144:0/2991126826 >> v1:172.21.15.116:6789/0 conn(0x7fc99c099c70 legacy=0x7fc99c09e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient: _finish_hunting 0 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 monclient: found mon.noname-c 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc993fff640 1 -- 172.21.15.144:0/2991126826 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c096a30 con 0x7fc99c099580 2024-09-18T12:26:57.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc99b7fe640 10 monclient: get_auth_request con 0x7fc99c098c70 auth_method 0 2024-09-18T12:26:57.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.048+0000 7fc99b7fe640 1 --2- 172.21.15.144:0/2991126826 >> v2:172.21.15.116:3300/0 conn(0x7fc99c098c70 0x7fc99c099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc993fff640 1 -- 172.21.15.144:0/2991126826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c099580 msgr2=0x7fc99c09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc993fff640 1 --2- 172.21.15.144:0/2991126826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c099580 0x7fc99c09dad0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc984009910 tx=0x7fc9840091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114afd0fdf2caa04 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 1 -- 172.21.15.144:0/2991126826 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984032440 con 0x7fc99c099580 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: dump: 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:57.060 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: _finish_auth 0 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc9a20a8640 5 monclient: authenticate success, global_id 4552 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc9a20a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:57.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 1 -- 172.21.15.144:0/2991126826 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9840325a0 con 0x7fc99c099580 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 1 -- 172.21.15.144:0/2991126826 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984032940 con 0x7fc99c099580 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc9a20a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc9a20a8640 4 set_mon_vals no callback set 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.051+0000 7fc99affd640 10 monclient: dump: 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:57.062 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 10 monclient: shutdown 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 -- 172.21.15.144:0/2991126826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c099580 msgr2=0x7fc99c09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/2991126826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c099580 0x7fc99c09dad0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc984009910 tx=0x7fc9840091d0 comp rx=0 tx=0).stop 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 -- 172.21.15.144:0/2991126826 shutdown_connections 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/2991126826 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c099580 0x7fc99c09dad0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/2991126826 >> v2:172.21.15.116:3300/0 conn(0x7fc99c098c70 0x7fc99c099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 -- 172.21.15.144:0/2991126826 >> 172.21.15.144:0/2991126826 conn(0x7fc99c070e90 msgr2=0x7fc99c0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 -- 172.21.15.144:0/2991126826 shutdown_connections 2024-09-18T12:26:57.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 1 -- 172.21.15.144:0/2991126826 wait complete. 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 10 monclient: build_initial_monmap 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.052+0000 7fc9a20a8640 10 monclient: monmap: 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:57.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 Processor -- start 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 -- start start 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient: init 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 0x7fc99c141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc99c098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc99c099580 0x7fc99c133e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc99c099580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99c099c70 0x7fc99c11e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc99c099c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c120a00 con 0x7fc99c099c70 2024-09-18T12:26:57.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c122540 con 0x7fc99c098c70 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c06eba0 con 0x7fc99c099580 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc9a20a8640 10 monclient(hunting): authenticate will time out at 852.622610s 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 0x7fc99c141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 0x7fc99c141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57834/0 (socket says 172.21.15.144:57834) 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 -- 172.21.15.144:0/1234215645 learned_addr learned my addr 172.21.15.144:0/1234215645 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): get_auth_request con 0x7fc99c098c70 auth_method 0 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:57.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc993fff640 1 -- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc99c099580 msgr2=0x7fc99c133e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc993fff640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc99c099580 0x7fc99c133e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_done global_id 4555 payload 824 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 -- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc99c099580 msgr2=0x7fc99c133e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc99c099580 0x7fc99c133e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 -- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99c099c70 msgr2=0x7fc99c11e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99c099c70 0x7fc99c11e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:57.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.053+0000 7fc99b7fe640 1 -- 172.21.15.144:0/1234215645 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc984009450 con 0x7fc99c098c70 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc99b7fe640 1 -- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 msgr2=0x7fc99c141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc99b7fe640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 0x7fc99c141000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c8e0 tx=0x7fc98c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc99b7fe640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 0x7fc99c141000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c8e0 tx=0x7fc98c008040 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fc98c007aa0 config(0 keys) v1 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00ce70 con 0x7fc99c098c70 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 10 monclient: dump: 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.068 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9a20a8640 5 monclient: authenticate success, global_id 4555 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98c007aa0 con 0x7fc99c098c70 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9a20a8640 10 monclient: _renew_subs 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9a20a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c122260 con 0x7fc99c098c70 2024-09-18T12:26:57.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9a20a8640 10 monclient: _renew_subs 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc99a7fc640 4 set_mon_vals no callback set 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9a20a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.056+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c1211c0 con 0x7fc99c098c70 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c007e20 con 0x7fc99c098c70 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9937fe640 10 monclient: dump: 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:57.070 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc98c011600 con 0x7fc99c098c70 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9937fe640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc97c038490 0x7fc97c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9a20a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9a20a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9a20a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c0008d0 con 0x7fc99c098c70 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc993fff640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc97c038490 0x7fc97c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:57.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.057+0000 7fc993fff640 10 monclient: get_auth_request con 0x7fc97c038490 auth_method 0 2024-09-18T12:26:57.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.059+0000 7fc993fff640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc97c038490 0x7fc97c03a940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc99c067560 tx=0x7fc984032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:57.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.061+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fc98c04bba0 con 0x7fc99c098c70 2024-09-18T12:26:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.679+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc98c0176c0 con 0x7fc99c098c70 2024-09-18T12:26:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.679+0000 7fc9937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.679+0000 7fc9937fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:57.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.834+0000 7fc9a20a8640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T12:26:57.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.834+0000 7fc9a20a8640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T12:26:57.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.834+0000 7fc9a20a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:57.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:57.834+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc99c064a20 con 0x7fc99c098c70 2024-09-18T12:26:58.372 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:26:58.373 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:26:58.373 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:26:58.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:26:58.376 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:26:58.376 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:26:58.376 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:26:58.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.479+0000 7f26eed9d640 10 monclient: get_monmap_and_config 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.479+0000 7f26eed9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.479+0000 7f26eed9d640 10 monclient: monmap: 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:26:58.480501+0000 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:26:58.480501+0000 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:26:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:26:58.482 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:26:58.482 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:26:58.482 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:26:58.482 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.480+0000 7f26eed9d640 1 Processor -- start 2024-09-18T12:26:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.480+0000 7f26eed9d640 1 -- start start 2024-09-18T12:26:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.480+0000 7f26eed9d640 10 monclient: init 2024-09-18T12:26:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T12:26:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26e8131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:26:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f26e809e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f26e809f220 0x7f26e8138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f26e809f220 addr v2:172.21.15.144:3300/0 2024-09-18T12:26:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e809faf0 con 0x7f26e8131ab0 2024-09-18T12:26:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e8132590 con 0x7f26e809f220 2024-09-18T12:26:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e809fd30 con 0x7f26e809e910 2024-09-18T12:26:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26eed9d640 10 monclient(hunting): authenticate will time out at 845.802167s 2024-09-18T12:26:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ecb39640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f26e8131ab0 0x7f26e8131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49626/0 (socket says 172.21.15.116:49626) 2024-09-18T12:26:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ecb39640 1 -- 172.21.15.116:0/1856497500 learned_addr learned my addr 172.21.15.116:0/1856497500 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:26:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 10 monclient(hunting): get_auth_request con 0x7f26e809e910 auth_method 0 2024-09-18T12:26:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 1 -- 172.21.15.116:0/1856497500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 msgr2=0x7f26e809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 1 -- 172.21.15.116:0/1856497500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 msgr2=0x7f26e809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 1 --2- 172.21.15.116:0/1856497500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26e7fff640 1 --2- 172.21.15.116:0/1856497500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 1 --2- 172.21.15.116:0/1856497500 >> v2:172.21.15.144:3300/0 conn(0x7f26e809f220 0x7f26e8138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): get_auth_request con 0x7f26e809f220 auth_method 0 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient(hunting): handle_auth_done global_id 4558 payload 274 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 1 -- 172.21.15.116:0/1856497500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 msgr2=0x7f26e809ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 1 --2- 172.21.15.116:0/1856497500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 1 -- 172.21.15.116:0/1856497500 >> v1:172.21.15.116:6789/0 conn(0x7f26e8131ab0 legacy=0x7f26e8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient: _finish_hunting 0 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 1 monclient: found mon.noname-c 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.481+0000 7f26ed33a640 1 -- 172.21.15.116:0/1856497500 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80a1110 con 0x7f26e809f220 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26ed33a640 1 -- 172.21.15.116:0/1856497500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 msgr2=0x7f26e8138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26ed33a640 1 --2- 172.21.15.116:0/1856497500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 0x7f26e8138bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009e60 tx=0x7f26e0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=807f45c6c5810e3a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 1 -- 172.21.15.116:0/1856497500 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0007760 con 0x7f26e809f220 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: dump: 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: _finish_auth 0 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26eed9d640 5 monclient: authenticate success, global_id 4558 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26eed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 1 -- 172.21.15.116:0/1856497500 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e002d050 con 0x7f26e809f220 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26eed9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:26:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26eed9d640 4 set_mon_vals no callback set 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 1 -- 172.21.15.116:0/1856497500 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0007b30 con 0x7f26e809f220 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.484+0000 7f26e77fe640 10 monclient: dump: 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:58.502 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 10 monclient: shutdown 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 -- 172.21.15.116:0/1856497500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 msgr2=0x7f26e8138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 --2- 172.21.15.116:0/1856497500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 0x7f26e8138bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009e60 tx=0x7f26e0009780 comp rx=0 tx=0).stop 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 -- 172.21.15.116:0/1856497500 shutdown_connections 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 --2- 172.21.15.116:0/1856497500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 0x7f26e8138bc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 --2- 172.21.15.116:0/1856497500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e809ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 -- 172.21.15.116:0/1856497500 >> 172.21.15.116:0/1856497500 conn(0x7f26e8067d30 msgr2=0x7f26e80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 -- 172.21.15.116:0/1856497500 shutdown_connections 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 1 -- 172.21.15.116:0/1856497500 wait complete. 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 10 monclient: build_initial_monmap 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.485+0000 7f26eed9d640 10 monclient: monmap: 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:26:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 Processor -- start 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 -- start start 2024-09-18T12:26:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient: init 2024-09-18T12:26:58.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:26:58.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:26:58.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e8131140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:58.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f26e809e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:26:58.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 0x7f26e8098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f26e809f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:26:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e8131ab0 0x7f26e8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f26e8131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:26:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8099780 con 0x7f26e8131ab0 2024-09-18T12:26:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8099a60 con 0x7f26e809f220 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8099d40 con 0x7f26e809e910 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.486+0000 7f26eed9d640 10 monclient(hunting): authenticate will time out at 845.807952s 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e8131140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e8131ab0 0x7f26e8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e8131140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51542/0 (socket says 172.21.15.116:51542) 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 1 -- 172.21.15.116:0/280382029 learned_addr learned my addr 172.21.15.116:0/280382029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 10 monclient(hunting): get_auth_request con 0x7f26e809e910 auth_method 0 2024-09-18T12:26:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): get_auth_request con 0x7f26e8131ab0 auth_method 0 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 0x7f26e8098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 10 monclient(hunting): get_auth_request con 0x7f26e809f220 auth_method 0 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 1 -- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 msgr2=0x7f26e8131140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:26:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 1 -- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 msgr2=0x7f26e8131140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e8131140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ecb39640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e8131140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:26:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 1 -- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 msgr2=0x7f26e8131140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e8131140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 1 -- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 msgr2=0x7f26e8098170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 0x7f26e8098170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient: _finish_hunting 0 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 1 monclient: found mon.noname-a 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.487+0000 7f26ed33a640 1 -- 172.21.15.116:0/280382029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0009a20 con 0x7f26e8131ab0 2024-09-18T12:26:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26ed33a640 1 -- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e8131ab0 msgr2=0x7f26e8096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26ed33a640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e8131ab0 0x7f26e8096a60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f26e0031000 tx=0x7f26e0037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 1 -- 172.21.15.116:0/280382029 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0031c10 con 0x7f26e8131ab0 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: dump: 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: _finish_auth 0 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 1 -- 172.21.15.116:0/280382029 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e0040070 con 0x7f26e8131ab0 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26eed9d640 5 monclient: authenticate success, global_id 4599 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26eed9d640 10 monclient: _renew_subs 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26eed9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e809a020 con 0x7f26e8131ab0 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 1 -- 172.21.15.116:0/280382029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0045430 con 0x7f26e8131ab0 2024-09-18T12:26:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e4ff9640 10 monclient: dump: 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:26:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26eed9d640 10 monclient: _renew_subs 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26eed9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e809ab40 con 0x7f26e8131ab0 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.489+0000 7f26e6ffd640 4 set_mon_vals no callback set 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26e4ff9640 1 -- 172.21.15.116:0/280382029 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f26e0042030 con 0x7f26e8131ab0 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26eed9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26eed9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26eed9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.490+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e804bf60 con 0x7f26e8131ab0 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.491+0000 7f26e4ff9640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f26cc038490 0x7f26cc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.491+0000 7f26e4ff9640 1 -- 172.21.15.116:0/280382029 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f26e006d490 con 0x7f26e8131ab0 2024-09-18T12:26:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.491+0000 7f26ecb39640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f26cc038490 0x7f26cc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:26:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.491+0000 7f26ecb39640 10 monclient: get_auth_request con 0x7f26cc038490 auth_method 0 2024-09-18T12:26:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.493+0000 7f26ecb39640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f26cc038490 0x7f26cc03a940 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f26dc007dd0 tx=0x7f26dc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:26:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.495+0000 7f26e4ff9640 1 -- 172.21.15.116:0/280382029 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f26e00449e0 con 0x7f26e8131ab0 2024-09-18T12:26:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.495+0000 7f26e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:26:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.495+0000 7f26e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:26:58.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:58.678+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v35) v1 ==== 152+0+0 (secure 0 0 0) 0x7fc98c0280c0 con 0x7fc99c098c70 2024-09-18T12:26:58.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:58.678+0000 7fc9937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T12:26:58.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:58.678+0000 7fc9937fe640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-18T12:26:58.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.737+0000 7f26eed9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:26:58.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.737+0000 7f26eed9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:26:58.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.737+0000 7f26eed9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:26:58.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:26:58.737+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f26e8066280 con 0x7f26e8131ab0 2024-09-18T12:26:58.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:58.790+0000 7fc9a20a8640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T12:26:58.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:58.790+0000 7fc9a20a8640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T12:26:58.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:58.790+0000 7fc9a20a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:26:58.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:26:58.790+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc99c141540 con 0x7fc99c098c70 2024-09-18T12:27:00.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:00.054+0000 7fc9927fc640 10 monclient: tick 2024-09-18T12:27:00.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:00.054+0000 7fc9927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:00.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:00.054+0000 7fc9927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:01.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:01.053+0000 7fc9927fc640 10 monclient: tick 2024-09-18T12:27:01.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:01.053+0000 7fc9927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:01.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:01.487+0000 7f26d37fe640 10 monclient: tick 2024-09-18T12:27:01.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:01.487+0000 7f26d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:01.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:01.487+0000 7f26d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:02.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:02.053+0000 7fc9927fc640 10 monclient: tick 2024-09-18T12:27:02.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:02.053+0000 7fc9927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:02.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:02.487+0000 7f26d37fe640 10 monclient: tick 2024-09-18T12:27:02.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:02.487+0000 7f26d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:03.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.054+0000 7fc9927fc640 10 monclient: tick 2024-09-18T12:27:03.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.054+0000 7fc9927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:03.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.487+0000 7f26d37fe640 10 monclient: tick 2024-09-18T12:27:03.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.487+0000 7f26d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:03.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.707+0000 7f26e4ff9640 1 -- 172.21.15.116:0/280382029 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f26e006be60 con 0x7f26e8131ab0 2024-09-18T12:27:03.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.708+0000 7f26e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:27:03.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.708+0000 7f26e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:03.709 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.710+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f26cc038490 msgr2=0x7f26cc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.710+0000 7f26eed9d640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f26cc038490 0x7f26cc03a940 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f26dc007dd0 tx=0x7f26dc009040 comp rx=0 tx=0).stop 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.710+0000 7f26eed9d640 10 monclient: shutdown 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.710+0000 7f26eed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.710+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e8131ab0 msgr2=0x7f26e8096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.710+0000 7f26eed9d640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e8131ab0 0x7f26e8096a60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f26e0031000 tx=0x7f26e0037000 comp rx=0 tx=0).stop 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 shutdown_connections 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f26cc038490 0x7f26cc03a940 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e8131ab0 0x7f26e8096a60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f26e809f220 0x7f26e8098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 --2- 172.21.15.116:0/280382029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f26e809e910 0x7f26e8131140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 >> 172.21.15.116:0/280382029 conn(0x7f26e8067d30 msgr2=0x7f26e812cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:03.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 shutdown_connections 2024-09-18T12:27:03.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:03.711+0000 7f26eed9d640 1 -- 172.21.15.116:0/280382029 wait complete. 2024-09-18T12:27:03.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.770+0000 7fc9937fe640 1 -- 172.21.15.144:0/1234215645 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v36) v1 ==== 152+0+0 (secure 0 0 0) 0x7fc98c02b070 con 0x7fc99c098c70 2024-09-18T12:27:03.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.770+0000 7fc9937fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T12:27:03.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.770+0000 7fc9937fe640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-18T12:27:03.772 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 4 size to 1 2024-09-18T12:27:03.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc97c038490 msgr2=0x7fc97c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:03.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc97c038490 0x7fc97c03a940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc99c067560 tx=0x7fc984032000 comp rx=0 tx=0).stop 2024-09-18T12:27:03.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 10 monclient: shutdown 2024-09-18T12:27:03.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:03.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 msgr2=0x7fc99c141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:03.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 0x7fc99c141000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c8e0 tx=0x7fc98c008040 comp rx=0 tx=0).stop 2024-09-18T12:27:03.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 shutdown_connections 2024-09-18T12:27:03.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc97c038490 0x7fc97c03a940 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc99c099c70 0x7fc99c11e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc99c099580 0x7fc99c133e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 --2- 172.21.15.144:0/1234215645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc99c098c70 0x7fc99c141000 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:03.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.784+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 >> 172.21.15.144:0/1234215645 conn(0x7fc99c070e90 msgr2=0x7fc99c06d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:03.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.785+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 shutdown_connections 2024-09-18T12:27:03.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:03.786+0000 7fc9a20a8640 1 -- 172.21.15.144:0/1234215645 wait complete. 2024-09-18T12:27:03.803 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_config_value foo min_size 1 2024-09-18T12:27:03.804 INFO:tasks.workunit.client.0.smithi144.stderr:+ local pool_name config_opt expected_val val 2024-09-18T12:27:03.804 INFO:tasks.workunit.client.0.smithi144.stderr:+ pool_name=foo 2024-09-18T12:27:03.804 INFO:tasks.workunit.client.0.smithi144.stderr:+ config_opt=min_size 2024-09-18T12:27:03.804 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_val=1 2024-09-18T12:27:03.804 INFO:tasks.workunit.client.0.smithi144.stderr:++ get_config_value_or_die foo min_size 2024-09-18T12:27:03.804 INFO:tasks.workunit.client.0.smithi144.stderr:++ local pool_name config_opt raw val 2024-09-18T12:27:03.804 INFO:tasks.workunit.client.0.smithi144.stderr:++ pool_name=foo 2024-09-18T12:27:03.805 INFO:tasks.workunit.client.0.smithi144.stderr:++ config_opt=min_size 2024-09-18T12:27:03.805 INFO:tasks.workunit.client.0.smithi144.stderr:+++ ceph osd pool get foo min_size 2024-09-18T12:27:04.159 INFO:tasks.workunit.client.0.smithi144.stderr:++ raw='min_size: 1' 2024-09-18T12:27:04.159 INFO:tasks.workunit.client.0.smithi144.stderr:++ [[ 0 -ne 0 ]] 2024-09-18T12:27:04.159 INFO:tasks.workunit.client.0.smithi144.stderr:+++ echo min_size: 1 2024-09-18T12:27:04.159 INFO:tasks.workunit.client.0.smithi144.stderr:+++ sed -e 's/[{} "]//g' 2024-09-18T12:27:04.160 INFO:tasks.workunit.client.0.smithi144.stderr:++ raw=min_size:1 2024-09-18T12:27:04.161 INFO:tasks.workunit.client.0.smithi144.stderr:+++ echo min_size:1 2024-09-18T12:27:04.161 INFO:tasks.workunit.client.0.smithi144.stderr:+++ cut -f2 -d: 2024-09-18T12:27:04.162 INFO:tasks.workunit.client.0.smithi144.stderr:++ val=1 2024-09-18T12:27:04.162 INFO:tasks.workunit.client.0.smithi144.stderr:++ echo 1 2024-09-18T12:27:04.162 INFO:tasks.workunit.client.0.smithi144.stderr:++ return 0 2024-09-18T12:27:04.162 INFO:tasks.workunit.client.0.smithi144.stderr:+ val=1 2024-09-18T12:27:04.162 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 1 != \1 ]] 2024-09-18T12:27:04.162 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set foo size 4 2024-09-18T12:27:04.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.236+0000 7fe063a34640 10 monclient: get_monmap_and_config 2024-09-18T12:27:04.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.236+0000 7fe063a34640 10 monclient: build_initial_monmap 2024-09-18T12:27:04.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.236+0000 7fe063a34640 10 monclient: monmap: 2024-09-18T12:27:04.237 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:04.237 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:04.237337+0000 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:04.237337+0000 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:04.238 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:04.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.236+0000 7fe063a34640 1 Processor -- start 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 1 -- start start 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient: init 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe05c09ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe05c09f020 0x7fe05c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c09f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe05c136d60 0x7fe05c139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c136d60 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c067420 con 0x7fe05c09ec50 2024-09-18T12:27:04.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05c09fdc0 con 0x7fe05c09f020 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05c0a0510 con 0x7fe05c136d60 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe063a34640 10 monclient(hunting): authenticate will time out at 859.806727s 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe05c136d60 0x7fe05c139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe0617d0640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe05c09ec50 0x7fe05c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:36714/0 (socket says 172.21.15.144:36714) 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe0617d0640 1 -- 172.21.15.144:0/1359255124 learned_addr learned my addr 172.21.15.144:0/1359255124 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 10 monclient(hunting): get_auth_request con 0x7fe05c136d60 auth_method 0 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:04.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe060fcf640 1 --2- 172.21.15.144:0/1359255124 >> v2:172.21.15.116:3300/0 conn(0x7fe05c09f020 0x7fe05c069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe060fcf640 10 monclient(hunting): get_auth_request con 0x7fe05c09f020 auth_method 0 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe060fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe060fcf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe060fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.237+0000 7fe061fd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe060fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe060fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe060fcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 10 monclient(hunting): handle_auth_done global_id 4570 payload 274 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 1 -- 172.21.15.144:0/1359255124 >> v2:172.21.15.116:3300/0 conn(0x7fe05c09f020 msgr2=0x7fe05c069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:04.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 1 --2- 172.21.15.144:0/1359255124 >> v2:172.21.15.116:3300/0 conn(0x7fe05c09f020 0x7fe05c069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:04.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 1 -- 172.21.15.144:0/1359255124 >> v1:172.21.15.144:6789/0 conn(0x7fe05c09ec50 legacy=0x7fe05c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:04.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 10 monclient: _finish_hunting 0 2024-09-18T12:27:04.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 1 monclient: found mon.noname-c 2024-09-18T12:27:04.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:04.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:04.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.238+0000 7fe061fd1640 1 -- 172.21.15.144:0/1359255124 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0a0200 con 0x7fe05c136d60 2024-09-18T12:27:04.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.240+0000 7fe061fd1640 1 -- 172.21.15.144:0/1359255124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c136d60 msgr2=0x7fe05c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:04.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.240+0000 7fe061fd1640 1 --2- 172.21.15.144:0/1359255124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c136d60 0x7fe05c139140 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe05c132a50 tx=0x7fe054009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cc8a099bc2dfc2a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:04.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 1 -- 172.21.15.144:0/1359255124 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe054007760 con 0x7fe05c136d60 2024-09-18T12:27:04.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:04.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:04.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: dump: 2024-09-18T12:27:04.249 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: _finish_auth 0 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:04.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 5 monclient: authenticate success, global_id 4570 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 1 -- 172.21.15.144:0/1359255124 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05402d050 con 0x7fe05c136d60 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 1 -- 172.21.15.144:0/1359255124 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe054007b30 con 0x7fe05c136d60 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 4 set_mon_vals no callback set 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe053fff640 10 monclient: dump: 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 10 monclient: shutdown 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 -- 172.21.15.144:0/1359255124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c136d60 msgr2=0x7fe05c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 --2- 172.21.15.144:0/1359255124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c136d60 0x7fe05c139140 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe05c132a50 tx=0x7fe054009780 comp rx=0 tx=0).stop 2024-09-18T12:27:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 -- 172.21.15.144:0/1359255124 shutdown_connections 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 --2- 172.21.15.144:0/1359255124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c136d60 0x7fe05c139140 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 --2- 172.21.15.144:0/1359255124 >> v2:172.21.15.116:3300/0 conn(0x7fe05c09f020 0x7fe05c069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 -- 172.21.15.144:0/1359255124 >> 172.21.15.144:0/1359255124 conn(0x7fe05c0a5d40 msgr2=0x7fe05c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 -- 172.21.15.144:0/1359255124 shutdown_connections 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.241+0000 7fe063a34640 1 -- 172.21.15.144:0/1359255124 wait complete. 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.242+0000 7fe063a34640 10 monclient: build_initial_monmap 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.242+0000 7fe063a34640 10 monclient: monmap: 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.242+0000 7fe063a34640 1 Processor -- start 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.242+0000 7fe063a34640 1 -- start start 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.242+0000 7fe063a34640 10 monclient: init 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 0x7fe05c1411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c09ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe05c09f020 0x7fe05c141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe05c136d60 0x7fe05c141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c09b790 con 0x7fe05c136d60 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c09c000 con 0x7fe05c09ec50 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0a3e50 con 0x7fe05c09f020 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe063a34640 10 monclient(hunting): authenticate will time out at 859.812358s 2024-09-18T12:27:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 0x7fe05c1411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 0x7fe05c1411d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40998/0 (socket says 172.21.15.144:40998) 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 -- 172.21.15.144:0/2762267502 learned_addr learned my addr 172.21.15.144:0/2762267502 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): get_auth_request con 0x7fe05c09ec50 auth_method 0 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient(hunting): handle_auth_done global_id 4573 payload 824 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe060fcf640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe05c09f020 0x7fe05c141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:04.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 -- 172.21.15.144:0/2762267502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe05c09f020 msgr2=0x7fe05c141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe05c09f020 0x7fe05c141710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 -- 172.21.15.144:0/2762267502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe05c136d60 msgr2=0x7fe05c141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe05c136d60 0x7fe05c141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient: _finish_hunting 0 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 monclient: found mon.noname-b 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.243+0000 7fe0617d0640 1 -- 172.21.15.144:0/2762267502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe054009a20 con 0x7fe05c09ec50 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.245+0000 7fe0617d0640 1 -- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 msgr2=0x7fe05c1411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.245+0000 7fe0617d0640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 0x7fe05c1411d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00c5e0 tx=0x7fe04c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c005cb0 con 0x7fe05c09ec50 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:04.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: dump: 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe063a34640 5 monclient: authenticate success, global_id 4573 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04c005e30 con 0x7fe05c09ec50 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe063a34640 10 monclient: _renew_subs 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe063a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c11b960 con 0x7fe05c09ec50 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c007340 con 0x7fe05c09ec50 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0517fa640 10 monclient: dump: 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe063a34640 10 monclient: _renew_subs 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe063a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe0537fe640 4 set_mon_vals no callback set 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.246+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c11c8c0 con 0x7fe05c09ec50 2024-09-18T12:27:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe04c01b400 con 0x7fe05c09ec50 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe063a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe063a34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe063a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c0008d0 con 0x7fe05c09ec50 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe0517fa640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe040038490 0x7fe04003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fe04c04b7a0 con 0x7fe05c09ec50 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe060fcf640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe040038490 0x7fe04003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.247+0000 7fe060fcf640 10 monclient: get_auth_request con 0x7fe040038490 auth_method 0 2024-09-18T12:27:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.249+0000 7fe060fcf640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe040038490 0x7fe04003a940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe058009360 tx=0x7fe058008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.251+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe04c01a720 con 0x7fe05c09ec50 2024-09-18T12:27:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.251+0000 7fe0517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.251+0000 7fe0517fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:04.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.406+0000 7fe063a34640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T12:27:04.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.406+0000 7fe063a34640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T12:27:04.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.406+0000 7fe063a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:04.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:04.406+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7fe05c0a13e0 con 0x7fe05c09ec50 2024-09-18T12:27:05.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:05.243+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v37) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe04c04b030 con 0x7fe05c09ec50 2024-09-18T12:27:05.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:05.243+0000 7fe0517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T12:27:05.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:05.243+0000 7fe0517fa640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-18T12:27:05.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:05.357+0000 7fe063a34640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T12:27:05.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:05.357+0000 7fe063a34640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T12:27:05.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:05.357+0000 7fe063a34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:05.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:05.357+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7fe05c072640 con 0x7fe05c09ec50 2024-09-18T12:27:06.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:27:06.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.815+0000 7fe0517fa640 1 -- 172.21.15.144:0/2762267502 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe04c04b020 con 0x7fe05c09ec50 2024-09-18T12:27:06.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.815+0000 7fe0517fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T12:27:06.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.815+0000 7fe0517fa640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-18T12:27:06.820 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 4 size to 4 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.820+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe040038490 msgr2=0x7fe04003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.820+0000 7fe063a34640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe040038490 0x7fe04003a940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe058009360 tx=0x7fe058008040 comp rx=0 tx=0).stop 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 10 monclient: shutdown 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 msgr2=0x7fe05c1411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 0x7fe05c1411d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00c5e0 tx=0x7fe04c004960 comp rx=0 tx=0).stop 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 shutdown_connections 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe040038490 0x7fe04003a940 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe05c136d60 0x7fe05c141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe05c09f020 0x7fe05c141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 --2- 172.21.15.144:0/2762267502 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe05c09ec50 0x7fe05c1411d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 >> 172.21.15.144:0/2762267502 conn(0x7fe05c0a5d40 msgr2=0x7fe05c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:06.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 shutdown_connections 2024-09-18T12:27:06.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:06.821+0000 7fe063a34640 1 -- 172.21.15.144:0/2762267502 wait complete. 2024-09-18T12:27:06.842 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_config_value foo min_size 2 2024-09-18T12:27:06.842 INFO:tasks.workunit.client.0.smithi144.stderr:+ local pool_name config_opt expected_val val 2024-09-18T12:27:06.842 INFO:tasks.workunit.client.0.smithi144.stderr:+ pool_name=foo 2024-09-18T12:27:06.842 INFO:tasks.workunit.client.0.smithi144.stderr:+ config_opt=min_size 2024-09-18T12:27:06.842 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_val=2 2024-09-18T12:27:06.842 INFO:tasks.workunit.client.0.smithi144.stderr:++ get_config_value_or_die foo min_size 2024-09-18T12:27:06.842 INFO:tasks.workunit.client.0.smithi144.stderr:++ local pool_name config_opt raw val 2024-09-18T12:27:06.843 INFO:tasks.workunit.client.0.smithi144.stderr:++ pool_name=foo 2024-09-18T12:27:06.843 INFO:tasks.workunit.client.0.smithi144.stderr:++ config_opt=min_size 2024-09-18T12:27:06.843 INFO:tasks.workunit.client.0.smithi144.stderr:+++ ceph osd pool get foo min_size 2024-09-18T12:27:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.844+0000 7f5a1b99d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.844+0000 7f5a1b99d640 10 monclient: build_initial_monmap 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.844+0000 7f5a1b99d640 10 monclient: monmap: 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:06.845277+0000 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:06.845277+0000 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:06.847 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:06.847 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:06.847 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:06.847 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:06.847 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:06.847 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.845+0000 7f5a1b99d640 1 Processor -- start 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.845+0000 7f5a1b99d640 1 -- start start 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.845+0000 7f5a1b99d640 10 monclient: init 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a1409a510 0x7f5a14096e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f5a1409a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a14097380 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5a140977a0 0x7f5a1412fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient(hunting): picked mon.noname-c con 0x7f5a140977a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1409d150 con 0x7f5a14097380 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1409cb00 con 0x7f5a1409a510 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1409a8e0 con 0x7f5a140977a0 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.846+0000 7f5a1b99d640 10 monclient(hunting): authenticate will time out at 854.167951s 2024-09-18T12:27:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a1409a510 0x7f5a14096e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a1409a510 0x7f5a14096e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55440/0 (socket says 172.21.15.116:55440) 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 1 -- 172.21.15.116:0/3257806771 learned_addr learned my addr 172.21.15.116:0/3257806771 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 10 monclient(hunting): get_auth_request con 0x7f5a1409a510 auth_method 0 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.847+0000 7f5a19739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19f3a640 1 --2- 172.21.15.116:0/3257806771 >> v2:172.21.15.144:3300/0 conn(0x7f5a140977a0 0x7f5a1412fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 10 monclient(hunting): handle_auth_done global_id 4608 payload 274 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 1 -- 172.21.15.116:0/3257806771 >> v2:172.21.15.144:3300/0 conn(0x7f5a140977a0 msgr2=0x7f5a1412fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 1 --2- 172.21.15.116:0/3257806771 >> v2:172.21.15.144:3300/0 conn(0x7f5a140977a0 0x7f5a1412fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 1 -- 172.21.15.116:0/3257806771 >> v1:172.21.15.144:6789/0 conn(0x7f5a14097380 legacy=0x7f5a1412f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 10 monclient: _finish_hunting 0 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 1 monclient: found mon.noname-a 2024-09-18T12:27:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.848+0000 7f5a19739640 1 -- 172.21.15.116:0/3257806771 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a14097c20 con 0x7f5a1409a510 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.851+0000 7f5a19739640 1 -- 172.21.15.116:0/3257806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1409a510 msgr2=0x7f5a14096e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.851+0000 7f5a19739640 1 --2- 172.21.15.116:0/3257806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1409a510 0x7f5a14096e40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5a14068990 tx=0x7f5a0c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa237ca6d70df756 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 1 -- 172.21.15.116:0/3257806771 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c032480 con 0x7f5a1409a510 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: dump: 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: _finish_auth 0 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a1b99d640 5 monclient: authenticate success, global_id 4608 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a1b99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 1 -- 172.21.15.116:0/3257806771 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0325e0 con 0x7f5a1409a510 2024-09-18T12:27:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a1b99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a1b99d640 4 set_mon_vals no callback set 2024-09-18T12:27:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 1 -- 172.21.15.116:0/3257806771 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0329c0 con 0x7f5a1409a510 2024-09-18T12:27:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.852+0000 7f5a0bfff640 10 monclient: dump: 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:06.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 10 monclient: shutdown 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 -- 172.21.15.116:0/3257806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1409a510 msgr2=0x7f5a14096e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/3257806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1409a510 0x7f5a14096e40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5a14068990 tx=0x7f5a0c009840 comp rx=0 tx=0).stop 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 -- 172.21.15.116:0/3257806771 shutdown_connections 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/3257806771 >> v2:172.21.15.144:3300/0 conn(0x7f5a140977a0 0x7f5a1412fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/3257806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1409a510 0x7f5a14096e40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 -- 172.21.15.116:0/3257806771 >> 172.21.15.116:0/3257806771 conn(0x7f5a14067110 msgr2=0x7f5a14067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 -- 172.21.15.116:0/3257806771 shutdown_connections 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.853+0000 7f5a1b99d640 1 -- 172.21.15.116:0/3257806771 wait complete. 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.854+0000 7f5a1b99d640 10 monclient: build_initial_monmap 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.854+0000 7f5a1b99d640 10 monclient: monmap: 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:06.858 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.855+0000 7f5a1b99d640 1 Processor -- start 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.855+0000 7f5a1b99d640 1 -- start start 2024-09-18T12:27:06.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.855+0000 7f5a1b99d640 10 monclient: init 2024-09-18T12:27:06.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:06.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:27:06.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a14097380 0x7f5a1412f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:06.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): picked mon.noname-b con 0x7f5a14097380 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 0x7f5a14121a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): picked mon.noname-a con 0x7f5a140977a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 0x7f5a14121fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1409a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1412e620 con 0x7f5a140977a0 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a14125070 con 0x7f5a14097380 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a14123b30 con 0x7f5a1409a510 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a1b99d640 10 monclient(hunting): authenticate will time out at 854.177411s 2024-09-18T12:27:06.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 0x7f5a14121fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 0x7f5a14121fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36504/0 (socket says 172.21.15.116:36504) 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 1 -- 172.21.15.116:0/1157401968 learned_addr learned my addr 172.21.15.116:0/1157401968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 10 monclient(hunting): get_auth_request con 0x7f5a1409a510 auth_method 0 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 1 -- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 msgr2=0x7f5a14121fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 1 -- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 msgr2=0x7f5a14121fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 0x7f5a14121fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.856+0000 7f5a19f3a640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 0x7f5a14121fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 0x7f5a14121a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): get_auth_request con 0x7f5a140977a0 auth_method 0 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:06.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient(hunting): handle_auth_done global_id 4611 payload 824 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 1 -- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 msgr2=0x7f5a14121fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 0x7f5a14121fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 1 -- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a14097380 msgr2=0x7f5a1412f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a14097380 0x7f5a1412f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient: _finish_hunting 0 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 1 monclient: found mon.noname-a 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:06.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:06.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.857+0000 7f5a18f38640 1 -- 172.21.15.116:0/1157401968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c009ae0 con 0x7f5a140977a0 2024-09-18T12:27:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a18f38640 1 -- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 msgr2=0x7f5a14121a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a18f38640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 0x7f5a14121a60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009fd0 tx=0x7f5a10009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a18f38640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 0x7f5a14121a60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009fd0 tx=0x7f5a10009060 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f5a1000fc40 mon_map magic: 0 v1 2024-09-18T12:27:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a097fa640 1 -- 172.21.15.116:0/1157401968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000f720 con 0x7f5a140977a0 2024-09-18T12:27:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:27:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a097fa640 10 monclient: dump: 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.860+0000 7f5a097fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:06.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a1b99d640 5 monclient: authenticate success, global_id 4611 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a1b99d640 10 monclient: _renew_subs 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a1b99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a14125d30 con 0x7f5a140977a0 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a1b99d640 10 monclient: _renew_subs 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a1b99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.861+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a141235b0 con 0x7f5a140977a0 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.862+0000 7f5a097fa640 1 -- 172.21.15.116:0/1157401968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1000f880 con 0x7f5a140977a0 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.862+0000 7f5a097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.862+0000 7f5a0affd640 4 set_mon_vals no callback set 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.862+0000 7f5a097fa640 1 -- 172.21.15.116:0/1157401968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000fc40 con 0x7f5a140977a0 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.862+0000 7f5a097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.862+0000 7f5a097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.862+0000 7f5a097fa640 10 monclient: dump: 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.863+0000 7f5a097fa640 1 -- 172.21.15.116:0/1157401968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5a10011610 con 0x7f5a140977a0 2024-09-18T12:27:06.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.863+0000 7f5a097fa640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5a000384e0 0x7f5a0003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:06.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.863+0000 7f5a1b99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:06.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.863+0000 7f5a1b99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:06.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.863+0000 7f5a1b99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:06.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.863+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f4005010 con 0x7f5a140977a0 2024-09-18T12:27:06.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.863+0000 7f5a19739640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5a000384e0 0x7f5a0003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:06.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.864+0000 7f5a19739640 10 monclient: get_auth_request con 0x7f5a000384e0 auth_method 0 2024-09-18T12:27:06.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.871+0000 7f5a19739640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5a000384e0 0x7f5a0003a990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c034000 tx=0x7f5a0c037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:06.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:06.872+0000 7f5a097fa640 1 -- 172.21.15.116:0/1157401968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f5a1004b510 con 0x7f5a140977a0 2024-09-18T12:27:07.219 INFO:tasks.workunit.client.0.smithi144.stderr:++ raw='min_size: 2' 2024-09-18T12:27:07.219 INFO:tasks.workunit.client.0.smithi144.stderr:++ [[ 0 -ne 0 ]] 2024-09-18T12:27:07.219 INFO:tasks.workunit.client.0.smithi144.stderr:+++ echo min_size: 2 2024-09-18T12:27:07.219 INFO:tasks.workunit.client.0.smithi144.stderr:+++ sed -e 's/[{} "]//g' 2024-09-18T12:27:07.220 INFO:tasks.workunit.client.0.smithi144.stderr:++ raw=min_size:2 2024-09-18T12:27:07.221 INFO:tasks.workunit.client.0.smithi144.stderr:+++ echo min_size:2 2024-09-18T12:27:07.221 INFO:tasks.workunit.client.0.smithi144.stderr:+++ cut -f2 -d: 2024-09-18T12:27:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:++ val=2 2024-09-18T12:27:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:++ echo 2 2024-09-18T12:27:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:++ return 0 2024-09-18T12:27:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:+ val=2 2024-09-18T12:27:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 2 != \2 ]] 2024-09-18T12:27:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set foo size 10 2024-09-18T12:27:07.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.296+0000 7fe0f18d4640 10 monclient: get_monmap_and_config 2024-09-18T12:27:07.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.296+0000 7fe0f18d4640 10 monclient: build_initial_monmap 2024-09-18T12:27:07.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.296+0000 7fe0f18d4640 10 monclient: monmap: 2024-09-18T12:27:07.297 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:07.297 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:07.297053+0000 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:07.297053+0000 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:07.298 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:07.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.296+0000 7fe0f18d4640 1 Processor -- start 2024-09-18T12:27:07.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.296+0000 7fe0f18d4640 1 -- start start 2024-09-18T12:27:07.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.296+0000 7fe0f18d4640 10 monclient: init 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0ec09ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe0ec09f020 0x7fe0ec069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ec09f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0ec136d60 0x7fe0ec139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ec136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec067420 con 0x7fe0ec09ec50 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec09fdc0 con 0x7fe0ec136d60 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec0a0510 con 0x7fe0ec09f020 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0f18d4640 10 monclient(hunting): authenticate will time out at 862.866254s 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe0ec09f020 0x7fe0ec069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eaffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe0ec09ec50 0x7fe0ec131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:35978/0 (socket says 172.21.15.144:35978) 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe0ec09f020 0x7fe0ec069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51438/0 (socket says 172.21.15.144:51438) 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eaffd640 1 -- 172.21.15.144:0/1021548212 learned_addr learned my addr 172.21.15.144:0/1021548212 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): get_auth_request con 0x7fe0ec09f020 auth_method 0 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 1 --2- 172.21.15.144:0/1021548212 >> v2:172.21.15.116:3300/0 conn(0x7fe0ec136d60 0x7fe0ec139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 10 monclient(hunting): get_auth_request con 0x7fe0ec136d60 auth_method 0 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:07.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_done global_id 4579 payload 274 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 1 -- 172.21.15.144:0/1021548212 >> v2:172.21.15.116:3300/0 conn(0x7fe0ec136d60 msgr2=0x7fe0ec139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 1 --2- 172.21.15.144:0/1021548212 >> v2:172.21.15.116:3300/0 conn(0x7fe0ec136d60 0x7fe0ec139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 1 -- 172.21.15.144:0/1021548212 >> v1:172.21.15.144:6789/0 conn(0x7fe0ec09ec50 legacy=0x7fe0ec131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 1 monclient: found mon.noname-c 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.297+0000 7fe0e27fc640 1 -- 172.21.15.144:0/1021548212 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec0a0200 con 0x7fe0ec09f020 2024-09-18T12:27:07.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0e27fc640 1 -- 172.21.15.144:0/1021548212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 msgr2=0x7fe0ec069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0e27fc640 1 --2- 172.21.15.144:0/1021548212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec069d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0a2b00 tx=0x7fe0d40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce90d1d1d13e8114 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 1 -- 172.21.15.144:0/1021548212 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d402adf0 con 0x7fe0ec09f020 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: dump: 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:07.307 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 1 -- 172.21.15.144:0/1021548212 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d4032440 con 0x7fe0ec09f020 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 1 -- 172.21.15.144:0/1021548212 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40327e0 con 0x7fe0ec09f020 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0f18d4640 5 monclient: authenticate success, global_id 4579 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0f18d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0f18d4640 4 set_mon_vals no callback set 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:07.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.300+0000 7fe0ea7fc640 10 monclient: dump: 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 10 monclient: shutdown 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 -- 172.21.15.144:0/1021548212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 msgr2=0x7fe0ec069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:07.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/1021548212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec069d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0a2b00 tx=0x7fe0d40090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 -- 172.21.15.144:0/1021548212 shutdown_connections 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/1021548212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec069d10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/1021548212 >> v2:172.21.15.116:3300/0 conn(0x7fe0ec136d60 0x7fe0ec139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 -- 172.21.15.144:0/1021548212 >> 172.21.15.144:0/1021548212 conn(0x7fe0ec0a5d40 msgr2=0x7fe0ec0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 -- 172.21.15.144:0/1021548212 shutdown_connections 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 1 -- 172.21.15.144:0/1021548212 wait complete. 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 10 monclient: build_initial_monmap 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.301+0000 7fe0f18d4640 10 monclient: monmap: 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:07.310 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:07.311 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:07.311 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:07.311 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:07.311 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 Processor -- start 2024-09-18T12:27:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 -- start start 2024-09-18T12:27:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient: init 2024-09-18T12:27:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:27:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 0x7fe0ec097500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ec09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec097a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ec09f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec136d60 0x7fe0ec097f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ec136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec09a9c0 con 0x7fe0ec136d60 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec09c000 con 0x7fe0ec09f020 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec09d880 con 0x7fe0ec09ec50 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0f18d4640 10 monclient(hunting): authenticate will time out at 862.871695s 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec097a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51444/0 (socket says 172.21.15.144:51444) 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 1 -- 172.21.15.144:0/665648831 learned_addr learned my addr 172.21.15.144:0/665648831 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 10 monclient(hunting): get_auth_request con 0x7fe0ec09f020 auth_method 0 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eb7fe640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec136d60 0x7fe0ec097f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eaffd640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 0x7fe0ec097500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eb7fe640 10 monclient(hunting): get_auth_request con 0x7fe0ec136d60 auth_method 0 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eaffd640 10 monclient(hunting): get_auth_request con 0x7fe0ec09ec50 auth_method 0 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.302+0000 7fe0eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 10 monclient(hunting): handle_auth_done global_id 4582 payload 824 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0eaffd640 1 -- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 msgr2=0x7fe0ec097500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 1 -- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 msgr2=0x7fe0ec097500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0eaffd640 1 -- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 msgr2=0x7fe0ec097500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0eaffd640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 0x7fe0ec097500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:07.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0eaffd640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 0x7fe0ec097500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 0x7fe0ec097500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 1 -- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec136d60 msgr2=0x7fe0ec097f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec136d60 0x7fe0ec097f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 1 monclient: found mon.noname-b 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0e27fc640 1 -- 172.21.15.144:0/665648831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d4009390 con 0x7fe0ec09f020 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.303+0000 7fe0eb7fe640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec136d60 0x7fe0ec097f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e27fc640 1 -- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 msgr2=0x7fe0ec097a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e27fc640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec097a40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009880 tx=0x7fe0d4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4031a10 con 0x7fe0ec09f020 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:07.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: dump: 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d4031b70 con 0x7fe0ec09f020 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0f18d4640 5 monclient: authenticate success, global_id 4582 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d403d420 con 0x7fe0ec09f020 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0f18d4640 10 monclient: _renew_subs 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0f18d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec099ca0 con 0x7fe0ec09f020 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e9ffb640 4 set_mon_vals no callback set 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0e37fe640 10 monclient: dump: 2024-09-18T12:27:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0f18d4640 10 monclient: _renew_subs 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0f18d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.305+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec098e90 con 0x7fe0ec09f020 2024-09-18T12:27:07.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe0d4039040 con 0x7fe0ec09f020 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0f18d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0f18d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0f18d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0e37fe640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0cc0384e0 0x7fe0cc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec0008d0 con 0x7fe0ec09f020 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fe0d406b9c0 con 0x7fe0ec09f020 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0eaffd640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0cc0384e0 0x7fe0cc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.306+0000 7fe0eaffd640 10 monclient: get_auth_request con 0x7fe0cc0384e0 auth_method 0 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.308+0000 7fe0eaffd640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0cc0384e0 0x7fe0cc03a990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc007d10 tx=0x7fe0dc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.311+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe0d4030b30 con 0x7fe0ec09f020 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.311+0000 7fe0e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.311+0000 7fe0e37fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.467+0000 7fe0f18d4640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T12:27:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.467+0000 7fe0f18d4640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T12:27:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.467+0000 7fe0f18d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.467+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7fe0ec0a13e0 con 0x7fe0ec09f020 2024-09-18T12:27:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.483+0000 7f5a097fa640 1 -- 172.21.15.116:0/1157401968 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5a10017a90 con 0x7f5a140977a0 2024-09-18T12:27:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.483+0000 7f5a097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.483+0000 7f5a097fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.695+0000 7f5a1b99d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:27:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.695+0000 7f5a1b99d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:27:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.695+0000 7f5a1b99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.695+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f59f4005840 con 0x7f5a140977a0 2024-09-18T12:27:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.695+0000 7f5a097fa640 1 -- 172.21.15.116:0/1157401968 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f5a10028620 con 0x7f5a140977a0 2024-09-18T12:27:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.695+0000 7f5a097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:27:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.695+0000 7f5a097fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:07.697 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5a000384e0 msgr2=0x7f5a0003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5a000384e0 0x7f5a0003a990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c034000 tx=0x7f5a0c037000 comp rx=0 tx=0).stop 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 10 monclient: shutdown 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 msgr2=0x7f5a14121a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 0x7f5a14121a60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009fd0 tx=0x7f5a10009060 comp rx=0 tx=0).stop 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 shutdown_connections 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5a000384e0 0x7f5a0003a990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5a1409a510 0x7f5a14121fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a140977a0 0x7f5a14121a60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 --2- 172.21.15.116:0/1157401968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a14097380 0x7f5a1412f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 >> 172.21.15.116:0/1157401968 conn(0x7f5a14067110 msgr2=0x7f5a14098ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:07.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 shutdown_connections 2024-09-18T12:27:07.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.697+0000 7f5a1b99d640 1 -- 172.21.15.116:0/1157401968 wait complete. 2024-09-18T12:27:07.708 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:27:07.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:27:07.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:27:07.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.800+0000 7f4a268a8640 10 monclient: get_monmap_and_config 2024-09-18T12:27:07.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.801+0000 7f4a268a8640 10 monclient: build_initial_monmap 2024-09-18T12:27:07.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.801+0000 7f4a268a8640 10 monclient: monmap: 2024-09-18T12:27:07.802 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:07.802 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:07.802 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:07.801677+0000 2024-09-18T12:27:07.802 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:07.801677+0000 2024-09-18T12:27:07.802 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:07.803 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:07.803 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:07.803 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:07.803 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:07.803 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:07.803 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:07.803 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.801+0000 7f4a268a8640 1 Processor -- start 2024-09-18T12:27:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.801+0000 7f4a268a8640 1 -- start start 2024-09-18T12:27:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.801+0000 7f4a268a8640 10 monclient: init 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a20131980 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a2009ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4a2009fe10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a2009ef70 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200a14d0 con 0x7f4a20131980 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a201325b0 con 0x7f4a2009ef70 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200a11f0 con 0x7f4a2009fe10 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a268a8640 10 monclient(hunting): authenticate will time out at 855.123339s 2024-09-18T12:27:07.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a2009ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4a20131980 0x7f4a20131d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44954/0 (socket says 172.21.15.116:44954) 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1ffff640 1 -- 172.21.15.116:0/815708904 learned_addr learned my addr 172.21.15.116:0/815708904 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f4a2009fe10 auth_method 0 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 msgr2=0x7f4a2009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 msgr2=0x7f4a2009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a2009ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.802+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a2009ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1ffff640 1 -- 172.21.15.116:0/815708904 >> v1:172.21.15.116:6789/0 conn(0x7f4a20131980 legacy=0x7f4a20131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810468210 0 0) 0x7f4a200a14d0 con 0x7f4a20131980 2024-09-18T12:27:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient(hunting): my global_id is 4626 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a080034e0 con 0x7f4a20131980 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a24e45640 1 -- 172.21.15.116:0/815708904 >> v1:172.21.15.144:6789/0 conn(0x7f4a2009ef70 legacy=0x7f4a20138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337449765 0 0) 0x7f4a080034e0 con 0x7f4a20131980 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 msgr2=0x7f4a2009ea30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a2009ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 >> v1:172.21.15.144:6789/0 conn(0x7f4a2009ef70 legacy=0x7f4a20138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2009f390 con 0x7f4a20131980 2024-09-18T12:27:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient: _finish_auth 0 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f4a10001860 con 0x7f4a20131980 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a268a8640 5 monclient: authenticate success, global_id 4626 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a268a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.804+0000 7f4a1effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient: dump: 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 >> v1:172.21.15.116:6789/0 conn(0x7f4a20131980 legacy=0x7f4a20131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ad0 0x7f4a08003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): picked mon.b con 0x7f4a08003ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): picked mon.c con 0x7f4a2009fe10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a08004e10 0x7f4a0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): picked mon.a con 0x7f4a08004e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200a11f0 con 0x7f4a08004e10 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080055f0 con 0x7f4a08003ad0 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0800bf40 con 0x7f4a2009fe10 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1effd640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a268a8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a08004e10 0x7f4a0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f4a2009fe10 auth_method 0 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 10 monclient(hunting): get_auth_request con 0x7f4a08004e10 auth_method 0 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ad0 0x7f4a08003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request con 0x7f4a08003ad0 auth_method 0 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_done global_id 4626 payload 293 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ad0 msgr2=0x7f4a08003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ad0 0x7f4a08003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a08004e10 msgr2=0x7f4a0800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:07.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a08004e10 0x7f4a0800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 1 monclient: found mon.c 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a080034e0 con 0x7f4a2009fe10 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.805+0000 7f4a24e45640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a08004e10 0x7f4a0800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.806+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 msgr2=0x7f4a08004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.806+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 msgr2=0x7f4a08004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.806+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f4a14006fd0 tx=0x7f4a14008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:07.806+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:07.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.830+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v39) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe0d4042020 con 0x7fe0ec09f020 2024-09-18T12:27:07.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.831+0000 7fe0e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T12:27:07.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.831+0000 7fe0e37fe640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-18T12:27:07.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.958+0000 7fe0f18d4640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T12:27:07.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.959+0000 7fe0f18d4640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T12:27:07.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.959+0000 7fe0f18d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:07.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:07.959+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7fe0ec0a13e0 con 0x7fe0ec09f020 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient: get_auth_request con 0x7f4a2009fe10 auth_method 0 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 0 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 1 -- 172.21.15.116:0/815708904 reap_dead start 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 msgr2=0x7f4a08004780 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a2009fe10 0x7f4a08004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a080099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): picked mon.b con 0x7f4a08003ee0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800a090 0x7f4a0800a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): picked mon.a con 0x7f4a0800a090 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800aba0 0x7f4a0800b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): picked mon.c con 0x7f4a0800aba0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0800b8f0 con 0x7f4a0800a090 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0800ba60 con 0x7f4a08003ee0 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0800bc20 con 0x7f4a0800aba0 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1effd640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800aba0 0x7f4a0800b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800a090 0x7f4a0800a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 10 monclient(hunting): get_auth_request con 0x7f4a0800aba0 auth_method 0 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f4a0800a090 auth_method 0 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a080099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request con 0x7f4a08003ee0 auth_method 0 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient(hunting): handle_auth_done global_id 4626 payload 293 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800aba0 msgr2=0x7f4a0800b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800aba0 0x7f4a0800b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 msgr2=0x7f4a080099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a080099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 monclient: found mon.a 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f4a200a14d0 con 0x7f4a0800a090 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a24e45640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800aba0 0x7f4a0800b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.007+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a080099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800a090 msgr2=0x7f4a0800a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800a090 0x7f4a0800a460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4a14006fd0 tx=0x7f4a14008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8da54146861fa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14034500 con 0x7f4a0800a090 2024-09-18T12:27:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1effd640 10 monclient: dump: 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1effd640 1 -- 172.21.15.116:0/815708904 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1402e070 con 0x7f4a0800a090 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a1effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 4 set_mon_vals no callback set 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 10 monclient: shutdown 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 -- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800a090 msgr2=0x7f4a0800a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800a090 0x7f4a0800a460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4a14006fd0 tx=0x7f4a14008040 comp rx=0 tx=0).stop 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 -- 172.21.15.116:0/815708904 shutdown_connections 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800aba0 0x7f4a0800b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800a090 0x7f4a0800a460 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 --2- 172.21.15.116:0/815708904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a080099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 -- 172.21.15.116:0/815708904 >> 172.21.15.116:0/815708904 conn(0x7f4a200a5050 msgr2=0x7f4a200a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 -- 172.21.15.116:0/815708904 shutdown_connections 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.010+0000 7f4a268a8640 1 -- 172.21.15.116:0/815708904 wait complete. 2024-09-18T12:27:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.011+0000 7f4a268a8640 10 monclient: build_initial_monmap 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.011+0000 7f4a268a8640 10 monclient: monmap: 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.011+0000 7f4a268a8640 1 Processor -- start 2024-09-18T12:27:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.011+0000 7f4a268a8640 1 -- start start 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.011+0000 7f4a268a8640 10 monclient: init 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a2009a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): picked mon.noname-b con 0x7f4a08003ee0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 0x7f4a20130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0800a090 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800aba0 0x7f4a2011b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0800aba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a20069eb0 con 0x7f4a0800aba0 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2009e1c0 con 0x7f4a08003ee0 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2011dd40 con 0x7f4a0800a090 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a268a8640 10 monclient(hunting): authenticate will time out at 855.333421s 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 0x7f4a20130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800aba0 0x7f4a2011b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 0x7f4a20130940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36556/0 (socket says 172.21.15.116:36556) 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/3592298804 learned_addr learned my addr 172.21.15.116:0/3592298804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f4a0800a090 auth_method 0 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): get_auth_request con 0x7f4a0800aba0 auth_method 0 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 msgr2=0x7f4a20130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 msgr2=0x7f4a20130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 0x7f4a20130940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 0x7f4a20130940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a2009a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request con 0x7f4a08003ee0 auth_method 0 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient(hunting): handle_auth_done global_id 4629 payload 824 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 msgr2=0x7f4a20130940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a0800a090 0x7f4a20130940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 msgr2=0x7f4a2009a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a08003ee0 0x7f4a2009a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 1 monclient: found mon.noname-a 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.012+0000 7f4a24e45640 1 -- 172.21.15.116:0/3592298804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2009b2f0 con 0x7f4a0800aba0 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.014+0000 7f4a24e45640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800aba0 msgr2=0x7f4a2011b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.014+0000 7f4a24e45640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800aba0 0x7f4a2011b6b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a200a60a0 tx=0x7f4a18009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a180070d0 con 0x7f4a0800aba0 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: dump: 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: _finish_auth 0 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 5 monclient: authenticate success, global_id 4629 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a18007230 con 0x7f4a0800aba0 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 10 monclient: _renew_subs 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2011da60 con 0x7f4a0800aba0 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a180075f0 con 0x7f4a0800aba0 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a07fff640 10 monclient: dump: 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.031 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 10 monclient: _renew_subs 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a1e7fc640 4 set_mon_vals no callback set 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a201360c0 con 0x7f4a0800aba0 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 0x7f4a2011bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.015+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a20065e00 con 0x7f4a20130e80 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4a18015440 con 0x7f4a0800aba0 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 0x7f4a2011bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient: get_auth_request con 0x7f4a20130e80 auth_method 0 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a07fff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a0c038490 0x7f4a0c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1ffff640 1 -- 172.21.15.116:0/3592298804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a201362b0 con 0x7f4a20130e80 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f4a1804b610 con 0x7f4a0800aba0 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a0c038490 0x7f4a0c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.016+0000 7f4a1f7fe640 10 monclient: get_auth_request con 0x7f4a0c038490 auth_method 0 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a1ffff640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 msgr2=0x7f4a2011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 0x7f4a2011bbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a10001ed0 tx=0x7f4a10009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10012d20 con 0x7f4a20130e80 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 10 monclient: dump: 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4a10012e80 con 0x7f4a20130e80 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 msgr2=0x7f4a2011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 0x7f4a2011bbf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a10001ed0 tx=0x7f4a10009040 comp rx=0 tx=0).stop 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 msgr2=0x7f4a2011bbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.018+0000 7f4a07fff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a20130e80 0x7f4a2011bbf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.019+0000 7f4a1f7fe640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a0c038490 0x7f4a0c03a940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a1400b280 tx=0x7f4a14037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a268a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a268a8640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 0x7f4a20096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a268a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a20131250 con 0x7f4a2011c130 2024-09-18T12:27:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 0x7f4a20096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient: get_auth_request con 0x7f4a2011c130 auth_method 0 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.024+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_done global_id 4635 payload 824 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.025+0000 7f4a1ffff640 1 -- 172.21.15.116:0/3592298804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4a2011c530 con 0x7f4a2011c130 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a1ffff640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 msgr2=0x7f4a20096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a1ffff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 0x7f4a20096a60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4a10001ed0 tx=0x7f4a10018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1001fe00 con 0x7f4a2011c130 2024-09-18T12:27:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 10 monclient: dump: 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f4a10012b10 con 0x7f4a2011c130 2024-09-18T12:27:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 msgr2=0x7f4a20096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 0x7f4a20096a60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4a10001ed0 tx=0x7f4a10018000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 msgr2=0x7f4a20096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a07fff640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2011c130 0x7f4a20096a60 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4a10001ed0 tx=0x7f4a10018000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:08.040 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 24, 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.041 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.042 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:08.043 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:08.044 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.046 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.047 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:08.049 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.052 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:27:08.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.053 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.053 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:08.053 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a0c038490 msgr2=0x7f4a0c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a0c038490 0x7f4a0c03a940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a1400b280 tx=0x7f4a14037000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 10 monclient: shutdown 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800aba0 msgr2=0x7f4a2011b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a24e45640 1 -- 172.21.15.116:0/3592298804 reap_dead start 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800aba0 0x7f4a2011b6b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a200a60a0 tx=0x7f4a18009040 comp rx=0 tx=0).stop 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 shutdown_connections 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 1 --2- 172.21.15.116:0/3592298804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0800aba0 0x7f4a2011b6b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.027+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 >> 172.21.15.116:0/3592298804 conn(0x7f4a200a5050 msgr2=0x7f4a201349c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.028+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 shutdown_connections 2024-09-18T12:27:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.028+0000 7f4a268a8640 1 -- 172.21.15.116:0/3592298804 wait complete. 2024-09-18T12:27:08.056 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:27:08.056 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:27:08.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:27:08.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.148+0000 7f2b8b84f640 10 monclient: get_monmap_and_config 2024-09-18T12:27:08.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.148+0000 7f2b8b84f640 10 monclient: build_initial_monmap 2024-09-18T12:27:08.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.148+0000 7f2b8b84f640 10 monclient: monmap: 2024-09-18T12:27:08.149 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:08.149 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.149 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:08.149142+0000 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:08.149142+0000 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:08.150 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 1 Processor -- start 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 1 -- start start 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient: init 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2b84131980 0x7f2b84131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b84131980 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b8409fda0 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2b84132290 0x7f2b8409f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b84132290 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:08.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b840a1180 con 0x7f2b8409fda0 2024-09-18T12:27:08.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b840a1460 con 0x7f2b84132290 2024-09-18T12:27:08.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b84139cd0 con 0x7f2b84131980 2024-09-18T12:27:08.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b8b84f640 10 monclient(hunting): authenticate will time out at 855.470782s 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b89dec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2b84132290 0x7f2b8409f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b89dec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2b84132290 0x7f2b8409f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55498/0 (socket says 172.21.15.116:55498) 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b89dec640 1 -- 172.21.15.116:0/3429520620 learned_addr learned my addr 172.21.15.116:0/3429520620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b89dec640 10 monclient(hunting): get_auth_request con 0x7f2b84132290 auth_method 0 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b89dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b89dec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b89dec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.149+0000 7f2b88dea640 1 --1- 172.21.15.116:0/3429520620 >> v1:172.21.15.116:6789/0 conn(0x7f2b8409fda0 0x7f2b8409e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44956/0 (socket says 172.21.15.116:44956) 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 10 monclient(hunting): handle_auth_done global_id 4638 payload 274 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b895eb640 1 --2- 172.21.15.116:0/3429520620 >> v2:172.21.15.144:3300/0 conn(0x7f2b84131980 0x7f2b84131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 1 -- 172.21.15.116:0/3429520620 >> v2:172.21.15.144:3300/0 conn(0x7f2b84131980 msgr2=0x7f2b84131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 1 --2- 172.21.15.116:0/3429520620 >> v2:172.21.15.144:3300/0 conn(0x7f2b84131980 0x7f2b84131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 1 -- 172.21.15.116:0/3429520620 >> v1:172.21.15.116:6789/0 conn(0x7f2b8409fda0 legacy=0x7f2b8409e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 1 monclient: found mon.noname-a 2024-09-18T12:27:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.150+0000 7f2b89dec640 1 -- 172.21.15.116:0/3429520620 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84139e10 con 0x7f2b84132290 2024-09-18T12:27:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.152+0000 7f2b89dec640 1 -- 172.21.15.116:0/3429520620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84132290 msgr2=0x7f2b8409f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.152+0000 7f2b89dec640 1 --2- 172.21.15.116:0/3429520620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84132290 0x7f2b8409f0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2b840a4fb0 tx=0x7f2b7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc15ce8426fe9ffc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 1 -- 172.21.15.116:0/3429520620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c007610 con 0x7f2b84132290 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: dump: 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.170 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: _finish_auth 0 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 1 -- 172.21.15.116:0/3429520620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c02d050 con 0x7f2b84132290 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 5 monclient: authenticate success, global_id 4638 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 4 set_mon_vals no callback set 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 1 -- 172.21.15.116:0/3429520620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0079a0 con 0x7f2b84132290 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b7bfff640 10 monclient: dump: 2024-09-18T12:27:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 10 monclient: shutdown 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 -- 172.21.15.116:0/3429520620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84132290 msgr2=0x7f2b8409f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 --2- 172.21.15.116:0/3429520620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84132290 0x7f2b8409f0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2b840a4fb0 tx=0x7f2b7c009840 comp rx=0 tx=0).stop 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 -- 172.21.15.116:0/3429520620 shutdown_connections 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 --2- 172.21.15.116:0/3429520620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84132290 0x7f2b8409f0d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 --2- 172.21.15.116:0/3429520620 >> v2:172.21.15.144:3300/0 conn(0x7f2b84131980 0x7f2b84131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 -- 172.21.15.116:0/3429520620 >> 172.21.15.116:0/3429520620 conn(0x7f2b84067bd0 msgr2=0x7f2b840a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 -- 172.21.15.116:0/3429520620 shutdown_connections 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.153+0000 7f2b8b84f640 1 -- 172.21.15.116:0/3429520620 wait complete. 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.154+0000 7f2b8b84f640 10 monclient: build_initial_monmap 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.154+0000 7f2b8b84f640 10 monclient: monmap: 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.154+0000 7f2b8b84f640 1 Processor -- start 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.154+0000 7f2b8b84f640 1 -- start start 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.154+0000 7f2b8b84f640 10 monclient: init 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 0x7f2b84137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b8409fda0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b84133660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): picked mon.noname-b con 0x7f2b84131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b84133ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b84132290 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b84136f20 con 0x7f2b8409fda0 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8409bf10 con 0x7f2b84131980 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840a4730 con 0x7f2b84132290 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b8b84f640 10 monclient(hunting): authenticate will time out at 855.476203s 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 0x7f2b84137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 0x7f2b84137ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55504/0 (socket says 172.21.15.116:55504) 2024-09-18T12:27:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b84133ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 learned_addr learned my addr 172.21.15.116:0/1681037634 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): get_auth_request con 0x7f2b8409fda0 auth_method 0 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 10 monclient(hunting): get_auth_request con 0x7f2b84132290 auth_method 0 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b84133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 10 monclient(hunting): get_auth_request con 0x7f2b84131980 auth_method 0 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 msgr2=0x7f2b84133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 msgr2=0x7f2b84133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b84133ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b84133ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient(hunting): handle_auth_done global_id 4641 payload 824 2024-09-18T12:27:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 msgr2=0x7f2b84133ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b84133ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 msgr2=0x7f2b84133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b84133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 monclient: found mon.noname-a 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.155+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c009ae0 con 0x7f2b8409fda0 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 msgr2=0x7f2b84137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 0x7f2b84137ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c8b0 tx=0x7f2b80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 msgr2=0x7f2b84137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 msgr2=0x7f2b84137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 0x7f2b84137ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c8b0 tx=0x7f2b80008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 0x7f2b84137ba0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c8b0 tx=0x7f2b80008040 comp rx=0 tx=0).stop 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 msgr2=0x7f2b84137ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b8409fda0 0x7f2b84137ba0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b680054c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2b84132290 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b68007eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2b84131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2b68008880 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:08.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6800b800 con 0x7f2b68008880 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6800b940 con 0x7f2b84131980 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6800bb00 con 0x7f2b84132290 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.157+0000 7f2b797fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b680054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): get_auth_request con 0x7f2b84132290 auth_method 0 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 10 monclient(hunting): get_auth_request con 0x7f2b68008880 auth_method 0 2024-09-18T12:27:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b68007eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b88dea640 10 monclient(hunting): get_auth_request con 0x7f2b84131980 auth_method 0 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b88dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b88dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b88dea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient(hunting): handle_auth_done global_id 4641 payload 293 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 msgr2=0x7f2b68007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b68007eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 msgr2=0x7f2b6800ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 1 monclient: found mon.noname-c 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6800bd80 con 0x7f2b84132290 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.158+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 msgr2=0x7f2b680054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b680054c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c880 tx=0x7f2b80019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 msgr2=0x7f2b680054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 msgr2=0x7f2b680054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b680054c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c880 tx=0x7f2b80019000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b680054c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c880 tx=0x7f2b80019000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 msgr2=0x7f2b680054c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84132290 0x7f2b680054c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2b68008880 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b8409fda0 0x7f2b6800e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2b8409fda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b68007eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2b84131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6800eed0 con 0x7f2b68008880 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6800f010 con 0x7f2b84131980 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6800f1a0 con 0x7f2b8409fda0 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b797fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b8409fda0 0x7f2b6800e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): get_auth_request con 0x7f2b68008880 auth_method 0 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 10 monclient(hunting): get_auth_request con 0x7f2b8409fda0 auth_method 0 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b68007eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient(hunting): handle_auth_done global_id 4641 payload 293 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b8409fda0 msgr2=0x7f2b6800e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b8409fda0 0x7f2b6800e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 msgr2=0x7f2b68007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b68007eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 monclient: found mon.noname-a 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6800f360 con 0x7f2b68008880 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b89dec640 10 monclient: get_auth_request con 0x7f2b84131980 auth_method 0 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.160+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2b84131980 0x7f2b68007eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:27:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b895eb640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 msgr2=0x7f2b6800ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b895eb640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ac60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800bc90 tx=0x7f2b8001b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000a020 con 0x7f2b68008880 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: dump: 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b8b84f640 5 monclient: authenticate success, global_id 4641 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8001f040 con 0x7f2b68008880 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:08.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80019610 con 0x7f2b68008880 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b797fa640 10 monclient: dump: 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.189 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b8b84f640 10 monclient: _renew_subs 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b8b84f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b7b7fe640 4 set_mon_vals no callback set 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b84135ce0 con 0x7f2b68008880 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b8b84f640 10 monclient: _renew_subs 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b8b84f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.163+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b84135110 con 0x7f2b68008880 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b8b84f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b8b84f640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b8b84f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b84136de0 con 0x7f2b84137ba0 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2b80033070 con 0x7f2b68008880 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 10 monclient: get_auth_request con 0x7f2b84137ba0 auth_method 0 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 msgr2=0x7f2b84133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 msgr2=0x7f2b84133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2b84132290 0x7f2b68042ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f2b8004db60 con 0x7f2b68008880 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2b84132290 0x7f2b68042ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.164+0000 7f2b89dec640 10 monclient: get_auth_request con 0x7f2b84132290 auth_method 0 2024-09-18T12:27:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.167+0000 7f2b89dec640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2b84132290 0x7f2b68042ed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b84064050 tx=0x7f2b7c036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 10 monclient: get_auth_request con 0x7f2b84137ba0 auth_method 0 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 msgr2=0x7f2b84133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 msgr2=0x7f2b84133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:08.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.365+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:27:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient: get_auth_request con 0x7f2b84137ba0 auth_method 0 2024-09-18T12:27:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 10 monclient(hunting): handle_auth_done global_id 44099 payload 824 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.766+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b84137160 con 0x7f2b84137ba0 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.768+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 msgr2=0x7f2b84133660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.768+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b74017900 tx=0x7f2b74007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.768+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74017ea0 con 0x7f2b84137ba0 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.768+0000 7f2b797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.768+0000 7f2b797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.768+0000 7f2b797fa640 10 monclient: dump: 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.769+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2b7401c070 con 0x7f2b84137ba0 2024-09-18T12:27:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.769+0000 7f2b797fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.769+0000 7f2b797fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.769+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 msgr2=0x7f2b84133660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.769+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b74017900 tx=0x7f2b74007e40 comp rx=0 tx=0).stop 2024-09-18T12:27:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.769+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 msgr2=0x7f2b84133660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.769+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84137ba0 0x7f2b84133660 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b74017900 tx=0x7f2b74007e40 comp rx=0 tx=0).stop 2024-09-18T12:27:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.774+0000 7f2b8b84f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.774+0000 7f2b8b84f640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 0x7f2b84133fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.774+0000 7f2b8b84f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.774+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b84065e00 con 0x7f2b84133ba0 2024-09-18T12:27:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 0x7f2b84133fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient: get_auth_request con 0x7f2b84133ba0 auth_method 0 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 10 monclient(hunting): handle_auth_done global_id 44102 payload 824 2024-09-18T12:27:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.775+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2b841344e0 con 0x7f2b84133ba0 2024-09-18T12:27:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b88dea640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 msgr2=0x7f2b84133fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b88dea640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 0x7f2b84133fa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2b74017900 tx=0x7f2b7401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74031070 con 0x7f2b84133ba0 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 10 monclient: dump: 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f2b7402d470 con 0x7f2b84133ba0 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 msgr2=0x7f2b84133fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 0x7f2b84133fa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2b74017900 tx=0x7f2b7401f000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 msgr2=0x7f2b84133fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.777+0000 7f2b797fa640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2b84133ba0 0x7f2b84133fa0 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2b74017900 tx=0x7f2b7401f000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 24, 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:08.781 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:08.784 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.786 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.787 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.788 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.789 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:08.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.792 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:08.793 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2b84132290 msgr2=0x7f2b68042ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2b84132290 0x7f2b68042ed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b84064050 tx=0x7f2b7c036000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 10 monclient: shutdown 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 msgr2=0x7f2b6800ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 --2- 172.21.15.116:0/1681037634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b68008880 0x7f2b6800ac60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800bc90 tx=0x7f2b8001b070 comp rx=0 tx=0).stop 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b89dec640 1 -- 172.21.15.116:0/1681037634 reap_dead start 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b89dec640 1 -- 172.21.15.116:0/1681037634 reap_dead start 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 shutdown_connections 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 >> 172.21.15.116:0/1681037634 conn(0x7f2b84067bd0 msgr2=0x7f2b840a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 shutdown_connections 2024-09-18T12:27:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.778+0000 7f2b8b84f640 1 -- 172.21.15.116:0/1681037634 wait complete. 2024-09-18T12:27:08.796 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:27:08.796 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:27:08.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:27:08.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.829+0000 7fe0e37fe640 1 -- 172.21.15.144:0/665648831 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v40) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe0d4042030 con 0x7fe0ec09f020 2024-09-18T12:27:08.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.829+0000 7fe0e37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T12:27:08.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.829+0000 7fe0e37fe640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-18T12:27:08.832 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 4 size to 10 2024-09-18T12:27:08.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0cc0384e0 msgr2=0x7fe0cc03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0cc0384e0 0x7fe0cc03a990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc007d10 tx=0x7fe0dc009040 comp rx=0 tx=0).stop 2024-09-18T12:27:08.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 10 monclient: shutdown 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 msgr2=0x7fe0ec097a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec097a40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009880 tx=0x7fe0d4032000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 shutdown_connections 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0cc0384e0 0x7fe0cc03a990 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc007d10 tx=0x7fe0dc009040 comp rx=0 tx=0).stop 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec136d60 0x7fe0ec097f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0ec09f020 0x7fe0ec097a40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 --2- 172.21.15.144:0/665648831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe0ec09ec50 0x7fe0ec097500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.835+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 >> 172.21.15.144:0/665648831 conn(0x7fe0ec0a5d40 msgr2=0x7fe0ec0a4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:08.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.838+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 shutdown_connections 2024-09-18T12:27:08.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:08.843+0000 7fe0f18d4640 1 -- 172.21.15.144:0/665648831 wait complete. 2024-09-18T12:27:08.864 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_config_value foo min_size 5 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:+ local pool_name config_opt expected_val val 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:+ pool_name=foo 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:+ config_opt=min_size 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_val=5 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:++ get_config_value_or_die foo min_size 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:++ local pool_name config_opt raw val 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:++ pool_name=foo 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:++ config_opt=min_size 2024-09-18T12:27:08.865 INFO:tasks.workunit.client.0.smithi144.stderr:+++ ceph osd pool get foo min_size 2024-09-18T12:27:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.936+0000 7f1f8a66f640 10 monclient: get_monmap_and_config 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.936+0000 7f1f8a66f640 10 monclient: build_initial_monmap 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.936+0000 7f1f8a66f640 10 monclient: monmap: 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:08.937628+0000 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:08.937628+0000 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:08.939 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.941+0000 7f1f8a66f640 1 Processor -- start 2024-09-18T12:27:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.941+0000 7f1f8a66f640 1 -- start start 2024-09-18T12:27:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.941+0000 7f1f8a66f640 10 monclient: init 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.941+0000 7f1f8a66f640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.941+0000 7f1f8a66f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.941+0000 7f1f8a66f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409ebd0 0x7f1f8409efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8409ebd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1f8409cf50 0x7f1f84068d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8409cf50 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f8409d320 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8409fea0 con 0x7f1f8409d320 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f840a1260 con 0x7f1f8409cf50 2024-09-18T12:27:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8409fc40 con 0x7f1f8409ebd0 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f8a66f640 10 monclient(hunting): authenticate will time out at 856.263122s 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f88c0c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1f8409d320 0x7f1f840692d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44972/0 (socket says 172.21.15.116:44972) 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f88c0c640 1 -- 172.21.15.116:0/3279485131 learned_addr learned my addr 172.21.15.116:0/3279485131 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.942+0000 7f1f837fe640 1 --2- 172.21.15.116:0/3279485131 >> v2:172.21.15.144:3300/0 conn(0x7f1f8409cf50 0x7f1f84068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f837fe640 10 monclient(hunting): get_auth_request con 0x7f1f8409cf50 auth_method 0 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 1 --2- 172.21.15.116:0/3279485131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409ebd0 0x7f1f8409efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): get_auth_request con 0x7f1f8409ebd0 auth_method 0 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient(hunting): handle_auth_done global_id 44105 payload 274 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 1 -- 172.21.15.116:0/3279485131 >> v2:172.21.15.144:3300/0 conn(0x7f1f8409cf50 msgr2=0x7f1f84068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 1 --2- 172.21.15.116:0/3279485131 >> v2:172.21.15.144:3300/0 conn(0x7f1f8409cf50 0x7f1f84068d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 1 -- 172.21.15.116:0/3279485131 >> v1:172.21.15.116:6789/0 conn(0x7f1f8409d320 legacy=0x7f1f840692d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 1 monclient: found mon.noname-b 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.943+0000 7f1f83fff640 1 -- 172.21.15.116:0/3279485131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840a0080 con 0x7f1f8409ebd0 2024-09-18T12:27:08.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f83fff640 1 -- 172.21.15.116:0/3279485131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409ebd0 msgr2=0x7f1f8409efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f83fff640 1 --2- 172.21.15.116:0/3279485131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409ebd0 0x7f1f8409efa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009850 tx=0x7f1f740090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f98372fe63db4b24 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 1 -- 172.21.15.116:0/3279485131 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7402add0 con 0x7f1f8409ebd0 2024-09-18T12:27:08.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: dump: 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f8a66f640 5 monclient: authenticate success, global_id 44105 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f8a66f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 1 -- 172.21.15.116:0/3279485131 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f74032440 con 0x7f1f8409ebd0 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 1 -- 172.21.15.116:0/3279485131 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740327e0 con 0x7f1f8409ebd0 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f82ffd640 10 monclient: dump: 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.959 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f8a66f640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.946+0000 7f1f8a66f640 4 set_mon_vals no callback set 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 10 monclient: shutdown 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 -- 172.21.15.116:0/3279485131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409ebd0 msgr2=0x7f1f8409efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/3279485131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409ebd0 0x7f1f8409efa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009850 tx=0x7f1f740090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 -- 172.21.15.116:0/3279485131 shutdown_connections 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/3279485131 >> v2:172.21.15.144:3300/0 conn(0x7f1f8409cf50 0x7f1f84068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/3279485131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409ebd0 0x7f1f8409efa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 -- 172.21.15.116:0/3279485131 >> 172.21.15.116:0/3279485131 conn(0x7f1f84067110 msgr2=0x7f1f84067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 -- 172.21.15.116:0/3279485131 shutdown_connections 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.947+0000 7f1f8a66f640 1 -- 172.21.15.116:0/3279485131 wait complete. 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.948+0000 7f1f8a66f640 10 monclient: build_initial_monmap 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.948+0000 7f1f8a66f640 10 monclient: monmap: 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.948+0000 7f1f8a66f640 1 Processor -- start 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.948+0000 7f1f8a66f640 1 -- start start 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.948+0000 7f1f8a66f640 10 monclient: init 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409cf50 0x7f1f8412fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8409cf50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 0x7f1f841302a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8409d320 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409ebd0 0x7f1f8412e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8409ebd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84130ef0 con 0x7f1f8409ebd0 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8413e950 con 0x7f1f8409cf50 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8409bab0 con 0x7f1f8409d320 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f8a66f640 10 monclient(hunting): authenticate will time out at 856.270143s 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 0x7f1f841302a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 0x7f1f841302a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36630/0 (socket says 172.21.15.116:36630) 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 -- 172.21.15.116:0/44355121 learned_addr learned my addr 172.21.15.116:0/44355121 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409ebd0 0x7f1f8412e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): get_auth_request con 0x7f1f8409d320 auth_method 0 2024-09-18T12:27:08.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 10 monclient(hunting): get_auth_request con 0x7f1f8409ebd0 auth_method 0 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409cf50 0x7f1f8412fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 10 monclient(hunting): get_auth_request con 0x7f1f8409cf50 auth_method 0 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient(hunting): handle_auth_done global_id 44108 payload 824 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409cf50 msgr2=0x7f1f8412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409cf50 0x7f1f8412fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409ebd0 msgr2=0x7f1f8412e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409ebd0 0x7f1f8412e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 monclient: found mon.noname-c 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f837fe640 1 -- 172.21.15.116:0/44355121 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f74009390 con 0x7f1f8409d320 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.949+0000 7f1f88c0c640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409ebd0 0x7f1f8412e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f837fe640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 msgr2=0x7f1f841302a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f837fe640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 0x7f1f841302a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800e960 tx=0x7f1f780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78009480 con 0x7f1f8409d320 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f6bfff640 10 monclient: dump: 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f6bfff640 10 monclient: _finish_auth 0 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7800caf0 con 0x7f1f8409d320 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f8a66f640 5 monclient: authenticate success, global_id 44108 2024-09-18T12:27:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f8a66f640 10 monclient: _renew_subs 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f8a66f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.951+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f84132900 con 0x7f1f8409d320 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f6bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7800ce70 con 0x7f1f8409d320 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f6bfff640 10 monclient: dump: 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.968 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f8a66f640 10 monclient: _renew_subs 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f8a66f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f84133b80 con 0x7f1f8409d320 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f827fc640 4 set_mon_vals no callback set 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f8a66f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1f78013680 con 0x7f1f8409d320 2024-09-18T12:27:08.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 0x7f1f8412f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f8a66f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f841309e0 con 0x7f1f8412ed90 2024-09-18T12:27:08.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f83fff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 0x7f1f8412f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f83fff640 10 monclient: get_auth_request con 0x7f1f8412ed90 auth_method 0 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.952+0000 7f1f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f6bfff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f70038510 0x7f1f7003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f1f7804bc20 con 0x7f1f8409d320 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f88c0c640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f70038510 0x7f1f7003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f88c0c640 10 monclient: get_auth_request con 0x7f1f70038510 auth_method 0 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f83fff640 10 monclient(hunting): handle_auth_done global_id 4600 payload 824 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.953+0000 7f1f83fff640 1 -- 172.21.15.116:0/44355121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f84133090 con 0x7f1f8412ed90 2024-09-18T12:27:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.955+0000 7f1f83fff640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 msgr2=0x7f1f8412f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.955+0000 7f1f83fff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 0x7f1f8412f160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f74034000 tx=0x7f1f74032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.955+0000 7f1f88c0c640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f70038510 0x7f1f7003a9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f8406eef0 tx=0x7f1f7c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.955+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740327e0 con 0x7f1f8412ed90 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.955+0000 7f1f6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.955+0000 7f1f6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.955+0000 7f1f6bfff640 10 monclient: dump: 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.974 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.956+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1f74041070 con 0x7f1f8412ed90 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.956+0000 7f1f6bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.956+0000 7f1f6bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.956+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 msgr2=0x7f1f8412f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.956+0000 7f1f6bfff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 0x7f1f8412f160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f74034000 tx=0x7f1f74032000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.956+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 msgr2=0x7f1f8412f160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.956+0000 7f1f6bfff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8412ed90 0x7f1f8412f160 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f74034000 tx=0x7f1f74032000 comp rx=0 tx=0).stop 2024-09-18T12:27:08.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f8a66f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:08.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 0x7f1f84096e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f8a66f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84065e00 con 0x7f1f84096a60 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f83fff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 0x7f1f84096e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f83fff640 10 monclient: get_auth_request con 0x7f1f84096a60 auth_method 0 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.961+0000 7f1f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.962+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.962+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.962+0000 7f1f83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.962+0000 7f1f83fff640 10 monclient(hunting): handle_auth_done global_id 4603 payload 824 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.962+0000 7f1f83fff640 1 -- 172.21.15.116:0/44355121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1f8412f6a0 con 0x7f1f84096a60 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f83fff640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 msgr2=0x7f1f84096e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f83fff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 0x7f1f84096e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1f841317b0 tx=0x7f1f7403f070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7403d600 con 0x7f1f84096a60 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 10 monclient: dump: 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:08.979 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f1f7403d760 con 0x7f1f84096a60 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 msgr2=0x7f1f84096e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 0x7f1f84096e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1f841317b0 tx=0x7f1f7403f070 comp rx=0 tx=0).stop 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 msgr2=0x7f1f84096e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.964+0000 7f1f6bfff640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f84096a60 0x7f1f84096e60 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1f841317b0 tx=0x7f1f7403f070 comp rx=0 tx=0).stop 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 24, 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:08.981 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.982 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.983 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:08.984 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:08.985 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:08.986 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.987 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.989 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:08.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.994 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:08.995 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:08.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:08.996 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:08.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:08.996 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:08.996 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:08.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f70038510 msgr2=0x7f1f7003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f70038510 0x7f1f7003a9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f8406eef0 tx=0x7f1f7c009040 comp rx=0 tx=0).stop 2024-09-18T12:27:08.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 10 monclient: shutdown 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 msgr2=0x7f1f841302a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 0x7f1f841302a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800e960 tx=0x7f1f780090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f88c0c640 1 -- 172.21.15.116:0/44355121 reap_dead start 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 shutdown_connections 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 --2- 172.21.15.116:0/44355121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f8409d320 0x7f1f841302a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 >> 172.21.15.116:0/44355121 conn(0x7f1f84067110 msgr2=0x7f1f8409d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 shutdown_connections 2024-09-18T12:27:08.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:08.965+0000 7f1f8a66f640 1 -- 172.21.15.116:0/44355121 wait complete. 2024-09-18T12:27:08.998 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:27:08.998 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:27:08.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:27:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.089+0000 7f25db2c9640 10 monclient: get_monmap_and_config 2024-09-18T12:27:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.089+0000 7f25db2c9640 10 monclient: build_initial_monmap 2024-09-18T12:27:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.089+0000 7f25db2c9640 10 monclient: monmap: 2024-09-18T12:27:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:09.089963+0000 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:09.089963+0000 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:09.091 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.089+0000 7f25db2c9640 1 Processor -- start 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 1 -- start start 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient: init 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25d4096c60 0x7f25d409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient(hunting): picked mon.noname-a con 0x7f25d4096c60 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f25d4097030 0x7f25d4097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient(hunting): picked mon.noname-c con 0x7f25d4097030 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25d412e5f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d409d2e0 con 0x7f25d412e5f0 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d409ac00 con 0x7f25d4096c60 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d409d6b0 con 0x7f25d4097030 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25db2c9640 10 monclient(hunting): authenticate will time out at 856.411587s 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25d4096c60 0x7f25d409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25d4096c60 0x7f25d409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55540/0 (socket says 172.21.15.116:55540) 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 -- 172.21.15.116:0/3565422908 learned_addr learned my addr 172.21.15.116:0/3565422908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): get_auth_request con 0x7f25d4096c60 auth_method 0 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d8864640 1 --2- 172.21.15.116:0/3565422908 >> v2:172.21.15.144:3300/0 conn(0x7f25d4097030 0x7f25d4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d8864640 10 monclient(hunting): get_auth_request con 0x7f25d4097030 auth_method 0 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d8864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d8864640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d8864640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient(hunting): handle_auth_done global_id 4650 payload 274 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 -- 172.21.15.116:0/3565422908 >> v2:172.21.15.144:3300/0 conn(0x7f25d4097030 msgr2=0x7f25d4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 --2- 172.21.15.116:0/3565422908 >> v2:172.21.15.144:3300/0 conn(0x7f25d4097030 0x7f25d4097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 -- 172.21.15.116:0/3565422908 >> v1:172.21.15.144:6789/0 conn(0x7f25d412e5f0 legacy=0x7f25d412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient: _finish_hunting 0 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 monclient: found mon.noname-a 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.090+0000 7f25d9065640 1 -- 172.21.15.116:0/3565422908 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d40992f0 con 0x7f25d4096c60 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d9065640 1 -- 172.21.15.116:0/3565422908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4096c60 msgr2=0x7f25d409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d9065640 1 --2- 172.21.15.116:0/3565422908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4096c60 0x7f25d409a6c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f25d409ecd0 tx=0x7f25cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb02f9f114d273c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 1 -- 172.21.15.116:0/3565422908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc007610 con 0x7f25d4096c60 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: dump: 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: _finish_auth 0 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25db2c9640 5 monclient: authenticate success, global_id 4650 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25db2c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 1 -- 172.21.15.116:0/3565422908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc02d050 con 0x7f25d4096c60 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25db2c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25db2c9640 4 set_mon_vals no callback set 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 1 -- 172.21.15.116:0/3565422908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc0079a0 con 0x7f25d4096c60 2024-09-18T12:27:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.093+0000 7f25d3fff640 10 monclient: dump: 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 10 monclient: shutdown 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 -- 172.21.15.116:0/3565422908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4096c60 msgr2=0x7f25d409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3565422908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4096c60 0x7f25d409a6c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f25d409ecd0 tx=0x7f25cc009840 comp rx=0 tx=0).stop 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 -- 172.21.15.116:0/3565422908 shutdown_connections 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3565422908 >> v2:172.21.15.144:3300/0 conn(0x7f25d4097030 0x7f25d4097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3565422908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4096c60 0x7f25d409a6c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 -- 172.21.15.116:0/3565422908 >> 172.21.15.116:0/3565422908 conn(0x7f25d40672b0 msgr2=0x7f25d40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 -- 172.21.15.116:0/3565422908 shutdown_connections 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.094+0000 7f25db2c9640 1 -- 172.21.15.116:0/3565422908 wait complete. 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.095+0000 7f25db2c9640 10 monclient: build_initial_monmap 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.095+0000 7f25db2c9640 10 monclient: monmap: 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.095+0000 7f25db2c9640 1 Processor -- start 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.095+0000 7f25db2c9640 1 -- start start 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.095+0000 7f25db2c9640 10 monclient: init 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 0x7f25d412db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): picked mon.noname-c con 0x7f25d4096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4097030 0x7f25d413e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): picked mon.noname-a con 0x7f25d4097030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d412e5f0 0x7f25d413e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): picked mon.noname-b con 0x7f25d412e5f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d409add0 con 0x7f25d4097030 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d411c270 con 0x7f25d412e5f0 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d411bf90 con 0x7f25d4096c60 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25db2c9640 10 monclient(hunting): authenticate will time out at 856.417126s 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 0x7f25d412db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4097030 0x7f25d413e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 0x7f25d412db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36646/0 (socket says 172.21.15.116:36646) 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4097030 0x7f25d413e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55544/0 (socket says 172.21.15.116:55544) 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 -- 172.21.15.116:0/3097705346 learned_addr learned my addr 172.21.15.116:0/3097705346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): get_auth_request con 0x7f25d4096c60 auth_method 0 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 10 monclient(hunting): get_auth_request con 0x7f25d4097030 auth_method 0 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d412e5f0 0x7f25d413e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 10 monclient(hunting): get_auth_request con 0x7f25d412e5f0 auth_method 0 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient(hunting): handle_auth_done global_id 44117 payload 824 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 -- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d412e5f0 msgr2=0x7f25d413e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d412e5f0 0x7f25d413e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 -- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4097030 msgr2=0x7f25d413e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4097030 0x7f25d413e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient: _finish_hunting 0 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 monclient: found mon.noname-c 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d9065640 1 -- 172.21.15.116:0/3097705346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc009ae0 con 0x7f25d4096c60 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.096+0000 7f25d8864640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4097030 0x7f25d413e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d9065640 1 -- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 msgr2=0x7f25d412db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d9065640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 0x7f25d412db80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009f50 tx=0x7f25cc036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 1 -- 172.21.15.116:0/3097705346 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc0328b0 con 0x7f25d4096c60 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: dump: 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25db2c9640 5 monclient: authenticate success, global_id 44117 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 1 -- 172.21.15.116:0/3097705346 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc032a10 con 0x7f25d4096c60 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25db2c9640 10 monclient: _renew_subs 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25db2c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d411ad50 con 0x7f25d4096c60 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 1 -- 172.21.15.116:0/3097705346 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc032dd0 con 0x7f25d4096c60 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d17fa640 10 monclient: dump: 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25db2c9640 10 monclient: _renew_subs 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.098+0000 7f25d37fe640 4 set_mon_vals no callback set 2024-09-18T12:27:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25db2c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d4119b00 con 0x7f25d4096c60 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25d17fa640 1 -- 172.21.15.116:0/3097705346 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f25cc038050 con 0x7f25d4096c60 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25db2c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25d17fa640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038490 0x7f25c003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25db2c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25db2c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d411a0a0 con 0x7f25d4096c60 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.099+0000 7f25d17fa640 1 -- 172.21.15.116:0/3097705346 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f25cc06c3f0 con 0x7f25d4096c60 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.100+0000 7f25d8864640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038490 0x7f25c003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.100+0000 7f25d8864640 10 monclient: get_auth_request con 0x7f25c0038490 auth_method 0 2024-09-18T12:27:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.102+0000 7f25d8864640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038490 0x7f25c003a940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f25d413f130 tx=0x7f25c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.104+0000 7f25d17fa640 1 -- 172.21.15.116:0/3097705346 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f25cc03f070 con 0x7f25d4096c60 2024-09-18T12:27:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.104+0000 7f25d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.104+0000 7f25d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:09.246 INFO:tasks.workunit.client.0.smithi144.stderr:++ raw='min_size: 5' 2024-09-18T12:27:09.246 INFO:tasks.workunit.client.0.smithi144.stderr:++ [[ 0 -ne 0 ]] 2024-09-18T12:27:09.246 INFO:tasks.workunit.client.0.smithi144.stderr:+++ echo min_size: 5 2024-09-18T12:27:09.246 INFO:tasks.workunit.client.0.smithi144.stderr:+++ sed -e 's/[{} "]//g' 2024-09-18T12:27:09.247 INFO:tasks.workunit.client.0.smithi144.stderr:++ raw=min_size:5 2024-09-18T12:27:09.247 INFO:tasks.workunit.client.0.smithi144.stderr:+++ echo min_size:5 2024-09-18T12:27:09.247 INFO:tasks.workunit.client.0.smithi144.stderr:+++ cut -f2 -d: 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:++ val=5 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:++ echo 5 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:++ return 0 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:+ val=5 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 5 != \5 ]] 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:09.249 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set foo size 0 2024-09-18T12:27:09.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.318+0000 7f25db2c9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:27:09.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.318+0000 7f25db2c9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:27:09.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.318+0000 7f25db2c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:09.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.318+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f25d404bf60 con 0x7f25d4096c60 2024-09-18T12:27:09.320 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.321 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=9,logm=68} crc {auth=705805638,config=3140514009,health=4264675569,kv=2187923472,logm=3819684974}) 2024-09-18T12:27:09.321 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=1,health=12,kv=9,logm=68} crc {auth=2245804959,config=1228700967,health=1921103840,kv=2187923472,logm=2161169978}) 2024-09-18T12:27:09.321 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.321 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=9,logm=68} crc {auth=705805638,config=3140514009,health=4264675569,kv=2187923472,logm=3819684974}) 2024-09-18T12:27:09.321 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=2072047025,config=3140514009,health=1921103840,kv=2187923472,logm=3686171559}) 2024-09-18T12:27:09.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f25d17fa640 1 -- 172.21.15.116:0/3097705346 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f25cc03b620 con 0x7f25d4096c60 2024-09-18T12:27:09.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f25d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:27:09.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.319+0000 7f25d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.320+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.320+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363844197}) 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.320+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3237852747}) 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.320+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.320+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363844197}) 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.320+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424260351}) 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.321+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.321+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987441443}) 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.321+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333641731}) 2024-09-18T12:27:09.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.321+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.321+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987441443}) 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.321+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316214442}) 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853488981}) 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3083594710}) 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853488981}) 2024-09-18T12:27:09.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4002455693}) 2024-09-18T12:27:09.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038490 msgr2=0x7f25c003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038490 0x7f25c003a940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f25d413f130 tx=0x7f25c8008040 comp rx=0 tx=0).stop 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 10 monclient: shutdown 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 msgr2=0x7f25d412db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 0x7f25d412db80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009f50 tx=0x7f25cc036000 comp rx=0 tx=0).stop 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 shutdown_connections 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038490 0x7f25c003a940 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25d412e5f0 0x7f25d413e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25d4097030 0x7f25d413e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 --2- 172.21.15.116:0/3097705346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25d4096c60 0x7f25d412db80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 >> 172.21.15.116:0/3097705346 conn(0x7f25d40672b0 msgr2=0x7f25d4069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:09.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 shutdown_connections 2024-09-18T12:27:09.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f25db2c9640 1 -- 172.21.15.116:0/3097705346 wait complete. 2024-09-18T12:27:09.330 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.330 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765418980}) 2024-09-18T12:27:09.330 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=428683600}) 2024-09-18T12:27:09.330 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.330 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765418980}) 2024-09-18T12:27:09.330 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.322+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938031204}) 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.323+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.323+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1636868341,mdsmap=1531151339,mgr=1692422016,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=785072485}) 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.323+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=87,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=223558336,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3576335585}) 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.323+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.323+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1636868341,mdsmap=1531151339,mgr=1692422016,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=785072485}) 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.323+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=2956678921,mdsmap=1531151339,mgr=1362417815}) 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.324+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.324+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mgrstat=2653605039,mon_config_key=1355063917,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=1370131380,osd_snap=2033388227,osdmap=4144479480}) 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.324+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=2,osdmap=76} crc {mgrstat=139681588,mon_config_key=3655569570,monmap=148614106,osd_metadata=3127564625,osd_snap=2033388227,osdmap=2704524336}) 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.324+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:27:09.331 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.324+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mgrstat=2653605039,mon_config_key=1355063917,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=1370131380,osd_snap=2033388227,osdmap=4144479480}) 2024-09-18T12:27:09.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:27:09.324+0000 7f82a2ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=76} crc {mgrstat=2610170656,mon_config_key=3655569570,monmap=148614106,osd_metadata=4175318930,osd_pg_creating=1370131381,osd_snap=2033388227,osdmap=4103511196}) 2024-09-18T12:27:09.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:27:09.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.322+0000 7f4a75d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.322+0000 7f4a75d9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.322+0000 7f4a75d9d640 10 monclient: monmap: 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:09.323320+0000 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:09.323320+0000 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:09.342 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 Processor -- start 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 -- start start 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient: init 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009ec50 0x7f4a70131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a7009ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a701321a0 0x7f4a70068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a701321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4a7009f020 0x7f4a70069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a7009f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a70067420 con 0x7f4a701321a0 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a7009fdc0 con 0x7f4a7009f020 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700a0510 con 0x7f4a7009ec50 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a75d9d640 10 monclient(hunting): authenticate will time out at 864.892773s 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4a7009f020 0x7f4a70069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009ec50 0x7f4a70131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a701321a0 0x7f4a70068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009ec50 0x7f4a70131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:46348/0 (socket says 172.21.15.144:46348) 2024-09-18T12:27:09.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6f7fe640 1 -- 172.21.15.144:0/3134163935 learned_addr learned my addr 172.21.15.144:0/3134163935 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4a7009f020 0x7f4a70069240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51462/0 (socket says 172.21.15.144:51462) 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4a7009ec50 auth_method 0 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6effd640 10 monclient(hunting): get_auth_request con 0x7f4a701321a0 auth_method 0 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6ffff640 10 monclient(hunting): get_auth_request con 0x7f4a7009f020 auth_method 0 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.323+0000 7f4a6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 10 monclient(hunting): handle_auth_done global_id 4618 payload 274 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 1 -- 172.21.15.144:0/3134163935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009ec50 msgr2=0x7f4a70131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 1 --2- 172.21.15.144:0/3134163935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009ec50 0x7f4a70131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 1 -- 172.21.15.144:0/3134163935 >> v2:172.21.15.116:3300/0 conn(0x7f4a701321a0 msgr2=0x7f4a70068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 1 --2- 172.21.15.144:0/3134163935 >> v2:172.21.15.116:3300/0 conn(0x7f4a701321a0 0x7f4a70068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 1 monclient: found mon.noname-c 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6ffff640 1 -- 172.21.15.144:0/3134163935 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700a0200 con 0x7f4a7009f020 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6effd640 1 --2- 172.21.15.144:0/3134163935 >> v2:172.21.15.116:3300/0 conn(0x7f4a701321a0 0x7f4a70068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.324+0000 7f4a6f7fe640 1 --2- 172.21.15.144:0/3134163935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009ec50 0x7f4a70131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6ffff640 1 -- 172.21.15.144:0/3134163935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009f020 msgr2=0x7f4a70069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6ffff640 1 --2- 172.21.15.144:0/3134163935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009f020 0x7f4a70069240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a700a0150 tx=0x7f4a68009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=634b1b8e22d14f24 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 1 -- 172.21.15.144:0/3134163935 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68007760 con 0x7f4a7009f020 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: dump: 2024-09-18T12:27:09.347 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 1 -- 172.21.15.144:0/3134163935 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6802d050 con 0x7f4a7009f020 2024-09-18T12:27:09.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 1 -- 172.21.15.144:0/3134163935 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68007af0 con 0x7f4a7009f020 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a75d9d640 5 monclient: authenticate success, global_id 4618 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a75d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a75d9d640 4 set_mon_vals no callback set 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.327+0000 7f4a6e7fc640 10 monclient: dump: 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:09.349 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 10 monclient: shutdown 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 -- 172.21.15.144:0/3134163935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009f020 msgr2=0x7f4a70069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 --2- 172.21.15.144:0/3134163935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009f020 0x7f4a70069240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a700a0150 tx=0x7f4a68009780 comp rx=0 tx=0).stop 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 -- 172.21.15.144:0/3134163935 shutdown_connections 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 --2- 172.21.15.144:0/3134163935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009f020 0x7f4a70069240 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 --2- 172.21.15.144:0/3134163935 >> v2:172.21.15.116:3300/0 conn(0x7f4a701321a0 0x7f4a70068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 --2- 172.21.15.144:0/3134163935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009ec50 0x7f4a70131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 -- 172.21.15.144:0/3134163935 >> 172.21.15.144:0/3134163935 conn(0x7f4a700a5d40 msgr2=0x7f4a700a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:09.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 -- 172.21.15.144:0/3134163935 shutdown_connections 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 1 -- 172.21.15.144:0/3134163935 wait complete. 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.328+0000 7f4a75d9d640 10 monclient: monmap: 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:09.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 Processor -- start 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 -- start start 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient: init 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009ec50 0x7f4a7009a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a7009ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009f020 0x7f4a701309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a7009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a701321a0 0x7f4a7011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a701321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70141c70 con 0x7f4a701321a0 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7009b350 con 0x7f4a7009ec50 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.329+0000 7f4a75d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700a45f0 con 0x7f4a7009f020 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a75d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a75d9d640 10 monclient(hunting): authenticate will time out at 864.899010s 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009ec50 0x7f4a7009a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009ec50 0x7f4a7009a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51472/0 (socket says 172.21.15.144:51472) 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 -- 172.21.15.144:0/1778208862 learned_addr learned my addr 172.21.15.144:0/1778208862 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4a7009ec50 auth_method 0 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:09.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6ffff640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a701321a0 0x7f4a7011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6effd640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009f020 0x7f4a701309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6effd640 10 monclient(hunting): get_auth_request con 0x7f4a7009f020 auth_method 0 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6ffff640 10 monclient(hunting): get_auth_request con 0x7f4a701321a0 auth_method 0 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_done global_id 4621 payload 824 2024-09-18T12:27:09.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 -- 172.21.15.144:0/1778208862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009f020 msgr2=0x7f4a701309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4a7009f020 0x7f4a701309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 -- 172.21.15.144:0/1778208862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a701321a0 msgr2=0x7f4a7011b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a701321a0 0x7f4a7011b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.330+0000 7f4a6f7fe640 1 -- 172.21.15.144:0/1778208862 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60009450 con 0x7f4a7009ec50 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a6f7fe640 1 -- 172.21.15.144:0/1778208862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009ec50 msgr2=0x7f4a7009a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a6f7fe640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009ec50 0x7f4a7009a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000ecb0 tx=0x7f4a600091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 1 -- 172.21.15.144:0/1778208862 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60009730 con 0x7f4a7009ec50 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:09.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: dump: 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:09.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 1 -- 172.21.15.144:0/1778208862 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6000cd50 con 0x7f4a7009ec50 2024-09-18T12:27:09.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a75d9d640 5 monclient: authenticate success, global_id 4621 2024-09-18T12:27:09.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:09.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 1 -- 172.21.15.144:0/1778208862 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600136a0 con 0x7f4a7009ec50 2024-09-18T12:27:09.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a75d9d640 10 monclient: _renew_subs 2024-09-18T12:27:09.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a75d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:09.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a75d9d640 1 -- 172.21.15.144:0/1778208862 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a64009390 con 0x7f4a7009ec50 2024-09-18T12:27:09.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a577fe640 10 monclient: dump: 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a6dffb640 4 set_mon_vals no callback set 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a75d9d640 10 monclient: _renew_subs 2024-09-18T12:27:09.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a75d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.332+0000 7f4a75d9d640 1 -- 172.21.15.144:0/1778208862 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a68009a20 con 0x7f4a7009ec50 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a6f7fe640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4a7009ec50 0x7f4a7009a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000ecb0 tx=0x7f4a600091d0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f4a6004bfa0 osd_map(40..40 src has 1..40) v4 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a577fe640 1 -- 172.21.15.144:0/1778208862 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4a60013a50 con 0x7f4a7009ec50 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a75d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a75d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a75d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a75d9d640 1 -- 172.21.15.144:0/1778208862 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7011e1f0 con 0x7f4a7009ec50 2024-09-18T12:27:09.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a577fe640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a5c0384c0 0x7f4a5c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:09.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a6effd640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a5c0384c0 0x7f4a5c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:09.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.333+0000 7f4a6effd640 10 monclient: get_auth_request con 0x7f4a5c0384c0 auth_method 0 2024-09-18T12:27:09.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.336+0000 7f4a6effd640 1 --2- 172.21.15.144:0/1778208862 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a5c0384c0 0x7f4a5c03a970 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009360 tx=0x7f4a64008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:09.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:09.338+0000 7f4a577fe640 1 -- 172.21.15.144:0/1778208862 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f4a6004bfa0 con 0x7f4a7009ec50 2024-09-18T12:27:10.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.161+0000 7f4a577fe640 1 -- 172.21.15.144:0/1778208862 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4a60023d20 con 0x7f4a7009ec50 2024-09-18T12:27:10.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.161+0000 7f4a577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.161+0000 7f4a577fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:10.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.318+0000 7f4a75d9d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-18T12:27:10.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.318+0000 7f4a75d9d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-18T12:27:10.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.318+0000 7f4a75d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.318+0000 7f4a75d9d640 1 -- 172.21.15.144:0/1778208862 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) v1 -- 0x7f4a700008d0 con 0x7f4a7009ec50 2024-09-18T12:27:10.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.319+0000 7f4a577fe640 1 -- 172.21.15.144:0/1778208862 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v40) v1 ==== 136+0+0 (secure 0 0 0) 0x7f4a60011e20 con 0x7f4a7009ec50 2024-09-18T12:27:10.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.319+0000 7f4a577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-18T12:27:10.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.319+0000 7f4a577fe640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-18T12:27:10.321 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-18T12:27:10.324 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:10.324 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-18T12:27:10.324 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:10.324 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set foo size 20 2024-09-18T12:27:10.338 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:27:10.338 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.398+0000 7f4213617640 10 monclient: get_monmap_and_config 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.398+0000 7f4213617640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.398+0000 7f4213617640 10 monclient: monmap: 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:10.398959+0000 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:10.398959+0000 2024-09-18T12:27:10.410 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.398+0000 7f4213617640 1 Processor -- start 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.398+0000 7f4213617640 1 -- start start 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.398+0000 7f4213617640 10 monclient: init 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T12:27:10.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f420c09ec50 0x7f420c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): picked mon.noname-c con 0x7f420c09ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c1321a0 0x7f420c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): picked mon.noname-b con 0x7f420c1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f420c09f020 0x7f420c069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): picked mon.noname-a con 0x7f420c09f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c067420 con 0x7f420c09f020 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c09fdc0 con 0x7f420c09ec50 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0a0510 con 0x7f420c1321a0 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4213617640 10 monclient(hunting): authenticate will time out at 865.968456s 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f420c09ec50 0x7f420c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f420c09ec50 0x7f420c131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51482/0 (socket says 172.21.15.144:51482) 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 -- 172.21.15.144:0/2606325013 learned_addr learned my addr 172.21.15.144:0/2606325013 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): get_auth_request con 0x7f420c09ec50 auth_method 0 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 1 --2- 172.21.15.144:0/2606325013 >> v2:172.21.15.116:3300/0 conn(0x7f420c09f020 0x7f420c069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4210bb2640 1 --2- 172.21.15.144:0/2606325013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c1321a0 0x7f420c068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 10 monclient(hunting): get_auth_request con 0x7f420c09f020 auth_method 0 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4210bb2640 10 monclient(hunting): get_auth_request con 0x7f420c1321a0 auth_method 0 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4210bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4210bb2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4210bb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4211bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient(hunting): handle_auth_done global_id 4624 payload 274 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 -- 172.21.15.144:0/2606325013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c1321a0 msgr2=0x7f420c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 --2- 172.21.15.144:0/2606325013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c1321a0 0x7f420c068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 -- 172.21.15.144:0/2606325013 >> v2:172.21.15.116:3300/0 conn(0x7f420c09f020 msgr2=0x7f420c069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 --2- 172.21.15.144:0/2606325013 >> v2:172.21.15.116:3300/0 conn(0x7f420c09f020 0x7f420c069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 monclient: found mon.noname-c 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f42113b3640 1 -- 172.21.15.144:0/2606325013 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c0a0200 con 0x7f420c09ec50 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.399+0000 7f4210bb2640 1 --2- 172.21.15.144:0/2606325013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c1321a0 0x7f420c068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f42113b3640 1 -- 172.21.15.144:0/2606325013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09ec50 msgr2=0x7f420c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f42113b3640 1 --2- 172.21.15.144:0/2606325013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09ec50 0x7f420c131c60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f420c0719d0 tx=0x7f42080091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bf79daceb44d200 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 1 -- 172.21.15.144:0/2606325013 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208032440 con 0x7f420c09ec50 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: dump: 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 1 -- 172.21.15.144:0/2606325013 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42080325a0 con 0x7f420c09ec50 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 1 -- 172.21.15.144:0/2606325013 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208032940 con 0x7f420c09ec50 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4213617640 5 monclient: authenticate success, global_id 4624 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4213617640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4213617640 4 set_mon_vals no callback set 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.402+0000 7f4203fff640 10 monclient: dump: 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.417 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 10 monclient: shutdown 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 -- 172.21.15.144:0/2606325013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09ec50 msgr2=0x7f420c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 --2- 172.21.15.144:0/2606325013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09ec50 0x7f420c131c60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f420c0719d0 tx=0x7f42080091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 -- 172.21.15.144:0/2606325013 shutdown_connections 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 --2- 172.21.15.144:0/2606325013 >> v2:172.21.15.116:3300/0 conn(0x7f420c09f020 0x7f420c069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 --2- 172.21.15.144:0/2606325013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c1321a0 0x7f420c068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 --2- 172.21.15.144:0/2606325013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09ec50 0x7f420c131c60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 -- 172.21.15.144:0/2606325013 >> 172.21.15.144:0/2606325013 conn(0x7f420c0a5d40 msgr2=0x7f420c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 -- 172.21.15.144:0/2606325013 shutdown_connections 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 1 -- 172.21.15.144:0/2606325013 wait complete. 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.403+0000 7f4213617640 10 monclient: monmap: 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.419 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 Processor -- start 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 -- start start 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient: init 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c09ec50 0x7f420c09a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): picked mon.noname-c con 0x7f420c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09f020 0x7f420c130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): picked mon.noname-b con 0x7f420c09f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:10.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f420c1321a0 0x7f420c11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): picked mon.noname-a con 0x7f420c1321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c09b3e0 con 0x7f420c1321a0 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c09e4e0 con 0x7f420c09f020 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c11de50 con 0x7f420c09ec50 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.404+0000 7f4213617640 10 monclient(hunting): authenticate will time out at 865.973985s 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09f020 0x7f420c130a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09f020 0x7f420c130a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51496/0 (socket says 172.21.15.144:51496) 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 -- 172.21.15.144:0/3617506796 learned_addr learned my addr 172.21.15.144:0/3617506796 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:10.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): get_auth_request con 0x7f420c09f020 auth_method 0 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c09ec50 0x7f420c09a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 10 monclient(hunting): get_auth_request con 0x7f420c09ec50 auth_method 0 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f420c1321a0 0x7f420c11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 10 monclient(hunting): get_auth_request con 0x7f420c1321a0 auth_method 0 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f42113b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4211bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient(hunting): handle_auth_done global_id 4627 payload 824 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 -- 172.21.15.144:0/3617506796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c09ec50 msgr2=0x7f420c09a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f420c09ec50 0x7f420c09a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 -- 172.21.15.144:0/3617506796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f420c1321a0 msgr2=0x7f420c11b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f420c1321a0 0x7f420c11b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 monclient: found mon.noname-b 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.405+0000 7f4210bb2640 1 -- 172.21.15.144:0/3617506796 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4208009450 con 0x7f420c09f020 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4210bb2640 1 -- 172.21.15.144:0/3617506796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09f020 msgr2=0x7f420c130a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4210bb2640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f420c09f020 0x7f420c130a50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00d8c0 tx=0x7f41fc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 1 -- 172.21.15.144:0/3617506796 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00deb0 con 0x7f420c09f020 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:10.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: dump: 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 1 -- 172.21.15.144:0/3617506796 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc00bae0 con 0x7f420c09f020 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4213617640 5 monclient: authenticate success, global_id 4627 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4213617640 10 monclient: _renew_subs 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4213617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4213617640 1 -- 172.21.15.144:0/3617506796 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420c141c50 con 0x7f420c09f020 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 1 -- 172.21.15.144:0/3617506796 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00bea0 con 0x7f420c09f020 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42017fa640 10 monclient: dump: 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.426 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f42037fe640 4 set_mon_vals no callback set 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4213617640 10 monclient: _renew_subs 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4213617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.407+0000 7f4213617640 1 -- 172.21.15.144:0/3617506796 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c1412f0 con 0x7f420c09f020 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f42017fa640 1 -- 172.21.15.144:0/3617506796 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f41fc014640 con 0x7f420c09f020 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f4213617640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f4213617640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f4213617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f42017fa640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41f0038490 0x7f41f003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f4213617640 1 -- 172.21.15.144:0/3617506796 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420c141510 con 0x7f420c09f020 2024-09-18T12:27:10.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f42017fa640 1 -- 172.21.15.144:0/3617506796 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f41fc04c170 con 0x7f420c09f020 2024-09-18T12:27:10.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.408+0000 7f42113b3640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41f0038490 0x7f41f003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.409+0000 7f42113b3640 10 monclient: get_auth_request con 0x7f41f0038490 auth_method 0 2024-09-18T12:27:10.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.411+0000 7f42113b3640 1 --2- 172.21.15.144:0/3617506796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f41f0038490 0x7f41f003a940 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f420c0a2b00 tx=0x7f4208032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.413+0000 7f42017fa640 1 -- 172.21.15.144:0/3617506796 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f41fc01e460 con 0x7f420c09f020 2024-09-18T12:27:10.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.413+0000 7f42017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.413+0000 7f42017fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.440+0000 7f70a791d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.440+0000 7f70a791d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.440+0000 7f70a791d640 10 monclient: monmap: 2024-09-18T12:27:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:10.441383+0000 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:10.441383+0000 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:10.442 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 1 Processor -- start 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 1 -- start start 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient: init 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70a00981c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 0x7f70a0136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient(hunting): picked mon.noname-b con 0x7f70a0131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f70a0132170 0x7f70a01390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient(hunting): picked mon.noname-c con 0x7f70a0132170 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a009ca60 con 0x7f70a00981c0 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a0131bd0 con 0x7f70a0132170 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a009cca0 con 0x7f70a0131da0 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.441+0000 7f70a791d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a791d640 10 monclient(hunting): authenticate will time out at 857.763008s 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 0x7f70a0136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 0x7f70a0136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36658/0 (socket says 172.21.15.116:36658) 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 -- 172.21.15.116:0/3350101100 learned_addr learned my addr 172.21.15.116:0/3350101100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): get_auth_request con 0x7f70a0131da0 auth_method 0 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a56b9640 1 --1- 172.21.15.116:0/3350101100 >> v1:172.21.15.144:6789/0 conn(0x7f70a00981c0 0x7f70a0098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:53772/0 (socket says 172.21.15.116:53772) 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 1 --2- 172.21.15.116:0/3350101100 >> v2:172.21.15.144:3300/0 conn(0x7f70a0132170 0x7f70a01390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 10 monclient(hunting): get_auth_request con 0x7f70a0132170 auth_method 0 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient(hunting): handle_auth_done global_id 44132 payload 274 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 -- 172.21.15.116:0/3350101100 >> v2:172.21.15.144:3300/0 conn(0x7f70a0132170 msgr2=0x7f70a01390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 --2- 172.21.15.116:0/3350101100 >> v2:172.21.15.144:3300/0 conn(0x7f70a0132170 0x7f70a01390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 -- 172.21.15.116:0/3350101100 >> v1:172.21.15.144:6789/0 conn(0x7f70a00981c0 legacy=0x7f70a0098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 monclient: found mon.noname-b 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a4eb8640 1 -- 172.21.15.116:0/3350101100 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a009e010 con 0x7f70a0131da0 2024-09-18T12:27:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.442+0000 7f70a5eba640 1 --2- 172.21.15.116:0/3350101100 >> v2:172.21.15.144:3300/0 conn(0x7f70a0132170 0x7f70a01390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f70a4eb8640 1 -- 172.21.15.116:0/3350101100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 msgr2=0x7f70a0136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f70a4eb8640 1 --2- 172.21.15.116:0/3350101100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 0x7f70a0136b70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7090009910 tx=0x7f70900091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40d62221d15e4b49 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 1 -- 172.21.15.116:0/3350101100 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709002ae90 con 0x7f70a0131da0 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: dump: 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 1 -- 172.21.15.116:0/3350101100 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090032440 con 0x7f70a0131da0 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f70a791d640 5 monclient: authenticate success, global_id 44132 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f70a791d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f70a791d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 1 -- 172.21.15.116:0/3350101100 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70900327e0 con 0x7f70a0131da0 2024-09-18T12:27:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f70a791d640 4 set_mon_vals no callback set 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.445+0000 7f7097fff640 10 monclient: dump: 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.457 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 10 monclient: shutdown 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 -- 172.21.15.116:0/3350101100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 msgr2=0x7f70a0136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 --2- 172.21.15.116:0/3350101100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 0x7f70a0136b70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7090009910 tx=0x7f70900091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 -- 172.21.15.116:0/3350101100 shutdown_connections 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 --2- 172.21.15.116:0/3350101100 >> v2:172.21.15.144:3300/0 conn(0x7f70a0132170 0x7f70a01390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 --2- 172.21.15.116:0/3350101100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a0131da0 0x7f70a0136b70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 -- 172.21.15.116:0/3350101100 >> 172.21.15.116:0/3350101100 conn(0x7f70a00a1e30 msgr2=0x7f70a00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 -- 172.21.15.116:0/3350101100 shutdown_connections 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 -- 172.21.15.116:0/3350101100 wait complete. 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 10 monclient: monmap: 2024-09-18T12:27:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.446+0000 7f70a791d640 1 Processor -- start 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 1 -- start start 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient: init 2024-09-18T12:27:10.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 0x7f70a0130da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): picked mon.noname-c con 0x7f70a00981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a0131da0 0x7f70a01312e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): picked mon.noname-a con 0x7f70a0131da0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a0132170 0x7f70a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): picked mon.noname-b con 0x7f70a0132170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a00a08d0 con 0x7f70a0131da0 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a0139360 con 0x7f70a0132170 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a006bc70 con 0x7f70a00981c0 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a791d640 10 monclient(hunting): authenticate will time out at 857.768659s 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 0x7f70a0130da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 0x7f70a0130da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36660/0 (socket says 172.21.15.116:36660) 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 1 -- 172.21.15.116:0/3259457772 learned_addr learned my addr 172.21.15.116:0/3259457772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): get_auth_request con 0x7f70a00981c0 auth_method 0 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a0131da0 0x7f70a01312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 10 monclient(hunting): get_auth_request con 0x7f70a0131da0 auth_method 0 2024-09-18T12:27:10.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a0132170 0x7f70a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a4eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 10 monclient(hunting): get_auth_request con 0x7f70a0132170 auth_method 0 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a5eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.447+0000 7f70a56b9640 10 monclient(hunting): handle_auth_done global_id 44135 payload 824 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 1 -- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a0132170 msgr2=0x7f70a006a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a0132170 0x7f70a006a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 1 -- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a0131da0 msgr2=0x7f70a01312e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a0131da0 0x7f70a01312e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 1 monclient: found mon.noname-c 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.448+0000 7f70a56b9640 1 -- 172.21.15.116:0/3259457772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7098009a20 con 0x7f70a00981c0 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a56b9640 1 -- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 msgr2=0x7f70a0130da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a56b9640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 0x7f70a0130da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c5b0 tx=0x7f709c007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 1 -- 172.21.15.116:0/3259457772 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c0074f0 con 0x7f70a00981c0 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: dump: 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.464 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 1 -- 172.21.15.116:0/3259457772 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c007670 con 0x7f70a00981c0 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a791d640 5 monclient: authenticate success, global_id 44135 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 1 -- 172.21.15.116:0/3259457772 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c005a20 con 0x7f70a00981c0 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a791d640 10 monclient: _renew_subs 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a791d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7090009450 con 0x7f70a00981c0 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70977fe640 4 set_mon_vals no callback set 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70957fa640 10 monclient: dump: 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.466 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a791d640 10 monclient: _renew_subs 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a791d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.450+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a0138c10 con 0x7f70a00981c0 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70957fa640 1 -- 172.21.15.116:0/3259457772 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f709c01b3b0 con 0x7f70a00981c0 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70a791d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70a791d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70a791d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a006cab0 con 0x7f70a00981c0 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70957fa640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70840384c0 0x7f708403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70957fa640 1 -- 172.21.15.116:0/3259457772 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f709c04af80 con 0x7f70a00981c0 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70a4eb8640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70840384c0 0x7f708403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.451+0000 7f70a4eb8640 10 monclient: get_auth_request con 0x7f70840384c0 auth_method 0 2024-09-18T12:27:10.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.454+0000 7f70a4eb8640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70840384c0 0x7f708403a970 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f70a0066560 tx=0x7f7090032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.456+0000 7f70957fa640 1 -- 172.21.15.116:0/3259457772 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f709c0187f0 con 0x7f70a00981c0 2024-09-18T12:27:10.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.456+0000 7f70957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.456+0000 7f70957fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:10.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.567+0000 7f4213617640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-18T12:27:10.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.567+0000 7f4213617640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-18T12:27:10.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.567+0000 7f4213617640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.567+0000 7f4213617640 1 -- 172.21.15.144:0/3617506796 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) v1 -- 0x7f420c0008d0 con 0x7f420c09f020 2024-09-18T12:27:10.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.568+0000 7f42017fa640 1 -- 172.21.15.144:0/3617506796 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v40) v1 ==== 137+0+0 (secure 0 0 0) 0x7f41fc01dbb0 con 0x7f420c09f020 2024-09-18T12:27:10.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.568+0000 7f42017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-18T12:27:10.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.568+0000 7f42017fa640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-18T12:27:10.569 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-18T12:27:10.572 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:10.572 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool delete foo 2024-09-18T12:27:10.573 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:10.573 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foo 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.644+0000 7f99d999d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.644+0000 7f99d999d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.645+0000 7f99d999d640 10 monclient: monmap: 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:10.645907+0000 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:10.645907+0000 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.646 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:10.647 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:10.647 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:10.647 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:10.647 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:10.647 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.645+0000 7f99d999d640 1 Processor -- start 2024-09-18T12:27:10.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.645+0000 7f99d999d640 1 -- start start 2024-09-18T12:27:10.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.645+0000 7f99d999d640 10 monclient: init 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f99d409ec50 0x7f99d4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient(hunting): picked mon.noname-a con 0x7f99d409ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99d41321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99d409f020 0x7f99d4069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient(hunting): picked mon.noname-c con 0x7f99d409f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4067420 con 0x7f99d41321a0 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d40a0480 con 0x7f99d409ec50 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d40a0200 con 0x7f99d409f020 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d999d640 10 monclient(hunting): authenticate will time out at 866.215179s 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99d409f020 0x7f99d4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f99d409f020 0x7f99d4069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51512/0 (socket says 172.21.15.144:51512) 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 -- 172.21.15.144:0/3081147777 learned_addr learned my addr 172.21.15.144:0/3081147777 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): get_auth_request con 0x7f99d409f020 auth_method 0 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 1 --2- 172.21.15.144:0/3081147777 >> v2:172.21.15.116:3300/0 conn(0x7f99d409ec50 0x7f99d4131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 10 monclient(hunting): get_auth_request con 0x7f99d409ec50 auth_method 0 2024-09-18T12:27:10.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient(hunting): handle_auth_done global_id 4636 payload 274 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 -- 172.21.15.144:0/3081147777 >> v2:172.21.15.116:3300/0 conn(0x7f99d409ec50 msgr2=0x7f99d4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 --2- 172.21.15.144:0/3081147777 >> v2:172.21.15.116:3300/0 conn(0x7f99d409ec50 0x7f99d4131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 -- 172.21.15.144:0/3081147777 >> v1:172.21.15.116:6789/0 conn(0x7f99d41321a0 legacy=0x7f99d4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 monclient: found mon.noname-c 2024-09-18T12:27:10.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.646+0000 7f99d37fe640 1 -- 172.21.15.144:0/3081147777 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d409fd80 con 0x7f99d409f020 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d37fe640 1 -- 172.21.15.144:0/3081147777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d409f020 msgr2=0x7f99d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d37fe640 1 --2- 172.21.15.144:0/3081147777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d409f020 0x7f99d4069410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99d409bf50 tx=0x7f99cc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df917835109395c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 1 -- 172.21.15.144:0/3081147777 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc007610 con 0x7f99d409f020 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: dump: 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.659 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 1 -- 172.21.15.144:0/3081147777 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc02d050 con 0x7f99d409f020 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d999d640 5 monclient: authenticate success, global_id 4636 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d999d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:10.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d999d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 1 -- 172.21.15.144:0/3081147777 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc0079a0 con 0x7f99d409f020 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d999d640 4 set_mon_vals no callback set 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.649+0000 7f99d1ffb640 10 monclient: dump: 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.661 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 10 monclient: shutdown 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 -- 172.21.15.144:0/3081147777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d409f020 msgr2=0x7f99d4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 --2- 172.21.15.144:0/3081147777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d409f020 0x7f99d4069410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99d409bf50 tx=0x7f99cc009840 comp rx=0 tx=0).stop 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 -- 172.21.15.144:0/3081147777 shutdown_connections 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 --2- 172.21.15.144:0/3081147777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d409f020 0x7f99d4069410 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 --2- 172.21.15.144:0/3081147777 >> v2:172.21.15.116:3300/0 conn(0x7f99d409ec50 0x7f99d4131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 -- 172.21.15.144:0/3081147777 >> 172.21.15.144:0/3081147777 conn(0x7f99d40a5cf0 msgr2=0x7f99d40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 -- 172.21.15.144:0/3081147777 shutdown_connections 2024-09-18T12:27:10.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 1 -- 172.21.15.144:0/3081147777 wait complete. 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.650+0000 7f99d999d640 10 monclient: monmap: 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 Processor -- start 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 -- start start 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient: init 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99d409ec50 0x7f99d409aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): picked mon.noname-c con 0x7f99d409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99d409f020 0x7f99d4130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): picked mon.noname-a con 0x7f99d409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d41321a0 0x7f99d4140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): picked mon.noname-b con 0x7f99d41321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d409c020 con 0x7f99d409f020 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d409df00 con 0x7f99d41321a0 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d409b730 con 0x7f99d409ec50 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d999d640 10 monclient(hunting): authenticate will time out at 866.220530s 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d41321a0 0x7f99d4140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d41321a0 0x7f99d4140d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51518/0 (socket says 172.21.15.144:51518) 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 -- 172.21.15.144:0/1474592484 learned_addr learned my addr 172.21.15.144:0/1474592484 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): get_auth_request con 0x7f99d41321a0 auth_method 0 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99d409ec50 0x7f99d409aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99d409f020 0x7f99d4130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 10 monclient(hunting): get_auth_request con 0x7f99d409ec50 auth_method 0 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 10 monclient(hunting): get_auth_request con 0x7f99d409f020 auth_method 0 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient(hunting): handle_auth_done global_id 4639 payload 824 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 -- 172.21.15.144:0/1474592484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99d409ec50 msgr2=0x7f99d409aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f99d409ec50 0x7f99d409aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 -- 172.21.15.144:0/1474592484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99d409f020 msgr2=0x7f99d4130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99d409f020 0x7f99d4130be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.651+0000 7f99d37fe640 1 -- 172.21.15.144:0/1474592484 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc009ae0 con 0x7f99d41321a0 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d37fe640 1 -- 172.21.15.144:0/1474592484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d41321a0 msgr2=0x7f99d4140d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d37fe640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99d41321a0 0x7f99d4140d00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f99cc02abb0 tx=0x7f99cc036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 1 -- 172.21.15.144:0/1474592484 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc007170 con 0x7f99d41321a0 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: dump: 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.668 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 1 -- 172.21.15.144:0/1474592484 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc032cc0 con 0x7f99d41321a0 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d999d640 5 monclient: authenticate success, global_id 4639 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d999d640 10 monclient: _renew_subs 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 1 -- 172.21.15.144:0/1474592484 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc03b680 con 0x7f99d41321a0 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d999d640 1 -- 172.21.15.144:0/1474592484 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d411c100 con 0x7f99d41321a0 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99baffd640 10 monclient: dump: 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.670 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d17fa640 4 set_mon_vals no callback set 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d999d640 10 monclient: _renew_subs 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.654+0000 7f99d999d640 1 -- 172.21.15.144:0/1474592484 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d411bc70 con 0x7f99d41321a0 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99baffd640 1 -- 172.21.15.144:0/1474592484 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f99cc038050 con 0x7f99d41321a0 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99d999d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99d999d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99d999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99baffd640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f99c0038490 0x7f99c003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99d999d640 1 -- 172.21.15.144:0/1474592484 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d411be60 con 0x7f99d41321a0 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99baffd640 1 -- 172.21.15.144:0/1474592484 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f99cc06c5a0 con 0x7f99d41321a0 2024-09-18T12:27:10.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99d2ffd640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f99c0038490 0x7f99c003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.655+0000 7f99d2ffd640 10 monclient: get_auth_request con 0x7f99c0038490 auth_method 0 2024-09-18T12:27:10.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.657+0000 7f99d2ffd640 1 --2- 172.21.15.144:0/1474592484 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f99c0038490 0x7f99c003a940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009d20 tx=0x7f99c40090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.659+0000 7f99baffd640 1 -- 172.21.15.144:0/1474592484 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f99cc03f080 con 0x7f99d41321a0 2024-09-18T12:27:10.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.659+0000 7f99baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.659+0000 7f99baffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:10.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.670+0000 7f70a791d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:27:10.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.670+0000 7f70a791d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:27:10.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.670+0000 7f70a791d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.670+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f70a006d080 con 0x7f70a00981c0 2024-09-18T12:27:10.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.670+0000 7f70957fa640 1 -- 172.21.15.116:0/3259457772 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f709c017020 con 0x7f70a00981c0 2024-09-18T12:27:10.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.670+0000 7f70957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:27:10.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.670+0000 7f70957fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:10.674 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70840384c0 msgr2=0x7f708403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70840384c0 0x7f708403a970 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f70a0066560 tx=0x7f7090032000 comp rx=0 tx=0).stop 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 10 monclient: shutdown 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 msgr2=0x7f70a0130da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 0x7f70a0130da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c5b0 tx=0x7f709c007f00 comp rx=0 tx=0).stop 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 shutdown_connections 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f70840384c0 0x7f708403a970 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f70a0132170 0x7f70a006a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70a0131da0 0x7f70a01312e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 --2- 172.21.15.116:0/3259457772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70a00981c0 0x7f70a0130da0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 >> 172.21.15.116:0/3259457772 conn(0x7f70a00a1e30 msgr2=0x7f70a0096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 shutdown_connections 2024-09-18T12:27:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.672+0000 7f70a791d640 1 -- 172.21.15.116:0/3259457772 wait complete. 2024-09-18T12:27:10.683 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:27:10.683 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:27:10.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:27:10.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:27:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.775+0000 7fbdd4f5d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.775+0000 7fbdd4f5d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.775+0000 7fbdd4f5d640 10 monclient: monmap: 2024-09-18T12:27:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:10.776326+0000 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:10.776326+0000 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:10.777 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 1 Processor -- start 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 1 -- start start 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient: init 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbdd0095830 0x7fbdd0097910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd0095830 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdd0095c00 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0096020 0x7fbdd012ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd0096020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd009ce80 con 0x7fbdd0095c00 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd0096b70 con 0x7fbdd0095830 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00965b0 con 0x7fbdd0096020 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.776+0000 7fbdd4f5d640 10 monclient(hunting): authenticate will time out at 858.097913s 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbdd0095830 0x7fbdd0097910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0096020 0x7fbdd012ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdc7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbdd0095c00 0x7fbdd012e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44976/0 (socket says 172.21.15.116:44976) 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdc7fff640 1 -- 172.21.15.116:0/1672595895 learned_addr learned my addr 172.21.15.116:0/1672595895 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): get_auth_request con 0x7fbdd0096020 auth_method 0 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 10 monclient(hunting): get_auth_request con 0x7fbdd0095830 auth_method 0 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient(hunting): handle_auth_done global_id 44141 payload 274 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 1 -- 172.21.15.116:0/1672595895 >> v2:172.21.15.116:3300/0 conn(0x7fbdd0095830 msgr2=0x7fbdd0097910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 1 --2- 172.21.15.116:0/1672595895 >> v2:172.21.15.116:3300/0 conn(0x7fbdd0095830 0x7fbdd0097910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 1 -- 172.21.15.116:0/1672595895 >> v1:172.21.15.116:6789/0 conn(0x7fbdd0095c00 legacy=0x7fbdd012e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 1 monclient: found mon.noname-b 2024-09-18T12:27:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdced9d640 1 -- 172.21.15.116:0/1672595895 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd0096890 con 0x7fbdd0096020 2024-09-18T12:27:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.777+0000 7fbdce59c640 1 --2- 172.21.15.116:0/1672595895 >> v2:172.21.15.116:3300/0 conn(0x7fbdd0095830 0x7fbdd0097910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdced9d640 1 -- 172.21.15.116:0/1672595895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0096020 msgr2=0x7fbdd012ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdced9d640 1 --2- 172.21.15.116:0/1672595895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0096020 0x7fbdd012ed60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009e60 tx=0x7fbdc8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12568c1896b55d0b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 1 -- 172.21.15.116:0/1672595895 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8007760 con 0x7fbdd0096020 2024-09-18T12:27:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: dump: 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 5 monclient: authenticate success, global_id 44141 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 1 -- 172.21.15.116:0/1672595895 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc802d050 con 0x7fbdd0096020 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 4 set_mon_vals no callback set 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 1 -- 172.21.15.116:0/1672595895 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8007af0 con 0x7fbdd0096020 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdcdd9b640 10 monclient: dump: 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.792 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 10 monclient: shutdown 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1672595895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0096020 msgr2=0x7fbdd012ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 --2- 172.21.15.116:0/1672595895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0096020 0x7fbdd012ed60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009e60 tx=0x7fbdc8009780 comp rx=0 tx=0).stop 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1672595895 shutdown_connections 2024-09-18T12:27:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 --2- 172.21.15.116:0/1672595895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0096020 0x7fbdd012ed60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 --2- 172.21.15.116:0/1672595895 >> v2:172.21.15.116:3300/0 conn(0x7fbdd0095830 0x7fbdd0097910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1672595895 >> 172.21.15.116:0/1672595895 conn(0x7fbdd0067b50 msgr2=0x7fbdd00a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1672595895 shutdown_connections 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.780+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1672595895 wait complete. 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 10 monclient: monmap: 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:10.794 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 1 Processor -- start 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 1 -- start start 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 10 monclient: init 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.781+0000 7fbdd4f5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0095830 0x7fbdd008b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd0095830 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdd0095c00 0x7fbdd008bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd0095c00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd0096020 0x7fbdd008c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd0096020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0097e80 con 0x7fbdd0096020 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd009aab0 con 0x7fbdd0095c00 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd008de90 con 0x7fbdd0095830 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdd4f5d640 10 monclient(hunting): authenticate will time out at 858.103059s 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0095830 0x7fbdd008b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0095830 0x7fbdd008b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36676/0 (socket says 172.21.15.116:36676) 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 -- 172.21.15.116:0/1195534215 learned_addr learned my addr 172.21.15.116:0/1195534215 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): get_auth_request con 0x7fbdd0095830 auth_method 0 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd0096020 0x7fbdd008c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 10 monclient(hunting): get_auth_request con 0x7fbdd0096020 auth_method 0 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdc7fff640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdd0095c00 0x7fbdd008bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdc7fff640 10 monclient(hunting): get_auth_request con 0x7fbdd0095c00 auth_method 0 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdc7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient(hunting): handle_auth_done global_id 44144 payload 824 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdd0095c00 msgr2=0x7fbdd008bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdd0095c00 0x7fbdd008bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd0096020 msgr2=0x7fbdd008c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd0096020 0x7fbdd008c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 monclient: found mon.noname-c 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdce59c640 1 -- 172.21.15.116:0/1195534215 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc0009390 con 0x7fbdd0095830 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.782+0000 7fbdc7fff640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbdd0095c00 0x7fbdd008bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdce59c640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0095830 msgr2=0x7fbdd008b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdce59c640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0095830 0x7fbdd008b780 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000eaa0 tx=0x7fbdc00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc000cc20 con 0x7fbdd0095830 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: dump: 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.799 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdd4f5d640 5 monclient: authenticate success, global_id 44144 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc000cd80 con 0x7fbdd0095830 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdd4f5d640 10 monclient: _renew_subs 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdd4f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc8009a20 con 0x7fbdd0095830 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc00136a0 con 0x7fbdd0095830 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdc67fc640 10 monclient: dump: 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdd4f5d640 10 monclient: _renew_subs 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdd4f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdcd59a640 4 set_mon_vals no callback set 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.784+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd0094d70 con 0x7fbdd0095830 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbdc0013a50 con 0x7fbdd0095830 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdd4f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdd4f5d640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 0x7fbdd009ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdd4f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0063720 con 0x7fbdd008c740 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc67fc640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbdb0038490 0x7fbdb003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fbdc004bf30 con 0x7fbdd0095830 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 0x7fbdd009ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 10 monclient: get_auth_request con 0x7fbdd008c740 auth_method 0 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdced9d640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbdb0038490 0x7fbdb003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.785+0000 7fbdced9d640 10 monclient: get_auth_request con 0x7fbdb0038490 auth_method 0 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.786+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_done global_id 4683 payload 824 2024-09-18T12:27:10.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.786+0000 7fbdc7fff640 1 -- 172.21.15.116:0/1195534215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdd008ef50 con 0x7fbdd008c740 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc7fff640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 msgr2=0x7fbdd009ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc7fff640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 0x7fbdd009ba70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e740 tx=0x7fbdb8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8005cb0 con 0x7fbdd008c740 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 10 monclient: dump: 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.804 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdced9d640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbdb0038490 0x7fbdb003a940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbdd009cc80 tx=0x7fbdc8037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbdb8005e10 con 0x7fbdd008c740 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 msgr2=0x7fbdd009ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 0x7fbdd009ba70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e740 tx=0x7fbdb8009040 comp rx=0 tx=0).stop 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 msgr2=0x7fbdd009ba70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.788+0000 7fbdc67fc640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd008c740 0x7fbdd009ba70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdd4f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdd4f5d640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 0x7fbdd009c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdd4f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd008f290 con 0x7fbdd009bfb0 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 0x7fbdd009c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient: get_auth_request con 0x7fbdd009bfb0 auth_method 0 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 10 monclient(hunting): handle_auth_done global_id 4686 payload 824 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.794+0000 7fbdc7fff640 1 -- 172.21.15.116:0/1195534215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbdd004bf60 con 0x7fbdd009bfb0 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.796+0000 7fbdc7fff640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 msgr2=0x7fbdd009c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.796+0000 7fbdc7fff640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 0x7fbdd009c3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e740 tx=0x7fbdb801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.796+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800c820 con 0x7fbdd009bfb0 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.796+0000 7fbdc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.796+0000 7fbdc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.796+0000 7fbdc67fc640 10 monclient: dump: 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fbdb8005aa0 con 0x7fbdd009bfb0 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdc67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdc67fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 msgr2=0x7fbdd009c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdc67fc640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 0x7fbdd009c3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e740 tx=0x7fbdb801a000 comp rx=0 tx=0).stop 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdc67fc640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 msgr2=0x7fbdd009c3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdc67fc640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdd009bfb0 0x7fbdd009c3b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:10.810 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 24, 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:10.811 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:10.812 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:10.813 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:10.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:10.816 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:10.819 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.820 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:10.822 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbdb0038490 msgr2=0x7fbdb003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbdb0038490 0x7fbdb003a940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbdd009cc80 tx=0x7fbdc8037000 comp rx=0 tx=0).stop 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 10 monclient: shutdown 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0095830 msgr2=0x7fbdd008b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 --2- 172.21.15.116:0/1195534215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbdd0095830 0x7fbdd008b780 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000eaa0 tx=0x7fbdc00090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdced9d640 1 -- 172.21.15.116:0/1195534215 reap_dead start 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdced9d640 1 -- 172.21.15.116:0/1195534215 reap_dead start 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 shutdown_connections 2024-09-18T12:27:10.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 >> 172.21.15.116:0/1195534215 conn(0x7fbdd0067b50 msgr2=0x7fbdd00992e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 shutdown_connections 2024-09-18T12:27:10.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.797+0000 7fbdd4f5d640 1 -- 172.21.15.116:0/1195534215 wait complete. 2024-09-18T12:27:10.825 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:27:10.825 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:27:10.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:27:10.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.816+0000 7f99d999d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-18T12:27:10.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.816+0000 7f99d999d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-18T12:27:10.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.816+0000 7f99d999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.816+0000 7f99d999d640 1 -- 172.21.15.144:0/1474592484 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) v1 -- 0x7f99d40008d0 con 0x7f99d41321a0 2024-09-18T12:27:10.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.817+0000 7f99baffd640 1 -- 172.21.15.144:0/1474592484 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v40) v1 ==== 274+0+0 (secure 0 0 0) 0x7f99cc032e30 con 0x7f99d41321a0 2024-09-18T12:27:10.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.817+0000 7f99baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-18T12:27:10.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.817+0000 7f99baffd640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T12:27:10.827 INFO:tasks.workunit.client.0.smithi144.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T12:27:10.827 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:10.827 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-18T12:27:10.827 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:10.827 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foo foo 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.895+0000 7fb439b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.895+0000 7fb439b9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.895+0000 7fb439b9d640 10 monclient: monmap: 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:10.896761+0000 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:10.896761+0000 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:10.897 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:10.898 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:10.898 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:10.898 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:10.898 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.896+0000 7fb439b9d640 1 Processor -- start 2024-09-18T12:27:10.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.896+0000 7fb439b9d640 1 -- start start 2024-09-18T12:27:10.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.896+0000 7fb439b9d640 10 monclient: init 2024-09-18T12:27:10.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T12:27:10.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb43409cad0 0x7fb43409c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb43409cad0 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb43409cea0 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb43409d2a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43409b710 con 0x7fb43409d2a0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43409c6e0 con 0x7fb43409cea0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43409e8f0 con 0x7fb43409cad0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb439b9d640 10 monclient(hunting): authenticate will time out at 866.466202s 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb432ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb43409cea0 0x7fb43409dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:35980/0 (socket says 172.21.15.144:35980) 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb432ffd640 1 -- 172.21.15.144:0/2563710025 learned_addr learned my addr 172.21.15.144:0/2563710025 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb43409cad0 0x7fb43409c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): get_auth_request con 0x7fb43409cad0 auth_method 0 2024-09-18T12:27:10.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient(hunting): handle_auth_done global_id 4645 payload 274 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 1 -- 172.21.15.144:0/2563710025 >> v1:172.21.15.144:6789/0 conn(0x7fb43409cea0 legacy=0x7fb43409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 1 -- 172.21.15.144:0/2563710025 >> v1:172.21.15.116:6789/0 conn(0x7fb43409d2a0 legacy=0x7fb43409e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 1 monclient: found mon.noname-c 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.897+0000 7fb4337fe640 1 -- 172.21.15.144:0/2563710025 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43409a650 con 0x7fb43409cad0 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4337fe640 1 -- 172.21.15.144:0/2563710025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409cad0 msgr2=0x7fb43409c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4337fe640 1 --2- 172.21.15.144:0/2563710025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409cad0 0x7fb43409c1a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb424009e60 tx=0x7fb424009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f57e30d08bee6d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 1 -- 172.21.15.144:0/2563710025 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424007840 con 0x7fb43409cad0 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: dump: 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.909 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 1 -- 172.21.15.144:0/2563710025 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42402d050 con 0x7fb43409cad0 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb439b9d640 5 monclient: authenticate success, global_id 4645 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb439b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 1 -- 172.21.15.144:0/2563710025 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb424007bd0 con 0x7fb43409cad0 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb439b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb439b9d640 4 set_mon_vals no callback set 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.900+0000 7fb4327fc640 10 monclient: dump: 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.911 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 10 monclient: shutdown 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 1 -- 172.21.15.144:0/2563710025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409cad0 msgr2=0x7fb43409c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 1 --2- 172.21.15.144:0/2563710025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409cad0 0x7fb43409c1a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb424009e60 tx=0x7fb424009780 comp rx=0 tx=0).stop 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 1 -- 172.21.15.144:0/2563710025 shutdown_connections 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 1 --2- 172.21.15.144:0/2563710025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409cad0 0x7fb43409c1a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 1 -- 172.21.15.144:0/2563710025 >> 172.21.15.144:0/2563710025 conn(0x7fb434070e90 msgr2=0x7fb4340a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 1 -- 172.21.15.144:0/2563710025 shutdown_connections 2024-09-18T12:27:10.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 1 -- 172.21.15.144:0/2563710025 wait complete. 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.901+0000 7fb439b9d640 10 monclient: monmap: 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 Processor -- start 2024-09-18T12:27:10.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 -- start start 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient: init 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43409cad0 0x7fb434141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb43409cad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43409cea0 0x7fb43412fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb43409cea0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409d2a0 0x7fb434130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb43409d2a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43411f2e0 con 0x7fb43409cea0 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43406eba0 con 0x7fb43409d2a0 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4341423f0 con 0x7fb43409cad0 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb439b9d640 10 monclient(hunting): authenticate will time out at 866.471749s 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409d2a0 0x7fb434130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409d2a0 0x7fb434130370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51548/0 (socket says 172.21.15.144:51548) 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 1 -- 172.21.15.144:0/4173744090 learned_addr learned my addr 172.21.15.144:0/4173744090 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 10 monclient(hunting): get_auth_request con 0x7fb43409d2a0 auth_method 0 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb432ffd640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43409cea0 0x7fb43412fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb433fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb432ffd640 10 monclient(hunting): get_auth_request con 0x7fb43409cea0 auth_method 0 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb432ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.902+0000 7fb4337fe640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43409cad0 0x7fb434141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb4337fe640 10 monclient(hunting): get_auth_request con 0x7fb43409cad0 auth_method 0 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb4337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb4337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb432ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb432ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb432ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 10 monclient(hunting): handle_auth_done global_id 4648 payload 824 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 1 -- 172.21.15.144:0/4173744090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43409cad0 msgr2=0x7fb434141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43409cad0 0x7fb434141510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 1 -- 172.21.15.144:0/4173744090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43409cea0 msgr2=0x7fb43412fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43409cea0 0x7fb43412fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 1 monclient: found mon.noname-b 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb433fff640 1 -- 172.21.15.144:0/4173744090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb424009a20 con 0x7fb43409d2a0 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.903+0000 7fb4337fe640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43409cad0 0x7fb434141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb433fff640 1 -- 172.21.15.144:0/4173744090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409d2a0 msgr2=0x7fb434130370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb433fff640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43409d2a0 0x7fb434130370 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00c5e0 tx=0x7fb42c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 1 -- 172.21.15.144:0/4173744090 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c012840 con 0x7fb43409d2a0 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: dump: 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.918 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb439b9d640 5 monclient: authenticate success, global_id 4648 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 1 -- 172.21.15.144:0/4173744090 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c0129a0 con 0x7fb43409d2a0 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb439b9d640 10 monclient: _renew_subs 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb439b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb439b9d640 1 -- 172.21.15.144:0/4173744090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43411ef70 con 0x7fb43409d2a0 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 1 -- 172.21.15.144:0/4173744090 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c012da0 con 0x7fb43409d2a0 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb41b7fe640 10 monclient: dump: 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:10.919 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb439b9d640 10 monclient: _renew_subs 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb439b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb439b9d640 1 -- 172.21.15.144:0/4173744090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43411e080 con 0x7fb43409d2a0 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.905+0000 7fb431ffb640 4 set_mon_vals no callback set 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb41b7fe640 1 -- 172.21.15.144:0/4173744090 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb42c011590 con 0x7fb43409d2a0 2024-09-18T12:27:10.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb439b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb439b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb439b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb439b9d640 1 -- 172.21.15.144:0/4173744090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4340008d0 con 0x7fb43409d2a0 2024-09-18T12:27:10.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb41b7fe640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb420038490 0x7fb42003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb41b7fe640 1 -- 172.21.15.144:0/4173744090 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fb42c04bae0 con 0x7fb43409d2a0 2024-09-18T12:27:10.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb4337fe640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb420038490 0x7fb42003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.906+0000 7fb4337fe640 10 monclient: get_auth_request con 0x7fb420038490 auth_method 0 2024-09-18T12:27:10.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.909+0000 7fb4337fe640 1 --2- 172.21.15.144:0/4173744090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb420038490 0x7fb42003a940 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb43409bec0 tx=0x7fb424037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.911+0000 7fb41b7fe640 1 -- 172.21.15.144:0/4173744090 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb42c04de60 con 0x7fb43409d2a0 2024-09-18T12:27:10.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.911+0000 7fb41b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:10.911+0000 7fb41b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.920+0000 7fd46cd00640 10 monclient: get_monmap_and_config 2024-09-18T12:27:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.920+0000 7fd46cd00640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.920+0000 7fd46cd00640 10 monclient: monmap: 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:10.921593+0000 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:10.921593+0000 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.921+0000 7fd46cd00640 1 Processor -- start 2024-09-18T12:27:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.921+0000 7fd46cd00640 1 -- start start 2024-09-18T12:27:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.921+0000 7fd46cd00640 10 monclient: init 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd468131980 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd46809fda0 0x7fd46809e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient(hunting): picked mon.noname-c con 0x7fd46809fda0 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd46809ef00 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4680a1180 con 0x7fd46809ef00 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4680a1460 con 0x7fd468131980 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd468132460 con 0x7fd46809fda0 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd46cd00640 10 monclient(hunting): authenticate will time out at 858.243262s 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd466d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd46809ef00 0x7fd468138cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44992/0 (socket says 172.21.15.116:44992) 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd466d9d640 1 -- 172.21.15.116:0/945306019 learned_addr learned my addr 172.21.15.116:0/945306019 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 1 --2- 172.21.15.116:0/945306019 >> v2:172.21.15.144:3300/0 conn(0x7fd46809fda0 0x7fd46809e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): get_auth_request con 0x7fd46809fda0 auth_method 0 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient(hunting): handle_auth_done global_id 4651 payload 274 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 1 -- 172.21.15.116:0/945306019 >> v1:172.21.15.144:6789/0 conn(0x7fd468131980 legacy=0x7fd468131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 1 -- 172.21.15.116:0/945306019 >> v1:172.21.15.116:6789/0 conn(0x7fd46809ef00 legacy=0x7fd468138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 1 monclient: found mon.noname-c 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.922+0000 7fd465d9b640 1 -- 172.21.15.116:0/945306019 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4681325a0 con 0x7fd46809fda0 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd465d9b640 1 -- 172.21.15.116:0/945306019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd46809fda0 msgr2=0x7fd46809e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd465d9b640 1 --2- 172.21.15.116:0/945306019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd46809fda0 0x7fd46809e9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009850 tx=0x7fd45c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f22aabcbd642d782 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 1 -- 172.21.15.116:0/945306019 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c032440 con 0x7fd46809fda0 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: dump: 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.936 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 1 -- 172.21.15.116:0/945306019 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c0325a0 con 0x7fd46809fda0 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46cd00640 5 monclient: authenticate success, global_id 4651 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46cd00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46cd00640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 1 -- 172.21.15.116:0/945306019 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c032940 con 0x7fd46809fda0 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46cd00640 4 set_mon_vals no callback set 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.925+0000 7fd46559a640 10 monclient: dump: 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.937 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 10 monclient: shutdown 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 1 -- 172.21.15.116:0/945306019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd46809fda0 msgr2=0x7fd46809e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 1 --2- 172.21.15.116:0/945306019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd46809fda0 0x7fd46809e9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009850 tx=0x7fd45c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 1 -- 172.21.15.116:0/945306019 shutdown_connections 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 1 --2- 172.21.15.116:0/945306019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd46809fda0 0x7fd46809e9c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 1 -- 172.21.15.116:0/945306019 >> 172.21.15.116:0/945306019 conn(0x7fd468067bd0 msgr2=0x7fd4680a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 1 -- 172.21.15.116:0/945306019 shutdown_connections 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.926+0000 7fd46cd00640 1 -- 172.21.15.116:0/945306019 wait complete. 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient: build_initial_monmap 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient: monmap: 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 Processor -- start 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 -- start start 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient: init 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd46809ef00 0x7fd4681382e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): picked mon.noname-a con 0x7fd46809ef00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46809fda0 0x7fd468134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): picked mon.noname-c con 0x7fd46809fda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd468131980 0x7fd468134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): picked mon.noname-b con 0x7fd468131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd468137900 con 0x7fd46809ef00 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4681398a0 con 0x7fd468131980 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd468137620 con 0x7fd46809fda0 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46cd00640 10 monclient(hunting): authenticate will time out at 858.248839s 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd46809ef00 0x7fd4681382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd465d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46809fda0 0x7fd468134010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd46809ef00 0x7fd4681382e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55624/0 (socket says 172.21.15.116:55624) 2024-09-18T12:27:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46659c640 1 -- 172.21.15.116:0/507944336 learned_addr learned my addr 172.21.15.116:0/507944336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46659c640 10 monclient(hunting): get_auth_request con 0x7fd46809ef00 auth_method 0 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd46659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd465d9b640 10 monclient(hunting): get_auth_request con 0x7fd46809fda0 auth_method 0 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd465d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd465d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.927+0000 7fd465d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd466d9d640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd468131980 0x7fd468134550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd466d9d640 10 monclient(hunting): get_auth_request con 0x7fd468131980 auth_method 0 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd466d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd466d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd466d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 10 monclient(hunting): handle_auth_done global_id 4692 payload 824 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46809fda0 msgr2=0x7fd468134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46809fda0 0x7fd468134010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd468131980 msgr2=0x7fd468134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd468131980 0x7fd468134550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 1 monclient: found mon.noname-a 2024-09-18T12:27:10.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd46659c640 1 -- 172.21.15.116:0/507944336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c009390 con 0x7fd46809ef00 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd466d9d640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd468131980 0x7fd468134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.928+0000 7fd465d9b640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46809fda0 0x7fd468134010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46659c640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd46809ef00 msgr2=0x7fd4681382e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46659c640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd46809ef00 0x7fd4681382e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd45800c8e0 tx=0x7fd4580026c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45800aa60 con 0x7fd46809ef00 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: dump: 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45800abe0 con 0x7fd46809ef00 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46cd00640 5 monclient: authenticate success, global_id 4692 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46cd00640 10 monclient: _renew_subs 2024-09-18T12:27:10.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46cd00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd468139450 con 0x7fd46809ef00 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458012420 con 0x7fd46809ef00 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd4567fc640 10 monclient: dump: 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd464d99640 4 set_mon_vals no callback set 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46cd00640 10 monclient: _renew_subs 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46cd00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.930+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4681389c0 con 0x7fd46809ef00 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd46cd00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd46cd00640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 0x7fd468134a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd46cd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd468065e00 con 0x7fd46802aa40 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd458012890 con 0x7fd46809ef00 2024-09-18T12:27:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 0x7fd468134a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient: get_auth_request con 0x7fd46802aa40 auth_method 0 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd4567fc640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd448038490 0x7fd44803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 10 monclient(hunting): handle_auth_done global_id 44153 payload 824 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd465d9b640 1 -- 172.21.15.116:0/507944336 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd46802ae10 con 0x7fd46802aa40 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.931+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fd45804bf20 con 0x7fd46809ef00 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.932+0000 7fd466d9d640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd448038490 0x7fd44803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.932+0000 7fd466d9d640 10 monclient: get_auth_request con 0x7fd448038490 auth_method 0 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd465d9b640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 msgr2=0x7fd468134a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd465d9b640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 0x7fd468134a90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd468068aa0 tx=0x7fd45c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c032440 con 0x7fd46802aa40 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 10 monclient: dump: 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.949 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd466d9d640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd448038490 0x7fd44803a940 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd460007fd0 tx=0x7fd460009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd45c031e90 con 0x7fd46802aa40 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 msgr2=0x7fd468134a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 0x7fd468134a90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd468068aa0 tx=0x7fd45c032000 comp rx=0 tx=0).stop 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 msgr2=0x7fd468134a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:10.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.934+0000 7fd4567fc640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46802aa40 0x7fd468134a90 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd468068aa0 tx=0x7fd45c032000 comp rx=0 tx=0).stop 2024-09-18T12:27:10.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd46cd00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:10.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd46cd00640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 0x7fd46805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:10.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd46cd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:10.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46809bf80 con 0x7fd46805e310 2024-09-18T12:27:10.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 0x7fd46805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:10.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient: get_auth_request con 0x7fd46805e310 auth_method 0 2024-09-18T12:27:10.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 10 monclient(hunting): handle_auth_done global_id 44156 payload 824 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.940+0000 7fd465d9b640 1 -- 172.21.15.116:0/507944336 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd46804bf60 con 0x7fd46805e310 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd465d9b640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 msgr2=0x7fd46805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd465d9b640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 0x7fd46805e710 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009130 tx=0x7fd45c03e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c047c30 con 0x7fd46805e310 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:10.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 10 monclient: dump: 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fd45c03c380 con 0x7fd46805e310 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:10.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 msgr2=0x7fd46805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 0x7fd46805e710 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009130 tx=0x7fd45c03e000 comp rx=0 tx=0).stop 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 msgr2=0x7fd46805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.942+0000 7fd4567fc640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd46805e310 0x7fd46805e710 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009130 tx=0x7fd45c03e000 comp rx=0 tx=0).stop 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 24, 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:10.967 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd448038490 msgr2=0x7fd44803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd448038490 0x7fd44803a940 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd460007fd0 tx=0x7fd460009040 comp rx=0 tx=0).stop 2024-09-18T12:27:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 10 monclient: shutdown 2024-09-18T12:27:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd46809ef00 msgr2=0x7fd4681382e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 --2- 172.21.15.116:0/507944336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd46809ef00 0x7fd4681382e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd45800c8e0 tx=0x7fd4580026c0 comp rx=0 tx=0).stop 2024-09-18T12:27:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd466d9d640 1 -- 172.21.15.116:0/507944336 reap_dead start 2024-09-18T12:27:10.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd466d9d640 1 -- 172.21.15.116:0/507944336 reap_dead start 2024-09-18T12:27:10.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 shutdown_connections 2024-09-18T12:27:10.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 >> 172.21.15.116:0/507944336 conn(0x7fd468067bd0 msgr2=0x7fd4680a4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:10.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 shutdown_connections 2024-09-18T12:27:10.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:10.943+0000 7fd46cd00640 1 -- 172.21.15.116:0/507944336 wait complete. 2024-09-18T12:27:10.969 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:27:10.969 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:27:10.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:27:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.062+0000 7f1ddbf1d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.062+0000 7f1ddbf1d640 10 monclient: build_initial_monmap 2024-09-18T12:27:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.062+0000 7f1ddbf1d640 10 monclient: monmap: 2024-09-18T12:27:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:11.063007+0000 2024-09-18T12:27:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:11.063007+0000 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:11.064 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.062+0000 7f1ddbf1d640 1 Processor -- start 2024-09-18T12:27:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 1 -- start start 2024-09-18T12:27:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient: init 2024-09-18T12:27:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1dd4096c60 0x7f1dd409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1dd4096c60 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1dd4097030 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1dd4133b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd40696a0 con 0x7f1dd4097030 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd409d2e0 con 0x7f1dd4133b40 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd409ac60 con 0x7f1dd4096c60 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1ddbf1d640 10 monclient(hunting): authenticate will time out at 858.384638s 2024-09-18T12:27:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd94b8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1dd4097030 0x7f1dd4097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45006/0 (socket says 172.21.15.116:45006) 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd94b8640 1 -- 172.21.15.116:0/1169851804 learned_addr learned my addr 172.21.15.116:0/1169851804 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd9cb9640 1 --2- 172.21.15.116:0/1169851804 >> v2:172.21.15.144:3300/0 conn(0x7f1dd4096c60 0x7f1dd409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd9cb9640 10 monclient(hunting): get_auth_request con 0x7f1dd4096c60 auth_method 0 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd9cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd9cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd9cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.063+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_done global_id 4657 payload 274 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1169851804 >> v1:172.21.15.144:6789/0 conn(0x7f1dd4133b40 legacy=0x7f1dd4135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1169851804 >> v1:172.21.15.116:6789/0 conn(0x7f1dd4097030 legacy=0x7f1dd4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 10 monclient: _finish_hunting 0 2024-09-18T12:27:11.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 1 monclient: found mon.noname-c 2024-09-18T12:27:11.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:11.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.064+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1169851804 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd409d6b0 con 0x7f1dd4096c60 2024-09-18T12:27:11.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.067+0000 7fb439b9d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-18T12:27:11.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.067+0000 7fb439b9d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-18T12:27:11.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.067+0000 7fb439b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.067+0000 7fb439b9d640 1 -- 172.21.15.144:0/4173744090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) v1 -- 0x7fb43409f660 con 0x7fb43409d2a0 2024-09-18T12:27:11.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.068+0000 7fb41b7fe640 1 -- 172.21.15.144:0/4173744090 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v40) v1 ==== 290+0+0 (secure 0 0 0) 0x7fb42c010ac0 con 0x7fb43409d2a0 2024-09-18T12:27:11.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.068+0000 7fb41b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-18T12:27:11.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.068+0000 7fb41b7fe640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T12:27:11.071 INFO:tasks.workunit.client.0.smithi144.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T12:27:11.075 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:11.075 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-18T12:27:11.075 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:11.076 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foo foo --force 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1169851804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 msgr2=0x7f1dd409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd9cb9640 1 --2- 172.21.15.116:0/1169851804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 0x7f1dd409a6c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009850 tx=0x7f1dc80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=607b07ae587fed81 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 1 -- 172.21.15.116:0/1169851804 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8032440 con 0x7f1dd4096c60 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: dump: 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.079 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: _finish_auth 0 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 1 -- 172.21.15.116:0/1169851804 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc80325a0 con 0x7f1dd4096c60 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 5 monclient: authenticate success, global_id 4657 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 1 -- 172.21.15.116:0/1169851804 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8032980 con 0x7f1dd4096c60 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1dd8cb7640 10 monclient: dump: 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:11.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 4 set_mon_vals no callback set 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 10 monclient: shutdown 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1169851804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 msgr2=0x7f1dd409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 1 --2- 172.21.15.116:0/1169851804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 0x7f1dd409a6c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009850 tx=0x7f1dc80090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1169851804 shutdown_connections 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 1 --2- 172.21.15.116:0/1169851804 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 0x7f1dd409a6c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1169851804 >> 172.21.15.116:0/1169851804 conn(0x7f1dd40672b0 msgr2=0x7f1dd40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1169851804 shutdown_connections 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.067+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1169851804 wait complete. 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.068+0000 7f1ddbf1d640 10 monclient: build_initial_monmap 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.068+0000 7f1ddbf1d640 10 monclient: monmap: 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.068+0000 7f1ddbf1d640 1 Processor -- start 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.068+0000 7f1ddbf1d640 1 -- start start 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.068+0000 7f1ddbf1d640 10 monclient: init 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 0x7f1dd406f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f1dd4096c60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dd4097030 0x7f1dd412e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1dd4097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd4133b40 0x7f1dd412ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1dd4133b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd411bda0 con 0x7f1dd4133b40 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd406e8e0 con 0x7f1dd4096c60 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd406d010 con 0x7f1dd4097030 2024-09-18T12:27:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1ddbf1d640 10 monclient(hunting): authenticate will time out at 858.390242s 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dd4097030 0x7f1dd412e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 0x7f1dd406f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dd4097030 0x7f1dd412e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36700/0 (socket says 172.21.15.116:36700) 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 1 -- 172.21.15.116:0/1606809392 learned_addr learned my addr 172.21.15.116:0/1606809392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): get_auth_request con 0x7f1dd4096c60 auth_method 0 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 10 monclient(hunting): get_auth_request con 0x7f1dd4097030 auth_method 0 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd4133b40 0x7f1dd412ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 10 monclient(hunting): get_auth_request con 0x7f1dd4133b40 auth_method 0 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient(hunting): handle_auth_done global_id 4660 payload 824 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dd4097030 msgr2=0x7f1dd412e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dd4097030 0x7f1dd412e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd4133b40 msgr2=0x7f1dd412ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd4133b40 0x7f1dd412ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient: _finish_hunting 0 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 1 monclient: found mon.noname-b 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1606809392 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc8009390 con 0x7f1dd4096c60 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dd94b8640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dd4097030 0x7f1dd412e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.069+0000 7f1dda4ba640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd4133b40 0x7f1dd412ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:27:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.071+0000 7f1dd9cb9640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 msgr2=0x7f1dd406f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.071+0000 7f1dd9cb9640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 0x7f1dd406f2c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8031000 tx=0x7f1dc8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8031990 con 0x7f1dd4096c60 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: dump: 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc8031b10 con 0x7f1dd4096c60 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1ddbf1d640 5 monclient: authenticate success, global_id 4660 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1ddbf1d640 10 monclient: _renew_subs 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1ddbf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd406ebc0 con 0x7f1dd4096c60 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8044410 con 0x7f1dd4096c60 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc5ffb640 10 monclient: dump: 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1ddbf1d640 10 monclient: _renew_subs 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1ddbf1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd409ba40 con 0x7f1dd4096c60 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.072+0000 7f1dc7fff640 4 set_mon_vals no callback set 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1ddbf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1ddbf1d640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 0x7f1dd406c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1ddbf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd4065f20 con 0x7f1dd406bf90 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1dc8039040 con 0x7f1dd4096c60 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 0x7f1dd406c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient: get_auth_request con 0x7f1dd406bf90 auth_method 0 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dc5ffb640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1db8038490 0x7f1db803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f1dc806c190 con 0x7f1dd4096c60 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_done global_id 4663 payload 824 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dd94b8640 1 -- 172.21.15.116:0/1606809392 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1dd409c3f0 con 0x7f1dd406bf90 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dda4ba640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1db8038490 0x7f1db803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.073+0000 7f1dda4ba640 10 monclient: get_auth_request con 0x7f1db8038490 auth_method 0 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.075+0000 7f1dd94b8640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 msgr2=0x7f1dd406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.075+0000 7f1dd94b8640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 0x7f1dd406c360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0007320 tx=0x7f1dd0004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.075+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000fd00 con 0x7f1dd406bf90 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.075+0000 7f1dc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.075+0000 7f1dc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.075+0000 7f1dc5ffb640 10 monclient: dump: 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dda4ba640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1db8038490 0x7f1db803a940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc007fd0 tx=0x7f1dcc0025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1dd000fe60 con 0x7f1dd406bf90 2024-09-18T12:27:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dc5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dc5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 msgr2=0x7f1dd406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dc5ffb640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 0x7f1dd406c360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0007320 tx=0x7f1dd0004960 comp rx=0 tx=0).stop 2024-09-18T12:27:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 msgr2=0x7f1dd406c360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.076+0000 7f1dc5ffb640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406bf90 0x7f1dd406c360 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0007320 tx=0x7f1dd0004960 comp rx=0 tx=0).stop 2024-09-18T12:27:11.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.081+0000 7f1ddbf1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.081+0000 7f1ddbf1d640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 0x7f1dd41318a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.081+0000 7f1ddbf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.081+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd406ed50 con 0x7f1dd406c8a0 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 0x7f1dd41318a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient: get_auth_request con 0x7f1dd406c8a0 auth_method 0 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 10 monclient(hunting): handle_auth_done global_id 4666 payload 824 2024-09-18T12:27:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.082+0000 7f1dd94b8640 1 -- 172.21.15.116:0/1606809392 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1dd404bf60 con 0x7f1dd406c8a0 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.084+0000 7f1dd94b8640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 msgr2=0x7f1dd41318a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.084+0000 7f1dd94b8640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 0x7f1dd41318a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0017000 tx=0x7f1dd0018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.084+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0021870 con 0x7f1dd406c8a0 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.084+0000 7f1dc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.084+0000 7f1dc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.084+0000 7f1dc5ffb640 10 monclient: dump: 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:11.097 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f1dd00219d0 con 0x7f1dd406c8a0 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dc5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dc5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 msgr2=0x7f1dd41318a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dc5ffb640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 0x7f1dd41318a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0017000 tx=0x7f1dd0018000 comp rx=0 tx=0).stop 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dc5ffb640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 msgr2=0x7f1dd41318a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dc5ffb640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd406c8a0 0x7f1dd41318a0 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0017000 tx=0x7f1dd0018000 comp rx=0 tx=0).stop 2024-09-18T12:27:11.098 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 24, 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:11.099 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:11.100 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:11.101 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:11.103 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.104 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.105 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:11.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.107 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:11.108 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:27:11.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:11.111 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1db8038490 msgr2=0x7f1db803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1db8038490 0x7f1db803a940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc007fd0 tx=0x7f1dcc0025b0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 10 monclient: shutdown 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 msgr2=0x7f1dd406f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 --2- 172.21.15.116:0/1606809392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dd4096c60 0x7f1dd406f2c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8031000 tx=0x7f1dc8032000 comp rx=0 tx=0).stop 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dda4ba640 1 -- 172.21.15.116:0/1606809392 reap_dead start 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1dda4ba640 1 -- 172.21.15.116:0/1606809392 reap_dead start 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 shutdown_connections 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 >> 172.21.15.116:0/1606809392 conn(0x7f1dd40672b0 msgr2=0x7f1dd41351d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 shutdown_connections 2024-09-18T12:27:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:11.085+0000 7f1ddbf1d640 1 -- 172.21.15.116:0/1606809392 wait complete. 2024-09-18T12:27:11.114 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:27:11.114 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:27:11.114 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-18T12:27:11.114 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:27:11.114 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:27:11.114 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:27:11.114 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:27:11.125 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:27:11.125 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T12:27:11.126 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T12:27:11.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.148+0000 7fce74914640 10 monclient: get_monmap_and_config 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.148+0000 7fce74914640 10 monclient: build_initial_monmap 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.148+0000 7fce74914640 10 monclient: monmap: 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:11.148933+0000 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:11.148933+0000 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:11.149 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:11.150 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:11.150 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:11.150 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:11.150 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:11.150 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.148+0000 7fce74914640 1 Processor -- start 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.148+0000 7fce74914640 1 -- start start 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.148+0000 7fce74914640 10 monclient: init 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce7009ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce7009f020 0x7fce70069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient(hunting): picked mon.noname-a con 0x7fce7009f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient(hunting): picked mon.noname-b con 0x7fce70136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce70067420 con 0x7fce7009ec50 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7009fdc0 con 0x7fce7009f020 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce700a0510 con 0x7fce70136d60 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce74914640 10 monclient(hunting): authenticate will time out at 866.718122s 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6e59c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fce7009ec50 legacy=0x7fce70131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce65d9b640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fce7009f020 msgr2=0x7fce70069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce65d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce7009f020 0x7fce70069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:11.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:46428/0 (socket says 172.21.15.144:46428) 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 -- 172.21.15.144:0/4135631521 learned_addr learned my addr 172.21.15.144:0/4135631521 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): get_auth_request con 0x7fce70136d60 auth_method 0 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient(hunting): handle_auth_done global_id 44162 payload 274 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 -- 172.21.15.144:0/4135631521 >> v2:172.21.15.116:3300/0 conn(0x7fce7009f020 msgr2=0x7fce70069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 --2- 172.21.15.144:0/4135631521 >> v2:172.21.15.116:3300/0 conn(0x7fce7009f020 0x7fce70069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 -- 172.21.15.144:0/4135631521 >> v1:172.21.15.116:6789/0 conn(0x7fce7009ec50 legacy=0x7fce70131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:11.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:27:11.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 monclient: found mon.noname-b 2024-09-18T12:27:11.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:11.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:11.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.149+0000 7fce6ed9d640 1 -- 172.21.15.144:0/4135631521 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce700a0200 con 0x7fce70136d60 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6ed9d640 1 -- 172.21.15.144:0/4135631521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 msgr2=0x7fce70139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6ed9d640 1 --2- 172.21.15.144:0/4135631521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70139140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce68009f20 tx=0x7fce68009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d60e480d2f1d924 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 1 -- 172.21.15.144:0/4135631521 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68007610 con 0x7fce70136d60 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: dump: 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.161 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 1 -- 172.21.15.144:0/4135631521 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6802d050 con 0x7fce70136d60 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 1 -- 172.21.15.144:0/4135631521 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce680079a0 con 0x7fce70136d60 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce74914640 5 monclient: authenticate success, global_id 44162 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce74914640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:11.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce74914640 4 set_mon_vals no callback set 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.152+0000 7fce6dd9b640 10 monclient: dump: 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.163 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 10 monclient: shutdown 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 -- 172.21.15.144:0/4135631521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 msgr2=0x7fce70139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 --2- 172.21.15.144:0/4135631521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70139140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce68009f20 tx=0x7fce68009840 comp rx=0 tx=0).stop 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 -- 172.21.15.144:0/4135631521 shutdown_connections 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 --2- 172.21.15.144:0/4135631521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70139140 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 --2- 172.21.15.144:0/4135631521 >> v2:172.21.15.116:3300/0 conn(0x7fce7009f020 0x7fce70069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 -- 172.21.15.144:0/4135631521 >> 172.21.15.144:0/4135631521 conn(0x7fce700a5d40 msgr2=0x7fce700a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 -- 172.21.15.144:0/4135631521 shutdown_connections 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.153+0000 7fce74914640 1 -- 172.21.15.144:0/4135631521 wait complete. 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient: build_initial_monmap 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient: monmap: 2024-09-18T12:27:11.164 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 Processor -- start 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 -- start start 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient: init 2024-09-18T12:27:11.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7009ec50 0x7fce70141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): picked mon.noname-a con 0x7fce7009ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce7009f020 0x7fce70141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): picked mon.noname-b con 0x7fce7009f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70097600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): picked mon.noname-c con 0x7fce70136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce70099170 con 0x7fce7009ec50 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7009c000 con 0x7fce7009f020 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7009b260 con 0x7fce70136d60 2024-09-18T12:27:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce74914640 10 monclient(hunting): authenticate will time out at 866.723933s 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce6e59c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7009ec50 msgr2=0x7fce70141a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.154+0000 7fce6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7009ec50 0x7fce70141a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce7009f020 0x7fce70141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce7009f020 0x7fce70141f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51568/0 (socket says 172.21.15.144:51568) 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 -- 172.21.15.144:0/1226343080 learned_addr learned my addr 172.21.15.144:0/1226343080 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): get_auth_request con 0x7fce7009f020 auth_method 0 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70097600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 10 monclient(hunting): get_auth_request con 0x7fce70136d60 auth_method 0 2024-09-18T12:27:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient(hunting): handle_auth_done global_id 4669 payload 824 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 -- 172.21.15.144:0/1226343080 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 msgr2=0x7fce70097600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce70136d60 0x7fce70097600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 -- 172.21.15.144:0/1226343080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7009ec50 msgr2=0x7fce70141a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7009ec50 0x7fce70141a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 monclient: found mon.noname-b 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.155+0000 7fce65d9b640 1 -- 172.21.15.144:0/1226343080 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68009ae0 con 0x7fce7009f020 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce65d9b640 1 -- 172.21.15.144:0/1226343080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce7009f020 msgr2=0x7fce70141f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce65d9b640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce7009f020 0x7fce70141f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fce5800b570 tx=0x7fce58009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 1 -- 172.21.15.144:0/1226343080 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58003cc0 con 0x7fce7009f020 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce65d9b640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce7009f020 0x7fce70141f80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fce5800b570 tx=0x7fce58009120 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fce58003e40 config(0 keys) v1 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 10 monclient: dump: 2024-09-18T12:27:11.169 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce74914640 5 monclient: authenticate success, global_id 4669 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 1 -- 172.21.15.144:0/1226343080 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce58003e40 con 0x7fce7009f020 2024-09-18T12:27:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce66ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce74914640 10 monclient: _renew_subs 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce74914640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce74914640 1 -- 172.21.15.144:0/1226343080 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7009b540 con 0x7fce7009f020 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce6d59a640 4 set_mon_vals no callback set 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce74914640 10 monclient: _renew_subs 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce74914640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.157+0000 7fce74914640 1 -- 172.21.15.144:0/1226343080 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7009af80 con 0x7fce7009f020 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce66ffd640 1 -- 172.21.15.144:0/1226343080 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58005340 con 0x7fce7009f020 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce66ffd640 10 monclient: dump: 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce66ffd640 1 -- 172.21.15.144:0/1226343080 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fce5801a460 con 0x7fce7009f020 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce74914640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce74914640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce74914640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.158+0000 7fce74914640 1 -- 172.21.15.144:0/1226343080 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce700008d0 con 0x7fce7009f020 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.159+0000 7fce66ffd640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce50038440 0x7fce5003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.159+0000 7fce6e59c640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce50038440 0x7fce5003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.159+0000 7fce6e59c640 10 monclient: get_auth_request con 0x7fce50038440 auth_method 0 2024-09-18T12:27:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.161+0000 7fce6e59c640 1 --2- 172.21.15.144:0/1226343080 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce50038440 0x7fce5003a8f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fce60006fd0 tx=0x7fce60006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.163+0000 7fce66ffd640 1 -- 172.21.15.144:0/1226343080 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fce5804b600 con 0x7fce7009f020 2024-09-18T12:27:11.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.780+0000 7fce66ffd640 1 -- 172.21.15.144:0/1226343080 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fce5801a710 con 0x7fce7009f020 2024-09-18T12:27:11.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.780+0000 7fce66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:11.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:11.780+0000 7fce66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:11.938 INFO:tasks.workunit.client.0.smithi144.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-18T12:27:11.938 INFO:tasks.workunit.client.0.smithi144.stderr:Invalid command: unused arguments: ['--force'] 2024-09-18T12:27:11.938 INFO:tasks.workunit.client.0.smithi144.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-18T12:27:11.938 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: invalid command 2024-09-18T12:27:11.940 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:11.940 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-18T12:27:11.940 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:11.940 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-18T12:27:12.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.013+0000 7f9d1b2b8640 10 monclient: get_monmap_and_config 2024-09-18T12:27:12.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.013+0000 7f9d1b2b8640 10 monclient: build_initial_monmap 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.013+0000 7f9d1b2b8640 10 monclient: monmap: 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:12.014472+0000 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:12.014472+0000 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:12.015 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 1 Processor -- start 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 1 -- start start 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient: init 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d140a0ad0 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d1409ee60 0x7f9d1409f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient(hunting): picked mon.noname-b con 0x7f9d1409ee60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9d14136e00 0x7f9d141391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient(hunting): picked mon.noname-c con 0x7f9d14136e00 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d1406a5b0 con 0x7f9d140a0ad0 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d1409f7a0 con 0x7f9d14136e00 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d140a0ea0 con 0x7f9d1409ee60 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.014+0000 7f9d1b2b8640 10 monclient(hunting): authenticate will time out at 867.583893s 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9d14136e00 0x7f9d141391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9d14136e00 0x7f9d141391e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51584/0 (socket says 172.21.15.144:51584) 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 -- 172.21.15.144:0/2940184824 learned_addr learned my addr 172.21.15.144:0/2940184824 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): get_auth_request con 0x7f9d14136e00 auth_method 0 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:12.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient(hunting): handle_auth_done global_id 4672 payload 274 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 -- 172.21.15.144:0/2940184824 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d1409ee60 msgr2=0x7f9d1409f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 --2- 172.21.15.144:0/2940184824 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d1409ee60 0x7f9d1409f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 -- 172.21.15.144:0/2940184824 >> v1:172.21.15.144:6789/0 conn(0x7f9d140a0ad0 legacy=0x7f9d140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient: _finish_hunting 0 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 monclient: found mon.noname-c 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.015+0000 7f9d19855640 1 -- 172.21.15.144:0/2940184824 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d140a0460 con 0x7f9d14136e00 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d19855640 1 -- 172.21.15.144:0/2940184824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d14136e00 msgr2=0x7f9d141391e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d19855640 1 --2- 172.21.15.144:0/2940184824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d14136e00 0x7f9d141391e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009f20 tx=0x7f9d0c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e6b79dcca8b2f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 1 -- 172.21.15.144:0/2940184824 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c007610 con 0x7f9d14136e00 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: dump: 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.026 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: _finish_auth 0 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 1 -- 172.21.15.144:0/2940184824 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0c02d050 con 0x7f9d14136e00 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 5 monclient: authenticate success, global_id 4672 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:12.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 1 -- 172.21.15.144:0/2940184824 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0079a0 con 0x7f9d14136e00 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 4 set_mon_vals no callback set 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d0bfff640 10 monclient: dump: 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.028 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 10 monclient: shutdown 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/2940184824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d14136e00 msgr2=0x7f9d141391e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 1 --2- 172.21.15.144:0/2940184824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d14136e00 0x7f9d141391e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009f20 tx=0x7f9d0c009840 comp rx=0 tx=0).stop 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/2940184824 shutdown_connections 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 1 --2- 172.21.15.144:0/2940184824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d14136e00 0x7f9d141391e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 1 --2- 172.21.15.144:0/2940184824 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d1409ee60 0x7f9d1409f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.018+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/2940184824 >> 172.21.15.144:0/2940184824 conn(0x7f9d14071370 msgr2=0x7f9d140a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:12.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.019+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/2940184824 shutdown_connections 2024-09-18T12:27:12.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.019+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/2940184824 wait complete. 2024-09-18T12:27:12.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.019+0000 7f9d1b2b8640 10 monclient: build_initial_monmap 2024-09-18T12:27:12.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.019+0000 7f9d1b2b8640 10 monclient: monmap: 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.019+0000 7f9d1b2b8640 1 Processor -- start 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.019+0000 7f9d1b2b8640 1 -- start start 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.019+0000 7f9d1b2b8640 10 monclient: init 2024-09-18T12:27:12.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d1409ee60 0x7f9d1411e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): picked mon.noname-b con 0x7f9d1409ee60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d140a0ad0 0x7f9d1411ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): picked mon.noname-c con 0x7f9d140a0ad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d14136e00 0x7f9d1411ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): picked mon.noname-a con 0x7f9d14136e00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d140a11e0 con 0x7f9d14136e00 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d140a4910 con 0x7f9d1409ee60 2024-09-18T12:27:12.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d14121c30 con 0x7f9d140a0ad0 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d1b2b8640 10 monclient(hunting): authenticate will time out at 867.589420s 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d1409ee60 0x7f9d1411e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d1409ee60 0x7f9d1411e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51598/0 (socket says 172.21.15.144:51598) 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 -- 172.21.15.144:0/3592724161 learned_addr learned my addr 172.21.15.144:0/3592724161 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): get_auth_request con 0x7f9d1409ee60 auth_method 0 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19855640 1 -- 172.21.15.144:0/3592724161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d14136e00 msgr2=0x7f9d1411ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19855640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d14136e00 0x7f9d1411ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:12.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient(hunting): handle_auth_done global_id 4675 payload 824 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 -- 172.21.15.144:0/3592724161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d140a0ad0 msgr2=0x7f9d1411ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9d140a0ad0 0x7f9d1411ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 -- 172.21.15.144:0/3592724161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d14136e00 msgr2=0x7f9d1411ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d14136e00 0x7f9d1411ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient: _finish_hunting 0 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 monclient: found mon.noname-b 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.020+0000 7f9d19054640 1 -- 172.21.15.144:0/3592724161 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c009ae0 con 0x7f9d1409ee60 2024-09-18T12:27:12.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.022+0000 7f9d19054640 1 -- 172.21.15.144:0/3592724161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d1409ee60 msgr2=0x7f9d1411e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.022+0000 7f9d19054640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d1409ee60 0x7f9d1411e4e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000c5e0 tx=0x7f9d10007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d19054640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d1409ee60 0x7f9d1411e4e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000c5e0 tx=0x7f9d10007ed0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f9d10012e40 mon_map magic: 0 v1 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 1 -- 172.21.15.144:0/3592724161 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10012920 con 0x7f9d1409ee60 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 10 monclient: dump: 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.039 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 1 -- 172.21.15.144:0/3592724161 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d10012a80 con 0x7f9d1409ee60 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d1b2b8640 5 monclient: authenticate success, global_id 4675 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d1b2b8640 10 monclient: _renew_subs 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d1b2b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/3592724161 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d14121920 con 0x7f9d1409ee60 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d097fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d0b7fe640 4 set_mon_vals no callback set 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d1b2b8640 10 monclient: _renew_subs 2024-09-18T12:27:12.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d1b2b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.023+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/3592724161 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d14121610 con 0x7f9d1409ee60 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d097fa640 1 -- 172.21.15.144:0/3592724161 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10012e40 con 0x7f9d1409ee60 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d097fa640 10 monclient: dump: 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.041 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d097fa640 1 -- 172.21.15.144:0/3592724161 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9d10011590 con 0x7f9d1409ee60 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d1b2b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d1b2b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d1b2b8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d1b2b8640 1 -- 172.21.15.144:0/3592724161 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d14121e40 con 0x7f9d1409ee60 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d097fa640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9cf80384c0 0x7f9cf803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d18853640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9cf80384c0 0x7f9cf803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:12.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.024+0000 7f9d18853640 10 monclient: get_auth_request con 0x7f9cf80384c0 auth_method 0 2024-09-18T12:27:12.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.026+0000 7f9d18853640 1 --2- 172.21.15.144:0/3592724161 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9cf80384c0 0x7f9cf803a970 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d040097e0 tx=0x7f9d04006d20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:12.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.028+0000 7f9d097fa640 1 -- 172.21.15.144:0/3592724161 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f9d1004bd80 con 0x7f9d1409ee60 2024-09-18T12:27:12.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.647+0000 7f9d097fa640 1 -- 172.21.15.144:0/3592724161 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9d14121e40 con 0x7f9d1409ee60 2024-09-18T12:27:12.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.647+0000 7f9d097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:12.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.647+0000 7f9d097fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:12.808 INFO:tasks.workunit.client.0.smithi144.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-18T12:27:12.809 INFO:tasks.workunit.client.0.smithi144.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-18T12:27:12.809 INFO:tasks.workunit.client.0.smithi144.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-18T12:27:12.809 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: invalid command 2024-09-18T12:27:12.810 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:12.811 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-18T12:27:12.811 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:12.811 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.884+0000 7fc3148c9640 10 monclient: get_monmap_and_config 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.884+0000 7fc3148c9640 10 monclient: build_initial_monmap 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.884+0000 7fc3148c9640 10 monclient: monmap: 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:12.885518+0000 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:12.885518+0000 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 1 Processor -- start 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 1 -- start start 2024-09-18T12:27:12.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient: init 2024-09-18T12:27:12.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:12.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T12:27:12.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc310098c70 0x7fc310099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc310098c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:12.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc310099580 0x7fc31009dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc310099580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc310099c70 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31009f680 con 0x7fc310099c70 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc31009f420 con 0x7fc310098c70 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc310068a20 con 0x7fc310099580 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc3148c9640 10 monclient(hunting): authenticate will time out at 868.454670s 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30ed9d640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fc310099c70 legacy=0x7fc31009e010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc310098c70 0x7fc310099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc310098c70 0x7fc310099040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51604/0 (socket says 172.21.15.144:51604) 2024-09-18T12:27:12.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 1 -- 172.21.15.144:0/1955636069 learned_addr learned my addr 172.21.15.144:0/1955636069 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 10 monclient(hunting): get_auth_request con 0x7fc310098c70 auth_method 0 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.885+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 10 monclient(hunting): handle_auth_done global_id 4678 payload 274 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 1 -- 172.21.15.144:0/1955636069 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc310099580 msgr2=0x7fc31009dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 1 --2- 172.21.15.144:0/1955636069 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc310099580 0x7fc31009dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 1 -- 172.21.15.144:0/1955636069 >> v1:172.21.15.116:6789/0 conn(0x7fc310099c70 legacy=0x7fc31009e010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:12.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 1 monclient: found mon.noname-c 2024-09-18T12:27:12.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:12.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.886+0000 7fc30e59c640 1 -- 172.21.15.144:0/1955636069 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc310096a30 con 0x7fc310098c70 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.888+0000 7fc30e59c640 1 -- 172.21.15.144:0/1955636069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 msgr2=0x7fc310099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.888+0000 7fc30e59c640 1 --2- 172.21.15.144:0/1955636069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 0x7fc310099040 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc300009910 tx=0x7fc3000091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb6fbb603b345c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 1 -- 172.21.15.144:0/1955636069 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300032440 con 0x7fc310098c70 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: dump: 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.897 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 1 -- 172.21.15.144:0/1955636069 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3000325a0 con 0x7fc310098c70 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 5 monclient: authenticate success, global_id 4678 2024-09-18T12:27:12.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 1 -- 172.21.15.144:0/1955636069 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300032980 con 0x7fc310098c70 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 4 set_mon_vals no callback set 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc30dd9b640 10 monclient: dump: 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.899 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 10 monclient: shutdown 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 -- 172.21.15.144:0/1955636069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 msgr2=0x7fc310099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 --2- 172.21.15.144:0/1955636069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 0x7fc310099040 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc300009910 tx=0x7fc3000091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 -- 172.21.15.144:0/1955636069 shutdown_connections 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 --2- 172.21.15.144:0/1955636069 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc310099580 0x7fc31009dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 --2- 172.21.15.144:0/1955636069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 0x7fc310099040 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 -- 172.21.15.144:0/1955636069 >> 172.21.15.144:0/1955636069 conn(0x7fc310070e90 msgr2=0x7fc3100a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 -- 172.21.15.144:0/1955636069 shutdown_connections 2024-09-18T12:27:12.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.889+0000 7fc3148c9640 1 -- 172.21.15.144:0/1955636069 wait complete. 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 10 monclient: build_initial_monmap 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 10 monclient: monmap: 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 1 Processor -- start 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 1 -- start start 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 10 monclient: init 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.890+0000 7fc3148c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 0x7fc310133e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc310098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc310099580 0x7fc31011db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc310099580 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc310099c70 0x7fc31011e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 0x7fc310133e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 0x7fc310133e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51610/0 (socket says 172.21.15.144:51610) 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc310099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:12.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 -- 172.21.15.144:0/3225010613 learned_addr learned my addr 172.21.15.144:0/3225010613 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 1 -- 172.21.15.144:0/3225010613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc310142040 con 0x7fc310099580 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 1 -- 172.21.15.144:0/3225010613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc310097cd0 con 0x7fc310098c70 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 1 -- 172.21.15.144:0/3225010613 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc310121200 con 0x7fc310099c70 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc3148c9640 10 monclient(hunting): authenticate will time out at 868.460209s 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): get_auth_request con 0x7fc310098c70 auth_method 0 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc305d9b640 1 -- 172.21.15.144:0/3225010613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc310099580 msgr2=0x7fc31011db50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:12.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc305d9b640 1 --2- 172.21.15.144:0/3225010613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc310099580 0x7fc31011db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient(hunting): handle_auth_done global_id 4681 payload 824 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 -- 172.21.15.144:0/3225010613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc310099c70 msgr2=0x7fc31011e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 --2- 172.21.15.144:0/3225010613 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc310099c70 0x7fc31011e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 -- 172.21.15.144:0/3225010613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc310099580 msgr2=0x7fc31011db50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 --2- 172.21.15.144:0/3225010613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc310099580 0x7fc31011db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 monclient: found mon.noname-b 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.891+0000 7fc30e59c640 1 -- 172.21.15.144:0/3225010613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc300009450 con 0x7fc310098c70 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc30e59c640 1 -- 172.21.15.144:0/3225010613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 msgr2=0x7fc310133e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc30e59c640 1 --2- 172.21.15.144:0/3225010613 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc310098c70 0x7fc310133e00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc300035040 tx=0x7fc300032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 1 -- 172.21.15.144:0/3225010613 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000319a0 con 0x7fc310098c70 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: dump: 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.905 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 1 -- 172.21.15.144:0/3225010613 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc300031b20 con 0x7fc310098c70 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc3148c9640 5 monclient: authenticate success, global_id 4681 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc3148c9640 10 monclient: _renew_subs 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc3148c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc3148c9640 1 -- 172.21.15.144:0/3225010613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31011ffc0 con 0x7fc310098c70 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 1 -- 172.21.15.144:0/3225010613 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30003c400 con 0x7fc310098c70 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:12.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc306ffd640 10 monclient: dump: 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc30d59a640 4 set_mon_vals no callback set 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc3148c9640 10 monclient: _renew_subs 2024-09-18T12:27:12.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.893+0000 7fc3148c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.894+0000 7fc3148c9640 1 -- 172.21.15.144:0/3225010613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31011f3f0 con 0x7fc310098c70 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.894+0000 7fc306ffd640 1 -- 172.21.15.144:0/3225010613 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc30003c830 con 0x7fc310098c70 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc3148c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc3148c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc3148c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc3148c9640 1 -- 172.21.15.144:0/3225010613 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3100008d0 con 0x7fc310098c70 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc306ffd640 1 --2- 172.21.15.144:0/3225010613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2f0038440 0x7fc2f003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc306ffd640 1 -- 172.21.15.144:0/3225010613 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fc30006c290 con 0x7fc310098c70 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc305d9b640 1 --2- 172.21.15.144:0/3225010613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2f0038440 0x7fc2f003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:12.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.895+0000 7fc305d9b640 10 monclient: get_auth_request con 0x7fc2f0038440 auth_method 0 2024-09-18T12:27:12.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.897+0000 7fc305d9b640 1 --2- 172.21.15.144:0/3225010613 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2f0038440 0x7fc2f003a8f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80025d0 tx=0x7fc2f8002a60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:12.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.899+0000 7fc306ffd640 1 -- 172.21.15.144:0/3225010613 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc30003b3a0 con 0x7fc310098c70 2024-09-18T12:27:12.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.899+0000 7fc306ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:12.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:12.899+0000 7fc306ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:13.053 INFO:tasks.workunit.client.0.smithi144.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-18T12:27:13.054 INFO:tasks.workunit.client.0.smithi144.stderr:Invalid command: unused arguments: ['foo'] 2024-09-18T12:27:13.054 INFO:tasks.workunit.client.0.smithi144.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-18T12:27:13.054 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: invalid command 2024-09-18T12:27:13.055 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:13.055 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-18T12:27:13.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.129+0000 7f94b7b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:13.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.129+0000 7f94b7b9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.129+0000 7f94b7b9d640 10 monclient: monmap: 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:13.130460+0000 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:13.130460+0000 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:13.131 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:13.132 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:13.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 1 Processor -- start 2024-09-18T12:27:13.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 1 -- start start 2024-09-18T12:27:13.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient: init 2024-09-18T12:27:13.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:13.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T12:27:13.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f94b00a0ad0 0x7f94b00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f94b00a0ad0 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94b009ee60 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94b009f260 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b006a5b0 con 0x7f94b009f260 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b00a0460 con 0x7f94b009ee60 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b00a0ea0 con 0x7f94b00a0ad0 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.130+0000 7f94b7b9d640 10 monclient(hunting): authenticate will time out at 868.699958s 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.131+0000 7f94b613a640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f94b009f260 legacy=0x7f94b01321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.131+0000 7f94b5939640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f94b00a0ad0 msgr2=0x7f94b00691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.131+0000 7f94b5939640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f94b00a0ad0 0x7f94b00691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:13.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.131+0000 7f94b5138640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f94b009ee60 0x7f94b0131ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:35992/0 (socket says 172.21.15.144:35992) 2024-09-18T12:27:13.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.131+0000 7f94b5138640 1 -- 172.21.15.144:0/435829897 learned_addr learned my addr 172.21.15.144:0/435829897 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b5138640 1 -- 172.21.15.144:0/435829897 >> v1:172.21.15.144:6789/0 conn(0x7f94b009ee60 legacy=0x7f94b0131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486378985 0 0) 0x7f94b00a0460 con 0x7f94b009ee60 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): my global_id is 4684 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f949c0034e0 con 0x7f94b009ee60 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3527623942 0 0) 0x7f949c0034e0 con 0x7f94b009ee60 2024-09-18T12:27:13.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 >> v2:172.21.15.116:3300/0 conn(0x7f94b00a0ad0 msgr2=0x7f94b00691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 --2- 172.21.15.144:0/435829897 >> v2:172.21.15.116:3300/0 conn(0x7f94b00a0ad0 0x7f94b00691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 >> v1:172.21.15.116:6789/0 conn(0x7f94b009f260 legacy=0x7f94b01321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: _finish_hunting 0 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b00a1bb0 con 0x7f94b009ee60 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: _finish_auth 0 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f94a40017a0 con 0x7f94b009ee60 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b7b9d640 5 monclient: authenticate success, global_id 4684 2024-09-18T12:27:13.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b7b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: dump: 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:13.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 >> v1:172.21.15.144:6789/0 conn(0x7f94b009ee60 legacy=0x7f94b0131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f949c0039d0 0x7f949c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): picked mon.a con 0x7f949c0039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f949c004780 0x7f949c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): picked mon.c con 0x7f949c004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f949c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): picked mon.b con 0x7f949c005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b00a0ea0 con 0x7f949c0039d0 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949c00c2f0 con 0x7f949c005330 2024-09-18T12:27:13.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949c00c4b0 con 0x7f949c004780 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b4937640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b7b9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b5939640 1 -- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f949c0039d0 msgr2=0x7f949c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b5939640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f949c0039d0 0x7f949c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f949c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 10 monclient(hunting): get_auth_request con 0x7f949c005330 auth_method 0 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:13.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.133+0000 7f94b613a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 10 monclient(hunting): handle_auth_done global_id 4684 payload 293 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 1 -- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f949c004780 msgr2=0x7f949c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f949c004780 0x7f949c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 1 -- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f949c0039d0 msgr2=0x7f949c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f949c0039d0 0x7f949c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 10 monclient: _finish_hunting 0 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 1 monclient: found mon.b 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 1 -- 172.21.15.144:0/435829897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c0034e0 con 0x7f949c005330 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 10 monclient: _finish_auth 0 2024-09-18T12:27:13.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.134+0000 7f94b613a640 1 -- 172.21.15.144:0/435829897 reap_dead start 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b613a640 1 -- 172.21.15.144:0/435829897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 msgr2=0x7f949c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b613a640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f949c00ba10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009f90 tx=0x7f94a80095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4dcf8953b16d05e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8032430 con 0x7f949c005330 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 10 monclient: dump: 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.147 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a802d070 con 0x7f949c005330 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b7b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b7b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 1 -- 172.21.15.144:0/435829897 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8032820 con 0x7f949c005330 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b7b9d640 4 set_mon_vals no callback set 2024-09-18T12:27:13.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.136+0000 7f94b4937640 10 monclient: dump: 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 10 monclient: shutdown 2024-09-18T12:27:13.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 1 -- 172.21.15.144:0/435829897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 msgr2=0x7f949c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f949c00ba10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009f90 tx=0x7f94a80095d0 comp rx=0 tx=0).stop 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 1 -- 172.21.15.144:0/435829897 shutdown_connections 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/435829897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f949c00ba10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 1 -- 172.21.15.144:0/435829897 >> 172.21.15.144:0/435829897 conn(0x7f94b0071370 msgr2=0x7f94b00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 1 -- 172.21.15.144:0/435829897 shutdown_connections 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 1 -- 172.21.15.144:0/435829897 wait complete. 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.137+0000 7f94b7b9d640 10 monclient: monmap: 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:13.150 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 Processor -- start 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 -- start start 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient: init 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f94b0134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f949c005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94b0135130 0x7f94b011e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f94b0135130 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94b011ec30 0x7f94b011f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f94b011ec30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b00a11e0 con 0x7f94b011ec30 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0141c30 con 0x7f949c005330 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0121ff0 con 0x7f94b0135130 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b7b9d640 10 monclient(hunting): authenticate will time out at 868.707747s 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f94b0134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b613a640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94b011ec30 msgr2=0x7f94b011f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:13.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b613a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94b011ec30 0x7f94b011f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f94b0134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51618/0 (socket says 172.21.15.144:51618) 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 1 -- 172.21.15.144:0/2649355949 learned_addr learned my addr 172.21.15.144:0/2649355949 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 10 monclient(hunting): get_auth_request con 0x7f949c005330 auth_method 0 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.138+0000 7f94b5939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:13.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 10 monclient(hunting): handle_auth_done global_id 4687 payload 824 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 1 -- 172.21.15.144:0/2649355949 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94b0135130 msgr2=0x7f94b011e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94b0135130 0x7f94b011e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 1 -- 172.21.15.144:0/2649355949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94b011ec30 msgr2=0x7f94b011f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94b011ec30 0x7f94b011f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 10 monclient: _finish_hunting 0 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 1 monclient: found mon.noname-b 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:13.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.139+0000 7f94b5939640 1 -- 172.21.15.144:0/2649355949 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b01201e0 con 0x7f949c005330 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b5939640 1 -- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 msgr2=0x7f94b0134bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b5939640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f94b0134bf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00b810 tx=0x7f94ac0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac003e20 con 0x7f949c005330 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: dump: 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:13.157 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac006150 con 0x7f949c005330 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b7b9d640 5 monclient: authenticate success, global_id 4687 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b7b9d640 10 monclient: _renew_subs 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac006300 con 0x7f949c005330 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:13.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b0121d10 con 0x7f949c005330 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a3fff640 4 set_mon_vals no callback set 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94a1ffb640 10 monclient: dump: 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:13.159 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b7b9d640 10 monclient: _renew_subs 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.141+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b01361c0 con 0x7f949c005330 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f94ac012020 con 0x7f949c005330 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94a1ffb640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9494038490 0x7f949403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f94ac04ae30 con 0x7f949c005330 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94b5138640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9494038490 0x7f949403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94b7b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94b7b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94b7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b00008d0 con 0x7f949c005330 2024-09-18T12:27:13.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.142+0000 7f94b5138640 10 monclient: get_auth_request con 0x7f9494038490 auth_method 0 2024-09-18T12:27:13.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.145+0000 7f94b5138640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9494038490 0x7f949403a940 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f94a4000f60 tx=0x7f94a4001e50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:13.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.147+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f94ac0153e0 con 0x7f949c005330 2024-09-18T12:27:13.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.147+0000 7f94a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:13.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.147+0000 7f94a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.299+0000 7f94b7b9d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.299+0000 7f94b7b9d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.299+0000 7f94b7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:13.299+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94b0064a20 con 0x7f949c005330 2024-09-18T12:27:16.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:16.139+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:16.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:16.139+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:16.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:16.139+0000 7f94a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:17.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:17.139+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:17.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:17.139+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:18.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:18.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:18.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:18.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:19.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:19.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:19.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:19.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:20.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:20.139+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:20.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:20.139+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:21.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:21.139+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:21.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:21.139+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:22.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:22.138+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:22.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:22.138+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:23.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:23.138+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:23.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:23.138+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:24.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:24.139+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:24.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:24.139+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:25.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:25.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:25.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:25.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:26.128 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T12:27:26.128 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T12:27:26.128 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:27:26.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:27:26.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:26.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:26.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:26.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:26.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:26.140+0000 7f94a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.226+0000 7f4f0b59d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.226+0000 7f4f0b59d640 10 monclient: build_initial_monmap 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.226+0000 7f4f0b59d640 10 monclient: monmap: 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:26.227733+0000 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:26.227733+0000 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:26.229 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.226+0000 7f4f0b59d640 1 Processor -- start 2024-09-18T12:27:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.226+0000 7f4f0b59d640 1 -- start start 2024-09-18T12:27:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.226+0000 7f4f0b59d640 10 monclient: init 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f04131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 0x7f4f0409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f0409e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-b con 0x7f4f0409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f0409faf0 con 0x7f4f0409e910 2024-09-18T12:27:26.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f0409fd30 con 0x7f4f04131ab0 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040a1110 con 0x7f4f0409f250 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f0b59d640 10 monclient(hunting): authenticate will time out at 873.549420s 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f08b38640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 msgr2=0x7f4f0409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f08b38640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 0x7f4f0409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48936/0 (socket says 172.21.15.116:48936) 2024-09-18T12:27:26.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 learned_addr learned my addr 172.21.15.116:0/4264594239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request con 0x7f4f0409f250 auth_method 0 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 10 monclient(hunting): get_auth_request con 0x7f4f04131ab0 auth_method 0 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:26.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:27:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:26.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.227+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 msgr2=0x7f4f0409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f08b38640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 0x7f4f0409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request con 0x7f4f0409f250 auth_method 0 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 10 monclient(hunting): get_auth_request con 0x7f4f04131ab0 auth_method 0 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.428+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 msgr2=0x7f4f0409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f08b38640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 0x7f4f0409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request con 0x7f4f0409f250 auth_method 0 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 10 monclient(hunting): get_auth_request con 0x7f4f04131ab0 auth_method 0 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:26.829+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:27:27.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:27.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:27.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:27.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 msgr2=0x7f4f0409ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f08b38640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 0x7f4f0409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:27:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request con 0x7f4f0409f250 auth_method 0 2024-09-18T12:27:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 10 monclient(hunting): get_auth_request con 0x7f4f04131ab0 auth_method 0 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:27.630+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:27:28.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:28.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:28.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:28.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:29.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:29.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:29.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:29.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 10 monclient(hunting): tick 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 monclient(hunting): continuing hunt 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 msgr2=0x7f4f04068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f0409f250 0x7f4f04068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 msgr2=0x7f4f04131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.144:3300/0 conn(0x7f4f04131ab0 0x7f4f04131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 -- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 msgr2=0x7f4f0409ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 --2- 172.21.15.116:0/4264594239 >> v2:172.21.15.116:3300/0 conn(0x7f4f0409e910 0x7f4f0409ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 10 monclient: _add_conns ranks=[0,1] 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ee8003650 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ee80072d0 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 -- 172.21.15.116:0/4264594239 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f040a13f0 con 0x7f4ee8003650 2024-09-18T12:27:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 1 -- 172.21.15.116:0/4264594239 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee800c000 con 0x7f4ee80072d0 2024-09-18T12:27:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4efaffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:29.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.227+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v1:172.21.15.116:6789/0 conn(0x7f4ee8003650 legacy=0x7f4ee8005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:29.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.230+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 >> v1:172.21.15.144:6789/0 conn(0x7f4ee80072d0 legacy=0x7f4ee8009780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:29.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.428+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v1:172.21.15.116:6789/0 conn(0x7f4ee8003650 legacy=0x7f4ee8005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:29.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:29.829+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v1:172.21.15.116:6789/0 conn(0x7f4ee8003650 legacy=0x7f4ee8005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:30.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:30.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:30.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:30.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:30.630+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> v1:172.21.15.116:6789/0 conn(0x7f4ee8003650 legacy=0x7f4ee8005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:31.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.138+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390325444 0 0) 0x7f4ee800c000 con 0x7f4ee80072d0 2024-09-18T12:27:31.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.138+0000 7f4efbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.138+0000 7f4efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.138+0000 7f4efbfff640 10 monclient(hunting): my global_id is 4690 2024-09-18T12:27:31.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.138+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef00034e0 con 0x7f4ee80072d0 2024-09-18T12:27:31.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688628873 0 0) 0x7f4ef00034e0 con 0x7f4ee80072d0 2024-09-18T12:27:31.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 >> v1:172.21.15.116:6789/0 conn(0x7f4ee8003650 legacy=0x7f4ee8005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee800d1c0 con 0x7f4ee80072d0 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f4ef40042c0 con 0x7f4ee80072d0 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:27:31.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: dump: 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 >> v1:172.21.15.144:6789/0 conn(0x7f4ee80072d0 legacy=0x7f4ee8009780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4f09b3a640 1 -- 172.21.15.116:0/4264594239 reap_dead start 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4ef0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): picked mon.a con 0x7f4ef00039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0004780 0x7f4ef0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): picked mon.c con 0x7f4ef0004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0005330 0x7f4ef000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): picked mon.b con 0x7f4ef0005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000c210 con 0x7f4ef00039d0 2024-09-18T12:27:31.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000c380 con 0x7f4ef0005330 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000c540 con 0x7f4ef0004780 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4efbfff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4f09339640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 msgr2=0x7f4ef0003da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.139+0000 7f4f09339640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4ef0003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0004780 0x7f4ef0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): get_auth_request con 0x7f4ef0004780 auth_method 0 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:31.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0005330 0x7f4ef000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request con 0x7f4ef0005330 auth_method 0 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient(hunting): handle_auth_done global_id 4690 payload 293 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0005330 msgr2=0x7f4ef000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:31.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0005330 0x7f4ef000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 msgr2=0x7f4ef0003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4ef0003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 1 monclient: found mon.c 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00034e0 con 0x7f4ef0004780 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f08b38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f0b59d640 5 monclient: authenticate success, global_id 4690 2024-09-18T12:27:31.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.140+0000 7f4f0b59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:31.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.140+0000 7f94a0ff9640 10 monclient: tick 2024-09-18T12:27:31.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.140+0000 7f94a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4f08b38640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0004780 msgr2=0x7f4ef0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4f08b38640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0004780 0x7f4ef0004bf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4000c00 tx=0x7f4ef4007b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b5f354441c8e809 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4034070 con 0x7f4ef0004780 2024-09-18T12:27:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 10 monclient: dump: 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef4007150 con 0x7f4ef0004780 2024-09-18T12:27:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 1 -- 172.21.15.116:0/4264594239 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4007530 con 0x7f4ef0004780 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4efbfff640 10 monclient: dump: 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4f0b59d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.143+0000 7f4f0b59d640 4 set_mon_vals no callback set 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 10 monclient: shutdown 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 -- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0004780 msgr2=0x7f4ef0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0004780 0x7f4ef0004bf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4000c00 tx=0x7f4ef4007b20 comp rx=0 tx=0).stop 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 -- 172.21.15.116:0/4264594239 shutdown_connections 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0005330 0x7f4ef000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0004780 0x7f4ef0004bf0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/4264594239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4ef0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 -- 172.21.15.116:0/4264594239 >> 172.21.15.116:0/4264594239 conn(0x7f4f04067d30 msgr2=0x7f4f040a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 -- 172.21.15.116:0/4264594239 shutdown_connections 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.144+0000 7f4f0b59d640 1 -- 172.21.15.116:0/4264594239 wait complete. 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.145+0000 7f4f0b59d640 10 monclient: build_initial_monmap 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.145+0000 7f4f0b59d640 10 monclient: monmap: 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:31.160 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.145+0000 7f4f0b59d640 1 Processor -- start 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.145+0000 7f4f0b59d640 1 -- start start 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.145+0000 7f4f0b59d640 10 monclient: init 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4f0409a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef00039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0004780 0x7f4f04130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef0004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 0x7f4f0411b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef0005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0411c7c0 con 0x7f4ef00039d0 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0409bde0 con 0x7f4ef0004780 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04134200 con 0x7f4ef0005330 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f0b59d640 10 monclient(hunting): authenticate will time out at 878.468545s 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09339640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 msgr2=0x7f4f0409a940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09339640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4f0409a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 0x7f4f0411b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 0x7f4f0411b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48966/0 (socket says 172.21.15.116:48966) 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 1 -- 172.21.15.116:0/3080832259 learned_addr learned my addr 172.21.15.116:0/3080832259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request con 0x7f4ef0005330 auth_method 0 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f08b38640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0004780 0x7f4f04130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f08b38640 10 monclient(hunting): get_auth_request con 0x7f4ef0004780 auth_method 0 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f08b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f08b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.146+0000 7f4f08b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f08b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 10 monclient(hunting): handle_auth_done global_id 44168 payload 824 2024-09-18T12:27:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 1 -- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0004780 msgr2=0x7f4f04130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0004780 0x7f4f04130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 1 -- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 msgr2=0x7f4f0409a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4f0409a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 1 monclient: found mon.noname-c 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.147+0000 7f4f09b3a640 1 -- 172.21.15.116:0/3080832259 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f04141580 con 0x7f4ef0005330 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f09b3a640 1 -- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 msgr2=0x7f4f0411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f09b3a640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 0x7f4f0411b6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8007240 tx=0x7f4efc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 1 -- 172.21.15.116:0/3080832259 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc018660 con 0x7f4ef0005330 2024-09-18T12:27:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: dump: 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 1 -- 172.21.15.116:0/3080832259 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc00f040 con 0x7f4ef0005330 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f0b59d640 5 monclient: authenticate success, global_id 44168 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 1 -- 172.21.15.116:0/3080832259 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc018a30 con 0x7f4ef0005330 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f0b59d640 10 monclient: _renew_subs 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0411daa0 con 0x7f4ef0005330 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4ef97fa640 10 monclient: dump: 2024-09-18T12:27:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4efb7fe640 4 set_mon_vals no callback set 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f0b59d640 10 monclient: _renew_subs 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.149+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0411cb40 con 0x7f4ef0005330 2024-09-18T12:27:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.150+0000 7f4ef97fa640 1 -- 172.21.15.116:0/3080832259 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4efc021450 con 0x7f4ef0005330 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.150+0000 7f4f0b59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.150+0000 7f4f0b59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.150+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.150+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0411df40 con 0x7f4ef0005330 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.151+0000 7f4ef97fa640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4ee8038cc0 0x7f4ee803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.151+0000 7f4ef97fa640 1 -- 172.21.15.116:0/3080832259 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f4efc04c720 con 0x7f4ef0005330 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.151+0000 7f4f09339640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4ee8038cc0 0x7f4ee803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.151+0000 7f4f09339640 10 monclient: get_auth_request con 0x7f4ee8038cc0 auth_method 0 2024-09-18T12:27:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.153+0000 7f4f09339640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4ee8038cc0 0x7f4ee803b170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f00006d10 tx=0x7f4f00008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.156+0000 7f4ef97fa640 1 -- 172.21.15.116:0/3080832259 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4efc016940 con 0x7f4ef0005330 2024-09-18T12:27:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.156+0000 7f4ef97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.156+0000 7f4ef97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:31.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.189+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v41) v1 ==== 155+0+0 (secure 0 0 0) 0x7f94ac0240c0 con 0x7f949c005330 2024-09-18T12:27:31.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.189+0000 7f94a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:31.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.189+0000 7f94a1ffb640 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-18T12:27:31.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.316+0000 7f94b7b9d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:31.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.316+0000 7f94b7b9d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:31.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.316+0000 7f94b7b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:31.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.316+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94b0120db0 con 0x7f949c005330 2024-09-18T12:27:31.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.317+0000 7f94a1ffb640 1 -- 172.21.15.144:0/2649355949 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v41) v1 ==== 162+0+0 (secure 0 0 0) 0x7f94ac02c0c0 con 0x7f949c005330 2024-09-18T12:27:31.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.317+0000 7f94a1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:31.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.317+0000 7f94a1ffb640 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-18T12:27:31.319 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'foooo' does not exist 2024-09-18T12:27:31.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9494038490 msgr2=0x7f949403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9494038490 0x7f949403a940 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f94a4000f60 tx=0x7f94a4001e50 comp rx=0 tx=0).stop 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 10 monclient: shutdown 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 msgr2=0x7f94b0134bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f94b0134bf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00b810 tx=0x7f94ac0091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 shutdown_connections 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9494038490 0x7f949403a940 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94b011ec30 0x7f94b011f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f94b0135130 0x7f94b011e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 --2- 172.21.15.144:0/2649355949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f949c005330 0x7f94b0134bf0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 >> 172.21.15.144:0/2649355949 conn(0x7f94b0071370 msgr2=0x7f94b009ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 shutdown_connections 2024-09-18T12:27:31.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.319+0000 7f94b7b9d640 1 -- 172.21.15.144:0/2649355949 wait complete. 2024-09-18T12:27:31.328 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-18T12:27:31.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.402+0000 7f6c9d40f640 10 monclient: get_monmap_and_config 2024-09-18T12:27:31.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.402+0000 7f6c9d40f640 10 monclient: build_initial_monmap 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.402+0000 7f6c9d40f640 10 monclient: monmap: 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:31.403427+0000 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:31.403427+0000 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:31.404 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:31.405 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:31.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 1 Processor -- start 2024-09-18T12:27:31.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 1 -- start start 2024-09-18T12:27:31.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient: init 2024-09-18T12:27:31.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c9809ec50 0x7f6c98131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9809ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c981321a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809f020 0x7f6c98069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c98067420 con 0x7f6c981321a0 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9809fdc0 con 0x7f6c9809ec50 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980a0510 con 0x7f6c9809f020 2024-09-18T12:27:31.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c9d40f640 10 monclient(hunting): authenticate will time out at 886.972631s 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c96ffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f6c9809ec50 msgr2=0x7f6c98131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c96ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c9809ec50 0x7f6c98131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c967fc640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f6c981321a0 0x7f6c98068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:32970/0 (socket says 172.21.15.144:32970) 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c967fc640 1 -- 172.21.15.144:0/2252925867 learned_addr learned my addr 172.21.15.144:0/2252925867 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 1 --2- 172.21.15.144:0/2252925867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809f020 0x7f6c98069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 10 monclient(hunting): get_auth_request con 0x7f6c9809f020 auth_method 0 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.403+0000 7f6c977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 10 monclient(hunting): handle_auth_done global_id 44171 payload 274 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 1 -- 172.21.15.144:0/2252925867 >> v2:172.21.15.116:3300/0 conn(0x7f6c9809ec50 msgr2=0x7f6c98131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 1 --2- 172.21.15.144:0/2252925867 >> v2:172.21.15.116:3300/0 conn(0x7f6c9809ec50 0x7f6c98131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 1 -- 172.21.15.144:0/2252925867 >> v1:172.21.15.144:6789/0 conn(0x7f6c981321a0 legacy=0x7f6c98068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.404+0000 7f6c977fe640 1 -- 172.21.15.144:0/2252925867 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980a0200 con 0x7f6c9809f020 2024-09-18T12:27:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.404+0000 7f4f0b59d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:27:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.404+0000 7f4f0b59d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:27:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.404+0000 7f4f0b59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.404+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4f04135b40 con 0x7f4ef0005330 2024-09-18T12:27:31.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.405+0000 7f4ef97fa640 1 -- 172.21.15.116:0/3080832259 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f4efc01f360 con 0x7f4ef0005330 2024-09-18T12:27:31.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.405+0000 7f4ef97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:27:31.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.405+0000 7f4ef97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:31.412 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:27:31.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4ee8038cc0 msgr2=0x7f4ee803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4ee8038cc0 0x7f4ee803b170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f00006d10 tx=0x7f4f00008040 comp rx=0 tx=0).stop 2024-09-18T12:27:31.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 10 monclient: shutdown 2024-09-18T12:27:31.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 msgr2=0x7f4f0411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 0x7f4f0411b6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8007240 tx=0x7f4efc00b040 comp rx=0 tx=0).stop 2024-09-18T12:27:31.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 shutdown_connections 2024-09-18T12:27:31.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4ee8038cc0 0x7f4ee803b170 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ef0005330 0x7f4f0411b6b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ef0004780 0x7f4f04130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 --2- 172.21.15.116:0/3080832259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef00039d0 0x7f4f0409a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 >> 172.21.15.116:0/3080832259 conn(0x7f4f04067d30 msgr2=0x7f4f0409e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 shutdown_connections 2024-09-18T12:27:31.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.407+0000 7f4f0b59d640 1 -- 172.21.15.116:0/3080832259 wait complete. 2024-09-18T12:27:31.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.406+0000 7f6c977fe640 1 -- 172.21.15.144:0/2252925867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809f020 msgr2=0x7f6c98069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.406+0000 7f6c977fe640 1 --2- 172.21.15.144:0/2252925867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809f020 0x7f6c98069410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009e60 tx=0x7f6c90009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12a78aa3205fd0e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 1 -- 172.21.15.144:0/2252925867 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90007840 con 0x7f6c9809f020 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: dump: 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.421 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 5 monclient: authenticate success, global_id 44171 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 1 -- 172.21.15.144:0/2252925867 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9002d050 con 0x7f6c9809f020 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 4 set_mon_vals no callback set 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 1 -- 172.21.15.144:0/2252925867 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90007c10 con 0x7f6c9809f020 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c95ffb640 10 monclient: dump: 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 10 monclient: shutdown 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 -- 172.21.15.144:0/2252925867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809f020 msgr2=0x7f6c98069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/2252925867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809f020 0x7f6c98069410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009e60 tx=0x7f6c90009780 comp rx=0 tx=0).stop 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 -- 172.21.15.144:0/2252925867 shutdown_connections 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/2252925867 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809f020 0x7f6c98069410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/2252925867 >> v2:172.21.15.116:3300/0 conn(0x7f6c9809ec50 0x7f6c98131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 -- 172.21.15.144:0/2252925867 >> 172.21.15.144:0/2252925867 conn(0x7f6c980a5d40 msgr2=0x7f6c980a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 -- 172.21.15.144:0/2252925867 shutdown_connections 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.407+0000 7f6c9d40f640 1 -- 172.21.15.144:0/2252925867 wait complete. 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.408+0000 7f6c9d40f640 10 monclient: build_initial_monmap 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.408+0000 7f6c9d40f640 10 monclient: monmap: 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:31.424 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.408+0000 7f6c9d40f640 1 Processor -- start 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.408+0000 7f6c9d40f640 1 -- start start 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.408+0000 7f6c9d40f640 10 monclient: init 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:31.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809ec50 0x7f6c9809a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c9809f020 0x7f6c9811bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 0x7f6c98130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): picked mon.noname-b con 0x7f6c981321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9809b5a0 con 0x7f6c9809f020 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9811c170 con 0x7f6c981321a0 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9811dcd0 con 0x7f6c9809ec50 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c9d40f640 10 monclient(hunting): authenticate will time out at 886.978244s 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c967fc640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c9809f020 msgr2=0x7f6c9811bc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c9809f020 0x7f6c9811bc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 0x7f6c98130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 0x7f6c98130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57182/0 (socket says 172.21.15.144:57182) 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 -- 172.21.15.144:0/749994942 learned_addr learned my addr 172.21.15.144:0/749994942 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): get_auth_request con 0x7f6c981321a0 auth_method 0 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809ec50 0x7f6c9809a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 10 monclient(hunting): get_auth_request con 0x7f6c9809ec50 auth_method 0 2024-09-18T12:27:31.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient(hunting): handle_auth_done global_id 4696 payload 824 2024-09-18T12:27:31.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 -- 172.21.15.144:0/749994942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809ec50 msgr2=0x7f6c9809a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809ec50 0x7f6c9809a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 -- 172.21.15.144:0/749994942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c9809f020 msgr2=0x7f6c9811bc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c9809f020 0x7f6c9811bc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.409+0000 7f6c977fe640 1 -- 172.21.15.144:0/749994942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009a20 con 0x7f6c981321a0 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.411+0000 7f6c977fe640 1 -- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 msgr2=0x7f6c98130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.411+0000 7f6c977fe640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 0x7f6c98130b80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009e60 tx=0x7f6c90037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.411+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90031cd0 con 0x7f6c981321a0 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: dump: 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:31.430 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90040070 con 0x7f6c981321a0 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 5 monclient: authenticate success, global_id 4696 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 10 monclient: _renew_subs 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9811df80 con 0x7f6c981321a0 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90045640 con 0x7f6c981321a0 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 10 monclient: dump: 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.432 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c957fa640 4 set_mon_vals no callback set 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 10 monclient: _renew_subs 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9811c8b0 con 0x7f6c981321a0 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6c90003020 con 0x7f6c981321a0 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.412+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c980008d0 con 0x7f6c981321a0 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.413+0000 7f6c8effd640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c78038440 0x7f6c7803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.413+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5358+0+0 (secure 0 0 0) 0x7f6c9006d6c0 con 0x7f6c981321a0 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.413+0000 7f6c96ffd640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c78038440 0x7f6c7803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.413+0000 7f6c96ffd640 10 monclient: get_auth_request con 0x7f6c78038440 auth_method 0 2024-09-18T12:27:31.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.415+0000 7f6c96ffd640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c78038440 0x7f6c7803a8f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c880038f0 tx=0x7f6c880090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.417+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6c90031e30 con 0x7f6c981321a0 2024-09-18T12:27:31.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.417+0000 7f6c8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.417+0000 7f6c8effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:31.435 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:27:31.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:27:31.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:27:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.527+0000 7f815c35d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.527+0000 7f815c35d640 10 monclient: build_initial_monmap 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.527+0000 7f815c35d640 10 monclient: monmap: 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:31.529410+0000 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:31.529410+0000 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:31.531 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 1 Processor -- start 2024-09-18T12:27:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 1 -- start start 2024-09-18T12:27:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient: init 2024-09-18T12:27:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 0x7f8154131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient(hunting): picked mon.noname-b con 0x7f8154131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f815409e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8154136bb0 0x7f8154138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient(hunting): picked mon.noname-a con 0x7f8154136bb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815409faf0 con 0x7f815409e910 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f815409fd30 con 0x7f8154136bb0 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81540a1110 con 0x7f8154131ab0 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815c35d640 10 monclient(hunting): authenticate will time out at 878.850944s 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f815a8fa640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f8154136bb0 msgr2=0x7f8154138f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.528+0000 7f81598f8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f815409e910 legacy=0x7f815409ece0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a8fa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8154136bb0 0x7f8154138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 0x7f8154131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 0x7f8154131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48980/0 (socket says 172.21.15.116:48980) 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 -- 172.21.15.116:0/1803739502 learned_addr learned my addr 172.21.15.116:0/1803739502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): get_auth_request con 0x7f8154131ab0 auth_method 0 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient(hunting): handle_auth_done global_id 44177 payload 274 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 -- 172.21.15.116:0/1803739502 >> v2:172.21.15.116:3300/0 conn(0x7f8154136bb0 msgr2=0x7f8154138f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 --2- 172.21.15.116:0/1803739502 >> v2:172.21.15.116:3300/0 conn(0x7f8154136bb0 0x7f8154138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 -- 172.21.15.116:0/1803739502 >> v1:172.21.15.116:6789/0 conn(0x7f815409e910 legacy=0x7f815409ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 monclient: found mon.noname-b 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.529+0000 7f815a0f9640 1 -- 172.21.15.116:0/1803739502 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81540a13f0 con 0x7f8154131ab0 2024-09-18T12:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815a0f9640 1 -- 172.21.15.116:0/1803739502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 msgr2=0x7f8154131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815a0f9640 1 --2- 172.21.15.116:0/1803739502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 0x7f8154131e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8150009910 tx=0x7f81500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a724b08d16f5e865 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 1 -- 172.21.15.116:0/1803739502 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815002ae20 con 0x7f8154131ab0 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: dump: 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.544 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 5 monclient: authenticate success, global_id 44177 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 1 -- 172.21.15.116:0/1803739502 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815002f4b0 con 0x7f8154131ab0 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 4 set_mon_vals no callback set 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 1 -- 172.21.15.116:0/1803739502 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815002f850 con 0x7f8154131ab0 2024-09-18T12:27:31.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f81590f7640 10 monclient: dump: 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 10 monclient: shutdown 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 -- 172.21.15.116:0/1803739502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 msgr2=0x7f8154131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 --2- 172.21.15.116:0/1803739502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 0x7f8154131e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8150009910 tx=0x7f81500091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 -- 172.21.15.116:0/1803739502 shutdown_connections 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 --2- 172.21.15.116:0/1803739502 >> v2:172.21.15.116:3300/0 conn(0x7f8154136bb0 0x7f8154138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 --2- 172.21.15.116:0/1803739502 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154131ab0 0x7f8154131e80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 -- 172.21.15.116:0/1803739502 >> 172.21.15.116:0/1803739502 conn(0x7f8154067d30 msgr2=0x7f81540a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 -- 172.21.15.116:0/1803739502 shutdown_connections 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.532+0000 7f815c35d640 1 -- 172.21.15.116:0/1803739502 wait complete. 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.533+0000 7f815c35d640 10 monclient: build_initial_monmap 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.533+0000 7f815c35d640 10 monclient: monmap: 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:31.547 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.533+0000 7f815c35d640 1 Processor -- start 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.533+0000 7f815c35d640 1 -- start start 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.533+0000 7f815c35d640 10 monclient: init 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815409e910 0x7f81540975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): picked mon.noname-a con 0x7f815409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8154131ab0 0x7f8154097b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): picked mon.noname-b con 0x7f8154131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 0x7f8154098070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): picked mon.noname-c con 0x7f8154136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815409afd0 con 0x7f815409e910 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8154099710 con 0x7f8154131ab0 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8154099460 con 0x7f8154136bb0 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a0f9640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815409e910 msgr2=0x7f81540975f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815c35d640 10 monclient(hunting): authenticate will time out at 878.856186s 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a0f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815409e910 0x7f81540975f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 0x7f8154098070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 0x7f8154098070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48984/0 (socket says 172.21.15.116:48984) 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 -- 172.21.15.116:0/2214027400 learned_addr learned my addr 172.21.15.116:0/2214027400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): get_auth_request con 0x7f8154136bb0 auth_method 0 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f81598f8640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8154131ab0 0x7f8154097b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f81598f8640 10 monclient(hunting): get_auth_request con 0x7f8154131ab0 auth_method 0 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f81598f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f81598f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f81598f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient(hunting): handle_auth_done global_id 44180 payload 824 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8154131ab0 msgr2=0x7f8154097b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8154131ab0 0x7f8154097b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815409e910 msgr2=0x7f81540975f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f815409e910 0x7f81540975f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 monclient: found mon.noname-c 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.534+0000 7f815a8fa640 1 -- 172.21.15.116:0/2214027400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150009450 con 0x7f8154136bb0 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815a8fa640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 msgr2=0x7f8154098070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815a8fa640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 0x7f8154098070 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f814c00bf50 tx=0x7f814c009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c00cb80 con 0x7f8154136bb0 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: dump: 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814c00cd00 con 0x7f8154136bb0 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815c35d640 5 monclient: authenticate success, global_id 44180 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815c35d640 10 monclient: _renew_subs 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815c35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815409bde0 con 0x7f8154136bb0 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c016430 con 0x7f8154136bb0 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f814a7fc640 10 monclient: dump: 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f81588f6640 4 set_mon_vals no callback set 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815c35d640 10 monclient: _renew_subs 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815c35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.536+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8154098f50 con 0x7f8154136bb0 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815c35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815c35d640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 0x7f815402aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815c35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8154063740 con 0x7f815409b6d0 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f814c012020 con 0x7f8154136bb0 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 0x7f815402aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 10 monclient: get_auth_request con 0x7f815409b6d0 auth_method 0 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f814a7fc640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8138038440 0x7f813803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.537+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5358+0+0 (secure 0 0 0) 0x7f814c04b770 con 0x7f8154136bb0 2024-09-18T12:27:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.538+0000 7f815a0f9640 10 monclient(hunting): handle_auth_done global_id 44183 payload 824 2024-09-18T12:27:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.538+0000 7f815a0f9640 1 -- 172.21.15.116:0/2214027400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f815404bf60 con 0x7f815409b6d0 2024-09-18T12:27:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.538+0000 7f81598f8640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8138038440 0x7f813803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.538+0000 7f81598f8640 10 monclient: get_auth_request con 0x7f8138038440 auth_method 0 2024-09-18T12:27:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f815a0f9640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 msgr2=0x7f815402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f815a0f9640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 0x7f815402aa40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8150034000 tx=0x7f8150032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815002ae20 con 0x7f815409b6d0 2024-09-18T12:27:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 10 monclient: dump: 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f81598f8640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8138038440 0x7f813803a8f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8144006fd0 tx=0x7f8144008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f815002f620 con 0x7f815409b6d0 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 msgr2=0x7f815402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 0x7f815402aa40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8150034000 tx=0x7f8150032000 comp rx=0 tx=0).stop 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 msgr2=0x7f815402aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.540+0000 7f814a7fc640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f815409b6d0 0x7f815402aa40 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8150034000 tx=0x7f8150032000 comp rx=0 tx=0).stop 2024-09-18T12:27:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815c35d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815c35d640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 0x7f815405e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815c35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8154133da0 con 0x7f81540985b0 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 0x7f815405e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient: get_auth_request con 0x7f81540985b0 auth_method 0 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 10 monclient(hunting): handle_auth_done global_id 44186 payload 824 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.546+0000 7f815a0f9640 1 -- 172.21.15.116:0/2214027400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8154066280 con 0x7f81540985b0 2024-09-18T12:27:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.548+0000 7f815a0f9640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 msgr2=0x7f815405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.548+0000 7f815a0f9640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 0x7f815405e310 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8150009910 tx=0x7f815003d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815003d8a0 con 0x7f81540985b0 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 10 monclient: dump: 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f815003da00 con 0x7f81540985b0 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 msgr2=0x7f815405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 0x7f815405e310 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8150009910 tx=0x7f815003d000 comp rx=0 tx=0).stop 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 msgr2=0x7f815405e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f814a7fc640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81540985b0 0x7f815405e310 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8150009910 tx=0x7f815003d000 comp rx=0 tx=0).stop 2024-09-18T12:27:31.564 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 26, 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:31.565 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:31.566 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:31.567 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:31.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:31.569 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:31.569 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:31.569 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:31.569 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:31.569 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:31.569 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:31.570 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:31.570 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:31.570 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:31.570 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:31.570 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:31.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:31.572 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.578 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:31.579 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:31.579 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:27:31.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.579 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.579 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:31.579 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8138038440 msgr2=0x7f813803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8138038440 0x7f813803a8f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8144006fd0 tx=0x7f8144008040 comp rx=0 tx=0).stop 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 10 monclient: shutdown 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 msgr2=0x7f8154098070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815a8fa640 1 -- 172.21.15.116:0/2214027400 reap_dead start 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 0x7f8154098070 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f814c00bf50 tx=0x7f814c009700 comp rx=0 tx=0).stop 2024-09-18T12:27:31.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 shutdown_connections 2024-09-18T12:27:31.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 --2- 172.21.15.116:0/2214027400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8154136bb0 0x7f8154098070 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 >> 172.21.15.116:0/2214027400 conn(0x7f8154067d30 msgr2=0x7f81540a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 shutdown_connections 2024-09-18T12:27:31.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.549+0000 7f815c35d640 1 -- 172.21.15.116:0/2214027400 wait complete. 2024-09-18T12:27:31.582 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:27:31.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:27:31.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:27:31.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.583+0000 7f6c9d40f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:31.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.583+0000 7f6c9d40f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:31.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.583+0000 7f6c9d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:31.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:31.583+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6c980a13e0 con 0x7f6c981321a0 2024-09-18T12:27:31.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.673+0000 7f15b099d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:31.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.673+0000 7f15b099d640 10 monclient: build_initial_monmap 2024-09-18T12:27:31.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.673+0000 7f15b099d640 10 monclient: monmap: 2024-09-18T12:27:31.675 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:31.675 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.675 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:27:31.675107+0000 2024-09-18T12:27:31.675 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:27:31.675107+0000 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:31.676 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.673+0000 7f15b099d640 1 Processor -- start 2024-09-18T12:27:31.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 1 -- start start 2024-09-18T12:27:31.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient: init 2024-09-18T12:27:31.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): picked mon.noname-b con 0x7f15ac096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f15ac09aaa0 0x7f15ac0971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): picked mon.noname-c con 0x7f15ac09aaa0 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f15ac097720 0x7f15ac12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): picked mon.noname-a con 0x7f15ac097720 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac099620 con 0x7f15ac097720 2024-09-18T12:27:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac099340 con 0x7f15ac09aaa0 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac099900 con 0x7f15ac096e10 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15b099d640 10 monclient(hunting): authenticate will time out at 878.996670s 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aad9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f15ac097720 msgr2=0x7f15ac12e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f15ac097720 0x7f15ac12e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac09a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:49018/0 (socket says 172.21.15.116:49018) 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 1 -- 172.21.15.116:0/1447207221 learned_addr learned my addr 172.21.15.116:0/1447207221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:31.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 10 monclient(hunting): get_auth_request con 0x7f15ac096e10 auth_method 0 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.674+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/1447207221 >> v2:172.21.15.144:3300/0 conn(0x7f15ac09aaa0 0x7f15ac0971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15a9d9b640 10 monclient(hunting): get_auth_request con 0x7f15ac09aaa0 auth_method 0 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 10 monclient(hunting): handle_auth_done global_id 44189 payload 274 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 1 -- 172.21.15.116:0/1447207221 >> v2:172.21.15.144:3300/0 conn(0x7f15ac09aaa0 msgr2=0x7f15ac0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 1 --2- 172.21.15.116:0/1447207221 >> v2:172.21.15.144:3300/0 conn(0x7f15ac09aaa0 0x7f15ac0971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 1 -- 172.21.15.116:0/1447207221 >> v2:172.21.15.116:3300/0 conn(0x7f15ac097720 msgr2=0x7f15ac12e5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 1 --2- 172.21.15.116:0/1447207221 >> v2:172.21.15.116:3300/0 conn(0x7f15ac097720 0x7f15ac12e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 1 monclient: found mon.noname-b 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.675+0000 7f15aa59c640 1 -- 172.21.15.116:0/1447207221 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac097b10 con 0x7f15ac096e10 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.677+0000 7f15aa59c640 1 -- 172.21.15.116:0/1447207221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 msgr2=0x7f15ac09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.677+0000 7f15aa59c640 1 --2- 172.21.15.116:0/1447207221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac09a560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0a0d60 tx=0x7f159c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde1e9bb3debf6c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 1 -- 172.21.15.116:0/1447207221 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c032440 con 0x7f15ac096e10 2024-09-18T12:27:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: dump: 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 5 monclient: authenticate success, global_id 44189 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 1 -- 172.21.15.116:0/1447207221 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c0325a0 con 0x7f15ac096e10 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 4 set_mon_vals no callback set 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 1 -- 172.21.15.116:0/1447207221 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c032940 con 0x7f15ac096e10 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15a959a640 10 monclient: dump: 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 10 monclient: shutdown 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 -- 172.21.15.116:0/1447207221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 msgr2=0x7f15ac09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 --2- 172.21.15.116:0/1447207221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac09a560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0a0d60 tx=0x7f159c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 -- 172.21.15.116:0/1447207221 shutdown_connections 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 --2- 172.21.15.116:0/1447207221 >> v2:172.21.15.116:3300/0 conn(0x7f15ac097720 0x7f15ac12e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 --2- 172.21.15.116:0/1447207221 >> v2:172.21.15.144:3300/0 conn(0x7f15ac09aaa0 0x7f15ac0971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 --2- 172.21.15.116:0/1447207221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac09a560 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 -- 172.21.15.116:0/1447207221 >> 172.21.15.116:0/1447207221 conn(0x7f15ac0672b0 msgr2=0x7f15ac0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 -- 172.21.15.116:0/1447207221 shutdown_connections 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.678+0000 7f15b099d640 1 -- 172.21.15.116:0/1447207221 wait complete. 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient: build_initial_monmap 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient: monmap: 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:27:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 Processor -- start 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 -- start start 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient: init 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac12e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): picked mon.noname-c con 0x7f15ac096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac097720 0x7f15ac11bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): picked mon.noname-a con 0x7f15ac097720 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac09aaa0 0x7f15ac11c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): picked mon.noname-b con 0x7f15ac09aaa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac11d6f0 con 0x7f15ac097720 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac11f1b0 con 0x7f15ac09aaa0 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac11f490 con 0x7f15ac096e10 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15b099d640 10 monclient(hunting): authenticate will time out at 879.001925s 2024-09-18T12:27:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15a9d9b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac097720 msgr2=0x7f15ac11bc90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac097720 0x7f15ac11bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.679+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac12e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac12e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:49030/0 (socket says 172.21.15.116:49030) 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 -- 172.21.15.116:0/2089612675 learned_addr learned my addr 172.21.15.116:0/2089612675 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): get_auth_request con 0x7f15ac096e10 auth_method 0 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac09aaa0 0x7f15ac11c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 10 monclient(hunting): get_auth_request con 0x7f15ac09aaa0 auth_method 0 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient(hunting): handle_auth_done global_id 44192 payload 824 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac09aaa0 msgr2=0x7f15ac11c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac09aaa0 0x7f15ac11c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac097720 msgr2=0x7f15ac11bc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac097720 0x7f15ac11bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 monclient: found mon.noname-c 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.680+0000 7f15aa59c640 1 -- 172.21.15.116:0/2089612675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c009450 con 0x7f15ac096e10 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15aa59c640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 msgr2=0x7f15ac12e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15aa59c640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac12e170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f159c009210 tx=0x7f159c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c031a90 con 0x7f15ac096e10 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: dump: 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c031c10 con 0x7f15ac096e10 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15b099d640 5 monclient: authenticate success, global_id 44192 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:31.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15b099d640 10 monclient: _renew_subs 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c03d440 con 0x7f15ac096e10 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15b099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac11f770 con 0x7f15ac096e10 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f159a7fc640 10 monclient: dump: 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.701 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15a8d99640 4 set_mon_vals no callback set 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15b099d640 10 monclient: _renew_subs 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15b099d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.682+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac1200f0 con 0x7f15ac096e10 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15b099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f159c039040 con 0x7f15ac096e10 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15b099d640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 0x7f15ac06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac065f20 con 0x7f15ac11c710 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f159a7fc640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f158c038440 0x7f158c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 0x7f15ac06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 10 monclient: get_auth_request con 0x7f15ac11c710 auth_method 0 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5358+0+0 (secure 0 0 0) 0x7f159c06ba30 con 0x7f15ac096e10 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15aad9d640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f158c038440 0x7f158c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.683+0000 7f15aad9d640 10 monclient: get_auth_request con 0x7f158c038440 auth_method 0 2024-09-18T12:27:31.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.684+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_done global_id 4708 payload 824 2024-09-18T12:27:31.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.684+0000 7f15a9d9b640 1 -- 172.21.15.116:0/2089612675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac11f9b0 con 0x7f15ac11c710 2024-09-18T12:27:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.686+0000 7f15a9d9b640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 msgr2=0x7f15ac06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.686+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 0x7f15ac06bf40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f15ac12f5e0 tx=0x7f15a0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.686+0000 7f15aad9d640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f158c038440 0x7f158c03a8f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f15ac11d3b0 tx=0x7f15a40096a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a000b810 con 0x7f15ac11c710 2024-09-18T12:27:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 10 monclient: dump: 2024-09-18T12:27:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f15a000b970 con 0x7f15ac11c710 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 msgr2=0x7f15ac06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 0x7f15ac06bf40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f15ac12f5e0 tx=0x7f15a0008040 comp rx=0 tx=0).stop 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 msgr2=0x7f15ac06bf40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.688+0000 7f159a7fc640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac11c710 0x7f15ac06bf40 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f15ac12f5e0 tx=0x7f15a0008040 comp rx=0 tx=0).stop 2024-09-18T12:27:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15b099d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:27:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15b099d640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 0x7f15ac06c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15b099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac11cae0 con 0x7f15ac06c480 2024-09-18T12:27:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 0x7f15ac06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient: get_auth_request con 0x7f15ac06c480 auth_method 0 2024-09-18T12:27:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 10 monclient(hunting): handle_auth_done global_id 4711 payload 824 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.694+0000 7f15a9d9b640 1 -- 172.21.15.116:0/2089612675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f15ac04bf60 con 0x7f15ac06c480 2024-09-18T12:27:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15a9d9b640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 msgr2=0x7f15ac06c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 0x7f15ac06c880 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f15a0007fd0 tx=0x7f15a001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0021610 con 0x7f15ac06c480 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 10 monclient: dump: 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:27:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:31.714 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:27:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f15a0021770 con 0x7f15ac06c480 2024-09-18T12:27:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:27:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 msgr2=0x7f15ac06c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 0x7f15ac06c880 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f15a0007fd0 tx=0x7f15a001a000 comp rx=0 tx=0).stop 2024-09-18T12:27:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 msgr2=0x7f15ac06c880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f159a7fc640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac06c480 0x7f15ac06c880 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.716 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:27:31.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:31.716 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:31.716 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:27:31.716 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 26, 2024-09-18T12:27:31.716 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:27:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:27:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:27:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:27:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:27:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:27:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:27:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.724 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:27:31.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.727 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:27:31.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:27:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:27:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:27:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:27:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:27:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:27:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:27:31.731 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f158c038440 msgr2=0x7f158c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f158c038440 0x7f158c03a8f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f15ac11d3b0 tx=0x7f15a40096a0 comp rx=0 tx=0).stop 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 10 monclient: shutdown 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 msgr2=0x7f15ac12e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 --2- 172.21.15.116:0/2089612675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac096e10 0x7f15ac12e170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f159c009210 tx=0x7f159c032000 comp rx=0 tx=0).stop 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15aad9d640 1 -- 172.21.15.116:0/2089612675 reap_dead start 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15aad9d640 1 -- 172.21.15.116:0/2089612675 reap_dead start 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 shutdown_connections 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 >> 172.21.15.116:0/2089612675 conn(0x7f15ac0672b0 msgr2=0x7f15ac09b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 shutdown_connections 2024-09-18T12:27:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:27:31.697+0000 7f15b099d640 1 -- 172.21.15.116:0/2089612675 wait complete. 2024-09-18T12:27:31.734 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:27:31.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:27:32.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.562+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v43) v1 ==== 152+0+0 (secure 0 0 0) 0x7f6c900393c0 con 0x7f6c981321a0 2024-09-18T12:27:32.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.563+0000 7f6c8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:32.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.563+0000 7f6c8effd640 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-18T12:27:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.668+0000 7f6c9d40f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.668+0000 7f6c9d40f640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.668+0000 7f6c9d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.668+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6c980a13e0 con 0x7f6c981321a0 2024-09-18T12:27:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.668+0000 7f6c8effd640 1 -- 172.21.15.144:0/749994942 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) v1 ==== 159+0+0 (secure 0 0 0) 0x7f6c9003a3a0 con 0x7f6c981321a0 2024-09-18T12:27:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.668+0000 7f6c8effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.668+0000 7f6c8effd640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-18T12:27:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'fooo' does not exist 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c78038440 msgr2=0x7f6c7803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c78038440 0x7f6c7803a8f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c880038f0 tx=0x7f6c880090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 10 monclient: shutdown 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 msgr2=0x7f6c98130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 0x7f6c98130b80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009e60 tx=0x7f6c90037000 comp rx=0 tx=0).stop 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 shutdown_connections 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c78038440 0x7f6c7803a8f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c981321a0 0x7f6c98130b80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c9809f020 0x7f6c9811bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 --2- 172.21.15.144:0/749994942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c9809ec50 0x7f6c9809a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 >> 172.21.15.144:0/749994942 conn(0x7f6c980a5d40 msgr2=0x7f6c98134a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 shutdown_connections 2024-09-18T12:27:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.670+0000 7f6c9d40f640 1 -- 172.21.15.144:0/749994942 wait complete. 2024-09-18T12:27:32.680 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-18T12:27:32.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.756+0000 7f3cc6314640 10 monclient: get_monmap_and_config 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.756+0000 7f3cc6314640 10 monclient: build_initial_monmap 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.756+0000 7f3cc6314640 10 monclient: monmap: 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:32.757561+0000 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:32.757561+0000 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:32.758 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:32.759 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:32.759 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:32.759 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:32.759 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:32.759 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:32.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 1 Processor -- start 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 1 -- start start 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient: init 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3cc0098c70 0x7f3cc0099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc0098c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3cc0099580 0x7f3cc009dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc0099580 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cc0099c70 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc009f680 con 0x7f3cc0099c70 2024-09-18T12:27:32.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc009f420 con 0x7f3cc0099580 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc0068a20 con 0x7f3cc0098c70 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.757+0000 7f3cc6314640 10 monclient(hunting): authenticate will time out at 888.326938s 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbf7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f3cc0099580 msgr2=0x7f3cc009dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3cc0098c70 0x7f3cc0099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cc48b1640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3cc0099c70 0x7f3cc009e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:32980/0 (socket says 172.21.15.144:32980) 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cc48b1640 1 -- 172.21.15.144:0/1482684920 learned_addr learned my addr 172.21.15.144:0/1482684920 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbf7fe640 1 --2- 172.21.15.144:0/1482684920 >> v2:172.21.15.116:3300/0 conn(0x7f3cc0099580 0x7f3cc009dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): get_auth_request con 0x7f3cc0098c70 auth_method 0 2024-09-18T12:27:32.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient(hunting): handle_auth_done global_id 4714 payload 274 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 1 -- 172.21.15.144:0/1482684920 >> v2:172.21.15.116:3300/0 conn(0x7f3cc0099580 msgr2=0x7f3cc009dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 1 --2- 172.21.15.144:0/1482684920 >> v2:172.21.15.116:3300/0 conn(0x7f3cc0099580 0x7f3cc009dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 1 -- 172.21.15.144:0/1482684920 >> v1:172.21.15.144:6789/0 conn(0x7f3cc0099c70 legacy=0x7f3cc009e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:32.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient: _finish_hunting 0 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 1 monclient: found mon.noname-c 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.758+0000 7f3cbffff640 1 -- 172.21.15.144:0/1482684920 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc0096a30 con 0x7f3cc0098c70 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbffff640 1 -- 172.21.15.144:0/1482684920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0098c70 msgr2=0x7f3cc0099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbffff640 1 --2- 172.21.15.144:0/1482684920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0098c70 0x7f3cc0099040 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009850 tx=0x7f3cb00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4054dc280566bb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 1 -- 172.21.15.144:0/1482684920 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0032440 con 0x7f3cc0098c70 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: dump: 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.769 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cc6314640 5 monclient: authenticate success, global_id 4714 2024-09-18T12:27:32.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 1 -- 172.21.15.144:0/1482684920 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb00325a0 con 0x7f3cc0098c70 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cc6314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cc6314640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cc6314640 4 set_mon_vals no callback set 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 1 -- 172.21.15.144:0/1482684920 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0032980 con 0x7f3cc0098c70 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.761+0000 7f3cbeffd640 10 monclient: dump: 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:32.771 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 10 monclient: shutdown 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 -- 172.21.15.144:0/1482684920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0098c70 msgr2=0x7f3cc0099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 --2- 172.21.15.144:0/1482684920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0098c70 0x7f3cc0099040 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009850 tx=0x7f3cb00090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 -- 172.21.15.144:0/1482684920 shutdown_connections 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 --2- 172.21.15.144:0/1482684920 >> v2:172.21.15.116:3300/0 conn(0x7f3cc0099580 0x7f3cc009dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 --2- 172.21.15.144:0/1482684920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0098c70 0x7f3cc0099040 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 -- 172.21.15.144:0/1482684920 >> 172.21.15.144:0/1482684920 conn(0x7f3cc0070e90 msgr2=0x7f3cc00a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 -- 172.21.15.144:0/1482684920 shutdown_connections 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 -- 172.21.15.144:0/1482684920 wait complete. 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 10 monclient: build_initial_monmap 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 10 monclient: monmap: 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:32.773 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.762+0000 7f3cc6314640 1 Processor -- start 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 1 -- start start 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient: init 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0098c70 0x7f3cc0133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc0098c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 0x7f3cc0134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): picked mon.noname-b con 0x7f3cc0099580 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:32.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3cc0099c70 0x7f3cc011db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc0099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc0121710 con 0x7f3cc0098c70 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc0120e70 con 0x7f3cc0099580 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc0121430 con 0x7f3cc0099c70 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cc6314640 10 monclient(hunting): authenticate will time out at 888.332561s 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 0x7f3cc0134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbffff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0098c70 msgr2=0x7f3cc0133d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 0x7f3cc0134290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57204/0 (socket says 172.21.15.144:57204) 2024-09-18T12:27:32.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 -- 172.21.15.144:0/2680737967 learned_addr learned my addr 172.21.15.144:0/2680737967 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbffff640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0098c70 0x7f3cc0133d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): get_auth_request con 0x7f3cc0099580 auth_method 0 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient(hunting): handle_auth_done global_id 4717 payload 824 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 -- 172.21.15.144:0/2680737967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3cc0099c70 msgr2=0x7f3cc011db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3cc0099c70 0x7f3cc011db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 -- 172.21.15.144:0/2680737967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0098c70 msgr2=0x7f3cc0133d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:32.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0098c70 0x7f3cc0133d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:32.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:32.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:32.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:32.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:32.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.763+0000 7f3cbf7fe640 1 -- 172.21.15.144:0/2680737967 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0009390 con 0x7f3cc0099580 2024-09-18T12:27:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cbf7fe640 1 -- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 msgr2=0x7f3cc0134290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cbf7fe640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 0x7f3cc0134290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400b600 tx=0x7f3cb40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4015070 con 0x7f3cc0099580 2024-09-18T12:27:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: dump: 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:32.780 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: _finish_auth 0 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4003d90 con 0x7f3cc0099580 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cc6314640 5 monclient: authenticate success, global_id 4717 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cc6314640 10 monclient: _renew_subs 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4006380 con 0x7f3cc0099580 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cc6314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc0134b30 con 0x7f3cc0099580 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:32.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3ca7fff640 10 monclient: dump: 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cbe7fc640 4 set_mon_vals no callback set 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cc6314640 10 monclient: _renew_subs 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cc6314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.766+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc01367a0 con 0x7f3cc0099580 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3cb4010440 con 0x7f3cc0099580 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3cc6314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3cc6314640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3cc6314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc011f340 con 0x7f3cc0099580 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3ca7fff640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cac038440 0x7f3cac03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4776+0+0 (secure 0 0 0) 0x7f3cb404b370 con 0x7f3cc0099580 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3cbffff640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cac038440 0x7f3cac03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.767+0000 7f3cbffff640 10 monclient: get_auth_request con 0x7f3cac038440 auth_method 0 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.769+0000 7f3cbffff640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cac038440 0x7f3cac03a8f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0034000 tx=0x7f3cb0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.771+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3cb4018ab0 con 0x7f3cc0099580 2024-09-18T12:27:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.771+0000 7f3ca7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.771+0000 7f3ca7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:32.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.948+0000 7f3cc6314640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:32.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.948+0000 7f3cc6314640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:32.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.948+0000 7f3cc6314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:32.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:32.948+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3cc00008d0 con 0x7f3cc0099580 2024-09-18T12:27:33.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.574+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v44) v1 ==== 149+0+0 (secure 0 0 0) 0x7f3cb4018690 con 0x7f3cc0099580 2024-09-18T12:27:33.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.574+0000 7f3ca7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:33.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.574+0000 7f3ca7fff640 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-18T12:27:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.680+0000 7f3cc6314640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:33.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.680+0000 7f3cc6314640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:33.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.680+0000 7f3cc6314640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:33.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.680+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3cc011ff30 con 0x7f3cc0099580 2024-09-18T12:27:33.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.680+0000 7f3ca7fff640 1 -- 172.21.15.144:0/2680737967 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v44) v1 ==== 156+0+0 (secure 0 0 0) 0x7f3cb4018ab0 con 0x7f3cc0099580 2024-09-18T12:27:33.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.681+0000 7f3ca7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:33.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.681+0000 7f3ca7fff640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-18T12:27:33.682 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'foo' does not exist 2024-09-18T12:27:33.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cac038440 msgr2=0x7f3cac03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:33.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cac038440 0x7f3cac03a8f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0034000 tx=0x7f3cb0032000 comp rx=0 tx=0).stop 2024-09-18T12:27:33.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 10 monclient: shutdown 2024-09-18T12:27:33.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:33.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 msgr2=0x7f3cc0134290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:33.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 0x7f3cc0134290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400b600 tx=0x7f3cb40091a0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 shutdown_connections 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cac038440 0x7f3cac03a8f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3cc0099c70 0x7f3cc011db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3cc0099580 0x7f3cc0134290 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 --2- 172.21.15.144:0/2680737967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0098c70 0x7f3cc0133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.682+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 >> 172.21.15.144:0/2680737967 conn(0x7f3cc0070e90 msgr2=0x7f3cc0134ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.683+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 shutdown_connections 2024-09-18T12:27:33.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.683+0000 7f3cc6314640 1 -- 172.21.15.144:0/2680737967 wait complete. 2024-09-18T12:27:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.765+0000 7f145c91d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.765+0000 7f145c91d640 10 monclient: build_initial_monmap 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.765+0000 7f145c91d640 10 monclient: monmap: 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:33.766842+0000 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:33.766842+0000 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:33.767 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:33.768 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:33.768 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:33.768 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:33.768 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:33.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.766+0000 7f145c91d640 1 Processor -- start 2024-09-18T12:27:33.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.766+0000 7f145c91d640 1 -- start start 2024-09-18T12:27:33.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.766+0000 7f145c91d640 10 monclient: init 2024-09-18T12:27:33.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:33.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T12:27:33.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1458098c70 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:33.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1458099c70 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14581383e0 0x7f145813a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient(hunting): picked mon.noname-c con 0x7f14581383e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145809f680 con 0x7f1458099c70 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145809f420 con 0x7f1458098c70 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1458068a20 con 0x7f14581383e0 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145c91d640 10 monclient(hunting): authenticate will time out at 889.336178s 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f144ffff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1458099c70 legacy=0x7f1458136cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f14581383e0 0x7f145813a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145659c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1458098c70 0x7f1458099040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:32986/0 (socket says 172.21.15.144:32986) 2024-09-18T12:27:33.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f145659c640 1 -- 172.21.15.144:0/2565501645 learned_addr learned my addr 172.21.15.144:0/2565501645 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): get_auth_request con 0x7f14581383e0 auth_method 0 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient(hunting): handle_auth_done global_id 4720 payload 274 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 1 -- 172.21.15.144:0/2565501645 >> v1:172.21.15.144:6789/0 conn(0x7f1458098c70 legacy=0x7f1458099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 1 -- 172.21.15.144:0/2565501645 >> v1:172.21.15.116:6789/0 conn(0x7f1458099c70 legacy=0x7f1458136cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:27:33.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 1 monclient: found mon.noname-c 2024-09-18T12:27:33.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:33.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:33.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.767+0000 7f1456d9d640 1 -- 172.21.15.144:0/2565501645 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458096a30 con 0x7f14581383e0 2024-09-18T12:27:33.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1456d9d640 1 -- 172.21.15.144:0/2565501645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14581383e0 msgr2=0x7f145813a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:33.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1456d9d640 1 --2- 172.21.15.144:0/2565501645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14581383e0 0x7f145813a7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1458096c20 tx=0x7f1450009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb36e4fab50d6ec8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:33.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 1 -- 172.21.15.144:0/2565501645 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450007840 con 0x7f14581383e0 2024-09-18T12:27:33.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: dump: 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: _finish_auth 0 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 1 -- 172.21.15.144:0/2565501645 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145002d050 con 0x7f14581383e0 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 5 monclient: authenticate success, global_id 4720 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 1 -- 172.21.15.144:0/2565501645 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450007bd0 con 0x7f14581383e0 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 4 set_mon_vals no callback set 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f1455d9b640 10 monclient: dump: 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:33.780 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 10 monclient: shutdown 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 1 -- 172.21.15.144:0/2565501645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14581383e0 msgr2=0x7f145813a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.770+0000 7f145c91d640 1 --2- 172.21.15.144:0/2565501645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14581383e0 0x7f145813a7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1458096c20 tx=0x7f1450009780 comp rx=0 tx=0).stop 2024-09-18T12:27:33.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 1 -- 172.21.15.144:0/2565501645 shutdown_connections 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 1 --2- 172.21.15.144:0/2565501645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f14581383e0 0x7f145813a7c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 1 -- 172.21.15.144:0/2565501645 >> 172.21.15.144:0/2565501645 conn(0x7f1458070e90 msgr2=0x7f14580a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 1 -- 172.21.15.144:0/2565501645 shutdown_connections 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 1 -- 172.21.15.144:0/2565501645 wait complete. 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 10 monclient: build_initial_monmap 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 10 monclient: monmap: 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:33.782 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 1 Processor -- start 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 1 -- start start 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.771+0000 7f145c91d640 10 monclient: init 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 0x7f1458137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): picked mon.noname-b con 0x7f1458098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1458099c70 0x7f145811e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): picked mon.noname-c con 0x7f1458099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:33.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14581383e0 0x7f145811eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): picked mon.noname-a con 0x7f14581383e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1458122a90 con 0x7f14581383e0 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14581223c0 con 0x7f1458098c70 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14580a1c30 con 0x7f1458099c70 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145c91d640 10 monclient(hunting): authenticate will time out at 889.341411s 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 0x7f1458137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f1456d9d640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14581383e0 msgr2=0x7f145811eca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 0x7f1458137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57214/0 (socket says 172.21.15.144:57214) 2024-09-18T12:27:33.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 -- 172.21.15.144:0/3759071801 learned_addr learned my addr 172.21.15.144:0/3759071801 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f1456d9d640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14581383e0 0x7f145811eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): get_auth_request con 0x7f1458098c70 auth_method 0 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f144ffff640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1458099c70 0x7f145811e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f144ffff640 10 monclient(hunting): get_auth_request con 0x7f1458099c70 auth_method 0 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f144ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f144ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:33.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f144ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient(hunting): handle_auth_done global_id 4723 payload 824 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 -- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1458099c70 msgr2=0x7f145811e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1458099c70 0x7f145811e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 -- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14581383e0 msgr2=0x7f145811eca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14581383e0 0x7f145811eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 monclient: found mon.noname-b 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.772+0000 7f145659c640 1 -- 172.21.15.144:0/3759071801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1450009a20 con 0x7f1458098c70 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145659c640 1 -- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 msgr2=0x7f1458137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145659c640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 0x7f1458137dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f144800c5e0 tx=0x7f1448004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:33.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145659c640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 0x7f1458137dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f144800c5e0 tx=0x7f1448004960 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f1448005cb0 mon_map magic: 0 v1 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448005cb0 con 0x7f1458098c70 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 10 monclient: dump: 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:33.787 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448005e30 con 0x7f1458098c70 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145c91d640 5 monclient: authenticate success, global_id 4723 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145c91d640 10 monclient: _renew_subs 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145c91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1458120720 con 0x7f1458098c70 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f144e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145559a640 4 set_mon_vals no callback set 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145c91d640 10 monclient: _renew_subs 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145c91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.775+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1458137320 con 0x7f1458098c70 2024-09-18T12:27:33.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448007340 con 0x7f1458098c70 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144e7fc640 10 monclient: dump: 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:33.789 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f144801b400 con 0x7f1458098c70 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f145c91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f145c91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f145c91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14580008d0 con 0x7f1458098c70 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144e7fc640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14380384c0 0x7f143803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144ffff640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14380384c0 0x7f143803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.776+0000 7f144ffff640 10 monclient: get_auth_request con 0x7f14380384c0 auth_method 0 2024-09-18T12:27:33.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.778+0000 7f144ffff640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14380384c0 0x7f143803a970 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f145806a840 tx=0x7f1440006f10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:33.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:33.780+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f144804b2b0 con 0x7f1458098c70 2024-09-18T12:27:34.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.398+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f144801a960 con 0x7f1458098c70 2024-09-18T12:27:34.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.398+0000 7f144e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:34.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.398+0000 7f144e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:34.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.552+0000 7f145c91d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.552+0000 7f145c91d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.552+0000 7f145c91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.552+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1458069890 con 0x7f1458098c70 2024-09-18T12:27:34.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.553+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v44) v1 ==== 156+0+0 (secure 0 0 0) 0x7f144801a360 con 0x7f1458098c70 2024-09-18T12:27:34.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.553+0000 7f144e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.553+0000 7f144e7fc640 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-18T12:27:34.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.645+0000 7f145c91d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.645+0000 7f145c91d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.645+0000 7f145c91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.645+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1458064a20 con 0x7f1458098c70 2024-09-18T12:27:34.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.645+0000 7f144e7fc640 1 -- 172.21.15.144:0/3759071801 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v44) v1 ==== 156+0+0 (secure 0 0 0) 0x7f144801a540 con 0x7f1458098c70 2024-09-18T12:27:34.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.645+0000 7f144e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.645+0000 7f144e7fc640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-18T12:27:34.647 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'foo' does not exist 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14380384c0 msgr2=0x7f143803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14380384c0 0x7f143803a970 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f145806a840 tx=0x7f1440006f10 comp rx=0 tx=0).stop 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 10 monclient: shutdown 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 msgr2=0x7f1458137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 0x7f1458137dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f144800c5e0 tx=0x7f1448004960 comp rx=0 tx=0).stop 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 shutdown_connections 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f14380384c0 0x7f143803a970 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14581383e0 0x7f145811eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1458099c70 0x7f145811e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 --2- 172.21.15.144:0/3759071801 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1458098c70 0x7f1458137dc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 >> 172.21.15.144:0/3759071801 conn(0x7f1458070e90 msgr2=0x7f1458097330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:34.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 shutdown_connections 2024-09-18T12:27:34.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.647+0000 7f145c91d640 1 -- 172.21.15.144:0/3759071801 wait complete. 2024-09-18T12:27:34.657 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-18T12:27:34.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.732+0000 7f466df1d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:34.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.732+0000 7f466df1d640 10 monclient: build_initial_monmap 2024-09-18T12:27:34.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.732+0000 7f466df1d640 10 monclient: monmap: 2024-09-18T12:27:34.733 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:34.733311+0000 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:34.733311+0000 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:34.734 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:34.735 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:34.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 1 Processor -- start 2024-09-18T12:27:34.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 1 -- start start 2024-09-18T12:27:34.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient: init 2024-09-18T12:27:34.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:34.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 0x7f466809c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient(hunting): picked mon.noname-b con 0x7f466809cad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f466809cea0 0x7f466809dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient(hunting): picked mon.noname-a con 0x7f466809cea0 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f466809d270 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466809b710 con 0x7f466809d270 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466809c6e0 con 0x7f466809cea0 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466809a650 con 0x7f466809cad0 2024-09-18T12:27:34.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f466df1d640 10 monclient(hunting): authenticate will time out at 890.302648s 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f4667fff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f466809d270 legacy=0x7f466809e010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f4666ffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f466809cea0 msgr2=0x7f466809dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f4666ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f466809cea0 0x7f466809dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.733+0000 7f46677fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 0x7f466809c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 0x7f466809c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:51612/0 (socket says 172.21.15.144:51612) 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 1 -- 172.21.15.144:0/3197437934 learned_addr learned my addr 172.21.15.144:0/3197437934 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): get_auth_request con 0x7f466809cad0 auth_method 0 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:34.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient(hunting): handle_auth_done global_id 44198 payload 274 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 1 -- 172.21.15.144:0/3197437934 >> v2:172.21.15.116:3300/0 conn(0x7f466809cea0 msgr2=0x7f466809dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 1 --2- 172.21.15.144:0/3197437934 >> v2:172.21.15.116:3300/0 conn(0x7f466809cea0 0x7f466809dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 1 -- 172.21.15.144:0/3197437934 >> v1:172.21.15.116:6789/0 conn(0x7f466809d270 legacy=0x7f466809e010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:34.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.734+0000 7f46677fe640 1 -- 172.21.15.144:0/3197437934 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466809f660 con 0x7f466809cad0 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46677fe640 1 -- 172.21.15.144:0/3197437934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 msgr2=0x7f466809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46677fe640 1 --2- 172.21.15.144:0/3197437934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 0x7f466809c1a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4658009910 tx=0x7f46580091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1810f5b8a2a50cd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 1 -- 172.21.15.144:0/3197437934 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658032440 con 0x7f466809cad0 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: dump: 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:34.747 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 5 monclient: authenticate success, global_id 44198 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 1 -- 172.21.15.144:0/3197437934 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46580325a0 con 0x7f466809cad0 2024-09-18T12:27:34.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 4 set_mon_vals no callback set 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 1 -- 172.21.15.144:0/3197437934 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658032980 con 0x7f466809cad0 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f46667fc640 10 monclient: dump: 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.749 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 10 monclient: shutdown 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 1 -- 172.21.15.144:0/3197437934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 msgr2=0x7f466809c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 1 --2- 172.21.15.144:0/3197437934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 0x7f466809c1a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4658009910 tx=0x7f46580091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 1 -- 172.21.15.144:0/3197437934 shutdown_connections 2024-09-18T12:27:34.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 1 --2- 172.21.15.144:0/3197437934 >> v2:172.21.15.116:3300/0 conn(0x7f466809cea0 0x7f466809dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 1 --2- 172.21.15.144:0/3197437934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809cad0 0x7f466809c1a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.737+0000 7f466df1d640 1 -- 172.21.15.144:0/3197437934 >> 172.21.15.144:0/3197437934 conn(0x7f4668070e90 msgr2=0x7f46680a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.738+0000 7f466df1d640 1 -- 172.21.15.144:0/3197437934 shutdown_connections 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.738+0000 7f466df1d640 1 -- 172.21.15.144:0/3197437934 wait complete. 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.738+0000 7f466df1d640 10 monclient: build_initial_monmap 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.738+0000 7f466df1d640 10 monclient: monmap: 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:34.751 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.738+0000 7f466df1d640 1 Processor -- start 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 1 -- start start 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient: init 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f466809cad0 0x7f466813f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): picked mon.noname-a con 0x7f466809cad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 0x7f4668131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:34.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): picked mon.noname-b con 0x7f466809cea0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809d270 0x7f46681322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): picked mon.noname-c con 0x7f466809d270 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466811eeb0 con 0x7f466809cad0 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466806deb0 con 0x7f466809cea0 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466811d3f0 con 0x7f466809d270 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f466df1d640 10 monclient(hunting): authenticate will time out at 890.308479s 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f46677fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f466809cad0 msgr2=0x7f466813f030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:34.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f46677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f466809cad0 0x7f466813f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 0x7f4668131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 0x7f4668131d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57224/0 (socket says 172.21.15.144:57224) 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 -- 172.21.15.144:0/111401619 learned_addr learned my addr 172.21.15.144:0/111401619 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): get_auth_request con 0x7f466809cea0 auth_method 0 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4667fff640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809d270 0x7f46681322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4667fff640 10 monclient(hunting): get_auth_request con 0x7f466809d270 auth_method 0 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:34.754 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4667fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient(hunting): handle_auth_done global_id 4726 payload 824 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 -- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809d270 msgr2=0x7f46681322d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809d270 0x7f46681322d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 -- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f466809cad0 msgr2=0x7f466813f030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f466809cad0 0x7f466813f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 monclient: found mon.noname-b 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.755 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4666ffd640 1 -- 172.21.15.144:0/111401619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658009450 con 0x7f466809cea0 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.739+0000 7f4667fff640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809d270 0x7f46681322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f4666ffd640 1 -- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 msgr2=0x7f4668131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f4666ffd640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 0x7f4668131d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f465c00b630 tx=0x7f465c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c015070 con 0x7f466809cea0 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: dump: 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:34.756 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c003d90 con 0x7f466809cea0 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f466df1d640 5 monclient: authenticate success, global_id 4726 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f466df1d640 10 monclient: _renew_subs 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f466df1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466806eba0 con 0x7f466809cea0 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c006380 con 0x7f466809cea0 2024-09-18T12:27:34.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f464f7fe640 10 monclient: dump: 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f466df1d640 10 monclient: _renew_subs 2024-09-18T12:27:34.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f466df1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466811f750 con 0x7f466809cea0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.742+0000 7f4665ffb640 4 set_mon_vals no callback set 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f465c010440 con 0x7f466809cea0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f466df1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f466df1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f466df1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46680008d0 con 0x7f466809cea0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f464f7fe640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4654038440 0x7f465403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f465c04b050 con 0x7f466809cea0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f46677fe640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4654038440 0x7f465403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.743+0000 7f46677fe640 10 monclient: get_auth_request con 0x7f4654038440 auth_method 0 2024-09-18T12:27:34.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.745+0000 7f46677fe640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4654038440 0x7f465403a8f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4658034000 tx=0x7f4658032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:34.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.747+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f465c07d030 con 0x7f466809cea0 2024-09-18T12:27:34.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.747+0000 7f464f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:34.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.747+0000 7f464f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:34.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.901+0000 7f466df1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.901+0000 7f466df1d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.901+0000 7f466df1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.901+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f466809b430 con 0x7f466809cea0 2024-09-18T12:27:34.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.901+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v44) v1 ==== 159+0+0 (secure 0 0 0) 0x7f465c006770 con 0x7f466809cea0 2024-09-18T12:27:34.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.901+0000 7f464f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.901+0000 7f464f7fe640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-18T12:27:34.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.994+0000 7f466df1d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.994+0000 7f466df1d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.994+0000 7f466df1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:34.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.994+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46680a2060 con 0x7f466809cea0 2024-09-18T12:27:34.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.994+0000 7f464f7fe640 1 -- 172.21.15.144:0/111401619 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v44) v1 ==== 159+0+0 (secure 0 0 0) 0x7f465c0290c0 con 0x7f466809cea0 2024-09-18T12:27:34.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.994+0000 7f464f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:34.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.994+0000 7f464f7fe640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-18T12:27:34.996 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'fooo' does not exist 2024-09-18T12:27:34.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4654038440 msgr2=0x7f465403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4654038440 0x7f465403a8f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4658034000 tx=0x7f4658032000 comp rx=0 tx=0).stop 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 10 monclient: shutdown 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 msgr2=0x7f4668131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 0x7f4668131d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f465c00b630 tx=0x7f465c0091a0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 shutdown_connections 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4654038440 0x7f465403a8f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f466809d270 0x7f46681322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f466809cea0 0x7f4668131d90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 --2- 172.21.15.144:0/111401619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f466809cad0 0x7f466813f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:34.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 >> 172.21.15.144:0/111401619 conn(0x7f4668070e90 msgr2=0x7f4668098a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:34.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 shutdown_connections 2024-09-18T12:27:34.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:34.996+0000 7f466df1d640 1 -- 172.21.15.144:0/111401619 wait complete. 2024-09-18T12:27:35.006 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-18T12:27:35.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.079+0000 7f57a519d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.079+0000 7f57a519d640 10 monclient: build_initial_monmap 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.079+0000 7f57a519d640 10 monclient: monmap: 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:35.080613+0000 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:35.080613+0000 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:35.081 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:35.082 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:35.082 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 1 Processor -- start 2024-09-18T12:27:35.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 1 -- start start 2024-09-18T12:27:35.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 10 monclient: init 2024-09-18T12:27:35.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:35.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a0131c90 0x7f57a0132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 10 monclient(hunting): picked mon.noname-b con 0x7f57a0131c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57a009d210 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.080+0000 7f57a519d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57a00a0ab0 0x7f57a01325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 10 monclient(hunting): picked mon.noname-c con 0x7f57a00a0ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a00a1800 con 0x7f57a009d210 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a009fea0 con 0x7f57a00a0ab0 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00a1520 con 0x7f57a0131c90 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f57a519d640 10 monclient(hunting): authenticate will time out at 890.650052s 2024-09-18T12:27:35.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57a00a0ab0 0x7f57a01325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f5797fff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f57a009d210 legacy=0x7f57a0068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57a00a0ab0 0x7f57a01325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57226/0 (socket says 172.21.15.144:57226) 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 -- 172.21.15.144:0/2976328130 learned_addr learned my addr 172.21.15.144:0/2976328130 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): get_auth_request con 0x7f57a00a0ab0 auth_method 0 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579ed9d640 1 --2- 172.21.15.144:0/2976328130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a0131c90 0x7f57a0132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579ed9d640 10 monclient(hunting): get_auth_request con 0x7f57a0131c90 auth_method 0 2024-09-18T12:27:35.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient(hunting): handle_auth_done global_id 4729 payload 274 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 -- 172.21.15.144:0/2976328130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a0131c90 msgr2=0x7f57a0132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 --2- 172.21.15.144:0/2976328130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a0131c90 0x7f57a0132060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 -- 172.21.15.144:0/2976328130 >> v1:172.21.15.116:6789/0 conn(0x7f57a009d210 legacy=0x7f57a0068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient: _finish_hunting 0 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 monclient: found mon.noname-c 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.081+0000 7f579f59e640 1 -- 172.21.15.144:0/2976328130 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a0069510 con 0x7f57a00a0ab0 2024-09-18T12:27:35.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579f59e640 1 -- 172.21.15.144:0/2976328130 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a00a0ab0 msgr2=0x7f57a01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:35.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579f59e640 1 --2- 172.21.15.144:0/2976328130 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a00a0ab0 0x7f57a01325a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5798009f20 tx=0x7f5798009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c981b1721d4e0023 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:35.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 1 -- 172.21.15.144:0/2976328130 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798007610 con 0x7f57a00a0ab0 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: dump: 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.092 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: _finish_auth 0 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 1 -- 172.21.15.144:0/2976328130 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579802d050 con 0x7f57a00a0ab0 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 1 -- 172.21.15.144:0/2976328130 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57980079a0 con 0x7f57a00a0ab0 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f579e59c640 10 monclient: dump: 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.093 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 5 monclient: authenticate success, global_id 4729 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 4 set_mon_vals no callback set 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 10 monclient: shutdown 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 1 -- 172.21.15.144:0/2976328130 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a00a0ab0 msgr2=0x7f57a01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 1 --2- 172.21.15.144:0/2976328130 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a00a0ab0 0x7f57a01325a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5798009f20 tx=0x7f5798009840 comp rx=0 tx=0).stop 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 1 -- 172.21.15.144:0/2976328130 shutdown_connections 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 1 --2- 172.21.15.144:0/2976328130 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a00a0ab0 0x7f57a01325a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 1 --2- 172.21.15.144:0/2976328130 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a0131c90 0x7f57a0132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.084+0000 7f57a519d640 1 -- 172.21.15.144:0/2976328130 >> 172.21.15.144:0/2976328130 conn(0x7f57a0071370 msgr2=0x7f57a00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.085+0000 7f57a519d640 1 -- 172.21.15.144:0/2976328130 shutdown_connections 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.085+0000 7f57a519d640 1 -- 172.21.15.144:0/2976328130 wait complete. 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.085+0000 7f57a519d640 10 monclient: build_initial_monmap 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.085+0000 7f57a519d640 10 monclient: monmap: 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:35.095 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.085+0000 7f57a519d640 1 Processor -- start 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.085+0000 7f57a519d640 1 -- start start 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.085+0000 7f57a519d640 10 monclient: init 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 0x7f57a0141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): picked mon.noname-c con 0x7f57a00a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:35.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 0x7f57a0141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): picked mon.noname-a con 0x7f57a0131c90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): picked mon.noname-b con 0x7f57a0141bd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00a0fb0 con 0x7f57a0131c90 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a009ef10 con 0x7f57a0141bd0 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a011ff80 con 0x7f57a00a0ab0 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f57a519d640 10 monclient(hunting): authenticate will time out at 890.655318s 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f5797fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 msgr2=0x7f57a0141690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:35.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f5797fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 0x7f57a0141690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57242/0 (socket says 172.21.15.144:57242) 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 learned_addr learned my addr 172.21.15.144:0/676741243 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): get_auth_request con 0x7f57a0141bd0 auth_method 0 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579ed9d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 0x7f57a0141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient(hunting): handle_auth_done global_id 4732 payload 824 2024-09-18T12:27:35.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 msgr2=0x7f57a0141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 0x7f57a0141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 msgr2=0x7f57a0141690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 0x7f57a0141690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient: _finish_hunting 0 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 monclient: found mon.noname-b 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5798009ae0 con 0x7f57a0141bd0 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 msgr2=0x7f57a011e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 msgr2=0x7f57a011e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f5798009ab0 tx=0x7f5798036000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:35.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.086+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:35.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient: get_auth_request con 0x7f57a0141bd0 auth_method 0 2024-09-18T12:27:35.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 0 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T12:27:35.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:35.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 msgr2=0x7f57a011e480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57a0141bd0 0x7f57a011e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 0x7f57800054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f57a0131c90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 0x7f5780007f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f57a00a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5780008930 0x7f578000adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f5780008930 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578000b8b0 con 0x7f57a0131c90 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578000b9f0 con 0x7f5780008930 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578000bbb0 con 0x7f57a00a0ab0 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5796ffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579ed9d640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 msgr2=0x7f57800054e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5780008930 0x7f578000adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579ed9d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 0x7f57800054e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): get_auth_request con 0x7f5780008930 auth_method 0 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:35.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5797fff640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 0x7f5780007f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5797fff640 10 monclient(hunting): get_auth_request con 0x7f57a00a0ab0 auth_method 0 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5797fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f5797fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient(hunting): handle_auth_done global_id 4732 payload 293 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 msgr2=0x7f5780007f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 0x7f5780007f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 msgr2=0x7f57800054e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 0x7f57800054e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient: _finish_hunting 0 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 monclient: found mon.noname-b 2024-09-18T12:27:35.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:35.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.287+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578000be30 con 0x7f5780008930 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.289+0000 7f579f59e640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5780008930 msgr2=0x7f578000adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.289+0000 7f579f59e640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5780008930 0x7f578000adb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5798009ab0 tx=0x7f579803b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798041070 con 0x7f5780008930 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: dump: 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.296 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5798045430 con 0x7f5780008930 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57980457f0 con 0x7f5780008930 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f57a519d640 5 monclient: authenticate success, global_id 4732 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f57a519d640 10 monclient: _renew_subs 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f57a519d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a0121a40 con 0x7f5780008930 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 10 monclient: dump: 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f579d59a640 4 set_mon_vals no callback set 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f57a519d640 10 monclient: _renew_subs 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f57a519d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a0120b50 con 0x7f5780008930 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.290+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5798038040 con 0x7f5780008930 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f5796ffd640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f57a0141bd0 0x7f57800429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f57a519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f57a519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f57a519d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f579806b7e0 con 0x7f5780008930 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a0121ec0 con 0x7f5780008930 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f579ed9d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f57a0141bd0 0x7f57800429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.291+0000 7f579ed9d640 10 monclient: get_auth_request con 0x7f57a0141bd0 auth_method 0 2024-09-18T12:27:35.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.293+0000 7f579ed9d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f57a0141bd0 0x7f57800429d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5790006e70 tx=0x7f5790008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:35.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.295+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5798036030 con 0x7f5780008930 2024-09-18T12:27:35.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.295+0000 7f5796ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:35.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.295+0000 7f5796ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:35.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.452+0000 7f57a519d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.452+0000 7f57a519d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.452+0000 7f57a519d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.452+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57a0141fa0 con 0x7f5780008930 2024-09-18T12:27:35.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.452+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v44) v1 ==== 159+0+0 (secure 0 0 0) 0x7f5798036030 con 0x7f5780008930 2024-09-18T12:27:35.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.452+0000 7f5796ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.452+0000 7f5796ffd640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-18T12:27:35.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.543+0000 7f57a519d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.543+0000 7f57a519d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.543+0000 7f57a519d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.543+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57a0141fa0 con 0x7f5780008930 2024-09-18T12:27:35.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.543+0000 7f5796ffd640 1 -- 172.21.15.144:0/676741243 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v44) v1 ==== 159+0+0 (secure 0 0 0) 0x7f5798036020 con 0x7f5780008930 2024-09-18T12:27:35.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.543+0000 7f5796ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.543+0000 7f5796ffd640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-18T12:27:35.546 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'fooo' does not exist 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.545+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f57a0141bd0 msgr2=0x7f57800429d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.545+0000 7f57a519d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f57a0141bd0 0x7f57800429d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5790006e70 tx=0x7f5790008040 comp rx=0 tx=0).stop 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.545+0000 7f57a519d640 10 monclient: shutdown 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.545+0000 7f57a519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.545+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5780008930 msgr2=0x7f578000adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.545+0000 7f57a519d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5780008930 0x7f578000adb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5798009ab0 tx=0x7f579803b000 comp rx=0 tx=0).stop 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 shutdown_connections 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f57a0141bd0 0x7f57800429d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5780008930 0x7f578000adb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f57a00a0ab0 0x7f5780007f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 --2- 172.21.15.144:0/676741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a0131c90 0x7f57800054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 >> 172.21.15.144:0/676741243 conn(0x7f57a0071370 msgr2=0x7f57a00a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:35.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 shutdown_connections 2024-09-18T12:27:35.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.546+0000 7f57a519d640 1 -- 172.21.15.144:0/676741243 wait complete. 2024-09-18T12:27:35.555 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-18T12:27:35.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.629+0000 7f0fcbd00640 10 monclient: get_monmap_and_config 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.629+0000 7f0fcbd00640 10 monclient: build_initial_monmap 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.629+0000 7f0fcbd00640 10 monclient: monmap: 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:35.630310+0000 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:35.630310+0000 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:35.631 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:35.632 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:35.632 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:35.632 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.629+0000 7f0fcbd00640 1 Processor -- start 2024-09-18T12:27:35.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 1 -- start start 2024-09-18T12:27:35.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient: init 2024-09-18T12:27:35.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:35.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0fc4096a30 0x7f0fc409d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc4096a30 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fc4096e00 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc4137200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc4134e20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc409c400 con 0x7f0fc4096e00 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc409b410 con 0x7f0fc4096a30 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc409b970 con 0x7f0fc4134e20 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:35.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fcbd00640 10 monclient(hunting): authenticate will time out at 891.199612s 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fc929b640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f0fc4096e00 legacy=0x7f0fc4097200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fc9a9c640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f0fc4096a30 msgr2=0x7f0fc409d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fc9a9c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0fc4096a30 0x7f0fc409d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fca29d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc4137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fca29d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc4137200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:51660/0 (socket says 172.21.15.144:51660) 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fca29d640 1 -- 172.21.15.144:0/2774703202 learned_addr learned my addr 172.21.15.144:0/2774703202 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fca29d640 10 monclient(hunting): get_auth_request con 0x7f0fc4134e20 auth_method 0 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fca29d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fca29d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.630+0000 7f0fca29d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:35.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 10 monclient(hunting): handle_auth_done global_id 44207 payload 274 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 1 -- 172.21.15.144:0/2774703202 >> v2:172.21.15.116:3300/0 conn(0x7f0fc4096a30 msgr2=0x7f0fc409d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 1 --2- 172.21.15.144:0/2774703202 >> v2:172.21.15.116:3300/0 conn(0x7f0fc4096a30 0x7f0fc409d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 1 -- 172.21.15.144:0/2774703202 >> v1:172.21.15.116:6789/0 conn(0x7f0fc4096e00 legacy=0x7f0fc4097200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 10 monclient: _finish_hunting 0 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 1 monclient: found mon.noname-b 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:35.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.631+0000 7f0fca29d640 1 -- 172.21.15.144:0/2774703202 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc409c120 con 0x7f0fc4134e20 2024-09-18T12:27:35.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fca29d640 1 -- 172.21.15.144:0/2774703202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 msgr2=0x7f0fc4137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fca29d640 1 --2- 172.21.15.144:0/2774703202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc4137200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009910 tx=0x7f0fbc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=996372bc3a7808df server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 1 -- 172.21.15.144:0/2774703202 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc032440 con 0x7f0fc4134e20 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: dump: 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.643 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.644 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.644 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: _finish_auth 0 2024-09-18T12:27:35.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:35.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 1 -- 172.21.15.144:0/2774703202 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc0325a0 con 0x7f0fc4134e20 2024-09-18T12:27:35.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fcbd00640 5 monclient: authenticate success, global_id 44207 2024-09-18T12:27:35.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fcbd00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fcbd00640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 1 -- 172.21.15.144:0/2774703202 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc032940 con 0x7f0fc4134e20 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fcbd00640 4 set_mon_vals no callback set 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.634+0000 7f0fc8a9a640 10 monclient: dump: 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.645 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 10 monclient: shutdown 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 -- 172.21.15.144:0/2774703202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 msgr2=0x7f0fc4137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/2774703202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc4137200 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009910 tx=0x7f0fbc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 -- 172.21.15.144:0/2774703202 shutdown_connections 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/2774703202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc4137200 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/2774703202 >> v2:172.21.15.116:3300/0 conn(0x7f0fc4096a30 0x7f0fc409d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 -- 172.21.15.144:0/2774703202 >> 172.21.15.144:0/2774703202 conn(0x7f0fc40711a0 msgr2=0x7f0fc40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 -- 172.21.15.144:0/2774703202 shutdown_connections 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 1 -- 172.21.15.144:0/2774703202 wait complete. 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 10 monclient: build_initial_monmap 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.635+0000 7f0fcbd00640 10 monclient: monmap: 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:35.647 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 Processor -- start 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 -- start start 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient: init 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc4096a30 0x7f0fc411c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc4096a30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:35.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 0x7f0fc411c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc4096e00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc411cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc4134e20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc411e190 con 0x7f0fc4096a30 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc409bba0 con 0x7f0fc4096e00 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc40a0b70 con 0x7f0fc4134e20 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fcbd00640 10 monclient(hunting): authenticate will time out at 891.205541s 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc9a9c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc4096a30 msgr2=0x7f0fc411c290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc9a9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc4096a30 0x7f0fc411c290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:35.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 0x7f0fc411c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 0x7f0fc411c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57274/0 (socket says 172.21.15.144:57274) 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 -- 172.21.15.144:0/3412298331 learned_addr learned my addr 172.21.15.144:0/3412298331 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): get_auth_request con 0x7f0fc4096e00 auth_method 0 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:35.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fca29d640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc411cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fca29d640 10 monclient(hunting): get_auth_request con 0x7f0fc4134e20 auth_method 0 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fca29d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fca29d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fca29d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient(hunting): handle_auth_done global_id 4735 payload 824 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 -- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 msgr2=0x7f0fc411cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc411cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 -- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc4096a30 msgr2=0x7f0fc411c290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc4096a30 0x7f0fc411c290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient: _finish_hunting 0 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 monclient: found mon.noname-b 2024-09-18T12:27:35.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fc929b640 1 -- 172.21.15.144:0/3412298331 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc009450 con 0x7f0fc4096e00 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.636+0000 7f0fca29d640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc411cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fc929b640 1 -- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 msgr2=0x7f0fc411c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fc929b640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 0x7f0fc411c7d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400b600 tx=0x7f0fb40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4003d90 con 0x7f0fc4096e00 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: dump: 2024-09-18T12:27:35.652 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:35.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fcbd00640 5 monclient: authenticate success, global_id 4735 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fcbd00640 10 monclient: _renew_subs 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb4006150 con 0x7f0fc4096e00 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fcbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc411f6b0 con 0x7f0fc4096e00 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4006510 con 0x7f0fc4096e00 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fb9ffb640 10 monclient: dump: 2024-09-18T12:27:35.654 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fcbd00640 10 monclient: _renew_subs 2024-09-18T12:27:35.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fbbfff640 4 set_mon_vals no callback set 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fcbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.639+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc411e7c0 con 0x7f0fc4096e00 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0fb40103c0 con 0x7f0fc4096e00 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fb9ffb640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0fa8038440 0x7f0fa803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f0fb404ae00 con 0x7f0fc4096e00 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fcbd00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fc9a9c640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0fa8038440 0x7f0fa803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fcbd00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fcbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc411f8f0 con 0x7f0fc4096e00 2024-09-18T12:27:35.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.640+0000 7f0fc9a9c640 10 monclient: get_auth_request con 0x7f0fa8038440 auth_method 0 2024-09-18T12:27:35.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.643+0000 7f0fc9a9c640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0fa8038440 0x7f0fa803a8f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0003c70 tx=0x7f0fc0009310 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:35.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.644+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0fb4017070 con 0x7f0fc4096e00 2024-09-18T12:27:35.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.644+0000 7f0fb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:35.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.644+0000 7f0fb9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:35.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.803+0000 7f0fcbd00640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.803+0000 7f0fcbd00640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.803+0000 7f0fcbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.803+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0fc40008d0 con 0x7f0fc4096e00 2024-09-18T12:27:35.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.804+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v44) v1 ==== 162+0+0 (secure 0 0 0) 0x7f0fb400e8c0 con 0x7f0fc4096e00 2024-09-18T12:27:35.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.804+0000 7f0fb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.804+0000 7f0fb9ffb640 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-18T12:27:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.899+0000 7f0fcbd00640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.899+0000 7f0fcbd00640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.899+0000 7f0fcbd00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.899+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0fc411f3a0 con 0x7f0fc4096e00 2024-09-18T12:27:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.899+0000 7f0fb9ffb640 1 -- 172.21.15.144:0/3412298331 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v44) v1 ==== 162+0+0 (secure 0 0 0) 0x7f0fb400e2c0 con 0x7f0fc4096e00 2024-09-18T12:27:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.899+0000 7f0fb9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T12:27:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.899+0000 7f0fb9ffb640 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-18T12:27:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'fuggg' does not exist 2024-09-18T12:27:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0fa8038440 msgr2=0x7f0fa803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0fa8038440 0x7f0fa803a8f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0003c70 tx=0x7f0fc0009310 comp rx=0 tx=0).stop 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 10 monclient: shutdown 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 msgr2=0x7f0fc411c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 0x7f0fc411c7d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400b600 tx=0x7f0fb40091a0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 shutdown_connections 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0fa8038440 0x7f0fa803a8f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0fc4134e20 0x7f0fc411cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0fc4096e00 0x7f0fc411c7d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 --2- 172.21.15.144:0/3412298331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc4096a30 0x7f0fc411c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 >> 172.21.15.144:0/3412298331 conn(0x7f0fc40711a0 msgr2=0x7f0fc406e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 shutdown_connections 2024-09-18T12:27:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:35.901+0000 7f0fcbd00640 1 -- 172.21.15.144:0/3412298331 wait complete. 2024-09-18T12:27:35.911 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo OK 2024-09-18T12:27:35.911 INFO:tasks.workunit.client.0.smithi144.stdout:OK 2024-09-18T12:27:35.912 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:27:35.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:27:35.942 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-18T12:27:35.944 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-18T12:27:35.944 DEBUG:teuthology.orchestra.run.smithi144:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-18T12:27:36.003 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush dump 2024-09-18T12:27:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.074+0000 7fbd28ad4640 10 monclient: get_monmap_and_config 2024-09-18T12:27:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.074+0000 7fbd28ad4640 10 monclient: build_initial_monmap 2024-09-18T12:27:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.074+0000 7fbd28ad4640 10 monclient: monmap: 2024-09-18T12:27:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:36.075031+0000 2024-09-18T12:27:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:36.075031+0000 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:36.076 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.074+0000 7fbd28ad4640 1 Processor -- start 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.074+0000 7fbd28ad4640 1 -- start start 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.074+0000 7fbd28ad4640 10 monclient: init 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd24131b40 0x7fbd2409ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fbd24131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd2409f240 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbd24136e70 0x7fbd24139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient(hunting): picked mon.noname-a con 0x7fbd24136e70 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2406a230 con 0x7fbd2409f240 2024-09-18T12:27:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd240a0300 con 0x7fbd24136e70 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240a00c0 con 0x7fbd24131b40 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd28ad4640 10 monclient(hunting): authenticate will time out at 891.644320s 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd22d9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fbd24136e70 msgr2=0x7fbd24139250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd19d9b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fbd2409f240 0x7fbd24131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33000/0 (socket says 172.21.15.144:33000) 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd19d9b640 1 -- 172.21.15.144:0/1169282499 learned_addr learned my addr 172.21.15.144:0/1169282499 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd22d9d640 1 --2- 172.21.15.144:0/1169282499 >> v2:172.21.15.116:3300/0 conn(0x7fbd24136e70 0x7fbd24139250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd2259c640 1 --2- 172.21.15.144:0/1169282499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd24131b40 0x7fbd2409ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd2259c640 10 monclient(hunting): get_auth_request con 0x7fbd24131b40 auth_method 0 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd2259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.075+0000 7fbd2259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 10 monclient(hunting): handle_auth_done global_id 44213 payload 274 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 1 -- 172.21.15.144:0/1169282499 >> v2:172.21.15.116:3300/0 conn(0x7fbd24136e70 msgr2=0x7fbd24139250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 1 --2- 172.21.15.144:0/1169282499 >> v2:172.21.15.116:3300/0 conn(0x7fbd24136e70 0x7fbd24139250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 1 -- 172.21.15.144:0/1169282499 >> v1:172.21.15.144:6789/0 conn(0x7fbd2409f240 legacy=0x7fbd24131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 1 monclient: found mon.noname-b 2024-09-18T12:27:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.076+0000 7fbd2259c640 1 -- 172.21.15.144:0/1169282499 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2409bee0 con 0x7fbd24131b40 2024-09-18T12:27:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd2259c640 1 -- 172.21.15.144:0/1169282499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd24131b40 msgr2=0x7fbd2409ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd2259c640 1 --2- 172.21.15.144:0/1169282499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd24131b40 0x7fbd2409ed00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009850 tx=0x7fbd140090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bddbf3d56f5d8c27 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 1 -- 172.21.15.144:0/1169282499 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14032440 con 0x7fbd24131b40 2024-09-18T12:27:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: dump: 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:36.089 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: _finish_auth 0 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 5 monclient: authenticate success, global_id 44213 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 1 -- 172.21.15.144:0/1169282499 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd140325a0 con 0x7fbd24131b40 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 4 set_mon_vals no callback set 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 1 -- 172.21.15.144:0/1169282499 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14032980 con 0x7fbd24131b40 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd21d9b640 10 monclient: dump: 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:36.091 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 10 monclient: shutdown 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 -- 172.21.15.144:0/1169282499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd24131b40 msgr2=0x7fbd2409ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/1169282499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd24131b40 0x7fbd2409ed00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009850 tx=0x7fbd140090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 -- 172.21.15.144:0/1169282499 shutdown_connections 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/1169282499 >> v2:172.21.15.116:3300/0 conn(0x7fbd24136e70 0x7fbd24139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/1169282499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd24131b40 0x7fbd2409ed00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 -- 172.21.15.144:0/1169282499 >> 172.21.15.144:0/1169282499 conn(0x7fbd240a5100 msgr2=0x7fbd240a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 -- 172.21.15.144:0/1169282499 shutdown_connections 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.079+0000 7fbd28ad4640 1 -- 172.21.15.144:0/1169282499 wait complete. 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.080+0000 7fbd28ad4640 10 monclient: build_initial_monmap 2024-09-18T12:27:36.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.080+0000 7fbd28ad4640 10 monclient: monmap: 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:36.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.080+0000 7fbd28ad4640 1 Processor -- start 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.080+0000 7fbd28ad4640 1 -- start start 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.080+0000 7fbd28ad4640 10 monclient: init 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2409f240 0x7fbd2412eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): picked mon.noname-c con 0x7fbd2409f240 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24131b40 0x7fbd24097950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): picked mon.noname-a con 0x7fbd24131b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 0x7fbd24097e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fbd24136e70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:36.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24099ed0 con 0x7fbd24131b40 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2409ddc0 con 0x7fbd24136e70 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240a0a30 con 0x7fbd2409f240 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd28ad4640 10 monclient(hunting): authenticate will time out at 891.650302s 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd19d9b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24131b40 msgr2=0x7fbd24097950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 0x7fbd24097e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd19d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24131b40 0x7fbd24097950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 0x7fbd24097e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57284/0 (socket says 172.21.15.144:57284) 2024-09-18T12:27:36.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 -- 172.21.15.144:0/3373781497 learned_addr learned my addr 172.21.15.144:0/3373781497 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): get_auth_request con 0x7fbd24136e70 auth_method 0 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd2259c640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2409f240 0x7fbd2412eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd2259c640 10 monclient(hunting): get_auth_request con 0x7fbd2409f240 auth_method 0 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:36.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd2259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd2259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient(hunting): handle_auth_done global_id 4738 payload 824 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 -- 172.21.15.144:0/3373781497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2409f240 msgr2=0x7fbd2412eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2409f240 0x7fbd2412eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 -- 172.21.15.144:0/3373781497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24131b40 msgr2=0x7fbd24097950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24131b40 0x7fbd24097950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 monclient: found mon.noname-b 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.081+0000 7fbd22d9d640 1 -- 172.21.15.144:0/3373781497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14009390 con 0x7fbd24136e70 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.083+0000 7fbd22d9d640 1 -- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 msgr2=0x7fbd24097e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:36.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.083+0000 7fbd22d9d640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 0x7fbd24097e90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00bb40 tx=0x7fbd1c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.083+0000 7fbd22d9d640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 0x7fbd24097e90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00bb40 tx=0x7fbd1c0097d0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fbd1c00cbf0 mon_map magic: 0 v1 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c00cbf0 con 0x7fbd24136e70 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: dump: 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:36.098 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: _finish_auth 0 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c00cd70 con 0x7fbd24136e70 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd28ad4640 5 monclient: authenticate success, global_id 4738 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd28ad4640 10 monclient: _renew_subs 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd28ad4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd240999e0 con 0x7fbd24136e70 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd20d99640 4 set_mon_vals no callback set 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd28ad4640 10 monclient: _renew_subs 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd28ad4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:36.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24142020 con 0x7fbd24136e70 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c006150 con 0x7fbd24136e70 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 10 monclient: dump: 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:36.100 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.084+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbd1c012020 con 0x7fbd24136e70 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.085+0000 7fbd28ad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.085+0000 7fbd28ad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.085+0000 7fbd28ad4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.085+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd240636e0 con 0x7fbd24136e70 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.085+0000 7fbd1affd640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbd04038490 0x7fbd0403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.085+0000 7fbd2259c640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbd04038490 0x7fbd0403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.085+0000 7fbd2259c640 10 monclient: get_auth_request con 0x7fbd04038490 auth_method 0 2024-09-18T12:27:36.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.087+0000 7fbd2259c640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbd04038490 0x7fbd0403a940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd14034000 tx=0x7fbd14032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:36.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.089+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fbd1c04b590 con 0x7fbd24136e70 2024-09-18T12:27:36.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.706+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbd1c01dac0 con 0x7fbd24136e70 2024-09-18T12:27:36.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.706+0000 7fbd1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:36.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.706+0000 7fbd1affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:36.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.864+0000 7fbd28ad4640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-18T12:27:36.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.864+0000 7fbd28ad4640 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-18T12:27:36.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.864+0000 7fbd28ad4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:36.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.864+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7fbd2409a0a0 con 0x7fbd24136e70 2024-09-18T12:27:36.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.864+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v45) v1 ==== 62+0+7238 (secure 0 0 0) 0x7fbd1c01d300 con 0x7fbd24136e70 2024-09-18T12:27:36.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.864+0000 7fbd1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-18T12:27:36.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.864+0000 7fbd1affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.958+0000 7fbd28ad4640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.958+0000 7fbd28ad4640 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.958+0000 7fbd28ad4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.958+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7fbd24072640 con 0x7fbd24136e70 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.959+0000 7fbd1affd640 1 -- 172.21.15.144:0/3373781497 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v45) v1 ==== 62+0+7238 (secure 0 0 0) 0x7fbd1c01d4e0 con 0x7fbd24136e70 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.959+0000 7fbd1affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.959+0000 7fbd1affd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:36.961 INFO:tasks.workunit.client.0.smithi144.stdout:{ 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: "devices": [ 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 0, 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "osd.0", 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: "class": "ssd" 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 1, 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "osd.1", 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: "class": "ssd" 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.962 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 2, 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "osd.2", 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "class": "ssd" 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 3, 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "osd.3", 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "class": "ssd" 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 4, 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "osd.4", 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: "class": "ssd" 2024-09-18T12:27:36.963 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 5, 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "osd.5", 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: "class": "ssd" 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: "types": [ 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 0, 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "osd" 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.964 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 1, 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "host" 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 2, 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "chassis" 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 3, 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "rack" 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.965 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 4, 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "row" 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 5, 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "pdu" 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 6, 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "pod" 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 7, 2024-09-18T12:27:36.966 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "room" 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 8, 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "datacenter" 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 9, 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "zone" 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 10, 2024-09-18T12:27:36.967 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "region" 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 11, 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "root" 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "buckets": [ 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -1, 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "default", 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 11, 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "type_name": "root", 2024-09-18T12:27:36.968 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 34560, 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "alg": "straw2", 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "hash": "rjenkins1", 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "items": [ 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -3, 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 0 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -5, 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.969 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 1 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -2, 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "default~ssd", 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 11, 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "type_name": "root", 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 34560, 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "alg": "straw2", 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "hash": "rjenkins1", 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: "items": [ 2024-09-18T12:27:36.970 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -4, 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 0 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -6, 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 1 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.971 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -3, 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "smithi116", 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 1, 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "type_name": "host", 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "alg": "straw2", 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "hash": "rjenkins1", 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "items": [ 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 0, 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.972 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 0 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 1, 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 1 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 2, 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 2 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.973 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -4, 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "smithi116~ssd", 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 1, 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "type_name": "host", 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "alg": "straw2", 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "hash": "rjenkins1", 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "items": [ 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 0, 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 0 2024-09-18T12:27:36.974 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 1, 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 1 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 2, 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 2 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.975 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -5, 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "smithi144", 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 1, 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "type_name": "host", 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "alg": "straw2", 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "hash": "rjenkins1", 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "items": [ 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 4, 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 0 2024-09-18T12:27:36.976 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 5, 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 1 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 3, 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 2 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.977 INFO:tasks.workunit.client.0.smithi144.stdout: "id": -6, 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "name": "smithi144~ssd", 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "type_id": 1, 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "type_name": "host", 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 17280, 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "alg": "straw2", 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "hash": "rjenkins1", 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "items": [ 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 4, 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 0 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.978 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 5, 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 1 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: "id": 3, 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: "weight": 5760, 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: "pos": 2 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:27:36.979 INFO:tasks.workunit.client.0.smithi144.stdout: "rules": [ 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_id": 0, 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_name": "replicated_rule", 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "type": 1, 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "steps": [ 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "take", 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "item": -1, 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "item_name": "default" 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.980 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "choose_firstn", 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 0, 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "type": "osd" 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "emit" 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_id": 1, 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_name": "erasure-code", 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "type": 3, 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "steps": [ 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.981 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "set_chooseleaf_tries", 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 5 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "set_choose_tries", 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 100 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "take", 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "item": -1, 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "item_name": "default" 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "choose_indep", 2024-09-18T12:27:36.982 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 0, 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: "type": "osd" 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "emit" 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: "tunables": { 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: "choose_local_tries": 0, 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: "choose_local_fallback_tries": 0, 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: "choose_total_tries": 50, 2024-09-18T12:27:36.983 INFO:tasks.workunit.client.0.smithi144.stdout: "chooseleaf_descend_once": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "chooseleaf_vary_r": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "chooseleaf_stable": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "straw_calc_version": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "allowed_bucket_algs": 54, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "profile": "jewel", 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "optimal_tunables": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "legacy_tunables": 0, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "minimum_required_version": "jewel", 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "require_feature_tunables": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "require_feature_tunables2": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "has_v2_rules": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "require_feature_tunables3": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "has_v3_rules": 0, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "has_v4_buckets": 1, 2024-09-18T12:27:36.984 INFO:tasks.workunit.client.0.smithi144.stdout: "require_feature_tunables5": 1, 2024-09-18T12:27:36.985 INFO:tasks.workunit.client.0.smithi144.stdout: "has_v5_rules": 0 2024-09-18T12:27:36.985 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:36.985 INFO:tasks.workunit.client.0.smithi144.stdout: "choose_args": {} 2024-09-18T12:27:36.985 INFO:tasks.workunit.client.0.smithi144.stdout:} 2024-09-18T12:27:36.985 INFO:tasks.workunit.client.0.smithi144.stdout: 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbd04038490 msgr2=0x7fbd0403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbd04038490 0x7fbd0403a940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd14034000 tx=0x7fbd14032000 comp rx=0 tx=0).stop 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 10 monclient: shutdown 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 msgr2=0x7fbd24097e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 0x7fbd24097e90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00bb40 tx=0x7fbd1c0097d0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 shutdown_connections 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbd04038490 0x7fbd0403a940 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbd24136e70 0x7fbd24097e90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd24131b40 0x7fbd24097950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 --2- 172.21.15.144:0/3373781497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbd2409f240 0x7fbd2412eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 >> 172.21.15.144:0/3373781497 conn(0x7fbd240a5100 msgr2=0x7fbd24130680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 shutdown_connections 2024-09-18T12:27:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:36.961+0000 7fbd28ad4640 1 -- 172.21.15.144:0/3373781497 wait complete. 2024-09-18T12:27:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule dump 2024-09-18T12:27:37.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.043+0000 7f588be34640 10 monclient: get_monmap_and_config 2024-09-18T12:27:37.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.043+0000 7f588be34640 10 monclient: build_initial_monmap 2024-09-18T12:27:37.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.043+0000 7f588be34640 10 monclient: monmap: 2024-09-18T12:27:37.044 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:37.044 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:37.044 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:37.044172+0000 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:37.044172+0000 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:37.045 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.043+0000 7f588be34640 1 Processor -- start 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 1 -- start start 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient: init 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5884131c90 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f588409d210 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient(hunting): picked mon.noname-b con 0x7f58840a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58840a1800 con 0x7f588409d210 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588409fea0 con 0x7f5884131c90 2024-09-18T12:27:37.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:37.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58840a1520 con 0x7f58840a0ab0 2024-09-18T12:27:37.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:37.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f588be34640 10 monclient(hunting): authenticate will time out at 892.613734s 2024-09-18T12:27:37.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f58893cf640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f588409d210 legacy=0x7f5884136fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:37.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f5889bd0640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f5884131c90 0x7f5884132060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33016/0 (socket says 172.21.15.144:33016) 2024-09-18T12:27:37.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.044+0000 7f5889bd0640 1 -- 172.21.15.144:0/780130926 learned_addr learned my addr 172.21.15.144:0/780130926 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 1 --2- 172.21.15.144:0/780130926 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): get_auth_request con 0x7f58840a0ab0 auth_method 0 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient(hunting): handle_auth_done global_id 44219 payload 274 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 1 -- 172.21.15.144:0/780130926 >> v1:172.21.15.144:6789/0 conn(0x7f5884131c90 legacy=0x7f5884132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 1 -- 172.21.15.144:0/780130926 >> v1:172.21.15.116:6789/0 conn(0x7f588409d210 legacy=0x7f5884136fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient: _finish_hunting 0 2024-09-18T12:27:37.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 1 monclient: found mon.noname-b 2024-09-18T12:27:37.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:37.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:37.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.045+0000 7f588a3d1640 1 -- 172.21.15.144:0/780130926 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5884132770 con 0x7f58840a0ab0 2024-09-18T12:27:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588a3d1640 1 -- 172.21.15.144:0/780130926 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 msgr2=0x7f5884069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588a3d1640 1 --2- 172.21.15.144:0/780130926 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884069e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5884072270 tx=0x7f587c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f3819489b4c658 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 1 -- 172.21.15.144:0/780130926 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c007840 con 0x7f58840a0ab0 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: dump: 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: _finish_auth 0 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 5 monclient: authenticate success, global_id 44219 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 1 -- 172.21.15.144:0/780130926 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587c02d050 con 0x7f58840a0ab0 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 4 set_mon_vals no callback set 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 1 -- 172.21.15.144:0/780130926 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c007c10 con 0x7f58840a0ab0 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f5888bce640 10 monclient: dump: 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 10 monclient: shutdown 2024-09-18T12:27:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 1 -- 172.21.15.144:0/780130926 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 msgr2=0x7f5884069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 1 --2- 172.21.15.144:0/780130926 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884069e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5884072270 tx=0x7f587c009780 comp rx=0 tx=0).stop 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 1 -- 172.21.15.144:0/780130926 shutdown_connections 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.048+0000 7f588be34640 1 --2- 172.21.15.144:0/780130926 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884069e10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 1 -- 172.21.15.144:0/780130926 >> 172.21.15.144:0/780130926 conn(0x7f5884071370 msgr2=0x7f58840a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 1 -- 172.21.15.144:0/780130926 shutdown_connections 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 1 -- 172.21.15.144:0/780130926 wait complete. 2024-09-18T12:27:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 10 monclient: build_initial_monmap 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 10 monclient: monmap: 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 1 Processor -- start 2024-09-18T12:27:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 1 -- start start 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.049+0000 7f588be34640 10 monclient: init 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): picked mon.noname-c con 0x7f58840a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 0x7f5884139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): picked mon.noname-b con 0x7f5884131c90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5884141380 0x7f5884134970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): picked mon.noname-a con 0x7f5884141380 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588409f330 con 0x7f5884141380 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58841382b0 con 0x7f5884131c90 2024-09-18T12:27:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58840a0fb0 con 0x7f58840a0ab0 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588be34640 10 monclient(hunting): authenticate will time out at 892.619441s 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 0x7f5884139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588a3d1640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5884141380 msgr2=0x7f5884134970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 0x7f5884139d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57300/0 (socket says 172.21.15.144:57300) 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 -- 172.21.15.144:0/393695992 learned_addr learned my addr 172.21.15.144:0/393695992 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): get_auth_request con 0x7f5884131c90 auth_method 0 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f588a3d1640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5884141380 0x7f5884134970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f5889bd0640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient(hunting): handle_auth_done global_id 4741 payload 824 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 -- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 msgr2=0x7f5884140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884140e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 -- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5884141380 msgr2=0x7f5884134970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5884141380 0x7f5884134970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient: _finish_hunting 0 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 monclient: found mon.noname-b 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f58893cf640 1 -- 172.21.15.144:0/393695992 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c009a20 con 0x7f5884131c90 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f5889bd0640 10 monclient: get_auth_request con 0x7f58840a0ab0 auth_method 0 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.050+0000 7f5889bd0640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884140e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f58893cf640 1 -- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 msgr2=0x7f5884139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f58893cf640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 0x7f5884139d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f587400b600 tx=0x7f58740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f58893cf640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 0x7f5884139d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f587400b600 tx=0x7f58740091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f5874015070 mon_map magic: 0 v1 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874015070 con 0x7f5884131c90 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 10 monclient: dump: 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5874003d90 con 0x7f5884131c90 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f588be34640 5 monclient: authenticate success, global_id 4741 2024-09-18T12:27:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f5879ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f588be34640 10 monclient: _renew_subs 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f588be34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5884136410 con 0x7f5884131c90 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f587bfff640 4 set_mon_vals no callback set 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f588be34640 10 monclient: _renew_subs 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f588be34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.053+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5884136fe0 con 0x7f5884131c90 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874006380 con 0x7f5884131c90 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5879ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5879ffb640 10 monclient: dump: 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5874010440 con 0x7f5884131c90 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5879ffb640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5868038490 0x7f586803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5889bd0640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5868038490 0x7f586803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f5889bd0640 10 monclient: get_auth_request con 0x7f5868038490 auth_method 0 2024-09-18T12:27:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f588be34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f588be34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f588be34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.054+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5884138050 con 0x7f5884131c90 2024-09-18T12:27:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.056+0000 7f5889bd0640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5868038490 0x7f586803a940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5880005150 tx=0x7f5880009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.059+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f587404b210 con 0x7f5884131c90 2024-09-18T12:27:37.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.676+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5874018d60 con 0x7f5884131c90 2024-09-18T12:27:37.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.676+0000 7f5879ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:37.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.676+0000 7f5879ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.830+0000 7f588be34640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-18T12:27:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.830+0000 7f588be34640 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-18T12:27:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.830+0000 7f588be34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.830+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7f58840008d0 con 0x7f5884131c90 2024-09-18T12:27:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.831+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v45) v1 ==== 67+0+1058 (secure 0 0 0) 0x7f5874018780 con 0x7f5884131c90 2024-09-18T12:27:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.831+0000 7f5879ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-18T12:27:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.831+0000 7f5879ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:37.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.924+0000 7f588be34640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-18T12:27:37.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.924+0000 7f588be34640 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-18T12:27:37.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.924+0000 7f588be34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.924+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7f58841383f0 con 0x7f5884131c90 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.924+0000 7f5879ffb640 1 -- 172.21.15.144:0/393695992 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v45) v1 ==== 67+0+1058 (secure 0 0 0) 0x7f5874018780 con 0x7f5884131c90 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.924+0000 7f5879ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.924+0000 7f5879ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stdout:[ 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_id": 0, 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_name": "replicated_rule", 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stdout: "type": 1, 2024-09-18T12:27:37.926 INFO:tasks.workunit.client.0.smithi144.stdout: "steps": [ 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "take", 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: "item": -1, 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: "item_name": "default" 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "choose_firstn", 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 0, 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: "type": "osd" 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "emit" 2024-09-18T12:27:37.927 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_id": 1, 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "rule_name": "erasure-code", 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "type": 3, 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "steps": [ 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "set_chooseleaf_tries", 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 5 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "set_choose_tries", 2024-09-18T12:27:37.928 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 100 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "take", 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: "item": -1, 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: "item_name": "default" 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "choose_indep", 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: "num": 0, 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: "type": "osd" 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: "op": "emit" 2024-09-18T12:27:37.929 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:37.930 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:27:37.930 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:27:37.930 INFO:tasks.workunit.client.0.smithi144.stdout:] 2024-09-18T12:27:37.930 INFO:tasks.workunit.client.0.smithi144.stdout: 2024-09-18T12:27:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5868038490 msgr2=0x7f586803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5868038490 0x7f586803a940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5880005150 tx=0x7f5880009040 comp rx=0 tx=0).stop 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 10 monclient: shutdown 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 msgr2=0x7f5884139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 0x7f5884139d10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f587400b600 tx=0x7f58740091a0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 shutdown_connections 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5868038490 0x7f586803a940 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5884141380 0x7f5884134970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5884131c90 0x7f5884139d10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 --2- 172.21.15.144:0/393695992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f58840a0ab0 0x7f5884140e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 >> 172.21.15.144:0/393695992 conn(0x7f5884071370 msgr2=0x7f588409e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 shutdown_connections 2024-09-18T12:27:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:37.926+0000 7f588be34640 1 -- 172.21.15.144:0/393695992 wait complete. 2024-09-18T12:27:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls 2024-09-18T12:27:38.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.008+0000 7f0e5215d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:38.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.008+0000 7f0e5215d640 10 monclient: build_initial_monmap 2024-09-18T12:27:38.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.008+0000 7f0e5215d640 10 monclient: monmap: 2024-09-18T12:27:38.009 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:38.009167+0000 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:38.009167+0000 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:38.010 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 1 Processor -- start 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 1 -- start start 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient: init 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e4c09ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e4c09f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c069d10 0x7f0e4c13a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e4c069d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4c067420 con 0x7f0e4c09f020 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4c0a0480 con 0x7f0e4c09ec50 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c06a170 con 0x7f0e4c069d10 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e5215d640 10 monclient(hunting): authenticate will time out at 893.578654s 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e43fff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f0e4c09f020 legacy=0x7f0e4c136d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e4b7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f0e4c09ec50 0x7f0e4c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59656/0 (socket says 172.21.15.144:59656) 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e4b7fe640 1 -- 172.21.15.144:0/2425263433 learned_addr learned my addr 172.21.15.144:0/2425263433 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.009+0000 7f0e4bfff640 1 --2- 172.21.15.144:0/2425263433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c069d10 0x7f0e4c13a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): get_auth_request con 0x7f0e4c069d10 auth_method 0 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:38.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient(hunting): handle_auth_done global_id 44222 payload 274 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 1 -- 172.21.15.144:0/2425263433 >> v1:172.21.15.144:6789/0 conn(0x7f0e4c09ec50 legacy=0x7f0e4c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 1 -- 172.21.15.144:0/2425263433 >> v1:172.21.15.116:6789/0 conn(0x7f0e4c09f020 legacy=0x7f0e4c136d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 1 monclient: found mon.noname-b 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:38.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.010+0000 7f0e4bfff640 1 -- 172.21.15.144:0/2425263433 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c0a0200 con 0x7f0e4c069d10 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4bfff640 1 -- 172.21.15.144:0/2425263433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c069d10 msgr2=0x7f0e4c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4bfff640 1 --2- 172.21.15.144:0/2425263433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c069d10 0x7f0e4c13a490 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e44009e90 tx=0x7f0e44009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb606a25203612a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 1 -- 172.21.15.144:0/2425263433 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44007840 con 0x7f0e4c069d10 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: dump: 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.021 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: _finish_auth 0 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 1 -- 172.21.15.144:0/2425263433 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4402d050 con 0x7f0e4c069d10 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 1 -- 172.21.15.144:0/2425263433 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44007bd0 con 0x7f0e4c069d10 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e5215d640 5 monclient: authenticate success, global_id 44222 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e5215d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e5215d640 4 set_mon_vals no callback set 2024-09-18T12:27:38.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.013+0000 7f0e4affd640 10 monclient: dump: 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 10 monclient: shutdown 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 1 -- 172.21.15.144:0/2425263433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c069d10 msgr2=0x7f0e4c13a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 1 --2- 172.21.15.144:0/2425263433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c069d10 0x7f0e4c13a490 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e44009e90 tx=0x7f0e44009780 comp rx=0 tx=0).stop 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 1 -- 172.21.15.144:0/2425263433 shutdown_connections 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 1 --2- 172.21.15.144:0/2425263433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c069d10 0x7f0e4c13a490 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 1 -- 172.21.15.144:0/2425263433 >> 172.21.15.144:0/2425263433 conn(0x7f0e4c0a5cf0 msgr2=0x7f0e4c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 1 -- 172.21.15.144:0/2425263433 shutdown_connections 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.014+0000 7f0e5215d640 1 -- 172.21.15.144:0/2425263433 wait complete. 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient: build_initial_monmap 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient: monmap: 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:38.024 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 Processor -- start 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 -- start start 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient: init 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:38.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 0x7f0e4c11def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e4c069d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e4c09ec50 0x7f0e4c11e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e4c09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c09f020 0x7f0e4c11e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e4c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c09c020 con 0x7f0e4c09ec50 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c09e0f0 con 0x7f0e4c069d10 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c1214a0 con 0x7f0e4c09f020 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:38.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e5215d640 10 monclient(hunting): authenticate will time out at 893.584847s 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e43fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e4c09ec50 msgr2=0x7f0e4c11e430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e4b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 0x7f0e4c11def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e43fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e4c09ec50 0x7f0e4c11e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e4b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 0x7f0e4c11def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51466/0 (socket says 172.21.15.144:51466) 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e4b7fe640 1 -- 172.21.15.144:0/929820519 learned_addr learned my addr 172.21.15.144:0/929820519 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e4b7fe640 10 monclient(hunting): get_auth_request con 0x7f0e4c069d10 auth_method 0 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.015+0000 7f0e4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:38.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4bfff640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c09f020 0x7f0e4c11e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4bfff640 10 monclient(hunting): get_auth_request con 0x7f0e4c09f020 auth_method 0 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 10 monclient(hunting): handle_auth_done global_id 4744 payload 824 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 1 -- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c09f020 msgr2=0x7f0e4c11e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c09f020 0x7f0e4c11e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 1 -- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e4c09ec50 msgr2=0x7f0e4c11e430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e4c09ec50 0x7f0e4c11e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4b7fe640 1 -- 172.21.15.144:0/929820519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44009a20 con 0x7f0e4c069d10 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.016+0000 7f0e4bfff640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c09f020 0x7f0e4c11e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e4b7fe640 1 -- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 msgr2=0x7f0e4c11def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e4b7fe640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 0x7f0e4c11def0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c5e0 tx=0x7f0e3c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c005cb0 con 0x7f0e4c069d10 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: dump: 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.029 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e5215d640 5 monclient: authenticate success, global_id 4744 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c005e30 con 0x7f0e4c069d10 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e5215d640 10 monclient: _renew_subs 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e5215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4c120540 con 0x7f0e4c069d10 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c007340 con 0x7f0e4c069d10 2024-09-18T12:27:38.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e437fe640 10 monclient: dump: 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e5215d640 10 monclient: _renew_subs 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e5215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e4a7fc640 4 set_mon_vals no callback set 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.018+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4c1314c0 con 0x7f0e4c069d10 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0e3c01b400 con 0x7f0e4c069d10 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e5215d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e5215d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e5215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e4c0008d0 con 0x7f0e4c069d10 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e437fe640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e2c038440 0x7f0e2c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f0e3c04b250 con 0x7f0e4c069d10 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e43fff640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e2c038440 0x7f0e2c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.019+0000 7f0e43fff640 10 monclient: get_auth_request con 0x7f0e2c038440 auth_method 0 2024-09-18T12:27:38.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.022+0000 7f0e43fff640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e2c038440 0x7f0e2c03a8f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e340038f0 tx=0x7f0e34006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.024+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0e3c01a720 con 0x7f0e4c069d10 2024-09-18T12:27:38.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.024+0000 7f0e437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.024+0000 7f0e437fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:38.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.179+0000 7f0e5215d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:27:38.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.179+0000 7f0e5215d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:27:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.179+0000 7f0e5215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.179+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f0e4c09fd80 con 0x7f0e4c069d10 2024-09-18T12:27:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.179+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v45) v1 ==== 65+0+29 (secure 0 0 0) 0x7f0e3c01a360 con 0x7f0e4c069d10 2024-09-18T12:27:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.179+0000 7f0e437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:27:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.179+0000 7f0e437fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:38.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.272+0000 7f0e5215d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:27:38.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.272+0000 7f0e5215d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:27:38.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.272+0000 7f0e5215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.272+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f0e4c072640 con 0x7f0e4c069d10 2024-09-18T12:27:38.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.272+0000 7f0e437fe640 1 -- 172.21.15.144:0/929820519 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v45) v1 ==== 65+0+29 (secure 0 0 0) 0x7f0e3c01a540 con 0x7f0e4c069d10 2024-09-18T12:27:38.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.272+0000 7f0e437fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:27:38.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.272+0000 7f0e437fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:38.274 INFO:tasks.workunit.client.0.smithi144.stdout:replicated_rule 2024-09-18T12:27:38.275 INFO:tasks.workunit.client.0.smithi144.stdout:erasure-code 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e2c038440 msgr2=0x7f0e2c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e2c038440 0x7f0e2c03a8f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e340038f0 tx=0x7f0e34006d80 comp rx=0 tx=0).stop 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 10 monclient: shutdown 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 msgr2=0x7f0e4c11def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 0x7f0e4c11def0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c5e0 tx=0x7f0e3c004960 comp rx=0 tx=0).stop 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 shutdown_connections 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e2c038440 0x7f0e2c03a8f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e4c09f020 0x7f0e4c11e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e4c09ec50 0x7f0e4c11e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 --2- 172.21.15.144:0/929820519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4c069d10 0x7f0e4c11def0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 >> 172.21.15.144:0/929820519 conn(0x7f0e4c0a5cf0 msgr2=0x7f0e4c09b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:38.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 shutdown_connections 2024-09-18T12:27:38.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.274+0000 7f0e5215d640 1 -- 172.21.15.144:0/929820519 wait complete. 2024-09-18T12:27:38.284 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule list 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.356+0000 7fe923cdd640 10 monclient: get_monmap_and_config 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.356+0000 7fe923cdd640 10 monclient: build_initial_monmap 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.356+0000 7fe923cdd640 10 monclient: monmap: 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:38.357741+0000 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:38.357741+0000 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.358 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:38.359 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:38.359 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:38.359 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:38.359 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:38.359 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.357+0000 7fe923cdd640 1 Processor -- start 2024-09-18T12:27:38.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.357+0000 7fe923cdd640 1 -- start start 2024-09-18T12:27:38.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.357+0000 7fe923cdd640 10 monclient: init 2024-09-18T12:27:38.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe91c06e000 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe91c06a710 0x7fe91c06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient(hunting): picked mon.noname-c con 0x7fe91c06a710 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c09e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient(hunting): picked mon.noname-b con 0x7fe91c09e170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c0699c0 con 0x7fe91c06e000 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91c06a360 con 0x7fe91c06a710 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c06be20 con 0x7fe91c09e170 2024-09-18T12:27:38.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe923cdd640 10 monclient(hunting): authenticate will time out at 893.927135s 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe91c06a710 0x7fe91c06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921a79640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe91c06e000 0x7fe91c09da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59658/0 (socket says 172.21.15.144:59658) 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921a79640 1 -- 172.21.15.144:0/589550582 learned_addr learned my addr 172.21.15.144:0/589550582 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): get_auth_request con 0x7fe91c06a710 auth_method 0 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe92227a640 1 --2- 172.21.15.144:0/589550582 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c09e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe92227a640 10 monclient(hunting): get_auth_request con 0x7fe91c09e170 auth_method 0 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe92227a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe92227a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe92227a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient(hunting): handle_auth_done global_id 4747 payload 274 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 1 -- 172.21.15.144:0/589550582 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 msgr2=0x7fe91c09e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 1 --2- 172.21.15.144:0/589550582 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c09e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 1 -- 172.21.15.144:0/589550582 >> v1:172.21.15.144:6789/0 conn(0x7fe91c06e000 legacy=0x7fe91c09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient: _finish_hunting 0 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 1 monclient: found mon.noname-c 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.358+0000 7fe921278640 1 -- 172.21.15.144:0/589550582 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c069f80 con 0x7fe91c06a710 2024-09-18T12:27:38.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe921278640 1 -- 172.21.15.144:0/589550582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06a710 msgr2=0x7fe91c06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:38.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe921278640 1 --2- 172.21.15.144:0/589550582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06a710 0x7fe91c06e3d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009850 tx=0x7fe90c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=733991ed44527597 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 1 -- 172.21.15.144:0/589550582 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c02add0 con 0x7fe91c06a710 2024-09-18T12:27:38.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: dump: 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: _finish_auth 0 2024-09-18T12:27:38.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 1 -- 172.21.15.144:0/589550582 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c032440 con 0x7fe91c06a710 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 5 monclient: authenticate success, global_id 4747 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 1 -- 172.21.15.144:0/589550582 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0327e0 con 0x7fe91c06a710 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 4 set_mon_vals no callback set 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe920a77640 10 monclient: dump: 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.372 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 10 monclient: shutdown 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 1 -- 172.21.15.144:0/589550582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06a710 msgr2=0x7fe91c06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.361+0000 7fe923cdd640 1 --2- 172.21.15.144:0/589550582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06a710 0x7fe91c06e3d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009850 tx=0x7fe90c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 -- 172.21.15.144:0/589550582 shutdown_connections 2024-09-18T12:27:38.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 --2- 172.21.15.144:0/589550582 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c09e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 --2- 172.21.15.144:0/589550582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06a710 0x7fe91c06e3d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 -- 172.21.15.144:0/589550582 >> 172.21.15.144:0/589550582 conn(0x7fe91c071340 msgr2=0x7fe91c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 -- 172.21.15.144:0/589550582 shutdown_connections 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 -- 172.21.15.144:0/589550582 wait complete. 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 10 monclient: build_initial_monmap 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 10 monclient: monmap: 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:38.374 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 Processor -- start 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 1 -- start start 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.362+0000 7fe923cdd640 10 monclient: init 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe91c06a710 0x7fe91c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): picked mon.noname-a con 0x7fe91c06a710 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 0x7fe91c133ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): picked mon.noname-b con 0x7fe91c06e000 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c11e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): picked mon.noname-c con 0x7fe91c09e170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c1222d0 con 0x7fe91c06a710 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c120a40 con 0x7fe91c06e000 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c121ff0 con 0x7fe91c09e170 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe923cdd640 10 monclient(hunting): authenticate will time out at 893.932410s 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 0x7fe91c133ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921a79640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe91c06a710 msgr2=0x7fe91c141090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 0x7fe91c133ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51474/0 (socket says 172.21.15.144:51474) 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 -- 172.21.15.144:0/4190102905 learned_addr learned my addr 172.21.15.144:0/4190102905 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): get_auth_request con 0x7fe91c06e000 auth_method 0 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921a79640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe91c06a710 0x7fe91c141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe92227a640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c11e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe92227a640 10 monclient(hunting): get_auth_request con 0x7fe91c09e170 auth_method 0 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe92227a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe92227a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe92227a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient(hunting): handle_auth_done global_id 4750 payload 824 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 -- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 msgr2=0x7fe91c11e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c11e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 -- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe91c06a710 msgr2=0x7fe91c141090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:38.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe91c06a710 0x7fe91c141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient: _finish_hunting 0 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 monclient: found mon.noname-b 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.363+0000 7fe921278640 1 -- 172.21.15.144:0/4190102905 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c009390 con 0x7fe91c06e000 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.366+0000 7fe921278640 1 -- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 msgr2=0x7fe91c133ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.366+0000 7fe921278640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 0x7fe91c133ec0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009850 tx=0x7fe90c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.366+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c031d40 con 0x7fe91c06e000 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:38.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: dump: 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:38.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c03d440 con 0x7fe91c06e000 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe923cdd640 5 monclient: authenticate success, global_id 4750 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe923cdd640 10 monclient: _renew_subs 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe923cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c120530 con 0x7fe91c06e000 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c031ea0 con 0x7fe91c06e000 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe911ffb640 10 monclient: dump: 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.380 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe923cdd640 10 monclient: _renew_subs 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe913fff640 4 set_mon_vals no callback set 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe923cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.368+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c141e20 con 0x7fe91c06e000 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe90c039040 con 0x7fe91c06e000 2024-09-18T12:27:38.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe923cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe923cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe923cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91c122490 con 0x7fe91c06e000 2024-09-18T12:27:38.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe911ffb640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe900038490 0x7fe90003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fe90c06b4c0 con 0x7fe91c06e000 2024-09-18T12:27:38.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe921a79640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe900038490 0x7fe90003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.369+0000 7fe921a79640 10 monclient: get_auth_request con 0x7fe900038490 auth_method 0 2024-09-18T12:27:38.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.371+0000 7fe921a79640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe900038490 0x7fe90003a940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe9180047d0 tx=0x7fe918007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.373+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe90c02f2a0 con 0x7fe91c06e000 2024-09-18T12:27:38.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.373+0000 7fe911ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.373+0000 7fe911ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:38.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.530+0000 7fe923cdd640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-18T12:27:38.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.530+0000 7fe923cdd640 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-18T12:27:38.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.530+0000 7fe923cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.530+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7fe91c0008d0 con 0x7fe91c06e000 2024-09-18T12:27:38.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.531+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v45) v1 ==== 67+0+29 (secure 0 0 0) 0x7fe90c037030 con 0x7fe91c06e000 2024-09-18T12:27:38.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.531+0000 7fe911ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-18T12:27:38.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.531+0000 7fe911ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:38.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.627+0000 7fe923cdd640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.627+0000 7fe923cdd640 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.627+0000 7fe923cdd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.627+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7fe91c122130 con 0x7fe91c06e000 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.627+0000 7fe911ffb640 1 -- 172.21.15.144:0/4190102905 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v45) v1 ==== 67+0+29 (secure 0 0 0) 0x7fe91c122130 con 0x7fe91c06e000 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.627+0000 7fe911ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.627+0000 7fe911ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stdout:replicated_rule 2024-09-18T12:27:38.629 INFO:tasks.workunit.client.0.smithi144.stdout:erasure-code 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe900038490 msgr2=0x7fe90003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe900038490 0x7fe90003a940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe9180047d0 tx=0x7fe918007b80 comp rx=0 tx=0).stop 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 10 monclient: shutdown 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 msgr2=0x7fe91c133ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 0x7fe91c133ec0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009850 tx=0x7fe90c032000 comp rx=0 tx=0).stop 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 shutdown_connections 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe900038490 0x7fe90003a940 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe91c09e170 0x7fe91c11e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe91c06e000 0x7fe91c133ec0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 --2- 172.21.15.144:0/4190102905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe91c06a710 0x7fe91c141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 >> 172.21.15.144:0/4190102905 conn(0x7fe91c071340 msgr2=0x7fe91c09c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:38.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 shutdown_connections 2024-09-18T12:27:38.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.629+0000 7fe923cdd640 1 -- 172.21.15.144:0/4190102905 wait complete. 2024-09-18T12:27:38.639 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-18T12:27:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.714+0000 7fba00f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.714+0000 7fba00f9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.714+0000 7fba00f9d640 10 monclient: monmap: 2024-09-18T12:27:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:38.714981+0000 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:38.714981+0000 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.714+0000 7fba00f9d640 1 Processor -- start 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.714+0000 7fba00f9d640 1 -- start start 2024-09-18T12:27:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.714+0000 7fba00f9d640 10 monclient: init 2024-09-18T12:27:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T12:27:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9fc09ec50 0x7fb9fc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9fc09ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc1321a0 0x7fb9fc068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9fc1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9fc09f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc067420 con 0x7fb9fc09f020 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9fc09fdc0 con 0x7fb9fc09ec50 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc0a0510 con 0x7fb9fc1321a0 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fba00f9d640 10 monclient(hunting): authenticate will time out at 894.284164s 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9fc09ec50 0x7fb9fc131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb9fc09ec50 0x7fb9fc131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51480/0 (socket says 172.21.15.144:51480) 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fad9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb9fc09f020 0x7fb9fc069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59674/0 (socket says 172.21.15.144:59674) 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 -- 172.21.15.144:0/3055659256 learned_addr learned my addr 172.21.15.144:0/3055659256 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): get_auth_request con 0x7fb9fc09ec50 auth_method 0 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9f1d9b640 1 --2- 172.21.15.144:0/3055659256 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc1321a0 0x7fb9fc068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_done global_id 4753 payload 274 2024-09-18T12:27:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 -- 172.21.15.144:0/3055659256 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc1321a0 msgr2=0x7fb9fc068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/3055659256 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc1321a0 0x7fb9fc068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 -- 172.21.15.144:0/3055659256 >> v1:172.21.15.144:6789/0 conn(0x7fb9fc09f020 legacy=0x7fb9fc069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 monclient: found mon.noname-c 2024-09-18T12:27:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.715+0000 7fb9fa59c640 1 -- 172.21.15.144:0/3055659256 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc0a0200 con 0x7fb9fc09ec50 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9fa59c640 1 -- 172.21.15.144:0/3055659256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc09ec50 msgr2=0x7fb9fc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/3055659256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc09ec50 0x7fb9fc131c60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009910 tx=0x7fb9ec0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=564a6b4b866c056f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 1 -- 172.21.15.144:0/3055659256 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec032440 con 0x7fb9fc09ec50 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: dump: 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: _finish_auth 0 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 1 -- 172.21.15.144:0/3055659256 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec0325a0 con 0x7fb9fc09ec50 2024-09-18T12:27:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fba00f9d640 5 monclient: authenticate success, global_id 4753 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fba00f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fba00f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 1 -- 172.21.15.144:0/3055659256 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec032940 con 0x7fb9fc09ec50 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fba00f9d640 4 set_mon_vals no callback set 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.718+0000 7fb9f9d9b640 10 monclient: dump: 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.729 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 10 monclient: shutdown 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 -- 172.21.15.144:0/3055659256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc09ec50 msgr2=0x7fb9fc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 --2- 172.21.15.144:0/3055659256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc09ec50 0x7fb9fc131c60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009910 tx=0x7fb9ec0091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 -- 172.21.15.144:0/3055659256 shutdown_connections 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 --2- 172.21.15.144:0/3055659256 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc1321a0 0x7fb9fc068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 --2- 172.21.15.144:0/3055659256 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc09ec50 0x7fb9fc131c60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 -- 172.21.15.144:0/3055659256 >> 172.21.15.144:0/3055659256 conn(0x7fb9fc0a5d40 msgr2=0x7fb9fc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 -- 172.21.15.144:0/3055659256 shutdown_connections 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 1 -- 172.21.15.144:0/3055659256 wait complete. 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.719+0000 7fba00f9d640 10 monclient: monmap: 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:38.731 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 Processor -- start 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 -- start start 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient: init 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 0x7fb9fc09aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9fc09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 0x7fb9fc130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9fc09f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 0x7fb9fc11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9fc1321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc09c000 con 0x7fb9fc09f020 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc11dd70 con 0x7fb9fc1321a0 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc11e050 con 0x7fb9fc09ec50 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fba00f9d640 10 monclient(hunting): authenticate will time out at 894.289745s 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 0x7fb9fc11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 0x7fb9fc11b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51498/0 (socket says 172.21.15.144:51498) 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9f1d9b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 msgr2=0x7fb9fc130c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 learned_addr learned my addr 172.21.15.144:0/2441484627 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 10 monclient(hunting): get_auth_request con 0x7fb9fc1321a0 auth_method 0 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9f1d9b640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 0x7fb9fc130c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 0x7fb9fc09aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fa59c640 10 monclient(hunting): get_auth_request con 0x7fb9fc09ec50 auth_method 0 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.720+0000 7fb9fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:38.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 10 monclient(hunting): handle_auth_done global_id 4756 payload 824 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 msgr2=0x7fb9fc09aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 0x7fb9fc09aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 msgr2=0x7fb9fc130c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 0x7fb9fc130c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 1 monclient: found mon.noname-b 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.721+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec009450 con 0x7fb9fc1321a0 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 msgr2=0x7fb9fc11b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9fad9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 0x7fb9fc11b760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4007fd0 tx=0x7fb9f4004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400ce40 con 0x7fb9fc1321a0 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: dump: 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f400a640 con 0x7fb9fc1321a0 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fba00f9d640 5 monclient: authenticate success, global_id 4756 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fba00f9d640 10 monclient: _renew_subs 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fba00f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc11e330 con 0x7fb9fc1321a0 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400aa00 con 0x7fb9fc1321a0 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f2ffd640 10 monclient: dump: 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:38.737 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fba00f9d640 10 monclient: _renew_subs 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fb9f959a640 4 set_mon_vals no callback set 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fba00f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.723+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc11ce90 con 0x7fb9fc1321a0 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fba00f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fba00f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb9f40070f0 con 0x7fb9fc1321a0 2024-09-18T12:27:38.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fba00f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc0008d0 con 0x7fb9fc1321a0 2024-09-18T12:27:38.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fb9f2ffd640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9dc038440 0x7fb9dc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:38.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fb9f404b3a0 con 0x7fb9fc1321a0 2024-09-18T12:27:38.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9dc038440 0x7fb9dc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:38.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.724+0000 7fb9fa59c640 10 monclient: get_auth_request con 0x7fb9dc038440 auth_method 0 2024-09-18T12:27:38.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.727+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9dc038440 0x7fb9dc03a8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0a2b00 tx=0x7fb9ec032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:38.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.728+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb9f401a360 con 0x7fb9fc1321a0 2024-09-18T12:27:38.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.728+0000 7fb9f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:38.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.728+0000 7fb9f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.886+0000 7fba00f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.886+0000 7fba00f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.886+0000 7fba00f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:38.886+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fb9fc064900 con 0x7fb9fc1321a0 2024-09-18T12:27:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.584+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v46) v1 ==== 126+0+0 (secure 0 0 0) 0x7fb9f404b020 con 0x7fb9fc1321a0 2024-09-18T12:27:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.584+0000 7fb9f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.584+0000 7fb9f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.686+0000 7fba00f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.686+0000 7fba00f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.686+0000 7fba00f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.686+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fb9fc141f20 con 0x7fb9fc1321a0 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 msgr2=0x7fb9fc11b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 msgr2=0x7fb9fc11b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fad9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 0x7fb9fc11b760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4007fd0 tx=0x7fb9f4004ef0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fad9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 0x7fb9fc11b760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4007fd0 tx=0x7fb9f4004ef0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:39.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 msgr2=0x7fb9fc11b760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9fc1321a0 0x7fb9fc11b760 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 0x7fb9dc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient(hunting): picked mon.c con 0x7fb9fc09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 0x7fb9dc042cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient(hunting): picked mon.a con 0x7fb9fc09f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9dc043610 0x7fb9dc045a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient(hunting): picked mon.b con 0x7fb9dc043610 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0464b0 con 0x7fb9fc09f020 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0465f0 con 0x7fb9dc043610 2024-09-18T12:27:39.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc046730 con 0x7fb9fc09ec50 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f2ffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9dc043610 0x7fb9dc045a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fad9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 msgr2=0x7fb9dc042cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fad9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 0x7fb9dc042cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): get_auth_request con 0x7fb9dc043610 auth_method 0 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:39.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f1d9b640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 0x7fb9dc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f1d9b640 10 monclient(hunting): get_auth_request con 0x7fb9fc09ec50 auth_method 0 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9f1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient(hunting): handle_auth_done global_id 4756 payload 293 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 msgr2=0x7fb9dc0406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 0x7fb9dc0406d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 msgr2=0x7fb9dc042cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 0x7fb9dc042cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 monclient: found mon.b 2024-09-18T12:27:39.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:39.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fb9dc0468f0 con 0x7fb9dc043610 2024-09-18T12:27:39.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 1 -- 172.21.15.144:0/2441484627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fb9fc0a0200 con 0x7fb9dc043610 2024-09-18T12:27:39.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 10 monclient: _finish_auth 0 2024-09-18T12:27:39.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.687+0000 7fb9fa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.689+0000 7fb9fa59c640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9dc043610 msgr2=0x7fb9dc045a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.689+0000 7fb9fa59c640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9dc043610 0x7fb9dc045a90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec003080 tx=0x7fb9ec02c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.689+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec03c650 con 0x7fb9dc043610 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 10 monclient: dump: 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.696 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec03cca0 con 0x7fb9dc043610 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f8d99640 4 set_mon_vals no callback set 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb9ec039040 con 0x7fb9dc043610 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 1 -- 172.21.15.144:0/2441484627 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v46) v1 ==== 149+0+0 (secure 0 0 0) 0x7fb9ec06d400 con 0x7fb9dc043610 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.690+0000 7fb9f2ffd640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo already exists 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.692+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9dc038440 msgr2=0x7fb9dc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.692+0000 7fba00f9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9dc038440 0x7fb9dc03a8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc0a2b00 tx=0x7fb9ec032000 comp rx=0 tx=0).stop 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.692+0000 7fba00f9d640 10 monclient: shutdown 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.692+0000 7fba00f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.692+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9dc043610 msgr2=0x7fb9dc045a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.692+0000 7fba00f9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9dc043610 0x7fb9dc045a90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec003080 tx=0x7fb9ec02c000 comp rx=0 tx=0).stop 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 shutdown_connections 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9dc043610 0x7fb9dc045a90 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec003080 tx=0x7fb9ec02c000 comp rx=0 tx=0).stop 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9fc09f020 0x7fb9dc042cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9fc09ec50 0x7fb9dc0406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 --2- 172.21.15.144:0/2441484627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9dc038440 0x7fb9dc03a8f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 >> 172.21.15.144:0/2441484627 conn(0x7fb9fc0a5d40 msgr2=0x7fb9fc09e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:39.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 shutdown_connections 2024-09-18T12:27:39.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.693+0000 7fba00f9d640 1 -- 172.21.15.144:0/2441484627 wait complete. 2024-09-18T12:27:39.703 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.778+0000 7f9f71f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.778+0000 7f9f71f9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.778+0000 7f9f71f9d640 10 monclient: monmap: 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:39.779714+0000 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:39.779714+0000 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:39.780 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:39.781 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:39.781 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:39.781 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:39.781 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.779+0000 7f9f71f9d640 1 Processor -- start 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.779+0000 7f9f71f9d640 1 -- start start 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.779+0000 7f9f71f9d640 10 monclient: init 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.779+0000 7f9f71f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.779+0000 7f9f71f9d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f6c0a0ad0 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9f6c09ee60 0x7f9f6c09f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f9f6c09ee60 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f6c136e00 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:39.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6c06a5b0 con 0x7f9f6c136e00 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6c0a0460 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6c09f7a0 con 0x7f9f6c09ee60 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f71f9d640 10 monclient(hunting): authenticate will time out at 895.349077s 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9f6c09ee60 0x7f9f6c09f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6b7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f9f6c0a0ad0 0x7f9f6c0691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59686/0 (socket says 172.21.15.144:59686) 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6b7fe640 1 -- 172.21.15.144:0/948074388 learned_addr learned my addr 172.21.15.144:0/948074388 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): get_auth_request con 0x7f9f6c09ee60 auth_method 0 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:39.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6bfff640 1 -- 172.21.15.144:0/948074388 >> v1:172.21.15.116:6789/0 conn(0x7f9f6c136e00 legacy=0x7f9f6c1391e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient(hunting): handle_auth_done global_id 4759 payload 274 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 1 -- 172.21.15.144:0/948074388 >> v1:172.21.15.144:6789/0 conn(0x7f9f6c0a0ad0 legacy=0x7f9f6c0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 1 -- 172.21.15.144:0/948074388 >> v1:172.21.15.116:6789/0 conn(0x7f9f6c136e00 legacy=0x7f9f6c1391e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 1 monclient: found mon.noname-c 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.780+0000 7f9f6affd640 1 -- 172.21.15.144:0/948074388 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0a0ea0 con 0x7f9f6c09ee60 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6affd640 1 -- 172.21.15.144:0/948074388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c09ee60 msgr2=0x7f9f6c09f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6affd640 1 --2- 172.21.15.144:0/948074388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c09ee60 0x7f9f6c09f260 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009850 tx=0x7f9f600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38d26116947f5196 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 1 -- 172.21.15.144:0/948074388 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60032440 con 0x7f9f6c09ee60 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: dump: 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:39.791 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 5 monclient: authenticate success, global_id 4759 2024-09-18T12:27:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 1 -- 172.21.15.144:0/948074388 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f600325a0 con 0x7f9f6c09ee60 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 1 -- 172.21.15.144:0/948074388 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60032940 con 0x7f9f6c09ee60 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 4 set_mon_vals no callback set 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f6a7fc640 10 monclient: dump: 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.793 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 10 monclient: shutdown 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 1 -- 172.21.15.144:0/948074388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c09ee60 msgr2=0x7f9f6c09f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/948074388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c09ee60 0x7f9f6c09f260 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009850 tx=0x7f9f600090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 1 -- 172.21.15.144:0/948074388 shutdown_connections 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/948074388 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c09ee60 0x7f9f6c09f260 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 1 -- 172.21.15.144:0/948074388 >> 172.21.15.144:0/948074388 conn(0x7f9f6c071370 msgr2=0x7f9f6c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:39.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 1 -- 172.21.15.144:0/948074388 shutdown_connections 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.783+0000 7f9f71f9d640 1 -- 172.21.15.144:0/948074388 wait complete. 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.784+0000 7f9f71f9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.784+0000 7f9f71f9d640 10 monclient: monmap: 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.784+0000 7f9f71f9d640 1 Processor -- start 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.784+0000 7f9f71f9d640 1 -- start start 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.784+0000 7f9f71f9d640 10 monclient: init 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f6c09ee60 0x7f9f6c135cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f9f6c09ee60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 0x7f9f6c11e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f9f6c0a0ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f6c136e00 0x7f9f6c11ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f9f6c136e00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c122410 con 0x7f9f6c09ee60 2024-09-18T12:27:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0a11e0 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c09e8d0 con 0x7f9f6c136e00 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f71f9d640 10 monclient(hunting): authenticate will time out at 895.354334s 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 0x7f9f6c11e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 0x7f9f6c11e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51510/0 (socket says 172.21.15.144:51510) 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 -- 172.21.15.144:0/3200735405 learned_addr learned my addr 172.21.15.144:0/3200735405 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): get_auth_request con 0x7f9f6c0a0ad0 auth_method 0 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:39.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6b7fe640 1 -- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f6c09ee60 msgr2=0x7f9f6c135cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6b7fe640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f6c09ee60 0x7f9f6c135cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient(hunting): handle_auth_done global_id 4762 payload 824 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6bfff640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f6c136e00 0x7f9f6c11ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 -- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f6c136e00 msgr2=0x7f9f6c11ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f6c136e00 0x7f9f6c11ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 -- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f6c09ee60 msgr2=0x7f9f6c135cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f6c09ee60 0x7f9f6c135cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 monclient: found mon.noname-b 2024-09-18T12:27:39.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.785+0000 7f9f6affd640 1 -- 172.21.15.144:0/3200735405 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60009390 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.787+0000 7f9f6affd640 1 -- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 msgr2=0x7f9f6c11e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.787+0000 7f9f6affd640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 0x7f9f6c11e740 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9f6002a5c0 tx=0x7f9f60032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60031a20 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: dump: 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:39.799 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f60031b80 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f71f9d640 5 monclient: authenticate success, global_id 4762 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f71f9d640 10 monclient: _renew_subs 2024-09-18T12:27:39.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f71f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c142090 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6003c410 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 10 monclient: dump: 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:39.801 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f71f9d640 10 monclient: _renew_subs 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f697fa640 4 set_mon_vals no callback set 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f71f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c120ef0 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.788+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9f60039070 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f71f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f71f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f71f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c1221c0 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f537fe640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9f58038440 0x7f9f5803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:39.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f9f6006bba0 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f6b7fe640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9f58038440 0x7f9f5803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:39.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.789+0000 7f9f6b7fe640 10 monclient: get_auth_request con 0x7f9f58038440 auth_method 0 2024-09-18T12:27:39.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.791+0000 7f9f6b7fe640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9f58038440 0x7f9f5803a8f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c0047d0 tx=0x7f9f5c007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:39.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.793+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9f6003a900 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.793+0000 7f9f537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:39.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.793+0000 7f9f537fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.949+0000 7f9f71f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.949+0000 7f9f71f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.949+0000 7f9f71f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.949+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f9f6c1221c0 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.949+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v46) v1 ==== 149+0+0 (secure 0 0 0) 0x7f9f60037020 con 0x7f9f6c0a0ad0 2024-09-18T12:27:39.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.949+0000 7f9f537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:39.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:39.949+0000 7f9f537fe640 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-18T12:27:40.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.044+0000 7f9f71f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:40.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.044+0000 7f9f71f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:40.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.044+0000 7f9f71f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.044+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7f9f6c1221c0 con 0x7f9f6c0a0ad0 2024-09-18T12:27:40.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.045+0000 7f9f537fe640 1 -- 172.21.15.144:0/3200735405 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v46) v1 ==== 149+0+0 (secure 0 0 0) 0x7f9f60040070 con 0x7f9f6c0a0ad0 2024-09-18T12:27:40.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.045+0000 7f9f537fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T12:27:40.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.045+0000 7f9f537fe640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-18T12:27:40.047 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo already exists 2024-09-18T12:27:40.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9f58038440 msgr2=0x7f9f5803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9f58038440 0x7f9f5803a8f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c0047d0 tx=0x7f9f5c007b80 comp rx=0 tx=0).stop 2024-09-18T12:27:40.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 10 monclient: shutdown 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 msgr2=0x7f9f6c11e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 0x7f9f6c11e740 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9f6002a5c0 tx=0x7f9f60032000 comp rx=0 tx=0).stop 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 shutdown_connections 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9f58038440 0x7f9f5803a8f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9f6c136e00 0x7f9f6c11ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9f6c0a0ad0 0x7f9f6c11e740 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 --2- 172.21.15.144:0/3200735405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f6c09ee60 0x7f9f6c135cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 >> 172.21.15.144:0/3200735405 conn(0x7f9f6c071370 msgr2=0x7f9f6c09ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 shutdown_connections 2024-09-18T12:27:40.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.047+0000 7f9f71f9d640 1 -- 172.21.15.144:0/3200735405 wait complete. 2024-09-18T12:27:40.056 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-18T12:27:40.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.131+0000 7f3230c57640 10 monclient: get_monmap_and_config 2024-09-18T12:27:40.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.131+0000 7f3230c57640 10 monclient: build_initial_monmap 2024-09-18T12:27:40.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.131+0000 7f3230c57640 10 monclient: monmap: 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:40.132361+0000 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:40.132361+0000 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:40.133 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:40.134 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:40.134 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:40.134 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 1 Processor -- start 2024-09-18T12:27:40.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 1 -- start start 2024-09-18T12:27:40.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient: init 2024-09-18T12:27:40.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:40.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T12:27:40.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f322c069a30 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f322c09ce10 0x7f322c098310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient(hunting): picked mon.noname-a con 0x7f322c09ce10 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f322c098850 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c09cbd0 con 0x7f322c069a30 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c132730 con 0x7f322c098850 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c09e1f0 con 0x7f322c09ce10 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3230c57640 10 monclient(hunting): authenticate will time out at 895.701747s 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3229d9b640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f322c09ce10 msgr2=0x7f322c098310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f322a59c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f322c069a30 legacy=0x7f322c132020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:40.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f3229d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f322c09ce10 0x7f322c098310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:40.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f322ad9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f322c098850 0x7f322c138db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59698/0 (socket says 172.21.15.144:59698) 2024-09-18T12:27:40.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.132+0000 7f322ad9d640 1 -- 172.21.15.144:0/2027657868 learned_addr learned my addr 172.21.15.144:0/2027657868 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:40.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322ad9d640 1 -- 172.21.15.144:0/2027657868 >> v1:172.21.15.144:6789/0 conn(0x7f322c098850 legacy=0x7f322c138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:40.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276811533 0 0) 0x7f322c132730 con 0x7f322c098850 2024-09-18T12:27:40.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:40.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): my global_id is 4765 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32140034e0 con 0x7f322c098850 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 874921050 0 0) 0x7f32140034e0 con 0x7f322c098850 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 >> v2:172.21.15.116:3300/0 conn(0x7f322c09ce10 msgr2=0x7f322c098310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 --2- 172.21.15.144:0/2027657868 >> v2:172.21.15.116:3300/0 conn(0x7f322c09ce10 0x7f322c098310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 >> v1:172.21.15.116:6789/0 conn(0x7f322c069a30 legacy=0x7f322c132020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: _finish_hunting 0 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c069e00 con 0x7f322c098850 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: _finish_auth 0 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:40.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f3224001b20 con 0x7f322c098850 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f3230c57640 5 monclient: authenticate success, global_id 4765 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f3230c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: dump: 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.150 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 >> v1:172.21.15.144:6789/0 conn(0x7f322c098850 legacy=0x7f322c138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3214003aa0 0x7f3214003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): picked mon.b con 0x7f3214003aa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3214004780 0x7f3214004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): picked mon.c con 0x7f3214004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32140099c0 0x7f321400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): picked mon.a con 0x7f32140099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:40.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c09e1f0 con 0x7f32140099c0 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321400c3c0 con 0x7f3214003aa0 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321400c580 con 0x7f3214004780 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322959a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f3230c57640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3214003aa0 0x7f3214003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): get_auth_request con 0x7f3214003aa0 auth_method 0 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322ad9d640 1 -- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32140099c0 msgr2=0x7f321400bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322ad9d640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32140099c0 0x7f321400bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient(hunting): handle_auth_done global_id 4765 payload 293 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 1 -- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3214004780 msgr2=0x7f3214004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3214004780 0x7f3214004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 1 -- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32140099c0 msgr2=0x7f321400bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32140099c0 0x7f321400bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:40.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 1 monclient: found mon.b 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322ad9d640 1 -- 172.21.15.144:0/2027657868 reap_dead start 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 1 -- 172.21.15.144:0/2027657868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32140034e0 con 0x7f3214003aa0 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 10 monclient: _finish_auth 0 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.135+0000 7f322a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322a59c640 1 -- 172.21.15.144:0/2027657868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3214003aa0 msgr2=0x7f3214003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322a59c640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3214003aa0 0x7f3214003e90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f32200098f0 tx=0x7f3220006ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddec9ce13ab031d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220003b80 con 0x7f3214003aa0 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 10 monclient: dump: 2024-09-18T12:27:40.154 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3220003ce0 con 0x7f3214003aa0 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f3230c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 1 -- 172.21.15.144:0/2027657868 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322002aac0 con 0x7f3214003aa0 2024-09-18T12:27:40.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f322959a640 10 monclient: dump: 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f3230c57640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.138+0000 7f3230c57640 4 set_mon_vals no callback set 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 10 monclient: shutdown 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 1 -- 172.21.15.144:0/2027657868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3214003aa0 msgr2=0x7f3214003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3214003aa0 0x7f3214003e90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f32200098f0 tx=0x7f3220006ed0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 1 -- 172.21.15.144:0/2027657868 shutdown_connections 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 1 --2- 172.21.15.144:0/2027657868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3214003aa0 0x7f3214003e90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 1 -- 172.21.15.144:0/2027657868 >> 172.21.15.144:0/2027657868 conn(0x7f322c09e980 msgr2=0x7f322c09ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 1 -- 172.21.15.144:0/2027657868 shutdown_connections 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 1 -- 172.21.15.144:0/2027657868 wait complete. 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 10 monclient: build_initial_monmap 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.139+0000 7f3230c57640 10 monclient: monmap: 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:40.157 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 Processor -- start 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 -- start start 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient: init 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:40.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 0x7f322c140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): picked mon.noname-a con 0x7f3214003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 0x7f322c12ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): picked mon.noname-b con 0x7f322c12e980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f322c11b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): picked mon.noname-c con 0x7f322c11b430 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c09d5d0 con 0x7f3214003aa0 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0a04b0 con 0x7f322c12e980 2024-09-18T12:27:40.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c141b00 con 0x7f322c11b430 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3230c57640 10 monclient(hunting): authenticate will time out at 895.709806s 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f322a59c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 msgr2=0x7f322c140c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f322a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 0x7f322c140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 0x7f322c12ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 0x7f322c12ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51522/0 (socket says 172.21.15.144:51522) 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 learned_addr learned my addr 172.21.15.144:0/3643962771 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 10 monclient(hunting): get_auth_request con 0x7f322c12e980 auth_method 0 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f322ad9d640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f322c11b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.140+0000 7f3229d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f322ad9d640 10 monclient(hunting): get_auth_request con 0x7f322c11b430 auth_method 0 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f322ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f322ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f322ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f322ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f322ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f322ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 10 monclient(hunting): handle_auth_done global_id 4768 payload 824 2024-09-18T12:27:40.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 msgr2=0x7f322c11b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f322c11b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 msgr2=0x7f322c140c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 0x7f322c140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 1 monclient: found mon.noname-b 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.141+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c11edb0 con 0x7f322c12e980 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 msgr2=0x7f322c12ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 0x7f322c12ed90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b810 tx=0x7f321c009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c00be20 con 0x7f322c12e980 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: dump: 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:40.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c003d90 con 0x7f322c12e980 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3230c57640 5 monclient: authenticate success, global_id 4768 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3230c57640 10 monclient: _renew_subs 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c005340 con 0x7f322c12e980 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3230c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c11d010 con 0x7f322c12e980 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f321a7fc640 10 monclient: dump: 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.164 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3228d99640 4 set_mon_vals no callback set 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3230c57640 10 monclient: _renew_subs 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3230c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.143+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c11dbe0 con 0x7f322c12e980 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f321c012070 con 0x7f322c12e980 2024-09-18T12:27:40.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f320c038470 0x7f320c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f321c04ac30 con 0x7f322c12e980 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3230c57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3230c57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3230c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f322a59c640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f320c038470 0x7f320c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c0008d0 con 0x7f322c12e980 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f322a59c640 10 monclient: get_auth_request con 0x7f320c038470 auth_method 0 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 msgr2=0x7f322c12ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 msgr2=0x7f322c12ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 0x7f322c12ed90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b810 tx=0x7f321c009220 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 0x7f322c12ed90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b810 tx=0x7f321c009220 comp rx=0 tx=0).stop 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:40.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 msgr2=0x7f322c12ed90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f322c12e980 0x7f322c12ed90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f320c03fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient(hunting): picked mon.c con 0x7f322c11b430 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 0x7f320c042730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient(hunting): picked mon.a con 0x7f3214003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f320c043100 0x7f320c045580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient(hunting): picked mon.b con 0x7f320c043100 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c046080 con 0x7f3214003aa0 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0461c0 con 0x7f320c043100 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c046380 con 0x7f322c11b430 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f321a7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.144+0000 7f322ad9d640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 msgr2=0x7f320c042730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322ad9d640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 0x7f320c042730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f320c043100 0x7f320c045580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f320c03fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 10 monclient(hunting): get_auth_request con 0x7f320c043100 auth_method 0 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): get_auth_request con 0x7f322c11b430 auth_method 0 2024-09-18T12:27:40.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f322a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient(hunting): handle_auth_done global_id 4768 payload 293 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f320c043100 msgr2=0x7f320c045580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f320c043100 0x7f320c045580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 msgr2=0x7f320c042730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:40.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 0x7f320c042730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 monclient: found mon.c 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f320c046540 con 0x7f322c11b430 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c0008d0 con 0x7f322c11b430 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 10 monclient: _finish_auth 0 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.145+0000 7f3229d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:40.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.147+0000 7f322a59c640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f320c038470 0x7f320c03a920 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f322002f000 tx=0x7f3220030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:40.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.147+0000 7f3229d9b640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 msgr2=0x7f320c03fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.147+0000 7f3229d9b640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f320c03fe10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f321c049630 tx=0x7f321c049590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c005730 con 0x7f322c11b430 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321a7fc640 10 monclient: dump: 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.172 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c01dd20 con 0x7f322c11b430 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321bfff640 4 set_mon_vals no callback set 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.148+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f321c00c520 con 0x7f322c11b430 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.153+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f321c020080 con 0x7f322c11b430 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.153+0000 7f321a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:40.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.153+0000 7f321a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:40.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.310+0000 7f3230c57640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T12:27:40.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.310+0000 7f3230c57640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T12:27:40.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.310+0000 7f3230c57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:40.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.310+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f322c098d50 con 0x7f322c11b430 2024-09-18T12:27:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.586+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.2 v2:172.21.15.116:3301/0 5 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v47) v1 ==== 126+0+0 (secure 0 0 0) 0x7f321c01a0c0 con 0x7f322c11b430 2024-09-18T12:27:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.586+0000 7f321a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T12:27:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.586+0000 7f321a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:40.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.683+0000 7f3230c57640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T12:27:40.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.683+0000 7f3230c57640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T12:27:40.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.683+0000 7f3230c57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:27:40.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.683+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f322c11efd0 con 0x7f322c11b430 2024-09-18T12:27:40.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.684+0000 7f321a7fc640 1 -- 172.21.15.144:0/3643962771 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v47) v1 ==== 149+0+0 (secure 0 0 0) 0x7f321c020080 con 0x7f322c11b430 2024-09-18T12:27:40.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.684+0000 7f321a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T12:27:40.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.684+0000 7f321a7fc640 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-18T12:27:40.685 INFO:tasks.workunit.client.0.smithi144.stderr:rule bar already exists 2024-09-18T12:27:40.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f320c038470 msgr2=0x7f320c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f320c038470 0x7f320c03a920 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f322002f000 tx=0x7f3220030000 comp rx=0 tx=0).stop 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 10 monclient: shutdown 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 msgr2=0x7f320c03fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f320c03fe10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f321c049630 tx=0x7f321c049590 comp rx=0 tx=0).stop 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 shutdown_connections 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f320c043100 0x7f320c045580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3214003aa0 0x7f320c042730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f322c11b430 0x7f320c03fe10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 --2- 172.21.15.144:0/3643962771 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f320c038470 0x7f320c03a920 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 >> 172.21.15.144:0/3643962771 conn(0x7f322c09e980 msgr2=0x7f322c0970f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:40.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 shutdown_connections 2024-09-18T12:27:40.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.686+0000 7f3230c57640 1 -- 172.21.15.144:0/3643962771 wait complete. 2024-09-18T12:27:40.696 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm-device-class all 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.768+0000 7fc09d657640 10 monclient: get_monmap_and_config 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.768+0000 7fc09d657640 10 monclient: build_initial_monmap 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.768+0000 7fc09d657640 10 monclient: monmap: 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:40.769534+0000 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:40.769534+0000 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:40.770 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:40.771 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:40.771 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:40.771 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:40.771 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 1 Processor -- start 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 1 -- start start 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient: init 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc09809ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc09809f020 0x7fc098069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient(hunting): picked mon.noname-a con 0x7fc09809f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:40.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc098136d60 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc098067420 con 0x7fc09809ec50 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980a0480 con 0x7fc098136d60 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980a0200 con 0x7fc09809f020 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc09d657640 10 monclient(hunting): authenticate will time out at 896.338895s 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc096ffd640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fc09809ec50 legacy=0x7fc098131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.769+0000 7fc08ffff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fc09809f020 msgr2=0x7fc098069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.770+0000 7fc08ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc09809f020 0x7fc098069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.770+0000 7fc0977fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc098136d60 0x7fc098139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59700/0 (socket says 172.21.15.144:59700) 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.770+0000 7fc0977fe640 1 -- 172.21.15.144:0/3757350814 learned_addr learned my addr 172.21.15.144:0/3757350814 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 -- 172.21.15.144:0/3757350814 >> v1:172.21.15.144:6789/0 conn(0x7fc098136d60 legacy=0x7fc098139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:40.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650162205 0 0) 0x7fc0980a0480 con 0x7fc098136d60 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): my global_id is 4771 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0840034e0 con 0x7fc098136d60 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746340057 0 0) 0x7fc0840034e0 con 0x7fc098136d60 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 >> v2:172.21.15.116:3300/0 conn(0x7fc09809f020 msgr2=0x7fc098069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 --2- 172.21.15.144:0/3757350814 >> v2:172.21.15.116:3300/0 conn(0x7fc09809f020 0x7fc098069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 >> v1:172.21.15.116:6789/0 conn(0x7fc09809ec50 legacy=0x7fc098131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: _finish_hunting 0 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:27:40.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09809fd80 con 0x7fc098136d60 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc09d657640 5 monclient: authenticate success, global_id 4771 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fc090001be0 con 0x7fc098136d60 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: dump: 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.779 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 >> v1:172.21.15.144:6789/0 conn(0x7fc098136d60 legacy=0x7fc098139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0840039d0 0x7fc084003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): picked mon.a con 0x7fc0840039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc084004780 0x7fc084004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): picked mon.c con 0x7fc084004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:40.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc08400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): picked mon.b con 0x7fc084005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0980a0200 con 0x7fc0840039d0 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08400c2f0 con 0x7fc084005330 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08400c4b0 con 0x7fc084004780 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0967fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc09d657640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc08400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc096ffd640 1 -- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0840039d0 msgr2=0x7fc084003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): get_auth_request con 0x7fc084005330 auth_method 0 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc096ffd640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0840039d0 0x7fc084003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient(hunting): handle_auth_done global_id 4771 payload 293 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 -- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc084004780 msgr2=0x7fc084004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc084004780 0x7fc084004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 -- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0840039d0 msgr2=0x7fc084003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0840039d0 0x7fc084003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 monclient: found mon.b 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 1 -- 172.21.15.144:0/3757350814 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0840034e0 con 0x7fc084005330 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.772+0000 7fc0977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:40.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.773+0000 7fc0977fe640 1 -- 172.21.15.144:0/3757350814 reap_dead start 2024-09-18T12:27:40.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0977fe640 1 -- 172.21.15.144:0/3757350814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 msgr2=0x7fc08400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0977fe640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc08400ba10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc09809bff0 tx=0x7fc09000a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db2238d1dafb53f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09002b760 con 0x7fc084005330 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 10 monclient: dump: 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.790 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc09d657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09002b900 con 0x7fc084005330 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc09d657640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc09d657640 4 set_mon_vals no callback set 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 1 -- 172.21.15.144:0/3757350814 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09002bcc0 con 0x7fc084005330 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.775+0000 7fc0967fc640 10 monclient: dump: 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.791 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 10 monclient: shutdown 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 1 -- 172.21.15.144:0/3757350814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 msgr2=0x7fc08400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc08400ba10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc09809bff0 tx=0x7fc09000a330 comp rx=0 tx=0).stop 2024-09-18T12:27:40.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 1 -- 172.21.15.144:0/3757350814 shutdown_connections 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 1 --2- 172.21.15.144:0/3757350814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc08400ba10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 1 -- 172.21.15.144:0/3757350814 >> 172.21.15.144:0/3757350814 conn(0x7fc0980a5cf0 msgr2=0x7fc0980a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 1 -- 172.21.15.144:0/3757350814 shutdown_connections 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 1 -- 172.21.15.144:0/3757350814 wait complete. 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 10 monclient: build_initial_monmap 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.776+0000 7fc09d657640 10 monclient: monmap: 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.793 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 Processor -- start 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 -- start start 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient: init 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc0981309e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): picked mon.noname-b con 0x7fc084005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc098130f20 0x7fc098099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): picked mon.noname-c con 0x7fc098130f20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098099eb0 0x7fc09809a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): picked mon.noname-a con 0x7fc098099eb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09811d140 con 0x7fc098099eb0 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09809c020 con 0x7fc084005330 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09809d810 con 0x7fc098130f20 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc09d657640 10 monclient(hunting): authenticate will time out at 896.346751s 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc0981309e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc0981309e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51550/0 (socket says 172.21.15.144:51550) 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 1 -- 172.21.15.144:0/2888315038 learned_addr learned my addr 172.21.15.144:0/2888315038 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 10 monclient(hunting): get_auth_request con 0x7fc084005330 auth_method 0 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:40.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc0977fe640 1 -- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098099eb0 msgr2=0x7fc09809a2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc0977fe640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098099eb0 0x7fc09809a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.777+0000 7fc096ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 10 monclient(hunting): handle_auth_done global_id 4774 payload 824 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc08ffff640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc098130f20 0x7fc098099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 1 -- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc098130f20 msgr2=0x7fc098099970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc098130f20 0x7fc098099970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 1 -- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098099eb0 msgr2=0x7fc09809a2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:40.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098099eb0 0x7fc09809a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 1 monclient: found mon.noname-b 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.778+0000 7fc096ffd640 1 -- 172.21.15.144:0/2888315038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09809b9d0 con 0x7fc084005330 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc096ffd640 1 -- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 msgr2=0x7fc0981309e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc096ffd640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc0981309e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc08800b600 tx=0x7fc0880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088015070 con 0x7fc084005330 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: dump: 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.797 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: _finish_auth 0 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc088003d90 con 0x7fc084005330 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc09d657640 5 monclient: authenticate success, global_id 4774 2024-09-18T12:27:40.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc09d657640 10 monclient: _renew_subs 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088006380 con 0x7fc084005330 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc09d657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09809b4c0 con 0x7fc084005330 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc08effd640 10 monclient: dump: 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:40.799 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc095ffb640 4 set_mon_vals no callback set 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc09d657640 10 monclient: _renew_subs 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc09d657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.780+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc098141f80 con 0x7fc084005330 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc088010440 con 0x7fc084005330 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc08effd640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc078038440 0x7fc07803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc09d657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc09d657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:40.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc08804b2a0 con 0x7fc084005330 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc09d657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0980008d0 con 0x7fc084005330 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc08ffff640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc078038440 0x7fc07803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.781+0000 7fc08ffff640 10 monclient: get_auth_request con 0x7fc078038440 auth_method 0 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.783+0000 7fc08ffff640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc078038440 0x7fc07803a8f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc080003870 tx=0x7fc080008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.785+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc088006770 con 0x7fc084005330 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.785+0000 7fc08effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:40.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.785+0000 7fc08effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:40.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.947+0000 7fc09d657640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T12:27:40.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.947+0000 7fc09d657640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T12:27:40.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.947+0000 7fc09d657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:40.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:40.947+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7fc0980a13e0 con 0x7fc084005330 2024-09-18T12:27:41.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:41.597+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v48) v1 ==== 131+0+0 (secure 0 0 0) 0x7fc088018690 con 0x7fc084005330 2024-09-18T12:27:41.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:41.597+0000 7fc08effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T12:27:41.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:41.597+0000 7fc08effd640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-18T12:27:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:41.697+0000 7fc09d657640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T12:27:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:41.697+0000 7fc09d657640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T12:27:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:41.697+0000 7fc09d657640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:41.697+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7fc0980a4ec0 con 0x7fc084005330 2024-09-18T12:27:42.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.603+0000 7fc08effd640 1 -- 172.21.15.144:0/2888315038 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v49) v1 ==== 282+0+0 (secure 0 0 0) 0x7fc088018ab0 con 0x7fc084005330 2024-09-18T12:27:42.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.603+0000 7fc08effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T12:27:42.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.603+0000 7fc08effd640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-18T12:27:42.605 INFO:tasks.workunit.client.0.smithi144.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-18T12:27:42.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc078038440 msgr2=0x7fc07803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc078038440 0x7fc07803a8f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc080003870 tx=0x7fc080008040 comp rx=0 tx=0).stop 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 10 monclient: shutdown 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 msgr2=0x7fc0981309e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc0981309e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc08800b600 tx=0x7fc0880091a0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 shutdown_connections 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc078038440 0x7fc07803a8f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098099eb0 0x7fc09809a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc098130f20 0x7fc098099970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 --2- 172.21.15.144:0/2888315038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc084005330 0x7fc0981309e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 >> 172.21.15.144:0/2888315038 conn(0x7fc0980a5cf0 msgr2=0x7fc0980a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 shutdown_connections 2024-09-18T12:27:42.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.606+0000 7fc09d657640 1 -- 172.21.15.144:0/2888315038 wait complete. 2024-09-18T12:27:42.618 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-18T12:27:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.696+0000 7f645f31d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.696+0000 7f645f31d640 10 monclient: build_initial_monmap 2024-09-18T12:27:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.696+0000 7f645f31d640 10 monclient: monmap: 2024-09-18T12:27:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:42.697088+0000 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:42.697088+0000 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:42.698 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.696+0000 7f645f31d640 1 Processor -- start 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 1 -- start start 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient: init 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6458098c70 0x7f6458099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient(hunting): picked mon.noname-a con 0x7f6458098c70 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458099580 0x7f645809dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient(hunting): picked mon.noname-b con 0x7f6458099580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6458099c70 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645809f680 con 0x7f6458099c70 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645809f420 con 0x7f6458098c70 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6458068a20 con 0x7f6458099580 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645f31d640 10 monclient(hunting): authenticate will time out at 898.266545s 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645d0b9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f6458098c70 msgr2=0x7f6458099040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645d0b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6458098c70 0x7f6458099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645d8ba640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f6458099c70 0x7f645809e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59716/0 (socket says 172.21.15.144:59716) 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645d8ba640 1 -- 172.21.15.144:0/1596627419 learned_addr learned my addr 172.21.15.144:0/1596627419 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645c8b8640 1 --2- 172.21.15.144:0/1596627419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458099580 0x7f645809dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645c8b8640 10 monclient(hunting): get_auth_request con 0x7f6458099580 auth_method 0 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645c8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645c8b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645c8b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645c8b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.697+0000 7f645c8b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 10 monclient(hunting): handle_auth_done global_id 44240 payload 274 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 1 -- 172.21.15.144:0/1596627419 >> v2:172.21.15.116:3300/0 conn(0x7f6458098c70 msgr2=0x7f6458099040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:42.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 1 --2- 172.21.15.144:0/1596627419 >> v2:172.21.15.116:3300/0 conn(0x7f6458098c70 0x7f6458099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 1 -- 172.21.15.144:0/1596627419 >> v1:172.21.15.144:6789/0 conn(0x7f6458099c70 legacy=0x7f645809e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 10 monclient: _finish_hunting 0 2024-09-18T12:27:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 1 monclient: found mon.noname-b 2024-09-18T12:27:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.698+0000 7f645c8b8640 1 -- 172.21.15.144:0/1596627419 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6458096a30 con 0x7f6458099580 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f645c8b8640 1 -- 172.21.15.144:0/1596627419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458099580 msgr2=0x7f645809dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f645c8b8640 1 --2- 172.21.15.144:0/1596627419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458099580 0x7f645809dad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f644c009850 tx=0x7f644c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3003d2aed5eb2823 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 1 -- 172.21.15.144:0/1596627419 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c032440 con 0x7f6458099580 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: dump: 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: _finish_auth 0 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f645f31d640 5 monclient: authenticate success, global_id 44240 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f645f31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 1 -- 172.21.15.144:0/1596627419 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c0325a0 con 0x7f6458099580 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f645f31d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f645f31d640 4 set_mon_vals no callback set 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 1 -- 172.21.15.144:0/1596627419 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c032980 con 0x7f6458099580 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.701+0000 7f644bfff640 10 monclient: dump: 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 10 monclient: shutdown 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 -- 172.21.15.144:0/1596627419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458099580 msgr2=0x7f645809dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 --2- 172.21.15.144:0/1596627419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458099580 0x7f645809dad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f644c009850 tx=0x7f644c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 -- 172.21.15.144:0/1596627419 shutdown_connections 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 --2- 172.21.15.144:0/1596627419 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458099580 0x7f645809dad0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 --2- 172.21.15.144:0/1596627419 >> v2:172.21.15.116:3300/0 conn(0x7f6458098c70 0x7f6458099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 -- 172.21.15.144:0/1596627419 >> 172.21.15.144:0/1596627419 conn(0x7f6458070e90 msgr2=0x7f64580a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 -- 172.21.15.144:0/1596627419 shutdown_connections 2024-09-18T12:27:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 1 -- 172.21.15.144:0/1596627419 wait complete. 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 10 monclient: build_initial_monmap 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.702+0000 7f645f31d640 10 monclient: monmap: 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 Processor -- start 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 -- start start 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient: init 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458098c70 0x7f6458133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): picked mon.noname-c con 0x7f6458098c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6458099580 0x7f645811db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): picked mon.noname-a con 0x7f6458099580 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 0x7f645811e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): picked mon.noname-b con 0x7f6458099c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6458141fc0 con 0x7f6458099580 2024-09-18T12:27:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645811f7d0 con 0x7f6458099c70 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6458134ca0 con 0x7f6458098c70 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645f31d640 10 monclient(hunting): authenticate will time out at 898.272736s 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 0x7f645811e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645c8b8640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6458099580 msgr2=0x7f645811db50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 0x7f645811e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51566/0 (socket says 172.21.15.144:51566) 2024-09-18T12:27:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 1 -- 172.21.15.144:0/1772291162 learned_addr learned my addr 172.21.15.144:0/1772291162 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 10 monclient(hunting): get_auth_request con 0x7f6458099c70 auth_method 0 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645c8b8640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6458099580 0x7f645811db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d0b9640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458098c70 0x7f6458133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d0b9640 10 monclient(hunting): get_auth_request con 0x7f6458098c70 auth_method 0 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d0b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.703+0000 7f645d0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 10 monclient(hunting): handle_auth_done global_id 4777 payload 824 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 1 -- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458098c70 msgr2=0x7f6458133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458098c70 0x7f6458133cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 1 -- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6458099580 msgr2=0x7f645811db50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6458099580 0x7f645811db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 10 monclient: _finish_hunting 0 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 1 monclient: found mon.noname-b 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.704+0000 7f645d8ba640 1 -- 172.21.15.144:0/1772291162 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c009390 con 0x7f6458099c70 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645d8ba640 1 -- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 msgr2=0x7f645811e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645d8ba640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 0x7f645811e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f645000ccc0 tx=0x7f6450005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000ce80 con 0x7f6458099c70 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: dump: 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: _finish_auth 0 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645f31d640 5 monclient: authenticate success, global_id 4777 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645000aa60 con 0x7f6458099c70 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645f31d640 10 monclient: _renew_subs 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:42.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645811f2c0 con 0x7f6458099c70 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000ae20 con 0x7f6458099c70 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f64497fa640 10 monclient: dump: 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:42.722 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645f31d640 10 monclient: _renew_subs 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645811fe90 con 0x7f6458099c70 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.706+0000 7f644b7fe640 4 set_mon_vals no callback set 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6450007300 con 0x7f6458099c70 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f645f31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f645f31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f645f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6458121830 con 0x7f6458099c70 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f64497fa640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f643c038440 0x7f643c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:42.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4248+0+0 (secure 0 0 0) 0x7f645004b720 con 0x7f6458099c70 2024-09-18T12:27:42.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f645d0b9640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f643c038440 0x7f643c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:42.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.707+0000 7f645d0b9640 10 monclient: get_auth_request con 0x7f643c038440 auth_method 0 2024-09-18T12:27:42.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.710+0000 7f645d0b9640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f643c038440 0x7f643c03a8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f64540038f0 tx=0x7f64540090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:42.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.712+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f645001a590 con 0x7f6458099c70 2024-09-18T12:27:42.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.712+0000 7f64497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:42.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.712+0000 7f64497fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:42.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.867+0000 7f645f31d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T12:27:42.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.867+0000 7f645f31d640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T12:27:42.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.867+0000 7f645f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:42.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:42.868+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7f64580008d0 con 0x7f6458099c70 2024-09-18T12:27:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:43.605+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v50) v1 ==== 135+0+0 (secure 0 0 0) 0x7f6450010af0 con 0x7f6458099c70 2024-09-18T12:27:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:43.605+0000 7f64497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T12:27:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:43.605+0000 7f64497fa640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-18T12:27:43.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:43.701+0000 7f645f31d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T12:27:43.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:43.701+0000 7f645f31d640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T12:27:43.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:43.701+0000 7f645f31d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:43.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:43.701+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7f6458064a20 con 0x7f6458099c70 2024-09-18T12:27:44.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.609+0000 7f64497fa640 1 -- 172.21.15.144:0/1772291162 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v51) v1 ==== 237+0+0 (secure 0 0 0) 0x7f6450010cd0 con 0x7f6458099c70 2024-09-18T12:27:44.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.609+0000 7f64497fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T12:27:44.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.609+0000 7f64497fa640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-18T12:27:44.611 INFO:tasks.workunit.client.0.smithi144.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-18T12:27:44.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f643c038440 msgr2=0x7f643c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f643c038440 0x7f643c03a8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f64540038f0 tx=0x7f64540090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 10 monclient: shutdown 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 msgr2=0x7f645811e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 0x7f645811e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f645000ccc0 tx=0x7f6450005370 comp rx=0 tx=0).stop 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 shutdown_connections 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f643c038440 0x7f643c03a8f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6458099c70 0x7f645811e090 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6458099580 0x7f645811db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 --2- 172.21.15.144:0/1772291162 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6458098c70 0x7f6458133cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 >> 172.21.15.144:0/1772291162 conn(0x7f6458070e90 msgr2=0x7f645809f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:44.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 shutdown_connections 2024-09-18T12:27:44.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.612+0000 7f645f31d640 1 -- 172.21.15.144:0/1772291162 wait complete. 2024-09-18T12:27:44.624 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-18T12:27:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.702+0000 7f7eec700640 10 monclient: get_monmap_and_config 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.702+0000 7f7eec700640 10 monclient: build_initial_monmap 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.702+0000 7f7eec700640 10 monclient: monmap: 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:44.703572+0000 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:44.703572+0000 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:44.705 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:44.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 1 Processor -- start 2024-09-18T12:27:44.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 1 -- start start 2024-09-18T12:27:44.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient: init 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7ee409f320 0x7f7ee41321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee409f320 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ee4070920 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ee4071430 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee4096b60 con 0x7f7ee4070920 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee409e4d0 con 0x7f7ee4071430 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee409e7b0 con 0x7f7ee409f320 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:44.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.703+0000 7f7eec700640 10 monclient(hunting): authenticate will time out at 900.272945s 2024-09-18T12:27:44.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.704+0000 7f7eeac9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7ee4071430 0x7f7ee4138fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59720/0 (socket says 172.21.15.144:59720) 2024-09-18T12:27:44.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.704+0000 7f7eea49c640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f7ee409f320 msgr2=0x7f7ee41321b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:44.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.704+0000 7f7eeac9d640 1 -- 172.21.15.144:0/4262875689 learned_addr learned my addr 172.21.15.144:0/4262875689 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:44.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.704+0000 7f7ee9c9b640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f7ee4070920 legacy=0x7f7ee4070d20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:44.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.704+0000 7f7eea49c640 1 --2- 172.21.15.144:0/4262875689 >> v2:172.21.15.116:3300/0 conn(0x7f7ee409f320 0x7f7ee41321b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 1 -- 172.21.15.144:0/4262875689 >> v1:172.21.15.144:6789/0 conn(0x7f7ee4071430 legacy=0x7f7ee4138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310991007 0 0) 0x7f7ee409e4d0 con 0x7f7ee4071430 2024-09-18T12:27:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): my global_id is 4780 2024-09-18T12:27:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed00034e0 con 0x7f7ee4071430 2024-09-18T12:27:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600023823 0 0) 0x7f7ed00034e0 con 0x7f7ee4071430 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 >> v2:172.21.15.116:3300/0 conn(0x7f7ee409f320 msgr2=0x7f7ee41321b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 --2- 172.21.15.144:0/4262875689 >> v2:172.21.15.116:3300/0 conn(0x7f7ee409f320 0x7f7ee41321b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 >> v1:172.21.15.116:6789/0 conn(0x7f7ee4070920 legacy=0x7f7ee4070d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: _finish_hunting 0 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee409e1f0 con 0x7f7ee4071430 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: _finish_auth 0 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f7edc001b20 con 0x7f7ee4071430 2024-09-18T12:27:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eec700640 5 monclient: authenticate success, global_id 4780 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eec700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: dump: 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 >> v1:172.21.15.144:6789/0 conn(0x7f7ee4071430 legacy=0x7f7ee4138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00039d0 0x7f7ed0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): picked mon.c con 0x7f7ed00039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed0004780 0x7f7ed0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): picked mon.a con 0x7f7ed0004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ed000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): picked mon.b con 0x7f7ed0005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee409e7b0 con 0x7f7ed0004780 2024-09-18T12:27:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed000c2f0 con 0x7f7ed0005330 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed000c4b0 con 0x7f7ed00039d0 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee949a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eec700640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee9c9b640 1 -- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed0004780 msgr2=0x7f7ed0004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ed000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7ee9c9b640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed0004780 0x7f7ed0004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 10 monclient(hunting): get_auth_request con 0x7f7ed0005330 auth_method 0 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.706+0000 7f7eeac9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eea49c640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00039d0 0x7f7ed0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 10 monclient(hunting): handle_auth_done global_id 4780 payload 293 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 1 -- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00039d0 msgr2=0x7f7ed0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ed00039d0 0x7f7ed0003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 1 -- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed0004780 msgr2=0x7f7ed0004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed0004780 0x7f7ed0004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 10 monclient: _finish_hunting 0 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 1 monclient: found mon.b 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 1 -- 172.21.15.144:0/4262875689 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed00034e0 con 0x7f7ed0005330 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 10 monclient: _finish_auth 0 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.707+0000 7f7eeac9d640 1 -- 172.21.15.144:0/4262875689 reap_dead start 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.709+0000 7f7eeac9d640 1 -- 172.21.15.144:0/4262875689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 msgr2=0x7f7ed000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.709+0000 7f7eeac9d640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ed000ba10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7edc001850 tx=0x7f7edc00a240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d846d03cc11f5ae server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc02b630 con 0x7f7ed0005330 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 10 monclient: dump: 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc02b7b0 con 0x7f7ed0005330 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 4 set_mon_vals no callback set 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 1 -- 172.21.15.144:0/4262875689 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc02bb70 con 0x7f7ed0005330 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7ee949a640 10 monclient: dump: 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 10 monclient: shutdown 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 1 -- 172.21.15.144:0/4262875689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 msgr2=0x7f7ed000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ed000ba10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7edc001850 tx=0x7f7edc00a240 comp rx=0 tx=0).stop 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 1 -- 172.21.15.144:0/4262875689 shutdown_connections 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 1 --2- 172.21.15.144:0/4262875689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ed000ba10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 1 -- 172.21.15.144:0/4262875689 >> 172.21.15.144:0/4262875689 conn(0x7f7ee40a2b70 msgr2=0x7f7ee409ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 1 -- 172.21.15.144:0/4262875689 shutdown_connections 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.710+0000 7f7eec700640 1 -- 172.21.15.144:0/4262875689 wait complete. 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient: build_initial_monmap 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient: monmap: 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 Processor -- start 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 -- start start 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient: init 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ee411c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): picked mon.noname-b con 0x7f7ed0005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee411c980 0x7f7ee411cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee411c980 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ee411d290 0x7f7ee41472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee411d290 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee409d5b0 con 0x7f7ee411c980 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee40979e0 con 0x7f7ed0005330 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee411f820 con 0x7f7ee411d290 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.711+0000 7f7eec700640 10 monclient(hunting): authenticate will time out at 900.280997s 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ee411c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7ee9c9b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee411c980 msgr2=0x7f7ee411cd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:44.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ee411c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51582/0 (socket says 172.21.15.144:51582) 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 -- 172.21.15.144:0/3096237579 learned_addr learned my addr 172.21.15.144:0/3096237579 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): get_auth_request con 0x7f7ed0005330 auth_method 0 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7ee9c9b640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee411c980 0x7f7ee411cd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ee411d290 0x7f7ee41472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 10 monclient(hunting): get_auth_request con 0x7f7ee411d290 auth_method 0 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:44.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eeac9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient(hunting): handle_auth_done global_id 4783 payload 824 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 -- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ee411d290 msgr2=0x7f7ee41472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ee411d290 0x7f7ee41472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 -- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee411c980 msgr2=0x7f7ee411cd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee411c980 0x7f7ee411cd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 monclient: found mon.noname-b 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:44.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.712+0000 7f7eea49c640 1 -- 172.21.15.144:0/3096237579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee411fde0 con 0x7f7ed0005330 2024-09-18T12:27:44.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eea49c640 1 -- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 msgr2=0x7f7ee411c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:44.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eea49c640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ee411c440 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400c880 tx=0x7f7ed4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:44.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed400ce10 con 0x7f7ed0005330 2024-09-18T12:27:44.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:44.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: dump: 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:44.732 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eec700640 5 monclient: authenticate success, global_id 4783 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eec700640 10 monclient: _renew_subs 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed4003d90 con 0x7f7ed0005330 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eec700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee412f8b0 con 0x7f7ed0005330 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4005340 con 0x7f7ed0005330 2024-09-18T12:27:44.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7eda7fc640 10 monclient: dump: 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eec700640 10 monclient: _renew_subs 2024-09-18T12:27:44.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.714+0000 7f7ee8c99640 4 set_mon_vals no callback set 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eec700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee411e930 con 0x7f7ed0005330 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eec700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7ed4007620 con 0x7f7ed0005330 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eec700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eec700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.715+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee40008d0 con 0x7f7ed0005330 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.716+0000 7f7eda7fc640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ec8038440 0x7f7ec803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.716+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4472+0+0 (secure 0 0 0) 0x7f7ed404b770 con 0x7f7ed0005330 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.716+0000 7f7ee9c9b640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ec8038440 0x7f7ec803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:44.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.716+0000 7f7ee9c9b640 10 monclient: get_auth_request con 0x7f7ec8038440 auth_method 0 2024-09-18T12:27:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.718+0000 7f7ee9c9b640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ec8038440 0x7f7ec803a8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00038f0 tx=0x7f7ee0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.720+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7ed4005730 con 0x7f7ed0005330 2024-09-18T12:27:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.720+0000 7f7eda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.720+0000 7f7eda7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:44.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.879+0000 7f7eec700640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T12:27:44.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.879+0000 7f7eec700640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T12:27:44.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.879+0000 7f7eec700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:44.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:44.879+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f7ee4064b20 con 0x7f7ed0005330 2024-09-18T12:27:45.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:45.612+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v52) v1 ==== 135+0+0 (secure 0 0 0) 0x7f7ed401a830 con 0x7f7ed0005330 2024-09-18T12:27:45.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:45.613+0000 7f7eda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T12:27:45.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:45.613+0000 7f7eda7fc640 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-18T12:27:45.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:45.709+0000 7f7eec700640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T12:27:45.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:45.709+0000 7f7eec700640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T12:27:45.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:45.709+0000 7f7eec700640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:45.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:45.709+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7f7ee4147810 con 0x7f7ed0005330 2024-09-18T12:27:46.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.558+0000 7f7eda7fc640 1 -- 172.21.15.144:0/3096237579 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v53) v1 ==== 237+0+0 (secure 0 0 0) 0x7f7ed401aa10 con 0x7f7ed0005330 2024-09-18T12:27:46.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.558+0000 7f7eda7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T12:27:46.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.558+0000 7f7eda7fc640 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-18T12:27:46.559 INFO:tasks.workunit.client.0.smithi144.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-18T12:27:46.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ec8038440 msgr2=0x7f7ec803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:46.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ec8038440 0x7f7ec803a8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00038f0 tx=0x7f7ee0006d80 comp rx=0 tx=0).stop 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 10 monclient: shutdown 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 msgr2=0x7f7ee411c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ee411c440 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400c880 tx=0x7f7ed4008040 comp rx=0 tx=0).stop 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 shutdown_connections 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ec8038440 0x7f7ec803a8f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ee411d290 0x7f7ee41472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee411c980 0x7f7ee411cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 --2- 172.21.15.144:0/3096237579 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ed0005330 0x7f7ee411c440 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 >> 172.21.15.144:0/3096237579 conn(0x7f7ee40a2b70 msgr2=0x7f7ee412fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 shutdown_connections 2024-09-18T12:27:46.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.561+0000 7f7eec700640 1 -- 172.21.15.144:0/3096237579 wait complete. 2024-09-18T12:27:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-18T12:27:46.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.657+0000 7fa7ed6a8640 10 monclient: get_monmap_and_config 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.657+0000 7fa7ed6a8640 10 monclient: build_initial_monmap 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.657+0000 7fa7ed6a8640 10 monclient: monmap: 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:46.658697+0000 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:46.658697+0000 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:46.659 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:46.660 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:46.660 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:46.660 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:46.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 1 Processor -- start 2024-09-18T12:27:46.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 1 -- start start 2024-09-18T12:27:46.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient: init 2024-09-18T12:27:46.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7e809ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7e809f020 0x7fa7e8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e809f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7e8136d60 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e8067420 con 0x7fa7e809ec50 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e809fdc0 con 0x7fa7e8136d60 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:46.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e80a0510 con 0x7fa7e809f020 2024-09-18T12:27:46.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:46.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.658+0000 7fa7ed6a8640 10 monclient(hunting): authenticate will time out at 902.227928s 2024-09-18T12:27:46.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.659+0000 7fa7e6ffd640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fa7e809ec50 legacy=0x7fa7e8131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:46.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.659+0000 7fa7de7fc640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fa7e809f020 msgr2=0x7fa7e8069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:46.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.659+0000 7fa7de7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7e809f020 0x7fa7e8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:46.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.659+0000 7fa7e77fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa7e8136d60 0x7fa7e8139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59728/0 (socket says 172.21.15.144:59728) 2024-09-18T12:27:46.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.659+0000 7fa7e77fe640 1 -- 172.21.15.144:0/2364473568 learned_addr learned my addr 172.21.15.144:0/2364473568 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:46.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e77fe640 1 -- 172.21.15.144:0/2364473568 >> v1:172.21.15.144:6789/0 conn(0x7fa7e8136d60 legacy=0x7fa7e8139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:46.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68395666 0 0) 0x7fa7e809fdc0 con 0x7fa7e8136d60 2024-09-18T12:27:46.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:46.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:46.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): my global_id is 4786 2024-09-18T12:27:46.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d40034e0 con 0x7fa7e8136d60 2024-09-18T12:27:46.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256556583 0 0) 0x7fa7d40034e0 con 0x7fa7e8136d60 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 >> v2:172.21.15.116:3300/0 conn(0x7fa7e809f020 msgr2=0x7fa7e8069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 --2- 172.21.15.144:0/2364473568 >> v2:172.21.15.116:3300/0 conn(0x7fa7e809f020 0x7fa7e8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 >> v1:172.21.15.116:6789/0 conn(0x7fa7e809ec50 legacy=0x7fa7e8131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: _finish_hunting 0 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80a0200 con 0x7fa7e8136d60 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fa7e0001b20 con 0x7fa7e8136d60 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7ed6a8640 5 monclient: authenticate success, global_id 4786 2024-09-18T12:27:46.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7ed6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: dump: 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:46.674 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 >> v1:172.21.15.144:6789/0 conn(0x7fa7e8136d60 legacy=0x7fa7e8139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7d4003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): picked mon.b con 0x7fa7d4003aa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d4004780 0x7fa7d4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): picked mon.a con 0x7fa7d4004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:46.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d40099c0 0x7fa7d400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): picked mon.c con 0x7fa7d40099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:46.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80a0510 con 0x7fa7d4004780 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d400c3c0 con 0x7fa7d4003aa0 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d400c580 con 0x7fa7d40099c0 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7ed6a8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7d4003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): get_auth_request con 0x7fa7d4003aa0 auth_method 0 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:46.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7de7fc640 1 -- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d4004780 msgr2=0x7fa7d4004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7de7fc640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d4004780 0x7fa7d4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.661+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_done global_id 4786 payload 293 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d40099c0 msgr2=0x7fa7d400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d40099c0 0x7fa7d400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d4004780 msgr2=0x7fa7d4004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d4004780 0x7fa7d4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 1 monclient: found mon.b 2024-09-18T12:27:46.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/2364473568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40034e0 con 0x7fa7d4003aa0 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.662+0000 7fa7e77fe640 1 -- 172.21.15.144:0/2364473568 reap_dead start 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/2364473568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 msgr2=0x7fa7d4003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e6ffd640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7d4003e90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80099c0 tx=0x7fa7d8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31f232b4dff5fdda server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8003b80 con 0x7fa7d4003aa0 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 10 monclient: dump: 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:46.679 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7ed6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d8003d20 con 0x7fa7d4003aa0 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7ed6a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:46.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7ed6a8640 4 set_mon_vals no callback set 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 1 -- 172.21.15.144:0/2364473568 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d802ab80 con 0x7fa7d4003aa0 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.664+0000 7fa7e67fc640 10 monclient: dump: 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:46.681 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 10 monclient: shutdown 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/2364473568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 msgr2=0x7fa7d4003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7d4003e90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80099c0 tx=0x7fa7d8009060 comp rx=0 tx=0).stop 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/2364473568 shutdown_connections 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/2364473568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7d4003e90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/2364473568 >> 172.21.15.144:0/2364473568 conn(0x7fa7e80a5d40 msgr2=0x7fa7e80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/2364473568 shutdown_connections 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/2364473568 wait complete. 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 10 monclient: build_initial_monmap 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.665+0000 7fa7ed6a8640 10 monclient: monmap: 2024-09-18T12:27:46.682 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 Processor -- start 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 -- start start 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient: init 2024-09-18T12:27:46.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7e81411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d4003aa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e8141710 0x7fa7e8141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e8141710 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7e80998a0 0x7fa7e8099d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e80998a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e811c900 con 0x7fa7e8141710 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e809c000 con 0x7fa7d4003aa0 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:46.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e809dc20 con 0x7fa7e80998a0 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7ed6a8640 10 monclient(hunting): authenticate will time out at 902.235931s 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.666+0000 7fa7e6ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7e81411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7de7fc640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e8141710 msgr2=0x7fa7e8141ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7e81411d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:51594/0 (socket says 172.21.15.144:51594) 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/1319041548 learned_addr learned my addr 172.21.15.144:0/1319041548 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): get_auth_request con 0x7fa7d4003aa0 auth_method 0 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:46.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7de7fc640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e8141710 0x7fa7e8141ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:46.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:46.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e77fe640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7e80998a0 0x7fa7e8099d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e77fe640 10 monclient(hunting): get_auth_request con 0x7fa7e80998a0 auth_method 0 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient(hunting): handle_auth_done global_id 4789 payload 824 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7e80998a0 msgr2=0x7fa7e8099d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7e80998a0 0x7fa7e8099d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e8141710 msgr2=0x7fa7e8141ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e8141710 0x7fa7e8141ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:46.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 monclient: found mon.noname-b 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.667+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/1319041548 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e8142020 con 0x7fa7d4003aa0 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7e6ffd640 1 -- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 msgr2=0x7fa7e81411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7e6ffd640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7e81411d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009420 tx=0x7fa7d8030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8005590 con 0x7fa7d4003aa0 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: dump: 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:46.690 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d80056f0 con 0x7fa7d4003aa0 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7ed6a8640 5 monclient: authenticate success, global_id 4789 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:46.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80336a0 con 0x7fa7d4003aa0 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7ed6a8640 10 monclient: _renew_subs 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7ed6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e809b720 con 0x7fa7d4003aa0 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7df7fe640 10 monclient: dump: 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:46.692 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7e5ffb640 4 set_mon_vals no callback set 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7ed6a8640 10 monclient: _renew_subs 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7ed6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.669+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e811dc40 con 0x7fa7d4003aa0 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa7d8030070 con 0x7fa7d4003aa0 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7ed6a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7ed6a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7ed6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e80008d0 con 0x7fa7d4003aa0 2024-09-18T12:27:46.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7df7fe640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7c8038470 0x7fa7c803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:46.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4704+0+0 (secure 0 0 0) 0x7fa7d803a070 con 0x7fa7d4003aa0 2024-09-18T12:27:46.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7de7fc640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7c8038470 0x7fa7c803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:46.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.670+0000 7fa7de7fc640 10 monclient: get_auth_request con 0x7fa7c8038470 auth_method 0 2024-09-18T12:27:46.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.673+0000 7fa7de7fc640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7c8038470 0x7fa7c803a920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa7e809ff10 tx=0x7fa7d0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:46.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.675+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa7d809d030 con 0x7fa7d4003aa0 2024-09-18T12:27:46.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.675+0000 7fa7df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:46.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.675+0000 7fa7df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:46.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.835+0000 7fa7ed6a8640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T12:27:46.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.835+0000 7fa7ed6a8640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T12:27:46.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.835+0000 7fa7ed6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:46.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:46.835+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7fa7e80a13e0 con 0x7fa7d4003aa0 2024-09-18T12:27:47.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.622+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v54) v1 ==== 150+0+0 (secure 0 0 0) 0x7fa7d803c370 con 0x7fa7d4003aa0 2024-09-18T12:27:47.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.622+0000 7fa7df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T12:27:47.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.622+0000 7fa7df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:47.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.720+0000 7fa7ed6a8640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T12:27:47.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.720+0000 7fa7ed6a8640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T12:27:47.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.720+0000 7fa7ed6a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:47.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.720+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7fa7e8072640 con 0x7fa7d4003aa0 2024-09-18T12:27:47.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.720+0000 7fa7df7fe640 1 -- 172.21.15.144:0/1319041548 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v54) v1 ==== 177+0+0 (secure 0 0 0) 0x7fa7d8032bc0 con 0x7fa7d4003aa0 2024-09-18T12:27:47.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.720+0000 7fa7df7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T12:27:47.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.720+0000 7fa7df7fe640 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-18T12:27:47.722 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo-ssd already exists 2024-09-18T12:27:47.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7c8038470 msgr2=0x7fa7c803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:47.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7c8038470 0x7fa7c803a920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa7e809ff10 tx=0x7fa7d0006d80 comp rx=0 tx=0).stop 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 10 monclient: shutdown 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 msgr2=0x7fa7e81411d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7e81411d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8009420 tx=0x7fa7d8030000 comp rx=0 tx=0).stop 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 shutdown_connections 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7c8038470 0x7fa7c803a920 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7e80998a0 0x7fa7e8099d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e8141710 0x7fa7e8141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 --2- 172.21.15.144:0/1319041548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d4003aa0 0x7fa7e81411d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 >> 172.21.15.144:0/1319041548 conn(0x7fa7e80a5d40 msgr2=0x7fa7e80a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 shutdown_connections 2024-09-18T12:27:47.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.722+0000 7fa7ed6a8640 1 -- 172.21.15.144:0/1319041548 wait complete. 2024-09-18T12:27:47.732 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-18T12:27:47.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.809+0000 7f48330dd640 10 monclient: get_monmap_and_config 2024-09-18T12:27:47.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.809+0000 7f48330dd640 10 monclient: build_initial_monmap 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.809+0000 7f48330dd640 10 monclient: monmap: 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:47.810372+0000 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:47.810372+0000 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:47.811 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:47.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 1 Processor -- start 2024-09-18T12:27:47.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 1 -- start start 2024-09-18T12:27:47.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient: init 2024-09-18T12:27:47.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:47.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T12:27:47.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f482c131c90 0x7f482c132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:47.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient(hunting): picked mon.noname-a con 0x7f482c131c90 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f482c09d210 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c1325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient(hunting): picked mon.noname-b con 0x7f482c0a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c0a1800 con 0x7f482c09d210 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c09fea0 con 0x7f482c131c90 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0a1520 con 0x7f482c0a0ab0 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f48330dd640 10 monclient(hunting): authenticate will time out at 903.379734s 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f482bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f482c09d210 0x7f482c068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49522/0 (socket says 172.21.15.144:49522) 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f482bfff640 1 -- 172.21.15.144:0/590892155 learned_addr learned my addr 172.21.15.144:0/590892155 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f4830e79640 1 -- 172.21.15.144:0/590892155 >> v2:172.21.15.116:3300/0 conn(0x7f482c131c90 msgr2=0x7f482c132060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:47.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f4830e79640 1 --2- 172.21.15.144:0/590892155 >> v2:172.21.15.116:3300/0 conn(0x7f482c131c90 0x7f482c132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.810+0000 7f483167a640 1 --2- 172.21.15.144:0/590892155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c1325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): get_auth_request con 0x7f482c0a0ab0 auth_method 0 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient(hunting): handle_auth_done global_id 44252 payload 274 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 1 -- 172.21.15.144:0/590892155 >> v2:172.21.15.116:3300/0 conn(0x7f482c131c90 msgr2=0x7f482c132060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 1 --2- 172.21.15.144:0/590892155 >> v2:172.21.15.116:3300/0 conn(0x7f482c131c90 0x7f482c132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 1 -- 172.21.15.144:0/590892155 >> v1:172.21.15.144:6789/0 conn(0x7f482c09d210 legacy=0x7f482c068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:47.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient: _finish_hunting 0 2024-09-18T12:27:47.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 1 monclient: found mon.noname-b 2024-09-18T12:27:47.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:47.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:47.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.811+0000 7f483167a640 1 -- 172.21.15.144:0/590892155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c069510 con 0x7f482c0a0ab0 2024-09-18T12:27:47.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f483167a640 1 -- 172.21.15.144:0/590892155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 msgr2=0x7f482c1325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:47.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f483167a640 1 --2- 172.21.15.144:0/590892155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c1325a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f482c076b50 tx=0x7f4824009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b723d9da48f708b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:47.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 1 -- 172.21.15.144:0/590892155 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824007840 con 0x7f482c0a0ab0 2024-09-18T12:27:47.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:47.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:47.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: dump: 2024-09-18T12:27:47.824 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:47.824 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:47.824 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 1 -- 172.21.15.144:0/590892155 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482402d050 con 0x7f482c0a0ab0 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 5 monclient: authenticate success, global_id 44252 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:47.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 1 -- 172.21.15.144:0/590892155 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824007c10 con 0x7f482c0a0ab0 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 4 set_mon_vals no callback set 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f482b7fe640 10 monclient: dump: 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:47.826 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 10 monclient: shutdown 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 -- 172.21.15.144:0/590892155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 msgr2=0x7f482c1325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 --2- 172.21.15.144:0/590892155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c1325a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f482c076b50 tx=0x7f4824009780 comp rx=0 tx=0).stop 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 -- 172.21.15.144:0/590892155 shutdown_connections 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 --2- 172.21.15.144:0/590892155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c1325a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 --2- 172.21.15.144:0/590892155 >> v2:172.21.15.116:3300/0 conn(0x7f482c131c90 0x7f482c132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 -- 172.21.15.144:0/590892155 >> 172.21.15.144:0/590892155 conn(0x7f482c071370 msgr2=0x7f482c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:47.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 -- 172.21.15.144:0/590892155 shutdown_connections 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.814+0000 7f48330dd640 1 -- 172.21.15.144:0/590892155 wait complete. 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.815+0000 7f48330dd640 10 monclient: build_initial_monmap 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.815+0000 7f48330dd640 10 monclient: monmap: 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:47.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.815+0000 7f48330dd640 1 Processor -- start 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.815+0000 7f48330dd640 1 -- start start 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.815+0000 7f48330dd640 10 monclient: init 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): picked mon.noname-c con 0x7f482c0a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c131c90 0x7f482c1418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): picked mon.noname-a con 0x7f482c131c90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 0x7f482c11e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): picked mon.noname-b con 0x7f482c11e2f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:47.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0a0fb0 con 0x7f482c131c90 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c09ef30 con 0x7f482c11e2f0 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c120a00 con 0x7f482c0a0ab0 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f48330dd640 10 monclient(hunting): authenticate will time out at 903.385438s 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f482bfff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c131c90 msgr2=0x7f482c1418c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f482bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c131c90 0x7f482c1418c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 0x7f482c11e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 0x7f482c11e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49924/0 (socket says 172.21.15.144:49924) 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 -- 172.21.15.144:0/774907918 learned_addr learned my addr 172.21.15.144:0/774907918 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): get_auth_request con 0x7f482c11e2f0 auth_method 0 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:47.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f4830e79640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f4830e79640 10 monclient(hunting): get_auth_request con 0x7f482c0a0ab0 auth_method 0 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f4830e79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f4830e79640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f4830e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient(hunting): handle_auth_done global_id 4792 payload 824 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 -- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 msgr2=0x7f482c141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 -- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c131c90 msgr2=0x7f482c1418c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c131c90 0x7f482c1418c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient: _finish_hunting 0 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 monclient: found mon.noname-b 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f483167a640 1 -- 172.21.15.144:0/774907918 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824009a20 con 0x7f482c11e2f0 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.816+0000 7f4830e79640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f483167a640 1 -- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 msgr2=0x7f482c11e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f483167a640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 0x7f482c11e6c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f48240099f0 tx=0x7f4824037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824031cd0 con 0x7f482c11e2f0 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:47.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: dump: 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: _finish_auth 0 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f48330dd640 5 monclient: authenticate success, global_id 4792 2024-09-18T12:27:47.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824040070 con 0x7f482c11e2f0 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f48330dd640 10 monclient: _renew_subs 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f48330dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c121960 con 0x7f482c11e2f0 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824045640 con 0x7f482c11e2f0 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f4828ff9640 10 monclient: dump: 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:47.834 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f48330dd640 10 monclient: _renew_subs 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f48330dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c120740 con 0x7f482c11e2f0 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.819+0000 7f482affd640 4 set_mon_vals no callback set 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4824003020 con 0x7f482c11e2f0 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f48330dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f48330dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f48330dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:47.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c0008d0 con 0x7f482c11e2f0 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f4828ff9640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4810038440 0x7f481003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4767+0+0 (secure 0 0 0) 0x7f482406d5c0 con 0x7f482c11e2f0 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f4830e79640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4810038440 0x7f481003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.820+0000 7f4830e79640 10 monclient: get_auth_request con 0x7f4810038440 auth_method 0 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.822+0000 7f4830e79640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4810038440 0x7f481003a8f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4820004fd0 tx=0x7f4820004980 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.824+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4824031e30 con 0x7f482c11e2f0 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.824+0000 7f4828ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:47.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.824+0000 7f4828ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:47.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.981+0000 7f48330dd640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T12:27:47.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.981+0000 7f48330dd640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T12:27:47.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.981+0000 7f48330dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:47.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:47.981+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f482c069810 con 0x7f482c11e2f0 2024-09-18T12:27:48.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.627+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v55) v1 ==== 150+0+0 (secure 0 0 0) 0x7f48240393c0 con 0x7f482c11e2f0 2024-09-18T12:27:48.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.627+0000 7f4828ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T12:27:48.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.627+0000 7f4828ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:48.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.722+0000 7f48330dd640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T12:27:48.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.722+0000 7f48330dd640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T12:27:48.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.722+0000 7f48330dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:48.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.722+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f482c069810 con 0x7f482c11e2f0 2024-09-18T12:27:48.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.723+0000 7f4828ff9640 1 -- 172.21.15.144:0/774907918 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v55) v1 ==== 177+0+0 (secure 0 0 0) 0x7f482403a3a0 con 0x7f482c11e2f0 2024-09-18T12:27:48.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.723+0000 7f4828ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T12:27:48.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.723+0000 7f4828ff9640 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-18T12:27:48.725 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo-hdd already exists 2024-09-18T12:27:48.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4810038440 msgr2=0x7f481003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:48.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4810038440 0x7f481003a8f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4820004fd0 tx=0x7f4820004980 comp rx=0 tx=0).stop 2024-09-18T12:27:48.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 10 monclient: shutdown 2024-09-18T12:27:48.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:48.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 msgr2=0x7f482c11e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:48.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 0x7f482c11e6c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f48240099f0 tx=0x7f4824037000 comp rx=0 tx=0).stop 2024-09-18T12:27:48.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 shutdown_connections 2024-09-18T12:27:48.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4810038440 0x7f481003a8f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f482c11e2f0 0x7f482c11e6c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c131c90 0x7f482c1418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 --2- 172.21.15.144:0/774907918 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f482c0a0ab0 0x7f482c141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 >> 172.21.15.144:0/774907918 conn(0x7f482c071370 msgr2=0x7f482c0a3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:48.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 shutdown_connections 2024-09-18T12:27:48.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.725+0000 7f48330dd640 1 -- 172.21.15.144:0/774907918 wait complete. 2024-09-18T12:27:48.735 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-18T12:27:48.735 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo-ssd 2024-09-18T12:27:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.808+0000 7f31b131d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.808+0000 7f31b131d640 10 monclient: build_initial_monmap 2024-09-18T12:27:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.808+0000 7f31b131d640 10 monclient: monmap: 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:48.809303+0000 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:48.809303+0000 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:48.810 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:48.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 1 Processor -- start 2024-09-18T12:27:48.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 1 -- start start 2024-09-18T12:27:48.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient: init 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient(hunting): picked mon.noname-b con 0x7f31ac09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31ac1321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31ac09f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ac067420 con 0x7f31ac1321a0 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ac09fdc0 con 0x7f31ac09f020 2024-09-18T12:27:48.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac0a0510 con 0x7f31ac09ec50 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31b131d640 10 monclient(hunting): authenticate will time out at 904.378704s 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31ab7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f31ac09f020 0x7f31ac069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49524/0 (socket says 172.21.15.144:49524) 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31a3fff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f31ac1321a0 legacy=0x7f31ac068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31ab7fe640 1 -- 172.21.15.144:0/1658766155 learned_addr learned my addr 172.21.15.144:0/1658766155 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31aaffd640 1 --2- 172.21.15.144:0/1658766155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31aaffd640 10 monclient(hunting): get_auth_request con 0x7f31ac09ec50 auth_method 0 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:48.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.809+0000 7f31aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 10 monclient(hunting): handle_auth_done global_id 44258 payload 274 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 1 -- 172.21.15.144:0/1658766155 >> v1:172.21.15.144:6789/0 conn(0x7f31ac09f020 legacy=0x7f31ac069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 1 -- 172.21.15.144:0/1658766155 >> v1:172.21.15.116:6789/0 conn(0x7f31ac1321a0 legacy=0x7f31ac068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 1 monclient: found mon.noname-b 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:27:48.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.810+0000 7f31aaffd640 1 -- 172.21.15.144:0/1658766155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac0a0200 con 0x7f31ac09ec50 2024-09-18T12:27:48.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aaffd640 1 -- 172.21.15.144:0/1658766155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 msgr2=0x7f31ac131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:48.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aaffd640 1 --2- 172.21.15.144:0/1658766155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac131c60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f319c009880 tx=0x7f319c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8898255157159411 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:48.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 1 -- 172.21.15.144:0/1658766155 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c02aea0 con 0x7f31ac09ec50 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: dump: 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:48.823 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 5 monclient: authenticate success, global_id 44258 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 1 -- 172.21.15.144:0/1658766155 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c032440 con 0x7f31ac09ec50 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 4 set_mon_vals no callback set 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 1 -- 172.21.15.144:0/1658766155 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c032820 con 0x7f31ac09ec50 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:48.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31aa7fc640 10 monclient: dump: 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:48.825 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 10 monclient: shutdown 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 1 -- 172.21.15.144:0/1658766155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 msgr2=0x7f31ac131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 1 --2- 172.21.15.144:0/1658766155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac131c60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f319c009880 tx=0x7f319c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 1 -- 172.21.15.144:0/1658766155 shutdown_connections 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.813+0000 7f31b131d640 1 --2- 172.21.15.144:0/1658766155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac131c60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 1 -- 172.21.15.144:0/1658766155 >> 172.21.15.144:0/1658766155 conn(0x7f31ac0a5d40 msgr2=0x7f31ac0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 1 -- 172.21.15.144:0/1658766155 shutdown_connections 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 1 -- 172.21.15.144:0/1658766155 wait complete. 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 10 monclient: build_initial_monmap 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 10 monclient: monmap: 2024-09-18T12:27:48.826 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 1 Processor -- start 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 1 -- start start 2024-09-18T12:27:48.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.814+0000 7f31b131d640 10 monclient: init 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac09aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): picked mon.noname-c con 0x7f31ac09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac09f020 0x7f31ac130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): picked mon.noname-a con 0x7f31ac09f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 0x7f31ac131170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): picked mon.noname-b con 0x7f31ac1321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:48.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac11ca60 con 0x7f31ac09f020 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac09c000 con 0x7f31ac1321a0 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac09df10 con 0x7f31ac09ec50 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31b131d640 10 monclient(hunting): authenticate will time out at 904.384269s 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 0x7f31ac131170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31a3fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac09f020 msgr2=0x7f31ac130c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 0x7f31ac131170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49930/0 (socket says 172.21.15.144:49930) 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 -- 172.21.15.144:0/1563744903 learned_addr learned my addr 172.21.15.144:0/1563744903 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:48.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): get_auth_request con 0x7f31ac1321a0 auth_method 0 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31a3fff640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac09f020 0x7f31ac130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31aaffd640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac09aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient(hunting): handle_auth_done global_id 4795 payload 824 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 -- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 msgr2=0x7f31ac09aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:48.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac09aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 -- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac09f020 msgr2=0x7f31ac130c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac09f020 0x7f31ac130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.815+0000 7f31ab7fe640 1 -- 172.21.15.144:0/1563744903 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c009390 con 0x7f31ac1321a0 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.817+0000 7f31ab7fe640 1 -- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 msgr2=0x7f31ac131170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.817+0000 7f31ab7fe640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 0x7f31ac131170 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f31a400ccc0 tx=0x7f31a4005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:48.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.817+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a400ce80 con 0x7f31ac1321a0 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: dump: 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:48.832 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a400aa60 con 0x7f31ac1321a0 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 5 monclient: authenticate success, global_id 4795 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 10 monclient: _renew_subs 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac11dd50 con 0x7f31ac1321a0 2024-09-18T12:27:48.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a400ae20 con 0x7f31ac1321a0 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 10 monclient: dump: 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:48.834 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a9ffb640 4 set_mon_vals no callback set 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 10 monclient: _renew_subs 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac11cdf0 con 0x7f31ac1321a0 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f31a4007300 con 0x7f31ac1321a0 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:48.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:48.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.818+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ac11e230 con 0x7f31ac1321a0 2024-09-18T12:27:48.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.819+0000 7f31a2ffd640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f318c038440 0x7f318c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:48.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.819+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f31a404bac0 con 0x7f31ac1321a0 2024-09-18T12:27:48.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.819+0000 7f31aaffd640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f318c038440 0x7f318c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:48.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.819+0000 7f31aaffd640 10 monclient: get_auth_request con 0x7f318c038440 auth_method 0 2024-09-18T12:27:48.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.821+0000 7f31aaffd640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f318c038440 0x7f318c03a8f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f319c034000 tx=0x7f319c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:48.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.823+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f31a401b400 con 0x7f31ac1321a0 2024-09-18T12:27:48.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.823+0000 7f31a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:48.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.823+0000 7f31a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:48.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.980+0000 7f31b131d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:48.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.980+0000 7f31b131d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:48.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.980+0000 7f31b131d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:48.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.980+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f31ac0008d0 con 0x7f31ac1321a0 2024-09-18T12:27:48.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.980+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7f31a4010af0 con 0x7f31ac1321a0 2024-09-18T12:27:48.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.980+0000 7f31a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:48.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:48.980+0000 7f31a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:49.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.076+0000 7f31b131d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.076+0000 7f31b131d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.076+0000 7f31b131d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.076+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f31ac0a13e0 con 0x7f31ac1321a0 2024-09-18T12:27:49.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.077+0000 7f31a2ffd640 1 -- 172.21.15.144:0/1563744903 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7f31a4010cd0 con 0x7f31ac1321a0 2024-09-18T12:27:49.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.077+0000 7f31a2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.077+0000 7f31a2ffd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:49.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.078+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f318c038440 msgr2=0x7f318c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.078+0000 7f31b131d640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f318c038440 0x7f318c03a8f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f319c034000 tx=0x7f319c032000 comp rx=0 tx=0).stop 2024-09-18T12:27:49.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.078+0000 7f31b131d640 10 monclient: shutdown 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.078+0000 7f31b131d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.078+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 msgr2=0x7f31ac131170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.078+0000 7f31b131d640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 0x7f31ac131170 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f31a400ccc0 tx=0x7f31a4005370 comp rx=0 tx=0).stop 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 shutdown_connections 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f318c038440 0x7f318c03a8f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31ac1321a0 0x7f31ac131170 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31ac09f020 0x7f31ac130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 --2- 172.21.15.144:0/1563744903 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31ac09ec50 0x7f31ac09aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 >> 172.21.15.144:0/1563744903 conn(0x7f31ac0a5d40 msgr2=0x7f31ac070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 shutdown_connections 2024-09-18T12:27:49.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.079+0000 7f31b131d640 1 -- 172.21.15.144:0/1563744903 wait complete. 2024-09-18T12:27:49.088 INFO:tasks.workunit.client.0.smithi144.stdout:foo-ssd 2024-09-18T12:27:49.089 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-18T12:27:49.089 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false grep foo-hdd 2024-09-18T12:27:49.089 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:49.089 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo-hdd 2024-09-18T12:27:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.163+0000 7f0a6ea60640 10 monclient: get_monmap_and_config 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.163+0000 7f0a6ea60640 10 monclient: build_initial_monmap 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.163+0000 7f0a6ea60640 10 monclient: monmap: 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:49.163942+0000 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:49.163942+0000 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:49.166 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:49.166 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.163+0000 7f0a6ea60640 1 Processor -- start 2024-09-18T12:27:49.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.163+0000 7f0a6ea60640 1 -- start start 2024-09-18T12:27:49.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.163+0000 7f0a6ea60640 10 monclient: init 2024-09-18T12:27:49.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a6809ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0a6809f020 0x7f0a68069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6809f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a68136d60 0x7f0a68139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient(hunting): picked mon.noname-a con 0x7f0a68136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a68067420 con 0x7f0a6809ec50 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6809fdc0 con 0x7f0a68136d60 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a680a0510 con 0x7f0a6809f020 2024-09-18T12:27:49.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6ea60640 10 monclient(hunting): authenticate will time out at 904.733314s 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a67fff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f0a6809ec50 legacy=0x7f0a68131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0a6809f020 0x7f0a68069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6cffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f0a68136d60 msgr2=0x7f0a68139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0a6809f020 0x7f0a68069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49940/0 (socket says 172.21.15.144:49940) 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a6cffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a68136d60 0x7f0a68139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 -- 172.21.15.144:0/2729213455 learned_addr learned my addr 172.21.15.144:0/2729213455 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): get_auth_request con 0x7f0a6809f020 auth_method 0 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:49.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient(hunting): handle_auth_done global_id 4798 payload 274 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 -- 172.21.15.144:0/2729213455 >> v2:172.21.15.116:3300/0 conn(0x7f0a68136d60 msgr2=0x7f0a68139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 --2- 172.21.15.144:0/2729213455 >> v2:172.21.15.116:3300/0 conn(0x7f0a68136d60 0x7f0a68139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 -- 172.21.15.144:0/2729213455 >> v1:172.21.15.116:6789/0 conn(0x7f0a6809ec50 legacy=0x7f0a68131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 monclient: found mon.noname-c 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.164+0000 7f0a677fe640 1 -- 172.21.15.144:0/2729213455 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a680a0200 con 0x7f0a6809f020 2024-09-18T12:27:49.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a677fe640 1 -- 172.21.15.144:0/2729213455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a6809f020 msgr2=0x7f0a68069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:49.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a677fe640 1 --2- 172.21.15.144:0/2729213455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a6809f020 0x7f0a68069d10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0a680a2b00 tx=0x7f0a5c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ae042c98700c7f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 1 -- 172.21.15.144:0/2729213455 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c032440 con 0x7f0a6809f020 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: dump: 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.177 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: _finish_auth 0 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 1 -- 172.21.15.144:0/2729213455 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c0325a0 con 0x7f0a6809f020 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a6ea60640 5 monclient: authenticate success, global_id 4798 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a6ea60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a6ea60640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 1 -- 172.21.15.144:0/2729213455 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c032980 con 0x7f0a6809f020 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a6ea60640 4 set_mon_vals no callback set 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.167+0000 7f0a66ffd640 10 monclient: dump: 2024-09-18T12:27:49.178 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 10 monclient: shutdown 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2729213455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a6809f020 msgr2=0x7f0a68069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2729213455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a6809f020 0x7f0a68069d10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0a680a2b00 tx=0x7f0a5c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2729213455 shutdown_connections 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2729213455 >> v2:172.21.15.116:3300/0 conn(0x7f0a68136d60 0x7f0a68139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2729213455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a6809f020 0x7f0a68069d10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2729213455 >> 172.21.15.144:0/2729213455 conn(0x7f0a680a5d40 msgr2=0x7f0a680a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2729213455 shutdown_connections 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2729213455 wait complete. 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 10 monclient: build_initial_monmap 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.168+0000 7f0a6ea60640 10 monclient: monmap: 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:49.180 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 Processor -- start 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 -- start start 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient: init 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6809ec50 0x7f0a68097460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): picked mon.noname-a con 0x7f0a6809ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6809f020 0x7f0a680979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 0x7f0a68097ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): picked mon.noname-b con 0x7f0a68136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a68099040 con 0x7f0a6809ec50 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6809add0 con 0x7f0a68136d60 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6809b070 con 0x7f0a6809f020 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6ea60640 10 monclient(hunting): authenticate will time out at 904.738813s 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6cffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 0x7f0a68097ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a67fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6809ec50 msgr2=0x7f0a68097460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6809ec50 0x7f0a68097460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:49.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6cffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 0x7f0a68097ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49950/0 (socket says 172.21.15.144:49950) 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6cffd640 1 -- 172.21.15.144:0/2145552966 learned_addr learned my addr 172.21.15.144:0/2145552966 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6cffd640 10 monclient(hunting): get_auth_request con 0x7f0a68136d60 auth_method 0 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6cffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6cffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.169+0000 7f0a6cffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6809f020 0x7f0a680979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 10 monclient(hunting): get_auth_request con 0x7f0a6809f020 auth_method 0 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:49.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 10 monclient(hunting): handle_auth_done global_id 4801 payload 824 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 1 -- 172.21.15.144:0/2145552966 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6809f020 msgr2=0x7f0a680979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6809f020 0x7f0a680979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 1 -- 172.21.15.144:0/2145552966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6809ec50 msgr2=0x7f0a68097460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6809ec50 0x7f0a68097460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 10 monclient: _finish_hunting 0 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 1 monclient: found mon.noname-b 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.170+0000 7f0a6cffd640 1 -- 172.21.15.144:0/2145552966 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c009450 con 0x7f0a68136d60 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6cffd640 1 -- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 msgr2=0x7f0a68097ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:49.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6cffd640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 0x7f0a68097ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000bb10 tx=0x7f0a600097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000cbc0 con 0x7f0a68136d60 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: dump: 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.185 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: _finish_auth 0 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6ea60640 5 monclient: authenticate success, global_id 4801 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6000cd40 con 0x7f0a68136d60 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6ea60640 10 monclient: _renew_subs 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6ea60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6809b720 con 0x7f0a68136d60 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60006150 con 0x7f0a68136d60 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a4ffff640 10 monclient: dump: 2024-09-18T12:27:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6ea60640 10 monclient: _renew_subs 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6ea60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a68099440 con 0x7f0a68136d60 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.172+0000 7f0a667fc640 4 set_mon_vals no callback set 2024-09-18T12:27:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.173+0000 7f0a6ea60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.173+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0a60012020 con 0x7f0a68136d60 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.173+0000 7f0a6ea60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.173+0000 7f0a6ea60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.173+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a680008d0 con 0x7f0a68136d60 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.173+0000 7f0a4ffff640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0a54038440 0x7f0a5403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.174+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f0a6004b710 con 0x7f0a68136d60 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.174+0000 7f0a67fff640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0a54038440 0x7f0a5403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.174+0000 7f0a67fff640 10 monclient: get_auth_request con 0x7f0a54038440 auth_method 0 2024-09-18T12:27:49.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.176+0000 7f0a67fff640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0a54038440 0x7f0a5403a8f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0a580038f0 tx=0x7f0a58006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.178+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0a6001da80 con 0x7f0a68136d60 2024-09-18T12:27:49.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.178+0000 7f0a4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.178+0000 7f0a4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:49.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.334+0000 7f0a6ea60640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.334+0000 7f0a6ea60640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.334+0000 7f0a6ea60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.334+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f0a680a13e0 con 0x7f0a68136d60 2024-09-18T12:27:49.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.335+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7f0a6001da80 con 0x7f0a68136d60 2024-09-18T12:27:49.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.335+0000 7f0a4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.335+0000 7f0a4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:49.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.429+0000 7f0a6ea60640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.429+0000 7f0a6ea60640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.429+0000 7f0a6ea60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.429+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f0a68134180 con 0x7f0a68136d60 2024-09-18T12:27:49.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.429+0000 7f0a4ffff640 1 -- 172.21.15.144:0/2145552966 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7f0a6001da80 con 0x7f0a68136d60 2024-09-18T12:27:49.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.429+0000 7f0a4ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T12:27:49.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.429+0000 7f0a4ffff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:49.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0a54038440 msgr2=0x7f0a5403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0a54038440 0x7f0a5403a8f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0a580038f0 tx=0x7f0a58006d80 comp rx=0 tx=0).stop 2024-09-18T12:27:49.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 10 monclient: shutdown 2024-09-18T12:27:49.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:49.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 msgr2=0x7f0a68097ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 0x7f0a68097ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000bb10 tx=0x7f0a600097d0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 shutdown_connections 2024-09-18T12:27:49.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0a54038440 0x7f0a5403a8f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0a68136d60 0x7f0a68097ee0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0a6809f020 0x7f0a680979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 --2- 172.21.15.144:0/2145552966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6809ec50 0x7f0a68097460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 >> 172.21.15.144:0/2145552966 conn(0x7f0a680a5d40 msgr2=0x7f0a68068c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:49.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 shutdown_connections 2024-09-18T12:27:49.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.431+0000 7f0a6ea60640 1 -- 172.21.15.144:0/2145552966 wait complete. 2024-09-18T12:27:49.441 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:49.442 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-18T12:27:49.442 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo-hdd 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.514+0000 7fada9354640 10 monclient: get_monmap_and_config 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.514+0000 7fada9354640 10 monclient: build_initial_monmap 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.515+0000 7fada9354640 10 monclient: monmap: 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:49.515919+0000 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:49.515919+0000 2024-09-18T12:27:49.516 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:49.517 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.517 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:49.517 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:49.517 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:49.517 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:49.517 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:49.517 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.515+0000 7fada9354640 1 Processor -- start 2024-09-18T12:27:49.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.515+0000 7fada9354640 1 -- start start 2024-09-18T12:27:49.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.515+0000 7fada9354640 10 monclient: init 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fada409ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fada409f020 0x7fada4069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient(hunting): picked mon.noname-c con 0x7fada409f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada4136d60 0x7fada4139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient(hunting): picked mon.noname-b con 0x7fada4136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4067420 con 0x7fada409ec50 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada409fdc0 con 0x7fada409f020 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada40a0510 con 0x7fada4136d60 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada9354640 10 monclient(hunting): authenticate will time out at 905.085231s 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fada409f020 0x7fada4069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada2ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fada409ec50 0x7fada4131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49530/0 (socket says 172.21.15.144:49530) 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): get_auth_request con 0x7fada409f020 auth_method 0 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada2ffd640 1 -- 172.21.15.144:0/1470457605 learned_addr learned my addr 172.21.15.144:0/1470457605 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:49.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient(hunting): handle_auth_done global_id 4804 payload 274 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 1 -- 172.21.15.144:0/1470457605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada4136d60 msgr2=0x7fada4139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 1 --2- 172.21.15.144:0/1470457605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada4136d60 0x7fada4139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 1 -- 172.21.15.144:0/1470457605 >> v1:172.21.15.144:6789/0 conn(0x7fada409ec50 legacy=0x7fada4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 1 monclient: found mon.noname-c 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.516+0000 7fada27fc640 1 -- 172.21.15.144:0/1470457605 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada40a0200 con 0x7fada409f020 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada27fc640 1 -- 172.21.15.144:0/1470457605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada409f020 msgr2=0x7fada4069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada27fc640 1 --2- 172.21.15.144:0/1470457605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada409f020 0x7fada4069d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009850 tx=0x7fad8c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e669a0a527c09d2d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 1 -- 172.21.15.144:0/1470457605 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c032440 con 0x7fada409f020 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: dump: 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.528 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 1 -- 172.21.15.144:0/1470457605 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c0325a0 con 0x7fada409f020 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada9354640 5 monclient: authenticate success, global_id 4804 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada9354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:49.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 1 -- 172.21.15.144:0/1470457605 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c032940 con 0x7fada409f020 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada9354640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada9354640 4 set_mon_vals no callback set 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.519+0000 7fada1ffb640 10 monclient: dump: 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.530 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 10 monclient: shutdown 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 -- 172.21.15.144:0/1470457605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada409f020 msgr2=0x7fada4069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 --2- 172.21.15.144:0/1470457605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada409f020 0x7fada4069d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009850 tx=0x7fad8c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 -- 172.21.15.144:0/1470457605 shutdown_connections 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 --2- 172.21.15.144:0/1470457605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada4136d60 0x7fada4139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 --2- 172.21.15.144:0/1470457605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada409f020 0x7fada4069d10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 -- 172.21.15.144:0/1470457605 >> 172.21.15.144:0/1470457605 conn(0x7fada40a5d40 msgr2=0x7fada40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 -- 172.21.15.144:0/1470457605 shutdown_connections 2024-09-18T12:27:49.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 1 -- 172.21.15.144:0/1470457605 wait complete. 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 10 monclient: build_initial_monmap 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.520+0000 7fada9354640 10 monclient: monmap: 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 Processor -- start 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 -- start start 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient: init 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada409ec50 0x7fada409b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): picked mon.noname-c con 0x7fada409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fada409f020 0x7fada4097190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): picked mon.noname-a con 0x7fada409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 0x7fada40976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): picked mon.noname-b con 0x7fada4136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada409c000 con 0x7fada409f020 2024-09-18T12:27:49.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada40a3a10 con 0x7fada4136d60 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada409b9e0 con 0x7fada409ec50 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada9354640 10 monclient(hunting): authenticate will time out at 905.090577s 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 0x7fada40976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 0x7fada40976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49960/0 (socket says 172.21.15.144:49960) 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 -- 172.21.15.144:0/1287876019 learned_addr learned my addr 172.21.15.144:0/1287876019 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): get_auth_request con 0x7fada4136d60 auth_method 0 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada27fc640 1 -- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fada409f020 msgr2=0x7fada4097190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:49.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada27fc640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fada409f020 0x7fada4097190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient(hunting): handle_auth_done global_id 4807 payload 824 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 -- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada409ec50 msgr2=0x7fada409b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada409ec50 0x7fada409b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 -- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fada409f020 msgr2=0x7fada4097190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fada409f020 0x7fada4097190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.521+0000 7fada37fe640 1 -- 172.21.15.144:0/1287876019 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c009390 con 0x7fada4136d60 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada37fe640 1 -- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 msgr2=0x7fada40976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada37fe640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 0x7fada40976d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00bc30 tx=0x7fad9c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00ca70 con 0x7fada4136d60 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: dump: 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.536 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: _finish_auth 0 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c00cbd0 con 0x7fada4136d60 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada9354640 5 monclient: authenticate success, global_id 4807 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada9354640 10 monclient: _renew_subs 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada9354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada409a7e0 con 0x7fada4136d60 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c015470 con 0x7fada4136d60 2024-09-18T12:27:49.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fad9affd640 10 monclient: dump: 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada9354640 10 monclient: _renew_subs 2024-09-18T12:27:49.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada9354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada17fa640 4 set_mon_vals no callback set 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.524+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada4099880 con 0x7fada4136d60 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fad9c012070 con 0x7fada4136d60 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fada9354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fada9354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fada9354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada4099ad0 con 0x7fada4136d60 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fad9affd640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad84038440 0x7fad8403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fad9c04b530 con 0x7fada4136d60 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fada2ffd640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad84038440 0x7fad8403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.525+0000 7fada2ffd640 10 monclient: get_auth_request con 0x7fad84038440 auth_method 0 2024-09-18T12:27:49.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.527+0000 7fada2ffd640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad84038440 0x7fad8403a8f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad94004fe0 tx=0x7fad94007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.529+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fada4099ad0 con 0x7fada4136d60 2024-09-18T12:27:49.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.529+0000 7fad9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.529+0000 7fad9affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:49.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.687+0000 7fada9354640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:49.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.687+0000 7fada9354640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:49.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.687+0000 7fada9354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.687+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fada4099ad0 con 0x7fada4136d60 2024-09-18T12:27:49.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.688+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7fada4099ad0 con 0x7fada4136d60 2024-09-18T12:27:49.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.688+0000 7fad9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:49.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.688+0000 7fad9affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.781+0000 7fada9354640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.781+0000 7fada9354640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.781+0000 7fada9354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.781+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fada4099ad0 con 0x7fada4136d60 2024-09-18T12:27:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.781+0000 7fad9affd640 1 -- 172.21.15.144:0/1287876019 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7fada4099ad0 con 0x7fada4136d60 2024-09-18T12:27:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.781+0000 7fad9affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.781+0000 7fad9affd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:49.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad84038440 msgr2=0x7fad8403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad84038440 0x7fad8403a8f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad94004fe0 tx=0x7fad94007b80 comp rx=0 tx=0).stop 2024-09-18T12:27:49.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 10 monclient: shutdown 2024-09-18T12:27:49.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:49.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 msgr2=0x7fada40976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 0x7fada40976d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fad9c00bc30 tx=0x7fad9c009840 comp rx=0 tx=0).stop 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 shutdown_connections 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad84038440 0x7fad8403a8f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fada4136d60 0x7fada40976d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fada409f020 0x7fada4097190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 --2- 172.21.15.144:0/1287876019 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fada409ec50 0x7fada409b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.783+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 >> 172.21.15.144:0/1287876019 conn(0x7fada40a5d40 msgr2=0x7fada40a44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.784+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 shutdown_connections 2024-09-18T12:27:49.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.784+0000 7fada9354640 1 -- 172.21.15.144:0/1287876019 wait complete. 2024-09-18T12:27:49.793 INFO:tasks.workunit.client.0.smithi144.stdout:foo-hdd 2024-09-18T12:27:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-18T12:27:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false grep foo-ssd 2024-09-18T12:27:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:27:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo-ssd 2024-09-18T12:27:49.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.866+0000 7f07b7f1d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:49.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.866+0000 7f07b7f1d640 10 monclient: build_initial_monmap 2024-09-18T12:27:49.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.866+0000 7f07b7f1d640 10 monclient: monmap: 2024-09-18T12:27:49.867 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:49.867 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:49.867117+0000 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:49.867117+0000 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:49.868 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.866+0000 7f07b7f1d640 1 Processor -- start 2024-09-18T12:27:49.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 1 -- start start 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient: init 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07b0131c90 0x7f07b0132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f07b0131c90 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07b009d210 0x7f07b01325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f07b009d210 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07b00a0ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b00a1800 con 0x7f07b00a0ab0 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b009fea0 con 0x7f07b0131c90 2024-09-18T12:27:49.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b00a1520 con 0x7f07b009d210 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b7f1d640 10 monclient(hunting): authenticate will time out at 905.436496s 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07b009d210 0x7f07b01325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b64ba640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f07b00a0ab0 legacy=0x7f07b0068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b5cb9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f07b0131c90 msgr2=0x7f07b0132060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07b009d210 0x7f07b01325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49964/0 (socket says 172.21.15.144:49964) 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 -- 172.21.15.144:0/1393361098 learned_addr learned my addr 172.21.15.144:0/1393361098 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): get_auth_request con 0x7f07b009d210 auth_method 0 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:49.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b5cb9640 1 --2- 172.21.15.144:0/1393361098 >> v2:172.21.15.116:3300/0 conn(0x7f07b0131c90 0x7f07b0132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient(hunting): handle_auth_done global_id 4810 payload 274 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 -- 172.21.15.144:0/1393361098 >> v2:172.21.15.116:3300/0 conn(0x7f07b0131c90 msgr2=0x7f07b0132060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 --2- 172.21.15.144:0/1393361098 >> v2:172.21.15.116:3300/0 conn(0x7f07b0131c90 0x7f07b0132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 -- 172.21.15.144:0/1393361098 >> v1:172.21.15.116:6789/0 conn(0x7f07b00a0ab0 legacy=0x7f07b0068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient: _finish_hunting 0 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 monclient: found mon.noname-c 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.867+0000 7f07b54b8640 1 -- 172.21.15.144:0/1393361098 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b0069510 con 0x7f07b009d210 2024-09-18T12:27:49.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.870+0000 7f07b54b8640 1 -- 172.21.15.144:0/1393361098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b009d210 msgr2=0x7f07b01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:49.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.870+0000 7f07b54b8640 1 --2- 172.21.15.144:0/1393361098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b009d210 0x7f07b01325a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f07b0076b50 tx=0x7f07a00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=198ee8338421d541 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 1 -- 172.21.15.144:0/1393361098 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0032440 con 0x7f07b009d210 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: dump: 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.879 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: _finish_auth 0 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 1 -- 172.21.15.144:0/1393361098 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a00325a0 con 0x7f07b009d210 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 5 monclient: authenticate success, global_id 4810 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 1 -- 172.21.15.144:0/1393361098 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0032980 con 0x7f07b009d210 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 4 set_mon_vals no callback set 2024-09-18T12:27:49.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b4cb7640 10 monclient: dump: 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 10 monclient: shutdown 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:49.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 -- 172.21.15.144:0/1393361098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b009d210 msgr2=0x7f07b01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/1393361098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b009d210 0x7f07b01325a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f07b0076b50 tx=0x7f07a00091d0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 -- 172.21.15.144:0/1393361098 shutdown_connections 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/1393361098 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b009d210 0x7f07b01325a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/1393361098 >> v2:172.21.15.116:3300/0 conn(0x7f07b0131c90 0x7f07b0132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 -- 172.21.15.144:0/1393361098 >> 172.21.15.144:0/1393361098 conn(0x7f07b0071370 msgr2=0x7f07b00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 -- 172.21.15.144:0/1393361098 shutdown_connections 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.871+0000 7f07b7f1d640 1 -- 172.21.15.144:0/1393361098 wait complete. 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.872+0000 7f07b7f1d640 10 monclient: build_initial_monmap 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.872+0000 7f07b7f1d640 10 monclient: monmap: 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:49.882 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.872+0000 7f07b7f1d640 1 Processor -- start 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.872+0000 7f07b7f1d640 1 -- start start 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.872+0000 7f07b7f1d640 10 monclient: init 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07b00a0ab0 0x7f07b01414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f07b00a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b0131c90 0x7f07b0130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f07b0131c90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 0x7f07b011eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f07b0131500 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0122120 con 0x7f07b0131c90 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b01423b0 con 0x7f07b0131500 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0120d80 con 0x7f07b00a0ab0 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b7f1d640 10 monclient(hunting): authenticate will time out at 905.442124s 2024-09-18T12:27:49.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 0x7f07b011eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b54b8640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b0131c90 msgr2=0x7f07b0130fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 0x7f07b011eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49966/0 (socket says 172.21.15.144:49966) 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 -- 172.21.15.144:0/4031051486 learned_addr learned my addr 172.21.15.144:0/4031051486 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): get_auth_request con 0x7f07b0131500 auth_method 0 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b54b8640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b0131c90 0x7f07b0130fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient(hunting): handle_auth_done global_id 4813 payload 824 2024-09-18T12:27:49.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b5cb9640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07b00a0ab0 0x7f07b01414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 -- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07b00a0ab0 msgr2=0x7f07b01414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07b00a0ab0 0x7f07b01414d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 -- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b0131c90 msgr2=0x7f07b0130fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b0131c90 0x7f07b0130fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient: _finish_hunting 0 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 monclient: found mon.noname-b 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.873+0000 7f07b64ba640 1 -- 172.21.15.144:0/4031051486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0009450 con 0x7f07b0131500 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07b64ba640 1 -- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 msgr2=0x7f07b011eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07b64ba640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 0x7f07b011eb40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f07a800bb10 tx=0x7f07a80097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a800cbc0 con 0x7f07b0131500 2024-09-18T12:27:49.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: dump: 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a800cd40 con 0x7f07b0131500 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07b7f1d640 5 monclient: authenticate success, global_id 4813 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8006150 con 0x7f07b0131500 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07b7f1d640 10 monclient: _renew_subs 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07b7f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b00a0fb0 con 0x7f07b0131500 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a5ffb640 10 monclient: dump: 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:49.888 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.875+0000 7f07a7fff640 4 set_mon_vals no callback set 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.876+0000 7f07b7f1d640 10 monclient: _renew_subs 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.876+0000 7f07b7f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.876+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b01211c0 con 0x7f07b0131500 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.876+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f07a8012020 con 0x7f07b0131500 2024-09-18T12:27:49.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07a5ffb640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0794038470 0x7f079403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:49.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f07a804b780 con 0x7f07b0131500 2024-09-18T12:27:49.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07b7f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07b7f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07b7f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:49.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07b5cb9640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0794038470 0x7f079403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:49.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b00008d0 con 0x7f07b0131500 2024-09-18T12:27:49.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.877+0000 7f07b5cb9640 10 monclient: get_auth_request con 0x7f0794038470 auth_method 0 2024-09-18T12:27:49.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.879+0000 7f07b5cb9640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0794038470 0x7f079403a920 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f07ac006fd0 tx=0x7f07ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:49.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.881+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f07a801b070 con 0x7f07b0131500 2024-09-18T12:27:49.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.881+0000 7f07a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:49.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:49.881+0000 7f07a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:50.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.037+0000 7f07b7f1d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:50.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.037+0000 7f07b7f1d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:50.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.037+0000 7f07b7f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:50.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.037+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f07b009ba50 con 0x7f07b0131500 2024-09-18T12:27:50.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.037+0000 7f07b64ba640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 0x7f07b011eb40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f07a800bb10 tx=0x7f07a80097d0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f07a801d4e0 mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) v1 2024-09-18T12:27:50.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.866+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7f07a801d4e0 con 0x7f07b0131500 2024-09-18T12:27:50.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.866+0000 7f07a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:50.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.866+0000 7f07a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:50.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.964+0000 7f07b7f1d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:50.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.964+0000 7f07b7f1d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:50.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.964+0000 7f07b7f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:50.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.964+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7f07b00a55b0 con 0x7f07b0131500 2024-09-18T12:27:50.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.964+0000 7f07a5ffb640 1 -- 172.21.15.144:0/4031051486 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) v1 ==== 90+0+8 (secure 0 0 0) 0x7f07a801d300 con 0x7f07b0131500 2024-09-18T12:27:50.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.964+0000 7f07a5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T12:27:50.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.964+0000 7f07a5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0794038470 msgr2=0x7f079403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0794038470 0x7f079403a920 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f07ac006fd0 tx=0x7f07ac008040 comp rx=0 tx=0).stop 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 10 monclient: shutdown 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 msgr2=0x7f07b011eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 0x7f07b011eb40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f07a800bb10 tx=0x7f07a80097d0 comp rx=0 tx=0).stop 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 shutdown_connections 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0794038470 0x7f079403a920 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07b0131500 0x7f07b011eb40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:50.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b0131c90 0x7f07b0130fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:50.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 --2- 172.21.15.144:0/4031051486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07b00a0ab0 0x7f07b01414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:50.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 >> 172.21.15.144:0/4031051486 conn(0x7f07b0071370 msgr2=0x7f07b00a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:50.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 shutdown_connections 2024-09-18T12:27:50.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:50.966+0000 7f07b7f1d640 1 -- 172.21.15.144:0/4031051486 wait complete. 2024-09-18T12:27:50.977 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:27:51.022 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-18T12:27:51.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.053+0000 7f9dbfeaf640 10 monclient: get_monmap_and_config 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.053+0000 7f9dbfeaf640 10 monclient: build_initial_monmap 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.053+0000 7f9dbfeaf640 10 monclient: monmap: 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:51.054669+0000 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:51.054669+0000 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:51.055 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:51.056 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:51.056 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:51.056 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.054+0000 7f9dbfeaf640 1 Processor -- start 2024-09-18T12:27:51.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.054+0000 7f9dbfeaf640 1 -- start start 2024-09-18T12:27:51.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.054+0000 7f9dbfeaf640 10 monclient: init 2024-09-18T12:27:51.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:51.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9db809ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9db809f020 0x7f9db8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9db809f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9db8136d60 0x7f9db8139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9db8136d60 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db8067420 con 0x7f9db809ec50 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db809fdc0 con 0x7f9db809f020 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db80a0510 con 0x7f9db8136d60 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbfeaf640 10 monclient(hunting): authenticate will time out at 906.624136s 2024-09-18T12:27:51.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9db8136d60 0x7f9db8139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbdc4b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f9db809ec50 0x7f9db8131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49544/0 (socket says 172.21.15.144:49544) 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbd44a640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f9db809f020 msgr2=0x7f9db8069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbdc4b640 1 -- 172.21.15.144:0/1789966629 learned_addr learned my addr 172.21.15.144:0/1789966629 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): get_auth_request con 0x7f9db8136d60 auth_method 0 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbd44a640 1 --2- 172.21.15.144:0/1789966629 >> v2:172.21.15.116:3300/0 conn(0x7f9db809f020 0x7f9db8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient(hunting): handle_auth_done global_id 4816 payload 274 2024-09-18T12:27:51.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 1 -- 172.21.15.144:0/1789966629 >> v2:172.21.15.116:3300/0 conn(0x7f9db809f020 msgr2=0x7f9db8069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:51.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 1 --2- 172.21.15.144:0/1789966629 >> v2:172.21.15.116:3300/0 conn(0x7f9db809f020 0x7f9db8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 1 -- 172.21.15.144:0/1789966629 >> v1:172.21.15.144:6789/0 conn(0x7f9db809ec50 legacy=0x7f9db8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:51.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient: _finish_hunting 0 2024-09-18T12:27:51.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 1 monclient: found mon.noname-c 2024-09-18T12:27:51.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:51.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.055+0000 7f9dbe44c640 1 -- 172.21.15.144:0/1789966629 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80a0200 con 0x7f9db8136d60 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbe44c640 1 -- 172.21.15.144:0/1789966629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db8136d60 msgr2=0x7f9db8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbe44c640 1 --2- 172.21.15.144:0/1789966629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db8136d60 0x7f9db8139140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9db8132a50 tx=0x7f9db40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=274e47e46b6b612b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 1 -- 172.21.15.144:0/1789966629 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db4032440 con 0x7f9db8136d60 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: dump: 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.067 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: _finish_auth 0 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 1 -- 172.21.15.144:0/1789966629 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db40325a0 con 0x7f9db8136d60 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 5 monclient: authenticate success, global_id 4816 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 1 -- 172.21.15.144:0/1789966629 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db4032940 con 0x7f9db8136d60 2024-09-18T12:27:51.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbcc49640 10 monclient: dump: 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 4 set_mon_vals no callback set 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 10 monclient: shutdown 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/1789966629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db8136d60 msgr2=0x7f9db8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/1789966629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db8136d60 0x7f9db8139140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9db8132a50 tx=0x7f9db40090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/1789966629 shutdown_connections 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/1789966629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db8136d60 0x7f9db8139140 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/1789966629 >> v2:172.21.15.116:3300/0 conn(0x7f9db809f020 0x7f9db8069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.058+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/1789966629 >> 172.21.15.144:0/1789966629 conn(0x7f9db80a5d40 msgr2=0x7f9db80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.059+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/1789966629 shutdown_connections 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.059+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/1789966629 wait complete. 2024-09-18T12:27:51.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.059+0000 7f9dbfeaf640 10 monclient: build_initial_monmap 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.059+0000 7f9dbfeaf640 10 monclient: monmap: 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:51.071 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.059+0000 7f9dbfeaf640 1 Processor -- start 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.059+0000 7f9dbfeaf640 1 -- start start 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.059+0000 7f9dbfeaf640 10 monclient: init 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9db809ec50 0x7f9db809b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9db809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 0x7f9db8097010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9db809f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:51.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9db8136d60 0x7f9db8097550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9db8136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db809b7e0 con 0x7f9db8136d60 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db809c000 con 0x7f9db809f020 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db8098fc0 con 0x7f9db809ec50 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbfeaf640 10 monclient(hunting): authenticate will time out at 906.629263s 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 0x7f9db8097010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 0x7f9db8097010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49982/0 (socket says 172.21.15.144:49982) 2024-09-18T12:27:51.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 -- 172.21.15.144:0/3574767539 learned_addr learned my addr 172.21.15.144:0/3574767539 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): get_auth_request con 0x7f9db809f020 auth_method 0 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbe44c640 1 -- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9db8136d60 msgr2=0x7f9db8097550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbe44c640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9db8136d60 0x7f9db8097550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbdc4b640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9db809ec50 0x7f9db809b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbdc4b640 10 monclient(hunting): get_auth_request con 0x7f9db809ec50 auth_method 0 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbdc4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:51.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbdc4b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbdc4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient(hunting): handle_auth_done global_id 4819 payload 824 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 -- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9db809ec50 msgr2=0x7f9db809b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9db809ec50 0x7f9db809b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 -- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9db8136d60 msgr2=0x7f9db8097550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9db8136d60 0x7f9db8097550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient: _finish_hunting 0 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 monclient: found mon.noname-b 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.060+0000 7f9dbd44a640 1 -- 172.21.15.144:0/3574767539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db4009390 con 0x7f9db809f020 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dbd44a640 1 -- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 msgr2=0x7f9db8097010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:51.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dbd44a640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 0x7f9db8097010 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9da800b600 tx=0x7f9da80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8015070 con 0x7f9db809f020 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: dump: 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.076 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: _finish_auth 0 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da8003d90 con 0x7f9db809f020 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dbfeaf640 5 monclient: authenticate success, global_id 4819 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dbfeaf640 10 monclient: _renew_subs 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dbfeaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db80a3940 con 0x7f9db809f020 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8006380 con 0x7f9db809f020 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:51.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9dadffb640 10 monclient: dump: 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.078 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.080 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.080 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.062+0000 7f9daffff640 4 set_mon_vals no callback set 2024-09-18T12:27:51.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.063+0000 7f9dbfeaf640 10 monclient: _renew_subs 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.063+0000 7f9dbfeaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.063+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db8099680 con 0x7f9db809f020 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.063+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9da8010440 con 0x7f9db809f020 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dadffb640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d9c038440 0x7f9d9c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f9da804b270 con 0x7f9db809f020 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dbdc4b640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d9c038440 0x7f9d9c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dbdc4b640 10 monclient: get_auth_request con 0x7f9d9c038440 auth_method 0 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dbfeaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dbfeaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dbfeaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.064+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db809a6f0 con 0x7f9db809f020 2024-09-18T12:27:51.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.066+0000 7f9dbdc4b640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d9c038440 0x7f9d9c03a8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9db000e160 tx=0x7f9db0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:51.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.068+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9da8006770 con 0x7f9db809f020 2024-09-18T12:27:51.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.068+0000 7f9dadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:51.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.068+0000 7f9dadffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:51.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.232+0000 7f9dbfeaf640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T12:27:51.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.232+0000 7f9dbfeaf640 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T12:27:51.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.232+0000 7f9dbfeaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.232+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f9db80008d0 con 0x7f9db809f020 2024-09-18T12:27:51.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.636+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v56) v1 ==== 151+0+0 (secure 0 0 0) 0x7f9da8018690 con 0x7f9db809f020 2024-09-18T12:27:51.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.636+0000 7f9dadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T12:27:51.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.636+0000 7f9dadffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:27:51.735 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:27:51.735 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:27:51.735 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:27:51.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:27:51.739 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:27:51.739 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T12:27:51.739 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T12:27:51.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:27:51.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.738+0000 7f9dbfeaf640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T12:27:51.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.738+0000 7f9dbfeaf640 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T12:27:51.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.738+0000 7f9dbfeaf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.738+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7f9db80a13e0 con 0x7f9db809f020 2024-09-18T12:27:51.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.738+0000 7f9dadffb640 1 -- 172.21.15.144:0/3574767539 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v56) v1 ==== 151+0+0 (secure 0 0 0) 0x7f9da8018ab0 con 0x7f9db809f020 2024-09-18T12:27:51.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.739+0000 7f9dadffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T12:27:51.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.739+0000 7f9dadffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:27:51.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d9c038440 msgr2=0x7f9d9c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:51.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d9c038440 0x7f9d9c03a8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9db000e160 tx=0x7f9db0009040 comp rx=0 tx=0).stop 2024-09-18T12:27:51.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 10 monclient: shutdown 2024-09-18T12:27:51.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 msgr2=0x7f9db8097010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 0x7f9db8097010 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9da800b600 tx=0x7f9da80091a0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 shutdown_connections 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9d9c038440 0x7f9d9c03a8f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9db8136d60 0x7f9db8097550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9db809f020 0x7f9db8097010 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 --2- 172.21.15.144:0/3574767539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9db809ec50 0x7f9db809b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.740+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 >> 172.21.15.144:0/3574767539 conn(0x7f9db80a5d40 msgr2=0x7f9db809cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:51.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.741+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 shutdown_connections 2024-09-18T12:27:51.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.741+0000 7f9dbfeaf640 1 -- 172.21.15.144:0/3574767539 wait complete. 2024-09-18T12:27:51.750 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.824+0000 7f3271b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.824+0000 7f3271b9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.824+0000 7f3271b9d640 10 monclient: monmap: 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:27:51.825776+0000 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:27:51.825776+0000 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:27:51.826 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:27:51.827 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:27:51.827 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:27:51.827 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:27:51.827 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.825+0000 7f3271b9d640 1 Processor -- start 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.825+0000 7f3271b9d640 1 -- start start 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.825+0000 7f3271b9d640 10 monclient: init 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f326c09f320 addr v1:172.21.15.144:6789/0 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f326c12e740 0x7f326c12eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f326c12e740 addr v2:172.21.15.116:3300/0 2024-09-18T12:27:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f326c12f080 0x7f326c138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f326c12f080 addr v2:172.21.15.144:3300/0 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c096b60 con 0x7f326c09f320 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c09e4d0 con 0x7f326c12e740 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c1328c0 con 0x7f326c12f080 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f3271b9d640 10 monclient(hunting): authenticate will time out at 907.395131s 2024-09-18T12:27:51.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f326c12f080 0x7f326c138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326affd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f326c12e740 msgr2=0x7f326c12eb40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326b7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f326c09f320 0x7f326c1321b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49554/0 (socket says 172.21.15.144:49554) 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326b7fe640 1 -- 172.21.15.144:0/1447829564 learned_addr learned my addr 172.21.15.144:0/1447829564 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c12f080 auth_method 0 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326affd640 1 --2- 172.21.15.144:0/1447829564 >> v2:172.21.15.116:3300/0 conn(0x7f326c12e740 0x7f326c12eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:51.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient(hunting): handle_auth_done global_id 4822 payload 274 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 1 -- 172.21.15.144:0/1447829564 >> v2:172.21.15.116:3300/0 conn(0x7f326c12e740 msgr2=0x7f326c12eb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 1 --2- 172.21.15.144:0/1447829564 >> v2:172.21.15.116:3300/0 conn(0x7f326c12e740 0x7f326c12eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 1 -- 172.21.15.144:0/1447829564 >> v1:172.21.15.144:6789/0 conn(0x7f326c09f320 legacy=0x7f326c1321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 1 monclient: found mon.noname-c 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.826+0000 7f326bfff640 1 -- 172.21.15.144:0/1447829564 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c09e7b0 con 0x7f326c12f080 2024-09-18T12:27:51.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326bfff640 1 -- 172.21.15.144:0/1447829564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c12f080 msgr2=0x7f326c138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326bfff640 1 --2- 172.21.15.144:0/1447829564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c12f080 0x7f326c138fb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f326c096f80 tx=0x7f32640090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b3aa1cbbfbf3b16 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 1 -- 172.21.15.144:0/1447829564 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264032440 con 0x7f326c12f080 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: dump: 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.839 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 1 -- 172.21.15.144:0/1447829564 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32640325a0 con 0x7f326c12f080 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f3271b9d640 5 monclient: authenticate success, global_id 4822 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f3271b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 1 -- 172.21.15.144:0/1447829564 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264032940 con 0x7f326c12f080 2024-09-18T12:27:51.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f3271b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f3271b9d640 4 set_mon_vals no callback set 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.829+0000 7f326a7fc640 10 monclient: dump: 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.841 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 10 monclient: shutdown 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 -- 172.21.15.144:0/1447829564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c12f080 msgr2=0x7f326c138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 --2- 172.21.15.144:0/1447829564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c12f080 0x7f326c138fb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f326c096f80 tx=0x7f32640090f0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 -- 172.21.15.144:0/1447829564 shutdown_connections 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 --2- 172.21.15.144:0/1447829564 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c12f080 0x7f326c138fb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 --2- 172.21.15.144:0/1447829564 >> v2:172.21.15.116:3300/0 conn(0x7f326c12e740 0x7f326c12eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 -- 172.21.15.144:0/1447829564 >> 172.21.15.144:0/1447829564 conn(0x7f326c0a2b70 msgr2=0x7f326c09ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:27:51.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 -- 172.21.15.144:0/1447829564 shutdown_connections 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 1 -- 172.21.15.144:0/1447829564 wait complete. 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 10 monclient: build_initial_monmap 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.830+0000 7f3271b9d640 10 monclient: monmap: 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 Processor -- start 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 -- start start 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient: init 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 0x7f326c12cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f326c09f320 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f326c12e740 0x7f326c09a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f326c12e740 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 0x7f326c09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f326c12f080 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c12c600 con 0x7f326c12f080 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c09ba90 con 0x7f326c09f320 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c139210 con 0x7f326c12e740 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f3271b9d640 10 monclient(hunting): authenticate will time out at 907.400611s 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 0x7f326c09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 0x7f326c09aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:51654/0 (socket says 172.21.15.144:51654) 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 1 -- 172.21.15.144:0/3031576628 learned_addr learned my addr 172.21.15.144:0/3031576628 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c12f080 auth_method 0 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:51.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 1 -- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 msgr2=0x7f326c09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 1 -- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 msgr2=0x7f326c09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 0x7f326c09aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326bfff640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 0x7f326c09aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326b7fe640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 0x7f326c12cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326b7fe640 10 monclient(hunting): get_auth_request con 0x7f326c09f320 auth_method 0 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.831+0000 7f326b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 10 monclient(hunting): handle_auth_done global_id 4825 payload 824 2024-09-18T12:27:51.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 1 -- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f326c12e740 msgr2=0x7f326c09a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f326c12e740 0x7f326c09a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 1 -- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 msgr2=0x7f326c09aa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 0x7f326c09aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.832+0000 7f326b7fe640 1 -- 172.21.15.144:0/3031576628 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3264009390 con 0x7f326c09f320 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f326b7fe640 1 -- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 msgr2=0x7f326c12cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f326b7fe640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 0x7f326c12cfe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f325c00cbe0 tx=0x7f325c005330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f326b7fe640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 0x7f326c12cfe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f325c00cbe0 tx=0x7f325c005330 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f325c00ce00 mon_map magic: 0 v1 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c00ce00 con 0x7f326c09f320 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 10 monclient: dump: 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 10 monclient: _finish_auth 0 2024-09-18T12:27:51.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c00aa60 con 0x7f326c09f320 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3271b9d640 5 monclient: authenticate success, global_id 4825 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3271b9d640 10 monclient: _renew_subs 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3271b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c09d5b0 con 0x7f326c09f320 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f32537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3271b9d640 10 monclient: _renew_subs 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3271b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c09c0c0 con 0x7f326c09f320 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.834+0000 7f3269ffb640 4 set_mon_vals no callback set 2024-09-18T12:27:51.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c00ae20 con 0x7f326c09f320 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f32537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f32537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f32537fe640 10 monclient: dump: 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:27:51.850 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f325c007300 con 0x7f326c09f320 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f3271b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f3271b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f3271b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c12c840 con 0x7f326c09f320 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f32537fe640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3258038510 0x7f325803a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f326affd640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3258038510 0x7f325803a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:27:51.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.835+0000 7f326affd640 10 monclient: get_auth_request con 0x7f3258038510 auth_method 0 2024-09-18T12:27:51.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.838+0000 7f326affd640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3258038510 0x7f325803a9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3260006fd0 tx=0x7f3260006cf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:27:51.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:51.840+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f325c04bb10 con 0x7f326c09f320 2024-09-18T12:27:52.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:52.457+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f325c0075b0 con 0x7f326c09f320 2024-09-18T12:27:52.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:52.457+0000 7f32537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:27:52.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:52.457+0000 7f32537fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:27:52.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:52.613+0000 7f3271b9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T12:27:52.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:52.613+0000 7f3271b9d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T12:27:52.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:52.613+0000 7f3271b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:27:52.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:52.613+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f326c0008d0 con 0x7f326c09f320 2024-09-18T12:27:54.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:54.832+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:27:54.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:54.832+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:54.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:54.832+0000 7f32527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:27:55.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:55.831+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:27:55.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:55.831+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:56.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:56.831+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:27:56.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:56.832+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:57.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:57.832+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:27:57.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:57.832+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:58.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:58.832+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:27:58.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:58.832+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:27:59.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:59.832+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:27:59.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:27:59.832+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:00.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:00.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:00.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:00.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:01.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:01.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:01.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:01.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:02.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:02.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:02.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:02.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:03.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:03.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:03.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:03.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:04.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:04.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:04.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:04.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:04.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:04.833+0000 7f32527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:05.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:05.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:05.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:05.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:06.740 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T12:28:06.740 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T12:28:06.741 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:28:06.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:28:06.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:06.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:06.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:06.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:06.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.843+0000 7fdf3c034640 10 monclient: get_monmap_and_config 2024-09-18T12:28:06.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.843+0000 7fdf3c034640 10 monclient: build_initial_monmap 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.843+0000 7fdf3c034640 10 monclient: monmap: 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:06.844522+0000 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:06.844522+0000 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:06.845 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:06.846 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:06.846 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.844+0000 7fdf3c034640 1 Processor -- start 2024-09-18T12:28:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.844+0000 7fdf3c034640 1 -- start start 2024-09-18T12:28:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.844+0000 7fdf3c034640 10 monclient: init 2024-09-18T12:28:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-18T12:28:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient(hunting): picked mon.noname-b con 0x7fdf34096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient(hunting): picked mon.noname-c con 0x7fdf34097030 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf3412e5f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3409d2e0 con 0x7fdf3412e5f0 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3409ac00 con 0x7fdf34097030 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3409d6b0 con 0x7fdf34096c60 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf3c034640 10 monclient(hunting): authenticate will time out at 914.166248s 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54388/0 (socket says 172.21.15.116:54388) 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 learned_addr learned my addr 172.21.15.116:0/203397756 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf34097030 auth_method 0 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.845+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:06.847+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.144:6789/0 conn(0x7fdf3412e5f0 legacy=0x7fdf3412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf34097030 auth_method 0 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:07.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf34097030 auth_method 0 2024-09-18T12:28:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:28:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:07.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:07.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:07.834+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:07.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:07.834+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf34097030 auth_method 0 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:08.248+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:08.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:08.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:08.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:08.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:09.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:09.833+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:09.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:09.833+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:09.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 10 monclient(hunting): tick 2024-09-18T12:28:09.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:28:09.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf3409a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf3409a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 msgr2=0x7fdf34097400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf34097030 0x7fdf34097400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.144:6789/0 conn(0x7fdf3412e5f0 legacy=0x7fdf3412ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 10 monclient: _add_conns ranks=[0,3] 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf20005940 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-a con 0x7fdf20009660 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf340992f0 con 0x7fdf20005940 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3409ac00 con 0x7fdf20009660 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf337fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf20009660 auth_method 0 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.845+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:09.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:09.847+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20005940 legacy=0x7fdf20007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf20009660 auth_method 0 2024-09-18T12:28:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf20009660 auth_method 0 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:10.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:10.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:10.834+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:10.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:10.834+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf20009660 auth_method 0 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.249+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.828+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20005940 legacy=0x7fdf20007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-18T12:28:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.828+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20005940 legacy=0x7fdf20007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:28:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.828+0000 7fdf39dd0640 1 --1- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20005940 0x7fdf20007d90 :-1 s=OPENED pgs=20 cs=1 l=1).handle_message read tag failed 2024-09-18T12:28:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.828+0000 7fdf39dd0640 1 --1- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20005940 0x7fdf20007d90 :-1 s=OPENED pgs=20 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:28:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:11.828+0000 7fdf38dce640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.116:6789/0 2024-09-18T12:28:11.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:11.834+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:11.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:11.834+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:12.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:12.834+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:12.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:12.834+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): tick 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 msgr2=0x7fdf2000baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20009660 0x7fdf2000baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20005940 legacy=0x7fdf20007d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T12:28:12.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 reap_dead start 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-b con 0x7fdf34096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf20004e70 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-c con 0x7fdf2000bbb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf200045b0 con 0x7fdf20004e70 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3409ac00 con 0x7fdf2000bbb0 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3409d6b0 con 0x7fdf34096c60 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf337fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:12.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request con 0x7fdf2000bbb0 auth_method 0 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:12.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:28:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.846+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:12.848+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.144:6789/0 conn(0x7fdf20004e70 legacy=0x7fdf20003570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request con 0x7fdf2000bbb0 auth_method 0 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:28:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.047+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:13.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:13.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:13.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request con 0x7fdf2000bbb0 auth_method 0 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:28:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:13.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:13.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:13.448+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:13.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:13.834+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:13.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:13.834+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf34096c60 auth_method 0 2024-09-18T12:28:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request con 0x7fdf2000bbb0 auth_method 0 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:14.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:14.249+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:14.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:14.835+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:14.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:14.835+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:14.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:14.835+0000 7f32527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:15.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:15.834+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:15.834+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:15.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 10 monclient(hunting): tick 2024-09-18T12:28:15.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf34096c60 0x7fdf20004930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 msgr2=0x7fdf20005240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.144:3300/0 conn(0x7fdf2000bbb0 0x7fdf20005240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.144:6789/0 conn(0x7fdf20004e70 legacy=0x7fdf20003570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 10 monclient: _add_conns ranks=[0,3] 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf20003c80 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-a con 0x7fdf200041b0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf20008000 con 0x7fdf20003c80 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3409ac00 con 0x7fdf200041b0 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf337fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf200041b0 auth_method 0 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:15.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:15.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:15.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:15.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.846+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:15.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:15.848+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20003c80 legacy=0x7fdf200101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf200041b0 auth_method 0 2024-09-18T12:28:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.046+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf200041b0 auth_method 0 2024-09-18T12:28:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:16.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:16.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:16.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:16.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:16.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.447+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.827+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20003c80 legacy=0x7fdf200101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-18T12:28:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.827+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20003c80 legacy=0x7fdf200101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:28:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.827+0000 7fdf39dd0640 1 --1- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20003c80 0x7fdf200101a0 :-1 s=OPENED pgs=21 cs=1 l=1).handle_message read tag failed 2024-09-18T12:28:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.827+0000 7fdf39dd0640 1 --1- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20003c80 0x7fdf200101a0 :-1 s=OPENED pgs=21 cs=1 l=1).fault on lossy channel, failing 2024-09-18T12:28:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:16.827+0000 7fdf38dce640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.116:6789/0 2024-09-18T12:28:16.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:16.835+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:16.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:16.835+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:17.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:17.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf200041b0 auth_method 0 2024-09-18T12:28:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:17.248+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:17.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:17.834+0000 7f32527fc640 10 monclient: tick 2024-09-18T12:28:17.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:17.834+0000 7f32527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.337+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v58) v1 ==== 177+0+0 (secure 0 0 0) 0x7f325c010af0 con 0x7f326c09f320 2024-09-18T12:28:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.338+0000 7f32537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T12:28:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.338+0000 7f32537fe640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-18T12:28:18.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.434+0000 7f3271b9d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T12:28:18.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.434+0000 7f3271b9d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T12:28:18.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.434+0000 7f3271b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:18.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.434+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f326c064b20 con 0x7f326c09f320 2024-09-18T12:28:18.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.436+0000 7f32537fe640 1 -- 172.21.15.144:0/3031576628 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v58) v1 ==== 184+0+0 (secure 0 0 0) 0x7f325c010cd0 con 0x7f326c09f320 2024-09-18T12:28:18.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.436+0000 7f32537fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T12:28:18.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.436+0000 7f32537fe640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-18T12:28:18.438 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'ec-foo' already exists 2024-09-18T12:28:18.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3258038510 msgr2=0x7f325803a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3258038510 0x7f325803a9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3260006fd0 tx=0x7f3260006cf0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 10 monclient: shutdown 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 msgr2=0x7f326c12cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 0x7f326c12cfe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f325c00cbe0 tx=0x7f325c005330 comp rx=0 tx=0).stop 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 shutdown_connections 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3258038510 0x7f325803a9c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f326c12f080 0x7f326c09aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f326c12e740 0x7f326c09a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 --2- 172.21.15.144:0/3031576628 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f326c09f320 0x7f326c12cfe0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 >> 172.21.15.144:0/3031576628 conn(0x7f326c0a2b70 msgr2=0x7f326c070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:18.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 shutdown_connections 2024-09-18T12:28:18.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.438+0000 7f3271b9d640 1 -- 172.21.15.144:0/3031576628 wait complete. 2024-09-18T12:28:18.450 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.524+0000 7f482a499640 10 monclient: get_monmap_and_config 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.524+0000 7f482a499640 10 monclient: build_initial_monmap 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.524+0000 7f482a499640 10 monclient: monmap: 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:18.526870+0000 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:18.526870+0000 2024-09-18T12:28:18.527 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:18.528 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:18.528 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:18.528 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:18.528 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:18.528 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:18.528 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:18.528 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:18.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.525+0000 7f482a499640 1 Processor -- start 2024-09-18T12:28:18.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.525+0000 7f482a499640 1 -- start start 2024-09-18T12:28:18.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.525+0000 7f482a499640 10 monclient: init 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f482409cad0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f482409c1a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 0x7f48241383d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient(hunting): picked mon.noname-b con 0x7f4824135ff0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482409d5b0 con 0x7f482409cad0 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482409b710 con 0x7f482409c1a0 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482409a650 con 0x7f4824135ff0 2024-09-18T12:28:18.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f482a499640 10 monclient(hunting): authenticate will time out at 934.096314s 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f48237fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f482409c1a0 0x7f48241348d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:47988/0 (socket says 172.21.15.144:47988) 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f48237fe640 1 -- 172.21.15.144:0/1439593332 learned_addr learned my addr 172.21.15.144:0/1439593332 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 0x7f48241383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): get_auth_request con 0x7f4824135ff0 auth_method 0 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient(hunting): handle_auth_done global_id 44267 payload 274 2024-09-18T12:28:18.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 1 -- 172.21.15.144:0/1439593332 >> v1:172.21.15.144:6789/0 conn(0x7f482409c1a0 legacy=0x7f48241348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:18.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 1 -- 172.21.15.144:0/1439593332 >> v1:172.21.15.116:6789/0 conn(0x7f482409cad0 legacy=0x7f482409cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:18.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient: _finish_hunting 0 2024-09-18T12:28:18.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 1 monclient: found mon.noname-b 2024-09-18T12:28:18.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:18.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:28:18.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.526+0000 7f4828a36640 1 -- 172.21.15.144:0/1439593332 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482409f660 con 0x7f4824135ff0 2024-09-18T12:28:18.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.529+0000 7f4828a36640 1 -- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 msgr2=0x7f48241383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:18.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.529+0000 7f4828a36640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 0x7f48241383d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f481c009e90 tx=0x7f481c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e89505797cea26a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:18.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.529+0000 7f4828a36640 1 -- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 msgr2=0x7f48241383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-18T12:28:18.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.529+0000 7f4828a36640 1 -- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 msgr2=0x7f48241383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:28:18.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.529+0000 7f4828a36640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 0x7f48241383d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f481c009e90 tx=0x7f481c009780 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:18.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.529+0000 7f4828a36640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 0x7f48241383d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f481c009e90 tx=0x7f481c009780 comp rx=0 tx=0).stop 2024-09-18T12:28:18.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:18.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:18.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 1 -- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 msgr2=0x7f48241383d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:18.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824135ff0 0x7f48241383d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient: _add_conns ranks=[4,3] 2024-09-18T12:28:18.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 1 --2- 172.21.15.144:0/1439593332 >> v2:172.21.15.144:3300/0 conn(0x7f480c0034c0 0x7f480c005930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f480c0034c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 1 --2- 172.21.15.144:0/1439593332 >> v2:172.21.15.116:3300/0 conn(0x7f480c006260 0x7f480c008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f480c006260 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 1 -- 172.21.15.144:0/1439593332 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480c008fe0 con 0x7f480c006260 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 1 -- 172.21.15.144:0/1439593332 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480c009150 con 0x7f480c0034c0 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4822ffd640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 1 --2- 172.21.15.144:0/1439593332 >> v2:172.21.15.144:3300/0 conn(0x7f480c0034c0 0x7f480c005930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): get_auth_request con 0x7f480c0034c0 auth_method 0 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 1 --2- 172.21.15.144:0/1439593332 >> v2:172.21.15.116:3300/0 conn(0x7f480c006260 0x7f480c008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 10 monclient(hunting): get_auth_request con 0x7f480c006260 auth_method 0 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f48237fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient(hunting): handle_auth_done global_id 44267 payload 293 2024-09-18T12:28:18.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 1 -- 172.21.15.144:0/1439593332 >> v2:172.21.15.116:3300/0 conn(0x7f480c006260 msgr2=0x7f480c008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 1 --2- 172.21.15.144:0/1439593332 >> v2:172.21.15.116:3300/0 conn(0x7f480c006260 0x7f480c008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 1 monclient: found mon.noname-c 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.530+0000 7f4823fff640 1 -- 172.21.15.144:0/1439593332 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c009380 con 0x7f480c0034c0 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4823fff640 1 -- 172.21.15.144:0/1439593332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 msgr2=0x7f480c005930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4823fff640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f480c005930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f480c003430 tx=0x7f4814007ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=782f1a1a57fadeaf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 1 -- 172.21.15.144:0/1439593332 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481400c8c0 con 0x7f480c0034c0 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: dump: 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:18.547 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f482a499640 5 monclient: authenticate success, global_id 44267 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 1 -- 172.21.15.144:0/1439593332 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481400ca20 con 0x7f480c0034c0 2024-09-18T12:28:18.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f482a499640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f482a499640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f482a499640 4 set_mon_vals no callback set 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 1 -- 172.21.15.144:0/1439593332 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481400ce20 con 0x7f480c0034c0 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.532+0000 7f4822ffd640 10 monclient: dump: 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:18.549 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 10 monclient: shutdown 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 1 -- 172.21.15.144:0/1439593332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 msgr2=0x7f480c005930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f480c005930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f480c003430 tx=0x7f4814007ca0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f4828a36640 1 -- 172.21.15.144:0/1439593332 reap_dead start 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 1 -- 172.21.15.144:0/1439593332 shutdown_connections 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 1 --2- 172.21.15.144:0/1439593332 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f480c005930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 1 -- 172.21.15.144:0/1439593332 >> 172.21.15.144:0/1439593332 conn(0x7f4824070e90 msgr2=0x7f48240a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:18.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 1 -- 172.21.15.144:0/1439593332 shutdown_connections 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 1 -- 172.21.15.144:0/1439593332 wait complete. 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 10 monclient: build_initial_monmap 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.533+0000 7f482a499640 10 monclient: monmap: 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:18.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 Processor -- start 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 -- start start 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient: init 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f4824134f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): picked mon.noname-b con 0x7f480c0034c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824135460 0x7f4824068ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): picked mon.noname-a con 0x7f4824135460 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824069010 0x7f4824069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): picked mon.noname-c con 0x7f4824069010 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4824136870 con 0x7f4824135460 2024-09-18T12:28:18.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482406eba0 con 0x7f480c0034c0 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48240a2880 con 0x7f4824069010 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f482a499640 10 monclient(hunting): authenticate will time out at 934.104897s 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f4823fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f4824134f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f4823fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f4824134f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46664/0 (socket says 172.21.15.144:46664) 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f4823fff640 1 -- 172.21.15.144:0/2408621106 learned_addr learned my addr 172.21.15.144:0/2408621106 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f4823fff640 10 monclient(hunting): get_auth_request con 0x7f480c0034c0 auth_method 0 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f4823fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f4823fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.534+0000 7f4823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824069010 0x7f4824069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824135460 0x7f4824068ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 10 monclient(hunting): get_auth_request con 0x7f4824069010 auth_method 0 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 10 monclient(hunting): get_auth_request con 0x7f4824135460 auth_method 0 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f48237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4828a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 10 monclient(hunting): handle_auth_done global_id 4828 payload 824 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 1 -- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824069010 msgr2=0x7f4824069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824069010 0x7f4824069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 1 -- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824135460 msgr2=0x7f4824068ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824135460 0x7f4824068ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 1 monclient: found mon.noname-b 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:18.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.535+0000 7f4823fff640 1 -- 172.21.15.144:0/2408621106 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824122040 con 0x7f480c0034c0 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f4823fff640 1 -- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 msgr2=0x7f4824134f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f4823fff640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f4824134f20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f480c009290 tx=0x7f4814015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814006790 con 0x7f480c0034c0 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: dump: 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.556 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: _finish_auth 0 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f482a499640 5 monclient: authenticate success, global_id 4828 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48140068f0 con 0x7f480c0034c0 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814006c90 con 0x7f480c0034c0 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f482a499640 10 monclient: _renew_subs 2024-09-18T12:28:18.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f482a499640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4824122600 con 0x7f480c0034c0 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f480bfff640 10 monclient: dump: 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.558 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f48227fc640 4 set_mon_vals no callback set 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f482a499640 10 monclient: _renew_subs 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f482a499640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.537+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482406a620 con 0x7f480c0034c0 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f481401e660 con 0x7f480c0034c0 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f480bfff640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48100384e0 0x7f481003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f482a499640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f482a499640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f482a499640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48240008d0 con 0x7f480c0034c0 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5496+0+0 (secure 0 0 0) 0x7f481404d930 con 0x7f480c0034c0 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f48237fe640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48100384e0 0x7f481003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.538+0000 7f48237fe640 10 monclient: get_auth_request con 0x7f48100384e0 auth_method 0 2024-09-18T12:28:18.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.541+0000 7f48237fe640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48100384e0 0x7f481003a990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4818007510 tx=0x7f4818007d90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:18.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.543+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f481402e3a0 con 0x7f480c0034c0 2024-09-18T12:28:18.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.543+0000 7f480bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:18.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.543+0000 7f480bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.702+0000 7f482a499640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.702+0000 7f482a499640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.702+0000 7f482a499640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:18.702+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4824121150 con 0x7f480c0034c0 2024-09-18T12:28:18.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): tick 2024-09-18T12:28:18.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:28:18.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:28:18.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 msgr2=0x7fdf200138d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf200041b0 0x7fdf200138d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20003c80 legacy=0x7fdf200101a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf3a5d1640 1 -- 172.21.15.116:0/203397756 reap_dead start 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20013fe0 0x7fdf200143b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-a con 0x7fdf20013fe0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20005240 0x7fdf20004930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-b con 0x7fdf20005240 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf20014990 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf20014f70 con 0x7fdf20014990 2024-09-18T12:28:18.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3409ac00 con 0x7fdf20013fe0 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 1 -- 172.21.15.116:0/203397756 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3409d6b0 con 0x7fdf20005240 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf337fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> v2:172.21.15.116:3300/0 conn(0x7fdf20013fe0 0x7fdf200143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf20013fe0 auth_method 0 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20005240 0x7fdf20004930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf20005240 auth_method 0 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf395cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_done global_id 54102 payload 274 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20005240 msgr2=0x7fdf20004930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20005240 0x7fdf20004930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> v1:172.21.15.116:6789/0 conn(0x7fdf20014990 legacy=0x7fdf20003550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient: _finish_hunting 0 2024-09-18T12:28:18.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 1 monclient: found mon.noname-a 2024-09-18T12:28:18.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:18.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:18.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.846+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf20003c60 con 0x7fdf20013fe0 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.848+0000 7fdf39dd0640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20013fe0 msgr2=0x7fdf200143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20013fe0 0x7fdf200143b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009f40 tx=0x7fdf24009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3ff8e792dd2877 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 1 -- 172.21.15.116:0/203397756 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24036420 con 0x7fdf20013fe0 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: dump: 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.856 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: _finish_auth 0 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 1 -- 172.21.15.116:0/203397756 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2400f040 con 0x7fdf20013fe0 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf3c034640 5 monclient: authenticate success, global_id 54102 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf3c034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf3c034640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:18.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 1 -- 172.21.15.116:0/203397756 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf240367d0 con 0x7fdf20013fe0 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf3c034640 4 set_mon_vals no callback set 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.849+0000 7fdf38dce640 10 monclient: dump: 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.858 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 10 monclient: shutdown 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 -- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20013fe0 msgr2=0x7fdf200143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20013fe0 0x7fdf200143b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009f40 tx=0x7fdf24009040 comp rx=0 tx=0).stop 2024-09-18T12:28:18.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 -- 172.21.15.116:0/203397756 shutdown_connections 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20005240 0x7fdf20004930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 --2- 172.21.15.116:0/203397756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20013fe0 0x7fdf200143b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 -- 172.21.15.116:0/203397756 >> 172.21.15.116:0/203397756 conn(0x7fdf340672b0 msgr2=0x7fdf340676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 -- 172.21.15.116:0/203397756 shutdown_connections 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 1 -- 172.21.15.116:0/203397756 wait complete. 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 10 monclient: build_initial_monmap 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.850+0000 7fdf3c034640 10 monclient: monmap: 2024-09-18T12:28:18.860 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:18.861 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:18.861 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:18.861 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:18.861 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:18.861 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 Processor -- start 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 -- start start 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient: init 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:28:18.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 0x7fdf3412d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): picked mon.noname-a con 0x7fdf20005240 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf20013fe0 0x7fdf34117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): picked mon.noname-b con 0x7fdf20013fe0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20014990 0x7fdf341179c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): picked mon.noname-c con 0x7fdf20014990 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf34130920 con 0x7fdf20005240 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3411aaf0 con 0x7fdf20013fe0 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf34118d50 con 0x7fdf20014990 2024-09-18T12:28:18.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:18.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.851+0000 7fdf3c034640 10 monclient(hunting): authenticate will time out at 926.173959s 2024-09-18T12:28:18.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 0x7fdf3412d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 0x7fdf3412d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50664/0 (socket says 172.21.15.116:50664) 2024-09-18T12:28:18.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 -- 172.21.15.116:0/286168442 learned_addr learned my addr 172.21.15.116:0/286168442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:18.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request con 0x7fdf20005240 auth_method 0 2024-09-18T12:28:18.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20014990 0x7fdf341179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request con 0x7fdf20014990 auth_method 0 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf3a5d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:18.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf395cf640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf20013fe0 0x7fdf34117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf395cf640 10 monclient(hunting): get_auth_request con 0x7fdf20013fe0 auth_method 0 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf395cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf395cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf395cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient(hunting): handle_auth_done global_id 54105 payload 824 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 -- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20014990 msgr2=0x7fdf341179c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20014990 0x7fdf341179c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 -- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf20013fe0 msgr2=0x7fdf34117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf20013fe0 0x7fdf34117480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient: _finish_hunting 0 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 monclient: found mon.noname-a 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:18.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:18.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.852+0000 7fdf39dd0640 1 -- 172.21.15.116:0/286168442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf20008000 con 0x7fdf20005240 2024-09-18T12:28:18.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf39dd0640 1 -- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 msgr2=0x7fdf3412d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:18.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf39dd0640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 0x7fdf3412d6f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009fa0 tx=0x7fdf24039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:18.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 1 -- 172.21.15.116:0/286168442 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2403f400 con 0x7fdf20005240 2024-09-18T12:28:18.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 10 monclient: dump: 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.870 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 1 -- 172.21.15.116:0/286168442 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2403f580 con 0x7fdf20005240 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf3c034640 5 monclient: authenticate success, global_id 54105 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf3c034640 10 monclient: _renew_subs 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf3c034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3411a810 con 0x7fdf20005240 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.854+0000 7fdf31ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf31ffb640 1 -- 172.21.15.116:0/286168442 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2403f730 con 0x7fdf20005240 2024-09-18T12:28:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf31ffb640 10 monclient: dump: 2024-09-18T12:28:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf3c034640 10 monclient: _renew_subs 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf3c034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf34119920 con 0x7fdf20005240 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.855+0000 7fdf33fff640 4 set_mon_vals no callback set 2024-09-18T12:28:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf31ffb640 1 -- 172.21.15.116:0/286168442 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdf2403d040 con 0x7fdf20005240 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf3c034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf3c034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf3c034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3411acb0 con 0x7fdf20005240 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf31ffb640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf2003b660 0x7fdf2003db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf31ffb640 1 -- 172.21.15.116:0/286168442 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5496+0+0 (secure 0 0 0) 0x7fdf24045020 con 0x7fdf20005240 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf395cf640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf2003b660 0x7fdf2003db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.856+0000 7fdf395cf640 10 monclient: get_auth_request con 0x7fdf2003b660 auth_method 0 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.859+0000 7fdf395cf640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf2003b660 0x7fdf2003db10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdf28006e70 tx=0x7fdf28028040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.861+0000 7fdf31ffb640 1 -- 172.21.15.116:0/286168442 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdf2403dd60 con 0x7fdf20005240 2024-09-18T12:28:18.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.861+0000 7fdf31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:18.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:18.861+0000 7fdf31ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.079+0000 7fdf3c034640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:28:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.079+0000 7fdf3c034640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.079+0000 7fdf3c034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.079+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdf3411acb0 con 0x7fdf20005240 2024-09-18T12:28:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.080+0000 7fdf31ffb640 1 -- 172.21.15.116:0/286168442 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fdf2403b620 con 0x7fdf20005240 2024-09-18T12:28:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.080+0000 7fdf31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.080+0000 7fdf31ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:19.082 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:28:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf2003b660 msgr2=0x7fdf2003db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf2003b660 0x7fdf2003db10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdf28006e70 tx=0x7fdf28028040 comp rx=0 tx=0).stop 2024-09-18T12:28:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 10 monclient: shutdown 2024-09-18T12:28:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 msgr2=0x7fdf3412d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 0x7fdf3412d6f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009fa0 tx=0x7fdf24039000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 shutdown_connections 2024-09-18T12:28:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf2003b660 0x7fdf2003db10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf20014990 0x7fdf341179c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf20013fe0 0x7fdf34117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 --2- 172.21.15.116:0/286168442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf20005240 0x7fdf3412d6f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 >> 172.21.15.116:0/286168442 conn(0x7fdf340672b0 msgr2=0x7fdf3409d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 shutdown_connections 2024-09-18T12:28:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.082+0000 7fdf3c034640 1 -- 172.21.15.116:0/286168442 wait complete. 2024-09-18T12:28:19.094 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:28:19.094 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:28:19.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:28:19.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.185+0000 7f1eabf9d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:19.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.185+0000 7f1eabf9d640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.185+0000 7f1eabf9d640 10 monclient: monmap: 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:19.187393+0000 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:19.187393+0000 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:19.188 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 1 Processor -- start 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 1 -- start start 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient: init 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ea4096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1ea40971e0 0x7f1ea40975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea40971e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:19.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ea4133b30 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea4099620 con 0x7f1ea4096e10 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea4099340 con 0x7f1ea4133b30 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea4099900 con 0x7f1ea40971e0 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.186+0000 7f1eabf9d640 10 monclient(hunting): authenticate will time out at 926.508976s 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9d39640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1ea4096e10 0x7f1ea409a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41442/0 (socket says 172.21.15.116:41442) 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9d39640 1 -- 172.21.15.116:0/1198943471 learned_addr learned my addr 172.21.15.116:0/1198943471 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 1 --2- 172.21.15.116:0/1198943471 >> v2:172.21.15.144:3300/0 conn(0x7f1ea40971e0 0x7f1ea40975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): get_auth_request con 0x7f1ea40971e0 auth_method 0 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient(hunting): handle_auth_done global_id 4834 payload 274 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 1 -- 172.21.15.116:0/1198943471 >> v1:172.21.15.144:6789/0 conn(0x7f1ea4133b30 legacy=0x7f1ea4135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 1 -- 172.21.15.116:0/1198943471 >> v1:172.21.15.116:6789/0 conn(0x7f1ea4096e10 legacy=0x7f1ea409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 1 monclient: found mon.noname-c 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.187+0000 7f1ea9538640 1 -- 172.21.15.116:0/1198943471 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea409ac70 con 0x7f1ea40971e0 2024-09-18T12:28:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea9538640 1 -- 172.21.15.116:0/1198943471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea40971e0 msgr2=0x7f1ea40975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea9538640 1 --2- 172.21.15.116:0/1198943471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea40971e0 0x7f1ea40975e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009850 tx=0x7f1e940090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=513bc39e1061bf98 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 1 -- 172.21.15.116:0/1198943471 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9402ae70 con 0x7f1ea40971e0 2024-09-18T12:28:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: dump: 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.206 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.207 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1eabf9d640 5 monclient: authenticate success, global_id 4834 2024-09-18T12:28:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1eabf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 1 -- 172.21.15.116:0/1198943471 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e94032440 con 0x7f1ea40971e0 2024-09-18T12:28:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1eabf9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:19.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1eabf9d640 4 set_mon_vals no callback set 2024-09-18T12:28:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 1 -- 172.21.15.116:0/1198943471 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94032820 con 0x7f1ea40971e0 2024-09-18T12:28:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.190+0000 7f1ea8d37640 10 monclient: dump: 2024-09-18T12:28:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.209 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 10 monclient: shutdown 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 1 -- 172.21.15.116:0/1198943471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea40971e0 msgr2=0x7f1ea40975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 1 --2- 172.21.15.116:0/1198943471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea40971e0 0x7f1ea40975e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009850 tx=0x7f1e940090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 1 -- 172.21.15.116:0/1198943471 shutdown_connections 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 1 --2- 172.21.15.116:0/1198943471 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea40971e0 0x7f1ea40975e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 1 -- 172.21.15.116:0/1198943471 >> 172.21.15.116:0/1198943471 conn(0x7f1ea40672b0 msgr2=0x7f1ea40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 1 -- 172.21.15.116:0/1198943471 shutdown_connections 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 1 -- 172.21.15.116:0/1198943471 wait complete. 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.191+0000 7f1eabf9d640 10 monclient: monmap: 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.211 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 Processor -- start 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 -- start start 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient: init 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4096e10 0x7f1ea409b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea4096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea40971e0 0x7f1ea4117950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea40971e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 0x7f1ea412e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea4133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea4117e90 con 0x7f1ea40971e0 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea412f7f0 con 0x7f1ea4096e10 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea411dc20 con 0x7f1ea4133b30 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eabf9d640 10 monclient(hunting): authenticate will time out at 926.514755s 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea40971e0 0x7f1ea4117950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea40971e0 0x7f1ea4117950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50668/0 (socket says 172.21.15.116:50668) 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 1 -- 172.21.15.116:0/4124370920 learned_addr learned my addr 172.21.15.116:0/4124370920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 10 monclient(hunting): get_auth_request con 0x7f1ea40971e0 auth_method 0 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eaa53a640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 0x7f1ea412e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eaa53a640 10 monclient(hunting): get_auth_request con 0x7f1ea4133b30 auth_method 0 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eaa53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eaa53a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1eaa53a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.192+0000 7f1ea9538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1eaa53a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1eaa53a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1eaa53a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9d39640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4096e10 0x7f1ea409b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1eaa53a640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 msgr2=0x7f1ea412e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1eaa53a640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 msgr2=0x7f1ea412e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1eaa53a640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 0x7f1ea412e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1eaa53a640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 0x7f1ea412e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:19.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9d39640 10 monclient(hunting): get_auth_request con 0x7f1ea4096e10 auth_method 0 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 10 monclient(hunting): handle_auth_done global_id 54108 payload 824 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 msgr2=0x7f1ea412e800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1ea4133b30 0x7f1ea412e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4096e10 msgr2=0x7f1ea409b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ea4096e10 0x7f1ea409b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 1 monclient: found mon.noname-a 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.193+0000 7f1ea9538640 1 -- 172.21.15.116:0/4124370920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94009390 con 0x7f1ea40971e0 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1ea9538640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea40971e0 msgr2=0x7f1ea4117950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1ea9538640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea40971e0 0x7f1ea4117950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009850 tx=0x7f1e94032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94031940 con 0x7f1ea40971e0 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: dump: 2024-09-18T12:28:19.217 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1eabf9d640 5 monclient: authenticate success, global_id 54108 2024-09-18T12:28:19.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e94031aa0 con 0x7f1ea40971e0 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1eabf9d640 10 monclient: _renew_subs 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1eabf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea411df00 con 0x7f1ea40971e0 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94031e20 con 0x7f1ea40971e0 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e99ffb640 10 monclient: dump: 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.219 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1e9bfff640 4 set_mon_vals no callback set 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1eabf9d640 10 monclient: _renew_subs 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1eabf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.195+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea412ed40 con 0x7f1ea40971e0 2024-09-18T12:28:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1e94039070 con 0x7f1ea40971e0 2024-09-18T12:28:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1eabf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1eabf9d640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 0x7f1ea4141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1eabf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea4065f20 con 0x7f1ea41410b0 2024-09-18T12:28:19.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1e99ffb640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1e88038490 0x7f1e8803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 0x7f1ea4141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient: get_auth_request con 0x7f1ea41410b0 auth_method 0 2024-09-18T12:28:19.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5496+0+0 (secure 0 0 0) 0x7f1e9406c560 con 0x7f1ea40971e0 2024-09-18T12:28:19.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.196+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_done global_id 54111 payload 824 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.197+0000 7f1ea9d39640 1 -- 172.21.15.116:0/4124370920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ea412ed40 con 0x7f1ea41410b0 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.197+0000 7f1eaa53a640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1e88038490 0x7f1e8803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.197+0000 7f1eaa53a640 10 monclient: get_auth_request con 0x7f1e88038490 auth_method 0 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1ea9d39640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 msgr2=0x7f1ea4141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1ea9d39640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 0x7f1ea4141480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000e740 tx=0x7f1ea0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0004e60 con 0x7f1ea41410b0 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 10 monclient: dump: 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1eaa53a640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1e88038490 0x7f1e8803a940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1ea411e6d0 tx=0x7f1e9c0025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1ea0007170 con 0x7f1ea41410b0 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 msgr2=0x7f1ea4141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 0x7f1ea4141480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000e740 tx=0x7f1ea0009040 comp rx=0 tx=0).stop 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 msgr2=0x7f1ea4141480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.199+0000 7f1e99ffb640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41410b0 0x7f1ea4141480 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000e740 tx=0x7f1ea0009040 comp rx=0 tx=0).stop 2024-09-18T12:28:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1eabf9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1eabf9d640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 0x7f1ea406b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1eabf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea411e090 con 0x7f1ea41419c0 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 0x7f1ea406b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient: get_auth_request con 0x7f1ea41419c0 auth_method 0 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 10 monclient(hunting): handle_auth_done global_id 54114 payload 824 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.205+0000 7f1ea9d39640 1 -- 172.21.15.116:0/4124370920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1ea404bf60 con 0x7f1ea41419c0 2024-09-18T12:28:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.207+0000 7f1ea9d39640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 msgr2=0x7f1ea406b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.207+0000 7f1ea9d39640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 0x7f1ea406b6b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0017000 tx=0x7f1ea0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.207+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0007760 con 0x7f1ea41419c0 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.207+0000 7f1e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.207+0000 7f1e99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.207+0000 7f1e99ffb640 10 monclient: dump: 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f1ea00208a0 con 0x7f1ea41419c0 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1e99ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1e99ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 msgr2=0x7f1ea406b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1e99ffb640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 0x7f1ea406b6b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0017000 tx=0x7f1ea0018000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1e99ffb640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 msgr2=0x7f1ea406b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1e99ffb640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea41419c0 0x7f1ea406b6b0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0017000 tx=0x7f1ea0018000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 34, 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:19.231 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.233 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:19.234 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:19.235 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:19.239 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.241 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.241 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:19.241 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:19.241 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.242 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.242 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.242 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:19.243 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:19.243 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.243 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.243 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.243 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.244 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:19.245 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:19.245 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.245 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.245 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:19.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1e88038490 msgr2=0x7f1e8803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1e88038490 0x7f1e8803a940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1ea411e6d0 tx=0x7f1e9c0025b0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 10 monclient: shutdown 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea40971e0 msgr2=0x7f1ea4117950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 --2- 172.21.15.116:0/4124370920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea40971e0 0x7f1ea4117950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009850 tx=0x7f1e94032000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eaa53a640 1 -- 172.21.15.116:0/4124370920 reap_dead start 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eaa53a640 1 -- 172.21.15.116:0/4124370920 reap_dead start 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 shutdown_connections 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 >> 172.21.15.116:0/4124370920 conn(0x7f1ea40672b0 msgr2=0x7f1ea409fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 shutdown_connections 2024-09-18T12:28:19.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.208+0000 7f1eabf9d640 1 -- 172.21.15.116:0/4124370920 wait complete. 2024-09-18T12:28:19.250 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:28:19.250 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:28:19.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:28:19.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.336+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v59) v1 ==== 154+0+0 (secure 0 0 0) 0x7f4814003000 con 0x7f480c0034c0 2024-09-18T12:28:19.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.336+0000 7f480bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:19.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.336+0000 7f480bfff640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-18T12:28:19.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.343+0000 7fe698e34640 10 monclient: get_monmap_and_config 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.343+0000 7fe698e34640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.343+0000 7fe698e34640 10 monclient: monmap: 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:19.344778+0000 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:19.344778+0000 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:19.345 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:19.346 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:19.346 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:19.346 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.343+0000 7fe698e34640 1 Processor -- start 2024-09-18T12:28:19.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.343+0000 7fe698e34640 1 -- start start 2024-09-18T12:28:19.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.343+0000 7fe698e34640 10 monclient: init 2024-09-18T12:28:19.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe694131ab0 0x7fe694131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient(hunting): picked mon.noname-a con 0x7fe694131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe69409e910 0x7fe69409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient(hunting): picked mon.noname-c con 0x7fe69409e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe69409f250 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69409faf0 con 0x7fe69409f250 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69409fd30 con 0x7fe694131ab0 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6940a1110 con 0x7fe69409e910 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe698e34640 10 monclient(hunting): authenticate will time out at 926.666496s 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe694131ab0 0x7fe694131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe69409e910 0x7fe69409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe69409e910 0x7fe69409ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:51344/0 (socket says 172.21.15.116:51344) 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 1 -- 172.21.15.116:0/2141437254 learned_addr learned my addr 172.21.15.116:0/2141437254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 10 monclient(hunting): get_auth_request con 0x7fe69409e910 auth_method 0 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): get_auth_request con 0x7fe694131ab0 auth_method 0 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe68bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient(hunting): handle_auth_done global_id 54117 payload 274 2024-09-18T12:28:19.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 1 -- 172.21.15.116:0/2141437254 >> v2:172.21.15.144:3300/0 conn(0x7fe69409e910 msgr2=0x7fe69409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 1 --2- 172.21.15.116:0/2141437254 >> v2:172.21.15.144:3300/0 conn(0x7fe69409e910 0x7fe69409ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 1 -- 172.21.15.116:0/2141437254 >> v1:172.21.15.116:6789/0 conn(0x7fe69409f250 legacy=0x7fe694068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 1 monclient: found mon.noname-a 2024-09-18T12:28:19.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.344+0000 7fe69259c640 1 -- 172.21.15.116:0/2141437254 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6940a13f0 con 0x7fe694131ab0 2024-09-18T12:28:19.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.347+0000 7fe69259c640 1 -- 172.21.15.116:0/2141437254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 msgr2=0x7fe694131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.347+0000 7fe69259c640 1 --2- 172.21.15.116:0/2141437254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe694131e80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe69409bdb0 tx=0x7fe6840090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c31706a72d26373 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.348+0000 7fe691d9b640 1 -- 172.21.15.116:0/2141437254 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68402adf0 con 0x7fe694131ab0 2024-09-18T12:28:19.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.348+0000 7fe691d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.348+0000 7fe691d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:19.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.348+0000 7fe691d9b640 10 monclient: dump: 2024-09-18T12:28:19.361 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.348+0000 7fe691d9b640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.348+0000 7fe691d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 5 monclient: authenticate success, global_id 54117 2024-09-18T12:28:19.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe691d9b640 1 -- 172.21.15.116:0/2141437254 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe684032440 con 0x7fe694131ab0 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe691d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe691d9b640 1 -- 172.21.15.116:0/2141437254 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6840327e0 con 0x7fe694131ab0 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 4 set_mon_vals no callback set 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe691d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe691d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe691d9b640 10 monclient: dump: 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.363 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.364 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.364 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.364 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.364 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 10 monclient: shutdown 2024-09-18T12:28:19.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 -- 172.21.15.116:0/2141437254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 msgr2=0x7fe694131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 --2- 172.21.15.116:0/2141437254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe694131e80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe69409bdb0 tx=0x7fe6840090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 -- 172.21.15.116:0/2141437254 shutdown_connections 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 --2- 172.21.15.116:0/2141437254 >> v2:172.21.15.144:3300/0 conn(0x7fe69409e910 0x7fe69409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 --2- 172.21.15.116:0/2141437254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe694131e80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 -- 172.21.15.116:0/2141437254 >> 172.21.15.116:0/2141437254 conn(0x7fe694067d30 msgr2=0x7fe6940a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 -- 172.21.15.116:0/2141437254 shutdown_connections 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.349+0000 7fe698e34640 1 -- 172.21.15.116:0/2141437254 wait complete. 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.350+0000 7fe698e34640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.350+0000 7fe698e34640 10 monclient: monmap: 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.350+0000 7fe698e34640 1 Processor -- start 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.350+0000 7fe698e34640 1 -- start start 2024-09-18T12:28:19.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.350+0000 7fe698e34640 10 monclient: init 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe69409e910 0x7fe69409ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): picked mon.noname-b con 0x7fe69409e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69409f250 0x7fe694130e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): picked mon.noname-c con 0x7fe69409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe69411b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): picked mon.noname-a con 0x7fe694131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694142100 con 0x7fe694131ab0 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69409b950 con 0x7fe69409e910 2024-09-18T12:28:19.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69409bde0 con 0x7fe69409f250 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe698e34640 10 monclient(hunting): authenticate will time out at 926.673151s 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe69411b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69409f250 0x7fe694130e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69409f250 0x7fe694130e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34152/0 (socket says 172.21.15.116:34152) 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe69411b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50692/0 (socket says 172.21.15.116:50692) 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe69409e910 0x7fe69409ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 1 -- 172.21.15.116:0/2152425537 learned_addr learned my addr 172.21.15.116:0/2152425537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 10 monclient(hunting): get_auth_request con 0x7fe69409f250 auth_method 0 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): get_auth_request con 0x7fe694131ab0 auth_method 0 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 10 monclient(hunting): get_auth_request con 0x7fe69409e910 auth_method 0 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient(hunting): handle_auth_done global_id 54120 payload 824 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69409f250 msgr2=0x7fe694130e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69409f250 0x7fe694130e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe69409e910 msgr2=0x7fe69409ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe69409e910 0x7fe69409ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 monclient: found mon.noname-a 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe692d9d640 1 -- 172.21.15.116:0/2152425537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe684009390 con 0x7fe694131ab0 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe68bfff640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69409f250 0x7fe694130e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.351+0000 7fe69259c640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe69409e910 0x7fe69409ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.353+0000 7fe692d9d640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 msgr2=0x7fe69411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.353+0000 7fe692d9d640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe69411b6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c7e0 tx=0x7fe68c0025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.353+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c00a850 con 0x7fe694131ab0 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: dump: 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.372 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c00a9d0 con 0x7fe694131ab0 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 5 monclient: authenticate success, global_id 54120 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 10 monclient: _renew_subs 2024-09-18T12:28:19.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6940a44c0 con 0x7fe694131ab0 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c00ad90 con 0x7fe694131ab0 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 10 monclient: dump: 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 10 monclient: _renew_subs 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe69159a640 4 set_mon_vals no callback set 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe694131380 con 0x7fe694131ab0 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe68c012630 con 0x7fe694131ab0 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 0x7fe69411bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69411e100 con 0x7fe694141620 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.354+0000 7fe69259c640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 0x7fe69411bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient: get_auth_request con 0x7fe694141620 auth_method 0 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe68a7fc640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe674038490 0x7fe67403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fe68c04bbc0 con 0x7fe694131ab0 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 10 monclient(hunting): handle_auth_done global_id 44285 payload 824 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe69259c640 1 -- 172.21.15.116:0/2152425537 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe69411d120 con 0x7fe694141620 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe68bfff640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe674038490 0x7fe67403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.355+0000 7fe68bfff640 10 monclient: get_auth_request con 0x7fe674038490 auth_method 0 2024-09-18T12:28:19.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe69259c640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 msgr2=0x7fe69411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe69259c640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 0x7fe69411bbf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe684034000 tx=0x7fe684032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68403c440 con 0x7fe694141620 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 10 monclient: dump: 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.377 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68bfff640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe674038490 0x7fe67403a940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe69411dbf0 tx=0x7fe67c0091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe68403c5a0 con 0x7fe694141620 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:19.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 msgr2=0x7fe69411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 0x7fe69411bbf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe684034000 tx=0x7fe684032000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 msgr2=0x7fe69411bbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:19.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.357+0000 7fe68a7fc640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe694141620 0x7fe69411bbf0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe684034000 tx=0x7fe684032000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe698e34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe698e34640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 0x7fe694096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe698e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe694063740 con 0x7fe69411c130 2024-09-18T12:28:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 0x7fe694096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient: get_auth_request con 0x7fe69411c130 auth_method 0 2024-09-18T12:28:19.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 10 monclient(hunting): handle_auth_done global_id 44288 payload 824 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.363+0000 7fe69259c640 1 -- 172.21.15.116:0/2152425537 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe69411c530 con 0x7fe69411c130 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe69259c640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 msgr2=0x7fe694096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe69259c640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 0x7fe694096a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe684009130 tx=0x7fe68403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684055070 con 0x7fe69411c130 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 10 monclient: dump: 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fe684051450 con 0x7fe69411c130 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 msgr2=0x7fe694096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 0x7fe694096a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe684009130 tx=0x7fe68403d000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 msgr2=0x7fe694096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe68a7fc640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe69411c130 0x7fe694096a60 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe684009130 tx=0x7fe68403d000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:28:19.384 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 34, 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.385 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.386 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.388 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:19.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:19.390 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.393 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.395 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:28:19.396 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:19.397 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe674038490 msgr2=0x7fe67403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe674038490 0x7fe67403a940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe69411dbf0 tx=0x7fe67c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 10 monclient: shutdown 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 msgr2=0x7fe69411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 --2- 172.21.15.116:0/2152425537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694131ab0 0x7fe69411b6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c7e0 tx=0x7fe68c0025b0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe692d9d640 1 -- 172.21.15.116:0/2152425537 reap_dead start 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe692d9d640 1 -- 172.21.15.116:0/2152425537 reap_dead start 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 shutdown_connections 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 >> 172.21.15.116:0/2152425537 conn(0x7fe694067d30 msgr2=0x7fe6940a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 shutdown_connections 2024-09-18T12:28:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.366+0000 7fe698e34640 1 -- 172.21.15.116:0/2152425537 wait complete. 2024-09-18T12:28:19.400 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:28:19.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:28:19.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:28:19.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.432+0000 7f482a499640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:19.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.432+0000 7f482a499640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:19.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.432+0000 7f482a499640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.432+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4824121150 con 0x7f480c0034c0 2024-09-18T12:28:19.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.433+0000 7f480bfff640 1 -- 172.21.15.144:0/2408621106 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v59) v1 ==== 161+0+0 (secure 0 0 0) 0x7f481401a3d0 con 0x7f480c0034c0 2024-09-18T12:28:19.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.433+0000 7f480bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:19.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.433+0000 7f480bfff640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-18T12:28:19.436 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'ec-foo' does not exist 2024-09-18T12:28:19.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48100384e0 msgr2=0x7f481003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48100384e0 0x7f481003a990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4818007510 tx=0x7f4818007d90 comp rx=0 tx=0).stop 2024-09-18T12:28:19.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 10 monclient: shutdown 2024-09-18T12:28:19.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 msgr2=0x7f4824134f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f4824134f20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f480c009290 tx=0x7f4814015000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 shutdown_connections 2024-09-18T12:28:19.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48100384e0 0x7f481003a990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4824069010 0x7f4824069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824135460 0x7f4824068ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 --2- 172.21.15.144:0/2408621106 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480c0034c0 0x7f4824134f20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.435+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 >> 172.21.15.144:0/2408621106 conn(0x7f4824070e90 msgr2=0x7f4824096a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.436+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 shutdown_connections 2024-09-18T12:28:19.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.436+0000 7f482a499640 1 -- 172.21.15.144:0/2408621106 wait complete. 2024-09-18T12:28:19.447 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls 2024-09-18T12:28:19.447 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo 2024-09-18T12:28:19.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.491+0000 7fa42c678640 10 monclient: get_monmap_and_config 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.491+0000 7fa42c678640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.491+0000 7fa42c678640 10 monclient: monmap: 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:19.493556+0000 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:19.493556+0000 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:19.495 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.492+0000 7fa42c678640 1 Processor -- start 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.492+0000 7fa42c678640 1 -- start start 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.492+0000 7fa42c678640 10 monclient: init 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa42409c9c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa42409e340 0x7fa42409e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient(hunting): picked mon.noname-a con 0x7fa42409e340 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa424134aa0 0x7fa424136e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient(hunting): picked mon.noname-c con 0x7fa424134aa0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42409bdd0 con 0x7fa42409c9c0 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42409ec90 con 0x7fa42409e340 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42409df40 con 0x7fa424134aa0 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42c678640 10 monclient(hunting): authenticate will time out at 926.815131s 2024-09-18T12:28:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa42409e340 0x7fa42409e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa42409e340 0x7fa42409e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50706/0 (socket says 172.21.15.116:50706) 2024-09-18T12:28:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 -- 172.21.15.116:0/765540360 learned_addr learned my addr 172.21.15.116:0/765540360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): get_auth_request con 0x7fa42409e340 auth_method 0 2024-09-18T12:28:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 1 --2- 172.21.15.116:0/765540360 >> v2:172.21.15.144:3300/0 conn(0x7fa424134aa0 0x7fa424136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 10 monclient(hunting): get_auth_request con 0x7fa424134aa0 auth_method 0 2024-09-18T12:28:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa42ac15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient(hunting): handle_auth_done global_id 54123 payload 274 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 -- 172.21.15.116:0/765540360 >> v2:172.21.15.144:3300/0 conn(0x7fa424134aa0 msgr2=0x7fa424136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 --2- 172.21.15.116:0/765540360 >> v2:172.21.15.144:3300/0 conn(0x7fa424134aa0 0x7fa424136e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 -- 172.21.15.116:0/765540360 >> v1:172.21.15.144:6789/0 conn(0x7fa42409c9c0 legacy=0x7fa42409d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 monclient: found mon.noname-a 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.493+0000 7fa429c13640 1 -- 172.21.15.116:0/765540360 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42409a880 con 0x7fa42409e340 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429c13640 1 -- 172.21.15.116:0/765540360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409e340 msgr2=0x7fa42409e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429c13640 1 --2- 172.21.15.116:0/765540360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409e340 0x7fa42409e750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa424066200 tx=0x7fa4140091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59c5d6963037902 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 1 -- 172.21.15.116:0/765540360 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414032440 con 0x7fa42409e340 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: dump: 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.511 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.512 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 5 monclient: authenticate success, global_id 54123 2024-09-18T12:28:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:19.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 1 -- 172.21.15.116:0/765540360 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4140325a0 con 0x7fa42409e340 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 4 set_mon_vals no callback set 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 1 -- 172.21.15.116:0/765540360 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414032980 con 0x7fa42409e340 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa429412640 10 monclient: dump: 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.513 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.514 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.514 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.514 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.514 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.514 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.514 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.515 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 10 monclient: shutdown 2024-09-18T12:28:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 1 -- 172.21.15.116:0/765540360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409e340 msgr2=0x7fa42409e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.496+0000 7fa42c678640 1 --2- 172.21.15.116:0/765540360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409e340 0x7fa42409e750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa424066200 tx=0x7fa4140091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 -- 172.21.15.116:0/765540360 shutdown_connections 2024-09-18T12:28:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 --2- 172.21.15.116:0/765540360 >> v2:172.21.15.144:3300/0 conn(0x7fa424134aa0 0x7fa424136e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 --2- 172.21.15.116:0/765540360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409e340 0x7fa42409e750 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 -- 172.21.15.116:0/765540360 >> 172.21.15.116:0/765540360 conn(0x7fa4240a2e40 msgr2=0x7fa4240a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 -- 172.21.15.116:0/765540360 shutdown_connections 2024-09-18T12:28:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 -- 172.21.15.116:0/765540360 wait complete. 2024-09-18T12:28:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 10 monclient: monmap: 2024-09-18T12:28:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:19.519 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 Processor -- start 2024-09-18T12:28:19.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 1 -- start start 2024-09-18T12:28:19.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.497+0000 7fa42c678640 10 monclient: init 2024-09-18T12:28:19.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:28:19.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409c9c0 0x7fa42411a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): picked mon.noname-a con 0x7fa42409c9c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:19.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42409e340 0x7fa42411a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): picked mon.noname-b con 0x7fa42409e340 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa424134aa0 0x7fa42411aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): picked mon.noname-c con 0x7fa424134aa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42411da20 con 0x7fa42409c9c0 2024-09-18T12:28:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa424099cc0 con 0x7fa42409e340 2024-09-18T12:28:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa424136d70 con 0x7fa424134aa0 2024-09-18T12:28:19.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42c678640 10 monclient(hunting): authenticate will time out at 926.820427s 2024-09-18T12:28:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409c9c0 0x7fa42411a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa424134aa0 0x7fa42411aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409c9c0 0x7fa42411a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50716/0 (socket says 172.21.15.116:50716) 2024-09-18T12:28:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 -- 172.21.15.116:0/2371435627 learned_addr learned my addr 172.21.15.116:0/2371435627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:19.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): get_auth_request con 0x7fa42409c9c0 auth_method 0 2024-09-18T12:28:19.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 10 monclient(hunting): get_auth_request con 0x7fa424134aa0 auth_method 0 2024-09-18T12:28:19.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42409e340 0x7fa42411a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 10 monclient(hunting): get_auth_request con 0x7fa42409e340 auth_method 0 2024-09-18T12:28:19.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42ac15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient(hunting): handle_auth_done global_id 54126 payload 824 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa424134aa0 msgr2=0x7fa42411aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa424134aa0 0x7fa42411aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42409e340 msgr2=0x7fa42411a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42409e340 0x7fa42411a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 monclient: found mon.noname-a 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.498+0000 7fa42a414640 1 -- 172.21.15.116:0/2371435627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa414009450 con 0x7fa42409c9c0 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42a414640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409c9c0 msgr2=0x7fa42411a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42a414640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409c9c0 0x7fa42411a120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c5e0 tx=0x7fa420007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420012840 con 0x7fa42409c9c0 2024-09-18T12:28:19.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:19.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: dump: 2024-09-18T12:28:19.529 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.529 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.529 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.530 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.530 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.530 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.530 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.530 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.531 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4200129a0 con 0x7fa42409c9c0 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 5 monclient: authenticate success, global_id 54126 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 10 monclient: _renew_subs 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42411c8f0 con 0x7fa42409c9c0 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420012da0 con 0x7fa42409c9c0 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa41a7fc640 10 monclient: dump: 2024-09-18T12:28:19.532 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.533 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.533 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.533 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.533 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.533 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.533 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.533 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.534 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.534 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa428c11640 4 set_mon_vals no callback set 2024-09-18T12:28:19.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 10 monclient: _renew_subs 2024-09-18T12:28:19.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4241347d0 con 0x7fa42409c9c0 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 0x7fa42406c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.501+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa424065e00 con 0x7fa42411b0e0 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa420011590 con 0x7fa42409c9c0 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 0x7fa42406c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient: get_auth_request con 0x7fa42411b0e0 auth_method 0 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa41a7fc640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa408038490 0x7fa40803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fa42004ba80 con 0x7fa42409c9c0 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 10 monclient(hunting): handle_auth_done global_id 4852 payload 824 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa429c13640 1 -- 172.21.15.116:0/2371435627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa42411dc20 con 0x7fa42411b0e0 2024-09-18T12:28:19.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa42ac15640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa408038490 0x7fa40803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.502+0000 7fa42ac15640 10 monclient: get_auth_request con 0x7fa408038490 auth_method 0 2024-09-18T12:28:19.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.504+0000 7fa429c13640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 msgr2=0x7fa42406c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.504+0000 7fa429c13640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 0x7fa42406c8c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa42404ca20 tx=0x7fa414032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.504+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4140319a0 con 0x7fa42411b0e0 2024-09-18T12:28:19.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.504+0000 7fa41a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.504+0000 7fa41a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:19.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.504+0000 7fa41a7fc640 10 monclient: dump: 2024-09-18T12:28:19.549 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.550 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.550 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.550 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.550 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.550 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.551 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.551 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.551 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.551 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.504+0000 7fa42ac15640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa408038490 0x7fa40803a940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009fd0 tx=0x7fa41c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.505+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa414031b00 con 0x7fa42411b0e0 2024-09-18T12:28:19.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.505+0000 7fa41a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.505+0000 7fa41a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:19.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.505+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 msgr2=0x7fa42406c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.505+0000 7fa41a7fc640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 0x7fa42406c8c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa42404ca20 tx=0x7fa414032000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.505+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 msgr2=0x7fa42406c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:19.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.505+0000 7fa41a7fc640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42411b0e0 0x7fa42406c8c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.510+0000 7fa42c678640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:19.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.510+0000 7fa42c678640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 0x7fa42406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.510+0000 7fa42c678640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.510+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42411d850 con 0x7fa42406ce00 2024-09-18T12:28:19.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 0x7fa42406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient: get_auth_request con 0x7fa42406ce00 auth_method 0 2024-09-18T12:28:19.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 10 monclient(hunting): handle_auth_done global_id 4855 payload 824 2024-09-18T12:28:19.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.511+0000 7fa429c13640 1 -- 172.21.15.116:0/2371435627 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa42404bf60 con 0x7fa42406ce00 2024-09-18T12:28:19.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.513+0000 7fa429c13640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 msgr2=0x7fa42406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.513+0000 7fa429c13640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 0x7fa42406d1d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa42411be10 tx=0x7fa41403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.513+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414031ed0 con 0x7fa42406ce00 2024-09-18T12:28:19.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.513+0000 7fa41a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.513+0000 7fa41a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:19.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.513+0000 7fa41a7fc640 10 monclient: dump: 2024-09-18T12:28:19.557 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.557 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.557 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.557 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.557 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.557 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.558 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.558 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.558 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.558 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fa414031790 con 0x7fa42406ce00 2024-09-18T12:28:19.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa41a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:19.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa41a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:19.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 msgr2=0x7fa42406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa41a7fc640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 0x7fa42406d1d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa42411be10 tx=0x7fa41403d000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa41a7fc640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 msgr2=0x7fa42406d1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:19.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa41a7fc640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa42406ce00 0x7fa42406d1d0 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa42411be10 tx=0x7fa41403d000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.560 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:28:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 34, 2024-09-18T12:28:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:28:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:28:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:28:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.565 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.565 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:19.565 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:19.565 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.565 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.565 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.566 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.566 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.566 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.566 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.566 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.566 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:19.567 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:19.568 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:19.570 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.572 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:19.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:19.574 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.575 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:19.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:19.577 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:19.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:19.577 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:19.577 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa408038490 msgr2=0x7fa40803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa408038490 0x7fa40803a940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009fd0 tx=0x7fa41c00b040 comp rx=0 tx=0).stop 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 10 monclient: shutdown 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409c9c0 msgr2=0x7fa42411a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 --2- 172.21.15.116:0/2371435627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa42409c9c0 0x7fa42411a120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c5e0 tx=0x7fa420007ed0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42ac15640 1 -- 172.21.15.116:0/2371435627 reap_dead start 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42ac15640 1 -- 172.21.15.116:0/2371435627 reap_dead start 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 shutdown_connections 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 >> 172.21.15.116:0/2371435627 conn(0x7fa4240a2e40 msgr2=0x7fa4240a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 shutdown_connections 2024-09-18T12:28:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.514+0000 7fa42c678640 1 -- 172.21.15.116:0/2371435627 wait complete. 2024-09-18T12:28:19.581 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:28:19.581 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:28:19.581 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:28:19.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.519+0000 7f6cb5e6f640 10 monclient: get_monmap_and_config 2024-09-18T12:28:19.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.519+0000 7f6cb5e6f640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.519+0000 7f6cb5e6f640 10 monclient: monmap: 2024-09-18T12:28:19.586 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:19.586 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.586 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:19.521251+0000 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:19.521251+0000 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 1 Processor -- start 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 1 -- start start 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient: init 2024-09-18T12:28:19.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb0069a30 0x7f6cb012f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb0069a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6cb009baa0 0x7f6cb009bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb009baa0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cb0070920 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb009e550 con 0x7f6cb0070920 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb0069e00 con 0x7f6cb009baa0 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb009d100 con 0x7f6cb0069a30 2024-09-18T12:28:19.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6cb5e6f640 10 monclient(hunting): authenticate will time out at 935.090616s 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6cb009baa0 0x7f6cb009bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6cb009baa0 0x7f6cb009bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46666/0 (socket says 172.21.15.144:46666) 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 -- 172.21.15.144:0/238406010 learned_addr learned my addr 172.21.15.144:0/238406010 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): get_auth_request con 0x7f6cb009baa0 auth_method 0 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 1 --2- 172.21.15.144:0/238406010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb0069a30 0x7f6cb012f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 10 monclient(hunting): get_auth_request con 0x7f6cb0069a30 auth_method 0 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient(hunting): handle_auth_done global_id 4858 payload 274 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 -- 172.21.15.144:0/238406010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb0069a30 msgr2=0x7f6cb012f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 --2- 172.21.15.144:0/238406010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb0069a30 0x7f6cb012f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 -- 172.21.15.144:0/238406010 >> v1:172.21.15.116:6789/0 conn(0x7f6cb0070920 legacy=0x7f6cb0070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 monclient: found mon.noname-c 2024-09-18T12:28:19.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.520+0000 7f6caeffd640 1 -- 172.21.15.144:0/238406010 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb009e270 con 0x7f6cb009baa0 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6caeffd640 1 -- 172.21.15.144:0/238406010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb009baa0 msgr2=0x7f6cb009bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6caeffd640 1 --2- 172.21.15.144:0/238406010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb009baa0 0x7f6cb009bea0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009910 tx=0x7f6ca40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3874946ff9d3874 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 1 -- 172.21.15.144:0/238406010 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca402ae90 con 0x7f6cb009baa0 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: dump: 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.591 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 1 -- 172.21.15.144:0/238406010 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca4032440 con 0x7f6cb009baa0 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cb5e6f640 5 monclient: authenticate success, global_id 4858 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cb5e6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cb5e6f640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 1 -- 172.21.15.144:0/238406010 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4032820 con 0x7f6cb009baa0 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cb5e6f640 4 set_mon_vals no callback set 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.523+0000 7f6cae7fc640 10 monclient: dump: 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.593 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 10 monclient: shutdown 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/238406010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb009baa0 msgr2=0x7f6cb009bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/238406010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb009baa0 0x7f6cb009bea0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009910 tx=0x7f6ca40091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/238406010 shutdown_connections 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/238406010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb009baa0 0x7f6cb009bea0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/238406010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb0069a30 0x7f6cb012f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/238406010 >> 172.21.15.144:0/238406010 conn(0x7f6cb009e9f0 msgr2=0x7f6cb009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/238406010 shutdown_connections 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/238406010 wait complete. 2024-09-18T12:28:19.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.524+0000 7f6cb5e6f640 10 monclient: monmap: 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 Processor -- start 2024-09-18T12:28:19.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 -- start start 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient: init 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb0069a30 0x7f6cb012eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb0069a30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 0x7f6cb0140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb0070920 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb009baa0 0x7f6cb0096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb009baa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb011df80 con 0x7f6cb0069a30 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb012f840 con 0x7f6cb0070920 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb011dca0 con 0x7f6cb009baa0 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6cb5e6f640 10 monclient(hunting): authenticate will time out at 935.095902s 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.525+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 0x7f6cb0140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 0x7f6cb0140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46670/0 (socket says 172.21.15.144:46670) 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 -- 172.21.15.144:0/2526327962 learned_addr learned my addr 172.21.15.144:0/2526327962 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): get_auth_request con 0x7f6cb0070920 auth_method 0 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb009baa0 0x7f6cb0096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb0069a30 0x7f6cb012eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 10 monclient(hunting): get_auth_request con 0x7f6cb0069a30 auth_method 0 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 10 monclient(hunting): get_auth_request con 0x7f6cb009baa0 auth_method 0 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient(hunting): handle_auth_done global_id 4861 payload 824 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 -- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb009baa0 msgr2=0x7f6cb0096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb009baa0 0x7f6cb0096a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 -- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb0069a30 msgr2=0x7f6cb012eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb0069a30 0x7f6cb012eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 monclient: found mon.noname-b 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.526+0000 7f6caeffd640 1 -- 172.21.15.144:0/2526327962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca4009450 con 0x7f6cb0070920 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6caeffd640 1 -- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 msgr2=0x7f6cb0140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6caeffd640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 0x7f6cb0140f50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009910 tx=0x7f6ca4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4000d10 con 0x7f6cb0070920 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: dump: 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.600 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cb5e6f640 5 monclient: authenticate success, global_id 4861 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca402ae90 con 0x7f6cb0070920 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cb5e6f640 10 monclient: _renew_subs 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cb5e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb011e260 con 0x7f6cb0070920 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4031ea0 con 0x7f6cb0070920 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6c977fe640 10 monclient: dump: 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.602 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cb5e6f640 10 monclient: _renew_subs 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cb5e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb00984b0 con 0x7f6cb0070920 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.528+0000 7f6cadffb640 4 set_mon_vals no callback set 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6ca4039040 con 0x7f6cb0070920 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6cb5e6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6cb5e6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6cb5e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb00008d0 con 0x7f6cb0070920 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6c977fe640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c9c038490 0x7f6c9c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f6ca406b700 con 0x7f6cb0070920 2024-09-18T12:28:19.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6caf7fe640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c9c038490 0x7f6c9c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.529+0000 7f6caf7fe640 10 monclient: get_auth_request con 0x7f6c9c038490 auth_method 0 2024-09-18T12:28:19.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.532+0000 7f6caf7fe640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c9c038490 0x7f6c9c03a940 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d20 tx=0x7f6ca00090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.534+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6ca402f2a0 con 0x7f6cb0070920 2024-09-18T12:28:19.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.534+0000 7f6c977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.534+0000 7f6c977fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.675+0000 7fc75b634640 10 monclient: get_monmap_and_config 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.675+0000 7fc75b634640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.675+0000 7fc75b634640 10 monclient: monmap: 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:19.676828+0000 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:19.676828+0000 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:19.678 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:19.678 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:19.678 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:19.678 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.675+0000 7fc75b634640 1 Processor -- start 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.675+0000 7fc75b634640 1 -- start start 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.675+0000 7fc75b634640 10 monclient: init 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc754131ab0 0x7fc754131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): picked mon.noname-a con 0x7fc754131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc75409e910 0x7fc75409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): picked mon.noname-c con 0x7fc75409e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): picked mon.noname-b con 0x7fc75409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75409faf0 con 0x7fc754131ab0 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75409fd30 con 0x7fc75409e910 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540a1110 con 0x7fc75409f250 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc75b634640 10 monclient(hunting): authenticate will time out at 926.998440s 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc754131ab0 0x7fc754131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc754131ab0 0x7fc754131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50722/0 (socket says 172.21.15.116:50722) 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34190/0 (socket says 172.21.15.116:34190) 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 1 -- 172.21.15.116:0/3889212389 learned_addr learned my addr 172.21.15.116:0/3889212389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:19.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 10 monclient(hunting): get_auth_request con 0x7fc754131ab0 auth_method 0 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): get_auth_request con 0x7fc75409f250 auth_method 0 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc758bcf640 1 --2- 172.21.15.116:0/3889212389 >> v2:172.21.15.144:3300/0 conn(0x7fc75409e910 0x7fc75409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc758bcf640 10 monclient(hunting): get_auth_request con 0x7fc75409e910 auth_method 0 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc758bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc758bcf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc758bcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient(hunting): handle_auth_done global_id 44300 payload 274 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 -- 172.21.15.116:0/3889212389 >> v2:172.21.15.144:3300/0 conn(0x7fc75409e910 msgr2=0x7fc75409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 --2- 172.21.15.116:0/3889212389 >> v2:172.21.15.144:3300/0 conn(0x7fc75409e910 0x7fc75409ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 -- 172.21.15.116:0/3889212389 >> v2:172.21.15.116:3300/0 conn(0x7fc754131ab0 msgr2=0x7fc754131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 --2- 172.21.15.116:0/3889212389 >> v2:172.21.15.116:3300/0 conn(0x7fc754131ab0 0x7fc754131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 monclient: found mon.noname-b 2024-09-18T12:28:19.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:28:19.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc759bd1640 1 -- 172.21.15.116:0/3889212389 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7540a13f0 con 0x7fc75409f250 2024-09-18T12:28:19.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc758bcf640 1 --2- 172.21.15.116:0/3889212389 >> v2:172.21.15.144:3300/0 conn(0x7fc75409e910 0x7fc75409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:19.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.676+0000 7fc7593d0640 1 --2- 172.21.15.116:0/3889212389 >> v2:172.21.15.116:3300/0 conn(0x7fc754131ab0 0x7fc754131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc759bd1640 1 -- 172.21.15.116:0/3889212389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 msgr2=0x7fc754068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc759bd1640 1 --2- 172.21.15.116:0/3889212389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754068ce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc75409bdb0 tx=0x7fc74c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44d0c142b73e31e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 1 -- 172.21.15.116:0/3889212389 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c007760 con 0x7fc75409f250 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: dump: 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 1 -- 172.21.15.116:0/3889212389 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c02d050 con 0x7fc75409f250 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 1 -- 172.21.15.116:0/3889212389 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c007b30 con 0x7fc75409f250 2024-09-18T12:28:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc75b634640 5 monclient: authenticate success, global_id 44300 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc75b634640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc75b634640 4 set_mon_vals no callback set 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.679+0000 7fc74bfff640 10 monclient: dump: 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 10 monclient: shutdown 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 -- 172.21.15.116:0/3889212389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 msgr2=0x7fc754068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 --2- 172.21.15.116:0/3889212389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754068ce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc75409bdb0 tx=0x7fc74c009780 comp rx=0 tx=0).stop 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 -- 172.21.15.116:0/3889212389 shutdown_connections 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 --2- 172.21.15.116:0/3889212389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754068ce0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 --2- 172.21.15.116:0/3889212389 >> v2:172.21.15.144:3300/0 conn(0x7fc75409e910 0x7fc75409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 --2- 172.21.15.116:0/3889212389 >> v2:172.21.15.116:3300/0 conn(0x7fc754131ab0 0x7fc754131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 -- 172.21.15.116:0/3889212389 >> 172.21.15.116:0/3889212389 conn(0x7fc754067d30 msgr2=0x7fc7540a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 -- 172.21.15.116:0/3889212389 shutdown_connections 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 1 -- 172.21.15.116:0/3889212389 wait complete. 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.680+0000 7fc75b634640 10 monclient: monmap: 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 Processor -- start 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 -- start start 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient: init 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75409e910 0x7fc75409a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): picked mon.noname-b con 0x7fc75409e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): picked mon.noname-c con 0x7fc75409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 0x7fc75411b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): picked mon.noname-a con 0x7fc754131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7541345f0 con 0x7fc754131ab0 2024-09-18T12:28:19.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75411cbc0 con 0x7fc75409e910 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75411e2c0 con 0x7fc75409f250 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc75b634640 10 monclient(hunting): authenticate will time out at 927.003862s 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc759bd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 0x7fc75411b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc758bcf640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc759bd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 0x7fc75411b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50734/0 (socket says 172.21.15.116:50734) 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.681+0000 7fc758bcf640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754130900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34194/0 (socket says 172.21.15.116:34194) 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 1 -- 172.21.15.116:0/2021168809 learned_addr learned my addr 172.21.15.116:0/2021168809 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): get_auth_request con 0x7fc754131ab0 auth_method 0 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 10 monclient(hunting): get_auth_request con 0x7fc75409f250 auth_method 0 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75409e910 0x7fc75409a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 10 monclient(hunting): get_auth_request con 0x7fc75409e910 auth_method 0 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc7593d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient(hunting): handle_auth_done global_id 54135 payload 824 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 1 -- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 msgr2=0x7fc754130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754130900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 1 -- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75409e910 msgr2=0x7fc75409a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75409e910 0x7fc75409a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 1 monclient: found mon.noname-a 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc759bd1640 1 -- 172.21.15.116:0/2021168809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750009450 con 0x7fc754131ab0 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.682+0000 7fc758bcf640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754130900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc759bd1640 1 -- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 msgr2=0x7fc75411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc759bd1640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 0x7fc75411b6b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009e90 tx=0x7fc74c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 1 -- 172.21.15.116:0/2021168809 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c031b70 con 0x7fc754131ab0 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: dump: 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 1 -- 172.21.15.116:0/2021168809 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c040070 con 0x7fc754131ab0 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc75b634640 5 monclient: authenticate success, global_id 54135 2024-09-18T12:28:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc75b634640 10 monclient: _renew_subs 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc75b634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c009a20 con 0x7fc754131ab0 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 1 -- 172.21.15.116:0/2021168809 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c045430 con 0x7fc754131ab0 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc7497fa640 10 monclient: dump: 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc74b7fe640 4 set_mon_vals no callback set 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc75b634640 10 monclient: _renew_subs 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc75b634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.684+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc754135f70 con 0x7fc754131ab0 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc7497fa640 1 -- 172.21.15.116:0/2021168809 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc74c042030 con 0x7fc754131ab0 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc7497fa640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc738038490 0x7fc73803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc75b634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc75b634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc75b634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc754136190 con 0x7fc754131ab0 2024-09-18T12:28:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc7497fa640 1 -- 172.21.15.116:0/2021168809 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fc74c06d1a0 con 0x7fc754131ab0 2024-09-18T12:28:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc7593d0640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc738038490 0x7fc73803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.685+0000 7fc7593d0640 10 monclient: get_auth_request con 0x7fc738038490 auth_method 0 2024-09-18T12:28:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.688+0000 7fc7593d0640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc738038490 0x7fc73803a940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc750006c90 tx=0x7fc7500091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.690+0000 7fc7497fa640 1 -- 172.21.15.116:0/2021168809 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc74c03ab80 con 0x7fc754131ab0 2024-09-18T12:28:19.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.690+0000 7fc7497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.690+0000 7fc7497fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:19.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.691+0000 7f6cb5e6f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:19.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.691+0000 7f6cb5e6f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:19.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.691+0000 7f6cb5e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.691+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f6cb009dc80 con 0x7f6cb0070920 2024-09-18T12:28:19.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.692+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (secure 0 0 0) 0x7f6ca4037050 con 0x7f6cb0070920 2024-09-18T12:28:19.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.692+0000 7f6c977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:19.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.692+0000 7f6c977fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:19.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.786+0000 7f6cb5e6f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:19.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.786+0000 7f6cb5e6f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:19.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.786+0000 7f6cb5e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.786+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f6cb009dc80 con 0x7f6cb0070920 2024-09-18T12:28:19.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.787+0000 7f6c977fe640 1 -- 172.21.15.144:0/2526327962 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) v1 ==== 65+0+60 (secure 0 0 0) 0x7f6ca4042030 con 0x7f6cb0070920 2024-09-18T12:28:19.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.787+0000 7f6c977fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:19.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.787+0000 7f6c977fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:28:19.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.788+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c9c038490 msgr2=0x7f6c9c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.788+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c9c038490 0x7f6c9c03a940 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d20 tx=0x7f6ca00090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.788+0000 7f6cb5e6f640 10 monclient: shutdown 2024-09-18T12:28:19.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.788+0000 7f6cb5e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.788+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 msgr2=0x7f6cb0140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.788+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 0x7f6cb0140f50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009910 tx=0x7f6ca4032000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 shutdown_connections 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c9c038490 0x7f6c9c03a940 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6cb009baa0 0x7f6cb0096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6cb0070920 0x7f6cb0140f50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 --2- 172.21.15.144:0/2526327962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb0069a30 0x7f6cb012eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 >> 172.21.15.144:0/2526327962 conn(0x7f6cb009e9f0 msgr2=0x7f6cb00a0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 shutdown_connections 2024-09-18T12:28:19.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.789+0000 7f6cb5e6f640 1 -- 172.21.15.144:0/2526327962 wait complete. 2024-09-18T12:28:19.799 INFO:tasks.workunit.client.0.smithi144.stdout:foo 2024-09-18T12:28:19.800 INFO:tasks.workunit.client.0.smithi144.stdout:foo-ssd 2024-09-18T12:28:19.800 INFO:tasks.workunit.client.0.smithi144.stdout:foo-hdd 2024-09-18T12:28:19.800 INFO:tasks.workunit.client.0.smithi144.stdout:ec-foo 2024-09-18T12:28:19.800 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-18T12:28:19.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.873+0000 7fbc32354640 10 monclient: get_monmap_and_config 2024-09-18T12:28:19.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.873+0000 7fbc32354640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.873+0000 7fbc32354640 10 monclient: monmap: 2024-09-18T12:28:19.876 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:19.876 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.876 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:19.875829+0000 2024-09-18T12:28:19.876 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:19.875829+0000 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:19.877 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.874+0000 7fbc32354640 1 Processor -- start 2024-09-18T12:28:19.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.874+0000 7fbc32354640 1 -- start start 2024-09-18T12:28:19.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.874+0000 7fbc32354640 10 monclient: init 2024-09-18T12:28:19.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbc2c0a0ad0 0x7fbc2c0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient(hunting): picked mon.noname-a con 0x7fbc2c0a0ad0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc2c09ee60 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc2c09f260 0x7fbc2c1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient(hunting): picked mon.noname-c con 0x7fbc2c09f260 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c06a5b0 con 0x7fbc2c09ee60 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c0a0ea0 con 0x7fbc2c0a0ad0 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c0a1bb0 con 0x7fbc2c09f260 2024-09-18T12:28:19.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc32354640 10 monclient(hunting): authenticate will time out at 935.445271s 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc2c09f260 0x7fbc2c1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc2c09f260 0x7fbc2c1321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46686/0 (socket says 172.21.15.144:46686) 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 -- 172.21.15.144:0/976239262 learned_addr learned my addr 172.21.15.144:0/976239262 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): get_auth_request con 0x7fbc2c09f260 auth_method 0 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 1 --2- 172.21.15.144:0/976239262 >> v2:172.21.15.116:3300/0 conn(0x7fbc2c0a0ad0 0x7fbc2c0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 10 monclient(hunting): get_auth_request con 0x7fbc2c0a0ad0 auth_method 0 2024-09-18T12:28:19.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient(hunting): handle_auth_done global_id 4870 payload 274 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 -- 172.21.15.144:0/976239262 >> v2:172.21.15.116:3300/0 conn(0x7fbc2c0a0ad0 msgr2=0x7fbc2c0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 --2- 172.21.15.144:0/976239262 >> v2:172.21.15.116:3300/0 conn(0x7fbc2c0a0ad0 0x7fbc2c0691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 -- 172.21.15.144:0/976239262 >> v1:172.21.15.116:6789/0 conn(0x7fbc2c09ee60 legacy=0x7fbc2c131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 monclient: found mon.noname-c 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:19.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.875+0000 7fbc308f1640 1 -- 172.21.15.144:0/976239262 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0a0460 con 0x7fbc2c09f260 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc308f1640 1 -- 172.21.15.144:0/976239262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c09f260 msgr2=0x7fbc2c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc308f1640 1 --2- 172.21.15.144:0/976239262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c09f260 0x7fbc2c1321e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0675c0 tx=0x7fbc24009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fddaec726870bb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 1 -- 172.21.15.144:0/976239262 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24007610 con 0x7fbc2c09f260 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: dump: 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.888 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 1 -- 172.21.15.144:0/976239262 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2402d050 con 0x7fbc2c09f260 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc32354640 5 monclient: authenticate success, global_id 4870 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc32354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc32354640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:19.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc32354640 4 set_mon_vals no callback set 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 1 -- 172.21.15.144:0/976239262 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc240079a0 con 0x7fbc2c09f260 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.878+0000 7fbc2affd640 10 monclient: dump: 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.890 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 10 monclient: shutdown 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 -- 172.21.15.144:0/976239262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c09f260 msgr2=0x7fbc2c1321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 --2- 172.21.15.144:0/976239262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c09f260 0x7fbc2c1321e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0675c0 tx=0x7fbc24009840 comp rx=0 tx=0).stop 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 -- 172.21.15.144:0/976239262 shutdown_connections 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 --2- 172.21.15.144:0/976239262 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c09f260 0x7fbc2c1321e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 --2- 172.21.15.144:0/976239262 >> v2:172.21.15.116:3300/0 conn(0x7fbc2c0a0ad0 0x7fbc2c0691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 -- 172.21.15.144:0/976239262 >> 172.21.15.144:0/976239262 conn(0x7fbc2c071370 msgr2=0x7fbc2c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 -- 172.21.15.144:0/976239262 shutdown_connections 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 -- 172.21.15.144:0/976239262 wait complete. 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 10 monclient: build_initial_monmap 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 10 monclient: monmap: 2024-09-18T12:28:19.891 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.879+0000 7fbc32354640 1 Processor -- start 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 1 -- start start 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient: init 2024-09-18T12:28:19.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 0x7fbc2c13d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): picked mon.noname-a con 0x7fbc2c09ee60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc2c09f260 0x7fbc2c13cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): picked mon.noname-c con 0x7fbc2c09f260 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c0a0ad0 0x7fbc2c140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): picked mon.noname-b con 0x7fbc2c0a0ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c122700 con 0x7fbc2c09ee60 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c122140 con 0x7fbc2c0a0ad0 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:19.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c11fbd0 con 0x7fbc2c09f260 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc32354640 10 monclient(hunting): authenticate will time out at 935.450666s 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c0a0ad0 0x7fbc2c140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c0a0ad0 0x7fbc2c140c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46700/0 (socket says 172.21.15.144:46700) 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 1 -- 172.21.15.144:0/3798622899 learned_addr learned my addr 172.21.15.144:0/3798622899 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 10 monclient(hunting): get_auth_request con 0x7fbc2c0a0ad0 auth_method 0 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 0x7fbc2c13d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 10 monclient(hunting): get_auth_request con 0x7fbc2c09ee60 auth_method 0 2024-09-18T12:28:19.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc2c09f260 0x7fbc2c13cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc308f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 10 monclient(hunting): get_auth_request con 0x7fbc2c09f260 auth_method 0 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.880+0000 7fbc2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 10 monclient(hunting): handle_auth_done global_id 54141 payload 824 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 1 -- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc2c09f260 msgr2=0x7fbc2c13cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc2c09f260 0x7fbc2c13cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 1 -- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c0a0ad0 msgr2=0x7fbc2c140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c0a0ad0 0x7fbc2c140c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 1 monclient: found mon.noname-a 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc2bfff640 1 -- 172.21.15.144:0/3798622899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc24009ae0 con 0x7fbc2c09ee60 2024-09-18T12:28:19.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.881+0000 7fbc308f1640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c0a0ad0 0x7fbc2c140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:19.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc2bfff640 1 -- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 msgr2=0x7fbc2c13d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc2bfff640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 0x7fbc2c13d710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00e990 tx=0x7fbc1c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c00cb50 con 0x7fbc2c09ee60 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: dump: 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.899 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: _finish_auth 0 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c00ccb0 con 0x7fbc2c09ee60 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc32354640 5 monclient: authenticate success, global_id 54141 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc32354640 10 monclient: _renew_subs 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc32354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c121e60 con 0x7fbc2c09ee60 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c013470 con 0x7fbc2c09ee60 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc13fff640 10 monclient: dump: 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:19.901 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc32354640 10 monclient: _renew_subs 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc2a7fc640 4 set_mon_vals no callback set 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc32354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.883+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c13d4a0 con 0x7fbc2c09ee60 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc2bfff640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 0x7fbc2c13d710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00e990 tx=0x7fbc1c0090f0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fbc1c04bc00 osd_map(59..59 src has 1..59) v4 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbc1c013860 con 0x7fbc2c09ee60 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc32354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc32354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc32354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2c0008d0 con 0x7fbc2c09ee60 2024-09-18T12:28:19.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc13fff640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc18038490 0x7fbc1803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:19.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc2b7fe640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc18038490 0x7fbc1803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:19.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.884+0000 7fbc2b7fe640 10 monclient: get_auth_request con 0x7fbc18038490 auth_method 0 2024-09-18T12:28:19.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.887+0000 7fbc2b7fe640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc18038490 0x7fbc1803a940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c09ca60 tx=0x7fbc20007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:19.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:19.889+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fbc1c04bc00 con 0x7fbc2c09ee60 2024-09-18T12:28:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.902+0000 7fc75b634640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:28:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.902+0000 7fc75b634640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:28:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.902+0000 7fc75b634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.902+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc75404bf60 con 0x7fc754131ab0 2024-09-18T12:28:19.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.903+0000 7fc7497fa640 1 -- 172.21.15.116:0/2021168809 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc74c03a3a0 con 0x7fc754131ab0 2024-09-18T12:28:19.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.903+0000 7fc7497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:28:19.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.903+0000 7fc7497fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:19.906 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.903+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.906 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.903+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=713539544,config=3140514009,health=2846003242,kv=96606817,logm=1494795029}) 2024-09-18T12:28:19.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.903+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=4005897190,config=3140514009,health=3279035843,kv=2187923472,logm=260162052}) 2024-09-18T12:28:19.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.903+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.903+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=713539544,config=3140514009,health=2846003242,kv=96606817,logm=1494795029}) 2024-09-18T12:28:19.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.903+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=11,kv=9,logm=68} crc {auth=2776203829,config=3140514009,health=3549464964,kv=2187923472,logm=1943627940}) 2024-09-18T12:28:19.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.904+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.904+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735826155}) 2024-09-18T12:28:19.908 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.904+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1977142801}) 2024-09-18T12:28:19.908 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.904+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.908 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.904+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735826155}) 2024-09-18T12:28:19.908 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.904+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3240868076}) 2024-09-18T12:28:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc738038490 msgr2=0x7fc73803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc738038490 0x7fc73803a940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc750006c90 tx=0x7fc7500091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 10 monclient: shutdown 2024-09-18T12:28:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 msgr2=0x7fc75411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:19.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 0x7fc75411b6b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009e90 tx=0x7fc74c037000 comp rx=0 tx=0).stop 2024-09-18T12:28:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 shutdown_connections 2024-09-18T12:28:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc738038490 0x7fc73803a940 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc754131ab0 0x7fc75411b6b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc75409f250 0x7fc754130900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 --2- 172.21.15.116:0/2021168809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc75409e910 0x7fc75409a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:19.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 >> 172.21.15.116:0/2021168809 conn(0x7fc754067d30 msgr2=0x7fc754134880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 shutdown_connections 2024-09-18T12:28:19.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:19.906+0000 7fc75b634640 1 -- 172.21.15.116:0/2021168809 wait complete. 2024-09-18T12:28:19.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:28:19.920 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.920 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327276050}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3332307564}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327276050}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3901122602}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564702036}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975251547}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564702036}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.916+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1161535355}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.917+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.917+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661130289}) 2024-09-18T12:28:19.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.917+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3814271020}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.917+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.917+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661130289}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.917+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446548154}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046402152}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997074420}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046402152}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1141352661}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433715065}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=756535769}) 2024-09-18T12:28:19.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.923 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433715065}) 2024-09-18T12:28:19.923 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.918+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151002184}) 2024-09-18T12:28:19.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.919+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.919+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=902500691}) 2024-09-18T12:28:19.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.919+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3158219811}) 2024-09-18T12:28:19.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.919+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.919+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=902500691}) 2024-09-18T12:28:19.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.919+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508179550}) 2024-09-18T12:28:19.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.920+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.920+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=26} crc {logm=3058137973,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=896561286,mon_config_key=3629230923,monmap=148614106,osd_metadata=943163813,osd_pg_creating=2420761471,osd_snap=4249038086,osdmap=643115645}) 2024-09-18T12:28:19.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.920+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=14,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=22} crc {logm=972167330,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgrstat=1418688822,mon_config_key=3655569570,monmap=148614106,osd_metadata=4229638783,osd_pg_creating=2420761471,osd_snap=4249038086,osdmap=1122776310}) 2024-09-18T12:28:19.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.920+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.920+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=26} crc {logm=3058137973,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=896561286,mon_config_key=3629230923,monmap=148614106,osd_metadata=943163813,osd_pg_creating=2420761471,osd_snap=4249038086,osdmap=643115645}) 2024-09-18T12:28:19.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.920+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=24} crc {logm=2643540013,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgrstat=2339227461,mon_config_key=194675655,monmap=148614106,osd_metadata=687973059,osd_pg_creating=2420761471,osd_snap=2033388227,osdmap=2849958548}) 2024-09-18T12:28:19.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.921+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.921+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=1393934419}) 2024-09-18T12:28:19.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.921+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=76} crc {osdmap=3313289846}) 2024-09-18T12:28:19.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.921+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:28:19.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.921+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=1393934419}) 2024-09-18T12:28:19.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:28:19.921+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=73} crc {osdmap=602073272}) 2024-09-18T12:28:20.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:20.712+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbc1c023ab0 con 0x7fbc2c09ee60 2024-09-18T12:28:20.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:20.712+0000 7fbc13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:20.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:20.712+0000 7fbc13fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:20.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:20.873+0000 7fbc32354640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:20.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:20.873+0000 7fbc32354640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:20.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:20.873+0000 7fbc32354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:20.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:20.873+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fbc2c064a20 con 0x7fbc2c09ee60 2024-09-18T12:28:20.919 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:28:20.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:28:21.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.015+0000 7f4aad460640 10 monclient: get_monmap_and_config 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.015+0000 7f4aad460640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.015+0000 7f4aad460640 10 monclient: monmap: 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:21.017579+0000 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:21.017579+0000 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:21.019 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:21.019 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.016+0000 7f4aad460640 1 Processor -- start 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.016+0000 7f4aad460640 1 -- start start 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.016+0000 7f4aad460640 10 monclient: init 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4aa8131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4aa809e910 0x7f4aa809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa809e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4aa809f250 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa809faf0 con 0x7f4aa8131ab0 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa809fd30 con 0x7f4aa809f250 2024-09-18T12:28:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa8132590 con 0x7f4aa809e910 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aad460640 10 monclient(hunting): authenticate will time out at 928.339200s 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4aa809e910 0x7f4aa809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aa6ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4aa8131ab0 0x7f4aa8131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41460/0 (socket says 172.21.15.116:41460) 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4aa809e910 0x7f4aa809ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50760/0 (socket says 172.21.15.116:50760) 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4aa6ffd640 1 -- 172.21.15.116:0/136613356 learned_addr learned my addr 172.21.15.116:0/136613356 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request con 0x7f4aa809e910 auth_method 0 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_done global_id 54144 payload 274 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 1 -- 172.21.15.116:0/136613356 >> v1:172.21.15.144:6789/0 conn(0x7f4aa809f250 legacy=0x7f4aa8138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 1 -- 172.21.15.116:0/136613356 >> v1:172.21.15.116:6789/0 conn(0x7f4aa8131ab0 legacy=0x7f4aa8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 1 monclient: found mon.noname-a 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.017+0000 7f4a9ffff640 1 -- 172.21.15.116:0/136613356 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa80a1110 con 0x7f4aa809e910 2024-09-18T12:28:21.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4a9ffff640 1 -- 172.21.15.116:0/136613356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 msgr2=0x7f4aa809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4a9ffff640 1 --2- 172.21.15.116:0/136613356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa809ed10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009850 tx=0x7f4a900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34737f4043b71bd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 1 -- 172.21.15.116:0/136613356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90032440 con 0x7f4aa809e910 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: dump: 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 1 -- 172.21.15.116:0/136613356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a900325a0 con 0x7f4aa809e910 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aad460640 5 monclient: authenticate success, global_id 54144 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aad460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aad460640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 1 -- 172.21.15.116:0/136613356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90032940 con 0x7f4aa809e910 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aad460640 4 set_mon_vals no callback set 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.020+0000 7f4aa67fc640 10 monclient: dump: 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 10 monclient: shutdown 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 1 -- 172.21.15.116:0/136613356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 msgr2=0x7f4aa809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 1 --2- 172.21.15.116:0/136613356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa809ed10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009850 tx=0x7f4a900090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 1 -- 172.21.15.116:0/136613356 shutdown_connections 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 1 --2- 172.21.15.116:0/136613356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa809ed10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 1 -- 172.21.15.116:0/136613356 >> 172.21.15.116:0/136613356 conn(0x7f4aa8067d30 msgr2=0x7f4aa80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 1 -- 172.21.15.116:0/136613356 shutdown_connections 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 1 -- 172.21.15.116:0/136613356 wait complete. 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.021+0000 7f4aad460640 10 monclient: monmap: 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.032 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 Processor -- start 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 -- start start 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient: init 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa8130070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa809e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4aa809f250 0x7f4aa81305b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4aa8131ab0 0x7f4aa8130af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa8131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8098d10 con 0x7f4aa809e910 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8099370 con 0x7f4aa8131ab0 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8097310 con 0x7f4aa809f250 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aad460640 10 monclient(hunting): authenticate will time out at 928.344570s 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa8130070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4aa809f250 0x7f4aa81305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa8130070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50768/0 (socket says 172.21.15.116:50768) 2024-09-18T12:28:21.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 1 -- 172.21.15.116:0/640058084 learned_addr learned my addr 172.21.15.116:0/640058084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request con 0x7f4aa809f250 auth_method 0 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 10 monclient(hunting): get_auth_request con 0x7f4aa809e910 auth_method 0 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa77fe640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4aa8131ab0 0x7f4aa8130af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_done global_id 44309 payload 824 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 1 -- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4aa8131ab0 msgr2=0x7f4aa8130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4aa8131ab0 0x7f4aa8130af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 1 -- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 msgr2=0x7f4aa8130070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa8130070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 1 monclient: found mon.noname-c 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4a9ffff640 1 -- 172.21.15.116:0/640058084 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90009390 con 0x7f4aa809f250 2024-09-18T12:28:21.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa77fe640 10 monclient: get_auth_request con 0x7f4aa8131ab0 auth_method 0 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa6ffd640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa8130070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.022+0000 7f4aa77fe640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4aa8131ab0 0x7f4aa8130af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9ffff640 1 -- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4aa809f250 msgr2=0x7f4aa81305b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9ffff640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4aa809f250 0x7f4aa81305b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a90034000 tx=0x7f4a90032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 1 -- 172.21.15.116:0/640058084 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90031990 con 0x7f4aa809f250 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: dump: 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 1 -- 172.21.15.116:0/640058084 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a90031b10 con 0x7f4aa809f250 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aad460640 5 monclient: authenticate success, global_id 44309 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aad460640 10 monclient: _renew_subs 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aad460640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa809be40 con 0x7f4aa809f250 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 1 -- 172.21.15.116:0/640058084 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9003c410 con 0x7f4aa809f250 2024-09-18T12:28:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4a9effd640 10 monclient: dump: 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aa5ffb640 4 set_mon_vals no callback set 2024-09-18T12:28:21.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aad460640 10 monclient: _renew_subs 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aad460640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.025+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa809aaf0 con 0x7f4aa809f250 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4a9effd640 1 -- 172.21.15.116:0/640058084 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4a90039070 con 0x7f4aa809f250 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4aad460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4aad460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4aad460640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa804bf60 con 0x7f4aa809f250 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4a9effd640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a88038490 0x7f4a8803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4a9effd640 1 -- 172.21.15.116:0/640058084 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f4a9006c310 con 0x7f4aa809f250 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4aa6ffd640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a88038490 0x7f4a8803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.026+0000 7f4aa6ffd640 10 monclient: get_auth_request con 0x7f4a88038490 auth_method 0 2024-09-18T12:28:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.029+0000 7f4aa6ffd640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a88038490 0x7f4a8803a940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a980050e0 tx=0x7f4a98004960 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.031+0000 7f4a9effd640 1 -- 172.21.15.116:0/640058084 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4a9003c840 con 0x7f4aa809f250 2024-09-18T12:28:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.031+0000 7f4a9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.031+0000 7f4a9effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.246+0000 7f4aad460640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:28:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.246+0000 7f4aad460640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.246+0000 7f4aad460640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.246+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4aa804bf60 con 0x7f4aa809f250 2024-09-18T12:28:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.246+0000 7f4a9effd640 1 -- 172.21.15.116:0/640058084 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f4a90040080 con 0x7f4aa809f250 2024-09-18T12:28:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.246+0000 7f4a9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.246+0000 7f4a9effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:21.249 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:28:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a88038490 msgr2=0x7f4a8803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a88038490 0x7f4a8803a940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a980050e0 tx=0x7f4a98004960 comp rx=0 tx=0).stop 2024-09-18T12:28:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 10 monclient: shutdown 2024-09-18T12:28:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4aa809f250 msgr2=0x7f4aa81305b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4aa809f250 0x7f4aa81305b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4a90034000 tx=0x7f4a90032000 comp rx=0 tx=0).stop 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 shutdown_connections 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4a88038490 0x7f4a8803a940 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4aa8131ab0 0x7f4aa8130af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4aa809f250 0x7f4aa81305b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 --2- 172.21.15.116:0/640058084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa809e910 0x7f4aa8130070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 >> 172.21.15.116:0/640058084 conn(0x7f4aa8067d30 msgr2=0x7f4aa809c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 shutdown_connections 2024-09-18T12:28:21.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.248+0000 7f4aad460640 1 -- 172.21.15.116:0/640058084 wait complete. 2024-09-18T12:28:21.260 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:28:21.260 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:28:21.260 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:28:21.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:28:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.351+0000 7f3b98e4f640 10 monclient: get_monmap_and_config 2024-09-18T12:28:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.351+0000 7f3b98e4f640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.351+0000 7f3b98e4f640 10 monclient: monmap: 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:21.353102+0000 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:21.353102+0000 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:21.354 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 1 Processor -- start 2024-09-18T12:28:21.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 1 -- start start 2024-09-18T12:28:21.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient: init 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b941318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b9409ea50 0x7f3b9409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient(hunting): picked mon.noname-a con 0x7f3b9409ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b94136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b94131cc0 con 0x7f3b941318f0 2024-09-18T12:28:21.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9409fbf0 con 0x7f3b94136c20 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9409fe30 con 0x7f3b9409ea50 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b98e4f640 10 monclient(hunting): authenticate will time out at 928.674866s 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b9259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3b941318f0 0x7f3b94131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41464/0 (socket says 172.21.15.116:41464) 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b9259c640 1 -- 172.21.15.116:0/2224911429 learned_addr learned my addr 172.21.15.116:0/2224911429 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b91d9b640 1 --2- 172.21.15.116:0/2224911429 >> v2:172.21.15.116:3300/0 conn(0x7f3b9409ea50 0x7f3b9409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b91d9b640 10 monclient(hunting): get_auth_request con 0x7f3b9409ea50 auth_method 0 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.352+0000 7f3b91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_done global_id 54150 payload 274 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 1 -- 172.21.15.116:0/2224911429 >> v1:172.21.15.144:6789/0 conn(0x7f3b94136c20 legacy=0x7f3b94139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 1 -- 172.21.15.116:0/2224911429 >> v1:172.21.15.116:6789/0 conn(0x7f3b941318f0 legacy=0x7f3b94131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 1 monclient: found mon.noname-a 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.353+0000 7f3b91d9b640 1 -- 172.21.15.116:0/2224911429 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940a1210 con 0x7f3b9409ea50 2024-09-18T12:28:21.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.355+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v60) v1 ==== 110+0+0 (secure 0 0 0) 0x7fbc1c021030 con 0x7fbc2c09ee60 2024-09-18T12:28:21.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.355+0000 7fbc13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.355+0000 7fbc13fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b91d9b640 1 -- 172.21.15.116:0/2224911429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9409ea50 msgr2=0x7f3b9409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b91d9b640 1 --2- 172.21.15.116:0/2224911429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9409ea50 0x7f3b9409ee20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009850 tx=0x7f3b880090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b920439505f1377 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 1 -- 172.21.15.116:0/2224911429 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88032440 con 0x7f3b9409ea50 2024-09-18T12:28:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: dump: 2024-09-18T12:28:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 5 monclient: authenticate success, global_id 54150 2024-09-18T12:28:21.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 1 -- 172.21.15.116:0/2224911429 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b880325a0 con 0x7f3b9409ea50 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 1 -- 172.21.15.116:0/2224911429 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88032940 con 0x7f3b9409ea50 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 4 set_mon_vals no callback set 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b9159a640 10 monclient: dump: 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.369 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 10 monclient: shutdown 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 1 -- 172.21.15.116:0/2224911429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9409ea50 msgr2=0x7f3b9409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.356+0000 7f3b98e4f640 1 --2- 172.21.15.116:0/2224911429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9409ea50 0x7f3b9409ee20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009850 tx=0x7f3b880090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 1 -- 172.21.15.116:0/2224911429 shutdown_connections 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 1 --2- 172.21.15.116:0/2224911429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9409ea50 0x7f3b9409ee20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 1 -- 172.21.15.116:0/2224911429 >> 172.21.15.116:0/2224911429 conn(0x7f3b94067110 msgr2=0x7f3b94067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 1 -- 172.21.15.116:0/2224911429 shutdown_connections 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 1 -- 172.21.15.116:0/2224911429 wait complete. 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 10 monclient: monmap: 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:21.371 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 1 Processor -- start 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 1 -- start start 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.357+0000 7f3b98e4f640 10 monclient: init 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b9409ea50 0x7f3b941327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): picked mon.noname-c con 0x7f3b9409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b941318f0 0x7f3b94132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): picked mon.noname-a con 0x7f3b941318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b94136c20 0x7f3b94133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): picked mon.noname-b con 0x7f3b94136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9413a460 con 0x7f3b941318f0 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941360c0 con 0x7f3b94136c20 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b94135de0 con 0x7f3b9409ea50 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b98e4f640 10 monclient(hunting): authenticate will time out at 928.680426s 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b9409ea50 0x7f3b941327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b941318f0 0x7f3b94132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b9409ea50 0x7f3b941327d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34206/0 (socket says 172.21.15.116:34206) 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b941318f0 0x7f3b94132d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50780/0 (socket says 172.21.15.116:50780) 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 -- 172.21.15.116:0/1259461123 learned_addr learned my addr 172.21.15.116:0/1259461123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): get_auth_request con 0x7f3b9409ea50 auth_method 0 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 10 monclient(hunting): get_auth_request con 0x7f3b941318f0 auth_method 0 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b94136c20 0x7f3b94133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 10 monclient(hunting): get_auth_request con 0x7f3b94136c20 auth_method 0 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient(hunting): handle_auth_done global_id 44312 payload 824 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b94136c20 msgr2=0x7f3b94133250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b94136c20 0x7f3b94133250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b941318f0 msgr2=0x7f3b94132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b941318f0 0x7f3b94132d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 monclient: found mon.noname-c 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b9259c640 1 -- 172.21.15.116:0/1259461123 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b88009390 con 0x7f3b9409ea50 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.358+0000 7f3b91d9b640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b941318f0 0x7f3b94132d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b9259c640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b9409ea50 msgr2=0x7f3b941327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b9259c640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b9409ea50 0x7f3b941327d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400c610 tx=0x7f3b84004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84005cb0 con 0x7f3b9409ea50 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: dump: 2024-09-18T12:28:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b98e4f640 5 monclient: authenticate success, global_id 44312 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b84005e30 con 0x7f3b9409ea50 2024-09-18T12:28:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84007340 con 0x7f3b9409ea50 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b98e4f640 10 monclient: _renew_subs 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b98e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b94134320 con 0x7f3b9409ea50 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b90d99640 4 set_mon_vals no callback set 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b827fc640 10 monclient: dump: 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.378 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b98e4f640 10 monclient: _renew_subs 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b98e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.361+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b941366f0 con 0x7f3b9409ea50 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b98e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3b8401b400 con 0x7f3b9409ea50 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b98e4f640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 0x7f3b94136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b98e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b94065e00 con 0x7f3b94133790 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 0x7f3b94136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient: get_auth_request con 0x7f3b94133790 auth_method 0 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b827fc640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b74038490 0x7f3b7403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f3b8404b520 con 0x7f3b9409ea50 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_done global_id 54156 payload 824 2024-09-18T12:28:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b91d9b640 1 -- 172.21.15.116:0/1259461123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b9404bf60 con 0x7f3b94133790 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b92d9d640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b74038490 0x7f3b7403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.362+0000 7f3b92d9d640 10 monclient: get_auth_request con 0x7f3b74038490 auth_method 0 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.364+0000 7f3b91d9b640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 msgr2=0x7f3b94136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.364+0000 7f3b91d9b640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 0x7f3b94136ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3b88034000 tx=0x7f3b88032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.364+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8803bed0 con 0x7f3b94133790 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.364+0000 7f3b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.364+0000 7f3b827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.364+0000 7f3b827fc640 10 monclient: dump: 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.382 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.364+0000 7f3b92d9d640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b74038490 0x7f3b7403a940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c007fd0 tx=0x7f3b8c0025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.365+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3b88040070 con 0x7f3b94133790 2024-09-18T12:28:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.365+0000 7f3b827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.365+0000 7f3b827fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.365+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 msgr2=0x7f3b94136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.365+0000 7f3b827fc640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 0x7f3b94136ff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3b88034000 tx=0x7f3b88032000 comp rx=0 tx=0).stop 2024-09-18T12:28:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.365+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 msgr2=0x7f3b94136ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.365+0000 7f3b827fc640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94133790 0x7f3b94136ff0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b98e4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b98e4f640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 0x7f3b94137930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b98e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940638d0 con 0x7f3b94137530 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b91d9b640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 0x7f3b94137930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b91d9b640 10 monclient: get_auth_request con 0x7f3b94137530 auth_method 0 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.370+0000 7f3b91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.371+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.371+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.371+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.371+0000 7f3b91d9b640 10 monclient(hunting): handle_auth_done global_id 54159 payload 824 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.371+0000 7f3b91d9b640 1 -- 172.21.15.116:0/1259461123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b94133b60 con 0x7f3b94137530 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b91d9b640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 msgr2=0x7f3b94137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b91d9b640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 0x7f3b94137930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009850 tx=0x7f3b8803e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88057070 con 0x7f3b94137530 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 10 monclient: dump: 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f3b88000bc0 con 0x7f3b94137530 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 msgr2=0x7f3b94137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 0x7f3b94137930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009850 tx=0x7f3b8803e070 comp rx=0 tx=0).stop 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 msgr2=0x7f3b94137930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.373+0000 7f3b827fc640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b94137530 0x7f3b94137930 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009850 tx=0x7f3b8803e070 comp rx=0 tx=0).stop 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 34, 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:28:21.389 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.390 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.391 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.393 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.395 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:21.396 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:21.398 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.399 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.400 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:28:21.401 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.401 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.401 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:21.401 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b74038490 msgr2=0x7f3b7403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b74038490 0x7f3b7403a940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c007fd0 tx=0x7f3b8c0025b0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 10 monclient: shutdown 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b9409ea50 msgr2=0x7f3b941327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 --2- 172.21.15.116:0/1259461123 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b9409ea50 0x7f3b941327d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400c610 tx=0x7f3b84004960 comp rx=0 tx=0).stop 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b92d9d640 1 -- 172.21.15.116:0/1259461123 reap_dead start 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b92d9d640 1 -- 172.21.15.116:0/1259461123 reap_dead start 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 shutdown_connections 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 >> 172.21.15.116:0/1259461123 conn(0x7f3b94067110 msgr2=0x7f3b9412bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 shutdown_connections 2024-09-18T12:28:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.374+0000 7f3b98e4f640 1 -- 172.21.15.116:0/1259461123 wait complete. 2024-09-18T12:28:21.403 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:28:21.404 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:28:21.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:28:21.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.451+0000 7fbc32354640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.451+0000 7fbc32354640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.451+0000 7fbc32354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.451+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7fbc2c120f90 con 0x7fbc2c09ee60 2024-09-18T12:28:21.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.452+0000 7fbc13fff640 1 -- 172.21.15.144:0/3798622899 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v60) v1 ==== 139+0+0 (secure 0 0 0) 0x7fbc1c021020 con 0x7fbc2c09ee60 2024-09-18T12:28:21.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.452+0000 7fbc13fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.452+0000 7fbc13fff640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-18T12:28:21.454 INFO:tasks.workunit.client.0.smithi144.stderr:already renamed to 'foo-asdf' 2024-09-18T12:28:21.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc18038490 msgr2=0x7fbc1803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc18038490 0x7fbc1803a940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c09ca60 tx=0x7fbc20007e10 comp rx=0 tx=0).stop 2024-09-18T12:28:21.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 10 monclient: shutdown 2024-09-18T12:28:21.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 msgr2=0x7fbc2c13d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 0x7fbc2c13d710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00e990 tx=0x7fbc1c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 shutdown_connections 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc18038490 0x7fbc1803a940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc2c0a0ad0 0x7fbc2c140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc2c09f260 0x7fbc2c13cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 --2- 172.21.15.144:0/3798622899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc2c09ee60 0x7fbc2c13d710 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.453+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 >> 172.21.15.144:0/3798622899 conn(0x7fbc2c071370 msgr2=0x7fbc2c0a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.454+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 shutdown_connections 2024-09-18T12:28:21.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.454+0000 7fbc32354640 1 -- 172.21.15.144:0/3798622899 wait complete. 2024-09-18T12:28:21.464 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-18T12:28:21.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.495+0000 7fbe3095d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:21.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.495+0000 7fbe3095d640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.495+0000 7fbe3095d640 10 monclient: monmap: 2024-09-18T12:28:21.497 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:21.497365+0000 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:21.497365+0000 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:21.498 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 1 Processor -- start 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 1 -- start start 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient: init 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe2c09c860 0x7fbe2c09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2c09c860 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:21.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe2c09aba0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 0x7fbe2c09df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2c09afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c06a320 con 0x7fbe2c09aba0 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2c09a9d0 con 0x7fbe2c09c860 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c068a20 con 0x7fbe2c09afa0 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.496+0000 7fbe3095d640 10 monclient(hunting): authenticate will time out at 928.818938s 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 0x7fbe2c09df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe2c09c860 0x7fbe2c09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 0x7fbe2c09df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34222/0 (socket says 172.21.15.116:34222) 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe2c09c860 0x7fbe2c09cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50816/0 (socket says 172.21.15.116:50816) 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/3452409198 learned_addr learned my addr 172.21.15.116:0/3452409198 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): get_auth_request con 0x7fbe2c09afa0 auth_method 0 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 10 monclient(hunting): get_auth_request con 0x7fbe2c09c860 auth_method 0 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_done global_id 44315 payload 274 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/3452409198 >> v2:172.21.15.116:3300/0 conn(0x7fbe2c09c860 msgr2=0x7fbe2c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 --2- 172.21.15.116:0/3452409198 >> v2:172.21.15.116:3300/0 conn(0x7fbe2c09c860 0x7fbe2c09cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/3452409198 >> v1:172.21.15.144:6789/0 conn(0x7fbe2c09aba0 legacy=0x7fbe2c09d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 monclient: found mon.noname-b 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/3452409198 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c068e60 con 0x7fbe2c09afa0 2024-09-18T12:28:21.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.497+0000 7fbe2a59c640 1 --2- 172.21.15.116:0/3452409198 >> v2:172.21.15.116:3300/0 conn(0x7fbe2c09c860 0x7fbe2c09cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:21.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/3452409198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 msgr2=0x7fbe2c09df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2ad9d640 1 --2- 172.21.15.116:0/3452409198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 0x7fbe2c09df40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009f20 tx=0x7fbe24009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a70e7846b1e23d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 1 -- 172.21.15.116:0/3452409198 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24007610 con 0x7fbe2c09afa0 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: dump: 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.512 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.513 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.513 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.513 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.513 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.513 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.514 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 1 -- 172.21.15.116:0/3452409198 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2402d050 con 0x7fbe2c09afa0 2024-09-18T12:28:21.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 5 monclient: authenticate success, global_id 44315 2024-09-18T12:28:21.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:21.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:21.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 1 -- 172.21.15.116:0/3452409198 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe240079a0 con 0x7fbe2c09afa0 2024-09-18T12:28:21.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 4 set_mon_vals no callback set 2024-09-18T12:28:21.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe2959a640 10 monclient: dump: 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 10 monclient: shutdown 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 -- 172.21.15.116:0/3452409198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 msgr2=0x7fbe2c09df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 --2- 172.21.15.116:0/3452409198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 0x7fbe2c09df40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009f20 tx=0x7fbe24009840 comp rx=0 tx=0).stop 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 -- 172.21.15.116:0/3452409198 shutdown_connections 2024-09-18T12:28:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 --2- 172.21.15.116:0/3452409198 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09afa0 0x7fbe2c09df40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 --2- 172.21.15.116:0/3452409198 >> v2:172.21.15.116:3300/0 conn(0x7fbe2c09c860 0x7fbe2c09cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 -- 172.21.15.116:0/3452409198 >> 172.21.15.116:0/3452409198 conn(0x7fbe2c0a2e40 msgr2=0x7fbe2c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 -- 172.21.15.116:0/3452409198 shutdown_connections 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.500+0000 7fbe3095d640 1 -- 172.21.15.116:0/3452409198 wait complete. 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.501+0000 7fbe3095d640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.501+0000 7fbe3095d640 10 monclient: monmap: 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.501+0000 7fbe3095d640 1 Processor -- start 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.501+0000 7fbe3095d640 1 -- start start 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.501+0000 7fbe3095d640 10 monclient: init 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09aba0 0x7fbe2c140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2c09aba0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe2c09afa0 0x7fbe2c131c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2c09afa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe2c09c860 0x7fbe2c06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2c09c860 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c0a12e0 con 0x7fbe2c09c860 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c141cc0 con 0x7fbe2c09afa0 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c11f020 con 0x7fbe2c09aba0 2024-09-18T12:28:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe3095d640 10 monclient(hunting): authenticate will time out at 928.824206s 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09aba0 0x7fbe2c140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe2c09c860 0x7fbe2c06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe2c09c860 0x7fbe2c06d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50828/0 (socket says 172.21.15.116:50828) 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09aba0 0x7fbe2c140de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34226/0 (socket says 172.21.15.116:34226) 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/819189796 learned_addr learned my addr 172.21.15.116:0/819189796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): get_auth_request con 0x7fbe2c09c860 auth_method 0 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 10 monclient(hunting): get_auth_request con 0x7fbe2c09aba0 auth_method 0 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe2c09afa0 0x7fbe2c131c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 10 monclient(hunting): get_auth_request con 0x7fbe2c09afa0 auth_method 0 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient(hunting): handle_auth_done global_id 54165 payload 824 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09aba0 msgr2=0x7fbe2c140de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09aba0 0x7fbe2c140de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe2c09afa0 msgr2=0x7fbe2c131c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe2c09afa0 0x7fbe2c131c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 monclient: found mon.noname-a 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/819189796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c009390 con 0x7fbe2c09c860 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.502+0000 7fbe2a59c640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c09aba0 0x7fbe2c140de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.504+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe2c09c860 msgr2=0x7fbe2c06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.504+0000 7fbe2ad9d640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe2c09c860 0x7fbe2c06d7e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009f20 tx=0x7fbe24036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.504+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24007170 con 0x7fbe2c09c860 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.504+0000 7fbe1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.504+0000 7fbe1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 10 monclient: dump: 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.525 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 5 monclient: authenticate success, global_id 54165 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe24032cc0 con 0x7fbe2c09c860 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 10 monclient: _renew_subs 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe24009ae0 con 0x7fbe2c09c860 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2403b680 con 0x7fbe2c09c860 2024-09-18T12:28:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 10 monclient: dump: 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 10 monclient: _renew_subs 2024-09-18T12:28:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe28d99640 4 set_mon_vals no callback set 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2c11f300 con 0x7fbe2c09c860 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 0x7fbe2c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c065e00 con 0x7fbe2c132160 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbe24038050 con 0x7fbe2c09c860 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.505+0000 7fbe2a59c640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 0x7fbe2c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient: get_auth_request con 0x7fbe2c132160 auth_method 0 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_done global_id 44321 payload 824 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe1a7fc640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe0c038490 0x7fbe0c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe2a59c640 1 -- 172.21.15.116:0/819189796 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe2c099c80 con 0x7fbe2c132160 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fbe2406c4b0 con 0x7fbe2c09c860 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe29d9b640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe0c038490 0x7fbe0c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.506+0000 7fbe29d9b640 10 monclient: get_auth_request con 0x7fbe0c038490 auth_method 0 2024-09-18T12:28:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.508+0000 7fbe2a59c640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 msgr2=0x7fbe2c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.508+0000 7fbe2a59c640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 0x7fbe2c06dd20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c011f20 tx=0x7fbe1c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.508+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c014940 con 0x7fbe2c132160 2024-09-18T12:28:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.508+0000 7fbe1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.508+0000 7fbe1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.508+0000 7fbe1a7fc640 10 monclient: dump: 2024-09-18T12:28:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.508+0000 7fbe29d9b640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe0c038490 0x7fbe0c03a940 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe20007fd0 tx=0x7fbe20007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.509+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbe1c014aa0 con 0x7fbe2c132160 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.509+0000 7fbe1a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.509+0000 7fbe1a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.509+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 msgr2=0x7fbe2c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.509+0000 7fbe1a7fc640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 0x7fbe2c06dd20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c011f20 tx=0x7fbe1c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.509+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 msgr2=0x7fbe2c06dd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:21.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.509+0000 7fbe1a7fc640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c132160 0x7fbe2c06dd20 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c011f20 tx=0x7fbe1c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe3095d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe3095d640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 0x7fbe2c132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe3095d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c06e660 con 0x7fbe2c06e260 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe2a59c640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 0x7fbe2c132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe2a59c640 10 monclient: get_auth_request con 0x7fbe2c06e260 auth_method 0 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.514+0000 7fbe2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.515+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.515+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.515+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.515+0000 7fbe2a59c640 10 monclient(hunting): handle_auth_done global_id 44324 payload 824 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.515+0000 7fbe2a59c640 1 -- 172.21.15.116:0/819189796 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbe2c04bf60 con 0x7fbe2c06e260 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe2a59c640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 msgr2=0x7fbe2c132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe2a59c640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 0x7fbe2c132a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c01b000 tx=0x7fbe1c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c01c8e0 con 0x7fbe2c06e260 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 10 monclient: dump: 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.535 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fbe1c01ca40 con 0x7fbe2c06e260 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 msgr2=0x7fbe2c132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 0x7fbe2c132a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c01b000 tx=0x7fbe1c01c000 comp rx=0 tx=0).stop 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 msgr2=0x7fbe2c132a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:21.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.517+0000 7fbe1a7fc640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe2c06e260 0x7fbe2c132a20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 34, 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.537 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.538 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.539 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:21.540 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:21.543 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:21.544 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.545 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.546 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.547 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.548 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.549 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.550 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.551 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:21.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.552 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.552 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:21.552 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:21.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe0c038490 msgr2=0x7fbe0c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe0c038490 0x7fbe0c03a940 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe20007fd0 tx=0x7fbe20007e10 comp rx=0 tx=0).stop 2024-09-18T12:28:21.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 10 monclient: shutdown 2024-09-18T12:28:21.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe2c09c860 msgr2=0x7fbe2c06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 --2- 172.21.15.116:0/819189796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe2c09c860 0x7fbe2c06d7e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009f20 tx=0x7fbe24036000 comp rx=0 tx=0).stop 2024-09-18T12:28:21.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/819189796 reap_dead start 2024-09-18T12:28:21.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe2ad9d640 1 -- 172.21.15.116:0/819189796 reap_dead start 2024-09-18T12:28:21.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 shutdown_connections 2024-09-18T12:28:21.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 >> 172.21.15.116:0/819189796 conn(0x7fbe2c0a2e40 msgr2=0x7fbe2c0a14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 shutdown_connections 2024-09-18T12:28:21.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.518+0000 7fbe3095d640 1 -- 172.21.15.116:0/819189796 wait complete. 2024-09-18T12:28:21.555 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:28:21.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:28:21.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:28:21.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.539+0000 7f1e1124f640 10 monclient: get_monmap_and_config 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.539+0000 7f1e1124f640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.539+0000 7f1e1124f640 10 monclient: monmap: 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:21.541186+0000 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:21.541186+0000 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:21.567 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.539+0000 7f1e1124f640 1 Processor -- start 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.539+0000 7f1e1124f640 1 -- start start 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.539+0000 7f1e1124f640 10 monclient: init 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c09f320 0x7f1e0c1321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient(hunting): picked mon.noname-b con 0x7f1e0c09f320 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1e0c070920 0x7f1e0c070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient(hunting): picked mon.noname-c con 0x7f1e0c070920 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e0c071230 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c096b60 con 0x7f1e0c071230 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0c09e4d0 con 0x7f1e0c070920 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c09e7b0 con 0x7f1e0c09f320 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e1124f640 10 monclient(hunting): authenticate will time out at 937.110412s 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1e0c070920 0x7f1e0c070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0b59e640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1e0c071230 0x7f1e0c12e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48004/0 (socket says 172.21.15.144:48004) 2024-09-18T12:28:21.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): get_auth_request con 0x7f1e0c070920 auth_method 0 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0b59e640 1 -- 172.21.15.144:0/3254297490 learned_addr learned my addr 172.21.15.144:0/3254297490 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 1 --2- 172.21.15.144:0/3254297490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c09f320 0x7f1e0c1321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 10 monclient(hunting): get_auth_request con 0x7f1e0c09f320 auth_method 0 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient(hunting): handle_auth_done global_id 4882 payload 274 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 1 -- 172.21.15.144:0/3254297490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c09f320 msgr2=0x7f1e0c1321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 1 --2- 172.21.15.144:0/3254297490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c09f320 0x7f1e0c1321b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 1 -- 172.21.15.144:0/3254297490 >> v1:172.21.15.144:6789/0 conn(0x7f1e0c071230 legacy=0x7f1e0c12e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 1 monclient: found mon.noname-c 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.540+0000 7f1e0259c640 1 -- 172.21.15.144:0/3254297490 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c09e1f0 con 0x7f1e0c070920 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.543+0000 7f1e0259c640 1 -- 172.21.15.144:0/3254297490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c070920 msgr2=0x7f1e0c070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.543+0000 7f1e0259c640 1 --2- 172.21.15.144:0/3254297490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c070920 0x7f1e0c070cf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009850 tx=0x7f1df40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34a9b67d11717073 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 1 -- 172.21.15.144:0/3254297490 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df402add0 con 0x7f1e0c070920 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: dump: 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.572 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 1 -- 172.21.15.144:0/3254297490 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df4032440 con 0x7f1e0c070920 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 5 monclient: authenticate success, global_id 4882 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 1 -- 172.21.15.144:0/3254297490 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df4032820 con 0x7f1e0c070920 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 4 set_mon_vals no callback set 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e0a59c640 10 monclient: dump: 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.574 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 10 monclient: shutdown 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 -- 172.21.15.144:0/3254297490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c070920 msgr2=0x7f1e0c070cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3254297490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c070920 0x7f1e0c070cf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009850 tx=0x7f1df40090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 -- 172.21.15.144:0/3254297490 shutdown_connections 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3254297490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c070920 0x7f1e0c070cf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3254297490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c09f320 0x7f1e0c1321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 -- 172.21.15.144:0/3254297490 >> 172.21.15.144:0/3254297490 conn(0x7f1e0c0a2b70 msgr2=0x7f1e0c09ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 -- 172.21.15.144:0/3254297490 shutdown_connections 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.544+0000 7f1e1124f640 1 -- 172.21.15.144:0/3254297490 wait complete. 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.545+0000 7f1e1124f640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.545+0000 7f1e1124f640 10 monclient: monmap: 2024-09-18T12:28:21.576 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.545+0000 7f1e1124f640 1 Processor -- start 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.545+0000 7f1e1124f640 1 -- start start 2024-09-18T12:28:21.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.545+0000 7f1e1124f640 10 monclient: init 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e0c070920 0x7f1e0c140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): picked mon.noname-a con 0x7f1e0c070920 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c071230 0x7f1e0c12fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): picked mon.noname-c con 0x7f1e0c071230 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 0x7f1e0c119970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): picked mon.noname-b con 0x7f1e0c09f320 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c11d0e0 con 0x7f1e0c070920 2024-09-18T12:28:21.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c11b480 con 0x7f1e0c09f320 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c130d40 con 0x7f1e0c071230 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e1124f640 10 monclient(hunting): authenticate will time out at 937.116168s 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 0x7f1e0c119970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 0x7f1e0c119970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46730/0 (socket says 172.21.15.144:46730) 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 -- 172.21.15.144:0/3726497736 learned_addr learned my addr 172.21.15.144:0/3726497736 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): get_auth_request con 0x7f1e0c09f320 auth_method 0 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0259c640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c071230 0x7f1e0c12fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0ad9d640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e0c070920 0x7f1e0c140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0259c640 10 monclient(hunting): get_auth_request con 0x7f1e0c071230 auth_method 0 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0ad9d640 10 monclient(hunting): get_auth_request con 0x7f1e0c070920 auth_method 0 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient(hunting): handle_auth_done global_id 4885 payload 824 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 -- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c071230 msgr2=0x7f1e0c12fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c071230 0x7f1e0c12fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 -- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e0c070920 msgr2=0x7f1e0c140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e0c070920 0x7f1e0c140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 monclient: found mon.noname-b 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.546+0000 7f1e0b59e640 1 -- 172.21.15.144:0/3726497736 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df4009390 con 0x7f1e0c09f320 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e0b59e640 1 -- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 msgr2=0x7f1e0c119970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e0b59e640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 0x7f1e0c119970 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400ccf0 tx=0x7f1e04005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0400aa60 con 0x7f1e0c09f320 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: dump: 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e1124f640 5 monclient: authenticate success, global_id 4885 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0400abe0 con 0x7f1e0c09f320 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e040070f0 con 0x7f1e0c09f320 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e1124f640 10 monclient: _renew_subs 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e1124f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c09d5b0 con 0x7f1e0c09f320 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e09d9b640 4 set_mon_vals no callback set 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e037fe640 10 monclient: dump: 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.583 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e1124f640 10 monclient: _renew_subs 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e1124f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.548+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c11bf10 con 0x7f1e0c09f320 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.549+0000 7f1e1124f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.549+0000 7f1e1124f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.549+0000 7f1e1124f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.549+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1e040074e0 con 0x7f1e0c09f320 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.549+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0c11c190 con 0x7f1e0c09f320 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.550+0000 7f1e037fe640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dec038490 0x7f1dec03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.550+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f1e0404bbd0 con 0x7f1e0c09f320 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.550+0000 7f1e0ad9d640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dec038490 0x7f1dec03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.550+0000 7f1e0ad9d640 10 monclient: get_auth_request con 0x7f1dec038490 auth_method 0 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.552+0000 7f1e0ad9d640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dec038490 0x7f1dec03a940 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009cb0 tx=0x7f1dfc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.554+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1e04007790 con 0x7f1e0c09f320 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.554+0000 7f1e037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.554+0000 7f1e037fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:21.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.653+0000 7fb3b5e78640 10 monclient: get_monmap_and_config 2024-09-18T12:28:21.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.653+0000 7fb3b5e78640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.653+0000 7fb3b5e78640 10 monclient: monmap: 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:21.655501+0000 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:21.655501+0000 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:21.656 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.654+0000 7fb3b5e78640 1 Processor -- start 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.654+0000 7fb3b5e78640 1 -- start start 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.654+0000 7fb3b5e78640 10 monclient: init 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3b009c860 0x7fb3b009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b009c860 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:21.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b009d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b009aba0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3b009afa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b006a320 con 0x7fb3b009afa0 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b009a9d0 con 0x7fb3b009c860 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0068a20 con 0x7fb3b009aba0 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3b5e78640 10 monclient(hunting): authenticate will time out at 928.977198s 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3b009c860 0x7fb3b009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b009d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3affff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb3b009afa0 0x7fb3b009dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41480/0 (socket says 172.21.15.116:41480) 2024-09-18T12:28:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3affff640 1 -- 172.21.15.116:0/439113214 learned_addr learned my addr 172.21.15.116:0/439113214 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): get_auth_request con 0x7fb3b009c860 auth_method 0 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request con 0x7fb3b009aba0 auth_method 0 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_done global_id 54171 payload 274 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 1 -- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 msgr2=0x7fb3b009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 1 --2- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b009d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 1 -- 172.21.15.116:0/439113214 >> v1:172.21.15.116:6789/0 conn(0x7fb3b009afa0 legacy=0x7fb3b009dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 1 monclient: found mon.noname-a 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3af7fe640 1 -- 172.21.15.116:0/439113214 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0068e60 con 0x7fb3b009c860 2024-09-18T12:28:21.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.655+0000 7fb3aeffd640 1 --2- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3af7fe640 1 -- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 msgr2=0x7fb3b009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3af7fe640 1 --2- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 0x7fb3b009cc30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009850 tx=0x7fb3a00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc424c34d857ce8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 1 -- 172.21.15.116:0/439113214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a002ae60 con 0x7fb3b009c860 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: dump: 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 1 -- 172.21.15.116:0/439113214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a0032440 con 0x7fb3b009c860 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3b5e78640 5 monclient: authenticate success, global_id 54171 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3b5e78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3b5e78640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 1 -- 172.21.15.116:0/439113214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00327e0 con 0x7fb3b009c860 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3b5e78640 4 set_mon_vals no callback set 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:21.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.658+0000 7fb3ae7fc640 10 monclient: dump: 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 10 monclient: shutdown 2024-09-18T12:28:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 -- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 msgr2=0x7fb3b009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 --2- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 0x7fb3b009cc30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009850 tx=0x7fb3a00090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 -- 172.21.15.116:0/439113214 shutdown_connections 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 --2- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 --2- 172.21.15.116:0/439113214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 0x7fb3b009cc30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 -- 172.21.15.116:0/439113214 >> 172.21.15.116:0/439113214 conn(0x7fb3b00a2e40 msgr2=0x7fb3b00a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 -- 172.21.15.116:0/439113214 shutdown_connections 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 1 -- 172.21.15.116:0/439113214 wait complete. 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.659+0000 7fb3b5e78640 10 monclient: monmap: 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 Processor -- start 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 -- start start 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient: init 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b0141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b009aba0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b009afa0 0x7fb3b0133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b009afa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 0x7fb3b011d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b009c860 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0099cc0 con 0x7fb3b009c860 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0134b20 con 0x7fb3b009afa0 2024-09-18T12:28:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0141ee0 con 0x7fb3b009aba0 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3b5e78640 10 monclient(hunting): authenticate will time out at 928.982862s 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3af7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b0141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3af7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b0141070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34256/0 (socket says 172.21.15.116:34256) 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 0x7fb3b011d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3af7fe640 1 -- 172.21.15.116:0/1549833351 learned_addr learned my addr 172.21.15.116:0/1549833351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3af7fe640 10 monclient(hunting): get_auth_request con 0x7fb3b009aba0 auth_method 0 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.660+0000 7fb3affff640 10 monclient(hunting): get_auth_request con 0x7fb3b009c860 auth_method 0 2024-09-18T12:28:21.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3affff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3aeffd640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b009afa0 0x7fb3b0133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request con 0x7fb3b009afa0 auth_method 0 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 10 monclient(hunting): handle_auth_done global_id 44336 payload 824 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b009afa0 msgr2=0x7fb3b0133bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b009afa0 0x7fb3b0133bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 msgr2=0x7fb3b011d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 0x7fb3b011d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 1 monclient: found mon.noname-c 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3af7fe640 1 -- 172.21.15.116:0/1549833351 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a4009390 con 0x7fb3b009aba0 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3aeffd640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b009afa0 0x7fb3b0133bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.661+0000 7fb3affff640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b009c860 0x7fb3b011d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:28:21.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3af7fe640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 msgr2=0x7fb3b0141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3af7fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b0141070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009360 tx=0x7fb3a0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0031d40 con 0x7fb3b009aba0 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: dump: 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.682 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3b5e78640 5 monclient: authenticate success, global_id 44336 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a003d440 con 0x7fb3b009aba0 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3b5e78640 10 monclient: _renew_subs 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3b5e78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a0009390 con 0x7fb3b009aba0 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0031ea0 con 0x7fb3b009aba0 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3977fe640 10 monclient: dump: 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3b5e78640 10 monclient: _renew_subs 2024-09-18T12:28:21.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3b5e78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b011f200 con 0x7fb3b009aba0 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.663+0000 7fb3adffb640 4 set_mon_vals no callback set 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3b5e78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3b5e78640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 0x7fb3b011ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3b5e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0065e00 con 0x7fb3b0134130 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb3a0039040 con 0x7fb3b009aba0 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 0x7fb3b011ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient: get_auth_request con 0x7fb3b0134130 auth_method 0 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3977fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb39c038490 0x7fb39c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fb3a006b830 con 0x7fb3b009aba0 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3affff640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb39c038490 0x7fb39c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_done global_id 4891 payload 824 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3aeffd640 1 -- 172.21.15.116:0/1549833351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3b004bf60 con 0x7fb3b0134130 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.664+0000 7fb3affff640 10 monclient: get_auth_request con 0x7fb39c038490 auth_method 0 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3aeffd640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 msgr2=0x7fb3b011ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3aeffd640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 0x7fb3b011ddf0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40118c0 tx=0x7fb3a4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400fd60 con 0x7fb3b0134130 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 10 monclient: dump: 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3affff640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb39c038490 0x7fb39c03a940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800d6a0 tx=0x7fb3a8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb3a400fec0 con 0x7fb3b0134130 2024-09-18T12:28:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 msgr2=0x7fb3b011ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 0x7fb3b011ddf0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40118c0 tx=0x7fb3a4008040 comp rx=0 tx=0).stop 2024-09-18T12:28:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 msgr2=0x7fb3b011ddf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.667+0000 7fb3977fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b0134130 0x7fb3b011ddf0 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40118c0 tx=0x7fb3a4008040 comp rx=0 tx=0).stop 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3b5e78640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3b5e78640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 0x7fb3b006b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3b5e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0134500 con 0x7fb3b011e330 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 0x7fb3b006b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient: get_auth_request con 0x7fb3b011e330 auth_method 0 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_done global_id 4894 payload 824 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.673+0000 7fb3aeffd640 1 -- 172.21.15.116:0/1549833351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb3b011e730 con 0x7fb3b011e330 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3aeffd640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 msgr2=0x7fb3b006b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3aeffd640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 0x7fb3b006b8b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40118c0 tx=0x7fb3a401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4006250 con 0x7fb3b011e330 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 10 monclient: dump: 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fb3a40063b0 con 0x7fb3b011e330 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:21.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 msgr2=0x7fb3b006b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 0x7fb3b006b8b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40118c0 tx=0x7fb3a401a000 comp rx=0 tx=0).stop 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 msgr2=0x7fb3b006b8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3977fe640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b011e330 0x7fb3b006b8b0 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40118c0 tx=0x7fb3a401a000 comp rx=0 tx=0).stop 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 34, 2024-09-18T12:28:21.693 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.694 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.695 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:21.696 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:21.697 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:21.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.701 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.702 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.704 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.705 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:21.706 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb39c038490 msgr2=0x7fb39c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb39c038490 0x7fb39c03a940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800d6a0 tx=0x7fb3a8009040 comp rx=0 tx=0).stop 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 10 monclient: shutdown 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 msgr2=0x7fb3b0141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 1 --2- 172.21.15.116:0/1549833351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b009aba0 0x7fb3b0141070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009360 tx=0x7fb3a0032000 comp rx=0 tx=0).stop 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3affff640 1 -- 172.21.15.116:0/1549833351 reap_dead start 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3affff640 1 -- 172.21.15.116:0/1549833351 reap_dead start 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 shutdown_connections 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.676+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 >> 172.21.15.116:0/1549833351 conn(0x7fb3b00a2e40 msgr2=0x7fb3b0134d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.677+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 shutdown_connections 2024-09-18T12:28:21.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.677+0000 7fb3b5e78640 1 -- 172.21.15.116:0/1549833351 wait complete. 2024-09-18T12:28:21.709 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:28:21.709 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:28:21.709 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:28:21.709 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:28:21.709 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:28:21.709 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:28:21.710 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:28:21.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.713+0000 7f1e1124f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.713+0000 7f1e1124f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.713+0000 7f1e1124f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.713+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f1e0c1416d0 con 0x7f1e0c09f320 2024-09-18T12:28:21.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.714+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v60) v1 ==== 139+0+0 (secure 0 0 0) 0x7f1e04010af0 con 0x7f1e0c09f320 2024-09-18T12:28:21.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.715+0000 7f1e037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.715+0000 7f1e037fe640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-18T12:28:21.720 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:28:21.720 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:28:21.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:28:21.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.810+0000 7f1e1124f640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.810+0000 7f1e1124f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.810+0000 7f1e1124f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.810+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f1e0c0008d0 con 0x7f1e0c09f320 2024-09-18T12:28:21.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.811+0000 7f1e037fe640 1 -- 172.21.15.144:0/3726497736 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v60) v1 ==== 139+0+0 (secure 0 0 0) 0x7f1e0401a360 con 0x7f1e0c09f320 2024-09-18T12:28:21.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.811+0000 7f1e037fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T12:28:21.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.811+0000 7f1e037fe640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-18T12:28:21.813 INFO:tasks.workunit.client.0.smithi144.stderr:already renamed to 'foo-asdf' 2024-09-18T12:28:21.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dec038490 msgr2=0x7f1dec03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dec038490 0x7f1dec03a940 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009cb0 tx=0x7f1dfc008040 comp rx=0 tx=0).stop 2024-09-18T12:28:21.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 10 monclient: shutdown 2024-09-18T12:28:21.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 msgr2=0x7f1e0c119970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 0x7f1e0c119970 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400ccf0 tx=0x7f1e04005370 comp rx=0 tx=0).stop 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 shutdown_connections 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dec038490 0x7f1dec03a940 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1e0c09f320 0x7f1e0c119970 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1e0c071230 0x7f1e0c12fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 --2- 172.21.15.144:0/3726497736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e0c070920 0x7f1e0c140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 >> 172.21.15.144:0/3726497736 conn(0x7f1e0c0a2b70 msgr2=0x7f1e0c134a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 shutdown_connections 2024-09-18T12:28:21.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.813+0000 7f1e1124f640 1 -- 172.21.15.144:0/3726497736 wait complete. 2024-09-18T12:28:21.824 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-18T12:28:21.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.851+0000 7f3f4cd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.851+0000 7f3f4cd9d640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.851+0000 7f3f4cd9d640 10 monclient: monmap: 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:21.853502+0000 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:21.853502+0000 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:21.854 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:21.855 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:21.855 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.852+0000 7f3f4cd9d640 1 Processor -- start 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.852+0000 7f3f4cd9d640 1 -- start start 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.852+0000 7f3f4cd9d640 10 monclient: init 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4806eb80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f48068a20 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3f4806f490 0x7f3f48138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4806f490 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f48069b80 con 0x7f3f48068a20 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f48099f90 con 0x7f3f4806f490 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48069e00 con 0x7f3f4806eb80 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4659c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 msgr2=0x7f3f4806ef50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4cd9d640 10 monclient(hunting): authenticate will time out at 929.175167s 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f4659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3f4806f490 0x7f3f48138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3f4806f490 0x7f3f48138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:51500/0 (socket says 172.21.15.116:51500) 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f45d9b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3f48068a20 0x7f3f4809c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:56522/0 (socket says 172.21.15.116:56522) 2024-09-18T12:28:21.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 -- 172.21.15.116:0/2295201202 learned_addr learned my addr 172.21.15.116:0/2295201202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): get_auth_request con 0x7f3f4806f490 auth_method 0 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient(hunting): handle_auth_done global_id 4897 payload 274 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 -- 172.21.15.116:0/2295201202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 msgr2=0x7f3f4806ef50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 --2- 172.21.15.116:0/2295201202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 -- 172.21.15.116:0/2295201202 >> v1:172.21.15.144:6789/0 conn(0x7f3f48068a20 legacy=0x7f3f4809c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 monclient: found mon.noname-c 2024-09-18T12:28:21.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.853+0000 7f3f46d9d640 1 -- 172.21.15.116:0/2295201202 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4809cc50 con 0x7f3f4806f490 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.856+0000 7f3f46d9d640 1 -- 172.21.15.116:0/2295201202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806f490 msgr2=0x7f3f48138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.856+0000 7f3f46d9d640 1 --2- 172.21.15.116:0/2295201202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806f490 0x7f3f48138ac0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009910 tx=0x7f3f400091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf8391db4faee14c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 1 -- 172.21.15.116:0/2295201202 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40032440 con 0x7f3f4806f490 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: dump: 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 1 -- 172.21.15.116:0/2295201202 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f400325a0 con 0x7f3f4806f490 2024-09-18T12:28:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 5 monclient: authenticate success, global_id 4897 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 4 set_mon_vals no callback set 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 1 -- 172.21.15.116:0/2295201202 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40032940 con 0x7f3f4806f490 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4559a640 10 monclient: dump: 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 10 monclient: shutdown 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/2295201202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806f490 msgr2=0x7f3f48138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/2295201202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806f490 0x7f3f48138ac0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009910 tx=0x7f3f400091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/2295201202 shutdown_connections 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/2295201202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f4806f490 0x7f3f48138ac0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/2295201202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.857+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/2295201202 >> 172.21.15.116:0/2295201202 conn(0x7f3f48067cd0 msgr2=0x7f3f480a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.858+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/2295201202 shutdown_connections 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.858+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/2295201202 wait complete. 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.858+0000 7f3f4cd9d640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.858+0000 7f3f4cd9d640 10 monclient: monmap: 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:21.870 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.858+0000 7f3f4cd9d640 1 Processor -- start 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.858+0000 7f3f4cd9d640 1 -- start start 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.858+0000 7f3f4cd9d640 10 monclient: init 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f48068a20 0x7f3f48138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f48068a20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4811e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4806eb80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f4806f490 0x7f3f4811e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4806f490 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48137f90 con 0x7f3f4806f490 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4809cf40 con 0x7f3f48068a20 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480a20a0 con 0x7f3f4806eb80 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4cd9d640 10 monclient(hunting): authenticate will time out at 929.181265s 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f45d9b640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 msgr2=0x7f3f4811e2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f45d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4811e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f4806f490 0x7f3f4811e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f4806f490 0x7f3f4811e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50876/0 (socket says 172.21.15.116:50876) 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f48068a20 0x7f3f48138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 1 -- 172.21.15.116:0/3657425799 learned_addr learned my addr 172.21.15.116:0/3657425799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:21.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 10 monclient(hunting): get_auth_request con 0x7f3f4806f490 auth_method 0 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): get_auth_request con 0x7f3f48068a20 auth_method 0 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient(hunting): handle_auth_done global_id 4900 payload 824 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 1 -- 172.21.15.116:0/3657425799 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 msgr2=0x7f3f4811e2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4811e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 1 -- 172.21.15.116:0/3657425799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f4806f490 msgr2=0x7f3f4811e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f4806f490 0x7f3f4811e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 1 monclient: found mon.noname-b 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.859+0000 7f3f4659c640 1 -- 172.21.15.116:0/3657425799 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f40009450 con 0x7f3f48068a20 2024-09-18T12:28:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.861+0000 7f3f4659c640 1 -- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f48068a20 msgr2=0x7f3f48138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.861+0000 7f3f4659c640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f48068a20 0x7f3f48138a10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800bf20 tx=0x7f3f38009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 1 -- 172.21.15.116:0/3657425799 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38016430 con 0x7f3f48068a20 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: dump: 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 1 -- 172.21.15.116:0/3657425799 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f380165b0 con 0x7f3f48068a20 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 5 monclient: authenticate success, global_id 4900 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 10 monclient: _renew_subs 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f48121f40 con 0x7f3f48068a20 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 1 -- 172.21.15.116:0/3657425799 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f380167a0 con 0x7f3f48068a20 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f367fc640 10 monclient: dump: 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 10 monclient: _renew_subs 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f37fff640 4 set_mon_vals no callback set 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4811fe50 con 0x7f3f48068a20 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.862+0000 7f3f4cd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f4cd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f4cd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4804bf60 con 0x7f3f48068a20 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f367fc640 1 -- 172.21.15.116:0/3657425799 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3f38012020 con 0x7f3f48068a20 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f367fc640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3f28038440 0x7f3f2803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f367fc640 1 -- 172.21.15.116:0/3657425799 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f3f3804ba60 con 0x7f3f48068a20 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f45d9b640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3f28038440 0x7f3f2803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.863+0000 7f3f45d9b640 10 monclient: get_auth_request con 0x7f3f28038440 auth_method 0 2024-09-18T12:28:21.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.865+0000 7f3f45d9b640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3f28038440 0x7f3f2803a8f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3f4811f8c0 tx=0x7f3f3c007ac0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.868+0000 7f3f367fc640 1 -- 172.21.15.116:0/3657425799 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3f380153e0 con 0x7f3f48068a20 2024-09-18T12:28:21.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.868+0000 7f3f367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:21.868+0000 7f3f367fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:21.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.897+0000 7fefdb11d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:21.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.897+0000 7fefdb11d640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.897+0000 7fefdb11d640 10 monclient: monmap: 2024-09-18T12:28:21.899 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:21.899 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.899 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:21.899226+0000 2024-09-18T12:28:21.899 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:21.899226+0000 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:21.900 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 1 Processor -- start 2024-09-18T12:28:21.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 1 -- start start 2024-09-18T12:28:21.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient: init 2024-09-18T12:28:21.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fefd40a19a0 0x7fefd409b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient(hunting): picked mon.noname-a con 0x7fefd40a19a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd409be20 0x7fefd409c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient(hunting): picked mon.noname-b con 0x7fefd409be20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefd4099250 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd409c730 con 0x7fefd4099250 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd40a1d70 con 0x7fefd40a19a0 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd4098600 con 0x7fefd409be20 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefdb11d640 10 monclient(hunting): authenticate will time out at 937.468611s 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd3fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd409be20 msgr2=0x7fefd409c1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd409be20 0x7fefd409c1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd96ba640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fefd4099250 0x7fefd40996d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48010/0 (socket says 172.21.15.144:48010) 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd96ba640 1 -- 172.21.15.144:0/3481756109 learned_addr learned my addr 172.21.15.144:0/3481756109 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 1 --2- 172.21.15.144:0/3481756109 >> v2:172.21.15.116:3300/0 conn(0x7fefd40a19a0 0x7fefd409b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 10 monclient(hunting): get_auth_request con 0x7fefd40a19a0 auth_method 0 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.898+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_done global_id 54180 payload 274 2024-09-18T12:28:21.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 1 -- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd409be20 msgr2=0x7fefd409c1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:21.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 1 --2- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd409be20 0x7fefd409c1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 1 -- 172.21.15.144:0/3481756109 >> v1:172.21.15.144:6789/0 conn(0x7fefd4099250 legacy=0x7fefd40996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:21.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 1 monclient: found mon.noname-a 2024-09-18T12:28:21.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:21.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.899+0000 7fefd8eb9640 1 -- 172.21.15.144:0/3481756109 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd4097a70 con 0x7fefd40a19a0 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd8eb9640 1 -- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd40a19a0 msgr2=0x7fefd409b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd8eb9640 1 --2- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd40a19a0 0x7fefd409b8e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fefd409c8c0 tx=0x7fefc40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b167fbce7dec385b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 1 -- 172.21.15.144:0/3481756109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4032440 con 0x7fefd40a19a0 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: dump: 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.911 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 1 -- 172.21.15.144:0/3481756109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc40325a0 con 0x7fefd40a19a0 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 5 monclient: authenticate success, global_id 54180 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:21.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 1 -- 172.21.15.144:0/3481756109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4032940 con 0x7fefd40a19a0 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 4 set_mon_vals no callback set 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefd37fe640 10 monclient: dump: 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.913 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 10 monclient: shutdown 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 -- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd40a19a0 msgr2=0x7fefd409b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 --2- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd40a19a0 0x7fefd409b8e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fefd409c8c0 tx=0x7fefc40090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 -- 172.21.15.144:0/3481756109 shutdown_connections 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 --2- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd409be20 0x7fefd409c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 --2- 172.21.15.144:0/3481756109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd40a19a0 0x7fefd409b8e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 -- 172.21.15.144:0/3481756109 >> 172.21.15.144:0/3481756109 conn(0x7fefd40711a0 msgr2=0x7fefd40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:21.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 -- 172.21.15.144:0/3481756109 shutdown_connections 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.902+0000 7fefdb11d640 1 -- 172.21.15.144:0/3481756109 wait complete. 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.903+0000 7fefdb11d640 10 monclient: build_initial_monmap 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.903+0000 7fefdb11d640 10 monclient: monmap: 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:21.915 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.903+0000 7fefdb11d640 1 Processor -- start 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.903+0000 7fefdb11d640 1 -- start start 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.903+0000 7fefdb11d640 10 monclient: init 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 0x7fefd4136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): picked mon.noname-b con 0x7fefd4099250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd409be20 0x7fefd4117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): picked mon.noname-a con 0x7fefd409be20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd40a19a0 0x7fefd4117a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): picked mon.noname-c con 0x7fefd40a19a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd409cc10 con 0x7fefd409be20 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd4130780 con 0x7fefd4099250 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd411a750 con 0x7fefd40a19a0 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefdb11d640 10 monclient(hunting): authenticate will time out at 937.474238s 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 0x7fefd4136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd96ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd40a19a0 msgr2=0x7fefd4117a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd96ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd40a19a0 0x7fefd4117a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 0x7fefd4136fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:46738/0 (socket says 172.21.15.144:46738) 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 -- 172.21.15.144:0/502106774 learned_addr learned my addr 172.21.15.144:0/502106774 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): get_auth_request con 0x7fefd4099250 auth_method 0 2024-09-18T12:28:21.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd409be20 0x7fefd4117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 10 monclient(hunting): get_auth_request con 0x7fefd409be20 auth_method 0 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:21.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient(hunting): handle_auth_done global_id 4903 payload 824 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 -- 172.21.15.144:0/502106774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd40a19a0 msgr2=0x7fefd4117a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd40a19a0 0x7fefd4117a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 -- 172.21.15.144:0/502106774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd409be20 msgr2=0x7fefd4117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd409be20 0x7fefd4117550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient: _finish_hunting 0 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 monclient: found mon.noname-b 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.904+0000 7fefd8eb9640 1 -- 172.21.15.144:0/502106774 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4009390 con 0x7fefd4099250 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.906+0000 7fefd8eb9640 1 -- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 msgr2=0x7fefd4136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.906+0000 7fefd8eb9640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 0x7fefd4136fd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009880 tx=0x7fefc4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4031990 con 0x7fefd4099250 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: dump: 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:21.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc4031af0 con 0x7fefd4099250 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 5 monclient: authenticate success, global_id 4903 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 10 monclient: _renew_subs 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd412f7d0 con 0x7fefd4099250 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4031e90 con 0x7fefd4099250 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 10 monclient: dump: 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.921 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd2ffd640 4 set_mon_vals no callback set 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 10 monclient: _renew_subs 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd41186d0 con 0x7fefd4099250 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fefc4039040 con 0x7fefd4099250 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.907+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd411a330 con 0x7fefd4099250 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.908+0000 7fefd0ff9640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefc0038440 0x7fefc003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.908+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fefc4041030 con 0x7fefd4099250 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.908+0000 7fefd3fff640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefc0038440 0x7fefc003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.908+0000 7fefd3fff640 10 monclient: get_auth_request con 0x7fefc0038440 auth_method 0 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.910+0000 7fefd3fff640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefc0038440 0x7fefc003a8f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fefc80038f0 tx=0x7fefc80090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.912+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fefc4039b20 con 0x7fefd4099250 2024-09-18T12:28:21.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.912+0000 7fefd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:21.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:21.912+0000 7fefd0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:22.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:22.071+0000 7fefdb11d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T12:28:22.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:22.071+0000 7fefdb11d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T12:28:22.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:22.071+0000 7fefdb11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:22.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:22.071+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7fefd412f7d0 con 0x7fefd4099250 2024-09-18T12:28:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.083+0000 7f3f4cd9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:28:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.083+0000 7f3f4cd9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.083+0000 7f3f4cd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.083+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3f480638d0 con 0x7f3f48068a20 2024-09-18T12:28:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.084+0000 7f3f367fc640 1 -- 172.21.15.116:0/3657425799 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f3f3801d300 con 0x7f3f48068a20 2024-09-18T12:28:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.084+0000 7f3f367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.084+0000 7f3f367fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:22.086 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:28:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3f28038440 msgr2=0x7f3f2803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3f28038440 0x7f3f2803a8f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3f4811f8c0 tx=0x7f3f3c007ac0 comp rx=0 tx=0).stop 2024-09-18T12:28:22.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 10 monclient: shutdown 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f48068a20 msgr2=0x7f3f48138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f48068a20 0x7f3f48138a10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800bf20 tx=0x7f3f38009720 comp rx=0 tx=0).stop 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 shutdown_connections 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3f28038440 0x7f3f2803a8f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f4806f490 0x7f3f4811e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3f4806eb80 0x7f3f4811e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 --2- 172.21.15.116:0/3657425799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3f48068a20 0x7f3f48138a10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 >> 172.21.15.116:0/3657425799 conn(0x7f3f48067cd0 msgr2=0x7f3f4809b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 shutdown_connections 2024-09-18T12:28:22.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:22.086+0000 7f3f4cd9d640 1 -- 172.21.15.116:0/3657425799 wait complete. 2024-09-18T12:28:24.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:24.905+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:24.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:24.905+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:24.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:24.905+0000 7fefbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:25.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:28:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.202+0000 7f8f7ea34640 10 monclient: get_monmap_and_config 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.202+0000 7f8f7ea34640 10 monclient: build_initial_monmap 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.202+0000 7f8f7ea34640 10 monclient: monmap: 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:25.204623+0000 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:25.204623+0000 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:25.206 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:25.206 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:25.206 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:25.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.203+0000 7f8f7ea34640 1 Processor -- start 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.203+0000 7f8f7ea34640 1 -- start start 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.203+0000 7f8f7ea34640 10 monclient: init 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f780974f0 0x7f8f7812ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient(hunting): picked mon.noname-a con 0x7f8f780974f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f7809a8a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f78135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient(hunting): picked mon.noname-b con 0x7f8f78133070 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780978c0 con 0x7f8f7809a8a0 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7809ca20 con 0x7f8f780974f0 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78098ad0 con 0x7f8f78133070 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7ea34640 10 monclient(hunting): authenticate will time out at 932.526287s 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7cfd1640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 msgr2=0x7f8f78135450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f7cfd1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f78135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f780974f0 0x7f8f7812ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f780974f0 0x7f8f7812ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50882/0 (socket says 172.21.15.116:50882) 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f777fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8f7809a8a0 0x7f8f7809ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41482/0 (socket says 172.21.15.116:41482) 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 -- 172.21.15.116:0/363486193 learned_addr learned my addr 172.21.15.116:0/363486193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): get_auth_request con 0x7f8f780974f0 auth_method 0 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:25.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient(hunting): handle_auth_done global_id 54186 payload 274 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 -- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 msgr2=0x7f8f78135450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 --2- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f78135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 -- 172.21.15.116:0/363486193 >> v1:172.21.15.116:6789/0 conn(0x7f8f7809a8a0 legacy=0x7f8f7809ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 monclient: found mon.noname-a 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:25.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.204+0000 7f8f77fff640 1 -- 172.21.15.116:0/363486193 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f78098db0 con 0x7f8f780974f0 2024-09-18T12:28:25.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f77fff640 1 -- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 msgr2=0x7f8f7812ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:25.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f77fff640 1 --2- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812ab00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009910 tx=0x7f8f680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef59a8d6116dfe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:25.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 1 -- 172.21.15.116:0/363486193 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68032440 con 0x7f8f780974f0 2024-09-18T12:28:25.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:25.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: dump: 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:25.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 1 -- 172.21.15.116:0/363486193 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f680325a0 con 0x7f8f780974f0 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f7ea34640 5 monclient: authenticate success, global_id 54186 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f7ea34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 1 -- 172.21.15.116:0/363486193 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68032940 con 0x7f8f780974f0 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f7ea34640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f7ea34640 4 set_mon_vals no callback set 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.207+0000 7f8f76ffd640 10 monclient: dump: 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:25.219 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 10 monclient: shutdown 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 -- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 msgr2=0x7f8f7812ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812ab00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009910 tx=0x7f8f680091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 -- 172.21.15.116:0/363486193 shutdown_connections 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f78135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/363486193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812ab00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 -- 172.21.15.116:0/363486193 >> 172.21.15.116:0/363486193 conn(0x7f8f780672b0 msgr2=0x7f8f780676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 -- 172.21.15.116:0/363486193 shutdown_connections 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 1 -- 172.21.15.116:0/363486193 wait complete. 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 10 monclient: build_initial_monmap 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.208+0000 7f8f7ea34640 10 monclient: monmap: 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:25.221 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.209+0000 7f8f7ea34640 1 Processor -- start 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.209+0000 7f8f7ea34640 1 -- start start 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.209+0000 7f8f7ea34640 10 monclient: init 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): picked mon.noname-a con 0x7f8f780974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f7809a8a0 0x7f8f7812be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:25.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): picked mon.noname-b con 0x7f8f7809a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f7812c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): picked mon.noname-c con 0x7f8f78133070 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7812cbe0 con 0x7f8f780974f0 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7812a510 con 0x7f8f7809a8a0 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7812e1c0 con 0x7f8f78133070 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7ea34640 10 monclient(hunting): authenticate will time out at 932.532173s 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7cfd1640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 msgr2=0x7f8f7812c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f7cfd1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f7812c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:25.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50886/0 (socket says 172.21.15.116:50886) 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 -- 172.21.15.116:0/3047618191 learned_addr learned my addr 172.21.15.116:0/3047618191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): get_auth_request con 0x7f8f780974f0 auth_method 0 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient(hunting): handle_auth_done global_id 54189 payload 824 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 -- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 msgr2=0x7f8f7812c3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:25.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f7812c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 -- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f7809a8a0 msgr2=0x7f8f7812be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f7809a8a0 0x7f8f7812be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 monclient: found mon.noname-a 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.210+0000 7f8f77fff640 1 -- 172.21.15.116:0/3047618191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f68009450 con 0x7f8f780974f0 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.212+0000 7f8f77fff640 1 -- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 msgr2=0x7f8f7812b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.212+0000 7f8f77fff640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812b930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009910 tx=0x7f8f68032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.212+0000 7f8f6ffff640 1 -- 172.21.15.116:0/3047618191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6803ca40 con 0x7f8f780974f0 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:25.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: dump: 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: _finish_auth 0 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 1 -- 172.21.15.116:0/3047618191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6803cba0 con 0x7f8f780974f0 2024-09-18T12:28:25.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f7ea34640 5 monclient: authenticate success, global_id 54189 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f7ea34640 10 monclient: _renew_subs 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f7ea34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7812dee0 con 0x7f8f780974f0 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 1 -- 172.21.15.116:0/3047618191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f680313a0 con 0x7f8f780974f0 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 10 monclient: dump: 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f767fc640 4 set_mon_vals no callback set 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f7ea34640 10 monclient: _renew_subs 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f7ea34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7812f170 con 0x7f8f780974f0 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.213+0000 7f8f6ffff640 1 -- 172.21.15.116:0/3047618191 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8f68039040 con 0x7f8f780974f0 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f7ea34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f7ea34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f7ea34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:25.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7804bf60 con 0x7f8f780974f0 2024-09-18T12:28:25.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f6ffff640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f58038440 0x7f8f5803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:25.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f6ffff640 1 -- 172.21.15.116:0/3047618191 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f8f6806bd40 con 0x7f8f780974f0 2024-09-18T12:28:25.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f777fe640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f58038440 0x7f8f5803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:25.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.214+0000 7f8f777fe640 10 monclient: get_auth_request con 0x7f8f58038440 auth_method 0 2024-09-18T12:28:25.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.217+0000 7f8f777fe640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f58038440 0x7f8f5803a8f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8f60004ec0 tx=0x7f8f60007ac0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:25.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.218+0000 7f8f6ffff640 1 -- 172.21.15.116:0/3047618191 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8f6804b530 con 0x7f8f780974f0 2024-09-18T12:28:25.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.218+0000 7f8f6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:25.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.218+0000 7f8f6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.434+0000 7f8f7ea34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:28:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.434+0000 7f8f7ea34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.434+0000 7f8f7ea34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.434+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8f780638d0 con 0x7f8f780974f0 2024-09-18T12:28:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.434+0000 7f8f6ffff640 1 -- 172.21.15.116:0/3047618191 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f8f68040070 con 0x7f8f780974f0 2024-09-18T12:28:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.434+0000 7f8f6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.434+0000 7f8f6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:25.437 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f58038440 msgr2=0x7f8f5803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f58038440 0x7f8f5803a8f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8f60004ec0 tx=0x7f8f60007ac0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 10 monclient: shutdown 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 msgr2=0x7f8f7812b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812b930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009910 tx=0x7f8f68032000 comp rx=0 tx=0).stop 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 shutdown_connections 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f58038440 0x7f8f5803a8f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f78133070 0x7f8f7812c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f7809a8a0 0x7f8f7812be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 --2- 172.21.15.116:0/3047618191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f780974f0 0x7f8f7812b930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.436+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 >> 172.21.15.116:0/3047618191 conn(0x7f8f780672b0 msgr2=0x7f8f7809d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.437+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 shutdown_connections 2024-09-18T12:28:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:25.437+0000 7f8f7ea34640 1 -- 172.21.15.116:0/3047618191 wait complete. 2024-09-18T12:28:25.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:25.905+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:25.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:25.905+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:26.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:26.905+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:26.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:26.905+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:27.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:27.905+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:27.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:27.905+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:28.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:28:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.552+0000 7f411c100640 10 monclient: get_monmap_and_config 2024-09-18T12:28:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.552+0000 7f411c100640 10 monclient: build_initial_monmap 2024-09-18T12:28:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.552+0000 7f411c100640 10 monclient: monmap: 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:28.554526+0000 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:28.554526+0000 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:28.555 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.553+0000 7f411c100640 1 Processor -- start 2024-09-18T12:28:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.553+0000 7f411c100640 1 -- start start 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.553+0000 7f411c100640 10 monclient: init 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4114096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41140971e0 0x7f41140975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient(hunting): picked mon.noname-a con 0x7f41140971e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4114133b30 0x7f4114135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient(hunting): picked mon.noname-c con 0x7f4114133b30 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4114099620 con 0x7f4114096e10 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4114099340 con 0x7f41140971e0 2024-09-18T12:28:28.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4114099900 con 0x7f4114133b30 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411c100640 10 monclient(hunting): authenticate will time out at 935.876233s 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41140971e0 0x7f41140975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41140971e0 0x7f41140975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48114/0 (socket says 172.21.15.116:48114) 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f4119e9c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4114096e10 0x7f411409a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51682/0 (socket says 172.21.15.116:51682) 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 -- 172.21.15.116:0/1577254940 learned_addr learned my addr 172.21.15.116:0/1577254940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): get_auth_request con 0x7f41140971e0 auth_method 0 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:28.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411a69d640 1 --2- 172.21.15.116:0/1577254940 >> v2:172.21.15.144:3300/0 conn(0x7f4114133b30 0x7f4114135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411a69d640 10 monclient(hunting): get_auth_request con 0x7f4114133b30 auth_method 0 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411a69d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411a69d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411a69d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient(hunting): handle_auth_done global_id 54192 payload 274 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 -- 172.21.15.116:0/1577254940 >> v2:172.21.15.144:3300/0 conn(0x7f4114133b30 msgr2=0x7f4114135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 --2- 172.21.15.116:0/1577254940 >> v2:172.21.15.144:3300/0 conn(0x7f4114133b30 0x7f4114135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 -- 172.21.15.116:0/1577254940 >> v1:172.21.15.116:6789/0 conn(0x7f4114096e10 legacy=0x7f411409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient: _finish_hunting 0 2024-09-18T12:28:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 monclient: found mon.noname-a 2024-09-18T12:28:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:28.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.554+0000 7f411969b640 1 -- 172.21.15.116:0/1577254940 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411409ac70 con 0x7f41140971e0 2024-09-18T12:28:28.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f411969b640 1 -- 172.21.15.116:0/1577254940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41140971e0 msgr2=0x7f41140975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:28.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f411969b640 1 --2- 172.21.15.116:0/1577254940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41140971e0 0x7f41140975e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f411409b020 tx=0x7f41040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8abd61a90454b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:28.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 1 -- 172.21.15.116:0/1577254940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410402ae80 con 0x7f41140971e0 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: dump: 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:28.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: _finish_auth 0 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f411c100640 5 monclient: authenticate success, global_id 54192 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f411c100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 1 -- 172.21.15.116:0/1577254940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104032440 con 0x7f41140971e0 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f411c100640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 1 -- 172.21.15.116:0/1577254940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040327c0 con 0x7f41140971e0 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f411c100640 4 set_mon_vals no callback set 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.557+0000 7f4118e9a640 10 monclient: dump: 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:28.570 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 10 monclient: shutdown 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 -- 172.21.15.116:0/1577254940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41140971e0 msgr2=0x7f41140975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 --2- 172.21.15.116:0/1577254940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41140971e0 0x7f41140975e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f411409b020 tx=0x7f41040091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 -- 172.21.15.116:0/1577254940 shutdown_connections 2024-09-18T12:28:28.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 --2- 172.21.15.116:0/1577254940 >> v2:172.21.15.144:3300/0 conn(0x7f4114133b30 0x7f4114135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 --2- 172.21.15.116:0/1577254940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41140971e0 0x7f41140975e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 -- 172.21.15.116:0/1577254940 >> 172.21.15.116:0/1577254940 conn(0x7f41140672b0 msgr2=0x7f41140676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 -- 172.21.15.116:0/1577254940 shutdown_connections 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 1 -- 172.21.15.116:0/1577254940 wait complete. 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 10 monclient: build_initial_monmap 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.558+0000 7f411c100640 10 monclient: monmap: 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:28.572 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 Processor -- start 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 -- start start 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient: init 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 0x7f411412e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): picked mon.noname-a con 0x7f4114096e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41140971e0 0x7f411412f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): picked mon.noname-c con 0x7f41140971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:28.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4114133b30 0x7f411412fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): picked mon.noname-b con 0x7f4114133b30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411409b050 con 0x7f4114096e10 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411409fd30 con 0x7f4114133b30 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4114141df0 con 0x7f41140971e0 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411c100640 10 monclient(hunting): authenticate will time out at 935.881558s 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411969b640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41140971e0 msgr2=0x7f411412f710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411969b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41140971e0 0x7f411412f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 0x7f411412e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:28.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 0x7f411412e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48126/0 (socket says 172.21.15.116:48126) 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 -- 172.21.15.116:0/3840405070 learned_addr learned my addr 172.21.15.116:0/3840405070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): get_auth_request con 0x7f4114096e10 auth_method 0 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4114133b30 0x7f411412fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 10 monclient(hunting): get_auth_request con 0x7f4114133b30 auth_method 0 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:28.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f411a69d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient(hunting): handle_auth_done global_id 54195 payload 824 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 -- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41140971e0 msgr2=0x7f411412f710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41140971e0 0x7f411412f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 -- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4114133b30 msgr2=0x7f411412fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4114133b30 0x7f411412fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 monclient: found mon.noname-a 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.559+0000 7f4119e9c640 1 -- 172.21.15.116:0/3840405070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104009450 con 0x7f4114096e10 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4119e9c640 1 -- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 msgr2=0x7f411412e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4119e9c640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 0x7f411412e710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f411000c930 tx=0x7f4110004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 1 -- 172.21.15.116:0/3840405070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000a5e0 con 0x7f4114096e10 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: dump: 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 1 -- 172.21.15.116:0/3840405070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411000a740 con 0x7f4114096e10 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f411c100640 5 monclient: authenticate success, global_id 54195 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f411c100640 10 monclient: _renew_subs 2024-09-18T12:28:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f411c100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41141312e0 con 0x7f4114096e10 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 1 -- 172.21.15.116:0/3840405070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411000ab00 con 0x7f4114096e10 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f4109ffb640 10 monclient: dump: 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:28.579 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f410bfff640 4 set_mon_vals no callback set 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f411c100640 10 monclient: _renew_subs 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f411c100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.562+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411411c960 con 0x7f4114096e10 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f4109ffb640 1 -- 172.21.15.116:0/3840405070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f41100070f0 con 0x7f4114096e10 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f4109ffb640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40f8038440 0x7f40f803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f4109ffb640 1 -- 172.21.15.116:0/3840405070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f411004b770 con 0x7f4114096e10 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f411c100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f411c100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f411c100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411411da50 con 0x7f4114096e10 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f411969b640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40f8038440 0x7f40f803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.563+0000 7f411969b640 10 monclient: get_auth_request con 0x7f40f8038440 auth_method 0 2024-09-18T12:28:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.566+0000 7f411969b640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40f8038440 0x7f40f803a8f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4114076be0 tx=0x7f4104032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.568+0000 7f4109ffb640 1 -- 172.21.15.116:0/3840405070 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f411001b400 con 0x7f4114096e10 2024-09-18T12:28:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.568+0000 7f4109ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:28.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.568+0000 7f4109ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:28.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.780+0000 7f411c100640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:28:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.780+0000 7f411c100640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.780+0000 7f411c100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.780+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f411404bf60 con 0x7f4114096e10 2024-09-18T12:28:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.780+0000 7f4109ffb640 1 -- 172.21.15.116:0/3840405070 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4110010af0 con 0x7f4114096e10 2024-09-18T12:28:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.780+0000 7f4109ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.780+0000 7f4109ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:28.783 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40f8038440 msgr2=0x7f40f803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40f8038440 0x7f40f803a8f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4114076be0 tx=0x7f4104032000 comp rx=0 tx=0).stop 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 10 monclient: shutdown 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 msgr2=0x7f411412e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 0x7f411412e710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f411000c930 tx=0x7f4110004f70 comp rx=0 tx=0).stop 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 shutdown_connections 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40f8038440 0x7f40f803a8f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4114133b30 0x7f411412fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41140971e0 0x7f411412f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 --2- 172.21.15.116:0/3840405070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4114096e10 0x7f411412e710 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:28.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 >> 172.21.15.116:0/3840405070 conn(0x7f41140672b0 msgr2=0x7f4114132000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:28.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 shutdown_connections 2024-09-18T12:28:28.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:28.782+0000 7f411c100640 1 -- 172.21.15.116:0/3840405070 wait complete. 2024-09-18T12:28:28.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:28.905+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:28.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:28.905+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:29.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:29.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:29.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:29.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:30.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:30.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:30.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:30.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:31.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.897+0000 7fef4bc17640 10 monclient: get_monmap_and_config 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.897+0000 7fef4bc17640 10 monclient: build_initial_monmap 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.897+0000 7fef4bc17640 10 monclient: monmap: 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:31.898966+0000 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:31.898966+0000 2024-09-18T12:28:31.899 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:31.900 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:31.900 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:31.900 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:31.900 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:31.900 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:31.900 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:31.900 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:31.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.897+0000 7fef4bc17640 1 Processor -- start 2024-09-18T12:28:31.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 1 -- start start 2024-09-18T12:28:31.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient: init 2024-09-18T12:28:31.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:31.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient(hunting): picked mon.noname-c con 0x7fef440981c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef44131da0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 0x7fef44067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient(hunting): picked mon.noname-b con 0x7fef44132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef44098ad0 con 0x7fef44131da0 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef44131bd0 con 0x7fef440981c0 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4409ca60 con 0x7fef44132170 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4bc17640 10 monclient(hunting): authenticate will time out at 939.220683s 2024-09-18T12:28:31.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4a1b4640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 msgr2=0x7fef44067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.898+0000 7fef4a1b4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 0x7fef44067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:50478/0 (socket says 172.21.15.116:50478) 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 learned_addr learned my addr 172.21.15.116:0/2463929416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 10 monclient(hunting): get_auth_request con 0x7fef440981c0 auth_method 0 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:31.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.899+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:31.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:31.901+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v1:172.21.15.144:6789/0 conn(0x7fef44131da0 legacy=0x7fef44067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:31.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:31.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:31.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:31.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:32.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.099+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 msgr2=0x7fef44067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:32.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.099+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 0x7fef44067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 10 monclient(hunting): get_auth_request con 0x7fef440981c0 auth_method 0 2024-09-18T12:28:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.100+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:32.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.500+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 msgr2=0x7fef44067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.500+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 0x7fef44067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 10 monclient(hunting): get_auth_request con 0x7fef440981c0 auth_method 0 2024-09-18T12:28:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:32.501+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:32.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:32.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.301+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 msgr2=0x7fef44067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:33.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.301+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 0x7fef44067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:33.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 10 monclient(hunting): get_auth_request con 0x7fef440981c0 auth_method 0 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:33.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:33.302+0000 7fef499b3640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:33.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:33.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:33.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:33.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:34.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 10 monclient(hunting): tick 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 msgr2=0x7fef44067820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef44132170 0x7fef44067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 msgr2=0x7fef44098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.144:3300/0 conn(0x7fef440981c0 0x7fef44098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 -- 172.21.15.116:0/2463929416 >> v1:172.21.15.144:6789/0 conn(0x7fef44131da0 legacy=0x7fef44067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:34.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 10 monclient: _add_conns ranks=[0,3] 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef28003720 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fef28007440 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 -- 172.21.15.116:0/2463929416 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef4409cca0 con 0x7fef28003720 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 1 -- 172.21.15.116:0/2463929416 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef44131bd0 con 0x7fef28007440 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef3b7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 10 monclient(hunting): get_auth_request con 0x7fef28007440 auth_method 0 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:34.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:34.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:34.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.899+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:34.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:34.901+0000 7fef499b3640 1 -- 172.21.15.116:0/2463929416 >> v1:172.21.15.116:6789/0 conn(0x7fef28003720 legacy=0x7fef28005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:34.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:34.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:34.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:34.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:34.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:34.906+0000 7fefbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 10 monclient(hunting): get_auth_request con 0x7fef28007440 auth_method 0 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.100+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 10 monclient(hunting): get_auth_request con 0x7fef28007440 auth_method 0 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:35.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:35.501+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:28:35.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:35.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:35.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:35.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 10 monclient(hunting): get_auth_request con 0x7fef28007440 auth_method 0 2024-09-18T12:28:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.302+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:28:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.364+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87436027 0 0) 0x7fef4409cca0 con 0x7fef28003720 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.364+0000 7fef489b1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.364+0000 7fef489b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.364+0000 7fef489b1640 10 monclient(hunting): my global_id is 54198 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.364+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef300034e0 con 0x7fef28003720 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3950127594 0 0) 0x7fef300034e0 con 0x7fef28003720 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 msgr2=0x7fef280098f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 --2- 172.21.15.116:0/2463929416 >> v2:172.21.15.116:3300/0 conn(0x7fef28007440 0x7fef280098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: _finish_hunting 0 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:28:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2800b020 con 0x7fef28003720 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: _finish_auth 0 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fef4000c680 con 0x7fef28003720 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4bc17640 5 monclient: authenticate success, global_id 54198 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4bc17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: dump: 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.371 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 >> v1:172.21.15.116:6789/0 conn(0x7fef28003720 legacy=0x7fef28005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 reap_dead start 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef300039d0 0x7fef30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): picked mon.a con 0x7fef300039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef30004780 0x7fef30004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): picked mon.c con 0x7fef30004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30005250 0x7fef3000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): picked mon.b con 0x7fef30005250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef44131bd0 con 0x7fef300039d0 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3000c1c0 con 0x7fef30005250 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3000c380 con 0x7fef30004780 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:36.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef489b1640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4bc17640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef491b2640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef30004780 msgr2=0x7fef30004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef491b2640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef30004780 0x7fef30004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30005250 0x7fef3000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): get_auth_request con 0x7fef30005250 auth_method 0 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_done global_id 54198 payload 293 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.365+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef30004780 msgr2=0x7fef30004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef30004780 0x7fef30004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef300039d0 msgr2=0x7fef30003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef300039d0 0x7fef30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 10 monclient: _finish_hunting 0 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 1 monclient: found mon.b 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef300034e0 con 0x7fef30005250 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 10 monclient: _finish_auth 0 2024-09-18T12:28:36.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.366+0000 7fef4a1b4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.369+0000 7fef4a1b4640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30005250 msgr2=0x7fef3000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.369+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30005250 0x7fef3000b9d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fef4409d090 tx=0x7fef3c007530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7aed0566d283c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c02aaf0 con 0x7fef30005250 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 10 monclient: dump: 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c02ac90 con 0x7fef30005250 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 1 -- 172.21.15.116:0/2463929416 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0336c0 con 0x7fef30005250 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef4bc17640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef4bc17640 4 set_mon_vals no callback set 2024-09-18T12:28:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.370+0000 7fef489b1640 10 monclient: dump: 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 10 monclient: shutdown 2024-09-18T12:28:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 -- 172.21.15.116:0/2463929416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30005250 msgr2=0x7fef3000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30005250 0x7fef3000b9d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fef4409d090 tx=0x7fef3c007530 comp rx=0 tx=0).stop 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 -- 172.21.15.116:0/2463929416 shutdown_connections 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30005250 0x7fef3000b9d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef30004780 0x7fef30004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 --2- 172.21.15.116:0/2463929416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef300039d0 0x7fef30003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 -- 172.21.15.116:0/2463929416 >> 172.21.15.116:0/2463929416 conn(0x7fef440a1e30 msgr2=0x7fef440a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 -- 172.21.15.116:0/2463929416 shutdown_connections 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.371+0000 7fef4bc17640 1 -- 172.21.15.116:0/2463929416 wait complete. 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.372+0000 7fef4bc17640 10 monclient: build_initial_monmap 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.372+0000 7fef4bc17640 10 monclient: monmap: 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.372+0000 7fef4bc17640 1 Processor -- start 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 1 -- start start 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient: init 2024-09-18T12:28:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef300039d0 0x7fef44140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): picked mon.noname-c con 0x7fef300039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30004780 0x7fef441309a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): picked mon.noname-b con 0x7fef30004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 0x7fef4411a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): picked mon.noname-a con 0x7fef30005250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4411c580 con 0x7fef30005250 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4409b930 con 0x7fef30004780 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef44141bf0 con 0x7fef300039d0 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4bc17640 10 monclient(hunting): authenticate will time out at 943.695544s 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef499b3640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef300039d0 msgr2=0x7fef44140d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef499b3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef300039d0 0x7fef44140d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 0x7fef4411a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 0x7fef4411a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42808/0 (socket says 172.21.15.116:42808) 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 1 -- 172.21.15.116:0/1425984013 learned_addr learned my addr 172.21.15.116:0/1425984013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 10 monclient(hunting): get_auth_request con 0x7fef30005250 auth_method 0 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef491b2640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30004780 0x7fef441309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef491b2640 10 monclient(hunting): get_auth_request con 0x7fef30004780 auth_method 0 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef491b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef491b2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.373+0000 7fef491b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 10 monclient(hunting): handle_auth_done global_id 54201 payload 824 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 1 -- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef300039d0 msgr2=0x7fef44140d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef300039d0 0x7fef44140d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 1 -- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30004780 msgr2=0x7fef441309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30004780 0x7fef441309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 10 monclient: _finish_hunting 0 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 1 monclient: found mon.noname-a 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef4a1b4640 1 -- 172.21.15.116:0/1425984013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44098cb0 con 0x7fef30005250 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.374+0000 7fef491b2640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30004780 0x7fef441309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4a1b4640 1 -- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 msgr2=0x7fef4411a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4a1b4640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 0x7fef4411a420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0075b0 tx=0x7fef3c033000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 1 -- 172.21.15.116:0/1425984013 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c03f430 con 0x7fef30005250 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: dump: 2024-09-18T12:28:36.390 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 1 -- 172.21.15.116:0/1425984013 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c03f590 con 0x7fef30005250 2024-09-18T12:28:36.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4bc17640 5 monclient: authenticate success, global_id 54201 2024-09-18T12:28:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4bc17640 10 monclient: _renew_subs 2024-09-18T12:28:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4bc17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4411cbd0 con 0x7fef30005250 2024-09-18T12:28:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 1 -- 172.21.15.116:0/1425984013 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c03f930 con 0x7fef30005250 2024-09-18T12:28:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef39ffb640 10 monclient: dump: 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef3bfff640 4 set_mon_vals no callback set 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4bc17640 10 monclient: _renew_subs 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4bc17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.376+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef44141250 con 0x7fef30005250 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.377+0000 7fef4bc17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.377+0000 7fef39ffb640 1 -- 172.21.15.116:0/1425984013 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fef3c030070 con 0x7fef30005250 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.377+0000 7fef4bc17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.377+0000 7fef4bc17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.377+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4411dbd0 con 0x7fef30005250 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.378+0000 7fef39ffb640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef28038dc0 0x7fef2803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.378+0000 7fef39ffb640 1 -- 172.21.15.116:0/1425984013 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fef3c06ba80 con 0x7fef30005250 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.378+0000 7fef499b3640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef28038dc0 0x7fef2803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.378+0000 7fef499b3640 10 monclient: get_auth_request con 0x7fef28038dc0 auth_method 0 2024-09-18T12:28:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.381+0000 7fef499b3640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef28038dc0 0x7fef2803b270 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef40006df0 tx=0x7fef40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.383+0000 7fef39ffb640 1 -- 172.21.15.116:0/1425984013 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fef3c03e3d0 con 0x7fef30005250 2024-09-18T12:28:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.383+0000 7fef39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:36.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.383+0000 7fef39ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.598+0000 7fef4bc17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:28:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.598+0000 7fef4bc17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.598+0000 7fef4bc17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.598+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fef4411dbd0 con 0x7fef30005250 2024-09-18T12:28:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.599+0000 7fef39ffb640 1 -- 172.21.15.116:0/1425984013 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fef3c031370 con 0x7fef30005250 2024-09-18T12:28:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.599+0000 7fef39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.599+0000 7fef39ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:36.602 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:28:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef28038dc0 msgr2=0x7fef2803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef28038dc0 0x7fef2803b270 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef40006df0 tx=0x7fef40008040 comp rx=0 tx=0).stop 2024-09-18T12:28:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 10 monclient: shutdown 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 msgr2=0x7fef4411a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 0x7fef4411a420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0075b0 tx=0x7fef3c033000 comp rx=0 tx=0).stop 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 shutdown_connections 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef28038dc0 0x7fef2803b270 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef30005250 0x7fef4411a420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef30004780 0x7fef441309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 --2- 172.21.15.116:0/1425984013 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef300039d0 0x7fef44140d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 >> 172.21.15.116:0/1425984013 conn(0x7fef440a1e30 msgr2=0x7fef44096f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 shutdown_connections 2024-09-18T12:28:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.601+0000 7fef4bc17640 1 -- 172.21.15.116:0/1425984013 wait complete. 2024-09-18T12:28:36.613 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:28:36.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:28:36.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.704+0000 7f6d52d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.704+0000 7f6d52d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.704+0000 7f6d52d1d640 10 monclient: monmap: 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:36.706566+0000 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:36.706566+0000 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:36.707 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:36.708 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:36.708 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:36.708 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:36.708 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.705+0000 7f6d52d1d640 1 Processor -- start 2024-09-18T12:28:36.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.705+0000 7f6d52d1d640 1 -- start start 2024-09-18T12:28:36.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.705+0000 7f6d52d1d640 10 monclient: init 2024-09-18T12:28:36.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d4c1318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d4c09ea50 0x7f6d4c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4c09ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 0x7f6d4c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4c136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c132640 con 0x7f6d4c1318f0 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4c131cc0 con 0x7f6d4c09ea50 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c09fbf0 con 0x7f6d4c136c20 2024-09-18T12:28:36.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d52d1d640 10 monclient(hunting): authenticate will time out at 944.028172s 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d512ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 msgr2=0x7f6d4c139000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d512ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 0x7f6d4c139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d4c09ea50 0x7f6d4c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d4c09ea50 0x7f6d4c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42822/0 (socket says 172.21.15.116:42822) 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 -- 172.21.15.116:0/1092705309 learned_addr learned my addr 172.21.15.116:0/1092705309 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): get_auth_request con 0x7f6d4c09ea50 auth_method 0 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient(hunting): handle_auth_done global_id 54204 payload 274 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 -- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 msgr2=0x7f6d4c139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 --2- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 0x7f6d4c139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 -- 172.21.15.116:0/1092705309 >> v1:172.21.15.144:6789/0 conn(0x7f6d4c1318f0 legacy=0x7f6d4c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 monclient: found mon.noname-a 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.706+0000 7f6d43fff640 1 -- 172.21.15.116:0/1092705309 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c09fe30 con 0x7f6d4c09ea50 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d43fff640 1 -- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 msgr2=0x7f6d4c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d43fff640 1 --2- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c09ee20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009910 tx=0x7f6d3c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a937af286c6e94 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 1 -- 172.21.15.116:0/1092705309 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c032440 con 0x7f6d4c09ea50 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: dump: 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 1 -- 172.21.15.116:0/1092705309 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3c0325a0 con 0x7f6d4c09ea50 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d52d1d640 5 monclient: authenticate success, global_id 54204 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d52d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d52d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d52d1d640 4 set_mon_vals no callback set 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 1 -- 172.21.15.116:0/1092705309 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c032940 con 0x7f6d4c09ea50 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.709+0000 7f6d437fe640 10 monclient: dump: 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 10 monclient: shutdown 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 msgr2=0x7f6d4c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 --2- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c09ee20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009910 tx=0x7f6d3c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1092705309 shutdown_connections 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 --2- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 0x7f6d4c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 --2- 172.21.15.116:0/1092705309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c09ee20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1092705309 >> 172.21.15.116:0/1092705309 conn(0x7f6d4c067110 msgr2=0x7f6d4c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1092705309 shutdown_connections 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1092705309 wait complete. 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 10 monclient: monmap: 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.710+0000 7f6d52d1d640 1 Processor -- start 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 1 -- start start 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient: init 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c097570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4c09ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d4c1318f0 0x7f6d4c097ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4c1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 0x7f6d4c097ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4c136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c09bf10 con 0x7f6d4c09ea50 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c09db50 con 0x7f6d4c1318f0 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c099340 con 0x7f6d4c136c20 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d52d1d640 10 monclient(hunting): authenticate will time out at 944.033590s 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d512ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 msgr2=0x7f6d4c097ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d512ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 0x7f6d4c097ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c097570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c097570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42830/0 (socket says 172.21.15.116:42830) 2024-09-18T12:28:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 -- 172.21.15.116:0/1154350648 learned_addr learned my addr 172.21.15.116:0/1154350648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): get_auth_request con 0x7f6d4c09ea50 auth_method 0 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient(hunting): handle_auth_done global_id 54207 payload 824 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 msgr2=0x7f6d4c097ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d4c136c20 0x7f6d4c097ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d43fff640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d4c1318f0 0x7f6d4c097ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d4c1318f0 msgr2=0x7f6d4c097ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d4c1318f0 0x7f6d4c097ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient: _finish_hunting 0 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 monclient: found mon.noname-a 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.711+0000 7f6d50ab9640 1 -- 172.21.15.116:0/1154350648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c009450 con 0x7f6d4c09ea50 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d50ab9640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 msgr2=0x7f6d4c097570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d50ab9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c097570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800c8b0 tx=0x7f6d480026c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800cea0 con 0x7f6d4c09ea50 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: dump: 2024-09-18T12:28:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: _finish_auth 0 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d52d1d640 5 monclient: authenticate success, global_id 54207 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4800aa60 con 0x7f6d4c09ea50 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800ae40 con 0x7f6d4c09ea50 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d40ff9640 10 monclient: dump: 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d52d1d640 10 monclient: _renew_subs 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d52d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d42ffd640 4 set_mon_vals no callback set 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c131640 con 0x7f6d4c09ea50 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d52d1d640 10 monclient: _renew_subs 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d52d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.714+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c098e30 con 0x7f6d4c09ea50 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6d48012670 con 0x7f6d4c09ea50 2024-09-18T12:28:36.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d52d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d52d1d640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 0x7f6d4c02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d52d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c09ab00 con 0x7f6d4c098530 2024-09-18T12:28:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 0x7f6d4c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient: get_auth_request con 0x7f6d4c098530 auth_method 0 2024-09-18T12:28:36.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d40ff9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6d38038440 0x7f6d3803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f6d4804bf00 con 0x7f6d4c09ea50 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d512ba640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6d38038440 0x7f6d3803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 10 monclient(hunting): handle_auth_done global_id 54210 payload 824 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d43fff640 1 -- 172.21.15.116:0/1154350648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d4c04bf60 con 0x7f6d4c098530 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.715+0000 7f6d512ba640 10 monclient: get_auth_request con 0x7f6d38038440 auth_method 0 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.717+0000 7f6d43fff640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 msgr2=0x7f6d4c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.717+0000 7f6d43fff640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 0x7f6d4c02aa40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c034000 tx=0x7f6d3c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c03c940 con 0x7f6d4c098530 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d512ba640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6d38038440 0x7f6d3803a8f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6d44003940 tx=0x7f6d44006cb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 10 monclient: dump: 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6d3c03caa0 con 0x7f6d4c098530 2024-09-18T12:28:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:36.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:36.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 msgr2=0x7f6d4c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 0x7f6d4c02aa40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c034000 tx=0x7f6d3c032000 comp rx=0 tx=0).stop 2024-09-18T12:28:36.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 msgr2=0x7f6d4c02aa40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:36.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.718+0000 7f6d40ff9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c098530 0x7f6d4c02aa40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d52d1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d52d1d640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 0x7f6d4c05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d52d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c065e00 con 0x7f6d4c05e310 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 0x7f6d4c05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient: get_auth_request con 0x7f6d4c05e310 auth_method 0 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 10 monclient(hunting): handle_auth_done global_id 54213 payload 824 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.725+0000 7f6d43fff640 1 -- 172.21.15.116:0/1154350648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6d4c133ea0 con 0x7f6d4c05e310 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.726+0000 7f6d43fff640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 msgr2=0x7f6d4c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.726+0000 7f6d43fff640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 0x7f6d4c05e710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0686d0 tx=0x7f6d3c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c03ce20 con 0x7f6d4c05e310 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 10 monclient: dump: 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.742 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f6d3c03c730 con 0x7f6d4c05e310 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 msgr2=0x7f6d4c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 0x7f6d4c05e710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0686d0 tx=0x7f6d3c03d000 comp rx=0 tx=0).stop 2024-09-18T12:28:36.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 msgr2=0x7f6d4c05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d40ff9640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c05e310 0x7f6d4c05e710 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0686d0 tx=0x7f6d3c03d000 comp rx=0 tx=0).stop 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 38, 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:36.744 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:36.746 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:36.747 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:36.748 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:36.750 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.752 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:36.753 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:36.754 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.755 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:36.756 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6d38038440 msgr2=0x7f6d3803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6d38038440 0x7f6d3803a8f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6d44003940 tx=0x7f6d44006cb0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 10 monclient: shutdown 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 msgr2=0x7f6d4c097570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 --2- 172.21.15.116:0/1154350648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d4c09ea50 0x7f6d4c097570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800c8b0 tx=0x7f6d480026c0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d512ba640 1 -- 172.21.15.116:0/1154350648 reap_dead start 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d512ba640 1 -- 172.21.15.116:0/1154350648 reap_dead start 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 shutdown_connections 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 >> 172.21.15.116:0/1154350648 conn(0x7f6d4c067110 msgr2=0x7f6d4c0a3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 shutdown_connections 2024-09-18T12:28:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.727+0000 7f6d52d1d640 1 -- 172.21.15.116:0/1154350648 wait complete. 2024-09-18T12:28:36.759 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:28:36.759 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:28:36.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.851+0000 7efdba100640 10 monclient: get_monmap_and_config 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.851+0000 7efdba100640 10 monclient: build_initial_monmap 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.851+0000 7efdba100640 10 monclient: monmap: 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:36.852830+0000 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:36.852830+0000 2024-09-18T12:28:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:36.854 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 1 Processor -- start 2024-09-18T12:28:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 1 -- start start 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient: init 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdb409c9c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efdb409e340 0x7efdb409e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient(hunting): picked mon.noname-c con 0x7efdb409e340 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdb4134aa0 0x7efdb4136e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient(hunting): picked mon.noname-a con 0x7efdb4134aa0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb409bdd0 con 0x7efdb409c9c0 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb409ec90 con 0x7efdb4134aa0 2024-09-18T12:28:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb409df40 con 0x7efdb409e340 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdba100640 10 monclient(hunting): authenticate will time out at 944.174703s 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efdb409c9c0 0x7efdb409d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33824/0 (socket says 172.21.15.116:33824) 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb37fe640 1 -- 172.21.15.116:0/192548745 learned_addr learned my addr 172.21.15.116:0/192548745 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 1 --2- 172.21.15.116:0/192548745 >> v2:172.21.15.116:3300/0 conn(0x7efdb4134aa0 0x7efdb4136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 10 monclient(hunting): get_auth_request con 0x7efdb4134aa0 auth_method 0 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.852+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdabfff640 1 --2- 172.21.15.116:0/192548745 >> v2:172.21.15.144:3300/0 conn(0x7efdb409e340 0x7efdb409e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdabfff640 10 monclient(hunting): get_auth_request con 0x7efdb409e340 auth_method 0 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdabfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 10 monclient(hunting): handle_auth_done global_id 54216 payload 274 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 1 -- 172.21.15.116:0/192548745 >> v2:172.21.15.144:3300/0 conn(0x7efdb409e340 msgr2=0x7efdb409e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 1 --2- 172.21.15.116:0/192548745 >> v2:172.21.15.144:3300/0 conn(0x7efdb409e340 0x7efdb409e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 1 -- 172.21.15.116:0/192548745 >> v1:172.21.15.116:6789/0 conn(0x7efdb409c9c0 legacy=0x7efdb409d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 1 monclient: found mon.noname-a 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:36.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.853+0000 7efdb3fff640 1 -- 172.21.15.116:0/192548745 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb409a880 con 0x7efdb4134aa0 2024-09-18T12:28:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.855+0000 7efdb3fff640 1 -- 172.21.15.116:0/192548745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 msgr2=0x7efdb4136e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.855+0000 7efdb3fff640 1 --2- 172.21.15.116:0/192548745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4136e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efdb409b520 tx=0x7efdac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf69d95c7ed5fbc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 1 -- 172.21.15.116:0/192548745 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac007760 con 0x7efdb4134aa0 2024-09-18T12:28:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: dump: 2024-09-18T12:28:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 1 -- 172.21.15.116:0/192548745 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac02d050 con 0x7efdb4134aa0 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 5 monclient: authenticate success, global_id 54216 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 4 set_mon_vals no callback set 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 1 -- 172.21.15.116:0/192548745 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac007af0 con 0x7efdb4134aa0 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdb2ffd640 10 monclient: dump: 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.869 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 10 monclient: shutdown 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 -- 172.21.15.116:0/192548745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 msgr2=0x7efdb4136e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 --2- 172.21.15.116:0/192548745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4136e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efdb409b520 tx=0x7efdac009780 comp rx=0 tx=0).stop 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 -- 172.21.15.116:0/192548745 shutdown_connections 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 --2- 172.21.15.116:0/192548745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4136e80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 --2- 172.21.15.116:0/192548745 >> v2:172.21.15.144:3300/0 conn(0x7efdb409e340 0x7efdb409e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 -- 172.21.15.116:0/192548745 >> 172.21.15.116:0/192548745 conn(0x7efdb40a2e40 msgr2=0x7efdb40a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 -- 172.21.15.116:0/192548745 shutdown_connections 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.856+0000 7efdba100640 1 -- 172.21.15.116:0/192548745 wait complete. 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.857+0000 7efdba100640 10 monclient: build_initial_monmap 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.857+0000 7efdba100640 10 monclient: monmap: 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.857+0000 7efdba100640 1 Processor -- start 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.857+0000 7efdba100640 1 -- start start 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.857+0000 7efdba100640 10 monclient: init 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.857+0000 7efdba100640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb409c9c0 0x7efdb4068900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): picked mon.noname-b con 0x7efdb409c9c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdb409e340 0x7efdb4068e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): picked mon.noname-c con 0x7efdb409e340 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): picked mon.noname-a con 0x7efdb4134aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4136b40 con 0x7efdb4134aa0 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb411fc00 con 0x7efdb409c9c0 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb406a410 con 0x7efdb409e340 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdba100640 10 monclient(hunting): authenticate will time out at 944.180085s 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdabfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdb409e340 msgr2=0x7efdb4068e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdabfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdb409e340 0x7efdb4068e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4069380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42858/0 (socket says 172.21.15.116:42858) 2024-09-18T12:28:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 -- 172.21.15.116:0/4231278008 learned_addr learned my addr 172.21.15.116:0/4231278008 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): get_auth_request con 0x7efdb4134aa0 auth_method 0 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb37fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb409c9c0 0x7efdb4068900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb37fe640 10 monclient(hunting): get_auth_request con 0x7efdb409c9c0 auth_method 0 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient(hunting): handle_auth_done global_id 54219 payload 824 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdb409e340 msgr2=0x7efdb4068e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efdb409e340 0x7efdb4068e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb409c9c0 msgr2=0x7efdb4068900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb409c9c0 0x7efdb4068900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 monclient: found mon.noname-a 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb3fff640 1 -- 172.21.15.116:0/4231278008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac009a20 con 0x7efdb4134aa0 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.858+0000 7efdb37fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb409c9c0 0x7efdb4068900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdb3fff640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 msgr2=0x7efdb4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdb3fff640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4069380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efdac0097c0 tx=0x7efdac037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac031c10 con 0x7efdb4134aa0 2024-09-18T12:28:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: dump: 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdba100640 5 monclient: authenticate success, global_id 54219 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac040070 con 0x7efdb4134aa0 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac045480 con 0x7efdb4134aa0 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdba100640 10 monclient: _renew_subs 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdba100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb411f640 con 0x7efdb4134aa0 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdab7fe640 10 monclient: dump: 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.877 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdb27fc640 4 set_mon_vals no callback set 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.860+0000 7efdba100640 10 monclient: _renew_subs 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdba100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb4134600 con 0x7efdb4134aa0 2024-09-18T12:28:36.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7efdac042030 con 0x7efdb4134aa0 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdba100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdba100640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 0x7efdb406c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdba100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb4065e00 con 0x7efdb40698c0 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdab7fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd94038440 0x7efd9403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.861+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5146+0+0 (secure 0 0 0) 0x7efdac06d1b0 con 0x7efdb4134aa0 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 0x7efdb406c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient: get_auth_request con 0x7efdb40698c0 auth_method 0 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdabfff640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd94038440 0x7efd9403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdabfff640 10 monclient: get_auth_request con 0x7efd94038440 auth_method 0 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 10 monclient(hunting): handle_auth_done global_id 4921 payload 824 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.862+0000 7efdb37fe640 1 -- 172.21.15.116:0/4231278008 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdb411fd90 con 0x7efdb40698c0 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.864+0000 7efdb37fe640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 msgr2=0x7efdb406c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.864+0000 7efdb37fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 0x7efdb406c8c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efda400e740 tx=0x7efda4009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.864+0000 7efdabfff640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd94038440 0x7efd9403a8f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efd9c006fd0 tx=0x7efd9c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4005cb0 con 0x7efdb40698c0 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 10 monclient: dump: 2024-09-18T12:28:36.880 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efda4005e10 con 0x7efdb40698c0 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:36.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 msgr2=0x7efdb406c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 0x7efdb406c8c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efda400e740 tx=0x7efda4009040 comp rx=0 tx=0).stop 2024-09-18T12:28:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 msgr2=0x7efdb406c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:36.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.865+0000 7efdab7fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb40698c0 0x7efdb406c8c0 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efda400e740 tx=0x7efda4009040 comp rx=0 tx=0).stop 2024-09-18T12:28:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.870+0000 7efdba100640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:28:36.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdba100640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 0x7efdb406d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdba100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb404bf60 con 0x7efdb406ce00 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 0x7efdb406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient: get_auth_request con 0x7efdb406ce00 auth_method 0 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 10 monclient(hunting): handle_auth_done global_id 4924 payload 824 2024-09-18T12:28:36.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.871+0000 7efdb37fe640 1 -- 172.21.15.116:0/4231278008 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efdb4099ed0 con 0x7efdb406ce00 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdb37fe640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 msgr2=0x7efdb406d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdb37fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 0x7efdb406d200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efda4017000 tx=0x7efda401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4013e00 con 0x7efdb406ce00 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 10 monclient: dump: 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:36.885 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7efda400c610 con 0x7efdb406ce00 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 msgr2=0x7efdb406d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 0x7efdb406d200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efda4017000 tx=0x7efda401a000 comp rx=0 tx=0).stop 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 msgr2=0x7efdb406d200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.874+0000 7efdab7fe640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efdb406ce00 0x7efdb406d200 secure :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efda4017000 tx=0x7efda401a000 comp rx=0 tx=0).stop 2024-09-18T12:28:36.886 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 38, 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:28:36.887 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:28:36.888 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:28:36.889 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:28:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:28:36.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:28:36.894 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:28:36.895 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:36.897 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:28:36.898 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:28:36.899 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:28:36.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd94038440 msgr2=0x7efd9403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd94038440 0x7efd9403a8f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efd9c006fd0 tx=0x7efd9c008040 comp rx=0 tx=0).stop 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 10 monclient: shutdown 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 msgr2=0x7efdb4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 --2- 172.21.15.116:0/4231278008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdb4134aa0 0x7efdb4069380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efdac0097c0 tx=0x7efdac037000 comp rx=0 tx=0).stop 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdb3fff640 1 -- 172.21.15.116:0/4231278008 reap_dead start 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdb3fff640 1 -- 172.21.15.116:0/4231278008 reap_dead start 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 shutdown_connections 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 >> 172.21.15.116:0/4231278008 conn(0x7efdb40a2e40 msgr2=0x7efdb40a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 shutdown_connections 2024-09-18T12:28:36.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:36.875+0000 7efdba100640 1 -- 172.21.15.116:0/4231278008 wait complete. 2024-09-18T12:28:36.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:28:36.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:28:36.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:36.906+0000 7fefbb7fe640 10 monclient: tick 2024-09-18T12:28:36.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:36.906+0000 7fefbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:37.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.376+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v62) v1 ==== 110+0+0 (secure 0 0 0) 0x7fefd412f7d0 con 0x7fefd4099250 2024-09-18T12:28:37.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.376+0000 7fefd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T12:28:37.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.376+0000 7fefd0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:37.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.474+0000 7fefdb11d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T12:28:37.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.474+0000 7fefdb11d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T12:28:37.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.474+0000 7fefdb11d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.474+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7fefd412f7d0 con 0x7fefd4099250 2024-09-18T12:28:37.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.475+0000 7fefd0ff9640 1 -- 172.21.15.144:0/502106774 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v62) v1 ==== 139+0+0 (secure 0 0 0) 0x7fefd412f7d0 con 0x7fefd4099250 2024-09-18T12:28:37.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.475+0000 7fefd0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T12:28:37.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.475+0000 7fefd0ff9640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-18T12:28:37.477 INFO:tasks.workunit.client.0.smithi144.stderr:already renamed to 'bar-asdf' 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefc0038440 msgr2=0x7fefc003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefc0038440 0x7fefc003a8f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fefc80038f0 tx=0x7fefc80090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 10 monclient: shutdown 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 msgr2=0x7fefd4136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 0x7fefd4136fd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009880 tx=0x7fefc4032000 comp rx=0 tx=0).stop 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 shutdown_connections 2024-09-18T12:28:37.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefc0038440 0x7fefc003a8f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefd40a19a0 0x7fefd4117a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd409be20 0x7fefd4117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.477+0000 7fefdb11d640 1 --2- 172.21.15.144:0/502106774 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefd4099250 0x7fefd4136fd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.478+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 >> 172.21.15.144:0/502106774 conn(0x7fefd40711a0 msgr2=0x7fefd4137c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:37.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.478+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 shutdown_connections 2024-09-18T12:28:37.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.478+0000 7fefdb11d640 1 -- 172.21.15.144:0/502106774 wait complete. 2024-09-18T12:28:37.488 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls 2024-09-18T12:28:37.489 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo-asdf 2024-09-18T12:28:37.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.560+0000 7fa466e9f640 10 monclient: get_monmap_and_config 2024-09-18T12:28:37.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.560+0000 7fa466e9f640 10 monclient: build_initial_monmap 2024-09-18T12:28:37.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.560+0000 7fa466e9f640 10 monclient: monmap: 2024-09-18T12:28:37.561 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:37.561 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.561 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:37.561130+0000 2024-09-18T12:28:37.561 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:37.561130+0000 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:37.562 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.560+0000 7fa466e9f640 1 Processor -- start 2024-09-18T12:28:37.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.560+0000 7fa466e9f640 1 -- start start 2024-09-18T12:28:37.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.560+0000 7fa466e9f640 10 monclient: init 2024-09-18T12:28:37.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:37.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:28:37.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4600677e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa460094080 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa460094b90 0x7fa460137d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient(hunting): picked mon.noname-c con 0x7fa460094b90 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46009d570 con 0x7fa4600677e0 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa460095f10 con 0x7fa460094080 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4600692c0 con 0x7fa460094b90 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa466e9f640 10 monclient(hunting): authenticate will time out at 953.130507s 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa460094b90 0x7fa460137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa45ffff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa460094080 0x7fa460094480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48804/0 (socket says 172.21.15.144:48804) 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa45ffff640 1 -- 172.21.15.144:0/1787984837 learned_addr learned my addr 172.21.15.144:0/1787984837 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:37.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): get_auth_request con 0x7fa460094b90 auth_method 0 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient(hunting): handle_auth_done global_id 4927 payload 274 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 1 -- 172.21.15.144:0/1787984837 >> v1:172.21.15.144:6789/0 conn(0x7fa460094080 legacy=0x7fa460094480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 1 -- 172.21.15.144:0/1787984837 >> v1:172.21.15.116:6789/0 conn(0x7fa4600677e0 legacy=0x7fa460095800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 1 monclient: found mon.noname-c 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.561+0000 7fa46543c640 1 -- 172.21.15.144:0/1787984837 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46009b560 con 0x7fa460094b90 2024-09-18T12:28:37.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa46543c640 1 -- 172.21.15.144:0/1787984837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa460094b90 msgr2=0x7fa460137d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa46543c640 1 --2- 172.21.15.144:0/1787984837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa460094b90 0x7fa460137d60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa458009e60 tx=0x7fa458009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3d82aa77a52c02c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 1 -- 172.21.15.144:0/1787984837 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa458007840 con 0x7fa460094b90 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: dump: 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.571 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa466e9f640 5 monclient: authenticate success, global_id 4927 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 1 -- 172.21.15.144:0/1787984837 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45802d050 con 0x7fa460094b90 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa466e9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa466e9f640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa466e9f640 4 set_mon_vals no callback set 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 1 -- 172.21.15.144:0/1787984837 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa458007bd0 con 0x7fa460094b90 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:37.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.564+0000 7fa45f7fe640 10 monclient: dump: 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 10 monclient: shutdown 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:37.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 1 -- 172.21.15.144:0/1787984837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa460094b90 msgr2=0x7fa460137d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 1 --2- 172.21.15.144:0/1787984837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa460094b90 0x7fa460137d60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa458009e60 tx=0x7fa458009780 comp rx=0 tx=0).stop 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 1 -- 172.21.15.144:0/1787984837 shutdown_connections 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 1 --2- 172.21.15.144:0/1787984837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa460094b90 0x7fa460137d60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 1 -- 172.21.15.144:0/1787984837 >> 172.21.15.144:0/1787984837 conn(0x7fa4600a16a0 msgr2=0x7fa4600a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 1 -- 172.21.15.144:0/1787984837 shutdown_connections 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 1 -- 172.21.15.144:0/1787984837 wait complete. 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 10 monclient: build_initial_monmap 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.565+0000 7fa466e9f640 10 monclient: monmap: 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:37.574 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 Processor -- start 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 -- start start 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient: init 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 0x7fa460091830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): picked mon.noname-b con 0x7fa4600677e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa460094080 0x7fa46008f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): picked mon.noname-a con 0x7fa460094080 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa460094b90 0x7fa46008fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): picked mon.noname-c con 0x7fa460094b90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460090930 con 0x7fa460094080 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460095090 con 0x7fa4600677e0 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa460098050 con 0x7fa460094b90 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa466e9f640 10 monclient(hunting): authenticate will time out at 953.135946s 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.566+0000 7fa464c3b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 0x7fa460091830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 0x7fa460091830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40688/0 (socket says 172.21.15.144:40688) 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa46543c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa460094b90 msgr2=0x7fa46008fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 -- 172.21.15.144:0/2759872432 learned_addr learned my addr 172.21.15.144:0/2759872432 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:37.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): get_auth_request con 0x7fa4600677e0 auth_method 0 2024-09-18T12:28:37.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:37.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:37.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:37.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa46543c640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa460094b90 0x7fa46008fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa460094080 0x7fa46008f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 10 monclient(hunting): get_auth_request con 0x7fa460094080 auth_method 0 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa45ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient(hunting): handle_auth_done global_id 4930 payload 824 2024-09-18T12:28:37.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 -- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa460094b90 msgr2=0x7fa46008fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa460094b90 0x7fa46008fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 -- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa460094080 msgr2=0x7fa46008f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa460094080 0x7fa46008f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient: _finish_hunting 0 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 monclient: found mon.noname-b 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.567+0000 7fa464c3b640 1 -- 172.21.15.144:0/2759872432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa458009a20 con 0x7fa4600677e0 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.569+0000 7fa464c3b640 1 -- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 msgr2=0x7fa460091830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.569+0000 7fa464c3b640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 0x7fa460091830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c5e0 tx=0x7fa454007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:37.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.569+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:28:37.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454012840 con 0x7fa4600677e0 2024-09-18T12:28:37.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:37.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: dump: 2024-09-18T12:28:37.581 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.581 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.581 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: _finish_auth 0 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4540129c0 con 0x7fa4600677e0 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa466e9f640 5 monclient: authenticate success, global_id 4930 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa466e9f640 10 monclient: _renew_subs 2024-09-18T12:28:37.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa466e9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa460091ee0 con 0x7fa4600677e0 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454012d80 con 0x7fa4600677e0 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45cff9640 10 monclient: dump: 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.583 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa45effd640 4 set_mon_vals no callback set 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa466e9f640 10 monclient: _renew_subs 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa466e9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.571+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4600934b0 con 0x7fa4600677e0 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa454011590 con 0x7fa4600677e0 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa45cff9640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa444038490 0x7fa44403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5151+0+0 (secure 0 0 0) 0x7fa45404ba90 con 0x7fa4600677e0 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa466e9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa466e9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa466e9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa45ffff640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa444038490 0x7fa44403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa460092300 con 0x7fa4600677e0 2024-09-18T12:28:37.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.572+0000 7fa45ffff640 10 monclient: get_auth_request con 0x7fa444038490 auth_method 0 2024-09-18T12:28:37.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.575+0000 7fa45ffff640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa444038490 0x7fa44403a940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa46008b7e0 tx=0x7fa44c007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:37.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.577+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa460092300 con 0x7fa4600677e0 2024-09-18T12:28:37.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.577+0000 7fa45cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:37.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.577+0000 7fa45cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:37.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.734+0000 7fa466e9f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:37.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.734+0000 7fa466e9f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:37.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.734+0000 7fa466e9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.734+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fa460092300 con 0x7fa4600677e0 2024-09-18T12:28:37.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.734+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+70 (secure 0 0 0) 0x7fa460092300 con 0x7fa4600677e0 2024-09-18T12:28:37.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.734+0000 7fa45cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:37.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.734+0000 7fa45cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:37.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.830+0000 7fa466e9f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.830+0000 7fa466e9f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.830+0000 7fa466e9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.830+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fa460092300 con 0x7fa4600677e0 2024-09-18T12:28:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.830+0000 7fa45cff9640 1 -- 172.21.15.144:0/2759872432 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+70 (secure 0 0 0) 0x7fa460092300 con 0x7fa4600677e0 2024-09-18T12:28:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.830+0000 7fa45cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:37.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.830+0000 7fa45cff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:28:37.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa444038490 msgr2=0x7fa44403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa444038490 0x7fa44403a940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa46008b7e0 tx=0x7fa44c007f00 comp rx=0 tx=0).stop 2024-09-18T12:28:37.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 10 monclient: shutdown 2024-09-18T12:28:37.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 msgr2=0x7fa460091830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 0x7fa460091830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c5e0 tx=0x7fa454007ed0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 shutdown_connections 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa444038490 0x7fa44403a940 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa460094b90 0x7fa46008fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa460094080 0x7fa46008f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 --2- 172.21.15.144:0/2759872432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4600677e0 0x7fa460091830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 >> 172.21.15.144:0/2759872432 conn(0x7fa4600a16a0 msgr2=0x7fa46008e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:37.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 shutdown_connections 2024-09-18T12:28:37.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.832+0000 7fa466e9f640 1 -- 172.21.15.144:0/2759872432 wait complete. 2024-09-18T12:28:37.842 INFO:tasks.workunit.client.0.smithi144.stdout:foo-asdf 2024-09-18T12:28:37.842 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls 2024-09-18T12:28:37.843 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep bar-asdf 2024-09-18T12:28:37.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.914+0000 7f2d318c0640 10 monclient: get_monmap_and_config 2024-09-18T12:28:37.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.914+0000 7f2d318c0640 10 monclient: build_initial_monmap 2024-09-18T12:28:37.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.914+0000 7f2d318c0640 10 monclient: monmap: 2024-09-18T12:28:37.915 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:37.915 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.915 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:37.915124+0000 2024-09-18T12:28:37.915 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:37.915124+0000 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:37.916 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.914+0000 7f2d318c0640 1 Processor -- start 2024-09-18T12:28:37.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.914+0000 7f2d318c0640 1 -- start start 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.914+0000 7f2d318c0640 10 monclient: init 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d2c09ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d2c09f020 0x7f2d2c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient(hunting): picked mon.noname-c con 0x7f2d2c09f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d2c136d60 0x7f2d2c139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2c136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:37.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c067420 con 0x7f2d2c09ec50 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0a0480 con 0x7f2d2c136d60 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0a0200 con 0x7f2d2c09f020 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d318c0640 10 monclient(hunting): authenticate will time out at 953.484407s 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d2c09f020 0x7f2d2c069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d2c09f020 0x7f2d2c069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40704/0 (socket says 172.21.15.144:40704) 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2affd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f2d2c09ec50 0x7f2d2c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48806/0 (socket says 172.21.15.144:48806) 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 -- 172.21.15.144:0/3979365264 learned_addr learned my addr 172.21.15.144:0/3979365264 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): get_auth_request con 0x7f2d2c09f020 auth_method 0 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:37.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient(hunting): handle_auth_done global_id 4933 payload 274 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 -- 172.21.15.144:0/3979365264 >> v2:172.21.15.116:3300/0 conn(0x7f2d2c136d60 msgr2=0x7f2d2c139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 --2- 172.21.15.144:0/3979365264 >> v2:172.21.15.116:3300/0 conn(0x7f2d2c136d60 0x7f2d2c139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 -- 172.21.15.144:0/3979365264 >> v1:172.21.15.144:6789/0 conn(0x7f2d2c09ec50 legacy=0x7f2d2c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:28:37.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 monclient: found mon.noname-c 2024-09-18T12:28:37.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:37.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.915+0000 7f2d2a7fc640 1 -- 172.21.15.144:0/3979365264 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c09fd80 con 0x7f2d2c09f020 2024-09-18T12:28:37.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d2a7fc640 1 -- 172.21.15.144:0/3979365264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c09f020 msgr2=0x7f2d2c069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:37.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d2a7fc640 1 --2- 172.21.15.144:0/3979365264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c09f020 0x7f2d2c069d10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c06a250 tx=0x7f2d200091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7790fe3241ff14fd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 1 -- 172.21.15.144:0/3979365264 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20032440 con 0x7f2d2c09f020 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: dump: 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.928 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d318c0640 5 monclient: authenticate success, global_id 4933 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d318c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 1 -- 172.21.15.144:0/3979365264 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d200325a0 con 0x7f2d2c09f020 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d318c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d318c0640 4 set_mon_vals no callback set 2024-09-18T12:28:37.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 1 -- 172.21.15.144:0/3979365264 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20032980 con 0x7f2d2c09f020 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.918+0000 7f2d29ffb640 10 monclient: dump: 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.930 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 10 monclient: shutdown 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 -- 172.21.15.144:0/3979365264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c09f020 msgr2=0x7f2d2c069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3979365264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c09f020 0x7f2d2c069d10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c06a250 tx=0x7f2d200091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 -- 172.21.15.144:0/3979365264 shutdown_connections 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3979365264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c09f020 0x7f2d2c069d10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3979365264 >> v2:172.21.15.116:3300/0 conn(0x7f2d2c136d60 0x7f2d2c139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 -- 172.21.15.144:0/3979365264 >> 172.21.15.144:0/3979365264 conn(0x7f2d2c0a5cf0 msgr2=0x7f2d2c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 -- 172.21.15.144:0/3979365264 shutdown_connections 2024-09-18T12:28:37.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.919+0000 7f2d318c0640 1 -- 172.21.15.144:0/3979365264 wait complete. 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient: build_initial_monmap 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient: monmap: 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 Processor -- start 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 -- start start 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient: init 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c09ec50 0x7f2d2c1418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2c09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2c09f020 0x7f2d2c0986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): picked mon.noname-c con 0x7f2d2c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 0x7f2d2c098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): picked mon.noname-b con 0x7f2d2c136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0a45a0 con 0x7f2d2c09ec50 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c11c020 con 0x7f2d2c136d60 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c11bd40 con 0x7f2d2c09f020 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d318c0640 10 monclient(hunting): authenticate will time out at 953.489909s 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.920+0000 7f2d2a7fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2c09f020 msgr2=0x7f2d2c0986d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2a7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2c09f020 0x7f2d2c0986d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 0x7f2d2c098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 0x7f2d2c098c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40718/0 (socket says 172.21.15.144:40718) 2024-09-18T12:28:37.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 -- 172.21.15.144:0/3772969300 learned_addr learned my addr 172.21.15.144:0/3772969300 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): get_auth_request con 0x7f2d2c136d60 auth_method 0 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2affd640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c09ec50 0x7f2d2c1418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2affd640 10 monclient(hunting): get_auth_request con 0x7f2d2c09ec50 auth_method 0 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:37.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient(hunting): handle_auth_done global_id 4936 payload 824 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 -- 172.21.15.144:0/3772969300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2c09f020 msgr2=0x7f2d2c0986d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2c09f020 0x7f2d2c0986d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 -- 172.21.15.144:0/3772969300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c09ec50 msgr2=0x7f2d2c1418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c09ec50 0x7f2d2c1418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.921+0000 7f2d2b7fe640 1 -- 172.21.15.144:0/3772969300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d20009450 con 0x7f2d2c136d60 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d2b7fe640 1 -- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 msgr2=0x7f2d2c098c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d2b7fe640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 0x7f2d2c098c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00bb60 tx=0x7f2d1c009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00cc20 con 0x7f2d2c136d60 2024-09-18T12:28:37.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: dump: 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.937 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c00cda0 con 0x7f2d2c136d60 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d318c0640 5 monclient: authenticate success, global_id 4936 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d318c0640 10 monclient: _renew_subs 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d318c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c11b760 con 0x7f2d2c136d60 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:37.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c014660 con 0x7f2d2c136d60 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d12ffd640 10 monclient: dump: 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:37.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d318c0640 10 monclient: _renew_subs 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d318c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c09a200 con 0x7f2d2c136d60 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.923+0000 7f2d297fa640 4 set_mon_vals no callback set 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2d1c014ad0 con 0x7f2d2c136d60 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d318c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d318c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d318c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d12ffd640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d18038440 0x7f2d1803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c0008d0 con 0x7f2d2c136d60 2024-09-18T12:28:37.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f2d1c04ba40 con 0x7f2d2c136d60 2024-09-18T12:28:37.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d2affd640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d18038440 0x7f2d1803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:37.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.924+0000 7f2d2affd640 10 monclient: get_auth_request con 0x7f2d18038440 auth_method 0 2024-09-18T12:28:37.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.927+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:28:37.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.927+0000 7f2d2affd640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d18038440 0x7f2d1803a8f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2d24007110 tx=0x7f2d24009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:37.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.929+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2d1c01bd00 con 0x7f2d2c136d60 2024-09-18T12:28:37.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.929+0000 7f2d12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:37.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:37.929+0000 7f2d12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:38.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.083+0000 7f2d318c0640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:38.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.083+0000 7f2d318c0640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:38.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.083+0000 7f2d318c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:38.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.083+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f2d2c11c160 con 0x7f2d2c136d60 2024-09-18T12:28:38.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.084+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+70 (secure 0 0 0) 0x7f2d1c01b310 con 0x7f2d2c136d60 2024-09-18T12:28:38.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.084+0000 7f2d12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:38.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.084+0000 7f2d12ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:38.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.177+0000 7f2d318c0640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:38.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.177+0000 7f2d318c0640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:38.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.177+0000 7f2d318c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:38.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.177+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f2d2c0a13e0 con 0x7f2d2c136d60 2024-09-18T12:28:38.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.177+0000 7f2d12ffd640 1 -- 172.21.15.144:0/3772969300 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+70 (secure 0 0 0) 0x7f2d1c01b960 con 0x7f2d2c136d60 2024-09-18T12:28:38.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.177+0000 7f2d12ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:38.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.177+0000 7f2d12ffd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:28:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d18038440 msgr2=0x7f2d1803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d18038440 0x7f2d1803a8f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2d24007110 tx=0x7f2d24009040 comp rx=0 tx=0).stop 2024-09-18T12:28:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 10 monclient: shutdown 2024-09-18T12:28:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 msgr2=0x7f2d2c098c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:38.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 0x7f2d2c098c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00bb60 tx=0x7f2d1c009810 comp rx=0 tx=0).stop 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 shutdown_connections 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d18038440 0x7f2d1803a8f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2c136d60 0x7f2d2c098c10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d2c09f020 0x7f2d2c0986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 --2- 172.21.15.144:0/3772969300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d2c09ec50 0x7f2d2c1418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 >> 172.21.15.144:0/3772969300 conn(0x7f2d2c0a5cf0 msgr2=0x7f2d2c130ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 shutdown_connections 2024-09-18T12:28:38.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.179+0000 7f2d318c0640 1 -- 172.21.15.144:0/3772969300 wait complete. 2024-09-18T12:28:38.189 INFO:tasks.workunit.client.0.smithi144.stdout:bar-asdf 2024-09-18T12:28:38.190 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rm foo 2024-09-18T12:28:38.190 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'does not exist' 2024-09-18T12:28:38.550 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.440+0000 7f5feffff640 1 -- 172.21.15.144:0/1810656880 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v62) v1 ==== 103+0+0 (secure 0 0 0) 0x7f5ffc020ae0 con 0x7f6008131c90 2024-09-18T12:28:38.550 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.441+0000 7f5feffff640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-18T12:28:38.550 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.538+0000 7f5feffff640 1 -- 172.21.15.144:0/1810656880 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v62) v1 ==== 103+0+0 (secure 0 0 0) 0x7f5ffc020300 con 0x7f6008131c90 2024-09-18T12:28:38.550 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.538+0000 7f5feffff640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-18T12:28:38.550 INFO:tasks.workunit.client.0.smithi144.stdout:rule foo does not exist 2024-09-18T12:28:38.550 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rm bar 2024-09-18T12:28:38.550 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'does not exist' 2024-09-18T12:28:38.903 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.795+0000 7f97bffff640 1 -- 172.21.15.144:0/3262264643 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v62) v1 ==== 103+0+0 (secure 0 0 0) 0x7f97cc010a10 con 0x7f97d8099580 2024-09-18T12:28:38.903 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.795+0000 7f97bffff640 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-18T12:28:38.903 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.891+0000 7f97bffff640 1 -- 172.21.15.144:0/3262264643 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v62) v1 ==== 103+0+0 (secure 0 0 0) 0x7f97cc010c00 con 0x7f97d8099580 2024-09-18T12:28:38.904 INFO:tasks.workunit.client.0.smithi144.stdout:2024-09-18T12:28:38.891+0000 7f97bffff640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-18T12:28:38.904 INFO:tasks.workunit.client.0.smithi144.stdout:rule bar does not exist 2024-09-18T12:28:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-18T12:28:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.977+0000 7f2411d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.977+0000 7f2411d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.977+0000 7f2411d1d640 10 monclient: monmap: 2024-09-18T12:28:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:38.978064+0000 2024-09-18T12:28:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:38.978064+0000 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:38.979 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:38.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.977+0000 7f2411d1d640 1 Processor -- start 2024-09-18T12:28:38.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.977+0000 7f2411d1d640 1 -- start start 2024-09-18T12:28:38.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.977+0000 7f2411d1d640 10 monclient: init 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f240c131bd0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f240c069e10 0x7f240c131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f240c069e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f240c136d60 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0a06d0 con 0x7f240c131bd0 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c071780 con 0x7f240c136d60 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c06a210 con 0x7f240c069e10 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2411d1d640 10 monclient(hunting): authenticate will time out at 954.547524s 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f240c069e10 0x7f240c131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:38.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f240bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f240c136d60 0x7f240c139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48820/0 (socket says 172.21.15.144:48820) 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f240c069e10 0x7f240c131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40772/0 (socket says 172.21.15.144:40772) 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f240bfff640 1 -- 172.21.15.144:0/2235880885 learned_addr learned my addr 172.21.15.144:0/2235880885 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): get_auth_request con 0x7f240c069e10 auth_method 0 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient(hunting): handle_auth_done global_id 4951 payload 274 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 1 -- 172.21.15.144:0/2235880885 >> v1:172.21.15.144:6789/0 conn(0x7f240c136d60 legacy=0x7f240c139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 1 -- 172.21.15.144:0/2235880885 >> v1:172.21.15.116:6789/0 conn(0x7f240c131bd0 legacy=0x7f240c09ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 1 monclient: found mon.noname-c 2024-09-18T12:28:38.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:38.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:38.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.978+0000 7f2402ffd640 1 -- 172.21.15.144:0/2235880885 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0a0010 con 0x7f240c069e10 2024-09-18T12:28:38.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.981+0000 7f2402ffd640 1 -- 172.21.15.144:0/2235880885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c069e10 msgr2=0x7f240c131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:38.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.981+0000 7f2402ffd640 1 --2- 172.21.15.144:0/2235880885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c069e10 0x7f240c131fa0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009850 tx=0x7f23f40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21260d368dc35a8e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:38.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 1 -- 172.21.15.144:0/2235880885 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4032440 con 0x7f240c069e10 2024-09-18T12:28:38.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:38.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: dump: 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: _finish_auth 0 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 1 -- 172.21.15.144:0/2235880885 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f40325a0 con 0x7f240c069e10 2024-09-18T12:28:38.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 5 monclient: authenticate success, global_id 4951 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 1 -- 172.21.15.144:0/2235880885 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4032940 con 0x7f240c069e10 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 4 set_mon_vals no callback set 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f240affd640 10 monclient: dump: 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:38.991 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 10 monclient: shutdown 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:38.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 1 -- 172.21.15.144:0/2235880885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c069e10 msgr2=0x7f240c131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2235880885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c069e10 0x7f240c131fa0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009850 tx=0x7f23f40090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 1 -- 172.21.15.144:0/2235880885 shutdown_connections 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2235880885 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c069e10 0x7f240c131fa0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 1 -- 172.21.15.144:0/2235880885 >> 172.21.15.144:0/2235880885 conn(0x7f240c0713b0 msgr2=0x7f240c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 1 -- 172.21.15.144:0/2235880885 shutdown_connections 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.982+0000 7f2411d1d640 1 -- 172.21.15.144:0/2235880885 wait complete. 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.983+0000 7f2411d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.983+0000 7f2411d1d640 10 monclient: monmap: 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:38.993 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.983+0000 7f2411d1d640 1 Processor -- start 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.983+0000 7f2411d1d640 1 -- start start 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.983+0000 7f2411d1d640 10 monclient: init 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f240c069e10 0x7f240c141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f240c069e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 0x7f240c130b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:38.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f240c131bd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c136d60 0x7f240c1310b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f240c136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c09bec0 con 0x7f240c136d60 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0a4b60 con 0x7f240c131bd0 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c11cf60 con 0x7f240c069e10 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2411d1d640 10 monclient(hunting): authenticate will time out at 954.553182s 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 0x7f240c130b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 0x7f240c130b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40774/0 (socket says 172.21.15.144:40774) 2024-09-18T12:28:38.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 -- 172.21.15.144:0/2552246727 learned_addr learned my addr 172.21.15.144:0/2552246727 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): get_auth_request con 0x7f240c131bd0 auth_method 0 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f240b7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f240c069e10 msgr2=0x7f240c141240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f240b7fe640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f240c069e10 0x7f240c141240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f240bfff640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c136d60 0x7f240c1310b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient(hunting): handle_auth_done global_id 4954 payload 824 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 -- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f240c069e10 msgr2=0x7f240c141240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:38.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f240c069e10 0x7f240c141240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 -- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c136d60 msgr2=0x7f240c1310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c136d60 0x7f240c1310b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 monclient: found mon.noname-b 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.984+0000 7f2402ffd640 1 -- 172.21.15.144:0/2552246727 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4009390 con 0x7f240c131bd0 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2402ffd640 1 -- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 msgr2=0x7f240c130b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2402ffd640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 0x7f240c130b70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009850 tx=0x7f23f4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4031a70 con 0x7f240c131bd0 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:38.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: dump: 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: _finish_auth 0 2024-09-18T12:28:38.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2411d1d640 5 monclient: authenticate success, global_id 4954 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f4031bd0 con 0x7f240c131bd0 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2411d1d640 10 monclient: _renew_subs 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2411d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c09b520 con 0x7f240c131bd0 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4044410 con 0x7f240c131bd0 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2403fff640 10 monclient: dump: 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:38.999 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2411d1d640 10 monclient: _renew_subs 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f240a7fc640 4 set_mon_vals no callback set 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2411d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.986+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c141780 con 0x7f240c131bd0 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.987+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f23f4039040 con 0x7f240c131bd0 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.987+0000 7f2403fff640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f23ec038440 0x7f23ec03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.987+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f23f406b810 con 0x7f240c131bd0 2024-09-18T12:28:39.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.987+0000 7f2411d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.988+0000 7f2411d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.988+0000 7f2411d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.988+0000 7f240b7fe640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f23ec038440 0x7f23ec03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.988+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c1369d0 con 0x7f240c131bd0 2024-09-18T12:28:39.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.988+0000 7f240b7fe640 10 monclient: get_auth_request con 0x7f23ec038440 auth_method 0 2024-09-18T12:28:39.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.990+0000 7f240b7fe640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f23ec038440 0x7f23ec03a8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f23fc007fd0 tx=0x7f23fc007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:39.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.992+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f240c1369d0 con 0x7f240c131bd0 2024-09-18T12:28:39.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.992+0000 7f2403fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:38.992+0000 7f2403fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:39.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.148+0000 7f2411d1d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.148+0000 7f2411d1d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.148+0000 7f2411d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.148+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f240c1369d0 con 0x7f240c131bd0 2024-09-18T12:28:39.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.381+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v63) v1 ==== 110+0+0 (secure 0 0 0) 0x7f240c1369d0 con 0x7f240c131bd0 2024-09-18T12:28:39.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.381+0000 7f2403fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.381+0000 7f2403fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:39.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.476+0000 7f2411d1d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.476+0000 7f2411d1d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.476+0000 7f2411d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.476+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f240c1369d0 con 0x7f240c131bd0 2024-09-18T12:28:39.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.478+0000 7f2403fff640 1 -- 172.21.15.144:0/2552246727 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v63) v1 ==== 134+0+0 (secure 0 0 0) 0x7f240c1369d0 con 0x7f240c131bd0 2024-09-18T12:28:39.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.478+0000 7f2403fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.478+0000 7f2403fff640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-18T12:28:39.479 INFO:tasks.workunit.client.0.smithi144.stderr:already renamed to 'foo' 2024-09-18T12:28:39.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f23ec038440 msgr2=0x7f23ec03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f23ec038440 0x7f23ec03a8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f23fc007fd0 tx=0x7f23fc007a70 comp rx=0 tx=0).stop 2024-09-18T12:28:39.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 10 monclient: shutdown 2024-09-18T12:28:39.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:39.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 msgr2=0x7f240c130b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 0x7f240c130b70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009850 tx=0x7f23f4032000 comp rx=0 tx=0).stop 2024-09-18T12:28:39.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 shutdown_connections 2024-09-18T12:28:39.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f23ec038440 0x7f23ec03a8f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c136d60 0x7f240c1310b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c131bd0 0x7f240c130b70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 --2- 172.21.15.144:0/2552246727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f240c069e10 0x7f240c141240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.479+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 >> 172.21.15.144:0/2552246727 conn(0x7f240c0713b0 msgr2=0x7f240c09cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:39.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.480+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 shutdown_connections 2024-09-18T12:28:39.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.480+0000 7f2411d1d640 1 -- 172.21.15.144:0/2552246727 wait complete. 2024-09-18T12:28:39.489 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-18T12:28:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.563+0000 7f7b7159d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.563+0000 7f7b7159d640 10 monclient: build_initial_monmap 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.563+0000 7f7b7159d640 10 monclient: monmap: 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:39.564534+0000 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:39.564534+0000 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:39.565 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:39.566 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 1 Processor -- start 2024-09-18T12:28:39.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 1 -- start start 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient: init 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b6c131bd0 0x7f7b6c131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c131bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b6c09ebe0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b6c09efe0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0a06d0 con 0x7f7b6c09efe0 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c071780 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6c0a0010 con 0x7f7b6c131bd0 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b7159d640 10 monclient(hunting): authenticate will time out at 955.133780s 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b6affd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b6c131bd0 0x7f7b6c131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b6a7fc640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7b6c09ebe0 0x7f7b6c068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48830/0 (socket says 172.21.15.144:48830) 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/3347020329 learned_addr learned my addr 172.21.15.144:0/3347020329 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b6affd640 10 monclient(hunting): get_auth_request con 0x7f7b6c131bd0 auth_method 0 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b6affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.564+0000 7f7b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:39.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 10 monclient(hunting): handle_auth_done global_id 4957 payload 274 2024-09-18T12:28:39.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 1 -- 172.21.15.144:0/3347020329 >> v1:172.21.15.144:6789/0 conn(0x7f7b6c09ebe0 legacy=0x7f7b6c068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:39.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 1 -- 172.21.15.144:0/3347020329 >> v1:172.21.15.116:6789/0 conn(0x7f7b6c09efe0 legacy=0x7f7b6c069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:39.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:39.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 1 monclient: found mon.noname-c 2024-09-18T12:28:39.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:39.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.565+0000 7f7b6affd640 1 -- 172.21.15.144:0/3347020329 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c0a1910 con 0x7f7b6c131bd0 2024-09-18T12:28:39.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.567+0000 7f7b6affd640 1 -- 172.21.15.144:0/3347020329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c131bd0 msgr2=0x7f7b6c131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:39.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.567+0000 7f7b6affd640 1 --2- 172.21.15.144:0/3347020329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c131bd0 0x7f7b6c131fa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7b54009850 tx=0x7f7b540090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=388e1f409313c774 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:39.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 1 -- 172.21.15.144:0/3347020329 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54032440 con 0x7f7b6c131bd0 2024-09-18T12:28:39.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: dump: 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:39.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 1 -- 172.21.15.144:0/3347020329 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b540325a0 con 0x7f7b6c131bd0 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 1 -- 172.21.15.144:0/3347020329 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54032980 con 0x7f7b6c131bd0 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 5 monclient: authenticate success, global_id 4957 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 4 set_mon_vals no callback set 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b69ffb640 10 monclient: dump: 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.578 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 10 monclient: shutdown 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 1 -- 172.21.15.144:0/3347020329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c131bd0 msgr2=0x7f7b6c131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 1 --2- 172.21.15.144:0/3347020329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c131bd0 0x7f7b6c131fa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7b54009850 tx=0x7f7b540090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 1 -- 172.21.15.144:0/3347020329 shutdown_connections 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 1 --2- 172.21.15.144:0/3347020329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c131bd0 0x7f7b6c131fa0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 1 -- 172.21.15.144:0/3347020329 >> 172.21.15.144:0/3347020329 conn(0x7f7b6c0713b0 msgr2=0x7f7b6c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 1 -- 172.21.15.144:0/3347020329 shutdown_connections 2024-09-18T12:28:39.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.568+0000 7f7b7159d640 1 -- 172.21.15.144:0/3347020329 wait complete. 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.569+0000 7f7b7159d640 10 monclient: build_initial_monmap 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.569+0000 7f7b7159d640 10 monclient: monmap: 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.569+0000 7f7b7159d640 1 Processor -- start 2024-09-18T12:28:39.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.569+0000 7f7b7159d640 1 -- start start 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.569+0000 7f7b7159d640 10 monclient: init 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 0x7f7b6c09abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c09ebe0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b6c09efe0 0x7f7b6c130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c09efe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b6c131bd0 0x7f7b6c11b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c131bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c09bf30 con 0x7f7b6c09efe0 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c11e040 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c09e530 con 0x7f7b6c131bd0 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b7159d640 10 monclient(hunting): authenticate will time out at 955.139206s 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 0x7f7b6c09abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 0x7f7b6c09abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40794/0 (socket says 172.21.15.144:40794) 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 -- 172.21.15.144:0/541576423 learned_addr learned my addr 172.21.15.144:0/541576423 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): get_auth_request con 0x7f7b6c09ebe0 auth_method 0 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6b7fe640 1 -- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b6c131bd0 msgr2=0x7f7b6c11b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6b7fe640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b6c131bd0 0x7f7b6c11b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:39.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient(hunting): handle_auth_done global_id 4960 payload 824 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6a7fc640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b6c09efe0 0x7f7b6c130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 -- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b6c131bd0 msgr2=0x7f7b6c11b6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b6c131bd0 0x7f7b6c11b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 -- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b6c09efe0 msgr2=0x7f7b6c130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b6c09efe0 0x7f7b6c130c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 monclient: found mon.noname-b 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.570+0000 7f7b6affd640 1 -- 172.21.15.144:0/541576423 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b54009390 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b6affd640 1 -- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 msgr2=0x7f7b6c09abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b6affd640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 0x7f7b6c09abb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7b54009850 tx=0x7f7b54032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54031a70 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: dump: 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.584 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b54031bd0 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b7159d640 5 monclient: authenticate success, global_id 4960 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b7159d640 10 monclient: _renew_subs 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b54044410 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b7159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c09b7a0 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b62ffd640 10 monclient: dump: 2024-09-18T12:28:39.585 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b697fa640 4 set_mon_vals no callback set 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b7159d640 10 monclient: _renew_subs 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b7159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.572+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c11dd60 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.573+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7b54039040 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.573+0000 7f7b7159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.573+0000 7f7b7159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.573+0000 7f7b7159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.573+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c04bf60 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.573+0000 7f7b62ffd640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b4c038440 0x7f7b4c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.574+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f7b5406b800 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.574+0000 7f7b6a7fc640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b4c038440 0x7f7b4c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.574+0000 7f7b6a7fc640 10 monclient: get_auth_request con 0x7f7b4c038440 auth_method 0 2024-09-18T12:28:39.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.576+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:28:39.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.576+0000 7f7b6a7fc640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b4c038440 0x7f7b4c03a8f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c09b3a0 tx=0x7f7b5c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:39.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.578+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7b54039940 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.578+0000 7f7b62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.578+0000 7f7b62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:39.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.738+0000 7f7b7159d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.738+0000 7f7b7159d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.738+0000 7f7b7159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.738+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f7b6c04bf60 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.739+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v63) v1 ==== 134+0+0 (secure 0 0 0) 0x7f7b54037020 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.739+0000 7f7b62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.739+0000 7f7b62ffd640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-18T12:28:39.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.835+0000 7f7b7159d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.835+0000 7f7b7159d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.835+0000 7f7b7159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.835+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f7b6c04bf60 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.836+0000 7f7b62ffd640 1 -- 172.21.15.144:0/541576423 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v63) v1 ==== 134+0+0 (secure 0 0 0) 0x7f7b54043dd0 con 0x7f7b6c09ebe0 2024-09-18T12:28:39.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.837+0000 7f7b62ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T12:28:39.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.837+0000 7f7b62ffd640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-18T12:28:39.838 INFO:tasks.workunit.client.0.smithi144.stderr:already renamed to 'foo' 2024-09-18T12:28:39.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b4c038440 msgr2=0x7f7b4c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b4c038440 0x7f7b4c03a8f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c09b3a0 tx=0x7f7b5c009040 comp rx=0 tx=0).stop 2024-09-18T12:28:39.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 10 monclient: shutdown 2024-09-18T12:28:39.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 msgr2=0x7f7b6c09abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 0x7f7b6c09abb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7b54009850 tx=0x7f7b54032000 comp rx=0 tx=0).stop 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 shutdown_connections 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b4c038440 0x7f7b4c03a8f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b6c131bd0 0x7f7b6c11b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b6c09efe0 0x7f7b6c130c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 --2- 172.21.15.144:0/541576423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b6c09ebe0 0x7f7b6c09abb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 >> 172.21.15.144:0/541576423 conn(0x7f7b6c0713b0 msgr2=0x7f7b6c09cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 shutdown_connections 2024-09-18T12:28:39.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.839+0000 7f7b7159d640 1 -- 172.21.15.144:0/541576423 wait complete. 2024-09-18T12:28:39.849 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-18T12:28:39.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.923+0000 7fe16359d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:39.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.923+0000 7fe16359d640 10 monclient: build_initial_monmap 2024-09-18T12:28:39.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.923+0000 7fe16359d640 10 monclient: monmap: 2024-09-18T12:28:39.924 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:39.924 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.924 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:39.923965+0000 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:39.923965+0000 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:39.925 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.923+0000 7fe16359d640 1 Processor -- start 2024-09-18T12:28:39.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.923+0000 7fe16359d640 1 -- start start 2024-09-18T12:28:39.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.923+0000 7fe16359d640 10 monclient: init 2024-09-18T12:28:39.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:39.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe15c131bd0 0x7fe15c131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient(hunting): picked mon.noname-c con 0x7fe15c131bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe15c09ebe0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe15c09efe0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c0a06d0 con 0x7fe15c09efe0 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c071780 con 0x7fe15c09ebe0 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c0a0010 con 0x7fe15c131bd0 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe16359d640 10 monclient(hunting): authenticate will time out at 955.493415s 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe15c131bd0 0x7fe15c131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe15c131bd0 0x7fe15c131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40808/0 (socket says 172.21.15.144:40808) 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe160b38640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe15c09ebe0 0x7fe15c068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48842/0 (socket says 172.21.15.144:48842) 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 1 -- 172.21.15.144:0/1449110973 learned_addr learned my addr 172.21.15.144:0/1449110973 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): get_auth_request con 0x7fe15c131bd0 auth_method 0 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient(hunting): handle_auth_done global_id 4963 payload 274 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 1 -- 172.21.15.144:0/1449110973 >> v1:172.21.15.144:6789/0 conn(0x7fe15c09ebe0 legacy=0x7fe15c068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:39.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 1 -- 172.21.15.144:0/1449110973 >> v1:172.21.15.116:6789/0 conn(0x7fe15c09efe0 legacy=0x7fe15c069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:39.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient: _finish_hunting 0 2024-09-18T12:28:39.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 1 monclient: found mon.noname-c 2024-09-18T12:28:39.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:39.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.924+0000 7fe161339640 1 -- 172.21.15.144:0/1449110973 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0a1910 con 0x7fe15c131bd0 2024-09-18T12:28:39.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe161339640 1 -- 172.21.15.144:0/1449110973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 msgr2=0x7fe15c131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:39.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe161339640 1 --2- 172.21.15.144:0/1449110973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c131fa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe154009910 tx=0x7fe1540091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d16ce861446e5df server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:39.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 1 -- 172.21.15.144:0/1449110973 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154032440 con 0x7fe15c131bd0 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: dump: 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: _finish_auth 0 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe16359d640 5 monclient: authenticate success, global_id 4963 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 1 -- 172.21.15.144:0/1449110973 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1540325a0 con 0x7fe15c131bd0 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe16359d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe16359d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe16359d640 4 set_mon_vals no callback set 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 1 -- 172.21.15.144:0/1449110973 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154032980 con 0x7fe15c131bd0 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.927+0000 7fe15bfff640 10 monclient: dump: 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.937 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 10 monclient: shutdown 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 1 -- 172.21.15.144:0/1449110973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 msgr2=0x7fe15c131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 1 --2- 172.21.15.144:0/1449110973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c131fa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe154009910 tx=0x7fe1540091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 1 -- 172.21.15.144:0/1449110973 shutdown_connections 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 1 --2- 172.21.15.144:0/1449110973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c131fa0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 1 -- 172.21.15.144:0/1449110973 >> 172.21.15.144:0/1449110973 conn(0x7fe15c0713b0 msgr2=0x7fe15c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 1 -- 172.21.15.144:0/1449110973 shutdown_connections 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 1 -- 172.21.15.144:0/1449110973 wait complete. 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 10 monclient: build_initial_monmap 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.928+0000 7fe16359d640 10 monclient: monmap: 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:39.939 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 Processor -- start 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 -- start start 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient: init 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c09ebe0 0x7fe15c09add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): picked mon.noname-a con 0x7fe15c09ebe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe15c09efe0 0x7fe15c130d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): picked mon.noname-c con 0x7fe15c09efe0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c11b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): picked mon.noname-b con 0x7fe15c131bd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c09bec0 con 0x7fe15c09ebe0 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c09e550 con 0x7fe15c131bd0 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c09b820 con 0x7fe15c09efe0 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe16359d640 10 monclient(hunting): authenticate will time out at 955.498726s 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c11b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe160b38640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe15c09efe0 msgr2=0x7fe15c130d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c11b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40824/0 (socket says 172.21.15.144:40824) 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 1 -- 172.21.15.144:0/2874115999 learned_addr learned my addr 172.21.15.144:0/2874115999 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 10 monclient(hunting): get_auth_request con 0x7fe15c131bd0 auth_method 0 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:39.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe160b38640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe15c09efe0 0x7fe15c130d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161339640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c09ebe0 0x7fe15c09add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161339640 10 monclient(hunting): get_auth_request con 0x7fe15c09ebe0 auth_method 0 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.929+0000 7fe161339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 10 monclient(hunting): handle_auth_done global_id 4966 payload 824 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 1 -- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe15c09efe0 msgr2=0x7fe15c130d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe15c09efe0 0x7fe15c130d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 1 -- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c09ebe0 msgr2=0x7fe15c09add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:39.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c09ebe0 0x7fe15c09add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 1 monclient: found mon.noname-b 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161b3a640 1 -- 172.21.15.144:0/2874115999 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154009450 con 0x7fe15c131bd0 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.930+0000 7fe161339640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c09ebe0 0x7fe15c09add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.932+0000 7fe161b3a640 1 -- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 msgr2=0x7fe15c11b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.932+0000 7fe161b3a640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c11b6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00c5e0 tx=0x7fe14c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.932+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c012840 con 0x7fe15c131bd0 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:39.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: dump: 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: _finish_auth 0 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:39.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c0129a0 con 0x7fe15c131bd0 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe16359d640 5 monclient: authenticate success, global_id 4966 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe16359d640 10 monclient: _renew_subs 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe16359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c11deb0 con 0x7fe15c131bd0 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c012da0 con 0x7fe15c131bd0 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe1597fa640 10 monclient: dump: 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.946 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe15b7fe640 4 set_mon_vals no callback set 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe16359d640 10 monclient: _renew_subs 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe16359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.934+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c142090 con 0x7fe15c131bd0 2024-09-18T12:28:39.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe14c011590 con 0x7fe15c131bd0 2024-09-18T12:28:39.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe16359d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe16359d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe16359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:39.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c04bf60 con 0x7fe15c131bd0 2024-09-18T12:28:39.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe1597fa640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe148038490 0x7fe14803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5146+0+0 (secure 0 0 0) 0x7fe14c04bac0 con 0x7fe15c131bd0 2024-09-18T12:28:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe161339640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe148038490 0x7fe14803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.935+0000 7fe161339640 10 monclient: get_auth_request con 0x7fe148038490 auth_method 0 2024-09-18T12:28:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.938+0000 7fe161339640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe148038490 0x7fe14803a940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe154034000 tx=0x7fe154032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.940+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe14c0108c0 con 0x7fe15c131bd0 2024-09-18T12:28:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.940+0000 7fe1597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:39.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:39.940+0000 7fe1597fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:40.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.097+0000 7fe16359d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T12:28:40.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.097+0000 7fe16359d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T12:28:40.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.097+0000 7fe16359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.097+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7fe15c11e2d0 con 0x7fe15c131bd0 2024-09-18T12:28:40.389 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.387+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v64) v1 ==== 110+0+0 (secure 0 0 0) 0x7fe14c016070 con 0x7fe15c131bd0 2024-09-18T12:28:40.389 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.387+0000 7fe1597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T12:28:40.389 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.387+0000 7fe1597fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:40.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.482+0000 7fe16359d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T12:28:40.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.482+0000 7fe16359d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T12:28:40.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.482+0000 7fe16359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.482+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7fe15c072640 con 0x7fe15c131bd0 2024-09-18T12:28:40.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.484+0000 7fe1597fa640 1 -- 172.21.15.144:0/2874115999 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v64) v1 ==== 134+0+0 (secure 0 0 0) 0x7fe14c016070 con 0x7fe15c131bd0 2024-09-18T12:28:40.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.484+0000 7fe1597fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T12:28:40.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.484+0000 7fe1597fa640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-18T12:28:40.485 INFO:tasks.workunit.client.0.smithi144.stderr:already renamed to 'bar' 2024-09-18T12:28:40.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe148038490 msgr2=0x7fe14803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe148038490 0x7fe14803a940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe154034000 tx=0x7fe154032000 comp rx=0 tx=0).stop 2024-09-18T12:28:40.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 10 monclient: shutdown 2024-09-18T12:28:40.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 msgr2=0x7fe15c11b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c11b6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00c5e0 tx=0x7fe14c007ed0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 shutdown_connections 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe148038490 0x7fe14803a940 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe15c131bd0 0x7fe15c11b6f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe15c09efe0 0x7fe15c130d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 --2- 172.21.15.144:0/2874115999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c09ebe0 0x7fe15c09add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 >> 172.21.15.144:0/2874115999 conn(0x7fe15c0713b0 msgr2=0x7fe15c09d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 shutdown_connections 2024-09-18T12:28:40.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.486+0000 7fe16359d640 1 -- 172.21.15.144:0/2874115999 wait complete. 2024-09-18T12:28:40.496 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls 2024-09-18T12:28:40.496 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false grep foo-asdf 2024-09-18T12:28:40.497 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:28:40.497 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo-asdf 2024-09-18T12:28:40.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.568+0000 7fe5a202b640 10 monclient: get_monmap_and_config 2024-09-18T12:28:40.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.568+0000 7fe5a202b640 10 monclient: build_initial_monmap 2024-09-18T12:28:40.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.568+0000 7fe5a202b640 10 monclient: monmap: 2024-09-18T12:28:40.569 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:40.569 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.569 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:40.569062+0000 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:40.569062+0000 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:40.570 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.568+0000 7fe5a202b640 1 Processor -- start 2024-09-18T12:28:40.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 1 -- start start 2024-09-18T12:28:40.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient: init 2024-09-18T12:28:40.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:40.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T12:28:40.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe59c09ec50 0x7fe59c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): picked mon.noname-c con 0x7fe59c09ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c1321a0 0x7fe59c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): picked mon.noname-b con 0x7fe59c1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe59c09f020 0x7fe59c069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): picked mon.noname-a con 0x7fe59c09f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c067420 con 0x7fe59c09f020 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c0a0480 con 0x7fe59c09ec50 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0a0200 con 0x7fe59c1321a0 2024-09-18T12:28:40.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe5a202b640 10 monclient(hunting): authenticate will time out at 956.138560s 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe59c09ec50 0x7fe59c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe593fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c1321a0 msgr2=0x7fe59c068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe59c09ec50 0x7fe59c131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40834/0 (socket says 172.21.15.144:40834) 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 1 -- 172.21.15.144:0/209451113 learned_addr learned my addr 172.21.15.144:0/209451113 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): get_auth_request con 0x7fe59c09ec50 auth_method 0 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe593fff640 1 --2- 172.21.15.144:0/209451113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c1321a0 0x7fe59c068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:40.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 10 monclient(hunting): handle_auth_done global_id 4969 payload 274 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 1 -- 172.21.15.144:0/209451113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c1321a0 msgr2=0x7fe59c068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 1 --2- 172.21.15.144:0/209451113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c1321a0 0x7fe59c068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59bfff640 1 --2- 172.21.15.144:0/209451113 >> v2:172.21.15.116:3300/0 conn(0x7fe59c09f020 0x7fe59c069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.569+0000 7fe59b7fe640 1 -- 172.21.15.144:0/209451113 >> v2:172.21.15.116:3300/0 conn(0x7fe59c09f020 msgr2=0x7fe59c069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59b7fe640 1 --2- 172.21.15.144:0/209451113 >> v2:172.21.15.116:3300/0 conn(0x7fe59c09f020 0x7fe59c069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59b7fe640 1 -- 172.21.15.144:0/209451113 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c09fd80 con 0x7fe59c09ec50 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59bfff640 10 monclient: get_auth_request con 0x7fe59c09f020 auth_method 0 2024-09-18T12:28:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.570+0000 7fe59bfff640 1 --2- 172.21.15.144:0/209451113 >> v2:172.21.15.116:3300/0 conn(0x7fe59c09f020 0x7fe59c069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59b7fe640 1 -- 172.21.15.144:0/209451113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c09ec50 msgr2=0x7fe59c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59b7fe640 1 --2- 172.21.15.144:0/209451113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c09ec50 0x7fe59c131c60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0719d0 tx=0x7fe58c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a9a2d021599a72a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 1 -- 172.21.15.144:0/209451113 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c02ae40 con 0x7fe59c09ec50 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: dump: 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.581 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: _finish_auth 0 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe5a202b640 5 monclient: authenticate success, global_id 4969 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 1 -- 172.21.15.144:0/209451113 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58c02f4b0 con 0x7fe59c09ec50 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe5a202b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:40.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe5a202b640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 1 -- 172.21.15.144:0/209451113 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c02f850 con 0x7fe59c09ec50 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe5a202b640 4 set_mon_vals no callback set 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.572+0000 7fe59affd640 10 monclient: dump: 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.583 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 10 monclient: shutdown 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 -- 172.21.15.144:0/209451113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c09ec50 msgr2=0x7fe59c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 --2- 172.21.15.144:0/209451113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c09ec50 0x7fe59c131c60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0719d0 tx=0x7fe58c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 -- 172.21.15.144:0/209451113 shutdown_connections 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 --2- 172.21.15.144:0/209451113 >> v2:172.21.15.116:3300/0 conn(0x7fe59c09f020 0x7fe59c069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 --2- 172.21.15.144:0/209451113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c1321a0 0x7fe59c068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 --2- 172.21.15.144:0/209451113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c09ec50 0x7fe59c131c60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 -- 172.21.15.144:0/209451113 >> 172.21.15.144:0/209451113 conn(0x7fe59c0a5cf0 msgr2=0x7fe59c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 -- 172.21.15.144:0/209451113 shutdown_connections 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 1 -- 172.21.15.144:0/209451113 wait complete. 2024-09-18T12:28:40.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 10 monclient: build_initial_monmap 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.573+0000 7fe5a202b640 10 monclient: monmap: 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 Processor -- start 2024-09-18T12:28:40.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 -- start start 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient: init 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c09ec50 0x7fe59c130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): picked mon.noname-c con 0x7fe59c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe59c09f020 0x7fe59c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): picked mon.noname-a con 0x7fe59c09f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 0x7fe59c1415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): picked mon.noname-b con 0x7fe59c1321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c141b10 con 0x7fe59c09f020 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c09c020 con 0x7fe59c1321a0 2024-09-18T12:28:40.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c11df90 con 0x7fe59c09ec50 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe5a202b640 10 monclient(hunting): authenticate will time out at 956.143829s 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59b7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c09ec50 msgr2=0x7fe59c130d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 0x7fe59c1415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c09ec50 0x7fe59c130d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 0x7fe59c1415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40838/0 (socket says 172.21.15.144:40838) 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59bfff640 1 -- 172.21.15.144:0/453683136 learned_addr learned my addr 172.21.15.144:0/453683136 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59bfff640 10 monclient(hunting): get_auth_request con 0x7fe59c1321a0 auth_method 0 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.574+0000 7fe59bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:40.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe59c09f020 0x7fe59c141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 10 monclient(hunting): get_auth_request con 0x7fe59c09f020 auth_method 0 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe593fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 10 monclient(hunting): handle_auth_done global_id 4972 payload 824 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 1 -- 172.21.15.144:0/453683136 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c09ec50 msgr2=0x7fe59c130d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:40.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c09ec50 0x7fe59c130d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 1 -- 172.21.15.144:0/453683136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe59c09f020 msgr2=0x7fe59c141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe59c09f020 0x7fe59c141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 1 monclient: found mon.noname-b 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.575+0000 7fe59bfff640 1 -- 172.21.15.144:0/453683136 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c009450 con 0x7fe59c1321a0 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe59bfff640 1 -- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 msgr2=0x7fe59c1415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe59bfff640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 0x7fe59c1415d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe59400a380 tx=0x7fe594007900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59400d860 con 0x7fe59c1321a0 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:40.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: dump: 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59400d9c0 con 0x7fe59c1321a0 2024-09-18T12:28:40.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5a202b640 5 monclient: authenticate success, global_id 4972 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5a202b640 10 monclient: _renew_subs 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59400dd80 con 0x7fe59c1321a0 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5a202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c11ee80 con 0x7fe59c1321a0 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe59a7fc640 4 set_mon_vals no callback set 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5937fe640 10 monclient: dump: 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.591 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5a202b640 10 monclient: _renew_subs 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5a202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.577+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c131290 con 0x7fe59c1321a0 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe59401e3b0 con 0x7fe59c1321a0 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe5a202b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe5a202b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe5a202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c0008d0 con 0x7fe59c1321a0 2024-09-18T12:28:40.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe5937fe640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe57c038440 0x7fe57c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fe59404c320 con 0x7fe59c1321a0 2024-09-18T12:28:40.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe59b7fe640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe57c038440 0x7fe57c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.578+0000 7fe59b7fe640 10 monclient: get_auth_request con 0x7fe57c038440 auth_method 0 2024-09-18T12:28:40.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.581+0000 7fe59b7fe640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe57c038440 0x7fe57c03a8f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe59c09bf50 tx=0x7fe58c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:40.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.583+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe594012a80 con 0x7fe59c1321a0 2024-09-18T12:28:40.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.583+0000 7fe5937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:40.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.583+0000 7fe5937fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:40.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.738+0000 7fe5a202b640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:40.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.738+0000 7fe5a202b640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:40.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.738+0000 7fe5a202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.738+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fe59c0a13e0 con 0x7fe59c1321a0 2024-09-18T12:28:40.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.739+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v64) v1 ==== 65+0+60 (secure 0 0 0) 0x7fe59401a080 con 0x7fe59c1321a0 2024-09-18T12:28:40.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.739+0000 7fe5937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:40.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.739+0000 7fe5937fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:40.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.832+0000 7fe5a202b640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:40.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.832+0000 7fe5a202b640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:40.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.832+0000 7fe5a202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.832+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fe59c09ea30 con 0x7fe59c1321a0 2024-09-18T12:28:40.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.832+0000 7fe5937fe640 1 -- 172.21.15.144:0/453683136 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v64) v1 ==== 65+0+60 (secure 0 0 0) 0x7fe5940122a0 con 0x7fe59c1321a0 2024-09-18T12:28:40.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.832+0000 7fe5937fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:40.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.832+0000 7fe5937fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe57c038440 msgr2=0x7fe57c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe57c038440 0x7fe57c03a8f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe59c09bf50 tx=0x7fe58c032000 comp rx=0 tx=0).stop 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 10 monclient: shutdown 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 msgr2=0x7fe59c1415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 0x7fe59c1415d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe59400a380 tx=0x7fe594007900 comp rx=0 tx=0).stop 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 shutdown_connections 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe57c038440 0x7fe57c03a8f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe59c1321a0 0x7fe59c1415d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe59c09f020 0x7fe59c141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 --2- 172.21.15.144:0/453683136 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe59c09ec50 0x7fe59c130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 >> 172.21.15.144:0/453683136 conn(0x7fe59c0a5cf0 msgr2=0x7fe59c09d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:40.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 shutdown_connections 2024-09-18T12:28:40.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.834+0000 7fe5a202b640 1 -- 172.21.15.144:0/453683136 wait complete. 2024-09-18T12:28:40.844 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:28:40.845 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule ls 2024-09-18T12:28:40.845 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false grep bar-asdf 2024-09-18T12:28:40.845 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:28:40.845 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep bar-asdf 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.917+0000 7f15b2309640 10 monclient: get_monmap_and_config 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.917+0000 7f15b2309640 10 monclient: build_initial_monmap 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.917+0000 7f15b2309640 10 monclient: monmap: 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:40.917931+0000 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:40.917931+0000 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:40.919 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.917+0000 7f15b2309640 1 Processor -- start 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.917+0000 7f15b2309640 1 -- start start 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.917+0000 7f15b2309640 10 monclient: init 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15ac098150 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f15ac098520 0x7f15ac136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient(hunting): picked mon.noname-c con 0x7f15ac098520 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15ac098980 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac0a18f0 con 0x7f15ac098980 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac132790 con 0x7f15ac098150 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac09fd80 con 0x7f15ac098520 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15b2309640 10 monclient(hunting): authenticate will time out at 956.487435s 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15abfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f15ac098150 0x7f15ac132080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48858/0 (socket says 172.21.15.144:48858) 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15abfff640 1 -- 172.21.15.144:0/3712346699 learned_addr learned my addr 172.21.15.144:0/3712346699 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 1 --2- 172.21.15.144:0/3712346699 >> v2:172.21.15.144:3300/0 conn(0x7f15ac098520 0x7f15ac136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): get_auth_request con 0x7f15ac098520 auth_method 0 2024-09-18T12:28:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient(hunting): handle_auth_done global_id 4975 payload 274 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 1 -- 172.21.15.144:0/3712346699 >> v1:172.21.15.144:6789/0 conn(0x7f15ac098150 legacy=0x7f15ac132080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 1 -- 172.21.15.144:0/3712346699 >> v1:172.21.15.116:6789/0 conn(0x7f15ac098980 legacy=0x7f15ac139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 1 monclient: found mon.noname-c 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.918+0000 7f15ab7fe640 1 -- 172.21.15.144:0/3712346699 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0a1330 con 0x7f15ac098520 2024-09-18T12:28:40.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15ab7fe640 1 -- 172.21.15.144:0/3712346699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098520 msgr2=0x7f15ac136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:40.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15ab7fe640 1 --2- 172.21.15.144:0/3712346699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098520 0x7f15ac136cf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009850 tx=0x7f15a00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f3cb6d41933905 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:40.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 1 -- 172.21.15.144:0/3712346699 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0032440 con 0x7f15ac098520 2024-09-18T12:28:40.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:40.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: dump: 2024-09-18T12:28:40.929 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:40.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 1 -- 172.21.15.144:0/3712346699 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a00325a0 con 0x7f15ac098520 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15b2309640 5 monclient: authenticate success, global_id 4975 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15b2309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15b2309640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 1 -- 172.21.15.144:0/3712346699 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0032980 con 0x7f15ac098520 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15b2309640 4 set_mon_vals no callback set 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.921+0000 7f15aaffd640 10 monclient: dump: 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.931 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 10 monclient: shutdown 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 1 -- 172.21.15.144:0/3712346699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098520 msgr2=0x7f15ac136cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 1 --2- 172.21.15.144:0/3712346699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098520 0x7f15ac136cf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009850 tx=0x7f15a00090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 1 -- 172.21.15.144:0/3712346699 shutdown_connections 2024-09-18T12:28:40.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 1 --2- 172.21.15.144:0/3712346699 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098520 0x7f15ac136cf0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 1 -- 172.21.15.144:0/3712346699 >> 172.21.15.144:0/3712346699 conn(0x7f15ac070e90 msgr2=0x7f15ac0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 1 -- 172.21.15.144:0/3712346699 shutdown_connections 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 1 -- 172.21.15.144:0/3712346699 wait complete. 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 10 monclient: build_initial_monmap 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.922+0000 7f15b2309640 10 monclient: monmap: 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:40.933 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 Processor -- start 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 -- start start 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient: init 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 0x7f15ac139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): picked mon.noname-b con 0x7f15ac098150 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac098520 0x7f15ac135710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): picked mon.noname-c con 0x7f15ac098520 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac098980 0x7f15ac135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): picked mon.noname-a con 0x7f15ac098980 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac098e70 con 0x7f15ac098980 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0a4220 con 0x7f15ac098150 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac138bd0 con 0x7f15ac098520 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15b2309640 10 monclient(hunting): authenticate will time out at 956.492921s 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15abfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 0x7f15ac139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15ab7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac098520 msgr2=0x7f15ac135710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.923+0000 7f15abfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 0x7f15ac139970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40854/0 (socket says 172.21.15.144:40854) 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 1 -- 172.21.15.144:0/1168475915 learned_addr learned my addr 172.21.15.144:0/1168475915 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:40.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): get_auth_request con 0x7f15ac098150 auth_method 0 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15ab7fe640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac098520 0x7f15ac135710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15b08a6640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac098980 0x7f15ac135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15b08a6640 10 monclient(hunting): get_auth_request con 0x7f15ac098980 auth_method 0 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15b08a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15b08a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15b08a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:40.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient(hunting): handle_auth_done global_id 4978 payload 824 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 1 -- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac098520 msgr2=0x7f15ac135710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac098520 0x7f15ac135710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 1 -- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac098980 msgr2=0x7f15ac135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac098980 0x7f15ac135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 1 monclient: found mon.noname-b 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15abfff640 1 -- 172.21.15.144:0/1168475915 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a0009390 con 0x7f15ac098150 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.924+0000 7f15b08a6640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac098980 0x7f15ac135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15abfff640 1 -- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 msgr2=0x7f15ac139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:40.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15abfff640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 0x7f15ac139970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f159c00c610 tx=0x7f159c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c005cb0 con 0x7f15ac098150 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: dump: 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.938 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: _finish_auth 0 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c005e30 con 0x7f15ac098150 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15b2309640 5 monclient: authenticate success, global_id 4978 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15b2309640 10 monclient: _renew_subs 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c007340 con 0x7f15ac098150 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15b2309640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac139190 con 0x7f15ac098150 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f1593fff640 10 monclient: dump: 2024-09-18T12:28:40.939 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15aa7fc640 4 set_mon_vals no callback set 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15b2309640 10 monclient: _renew_subs 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15b2309640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.926+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac139eb0 con 0x7f15ac098150 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f159c01b400 con 0x7f15ac098150 2024-09-18T12:28:40.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f15b2309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f15b2309640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f15b2309640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac13a100 con 0x7f15ac098150 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f1593fff640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1598038440 0x7f159803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f159c04b5f0 con 0x7f15ac098150 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f15ab7fe640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1598038440 0x7f159803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.927+0000 7f15ab7fe640 10 monclient: get_auth_request con 0x7f1598038440 auth_method 0 2024-09-18T12:28:40.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.930+0000 7f15ab7fe640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1598038440 0x7f159803a8f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f15a0034000 tx=0x7f15a0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:40.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.932+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f159c01a940 con 0x7f15ac098150 2024-09-18T12:28:40.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.932+0000 7f1593fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:40.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:40.932+0000 7f1593fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:41.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.085+0000 7f15b2309640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:41.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.085+0000 7f15b2309640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:41.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.085+0000 7f15b2309640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:41.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.085+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f15ac137170 con 0x7f15ac098150 2024-09-18T12:28:41.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.085+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v64) v1 ==== 65+0+60 (secure 0 0 0) 0x7f159c01a360 con 0x7f15ac098150 2024-09-18T12:28:41.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.086+0000 7f1593fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:41.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.086+0000 7f1593fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:41.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.177+0000 7f15b2309640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:41.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.177+0000 7f15b2309640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:41.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.177+0000 7f15b2309640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:41.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.177+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f15ac138d40 con 0x7f15ac098150 2024-09-18T12:28:41.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.178+0000 7f1593fff640 1 -- 172.21.15.144:0/1168475915 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v64) v1 ==== 65+0+60 (secure 0 0 0) 0x7f159c01a940 con 0x7f15ac098150 2024-09-18T12:28:41.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.178+0000 7f1593fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T12:28:41.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.178+0000 7f1593fff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:28:41.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1598038440 msgr2=0x7f159803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:41.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1598038440 0x7f159803a8f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f15a0034000 tx=0x7f15a0032000 comp rx=0 tx=0).stop 2024-09-18T12:28:41.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 10 monclient: shutdown 2024-09-18T12:28:41.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:41.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 msgr2=0x7f15ac139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:41.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 0x7f15ac139970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f159c00c610 tx=0x7f159c004960 comp rx=0 tx=0).stop 2024-09-18T12:28:41.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 shutdown_connections 2024-09-18T12:28:41.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1598038440 0x7f159803a8f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ac098980 0x7f15ac135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ac098520 0x7f15ac135710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 --2- 172.21.15.144:0/1168475915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ac098150 0x7f15ac139970 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.179+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 >> 172.21.15.144:0/1168475915 conn(0x7f15ac070e90 msgr2=0x7f15ac096ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:41.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.180+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 shutdown_connections 2024-09-18T12:28:41.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.180+0000 7f15b2309640 1 -- 172.21.15.144:0/1168475915 wait complete. 2024-09-18T12:28:41.190 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:28:41.190 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rm foo 2024-09-18T12:28:41.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.262+0000 7fefe2020640 10 monclient: get_monmap_and_config 2024-09-18T12:28:41.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.262+0000 7fefe2020640 10 monclient: build_initial_monmap 2024-09-18T12:28:41.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.262+0000 7fefe2020640 10 monclient: monmap: 2024-09-18T12:28:41.263 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:41.263255+0000 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:41.263255+0000 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:41.264 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:41.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 1 Processor -- start 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 1 -- start start 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient: init 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fefdc09e560 0x7fefdc12e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient(hunting): picked mon.noname-c con 0x7fefdc09e560 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefdc09aeb0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc1391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient(hunting): picked mon.noname-b con 0x7fefdc136dd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefdc064c50 con 0x7fefdc09aeb0 2024-09-18T12:28:41.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefdc066090 con 0x7fefdc09e560 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc12f0c0 con 0x7fefdc136dd0 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefe2020640 10 monclient(hunting): authenticate will time out at 956.832591s 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fefdc09e560 0x7fefdc12e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdbfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 msgr2=0x7fefdc1391b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefd2ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fefdc09aeb0 0x7fefdc09b280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48866/0 (socket says 172.21.15.144:48866) 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefd2ffd640 1 -- 172.21.15.144:0/1318316983 learned_addr learned my addr 172.21.15.144:0/1318316983 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): get_auth_request con 0x7fefdc09e560 auth_method 0 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdbfff640 1 --2- 172.21.15.144:0/1318316983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:41.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_done global_id 4981 payload 274 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 1 -- 172.21.15.144:0/1318316983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 msgr2=0x7fefdc1391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 1 --2- 172.21.15.144:0/1318316983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc1391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.263+0000 7fefdb7fe640 1 -- 172.21.15.144:0/1318316983 >> v1:172.21.15.144:6789/0 conn(0x7fefdc09aeb0 legacy=0x7fefdc09b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.264+0000 7fefdb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.264+0000 7fefdb7fe640 1 monclient: found mon.noname-c 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.264+0000 7fefdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.264+0000 7fefdb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:41.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.264+0000 7fefdb7fe640 1 -- 172.21.15.144:0/1318316983 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc06ed70 con 0x7fefdc09e560 2024-09-18T12:28:41.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.266+0000 7fefdb7fe640 1 -- 172.21.15.144:0/1318316983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09e560 msgr2=0x7fefdc12e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:41.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.266+0000 7fefdb7fe640 1 --2- 172.21.15.144:0/1318316983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09e560 0x7fefdc12e7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009850 tx=0x7fefcc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc9568ea97e3b5a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:41.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.266+0000 7fefdaffd640 1 -- 172.21.15.144:0/1318316983 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc032440 con 0x7fefdc09e560 2024-09-18T12:28:41.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.266+0000 7fefdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:41.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:41.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 10 monclient: dump: 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:41.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 1 -- 172.21.15.144:0/1318316983 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefcc0325a0 con 0x7fefdc09e560 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 5 monclient: authenticate success, global_id 4981 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 1 -- 172.21.15.144:0/1318316983 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc032940 con 0x7fefdc09e560 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 4 set_mon_vals no callback set 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefdaffd640 10 monclient: dump: 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:41.277 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 10 monclient: shutdown 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 -- 172.21.15.144:0/1318316983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09e560 msgr2=0x7fefdc12e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 --2- 172.21.15.144:0/1318316983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09e560 0x7fefdc12e7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009850 tx=0x7fefcc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 -- 172.21.15.144:0/1318316983 shutdown_connections 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 --2- 172.21.15.144:0/1318316983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc1391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 --2- 172.21.15.144:0/1318316983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09e560 0x7fefdc12e7c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 -- 172.21.15.144:0/1318316983 >> 172.21.15.144:0/1318316983 conn(0x7fefdc0a1db0 msgr2=0x7fefdc09de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 -- 172.21.15.144:0/1318316983 shutdown_connections 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 1 -- 172.21.15.144:0/1318316983 wait complete. 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.267+0000 7fefe2020640 10 monclient: build_initial_monmap 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient: monmap: 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:41.279 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 Processor -- start 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 -- start start 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient: init 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 0x7fefdc0972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): picked mon.noname-b con 0x7fefdc09aeb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefdc09e560 0x7fefdc097820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): picked mon.noname-a con 0x7fefdc09e560 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc126520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): picked mon.noname-c con 0x7fefdc136dd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:41.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc09ebf0 con 0x7fefdc09e560 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc129e50 con 0x7fefdc09aeb0 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc0a0f00 con 0x7fefdc136dd0 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.268+0000 7fefe2020640 10 monclient(hunting): authenticate will time out at 956.837939s 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdbfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 msgr2=0x7fefdc126520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 0x7fefdc0972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdbfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc126520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 0x7fefdc0972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40864/0 (socket says 172.21.15.144:40864) 2024-09-18T12:28:41.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 -- 172.21.15.144:0/4020153726 learned_addr learned my addr 172.21.15.144:0/4020153726 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): get_auth_request con 0x7fefdc09aeb0 auth_method 0 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient(hunting): handle_auth_done global_id 4984 payload 824 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 -- 172.21.15.144:0/4020153726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 msgr2=0x7fefdc126520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc126520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 -- 172.21.15.144:0/4020153726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefdc09e560 msgr2=0x7fefdc097820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefdc09e560 0x7fefdc097820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:41.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 monclient: found mon.noname-b 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.269+0000 7fefdb7fe640 1 -- 172.21.15.144:0/4020153726 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc009390 con 0x7fefdc09aeb0 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefdb7fe640 1 -- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 msgr2=0x7fefdc0972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefdb7fe640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 0x7fefdc0972e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009360 tx=0x7fefcc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc0319a0 con 0x7fefdc09aeb0 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefdb7fe640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 0x7fefdc0972e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009360 tx=0x7fefcc032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fefcc031e80 mon_map magic: 0 v1 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 10 monclient: dump: 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:41.283 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 10 monclient: _finish_auth 0 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefe2020640 5 monclient: authenticate success, global_id 4984 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefcc031b00 con 0x7fefdc09aeb0 2024-09-18T12:28:41.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefd3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefe2020640 10 monclient: _renew_subs 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefe2020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc129b70 con 0x7fefdc09aeb0 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefda7fc640 4 set_mon_vals no callback set 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefe2020640 10 monclient: _renew_subs 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefe2020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.271+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc097d60 con 0x7fefdc09aeb0 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc031e80 con 0x7fefdc09aeb0 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefd3fff640 10 monclient: dump: 2024-09-18T12:28:41.285 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fefcc039040 con 0x7fefdc09aeb0 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefe2020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefe2020640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:41.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefe2020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:41.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc12a070 con 0x7fefdc09aeb0 2024-09-18T12:28:41.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.272+0000 7fefd3fff640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefbc038440 0x7fefbc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:41.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.273+0000 7fefd2ffd640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefbc038440 0x7fefbc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:41.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.273+0000 7fefd2ffd640 10 monclient: get_auth_request con 0x7fefbc038440 auth_method 0 2024-09-18T12:28:41.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.275+0000 7fefd2ffd640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefbc038440 0x7fefbc03a8f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fefdc127cb0 tx=0x7fefc4006bd0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:41.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.277+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fefcc06b8b0 con 0x7fefdc09aeb0 2024-09-18T12:28:41.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.894+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fefdc12a070 con 0x7fefdc09aeb0 2024-09-18T12:28:41.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.894+0000 7fefd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:41.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:41.894+0000 7fefd3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:42.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.050+0000 7fefe2020640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:42.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.050+0000 7fefe2020640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:42.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.050+0000 7fefe2020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:42.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.050+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fefdc12a070 con 0x7fefdc09aeb0 2024-09-18T12:28:42.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.839+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v66) v1 ==== 80+0+0 (secure 0 0 0) 0x7fefdc12a070 con 0x7fefdc09aeb0 2024-09-18T12:28:42.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.839+0000 7fefd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:42.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.839+0000 7fefd3fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:42.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.937+0000 7fefe2020640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:42.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.937+0000 7fefe2020640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:42.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.937+0000 7fefe2020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:42.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.937+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fefdc12a070 con 0x7fefdc09aeb0 2024-09-18T12:28:42.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.938+0000 7fefd3fff640 1 -- 172.21.15.144:0/4020153726 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v66) v1 ==== 103+0+0 (secure 0 0 0) 0x7fefdc12a070 con 0x7fefdc09aeb0 2024-09-18T12:28:42.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.938+0000 7fefd3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:42.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.938+0000 7fefd3fff640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-18T12:28:42.940 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo does not exist 2024-09-18T12:28:42.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefbc038440 msgr2=0x7fefbc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:42.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefbc038440 0x7fefbc03a8f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fefdc127cb0 tx=0x7fefc4006bd0 comp rx=0 tx=0).stop 2024-09-18T12:28:42.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 10 monclient: shutdown 2024-09-18T12:28:42.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:42.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 msgr2=0x7fefdc0972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:42.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 0x7fefdc0972e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009360 tx=0x7fefcc032000 comp rx=0 tx=0).stop 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 shutdown_connections 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fefbc038440 0x7fefbc03a8f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fefdc136dd0 0x7fefdc126520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefdc09e560 0x7fefdc097820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 --2- 172.21.15.144:0/4020153726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fefdc09aeb0 0x7fefdc0972e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 >> 172.21.15.144:0/4020153726 conn(0x7fefdc0a1db0 msgr2=0x7fefdc12fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 shutdown_connections 2024-09-18T12:28:42.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:42.941+0000 7fefe2020640 1 -- 172.21.15.144:0/4020153726 wait complete. 2024-09-18T12:28:42.951 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rm foo 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.024+0000 7fbfb3c57640 10 monclient: get_monmap_and_config 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.024+0000 7fbfb3c57640 10 monclient: build_initial_monmap 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.024+0000 7fbfb3c57640 10 monclient: monmap: 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:43.025584+0000 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:43.025584+0000 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:43.026 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 1 Processor -- start 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 1 -- start start 2024-09-18T12:28:43.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient: init 2024-09-18T12:28:43.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:43.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T12:28:43.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cad0 0x7fbfac09c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:43.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient(hunting): picked mon.noname-b con 0x7fbfac09cad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:43.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfac09cea0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:43.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfac09d270 0x7fbfac09e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient(hunting): picked mon.noname-a con 0x7fbfac09d270 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac09b710 con 0x7fbfac09cea0 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfac09c6e0 con 0x7fbfac09d270 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac09a650 con 0x7fbfac09cad0 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.025+0000 7fbfb3c57640 10 monclient(hunting): authenticate will time out at 958.594959s 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb19f3640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cad0 msgr2=0x7fbfac09c1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb19f3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cad0 0x7fbfac09c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfac09d270 0x7fbfac09e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfac09d270 0x7fbfac09e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:60584/0 (socket says 172.21.15.144:60584) 2024-09-18T12:28:43.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 -- 172.21.15.144:0/1510882523 learned_addr learned my addr 172.21.15.144:0/1510882523 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb11f2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbfac09cea0 0x7fbfac09dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:53702/0 (socket says 172.21.15.144:53702) 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): get_auth_request con 0x7fbfac09d270 auth_method 0 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient(hunting): handle_auth_done global_id 54243 payload 274 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 -- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cad0 msgr2=0x7fbfac09c1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 --2- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cad0 0x7fbfac09c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 -- 172.21.15.144:0/1510882523 >> v1:172.21.15.116:6789/0 conn(0x7fbfac09cea0 legacy=0x7fbfac09dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:43.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient: _finish_hunting 0 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 monclient: found mon.noname-a 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.026+0000 7fbfb21f4640 1 -- 172.21.15.144:0/1510882523 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac09f660 con 0x7fbfac09d270 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb21f4640 1 -- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 msgr2=0x7fbfac09e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb21f4640 1 --2- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac09e1e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009f20 tx=0x7fbfa4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd4aff7a18a9a24 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 1 -- 172.21.15.144:0/1510882523 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4007610 con 0x7fbfac09d270 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: dump: 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:43.034 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: _finish_auth 0 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb3c57640 5 monclient: authenticate success, global_id 54243 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb3c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:43.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 1 -- 172.21.15.144:0/1510882523 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa402d050 con 0x7fbfac09d270 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 1 -- 172.21.15.144:0/1510882523 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa40079a0 con 0x7fbfac09d270 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb3c57640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb3c57640 4 set_mon_vals no callback set 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.029+0000 7fbfb09f1640 10 monclient: dump: 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:43.036 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 10 monclient: shutdown 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 -- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 msgr2=0x7fbfac09e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac09e1e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009f20 tx=0x7fbfa4009840 comp rx=0 tx=0).stop 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 -- 172.21.15.144:0/1510882523 shutdown_connections 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac09e1e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/1510882523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cad0 0x7fbfac09c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 -- 172.21.15.144:0/1510882523 >> 172.21.15.144:0/1510882523 conn(0x7fbfac070e90 msgr2=0x7fbfac0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:43.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 -- 172.21.15.144:0/1510882523 shutdown_connections 2024-09-18T12:28:43.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 1 -- 172.21.15.144:0/1510882523 wait complete. 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 10 monclient: build_initial_monmap 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.030+0000 7fbfb3c57640 10 monclient: monmap: 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:43.042 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 Processor -- start 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 -- start start 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient: init 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 0x7fbfac13ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): picked mon.noname-b con 0x7fbfac09cad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cea0 0x7fbfac132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): picked mon.noname-c con 0x7fbfac09cea0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac11c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:43.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): picked mon.noname-a con 0x7fbfac09d270 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0a27f0 con 0x7fbfac09d270 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac11fee0 con 0x7fbfac09cad0 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac11e0d0 con 0x7fbfac09cea0 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb3c57640 10 monclient(hunting): authenticate will time out at 958.600725s 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 0x7fbfac13ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb11f2640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cea0 msgr2=0x7fbfac132ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 0x7fbfac13ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40880/0 (socket says 172.21.15.144:40880) 2024-09-18T12:28:43.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 1 -- 172.21.15.144:0/73457855 learned_addr learned my addr 172.21.15.144:0/73457855 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 10 monclient(hunting): get_auth_request con 0x7fbfac09cad0 auth_method 0 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb11f2640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cea0 0x7fbfac132ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb19f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb21f4640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac11c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb21f4640 10 monclient(hunting): get_auth_request con 0x7fbfac09d270 auth_method 0 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb21f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb21f4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:43.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.031+0000 7fbfb21f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 10 monclient(hunting): handle_auth_done global_id 4987 payload 824 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 1 -- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cea0 msgr2=0x7fbfac132ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cea0 0x7fbfac132ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 1 -- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 msgr2=0x7fbfac11c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac11c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 10 monclient: _finish_hunting 0 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 1 monclient: found mon.noname-b 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb19f3640 1 -- 172.21.15.144:0/73457855 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa4009ae0 con 0x7fbfac09cad0 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.032+0000 7fbfb21f4640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac11c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb19f3640 1 -- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 msgr2=0x7fbfac13ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:43.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb19f3640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 0x7fbfac13ec10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800b630 tx=0x7fbfa80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb19f3640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 0x7fbfac13ec10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800b630 tx=0x7fbfa80091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fbfa8015070 mon_map magic: 0 v1 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8015070 con 0x7fbfac09cad0 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 10 monclient: dump: 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:43.047 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8003d90 con 0x7fbfac09cad0 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb3c57640 5 monclient: authenticate success, global_id 4987 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb3c57640 10 monclient: _renew_subs 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb3c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac11fc00 con 0x7fbfac09cad0 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfa3fff640 4 set_mon_vals no callback set 2024-09-18T12:28:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb3c57640 10 monclient: _renew_subs 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb3c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.034+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac13f150 con 0x7fbfac09cad0 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8006380 con 0x7fbfac09cad0 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfa1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfa1ffb640 10 monclient: dump: 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbfa8010440 con 0x7fbfac09cad0 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfb3c57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfb3c57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfb3c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac0008d0 con 0x7fbfac09cad0 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfa1ffb640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf90038490 0x7fbf9003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfb11f2640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf90038490 0x7fbf9003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.035+0000 7fbfb11f2640 10 monclient: get_auth_request con 0x7fbf90038490 auth_method 0 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.038+0000 7fbfb11f2640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf90038490 0x7fbf9003a940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c004880 tx=0x7fbf9c007ac0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.040+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5086+0+0 (secure 0 0 0) 0x7fbfa804b380 con 0x7fbfac09cad0 2024-09-18T12:28:43.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.657+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbfa8006770 con 0x7fbfac09cad0 2024-09-18T12:28:43.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.657+0000 7fbfa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:43.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.657+0000 7fbfa1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.817+0000 7fbfb3c57640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:43.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.817+0000 7fbfb3c57640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:43.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.817+0000 7fbfb3c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:43.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.817+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fbfac09b430 con 0x7fbfac09cad0 2024-09-18T12:28:43.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.818+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v66) v1 ==== 103+0+0 (secure 0 0 0) 0x7fbfa8018780 con 0x7fbfac09cad0 2024-09-18T12:28:43.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.818+0000 7fbfa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:43.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.818+0000 7fbfa1ffb640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-18T12:28:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.914+0000 7fbfb3c57640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.914+0000 7fbfb3c57640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.914+0000 7fbfb3c57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.914+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fbfac120050 con 0x7fbfac09cad0 2024-09-18T12:28:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.915+0000 7fbfa1ffb640 1 -- 172.21.15.144:0/73457855 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v66) v1 ==== 103+0+0 (secure 0 0 0) 0x7fbfa8018d60 con 0x7fbfac09cad0 2024-09-18T12:28:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.915+0000 7fbfa1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T12:28:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.915+0000 7fbfa1ffb640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-18T12:28:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo does not exist 2024-09-18T12:28:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf90038490 msgr2=0x7fbf9003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf90038490 0x7fbf9003a940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c004880 tx=0x7fbf9c007ac0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 10 monclient: shutdown 2024-09-18T12:28:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 msgr2=0x7fbfac13ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 0x7fbfac13ec10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800b630 tx=0x7fbfa80091a0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 shutdown_connections 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf90038490 0x7fbf9003a940 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfac09d270 0x7fbfac11c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfac09cea0 0x7fbfac132ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 --2- 172.21.15.144:0/73457855 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfac09cad0 0x7fbfac13ec10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 >> 172.21.15.144:0/73457855 conn(0x7fbfac070e90 msgr2=0x7fbfac096a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 shutdown_connections 2024-09-18T12:28:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:43.917+0000 7fbfb3c57640 1 -- 172.21.15.144:0/73457855 wait complete. 2024-09-18T12:28:43.927 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rm bar 2024-09-18T12:28:44.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.002+0000 7ff388a46640 10 monclient: get_monmap_and_config 2024-09-18T12:28:44.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.002+0000 7ff388a46640 10 monclient: build_initial_monmap 2024-09-18T12:28:44.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.002+0000 7ff388a46640 10 monclient: monmap: 2024-09-18T12:28:44.003 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:44.003350+0000 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:44.003350+0000 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:44.004 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:44.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 1 Processor -- start 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 1 -- start start 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient: init 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff384131c90 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff38409d210 0x7ff384069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient(hunting): picked mon.noname-b con 0x7ff38409d210 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3840a0ab0 0x7ff384138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient(hunting): picked mon.noname-c con 0x7ff3840a0ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3840a1800 con 0x7ff384131c90 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:44.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38409fea0 con 0x7ff3840a0ab0 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840a1520 con 0x7ff38409d210 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff388a46640 10 monclient(hunting): authenticate will time out at 959.572796s 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff381d9b640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff38409d210 msgr2=0x7ff384069e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff382d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3840a0ab0 0x7ff384138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff381d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff38409d210 0x7ff384069e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff382d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3840a0ab0 0x7ff384138fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40892/0 (socket says 172.21.15.144:40892) 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff382d9d640 1 -- 172.21.15.144:0/1538288456 learned_addr learned my addr 172.21.15.144:0/1538288456 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff382d9d640 10 monclient(hunting): get_auth_request con 0x7ff3840a0ab0 auth_method 0 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff382d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff382d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:44.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.003+0000 7ff382d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 10 monclient(hunting): handle_auth_done global_id 4990 payload 274 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 1 -- 172.21.15.144:0/1538288456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff38409d210 msgr2=0x7ff384069e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 1 --2- 172.21.15.144:0/1538288456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff38409d210 0x7ff384069e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 1 -- 172.21.15.144:0/1538288456 >> v1:172.21.15.116:6789/0 conn(0x7ff384131c90 legacy=0x7ff384132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 1 monclient: found mon.noname-c 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:44.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.004+0000 7ff382d9d640 1 -- 172.21.15.144:0/1538288456 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff384132770 con 0x7ff3840a0ab0 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.006+0000 7ff382d9d640 1 -- 172.21.15.144:0/1538288456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3840a0ab0 msgr2=0x7ff384138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.006+0000 7ff382d9d640 1 --2- 172.21.15.144:0/1538288456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3840a0ab0 0x7ff384138fb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009f20 tx=0x7ff37c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea10989c26fc78e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 1 -- 172.21.15.144:0/1538288456 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c007610 con 0x7ff3840a0ab0 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: dump: 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:44.015 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: _finish_auth 0 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 1 -- 172.21.15.144:0/1538288456 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c02d050 con 0x7ff3840a0ab0 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 5 monclient: authenticate success, global_id 4990 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 1 -- 172.21.15.144:0/1538288456 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0079a0 con 0x7ff3840a0ab0 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:44.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 4 set_mon_vals no callback set 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff38159a640 10 monclient: dump: 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:44.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 10 monclient: shutdown 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 -- 172.21.15.144:0/1538288456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3840a0ab0 msgr2=0x7ff384138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 --2- 172.21.15.144:0/1538288456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3840a0ab0 0x7ff384138fb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009f20 tx=0x7ff37c009840 comp rx=0 tx=0).stop 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 -- 172.21.15.144:0/1538288456 shutdown_connections 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 --2- 172.21.15.144:0/1538288456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3840a0ab0 0x7ff384138fb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 --2- 172.21.15.144:0/1538288456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff38409d210 0x7ff384069e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 -- 172.21.15.144:0/1538288456 >> 172.21.15.144:0/1538288456 conn(0x7ff384071370 msgr2=0x7ff3840a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 -- 172.21.15.144:0/1538288456 shutdown_connections 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.007+0000 7ff388a46640 1 -- 172.21.15.144:0/1538288456 wait complete. 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient: build_initial_monmap 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient: monmap: 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:44.018 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 Processor -- start 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 -- start start 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient: init 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:44.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3840a0ab0 0x7ff384138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): picked mon.noname-a con 0x7ff3840a0ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff384131c90 0x7ff384133fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): picked mon.noname-c con 0x7ff384131c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 0x7ff384134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): picked mon.noname-b con 0x7ff384136790 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff384138880 con 0x7ff3840a0ab0 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840a0fb0 con 0x7ff384136790 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840a2910 con 0x7ff384131c90 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:44.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.008+0000 7ff388a46640 10 monclient(hunting): authenticate will time out at 959.577989s 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 0x7ff384134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff381d9b640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff384131c90 msgr2=0x7ff384133fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff381d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff384131c90 0x7ff384133fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 0x7ff384134520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40908/0 (socket says 172.21.15.144:40908) 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 -- 172.21.15.144:0/2438559898 learned_addr learned my addr 172.21.15.144:0/2438559898 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): get_auth_request con 0x7ff384136790 auth_method 0 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:44.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff38259c640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3840a0ab0 0x7ff384138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff38259c640 10 monclient(hunting): get_auth_request con 0x7ff3840a0ab0 auth_method 0 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff38259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff38259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff38259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient(hunting): handle_auth_done global_id 4993 payload 824 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 -- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff384131c90 msgr2=0x7ff384133fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff384131c90 0x7ff384133fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 -- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3840a0ab0 msgr2=0x7ff384138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3840a0ab0 0x7ff384138340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 monclient: found mon.noname-b 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:44.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff382d9d640 1 -- 172.21.15.144:0/2438559898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c009ae0 con 0x7ff384136790 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.009+0000 7ff38259c640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3840a0ab0 0x7ff384138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff382d9d640 1 -- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 msgr2=0x7ff384134520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff382d9d640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 0x7ff384134520 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009ab0 tx=0x7ff37c036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c031cb0 con 0x7ff384136790 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: dump: 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:44.023 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c031e10 con 0x7ff384136790 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff388a46640 5 monclient: authenticate success, global_id 4993 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c03c650 con 0x7ff384136790 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff388a46640 10 monclient: _renew_subs 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff388a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:44.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3841360d0 con 0x7ff384136790 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff3727fc640 10 monclient: dump: 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:44.025 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff380d99640 4 set_mon_vals no callback set 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff388a46640 10 monclient: _renew_subs 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff388a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.011+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff384135bc0 con 0x7ff384136790 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff37c03ca40 con 0x7ff384136790 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff388a46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff388a46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff388a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3840008d0 con 0x7ff384136790 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff3727fc640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff364038440 0x7ff36403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5086+0+0 (secure 0 0 0) 0x7ff37c06c3a0 con 0x7ff384136790 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.012+0000 7ff38259c640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff364038440 0x7ff36403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:44.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.013+0000 7ff38259c640 10 monclient: get_auth_request con 0x7ff364038440 auth_method 0 2024-09-18T12:28:44.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.015+0000 7ff38259c640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff364038440 0x7ff36403a8f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff374002860 tx=0x7ff374009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:44.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.017+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff37c02fae0 con 0x7ff384136790 2024-09-18T12:28:44.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.017+0000 7ff3727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:44.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.017+0000 7ff3727fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:44.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.175+0000 7ff388a46640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T12:28:44.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.175+0000 7ff388a46640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T12:28:44.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.175+0000 7ff388a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:44.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.175+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7ff3841394f0 con 0x7ff384136790 2024-09-18T12:28:44.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.843+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 v67) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff37c02f300 con 0x7ff384136790 2024-09-18T12:28:44.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.843+0000 7ff3727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T12:28:44.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.843+0000 7ff3727fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:44.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.943+0000 7ff388a46640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T12:28:44.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.943+0000 7ff388a46640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T12:28:44.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.943+0000 7ff388a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:44.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.943+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7ff38409cec0 con 0x7ff384136790 2024-09-18T12:28:44.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.945+0000 7ff3727fc640 1 -- 172.21.15.144:0/2438559898 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v67) v1 ==== 103+0+0 (secure 0 0 0) 0x7ff37c040030 con 0x7ff384136790 2024-09-18T12:28:44.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.945+0000 7ff3727fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T12:28:44.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.945+0000 7ff3727fc640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-18T12:28:44.946 INFO:tasks.workunit.client.0.smithi144.stderr:rule bar does not exist 2024-09-18T12:28:44.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff364038440 msgr2=0x7ff36403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:44.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff364038440 0x7ff36403a8f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff374002860 tx=0x7ff374009040 comp rx=0 tx=0).stop 2024-09-18T12:28:44.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 10 monclient: shutdown 2024-09-18T12:28:44.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 msgr2=0x7ff384134520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 0x7ff384134520 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009ab0 tx=0x7ff37c036000 comp rx=0 tx=0).stop 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 shutdown_connections 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff364038440 0x7ff36403a8f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff384136790 0x7ff384134520 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff384131c90 0x7ff384133fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 --2- 172.21.15.144:0/2438559898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3840a0ab0 0x7ff384138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 >> 172.21.15.144:0/2438559898 conn(0x7ff384071370 msgr2=0x7ff38409df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 shutdown_connections 2024-09-18T12:28:44.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:44.947+0000 7ff388a46640 1 -- 172.21.15.144:0/2438559898 wait complete. 2024-09-18T12:28:44.957 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create pinning_pool 1 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.030+0000 7fb6b0d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.030+0000 7fb6b0d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.030+0000 7fb6b0d1d640 10 monclient: monmap: 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:45.031857+0000 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:45.031857+0000 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:45.032 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:45.033 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:45.033 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:45.033 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:45.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.031+0000 7fb6b0d1d640 1 Processor -- start 2024-09-18T12:28:45.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.031+0000 7fb6b0d1d640 1 -- start start 2024-09-18T12:28:45.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.031+0000 7fb6b0d1d640 10 monclient: init 2024-09-18T12:28:45.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:45.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6ac0a19a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb6ac09a260 0x7fb6ac09bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6ac09a260 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb6ac1336f0 0x7fb6ac135ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6ac1336f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac0a1d70 con 0x7fb6ac0a19a0 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ac098600 con 0x7fb6ac1336f0 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ac097a70 con 0x7fb6ac09a260 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6b0d1d640 10 monclient(hunting): authenticate will time out at 960.601208s 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb6ac09a260 0x7fb6ac09bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:45.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb6ac09a260 0x7fb6ac09bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40918/0 (socket says 172.21.15.144:40918) 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 -- 172.21.15.144:0/3977372958 learned_addr learned my addr 172.21.15.144:0/3977372958 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6aa59c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb6ac0a19a0 0x7fb6ac09b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48874/0 (socket says 172.21.15.144:48874) 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): get_auth_request con 0x7fb6ac09a260 auth_method 0 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6aad9d640 1 --2- 172.21.15.144:0/3977372958 >> v2:172.21.15.116:3300/0 conn(0x7fb6ac1336f0 0x7fb6ac135ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6aad9d640 10 monclient(hunting): get_auth_request con 0x7fb6ac1336f0 auth_method 0 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:45.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient(hunting): handle_auth_done global_id 4996 payload 274 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 -- 172.21.15.144:0/3977372958 >> v2:172.21.15.116:3300/0 conn(0x7fb6ac1336f0 msgr2=0x7fb6ac135ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 --2- 172.21.15.144:0/3977372958 >> v2:172.21.15.116:3300/0 conn(0x7fb6ac1336f0 0x7fb6ac135ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 -- 172.21.15.144:0/3977372958 >> v1:172.21.15.144:6789/0 conn(0x7fb6ac0a19a0 legacy=0x7fb6ac09b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 monclient: found mon.noname-c 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:45.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.032+0000 7fb6a9d9b640 1 -- 172.21.15.144:0/3977372958 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac097cb0 con 0x7fb6ac09a260 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a9d9b640 1 -- 172.21.15.144:0/3977372958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 msgr2=0x7fb6ac09bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a9d9b640 1 --2- 172.21.15.144:0/3977372958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac09bff0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac09d7f0 tx=0x7fb6a000b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8379377c887111e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 1 -- 172.21.15.144:0/3977372958 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0032730 con 0x7fb6ac09a260 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: dump: 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.044 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: _finish_auth 0 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 1 -- 172.21.15.144:0/3977372958 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a0032890 con 0x7fb6ac09a260 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 1 -- 172.21.15.144:0/3977372958 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0032c30 con 0x7fb6ac09a260 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6b0d1d640 5 monclient: authenticate success, global_id 4996 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6b0d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:45.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6b0d1d640 4 set_mon_vals no callback set 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.035+0000 7fb6a959a640 10 monclient: dump: 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:45.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 10 monclient: shutdown 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/3977372958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 msgr2=0x7fb6ac09bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/3977372958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac09bff0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac09d7f0 tx=0x7fb6a000b460 comp rx=0 tx=0).stop 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/3977372958 shutdown_connections 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/3977372958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac09bff0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/3977372958 >> v2:172.21.15.116:3300/0 conn(0x7fb6ac1336f0 0x7fb6ac135ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/3977372958 >> 172.21.15.144:0/3977372958 conn(0x7fb6ac0711a0 msgr2=0x7fb6ac0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/3977372958 shutdown_connections 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/3977372958 wait complete. 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.036+0000 7fb6b0d1d640 10 monclient: monmap: 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:45.047 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 Processor -- start 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 -- start start 2024-09-18T12:28:45.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient: init 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac13ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6ac09a260 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6ac0a19a0 0x7fb6ac130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6ac0a19a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6ac1336f0 0x7fb6ac118780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6ac1336f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac13d410 con 0x7fb6ac0a19a0 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac09cc10 con 0x7fb6ac09a260 2024-09-18T12:28:45.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac09e1b0 con 0x7fb6ac1336f0 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6b0d1d640 10 monclient(hunting): authenticate will time out at 960.606766s 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac13ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aad9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6ac1336f0 msgr2=0x7fb6ac118780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6ac1336f0 0x7fb6ac118780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac13ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40922/0 (socket says 172.21.15.144:40922) 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 1 -- 172.21.15.144:0/279527443 learned_addr learned my addr 172.21.15.144:0/279527443 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 10 monclient(hunting): get_auth_request con 0x7fb6ac09a260 auth_method 0 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:45.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6a9d9b640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6ac0a19a0 0x7fb6ac130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6a9d9b640 10 monclient(hunting): get_auth_request con 0x7fb6ac0a19a0 auth_method 0 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.037+0000 7fb6a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 10 monclient(hunting): handle_auth_done global_id 4999 payload 824 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 1 -- 172.21.15.144:0/279527443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6ac1336f0 msgr2=0x7fb6ac118780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:45.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6ac1336f0 0x7fb6ac118780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 1 -- 172.21.15.144:0/279527443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6ac0a19a0 msgr2=0x7fb6ac130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6ac0a19a0 0x7fb6ac130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 1 monclient: found mon.noname-b 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.038+0000 7fb6aa59c640 1 -- 172.21.15.144:0/279527443 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a000b700 con 0x7fb6ac09a260 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6aa59c640 1 -- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 msgr2=0x7fb6ac13ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6aa59c640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac13ced0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c5b0 tx=0x7fb69c007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c007510 con 0x7fb6ac09a260 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:45.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: dump: 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c0057d0 con 0x7fb6ac09a260 2024-09-18T12:28:45.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6b0d1d640 5 monclient: authenticate success, global_id 4999 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6b0d1d640 10 monclient: _renew_subs 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6b0d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac11c280 con 0x7fb6ac09a260 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c007690 con 0x7fb6ac09a260 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb69a7fc640 10 monclient: dump: 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:45.054 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6b0d1d640 10 monclient: _renew_subs 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6a8d99640 4 set_mon_vals no callback set 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6b0d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.040+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac11a190 con 0x7fb6ac09a260 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb6b0d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb6b0d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb6b0d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb69c01b400 con 0x7fb6ac09a260 2024-09-18T12:28:45.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac0008d0 con 0x7fb6ac09a260 2024-09-18T12:28:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb69a7fc640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb68c038440 0x7fb68c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5031+0+0 (secure 0 0 0) 0x7fb69c04b330 con 0x7fb6ac09a260 2024-09-18T12:28:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb6a9d9b640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb68c038440 0x7fb68c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.041+0000 7fb6a9d9b640 10 monclient: get_auth_request con 0x7fb68c038440 auth_method 0 2024-09-18T12:28:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.044+0000 7fb6a9d9b640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb68c038440 0x7fb68c03a8f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac068b30 tx=0x7fb6a0038000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.045+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb69c005e60 con 0x7fb6ac09a260 2024-09-18T12:28:45.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.045+0000 7fb69a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:45.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.045+0000 7fb69a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:45.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.202+0000 7fb6b0d1d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T12:28:45.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.202+0000 7fb6b0d1d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T12:28:45.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.202+0000 7fb6b0d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:45.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.202+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7fb6ac06ebf0 con 0x7fb6ac09a260 2024-09-18T12:28:45.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.848+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v68) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb69c01a6e0 con 0x7fb6ac09a260 2024-09-18T12:28:45.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.848+0000 7fb69a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T12:28:45.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.848+0000 7fb69a7fc640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-18T12:28:45.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.945+0000 7fb6b0d1d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T12:28:45.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.945+0000 7fb6b0d1d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T12:28:45.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.945+0000 7fb6b0d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:45.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.945+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7fb6ac09e840 con 0x7fb6ac09a260 2024-09-18T12:28:45.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.946+0000 7fb69a7fc640 1 -- 172.21.15.144:0/279527443 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v68) v1 ==== 134+0+0 (secure 0 0 0) 0x7fb69c01ab00 con 0x7fb6ac09a260 2024-09-18T12:28:45.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.946+0000 7fb69a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T12:28:45.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.946+0000 7fb69a7fc640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-18T12:28:45.949 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'pinning_pool' already exists 2024-09-18T12:28:45.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb68c038440 msgr2=0x7fb68c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:45.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb68c038440 0x7fb68c03a8f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac068b30 tx=0x7fb6a0038000 comp rx=0 tx=0).stop 2024-09-18T12:28:45.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 10 monclient: shutdown 2024-09-18T12:28:45.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:45.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 msgr2=0x7fb6ac13ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac13ced0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00c5b0 tx=0x7fb69c007f00 comp rx=0 tx=0).stop 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 shutdown_connections 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb68c038440 0x7fb68c03a8f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb6ac1336f0 0x7fb6ac118780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6ac0a19a0 0x7fb6ac130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 --2- 172.21.15.144:0/279527443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6ac09a260 0x7fb6ac13ced0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 >> 172.21.15.144:0/279527443 conn(0x7fb6ac0711a0 msgr2=0x7fb6ac09b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 shutdown_connections 2024-09-18T12:28:45.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:45.948+0000 7fb6b0d1d640 1 -- 172.21.15.144:0/279527443 wait complete. 2024-09-18T12:28:45.959 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-18T12:28:45.959 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:28:45.959 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-18T12:28:46.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.033+0000 7f2333d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:46.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.033+0000 7f2333d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:46.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.033+0000 7f2333d1d640 10 monclient: monmap: 2024-09-18T12:28:46.034 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:46.034 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.034 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:46.034188+0000 2024-09-18T12:28:46.034 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:46.034188+0000 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:46.035 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.033+0000 7f2333d1d640 1 Processor -- start 2024-09-18T12:28:46.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 1 -- start start 2024-09-18T12:28:46.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient: init 2024-09-18T12:28:46.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:46.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:28:46.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f232c09ec50 0x7f232c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f232c09ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f232c1321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09f020 0x7f232c069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f232c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c067420 con 0x7f232c1321a0 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232c09fdc0 con 0x7f232c09ec50 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c0a0510 con 0x7f232c09f020 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2333d1d640 10 monclient(hunting): authenticate will time out at 961.603684s 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f23322ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09f020 msgr2=0x7f232c069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f23322ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09f020 0x7f232c069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:46.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.034+0000 7f2331ab9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f232c09ec50 0x7f232c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f232c09ec50 0x7f232c131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:60620/0 (socket says 172.21.15.144:60620) 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f23312b8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f232c1321a0 0x7f232c068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:53708/0 (socket says 172.21.15.144:53708) 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 1 -- 172.21.15.144:0/162742684 learned_addr learned my addr 172.21.15.144:0/162742684 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): get_auth_request con 0x7f232c09ec50 auth_method 0 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f23312b8640 1 -- 172.21.15.144:0/162742684 >> v1:172.21.15.116:6789/0 conn(0x7f232c1321a0 legacy=0x7f232c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f23312b8640 1 -- 172.21.15.144:0/162742684 >> v1:172.21.15.116:6789/0 conn(0x7f232c1321a0 legacy=0x7f232c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f23312b8640 1 --1- 172.21.15.144:0/162742684 >> v1:172.21.15.116:6789/0 conn(0x7f232c1321a0 0x7f232c068d00 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:46.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient(hunting): handle_auth_done global_id 54258 payload 274 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 1 -- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09f020 msgr2=0x7f232c069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 1 --2- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09f020 0x7f232c069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 1 -- 172.21.15.144:0/162742684 >> v1:172.21.15.116:6789/0 conn(0x7f232c1321a0 legacy=0x7f232c068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient: _finish_hunting 0 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 1 monclient: found mon.noname-a 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:46.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.035+0000 7f2331ab9640 1 -- 172.21.15.144:0/162742684 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c0a0200 con 0x7f232c09ec50 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2331ab9640 1 -- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c09ec50 msgr2=0x7f232c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2331ab9640 1 --2- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c09ec50 0x7f232c131c60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2328009910 tx=0x7f23280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37544b18a0cbfaa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 1 -- 172.21.15.144:0/162742684 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328032440 con 0x7f232c09ec50 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: dump: 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.047 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: _finish_auth 0 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 1 -- 172.21.15.144:0/162742684 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23280325a0 con 0x7f232c09ec50 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 5 monclient: authenticate success, global_id 54258 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:46.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 4 set_mon_vals no callback set 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 1 -- 172.21.15.144:0/162742684 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328032940 con 0x7f232c09ec50 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2330ab7640 10 monclient: dump: 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 10 monclient: shutdown 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 1 -- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c09ec50 msgr2=0x7f232c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 1 --2- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c09ec50 0x7f232c131c60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2328009910 tx=0x7f23280091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 1 -- 172.21.15.144:0/162742684 shutdown_connections 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.038+0000 7f2333d1d640 1 --2- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09f020 0x7f232c069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 1 --2- 172.21.15.144:0/162742684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c09ec50 0x7f232c131c60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 1 -- 172.21.15.144:0/162742684 >> 172.21.15.144:0/162742684 conn(0x7f232c0a5d40 msgr2=0x7f232c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 1 -- 172.21.15.144:0/162742684 shutdown_connections 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 1 -- 172.21.15.144:0/162742684 wait complete. 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 10 monclient: build_initial_monmap 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 10 monclient: monmap: 2024-09-18T12:28:46.050 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 1 Processor -- start 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 1 -- start start 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.039+0000 7f2333d1d640 10 monclient: init 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:46.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09ec50 0x7f232c11ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f232c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232c09f020 0x7f232c11d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f232c09f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c1321a0 0x7f232c11d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f232c1321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c11e9b0 con 0x7f232c1321a0 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c09c000 con 0x7f232c09f020 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c09dec0 con 0x7f232c09ec50 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:46.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2333d1d640 10 monclient(hunting): authenticate will time out at 961.609440s 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2331ab9640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09ec50 msgr2=0x7f232c11ce30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f2331ab9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09ec50 0x7f232c11ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232c09f020 0x7f232c11d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232c09f020 0x7f232c11d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40930/0 (socket says 172.21.15.144:40930) 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 -- 172.21.15.144:0/3436467698 learned_addr learned my addr 172.21.15.144:0/3436467698 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): get_auth_request con 0x7f232c09f020 auth_method 0 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 1 --2- 172.21.15.144:0/3436467698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c1321a0 0x7f232c11d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:46.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 10 monclient(hunting): get_auth_request con 0x7f232c1321a0 auth_method 0 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23322ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient(hunting): handle_auth_done global_id 5002 payload 824 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 -- 172.21.15.144:0/3436467698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09ec50 msgr2=0x7f232c11ce30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 --2- 172.21.15.144:0/3436467698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f232c09ec50 0x7f232c11ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 -- 172.21.15.144:0/3436467698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c1321a0 msgr2=0x7f232c11d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 --2- 172.21.15.144:0/3436467698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232c1321a0 0x7f232c11d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient: _finish_hunting 0 2024-09-18T12:28:46.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 monclient: found mon.noname-b 2024-09-18T12:28:46.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.040+0000 7f23312b8640 1 -- 172.21.15.144:0/3436467698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2328009450 con 0x7f232c09f020 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f23312b8640 1 -- 172.21.15.144:0/3436467698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232c09f020 msgr2=0x7f232c11d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f23312b8640 1 --2- 172.21.15.144:0/3436467698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232c09f020 0x7f232c11d370 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f231c00c610 tx=0x7f231c007ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 1 -- 172.21.15.144:0/3436467698 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c007510 con 0x7f232c09f020 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:46.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: dump: 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 1 -- 172.21.15.144:0/3436467698 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c0057d0 con 0x7f232c09f020 2024-09-18T12:28:46.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2333d1d640 5 monclient: authenticate success, global_id 5002 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 1 -- 172.21.15.144:0/3436467698 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c0076b0 con 0x7f232c09f020 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2333d1d640 10 monclient: _renew_subs 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2333d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2333d1d640 1 -- 172.21.15.144:0/3436467698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c120450 con 0x7f232c09f020 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2321ffb640 10 monclient: dump: 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.058 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2323fff640 4 set_mon_vals no callback set 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2333d1d640 10 monclient: _renew_subs 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2333d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.043+0000 7f2333d1d640 1 -- 172.21.15.144:0/3436467698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c11f4f0 con 0x7f232c09f020 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2321ffb640 1 -- 172.21.15.144:0/3436467698 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f231c01b400 con 0x7f232c09f020 2024-09-18T12:28:46.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2321ffb640 1 --2- 172.21.15.144:0/3436467698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2310038440 0x7f231003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2333d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:46.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2333d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:46.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2333d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2333d1d640 1 -- 172.21.15.144:0/3436467698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232c11f710 con 0x7f232c09f020 2024-09-18T12:28:46.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2321ffb640 1 -- 172.21.15.144:0/3436467698 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f231c04b4f0 con 0x7f232c09f020 2024-09-18T12:28:46.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2331ab9640 1 --2- 172.21.15.144:0/3436467698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2310038440 0x7f231003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.044+0000 7f2331ab9640 10 monclient: get_auth_request con 0x7f2310038440 auth_method 0 2024-09-18T12:28:46.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.046+0000 7f2331ab9640 1 --2- 172.21.15.144:0/3436467698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2310038440 0x7f231003a8f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f232c0a2b00 tx=0x7f2328032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:46.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.048+0000 7f2321ffb640 1 -- 172.21.15.144:0/3436467698 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f231c005e20 con 0x7f232c09f020 2024-09-18T12:28:46.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.048+0000 7f2321ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:46.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.048+0000 7f2321ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:46.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.204+0000 7f2333d1d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-18T12:28:46.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.204+0000 7f2333d1d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-18T12:28:46.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.204+0000 7f2333d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.204+0000 7f2333d1d640 1 -- 172.21.15.144:0/3436467698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) v1 -- 0x7f232c0008d0 con 0x7f232c09f020 2024-09-18T12:28:46.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.206+0000 7f2321ffb640 1 -- 172.21.15.144:0/3436467698 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v68) v1 ==== 132+0+0 (secure 0 0 0) 0x7f231c0290a0 con 0x7f232c09f020 2024-09-18T12:28:46.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.206+0000 7f2321ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-18T12:28:46.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.206+0000 7f2321ffb640 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-18T12:28:46.207 INFO:tasks.workunit.client.0.smithi144.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-18T12:28:46.210 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:28:46.211 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.284+0000 7f764ed54640 10 monclient: get_monmap_and_config 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.284+0000 7f764ed54640 10 monclient: build_initial_monmap 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.284+0000 7f764ed54640 10 monclient: monmap: 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:46.285826+0000 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:46.285826+0000 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:46.286 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.287 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:46.287 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:46.287 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:46.287 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:46.287 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:46.287 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.285+0000 7f764ed54640 1 Processor -- start 2024-09-18T12:28:46.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.285+0000 7f764ed54640 1 -- start start 2024-09-18T12:28:46.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.285+0000 7f764ed54640 10 monclient: init 2024-09-18T12:28:46.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:46.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76480a0ad0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 0x7f764809f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient(hunting): picked mon.noname-a con 0x7f764809ee60 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7648136e00 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764806a5b0 con 0x7f76480a0ad0 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480a0460 con 0x7f7648136e00 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764809f7a0 con 0x7f764809ee60 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764ed54640 10 monclient(hunting): authenticate will time out at 961.855169s 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764d2f1640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7648136e00 0x7f76481391e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48888/0 (socket says 172.21.15.144:48888) 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f764d2f1640 1 -- 172.21.15.144:0/2461170422 learned_addr learned my addr 172.21.15.144:0/2461170422 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:46.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 1 --2- 172.21.15.144:0/2461170422 >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 0x7f764809f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 10 monclient(hunting): get_auth_request con 0x7f764809ee60 auth_method 0 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 1 -- 172.21.15.144:0/2461170422 >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 msgr2=0x7f764809f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 1 -- 172.21.15.144:0/2461170422 >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 msgr2=0x7f764809f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 1 --2- 172.21.15.144:0/2461170422 >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 0x7f764809f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.286+0000 7f763ffff640 1 --2- 172.21.15.144:0/2461170422 >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 0x7f764809f260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:46.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f764d2f1640 1 -- 172.21.15.144:0/2461170422 >> v1:172.21.15.144:6789/0 conn(0x7f7648136e00 legacy=0x7f76481391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872395892 0 0) 0x7f76480a0460 con 0x7f7648136e00 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): my global_id is 5005 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76300034e0 con 0x7f7648136e00 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f764caf0640 1 -- 172.21.15.144:0/2461170422 >> v1:172.21.15.116:6789/0 conn(0x7f76480a0ad0 legacy=0x7f76480691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679883737 0 0) 0x7f76300034e0 con 0x7f7648136e00 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 msgr2=0x7f764809f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 --2- 172.21.15.144:0/2461170422 >> v2:172.21.15.116:3300/0 conn(0x7f764809ee60 0x7f764809f260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 >> v1:172.21.15.116:6789/0 conn(0x7f76480a0ad0 legacy=0x7f76480691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:28:46.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480a0ea0 con 0x7f7648136e00 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f764ed54640 5 monclient: authenticate success, global_id 5005 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f7640001b20 con 0x7f7648136e00 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: dump: 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.303 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 >> v1:172.21.15.144:6789/0 conn(0x7f7648136e00 legacy=0x7f76481391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7630003bc0 0x7f7630003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): picked mon.b con 0x7f7630003bc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7630004780 0x7f7630004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): picked mon.c con 0x7f7630004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76300099c0 0x7f763000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): picked mon.a con 0x7f76300099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764809f7a0 con 0x7f76300099c0 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763000c360 con 0x7f7630003bc0 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763000c520 con 0x7f7630004780 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f763f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.288+0000 7f764ed54640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7630003bc0 0x7f7630003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f763ffff640 1 -- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7630004780 msgr2=0x7f7630004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): get_auth_request con 0x7f7630003bc0 auth_method 0 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f763ffff640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7630004780 0x7f7630004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764d2f1640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76300099c0 0x7f763000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764d2f1640 10 monclient(hunting): get_auth_request con 0x7f76300099c0 auth_method 0 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764d2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764d2f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764d2f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient(hunting): handle_auth_done global_id 5005 payload 293 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 1 -- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7630004780 msgr2=0x7f7630004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7630004780 0x7f7630004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 1 -- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76300099c0 msgr2=0x7f763000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76300099c0 0x7f763000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient: _finish_hunting 0 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 1 monclient: found mon.b 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 1 -- 172.21.15.144:0/2461170422 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76300034e0 con 0x7f7630003bc0 2024-09-18T12:28:46.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764d2f1640 1 -- 172.21.15.144:0/2461170422 reap_dead start 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 10 monclient: _finish_auth 0 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.289+0000 7f764caf0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.291+0000 7f764caf0640 1 -- 172.21.15.144:0/2461170422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7630003bc0 msgr2=0x7f7630003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.291+0000 7f764caf0640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7630003bc0 0x7f7630003fd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7644001b10 tx=0x7f7644007c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d234c3b9c3833e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764402ab50 con 0x7f7630003bc0 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 10 monclient: dump: 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.308 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764402acb0 con 0x7f7630003bc0 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 1 -- 172.21.15.144:0/2461170422 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644007340 con 0x7f7630003bc0 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 4 set_mon_vals no callback set 2024-09-18T12:28:46.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f763f7fe640 10 monclient: dump: 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 10 monclient: shutdown 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 1 -- 172.21.15.144:0/2461170422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7630003bc0 msgr2=0x7f7630003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7630003bc0 0x7f7630003fd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7644001b10 tx=0x7f7644007c80 comp rx=0 tx=0).stop 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 1 -- 172.21.15.144:0/2461170422 shutdown_connections 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 1 --2- 172.21.15.144:0/2461170422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7630003bc0 0x7f7630003fd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 1 -- 172.21.15.144:0/2461170422 >> 172.21.15.144:0/2461170422 conn(0x7f7648071370 msgr2=0x7f76480a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 1 -- 172.21.15.144:0/2461170422 shutdown_connections 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.292+0000 7f764ed54640 1 -- 172.21.15.144:0/2461170422 wait complete. 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.293+0000 7f764ed54640 10 monclient: build_initial_monmap 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.293+0000 7f764ed54640 10 monclient: monmap: 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:46.311 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.293+0000 7f764ed54640 1 Processor -- start 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.293+0000 7f764ed54640 1 -- start start 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.293+0000 7f764ed54640 10 monclient: init 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:28:46.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7630003bc0 0x7f7648141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): picked mon.noname-a con 0x7f7630003bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 0x7f7648141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): picked mon.noname-b con 0x7f76481417b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f764811e620 0x7f764811ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): picked mon.noname-c con 0x7f764811e620 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648120070 con 0x7f7630003bc0 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480a11e0 con 0x7f76481417b0 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480a4b20 con 0x7f764811e620 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:46.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764ed54640 10 monclient(hunting): authenticate will time out at 961.863238s 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 0x7f7648141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 0x7f7648141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40950/0 (socket says 172.21.15.144:40950) 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 -- 172.21.15.144:0/2964681390 learned_addr learned my addr 172.21.15.144:0/2964681390 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): get_auth_request con 0x7f76481417b0 auth_method 0 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764d2f1640 1 -- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f764811e620 msgr2=0x7f764811ea70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764d2f1640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f764811e620 0x7f764811ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7630003bc0 0x7f7648141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 10 monclient(hunting): get_auth_request con 0x7f7630003bc0 auth_method 0 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f764caf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient(hunting): handle_auth_done global_id 5008 payload 824 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 -- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f764811e620 msgr2=0x7f764811ea70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f764811e620 0x7f764811ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 -- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7630003bc0 msgr2=0x7f7648141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7630003bc0 0x7f7648141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:46.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 monclient: found mon.noname-b 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.294+0000 7f763ffff640 1 -- 172.21.15.144:0/2964681390 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76481420c0 con 0x7f76481417b0 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763ffff640 1 -- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 msgr2=0x7f7648141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763ffff640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 0x7f7648141b80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f763800e9b0 tx=0x7f76380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638009560 con 0x7f76481417b0 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763cff9640 10 monclient: dump: 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.316 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763cff9640 10 monclient: _finish_auth 0 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f764ed54640 5 monclient: authenticate success, global_id 5008 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763800cad0 con 0x7f76481417b0 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f764ed54640 10 monclient: _renew_subs 2024-09-18T12:28:46.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.296+0000 7f764ed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7648122ef0 con 0x7f76481417b0 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f763cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763800ce90 con 0x7f76481417b0 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f763cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f763cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f763cff9640 10 monclient: dump: 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:46.318 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 10 monclient: _renew_subs 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f763effd640 4 set_mon_vals no callback set 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7648120c40 con 0x7f76481417b0 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7638013680 con 0x7f76481417b0 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.297+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76480008d0 con 0x7f76481417b0 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.298+0000 7f763cff9640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7634038440 0x7f763403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.298+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f763804bd80 con 0x7f76481417b0 2024-09-18T12:28:46.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.298+0000 7f764caf0640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7634038440 0x7f763403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:46.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.298+0000 7f764caf0640 10 monclient: get_auth_request con 0x7f7634038440 auth_method 0 2024-09-18T12:28:46.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.300+0000 7f764caf0640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7634038440 0x7f763403a8f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7644007fd0 tx=0x7f7644030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:46.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.302+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f763801ad80 con 0x7f76481417b0 2024-09-18T12:28:46.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.302+0000 7f763cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:46.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.302+0000 7f763cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:46.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.457+0000 7f764ed54640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:46.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.457+0000 7f764ed54640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:46.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.457+0000 7f764ed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.457+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7648121d40 con 0x7f76481417b0 2024-09-18T12:28:46.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.834+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v69) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7638017070 con 0x7f76481417b0 2024-09-18T12:28:46.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.834+0000 7f763cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:46.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.834+0000 7f763cff9640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-18T12:28:46.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.927+0000 7f764ed54640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:46.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.927+0000 7f764ed54640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:46.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.927+0000 7f764ed54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:46.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.927+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7648064a20 con 0x7f76481417b0 2024-09-18T12:28:46.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.929+0000 7f763cff9640 1 -- 172.21.15.144:0/2964681390 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v69) v1 ==== 179+0+0 (secure 0 0 0) 0x7f763801d030 con 0x7f76481417b0 2024-09-18T12:28:46.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.929+0000 7f763cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:28:46.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.929+0000 7f763cff9640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-18T12:28:46.930 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'pinning_pool' does not exist 2024-09-18T12:28:46.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.930+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7634038440 msgr2=0x7f763403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.930+0000 7f764ed54640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7634038440 0x7f763403a8f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7644007fd0 tx=0x7f7644030000 comp rx=0 tx=0).stop 2024-09-18T12:28:46.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.930+0000 7f764ed54640 10 monclient: shutdown 2024-09-18T12:28:46.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.930+0000 7f764ed54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.930+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 msgr2=0x7f7648141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.930+0000 7f764ed54640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 0x7f7648141b80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f763800e9b0 tx=0x7f76380090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 shutdown_connections 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7634038440 0x7f763403a8f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f764811e620 0x7f764811ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76481417b0 0x7f7648141b80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 --2- 172.21.15.144:0/2964681390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7630003bc0 0x7f7648141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 >> 172.21.15.144:0/2964681390 conn(0x7f7648071370 msgr2=0x7f764809ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 shutdown_connections 2024-09-18T12:28:46.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:46.931+0000 7f764ed54640 1 -- 172.21.15.144:0/2964681390 wait complete. 2024-09-18T12:28:46.941 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-18T12:28:46.941 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:28:46.941 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket foo osd 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.014+0000 7fb66575d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.014+0000 7fb66575d640 10 monclient: build_initial_monmap 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.014+0000 7fb66575d640 10 monclient: monmap: 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:47.014965+0000 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:47.014965+0000 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.015 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:47.016 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:47.016 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:47.016 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:47.016 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:47.016 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.014+0000 7fb66575d640 1 Processor -- start 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.014+0000 7fb66575d640 1 -- start start 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.014+0000 7fb66575d640 10 monclient: init 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb66009ec50 0x7fb660131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient(hunting): picked mon.noname-a con 0x7fb66009ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6601321a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb66009f020 0x7fb660069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient(hunting): picked mon.noname-c con 0x7fb66009f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb660067420 con 0x7fb6601321a0 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb66009fdc0 con 0x7fb66009ec50 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6600a0510 con 0x7fb66009f020 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb66575d640 10 monclient(hunting): authenticate will time out at 962.584243s 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb66009f020 0x7fb660069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65e7fc640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb6601321a0 0x7fb660068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:48896/0 (socket says 172.21.15.144:48896) 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65e7fc640 1 -- 172.21.15.144:0/3120875485 learned_addr learned my addr 172.21.15.144:0/3120875485 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): get_auth_request con 0x7fb66009f020 auth_method 0 2024-09-18T12:28:47.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65effd640 1 --2- 172.21.15.144:0/3120875485 >> v2:172.21.15.116:3300/0 conn(0x7fb66009ec50 0x7fb660131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65effd640 10 monclient(hunting): get_auth_request con 0x7fb66009ec50 auth_method 0 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient(hunting): handle_auth_done global_id 5011 payload 274 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 1 -- 172.21.15.144:0/3120875485 >> v2:172.21.15.116:3300/0 conn(0x7fb66009ec50 msgr2=0x7fb660131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:47.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 1 --2- 172.21.15.144:0/3120875485 >> v2:172.21.15.116:3300/0 conn(0x7fb66009ec50 0x7fb660131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 1 -- 172.21.15.144:0/3120875485 >> v1:172.21.15.144:6789/0 conn(0x7fb6601321a0 legacy=0x7fb660068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65f7fe640 1 -- 172.21.15.144:0/3120875485 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6600a0200 con 0x7fb66009f020 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.015+0000 7fb65effd640 1 --2- 172.21.15.144:0/3120875485 >> v2:172.21.15.116:3300/0 conn(0x7fb66009ec50 0x7fb660131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65f7fe640 1 -- 172.21.15.144:0/3120875485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009f020 msgr2=0x7fb660069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65f7fe640 1 --2- 172.21.15.144:0/3120875485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009f020 0x7fb660069410 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb660132a50 tx=0x7fb6580090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43233b81efa3c365 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 1 -- 172.21.15.144:0/3120875485 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65802adf0 con 0x7fb66009f020 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:47.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: dump: 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:47.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 1 -- 172.21.15.144:0/3120875485 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb658032440 con 0x7fb66009f020 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb66575d640 5 monclient: authenticate success, global_id 5011 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb66575d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb66575d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 1 -- 172.21.15.144:0/3120875485 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6580327e0 con 0x7fb66009f020 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb66575d640 4 set_mon_vals no callback set 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.018+0000 7fb65dffb640 10 monclient: dump: 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.026 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 10 monclient: shutdown 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 -- 172.21.15.144:0/3120875485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009f020 msgr2=0x7fb660069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 --2- 172.21.15.144:0/3120875485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009f020 0x7fb660069410 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb660132a50 tx=0x7fb6580090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 -- 172.21.15.144:0/3120875485 shutdown_connections 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 --2- 172.21.15.144:0/3120875485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009f020 0x7fb660069410 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 --2- 172.21.15.144:0/3120875485 >> v2:172.21.15.116:3300/0 conn(0x7fb66009ec50 0x7fb660131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 -- 172.21.15.144:0/3120875485 >> 172.21.15.144:0/3120875485 conn(0x7fb6600a5d40 msgr2=0x7fb6600a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 -- 172.21.15.144:0/3120875485 shutdown_connections 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 1 -- 172.21.15.144:0/3120875485 wait complete. 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 10 monclient: build_initial_monmap 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.019+0000 7fb66575d640 10 monclient: monmap: 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:47.028 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:47.029 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:47.029 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 Processor -- start 2024-09-18T12:28:47.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 -- start start 2024-09-18T12:28:47.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient: init 2024-09-18T12:28:47.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:47.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:47.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009ec50 0x7fb66009a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): picked mon.noname-b con 0x7fb66009ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:47.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66009f020 0x7fb660130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): picked mon.noname-c con 0x7fb66009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6601321a0 0x7fb660131170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6601321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66009c000 con 0x7fb6601321a0 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66009e200 con 0x7fb66009ec50 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66011e030 con 0x7fb66009f020 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb66575d640 10 monclient(hunting): authenticate will time out at 962.589704s 2024-09-18T12:28:47.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009ec50 0x7fb66009a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009ec50 0x7fb66009a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40964/0 (socket says 172.21.15.144:40964) 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 1 -- 172.21.15.144:0/2233223567 learned_addr learned my addr 172.21.15.144:0/2233223567 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 10 monclient(hunting): get_auth_request con 0x7fb66009ec50 auth_method 0 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65e7fc640 1 -- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66009f020 msgr2=0x7fb660130c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65e7fc640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66009f020 0x7fb660130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65f7fe640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6601321a0 0x7fb660131170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65f7fe640 10 monclient(hunting): get_auth_request con 0x7fb6601321a0 auth_method 0 2024-09-18T12:28:47.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.020+0000 7fb65f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 10 monclient(hunting): handle_auth_done global_id 5014 payload 824 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 1 -- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66009f020 msgr2=0x7fb660130c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb66009f020 0x7fb660130c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 1 -- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6601321a0 msgr2=0x7fb660131170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6601321a0 0x7fb660131170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 1 monclient: found mon.noname-b 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65effd640 1 -- 172.21.15.144:0/2233223567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb658009390 con 0x7fb66009ec50 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.021+0000 7fb65f7fe640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6601321a0 0x7fb660131170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb65effd640 1 -- 172.21.15.144:0/2233223567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009ec50 msgr2=0x7fb66009a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb65effd640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb66009ec50 0x7fb66009a9b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb65000d1d0 tx=0x7fb650009a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 1 -- 172.21.15.144:0/2233223567 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650012560 con 0x7fb66009ec50 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: dump: 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.037 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb66575d640 5 monclient: authenticate success, global_id 5014 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 1 -- 172.21.15.144:0/2233223567 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65000f040 con 0x7fb66009ec50 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb66575d640 10 monclient: _renew_subs 2024-09-18T12:28:47.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 1 -- 172.21.15.144:0/2233223567 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500128f0 con 0x7fb66009ec50 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb66575d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb66575d640 1 -- 172.21.15.144:0/2233223567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66011c6f0 con 0x7fb66009ec50 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb646ffd640 10 monclient: dump: 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.039 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb65d7fa640 4 set_mon_vals no callback set 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb66575d640 10 monclient: _renew_subs 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb66575d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.023+0000 7fb66575d640 1 -- 172.21.15.144:0/2233223567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66011cdf0 con 0x7fb66009ec50 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb646ffd640 1 -- 172.21.15.144:0/2233223567 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb650012d20 con 0x7fb66009ec50 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb66575d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb66575d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb66575d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb66575d640 1 -- 172.21.15.144:0/2233223567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66011e270 con 0x7fb66009ec50 2024-09-18T12:28:47.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb646ffd640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb64c038440 0x7fb64c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb646ffd640 1 -- 172.21.15.144:0/2233223567 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5031+0+0 (secure 0 0 0) 0x7fb65004be70 con 0x7fb66009ec50 2024-09-18T12:28:47.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb65e7fc640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb64c038440 0x7fb64c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.024+0000 7fb65e7fc640 10 monclient: get_auth_request con 0x7fb64c038440 auth_method 0 2024-09-18T12:28:47.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.027+0000 7fb65e7fc640 1 --2- 172.21.15.144:0/2233223567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb64c038440 0x7fb64c03a8f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb66004c190 tx=0x7fb654007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:47.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.029+0000 7fb646ffd640 1 -- 172.21.15.144:0/2233223567 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb6500112b0 con 0x7fb66009ec50 2024-09-18T12:28:47.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.029+0000 7fb646ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:47.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.029+0000 7fb646ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.186+0000 7fb66575d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-18T12:28:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.186+0000 7fb66575d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-18T12:28:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.186+0000 7fb66575d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.186+0000 7fb66575d640 1 -- 172.21.15.144:0/2233223567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) v1 -- 0x7fb6600008d0 con 0x7fb66009ec50 2024-09-18T12:28:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.188+0000 7fb646ffd640 1 -- 172.21.15.144:0/2233223567 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v69) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb65001a070 con 0x7fb66009ec50 2024-09-18T12:28:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.188+0000 7fb646ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-18T12:28:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.188+0000 7fb646ffd640 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-18T12:28:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-18T12:28:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:28:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket foo root 2024-09-18T12:28:47.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.265+0000 7f33994c0640 10 monclient: get_monmap_and_config 2024-09-18T12:28:47.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.265+0000 7f33994c0640 10 monclient: build_initial_monmap 2024-09-18T12:28:47.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.265+0000 7f33994c0640 10 monclient: monmap: 2024-09-18T12:28:47.266 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:47.266 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:47.266180+0000 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:47.266180+0000 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:47.267 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.265+0000 7f33994c0640 1 Processor -- start 2024-09-18T12:28:47.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 1 -- start start 2024-09-18T12:28:47.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient: init 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f339409ec50 0x7f3394131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient(hunting): picked mon.noname-a con 0x7f339409ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33941321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f339409f020 0x7f3394069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient(hunting): picked mon.noname-c con 0x7f339409f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3394067420 con 0x7f33941321a0 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339409fdc0 con 0x7f339409ec50 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33940a0510 con 0x7f339409f020 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33994c0640 10 monclient(hunting): authenticate will time out at 962.835565s 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f339409f020 0x7f3394069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f339409f020 0x7f3394069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48646/0 (socket says 172.21.15.144:48646) 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 1 -- 172.21.15.144:0/2709034917 learned_addr learned my addr 172.21.15.144:0/2709034917 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): get_auth_request con 0x7f339409f020 auth_method 0 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f3392ffd640 1 --2- 172.21.15.144:0/2709034917 >> v2:172.21.15.116:3300/0 conn(0x7f339409ec50 0x7f3394131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f3392ffd640 10 monclient(hunting): get_auth_request con 0x7f339409ec50 auth_method 0 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f3392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f3392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f3392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient(hunting): handle_auth_done global_id 5017 payload 274 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 1 -- 172.21.15.144:0/2709034917 >> v2:172.21.15.116:3300/0 conn(0x7f339409ec50 msgr2=0x7f3394131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 1 --2- 172.21.15.144:0/2709034917 >> v2:172.21.15.116:3300/0 conn(0x7f339409ec50 0x7f3394131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 1 -- 172.21.15.144:0/2709034917 >> v1:172.21.15.116:6789/0 conn(0x7f33941321a0 legacy=0x7f3394068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 1 monclient: found mon.noname-c 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.266+0000 7f33937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.267+0000 7f33937fe640 1 -- 172.21.15.144:0/2709034917 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33940a0200 con 0x7f339409f020 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33937fe640 1 -- 172.21.15.144:0/2709034917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339409f020 msgr2=0x7f3394069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33937fe640 1 --2- 172.21.15.144:0/2709034917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339409f020 0x7f3394069410 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3394132a50 tx=0x7f338c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef345e6af177d36e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 1 -- 172.21.15.144:0/2709034917 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c007610 con 0x7f339409f020 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: dump: 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.278 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33994c0640 5 monclient: authenticate success, global_id 5017 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 1 -- 172.21.15.144:0/2709034917 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c02d050 con 0x7f339409f020 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33994c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:47.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33994c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 1 -- 172.21.15.144:0/2709034917 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c0079a0 con 0x7f339409f020 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33994c0640 4 set_mon_vals no callback set 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.269+0000 7f33927fc640 10 monclient: dump: 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.280 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 10 monclient: shutdown 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 -- 172.21.15.144:0/2709034917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339409f020 msgr2=0x7f3394069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 --2- 172.21.15.144:0/2709034917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339409f020 0x7f3394069410 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3394132a50 tx=0x7f338c009840 comp rx=0 tx=0).stop 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 -- 172.21.15.144:0/2709034917 shutdown_connections 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 --2- 172.21.15.144:0/2709034917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f339409f020 0x7f3394069410 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 --2- 172.21.15.144:0/2709034917 >> v2:172.21.15.116:3300/0 conn(0x7f339409ec50 0x7f3394131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 -- 172.21.15.144:0/2709034917 >> 172.21.15.144:0/2709034917 conn(0x7f33940a5d40 msgr2=0x7f33940a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 -- 172.21.15.144:0/2709034917 shutdown_connections 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.270+0000 7f33994c0640 1 -- 172.21.15.144:0/2709034917 wait complete. 2024-09-18T12:28:47.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient: build_initial_monmap 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient: monmap: 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 Processor -- start 2024-09-18T12:28:47.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 -- start start 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient: init 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f339409ec50 0x7f33941309a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): picked mon.noname-a con 0x7f339409ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f339409f020 0x7f339411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): picked mon.noname-c con 0x7f339409f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 0x7f339411bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): picked mon.noname-b con 0x7f33941321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339409c000 con 0x7f339409ec50 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339411f090 con 0x7f33941321a0 2024-09-18T12:28:47.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339411edb0 con 0x7f339409f020 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33994c0640 10 monclient(hunting): authenticate will time out at 962.840816s 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f338bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f339409f020 msgr2=0x7f339411b760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f338bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f339409f020 0x7f339411b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33937fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 0x7f339411bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33937fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 0x7f339411bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48652/0 (socket says 172.21.15.144:48652) 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33937fe640 1 -- 172.21.15.144:0/3125711342 learned_addr learned my addr 172.21.15.144:0/3125711342 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f3392ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f339409ec50 0x7f33941309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33937fe640 10 monclient(hunting): get_auth_request con 0x7f33941321a0 auth_method 0 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33937fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f33937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f3392ffd640 10 monclient(hunting): get_auth_request con 0x7f339409ec50 auth_method 0 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f3392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f3392ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.271+0000 7f3392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f3392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f3392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f3392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 10 monclient(hunting): handle_auth_done global_id 5020 payload 824 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 1 -- 172.21.15.144:0/3125711342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f339409f020 msgr2=0x7f339411b760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:47.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f339409f020 0x7f339411b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 1 -- 172.21.15.144:0/3125711342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f339409ec50 msgr2=0x7f33941309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f339409ec50 0x7f33941309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 1 monclient: found mon.noname-b 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.272+0000 7f33937fe640 1 -- 172.21.15.144:0/3125711342 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c009ae0 con 0x7f33941321a0 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33937fe640 1 -- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 msgr2=0x7f339411bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33937fe640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 0x7f339411bca0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f338c009ab0 tx=0x7f338c036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c032c60 con 0x7f33941321a0 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: dump: 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: _finish_auth 0 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:47.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33994c0640 5 monclient: authenticate success, global_id 5020 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33994c0640 10 monclient: _renew_subs 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33994c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339411ead0 con 0x7f33941321a0 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c032dc0 con 0x7f33941321a0 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c03b640 con 0x7f33941321a0 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f338affd640 10 monclient: dump: 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.288 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33994c0640 10 monclient: _renew_subs 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33994c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f3391ffb640 4 set_mon_vals no callback set 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.274+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3394141ba0 con 0x7f33941321a0 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f338c003020 con 0x7f33941321a0 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f33994c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:47.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f33994c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:47.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f33994c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339411f1d0 con 0x7f33941321a0 2024-09-18T12:28:47.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f338affd640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3374038440 0x7f337403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:47.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f338c06c310 con 0x7f33941321a0 2024-09-18T12:28:47.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f3392ffd640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3374038440 0x7f337403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:47.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.275+0000 7f3392ffd640 10 monclient: get_auth_request con 0x7f3374038440 auth_method 0 2024-09-18T12:28:47.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.278+0000 7f3392ffd640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3374038440 0x7f337403a8f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3384007ba0 tx=0x7f3384008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:47.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.279+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f339411f1d0 con 0x7f33941321a0 2024-09-18T12:28:47.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.279+0000 7f338affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:47.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.280+0000 7f338affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:47.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.436+0000 7f33994c0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T12:28:47.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.436+0000 7f33994c0640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T12:28:47.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.436+0000 7f33994c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.436+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f339411f1d0 con 0x7f33941321a0 2024-09-18T12:28:47.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.856+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v70) v1 ==== 138+0+0 (secure 0 0 0) 0x7f339411f1d0 con 0x7f33941321a0 2024-09-18T12:28:47.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.856+0000 7f338affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T12:28:47.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.856+0000 7f338affd640 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-18T12:28:47.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.953+0000 7f33994c0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T12:28:47.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.953+0000 7f33994c0640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T12:28:47.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.953+0000 7f33994c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:47.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.953+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7f339411f1d0 con 0x7f33941321a0 2024-09-18T12:28:47.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.955+0000 7f338affd640 1 -- 172.21.15.144:0/3125711342 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v70) v1 ==== 126+0+0 (secure 0 0 0) 0x7f339411f1d0 con 0x7f33941321a0 2024-09-18T12:28:47.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.955+0000 7f338affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T12:28:47.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.955+0000 7f338affd640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-18T12:28:47.956 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'foo' already exists 2024-09-18T12:28:47.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3374038440 msgr2=0x7f337403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:47.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3374038440 0x7f337403a8f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3384007ba0 tx=0x7f3384008040 comp rx=0 tx=0).stop 2024-09-18T12:28:47.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 10 monclient: shutdown 2024-09-18T12:28:47.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 msgr2=0x7f339411bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 0x7f339411bca0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f338c009ab0 tx=0x7f338c036000 comp rx=0 tx=0).stop 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 shutdown_connections 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3374038440 0x7f337403a8f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f33941321a0 0x7f339411bca0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f339409f020 0x7f339411b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 --2- 172.21.15.144:0/3125711342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f339409ec50 0x7f33941309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 >> 172.21.15.144:0/3125711342 conn(0x7f33940a5d40 msgr2=0x7f33940a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 shutdown_connections 2024-09-18T12:28:47.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:47.957+0000 7f33994c0640 1 -- 172.21.15.144:0/3125711342 wait complete. 2024-09-18T12:28:47.967 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 2024-09-18T12:28:48.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.037+0000 7fa2b22c0640 10 monclient: get_monmap_and_config 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.037+0000 7fa2b22c0640 10 monclient: build_initial_monmap 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.037+0000 7fa2b22c0640 10 monclient: monmap: 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:48.038556+0000 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:48.038556+0000 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:48.039 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.038+0000 7fa2b22c0640 1 Processor -- start 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.038+0000 7fa2b22c0640 1 -- start start 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.038+0000 7fa2b22c0640 10 monclient: init 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2ac069a30 0x7fa2ac1320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa2ac069a30 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa2ac070920 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2ac071230 0x7fa2ac12e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:48.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa2ac071230 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ac09e560 con 0x7fa2ac069a30 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ac09cc60 con 0x7fa2ac071230 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac09d0a0 con 0x7fa2ac070920 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b22c0640 10 monclient(hunting): authenticate will time out at 963.608157s 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2a3fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 msgr2=0x7fa2ac070cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2a3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac070cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2ac071230 0x7fa2ac12e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa2ac071230 0x7fa2ac12e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48662/0 (socket says 172.21.15.144:48662) 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 -- 172.21.15.144:0/2957150896 learned_addr learned my addr 172.21.15.144:0/2957150896 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:48.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): get_auth_request con 0x7fa2ac071230 auth_method 0 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2abfff640 1 --2- 172.21.15.144:0/2957150896 >> v2:172.21.15.116:3300/0 conn(0x7fa2ac069a30 0x7fa2ac1320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2abfff640 10 monclient(hunting): get_auth_request con 0x7fa2ac069a30 auth_method 0 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2abfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient(hunting): handle_auth_done global_id 5023 payload 274 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 -- 172.21.15.144:0/2957150896 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 msgr2=0x7fa2ac070cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 --2- 172.21.15.144:0/2957150896 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac070cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:48.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 -- 172.21.15.144:0/2957150896 >> v2:172.21.15.116:3300/0 conn(0x7fa2ac069a30 msgr2=0x7fa2ac1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:48.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 --2- 172.21.15.144:0/2957150896 >> v2:172.21.15.116:3300/0 conn(0x7fa2ac069a30 0x7fa2ac1320d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:48.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:48.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 monclient: found mon.noname-c 2024-09-18T12:28:48.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:48.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:48.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2b085d640 1 -- 172.21.15.144:0/2957150896 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac09e280 con 0x7fa2ac071230 2024-09-18T12:28:48.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.039+0000 7fa2abfff640 1 --2- 172.21.15.144:0/2957150896 >> v2:172.21.15.116:3300/0 conn(0x7fa2ac069a30 0x7fa2ac1320d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:48.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2b085d640 1 -- 172.21.15.144:0/2957150896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac071230 msgr2=0x7fa2ac12e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:48.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2b085d640 1 --2- 172.21.15.144:0/2957150896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac071230 0x7fa2ac12e740 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac132bb0 tx=0x7fa2a4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc9ff0c2fe72c3c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:48.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 1 -- 172.21.15.144:0/2957150896 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4007610 con 0x7fa2ac071230 2024-09-18T12:28:48.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: dump: 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:48.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 1 -- 172.21.15.144:0/2957150896 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a402d050 con 0x7fa2ac071230 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2b22c0640 5 monclient: authenticate success, global_id 5023 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2b22c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 1 -- 172.21.15.144:0/2957150896 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a40079a0 con 0x7fa2ac071230 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2ab7fe640 10 monclient: dump: 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.053 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2b22c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.042+0000 7fa2b22c0640 4 set_mon_vals no callback set 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 10 monclient: shutdown 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2957150896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac071230 msgr2=0x7fa2ac12e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2957150896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac071230 0x7fa2ac12e740 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac132bb0 tx=0x7fa2a4009840 comp rx=0 tx=0).stop 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2957150896 shutdown_connections 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2957150896 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac071230 0x7fa2ac12e740 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:48.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2957150896 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac070cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2957150896 >> v2:172.21.15.116:3300/0 conn(0x7fa2ac069a30 0x7fa2ac1320d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2957150896 >> 172.21.15.144:0/2957150896 conn(0x7fa2ac09ea70 msgr2=0x7fa2ac09ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2957150896 shutdown_connections 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2957150896 wait complete. 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 10 monclient: build_initial_monmap 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 10 monclient: monmap: 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:48.055 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 Processor -- start 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.043+0000 7fa2b22c0640 1 -- start start 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient: init 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 0x7fa2ac12ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa2ac069a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac11ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa2ac070920 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2ac071230 0x7fa2ac11cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa2ac071230 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:48.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac11edd0 con 0x7fa2ac071230 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac09d5d0 con 0x7fa2ac069a30 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac0973a0 con 0x7fa2ac070920 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b22c0640 10 monclient(hunting): authenticate will time out at 963.613424s 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 0x7fa2ac12ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 0x7fa2ac12ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48664/0 (socket says 172.21.15.144:48664) 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 -- 172.21.15.144:0/2710497909 learned_addr learned my addr 172.21.15.144:0/2710497909 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): get_auth_request con 0x7fa2ac069a30 auth_method 0 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:48.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2a3fff640 1 -- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 msgr2=0x7fa2ac11ca90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2a3fff640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac11ca90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient(hunting): handle_auth_done global_id 5026 payload 824 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 -- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 msgr2=0x7fa2ac11ca90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac11ca90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 -- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2ac071230 msgr2=0x7fa2ac11cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2b085d640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2ac071230 0x7fa2ac11cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2ac071230 0x7fa2ac11cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:48.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 monclient: found mon.noname-b 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.044+0000 7fa2abfff640 1 -- 172.21.15.144:0/2710497909 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a4009ae0 con 0x7fa2ac069a30 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2abfff640 1 -- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 msgr2=0x7fa2ac12ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2abfff640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 0x7fa2ac12ffb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00d890 tx=0x7fa29c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2abfff640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 0x7fa2ac12ffb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00d890 tx=0x7fa29c008040 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fa29c00de80 mon_map magic: 0 v1 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c00de80 con 0x7fa2ac069a30 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 10 monclient: dump: 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.059 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 10 monclient: _finish_auth 0 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:48.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c00bae0 con 0x7fa2ac069a30 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2b22c0640 5 monclient: authenticate success, global_id 5026 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2b22c0640 10 monclient: _renew_subs 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2b22c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac120100 con 0x7fa2ac069a30 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2aaffd640 4 set_mon_vals no callback set 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2b22c0640 10 monclient: _renew_subs 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2b22c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.047+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac11e5d0 con 0x7fa2ac069a30 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c00bea0 con 0x7fa2ac069a30 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:48.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a8ff9640 10 monclient: dump: 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa29c014640 con 0x7fa2ac069a30 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2b22c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2b22c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:48.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2b22c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:48.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac0636e0 con 0x7fa2ac069a30 2024-09-18T12:28:48.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a8ff9640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa28c038490 0x7fa28c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:48.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a3fff640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa28c038490 0x7fa28c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:48.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.048+0000 7fa2a3fff640 10 monclient: get_auth_request con 0x7fa28c038490 auth_method 0 2024-09-18T12:28:48.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.051+0000 7fa2a3fff640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa28c038490 0x7fa28c03a940 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa2940038f0 tx=0x7fa294006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:48.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.052+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5058+0+0 (secure 0 0 0) 0x7fa29c04c080 con 0x7fa2ac069a30 2024-09-18T12:28:48.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.670+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa29c011570 con 0x7fa2ac069a30 2024-09-18T12:28:48.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.670+0000 7fa2a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:48.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.670+0000 7fa2a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:48.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.835+0000 7fa2b22c0640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:28:48.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.835+0000 7fa2b22c0640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T12:28:48.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.835+0000 7fa2b22c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:48.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.835+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa2ac120440 con 0x7fa2ac069a30 2024-09-18T12:28:48.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.866+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v71) v1 ==== 58+0+2 (secure 0 0 0) 0x7fa2ac120440 con 0x7fa2ac069a30 2024-09-18T12:28:48.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.866+0000 7fa2a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T12:28:48.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.866+0000 7fa2a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:48.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.969+0000 7fa2b22c0640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:28:48.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.969+0000 7fa2b22c0640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T12:28:48.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.969+0000 7fa2b22c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:48.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:48.969+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa2ac120440 con 0x7fa2ac069a30 2024-09-18T12:28:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.871+0000 7fa2a8ff9640 1 -- 172.21.15.144:0/2710497909 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v72) v1 ==== 58+0+2 (secure 0 0 0) 0x7fa2ac120440 con 0x7fa2ac069a30 2024-09-18T12:28:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.871+0000 7fa2a8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T12:28:49.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.871+0000 7fa2a8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa28c038490 msgr2=0x7fa28c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa28c038490 0x7fa28c03a940 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa2940038f0 tx=0x7fa294006d80 comp rx=0 tx=0).stop 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 10 monclient: shutdown 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 msgr2=0x7fa2ac12ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 0x7fa2ac12ffb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00d890 tx=0x7fa29c008040 comp rx=0 tx=0).stop 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 shutdown_connections 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa28c038490 0x7fa28c03a940 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2ac071230 0x7fa2ac11cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2ac070920 0x7fa2ac11ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 --2- 172.21.15.144:0/2710497909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2ac069a30 0x7fa2ac12ffb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 >> 172.21.15.144:0/2710497909 conn(0x7fa2ac09ea70 msgr2=0x7fa2ac0a0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:49.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 shutdown_connections 2024-09-18T12:28:49.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.874+0000 7fa2b22c0640 1 -- 172.21.15.144:0/2710497909 wait complete. 2024-09-18T12:28:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:+ o1=7 2024-09-18T12:28:49.887 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 2024-09-18T12:28:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.962+0000 7f53a9280640 10 monclient: get_monmap_and_config 2024-09-18T12:28:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.962+0000 7f53a9280640 10 monclient: build_initial_monmap 2024-09-18T12:28:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.962+0000 7f53a9280640 10 monclient: monmap: 2024-09-18T12:28:49.963 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:49.963362+0000 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:49.963362+0000 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:49.964 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 1 Processor -- start 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 1 -- start start 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient: init 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 0x7f53a409ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient(hunting): picked mon.noname-b con 0x7f53a4131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53a409f240 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:49.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53a4136e70 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a406a230 con 0x7f53a409f240 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a40a0300 con 0x7f53a4136e70 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40a0130 con 0x7f53a4131b40 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a9280640 10 monclient(hunting): authenticate will time out at 965.532595s 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a2ffd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 msgr2=0x7f53a409ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a37fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f53a4136e70 0x7f53a4139250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:53516/0 (socket says 172.21.15.144:53516) 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a37fe640 1 -- 172.21.15.144:0/4033673449 learned_addr learned my addr 172.21.15.144:0/4033673449 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:49.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.963+0000 7f53a2ffd640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 0x7f53a409ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:49.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 -- 172.21.15.144:0/4033673449 >> v1:172.21.15.144:6789/0 conn(0x7f53a4136e70 legacy=0x7f53a4139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a27fc640 1 -- 172.21.15.144:0/4033673449 >> v1:172.21.15.116:6789/0 conn(0x7f53a409f240 legacy=0x7f53a4131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815483800 0 0) 0x7f53a40a0300 con 0x7f53a4136e70 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): my global_id is 5029 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f538c0034e0 con 0x7f53a4136e70 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265570185 0 0) 0x7f53a406a230 con 0x7f53a409f240 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): my global_id is 54288 2024-09-18T12:28:49.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a40a0300 con 0x7f53a409f240 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376035527 0 0) 0x7f538c0034e0 con 0x7f53a4136e70 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 msgr2=0x7f53a409ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 0x7f53a409ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 >> v1:172.21.15.116:6789/0 conn(0x7f53a409f240 legacy=0x7f53a4131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a409bee0 con 0x7f53a4136e70 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f539c001b20 con 0x7f53a4136e70 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:49.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: dump: 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:49.980 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 >> v1:172.21.15.144:6789/0 conn(0x7f53a4136e70 legacy=0x7f53a4139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c004c60 0x7f538c005050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): picked mon.a con 0x7f538c004c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 0x7f538c003720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): picked mon.c con 0x7f53a4131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f538c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): picked mon.b con 0x7f538c003e60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:49.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40a0130 con 0x7f538c004c60 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0045e0 con 0x7f538c003e60 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c00bf90 con 0x7f53a4131b40 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a27fc640 1 -- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 msgr2=0x7f538c003720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a27fc640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 0x7f538c003720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f538c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): get_auth_request con 0x7f538c003e60 auth_method 0 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:49.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:49.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:49.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:49.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:49.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:28:49.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a2ffd640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c004c60 0x7f538c005050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:49.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request con 0x7f538c004c60 auth_method 0 2024-09-18T12:28:49.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient(hunting): handle_auth_done global_id 5029 payload 293 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 -- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 msgr2=0x7f538c003720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4131b40 0x7f538c003720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 -- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c004c60 msgr2=0x7f538c005050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c004c60 0x7f538c005050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 monclient: found mon.b 2024-09-18T12:28:49.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 1 -- 172.21.15.144:0/4033673449 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c0034e0 con 0x7f538c003e60 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a9280640 5 monclient: authenticate success, global_id 5029 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.966+0000 7f53a9280640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a37fe640 1 -- 172.21.15.144:0/4033673449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 msgr2=0x7f538c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a37fe640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f538c00b9d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f539c001850 tx=0x7f539c00a200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df193e80e84b899f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c02b600 con 0x7f538c003e60 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:49.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 10 monclient: dump: 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c02b7a0 con 0x7f538c003e60 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a9280640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:49.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 1 -- 172.21.15.144:0/4033673449 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c02bb60 con 0x7f538c003e60 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a9280640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a9280640 4 set_mon_vals no callback set 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.969+0000 7f53a1ffb640 10 monclient: dump: 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:49.987 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 10 monclient: shutdown 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 1 -- 172.21.15.144:0/4033673449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 msgr2=0x7f538c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 1 --2- 172.21.15.144:0/4033673449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f538c00b9d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f539c001850 tx=0x7f539c00a200 comp rx=0 tx=0).stop 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a37fe640 1 -- 172.21.15.144:0/4033673449 reap_dead start 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 1 -- 172.21.15.144:0/4033673449 shutdown_connections 2024-09-18T12:28:49.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 1 -- 172.21.15.144:0/4033673449 >> 172.21.15.144:0/4033673449 conn(0x7f53a40a5100 msgr2=0x7f53a40a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 1 -- 172.21.15.144:0/4033673449 shutdown_connections 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.970+0000 7f53a9280640 1 -- 172.21.15.144:0/4033673449 wait complete. 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient: build_initial_monmap 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient: monmap: 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:49.989 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 Processor -- start 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 -- start start 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient: init 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f53a4141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): picked mon.noname-b con 0x7f538c003e60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53a411bb40 0x7f53a4141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): picked mon.noname-a con 0x7f53a411bb40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4097910 0x7f53a4097db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): picked mon.noname-c con 0x7f53a4097910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:49.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a409b540 con 0x7f53a411bb40 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40a0a30 con 0x7f538c003e60 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a409df20 con 0x7f53a4097910 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.971+0000 7f53a9280640 10 monclient(hunting): authenticate will time out at 965.540957s 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f53a4141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f53a4141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48686/0 (socket says 172.21.15.144:48686) 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 -- 172.21.15.144:0/4233787153 learned_addr learned my addr 172.21.15.144:0/4233787153 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request con 0x7f538c003e60 auth_method 0 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a37fe640 1 -- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4097910 msgr2=0x7f53a4097db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:49.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a37fe640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4097910 0x7f53a4097db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient(hunting): handle_auth_done global_id 5032 payload 824 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a27fc640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53a411bb40 0x7f53a4141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 -- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4097910 msgr2=0x7f53a4097db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4097910 0x7f53a4097db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 -- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53a411bb40 msgr2=0x7f53a4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53a411bb40 0x7f53a4141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 monclient: found mon.noname-b 2024-09-18T12:28:49.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.972+0000 7f53a2ffd640 1 -- 172.21.15.144:0/4233787153 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40998d0 con 0x7f538c003e60 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a2ffd640 1 -- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 msgr2=0x7f53a4141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a2ffd640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f53a4141560 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f539400d860 tx=0x7f5394008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394003b80 con 0x7f538c003e60 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: dump: 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.993 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: _finish_auth 0 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a9280640 5 monclient: authenticate success, global_id 5032 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394003ce0 con 0x7f538c003e60 2024-09-18T12:28:49.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a9280640 10 monclient: _renew_subs 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a9280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a40993c0 con 0x7f538c003e60 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394005320 con 0x7f538c003e60 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f538affd640 10 monclient: dump: 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:49.995 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a9280640 10 monclient: _renew_subs 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a9280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a4099f90 con 0x7f538c003e60 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.974+0000 7f53a17fa640 4 set_mon_vals no callback set 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5394005690 con 0x7f538c003e60 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f53a9280640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f53a9280640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f53a9280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a40636e0 con 0x7f538c003e60 2024-09-18T12:28:49.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f538affd640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5390038470 0x7f539003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:49.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f539404b8a0 con 0x7f538c003e60 2024-09-18T12:28:49.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f53a27fc640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5390038470 0x7f539003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:49.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.975+0000 7f53a27fc640 10 monclient: get_auth_request con 0x7f5390038470 auth_method 0 2024-09-18T12:28:49.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.978+0000 7f53a27fc640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5390038470 0x7f539003a920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f53a4098f10 tx=0x7f5398009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:49.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.980+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f539401b400 con 0x7f538c003e60 2024-09-18T12:28:49.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.980+0000 7f538affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:49.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:49.980+0000 7f538affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:50.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.148+0000 7f53a9280640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:28:50.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.148+0000 7f53a9280640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T12:28:50.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.148+0000 7f53a9280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:50.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.148+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f53a4134290 con 0x7f538c003e60 2024-09-18T12:28:50.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.878+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v73) v1 ==== 58+0+2 (secure 0 0 0) 0x7f5394016070 con 0x7f538c003e60 2024-09-18T12:28:50.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.878+0000 7f538affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T12:28:50.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.878+0000 7f538affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:28:50.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.985+0000 7f53a9280640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:28:50.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.985+0000 7f53a9280640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T12:28:50.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.985+0000 7f53a9280640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:50.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:50.985+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f53a4072640 con 0x7f538c003e60 2024-09-18T12:28:51.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.878+0000 7f538affd640 1 -- 172.21.15.144:0/4233787153 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v74) v1 ==== 58+0+2 (secure 0 0 0) 0x7f5394028020 con 0x7f538c003e60 2024-09-18T12:28:51.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.878+0000 7f538affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T12:28:51.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.878+0000 7f538affd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:28:51.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5390038470 msgr2=0x7f539003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:51.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5390038470 0x7f539003a920 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f53a4098f10 tx=0x7f5398009040 comp rx=0 tx=0).stop 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 10 monclient: shutdown 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 msgr2=0x7f53a4141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f53a4141560 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f539400d860 tx=0x7f5394008040 comp rx=0 tx=0).stop 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 shutdown_connections 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5390038470 0x7f539003a920 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f53a4097910 0x7f53a4097db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53a411bb40 0x7f53a4141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 --2- 172.21.15.144:0/4233787153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c003e60 0x7f53a4141560 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 >> 172.21.15.144:0/4233787153 conn(0x7f53a40a5100 msgr2=0x7f53a41308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 shutdown_connections 2024-09-18T12:28:51.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.881+0000 7f53a9280640 1 -- 172.21.15.144:0/4233787153 wait complete. 2024-09-18T12:28:51.894 INFO:tasks.workunit.client.0.smithi144.stderr:+ o2=9 2024-09-18T12:28:51.894 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-18T12:28:51.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.972+0000 7f7f9ba80640 10 monclient: get_monmap_and_config 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.972+0000 7f7f9ba80640 10 monclient: build_initial_monmap 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.973+0000 7f7f9ba80640 10 monclient: monmap: 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:51.973913+0000 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:51.973913+0000 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:51.975 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.973+0000 7f7f9ba80640 1 Processor -- start 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.973+0000 7f7f9ba80640 1 -- start start 2024-09-18T12:28:51.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.973+0000 7f7f9ba80640 10 monclient: init 2024-09-18T12:28:51.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f9409cad0 0x7f7f9409c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient(hunting): picked mon.noname-c con 0x7f7f9409cad0 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f9409cea0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f9409d2a0 0x7f7f9409e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9409d2a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9409b710 con 0x7f7f9409cea0 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9409c6e0 con 0x7f7f9409d2a0 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:51.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9409a650 con 0x7f7f9409cad0 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9ba80640 10 monclient(hunting): authenticate will time out at 967.543352s 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f9409cad0 0x7f7f9409c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9901b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7f9409cea0 0x7f7f9409dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:53526/0 (socket says 172.21.15.144:53526) 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f9409cad0 0x7f7f9409c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48690/0 (socket says 172.21.15.144:48690) 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9901b640 1 -- 172.21.15.144:0/232411235 learned_addr learned my addr 172.21.15.144:0/232411235 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): get_auth_request con 0x7f7f9409cad0 auth_method 0 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:51.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:51.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:51.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:51.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient(hunting): handle_auth_done global_id 5035 payload 274 2024-09-18T12:28:51.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 1 -- 172.21.15.144:0/232411235 >> v2:172.21.15.116:3300/0 conn(0x7f7f9409d2a0 msgr2=0x7f7f9409e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:51.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9a01d640 1 --2- 172.21.15.144:0/232411235 >> v2:172.21.15.116:3300/0 conn(0x7f7f9409d2a0 0x7f7f9409e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:51.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 1 --2- 172.21.15.144:0/232411235 >> v2:172.21.15.116:3300/0 conn(0x7f7f9409d2a0 0x7f7f9409e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 1 -- 172.21.15.144:0/232411235 >> v1:172.21.15.144:6789/0 conn(0x7f7f9409cea0 legacy=0x7f7f9409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 1 monclient: found mon.noname-c 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.974+0000 7f7f9981c640 1 -- 172.21.15.144:0/232411235 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9409f660 con 0x7f7f9409cad0 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9981c640 1 -- 172.21.15.144:0/232411235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409cad0 msgr2=0x7f7f9409c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9981c640 1 --2- 172.21.15.144:0/232411235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409cad0 0x7f7f9409c1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7f90009850 tx=0x7f7f900090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=680457bcca2884ed server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 1 -- 172.21.15.144:0/232411235 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90032440 con 0x7f7f9409cad0 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:51.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: dump: 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: _finish_auth 0 2024-09-18T12:28:51.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:51.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 1 -- 172.21.15.144:0/232411235 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f900325a0 con 0x7f7f9409cad0 2024-09-18T12:28:51.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9ba80640 5 monclient: authenticate success, global_id 5035 2024-09-18T12:28:51.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:51.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9ba80640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9ba80640 4 set_mon_vals no callback set 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 1 -- 172.21.15.144:0/232411235 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90032940 con 0x7f7f9409cad0 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.977+0000 7f7f9881a640 10 monclient: dump: 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:51.989 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 10 monclient: shutdown 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 -- 172.21.15.144:0/232411235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409cad0 msgr2=0x7f7f9409c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/232411235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409cad0 0x7f7f9409c1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7f90009850 tx=0x7f7f900090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 -- 172.21.15.144:0/232411235 shutdown_connections 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/232411235 >> v2:172.21.15.116:3300/0 conn(0x7f7f9409d2a0 0x7f7f9409e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/232411235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409cad0 0x7f7f9409c1a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 -- 172.21.15.144:0/232411235 >> 172.21.15.144:0/232411235 conn(0x7f7f94070e90 msgr2=0x7f7f940a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 -- 172.21.15.144:0/232411235 shutdown_connections 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 1 -- 172.21.15.144:0/232411235 wait complete. 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 10 monclient: build_initial_monmap 2024-09-18T12:28:51.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.978+0000 7f7f9ba80640 10 monclient: monmap: 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 Processor -- start 2024-09-18T12:28:51.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 -- start start 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient: init 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f9409cad0 0x7f7f9413bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): picked mon.noname-c con 0x7f7f9409cad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9409cea0 0x7f7f9413b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9409cea0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 0x7f7f9413f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): picked mon.noname-b con 0x7f7f9409d2a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:51.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940a2710 con 0x7f7f9409cea0 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f94120590 con 0x7f7f9409d2a0 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9413c5e0 con 0x7f7f9409cad0 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9ba80640 10 monclient(hunting): authenticate will time out at 967.548838s 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9981c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f9409cad0 msgr2=0x7f7f9413bb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9981c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f9409cad0 0x7f7f9413bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9a01d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 0x7f7f9413f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9a01d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 0x7f7f9413f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48694/0 (socket says 172.21.15.144:48694) 2024-09-18T12:28:51.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9a01d640 1 -- 172.21.15.144:0/1272834916 learned_addr learned my addr 172.21.15.144:0/1272834916 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9a01d640 10 monclient(hunting): get_auth_request con 0x7f7f9409d2a0 auth_method 0 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9a01d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9a01d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.979+0000 7f7f9a01d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9409cea0 0x7f7f9413b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 10 monclient(hunting): get_auth_request con 0x7f7f9409cea0 auth_method 0 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:51.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9901b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 10 monclient(hunting): handle_auth_done global_id 5038 payload 824 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 1 -- 172.21.15.144:0/1272834916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f9409cad0 msgr2=0x7f7f9413bb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f9409cad0 0x7f7f9413bb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 1 -- 172.21.15.144:0/1272834916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9409cea0 msgr2=0x7f7f9413b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9409cea0 0x7f7f9413b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 10 monclient: _finish_hunting 0 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 1 monclient: found mon.noname-b 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:51.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.980+0000 7f7f9a01d640 1 -- 172.21.15.144:0/1272834916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f90009390 con 0x7f7f9409d2a0 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9a01d640 1 -- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 msgr2=0x7f7f9413f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9a01d640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 0x7f7f9413f0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00ceb0 tx=0x7f7f8c007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00e610 con 0x7f7f9409d2a0 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: dump: 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:51.998 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9ba80640 5 monclient: authenticate success, global_id 5038 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8c00e770 con 0x7f7f9409d2a0 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00eb30 con 0x7f7f9409d2a0 2024-09-18T12:28:51.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9ba80640 10 monclient: _renew_subs 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9ba80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f94120310 con 0x7f7f9409d2a0 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f8bfff640 4 set_mon_vals no callback set 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f89ffb640 10 monclient: dump: 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:52.000 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9ba80640 10 monclient: _renew_subs 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9ba80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.982+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9413b750 con 0x7f7f9409d2a0 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.983+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7f8c009030 con 0x7f7f9409d2a0 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.983+0000 7f7f89ffb640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f78038470 0x7f7f7803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.983+0000 7f7f9ba80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.983+0000 7f7f9ba80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.983+0000 7f7f9ba80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.983+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5518+0+0 (secure 0 0 0) 0x7f7f8c012070 con 0x7f7f9409d2a0 2024-09-18T12:28:52.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.983+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f940008d0 con 0x7f7f9409d2a0 2024-09-18T12:28:52.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.984+0000 7f7f9981c640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f78038470 0x7f7f7803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:52.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.984+0000 7f7f9981c640 10 monclient: get_auth_request con 0x7f7f78038470 auth_method 0 2024-09-18T12:28:52.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.986+0000 7f7f9981c640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f78038470 0x7f7f7803a920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7f9409eb00 tx=0x7f7f90032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:52.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.988+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7f8c0153d0 con 0x7f7f9409d2a0 2024-09-18T12:28:52.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.988+0000 7f7f89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:52.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:51.988+0000 7f7f89ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:52.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.146+0000 7f7f9ba80640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:52.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.146+0000 7f7f9ba80640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:52.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.146+0000 7f7f9ba80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:52.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.146+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f7f9409b430 con 0x7f7f9409d2a0 2024-09-18T12:28:52.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.887+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map v75) v1 ==== 203+0+0 (secure 0 0 0) 0x7f7f8c014330 con 0x7f7f9409d2a0 2024-09-18T12:28:52.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.887+0000 7f7f89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:52.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.887+0000 7f7f89ffb640 10 monclient: _finish_command 2 = system:0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map 2024-09-18T12:28:52.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.984+0000 7f7f9ba80640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:52.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.984+0000 7f7f9ba80640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:52.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.984+0000 7f7f9ba80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:52.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.984+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f7f940a2c70 con 0x7f7f9409d2a0 2024-09-18T12:28:52.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.986+0000 7f7f89ffb640 1 -- 172.21.15.144:0/1272834916 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v75) v1 ==== 201+0+0 (secure 0 0 0) 0x7f7f8c01d310 con 0x7f7f9409d2a0 2024-09-18T12:28:52.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.986+0000 7f7f89ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:52.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.986+0000 7f7f89ffb640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T12:28:52.987 INFO:tasks.workunit.client.0.smithi144.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T12:28:52.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f78038470 msgr2=0x7f7f7803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f78038470 0x7f7f7803a920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7f9409eb00 tx=0x7f7f90032000 comp rx=0 tx=0).stop 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 10 monclient: shutdown 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 msgr2=0x7f7f9413f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 0x7f7f9413f0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00ceb0 tx=0x7f7f8c007870 comp rx=0 tx=0).stop 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 shutdown_connections 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f78038470 0x7f7f7803a920 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f9409d2a0 0x7f7f9413f0d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f9409cea0 0x7f7f9413b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 --2- 172.21.15.144:0/1272834916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f9409cad0 0x7f7f9413bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 >> 172.21.15.144:0/1272834916 conn(0x7f7f94070e90 msgr2=0x7f7f9406e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:52.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 shutdown_connections 2024-09-18T12:28:52.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:52.988+0000 7f7f9ba80640 1 -- 172.21.15.144:0/1272834916 wait complete. 2024-09-18T12:28:52.998 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-18T12:28:53.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.072+0000 7f9209b00640 10 monclient: get_monmap_and_config 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.072+0000 7f9209b00640 10 monclient: build_initial_monmap 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.072+0000 7f9209b00640 10 monclient: monmap: 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:53.073570+0000 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:53.073570+0000 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:53.074 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:53.075 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 1 Processor -- start 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 1 -- start start 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient: init 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9204098c70 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9204099c70 0x7f9204136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient(hunting): picked mon.noname-c con 0x7f9204099c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204137200 0x7f92041395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:53.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient(hunting): picked mon.noname-b con 0x7f9204137200 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920409f680 con 0x7f9204098c70 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f920409f420 con 0x7f9204099c70 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9204068a20 con 0x7f9204137200 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.073+0000 7f9209b00640 10 monclient(hunting): authenticate will time out at 968.642955s 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9204099c70 0x7f9204136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:53.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9203fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204137200 msgr2=0x7f92041395e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9203fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204137200 0x7f92041395e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9204099c70 0x7f9204136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48708/0 (socket says 172.21.15.144:48708) 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 -- 172.21.15.144:0/2273790233 learned_addr learned my addr 172.21.15.144:0/2273790233 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): get_auth_request con 0x7f9204099c70 auth_method 0 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient(hunting): handle_auth_done global_id 5041 payload 274 2024-09-18T12:28:53.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 -- 172.21.15.144:0/2273790233 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204137200 msgr2=0x7f92041395e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:53.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 --2- 172.21.15.144:0/2273790233 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204137200 0x7f92041395e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 -- 172.21.15.144:0/2273790233 >> v1:172.21.15.116:6789/0 conn(0x7f9204098c70 legacy=0x7f9204099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:53.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:53.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 monclient: found mon.noname-c 2024-09-18T12:28:53.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:53.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:53.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.074+0000 7f9202ffd640 1 -- 172.21.15.144:0/2273790233 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9204096a30 con 0x7f9204099c70 2024-09-18T12:28:53.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9202ffd640 1 -- 172.21.15.144:0/2273790233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204099c70 msgr2=0x7f9204136cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9202ffd640 1 --2- 172.21.15.144:0/2273790233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204099c70 0x7f9204136cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009940 tx=0x7f91f80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cbeeff563fd52c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 1 -- 172.21.15.144:0/2273790233 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8032440 con 0x7f9204099c70 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: dump: 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:53.087 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 1 -- 172.21.15.144:0/2273790233 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f80325a0 con 0x7f9204099c70 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 5 monclient: authenticate success, global_id 5041 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 1 -- 172.21.15.144:0/2273790233 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8032940 con 0x7f9204099c70 2024-09-18T12:28:53.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 4 set_mon_vals no callback set 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f92027fc640 10 monclient: dump: 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:53.089 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 10 monclient: shutdown 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 -- 172.21.15.144:0/2273790233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204099c70 msgr2=0x7f9204136cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 --2- 172.21.15.144:0/2273790233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204099c70 0x7f9204136cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009940 tx=0x7f91f80091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 -- 172.21.15.144:0/2273790233 shutdown_connections 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 --2- 172.21.15.144:0/2273790233 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204137200 0x7f92041395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 --2- 172.21.15.144:0/2273790233 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204099c70 0x7f9204136cc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 -- 172.21.15.144:0/2273790233 >> 172.21.15.144:0/2273790233 conn(0x7f9204070e90 msgr2=0x7f92040a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 -- 172.21.15.144:0/2273790233 shutdown_connections 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.077+0000 7f9209b00640 1 -- 172.21.15.144:0/2273790233 wait complete. 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.078+0000 7f9209b00640 10 monclient: build_initial_monmap 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.078+0000 7f9209b00640 10 monclient: monmap: 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:53.090 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.078+0000 7f9209b00640 1 Processor -- start 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.078+0000 7f9209b00640 1 -- start start 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.078+0000 7f9209b00640 10 monclient: init 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:28:53.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 0x7f920411dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): picked mon.noname-b con 0x7f9204098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204099c70 0x7f920411e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): picked mon.noname-c con 0x7f9204099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9204137200 0x7f920411e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): picked mon.noname-a con 0x7f9204137200 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920406eba0 con 0x7f9204137200 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:53.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040a2190 con 0x7f9204098c70 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920411f4a0 con 0x7f9204099c70 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9209b00640 10 monclient(hunting): authenticate will time out at 968.648288s 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 0x7f920411dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 0x7f920411dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48716/0 (socket says 172.21.15.144:48716) 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 -- 172.21.15.144:0/736734440 learned_addr learned my addr 172.21.15.144:0/736734440 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): get_auth_request con 0x7f9204098c70 auth_method 0 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:53.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9202ffd640 1 -- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204099c70 msgr2=0x7f920411e2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9202ffd640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204099c70 0x7f920411e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9203fff640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9204137200 0x7f920411e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient(hunting): handle_auth_done global_id 5044 payload 824 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 -- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204099c70 msgr2=0x7f920411e2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204099c70 0x7f920411e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 -- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9204137200 msgr2=0x7f920411e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9204137200 0x7f920411e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 monclient: found mon.noname-b 2024-09-18T12:28:53.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f92037fe640 1 -- 172.21.15.144:0/736734440 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f8009450 con 0x7f9204098c70 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9203fff640 10 monclient: get_auth_request con 0x7f9204137200 auth_method 0 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.079+0000 7f9203fff640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9204137200 0x7f920411e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.081+0000 7f92037fe640 1 -- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 msgr2=0x7f920411dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.081+0000 7f92037fe640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 0x7f920411dd80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f91f400c8b0 tx=0x7f91f40026c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f400cea0 con 0x7f9204098c70 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 10 monclient: dump: 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:53.095 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9209b00640 5 monclient: authenticate success, global_id 5044 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9209b00640 10 monclient: _renew_subs 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9209b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:53.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f92037fe640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 0x7f920411dd80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f91f400c8b0 tx=0x7f91f40026c0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f91f400aa60 config(0 keys) v1 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9204120ac0 con 0x7f9204098c70 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9209b00640 10 monclient: _renew_subs 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9209b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92041207b0 con 0x7f9204098c70 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f400aa60 con 0x7f9204098c70 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f91eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.082+0000 7f9201ffb640 4 set_mon_vals no callback set 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f400ae40 con 0x7f9204098c70 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f91eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f9209b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f91eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f91eb7fe640 10 monclient: dump: 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:53.097 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f91f4012670 con 0x7f9204098c70 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f9209b00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f9209b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920409a070 con 0x7f9204098c70 2024-09-18T12:28:53.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f91eb7fe640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91f0038470 0x7f91f003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:53.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f9202ffd640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91f0038470 0x7f91f003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:53.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.083+0000 7f9202ffd640 10 monclient: get_auth_request con 0x7f91f0038470 auth_method 0 2024-09-18T12:28:53.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.085+0000 7f9202ffd640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91f0038470 0x7f91f003a920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f92040a0c20 tx=0x7f91f8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:53.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.087+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5800+0+0 (secure 0 0 0) 0x7f91f404c150 con 0x7f9204098c70 2024-09-18T12:28:53.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.705+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f91f40102e0 con 0x7f9204098c70 2024-09-18T12:28:53.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.705+0000 7f91eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:53.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.705+0000 7f91eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:53.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.863+0000 7f9209b00640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:53.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.863+0000 7f9209b00640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:53.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.863+0000 7f9209b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:53.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.863+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f92040008d0 con 0x7f9204098c70 2024-09-18T12:28:53.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.865+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v75) v1 ==== 201+0+0 (secure 0 0 0) 0x7f91f4022300 con 0x7f9204098c70 2024-09-18T12:28:53.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.865+0000 7f91eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:53.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.865+0000 7f91eb7fe640 10 monclient: _finish_command 2 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T12:28:53.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.961+0000 7f9209b00640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:53.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.961+0000 7f9209b00640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:53.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.961+0000 7f9209b00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:53.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.961+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f9204120e80 con 0x7f9204098c70 2024-09-18T12:28:53.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.962+0000 7f91eb7fe640 1 -- 172.21.15.144:0/736734440 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v75) v1 ==== 201+0+0 (secure 0 0 0) 0x7f91f4016030 con 0x7f9204098c70 2024-09-18T12:28:53.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.962+0000 7f91eb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T12:28:53.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.962+0000 7f91eb7fe640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T12:28:53.963 INFO:tasks.workunit.client.0.smithi144.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T12:28:53.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91f0038470 msgr2=0x7f91f003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:53.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91f0038470 0x7f91f003a920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f92040a0c20 tx=0x7f91f8032000 comp rx=0 tx=0).stop 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 10 monclient: shutdown 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 msgr2=0x7f920411dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 0x7f920411dd80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f91f400c8b0 tx=0x7f91f40026c0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 shutdown_connections 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91f0038470 0x7f91f003a920 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9204137200 0x7f920411e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9204099c70 0x7f920411e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 --2- 172.21.15.144:0/736734440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9204098c70 0x7f920411dd80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 >> 172.21.15.144:0/736734440 conn(0x7f9204070e90 msgr2=0x7f92040979c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 shutdown_connections 2024-09-18T12:28:53.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:53.964+0000 7f9209b00640 1 -- 172.21.15.144:0/736734440 wait complete. 2024-09-18T12:28:53.974 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-18T12:28:54.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.049+0000 7fb71e42b640 10 monclient: get_monmap_and_config 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.049+0000 7fb71e42b640 10 monclient: build_initial_monmap 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.049+0000 7fb71e42b640 10 monclient: monmap: 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:54.050621+0000 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:54.050621+0000 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:54.051 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 1 Processor -- start 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 1 -- start start 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient: init 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7180a19a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb71809a260 0x7fb71809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient(hunting): picked mon.noname-c con 0x7fb71809a260 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7181336f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180a1d70 con 0x7fb7181336f0 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb718098600 con 0x7fb7180a19a0 2024-09-18T12:28:54.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb718097a70 con 0x7fb71809a260 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.050+0000 7fb71e42b640 10 monclient(hunting): authenticate will time out at 969.619907s 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb71809a260 0x7fb71809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb717fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb7180a19a0 0x7fb71809b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:53542/0 (socket says 172.21.15.144:53542) 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb717fff640 1 -- 172.21.15.144:0/3850134965 learned_addr learned my addr 172.21.15.144:0/3850134965 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): get_auth_request con 0x7fb71809a260 auth_method 0 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:54.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient(hunting): handle_auth_done global_id 5047 payload 274 2024-09-18T12:28:54.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 1 -- 172.21.15.144:0/3850134965 >> v1:172.21.15.144:6789/0 conn(0x7fb7180a19a0 legacy=0x7fb71809b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:54.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 1 -- 172.21.15.144:0/3850134965 >> v1:172.21.15.116:6789/0 conn(0x7fb7181336f0 legacy=0x7fb718135ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:54.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:54.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 1 monclient: found mon.noname-c 2024-09-18T12:28:54.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:54.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:54.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.051+0000 7fb7177fe640 1 -- 172.21.15.144:0/3850134965 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb718097cb0 con 0x7fb71809a260 2024-09-18T12:28:54.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.053+0000 7fb7177fe640 1 -- 172.21.15.144:0/3850134965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb71809a260 msgr2=0x7fb71809bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:54.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.053+0000 7fb7177fe640 1 --2- 172.21.15.144:0/3850134965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb71809a260 0x7fb71809bff0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009850 tx=0x7fb70c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb695d8f176c066 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:54.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 1 -- 172.21.15.144:0/3850134965 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c032440 con 0x7fb71809a260 2024-09-18T12:28:54.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: dump: 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:54.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 1 -- 172.21.15.144:0/3850134965 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c0325a0 con 0x7fb71809a260 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 5 monclient: authenticate success, global_id 5047 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 1 -- 172.21.15.144:0/3850134965 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c032940 con 0x7fb71809a260 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 4 set_mon_vals no callback set 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb716ffd640 10 monclient: dump: 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:54.063 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 10 monclient: shutdown 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 1 -- 172.21.15.144:0/3850134965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb71809a260 msgr2=0x7fb71809bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3850134965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb71809a260 0x7fb71809bff0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009850 tx=0x7fb70c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 1 -- 172.21.15.144:0/3850134965 shutdown_connections 2024-09-18T12:28:54.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3850134965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb71809a260 0x7fb71809bff0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 1 -- 172.21.15.144:0/3850134965 >> 172.21.15.144:0/3850134965 conn(0x7fb7180711a0 msgr2=0x7fb7180a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 1 -- 172.21.15.144:0/3850134965 shutdown_connections 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.054+0000 7fb71e42b640 1 -- 172.21.15.144:0/3850134965 wait complete. 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.055+0000 7fb71e42b640 10 monclient: build_initial_monmap 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.055+0000 7fb71e42b640 10 monclient: monmap: 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:54.065 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.055+0000 7fb71e42b640 1 Processor -- start 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.055+0000 7fb71e42b640 1 -- start start 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.055+0000 7fb71e42b640 10 monclient: init 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb71809a260 0x7fb71813f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): picked mon.noname-c con 0x7fb71809a260 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 0x7fb71813faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7180a19a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7181336f0 0x7fb71813ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7181336f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:54.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7181385e0 con 0x7fb7181336f0 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71809cc10 con 0x7fb7180a19a0 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71809ed60 con 0x7fb71809a260 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71e42b640 10 monclient(hunting): authenticate will time out at 969.625196s 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 0x7fb71813faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb717fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb71809a260 msgr2=0x7fb71813f560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 0x7fb71813faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48738/0 (socket says 172.21.15.144:48738) 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 -- 172.21.15.144:0/3954866285 learned_addr learned my addr 172.21.15.144:0/3954866285 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): get_auth_request con 0x7fb7180a19a0 auth_method 0 2024-09-18T12:28:54.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb717fff640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb71809a260 0x7fb71813f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71c9c8640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7181336f0 0x7fb71813ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71c9c8640 10 monclient(hunting): get_auth_request con 0x7fb7181336f0 auth_method 0 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71c9c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71c9c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71c9c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient(hunting): handle_auth_done global_id 5050 payload 824 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 -- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb71809a260 msgr2=0x7fb71813f560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:54.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb71809a260 0x7fb71813f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 -- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7181336f0 msgr2=0x7fb71813ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7181336f0 0x7fb71813ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 monclient: found mon.noname-b 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb7177fe640 1 -- 172.21.15.144:0/3954866285 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c009390 con 0x7fb7180a19a0 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.056+0000 7fb71c9c8640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7181336f0 0x7fb71813ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb7177fe640 1 -- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 msgr2=0x7fb71813faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb7177fe640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 0x7fb71813faa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009850 tx=0x7fb70c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c031990 con 0x7fb7180a19a0 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:54.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 10 monclient: dump: 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 10 monclient: _finish_auth 0 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:54.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c031af0 con 0x7fb7180a19a0 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.058+0000 7fb6fffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c044410 con 0x7fb7180a19a0 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb6fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb6fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb6fffff640 10 monclient: dump: 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:54.071 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 5 monclient: authenticate success, global_id 5050 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 10 monclient: _renew_subs 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb718117550 con 0x7fb7180a19a0 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb7167fc640 4 set_mon_vals no callback set 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 10 monclient: _renew_subs 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb718138c30 con 0x7fb7180a19a0 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb70c039040 con 0x7fb7180a19a0 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:54.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.059+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7180008d0 con 0x7fb7180a19a0 2024-09-18T12:28:54.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.060+0000 7fb6fffff640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb704038440 0x7fb70403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:54.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.060+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5800+0+0 (secure 0 0 0) 0x7fb70c06b960 con 0x7fb7180a19a0 2024-09-18T12:28:54.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.060+0000 7fb717fff640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb704038440 0x7fb70403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:54.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.060+0000 7fb717fff640 10 monclient: get_auth_request con 0x7fb704038440 auth_method 0 2024-09-18T12:28:54.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.062+0000 7fb717fff640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb704038440 0x7fb70403a8f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7080047d0 tx=0x7fb708007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:54.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.064+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb70c042320 con 0x7fb7180a19a0 2024-09-18T12:28:54.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.064+0000 7fb6fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:54.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.064+0000 7fb6fffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.222+0000 7fb71e42b640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.222+0000 7fb71e42b640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.222+0000 7fb71e42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.222+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fb7180008d0 con 0x7fb7180a19a0 2024-09-18T12:28:54.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.894+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map v76) v1 ==== 203+0+0 (secure 0 0 0) 0x7fb70c037030 con 0x7fb7180a19a0 2024-09-18T12:28:54.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.894+0000 7fb6fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:54.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.894+0000 7fb6fffff640 10 monclient: _finish_command 2 = system:0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map 2024-09-18T12:28:54.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.992+0000 7fb71e42b640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:54.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.992+0000 7fb71e42b640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:54.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.992+0000 7fb71e42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:54.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.992+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fb7180008d0 con 0x7fb7180a19a0 2024-09-18T12:28:54.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.993+0000 7fb6fffff640 1 -- 172.21.15.144:0/3954866285 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v76) v1 ==== 201+0+0 (secure 0 0 0) 0x7fb70c037020 con 0x7fb7180a19a0 2024-09-18T12:28:54.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.993+0000 7fb6fffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:54.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.993+0000 7fb6fffff640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T12:28:54.995 INFO:tasks.workunit.client.0.smithi144.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T12:28:54.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb704038440 msgr2=0x7fb70403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:54.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb704038440 0x7fb70403a8f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7080047d0 tx=0x7fb708007b80 comp rx=0 tx=0).stop 2024-09-18T12:28:54.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 10 monclient: shutdown 2024-09-18T12:28:54.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:54.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 msgr2=0x7fb71813faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 0x7fb71813faa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009850 tx=0x7fb70c032000 comp rx=0 tx=0).stop 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 shutdown_connections 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb704038440 0x7fb70403a8f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7181336f0 0x7fb71813ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7180a19a0 0x7fb71813faa0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 --2- 172.21.15.144:0/3954866285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb71809a260 0x7fb71813f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 >> 172.21.15.144:0/3954866285 conn(0x7fb7180711a0 msgr2=0x7fb71809b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.995+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 shutdown_connections 2024-09-18T12:28:54.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:54.996+0000 7fb71e42b640 1 -- 172.21.15.144:0/3954866285 wait complete. 2024-09-18T12:28:55.006 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-18T12:28:55.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.080+0000 7f044c817640 10 monclient: get_monmap_and_config 2024-09-18T12:28:55.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.080+0000 7f044c817640 10 monclient: build_initial_monmap 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.080+0000 7f044c817640 10 monclient: monmap: 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:55.081415+0000 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:55.081415+0000 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:55.082 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 1 Processor -- start 2024-09-18T12:28:55.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 1 -- start start 2024-09-18T12:28:55.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient: init 2024-09-18T12:28:55.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:55.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0448098c70 0x7f0448099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient(hunting): picked mon.noname-c con 0x7f0448098c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0448099580 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044809e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient(hunting): picked mon.noname-b con 0x7f0448099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044809f680 con 0x7f0448099580 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044809f420 con 0x7f0448098c70 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0448068a20 con 0x7f0448099c70 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044c817640 10 monclient(hunting): authenticate will time out at 970.650766s 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0448098c70 0x7f0448099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0448098c70 0x7f0448099040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48754/0 (socket says 172.21.15.144:48754) 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f0445d9b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f0448099580 0x7f044809dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:53548/0 (socket says 172.21.15.144:53548) 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044659c640 1 -- 172.21.15.144:0/3942579500 learned_addr learned my addr 172.21.15.144:0/3942579500 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044659c640 10 monclient(hunting): get_auth_request con 0x7f0448098c70 auth_method 0 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f044659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f0446d9d640 1 -- 172.21.15.144:0/3942579500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 msgr2=0x7f044809e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.081+0000 7f0446d9d640 1 --2- 172.21.15.144:0/3942579500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044809e1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 10 monclient(hunting): handle_auth_done global_id 5053 payload 274 2024-09-18T12:28:55.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 1 -- 172.21.15.144:0/3942579500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 msgr2=0x7f044809e1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:55.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 1 --2- 172.21.15.144:0/3942579500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044809e1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 1 -- 172.21.15.144:0/3942579500 >> v1:172.21.15.144:6789/0 conn(0x7f0448099580 legacy=0x7f044809dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:55.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:55.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 1 monclient: found mon.noname-c 2024-09-18T12:28:55.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:55.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.082+0000 7f044659c640 1 -- 172.21.15.144:0/3942579500 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448096a30 con 0x7f0448098c70 2024-09-18T12:28:55.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.084+0000 7f044659c640 1 -- 172.21.15.144:0/3942579500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 msgr2=0x7f0448099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:55.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.084+0000 7f044659c640 1 --2- 172.21.15.144:0/3942579500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448099040 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f043c009940 tx=0x7f043c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c870800d82238cc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:55.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 1 -- 172.21.15.144:0/3942579500 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c032440 con 0x7f0448098c70 2024-09-18T12:28:55.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:55.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: dump: 2024-09-18T12:28:55.094 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: _finish_auth 0 2024-09-18T12:28:55.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 1 -- 172.21.15.144:0/3942579500 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c0325a0 con 0x7f0448098c70 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 5 monclient: authenticate success, global_id 5053 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 1 -- 172.21.15.144:0/3942579500 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c032940 con 0x7f0448098c70 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 4 set_mon_vals no callback set 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044559a640 10 monclient: dump: 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.096 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.097 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.097 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.097 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.097 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.097 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.097 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 10 monclient: shutdown 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 1 -- 172.21.15.144:0/3942579500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 msgr2=0x7f0448099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 1 --2- 172.21.15.144:0/3942579500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448099040 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f043c009940 tx=0x7f043c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 1 -- 172.21.15.144:0/3942579500 shutdown_connections 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 1 --2- 172.21.15.144:0/3942579500 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044809e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 1 --2- 172.21.15.144:0/3942579500 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448099040 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.086+0000 7f044c817640 1 -- 172.21.15.144:0/3942579500 >> 172.21.15.144:0/3942579500 conn(0x7f0448070e90 msgr2=0x7f04480a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.087+0000 7f044c817640 1 -- 172.21.15.144:0/3942579500 shutdown_connections 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.087+0000 7f044c817640 1 -- 172.21.15.144:0/3942579500 wait complete. 2024-09-18T12:28:55.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.087+0000 7f044c817640 10 monclient: build_initial_monmap 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.087+0000 7f044c817640 10 monclient: monmap: 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.087+0000 7f044c817640 1 Processor -- start 2024-09-18T12:28:55.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.087+0000 7f044c817640 1 -- start start 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.087+0000 7f044c817640 10 monclient: init 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): picked mon.noname-b con 0x7f0448098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448099580 0x7f0448135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): picked mon.noname-a con 0x7f0448099580 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044811ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): picked mon.noname-c con 0x7f0448099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:55.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0448122ae0 con 0x7f0448099580 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044806eba0 con 0x7f0448098c70 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0448097ca0 con 0x7f0448099c70 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044c817640 10 monclient(hunting): authenticate will time out at 970.657305s 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48768/0 (socket says 172.21.15.144:48768) 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 -- 172.21.15.144:0/3224109251 learned_addr learned my addr 172.21.15.144:0/3224109251 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): get_auth_request con 0x7f0448098c70 auth_method 0 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:55.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f0446d9d640 1 -- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 msgr2=0x7f044811ec00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f0446d9d640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044811ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f0445d9b640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448099580 0x7f0448135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f0445d9b640 10 monclient(hunting): get_auth_request con 0x7f0448099580 auth_method 0 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f0445d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f0445d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f0445d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient(hunting): handle_auth_done global_id 5056 payload 824 2024-09-18T12:28:55.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 -- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 msgr2=0x7f044811ec00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044811ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 -- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448099580 msgr2=0x7f0448135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448099580 0x7f0448135090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient: _finish_hunting 0 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 monclient: found mon.noname-b 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.088+0000 7f044659c640 1 -- 172.21.15.144:0/3224109251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c009450 con 0x7f0448098c70 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f044659c640 1 -- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 msgr2=0x7f0448141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f044659c640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448141290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f043c009940 tx=0x7f043c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c03c900 con 0x7f0448098c70 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f04367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f04367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:55.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f04367fc640 10 monclient: dump: 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f04367fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:55.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f04367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c03ca60 con 0x7f0448098c70 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f044c817640 5 monclient: authenticate success, global_id 5056 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f044c817640 10 monclient: _renew_subs 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f044c817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.090+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0448120680 con 0x7f0448098c70 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f04367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c03ce20 con 0x7f0448098c70 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f04367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f04367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f04367fc640 10 monclient: dump: 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.105 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f0444d99640 4 set_mon_vals no callback set 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f044c817640 10 monclient: _renew_subs 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f044c817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04481420c0 con 0x7f0448098c70 2024-09-18T12:28:55.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f043c039070 con 0x7f0448098c70 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f044c817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f044c817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f044c817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04481225b0 con 0x7f0448098c70 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.091+0000 7f04367fc640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0428038440 0x7f042803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.092+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6044+0+0 (secure 0 0 0) 0x7f043c06bff0 con 0x7f0448098c70 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.092+0000 7f0445d9b640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0428038440 0x7f042803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.092+0000 7f0445d9b640 10 monclient: get_auth_request con 0x7f0428038440 auth_method 0 2024-09-18T12:28:55.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.094+0000 7f0445d9b640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0428038440 0x7f042803a8f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04481203b0 tx=0x7f0438009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:55.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.096+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f043c03a300 con 0x7f0448098c70 2024-09-18T12:28:55.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.096+0000 7f04367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:55.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.096+0000 7f04367fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:55.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.253+0000 7f044c817640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:55.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.253+0000 7f044c817640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:55.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.253+0000 7f044c817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.253+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f04480008d0 con 0x7f0448098c70 2024-09-18T12:28:55.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.254+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v76) v1 ==== 201+0+0 (secure 0 0 0) 0x7f043c040020 con 0x7f0448098c70 2024-09-18T12:28:55.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.254+0000 7f04367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:55.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.254+0000 7f04367fc640 10 monclient: _finish_command 2 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T12:28:55.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.351+0000 7f044c817640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:55.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.351+0000 7f044c817640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:55.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.351+0000 7f044c817640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.351+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7f04480008d0 con 0x7f0448098c70 2024-09-18T12:28:55.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.352+0000 7f04367fc640 1 -- 172.21.15.144:0/3224109251 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v76) v1 ==== 201+0+0 (secure 0 0 0) 0x7f043c037070 con 0x7f0448098c70 2024-09-18T12:28:55.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.352+0000 7f04367fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T12:28:55.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.352+0000 7f04367fc640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T12:28:55.354 INFO:tasks.workunit.client.0.smithi144.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T12:28:55.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0428038440 msgr2=0x7f042803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0428038440 0x7f042803a8f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04481203b0 tx=0x7f0438009040 comp rx=0 tx=0).stop 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 10 monclient: shutdown 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 msgr2=0x7f0448141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448141290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f043c009940 tx=0x7f043c032000 comp rx=0 tx=0).stop 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 shutdown_connections 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0428038440 0x7f042803a8f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0448099c70 0x7f044811ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448099580 0x7f0448135090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 --2- 172.21.15.144:0/3224109251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0448098c70 0x7f0448141290 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 >> 172.21.15.144:0/3224109251 conn(0x7f0448070e90 msgr2=0x7f04480980f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:55.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.354+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 shutdown_connections 2024-09-18T12:28:55.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.355+0000 7f044c817640 1 -- 172.21.15.144:0/3224109251 wait complete. 2024-09-18T12:28:55.364 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket bar root 2024-09-18T12:28:55.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.437+0000 7fedeaadd640 10 monclient: get_monmap_and_config 2024-09-18T12:28:55.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.437+0000 7fedeaadd640 10 monclient: build_initial_monmap 2024-09-18T12:28:55.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.437+0000 7fedeaadd640 10 monclient: monmap: 2024-09-18T12:28:55.438 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:55.438 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.438 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:55.437973+0000 2024-09-18T12:28:55.438 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:55.437973+0000 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:55.439 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.437+0000 7fedeaadd640 1 Processor -- start 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.437+0000 7fedeaadd640 1 -- start start 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.437+0000 7fedeaadd640 10 monclient: init 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient(hunting): picked mon.noname-b con 0x7fede409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fede41321a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fede409f020 0x7fede4069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient(hunting): picked mon.noname-c con 0x7fede409f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:55.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede4067420 con 0x7fede41321a0 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede409fdc0 con 0x7fede409f020 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede40a0510 con 0x7fede409ec50 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fedeaadd640 10 monclient(hunting): authenticate will time out at 971.007388s 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede8879640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 msgr2=0x7fede4131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fede409f020 0x7fede4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede8879640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede3fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fede41321a0 0x7fede4068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:53560/0 (socket says 172.21.15.144:53560) 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede3fff640 1 -- 172.21.15.144:0/1312999112 learned_addr learned my addr 172.21.15.144:0/1312999112 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): get_auth_request con 0x7fede409f020 auth_method 0 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:55.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient(hunting): handle_auth_done global_id 5059 payload 274 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 1 -- 172.21.15.144:0/1312999112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 msgr2=0x7fede4131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 1 --2- 172.21.15.144:0/1312999112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 1 -- 172.21.15.144:0/1312999112 >> v1:172.21.15.144:6789/0 conn(0x7fede41321a0 legacy=0x7fede4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient: _finish_hunting 0 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 1 monclient: found mon.noname-c 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:55.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.438+0000 7fede907a640 1 -- 172.21.15.144:0/1312999112 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede40a0200 con 0x7fede409f020 2024-09-18T12:28:55.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede907a640 1 -- 172.21.15.144:0/1312999112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede409f020 msgr2=0x7fede4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:55.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede907a640 1 --2- 172.21.15.144:0/1312999112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede409f020 0x7fede4069410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feddc009e60 tx=0x7feddc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da0c08d1300d3eb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:55.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 1 -- 172.21.15.144:0/1312999112 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc007840 con 0x7fede409f020 2024-09-18T12:28:55.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: dump: 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:55.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fedeaadd640 5 monclient: authenticate success, global_id 5059 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 1 -- 172.21.15.144:0/1312999112 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc02d050 con 0x7fede409f020 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fedeaadd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fedeaadd640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fedeaadd640 4 set_mon_vals no callback set 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 1 -- 172.21.15.144:0/1312999112 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc007bd0 con 0x7fede409f020 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.441+0000 7fede37fe640 10 monclient: dump: 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.452 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 10 monclient: shutdown 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 -- 172.21.15.144:0/1312999112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede409f020 msgr2=0x7fede4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 --2- 172.21.15.144:0/1312999112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede409f020 0x7fede4069410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feddc009e60 tx=0x7feddc009780 comp rx=0 tx=0).stop 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 -- 172.21.15.144:0/1312999112 shutdown_connections 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 --2- 172.21.15.144:0/1312999112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede409f020 0x7fede4069410 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 --2- 172.21.15.144:0/1312999112 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede4131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 -- 172.21.15.144:0/1312999112 >> 172.21.15.144:0/1312999112 conn(0x7fede40a5d40 msgr2=0x7fede40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 -- 172.21.15.144:0/1312999112 shutdown_connections 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 1 -- 172.21.15.144:0/1312999112 wait complete. 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 10 monclient: build_initial_monmap 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.442+0000 7fedeaadd640 10 monclient: monmap: 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:55.454 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 Processor -- start 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 -- start start 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient: init 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede409a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): picked mon.noname-c con 0x7fede409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede409f020 0x7fede4130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): picked mon.noname-a con 0x7fede409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 0x7fede411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): picked mon.noname-b con 0x7fede41321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede411e330 con 0x7fede409f020 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede411c900 con 0x7fede41321a0 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede409c000 con 0x7fede409ec50 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fedeaadd640 10 monclient(hunting): authenticate will time out at 971.012851s 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede907a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 0x7fede411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede907a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 0x7fede411b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48790/0 (socket says 172.21.15.144:48790) 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede8879640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 msgr2=0x7fede409a930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede907a640 1 -- 172.21.15.144:0/3716161808 learned_addr learned my addr 172.21.15.144:0/3716161808 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede907a640 10 monclient(hunting): get_auth_request con 0x7fede41321a0 auth_method 0 2024-09-18T12:28:55.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede907a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede907a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede907a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.443+0000 7fede8879640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede409a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede3fff640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede409f020 0x7fede4130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 10 monclient(hunting): handle_auth_done global_id 5062 payload 824 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 1 -- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 msgr2=0x7fede409a930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede409a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 1 -- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede409f020 msgr2=0x7fede4130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede409f020 0x7fede4130b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 10 monclient: _finish_hunting 0 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 1 monclient: found mon.noname-b 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.444+0000 7fede907a640 1 -- 172.21.15.144:0/3716161808 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc009a20 con 0x7fede41321a0 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede907a640 1 -- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 msgr2=0x7fede411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede907a640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 0x7fede411b760 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feddc009e60 tx=0x7feddc037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc031db0 con 0x7fede41321a0 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: dump: 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.458 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc040070 con 0x7fede41321a0 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fedeaadd640 5 monclient: authenticate success, global_id 5062 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fedeaadd640 10 monclient: _renew_subs 2024-09-18T12:28:55.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc0456b0 con 0x7fede41321a0 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fedeaadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede409e280 con 0x7fede41321a0 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede0ff9640 10 monclient: dump: 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:55.460 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fede2ffd640 4 set_mon_vals no callback set 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fedeaadd640 10 monclient: _renew_subs 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fedeaadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.446+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede4141be0 con 0x7fede41321a0 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7feddc042030 con 0x7fede41321a0 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fedeaadd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fede0ff9640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedd0038440 0x7fedd003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fedeaadd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:55.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fedeaadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede40008d0 con 0x7fede41321a0 2024-09-18T12:28:55.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6044+0+0 (secure 0 0 0) 0x7feddc06d320 con 0x7fede41321a0 2024-09-18T12:28:55.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fede8879640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedd0038440 0x7fedd003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:55.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.447+0000 7fede8879640 10 monclient: get_auth_request con 0x7fedd0038440 auth_method 0 2024-09-18T12:28:55.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.450+0000 7fede8879640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedd0038440 0x7fedd003a8f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fedd40038f0 tx=0x7fedd4006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:55.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.452+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7feddc03b420 con 0x7fede41321a0 2024-09-18T12:28:55.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.452+0000 7fede0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:55.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.452+0000 7fede0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:55.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.608+0000 7fedeaadd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:55.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.608+0000 7fedeaadd640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:55.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.608+0000 7fedeaadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.608+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fede40a13e0 con 0x7fede41321a0 2024-09-18T12:28:55.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.899+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7feddc04a070 con 0x7fede41321a0 2024-09-18T12:28:55.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.899+0000 7fede0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:55.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.899+0000 7fede0ff9640 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-18T12:28:55.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.994+0000 7fedeaadd640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:55.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.994+0000 7fedeaadd640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:55.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.994+0000 7fedeaadd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:55.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.994+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fede409b460 con 0x7fede41321a0 2024-09-18T12:28:55.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.995+0000 7fede0ff9640 1 -- 172.21.15.144:0/3716161808 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v77) v1 ==== 126+0+0 (secure 0 0 0) 0x7feddc0303a0 con 0x7fede41321a0 2024-09-18T12:28:55.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.995+0000 7fede0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:55.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.995+0000 7fede0ff9640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-18T12:28:55.997 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'bar' already exists 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedd0038440 msgr2=0x7fedd003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedd0038440 0x7fedd003a8f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fedd40038f0 tx=0x7fedd4006d80 comp rx=0 tx=0).stop 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 10 monclient: shutdown 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 msgr2=0x7fede411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 0x7fede411b760 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feddc009e60 tx=0x7feddc037000 comp rx=0 tx=0).stop 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 shutdown_connections 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedd0038440 0x7fedd003a8f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fede41321a0 0x7fede411b760 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fede409f020 0x7fede4130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:55.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 --2- 172.21.15.144:0/3716161808 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fede409ec50 0x7fede409a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 >> 172.21.15.144:0/3716161808 conn(0x7fede40a5d40 msgr2=0x7fede4134ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:56.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 shutdown_connections 2024-09-18T12:28:56.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:55.997+0000 7fedeaadd640 1 -- 172.21.15.144:0/3716161808 wait complete. 2024-09-18T12:28:56.008 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket bar root 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.081+0000 7f0c22020640 10 monclient: get_monmap_and_config 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.081+0000 7f0c22020640 10 monclient: build_initial_monmap 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.081+0000 7f0c22020640 10 monclient: monmap: 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:56.082018+0000 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:56.082018+0000 2024-09-18T12:28:56.082 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:56.083 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.083 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:56.083 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:56.083 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:56.083 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:56.083 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:56.083 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.081+0000 7f0c22020640 1 Processor -- start 2024-09-18T12:28:56.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.081+0000 7f0c22020640 1 -- start start 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.081+0000 7f0c22020640 10 monclient: init 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 0x7f0c1c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient(hunting): picked mon.noname-b con 0x7f0c1c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c1c1321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c1c09f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c1c067420 con 0x7f0c1c1321a0 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c1c09fdc0 con 0x7f0c1c09f020 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0a0510 con 0x7f0c1c09ec50 2024-09-18T12:28:56.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:56.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c22020640 10 monclient(hunting): authenticate will time out at 971.651503s 2024-09-18T12:28:56.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c1b7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 msgr2=0x7f0c1c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:56.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c1b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 0x7f0c1c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:56.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c1bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f0c1c09f020 0x7f0c1c069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:53568/0 (socket says 172.21.15.144:53568) 2024-09-18T12:28:56.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.082+0000 7f0c1bfff640 1 -- 172.21.15.144:0/4006655729 learned_addr learned my addr 172.21.15.144:0/4006655729 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:56.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.084+0000 7f0c1bfff640 1 -- 172.21.15.144:0/4006655729 >> v1:172.21.15.144:6789/0 conn(0x7f0c1c09f020 legacy=0x7f0c1c069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:56.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.084+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670398995 0 0) 0x7f0c1c09fdc0 con 0x7f0c1c09f020 2024-09-18T12:28:56.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.084+0000 7f0c1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.084+0000 7f0c1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.084+0000 7f0c1a7fc640 10 monclient(hunting): my global_id is 5065 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.084+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c040034e0 con 0x7f0c1c09f020 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.084+0000 7f0c1affd640 1 -- 172.21.15.144:0/4006655729 >> v1:172.21.15.116:6789/0 conn(0x7f0c1c1321a0 legacy=0x7f0c1c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4000085847 0 0) 0x7f0c040034e0 con 0x7f0c1c09f020 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 msgr2=0x7f0c1c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 0x7f0c1c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 >> v1:172.21.15.116:6789/0 conn(0x7f0c1c1321a0 legacy=0x7f0c1c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:28:56.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c0a0200 con 0x7f0c1c09f020 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f0c140017a0 con 0x7f0c1c09f020 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: dump: 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.095 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 >> v1:172.21.15.144:6789/0 conn(0x7f0c1c09f020 legacy=0x7f0c1c069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 0x7f0c04003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): picked mon.c con 0x7f0c1c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:56.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c04004220 0x7f0c04004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): picked mon.a con 0x7f0c04004220 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c0400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): picked mon.b con 0x7f0c04004f10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0a0510 con 0x7f0c04004220 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0400bf10 con 0x7f0c04004f10 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0400c050 con 0x7f0c1c09ec50 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1b7fe640 1 -- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 msgr2=0x7f0c04003ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:56.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1b7fe640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 0x7f0c04003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c0400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): get_auth_request con 0x7f0c04004f10 auth_method 0 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c04004220 0x7f0c04004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 10 monclient(hunting): get_auth_request con 0x7f0c04004220 auth_method 0 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:56.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_done global_id 5065 payload 293 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 1 -- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 msgr2=0x7f0c04003ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c09ec50 0x7f0c04003ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 1 -- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c04004220 msgr2=0x7f0c04004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c04004220 0x7f0c04004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 1 monclient: found mon.b 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:56.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 1 -- 172.21.15.144:0/4006655729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c040034e0 con 0x7f0c04004f10 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 10 monclient: _finish_auth 0 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c22020640 5 monclient: authenticate success, global_id 5065 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.085+0000 7f0c22020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1bfff640 1 -- 172.21.15.144:0/4006655729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 msgr2=0x7f0c0400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1bfff640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c0400b9d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0c140014d0 tx=0x7f0c14007a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e32de7cca8e661c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c14005890 con 0x7f0c04004f10 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 10 monclient: dump: 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.100 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c140059f0 con 0x7f0c04004f10 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c22020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:56.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c22020640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 1 -- 172.21.15.144:0/4006655729 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c14005c00 con 0x7f0c04004f10 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c22020640 4 set_mon_vals no callback set 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.088+0000 7f0c1a7fc640 10 monclient: dump: 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.102 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 10 monclient: shutdown 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 1 -- 172.21.15.144:0/4006655729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 msgr2=0x7f0c0400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 1 --2- 172.21.15.144:0/4006655729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c0400b9d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0c140014d0 tx=0x7f0c14007a70 comp rx=0 tx=0).stop 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c1bfff640 1 -- 172.21.15.144:0/4006655729 reap_dead start 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 1 -- 172.21.15.144:0/4006655729 shutdown_connections 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 1 -- 172.21.15.144:0/4006655729 >> 172.21.15.144:0/4006655729 conn(0x7f0c1c0a5d40 msgr2=0x7f0c1c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 1 -- 172.21.15.144:0/4006655729 shutdown_connections 2024-09-18T12:28:56.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 1 -- 172.21.15.144:0/4006655729 wait complete. 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 10 monclient: build_initial_monmap 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.089+0000 7f0c22020640 10 monclient: monmap: 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 Processor -- start 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 -- start start 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient: init 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c1c09aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): picked mon.noname-b con 0x7f0c04004f10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1c11bc30 0x7f0c1c130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): picked mon.noname-a con 0x7f0c1c11bc30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c131090 0x7f0c1c140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): picked mon.noname-c con 0x7f0c1c131090 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c11c0a0 con 0x7f0c1c11bc30 2024-09-18T12:28:56.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c11cd10 con 0x7f0c04004f10 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c11dc70 con 0x7f0c1c131090 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c22020640 10 monclient(hunting): authenticate will time out at 971.659802s 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c1c09aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c131090 msgr2=0x7f0c1c140e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c131090 0x7f0c1c140e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c1c09aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48794/0 (socket says 172.21.15.144:48794) 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 1 -- 172.21.15.144:0/4178864852 learned_addr learned my addr 172.21.15.144:0/4178864852 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c04004f10 auth_method 0 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:56.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.090+0000 7f0c1affd640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1c11bc30 0x7f0c1c130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1affd640 10 monclient(hunting): get_auth_request con 0x7f0c1c11bc30 auth_method 0 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_done global_id 5068 payload 824 2024-09-18T12:28:56.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 1 -- 172.21.15.144:0/4178864852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c131090 msgr2=0x7f0c1c140e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c131090 0x7f0c1c140e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 1 -- 172.21.15.144:0/4178864852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1c11bc30 msgr2=0x7f0c1c130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1c11bc30 0x7f0c1c130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.091+0000 7f0c1b7fe640 1 -- 172.21.15.144:0/4178864852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c11df50 con 0x7f0c04004f10 2024-09-18T12:28:56.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c1b7fe640 1 -- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 msgr2=0x7f0c1c09aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:56.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c1b7fe640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c1c09aad0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00bb80 tx=0x7f0c0c0097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:56.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c00cc80 con 0x7f0c04004f10 2024-09-18T12:28:56.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: dump: 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.112 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: _finish_auth 0 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c006e90 con 0x7f0c04004f10 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c22020640 5 monclient: authenticate success, global_id 5068 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c22020640 10 monclient: _renew_subs 2024-09-18T12:28:56.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c22020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c11e230 con 0x7f0c04004f10 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c015470 con 0x7f0c04004f10 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c037fe640 10 monclient: dump: 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.114 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c22020640 10 monclient: _renew_subs 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c19ffb640 4 set_mon_vals no callback set 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c22020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.093+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1c141380 con 0x7f0c04004f10 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0c0c012070 con 0x7f0c04004f10 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c22020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c22020640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c22020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1c0008d0 con 0x7f0c04004f10 2024-09-18T12:28:56.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c037fe640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0c08038490 0x7f0c0803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f0c0c04bae0 con 0x7f0c04004f10 2024-09-18T12:28:56.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c1affd640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0c08038490 0x7f0c0803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.094+0000 7f0c1affd640 10 monclient: get_auth_request con 0x7f0c08038490 auth_method 0 2024-09-18T12:28:56.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.097+0000 7f0c1affd640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0c08038490 0x7f0c0803a940 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c09d0f0 tx=0x7f0c10007c80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:56.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.099+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0c0c04de60 con 0x7f0c04004f10 2024-09-18T12:28:56.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.099+0000 7f0c037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:56.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.099+0000 7f0c037fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.254+0000 7f0c22020640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.254+0000 7f0c22020640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.254+0000 7f0c22020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.254+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f0c1c0a13e0 con 0x7f0c04004f10 2024-09-18T12:28:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.255+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v77) v1 ==== 126+0+0 (secure 0 0 0) 0x7f0c0c01e3f0 con 0x7f0c04004f10 2024-09-18T12:28:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.255+0000 7f0c037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.255+0000 7f0c037fe640 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-18T12:28:56.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.349+0000 7f0c22020640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:56.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.349+0000 7f0c22020640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:56.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.349+0000 7f0c22020640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.349+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7f0c1c072640 con 0x7f0c04004f10 2024-09-18T12:28:56.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.350+0000 7f0c037fe640 1 -- 172.21.15.144:0/4178864852 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v77) v1 ==== 126+0+0 (secure 0 0 0) 0x7f0c0c0050f0 con 0x7f0c04004f10 2024-09-18T12:28:56.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.350+0000 7f0c037fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T12:28:56.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.350+0000 7f0c037fe640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-18T12:28:56.352 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'bar' already exists 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.352+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0c08038490 msgr2=0x7f0c0803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.352+0000 7f0c22020640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0c08038490 0x7f0c0803a940 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c09d0f0 tx=0x7f0c10007c80 comp rx=0 tx=0).stop 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.352+0000 7f0c22020640 10 monclient: shutdown 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.352+0000 7f0c22020640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.352+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 msgr2=0x7f0c1c09aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c1c09aad0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00bb80 tx=0x7f0c0c0097f0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 shutdown_connections 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0c08038490 0x7f0c0803a940 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0c1c131090 0x7f0c1c140e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c1c11bc30 0x7f0c1c130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 --2- 172.21.15.144:0/4178864852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0c04004f10 0x7f0c1c09aad0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 >> 172.21.15.144:0/4178864852 conn(0x7f0c1c0a5d40 msgr2=0x7f0c1c069490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:56.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 shutdown_connections 2024-09-18T12:28:56.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.353+0000 7f0c22020640 1 -- 172.21.15.144:0/4178864852 wait complete. 2024-09-18T12:28:56.362 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush link host1 root=bar 2024-09-18T12:28:56.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.435+0000 7f483971d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:56.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.435+0000 7f483971d640 10 monclient: build_initial_monmap 2024-09-18T12:28:56.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.435+0000 7f483971d640 10 monclient: monmap: 2024-09-18T12:28:56.436 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:56.436 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.436 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:56.436136+0000 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:56.436136+0000 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:56.437 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.435+0000 7f483971d640 1 Processor -- start 2024-09-18T12:28:56.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.435+0000 7f483971d640 1 -- start start 2024-09-18T12:28:56.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.435+0000 7f483971d640 10 monclient: init 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f483409ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f483409f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4834069d10 0x7f483413a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient(hunting): picked mon.noname-a con 0x7f4834069d10 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483406a120 con 0x7f483409ec50 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4834067420 con 0x7f483409f020 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483409fdc0 con 0x7f4834069d10 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f483971d640 10 monclient(hunting): authenticate will time out at 972.005360s 2024-09-18T12:28:56.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48327fc640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f483409f020 0x7f4834136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:53584/0 (socket says 172.21.15.144:53584) 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48327fc640 1 -- 172.21.15.144:0/688987092 learned_addr learned my addr 172.21.15.144:0/688987092 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 1 --2- 172.21.15.144:0/688987092 >> v2:172.21.15.116:3300/0 conn(0x7f4834069d10 0x7f483413a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): get_auth_request con 0x7f4834069d10 auth_method 0 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient(hunting): handle_auth_done global_id 54306 payload 274 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 1 -- 172.21.15.144:0/688987092 >> v1:172.21.15.144:6789/0 conn(0x7f483409f020 legacy=0x7f4834136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 1 -- 172.21.15.144:0/688987092 >> v1:172.21.15.116:6789/0 conn(0x7f483409ec50 legacy=0x7f4834131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:56.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 1 monclient: found mon.noname-a 2024-09-18T12:28:56.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:56.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:56.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.436+0000 7f48337fe640 1 -- 172.21.15.144:0/688987092 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48340a0510 con 0x7f4834069d10 2024-09-18T12:28:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f48337fe640 1 -- 172.21.15.144:0/688987092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834069d10 msgr2=0x7f483413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f48337fe640 1 --2- 172.21.15.144:0/688987092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834069d10 0x7f483413a490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4824009850 tx=0x7f48240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf90d69e1db44912 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 1 -- 172.21.15.144:0/688987092 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824032440 con 0x7f4834069d10 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: dump: 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 5 monclient: authenticate success, global_id 54306 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 1 -- 172.21.15.144:0/688987092 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48240325a0 con 0x7f4834069d10 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 4 set_mon_vals no callback set 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 1 -- 172.21.15.144:0/688987092 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824032980 con 0x7f4834069d10 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f4831ffb640 10 monclient: dump: 2024-09-18T12:28:56.449 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 10 monclient: shutdown 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:56.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 1 -- 172.21.15.144:0/688987092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834069d10 msgr2=0x7f483413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 1 --2- 172.21.15.144:0/688987092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834069d10 0x7f483413a490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4824009850 tx=0x7f48240090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 1 -- 172.21.15.144:0/688987092 shutdown_connections 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 1 --2- 172.21.15.144:0/688987092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834069d10 0x7f483413a490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 1 -- 172.21.15.144:0/688987092 >> 172.21.15.144:0/688987092 conn(0x7f48340a5d40 msgr2=0x7f48340a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 1 -- 172.21.15.144:0/688987092 shutdown_connections 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.440+0000 7f483971d640 1 -- 172.21.15.144:0/688987092 wait complete. 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.441+0000 7f483971d640 10 monclient: build_initial_monmap 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.441+0000 7f483971d640 10 monclient: monmap: 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:56.451 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.441+0000 7f483971d640 1 Processor -- start 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.441+0000 7f483971d640 1 -- start start 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.441+0000 7f483971d640 10 monclient: init 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4834069d10 0x7f483409aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): picked mon.noname-c con 0x7f4834069d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483409ec50 0x7f4834130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): picked mon.noname-a con 0x7f483409ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 0x7f483411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): picked mon.noname-b con 0x7f483409f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483411e4a0 con 0x7f483409ec50 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483411dee0 con 0x7f483409f020 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4834141e10 con 0x7f4834069d10 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f483971d640 10 monclient(hunting): authenticate will time out at 972.011337s 2024-09-18T12:28:56.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 0x7f483411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f4832ffd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4834069d10 msgr2=0x7f483409aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 0x7f483411b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48810/0 (socket says 172.21.15.144:48810) 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 -- 172.21.15.144:0/3948000739 learned_addr learned my addr 172.21.15.144:0/3948000739 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): get_auth_request con 0x7f483409f020 auth_method 0 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f4832ffd640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4834069d10 0x7f483409aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:56.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48327fc640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483409ec50 0x7f4834130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient(hunting): handle_auth_done global_id 5071 payload 824 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 -- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4834069d10 msgr2=0x7f483409aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4834069d10 0x7f483409aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 -- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483409ec50 msgr2=0x7f4834130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483409ec50 0x7f4834130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 monclient: found mon.noname-b 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48337fe640 1 -- 172.21.15.144:0/3948000739 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824009390 con 0x7f483409f020 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48327fc640 10 monclient: get_auth_request con 0x7f483409ec50 auth_method 0 2024-09-18T12:28:56.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.442+0000 7f48327fc640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483409ec50 0x7f4834130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.444+0000 7f48337fe640 1 -- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 msgr2=0x7f483411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.444+0000 7f48337fe640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 0x7f483411b760 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4824009360 tx=0x7f4824032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824031b70 con 0x7f483409f020 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: dump: 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.456 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: _finish_auth 0 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824031cd0 con 0x7f483409f020 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f483971d640 5 monclient: authenticate success, global_id 5071 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f483971d640 10 monclient: _renew_subs 2024-09-18T12:28:56.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f483971d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483411dc00 con 0x7f483409f020 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824044660 con 0x7f483409f020 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f481affd640 10 monclient: dump: 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:56.458 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f483971d640 10 monclient: _renew_subs 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f48317fa640 4 set_mon_vals no callback set 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f483971d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.445+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4834141640 con 0x7f483409f020 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4824039040 con 0x7f483409f020 2024-09-18T12:28:56.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f483971d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:56.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f483971d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:56.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f483971d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483411e2d0 con 0x7f483409f020 2024-09-18T12:28:56.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f481affd640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4820038440 0x7f482003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:56.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f482406bbd0 con 0x7f483409f020 2024-09-18T12:28:56.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f4832ffd640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4820038440 0x7f482003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:56.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.446+0000 7f4832ffd640 10 monclient: get_auth_request con 0x7f4820038440 auth_method 0 2024-09-18T12:28:56.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.448+0000 7f4832ffd640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4820038440 0x7f482003a8f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f482c007fd0 tx=0x7f482c007df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:56.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.450+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4824039ad0 con 0x7f483409f020 2024-09-18T12:28:56.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.450+0000 7f481affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:56.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.450+0000 7f481affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:56.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.607+0000 7f483971d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:56.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.607+0000 7f483971d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:56.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.607+0000 7f483971d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:56.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.607+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f483406a120 con 0x7f483409f020 2024-09-18T12:28:56.903 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:28:56.903 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:28:56.903 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:28:56.904 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:28:56.906 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:28:56.906 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:28:56.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:28:56.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.914+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v78) v1 ==== 168+0+0 (secure 0 0 0) 0x7f483406a120 con 0x7f483409f020 2024-09-18T12:28:56.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.914+0000 7f481affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:56.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:56.914+0000 7f481affd640 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T12:28:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.010+0000 7fb271434640 10 monclient: get_monmap_and_config 2024-09-18T12:28:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.010+0000 7fb271434640 10 monclient: build_initial_monmap 2024-09-18T12:28:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.010+0000 7fb271434640 10 monclient: monmap: 2024-09-18T12:28:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:28:57.010801+0000 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:28:57.010801+0000 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.010+0000 7fb271434640 1 Processor -- start 2024-09-18T12:28:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.010+0000 7fb271434640 1 -- start start 2024-09-18T12:28:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.010+0000 7fb271434640 10 monclient: init 2024-09-18T12:28:57.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.012+0000 7f483971d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:57.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.012+0000 7f483971d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:57.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.012+0000 7f483971d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:57.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.012+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f483406a120 con 0x7f483409f020 2024-09-18T12:28:57.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.013+0000 7f481affd640 1 -- 172.21.15.144:0/3948000739 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v78) v1 ==== 177+0+0 (secure 0 0 0) 0x7f483406a120 con 0x7f483409f020 2024-09-18T12:28:57.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.013+0000 7f481affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:57.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.013+0000 7f481affd640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T12:28:57.015 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T12:28:57.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.014+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4820038440 msgr2=0x7f482003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:57.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.014+0000 7f483971d640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4820038440 0x7f482003a8f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f482c007fd0 tx=0x7f482c007df0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.014+0000 7f483971d640 10 monclient: shutdown 2024-09-18T12:28:57.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.014+0000 7f483971d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:57.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.014+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 msgr2=0x7f483411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.014+0000 7f483971d640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 0x7f483411b760 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4824009360 tx=0x7f4824032000 comp rx=0 tx=0).stop 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 shutdown_connections 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4820038440 0x7f482003a8f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483409f020 0x7f483411b760 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483409ec50 0x7f4834130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 --2- 172.21.15.144:0/3948000739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4834069d10 0x7f483409aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 >> 172.21.15.144:0/3948000739 conn(0x7f48340a5d40 msgr2=0x7f48340a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 shutdown_connections 2024-09-18T12:28:57.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.015+0000 7f483971d640 1 -- 172.21.15.144:0/3948000739 wait complete. 2024-09-18T12:28:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T12:28:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb26c1318f0 0x7fb26c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient(hunting): picked mon.noname-a con 0x7fb26c1318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb26c09ea50 0x7fb26c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient(hunting): picked mon.noname-c con 0x7fb26c09ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb26c068d90 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c09fbf0 con 0x7fb26c068d90 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c131d50 con 0x7fb26c1318f0 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c09fe30 con 0x7fb26c09ea50 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb271434640 10 monclient(hunting): authenticate will time out at 964.332283s 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb26c1318f0 0x7fb26c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb26c068d90 0x7fb26c0691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60882/0 (socket says 172.21.15.116:60882) 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26b7fe640 1 -- 172.21.15.116:0/568907718 learned_addr learned my addr 172.21.15.116:0/568907718 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:28:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb26c1318f0 0x7fb26c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57790/0 (socket says 172.21.15.116:57790) 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): get_auth_request con 0x7fb26c1318f0 auth_method 0 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 1 --2- 172.21.15.116:0/568907718 >> v2:172.21.15.144:3300/0 conn(0x7fb26c09ea50 0x7fb26c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 10 monclient(hunting): get_auth_request con 0x7fb26c09ea50 auth_method 0 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient(hunting): handle_auth_done global_id 54309 payload 274 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 1 -- 172.21.15.116:0/568907718 >> v2:172.21.15.144:3300/0 conn(0x7fb26c09ea50 msgr2=0x7fb26c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 1 --2- 172.21.15.116:0/568907718 >> v2:172.21.15.144:3300/0 conn(0x7fb26c09ea50 0x7fb26c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 1 -- 172.21.15.116:0/568907718 >> v1:172.21.15.116:6789/0 conn(0x7fb26c068d90 legacy=0x7fb26c0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient: _finish_hunting 0 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 1 monclient: found mon.noname-a 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.011+0000 7fb26affd640 1 -- 172.21.15.116:0/568907718 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0a1210 con 0x7fb26c1318f0 2024-09-18T12:28:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb26affd640 1 -- 172.21.15.116:0/568907718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 msgr2=0x7fb26c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb26affd640 1 --2- 172.21.15.116:0/568907718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c131f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb26c06f600 tx=0x7fb25c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a0534e3efa4e9d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 1 -- 172.21.15.116:0/568907718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c02adf0 con 0x7fb26c1318f0 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: dump: 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.028 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb271434640 5 monclient: authenticate success, global_id 54309 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 1 -- 172.21.15.116:0/568907718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c032440 con 0x7fb26c1318f0 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb271434640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb271434640 4 set_mon_vals no callback set 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 1 -- 172.21.15.116:0/568907718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0327e0 con 0x7fb26c1318f0 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.014+0000 7fb269ffb640 10 monclient: dump: 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 10 monclient: shutdown 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 -- 172.21.15.116:0/568907718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 msgr2=0x7fb26c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 --2- 172.21.15.116:0/568907718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c131f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb26c06f600 tx=0x7fb25c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 -- 172.21.15.116:0/568907718 shutdown_connections 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 --2- 172.21.15.116:0/568907718 >> v2:172.21.15.144:3300/0 conn(0x7fb26c09ea50 0x7fb26c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 --2- 172.21.15.116:0/568907718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c131f30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 -- 172.21.15.116:0/568907718 >> 172.21.15.116:0/568907718 conn(0x7fb26c067110 msgr2=0x7fb26c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 -- 172.21.15.116:0/568907718 shutdown_connections 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.015+0000 7fb271434640 1 -- 172.21.15.116:0/568907718 wait complete. 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient: build_initial_monmap 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient: monmap: 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:28:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 Processor -- start 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 -- start start 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient: init 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 0x7fb26c09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): picked mon.noname-c con 0x7fb26c068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26c09ea50 0x7fb26c130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): picked mon.noname-b con 0x7fb26c09ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): picked mon.noname-a con 0x7fb26c1318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0a43e0 con 0x7fb26c1318f0 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c11e030 con 0x7fb26c09ea50 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c11ce60 con 0x7fb26c068d90 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.016+0000 7fb271434640 10 monclient(hunting): authenticate will time out at 964.337893s 2024-09-18T12:28:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 0x7fb26c09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 0x7fb26c09aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:37590/0 (socket says 172.21.15.116:37590) 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57800/0 (socket says 172.21.15.116:57800) 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 1 -- 172.21.15.116:0/3727520344 learned_addr learned my addr 172.21.15.116:0/3727520344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 10 monclient(hunting): get_auth_request con 0x7fb26c068d90 auth_method 0 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): get_auth_request con 0x7fb26c1318f0 auth_method 0 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26c09ea50 0x7fb26c130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 10 monclient(hunting): get_auth_request con 0x7fb26c09ea50 auth_method 0 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 1 -- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 msgr2=0x7fb26c09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 1 -- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 msgr2=0x7fb26c09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 0x7fb26c09aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26affd640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 0x7fb26c09aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient(hunting): handle_auth_done global_id 54312 payload 824 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 -- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 msgr2=0x7fb26c09aa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 0x7fb26c09aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 -- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26c09ea50 msgr2=0x7fb26c130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26c09ea50 0x7fb26c130c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 monclient: found mon.noname-a 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:28:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.017+0000 7fb26b7fe640 1 -- 172.21.15.116:0/3727520344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c009390 con 0x7fb26c1318f0 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb26b7fe640 1 -- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 msgr2=0x7fb26c140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb26b7fe640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c140e50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb26400cb60 tx=0x7fb2640052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 1 -- 172.21.15.116:0/3727520344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26400a850 con 0x7fb26c1318f0 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: dump: 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: _finish_auth 0 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 1 -- 172.21.15.116:0/3727520344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26400a9d0 con 0x7fb26c1318f0 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb271434640 5 monclient: authenticate success, global_id 54312 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb271434640 10 monclient: _renew_subs 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb271434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c11dd50 con 0x7fb26c1318f0 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 1 -- 172.21.15.116:0/3727520344 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26400ad90 con 0x7fb26c1318f0 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb252ffd640 10 monclient: dump: 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.039 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb271434640 10 monclient: _renew_subs 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb271434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c11bbf0 con 0x7fb26c1318f0 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.019+0000 7fb2697fa640 4 set_mon_vals no callback set 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb252ffd640 1 -- 172.21.15.116:0/3727520344 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb264007300 con 0x7fb26c1318f0 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb252ffd640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb258038490 0x7fb25803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb252ffd640 1 -- 172.21.15.116:0/3727520344 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6197+0+0 (secure 0 0 0) 0x7fb26404be60 con 0x7fb26c1318f0 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb26affd640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb258038490 0x7fb25803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb271434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb271434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb271434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.020+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c04bf60 con 0x7fb26c1318f0 2024-09-18T12:28:57.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.021+0000 7fb26affd640 10 monclient: get_auth_request con 0x7fb258038490 auth_method 0 2024-09-18T12:28:57.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.023+0000 7fb26affd640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb258038490 0x7fb25803a940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb25c035040 tx=0x7fb25c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:57.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.025+0000 7fb252ffd640 1 -- 172.21.15.116:0/3727520344 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb26401bae0 con 0x7fb26c1318f0 2024-09-18T12:28:57.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.025+0000 7fb252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:57.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.025+0000 7fb252ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:57.041 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush link host1 root=bar 2024-09-18T12:28:57.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.097+0000 7f7f73f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:28:57.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.097+0000 7f7f73f9d640 10 monclient: build_initial_monmap 2024-09-18T12:28:57.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.097+0000 7f7f73f9d640 10 monclient: monmap: 2024-09-18T12:28:57.098 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:57.098 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:28:57.098328+0000 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:28:57.098328+0000 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:28:57.099 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:57.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.097+0000 7f7f73f9d640 1 Processor -- start 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 1 -- start start 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient: init 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f6c098c70 0x7f7f6c099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f6c098c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 0x7f7f6c09dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f6c099580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f6c099c70 addr v1:172.21.15.116:6789/0 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c09f680 con 0x7f7f6c099c70 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6c09f420 con 0x7f7f6c098c70 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c068a20 con 0x7f7f6c099580 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f73f9d640 10 monclient(hunting): authenticate will time out at 972.667713s 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f6c098c70 0x7f7f6c099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f6c098c70 0x7f7f6c099040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48820/0 (socket says 172.21.15.144:48820) 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 1 -- 172.21.15.144:0/2331919775 learned_addr learned my addr 172.21.15.144:0/2331919775 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 10 monclient(hunting): get_auth_request con 0x7f7f6c098c70 auth_method 0 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:57.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71538640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 0x7f7f6c09dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.098+0000 7f7f71d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 10 monclient(hunting): get_auth_request con 0x7f7f6c099580 auth_method 0 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 10 monclient(hunting): handle_auth_done global_id 5080 payload 274 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 1 -- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 msgr2=0x7f7f6c09dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 1 -- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 msgr2=0x7f7f6c09dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 0x7f7f6c09dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 1 -- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 msgr2=0x7f7f6c09dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71538640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 0x7f7f6c09dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:57.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 0x7f7f6c09dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 1 -- 172.21.15.144:0/2331919775 >> v1:172.21.15.116:6789/0 conn(0x7f7f6c099c70 legacy=0x7f7f6c09e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:28:57.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 10 monclient: _finish_hunting 0 2024-09-18T12:28:57.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 1 monclient: found mon.noname-c 2024-09-18T12:28:57.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:57.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:28:57.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.099+0000 7f7f71d39640 1 -- 172.21.15.144:0/2331919775 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c096a30 con 0x7f7f6c098c70 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.101+0000 7f7f71d39640 1 -- 172.21.15.144:0/2331919775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c098c70 msgr2=0x7f7f6c099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.101+0000 7f7f71d39640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c098c70 0x7f7f6c099040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009910 tx=0x7f7f680091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47f6a44a835eed73 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 1 -- 172.21.15.144:0/2331919775 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6802ae90 con 0x7f7f6c098c70 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: dump: 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.111 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: _finish_auth 0 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 1 -- 172.21.15.144:0/2331919775 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f68032440 con 0x7f7f6c098c70 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 5 monclient: authenticate success, global_id 5080 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:57.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 1 -- 172.21.15.144:0/2331919775 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f680327e0 con 0x7f7f6c098c70 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 4 set_mon_vals no callback set 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f70d37640 10 monclient: dump: 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.113 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 10 monclient: shutdown 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 -- 172.21.15.144:0/2331919775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c098c70 msgr2=0x7f7f6c099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c098c70 0x7f7f6c099040 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009910 tx=0x7f7f680091d0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 -- 172.21.15.144:0/2331919775 shutdown_connections 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c099580 0x7f7f6c09dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/2331919775 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c098c70 0x7f7f6c099040 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 -- 172.21.15.144:0/2331919775 >> 172.21.15.144:0/2331919775 conn(0x7f7f6c070e90 msgr2=0x7f7f6c0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 -- 172.21.15.144:0/2331919775 shutdown_connections 2024-09-18T12:28:57.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.102+0000 7f7f73f9d640 1 -- 172.21.15.144:0/2331919775 wait complete. 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.103+0000 7f7f73f9d640 10 monclient: build_initial_monmap 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.103+0000 7f7f73f9d640 10 monclient: monmap: 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.103+0000 7f7f73f9d640 1 Processor -- start 2024-09-18T12:28:57.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.103+0000 7f7f73f9d640 1 -- start start 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.103+0000 7f7f73f9d640 10 monclient: init 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 0x7f7f6c141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f6c098c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f6c099580 0x7f7f6c133e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f6c099580 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 0x7f7f6c11e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f6c099c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c1222a0 con 0x7f7f6c099580 2024-09-18T12:28:57.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c06eba0 con 0x7f7f6c099c70 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c141e40 con 0x7f7f6c098c70 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f73f9d640 10 monclient(hunting): authenticate will time out at 972.673130s 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 0x7f7f6c11e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 0x7f7f6c11e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48834/0 (socket says 172.21.15.144:48834) 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 -- 172.21.15.144:0/1262214783 learned_addr learned my addr 172.21.15.144:0/1262214783 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): get_auth_request con 0x7f7f6c099c70 auth_method 0 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f6c099580 0x7f7f6c133e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 10 monclient(hunting): get_auth_request con 0x7f7f6c099580 auth_method 0 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 0x7f7f6c141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 10 monclient(hunting): get_auth_request con 0x7f7f6c098c70 auth_method 0 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 1 -- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 msgr2=0x7f7f6c141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:28:57.118 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 1 -- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 msgr2=0x7f7f6c141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 0x7f7f6c141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f71d39640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 0x7f7f6c141000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient(hunting): handle_auth_done global_id 5083 payload 824 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 -- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 msgr2=0x7f7f6c141000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 0x7f7f6c141000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 -- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f6c099580 msgr2=0x7f7f6c133e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f6c099580 0x7f7f6c133e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient: _finish_hunting 0 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 monclient: found mon.noname-b 2024-09-18T12:28:57.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.104+0000 7f7f7253a640 1 -- 172.21.15.144:0/1262214783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f68009450 con 0x7f7f6c099c70 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f7253a640 1 -- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 msgr2=0x7f7f6c11e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f7253a640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 0x7f7f6c11e950 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c890 tx=0x7f7f64002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6400ce80 con 0x7f7f6c099c70 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: dump: 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.120 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: _finish_auth 0 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6400aa00 con 0x7f7f6c099c70 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f73f9d640 5 monclient: authenticate success, global_id 5083 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f73f9d640 10 monclient: _renew_subs 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6400ae00 con 0x7f7f6c099c70 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f73f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c121fc0 con 0x7f7f6c099c70 2024-09-18T12:28:57.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f61ffb640 10 monclient: dump: 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.106+0000 7f7f63fff640 4 set_mon_vals no callback set 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.107+0000 7f7f73f9d640 10 monclient: _renew_subs 2024-09-18T12:28:57.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.107+0000 7f7f73f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.107+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c122630 con 0x7f7f6c099c70 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.107+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7f64012630 con 0x7f7f6c099c70 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.107+0000 7f7f61ffb640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f50038490 0x7f7f5003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.108+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6197+0+0 (secure 0 0 0) 0x7f7f6404bab0 con 0x7f7f6c099c70 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.108+0000 7f7f73f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.108+0000 7f7f73f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.108+0000 7f7f73f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.108+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c0008d0 con 0x7f7f6c099c70 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.108+0000 7f7f71d39640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f50038490 0x7f7f5003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:28:57.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.108+0000 7f7f71d39640 10 monclient: get_auth_request con 0x7f7f50038490 auth_method 0 2024-09-18T12:28:57.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.110+0000 7f7f71d39640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f50038490 0x7f7f5003a940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c0697a0 tx=0x7f7f68032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:28:57.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.112+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7f64016080 con 0x7f7f6c099c70 2024-09-18T12:28:57.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.112+0000 7f7f61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:28:57.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.112+0000 7f7f61ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:28:57.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.243+0000 7fb271434640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:28:57.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.243+0000 7fb271434640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:28:57.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.243+0000 7fb271434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:28:57.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:28:57.243+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb26c11bed0 con 0x7fb26c1318f0 2024-09-18T12:28:57.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.273+0000 7f7f73f9d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:57.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.273+0000 7f7f73f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:28:57.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.273+0000 7f7f73f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:28:57.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:28:57.273+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f7f6c064a20 con 0x7f7f6c099c70 2024-09-18T12:29:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:00.016+0000 7fb251ffb640 10 monclient: tick 2024-09-18T12:29:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:00.016+0000 7fb251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:00.016+0000 7fb251ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:00.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:00.104+0000 7f7f60ff9640 10 monclient: tick 2024-09-18T12:29:00.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:00.104+0000 7f7f60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:00.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:00.104+0000 7f7f60ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:01.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:01.017+0000 7fb251ffb640 10 monclient: tick 2024-09-18T12:29:01.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:01.017+0000 7fb251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:01.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:01.104+0000 7f7f60ff9640 10 monclient: tick 2024-09-18T12:29:01.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:01.104+0000 7f7f60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:02.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.017+0000 7fb251ffb640 10 monclient: tick 2024-09-18T12:29:02.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.017+0000 7fb251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:02.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.105+0000 7f7f60ff9640 10 monclient: tick 2024-09-18T12:29:02.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.105+0000 7f7f60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:02.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.224+0000 7fb252ffd640 1 -- 172.21.15.116:0/3727520344 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fb26402a080 con 0x7fb26c1318f0 2024-09-18T12:29:02.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.224+0000 7fb252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:29:02.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.224+0000 7fb252ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:02.226 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:29:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb258038490 msgr2=0x7fb25803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb258038490 0x7fb25803a940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb25c035040 tx=0x7fb25c032000 comp rx=0 tx=0).stop 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 10 monclient: shutdown 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 msgr2=0x7fb26c140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c140e50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb26400cb60 tx=0x7fb2640052b0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 shutdown_connections 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb258038490 0x7fb25803a940 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb26c1318f0 0x7fb26c140e50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb26c09ea50 0x7fb26c130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 --2- 172.21.15.116:0/3727520344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb26c068d90 0x7fb26c09aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 >> 172.21.15.116:0/3727520344 conn(0x7fb26c067110 msgr2=0x7fb26c0a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:02.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 shutdown_connections 2024-09-18T12:29:02.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:02.227+0000 7fb271434640 1 -- 172.21.15.116:0/3727520344 wait complete. 2024-09-18T12:29:02.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.231+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v78) v1 ==== 177+0+0 (secure 0 0 0) 0x7f7f640128e0 con 0x7f7f6c099c70 2024-09-18T12:29:02.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.231+0000 7f7f61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:29:02.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.231+0000 7f7f61ffb640 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T12:29:02.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.335+0000 7f7f73f9d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:29:02.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.335+0000 7f7f73f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:29:02.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.335+0000 7f7f73f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:02.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.335+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f7f6c141540 con 0x7f7f6c099c70 2024-09-18T12:29:02.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.336+0000 7f7f61ffb640 1 -- 172.21.15.144:0/1262214783 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v78) v1 ==== 177+0+0 (secure 0 0 0) 0x7f7f64012a10 con 0x7f7f6c099c70 2024-09-18T12:29:02.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.336+0000 7f7f61ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T12:29:02.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.336+0000 7f7f61ffb640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T12:29:02.337 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T12:29:02.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f50038490 msgr2=0x7f7f5003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:02.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f50038490 0x7f7f5003a940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c0697a0 tx=0x7f7f68032000 comp rx=0 tx=0).stop 2024-09-18T12:29:02.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 10 monclient: shutdown 2024-09-18T12:29:02.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:02.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 msgr2=0x7f7f6c11e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:02.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 0x7f7f6c11e950 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400c890 tx=0x7f7f64002640 comp rx=0 tx=0).stop 2024-09-18T12:29:02.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 shutdown_connections 2024-09-18T12:29:02.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f50038490 0x7f7f5003a940 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f6c099c70 0x7f7f6c11e950 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f6c099580 0x7f7f6c133e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 --2- 172.21.15.144:0/1262214783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f6c098c70 0x7f7f6c141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 >> 172.21.15.144:0/1262214783 conn(0x7f7f6c070e90 msgr2=0x7f7f6c0a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:02.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 shutdown_connections 2024-09-18T12:29:02.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.338+0000 7f7f73f9d640 1 -- 172.21.15.144:0/1262214783 wait complete. 2024-09-18T12:29:02.349 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush link host2 root=bar 2024-09-18T12:29:02.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.423+0000 7f65db900640 10 monclient: get_monmap_and_config 2024-09-18T12:29:02.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.423+0000 7f65db900640 10 monclient: build_initial_monmap 2024-09-18T12:29:02.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.423+0000 7f65db900640 10 monclient: monmap: 2024-09-18T12:29:02.424 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:02.424 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:02.424 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:02.424097+0000 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:02.424097+0000 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:02.425 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:02.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.423+0000 7f65db900640 1 Processor -- start 2024-09-18T12:29:02.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.423+0000 7f65db900640 1 -- start start 2024-09-18T12:29:02.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.423+0000 7f65db900640 10 monclient: init 2024-09-18T12:29:02.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:02.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65d409ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65d409f020 0x7f65d4069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient(hunting): picked mon.noname-a con 0x7f65d409f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65d4136d60 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d4067420 con 0x7f65d4136d60 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d409fdc0 con 0x7f65d409ec50 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40a0510 con 0x7f65d409f020 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:02.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65db900640 10 monclient(hunting): authenticate will time out at 977.993413s 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d969c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f65d409ec50 0x7f65d4131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34976/0 (socket says 172.21.15.144:34976) 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d969c640 1 -- 172.21.15.144:0/372723456 learned_addr learned my addr 172.21.15.144:0/372723456 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 1 --2- 172.21.15.144:0/372723456 >> v2:172.21.15.116:3300/0 conn(0x7f65d409f020 0x7f65d4069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 10 monclient(hunting): get_auth_request con 0x7f65d409f020 auth_method 0 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:02.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.424+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_done global_id 54318 payload 274 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 1 -- 172.21.15.144:0/372723456 >> v1:172.21.15.144:6789/0 conn(0x7f65d409ec50 legacy=0x7f65d4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 1 -- 172.21.15.144:0/372723456 >> v1:172.21.15.116:6789/0 conn(0x7f65d4136d60 legacy=0x7f65d4139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 10 monclient: _finish_hunting 0 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 1 monclient: found mon.noname-a 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:02.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.425+0000 7f65d8e9b640 1 -- 172.21.15.144:0/372723456 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40a0200 con 0x7f65d409f020 2024-09-18T12:29:02.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65d8e9b640 1 -- 172.21.15.144:0/372723456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d409f020 msgr2=0x7f65d4069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:02.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65d8e9b640 1 --2- 172.21.15.144:0/372723456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d409f020 0x7f65d4069d10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009850 tx=0x7f65c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4413d1a05200c4ad server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:02.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 1 -- 172.21.15.144:0/372723456 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4032440 con 0x7f65d409f020 2024-09-18T12:29:02.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:02.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:02.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: dump: 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: _finish_auth 0 2024-09-18T12:29:02.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 1 -- 172.21.15.144:0/372723456 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c40325a0 con 0x7f65d409f020 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 5 monclient: authenticate success, global_id 54318 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 4 set_mon_vals no callback set 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 1 -- 172.21.15.144:0/372723456 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4032980 con 0x7f65d409f020 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:02.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65cbfff640 10 monclient: dump: 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:02.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 10 monclient: shutdown 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 1 -- 172.21.15.144:0/372723456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d409f020 msgr2=0x7f65d4069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 1 --2- 172.21.15.144:0/372723456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d409f020 0x7f65d4069d10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009850 tx=0x7f65c40090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 1 -- 172.21.15.144:0/372723456 shutdown_connections 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 1 --2- 172.21.15.144:0/372723456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d409f020 0x7f65d4069d10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.428+0000 7f65db900640 1 -- 172.21.15.144:0/372723456 >> 172.21.15.144:0/372723456 conn(0x7f65d40a5d40 msgr2=0x7f65d40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.429+0000 7f65db900640 1 -- 172.21.15.144:0/372723456 shutdown_connections 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.429+0000 7f65db900640 1 -- 172.21.15.144:0/372723456 wait complete. 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.429+0000 7f65db900640 10 monclient: build_initial_monmap 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.429+0000 7f65db900640 10 monclient: monmap: 2024-09-18T12:29:02.442 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.429+0000 7f65db900640 1 Processor -- start 2024-09-18T12:29:02.443 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.429+0000 7f65db900640 1 -- start start 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.429+0000 7f65db900640 10 monclient: init 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 0x7f65d4132f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): picked mon.noname-c con 0x7f65d409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 0x7f65d4133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): picked mon.noname-b con 0x7f65d409f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d4136d60 0x7f65d4133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): picked mon.noname-a con 0x7f65d4136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:02.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d4135360 con 0x7f65d4136d60 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d4134ce0 con 0x7f65d409f020 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d409c000 con 0x7f65d409ec50 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65db900640 10 monclient(hunting): authenticate will time out at 977.999524s 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 0x7f65d4133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 0x7f65d4133440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40926/0 (socket says 172.21.15.144:40926) 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 -- 172.21.15.144:0/1560031890 learned_addr learned my addr 172.21.15.144:0/1560031890 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): get_auth_request con 0x7f65d409f020 auth_method 0 2024-09-18T12:29:02.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 0x7f65d4132f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 10 monclient(hunting): get_auth_request con 0x7f65d409ec50 auth_method 0 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:02.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d9e9d640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d4136d60 0x7f65d4133980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d9e9d640 10 monclient(hunting): get_auth_request con 0x7f65d4136d60 auth_method 0 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d9e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d9e9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d9e9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 1 -- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 msgr2=0x7f65d4132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 1 -- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 msgr2=0x7f65d4132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 0x7f65d4132f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d969c640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 0x7f65d4132f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient(hunting): handle_auth_done global_id 5086 payload 824 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 -- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 msgr2=0x7f65d4132f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 0x7f65d4132f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 -- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d4136d60 msgr2=0x7f65d4133980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d4136d60 0x7f65d4133980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient: _finish_hunting 0 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 monclient: found mon.noname-b 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d8e9b640 1 -- 172.21.15.144:0/1560031890 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c4009390 con 0x7f65d409f020 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.430+0000 7f65d9e9d640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d4136d60 0x7f65d4133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65d8e9b640 1 -- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 msgr2=0x7f65d4133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65d8e9b640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 0x7f65d4133440 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009360 tx=0x7f65c4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4031c70 con 0x7f65d409f020 2024-09-18T12:29:02.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: dump: 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:02.449 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65db900640 5 monclient: authenticate success, global_id 5086 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4031dd0 con 0x7f65d409f020 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65db900640 10 monclient: _renew_subs 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65db900640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d40a3c90 con 0x7f65d409f020 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c403d650 con 0x7f65d409f020 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:02.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65c97fa640 10 monclient: dump: 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:02.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65db900640 10 monclient: _renew_subs 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65cb7fe640 4 set_mon_vals no callback set 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65db900640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.433+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d4134550 con 0x7f65d409f020 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f65c4039040 con 0x7f65d409f020 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65db900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65db900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65db900640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d40008d0 con 0x7f65d409f020 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65c97fa640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65b8038490 0x7f65b803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6197+0+0 (secure 0 0 0) 0x7f65c406bdf0 con 0x7f65d409f020 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65d969c640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65b8038490 0x7f65b803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:02.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.434+0000 7f65d969c640 10 monclient: get_auth_request con 0x7f65b8038490 auth_method 0 2024-09-18T12:29:02.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.436+0000 7f65d969c640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65b8038490 0x7f65b803a940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f65d0007d10 tx=0x7f65d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:02.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.438+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f65c406f030 con 0x7f65d409f020 2024-09-18T12:29:02.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.438+0000 7f65c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:02.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.438+0000 7f65c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:02.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.597+0000 7f65db900640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:02.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.597+0000 7f65db900640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:02.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.597+0000 7f65db900640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:02.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:02.597+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f65d40008d0 con 0x7f65d409f020 2024-09-18T12:29:03.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.278+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v79) v1 ==== 168+0+0 (secure 0 0 0) 0x7f65c4037020 con 0x7f65d409f020 2024-09-18T12:29:03.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.278+0000 7f65c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.278+0000 7f65c97fa640 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T12:29:03.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.377+0000 7f65db900640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.377+0000 7f65db900640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.377+0000 7f65db900640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.377+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f65d40008d0 con 0x7f65d409f020 2024-09-18T12:29:03.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.379+0000 7f65c97fa640 1 -- 172.21.15.144:0/1560031890 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v79) v1 ==== 177+0+0 (secure 0 0 0) 0x7f65c4042070 con 0x7f65d409f020 2024-09-18T12:29:03.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.379+0000 7f65c97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.379+0000 7f65c97fa640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T12:29:03.380 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T12:29:03.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.380+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65b8038490 msgr2=0x7f65b803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.380+0000 7f65db900640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65b8038490 0x7f65b803a940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f65d0007d10 tx=0x7f65d0009040 comp rx=0 tx=0).stop 2024-09-18T12:29:03.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.380+0000 7f65db900640 10 monclient: shutdown 2024-09-18T12:29:03.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.380+0000 7f65db900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:03.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.380+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 msgr2=0x7f65d4133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.380+0000 7f65db900640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 0x7f65d4133440 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009360 tx=0x7f65c4032000 comp rx=0 tx=0).stop 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 shutdown_connections 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65b8038490 0x7f65b803a940 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65d4136d60 0x7f65d4133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65d409f020 0x7f65d4133440 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 --2- 172.21.15.144:0/1560031890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65d409ec50 0x7f65d4132f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 >> 172.21.15.144:0/1560031890 conn(0x7f65d40a5d40 msgr2=0x7f65d412bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 shutdown_connections 2024-09-18T12:29:03.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.381+0000 7f65db900640 1 -- 172.21.15.144:0/1560031890 wait complete. 2024-09-18T12:29:03.390 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush link host2 root=bar 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.463+0000 7f3bade4f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.463+0000 7f3bade4f640 10 monclient: build_initial_monmap 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.464+0000 7f3bade4f640 10 monclient: monmap: 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:03.464909+0000 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:03.464909+0000 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:03.465 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.466 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:03.466 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:03.466 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:03.466 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:03.466 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:03.466 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.464+0000 7f3bade4f640 1 Processor -- start 2024-09-18T12:29:03.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.464+0000 7f3bade4f640 1 -- start start 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.464+0000 7f3bade4f640 10 monclient: init 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ba809d140 0x7f3ba809d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): picked mon.noname-a con 0x7f3ba809d140 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ba809da50 0x7f3ba8094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ba809da50 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba80945c0 0x7f3ba8094a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ba80945c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba80670b0 con 0x7f3ba809d140 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba8097fb0 con 0x7f3ba809da50 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8094f40 con 0x7f3ba80945c0 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3bade4f640 10 monclient(hunting): authenticate will time out at 979.034277s 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ba809da50 0x7f3ba8094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ba809da50 0x7f3ba8094080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40932/0 (socket says 172.21.15.144:40932) 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 -- 172.21.15.144:0/89672617 learned_addr learned my addr 172.21.15.144:0/89672617 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:03.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): get_auth_request con 0x7f3ba809da50 auth_method 0 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba7fff640 1 --2- 172.21.15.144:0/89672617 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba80945c0 0x7f3ba8094a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba7fff640 10 monclient(hunting): get_auth_request con 0x7f3ba80945c0 auth_method 0 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba77fe640 1 --2- 172.21.15.144:0/89672617 >> v2:172.21.15.116:3300/0 conn(0x7f3ba809d140 0x7f3ba809d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient(hunting): handle_auth_done global_id 5089 payload 274 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 -- 172.21.15.144:0/89672617 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba80945c0 msgr2=0x7f3ba8094a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 --2- 172.21.15.144:0/89672617 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba80945c0 0x7f3ba8094a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 -- 172.21.15.144:0/89672617 >> v2:172.21.15.116:3300/0 conn(0x7f3ba809d140 msgr2=0x7f3ba809d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 --2- 172.21.15.144:0/89672617 >> v2:172.21.15.116:3300/0 conn(0x7f3ba809d140 0x7f3ba809d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 monclient: found mon.noname-c 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba6ffd640 1 -- 172.21.15.144:0/89672617 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8098570 con 0x7f3ba809da50 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba77fe640 10 monclient: get_auth_request con 0x7f3ba809d140 auth_method 0 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.465+0000 7f3ba77fe640 1 --2- 172.21.15.144:0/89672617 >> v2:172.21.15.116:3300/0 conn(0x7f3ba809d140 0x7f3ba809d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba6ffd640 1 -- 172.21.15.144:0/89672617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 msgr2=0x7f3ba8094080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba6ffd640 1 --2- 172.21.15.144:0/89672617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba8094080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8063900 tx=0x7f3b9c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b0e49eca20d175 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:03.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 1 -- 172.21.15.144:0/89672617 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c032440 con 0x7f3ba809da50 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: dump: 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.475 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3bade4f640 5 monclient: authenticate success, global_id 5089 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3bade4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 1 -- 172.21.15.144:0/89672617 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c0325a0 con 0x7f3ba809da50 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3bade4f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 1 -- 172.21.15.144:0/89672617 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c032940 con 0x7f3ba809da50 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3bade4f640 4 set_mon_vals no callback set 2024-09-18T12:29:03.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.468+0000 7f3ba67fc640 10 monclient: dump: 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 10 monclient: shutdown 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:03.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 -- 172.21.15.144:0/89672617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 msgr2=0x7f3ba8094080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 --2- 172.21.15.144:0/89672617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba8094080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8063900 tx=0x7f3b9c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 -- 172.21.15.144:0/89672617 shutdown_connections 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 --2- 172.21.15.144:0/89672617 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba80945c0 0x7f3ba8094a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 --2- 172.21.15.144:0/89672617 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba8094080 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 --2- 172.21.15.144:0/89672617 >> v2:172.21.15.116:3300/0 conn(0x7f3ba809d140 0x7f3ba809d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 -- 172.21.15.144:0/89672617 >> 172.21.15.144:0/89672617 conn(0x7f3ba80a2190 msgr2=0x7f3ba809e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 -- 172.21.15.144:0/89672617 shutdown_connections 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 1 -- 172.21.15.144:0/89672617 wait complete. 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 10 monclient: build_initial_monmap 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.469+0000 7f3bade4f640 10 monclient: monmap: 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:03.478 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:03.479 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 Processor -- start 2024-09-18T12:29:03.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 -- start start 2024-09-18T12:29:03.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient: init 2024-09-18T12:29:03.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:03.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:03.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ba80945c0 0x7f3ba8093550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): picked mon.noname-a con 0x7f3ba80945c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 0x7f3ba8140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ba809d140 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba81413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ba809da50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba80972b0 con 0x7f3ba80945c0 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8096b80 con 0x7f3ba809da50 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba808fd20 con 0x7f3ba809d140 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:03.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3bade4f640 10 monclient(hunting): authenticate will time out at 979.039697s 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba81413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba81413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40942/0 (socket says 172.21.15.144:40942) 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 1 -- 172.21.15.144:0/2467772622 learned_addr learned my addr 172.21.15.144:0/2467772622 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 10 monclient(hunting): get_auth_request con 0x7f3ba809da50 auth_method 0 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba6ffd640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 0x7f3ba8140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba77fe640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ba80945c0 0x7f3ba8093550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba6ffd640 10 monclient(hunting): get_auth_request con 0x7f3ba809d140 auth_method 0 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.470+0000 7f3ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba77fe640 10 monclient(hunting): get_auth_request con 0x7f3ba80945c0 auth_method 0 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 10 monclient(hunting): handle_auth_done global_id 5092 payload 824 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba6ffd640 1 -- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 msgr2=0x7f3ba8140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 1 -- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 msgr2=0x7f3ba8140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba6ffd640 1 -- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 msgr2=0x7f3ba8140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba6ffd640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 0x7f3ba8140eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba6ffd640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 0x7f3ba8140eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 0x7f3ba8140eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 1 -- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ba80945c0 msgr2=0x7f3ba8093550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ba80945c0 0x7f3ba8093550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 1 monclient: found mon.noname-b 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:03.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba7fff640 1 -- 172.21.15.144:0/2467772622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c009450 con 0x7f3ba809da50 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.471+0000 7f3ba77fe640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ba80945c0 0x7f3ba8093550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3ba7fff640 1 -- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 msgr2=0x7f3ba81413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3ba7fff640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba81413f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0007f20 tx=0x7f3ba0009860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba000ee30 con 0x7f3ba809da50 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: dump: 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.489 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba0013590 con 0x7f3ba809da50 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3bade4f640 5 monclient: authenticate success, global_id 5092 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3bade4f640 10 monclient: _renew_subs 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3bade4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba808fa40 con 0x7f3ba809da50 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0013950 con 0x7f3ba809da50 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3b8f7fe640 10 monclient: dump: 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.491 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3bade4f640 10 monclient: _renew_subs 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3bade4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba808e2a0 con 0x7f3ba809da50 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.473+0000 7f3ba5ffb640 4 set_mon_vals no callback set 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3ba0013cc0 con 0x7f3ba809da50 2024-09-18T12:29:03.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3bade4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3bade4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3bade4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba80008d0 con 0x7f3ba809da50 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3b8f7fe640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b94038490 0x7f3b9403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f3ba004c670 con 0x7f3ba809da50 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3ba77fe640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b94038490 0x7f3b9403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.474+0000 7f3ba77fe640 10 monclient: get_auth_request con 0x7f3b94038490 auth_method 0 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.476+0000 7f3ba77fe640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b94038490 0x7f3b9403a940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800ab50 tx=0x7f3b98008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.478+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3ba0012ac0 con 0x7f3ba809da50 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.478+0000 7f3b8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:03.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.478+0000 7f3b8f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:03.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.637+0000 7f3bade4f640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.637+0000 7f3bade4f640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.637+0000 7f3bade4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.637+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f3ba8093a90 con 0x7f3ba809da50 2024-09-18T12:29:03.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.638+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v79) v1 ==== 177+0+0 (secure 0 0 0) 0x7f3ba00122e0 con 0x7f3ba809da50 2024-09-18T12:29:03.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.638+0000 7f3b8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.638+0000 7f3b8f7fe640 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T12:29:03.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.735+0000 7f3bade4f640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.735+0000 7f3bade4f640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.735+0000 7f3bade4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.735+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f3ba8093c20 con 0x7f3ba809da50 2024-09-18T12:29:03.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.737+0000 7f3b8f7fe640 1 -- 172.21.15.144:0/2467772622 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v79) v1 ==== 177+0+0 (secure 0 0 0) 0x7f3ba00124c0 con 0x7f3ba809da50 2024-09-18T12:29:03.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.737+0000 7f3b8f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T12:29:03.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.737+0000 7f3b8f7fe640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T12:29:03.738 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T12:29:03.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b94038490 msgr2=0x7f3b9403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b94038490 0x7f3b9403a940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800ab50 tx=0x7f3b98008040 comp rx=0 tx=0).stop 2024-09-18T12:29:03.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 10 monclient: shutdown 2024-09-18T12:29:03.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:03.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 msgr2=0x7f3ba81413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba81413f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0007f20 tx=0x7f3ba0009860 comp rx=0 tx=0).stop 2024-09-18T12:29:03.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 shutdown_connections 2024-09-18T12:29:03.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b94038490 0x7f3b9403a940 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ba809da50 0x7f3ba81413f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ba809d140 0x7f3ba8140eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 --2- 172.21.15.144:0/2467772622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ba80945c0 0x7f3ba8093550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 >> 172.21.15.144:0/2467772622 conn(0x7f3ba80a2190 msgr2=0x7f3ba8099d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:03.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 shutdown_connections 2024-09-18T12:29:03.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.739+0000 7f3bade4f640 1 -- 172.21.15.144:0/2467772622 wait complete. 2024-09-18T12:29:03.749 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:03.750 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c osd.7 2024-09-18T12:29:03.750 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 2 2024-09-18T12:29:03.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.819+0000 7fc952ac0640 10 monclient: get_monmap_and_config 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.819+0000 7fc952ac0640 10 monclient: build_initial_monmap 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.819+0000 7fc952ac0640 10 monclient: monmap: 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:03.820570+0000 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:03.820570+0000 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:03.821 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:03.822 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:03.822 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:03.822 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 1 Processor -- start 2024-09-18T12:29:03.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 1 -- start start 2024-09-18T12:29:03.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient: init 2024-09-18T12:29:03.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:03.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T12:29:03.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc94c0a3c00 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:03.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc94c0994e0 0x7fc94c06a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient(hunting): picked mon.noname-c con 0x7fc94c0994e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc94c136c50 0x7fc94c139030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient(hunting): picked mon.noname-a con 0x7fc94c136c50 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c069f00 con 0x7fc94c0a3c00 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c0998e0 con 0x7fc94c136c50 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c069c20 con 0x7fc94c0994e0 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:03.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.820+0000 7fc952ac0640 10 monclient(hunting): authenticate will time out at 979.389916s 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc94bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc94c0994e0 0x7fc94c06a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc94bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc94c0994e0 0x7fc94c06a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40952/0 (socket says 172.21.15.144:40952) 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc94bfff640 1 -- 172.21.15.144:0/2780779766 learned_addr learned my addr 172.21.15.144:0/2780779766 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc94bfff640 10 monclient(hunting): get_auth_request con 0x7fc94c0994e0 auth_method 0 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc94bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc94bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc94bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 1 --2- 172.21.15.144:0/2780779766 >> v2:172.21.15.116:3300/0 conn(0x7fc94c136c50 0x7fc94c139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): get_auth_request con 0x7fc94c136c50 auth_method 0 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient(hunting): handle_auth_done global_id 54327 payload 274 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 1 -- 172.21.15.144:0/2780779766 >> v2:172.21.15.144:3300/0 conn(0x7fc94c0994e0 msgr2=0x7fc94c06a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 1 --2- 172.21.15.144:0/2780779766 >> v2:172.21.15.144:3300/0 conn(0x7fc94c0994e0 0x7fc94c06a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 1 -- 172.21.15.144:0/2780779766 >> v1:172.21.15.144:6789/0 conn(0x7fc94c0a3c00 legacy=0x7fc94c098dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient: _finish_hunting 0 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 1 monclient: found mon.noname-a 2024-09-18T12:29:03.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:03.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:03.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.821+0000 7fc95105d640 1 -- 172.21.15.144:0/2780779766 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c069940 con 0x7fc94c136c50 2024-09-18T12:29:03.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc95105d640 1 -- 172.21.15.144:0/2780779766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c136c50 msgr2=0x7fc94c139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc95105d640 1 --2- 172.21.15.144:0/2780779766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c136c50 0x7fc94c139030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc94c06ab40 tx=0x7fc9440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d6d4fe9f1b287d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 1 -- 172.21.15.144:0/2780779766 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94402adf0 con 0x7fc94c136c50 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: dump: 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.835 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc952ac0640 5 monclient: authenticate success, global_id 54327 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 1 -- 172.21.15.144:0/2780779766 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc944032440 con 0x7fc94c136c50 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:03.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc952ac0640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 1 -- 172.21.15.144:0/2780779766 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944032820 con 0x7fc94c136c50 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc952ac0640 4 set_mon_vals no callback set 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.824+0000 7fc94b7fe640 10 monclient: dump: 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.837 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 10 monclient: shutdown 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 -- 172.21.15.144:0/2780779766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c136c50 msgr2=0x7fc94c139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 --2- 172.21.15.144:0/2780779766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c136c50 0x7fc94c139030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc94c06ab40 tx=0x7fc9440090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 -- 172.21.15.144:0/2780779766 shutdown_connections 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 --2- 172.21.15.144:0/2780779766 >> v2:172.21.15.144:3300/0 conn(0x7fc94c0994e0 0x7fc94c06a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 --2- 172.21.15.144:0/2780779766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c136c50 0x7fc94c139030 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 -- 172.21.15.144:0/2780779766 >> 172.21.15.144:0/2780779766 conn(0x7fc94c071350 msgr2=0x7fc94c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:03.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 -- 172.21.15.144:0/2780779766 shutdown_connections 2024-09-18T12:29:03.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 -- 172.21.15.144:0/2780779766 wait complete. 2024-09-18T12:29:03.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 10 monclient: build_initial_monmap 2024-09-18T12:29:03.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 10 monclient: monmap: 2024-09-18T12:29:03.839 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:03.840 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:03.840 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:03.840 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:03.840 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:03.840 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.825+0000 7fc952ac0640 1 Processor -- start 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 1 -- start start 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient: init 2024-09-18T12:29:03.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 0x7fc94c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): picked mon.noname-b con 0x7fc94c0994e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0a3c00 0x7fc94c09b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): picked mon.noname-a con 0x7fc94c0a3c00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94c136c50 0x7fc94c09bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): picked mon.noname-c con 0x7fc94c136c50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c06e750 con 0x7fc94c0a3c00 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c099170 con 0x7fc94c0994e0 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:03.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c06bad0 con 0x7fc94c136c50 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc952ac0640 10 monclient(hunting): authenticate will time out at 979.395661s 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 0x7fc94c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 0x7fc94c06c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40958/0 (socket says 172.21.15.144:40958) 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 1 -- 172.21.15.144:0/1541772779 learned_addr learned my addr 172.21.15.144:0/1541772779 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): get_auth_request con 0x7fc94c0994e0 auth_method 0 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:03.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95085c640 10 monclient(hunting): handle_auth_done global_id 5098 payload 824 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc95105d640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94c136c50 0x7fc94c09bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.826+0000 7fc94bfff640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0a3c00 0x7fc94c09b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 1 -- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94c136c50 msgr2=0x7fc94c09bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94c136c50 0x7fc94c09bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 1 -- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0a3c00 msgr2=0x7fc94c09b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0a3c00 0x7fc94c09b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 1 monclient: found mon.noname-b 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95085c640 1 -- 172.21.15.144:0/1541772779 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc944009390 con 0x7fc94c0994e0 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95105d640 10 monclient: get_auth_request con 0x7fc94c136c50 auth_method 0 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.827+0000 7fc95105d640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94c136c50 0x7fc94c09bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc95085c640 1 -- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 msgr2=0x7fc94c06c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc95085c640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 0x7fc94c06c9b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00c8e0 tx=0x7fc93c0026c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00aa60 con 0x7fc94c0994e0 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: dump: 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.845 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: _finish_auth 0 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc952ac0640 5 monclient: authenticate success, global_id 5098 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc952ac0640 10 monclient: _renew_subs 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc952ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c06ec10 con 0x7fc94c0994e0 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c00abe0 con 0x7fc94c0994e0 2024-09-18T12:29:03.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c012420 con 0x7fc94c0994e0 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc948ff9640 10 monclient: dump: 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:03.847 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:03.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc94affd640 4 set_mon_vals no callback set 2024-09-18T12:29:03.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc952ac0640 10 monclient: _renew_subs 2024-09-18T12:29:03.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc952ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.829+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c06c190 con 0x7fc94c0994e0 2024-09-18T12:29:03.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc952ac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:03.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc952ac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:03.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc952ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc93c012810 con 0x7fc94c0994e0 2024-09-18T12:29:03.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c0636e0 con 0x7fc94c0994e0 2024-09-18T12:29:03.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc948ff9640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc92c038490 0x7fc92c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:03.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7fc93c04c2f0 con 0x7fc94c0994e0 2024-09-18T12:29:03.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc94bfff640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc92c038490 0x7fc92c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:03.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.830+0000 7fc94bfff640 10 monclient: get_auth_request con 0x7fc92c038490 auth_method 0 2024-09-18T12:29:03.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.833+0000 7fc94bfff640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc92c038490 0x7fc92c03a940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc934006cb0 tx=0x7fc934008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:03.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.835+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc93c0102e0 con 0x7fc94c0994e0 2024-09-18T12:29:03.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.835+0000 7fc948ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:03.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.835+0000 7fc948ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:03.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.991+0000 7fc952ac0640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:03.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.991+0000 7fc952ac0640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:03.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.991+0000 7fc952ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:03.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.991+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc94c09d0c0 con 0x7fc94c0994e0 2024-09-18T12:29:03.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.992+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7fc93c0165e0 con 0x7fc94c0994e0 2024-09-18T12:29:03.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.992+0000 7fc948ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:03.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:03.992+0000 7fc948ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:04.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.085+0000 7fc952ac0640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:04.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.085+0000 7fc952ac0640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.085+0000 7fc952ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.085+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fc94c072640 con 0x7fc94c0994e0 2024-09-18T12:29:04.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.085+0000 7fc948ff9640 1 -- 172.21.15.144:0/1541772779 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7fc93c016070 con 0x7fc94c0994e0 2024-09-18T12:29:04.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.085+0000 7fc948ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.085+0000 7fc948ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc92c038490 msgr2=0x7fc92c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc92c038490 0x7fc92c03a940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc934006cb0 tx=0x7fc934008040 comp rx=0 tx=0).stop 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 10 monclient: shutdown 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 msgr2=0x7fc94c06c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 0x7fc94c06c9b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00c8e0 tx=0x7fc93c0026c0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 shutdown_connections 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc92c038490 0x7fc92c03a940 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94c136c50 0x7fc94c09bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc94c0a3c00 0x7fc94c09b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 --2- 172.21.15.144:0/1541772779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc94c0994e0 0x7fc94c06c9b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 >> 172.21.15.144:0/1541772779 conn(0x7fc94c071350 msgr2=0x7fc94c0a1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:04.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 shutdown_connections 2024-09-18T12:29:04.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.087+0000 7fc952ac0640 1 -- 172.21.15.144:0/1541772779 wait complete. 2024-09-18T12:29:04.097 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:04.098 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c host1 2024-09-18T12:29:04.098 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 2 2024-09-18T12:29:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.167+0000 7f1606b5d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.167+0000 7f1606b5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.167+0000 7f1606b5d640 10 monclient: monmap: 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:04.168562+0000 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:04.168562+0000 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:04.169 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 1 Processor -- start 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 1 -- start start 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient: init 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f160009a780 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f1600134d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f16001300f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1600130500 0x7f1600137260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1600130500 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160006ec50 con 0x7f160009a780 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1600097a40 con 0x7f1600130500 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160009c570 con 0x7f16001300f0 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.168+0000 7f1606b5d640 10 monclient(hunting): authenticate will time out at 979.737904s 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16048f9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f160009a780 0x7f160012f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34996/0 (socket says 172.21.15.144:34996) 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16048f9640 1 -- 172.21.15.144:0/381735427 learned_addr learned my addr 172.21.15.144:0/381735427 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:04.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 1 --2- 172.21.15.144:0/381735427 >> v2:172.21.15.116:3300/0 conn(0x7f1600130500 0x7f1600137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f1600134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): get_auth_request con 0x7f1600130500 auth_method 0 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 10 monclient(hunting): get_auth_request con 0x7f16001300f0 auth_method 0 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 1 -- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 msgr2=0x7f1600134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 1 -- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 msgr2=0x7f1600134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:04.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f1600134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f15fffff640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f1600134d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient(hunting): handle_auth_done global_id 54330 payload 274 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 1 -- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 msgr2=0x7f1600134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f1600134d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 1 -- 172.21.15.144:0/381735427 >> v1:172.21.15.144:6789/0 conn(0x7f160009a780 legacy=0x7f160012f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient: _finish_hunting 0 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 1 monclient: found mon.noname-a 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:04.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.169+0000 7f16050fa640 1 -- 172.21.15.144:0/381735427 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1600097de0 con 0x7f1600130500 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f16050fa640 1 -- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1600130500 msgr2=0x7f1600137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f16050fa640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1600130500 0x7f1600137260 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e60 tx=0x7f15f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1cb9d82969fd25 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 1 -- 172.21.15.144:0/381735427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8007760 con 0x7f1600130500 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: dump: 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.181 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 1 -- 172.21.15.144:0/381735427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f802d050 con 0x7f1600130500 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f1606b5d640 5 monclient: authenticate success, global_id 54330 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f1606b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:04.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f1606b5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f1606b5d640 4 set_mon_vals no callback set 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 1 -- 172.21.15.144:0/381735427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8007b30 con 0x7f1600130500 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.172+0000 7f15ff7fe640 10 monclient: dump: 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.183 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 10 monclient: shutdown 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 -- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1600130500 msgr2=0x7f1600137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1600130500 0x7f1600137260 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e60 tx=0x7f15f8009780 comp rx=0 tx=0).stop 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 -- 172.21.15.144:0/381735427 shutdown_connections 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f1600134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 --2- 172.21.15.144:0/381735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1600130500 0x7f1600137260 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 -- 172.21.15.144:0/381735427 >> 172.21.15.144:0/381735427 conn(0x7f16000714d0 msgr2=0x7f16000a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 -- 172.21.15.144:0/381735427 shutdown_connections 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 -- 172.21.15.144:0/381735427 wait complete. 2024-09-18T12:29:04.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 10 monclient: monmap: 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.173+0000 7f1606b5d640 1 Processor -- start 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 1 -- start start 2024-09-18T12:29:04.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient: init 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160009a780 0x7f1600140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f160009a780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f160012cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f16001300f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 0x7f16000983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f1600130500 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160006ede0 con 0x7f160009a780 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16000a0e30 con 0x7f1600130500 2024-09-18T12:29:04.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160011f470 con 0x7f16001300f0 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f1606b5d640 10 monclient(hunting): authenticate will time out at 979.743646s 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 0x7f16000983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 0x7f16000983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40972/0 (socket says 172.21.15.144:40972) 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 1 -- 172.21.15.144:0/1193975708 learned_addr learned my addr 172.21.15.144:0/1193975708 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 10 monclient(hunting): get_auth_request con 0x7f1600130500 auth_method 0 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:04.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16050fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f15fffff640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f160012cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f15fffff640 10 monclient(hunting): get_auth_request con 0x7f16001300f0 auth_method 0 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.174+0000 7f16048f9640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160009a780 0x7f1600140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f15fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f15fffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f15fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 10 monclient(hunting): handle_auth_done global_id 5101 payload 824 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 1 -- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 msgr2=0x7f160012cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f160012cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 1 -- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160009a780 msgr2=0x7f1600140c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160009a780 0x7f1600140c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 10 monclient: _finish_hunting 0 2024-09-18T12:29:04.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 1 monclient: found mon.noname-b 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16050fa640 1 -- 172.21.15.144:0/1193975708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f8009a20 con 0x7f1600130500 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16048f9640 10 monclient: get_auth_request con 0x7f160009a780 auth_method 0 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.175+0000 7f16048f9640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160009a780 0x7f1600140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f16050fa640 1 -- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 msgr2=0x7f16000983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f16050fa640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 0x7f16000983b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e60 tx=0x7f15f8037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8031b90 con 0x7f1600130500 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: dump: 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.189 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: _finish_auth 0 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f1606b5d640 5 monclient: authenticate success, global_id 5101 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8040070 con 0x7f1600130500 2024-09-18T12:29:04.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f1606b5d640 10 monclient: _renew_subs 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f1606b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160009a340 con 0x7f1600130500 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8045430 con 0x7f1600130500 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15fcff9640 10 monclient: dump: 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.191 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f15feffd640 4 set_mon_vals no callback set 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f1606b5d640 10 monclient: _renew_subs 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f1606b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.177+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1600141aa0 con 0x7f1600130500 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f15f8003020 con 0x7f1600130500 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f1606b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f1606b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f1606b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16000636e0 con 0x7f1600130500 2024-09-18T12:29:04.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f15fcff9640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15e4038490 0x7f15e403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f15f806d4f0 con 0x7f1600130500 2024-09-18T12:29:04.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f16048f9640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15e4038490 0x7f15e403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.178+0000 7f16048f9640 10 monclient: get_auth_request con 0x7f15e4038490 auth_method 0 2024-09-18T12:29:04.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.181+0000 7f16048f9640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15e4038490 0x7f15e403a940 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f15f4005150 tx=0x7f15f4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.183+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f15f804d3f0 con 0x7f1600130500 2024-09-18T12:29:04.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.183+0000 7f15fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.183+0000 7f15fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:04.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.340+0000 7f1606b5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:04.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.340+0000 7f1606b5d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.340+0000 7f1606b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.340+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f160009a340 con 0x7f1600130500 2024-09-18T12:29:04.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.340+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f160009a340 con 0x7f1600130500 2024-09-18T12:29:04.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.340+0000 7f15fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.340+0000 7f15fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:04.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.434+0000 7f1606b5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:04.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.434+0000 7f1606b5d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.434+0000 7f1606b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.434+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f160009a340 con 0x7f1600130500 2024-09-18T12:29:04.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.434+0000 7f15fcff9640 1 -- 172.21.15.144:0/1193975708 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f160009a340 con 0x7f1600130500 2024-09-18T12:29:04.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.434+0000 7f15fcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.434+0000 7f15fcff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:04.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15e4038490 msgr2=0x7f15e403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15e4038490 0x7f15e403a940 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f15f4005150 tx=0x7f15f4009040 comp rx=0 tx=0).stop 2024-09-18T12:29:04.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 10 monclient: shutdown 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 msgr2=0x7f16000983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 0x7f16000983b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009e60 tx=0x7f15f8037000 comp rx=0 tx=0).stop 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 shutdown_connections 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15e4038490 0x7f15e403a940 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1600130500 0x7f16000983b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16001300f0 0x7f160012cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 --2- 172.21.15.144:0/1193975708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f160009a780 0x7f1600140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 >> 172.21.15.144:0/1193975708 conn(0x7f16000714d0 msgr2=0x7f160012e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 shutdown_connections 2024-09-18T12:29:04.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.436+0000 7f1606b5d640 1 -- 172.21.15.144:0/1193975708 wait complete. 2024-09-18T12:29:04.446 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:04.446 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c osd.9 2024-09-18T12:29:04.446 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 2 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.515+0000 7f582eb9d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.515+0000 7f582eb9d640 10 monclient: build_initial_monmap 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.515+0000 7f582eb9d640 10 monclient: monmap: 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:04.516846+0000 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:04.516846+0000 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:04.517 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.518 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:04.518 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:04.518 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:04.518 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:04.518 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:04.518 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.516+0000 7f582eb9d640 1 Processor -- start 2024-09-18T12:29:04.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.516+0000 7f582eb9d640 1 -- start start 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.516+0000 7f582eb9d640 10 monclient: init 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809f140 0x7f582809ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f582809f140 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f582809cf20 0x7f5828068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f582809cf20 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f58280a0430 0x7f58280693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f58280a0430 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280a1730 con 0x7f58280a0430 2024-09-18T12:29:04.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280a01f0 con 0x7f582809cf20 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58280a1b10 con 0x7f582809f140 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582eb9d640 10 monclient(hunting): authenticate will time out at 980.086287s 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f582809cf20 0x7f5828068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f582809cf20 0x7f5828068eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40988/0 (socket says 172.21.15.144:40988) 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 -- 172.21.15.144:0/2246397947 learned_addr learned my addr 172.21.15.144:0/2246397947 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): get_auth_request con 0x7f582809cf20 auth_method 0 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582c939640 1 --2- 172.21.15.144:0/2246397947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809f140 0x7f582809ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582d13a640 1 --2- 172.21.15.144:0/2246397947 >> v2:172.21.15.116:3300/0 conn(0x7f58280a0430 0x7f58280693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient(hunting): handle_auth_done global_id 5104 payload 274 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 -- 172.21.15.144:0/2246397947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809f140 msgr2=0x7f582809ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 --2- 172.21.15.144:0/2246397947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809f140 0x7f582809ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 -- 172.21.15.144:0/2246397947 >> v2:172.21.15.116:3300/0 conn(0x7f58280a0430 msgr2=0x7f58280693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 --2- 172.21.15.144:0/2246397947 >> v2:172.21.15.116:3300/0 conn(0x7f58280a0430 0x7f58280693f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:04.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 monclient: found mon.noname-c 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f5827fff640 1 -- 172.21.15.144:0/2246397947 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582809bfc0 con 0x7f582809cf20 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582c939640 10 monclient: get_auth_request con 0x7f582809f140 auth_method 0 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582d13a640 10 monclient: get_auth_request con 0x7f58280a0430 auth_method 0 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582c939640 1 --2- 172.21.15.144:0/2246397947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809f140 0x7f582809ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.517+0000 7f582d13a640 1 --2- 172.21.15.144:0/2246397947 >> v2:172.21.15.116:3300/0 conn(0x7f58280a0430 0x7f58280693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f5827fff640 1 -- 172.21.15.144:0/2246397947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f582809cf20 msgr2=0x7f5828068eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f5827fff640 1 --2- 172.21.15.144:0/2246397947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f582809cf20 0x7f5828068eb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f582806a500 tx=0x7f581c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cdd01c8efcecb8c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 1 -- 172.21.15.144:0/2246397947 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c02ae20 con 0x7f582809cf20 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: dump: 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:04.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 1 -- 172.21.15.144:0/2246397947 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c02f4b0 con 0x7f582809cf20 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f582eb9d640 5 monclient: authenticate success, global_id 5104 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f582eb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f582eb9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 1 -- 172.21.15.144:0/2246397947 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c02f850 con 0x7f582809cf20 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f582eb9d640 4 set_mon_vals no callback set 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.520+0000 7f58277fe640 10 monclient: dump: 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.532 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 10 monclient: shutdown 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 -- 172.21.15.144:0/2246397947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f582809cf20 msgr2=0x7f5828068eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 --2- 172.21.15.144:0/2246397947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f582809cf20 0x7f5828068eb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f582806a500 tx=0x7f581c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 -- 172.21.15.144:0/2246397947 shutdown_connections 2024-09-18T12:29:04.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 --2- 172.21.15.144:0/2246397947 >> v2:172.21.15.116:3300/0 conn(0x7f58280a0430 0x7f58280693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 --2- 172.21.15.144:0/2246397947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f582809cf20 0x7f5828068eb0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 --2- 172.21.15.144:0/2246397947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809f140 0x7f582809ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 -- 172.21.15.144:0/2246397947 >> 172.21.15.144:0/2246397947 conn(0x7f58280a5230 msgr2=0x7f58280a5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 -- 172.21.15.144:0/2246397947 shutdown_connections 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 1 -- 172.21.15.144:0/2246397947 wait complete. 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 10 monclient: build_initial_monmap 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.521+0000 7f582eb9d640 10 monclient: monmap: 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 Processor -- start 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 -- start start 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient: init 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 0x7f5828132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f582809cf20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f582809f140 0x7f582811c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f582809f140 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 0x7f582811cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f58280a0430 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5828134cb0 con 0x7f582809f140 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582811ff40 con 0x7f58280a0430 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58280a0b60 con 0x7f582809cf20 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582eb9d640 10 monclient(hunting): authenticate will time out at 980.091724s 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 0x7f582811cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 0x7f582811cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40990/0 (socket says 172.21.15.144:40990) 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 1 -- 172.21.15.144:0/546277349 learned_addr learned my addr 172.21.15.144:0/546277349 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 10 monclient(hunting): get_auth_request con 0x7f58280a0430 auth_method 0 2024-09-18T12:29:04.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582d13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582c939640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 0x7f5828132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f5827fff640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f582809f140 0x7f582811c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582c939640 10 monclient(hunting): get_auth_request con 0x7f582809cf20 auth_method 0 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582c939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582c939640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f582c939640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f5827fff640 10 monclient(hunting): get_auth_request con 0x7f582809f140 auth_method 0 2024-09-18T12:29:04.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f5827fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f5827fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.522+0000 7f5827fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582c939640 1 -- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 msgr2=0x7f5828132b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582c939640 1 -- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 msgr2=0x7f5828132b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 10 monclient(hunting): handle_auth_done global_id 5107 payload 824 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582c939640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 0x7f5828132b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 1 -- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 msgr2=0x7f5828132b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582c939640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 0x7f5828132b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 0x7f5828132b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 1 -- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f582809f140 msgr2=0x7f582811c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f582809f140 0x7f582811c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 10 monclient: _finish_hunting 0 2024-09-18T12:29:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 1 monclient: found mon.noname-b 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f582d13a640 1 -- 172.21.15.144:0/546277349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c009450 con 0x7f58280a0430 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.523+0000 7f5827fff640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f582809f140 0x7f582811c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582d13a640 1 -- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 msgr2=0x7f582811cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582d13a640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 0x7f582811cda0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f582000a380 tx=0x7f5820007900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582000d860 con 0x7f58280a0430 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: dump: 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: _finish_auth 0 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582000d9c0 con 0x7f58280a0430 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582eb9d640 5 monclient: authenticate success, global_id 5107 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582eb9d640 10 monclient: _renew_subs 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582811fc60 con 0x7f58280a0430 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582000dd80 con 0x7f58280a0430 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5824ff9640 10 monclient: dump: 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.541 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.542 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582eb9d640 10 monclient: _renew_subs 2024-09-18T12:29:04.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f5826ffd640 4 set_mon_vals no callback set 2024-09-18T12:29:04.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.525+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58281364f0 con 0x7f58280a0430 2024-09-18T12:29:04.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f582001e3b0 con 0x7f58280a0430 2024-09-18T12:29:04.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f5824ff9640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5814038490 0x7f581403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f582004c7a0 con 0x7f58280a0430 2024-09-18T12:29:04.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f582eb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f582eb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f582eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f582c939640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5814038490 0x7f581403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58280636e0 con 0x7f58280a0430 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.526+0000 7f582c939640 10 monclient: get_auth_request con 0x7f5814038490 auth_method 0 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.528+0000 7f582c939640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5814038490 0x7f581403a940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f581800a160 tx=0x7f5818008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.531+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5820012a80 con 0x7f58280a0430 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.531+0000 7f5824ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.531+0000 7f5824ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:04.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.684+0000 7f582eb9d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:04.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.684+0000 7f582eb9d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.684+0000 7f582eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.684+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f582809f510 con 0x7f58280a0430 2024-09-18T12:29:04.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.684+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f582001a070 con 0x7f58280a0430 2024-09-18T12:29:04.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.684+0000 7f5824ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.684+0000 7f5824ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:04.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.776+0000 7f582eb9d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:04.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.776+0000 7f582eb9d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.776+0000 7f582eb9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.776+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f582806f270 con 0x7f58280a0430 2024-09-18T12:29:04.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.776+0000 7f5824ff9640 1 -- 172.21.15.144:0/546277349 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7f58200122a0 con 0x7f58280a0430 2024-09-18T12:29:04.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.776+0000 7f5824ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:04.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.776+0000 7f5824ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:04.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5814038490 msgr2=0x7f581403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5814038490 0x7f581403a940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f581800a160 tx=0x7f5818008040 comp rx=0 tx=0).stop 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 10 monclient: shutdown 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 msgr2=0x7f582811cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 0x7f582811cda0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f582000a380 tx=0x7f5820007900 comp rx=0 tx=0).stop 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 shutdown_connections 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5814038490 0x7f581403a940 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f58280a0430 0x7f582811cda0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f582809f140 0x7f582811c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 --2- 172.21.15.144:0/546277349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f582809cf20 0x7f5828132b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 >> 172.21.15.144:0/546277349 conn(0x7f58280a5230 msgr2=0x7f5828134e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:04.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 shutdown_connections 2024-09-18T12:29:04.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.778+0000 7f582eb9d640 1 -- 172.21.15.144:0/546277349 wait complete. 2024-09-18T12:29:04.789 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:04.789 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c host2 2024-09-18T12:29:04.789 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 2 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.858+0000 7fb50a99d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.858+0000 7fb50a99d640 10 monclient: build_initial_monmap 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.858+0000 7fb50a99d640 10 monclient: monmap: 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:04.859702+0000 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:04.859702+0000 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:04.860 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:04.861 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:04.861 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.859+0000 7fb50a99d640 1 Processor -- start 2024-09-18T12:29:04.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.859+0000 7fb50a99d640 1 -- start start 2024-09-18T12:29:04.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.859+0000 7fb50a99d640 10 monclient: init 2024-09-18T12:29:04.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:04.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T12:29:04.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb504131c40 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb50409cfb0 0x7fb504137200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient(hunting): picked mon.noname-a con 0x7fb50409cfb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb50406ea40 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb50409c630 con 0x7fb504131c40 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb50406f2d0 con 0x7fb50406ea40 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50409c000 con 0x7fb50409cfb0 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb50a99d640 10 monclient(hunting): authenticate will time out at 980.429159s 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb508f3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb50406ea40 0x7fb504139750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:35004/0 (socket says 172.21.15.144:35004) 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb508f3a640 1 -- 172.21.15.144:0/2169138055 learned_addr learned my addr 172.21.15.144:0/2169138055 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:04.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 1 --2- 172.21.15.144:0/2169138055 >> v2:172.21.15.116:3300/0 conn(0x7fb50409cfb0 0x7fb504137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): get_auth_request con 0x7fb50409cfb0 auth_method 0 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient(hunting): handle_auth_done global_id 54336 payload 274 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 1 -- 172.21.15.144:0/2169138055 >> v1:172.21.15.144:6789/0 conn(0x7fb50406ea40 legacy=0x7fb504139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 1 -- 172.21.15.144:0/2169138055 >> v1:172.21.15.116:6789/0 conn(0x7fb504131c40 legacy=0x7fb504132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 1 monclient: found mon.noname-a 2024-09-18T12:29:04.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:04.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:04.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.860+0000 7fb5037fe640 1 -- 172.21.15.144:0/2169138055 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50409bd60 con 0x7fb50409cfb0 2024-09-18T12:29:04.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.863+0000 7fb5037fe640 1 -- 172.21.15.144:0/2169138055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 msgr2=0x7fb504137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:04.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.863+0000 7fb5037fe640 1 --2- 172.21.15.144:0/2169138055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504137200 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009850 tx=0x7fb4f80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99c81873d8999d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.863+0000 7fb502ffd640 1 -- 172.21.15.144:0/2169138055 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8032440 con 0x7fb50409cfb0 2024-09-18T12:29:04.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: dump: 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:04.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 1 -- 172.21.15.144:0/2169138055 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f80325a0 con 0x7fb50409cfb0 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 1 -- 172.21.15.144:0/2169138055 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8032940 con 0x7fb50409cfb0 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 5 monclient: authenticate success, global_id 54336 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 4 set_mon_vals no callback set 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb502ffd640 10 monclient: dump: 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.872 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 10 monclient: shutdown 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 1 -- 172.21.15.144:0/2169138055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 msgr2=0x7fb504137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 1 --2- 172.21.15.144:0/2169138055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504137200 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009850 tx=0x7fb4f80090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 1 -- 172.21.15.144:0/2169138055 shutdown_connections 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 1 --2- 172.21.15.144:0/2169138055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504137200 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 1 -- 172.21.15.144:0/2169138055 >> 172.21.15.144:0/2169138055 conn(0x7fb5040a12d0 msgr2=0x7fb5040a16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 1 -- 172.21.15.144:0/2169138055 shutdown_connections 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.864+0000 7fb50a99d640 1 -- 172.21.15.144:0/2169138055 wait complete. 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.865+0000 7fb50a99d640 10 monclient: build_initial_monmap 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.865+0000 7fb50a99d640 10 monclient: monmap: 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:04.874 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.865+0000 7fb50a99d640 1 Processor -- start 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.865+0000 7fb50a99d640 1 -- start start 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.865+0000 7fb50a99d640 10 monclient: init 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 0x7fb504141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): picked mon.noname-b con 0x7fb50406ea40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): picked mon.noname-a con 0x7fb50409cfb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb504131c40 0x7fb50411d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): picked mon.noname-c con 0x7fb504131c40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5040a05f0 con 0x7fb50409cfb0 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5041214b0 con 0x7fb50406ea40 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb504120e60 con 0x7fb504131c40 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb50a99d640 10 monclient(hunting): authenticate will time out at 980.435103s 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 0x7fb504141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 0x7fb504141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40998/0 (socket says 172.21.15.144:40998) 2024-09-18T12:29:04.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 -- 172.21.15.144:0/1137423884 learned_addr learned my addr 172.21.15.144:0/1137423884 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): get_auth_request con 0x7fb50406ea40 auth_method 0 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb508f3a640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb504131c40 0x7fb50411d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb508f3a640 10 monclient(hunting): get_auth_request con 0x7fb504131c40 auth_method 0 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb508f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb508f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb508f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:04.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb5037fe640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient(hunting): handle_auth_done global_id 5110 payload 824 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 -- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb504131c40 msgr2=0x7fb50411d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb504131c40 0x7fb50411d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 -- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 msgr2=0x7fb504130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 monclient: found mon.noname-b 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb503fff640 1 -- 172.21.15.144:0/1137423884 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f8009390 con 0x7fb50406ea40 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb5037fe640 10 monclient: get_auth_request con 0x7fb50409cfb0 auth_method 0 2024-09-18T12:29:04.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.866+0000 7fb5037fe640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb503fff640 1 -- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 msgr2=0x7fb504141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb503fff640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 0x7fb504141190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb4f400c610 tx=0x7fb4f4007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4012840 con 0x7fb50406ea40 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: dump: 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.879 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: _finish_auth 0 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb50a99d640 5 monclient: authenticate success, global_id 5110 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb50a99d640 10 monclient: _renew_subs 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb50a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb504121140 con 0x7fb50406ea40 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f40129e0 con 0x7fb50406ea40 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:04.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4012da0 con 0x7fb50406ea40 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb4ebfff640 10 monclient: dump: 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:04.881 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:04.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb50a99d640 10 monclient: _renew_subs 2024-09-18T12:29:04.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb5027fc640 4 set_mon_vals no callback set 2024-09-18T12:29:04.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb50a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.868+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5041416d0 con 0x7fb50406ea40 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.869+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb4f4011590 con 0x7fb50406ea40 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.869+0000 7fb50a99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.869+0000 7fb50a99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.869+0000 7fb50a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.869+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5040636e0 con 0x7fb50406ea40 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.870+0000 7fb4ebfff640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4f0038490 0x7fb4f003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.870+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7fb4f404b8a0 con 0x7fb50406ea40 2024-09-18T12:29:04.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.870+0000 7fb5037fe640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4f0038490 0x7fb4f003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:04.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.870+0000 7fb5037fe640 10 monclient: get_auth_request con 0x7fb4f0038490 auth_method 0 2024-09-18T12:29:04.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.872+0000 7fb5037fe640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4f0038490 0x7fb4f003a940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb504067ad0 tx=0x7fb4f8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:04.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.874+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb4f401a710 con 0x7fb50406ea40 2024-09-18T12:29:04.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.874+0000 7fb4ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:04.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:04.874+0000 7fb4ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:05.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.032+0000 7fb50a99d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:05.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.032+0000 7fb50a99d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:05.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.032+0000 7fb50a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.032+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb504141920 con 0x7fb50406ea40 2024-09-18T12:29:05.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.033+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7fb4f4019310 con 0x7fb50406ea40 2024-09-18T12:29:05.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.033+0000 7fb4ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:05.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.033+0000 7fb4ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:05.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.128+0000 7fb50a99d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:05.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.128+0000 7fb50a99d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:05.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.128+0000 7fb50a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.128+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb504072640 con 0x7fb50406ea40 2024-09-18T12:29:05.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.128+0000 7fb4ebfff640 1 -- 172.21.15.144:0/1137423884 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) v1 ==== 56+0+1474 (secure 0 0 0) 0x7fb4f40194f0 con 0x7fb50406ea40 2024-09-18T12:29:05.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.128+0000 7fb4ebfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:05.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.128+0000 7fb4ebfff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4f0038490 msgr2=0x7fb4f003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4f0038490 0x7fb4f003a940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb504067ad0 tx=0x7fb4f8032000 comp rx=0 tx=0).stop 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 10 monclient: shutdown 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 msgr2=0x7fb504141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 0x7fb504141190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb4f400c610 tx=0x7fb4f4007ed0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 shutdown_connections 2024-09-18T12:29:05.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb4f0038490 0x7fb4f003a940 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb504131c40 0x7fb50411d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50409cfb0 0x7fb504130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 --2- 172.21.15.144:0/1137423884 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb50406ea40 0x7fb504141190 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 >> 172.21.15.144:0/1137423884 conn(0x7fb5040a12d0 msgr2=0x7fb504098b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 shutdown_connections 2024-09-18T12:29:05.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.130+0000 7fb50a99d640 1 -- 172.21.15.144:0/1137423884 wait complete. 2024-09-18T12:29:05.140 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-18T12:29:05.140 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:29:05.140 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm host1 foo 2024-09-18T12:29:05.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.213+0000 7f31f78d4640 10 monclient: get_monmap_and_config 2024-09-18T12:29:05.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.213+0000 7f31f78d4640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.213+0000 7f31f78d4640 10 monclient: monmap: 2024-09-18T12:29:05.214 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:05.214 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.214 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:05.214191+0000 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:05.214191+0000 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:05.215 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.213+0000 7f31f78d4640 1 Processor -- start 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 1 -- start start 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient: init 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f31f00a19a0 0x7f31f009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient(hunting): picked mon.noname-a con 0x7f31f00a19a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 0x7f31f009c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient(hunting): picked mon.noname-b con 0x7f31f009be20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31f0099250 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f009c730 con 0x7f31f0099250 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00a1d70 con 0x7f31f00a19a0 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0098600 con 0x7f31f009be20 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f78d4640 10 monclient(hunting): authenticate will time out at 980.783514s 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5e71640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f31f0099250 0x7f31f00996d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:35014/0 (socket says 172.21.15.144:35014) 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5e71640 1 -- 172.21.15.144:0/3582943685 learned_addr learned my addr 172.21.15.144:0/3582943685 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 1 --2- 172.21.15.144:0/3582943685 >> v2:172.21.15.116:3300/0 conn(0x7f31f00a19a0 0x7f31f009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 0x7f31f009c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 10 monclient(hunting): get_auth_request con 0x7f31f00a19a0 auth_method 0 2024-09-18T12:29:05.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 10 monclient(hunting): get_auth_request con 0x7f31f009be20 auth_method 0 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f5670640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 1 -- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 msgr2=0x7f31f009c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 1 -- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 msgr2=0x7f31f009c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 0x7f31f009c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.214+0000 7f31f4e6f640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 0x7f31f009c1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 10 monclient(hunting): handle_auth_done global_id 54339 payload 274 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 1 -- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 msgr2=0x7f31f009c1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 0x7f31f009c1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 1 -- 172.21.15.144:0/3582943685 >> v1:172.21.15.144:6789/0 conn(0x7f31f0099250 legacy=0x7f31f00996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.215+0000 7f31f5670640 1 -- 172.21.15.144:0/3582943685 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0097a70 con 0x7f31f00a19a0 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.217+0000 7f31f5670640 1 -- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f00a19a0 msgr2=0x7f31f009b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.217+0000 7f31f5670640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f00a19a0 0x7f31f009b8e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f31f009c8c0 tx=0x7f31ec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fbea1f01a2e89c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 1 -- 172.21.15.144:0/3582943685 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec02adc0 con 0x7f31f00a19a0 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: dump: 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.227 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 5 monclient: authenticate success, global_id 54339 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 1 -- 172.21.15.144:0/3582943685 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec032440 con 0x7f31f00a19a0 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:05.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 4 set_mon_vals no callback set 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 1 -- 172.21.15.144:0/3582943685 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec0327e0 con 0x7f31f00a19a0 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31e7fff640 10 monclient: dump: 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.229 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 10 monclient: shutdown 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 1 -- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f00a19a0 msgr2=0x7f31f009b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f00a19a0 0x7f31f009b8e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f31f009c8c0 tx=0x7f31ec0090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 1 -- 172.21.15.144:0/3582943685 shutdown_connections 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f009be20 0x7f31f009c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 1 --2- 172.21.15.144:0/3582943685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f00a19a0 0x7f31f009b8e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.218+0000 7f31f78d4640 1 -- 172.21.15.144:0/3582943685 >> 172.21.15.144:0/3582943685 conn(0x7f31f00711a0 msgr2=0x7f31f00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.219+0000 7f31f78d4640 1 -- 172.21.15.144:0/3582943685 shutdown_connections 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.219+0000 7f31f78d4640 1 -- 172.21.15.144:0/3582943685 wait complete. 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.219+0000 7f31f78d4640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.219+0000 7f31f78d4640 10 monclient: monmap: 2024-09-18T12:29:05.230 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.219+0000 7f31f78d4640 1 Processor -- start 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.219+0000 7f31f78d4640 1 -- start start 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.219+0000 7f31f78d4640 10 monclient: init 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f0099250 0x7f31f0130110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): picked mon.noname-c con 0x7f31f0099250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f009be20 0x7f31f0132e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): picked mon.noname-a con 0x7f31f009be20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f00a19a0 0x7f31f01333b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): picked mon.noname-b con 0x7f31f00a19a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f009cc10 con 0x7f31f009be20 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0130780 con 0x7f31f00a19a0 2024-09-18T12:29:05.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0136800 con 0x7f31f0099250 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f78d4640 10 monclient(hunting): authenticate will time out at 980.789404s 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f00a19a0 0x7f31f01333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f00a19a0 0x7f31f01333b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41006/0 (socket says 172.21.15.144:41006) 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 -- 172.21.15.144:0/2672046635 learned_addr learned my addr 172.21.15.144:0/2672046635 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): get_auth_request con 0x7f31f00a19a0 auth_method 0 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5670640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f0099250 0x7f31f0130110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f4e6f640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f009be20 0x7f31f0132e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient(hunting): handle_auth_done global_id 5113 payload 824 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 -- 172.21.15.144:0/2672046635 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f0099250 msgr2=0x7f31f0130110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f31f0099250 0x7f31f0130110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 -- 172.21.15.144:0/2672046635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f009be20 msgr2=0x7f31f0132e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31f009be20 0x7f31f0132e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 monclient: found mon.noname-b 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.220+0000 7f31f5e71640 1 -- 172.21.15.144:0/2672046635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec009390 con 0x7f31f00a19a0 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.222+0000 7f31f5e71640 1 -- 172.21.15.144:0/2672046635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f00a19a0 msgr2=0x7f31f01333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.222+0000 7f31f5e71640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31f00a19a0 0x7f31f01333b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f31e800bb80 tx=0x7f31e80097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 1 -- 172.21.15.144:0/2672046635 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e800cc80 con 0x7f31f00a19a0 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: dump: 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.235 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31f78d4640 5 monclient: authenticate success, global_id 5113 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 1 -- 172.21.15.144:0/2672046635 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e800cde0 con 0x7f31f00a19a0 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31f78d4640 10 monclient: _renew_subs 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31f78d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31f78d4640 1 -- 172.21.15.144:0/2672046635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f0135140 con 0x7f31f00a19a0 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 1 -- 172.21.15.144:0/2672046635 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8014690 con 0x7f31f00a19a0 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 10 monclient: dump: 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.237 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e77fe640 4 set_mon_vals no callback set 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31f78d4640 10 monclient: _renew_subs 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31f78d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31f78d4640 1 -- 172.21.15.144:0/2672046635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f012f540 con 0x7f31f00a19a0 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.223+0000 7f31e57fa640 1 -- 172.21.15.144:0/2672046635 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f31e801d3b0 con 0x7f31f00a19a0 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31f78d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31f78d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31f78d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31f78d4640 1 -- 172.21.15.144:0/2672046635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f00008d0 con 0x7f31f00a19a0 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31e57fa640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31d4038490 0x7f31d403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31e57fa640 1 -- 172.21.15.144:0/2672046635 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f31e804bdf0 con 0x7f31f00a19a0 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31f5670640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31d4038490 0x7f31d403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.224+0000 7f31f5670640 10 monclient: get_auth_request con 0x7f31d4038490 auth_method 0 2024-09-18T12:29:05.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.226+0000 7f31f5670640 1 --2- 172.21.15.144:0/2672046635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31d4038490 0x7f31d403a940 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f31f009cf50 tx=0x7f31ec032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.228+0000 7f31e57fa640 1 -- 172.21.15.144:0/2672046635 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f31e804f030 con 0x7f31f00a19a0 2024-09-18T12:29:05.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.228+0000 7f31e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.228+0000 7f31e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:05.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:29:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.344+0000 7f7b9760f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.344+0000 7f7b9760f640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.344+0000 7f7b9760f640 10 monclient: monmap: 2024-09-18T12:29:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:05.345324+0000 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:05.345324+0000 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:05.346 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 1 Processor -- start 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 1 -- start start 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient: init 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b90096c60 0x7f7b9009a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient(hunting): picked mon.noname-a con 0x7f7b90096c60 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b90097030 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 0x7f7b90135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient(hunting): picked mon.noname-b con 0x7f7b90133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900696a0 con 0x7f7b90097030 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b9009ac00 con 0x7f7b90096c60 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9009d6b0 con 0x7f7b90133b40 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.345+0000 7f7b9760f640 10 monclient(hunting): authenticate will time out at 972.666989s 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 0x7f7b90135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b90096c60 0x7f7b9009a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 0x7f7b90135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:37620/0 (socket says 172.21.15.116:37620) 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 1 -- 172.21.15.116:0/4178061844 learned_addr learned my addr 172.21.15.116:0/4178061844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 10 monclient(hunting): get_auth_request con 0x7f7b90133b40 auth_method 0 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): get_auth_request con 0x7f7b90096c60 auth_method 0 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 1 -- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 msgr2=0x7f7b90135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 1 -- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 msgr2=0x7f7b90135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 1 --2- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 0x7f7b90135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b95bac640 1 --2- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 0x7f7b90135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient(hunting): handle_auth_done global_id 54342 payload 274 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 1 -- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 msgr2=0x7f7b90135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 1 --2- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 0x7f7b90135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 1 -- 172.21.15.116:0/4178061844 >> v1:172.21.15.144:6789/0 conn(0x7f7b90097030 legacy=0x7f7b90097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.346+0000 7f7b953ab640 1 -- 172.21.15.116:0/4178061844 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9009d2e0 con 0x7f7b90096c60 2024-09-18T12:29:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b953ab640 1 -- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90096c60 msgr2=0x7f7b9009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b953ab640 1 --2- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90096c60 0x7f7b9009a6c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009850 tx=0x7f7b8c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54aba28f4dcdc96e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 1 -- 172.21.15.116:0/4178061844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c02add0 con 0x7f7b90096c60 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: dump: 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 5 monclient: authenticate success, global_id 54342 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 1 -- 172.21.15.116:0/4178061844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8c032440 con 0x7f7b90096c60 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 1 -- 172.21.15.116:0/4178061844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0327e0 con 0x7f7b90096c60 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 4 set_mon_vals no callback set 2024-09-18T12:29:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b87fff640 10 monclient: dump: 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 10 monclient: shutdown 2024-09-18T12:29:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 1 -- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90096c60 msgr2=0x7f7b9009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 1 --2- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90096c60 0x7f7b9009a6c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009850 tx=0x7f7b8c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 1 -- 172.21.15.116:0/4178061844 shutdown_connections 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 1 --2- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90133b40 0x7f7b90135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 1 --2- 172.21.15.116:0/4178061844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90096c60 0x7f7b9009a6c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.349+0000 7f7b9760f640 1 -- 172.21.15.116:0/4178061844 >> 172.21.15.116:0/4178061844 conn(0x7f7b900672b0 msgr2=0x7f7b900676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.350+0000 7f7b9760f640 1 -- 172.21.15.116:0/4178061844 shutdown_connections 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.350+0000 7f7b9760f640 1 -- 172.21.15.116:0/4178061844 wait complete. 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.350+0000 7f7b9760f640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.350+0000 7f7b9760f640 10 monclient: monmap: 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.350+0000 7f7b9760f640 1 Processor -- start 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.350+0000 7f7b9760f640 1 -- start start 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.350+0000 7f7b9760f640 10 monclient: init 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 0x7f7b9009b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): picked mon.noname-c con 0x7f7b90096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b90097030 0x7f7b9012eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): picked mon.noname-b con 0x7f7b90097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90133b40 0x7f7b9012f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): picked mon.noname-a con 0x7f7b90133b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9009add0 con 0x7f7b90133b40 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b901337a0 con 0x7f7b90097030 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90118910 con 0x7f7b90096c60 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b9760f640 10 monclient(hunting): authenticate will time out at 972.672608s 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 0x7f7b9009b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 0x7f7b9009b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:37632/0 (socket says 172.21.15.116:37632) 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 1 -- 172.21.15.116:0/1067637387 learned_addr learned my addr 172.21.15.116:0/1067637387 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 10 monclient(hunting): get_auth_request con 0x7f7b90096c60 auth_method 0 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90133b40 0x7f7b9012f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): get_auth_request con 0x7f7b90133b40 auth_method 0 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 1 -- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 msgr2=0x7f7b9009b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 1 -- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 msgr2=0x7f7b9009b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 0x7f7b9009b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b90097030 0x7f7b9012eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b953ab640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 0x7f7b9009b810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 10 monclient(hunting): get_auth_request con 0x7f7b90097030 auth_method 0 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b94baa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 10 monclient(hunting): handle_auth_done global_id 54345 payload 824 2024-09-18T12:29:05.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.351+0000 7f7b95bac640 1 -- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 msgr2=0x7f7b9009b810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 0x7f7b9009b810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 1 -- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b90097030 msgr2=0x7f7b9012eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b90097030 0x7f7b9012eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.352+0000 7f7b95bac640 1 -- 172.21.15.116:0/1067637387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c009390 con 0x7f7b90133b40 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b95bac640 1 -- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90133b40 msgr2=0x7f7b9012f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b95bac640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90133b40 0x7f7b9012f080 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009650 tx=0x7f7b88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 1 -- 172.21.15.116:0/1067637387 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800dc00 con 0x7f7b90133b40 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: dump: 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b9760f640 5 monclient: authenticate success, global_id 54345 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b9760f640 10 monclient: _renew_subs 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b9760f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9011dc80 con 0x7f7b90133b40 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 1 -- 172.21.15.116:0/1067637387 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8800f040 con 0x7f7b90133b40 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 1 -- 172.21.15.116:0/1067637387 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88015400 con 0x7f7b90133b40 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b857fa640 10 monclient: dump: 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.369 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b877fe640 4 set_mon_vals no callback set 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b9760f640 10 monclient: _renew_subs 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b9760f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.354+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b90118450 con 0x7f7b90133b40 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b857fa640 1 -- 172.21.15.116:0/1067637387 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7b88003020 con 0x7f7b90133b40 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b9760f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b9760f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b9760f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9011e320 con 0x7f7b90133b40 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b857fa640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b74038490 0x7f7b7403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b857fa640 1 -- 172.21.15.116:0/1067637387 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f7b8804ba80 con 0x7f7b90133b40 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b953ab640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b74038490 0x7f7b7403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.355+0000 7f7b953ab640 10 monclient: get_auth_request con 0x7f7b74038490 auth_method 0 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.358+0000 7f7b953ab640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b74038490 0x7f7b7403a940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c000e10 tx=0x7f7b8c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.359+0000 7f7b857fa640 1 -- 172.21.15.116:0/1067637387 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7b88013ae0 con 0x7f7b90133b40 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.359+0000 7f7b857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.359+0000 7f7b857fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:05.391 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.390+0000 7f31f78d4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:05.391 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.390+0000 7f31f78d4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:05.391 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.390+0000 7f31f78d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.390+0000 7f31f78d4640 1 -- 172.21.15.144:0/2672046635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f31f0063210 con 0x7f31f00a19a0 2024-09-18T12:29:05.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.391+0000 7f31e57fa640 1 -- 172.21.15.144:0/2672046635 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v79) v1 ==== 120+0+0 (secure 0 0 0) 0x7f31e8019020 con 0x7f31f00a19a0 2024-09-18T12:29:05.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.391+0000 7f31e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:05.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.391+0000 7f31e57fa640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T12:29:05.393 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T12:29:05.396 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:29:05.397 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush unlink host1 foo 2024-09-18T12:29:05.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.469+0000 7f8e0c2f8640 10 monclient: get_monmap_and_config 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.469+0000 7f8e0c2f8640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.469+0000 7f8e0c2f8640 10 monclient: monmap: 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:05.470598+0000 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:05.470598+0000 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:05.471 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:05.472 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:05.472 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:05.472 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:05.472 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 1 Processor -- start 2024-09-18T12:29:05.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 1 -- start start 2024-09-18T12:29:05.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient: init 2024-09-18T12:29:05.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T12:29:05.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e0409f180 0x7f8e04130040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient(hunting): picked mon.noname-a con 0x7f8e0409f180 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e0409cee0 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e04134c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0409e610 con 0x7f8e0409cee0 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0409f550 con 0x7f8e04134c60 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0409e2a0 con 0x7f8e0409f180 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.470+0000 7f8e0c2f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0c2f8640 10 monclient(hunting): authenticate will time out at 981.040007s 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a895640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f8e04134c60 0x7f8e04137040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:35020/0 (socket says 172.21.15.144:35020) 2024-09-18T12:29:05.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a895640 1 -- 172.21.15.144:0/2941571037 learned_addr learned my addr 172.21.15.144:0/2941571037 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 1 --2- 172.21.15.144:0/2941571037 >> v2:172.21.15.116:3300/0 conn(0x7f8e0409f180 0x7f8e04130040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): get_auth_request con 0x7f8e0409f180 auth_method 0 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient(hunting): handle_auth_done global_id 54348 payload 274 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 1 -- 172.21.15.144:0/2941571037 >> v1:172.21.15.144:6789/0 conn(0x7f8e04134c60 legacy=0x7f8e04137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 1 -- 172.21.15.144:0/2941571037 >> v1:172.21.15.116:6789/0 conn(0x7f8e0409cee0 legacy=0x7f8e0409bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:05.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.471+0000 7f8e0a094640 1 -- 172.21.15.144:0/2941571037 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0409c250 con 0x7f8e0409f180 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e0a094640 1 -- 172.21.15.144:0/2941571037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 msgr2=0x7f8e04130040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e0a094640 1 --2- 172.21.15.144:0/2941571037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 0x7f8e04130040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009850 tx=0x7f8e000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d39e0ea310eb02 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 1 -- 172.21.15.144:0/2941571037 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00032440 con 0x7f8e0409f180 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: dump: 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.482 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 1 -- 172.21.15.144:0/2941571037 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e000325a0 con 0x7f8e0409f180 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 1 -- 172.21.15.144:0/2941571037 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00032940 con 0x7f8e0409f180 2024-09-18T12:29:05.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e0c2f8640 5 monclient: authenticate success, global_id 54348 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e0c2f8640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e0c2f8640 4 set_mon_vals no callback set 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.474+0000 7f8e09092640 10 monclient: dump: 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.484 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 10 monclient: shutdown 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/2941571037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 msgr2=0x7f8e04130040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/2941571037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 0x7f8e04130040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009850 tx=0x7f8e000090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/2941571037 shutdown_connections 2024-09-18T12:29:05.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/2941571037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 0x7f8e04130040 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/2941571037 >> 172.21.15.144:0/2941571037 conn(0x7f8e040a2b70 msgr2=0x7f8e0409ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/2941571037 shutdown_connections 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/2941571037 wait complete. 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.475+0000 7f8e0c2f8640 10 monclient: monmap: 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.486 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 Processor -- start 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 -- start start 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient: init 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 0x7f8e041342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): picked mon.noname-c con 0x7f8e0409cee0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 0x7f8e0406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): picked mon.noname-a con 0x7f8e0409f180 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:05.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 0x7f8e0406ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): picked mon.noname-b con 0x7f8e04134c60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0406bda0 con 0x7f8e0409f180 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0406ec70 con 0x7f8e04134c60 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e040a1460 con 0x7f8e0409cee0 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0c2f8640 10 monclient(hunting): authenticate will time out at 981.045654s 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 0x7f8e0406ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 0x7f8e0406ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41012/0 (socket says 172.21.15.144:41012) 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 1 -- 172.21.15.144:0/1517127277 learned_addr learned my addr 172.21.15.144:0/1517127277 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 10 monclient(hunting): get_auth_request con 0x7f8e04134c60 auth_method 0 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 0x7f8e041342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 0x7f8e0406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 10 monclient(hunting): get_auth_request con 0x7f8e0409cee0 auth_method 0 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a895640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 10 monclient(hunting): get_auth_request con 0x7f8e0409f180 auth_method 0 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 1 -- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 msgr2=0x7f8e041342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 1 -- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 msgr2=0x7f8e041342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:05.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 0x7f8e041342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e0a094640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 0x7f8e041342f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.476+0000 7f8e09893640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 10 monclient(hunting): handle_auth_done global_id 5119 payload 824 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 1 -- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 msgr2=0x7f8e041342f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 0x7f8e041342f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 1 -- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 msgr2=0x7f8e0406a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 0x7f8e0406a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 1 monclient: found mon.noname-b 2024-09-18T12:29:05.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.477+0000 7f8e0a895640 1 -- 172.21.15.144:0/1517127277 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00009390 con 0x7f8e04134c60 2024-09-18T12:29:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0a895640 1 -- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 msgr2=0x7f8e0406ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0a895640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 0x7f8e0406ad30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00ccf0 tx=0x7f8dfc005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc00aa60 con 0x7f8e04134c60 2024-09-18T12:29:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: dump: 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.495 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc00abe0 con 0x7f8e04134c60 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0c2f8640 5 monclient: authenticate success, global_id 5119 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0c2f8640 10 monclient: _renew_subs 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0c2f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0406d840 con 0x7f8e04134c60 2024-09-18T12:29:05.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc0070f0 con 0x7f8e04134c60 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8dfa7fc640 10 monclient: dump: 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0c2f8640 10 monclient: _renew_subs 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e08891640 4 set_mon_vals no callback set 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0c2f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.479+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0406e500 con 0x7f8e04134c60 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8dfc0074a0 con 0x7f8e04134c60 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8e0c2f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8e0c2f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8e0c2f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e040008d0 con 0x7f8e04134c60 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8dfa7fc640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8de8038490 0x7f8de803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f8dfc04bef0 con 0x7f8e04134c60 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8e0a094640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8de8038490 0x7f8de803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.480+0000 7f8e0a094640 10 monclient: get_auth_request con 0x7f8de8038490 auth_method 0 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.482+0000 7f8e0a094640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8de8038490 0x7f8de803a940 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8e040a3050 tx=0x7f8e00032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.484+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8dfc01a360 con 0x7f8e04134c60 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.484+0000 7f8dfa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.484+0000 7f8dfa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.574+0000 7f7b9760f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:29:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.574+0000 7f7b9760f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:29:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.574+0000 7f7b9760f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.574+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7b9004bf60 con 0x7f7b90133b40 2024-09-18T12:29:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.575+0000 7f7b857fa640 1 -- 172.21.15.116:0/1067637387 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7b88019070 con 0x7f7b90133b40 2024-09-18T12:29:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.575+0000 7f7b857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:29:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.575+0000 7f7b857fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:05.577 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:29:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b74038490 msgr2=0x7f7b7403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b74038490 0x7f7b7403a940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c000e10 tx=0x7f7b8c032000 comp rx=0 tx=0).stop 2024-09-18T12:29:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 10 monclient: shutdown 2024-09-18T12:29:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90133b40 msgr2=0x7f7b9012f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90133b40 0x7f7b9012f080 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009650 tx=0x7f7b88009250 comp rx=0 tx=0).stop 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 shutdown_connections 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b74038490 0x7f7b7403a940 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b90133b40 0x7f7b9012f080 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b90097030 0x7f7b9012eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 --2- 172.21.15.116:0/1067637387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b90096c60 0x7f7b9009b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 >> 172.21.15.116:0/1067637387 conn(0x7f7b900672b0 msgr2=0x7f7b9012fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 shutdown_connections 2024-09-18T12:29:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.577+0000 7f7b9760f640 1 -- 172.21.15.116:0/1067637387 wait complete. 2024-09-18T12:29:05.589 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:29:05.589 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:29:05.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:29:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.646+0000 7f8e0c2f8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.646+0000 7f8e0c2f8640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.646+0000 7f8e0c2f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:05.646+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f8e04130580 con 0x7f8e04134c60 2024-09-18T12:29:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.682+0000 7f6544714640 10 monclient: get_monmap_and_config 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.682+0000 7f6544714640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.682+0000 7f6544714640 10 monclient: monmap: 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:05.683577+0000 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:05.683577+0000 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:05.685 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.683+0000 7f6544714640 1 Processor -- start 2024-09-18T12:29:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.683+0000 7f6544714640 1 -- start start 2024-09-18T12:29:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.683+0000 7f6544714640 10 monclient: init 2024-09-18T12:29:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): picked mon.noname-b con 0x7f653c0974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f653c09a8a0 0x7f653c09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): picked mon.noname-a con 0x7f653c09a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f653c12dd40 0x7f653c12e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): picked mon.noname-c con 0x7f653c12dd40 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653c0978c0 con 0x7f653c09a8a0 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653c09ca20 con 0x7f653c12dd40 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653c09b1f0 con 0x7f653c0974f0 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6544714640 10 monclient(hunting): authenticate will time out at 973.005296s 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f653c09a8a0 0x7f653c09acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f653c09a8a0 0x7f653c09acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51924/0 (socket says 172.21.15.116:51924) 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 -- 172.21.15.116:0/1044832170 learned_addr learned my addr 172.21.15.116:0/1044832170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59428/0 (socket says 172.21.15.116:59428) 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): get_auth_request con 0x7f653c09a8a0 auth_method 0 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 10 monclient(hunting): get_auth_request con 0x7f653c0974f0 auth_method 0 2024-09-18T12:29:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 1 --2- 172.21.15.116:0/1044832170 >> v2:172.21.15.144:3300/0 conn(0x7f653c12dd40 0x7f653c12e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 10 monclient(hunting): get_auth_request con 0x7f653c12dd40 auth_method 0 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 1 -- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 msgr2=0x7f653c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 1 -- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 msgr2=0x7f653c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 1 --2- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f65424b0640 1 --2- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient(hunting): handle_auth_done global_id 54354 payload 274 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 -- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 msgr2=0x7f653c12ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 --2- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 -- 172.21.15.116:0/1044832170 >> v2:172.21.15.144:3300/0 conn(0x7f653c12dd40 msgr2=0x7f653c12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 --2- 172.21.15.116:0/1044832170 >> v2:172.21.15.144:3300/0 conn(0x7f653c12dd40 0x7f653c12e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6541caf640 1 -- 172.21.15.116:0/1044832170 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c098ad0 con 0x7f653c09a8a0 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.684+0000 7f6542cb1640 1 --2- 172.21.15.116:0/1044832170 >> v2:172.21.15.144:3300/0 conn(0x7f653c12dd40 0x7f653c12e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f6541caf640 1 -- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 msgr2=0x7f653c09acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f6541caf640 1 --2- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c09acb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f653c065310 tx=0x7f653000b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b11a407ba3f264 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 1 -- 172.21.15.116:0/1044832170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530032690 con 0x7f653c09a8a0 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: dump: 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 1 -- 172.21.15.116:0/1044832170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653002f030 con 0x7f653c09a8a0 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f6544714640 5 monclient: authenticate success, global_id 54354 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f6544714640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f6544714640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 1 -- 172.21.15.116:0/1044832170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530032a20 con 0x7f653c09a8a0 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f6544714640 4 set_mon_vals no callback set 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.687+0000 7f65414ae640 10 monclient: dump: 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 10 monclient: shutdown 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 -- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 msgr2=0x7f653c09acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 --2- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c09acb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f653c065310 tx=0x7f653000b4b0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 -- 172.21.15.116:0/1044832170 shutdown_connections 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 --2- 172.21.15.116:0/1044832170 >> v2:172.21.15.144:3300/0 conn(0x7f653c12dd40 0x7f653c12e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 --2- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c09acb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 --2- 172.21.15.116:0/1044832170 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c12ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 -- 172.21.15.116:0/1044832170 >> 172.21.15.116:0/1044832170 conn(0x7f653c0672b0 msgr2=0x7f653c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 -- 172.21.15.116:0/1044832170 shutdown_connections 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 1 -- 172.21.15.116:0/1044832170 wait complete. 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.688+0000 7f6544714640 10 monclient: monmap: 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 Processor -- start 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 -- start start 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient: init 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c06d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): picked mon.noname-c con 0x7f653c0974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c06d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): picked mon.noname-a con 0x7f653c09a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f653c12dd40 0x7f653c06db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): picked mon.noname-b con 0x7f653c12dd40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c11cf20 con 0x7f653c09a8a0 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c098300 con 0x7f653c12dd40 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653c09df70 con 0x7f653c0974f0 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6544714640 10 monclient(hunting): authenticate will time out at 973.010726s 2024-09-18T12:29:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c06d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c06d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c06d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59436/0 (socket says 172.21.15.116:59436) 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c06d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51930/0 (socket says 172.21.15.116:51930) 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 1 -- 172.21.15.116:0/302638833 learned_addr learned my addr 172.21.15.116:0/302638833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 10 monclient(hunting): get_auth_request con 0x7f653c0974f0 auth_method 0 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 10 monclient(hunting): get_auth_request con 0x7f653c09a8a0 auth_method 0 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 msgr2=0x7f653c06d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:29:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 msgr2=0x7f653c06d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c06d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f6541caf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.689+0000 7f65424b0640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c06d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 10 monclient(hunting): handle_auth_done global_id 54357 payload 824 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6542cb1640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f653c12dd40 0x7f653c06db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 msgr2=0x7f653c06d0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f653c0974f0 0x7f653c06d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f653c12dd40 msgr2=0x7f653c06db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f653c12dd40 0x7f653c06db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.690+0000 7f6541caf640 1 -- 172.21.15.116:0/302638833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6534009390 con 0x7f653c09a8a0 2024-09-18T12:29:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6541caf640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 msgr2=0x7f653c06d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6541caf640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c06d610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f653c068d80 tx=0x7f653003a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530035830 con 0x7f653c09a8a0 2024-09-18T12:29:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: dump: 2024-09-18T12:29:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6544714640 5 monclient: authenticate success, global_id 54357 2024-09-18T12:29:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530035990 con 0x7f653c09a8a0 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6544714640 10 monclient: _renew_subs 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6544714640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653000b6f0 con 0x7f653c09a8a0 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530035d50 con 0x7f653c09a8a0 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f653a7fc640 10 monclient: dump: 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6544714640 10 monclient: _renew_subs 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6544714640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c11c5f0 con 0x7f653c09a8a0 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.692+0000 7f6540cad640 4 set_mon_vals no callback set 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f6544714640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f6544714640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 0x7f653c0696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c065f20 con 0x7f653c06e090 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 0x7f653c0696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient: get_auth_request con 0x7f653c06e090 auth_method 0 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f653003c070 con 0x7f653c09a8a0 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 10 monclient(hunting): handle_auth_done global_id 54360 payload 824 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f65424b0640 1 -- 172.21.15.116:0/302638833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f653c11c340 con 0x7f653c06e090 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f653a7fc640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65280384e0 0x7f652803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.693+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f653006ced0 con 0x7f653c09a8a0 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.694+0000 7f6542cb1640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65280384e0 0x7f652803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.694+0000 7f6542cb1640 10 monclient: get_auth_request con 0x7f65280384e0 auth_method 0 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.695+0000 7f65424b0640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 msgr2=0x7f653c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.695+0000 7f65424b0640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 0x7f653c0696a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f652c009f40 tx=0x7f652c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c0155f0 con 0x7f653c06e090 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 10 monclient: dump: 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.716 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f652c015750 con 0x7f653c06e090 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 msgr2=0x7f653c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 0x7f653c0696a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f652c009f40 tx=0x7f652c00b040 comp rx=0 tx=0).stop 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 msgr2=0x7f653c0696a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f653a7fc640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c06e090 0x7f653c0696a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.696+0000 7f6542cb1640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65280384e0 0x7f652803a990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6534006e70 tx=0x7f6534008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.701+0000 7f6544714640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.701+0000 7f6544714640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 0x7f653c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.701+0000 7f6544714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.701+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c04bf60 con 0x7f653c069be0 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 0x7f653c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient: get_auth_request con 0x7f653c069be0 auth_method 0 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 10 monclient(hunting): handle_auth_done global_id 54363 payload 824 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.702+0000 7f65424b0640 1 -- 172.21.15.116:0/302638833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f653c0638b0 con 0x7f653c069be0 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f65424b0640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 msgr2=0x7f653c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f65424b0640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 0x7f653c069fe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f652c01a000 tx=0x7f652c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c027070 con 0x7f653c069be0 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 10 monclient: dump: 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f652c0153e0 con 0x7f653c069be0 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 msgr2=0x7f653c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 0x7f653c069fe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f652c01a000 tx=0x7f652c01d000 comp rx=0 tx=0).stop 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 msgr2=0x7f653c069fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.704+0000 7f653a7fc640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c069be0 0x7f653c069fe0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f652c01a000 tx=0x7f652c01d000 comp rx=0 tx=0).stop 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 44, 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:29:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.727 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:05.728 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:05.730 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:05.731 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:05.732 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:05.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:05.733 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:05.733 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65280384e0 msgr2=0x7f652803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65280384e0 0x7f652803a990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6534006e70 tx=0x7f6534008040 comp rx=0 tx=0).stop 2024-09-18T12:29:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 10 monclient: shutdown 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 msgr2=0x7f653c06d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 --2- 172.21.15.116:0/302638833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f653c09a8a0 0x7f653c06d610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f653c068d80 tx=0x7f653003a000 comp rx=0 tx=0).stop 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6542cb1640 1 -- 172.21.15.116:0/302638833 reap_dead start 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6542cb1640 1 -- 172.21.15.116:0/302638833 reap_dead start 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 shutdown_connections 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 >> 172.21.15.116:0/302638833 conn(0x7f653c0672b0 msgr2=0x7f653c0a0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 shutdown_connections 2024-09-18T12:29:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.705+0000 7f6544714640 1 -- 172.21.15.116:0/302638833 wait complete. 2024-09-18T12:29:05.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:29:05.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:29:05.736 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.828+0000 7f126135d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.828+0000 7f126135d640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.828+0000 7f126135d640 10 monclient: monmap: 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:05.828890+0000 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:05.828890+0000 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:05.830 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.828+0000 7f126135d640 1 Processor -- start 2024-09-18T12:29:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.828+0000 7f126135d640 1 -- start start 2024-09-18T12:29:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.828+0000 7f126135d640 10 monclient: init 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient(hunting): picked mon.noname-b con 0x7f125c1318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f125c09ea50 0x7f125c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient(hunting): picked mon.noname-a con 0x7f125c09ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f125c068d90 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f125c131cc0 con 0x7f125c068d90 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c09fbf0 con 0x7f125c09ea50 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c09fe30 con 0x7f125c1318f0 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f126135d640 10 monclient(hunting): authenticate will time out at 973.150436s 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f125c09ea50 0x7f125c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59448/0 (socket says 172.21.15.116:59448) 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f125c09ea50 0x7f125c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51962/0 (socket says 172.21.15.116:51962) 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 1 -- 172.21.15.116:0/1413191834 learned_addr learned my addr 172.21.15.116:0/1413191834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 10 monclient(hunting): get_auth_request con 0x7f125c1318f0 auth_method 0 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125b7fe640 1 --1- 172.21.15.116:0/1413191834 >> v1:172.21.15.116:6789/0 conn(0x7f125c068d90 0x7f125c0691f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48546/0 (socket says 172.21.15.116:48546) 2024-09-18T12:29:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): get_auth_request con 0x7f125c09ea50 auth_method 0 2024-09-18T12:29:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 1 -- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 msgr2=0x7f125c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 1 -- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 msgr2=0x7f125c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 1 --2- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125affd640 1 --2- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient(hunting): handle_auth_done global_id 54366 payload 274 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 1 -- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 msgr2=0x7f125c131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 1 --2- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 1 -- 172.21.15.116:0/1413191834 >> v1:172.21.15.116:6789/0 conn(0x7f125c068d90 legacy=0x7f125c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.829+0000 7f125a7fc640 1 -- 172.21.15.116:0/1413191834 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c0a1210 con 0x7f125c09ea50 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f125a7fc640 1 -- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c09ea50 msgr2=0x7f125c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f125a7fc640 1 --2- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c09ea50 0x7f125c09ee20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1250009e60 tx=0x7f1250009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd7ba37c0a69cda3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 1 -- 172.21.15.116:0/1413191834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250007760 con 0x7f125c09ea50 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: dump: 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 1 -- 172.21.15.116:0/1413191834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125002d050 con 0x7f125c09ea50 2024-09-18T12:29:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f126135d640 5 monclient: authenticate success, global_id 54366 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f126135d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f126135d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 1 -- 172.21.15.116:0/1413191834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250007af0 con 0x7f125c09ea50 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f126135d640 4 set_mon_vals no callback set 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.832+0000 7f1259ffb640 10 monclient: dump: 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 10 monclient: shutdown 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 -- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c09ea50 msgr2=0x7f125c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 --2- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c09ea50 0x7f125c09ee20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1250009e60 tx=0x7f1250009780 comp rx=0 tx=0).stop 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 -- 172.21.15.116:0/1413191834 shutdown_connections 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 --2- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c09ea50 0x7f125c09ee20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 --2- 172.21.15.116:0/1413191834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 -- 172.21.15.116:0/1413191834 >> 172.21.15.116:0/1413191834 conn(0x7f125c067110 msgr2=0x7f125c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 -- 172.21.15.116:0/1413191834 shutdown_connections 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 1 -- 172.21.15.116:0/1413191834 wait complete. 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 10 monclient: build_initial_monmap 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.833+0000 7f126135d640 10 monclient: monmap: 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 Processor -- start 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 -- start start 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient: init 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c068d90 0x7f125c09aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): picked mon.noname-a con 0x7f125c068d90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f125c09ea50 0x7f125c130bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): picked mon.noname-b con 0x7f125c09ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c11b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): picked mon.noname-c con 0x7f125c1318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c11dfb0 con 0x7f125c068d90 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c11cba0 con 0x7f125c09ea50 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c09bf10 con 0x7f125c1318f0 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f126135d640 10 monclient(hunting): authenticate will time out at 973.155823s 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c068d90 0x7f125c09aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c068d90 0x7f125c09aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51968/0 (socket says 172.21.15.116:51968) 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 1 -- 172.21.15.116:0/2882155585 learned_addr learned my addr 172.21.15.116:0/2882155585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 10 monclient(hunting): get_auth_request con 0x7f125c068d90 auth_method 0 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125b7fe640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c11b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125b7fe640 10 monclient(hunting): get_auth_request con 0x7f125c1318f0 auth_method 0 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.834+0000 7f125affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125b7fe640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 msgr2=0x7f125c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125b7fe640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 msgr2=0x7f125c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125b7fe640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c11b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125b7fe640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c11b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f125c09ea50 0x7f125c130bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125a7fc640 10 monclient(hunting): get_auth_request con 0x7f125c09ea50 auth_method 0 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 10 monclient(hunting): handle_auth_done global_id 54369 payload 824 2024-09-18T12:29:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 msgr2=0x7f125c11b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1318f0 0x7f125c11b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f125c09ea50 msgr2=0x7f125c130bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f125c09ea50 0x7f125c130bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 1 monclient: found mon.noname-a 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125affd640 1 -- 172.21.15.116:0/2882155585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1250009a20 con 0x7f125c068d90 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.835+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f125c09ea50 0x7f125c130bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f125affd640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c068d90 msgr2=0x7f125c09aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f125affd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c068d90 0x7f125c09aa00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f125400e870 tx=0x7f1254008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125400c6d0 con 0x7f125c068d90 2024-09-18T12:29:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: dump: 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125400c830 con 0x7f125c068d90 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f126135d640 5 monclient: authenticate success, global_id 54369 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f126135d640 10 monclient: _renew_subs 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f126135d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c141e00 con 0x7f125c068d90 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125400cbf0 con 0x7f125c068d90 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f1242ffd640 10 monclient: dump: 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f126135d640 10 monclient: _renew_subs 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f126135d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125c11dcd0 con 0x7f125c068d90 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.837+0000 7f12597fa640 4 set_mon_vals no callback set 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f126135d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f126135d640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f126135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c065e00 con 0x7f125c1310f0 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1254013410 con 0x7f125c068d90 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 10 monclient: get_auth_request con 0x7f125c1310f0 auth_method 0 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f1242ffd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1248038490 0x7f124803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.838+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f125404bce0 con 0x7f125c068d90 2024-09-18T12:29:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.839+0000 7f125b7fe640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1248038490 0x7f124803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.839+0000 7f125b7fe640 10 monclient: get_auth_request con 0x7f1248038490 auth_method 0 2024-09-18T12:29:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:05.841+0000 7f125b7fe640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1248038490 0x7f124803a940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f125c09b950 tx=0x7f124c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 10 monclient: get_auth_request con 0x7f125c1310f0 auth_method 0 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.039+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:29:06.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.333+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v80) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8dfc010af0 con 0x7f8e04134c60 2024-09-18T12:29:06.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.333+0000 7f8dfa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.333+0000 7f8dfa7fc640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-18T12:29:06.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.436+0000 7f8e0c2f8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.436+0000 7f8e0c2f8640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.436+0000 7f8e0c2f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:06.437 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.436+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f8e0409df10 con 0x7f8e04134c60 2024-09-18T12:29:06.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 10 monclient: get_auth_request con 0x7f125c1310f0 auth_method 0 2024-09-18T12:29:06.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:06.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:06.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:29:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:06.440+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:29:06.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.606+0000 7f8dfa7fc640 1 -- 172.21.15.144:0/1517127277 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v80) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8dfc010cd0 con 0x7f8e04134c60 2024-09-18T12:29:06.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.606+0000 7f8dfa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.606+0000 7f8dfa7fc640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-18T12:29:06.611 INFO:tasks.workunit.client.0.smithi144.stderr:item -11 does not appear in that position 2024-09-18T12:29:06.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8de8038490 msgr2=0x7f8de803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:06.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8de8038490 0x7f8de803a940 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8e040a3050 tx=0x7f8e00032000 comp rx=0 tx=0).stop 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 10 monclient: shutdown 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 msgr2=0x7f8e0406ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 0x7f8e0406ad30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00ccf0 tx=0x7f8dfc005370 comp rx=0 tx=0).stop 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 shutdown_connections 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8de8038490 0x7f8de803a940 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8e04134c60 0x7f8e0406ad30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e0409f180 0x7f8e0406a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 --2- 172.21.15.144:0/1517127277 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8e0409cee0 0x7f8e041342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 >> 172.21.15.144:0/1517127277 conn(0x7f8e040a2b70 msgr2=0x7f8e0409ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 shutdown_connections 2024-09-18T12:29:06.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.609+0000 7f8e0c2f8640 1 -- 172.21.15.144:0/1517127277 wait complete. 2024-09-18T12:29:06.619 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush unlink host1 foo 2024-09-18T12:29:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.692+0000 7f6f11b5d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.692+0000 7f6f11b5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.692+0000 7f6f11b5d640 10 monclient: monmap: 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:06.693677+0000 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:06.693677+0000 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:06.695 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 1 Processor -- start 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 1 -- start start 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 10 monclient: init 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 0x7f6f0c099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0c098c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c099580 0x7f6f0c09dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.693+0000 7f6f11b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f0c099580 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f0c099c70 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0c09f680 con 0x7f6f0c099c70 2024-09-18T12:29:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0c09f420 con 0x7f6f0c099580 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c068a20 con 0x7f6f0c098c70 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f11b5d640 10 monclient(hunting): authenticate will time out at 982.263041s 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c099580 0x7f6f0c09dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 0x7f6f0c099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c099580 0x7f6f0c09dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:52512/0 (socket says 172.21.15.144:52512) 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 -- 172.21.15.144:0/2086525532 learned_addr learned my addr 172.21.15.144:0/2086525532 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): get_auth_request con 0x7f6f0c099580 auth_method 0 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 10 monclient(hunting): get_auth_request con 0x7f6f0c098c70 auth_method 0 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 1 -- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 msgr2=0x7f6f0c099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 1 -- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 msgr2=0x7f6f0c099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 1 --2- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 0x7f6f0c099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0b7fe640 1 --2- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 0x7f6f0c099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient(hunting): handle_auth_done global_id 54372 payload 274 2024-09-18T12:29:06.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 -- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 msgr2=0x7f6f0c099040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:06.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 --2- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 0x7f6f0c099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 -- 172.21.15.144:0/2086525532 >> v1:172.21.15.116:6789/0 conn(0x7f6f0c099c70 legacy=0x7f6f0c09e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:06.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:06.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 monclient: found mon.noname-a 2024-09-18T12:29:06.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:06.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:06.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.694+0000 7f6f0affd640 1 -- 172.21.15.144:0/2086525532 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c096a30 con 0x7f6f0c099580 2024-09-18T12:29:06.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.697+0000 7f6f0affd640 1 -- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099580 msgr2=0x7f6f0c09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:06.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.697+0000 7f6f0affd640 1 --2- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099580 0x7f6f0c09dad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009910 tx=0x7f6f000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52c8943291fbd65 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:06.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.697+0000 7f6f0a7fc640 1 -- 172.21.15.144:0/2086525532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0002ae90 con 0x7f6f0c099580 2024-09-18T12:29:06.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: dump: 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:06.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 5 monclient: authenticate success, global_id 54372 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 1 -- 172.21.15.144:0/2086525532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00032440 con 0x7f6f0c099580 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 4 set_mon_vals no callback set 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 1 -- 172.21.15.144:0/2086525532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f000327e0 con 0x7f6f0c099580 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f0a7fc640 10 monclient: dump: 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:06.709 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 10 monclient: shutdown 2024-09-18T12:29:06.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 -- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099580 msgr2=0x7f6f0c09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099580 0x7f6f0c09dad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009910 tx=0x7f6f000091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 -- 172.21.15.144:0/2086525532 shutdown_connections 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099580 0x7f6f0c09dad0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/2086525532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c098c70 0x7f6f0c099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 -- 172.21.15.144:0/2086525532 >> 172.21.15.144:0/2086525532 conn(0x7f6f0c070e90 msgr2=0x7f6f0c0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 -- 172.21.15.144:0/2086525532 shutdown_connections 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.698+0000 7f6f11b5d640 1 -- 172.21.15.144:0/2086525532 wait complete. 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient: monmap: 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:06.711 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 Processor -- start 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 -- start start 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient: init 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:29:06.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 0x7f6f0c133c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0c098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 0x7f6f0c134160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0c099580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099c70 0x7f6f0c11db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f0c099c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c121300 con 0x7f6f0c099c70 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c06eba0 con 0x7f6f0c098c70 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c121020 con 0x7f6f0c099580 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:06.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.699+0000 7f6f11b5d640 10 monclient(hunting): authenticate will time out at 982.268989s 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 0x7f6f0c133c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 0x7f6f0c133c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41022/0 (socket says 172.21.15.144:41022) 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 -- 172.21.15.144:0/684982636 learned_addr learned my addr 172.21.15.144:0/684982636 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): get_auth_request con 0x7f6f0c098c70 auth_method 0 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0bfff640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099c70 0x7f6f0c11db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 0x7f6f0c134160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 10 monclient(hunting): get_auth_request con 0x7f6f0c099580 auth_method 0 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0bfff640 10 monclient(hunting): get_auth_request con 0x7f6f0c099c70 auth_method 0 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 1 -- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 msgr2=0x7f6f0c134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 1 -- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 msgr2=0x7f6f0c134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 0x7f6f0c134160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient(hunting): handle_auth_done global_id 5128 payload 824 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0affd640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 0x7f6f0c134160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:06.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 -- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 msgr2=0x7f6f0c134160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 0x7f6f0c134160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 -- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099c70 msgr2=0x7f6f0c11db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099c70 0x7f6f0c11db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0b7fe640 1 -- 172.21.15.144:0/684982636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00009450 con 0x7f6f0c098c70 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.700+0000 7f6f0bfff640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099c70 0x7f6f0c11db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f0b7fe640 1 -- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 msgr2=0x7f6f0c133c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f0b7fe640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 0x7f6f0c133c20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00d890 tx=0x7f6efc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:06.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00de20 con 0x7f6f0c098c70 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: dump: 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:06.717 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc00bae0 con 0x7f6f0c098c70 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f11b5d640 5 monclient: authenticate success, global_id 5128 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f11b5d640 10 monclient: _renew_subs 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f11b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c11f210 con 0x7f6f0c098c70 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00bea0 con 0x7f6f0c098c70 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6ef37fe640 10 monclient: dump: 2024-09-18T12:29:06.718 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f09ffb640 4 set_mon_vals no callback set 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f11b5d640 10 monclient: _renew_subs 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f11b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:06.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.702+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c136600 con 0x7f6f0c098c70 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.703+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6efc014640 con 0x7f6f0c098c70 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.703+0000 7f6f11b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.703+0000 7f6f11b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.703+0000 7f6f11b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.703+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c0008d0 con 0x7f6f0c098c70 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.704+0000 7f6ef37fe640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ef8038490 0x7f6ef803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.704+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f6efc04c420 con 0x7f6f0c098c70 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.704+0000 7f6f0affd640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ef8038490 0x7f6ef803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.704+0000 7f6f0affd640 10 monclient: get_auth_request con 0x7f6ef8038490 auth_method 0 2024-09-18T12:29:06.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.706+0000 7f6f0affd640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ef8038490 0x7f6ef803a940 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c06abc0 tx=0x7f6f00032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:06.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.708+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6efc011570 con 0x7f6f0c098c70 2024-09-18T12:29:06.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.708+0000 7f6ef37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:06.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.708+0000 7f6ef37fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:06.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.869+0000 7f6f11b5d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.869+0000 7f6f11b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.869+0000 7f6f11b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:06.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.869+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f6f0c121440 con 0x7f6f0c098c70 2024-09-18T12:29:06.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.896+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v81) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6efc01dbb0 con 0x7f6f0c098c70 2024-09-18T12:29:06.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.897+0000 7f6ef37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.897+0000 7f6ef37fe640 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-18T12:29:06.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.993+0000 7f6f11b5d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.993+0000 7f6f11b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.993+0000 7f6f11b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:06.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.993+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f6f0c11fde0 con 0x7f6f0c098c70 2024-09-18T12:29:06.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.995+0000 7f6ef37fe640 1 -- 172.21.15.144:0/684982636 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v81) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6efc01dd90 con 0x7f6f0c098c70 2024-09-18T12:29:06.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.995+0000 7f6ef37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T12:29:06.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.995+0000 7f6ef37fe640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-18T12:29:06.998 INFO:tasks.workunit.client.0.smithi144.stderr:item -11 does not appear in that position 2024-09-18T12:29:06.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ef8038490 msgr2=0x7f6ef803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ef8038490 0x7f6ef803a940 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c06abc0 tx=0x7f6f00032000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 10 monclient: shutdown 2024-09-18T12:29:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 msgr2=0x7f6f0c133c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 0x7f6f0c133c20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00d890 tx=0x7f6efc008040 comp rx=0 tx=0).stop 2024-09-18T12:29:07.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 shutdown_connections 2024-09-18T12:29:07.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ef8038490 0x7f6ef803a940 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c099c70 0x7f6f0c11db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6f0c099580 0x7f6f0c134160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 --2- 172.21.15.144:0/684982636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f0c098c70 0x7f6f0c133c20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 >> 172.21.15.144:0/684982636 conn(0x7f6f0c070e90 msgr2=0x7f6f0c134ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 shutdown_connections 2024-09-18T12:29:07.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:06.997+0000 7f6f11b5d640 1 -- 172.21.15.144:0/684982636 wait complete. 2024-09-18T12:29:07.008 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:07.008 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c host1 2024-09-18T12:29:07.008 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 1 2024-09-18T12:29:07.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.077+0000 7f1bdb0c9640 10 monclient: get_monmap_and_config 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.077+0000 7f1bdb0c9640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.077+0000 7f1bdb0c9640 10 monclient: monmap: 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:07.078595+0000 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:07.078595+0000 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:07.083 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 1 Processor -- start 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 1 -- start start 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient: init 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1bd40972c0 0x7f1bd409b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd40972c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1bd409bcb0 0x7f1bd409c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd409bcb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:07.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bd4099790 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd409c5f0 con 0x7f1bd4099790 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd409d6a0 con 0x7f1bd40972c0 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd409c880 con 0x7f1bd409bcb0 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bdb0c9640 10 monclient(hunting): authenticate will time out at 982.647890s 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.078+0000 7f1bd3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1bd409bcb0 0x7f1bd409c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1bd409bcb0 0x7f1bd409c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41030/0 (socket says 172.21.15.144:41030) 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 1 -- 172.21.15.144:0/1797788902 learned_addr learned my addr 172.21.15.144:0/1797788902 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 10 monclient(hunting): get_auth_request con 0x7f1bd409bcb0 auth_method 0 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/1797788902 >> v2:172.21.15.116:3300/0 conn(0x7f1bd40972c0 0x7f1bd409b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): get_auth_request con 0x7f1bd40972c0 auth_method 0 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_done global_id 54378 payload 274 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 1 -- 172.21.15.144:0/1797788902 >> v2:172.21.15.144:3300/0 conn(0x7f1bd409bcb0 msgr2=0x7f1bd409c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/1797788902 >> v2:172.21.15.144:3300/0 conn(0x7f1bd409bcb0 0x7f1bd409c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 1 -- 172.21.15.144:0/1797788902 >> v1:172.21.15.144:6789/0 conn(0x7f1bd4099790 legacy=0x7f1bd4099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 1 monclient: found mon.noname-a 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:07.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.079+0000 7f1bd8e65640 1 -- 172.21.15.144:0/1797788902 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4097690 con 0x7f1bd40972c0 2024-09-18T12:29:07.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd8e65640 1 -- 172.21.15.144:0/1797788902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd40972c0 msgr2=0x7f1bd409b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/1797788902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd40972c0 0x7f1bd409b770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40a1bd0 tx=0x7f1bc80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c36c93330cdef2b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 1 -- 172.21.15.144:0/1797788902 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc802adf0 con 0x7f1bd40972c0 2024-09-18T12:29:07.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:07.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: dump: 2024-09-18T12:29:07.096 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 1 -- 172.21.15.144:0/1797788902 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc8032440 con 0x7f1bd40972c0 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bdb0c9640 5 monclient: authenticate success, global_id 54378 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 1 -- 172.21.15.144:0/1797788902 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8032820 con 0x7f1bd40972c0 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bdb0c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bdb0c9640 4 set_mon_vals no callback set 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:07.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.082+0000 7f1bd37fe640 10 monclient: dump: 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 10 monclient: shutdown 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/1797788902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd40972c0 msgr2=0x7f1bd409b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/1797788902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd40972c0 0x7f1bd409b770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40a1bd0 tx=0x7f1bc80090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/1797788902 shutdown_connections 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/1797788902 >> v2:172.21.15.144:3300/0 conn(0x7f1bd409bcb0 0x7f1bd409c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/1797788902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd40972c0 0x7f1bd409b770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/1797788902 >> 172.21.15.144:0/1797788902 conn(0x7f1bd40714d0 msgr2=0x7f1bd40a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/1797788902 shutdown_connections 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/1797788902 wait complete. 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.083+0000 7f1bdb0c9640 10 monclient: monmap: 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:07.100 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 Processor -- start 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 -- start start 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient: init 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 0x7f1bd413dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd40972c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 0x7f1bd412fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd4099790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 0x7f1bd4132d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd409bcb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd4118070 con 0x7f1bd409bcb0 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd409abb0 con 0x7f1bd40972c0 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd413eaf0 con 0x7f1bd4099790 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bdb0c9640 10 monclient(hunting): authenticate will time out at 982.653934s 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.084+0000 7f1bd8e65640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 0x7f1bd413dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 0x7f1bd413dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41036/0 (socket says 172.21.15.144:41036) 2024-09-18T12:29:07.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 -- 172.21.15.144:0/574557209 learned_addr learned my addr 172.21.15.144:0/574557209 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): get_auth_request con 0x7f1bd40972c0 auth_method 0 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 0x7f1bd412fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 10 monclient(hunting): get_auth_request con 0x7f1bd4099790 auth_method 0 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd9666640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 0x7f1bd4132d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd9666640 10 monclient(hunting): get_auth_request con 0x7f1bd409bcb0 auth_method 0 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd9666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd9666640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd9666640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient(hunting): handle_auth_done global_id 5134 payload 824 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 msgr2=0x7f1bd412fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 0x7f1bd412fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 msgr2=0x7f1bd4132d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 0x7f1bd4132d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 monclient: found mon.noname-b 2024-09-18T12:29:07.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd8e65640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc8009390 con 0x7f1bd40972c0 2024-09-18T12:29:07.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.085+0000 7f1bd9666640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 0x7f1bd4132d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:07.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd8e65640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 msgr2=0x7f1bd413dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 0x7f1bd413dc80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009880 tx=0x7f1bc8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd8e65640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 msgr2=0x7f1bd413dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd8e65640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 msgr2=0x7f1bd413dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 0x7f1bd413dc80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009880 tx=0x7f1bc8032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 0x7f1bd413dc80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009880 tx=0x7f1bc8032000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 msgr2=0x7f1bd413dc80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd40972c0 0x7f1bd413dc80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 0x7f1bb8005490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd4099790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bb8005e10 0x7f1bb8008290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb8005e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:07.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 0x7f1bb800ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd409bcb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb800b7e0 con 0x7f1bd409bcb0 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb800b920 con 0x7f1bb8005e10 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb800bae0 con 0x7f1bd4099790 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.087+0000 7f1bd0ff9640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bb8005e10 0x7f1bb8008290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): get_auth_request con 0x7f1bb8005e10 auth_method 0 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient(hunting): handle_auth_done global_id 5134 payload 293 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 msgr2=0x7f1bb8005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 0x7f1bb8005490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 msgr2=0x7f1bb800ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 0x7f1bb800ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 1 monclient: found mon.noname-b 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.088+0000 7f1bd3fff640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb800df10 con 0x7f1bb8005e10 2024-09-18T12:29:07.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd3fff640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bb8005e10 msgr2=0x7f1bb8008290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd3fff640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bb8005e10 0x7f1bb8008290 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4134540 tx=0x7f1bc00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0011e90 con 0x7f1bb8005e10 2024-09-18T12:29:07.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:07.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: dump: 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.110 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bdb0c9640 5 monclient: authenticate success, global_id 5134 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc0009820 con 0x7f1bb8005e10 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bdb0c9640 10 monclient: _renew_subs 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bdb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd4130830 con 0x7f1bb8005e10 2024-09-18T12:29:07.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0014670 con 0x7f1bb8005e10 2024-09-18T12:29:07.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:07.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd0ff9640 10 monclient: dump: 2024-09-18T12:29:07.112 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.112 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.113 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.113 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.113 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.113 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.113 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.114 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.114 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.114 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bd2ffd640 4 set_mon_vals no callback set 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bdb0c9640 10 monclient: _renew_subs 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bdb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.090+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd41353e0 con 0x7f1bb8005e10 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1bc0014a60 con 0x7f1bb8005e10 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bdb0c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bdb0c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bdb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd41184e0 con 0x7f1bb8005e10 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bd0ff9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1bd40972c0 0x7f1bb8044e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f1bc0018030 con 0x7f1bb8005e10 2024-09-18T12:29:07.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1bd40972c0 0x7f1bb8044e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.091+0000 7f1bd8e65640 10 monclient: get_auth_request con 0x7f1bd40972c0 auth_method 0 2024-09-18T12:29:07.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.094+0000 7f1bd8e65640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1bd40972c0 0x7f1bb8044e20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8032a10 tx=0x7f1bc803b000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.096+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1bc0019290 con 0x7f1bb8005e10 2024-09-18T12:29:07.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.096+0000 7f1bd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.117 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.096+0000 7f1bd0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:07.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient: get_auth_request con 0x7f125c1310f0 auth_method 0 2024-09-18T12:29:07.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:07.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-09-18T12:29:07.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.241+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f125c11e0f0 con 0x7f125c1310f0 2024-09-18T12:29:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.243+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.243+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1250034000 tx=0x7f1250037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125004b070 con 0x7f125c1310f0 2024-09-18T12:29:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 10 monclient: dump: 2024-09-18T12:29:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1250042040 con 0x7f125c1310f0 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:07.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1250034000 tx=0x7f1250037000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 msgr2=0x7f125c11bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:07.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.244+0000 7f1242ffd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c1310f0 0x7f125c11bc60 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1250034000 tx=0x7f1250037000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.250+0000 7f126135d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:07.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.250+0000 7f126135d640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 0x7f125c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.250+0000 7f126135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.250+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c1314c0 con 0x7f125c11c1a0 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 0x7f125c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient: get_auth_request con 0x7f125c11c1a0 auth_method 0 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 10 monclient(hunting): handle_auth_done global_id 64106 payload 824 2024-09-18T12:29:07.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.251+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f125c11c5a0 con 0x7f125c11c1a0 2024-09-18T12:29:07.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.251+0000 7f1bdb0c9640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:07.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.251+0000 7f1bdb0c9640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:07.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.251+0000 7f1bdb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.251+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1bd40636e0 con 0x7f1bb8005e10 2024-09-18T12:29:07.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.252+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f1bc001aa90 con 0x7f1bb8005e10 2024-09-18T12:29:07.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.252+0000 7f1bd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:07.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.252+0000 7f1bd0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.253+0000 7f125a7fc640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 msgr2=0x7f125c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.253+0000 7f125a7fc640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 0x7f125c096a60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f125c068680 tx=0x7f125003e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.253+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250041050 con 0x7f125c11c1a0 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.253+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.253+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.253+0000 7f1242ffd640 10 monclient: dump: 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.256 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f1250031dd0 con 0x7f125c11c1a0 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f1242ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f1242ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:07.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 msgr2=0x7f125c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f1242ffd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 0x7f125c096a60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f125c068680 tx=0x7f125003e070 comp rx=0 tx=0).stop 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f1242ffd640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 msgr2=0x7f125c096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f1242ffd640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f125c11c1a0 0x7f125c096a60 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f125c068680 tx=0x7f125003e070 comp rx=0 tx=0).stop 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 44, 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:29:07.258 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:07.259 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:07.260 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:07.261 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:07.262 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.263 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:07.264 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:07.265 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:07.266 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:07.268 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:07.269 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.270 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:07.271 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1248038490 msgr2=0x7f124803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1248038490 0x7f124803a940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f125c09b950 tx=0x7f124c009040 comp rx=0 tx=0).stop 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 10 monclient: shutdown 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c068d90 msgr2=0x7f125c09aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 --2- 172.21.15.116:0/2882155585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f125c068d90 0x7f125c09aa00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f125400e870 tx=0x7f1254008040 comp rx=0 tx=0).stop 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f125b7fe640 1 -- 172.21.15.116:0/2882155585 reap_dead start 2024-09-18T12:29:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f125b7fe640 1 -- 172.21.15.116:0/2882155585 reap_dead start 2024-09-18T12:29:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 shutdown_connections 2024-09-18T12:29:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 >> 172.21.15.116:0/2882155585 conn(0x7f125c067110 msgr2=0x7f125c09d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 shutdown_connections 2024-09-18T12:29:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.254+0000 7f126135d640 1 -- 172.21.15.116:0/2882155585 wait complete. 2024-09-18T12:29:07.274 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:29:07.274 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:29:07.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:29:07.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.347+0000 7f1bdb0c9640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:07.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.347+0000 7f1bdb0c9640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:07.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.347+0000 7f1bdb0c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.347+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1bd41182b0 con 0x7f1bb8005e10 2024-09-18T12:29:07.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.347+0000 7f1bd0ff9640 1 -- 172.21.15.144:0/574557209 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v81) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f1bc001aa90 con 0x7f1bb8005e10 2024-09-18T12:29:07.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.347+0000 7f1bd0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:07.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.347+0000 7f1bd0ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:07.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1bd40972c0 msgr2=0x7f1bb8044e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1bd40972c0 0x7f1bb8044e20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8032a10 tx=0x7f1bc803b000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 10 monclient: shutdown 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bb8005e10 msgr2=0x7f1bb8008290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bb8005e10 0x7f1bb8008290 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4134540 tx=0x7f1bc00091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 shutdown_connections 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1bd40972c0 0x7f1bb8044e20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd409bcb0 0x7f1bb800ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bb8005e10 0x7f1bb8008290 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 --2- 172.21.15.144:0/574557209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1bd4099790 0x7f1bb8005490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 >> 172.21.15.144:0/574557209 conn(0x7f1bd40714d0 msgr2=0x7f1bd4135ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 shutdown_connections 2024-09-18T12:29:07.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.349+0000 7f1bdb0c9640 1 -- 172.21.15.144:0/574557209 wait complete. 2024-09-18T12:29:07.359 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush rm foo 2024-09-18T12:29:07.360 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:29:07.360 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm foo 2024-09-18T12:29:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.376+0000 7f9ac100f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.376+0000 7f9ac100f640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.376+0000 7f9ac100f640 10 monclient: monmap: 2024-09-18T12:29:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:07.377304+0000 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:07.377304+0000 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:07.378 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 1 Processor -- start 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 1 -- start start 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient: init 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9abc096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9abc0971e0 0x7f9abc0975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient(hunting): picked mon.noname-c con 0x7f9abc0971e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9abc133b30 0x7f9abc135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient(hunting): picked mon.noname-a con 0x7f9abc133b30 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc099620 con 0x7f9abc096e10 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc099340 con 0x7f9abc133b30 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc099900 con 0x7f9abc0971e0 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9ac100f640 10 monclient(hunting): authenticate will time out at 974.698829s 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9abad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9abc096e10 0x7f9abc09a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48552/0 (socket says 172.21.15.116:48552) 2024-09-18T12:29:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9abb59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9abc133b30 0x7f9abc135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9abad9d640 1 -- 172.21.15.116:0/1351952031 learned_addr learned my addr 172.21.15.116:0/1351952031 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9abb59e640 10 monclient(hunting): get_auth_request con 0x7f9abc133b30 auth_method 0 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9abb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9abb59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.377+0000 7f9abb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 1 --2- 172.21.15.116:0/1351952031 >> v2:172.21.15.144:3300/0 conn(0x7f9abc0971e0 0x7f9abc0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 10 monclient(hunting): get_auth_request con 0x7f9abc0971e0 auth_method 0 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 10 monclient(hunting): handle_auth_done global_id 54384 payload 274 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 1 -- 172.21.15.116:0/1351952031 >> v2:172.21.15.144:3300/0 conn(0x7f9abc0971e0 msgr2=0x7f9abc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 1 --2- 172.21.15.116:0/1351952031 >> v2:172.21.15.144:3300/0 conn(0x7f9abc0971e0 0x7f9abc0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 1 -- 172.21.15.116:0/1351952031 >> v1:172.21.15.116:6789/0 conn(0x7f9abc096e10 legacy=0x7f9abc09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 1 monclient: found mon.noname-a 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.378+0000 7f9abb59e640 1 -- 172.21.15.116:0/1351952031 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc09ac70 con 0x7f9abc133b30 2024-09-18T12:29:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.380+0000 7f9abb59e640 1 -- 172.21.15.116:0/1351952031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc133b30 msgr2=0x7f9abc135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.380+0000 7f9abb59e640 1 --2- 172.21.15.116:0/1351952031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc133b30 0x7f9abc135f10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9abc09b020 tx=0x7f9ab4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43933683ddad04c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 1 -- 172.21.15.116:0/1351952031 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4007760 con 0x7f9abc133b30 2024-09-18T12:29:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: dump: 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 5 monclient: authenticate success, global_id 54384 2024-09-18T12:29:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 1 -- 172.21.15.116:0/1351952031 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab402d050 con 0x7f9abc133b30 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 1 -- 172.21.15.116:0/1351952031 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4007af0 con 0x7f9abc133b30 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 4 set_mon_vals no callback set 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ab9d9b640 10 monclient: dump: 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 10 monclient: shutdown 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 -- 172.21.15.116:0/1351952031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc133b30 msgr2=0x7f9abc135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 --2- 172.21.15.116:0/1351952031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc133b30 0x7f9abc135f10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9abc09b020 tx=0x7f9ab4009780 comp rx=0 tx=0).stop 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 -- 172.21.15.116:0/1351952031 shutdown_connections 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 --2- 172.21.15.116:0/1351952031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc133b30 0x7f9abc135f10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 --2- 172.21.15.116:0/1351952031 >> v2:172.21.15.144:3300/0 conn(0x7f9abc0971e0 0x7f9abc0975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 -- 172.21.15.116:0/1351952031 >> 172.21.15.116:0/1351952031 conn(0x7f9abc0672b0 msgr2=0x7f9abc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 -- 172.21.15.116:0/1351952031 shutdown_connections 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.381+0000 7f9ac100f640 1 -- 172.21.15.116:0/1351952031 wait complete. 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.382+0000 7f9ac100f640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.382+0000 7f9ac100f640 10 monclient: monmap: 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.382+0000 7f9ac100f640 1 Processor -- start 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.382+0000 7f9ac100f640 1 -- start start 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.382+0000 7f9ac100f640 10 monclient: init 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 0x7f9abc141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): picked mon.noname-a con 0x7f9abc096e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9abc0971e0 0x7f9abc141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): picked mon.noname-c con 0x7f9abc0971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc133b30 0x7f9abc12f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): picked mon.noname-b con 0x7f9abc133b30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc09b050 con 0x7f9abc096e10 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc1333b0 con 0x7f9abc133b30 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc11c5d0 con 0x7f9abc0971e0 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9ac100f640 10 monclient(hunting): authenticate will time out at 974.704311s 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 0x7f9abc141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9abc0971e0 0x7f9abc141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 0x7f9abc141200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51988/0 (socket says 172.21.15.116:51988) 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 -- 172.21.15.116:0/998094376 learned_addr learned my addr 172.21.15.116:0/998094376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): get_auth_request con 0x7f9abc096e10 auth_method 0 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 10 monclient(hunting): get_auth_request con 0x7f9abc0971e0 auth_method 0 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc133b30 0x7f9abc12f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 10 monclient(hunting): get_auth_request con 0x7f9abc133b30 auth_method 0 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient(hunting): handle_auth_done global_id 54387 payload 824 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9abc0971e0 msgr2=0x7f9abc141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9abc0971e0 0x7f9abc141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc133b30 msgr2=0x7f9abc12f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc133b30 0x7f9abc12f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 monclient: found mon.noname-a 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9abad9d640 1 -- 172.21.15.116:0/998094376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4009a20 con 0x7f9abc096e10 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.383+0000 7f9aba59c640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9abc0971e0 0x7f9abc141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.385+0000 7f9abad9d640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 msgr2=0x7f9abc141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.385+0000 7f9abad9d640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 0x7f9abc141200 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c5e0 tx=0x7f9aac004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac005cb0 con 0x7f9abc096e10 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: dump: 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac005e30 con 0x7f9abc096e10 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 5 monclient: authenticate success, global_id 54387 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 10 monclient: _renew_subs 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc11da80 con 0x7f9abc096e10 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac007340 con 0x7f9abc096e10 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9aa2ffd640 10 monclient: dump: 2024-09-18T12:29:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 10 monclient: _renew_subs 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc11e650 con 0x7f9abc096e10 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ab959a640 4 set_mon_vals no callback set 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 0x7f9abc0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.386+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc065f20 con 0x7f9abc1303f0 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9aac01b400 con 0x7f9abc096e10 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 0x7f9abc0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient: get_auth_request con 0x7f9abc1303f0 auth_method 0 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aa2ffd640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9aa8038490 0x7f9aa803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f9aac04b940 con 0x7f9abc096e10 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 10 monclient(hunting): handle_auth_done global_id 5143 payload 824 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9aba59c640 1 -- 172.21.15.116:0/998094376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9abc04bf60 con 0x7f9abc1303f0 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9abb59e640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9aa8038490 0x7f9aa803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.387+0000 7f9abb59e640 10 monclient: get_auth_request con 0x7f9aa8038490 auth_method 0 2024-09-18T12:29:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.389+0000 7f9aba59c640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 msgr2=0x7f9abc0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.389+0000 7f9aba59c640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 0x7f9abc0298c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009f30 tx=0x7f9ab00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.389+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0014860 con 0x7f9abc1303f0 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.389+0000 7f9aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.389+0000 7f9aa2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.389+0000 7f9aa2ffd640 10 monclient: dump: 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.389+0000 7f9abb59e640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9aa8038490 0x7f9aa803a940 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9abc076be0 tx=0x7f9ab4037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.390+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9ab00149c0 con 0x7f9abc1303f0 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.390+0000 7f9aa2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.390+0000 7f9aa2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.390+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 msgr2=0x7f9abc0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.390+0000 7f9aa2ffd640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 0x7f9abc0298c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009f30 tx=0x7f9ab00090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.390+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 msgr2=0x7f9abc0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.390+0000 7f9aa2ffd640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1303f0 0x7f9abc0298c0 secure :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009f30 tx=0x7f9ab00090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.395+0000 7f9ac100f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.395+0000 7f9ac100f640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 0x7f9abc029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.395+0000 7f9ac100f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.395+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc11c150 con 0x7f9abc1307c0 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 0x7f9abc029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient: get_auth_request con 0x7f9abc1307c0 auth_method 0 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 10 monclient(hunting): handle_auth_done global_id 5146 payload 824 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.396+0000 7f9aba59c640 1 -- 172.21.15.116:0/998094376 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9abc06cd80 con 0x7f9abc1307c0 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.398+0000 7f9aba59c640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 msgr2=0x7f9abc029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.398+0000 7f9aba59c640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 0x7f9abc029e00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009f30 tx=0x7f9ab001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.398+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0024040 con 0x7f9abc1307c0 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.398+0000 7f9aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.398+0000 7f9aa2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.398+0000 7f9aa2ffd640 10 monclient: dump: 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f9ab00133a0 con 0x7f9abc1307c0 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9aa2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9aa2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 msgr2=0x7f9abc029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9aa2ffd640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 0x7f9abc029e00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009f30 tx=0x7f9ab001c000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9aa2ffd640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 msgr2=0x7f9abc029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:07.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9aa2ffd640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9abc1307c0 0x7f9abc029e00 secure :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009f30 tx=0x7f9ab001c000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.416 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:07.416 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:07.416 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:07.416 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:29:07.416 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 44, 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:07.417 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:07.418 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:07.419 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:07.419 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:07.419 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:07.419 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:07.419 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:07.420 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:07.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:07.420 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:07.420 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:07.420 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:07.421 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:07.422 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:07.422 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:07.422 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:07.422 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:07.422 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:07.422 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:07.423 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:07.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:07.426 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.427 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.429 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:07.429 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:07.429 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:07.429 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:07.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.429 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:07.430 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:07.430 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.430 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:07.430 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:07.431 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:07.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.431 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.431 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:07.431 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:07.432 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:07.432 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:07.432 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:07.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.432 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.433 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:07.433 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:07.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.433 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:07.433 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:07.433 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:07.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.434 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.434 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:29:07.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.434 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:07.434 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:07.435 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:29:07.435 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.435 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.435 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:07.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.435 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:07.436 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:07.436 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:29:07.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.436 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:07.436 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:29:07.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:07.437 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:07.437 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:07.437 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:07.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:07.437 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:07.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:07.438 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:07.438 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:07.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9aa8038490 msgr2=0x7f9aa803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9aa8038490 0x7f9aa803a940 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9abc076be0 tx=0x7f9ab4037000 comp rx=0 tx=0).stop 2024-09-18T12:29:07.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 10 monclient: shutdown 2024-09-18T12:29:07.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 msgr2=0x7f9abc141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 0x7f9abc141200 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c5e0 tx=0x7f9aac004960 comp rx=0 tx=0).stop 2024-09-18T12:29:07.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9abb59e640 1 -- 172.21.15.116:0/998094376 reap_dead start 2024-09-18T12:29:07.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 shutdown_connections 2024-09-18T12:29:07.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 --2- 172.21.15.116:0/998094376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9abc096e10 0x7f9abc141200 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 >> 172.21.15.116:0/998094376 conn(0x7f9abc0672b0 msgr2=0x7f9abc132c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 shutdown_connections 2024-09-18T12:29:07.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.399+0000 7f9ac100f640 1 -- 172.21.15.116:0/998094376 wait complete. 2024-09-18T12:29:07.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:29:07.443 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:29:07.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.432+0000 7f055895d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.432+0000 7f055895d640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.432+0000 7f055895d640 10 monclient: monmap: 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:07.433393+0000 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:07.433393+0000 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.454 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 1 Processor -- start 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 1 -- start start 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient: init 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0554131c90 0x7f0554132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient(hunting): picked mon.noname-c con 0x7f0554131c90 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:07.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f055409d210 0x7f05541325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient(hunting): picked mon.noname-a con 0x7f055409d210 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05540a0ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05540a1800 con 0x7f05540a0ab0 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055409fea0 con 0x7f055409d210 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05540a1520 con 0x7f0554131c90 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055895d640 10 monclient(hunting): authenticate will time out at 983.002829s 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0554131c90 0x7f0554132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0554131c90 0x7f0554132060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42848/0 (socket says 172.21.15.144:42848) 2024-09-18T12:29:07.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055259c640 1 -- 172.21.15.144:0/3288050502 learned_addr learned my addr 172.21.15.144:0/3288050502 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:07.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055259c640 10 monclient(hunting): get_auth_request con 0x7f0554131c90 auth_method 0 2024-09-18T12:29:07.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.433+0000 7f055259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f055259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f055259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f055259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 1 --2- 172.21.15.144:0/3288050502 >> v2:172.21.15.116:3300/0 conn(0x7f055409d210 0x7f05541325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): get_auth_request con 0x7f055409d210 auth_method 0 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient(hunting): handle_auth_done global_id 54390 payload 274 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 1 -- 172.21.15.144:0/3288050502 >> v2:172.21.15.144:3300/0 conn(0x7f0554131c90 msgr2=0x7f0554132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 1 --2- 172.21.15.144:0/3288050502 >> v2:172.21.15.144:3300/0 conn(0x7f0554131c90 0x7f0554132060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 1 -- 172.21.15.144:0/3288050502 >> v1:172.21.15.144:6789/0 conn(0x7f05540a0ab0 legacy=0x7f0554068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 1 monclient: found mon.noname-a 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.434+0000 7f0551d9b640 1 -- 172.21.15.144:0/3288050502 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0554069510 con 0x7f055409d210 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f0551d9b640 1 -- 172.21.15.144:0/3288050502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409d210 msgr2=0x7f05541325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f0551d9b640 1 --2- 172.21.15.144:0/3288050502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409d210 0x7f05541325a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0554076940 tx=0x7f053c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8944a56cce80af35 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 1 -- 172.21.15.144:0/3288050502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c02adc0 con 0x7f055409d210 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: dump: 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.460 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 1 -- 172.21.15.144:0/3288050502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c032440 con 0x7f055409d210 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 1 -- 172.21.15.144:0/3288050502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c032820 con 0x7f055409d210 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055895d640 5 monclient: authenticate success, global_id 54390 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055895d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055895d640 4 set_mon_vals no callback set 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.437+0000 7f055159a640 10 monclient: dump: 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 10 monclient: shutdown 2024-09-18T12:29:07.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 -- 172.21.15.144:0/3288050502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409d210 msgr2=0x7f05541325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 --2- 172.21.15.144:0/3288050502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409d210 0x7f05541325a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0554076940 tx=0x7f053c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 -- 172.21.15.144:0/3288050502 shutdown_connections 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 --2- 172.21.15.144:0/3288050502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409d210 0x7f05541325a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 --2- 172.21.15.144:0/3288050502 >> v2:172.21.15.144:3300/0 conn(0x7f0554131c90 0x7f0554132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 -- 172.21.15.144:0/3288050502 >> 172.21.15.144:0/3288050502 conn(0x7f0554071370 msgr2=0x7f05540a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 -- 172.21.15.144:0/3288050502 shutdown_connections 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 1 -- 172.21.15.144:0/3288050502 wait complete. 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.438+0000 7f055895d640 10 monclient: monmap: 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:07.463 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 Processor -- start 2024-09-18T12:29:07.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 -- start start 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient: init 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f05540a0ab0 0x7f0554141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): picked mon.noname-c con 0x7f05540a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554131c90 0x7f05541418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): picked mon.noname-b con 0x7f0554131c90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055411e720 0x7f055411ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): picked mon.noname-a con 0x7f055411e720 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554122210 con 0x7f055411e720 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05540a0fb0 con 0x7f0554131c90 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055409f160 con 0x7f05540a0ab0 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055895d640 10 monclient(hunting): authenticate will time out at 983.008700s 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554131c90 0x7f05541418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554131c90 0x7f05541418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42860/0 (socket says 172.21.15.144:42860) 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 1 -- 172.21.15.144:0/813415783 learned_addr learned my addr 172.21.15.144:0/813415783 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 10 monclient(hunting): get_auth_request con 0x7f0554131c90 auth_method 0 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f05540a0ab0 0x7f0554141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0551d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 10 monclient(hunting): get_auth_request con 0x7f05540a0ab0 auth_method 0 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055411e720 0x7f055411ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 10 monclient(hunting): get_auth_request con 0x7f055411e720 auth_method 0 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f055259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.439+0000 7f0552d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 10 monclient(hunting): handle_auth_done global_id 5152 payload 824 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 1 -- 172.21.15.144:0/813415783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f05540a0ab0 msgr2=0x7f0554141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f05540a0ab0 0x7f0554141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 1 -- 172.21.15.144:0/813415783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055411e720 msgr2=0x7f055411ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055411e720 0x7f055411ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 1 monclient: found mon.noname-b 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.440+0000 7f0551d9b640 1 -- 172.21.15.144:0/813415783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c009390 con 0x7f0554131c90 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f0551d9b640 1 -- 172.21.15.144:0/813415783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554131c90 msgr2=0x7f05541418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f0551d9b640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554131c90 0x7f05541418c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f053c009850 tx=0x7f053c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 1 -- 172.21.15.144:0/813415783 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c031c70 con 0x7f0554131c90 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: dump: 2024-09-18T12:29:07.469 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f055895d640 5 monclient: authenticate success, global_id 5152 2024-09-18T12:29:07.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f055895d640 10 monclient: _renew_subs 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f055895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f055895d640 1 -- 172.21.15.144:0/813415783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05541209e0 con 0x7f0554131c90 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f055895d640 10 monclient: _renew_subs 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f055895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f055895d640 1 -- 172.21.15.144:0/813415783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0554120090 con 0x7f0554131c90 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 1 -- 172.21.15.144:0/813415783 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c031dd0 con 0x7f0554131c90 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 1 -- 172.21.15.144:0/813415783 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c03d690 con 0x7f0554131c90 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f0550d99640 4 set_mon_vals no callback set 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.442+0000 7f054a7fc640 10 monclient: dump: 2024-09-18T12:29:07.471 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f054a7fc640 1 -- 172.21.15.144:0/813415783 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f053c039040 con 0x7f0554131c90 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f055895d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f055895d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f055895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f055895d640 1 -- 172.21.15.144:0/813415783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0538005010 con 0x7f0554131c90 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f054a7fc640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f05340384c0 0x7f053403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f054a7fc640 1 -- 172.21.15.144:0/813415783 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f053c06bd80 con 0x7f0554131c90 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f055259c640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f05340384c0 0x7f053403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.443+0000 7f055259c640 10 monclient: get_auth_request con 0x7f05340384c0 auth_method 0 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.446+0000 7f055259c640 1 --2- 172.21.15.144:0/813415783 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f05340384c0 0x7f053403a970 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0544009d20 tx=0x7f05440091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.447+0000 7f054a7fc640 1 -- 172.21.15.144:0/813415783 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f053c02f2a0 con 0x7f0554131c90 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.447+0000 7f054a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.447+0000 7f054a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:07.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.539+0000 7ff080d5d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.539+0000 7ff080d5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.539+0000 7ff080d5d640 10 monclient: monmap: 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:07.540192+0000 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:07.540192+0000 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 Processor -- start 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 -- start start 2024-09-18T12:29:07.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient: init 2024-09-18T12:29:07.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:29:07.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff07c09c860 0x7ff07c09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff07c09c860 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:07.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff07c09aba0 0x7ff07c09d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff07c09aba0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09afa0 0x7ff07c09dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff07c09afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07c09a9d0 con 0x7ff07c09aba0 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07c06a320 con 0x7ff07c09c860 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c068a20 con 0x7ff07c09afa0 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff080d5d640 10 monclient(hunting): authenticate will time out at 974.861766s 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff079d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff07c09aba0 0x7ff07c09d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff07ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09afa0 0x7ff07c09dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff079d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff07c09aba0 0x7ff07c09d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52000/0 (socket says 172.21.15.116:52000) 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff079d9b640 1 -- 172.21.15.116:0/2127190023 learned_addr learned my addr 172.21.15.116:0/2127190023 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff079d9b640 10 monclient(hunting): get_auth_request con 0x7ff07c09aba0 auth_method 0 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff079d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff079d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff07ad9d640 10 monclient(hunting): get_auth_request con 0x7ff07c09afa0 auth_method 0 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff07ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff07ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.540+0000 7ff07ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 1 --2- 172.21.15.116:0/2127190023 >> v2:172.21.15.144:3300/0 conn(0x7ff07c09c860 0x7ff07c09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 10 monclient(hunting): get_auth_request con 0x7ff07c09c860 auth_method 0 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 10 monclient(hunting): handle_auth_done global_id 54396 payload 274 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 1 -- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09afa0 msgr2=0x7ff07c09dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 1 --2- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09afa0 0x7ff07c09dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 1 -- 172.21.15.116:0/2127190023 >> v2:172.21.15.144:3300/0 conn(0x7ff07c09c860 msgr2=0x7ff07c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 1 --2- 172.21.15.116:0/2127190023 >> v2:172.21.15.144:3300/0 conn(0x7ff07c09c860 0x7ff07c09cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 1 monclient: found mon.noname-a 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff079d9b640 1 -- 172.21.15.116:0/2127190023 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c068e60 con 0x7ff07c09aba0 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.541+0000 7ff07ad9d640 1 --2- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09afa0 0x7ff07c09dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.543+0000 7ff079d9b640 1 -- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 msgr2=0x7ff07c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.543+0000 7ff079d9b640 1 --2- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c09d830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff07c071740 tx=0x7ff0700091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fcbd829208a2b2d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 1 -- 172.21.15.116:0/2127190023 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070032440 con 0x7ff07c09aba0 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: dump: 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 1 -- 172.21.15.116:0/2127190023 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0700325a0 con 0x7ff07c09aba0 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 5 monclient: authenticate success, global_id 54396 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 4 set_mon_vals no callback set 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 1 -- 172.21.15.116:0/2127190023 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070032980 con 0x7ff07c09aba0 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff07959a640 10 monclient: dump: 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.560 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 10 monclient: shutdown 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 -- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 msgr2=0x7ff07c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 --2- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c09d830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff07c071740 tx=0x7ff0700091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 -- 172.21.15.116:0/2127190023 shutdown_connections 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 --2- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09afa0 0x7ff07c09dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 --2- 172.21.15.116:0/2127190023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c09d830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 --2- 172.21.15.116:0/2127190023 >> v2:172.21.15.144:3300/0 conn(0x7ff07c09c860 0x7ff07c09cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 -- 172.21.15.116:0/2127190023 >> 172.21.15.116:0/2127190023 conn(0x7ff07c0a2e40 msgr2=0x7ff07c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 -- 172.21.15.116:0/2127190023 shutdown_connections 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.544+0000 7ff080d5d640 1 -- 172.21.15.116:0/2127190023 wait complete. 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient: monmap: 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:07.562 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 Processor -- start 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 -- start start 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient: init 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff07c09aba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff07c09afa0 0x7ff07c1415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff07c09afa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09c860 0x7ff07c131ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff07c09c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c141b00 con 0x7ff07c09aba0 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c099cc0 con 0x7ff07c09afa0 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c11f110 con 0x7ff07c09c860 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff080d5d640 10 monclient(hunting): authenticate will time out at 974.866922s 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.545+0000 7ff07a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c06ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52006/0 (socket says 172.21.15.116:52006) 2024-09-18T12:29:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09c860 0x7ff07c131ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 1 -- 172.21.15.116:0/3027389068 learned_addr learned my addr 172.21.15.116:0/3027389068 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 10 monclient(hunting): get_auth_request con 0x7ff07c09aba0 auth_method 0 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): get_auth_request con 0x7ff07c09c860 auth_method 0 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff07c09afa0 0x7ff07c1415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 10 monclient(hunting): get_auth_request con 0x7ff07c09afa0 auth_method 0 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_done global_id 64118 payload 824 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 1 -- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff07c09afa0 msgr2=0x7ff07c1415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff07c09afa0 0x7ff07c1415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 1 -- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 msgr2=0x7ff07c06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c06ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 1 monclient: found mon.noname-c 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07ad9d640 1 -- 172.21.15.116:0/3027389068 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff074009a20 con 0x7ff07c09c860 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.546+0000 7ff07a59c640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff07ad9d640 1 -- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09c860 msgr2=0x7ff07c131ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff07ad9d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09c860 0x7ff07c131ed0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff074009f10 tx=0x7ff07400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 1 -- 172.21.15.116:0/3027389068 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074014480 con 0x7ff07c09c860 2024-09-18T12:29:07.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:07.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: dump: 2024-09-18T12:29:07.569 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.569 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.570 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.570 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.570 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.570 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.570 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.570 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.571 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.571 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 1 -- 172.21.15.116:0/3027389068 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07400f040 con 0x7ff07c09c860 2024-09-18T12:29:07.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff080d5d640 5 monclient: authenticate success, global_id 64118 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 1 -- 172.21.15.116:0/3027389068 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074014850 con 0x7ff07c09c860 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff080d5d640 10 monclient: _renew_subs 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff080d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff070009450 con 0x7ff07c09c860 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff078d99640 4 set_mon_vals no callback set 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff06a7fc640 10 monclient: dump: 2024-09-18T12:29:07.572 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff080d5d640 10 monclient: _renew_subs 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff080d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.548+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c1202e0 con 0x7ff07c09c860 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff06a7fc640 1 -- 172.21.15.116:0/3027389068 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff074003020 con 0x7ff07c09c860 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff080d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff080d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff080d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c1206e0 con 0x7ff07c09c860 2024-09-18T12:29:07.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff06a7fc640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff05c038490 0x7ff05c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff06a7fc640 1 -- 172.21.15.116:0/3027389068 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7ff07404c7a0 con 0x7ff07c09c860 2024-09-18T12:29:07.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff07a59c640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff05c038490 0x7ff05c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.549+0000 7ff07a59c640 10 monclient: get_auth_request con 0x7ff05c038490 auth_method 0 2024-09-18T12:29:07.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.552+0000 7ff07a59c640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff05c038490 0x7ff05c03a940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0114a0 tx=0x7ff06c0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.554+0000 7ff06a7fc640 1 -- 172.21.15.116:0/3027389068 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff0740112c0 con 0x7ff07c09c860 2024-09-18T12:29:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.554+0000 7ff06a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.554+0000 7ff06a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:07.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.609+0000 7f055895d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:07.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.609+0000 7f055895d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:07.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.609+0000 7f055895d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.609+0000 7f055895d640 1 -- 172.21.15.144:0/813415783 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f05380052c0 con 0x7f0554131c90 2024-09-18T12:29:07.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.610+0000 7f054a7fc640 1 -- 172.21.15.144:0/813415783 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v81) v1 ==== 99+0+0 (secure 0 0 0) 0x7f053c046e00 con 0x7f0554131c90 2024-09-18T12:29:07.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.610+0000 7f054a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:07.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.610+0000 7f054a7fc640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T12:29:07.611 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T12:29:07.615 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:29:07.615 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush rm bar 2024-09-18T12:29:07.615 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:29:07.615 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm bar 2024-09-18T12:29:07.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.686+0000 7f19b18e8640 10 monclient: get_monmap_and_config 2024-09-18T12:29:07.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.686+0000 7f19b18e8640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.686+0000 7f19b18e8640 10 monclient: monmap: 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:07.687716+0000 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:07.687716+0000 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:07.689 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:07.690 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:07.690 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:07.690 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:07.690 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 1 Processor -- start 2024-09-18T12:29:07.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 1 -- start start 2024-09-18T12:29:07.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient: init 2024-09-18T12:29:07.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:29:07.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19ac09ec50 0x7f19ac131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient(hunting): picked mon.noname-c con 0x7f19ac09ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:07.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19ac1321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19ac09f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac067420 con 0x7f19ac1321a0 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac0a0480 con 0x7f19ac09f020 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ac0a0200 con 0x7f19ac09ec50 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.687+0000 7f19b18e8640 10 monclient(hunting): authenticate will time out at 983.256892s 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19ac09ec50 0x7f19ac131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19ac09ec50 0x7f19ac131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42868/0 (socket says 172.21.15.144:42868) 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 1 -- 172.21.15.144:0/1637237387 learned_addr learned my addr 172.21.15.144:0/1637237387 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 10 monclient(hunting): get_auth_request con 0x7f19ac09ec50 auth_method 0 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.688+0000 7f19aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19ab7fe640 1 -- 172.21.15.144:0/1637237387 >> v1:172.21.15.144:6789/0 conn(0x7f19ac09f020 legacy=0x7f19ac069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19a27fc640 1 -- 172.21.15.144:0/1637237387 >> v1:172.21.15.116:6789/0 conn(0x7f19ac1321a0 legacy=0x7f19ac068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 10 monclient(hunting): handle_auth_done global_id 5161 payload 274 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 1 -- 172.21.15.144:0/1637237387 >> v1:172.21.15.144:6789/0 conn(0x7f19ac09f020 legacy=0x7f19ac069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 1 -- 172.21.15.144:0/1637237387 >> v1:172.21.15.116:6789/0 conn(0x7f19ac1321a0 legacy=0x7f19ac068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 1 monclient: found mon.noname-c 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.690+0000 7f19aaffd640 1 -- 172.21.15.144:0/1637237387 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac09fd80 con 0x7f19ac09ec50 2024-09-18T12:29:07.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aaffd640 1 -- 172.21.15.144:0/1637237387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 msgr2=0x7f19ac131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aaffd640 1 --2- 172.21.15.144:0/1637237387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac131c60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f199c009a10 tx=0x7f199c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bc6931824ad7352 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 1 -- 172.21.15.144:0/1637237387 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c02f4b0 con 0x7f19ac09ec50 2024-09-18T12:29:07.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: dump: 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 5 monclient: authenticate success, global_id 5161 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 1 -- 172.21.15.144:0/1637237387 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c02f610 con 0x7f19ac09ec50 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 4 set_mon_vals no callback set 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 1 -- 172.21.15.144:0/1637237387 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c02f9b0 con 0x7f19ac09ec50 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19aa7fc640 10 monclient: dump: 2024-09-18T12:29:07.703 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 10 monclient: shutdown 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 1 -- 172.21.15.144:0/1637237387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 msgr2=0x7f19ac131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 1 --2- 172.21.15.144:0/1637237387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac131c60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f199c009a10 tx=0x7f199c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 1 -- 172.21.15.144:0/1637237387 shutdown_connections 2024-09-18T12:29:07.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 1 --2- 172.21.15.144:0/1637237387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac131c60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 1 -- 172.21.15.144:0/1637237387 >> 172.21.15.144:0/1637237387 conn(0x7f19ac0a5cf0 msgr2=0x7f19ac0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 1 -- 172.21.15.144:0/1637237387 shutdown_connections 2024-09-18T12:29:07.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.693+0000 7f19b18e8640 1 -- 172.21.15.144:0/1637237387 wait complete. 2024-09-18T12:29:07.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.694+0000 7f19b18e8640 10 monclient: build_initial_monmap 2024-09-18T12:29:07.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.694+0000 7f19b18e8640 10 monclient: monmap: 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.694+0000 7f19b18e8640 1 Processor -- start 2024-09-18T12:29:07.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.694+0000 7f19b18e8640 1 -- start start 2024-09-18T12:29:07.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.694+0000 7f19b18e8640 10 monclient: init 2024-09-18T12:29:07.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:07.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:07.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac11de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): picked mon.noname-b con 0x7f19ac09ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:07.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ac09f020 0x7f19ac11e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): picked mon.noname-a con 0x7f19ac09f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:07.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19ac1321a0 0x7f19ac11e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): picked mon.noname-c con 0x7f19ac1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:07.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac11fa60 con 0x7f19ac09f020 2024-09-18T12:29:07.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac09c020 con 0x7f19ac09ec50 2024-09-18T12:29:07.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:07.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac09e1c0 con 0x7f19ac1321a0 2024-09-18T12:29:07.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:07.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19b18e8640 10 monclient(hunting): authenticate will time out at 983.264268s 2024-09-18T12:29:07.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac11de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac11de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42870/0 (socket says 172.21.15.144:42870) 2024-09-18T12:29:07.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 -- 172.21.15.144:0/1198905729 learned_addr learned my addr 172.21.15.144:0/1198905729 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:07.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): get_auth_request con 0x7f19ac09ec50 auth_method 0 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ac09f020 0x7f19ac11e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19ac1321a0 0x7f19ac11e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 10 monclient(hunting): get_auth_request con 0x7f19ac09f020 auth_method 0 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 10 monclient(hunting): get_auth_request con 0x7f19ac1321a0 auth_method 0 2024-09-18T12:29:07.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:07.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:07.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:07.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:07.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:07.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient(hunting): handle_auth_done global_id 5164 payload 824 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 -- 172.21.15.144:0/1198905729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19ac1321a0 msgr2=0x7f19ac11e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19ac1321a0 0x7f19ac11e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 -- 172.21.15.144:0/1198905729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ac09f020 msgr2=0x7f19ac11e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19ac09f020 0x7f19ac11e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 monclient: found mon.noname-b 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.695+0000 7f19aaffd640 1 -- 172.21.15.144:0/1198905729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c009450 con 0x7f19ac09ec50 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19aaffd640 1 -- 172.21.15.144:0/1198905729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 msgr2=0x7f19ac11de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19aaffd640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac11de90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f199c009210 tx=0x7f199c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19a37fe640 1 -- 172.21.15.144:0/1198905729 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c032a30 con 0x7f19ac09ec50 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19a37fe640 10 monclient: dump: 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.715 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.716 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.716 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.716 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.716 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.716 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.716 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19a37fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19a37fe640 1 -- 172.21.15.144:0/1198905729 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c032bd0 con 0x7f19ac09ec50 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19b18e8640 5 monclient: authenticate success, global_id 5164 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19b18e8640 10 monclient: _renew_subs 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.697+0000 7f19b18e8640 1 -- 172.21.15.144:0/1198905729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac121500 con 0x7f19ac09ec50 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19a37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19a37fe640 1 -- 172.21.15.144:0/1198905729 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c03b400 con 0x7f19ac09ec50 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:07.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19a37fe640 10 monclient: dump: 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19a9ffb640 4 set_mon_vals no callback set 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19b18e8640 10 monclient: _renew_subs 2024-09-18T12:29:07.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19b18e8640 1 -- 172.21.15.144:0/1198905729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac131520 con 0x7f19ac09ec50 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19aaffd640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19ac09ec50 0x7f19ac11de90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f199c009210 tx=0x7f199c032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f199c039040 mgrmap(e 5) v1 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19a37fe640 1 -- 172.21.15.144:0/1198905729 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f199c039040 con 0x7f19ac09ec50 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19b18e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19b18e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.698+0000 7f19b18e8640 1 -- 172.21.15.144:0/1198905729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ac0008d0 con 0x7f19ac09ec50 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.699+0000 7f19a37fe640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f198c038490 0x7f198c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.699+0000 7f19a27fc640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f198c038490 0x7f198c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:07.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.699+0000 7f19a27fc640 10 monclient: get_auth_request con 0x7f198c038490 auth_method 0 2024-09-18T12:29:07.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.701+0000 7f19a27fc640 1 --2- 172.21.15.144:0/1198905729 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f198c038490 0x7f198c03a940 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1994007fd0 tx=0x7f1994009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:07.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:07.703+0000 7f19a37fe640 1 -- 172.21.15.144:0/1198905729 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f199c06be70 con 0x7f19ac09ec50 2024-09-18T12:29:07.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.769+0000 7ff080d5d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:29:07.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.769+0000 7ff080d5d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:29:07.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.769+0000 7ff080d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:07.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.769+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7ff07c04bf60 con 0x7ff07c09c860 2024-09-18T12:29:07.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.770+0000 7ff06a7fc640 1 -- 172.21.15.116:0/3027389068 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7ff074012340 con 0x7ff07c09c860 2024-09-18T12:29:07.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.770+0000 7ff06a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:29:07.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.770+0000 7ff06a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.770+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.770+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=523041582,config=3140514009,health=1535303502,kv=2187923472,logm=2895102968}) 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.770+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=1200399075,config=3140514009,health=1535303502,kv=2187923472,logm=4270533694}) 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.770+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.770+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=523041582,config=3140514009,health=1535303502,kv=2187923472,logm=2895102968}) 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.770+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=523041582,config=3140514009,health=1535303502,kv=2187923472,logm=3940401070}) 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.771+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.771+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884077374}) 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.771+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181889637}) 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.771+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.771+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884077374}) 2024-09-18T12:29:07.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.771+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328415813}) 2024-09-18T12:29:07.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176318803}) 2024-09-18T12:29:07.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828906716}) 2024-09-18T12:29:07.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176318803}) 2024-09-18T12:29:07.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154098493}) 2024-09-18T12:29:07.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff05c038490 msgr2=0x7ff05c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff05c038490 0x7ff05c03a940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0114a0 tx=0x7ff06c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 10 monclient: shutdown 2024-09-18T12:29:07.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:07.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09c860 msgr2=0x7ff07c131ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:07.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09c860 0x7ff07c131ed0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff074009f10 tx=0x7ff07400b040 comp rx=0 tx=0).stop 2024-09-18T12:29:07.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630333054}) 2024-09-18T12:29:07.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 shutdown_connections 2024-09-18T12:29:07.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536931456}) 2024-09-18T12:29:07.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff05c038490 0x7ff05c03a940 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff07c09c860 0x7ff07c131ed0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff07c09afa0 0x7ff07c1415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 --2- 172.21.15.116:0/3027389068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07c09aba0 0x7ff07c06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:07.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 >> 172.21.15.116:0/3027389068 conn(0x7ff07c0a2e40 msgr2=0x7ff07c09b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:07.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630333054}) 2024-09-18T12:29:07.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.772+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474604122}) 2024-09-18T12:29:07.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 shutdown_connections 2024-09-18T12:29:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:07.772+0000 7ff080d5d640 1 -- 172.21.15.116:0/3027389068 wait complete. 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.773+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.773+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331171589}) 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.773+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2832460932}) 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.773+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.773+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331171589}) 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.773+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3833239485}) 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332059551}) 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754268465}) 2024-09-18T12:29:07.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332059551}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1829174756}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002461426}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768333076}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002461426}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.774+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2451321800}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.775+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.775+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919173842}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.775+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524712470}) 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.775+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.775+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919173842}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.775+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269705509}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284098731}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2318853352}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284098731}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2344428965}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394674509}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517519096}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394674509}) 2024-09-18T12:29:07.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.776+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2033158972}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.777+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.777+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531666617}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.777+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81826424}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.777+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.777+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531666617}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.777+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604659062}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.778+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.778+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=73,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=1166548517,mdsmap=1531151339,mgr=2854130552,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1746713598,mon_config_key=3655569570}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.778+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=73,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=2} crc {logm=2768483442,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1920779167,mon_config_key=2947491177}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.778+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.778+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=73,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=1166548517,mdsmap=1531151339,mgr=2854130552,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1746713598,mon_config_key=3655569570}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.778+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=1} crc {logm=1023483685,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3683213454,mon_config_key=1611758519,monmap=1305321694}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.779+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.779+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3750608922,osd_metadata=2440437218,osd_pg_creating=4057149032,osd_snap=2033388227,osdmap=761406380}) 2024-09-18T12:29:07.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.779+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=2528230717,osd_metadata=2440437218,osd_pg_creating=4057149032,osd_snap=2033388227,osdmap=325048778}) 2024-09-18T12:29:07.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.779+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.779+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3750608922,osd_metadata=2440437218,osd_pg_creating=4057149032,osd_snap=2033388227,osdmap=761406380}) 2024-09-18T12:29:07.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.779+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=90} crc {monmap=148614106,osd_metadata=2377971079,osd_pg_creating=4057149032,osd_snap=2033388227,osdmap=2994286455}) 2024-09-18T12:29:07.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.781+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:29:07.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.781+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=42} crc {osdmap=2096739543}) 2024-09-18T12:29:07.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:29:07.781+0000 7f6d913f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=4002010718}) 2024-09-18T12:29:07.788 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:29:08.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.527+0000 7f19a37fe640 1 -- 172.21.15.144:0/1198905729 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f199c03b7b0 con 0x7f19ac09ec50 2024-09-18T12:29:08.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.527+0000 7f19a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:08.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.527+0000 7f19a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:08.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.685+0000 7f19b18e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:08.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.685+0000 7f19b18e8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:08.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.685+0000 7f19b18e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:08.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.685+0000 7f19b18e8640 1 -- 172.21.15.144:0/1198905729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f19ac0a13e0 con 0x7f19ac09ec50 2024-09-18T12:29:08.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.686+0000 7f19a37fe640 1 -- 172.21.15.144:0/1198905729 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v81) v1 ==== 99+0+0 (secure 0 0 0) 0x7f199c03f020 con 0x7f19ac09ec50 2024-09-18T12:29:08.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.686+0000 7f19a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:08.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.686+0000 7f19a37fe640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T12:29:08.688 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T12:29:08.691 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:29:08.691 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush unlink host1 bar 2024-09-18T12:29:08.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.764+0000 7f351739d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:08.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.764+0000 7f351739d640 10 monclient: build_initial_monmap 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.764+0000 7f351739d640 10 monclient: monmap: 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:08.764932+0000 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:08.764932+0000 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:08.766 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:08.767 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:08.767 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:08.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.764+0000 7f351739d640 1 Processor -- start 2024-09-18T12:29:08.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.764+0000 7f351739d640 1 -- start start 2024-09-18T12:29:08.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.764+0000 7f351739d640 10 monclient: init 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3510096a30 0x7f351009d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient(hunting): picked mon.noname-b con 0x7f3510096a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3510096e00 0x7f35100971d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient(hunting): picked mon.noname-c con 0x7f3510096e00 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f351012c6a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351009c400 con 0x7f351012c6a0 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351009b970 con 0x7f3510096e00 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351009c120 con 0x7f3510096a30 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f351739d640 10 monclient(hunting): authenticate will time out at 984.334350s 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3510096e00 0x7f35100971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3510096e00 0x7f35100971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42876/0 (socket says 172.21.15.144:42876) 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 -- 172.21.15.144:0/389200549 learned_addr learned my addr 172.21.15.144:0/389200549 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): get_auth_request con 0x7f3510096e00 auth_method 0 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:08.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3515139640 1 --2- 172.21.15.144:0/389200549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3510096a30 0x7f351009d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3515139640 10 monclient(hunting): get_auth_request con 0x7f3510096a30 auth_method 0 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3515139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3515139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3515139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient(hunting): handle_auth_done global_id 5167 payload 274 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 -- 172.21.15.144:0/389200549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3510096a30 msgr2=0x7f351009d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 --2- 172.21.15.144:0/389200549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3510096a30 0x7f351009d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 -- 172.21.15.144:0/389200549 >> v1:172.21.15.116:6789/0 conn(0x7f351012c6a0 legacy=0x7f351012cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient: _finish_hunting 0 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 monclient: found mon.noname-c 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:08.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.765+0000 7f3514938640 1 -- 172.21.15.144:0/389200549 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351009c6e0 con 0x7f3510096e00 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3514938640 1 -- 172.21.15.144:0/389200549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096e00 msgr2=0x7f35100971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3514938640 1 --2- 172.21.15.144:0/389200549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096e00 0x7f35100971d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3500009910 tx=0x7f35000091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7db7ff16feba4ad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 1 -- 172.21.15.144:0/389200549 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350002ae90 con 0x7f3510096e00 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: dump: 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:08.782 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.783 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.783 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.783 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:08.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:08.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:08.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 1 -- 172.21.15.144:0/389200549 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500032440 con 0x7f3510096e00 2024-09-18T12:29:08.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f351739d640 5 monclient: authenticate success, global_id 5167 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f351739d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 1 -- 172.21.15.144:0/389200549 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500032820 con 0x7f3510096e00 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f351739d640 4 set_mon_vals no callback set 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.768+0000 7f3507fff640 10 monclient: dump: 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:08.784 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 10 monclient: shutdown 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 -- 172.21.15.144:0/389200549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096e00 msgr2=0x7f35100971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 --2- 172.21.15.144:0/389200549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096e00 0x7f35100971d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3500009910 tx=0x7f35000091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 -- 172.21.15.144:0/389200549 shutdown_connections 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 --2- 172.21.15.144:0/389200549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096e00 0x7f35100971d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 --2- 172.21.15.144:0/389200549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3510096a30 0x7f351009d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 -- 172.21.15.144:0/389200549 >> 172.21.15.144:0/389200549 conn(0x7f35100711a0 msgr2=0x7f35100a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:08.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 -- 172.21.15.144:0/389200549 shutdown_connections 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 1 -- 172.21.15.144:0/389200549 wait complete. 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 10 monclient: build_initial_monmap 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.769+0000 7f351739d640 10 monclient: monmap: 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:08.786 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 Processor -- start 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 -- start start 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient: init 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 0x7f351013ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): picked mon.noname-b con 0x7f3510096a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3510096e00 0x7f351012b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): picked mon.noname-a con 0x7f3510096e00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:08.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f351012c6a0 0x7f3510130730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): picked mon.noname-c con 0x7f351012c6a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3510132930 con 0x7f3510096e00 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3510118c10 con 0x7f3510096a30 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351013fcb0 con 0x7f351012c6a0 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351739d640 10 monclient(hunting): authenticate will time out at 984.339828s 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 0x7f351013ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 0x7f351013ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42892/0 (socket says 172.21.15.144:42892) 2024-09-18T12:29:08.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 1 -- 172.21.15.144:0/3585386291 learned_addr learned my addr 172.21.15.144:0/3585386291 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:08.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 10 monclient(hunting): get_auth_request con 0x7f3510096a30 auth_method 0 2024-09-18T12:29:08.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f3515139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3514938640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3510096e00 0x7f351012b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.770+0000 7f351593a640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f351012c6a0 0x7f3510130730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3514938640 10 monclient(hunting): get_auth_request con 0x7f3510096e00 auth_method 0 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3514938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3514938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3514938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f351593a640 10 monclient(hunting): get_auth_request con 0x7f351012c6a0 auth_method 0 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f351593a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f351593a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f351593a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 10 monclient(hunting): handle_auth_done global_id 5170 payload 824 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 1 -- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f351012c6a0 msgr2=0x7f3510130730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f351012c6a0 0x7f3510130730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 1 -- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3510096e00 msgr2=0x7f351012b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3510096e00 0x7f351012b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 10 monclient: _finish_hunting 0 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 1 monclient: found mon.noname-b 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3515139640 1 -- 172.21.15.144:0/3585386291 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3500009450 con 0x7f3510096a30 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f351593a640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f351012c6a0 0x7f3510130730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.771+0000 7f3514938640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3510096e00 0x7f351012b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:08.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f3515139640 1 -- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 msgr2=0x7f351013ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f3515139640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 0x7f351013ee60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f350c00d890 tx=0x7f350c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00de20 con 0x7f3510096a30 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: dump: 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f351739d640 5 monclient: authenticate success, global_id 5170 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c00bae0 con 0x7f3510096a30 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f351739d640 10 monclient: _renew_subs 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f351739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3510132290 con 0x7f3510096a30 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00be80 con 0x7f3510096a30 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35057fa640 10 monclient: dump: 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f35077fe640 4 set_mon_vals no callback set 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f351739d640 10 monclient: _renew_subs 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f351739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.773+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351013f3a0 con 0x7f3510096a30 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f350c014640 con 0x7f3510096a30 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f351739d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f351739d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f351739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351013f620 con 0x7f3510096a30 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f35057fa640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34f40384e0 0x7f34f403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f350c04c460 con 0x7f3510096a30 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f3514938640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34f40384e0 0x7f34f403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.774+0000 7f3514938640 10 monclient: get_auth_request con 0x7f34f40384e0 auth_method 0 2024-09-18T12:29:08.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.777+0000 7f3514938640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34f40384e0 0x7f34f403a990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3500034000 tx=0x7f3500032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:08.796 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:29:08.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:29:08.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.779+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f350c011570 con 0x7f3510096a30 2024-09-18T12:29:08.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.779+0000 7f35057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:08.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.779+0000 7f35057fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:08.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.898+0000 7f5c0d48b640 10 monclient: get_monmap_and_config 2024-09-18T12:29:08.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.898+0000 7f5c0d48b640 10 monclient: build_initial_monmap 2024-09-18T12:29:08.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.899+0000 7f5c0d48b640 10 monclient: monmap: 2024-09-18T12:29:08.900 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:08.900 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.901 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:08.899646+0000 2024-09-18T12:29:08.901 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:08.899646+0000 2024-09-18T12:29:08.901 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:08.901 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:08.901 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:08.901 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:08.902 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:08.902 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:08.902 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:08.902 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:08.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.899+0000 7f5c0d48b640 1 Processor -- start 2024-09-18T12:29:08.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.899+0000 7f5c0d48b640 1 -- start start 2024-09-18T12:29:08.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.899+0000 7f5c0d48b640 10 monclient: init 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c0806eb80 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c08068a20 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08137ac0 0x7f5c08139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c08137ac0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c08069b80 con 0x7f5c08068a20 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c08069e00 con 0x7f5c0806eb80 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08099f90 con 0x7f5c08137ac0 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c0d48b640 10 monclient(hunting): authenticate will time out at 976.221294s 2024-09-18T12:29:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08137ac0 0x7f5c08139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5bfe7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5c08068a20 0x7f5c0809c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48566/0 (socket says 172.21.15.116:48566) 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5bfe7fc640 1 -- 172.21.15.116:0/3665308051 learned_addr learned my addr 172.21.15.116:0/3665308051 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): get_auth_request con 0x7f5c08137ac0 auth_method 0 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient(hunting): handle_auth_done global_id 64130 payload 274 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 1 -- 172.21.15.116:0/3665308051 >> v1:172.21.15.144:6789/0 conn(0x7f5c0806eb80 legacy=0x7f5c0806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 1 -- 172.21.15.116:0/3665308051 >> v1:172.21.15.116:6789/0 conn(0x7f5c08068a20 legacy=0x7f5c0809c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 1 monclient: found mon.noname-b 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.900+0000 7f5c077fe640 1 -- 172.21.15.116:0/3665308051 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0806f660 con 0x7f5c08137ac0 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c077fe640 1 -- 172.21.15.116:0/3665308051 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08137ac0 msgr2=0x7f5c08139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c077fe640 1 --2- 172.21.15.116:0/3665308051 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08137ac0 0x7f5c08139ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009e90 tx=0x7f5c00009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2676ced9ea65d0d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 1 -- 172.21.15.116:0/3665308051 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007840 con 0x7f5c08137ac0 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: dump: 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 1 -- 172.21.15.116:0/3665308051 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0002d050 con 0x7f5c08137ac0 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c0d48b640 5 monclient: authenticate success, global_id 64130 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c0d48b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 1 -- 172.21.15.116:0/3665308051 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007bd0 con 0x7f5c08137ac0 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c067fc640 10 monclient: dump: 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c0d48b640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.903+0000 7f5c0d48b640 4 set_mon_vals no callback set 2024-09-18T12:29:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 10 monclient: shutdown 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3665308051 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08137ac0 msgr2=0x7f5c08139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3665308051 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08137ac0 0x7f5c08139ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009e90 tx=0x7f5c00009780 comp rx=0 tx=0).stop 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3665308051 shutdown_connections 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3665308051 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08137ac0 0x7f5c08139ea0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3665308051 >> 172.21.15.116:0/3665308051 conn(0x7f5c08067cd0 msgr2=0x7f5c080a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3665308051 shutdown_connections 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3665308051 wait complete. 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 10 monclient: build_initial_monmap 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.904+0000 7f5c0d48b640 10 monclient: monmap: 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 Processor -- start 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 -- start start 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient: init 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 0x7f5c0806c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c08068a20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0806eb80 0x7f5c080979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0806eb80 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c08137ac0 0x7f5c08097ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c08137ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0806d020 con 0x7f5c0806eb80 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0809c7e0 con 0x7f5c08137ac0 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08099400 con 0x7f5c08068a20 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c0d48b640 10 monclient(hunting): authenticate will time out at 976.226739s 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 0x7f5c0806c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0806eb80 0x7f5c080979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 0x7f5c0806c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59534/0 (socket says 172.21.15.116:59534) 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 1 -- 172.21.15.116:0/3860065333 learned_addr learned my addr 172.21.15.116:0/3860065333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 10 monclient(hunting): get_auth_request con 0x7f5c08068a20 auth_method 0 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 10 monclient(hunting): get_auth_request con 0x7f5c0806eb80 auth_method 0 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c077fe640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c08137ac0 0x7f5c08097ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5bfe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c077fe640 10 monclient(hunting): get_auth_request con 0x7f5c08137ac0 auth_method 0 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.905+0000 7f5c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 1 -- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c08137ac0 msgr2=0x7f5c08097ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c08137ac0 0x7f5c08097ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 1 -- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0806eb80 msgr2=0x7f5c080979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0806eb80 0x7f5c080979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 1 monclient: found mon.noname-c 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5c06ffd640 1 -- 172.21.15.116:0/3860065333 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c00009a20 con 0x7f5c08068a20 2024-09-18T12:29:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.906+0000 7f5bfe7fc640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0806eb80 0x7f5c080979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c06ffd640 1 -- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 msgr2=0x7f5c0806c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c06ffd640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 0x7f5c0806c4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c5e0 tx=0x7f5bf8007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 1 -- 172.21.15.116:0/3860065333 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8012840 con 0x7f5c08068a20 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: dump: 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c0d48b640 5 monclient: authenticate success, global_id 64133 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 1 -- 172.21.15.116:0/3860065333 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf80129a0 con 0x7f5c08068a20 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c0d48b640 10 monclient: _renew_subs 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c0d48b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0806cd40 con 0x7f5c08068a20 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 1 -- 172.21.15.116:0/3860065333 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8012de0 con 0x7f5c08068a20 2024-09-18T12:29:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5bff7fe640 10 monclient: dump: 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c0d48b640 10 monclient: _renew_subs 2024-09-18T12:29:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c05ffb640 4 set_mon_vals no callback set 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c0d48b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.908+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0806bde0 con 0x7f5c08068a20 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.909+0000 7f5bff7fe640 1 -- 172.21.15.116:0/3860065333 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5bf8011590 con 0x7f5c08068a20 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.909+0000 7f5c0d48b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.909+0000 7f5c0d48b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.909+0000 7f5c0d48b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.909+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0806d260 con 0x7f5c08068a20 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.909+0000 7f5bff7fe640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5be80384c0 0x7f5be803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.909+0000 7f5bff7fe640 1 -- 172.21.15.116:0/3860065333 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7f5bf804bf30 con 0x7f5c08068a20 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.910+0000 7f5bfe7fc640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5be80384c0 0x7f5be803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.910+0000 7f5bfe7fc640 10 monclient: get_auth_request con 0x7f5be80384c0 auth_method 0 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.912+0000 7f5bfe7fc640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5be80384c0 0x7f5be803a970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c08099130 tx=0x7f5bf0007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.914+0000 7f5bff7fe640 1 -- 172.21.15.116:0/3860065333 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5bf8010aa0 con 0x7f5c08068a20 2024-09-18T12:29:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.914+0000 7f5bff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:08.914+0000 7f5bff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:08.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.941+0000 7f351739d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T12:29:08.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.941+0000 7f351739d640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T12:29:08.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.941+0000 7f351739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:08.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:08.941+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f35100008d0 con 0x7f3510096a30 2024-09-18T12:29:09.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.126+0000 7f5c0d48b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:29:09.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.126+0000 7f5c0d48b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:29:09.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.126+0000 7f5c0d48b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:09.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.126+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c0804bf60 con 0x7f5c08068a20 2024-09-18T12:29:09.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.127+0000 7f5bff7fe640 1 -- 172.21.15.116:0/3860065333 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f5bf8016070 con 0x7f5c08068a20 2024-09-18T12:29:09.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.127+0000 7f5bff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:29:09.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.127+0000 7f5bff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:09.129 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:29:09.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.128+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5be80384c0 msgr2=0x7f5be803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.128+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5be80384c0 0x7f5be803a970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c08099130 tx=0x7f5bf0007f00 comp rx=0 tx=0).stop 2024-09-18T12:29:09.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 10 monclient: shutdown 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 msgr2=0x7f5c0806c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 0x7f5c0806c4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c5e0 tx=0x7f5bf8007ed0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 shutdown_connections 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5be80384c0 0x7f5be803a970 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c08137ac0 0x7f5c08097ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0806eb80 0x7f5c080979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 --2- 172.21.15.116:0/3860065333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c08068a20 0x7f5c0806c4d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 >> 172.21.15.116:0/3860065333 conn(0x7f5c08067cd0 msgr2=0x7f5c08131c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 shutdown_connections 2024-09-18T12:29:09.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.129+0000 7f5c0d48b640 1 -- 172.21.15.116:0/3860065333 wait complete. 2024-09-18T12:29:09.140 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:29:09.140 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:29:09.140 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:29:09.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:29:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.233+0000 7fc94e26f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.233+0000 7fc94e26f640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.233+0000 7fc94e26f640 10 monclient: monmap: 2024-09-18T12:29:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:09.233878+0000 2024-09-18T12:29:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:09.233878+0000 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:09.235 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.233+0000 7fc94e26f640 1 Processor -- start 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.233+0000 7fc94e26f640 1 -- start start 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.233+0000 7fc94e26f640 10 monclient: init 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 0x7fc948131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient(hunting): picked mon.noname-b con 0x7fc9481318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc94809ea50 0x7fc94809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient(hunting): picked mon.noname-c con 0x7fc94809ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc948068d90 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc948131cc0 con 0x7fc948068d90 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94809fbf0 con 0x7fc94809ea50 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94809fe30 con 0x7fc9481318f0 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc94e26f640 10 monclient(hunting): authenticate will time out at 976.555478s 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 0x7fc948131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 0x7fc948131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59544/0 (socket says 172.21.15.116:59544) 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 -- 172.21.15.116:0/3186393543 learned_addr learned my addr 172.21.15.116:0/3186393543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): get_auth_request con 0x7fc9481318f0 auth_method 0 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc9477fe640 1 --2- 172.21.15.116:0/3186393543 >> v2:172.21.15.144:3300/0 conn(0x7fc94809ea50 0x7fc94809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc94809ea50 auth_method 0 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient(hunting): handle_auth_done global_id 64136 payload 274 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 -- 172.21.15.116:0/3186393543 >> v2:172.21.15.144:3300/0 conn(0x7fc94809ea50 msgr2=0x7fc94809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 --2- 172.21.15.116:0/3186393543 >> v2:172.21.15.144:3300/0 conn(0x7fc94809ea50 0x7fc94809ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 -- 172.21.15.116:0/3186393543 >> v1:172.21.15.144:6789/0 conn(0x7fc948068d90 legacy=0x7fc9480691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 monclient: found mon.noname-b 2024-09-18T12:29:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.234+0000 7fc947fff640 1 -- 172.21.15.116:0/3186393543 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480a1210 con 0x7fc9481318f0 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc947fff640 1 -- 172.21.15.116:0/3186393543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 msgr2=0x7fc948131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc947fff640 1 --2- 172.21.15.116:0/3186393543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 0x7fc948131f30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc938009910 tx=0x7fc9380091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e67826d01ab4469 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 1 -- 172.21.15.116:0/3186393543 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93802ae90 con 0x7fc9481318f0 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: dump: 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 1 -- 172.21.15.116:0/3186393543 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc938032440 con 0x7fc9481318f0 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc94e26f640 5 monclient: authenticate success, global_id 64136 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc94e26f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:09.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 1 -- 172.21.15.116:0/3186393543 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938032820 con 0x7fc9481318f0 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc94e26f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc94e26f640 4 set_mon_vals no callback set 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.237+0000 7fc946ffd640 10 monclient: dump: 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.251 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 10 monclient: shutdown 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 -- 172.21.15.116:0/3186393543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 msgr2=0x7fc948131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 --2- 172.21.15.116:0/3186393543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 0x7fc948131f30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc938009910 tx=0x7fc9380091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 -- 172.21.15.116:0/3186393543 shutdown_connections 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 --2- 172.21.15.116:0/3186393543 >> v2:172.21.15.144:3300/0 conn(0x7fc94809ea50 0x7fc94809ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 --2- 172.21.15.116:0/3186393543 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc9481318f0 0x7fc948131f30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 -- 172.21.15.116:0/3186393543 >> 172.21.15.116:0/3186393543 conn(0x7fc948067110 msgr2=0x7fc948067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 -- 172.21.15.116:0/3186393543 shutdown_connections 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 1 -- 172.21.15.116:0/3186393543 wait complete. 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.238+0000 7fc94e26f640 10 monclient: monmap: 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 Processor -- start 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 -- start start 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient: init 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948068d90 0x7fc94809ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): picked mon.noname-a con 0x7fc948068d90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94809ea50 0x7fc948130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): picked mon.noname-c con 0x7fc94809ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9481318f0 0x7fc948131370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): picked mon.noname-b con 0x7fc9481318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94809bf40 con 0x7fc948068d90 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480a4db0 con 0x7fc9481318f0 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94809b900 con 0x7fc94809ea50 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc94e26f640 10 monclient(hunting): authenticate will time out at 976.560924s 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc947fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948068d90 0x7fc94809ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc9477fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94809ea50 0x7fc948130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc9477fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94809ea50 0x7fc948130e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59550/0 (socket says 172.21.15.116:59550) 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc947fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948068d90 0x7fc94809ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52024/0 (socket says 172.21.15.116:52024) 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc9477fe640 1 -- 172.21.15.116:0/306306847 learned_addr learned my addr 172.21.15.116:0/306306847 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc9477fe640 10 monclient(hunting): get_auth_request con 0x7fc94809ea50 auth_method 0 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc947fff640 10 monclient(hunting): get_auth_request con 0x7fc948068d90 auth_method 0 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.239+0000 7fc947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc94c80c640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9481318f0 0x7fc948131370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc94c80c640 10 monclient(hunting): get_auth_request con 0x7fc9481318f0 auth_method 0 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc94c80c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc94c80c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc94c80c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient(hunting): handle_auth_done global_id 54414 payload 824 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94809ea50 msgr2=0x7fc948130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94809ea50 0x7fc948130e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9481318f0 msgr2=0x7fc948131370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9481318f0 0x7fc948131370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 1 monclient: found mon.noname-a 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc947fff640 1 -- 172.21.15.116:0/306306847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc938009450 con 0x7fc948068d90 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.240+0000 7fc9477fe640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc94809ea50 0x7fc948130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc947fff640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948068d90 msgr2=0x7fc94809ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc947fff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948068d90 0x7fc94809ada0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc938031000 tx=0x7fc938032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938031d40 con 0x7fc948068d90 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: dump: 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc94e26f640 5 monclient: authenticate success, global_id 54414 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc94e26f640 10 monclient: _renew_subs 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc94e26f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93802ae90 con 0x7fc948068d90 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94811c650 con 0x7fc948068d90 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93803d690 con 0x7fc948068d90 2024-09-18T12:29:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc93ffff640 10 monclient: dump: 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc94e26f640 10 monclient: _renew_subs 2024-09-18T12:29:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc9467fc640 4 set_mon_vals no callback set 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc94e26f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.242+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94811bc30 con 0x7fc948068d90 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc938039040 con 0x7fc948068d90 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc94e26f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc94e26f640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 0x7fc948141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc94e26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948065e00 con 0x7fc948140f90 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 0x7fc948141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient: get_auth_request con 0x7fc948140f90 auth_method 0 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc93ffff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc928038490 0x7fc92803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6213+0+0 (secure 0 0 0) 0x7fc93806b8b0 con 0x7fc948068d90 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 10 monclient(hunting): handle_auth_done global_id 54417 payload 824 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc9477fe640 1 -- 172.21.15.116:0/306306847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc94811bee0 con 0x7fc948140f90 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc94c80c640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc928038490 0x7fc92803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.243+0000 7fc94c80c640 10 monclient: get_auth_request con 0x7fc928038490 auth_method 0 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc9477fe640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 msgr2=0x7fc948141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc9477fe640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 0x7fc948141360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc94809bc10 tx=0x7fc930008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930010ec0 con 0x7fc948140f90 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 10 monclient: dump: 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc94c80c640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc928038490 0x7fc92803a940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc940003e50 tx=0x7fc940009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc930014640 con 0x7fc948140f90 2024-09-18T12:29:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 msgr2=0x7fc948141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 0x7fc948141360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc94809bc10 tx=0x7fc930008040 comp rx=0 tx=0).stop 2024-09-18T12:29:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 msgr2=0x7fc948141360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.246+0000 7fc93ffff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948140f90 0x7fc948141360 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc94809bc10 tx=0x7fc930008040 comp rx=0 tx=0).stop 2024-09-18T12:29:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc94e26f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc94e26f640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 0x7fc948096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc94e26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94811e060 con 0x7fc9481418a0 2024-09-18T12:29:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 0x7fc948096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient: get_auth_request con 0x7fc9481418a0 auth_method 0 2024-09-18T12:29:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 10 monclient(hunting): handle_auth_done global_id 54420 payload 824 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.252+0000 7fc9477fe640 1 -- 172.21.15.116:0/306306847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc94804bf60 con 0x7fc9481418a0 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.254+0000 7fc9477fe640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 msgr2=0x7fc948096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.254+0000 7fc9477fe640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 0x7fc948096a60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc930007b50 tx=0x7fc93001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.254+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930027400 con 0x7fc9481418a0 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.254+0000 7fc93ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.254+0000 7fc93ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.254+0000 7fc93ffff640 10 monclient: dump: 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fc9300133d0 con 0x7fc9481418a0 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc93ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc93ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 msgr2=0x7fc948096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc93ffff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 0x7fc948096a60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc930007b50 tx=0x7fc93001c000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc93ffff640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 msgr2=0x7fc948096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc93ffff640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9481418a0 0x7fc948096a60 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc930007b50 tx=0x7fc93001c000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:09.269 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 44, 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:09.276 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.277 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.278 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.279 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.281 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.282 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:09.283 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc928038490 msgr2=0x7fc92803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc928038490 0x7fc92803a940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc940003e50 tx=0x7fc940009040 comp rx=0 tx=0).stop 2024-09-18T12:29:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 10 monclient: shutdown 2024-09-18T12:29:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948068d90 msgr2=0x7fc94809ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 --2- 172.21.15.116:0/306306847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc948068d90 0x7fc94809ada0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc938031000 tx=0x7fc938032000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94c80c640 1 -- 172.21.15.116:0/306306847 reap_dead start 2024-09-18T12:29:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94c80c640 1 -- 172.21.15.116:0/306306847 reap_dead start 2024-09-18T12:29:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 shutdown_connections 2024-09-18T12:29:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 >> 172.21.15.116:0/306306847 conn(0x7fc948067110 msgr2=0x7fc94809df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 shutdown_connections 2024-09-18T12:29:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.255+0000 7fc94e26f640 1 -- 172.21.15.116:0/306306847 wait complete. 2024-09-18T12:29:09.286 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:29:09.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:29:09.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:29:09.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.309+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v82) v1 ==== 147+0+0 (secure 0 0 0) 0x7f350c01dbb0 con 0x7f3510096a30 2024-09-18T12:29:09.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.309+0000 7f35057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T12:29:09.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.309+0000 7f35057fa640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-18T12:29:09.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.379+0000 7f1cb511d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:09.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.379+0000 7f1cb511d640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.379+0000 7f1cb511d640 10 monclient: monmap: 2024-09-18T12:29:09.384 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:09.384 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:09.380559+0000 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:09.380559+0000 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.380+0000 7f1cb511d640 1 Processor -- start 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.380+0000 7f1cb511d640 1 -- start start 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.380+0000 7f1cb511d640 10 monclient: init 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.380+0000 7f1cb511d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cb00974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cb009a8a0 0x7f1cb009ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 10 monclient(hunting): picked mon.noname-a con 0x7f1cb009a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cb0133070 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb012b210 con 0x7f1cb0133070 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb00978c0 con 0x7f1cb00974f0 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb009b220 con 0x7f1cb009a8a0 2024-09-18T12:29:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1cb511d640 10 monclient(hunting): authenticate will time out at 976.702092s 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cb009a8a0 0x7f1cb009ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cb009a8a0 0x7f1cb009ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52048/0 (socket says 172.21.15.116:52048) 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 1 -- 172.21.15.116:0/831303144 learned_addr learned my addr 172.21.15.116:0/831303144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): get_auth_request con 0x7f1cb009a8a0 auth_method 0 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1caf59e640 1 --1- 172.21.15.116:0/831303144 >> v1:172.21.15.116:6789/0 conn(0x7f1cb0133070 0x7f1cb0135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48574/0 (socket says 172.21.15.116:48574) 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient(hunting): handle_auth_done global_id 54423 payload 274 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 1 -- 172.21.15.116:0/831303144 >> v1:172.21.15.144:6789/0 conn(0x7f1cb00974f0 legacy=0x7f1cb012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 1 -- 172.21.15.116:0/831303144 >> v1:172.21.15.116:6789/0 conn(0x7f1cb0133070 legacy=0x7f1cb0135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 1 monclient: found mon.noname-a 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.381+0000 7f1ca659c640 1 -- 172.21.15.116:0/831303144 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb009ca20 con 0x7f1cb009a8a0 2024-09-18T12:29:09.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1ca659c640 1 -- 172.21.15.116:0/831303144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 msgr2=0x7f1cb009ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1ca659c640 1 --2- 172.21.15.116:0/831303144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb009ace0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800bbc0 tx=0x7f1c9800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d3ce1f29b62525 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 1 -- 172.21.15.116:0/831303144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c980327e0 con 0x7f1cb009a8a0 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: dump: 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.398 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 1 -- 172.21.15.116:0/831303144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98032940 con 0x7f1cb009a8a0 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 5 monclient: authenticate success, global_id 54423 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:09.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 1 -- 172.21.15.116:0/831303144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98032ce0 con 0x7f1cb009a8a0 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 4 set_mon_vals no callback set 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cae59c640 10 monclient: dump: 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.400 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 10 monclient: shutdown 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 1 -- 172.21.15.116:0/831303144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 msgr2=0x7f1cb009ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.384+0000 7f1cb511d640 1 --2- 172.21.15.116:0/831303144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb009ace0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800bbc0 tx=0x7f1c9800b460 comp rx=0 tx=0).stop 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 1 -- 172.21.15.116:0/831303144 shutdown_connections 2024-09-18T12:29:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 1 --2- 172.21.15.116:0/831303144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb009ace0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 1 -- 172.21.15.116:0/831303144 >> 172.21.15.116:0/831303144 conn(0x7f1cb00672b0 msgr2=0x7f1cb00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 1 -- 172.21.15.116:0/831303144 shutdown_connections 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 1 -- 172.21.15.116:0/831303144 wait complete. 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 10 monclient: monmap: 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.385+0000 7f1cb511d640 1 Processor -- start 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 1 -- start start 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient: init 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb00974f0 0x7f1cb012b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): picked mon.noname-b con 0x7f1cb00974f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb012be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): picked mon.noname-a con 0x7f1cb009a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb0133070 0x7f1cb012c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): picked mon.noname-c con 0x7f1cb0133070 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb0098300 con 0x7f1cb009a8a0 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb012a4f0 con 0x7f1cb00974f0 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb012ee50 con 0x7f1cb0133070 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1cb511d640 10 monclient(hunting): authenticate will time out at 976.707448s 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb012be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb012be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52052/0 (socket says 172.21.15.116:52052) 2024-09-18T12:29:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 -- 172.21.15.116:0/2546566625 learned_addr learned my addr 172.21.15.116:0/2546566625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): get_auth_request con 0x7f1cb009a8a0 auth_method 0 2024-09-18T12:29:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb0133070 0x7f1cb012c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb00974f0 0x7f1cb012b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 10 monclient(hunting): get_auth_request con 0x7f1cb0133070 auth_method 0 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 10 monclient(hunting): get_auth_request con 0x7f1cb00974f0 auth_method 0 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient(hunting): handle_auth_done global_id 54426 payload 824 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb0133070 msgr2=0x7f1cb012c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb0133070 0x7f1cb012c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb00974f0 msgr2=0x7f1cb012b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cb00974f0 0x7f1cb012b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 monclient: found mon.noname-a 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1ca659c640 1 -- 172.21.15.116:0/2546566625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9800b700 con 0x7f1cb009a8a0 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.386+0000 7f1caf59e640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb0133070 0x7f1cb012c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca659c640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 msgr2=0x7f1cb012be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca659c640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb012be50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b4a0 tx=0x7f1c9803a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98035d80 con 0x7f1cb009a8a0 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: dump: 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98040480 con 0x7f1cb009a8a0 2024-09-18T12:29:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cb511d640 5 monclient: authenticate success, global_id 54426 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98040840 con 0x7f1cb009a8a0 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cb511d640 10 monclient: _renew_subs 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cb511d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb012eb70 con 0x7f1cb009a8a0 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1ca77fe640 10 monclient: dump: 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cadd9b640 4 set_mon_vals no callback set 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cb511d640 10 monclient: _renew_subs 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cb511d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.389+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb012dc10 con 0x7f1cb009a8a0 2024-09-18T12:29:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1cb511d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1cb511d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 0x7f1cb006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1c9803c040 con 0x7f1cb009a8a0 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1cb511d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb0065f20 con 0x7f1cb012c8d0 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 0x7f1cb006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient: get_auth_request con 0x7f1cb012c8d0 auth_method 0 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1ca77fe640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c90038490 0x7f1c9003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f1c9806f3e0 con 0x7f1cb009a8a0 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 10 monclient(hunting): handle_auth_done global_id 64145 payload 824 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caed9d640 1 -- 172.21.15.116:0/2546566625 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cb012efc0 con 0x7f1cb012c8d0 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caf59e640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c90038490 0x7f1c9003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.390+0000 7f1caf59e640 10 monclient: get_auth_request con 0x7f1c90038490 auth_method 0 2024-09-18T12:29:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1caed9d640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 msgr2=0x7f1cb006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1caed9d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 0x7f1cb006b6b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000f780 tx=0x7f1ca0007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca000d8b0 con 0x7f1cb012c8d0 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1ca77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1ca77fe640 10 monclient: dump: 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1ca000da10 con 0x7f1cb012c8d0 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1ca77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.392+0000 7f1ca77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.393+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 msgr2=0x7f1cb006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.393+0000 7f1ca77fe640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 0x7f1cb006b6b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000f780 tx=0x7f1ca0007e10 comp rx=0 tx=0).stop 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.393+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 msgr2=0x7f1cb006b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.393+0000 7f1ca77fe640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb012c8d0 0x7f1cb006b6b0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000f780 tx=0x7f1ca0007e10 comp rx=0 tx=0).stop 2024-09-18T12:29:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.393+0000 7f1caf59e640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c90038490 0x7f1c9003a940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0097d70 tx=0x7f1ca8004e20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.408+0000 7f351739d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T12:29:09.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.408+0000 7f351739d640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T12:29:09.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.408+0000 7f351739d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.408+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f3510097710 con 0x7f3510096a30 2024-09-18T12:29:09.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.409+0000 7f35057fa640 1 -- 172.21.15.144:0/3585386291 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v82) v1 ==== 141+0+0 (secure 0 0 0) 0x7f350c011340 con 0x7f3510096a30 2024-09-18T12:29:09.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.409+0000 7f35057fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T12:29:09.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.409+0000 7f35057fa640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-18T12:29:09.421 INFO:tasks.workunit.client.0.smithi144.stderr:item -11 does not appear in that position 2024-09-18T12:29:09.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34f40384e0 msgr2=0x7f34f403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34f40384e0 0x7f34f403a990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3500034000 tx=0x7f3500032000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 10 monclient: shutdown 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 msgr2=0x7f351013ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 0x7f351013ee60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f350c00d890 tx=0x7f350c008040 comp rx=0 tx=0).stop 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 shutdown_connections 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34f40384e0 0x7f34f403a990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f351012c6a0 0x7f3510130730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3510096e00 0x7f351012b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 --2- 172.21.15.144:0/3585386291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3510096a30 0x7f351013ee60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 >> 172.21.15.144:0/3585386291 conn(0x7f35100711a0 msgr2=0x7f3510132b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 shutdown_connections 2024-09-18T12:29:09.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.411+0000 7f351739d640 1 -- 172.21.15.144:0/3585386291 wait complete. 2024-09-18T12:29:09.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1cb511d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1cb511d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 0x7f1cb006bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1cb511d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb004bf60 con 0x7f1cb006bbf0 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 0x7f1cb006bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient: get_auth_request con 0x7f1cb006bbf0 auth_method 0 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 10 monclient(hunting): handle_auth_done global_id 64148 payload 824 2024-09-18T12:29:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.398+0000 7f1caed9d640 1 -- 172.21.15.116:0/2546566625 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1cb012cca0 con 0x7f1cb006bbf0 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1caed9d640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 msgr2=0x7f1cb006bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1caed9d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 0x7f1cb006bff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000f780 tx=0x7f1ca0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0007030 con 0x7f1cb006bbf0 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 10 monclient: dump: 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f1ca0007190 con 0x7f1cb006bbf0 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 msgr2=0x7f1cb006bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 0x7f1cb006bff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000f780 tx=0x7f1ca0019000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 msgr2=0x7f1cb006bff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1ca77fe640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cb006bbf0 0x7f1cb006bff0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000f780 tx=0x7f1ca0019000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 44, 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.428 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 7, 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.429 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:09.431 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:09.433 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.434 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.435 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.436 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.437 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.438 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:09.439 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:09.440 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:09.440 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c host1 2024-09-18T12:29:09.440 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 1 2024-09-18T12:29:09.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c90038490 msgr2=0x7f1c9003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c90038490 0x7f1c9003a940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0097d70 tx=0x7f1ca8004e20 comp rx=0 tx=0).stop 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 10 monclient: shutdown 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 msgr2=0x7f1cb012be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 --2- 172.21.15.116:0/2546566625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cb009a8a0 0x7f1cb012be50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800b4a0 tx=0x7f1c9803a000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1caf59e640 1 -- 172.21.15.116:0/2546566625 reap_dead start 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1caf59e640 1 -- 172.21.15.116:0/2546566625 reap_dead start 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 shutdown_connections 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 >> 172.21.15.116:0/2546566625 conn(0x7f1cb00672b0 msgr2=0x7f1cb009e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 shutdown_connections 2024-09-18T12:29:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.401+0000 7f1cb511d640 1 -- 172.21.15.116:0/2546566625 wait complete. 2024-09-18T12:29:09.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:29:09.443 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:29:09.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:29:09.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.492+0000 7f617f71d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:09.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.492+0000 7f617f71d640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.492+0000 7f617f71d640 10 monclient: monmap: 2024-09-18T12:29:09.493 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:09.493 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.493 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:09.493051+0000 2024-09-18T12:29:09.493 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:09.493051+0000 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:09.494 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.492+0000 7f617f71d640 1 Processor -- start 2024-09-18T12:29:09.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.492+0000 7f617f71d640 1 -- start start 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.492+0000 7f617f71d640 10 monclient: init 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6178131b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6178131f10 0x7f6178132330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient(hunting): picked mon.noname-a con 0x7f6178131f10 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6178136e70 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617806a230 con 0x7f6178136e70 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61780a0300 con 0x7f6178131b40 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61780a00c0 con 0x7f6178131f10 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617f71d640 10 monclient(hunting): authenticate will time out at 985.062402s 2024-09-18T12:29:09.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617d4b9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f6178131b40 0x7f617809ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:50472/0 (socket says 172.21.15.144:50472) 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617d4b9640 1 -- 172.21.15.144:0/1556698283 learned_addr learned my addr 172.21.15.144:0/1556698283 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 1 --2- 172.21.15.144:0/1556698283 >> v2:172.21.15.116:3300/0 conn(0x7f6178131f10 0x7f6178132330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): get_auth_request con 0x7f6178131f10 auth_method 0 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient(hunting): handle_auth_done global_id 54429 payload 274 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 1 -- 172.21.15.144:0/1556698283 >> v1:172.21.15.144:6789/0 conn(0x7f6178131b40 legacy=0x7f617809ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 1 -- 172.21.15.144:0/1556698283 >> v1:172.21.15.116:6789/0 conn(0x7f6178136e70 legacy=0x7f6178139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 1 monclient: found mon.noname-a 2024-09-18T12:29:09.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.493+0000 7f617ccb8640 1 -- 172.21.15.144:0/1556698283 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617809bee0 con 0x7f6178131f10 2024-09-18T12:29:09.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.496+0000 7f617ccb8640 1 -- 172.21.15.144:0/1556698283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 msgr2=0x7f6178132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.496+0000 7f617ccb8640 1 --2- 172.21.15.144:0/1556698283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 0x7f6178132330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6168009850 tx=0x7f61680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5697a99823a12d5b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 1 -- 172.21.15.144:0/1556698283 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168032440 con 0x7f6178131f10 2024-09-18T12:29:09.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:09.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: dump: 2024-09-18T12:29:09.505 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 1 -- 172.21.15.144:0/1556698283 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61680325a0 con 0x7f6178131f10 2024-09-18T12:29:09.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 5 monclient: authenticate success, global_id 54429 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 4 set_mon_vals no callback set 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 1 -- 172.21.15.144:0/1556698283 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168032940 con 0x7f6178131f10 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f616ffff640 10 monclient: dump: 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.507 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 10 monclient: shutdown 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 1 -- 172.21.15.144:0/1556698283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 msgr2=0x7f6178132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 1 --2- 172.21.15.144:0/1556698283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 0x7f6178132330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6168009850 tx=0x7f61680090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 1 -- 172.21.15.144:0/1556698283 shutdown_connections 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 1 --2- 172.21.15.144:0/1556698283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 0x7f6178132330 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 1 -- 172.21.15.144:0/1556698283 >> 172.21.15.144:0/1556698283 conn(0x7f61780a5100 msgr2=0x7f61780a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 1 -- 172.21.15.144:0/1556698283 shutdown_connections 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.497+0000 7f617f71d640 1 -- 172.21.15.144:0/1556698283 wait complete. 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.498+0000 7f617f71d640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.498+0000 7f617f71d640 10 monclient: monmap: 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:09.509 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.498+0000 7f617f71d640 1 Processor -- start 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.498+0000 7f617f71d640 1 -- start start 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.498+0000 7f617f71d640 10 monclient: init 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6178131b40 0x7f617812eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): picked mon.noname-c con 0x7f6178131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 0x7f6178097a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): picked mon.noname-a con 0x7f6178131f10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 0x7f6178097f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): picked mon.noname-b con 0x7f6178136e70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617811b890 con 0x7f6178131f10 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6178099a40 con 0x7f6178136e70 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61780a0a30 con 0x7f6178131b40 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617f71d640 10 monclient(hunting): authenticate will time out at 985.068097s 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 0x7f6178097f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 0x7f6178097f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42898/0 (socket says 172.21.15.144:42898) 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 -- 172.21.15.144:0/2476130434 learned_addr learned my addr 172.21.15.144:0/2476130434 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): get_auth_request con 0x7f6178136e70 auth_method 0 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 0x7f6178097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6178131b40 0x7f617812eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 10 monclient(hunting): get_auth_request con 0x7f6178131f10 auth_method 0 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 10 monclient(hunting): get_auth_request con 0x7f6178131b40 auth_method 0 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617ccb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617d4b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient(hunting): handle_auth_done global_id 5185 payload 824 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 -- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6178131b40 msgr2=0x7f617812eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6178131b40 0x7f617812eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 -- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 msgr2=0x7f6178097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 0x7f6178097a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 monclient: found mon.noname-b 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.499+0000 7f617dcba640 1 -- 172.21.15.144:0/2476130434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6168009390 con 0x7f6178136e70 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f617dcba640 1 -- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 msgr2=0x7f6178097f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f617dcba640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 0x7f6178097f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f617000c890 tx=0x7f6170002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617000ce80 con 0x7f6178136e70 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: dump: 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f617f71d640 5 monclient: authenticate success, global_id 5185 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617000aa00 con 0x7f6178136e70 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f617f71d640 10 monclient: _renew_subs 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f617f71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6178142120 con 0x7f6178136e70 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617000ae00 con 0x7f6178136e70 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616d7fa640 10 monclient: dump: 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.516 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f616f7fe640 4 set_mon_vals no callback set 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.501+0000 7f617f71d640 10 monclient: _renew_subs 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.502+0000 7f617f71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.502+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61780994c0 con 0x7f6178136e70 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.502+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6170012630 con 0x7f6178136e70 2024-09-18T12:29:09.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f617f71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f617f71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f617f71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61780636e0 con 0x7f6178136e70 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f616d7fa640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f615c038490 0x7f615c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f617004c0c0 con 0x7f6178136e70 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f617d4b9640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f615c038490 0x7f615c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.503+0000 7f617d4b9640 10 monclient: get_auth_request con 0x7f615c038490 auth_method 0 2024-09-18T12:29:09.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.505+0000 7f617d4b9640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f615c038490 0x7f615c03a940 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6174007fd0 tx=0x7f6174007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.507+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6170022d70 con 0x7f6178136e70 2024-09-18T12:29:09.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.507+0000 7f616d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.507+0000 7f616d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:09.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.536+0000 7f95fad09640 10 monclient: get_monmap_and_config 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.536+0000 7f95fad09640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.537+0000 7f95fad09640 10 monclient: monmap: 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:09.537657+0000 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:09.537657+0000 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:09.538 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:09.539 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.537+0000 7f95fad09640 1 Processor -- start 2024-09-18T12:29:09.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.537+0000 7f95fad09640 1 -- start start 2024-09-18T12:29:09.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.537+0000 7f95fad09640 10 monclient: init 2024-09-18T12:29:09.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient(hunting): picked mon.noname-b con 0x7f95f4131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95f409fda0 0x7f95f4132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient(hunting): picked mon.noname-a con 0x7f95f409fda0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95f409e9c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f409f510 con 0x7f95f409e9c0 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f40a1180 con 0x7f95f409fda0 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f40a1460 con 0x7f95f4131980 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95fad09640 10 monclient(hunting): authenticate will time out at 976.859224s 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95f409fda0 0x7f95f4132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95f409fda0 0x7f95f4132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52064/0 (socket says 172.21.15.116:52064) 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59580/0 (socket says 172.21.15.116:59580) 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 1 -- 172.21.15.116:0/1125962300 learned_addr learned my addr 172.21.15.116:0/1125962300 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 10 monclient(hunting): get_auth_request con 0x7f95f409fda0 auth_method 0 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): get_auth_request con 0x7f95f4131980 auth_method 0 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_done global_id 64154 payload 274 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 1 -- 172.21.15.116:0/1125962300 >> v2:172.21.15.116:3300/0 conn(0x7f95f409fda0 msgr2=0x7f95f4132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 1 --2- 172.21.15.116:0/1125962300 >> v2:172.21.15.116:3300/0 conn(0x7f95f409fda0 0x7f95f4132290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 1 -- 172.21.15.116:0/1125962300 >> v1:172.21.15.144:6789/0 conn(0x7f95f409e9c0 legacy=0x7f95f409ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 1 monclient: found mon.noname-b 2024-09-18T12:29:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:09.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f8aa5640 1 -- 172.21.15.116:0/1125962300 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f4139cd0 con 0x7f95f4131980 2024-09-18T12:29:09.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.538+0000 7f95f3fff640 1 --2- 172.21.15.116:0/1125962300 >> v2:172.21.15.116:3300/0 conn(0x7f95f409fda0 0x7f95f4132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f8aa5640 1 -- 172.21.15.116:0/1125962300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 msgr2=0x7f95f4131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f8aa5640 1 --2- 172.21.15.116:0/1125962300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4131d50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009850 tx=0x7f95e80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c7e6d1b6e6201ec server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 1 -- 172.21.15.116:0/1125962300 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e802add0 con 0x7f95f4131980 2024-09-18T12:29:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: dump: 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 1 -- 172.21.15.116:0/1125962300 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e8032440 con 0x7f95f4131980 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95fad09640 5 monclient: authenticate success, global_id 64154 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95fad09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 1 -- 172.21.15.116:0/1125962300 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e80327e0 con 0x7f95f4131980 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95f37fe640 10 monclient: dump: 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95fad09640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.541+0000 7f95fad09640 4 set_mon_vals no callback set 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 10 monclient: shutdown 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 -- 172.21.15.116:0/1125962300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 msgr2=0x7f95f4131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 --2- 172.21.15.116:0/1125962300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4131d50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009850 tx=0x7f95e80090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 -- 172.21.15.116:0/1125962300 shutdown_connections 2024-09-18T12:29:09.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 --2- 172.21.15.116:0/1125962300 >> v2:172.21.15.116:3300/0 conn(0x7f95f409fda0 0x7f95f4132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 --2- 172.21.15.116:0/1125962300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4131d50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 -- 172.21.15.116:0/1125962300 >> 172.21.15.116:0/1125962300 conn(0x7f95f4067bd0 msgr2=0x7f95f40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 -- 172.21.15.116:0/1125962300 shutdown_connections 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 -- 172.21.15.116:0/1125962300 wait complete. 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 10 monclient: monmap: 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.542+0000 7f95fad09640 1 Processor -- start 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 1 -- start start 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient: init 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f409e9c0 0x7f95f4137350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): picked mon.noname-b con 0x7f95f409e9c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f409fda0 0x7f95f4133820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): picked mon.noname-a con 0x7f95f409fda0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): picked mon.noname-c con 0x7f95f4131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41348c0 con 0x7f95f409fda0 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f409ddc0 con 0x7f95f409e9c0 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41342a0 con 0x7f95f4131980 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95fad09640 10 monclient(hunting): authenticate will time out at 976.864609s 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f409fda0 0x7f95f4133820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f409fda0 0x7f95f4133820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52078/0 (socket says 172.21.15.116:52078) 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 -- 172.21.15.116:0/4064083064 learned_addr learned my addr 172.21.15.116:0/4064083064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): get_auth_request con 0x7f95f409fda0 auth_method 0 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 10 monclient(hunting): get_auth_request con 0x7f95f4131980 auth_method 0 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f409e9c0 0x7f95f4137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f92a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 10 monclient(hunting): get_auth_request con 0x7f95f409e9c0 auth_method 0 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient(hunting): handle_auth_done global_id 54438 payload 824 2024-09-18T12:29:09.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 msgr2=0x7f95f4133d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4133d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f409e9c0 msgr2=0x7f95f4137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f409e9c0 0x7f95f4137350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 monclient: found mon.noname-a 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.543+0000 7f95f3fff640 1 -- 172.21.15.116:0/4064083064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e0009450 con 0x7f95f409fda0 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.544+0000 7f95f92a6640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f95f4131980 0x7f95f4133d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f3fff640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f409fda0 msgr2=0x7f95f4133820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f3fff640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f409fda0 0x7f95f4133820 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f95e000ec80 tx=0x7f95e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0009730 con 0x7f95f409fda0 2024-09-18T12:29:09.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: dump: 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e000eec0 con 0x7f95f409fda0 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95fad09640 5 monclient: authenticate success, global_id 54438 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0013470 con 0x7f95f409fda0 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95fad09640 10 monclient: _renew_subs 2024-09-18T12:29:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95fad09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e8009390 con 0x7f95f409fda0 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f0ff9640 10 monclient: dump: 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95f2ffd640 4 set_mon_vals no callback set 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95fad09640 10 monclient: _renew_subs 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95fad09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.546+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f41332e0 con 0x7f95f409fda0 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f95e00137e0 con 0x7f95f409fda0 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95fad09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95fad09640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 0x7f95f4141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95fad09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4065e00 con 0x7f95f4140f40 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f0ff9640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95d8038490 0x7f95d803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f95e004c2d0 con 0x7f95f409fda0 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 0x7f95f4141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient: get_auth_request con 0x7f95f4140f40 auth_method 0 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f92a6640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95d8038490 0x7f95d803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f92a6640 10 monclient: get_auth_request con 0x7f95d8038490 auth_method 0 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_done global_id 5191 payload 824 2024-09-18T12:29:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.547+0000 7f95f8aa5640 1 -- 172.21.15.116:0/4064083064 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95f4136560 con 0x7f95f4140f40 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f92a6640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95d8038490 0x7f95d803a940 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95ec007fd0 tx=0x7f95ec0025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f8aa5640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 msgr2=0x7f95f4141310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f8aa5640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 0x7f95f4141310 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f95e8034000 tx=0x7f95e8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8031cb0 con 0x7f95f4140f40 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 10 monclient: dump: 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f95e8031e10 con 0x7f95f4140f40 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 msgr2=0x7f95f4141310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 0x7f95f4141310 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f95e8034000 tx=0x7f95e8032000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 msgr2=0x7f95f4141310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.550+0000 7f95f0ff9640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4140f40 0x7f95f4141310 secure :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f95e8034000 tx=0x7f95e8032000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95fad09640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95fad09640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 0x7f95f4098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95fad09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f404bf60 con 0x7f95f4141850 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 0x7f95f4098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient: get_auth_request con 0x7f95f4141850 auth_method 0 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 10 monclient(hunting): handle_auth_done global_id 5194 payload 824 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.556+0000 7f95f8aa5640 1 -- 172.21.15.116:0/4064083064 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f95f4137b90 con 0x7f95f4141850 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.558+0000 7f95f8aa5640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 msgr2=0x7f95f4098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.558+0000 7f95f8aa5640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 0x7f95f4098a70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f95f4136a90 tx=0x7f95e803e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e803dbc0 con 0x7f95f4141850 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 10 monclient: dump: 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f95e803dd20 con 0x7f95f4141850 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 msgr2=0x7f95f4098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 0x7f95f4098a70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f95f4136a90 tx=0x7f95e803e000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 msgr2=0x7f95f4098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f0ff9640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f95f4141850 0x7f95f4098a70 secure :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f95f4136a90 tx=0x7f95e803e000 comp rx=0 tx=0).stop 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 44, 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:09.573 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 7, 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.574 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.575 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:09.576 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:09.577 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:09.578 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.581 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:09.582 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:09.583 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.584 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:29:09.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:09.586 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95d8038490 msgr2=0x7f95d803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95d8038490 0x7f95d803a940 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f95ec007fd0 tx=0x7f95ec0025b0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 10 monclient: shutdown 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f409fda0 msgr2=0x7f95f4133820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 --2- 172.21.15.116:0/4064083064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f409fda0 0x7f95f4133820 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f95e000ec80 tx=0x7f95e00091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f92a6640 1 -- 172.21.15.116:0/4064083064 reap_dead start 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95f92a6640 1 -- 172.21.15.116:0/4064083064 reap_dead start 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 shutdown_connections 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 >> 172.21.15.116:0/4064083064 conn(0x7f95f4067bd0 msgr2=0x7f95f409e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 shutdown_connections 2024-09-18T12:29:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:09.559+0000 7f95fad09640 1 -- 172.21.15.116:0/4064083064 wait complete. 2024-09-18T12:29:09.589 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:29:09.589 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:29:09.589 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-18T12:29:09.589 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:29:09.589 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:29:09.589 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:29:09.590 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:29:09.599 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:29:09.599 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T12:29:09.600 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T12:29:09.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:29:09.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.665+0000 7f617f71d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:09.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.665+0000 7f617f71d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:09.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.665+0000 7f617f71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.665+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f6178064a70 con 0x7f6178136e70 2024-09-18T12:29:09.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.665+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f6170016070 con 0x7f6178136e70 2024-09-18T12:29:09.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.665+0000 7f616d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:09.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.665+0000 7f616d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:09.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.756+0000 7f617f71d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:09.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.756+0000 7f617f71d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:09.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.756+0000 7f617f71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.756+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f6178137720 con 0x7f6178136e70 2024-09-18T12:29:09.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.757+0000 7f616d7fa640 1 -- 172.21.15.144:0/2476130434 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f6170011a80 con 0x7f6178136e70 2024-09-18T12:29:09.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.757+0000 7f616d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:09.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.757+0000 7f616d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:09.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.758+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f615c038490 msgr2=0x7f615c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.758+0000 7f617f71d640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f615c038490 0x7f615c03a940 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6174007fd0 tx=0x7f6174007f00 comp rx=0 tx=0).stop 2024-09-18T12:29:09.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.758+0000 7f617f71d640 10 monclient: shutdown 2024-09-18T12:29:09.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.758+0000 7f617f71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.758+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 msgr2=0x7f6178097f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.758+0000 7f617f71d640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 0x7f6178097f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f617000c890 tx=0x7f6170002640 comp rx=0 tx=0).stop 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 shutdown_connections 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f615c038490 0x7f615c03a940 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6178136e70 0x7f6178097f60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6178131f10 0x7f6178097a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 --2- 172.21.15.144:0/2476130434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6178131b40 0x7f617812eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 >> 172.21.15.144:0/2476130434 conn(0x7f61780a5100 msgr2=0x7f61781313e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 shutdown_connections 2024-09-18T12:29:09.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.759+0000 7f617f71d640 1 -- 172.21.15.144:0/2476130434 wait complete. 2024-09-18T12:29:09.769 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm osd.7 host1 2024-09-18T12:29:09.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.841+0000 7f2587500640 10 monclient: get_monmap_and_config 2024-09-18T12:29:09.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.841+0000 7f2587500640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.841+0000 7f2587500640 10 monclient: monmap: 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:09.842214+0000 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:09.842214+0000 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.841+0000 7f2587500640 1 Processor -- start 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 1 -- start start 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient: init 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f258009ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f258009f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2580069d10 0x7f258013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient(hunting): picked mon.noname-b con 0x7f2580069d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2580067420 con 0x7f258009ec50 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258009fdc0 con 0x7f258009f020 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258006a170 con 0x7f2580069d10 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2587500640 10 monclient(hunting): authenticate will time out at 985.411587s 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f258529c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f258009ec50 legacy=0x7f2580131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2584a9b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f258009f020 0x7f2580136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:50474/0 (socket says 172.21.15.144:50474) 2024-09-18T12:29:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.842+0000 7f2584a9b640 1 -- 172.21.15.144:0/4262176637 learned_addr learned my addr 172.21.15.144:0/4262176637 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.844+0000 7f2584a9b640 1 -- 172.21.15.144:0/4262176637 >> v1:172.21.15.144:6789/0 conn(0x7f258009f020 legacy=0x7f2580136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685591686 0 0) 0x7f258009fdc0 con 0x7f258009f020 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): my global_id is 5197 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f256c0034e0 con 0x7f258009f020 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1885982499 0 0) 0x7f256c0034e0 con 0x7f258009f020 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2580069d10 msgr2=0x7f258013a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2580069d10 0x7f258013a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 >> v1:172.21.15.116:6789/0 conn(0x7f258009ec50 legacy=0x7f2580131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25800a0510 con 0x7f258009f020 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f25700017a0 con 0x7f258009f020 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2587500640 5 monclient: authenticate success, global_id 5197 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2587500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:09.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: dump: 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 >> v1:172.21.15.144:6789/0 conn(0x7f258009f020 legacy=0x7f2580136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256c003a50 0x7f256c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): picked mon.b con 0x7f256c003a50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c004780 0x7f256c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): picked mon.a con 0x7f256c004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2580069d10 0x7f256c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): picked mon.c con 0x7f2580069d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0055e0 con 0x7f256c004780 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c00bf10 con 0x7f256c003a50 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c00c0a0 con 0x7f2580069d10 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2577fff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2587500640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2584a9b640 1 -- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c004780 msgr2=0x7f256c004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f2584a9b640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c004780 0x7f256c004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256c003a50 0x7f256c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): get_auth_request con 0x7f256c003a50 auth_method 0 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient(hunting): handle_auth_done global_id 5197 payload 293 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 1 -- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2580069d10 msgr2=0x7f256c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2580069d10 0x7f256c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 1 -- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c004780 msgr2=0x7f256c004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c004780 0x7f256c004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 1 monclient: found mon.b 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 1 -- 172.21.15.144:0/4262176637 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c0034e0 con 0x7f256c003a50 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.845+0000 7f258529c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f258529c640 1 -- 172.21.15.144:0/4262176637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256c003a50 msgr2=0x7f256c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f258529c640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256c003a50 0x7f256c003e40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f257c0099c0 tx=0x7f257c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98fe991681f12c7b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c003b80 con 0x7f256c003a50 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 10 monclient: dump: 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2587500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c003d20 con 0x7f256c003a50 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2587500640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:09.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 1 -- 172.21.15.144:0/4262176637 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c02ab80 con 0x7f256c003a50 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2587500640 4 set_mon_vals no callback set 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.848+0000 7f2577fff640 10 monclient: dump: 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.861 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 10 monclient: shutdown 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 1 -- 172.21.15.144:0/4262176637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256c003a50 msgr2=0x7f256c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 1 --2- 172.21.15.144:0/4262176637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f256c003a50 0x7f256c003e40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f257c0099c0 tx=0x7f257c009060 comp rx=0 tx=0).stop 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2585a9d640 1 -- 172.21.15.144:0/4262176637 reap_dead start 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 1 -- 172.21.15.144:0/4262176637 shutdown_connections 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 1 -- 172.21.15.144:0/4262176637 >> 172.21.15.144:0/4262176637 conn(0x7f25800a5d40 msgr2=0x7f25800a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 1 -- 172.21.15.144:0/4262176637 shutdown_connections 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 1 -- 172.21.15.144:0/4262176637 wait complete. 2024-09-18T12:29:09.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 10 monclient: build_initial_monmap 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.849+0000 7f2587500640 10 monclient: monmap: 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 Processor -- start 2024-09-18T12:29:09.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 -- start start 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient: init 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c003a50 0x7f258009a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): picked mon.noname-a con 0x7f256c003a50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f258011bc70 0x7f2580130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): picked mon.noname-c con 0x7f258011bc70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 0x7f2580140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): picked mon.noname-b con 0x7f258011cc60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258009e140 con 0x7f256c003a50 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258011ded0 con 0x7f258011cc60 2024-09-18T12:29:09.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258011e1b0 con 0x7f258011bc70 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2587500640 10 monclient(hunting): authenticate will time out at 985.419676s 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f258529c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c003a50 msgr2=0x7f258009a970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 0x7f2580140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f258529c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c003a50 0x7f258009a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 0x7f2580140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42914/0 (socket says 172.21.15.144:42914) 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 1 -- 172.21.15.144:0/4215173298 learned_addr learned my addr 172.21.15.144:0/4215173298 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 10 monclient(hunting): get_auth_request con 0x7f258011cc60 auth_method 0 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:09.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.850+0000 7f2585a9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 10 monclient(hunting): handle_auth_done global_id 5200 payload 824 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 1 -- 172.21.15.144:0/4215173298 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f258011bc70 msgr2=0x7f2580130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f258011bc70 0x7f2580130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 1 -- 172.21.15.144:0/4215173298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c003a50 msgr2=0x7f258009a970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c003a50 0x7f258009a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 10 monclient: _finish_hunting 0 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 1 monclient: found mon.noname-b 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.851+0000 7f2585a9d640 1 -- 172.21.15.144:0/4215173298 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258011dbc0 con 0x7f258011cc60 2024-09-18T12:29:09.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2585a9d640 1 -- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 msgr2=0x7f2580140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:09.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2585a9d640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 0x7f2580140dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f25800a1b50 tx=0x7f2578002880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257800ce80 con 0x7f258011cc60 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: dump: 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.869 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2587500640 5 monclient: authenticate success, global_id 5200 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25780074f0 con 0x7f258011cc60 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25780078b0 con 0x7f258011cc60 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2587500640 10 monclient: _renew_subs 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2587500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258011c730 con 0x7f258011cc60 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25777fe640 4 set_mon_vals no callback set 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f25757fa640 10 monclient: dump: 2024-09-18T12:29:09.870 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2587500640 10 monclient: _renew_subs 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2587500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.853+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2580131180 con 0x7f258011cc60 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f257801b3b0 con 0x7f258011cc60 2024-09-18T12:29:09.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f2587500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f2587500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f2587500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258009f5e0 con 0x7f258011cc60 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f25757fa640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2564038440 0x7f256403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f257804b1f0 con 0x7f258011cc60 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f258529c640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2564038440 0x7f256403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.854+0000 7f258529c640 10 monclient: get_auth_request con 0x7f2564038440 auth_method 0 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.857+0000 7f258529c640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2564038440 0x7f256403a8f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f257c032000 tx=0x7f257c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.858+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f258009f5e0 con 0x7f258011cc60 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.858+0000 7f25757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:09.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:09.858+0000 7f25757fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:10.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:10.017+0000 7f2587500640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-18T12:29:10.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:10.017+0000 7f2587500640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-18T12:29:10.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:10.017+0000 7f2587500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:10.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:10.017+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f25800008d0 con 0x7f258011cc60 2024-09-18T12:29:12.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:12.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:12.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:12.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:12.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:12.851+0000 7f256bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:13.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:13.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:13.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:13.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:14.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:14.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:14.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:14.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:15.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:15.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:15.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:15.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:16.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:16.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:16.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:16.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:17.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:17.850+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:17.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:17.850+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:18.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:18.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:18.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:18.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:19.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:19.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:19.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:19.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:20.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:20.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:20.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:20.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:21.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:21.851+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:21.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:21.851+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:22.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:22.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:22.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:22.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:22.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:22.852+0000 7f256bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:23.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:23.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:23.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:23.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:24.601 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T12:29:24.601 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T12:29:24.601 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:29:24.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.703+0000 7fda12f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.703+0000 7fda12f9d640 10 monclient: build_initial_monmap 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.703+0000 7fda12f9d640 10 monclient: monmap: 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:24.704744+0000 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:24.704744+0000 2024-09-18T12:29:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:24.706 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.703+0000 7fda12f9d640 1 Processor -- start 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.703+0000 7fda12f9d640 1 -- start start 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.703+0000 7fda12f9d640 10 monclient: init 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 0x7fda0c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fda0c131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fda0c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda0c09f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c09faf0 con 0x7fda0c09f220 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c09fd30 con 0x7fda0c131ab0 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0a1110 con 0x7fda0c09e910 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda12f9d640 10 monclient(hunting): authenticate will time out at 992.026509s 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda1153a640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fda0c09f220 legacy=0x7fda0c068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda10d39640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 msgr2=0x7fda0c131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda10d39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 0x7fda0c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43700/0 (socket says 172.21.15.116:43700) 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 learned_addr learned my addr 172.21.15.116:0/2407946895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 10 monclient(hunting): get_auth_request con 0x7fda0c09e910 auth_method 0 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:29:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.704+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:24.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:24.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:24.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda1153a640 1 -- 172.21.15.116:0/2407946895 >> v1:172.21.15.116:6789/0 conn(0x7fda0c09f220 legacy=0x7fda0c068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:24.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 msgr2=0x7fda0c131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:24.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 0x7fda0c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:29:24.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 10 monclient(hunting): get_auth_request con 0x7fda0c09e910 auth_method 0 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:24.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:24.905+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 msgr2=0x7fda0c131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda1153a640 1 -- 172.21.15.116:0/2407946895 >> v1:172.21.15.116:6789/0 conn(0x7fda0c09f220 legacy=0x7fda0c068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 0x7fda0c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 10 monclient(hunting): get_auth_request con 0x7fda0c09e910 auth_method 0 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:25.306+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:29:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:25.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:25.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 msgr2=0x7fda0c131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda1153a640 1 -- 172.21.15.116:0/2407946895 >> v1:172.21.15.116:6789/0 conn(0x7fda0c09f220 legacy=0x7fda0c068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 0x7fda0c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 10 monclient(hunting): get_auth_request con 0x7fda0c09e910 auth_method 0 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:26.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:26.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:26.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:26.107+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:29:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:26.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:26.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 10 monclient(hunting): tick 2024-09-18T12:29:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 monclient(hunting): continuing hunt 2024-09-18T12:29:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:29:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c09ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 msgr2=0x7fda0c131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.116:3300/0 conn(0x7fda0c131ab0 0x7fda0c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 -- 172.21.15.116:0/2407946895 >> v1:172.21.15.116:6789/0 conn(0x7fda0c09f220 legacy=0x7fda0c068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 10 monclient: _add_conns ranks=[4,1] 2024-09-18T12:29:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fd9f0003660 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9f0006120 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 -- 172.21.15.116:0/2407946895 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c0a13f0 con 0x7fd9f0006120 2024-09-18T12:29:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 1 -- 172.21.15.116:0/2407946895 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c09fd30 con 0x7fd9f0003660 2024-09-18T12:29:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda0a7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 10 monclient(hunting): get_auth_request con 0x7fd9f0003660 auth_method 0 2024-09-18T12:29:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:27.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:27.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:27.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:27.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:27.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.704+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:27.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.705+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.707+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> v1:172.21.15.144:6789/0 conn(0x7fd9f0006120 legacy=0x7fd9f00085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:27.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:27.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:27.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:27.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 10 monclient(hunting): get_auth_request con 0x7fd9f0003660 auth_method 0 2024-09-18T12:29:27.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:27.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:27.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:27.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:27.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:27.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:27.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:27.906+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:29:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:28.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 10 monclient(hunting): get_auth_request con 0x7fd9f0003660 auth_method 0 2024-09-18T12:29:28.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:28.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:28.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:28.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:28.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:28.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:28.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:28.307+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:29:28.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:28.852+0000 7f256bfff640 10 monclient: tick 2024-09-18T12:29:28.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:28.852+0000 7f256bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 10 monclient(hunting): get_auth_request con 0x7fd9f0003660 auth_method 0 2024-09-18T12:29:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:29:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:29:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.109+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:29:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.609+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79288357 0 0) 0x7fda0c0a13f0 con 0x7fd9f0006120 2024-09-18T12:29:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.609+0000 7fda0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.609+0000 7fda0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.609+0000 7fda0b7fe640 10 monclient(hunting): my global_id is 5203 2024-09-18T12:29:29.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.609+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9fc0034e0 con 0x7fd9f0006120 2024-09-18T12:29:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1232949143 0 0) 0x7fd9fc0034e0 con 0x7fd9f0006120 2024-09-18T12:29:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 msgr2=0x7fd9f0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:29.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 1 --2- 172.21.15.116:0/2407946895 >> v2:172.21.15.144:3300/0 conn(0x7fd9f0003660 0x7fd9f0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c09faf0 con 0x7fd9f0006120 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.610+0000 7fda0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fd9f800c680 con 0x7fd9f0006120 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda12f9d640 5 monclient: authenticate success, global_id 5203 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda12f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient: dump: 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:29.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 >> v1:172.21.15.144:6789/0 conn(0x7fd9f0006120 legacy=0x7fd9f00085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 1 -- 172.21.15.116:0/2407946895 reap_dead start 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fd9fc0039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): picked mon.c con 0x7fda0c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fd9fc004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): picked mon.a con 0x7fd9fc004030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0bfff640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 msgr2=0x7fd9fc004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fd9fc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0bfff640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fd9fc004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): picked mon.b con 0x7fd9fc004d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c09fd30 con 0x7fd9fc004030 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc00bc80 con 0x7fd9fc004d10 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc00be40 con 0x7fda0c09e910 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda0b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda12f9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fd9fc0039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): get_auth_request con 0x7fda0c09e910 auth_method 0 2024-09-18T12:29:29.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fd9fc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 10 monclient(hunting): get_auth_request con 0x7fd9fc004d10 auth_method 0 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda1153a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient(hunting): handle_auth_done global_id 5203 payload 293 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 msgr2=0x7fd9fc005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fd9fc005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 msgr2=0x7fd9fc004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fd9fc004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient: _finish_hunting 0 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 1 monclient: found mon.c 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9fc0034e0 con 0x7fda0c09e910 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 10 monclient: _finish_auth 0 2024-09-18T12:29:29.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.611+0000 7fda10d39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda10d39640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fd9fc0039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda10d39640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fd9fc0039d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fda00009ae0 tx=0x7fda00003d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ccdf181765e76 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0002bd50 con 0x7fda0c09e910 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 10 monclient: dump: 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.627 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda12f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0002f070 con 0x7fda0c09e910 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 1 -- 172.21.15.116:0/2407946895 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00035650 con 0x7fda0c09e910 2024-09-18T12:29:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda0b7fe640 10 monclient: dump: 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda12f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.614+0000 7fda12f9d640 4 set_mon_vals no callback set 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 10 monclient: shutdown 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 -- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fd9fc0039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fd9fc0039d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fda00009ae0 tx=0x7fda00003d20 comp rx=0 tx=0).stop 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 -- 172.21.15.116:0/2407946895 shutdown_connections 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fd9fc005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fd9fc0039d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 --2- 172.21.15.116:0/2407946895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fd9fc004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 -- 172.21.15.116:0/2407946895 >> 172.21.15.116:0/2407946895 conn(0x7fda0c067d30 msgr2=0x7fda0c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 -- 172.21.15.116:0/2407946895 shutdown_connections 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.615+0000 7fda12f9d640 1 -- 172.21.15.116:0/2407946895 wait complete. 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.616+0000 7fda12f9d640 10 monclient: build_initial_monmap 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.616+0000 7fda12f9d640 10 monclient: monmap: 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.616+0000 7fda12f9d640 1 Processor -- start 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 1 -- start start 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient: init 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fda0c11de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9fc004030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fda0c11e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd9fc004d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c11e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fda0c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c11f970 con 0x7fd9fc004030 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c09bde0 con 0x7fd9fc004d10 2024-09-18T12:29:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c09ddc0 con 0x7fda0c09e910 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda12f9d640 10 monclient(hunting): authenticate will time out at 996.939918s 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda10d39640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 msgr2=0x7fda0c11de30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.617+0000 7fda10d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fda0c11de30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c11e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c11e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48130/0 (socket says 172.21.15.116:48130) 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 -- 172.21.15.116:0/40429045 learned_addr learned my addr 172.21.15.116:0/40429045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): get_auth_request con 0x7fda0c09e910 auth_method 0 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fda0c11e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 10 monclient(hunting): get_auth_request con 0x7fd9fc004d10 auth_method 0 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient(hunting): handle_auth_done global_id 64160 payload 824 2024-09-18T12:29:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 -- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 msgr2=0x7fda0c11e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fda0c11e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 -- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 msgr2=0x7fda0c11de30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fda0c11de30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient: _finish_hunting 0 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 monclient: found mon.noname-c 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.618+0000 7fda1153a640 1 -- 172.21.15.116:0/40429045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c121410 con 0x7fda0c09e910 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda1153a640 1 -- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c11e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda1153a640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c11e8b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009e10 tx=0x7fda04006f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 1 -- 172.21.15.116:0/40429045 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0400f920 con 0x7fda0c09e910 2024-09-18T12:29:29.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: dump: 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.637 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: _finish_auth 0 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 1 -- 172.21.15.116:0/40429045 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0400fa80 con 0x7fda0c09e910 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda12f9d640 5 monclient: authenticate success, global_id 64160 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 1 -- 172.21.15.116:0/40429045 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0400fe40 con 0x7fda0c09e910 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda08ff9640 10 monclient: dump: 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.638 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda12f9d640 10 monclient: _renew_subs 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.621+0000 7fda12f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.621+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c131430 con 0x7fda0c09e910 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.620+0000 7fda0affd640 4 set_mon_vals no callback set 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.621+0000 7fda12f9d640 10 monclient: _renew_subs 2024-09-18T12:29:29.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.621+0000 7fda12f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.621+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c120650 con 0x7fda0c09e910 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.621+0000 7fda08ff9640 1 -- 172.21.15.116:0/40429045 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fda04015070 con 0x7fda0c09e910 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda12f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda12f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda12f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda08ff9640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9f0038ab0 0x7fd9f003af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c121810 con 0x7fda0c09e910 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda08ff9640 1 -- 172.21.15.116:0/40429045 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6173+0+0 (secure 0 0 0) 0x7fda0404bb70 con 0x7fda0c09e910 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda10d39640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9f0038ab0 0x7fd9f003af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.622+0000 7fda10d39640 10 monclient: get_auth_request con 0x7fd9f0038ab0 auth_method 0 2024-09-18T12:29:29.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.625+0000 7fda10d39640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9f0038ab0 0x7fd9f003af60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fda00036000 tx=0x7fda00037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.627+0000 7fda08ff9640 1 -- 172.21.15.116:0/40429045 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fda04016370 con 0x7fda0c09e910 2024-09-18T12:29:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.627+0000 7fda08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:29.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.627+0000 7fda08ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:29.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.666+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 removed item id 7 name 'osd.7' from crush map v83) v1 ==== 143+0+0 (secure 0 0 0) 0x7f2578017050 con 0x7f258011cc60 2024-09-18T12:29:29.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.666+0000 7f25757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-18T12:29:29.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.666+0000 7f25757fa640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-18T12:29:29.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.768+0000 7f2587500640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-18T12:29:29.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.768+0000 7f2587500640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-18T12:29:29.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.768+0000 7f2587500640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:29.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.768+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) v1 -- 0x7f2580072640 con 0x7f258011cc60 2024-09-18T12:29:29.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.769+0000 7f25757fa640 1 -- 172.21.15.144:0/4215173298 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 device 'osd.7' does not appear in the crush map v83) v1 ==== 145+0+0 (secure 0 0 0) 0x7f257800f860 con 0x7f258011cc60 2024-09-18T12:29:29.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.769+0000 7f25757fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-18T12:29:29.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.769+0000 7f25757fa640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-18T12:29:29.771 INFO:tasks.workunit.client.0.smithi144.stderr:device 'osd.7' does not appear in the crush map 2024-09-18T12:29:29.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2564038440 msgr2=0x7f256403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2564038440 0x7f256403a8f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f257c032000 tx=0x7f257c030000 comp rx=0 tx=0).stop 2024-09-18T12:29:29.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 10 monclient: shutdown 2024-09-18T12:29:29.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:29.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 msgr2=0x7f2580140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 0x7f2580140dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f25800a1b50 tx=0x7f2578002880 comp rx=0 tx=0).stop 2024-09-18T12:29:29.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 shutdown_connections 2024-09-18T12:29:29.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2564038440 0x7f256403a8f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f258011cc60 0x7f2580140dd0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f258011bc70 0x7f2580130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 --2- 172.21.15.144:0/4215173298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f256c003a50 0x7f258009a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 >> 172.21.15.144:0/4215173298 conn(0x7f25800a5d40 msgr2=0x7f25800a4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:29.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 shutdown_connections 2024-09-18T12:29:29.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.771+0000 7f2587500640 1 -- 172.21.15.144:0/4215173298 wait complete. 2024-09-18T12:29:29.782 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm host1 2024-09-18T12:29:29.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.849+0000 7fda12f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:29:29.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.849+0000 7fda12f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:29:29.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.849+0000 7fda12f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.849+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fda0c04bf60 con 0x7fda0c09e910 2024-09-18T12:29:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.849+0000 7fda08ff9640 1 -- 172.21.15.116:0/40429045 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fda04015320 con 0x7fda0c09e910 2024-09-18T12:29:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.849+0000 7fda08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:29:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.849+0000 7fda08ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:29.852 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9f0038ab0 msgr2=0x7fd9f003af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9f0038ab0 0x7fd9f003af60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fda00036000 tx=0x7fda00037000 comp rx=0 tx=0).stop 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 10 monclient: shutdown 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 msgr2=0x7fda0c11e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c11e8b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009e10 tx=0x7fda04006f90 comp rx=0 tx=0).stop 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 shutdown_connections 2024-09-18T12:29:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9f0038ab0 0x7fd9f003af60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fda0c09e910 0x7fda0c11e8b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9fc004d10 0x7fda0c11e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 --2- 172.21.15.116:0/40429045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9fc004030 0x7fda0c11de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.851+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 >> 172.21.15.116:0/40429045 conn(0x7fda0c067d30 msgr2=0x7fda0c09b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.852+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 shutdown_connections 2024-09-18T12:29:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.852+0000 7fda12f9d640 1 -- 172.21.15.116:0/40429045 wait complete. 2024-09-18T12:29:29.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.854+0000 7f36a4354640 10 monclient: get_monmap_and_config 2024-09-18T12:29:29.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.854+0000 7f36a4354640 10 monclient: build_initial_monmap 2024-09-18T12:29:29.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.854+0000 7f36a4354640 10 monclient: monmap: 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:29.855624+0000 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:29.855624+0000 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:29.856 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 1 Processor -- start 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 1 -- start start 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient: init 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f369c069a30 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c09bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient(hunting): picked mon.noname-b con 0x7f369c09baa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:29.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f369c134b70 0x7f369c136f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient(hunting): picked mon.noname-a con 0x7f369c134b70 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c09e550 con 0x7f369c069a30 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369c12ff50 con 0x7f369c134b70 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c069e00 con 0x7f369c09baa0 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.855+0000 7f36a4354640 10 monclient(hunting): authenticate will time out at 1005.424951s 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a28f1640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f369c134b70 msgr2=0x7f369c136f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a20f0640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f369c069a30 0x7f369c12f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49900/0 (socket says 172.21.15.144:49900) 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a20f0640 1 -- 172.21.15.144:0/2596142321 learned_addr learned my addr 172.21.15.144:0/2596142321 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a28f1640 1 --2- 172.21.15.144:0/2596142321 >> v2:172.21.15.116:3300/0 conn(0x7f369c134b70 0x7f369c136f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:29.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 1 --2- 172.21.15.144:0/2596142321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c09bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): get_auth_request con 0x7f369c09baa0 auth_method 0 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:29.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient(hunting): handle_auth_done global_id 64163 payload 274 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 1 -- 172.21.15.144:0/2596142321 >> v2:172.21.15.116:3300/0 conn(0x7f369c134b70 msgr2=0x7f369c136f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 1 --2- 172.21.15.144:0/2596142321 >> v2:172.21.15.116:3300/0 conn(0x7f369c134b70 0x7f369c136f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 1 -- 172.21.15.144:0/2596142321 >> v1:172.21.15.144:6789/0 conn(0x7f369c069a30 legacy=0x7f369c12f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient: _finish_hunting 0 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 1 monclient: found mon.noname-b 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.856+0000 7f36a18ef640 1 -- 172.21.15.144:0/2596142321 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c09d100 con 0x7f369c09baa0 2024-09-18T12:29:29.869 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:29:29.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:29:29.869 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:29:29.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a18ef640 1 -- 172.21.15.144:0/2596142321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 msgr2=0x7f369c09bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:29.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a18ef640 1 --2- 172.21.15.144:0/2596142321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c09bea0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f369c099060 tx=0x7f368c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45596112488392ef server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:29.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 1 -- 172.21.15.144:0/2596142321 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c02aea0 con 0x7f369c09baa0 2024-09-18T12:29:29.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: dump: 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: _finish_auth 0 2024-09-18T12:29:29.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 1 -- 172.21.15.144:0/2596142321 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c032440 con 0x7f369c09baa0 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 1 -- 172.21.15.144:0/2596142321 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c0327e0 con 0x7f369c09baa0 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a4354640 5 monclient: authenticate success, global_id 64163 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a4354640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a4354640 4 set_mon_vals no callback set 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.859+0000 7f36a10ee640 10 monclient: dump: 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 10 monclient: shutdown 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 -- 172.21.15.144:0/2596142321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 msgr2=0x7f369c09bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 --2- 172.21.15.144:0/2596142321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c09bea0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f369c099060 tx=0x7f368c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 -- 172.21.15.144:0/2596142321 shutdown_connections 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 --2- 172.21.15.144:0/2596142321 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c09bea0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 --2- 172.21.15.144:0/2596142321 >> v2:172.21.15.116:3300/0 conn(0x7f369c134b70 0x7f369c136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 -- 172.21.15.144:0/2596142321 >> 172.21.15.144:0/2596142321 conn(0x7f369c09e9f0 msgr2=0x7f369c09edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 -- 172.21.15.144:0/2596142321 shutdown_connections 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 -- 172.21.15.144:0/2596142321 wait complete. 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 10 monclient: build_initial_monmap 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 10 monclient: monmap: 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:29.874 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.860+0000 7f36a4354640 1 Processor -- start 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 1 -- start start 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient: init 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 0x7f369c133490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): picked mon.noname-b con 0x7f369c069a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c096f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): picked mon.noname-c con 0x7f369c09baa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f369c134b70 0x7f369c12e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): picked mon.noname-a con 0x7f369c134b70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:29.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c097a80 con 0x7f369c134b70 2024-09-18T12:29:29.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0a12f0 con 0x7f369c069a30 2024-09-18T12:29:29.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c11cd80 con 0x7f369c09baa0 2024-09-18T12:29:29.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:29.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a4354640 10 monclient(hunting): authenticate will time out at 1005.430447s 2024-09-18T12:29:29.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 0x7f369c133490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 0x7f369c133490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53710/0 (socket says 172.21.15.144:53710) 2024-09-18T12:29:29.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a28f1640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f369c134b70 msgr2=0x7f369c12e970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:29.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 -- 172.21.15.144:0/1313632578 learned_addr learned my addr 172.21.15.144:0/1313632578 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:29.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): get_auth_request con 0x7f369c069a30 auth_method 0 2024-09-18T12:29:29.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a28f1640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f369c134b70 0x7f369c12e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:29.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c096f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:29.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 10 monclient(hunting): get_auth_request con 0x7f369c09baa0 auth_method 0 2024-09-18T12:29:29.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a18ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:29.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient(hunting): handle_auth_done global_id 5209 payload 824 2024-09-18T12:29:29.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 -- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 msgr2=0x7f369c096f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:29.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c096f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 -- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f369c134b70 msgr2=0x7f369c12e970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:29.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f369c134b70 0x7f369c12e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient: _finish_hunting 0 2024-09-18T12:29:29.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 monclient: found mon.noname-b 2024-09-18T12:29:29.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:29.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:29.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.861+0000 7f36a20f0640 1 -- 172.21.15.144:0/1313632578 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c009390 con 0x7f369c069a30 2024-09-18T12:29:29.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.863+0000 7f36a20f0640 1 -- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 msgr2=0x7f369c133490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:29.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.863+0000 7f36a20f0640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 0x7f369c133490 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f369800c5e0 tx=0x7f3698004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:29.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698005cb0 con 0x7f369c069a30 2024-09-18T12:29:29.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:29.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: dump: 2024-09-18T12:29:29.883 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:29.883 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.883 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.883 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.884 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.884 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:29.884 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.884 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.884 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.884 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:29.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:29.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:29.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a4354640 5 monclient: authenticate success, global_id 5209 2024-09-18T12:29:29.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698005e30 con 0x7f369c069a30 2024-09-18T12:29:29.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a4354640 10 monclient: _renew_subs 2024-09-18T12:29:29.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a4354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:29.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c097e80 con 0x7f369c069a30 2024-09-18T12:29:29.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:29.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698007340 con 0x7f369c069a30 2024-09-18T12:29:29.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:29.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:29.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 10 monclient: dump: 2024-09-18T12:29:29.887 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:29.887 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.887 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.887 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:29.887 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:29.888 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:29.888 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:29.888 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:29.888 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:29.888 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:29.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a08ed640 4 set_mon_vals no callback set 2024-09-18T12:29:29.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a4354640 10 monclient: _renew_subs 2024-09-18T12:29:29.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a4354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:29.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c11cb50 con 0x7f369c069a30 2024-09-18T12:29:29.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.864+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f369801b400 con 0x7f369c069a30 2024-09-18T12:29:29.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36a4354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:29.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36a4354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:29.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36a4354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:29.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c0008d0 con 0x7f369c069a30 2024-09-18T12:29:29.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36927fc640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3680038440 0x7f368003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f369804b8a0 con 0x7f369c069a30 2024-09-18T12:29:29.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36a18ef640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3680038440 0x7f368003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.865+0000 7f36a18ef640 10 monclient: get_auth_request con 0x7f3680038440 auth_method 0 2024-09-18T12:29:29.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.867+0000 7f36a18ef640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3680038440 0x7f368003a8f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f369c09e0d0 tx=0x7f368c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:29.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.869+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f369801a960 con 0x7f369c069a30 2024-09-18T12:29:29.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.869+0000 7f36927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:29.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:29.869+0000 7f36927fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.983+0000 7fd0e5aa8640 10 monclient: get_monmap_and_config 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.983+0000 7fd0e5aa8640 10 monclient: build_initial_monmap 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.983+0000 7fd0e5aa8640 10 monclient: monmap: 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:29.984775+0000 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:29.984775+0000 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:29.986 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:29.986 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:29.986 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:29.986 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:29.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.983+0000 7fd0e5aa8640 1 Processor -- start 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.983+0000 7fd0e5aa8640 1 -- start start 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.983+0000 7fd0e5aa8640 10 monclient: init 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd0e0131ab0 0x7fd0e0131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e0131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0e009e910 0x7fd0e009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 0x7fd0e0068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e009f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e009faf0 con 0x7fd0e009e910 2024-09-18T12:29:29.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e009fd30 con 0x7fd0e0131ab0 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00a1110 con 0x7fd0e009f250 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0e5aa8640 10 monclient(hunting): authenticate will time out at 997.306384s 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0deffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fd0e009e910 msgr2=0x7fd0e009ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0deffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0e009e910 0x7fd0e009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 0x7fd0e0068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 0x7fd0e0068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48144/0 (socket says 172.21.15.116:48144) 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 -- 172.21.15.116:0/4207967101 learned_addr learned my addr 172.21.15.116:0/4207967101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): get_auth_request con 0x7fd0e009f250 auth_method 0 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 1 --2- 172.21.15.116:0/4207967101 >> v2:172.21.15.144:3300/0 conn(0x7fd0e0131ab0 0x7fd0e0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 10 monclient(hunting): get_auth_request con 0x7fd0e0131ab0 auth_method 0 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:29.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient(hunting): handle_auth_done global_id 64169 payload 274 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 -- 172.21.15.116:0/4207967101 >> v2:172.21.15.144:3300/0 conn(0x7fd0e0131ab0 msgr2=0x7fd0e0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 --2- 172.21.15.116:0/4207967101 >> v2:172.21.15.144:3300/0 conn(0x7fd0e0131ab0 0x7fd0e0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 -- 172.21.15.116:0/4207967101 >> v2:172.21.15.116:3300/0 conn(0x7fd0e009e910 msgr2=0x7fd0e009ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 --2- 172.21.15.116:0/4207967101 >> v2:172.21.15.116:3300/0 conn(0x7fd0e009e910 0x7fd0e009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 monclient: found mon.noname-b 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:29.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.984+0000 7fd0dffff640 1 -- 172.21.15.116:0/4207967101 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e00a13f0 con 0x7fd0e009f250 2024-09-18T12:29:30.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0dffff640 1 -- 172.21.15.116:0/4207967101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 msgr2=0x7fd0e0068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0dffff640 1 --2- 172.21.15.116:0/4207967101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 0x7fd0e0068ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0076760 tx=0x7fd0d80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9009313256d76f63 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 1 -- 172.21.15.116:0/4207967101 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8032440 con 0x7fd0e009f250 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: dump: 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.002 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 1 -- 172.21.15.116:0/4207967101 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d80325a0 con 0x7fd0e009f250 2024-09-18T12:29:30.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0e5aa8640 5 monclient: authenticate success, global_id 64169 2024-09-18T12:29:30.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0e5aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:30.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:30.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0e5aa8640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:30.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 1 -- 172.21.15.116:0/4207967101 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8032940 con 0x7fd0e009f250 2024-09-18T12:29:30.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0e5aa8640 4 set_mon_vals no callback set 2024-09-18T12:29:30.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.987+0000 7fd0de7fc640 10 monclient: dump: 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.005 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 10 monclient: shutdown 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/4207967101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 msgr2=0x7fd0e0068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/4207967101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 0x7fd0e0068ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0076760 tx=0x7fd0d80091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/4207967101 shutdown_connections 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/4207967101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e009f250 0x7fd0e0068ce0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/4207967101 >> v2:172.21.15.116:3300/0 conn(0x7fd0e009e910 0x7fd0e009ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/4207967101 >> v2:172.21.15.144:3300/0 conn(0x7fd0e0131ab0 0x7fd0e0131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/4207967101 >> 172.21.15.116:0/4207967101 conn(0x7fd0e0067d30 msgr2=0x7fd0e00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/4207967101 shutdown_connections 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/4207967101 wait complete. 2024-09-18T12:29:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 10 monclient: build_initial_monmap 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.988+0000 7fd0e5aa8640 10 monclient: monmap: 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.008 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 Processor -- start 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 -- start start 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient: init 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:30.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e009e910 0x7fd0e012ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e009e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0e009f250 0x7fd0e0140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e009f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0131ab0 0x7fd0e0099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e0131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e009bde0 con 0x7fd0e009e910 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e0130aa0 con 0x7fd0e009f250 2024-09-18T12:29:30.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0df7fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e009e910 msgr2=0x7fd0e012ffd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e011b6b0 con 0x7fd0e0131ab0 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0e5aa8640 10 monclient(hunting): authenticate will time out at 997.311791s 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e009e910 0x7fd0e012ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0131ab0 0x7fd0e0099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0131ab0 0x7fd0e0099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48150/0 (socket says 172.21.15.116:48150) 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 1 -- 172.21.15.116:0/102047940 learned_addr learned my addr 172.21.15.116:0/102047940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 10 monclient(hunting): get_auth_request con 0x7fd0e0131ab0 auth_method 0 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0deffd640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0e009f250 0x7fd0e0140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0deffd640 10 monclient(hunting): get_auth_request con 0x7fd0e009f250 auth_method 0 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0deffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.989+0000 7fd0deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 10 monclient(hunting): handle_auth_done global_id 64172 payload 824 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0e009f250 msgr2=0x7fd0e0140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0e009f250 0x7fd0e0140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e009e910 msgr2=0x7fd0e012ffd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0e009e910 0x7fd0e012ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 1 monclient: found mon.noname-c 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.990+0000 7fd0dffff640 1 -- 172.21.15.116:0/102047940 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8009450 con 0x7fd0e0131ab0 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0dffff640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0131ab0 msgr2=0x7fd0e0099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0dffff640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0131ab0 0x7fd0e0099ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8035040 tx=0x7fd0d8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8032440 con 0x7fd0e0131ab0 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: dump: 2024-09-18T12:29:30.013 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d8031c70 con 0x7fd0e0131ab0 2024-09-18T12:29:30.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0e5aa8640 5 monclient: authenticate success, global_id 64172 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0e5aa8640 10 monclient: _renew_subs 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0e5aa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e011d0f0 con 0x7fd0e0131ab0 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d803d440 con 0x7fd0e0131ab0 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0c77fe640 10 monclient: dump: 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.015 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0e5aa8640 10 monclient: _renew_subs 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0e5aa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0ddffb640 4 set_mon_vals no callback set 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.992+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e011c200 con 0x7fd0e0131ab0 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0e5aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 0x7fd0e009a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e0063740 con 0x7fd0e0130510 2024-09-18T12:29:30.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd0d8039040 con 0x7fd0e0131ab0 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 0x7fd0e009a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient: get_auth_request con 0x7fd0e0130510 auth_method 0 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0c77fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0cc038440 0x7fd0cc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fd0d806bd60 con 0x7fd0e0131ab0 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_done global_id 64175 payload 824 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0df7fe640 1 -- 172.21.15.116:0/102047940 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0e004bf60 con 0x7fd0e0130510 2024-09-18T12:29:30.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0deffd640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0cc038440 0x7fd0cc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.993+0000 7fd0deffd640 10 monclient: get_auth_request con 0x7fd0cc038440 auth_method 0 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.995+0000 7fd0df7fe640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 msgr2=0x7fd0e009a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.995+0000 7fd0df7fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 0x7fd0e009a410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000efa0 tx=0x7fd0d000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.995+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000ce90 con 0x7fd0e0130510 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 10 monclient: dump: 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.018 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0deffd640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0cc038440 0x7fd0cc03a8f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd0e009e770 tx=0x7fd0d40090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd0d000f040 con 0x7fd0e0130510 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 msgr2=0x7fd0e009a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 0x7fd0e009a410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000efa0 tx=0x7fd0d000b040 comp rx=0 tx=0).stop 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 msgr2=0x7fd0e009a410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:30.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:29.996+0000 7fd0c77fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0130510 0x7fd0e009a410 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000efa0 tx=0x7fd0d000b040 comp rx=0 tx=0).stop 2024-09-18T12:29:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.001+0000 7fd0e5aa8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.001+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 0x7fd0e009a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.001+0000 7fd0e5aa8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.001+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e011d470 con 0x7fd0e01413e0 2024-09-18T12:29:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 0x7fd0e009a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient: get_auth_request con 0x7fd0e01413e0 auth_method 0 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 10 monclient(hunting): handle_auth_done global_id 64178 payload 824 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.002+0000 7fd0df7fe640 1 -- 172.21.15.116:0/102047940 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd0e0066280 con 0x7fd0e01413e0 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0df7fe640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 msgr2=0x7fd0e009a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0df7fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 0x7fd0e009a950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000efa0 tx=0x7fd0d001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d001ed30 con 0x7fd0e01413e0 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 10 monclient: dump: 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fd0d0024070 con 0x7fd0e01413e0 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:30.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 msgr2=0x7fd0e009a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 0x7fd0e009a950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000efa0 tx=0x7fd0d001a000 comp rx=0 tx=0).stop 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 msgr2=0x7fd0e009a950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.004+0000 7fd0c77fe640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e01413e0 0x7fd0e009a950 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000efa0 tx=0x7fd0d001a000 comp rx=0 tx=0).stop 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 46, 2024-09-18T12:29:30.024 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:30.025 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:30.032 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:30.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:30.036 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:30.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.029+0000 7f36a4354640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T12:29:30.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.029+0000 7f36a4354640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T12:29:30.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.029+0000 7f36a4354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:30.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.029+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f369c064b00 con 0x7f369c069a30 2024-09-18T12:29:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0cc038440 msgr2=0x7fd0cc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd0cc038440 0x7fd0cc03a8f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd0e009e770 tx=0x7fd0d40090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 10 monclient: shutdown 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0131ab0 msgr2=0x7fd0e0099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 --2- 172.21.15.116:0/102047940 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0e0131ab0 0x7fd0e0099ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8035040 tx=0x7fd0d8032000 comp rx=0 tx=0).stop 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0dffff640 1 -- 172.21.15.116:0/102047940 reap_dead start 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0dffff640 1 -- 172.21.15.116:0/102047940 reap_dead start 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 shutdown_connections 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 >> 172.21.15.116:0/102047940 conn(0x7fd0e0067d30 msgr2=0x7fd0e009c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 shutdown_connections 2024-09-18T12:29:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.005+0000 7fd0e5aa8640 1 -- 172.21.15.116:0/102047940 wait complete. 2024-09-18T12:29:30.039 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:29:30.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:29:30.040 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.131+0000 7fb44435d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.131+0000 7fb44435d640 10 monclient: build_initial_monmap 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.131+0000 7fb44435d640 10 monclient: monmap: 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:29:30.133061+0000 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:29:30.133061+0000 2024-09-18T12:29:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:30.134 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.134 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:30.134 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:30.134 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:30.134 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:30.134 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:30.134 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.131+0000 7fb44435d640 1 Processor -- start 2024-09-18T12:29:30.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 1 -- start start 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient: init 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 0x7fb43c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient(hunting): picked mon.noname-b con 0x7fb43c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb43c09e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb43c136bb0 0x7fb43c138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient(hunting): picked mon.noname-c con 0x7fb43c136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43c09faf0 con 0x7fb43c09e910 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43c09fd30 con 0x7fb43c136bb0 2024-09-18T12:29:30.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c0a1110 con 0x7fb43c131ab0 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb44435d640 10 monclient(hunting): authenticate will time out at 997.454723s 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 0x7fb43c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 0x7fb43c131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48166/0 (socket says 172.21.15.116:48166) 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 1 -- 172.21.15.116:0/2061850879 learned_addr learned my addr 172.21.15.116:0/2061850879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 10 monclient(hunting): get_auth_request con 0x7fb43c131ab0 auth_method 0 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.132+0000 7fb4420f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4428fa640 1 --2- 172.21.15.116:0/2061850879 >> v2:172.21.15.144:3300/0 conn(0x7fb43c136bb0 0x7fb43c138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 10 monclient(hunting): handle_auth_done global_id 64181 payload 274 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 1 -- 172.21.15.116:0/2061850879 >> v2:172.21.15.144:3300/0 conn(0x7fb43c136bb0 msgr2=0x7fb43c138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 1 --2- 172.21.15.116:0/2061850879 >> v2:172.21.15.144:3300/0 conn(0x7fb43c136bb0 0x7fb43c138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 1 -- 172.21.15.116:0/2061850879 >> v1:172.21.15.144:6789/0 conn(0x7fb43c09e910 legacy=0x7fb43c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 10 monclient: _finish_hunting 0 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 1 monclient: found mon.noname-b 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.133+0000 7fb4420f9640 1 -- 172.21.15.116:0/2061850879 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c0a13f0 con 0x7fb43c131ab0 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.135+0000 7fb4420f9640 1 -- 172.21.15.116:0/2061850879 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 msgr2=0x7fb43c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.135+0000 7fb4420f9640 1 --2- 172.21.15.116:0/2061850879 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 0x7fb43c131e80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f20 tx=0x7fb42c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f5cdf4e772016ca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 1 -- 172.21.15.116:0/2061850879 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c007610 con 0x7fb43c131ab0 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: dump: 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: _finish_auth 0 2024-09-18T12:29:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 1 -- 172.21.15.116:0/2061850879 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c02d050 con 0x7fb43c131ab0 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 5 monclient: authenticate success, global_id 64181 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 1 -- 172.21.15.116:0/2061850879 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c0079a0 con 0x7fb43c131ab0 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb4410f7640 10 monclient: dump: 2024-09-18T12:29:30.147 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 4 set_mon_vals no callback set 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 10 monclient: shutdown 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 -- 172.21.15.116:0/2061850879 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 msgr2=0x7fb43c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 --2- 172.21.15.116:0/2061850879 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 0x7fb43c131e80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f20 tx=0x7fb42c009840 comp rx=0 tx=0).stop 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 -- 172.21.15.116:0/2061850879 shutdown_connections 2024-09-18T12:29:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 --2- 172.21.15.116:0/2061850879 >> v2:172.21.15.144:3300/0 conn(0x7fb43c136bb0 0x7fb43c138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 --2- 172.21.15.116:0/2061850879 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c131ab0 0x7fb43c131e80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 -- 172.21.15.116:0/2061850879 >> 172.21.15.116:0/2061850879 conn(0x7fb43c067d30 msgr2=0x7fb43c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 -- 172.21.15.116:0/2061850879 shutdown_connections 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.136+0000 7fb44435d640 1 -- 172.21.15.116:0/2061850879 wait complete. 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 10 monclient: build_initial_monmap 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 10 monclient: monmap: 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:29:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 1 Processor -- start 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 1 -- start start 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 10 monclient: init 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c09e910 0x7fb43c0974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.137+0000 7fb44435d640 10 monclient(hunting): picked mon.noname-c con 0x7fb43c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c131ab0 0x7fb43c097a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 10 monclient(hunting): picked mon.noname-b con 0x7fb43c131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43c136bb0 0x7fb43c097f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 10 monclient(hunting): picked mon.noname-a con 0x7fb43c136bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c09ac30 con 0x7fb43c136bb0 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c09bde0 con 0x7fb43c131ab0 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c09d900 con 0x7fb43c09e910 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb44435d640 10 monclient(hunting): authenticate will time out at 997.460060s 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4428fa640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43c136bb0 msgr2=0x7fb43c097f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c09e910 0x7fb43c0974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4428fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43c136bb0 0x7fb43c097f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c09e910 0x7fb43c0974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48172/0 (socket says 172.21.15.116:48172) 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 -- 172.21.15.116:0/405561417 learned_addr learned my addr 172.21.15.116:0/405561417 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:29:30.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): get_auth_request con 0x7fb43c09e910 auth_method 0 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c131ab0 0x7fb43c097a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 10 monclient(hunting): get_auth_request con 0x7fb43c131ab0 auth_method 0 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient(hunting): handle_auth_done global_id 64184 payload 824 2024-09-18T12:29:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c131ab0 msgr2=0x7fb43c097a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c131ab0 0x7fb43c097a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43c136bb0 msgr2=0x7fb43c097f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb43c136bb0 0x7fb43c097f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient: _finish_hunting 0 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 monclient: found mon.noname-c 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:29:30.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.138+0000 7fb4420f9640 1 -- 172.21.15.116:0/405561417 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c009ae0 con 0x7fb43c09e910 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb4420f9640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c09e910 msgr2=0x7fb43c0974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb4420f9640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c09e910 0x7fb43c0974c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f80 tx=0x7fb42c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c031a70 con 0x7fb43c09e910 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: dump: 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.158 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c031bd0 con 0x7fb43c09e910 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c03c420 con 0x7fb43c09e910 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb44435d640 5 monclient: authenticate success, global_id 64184 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb44435d640 10 monclient: _renew_subs 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb44435d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:30.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43c098e20 con 0x7fb43c09e910 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb4408f6640 4 set_mon_vals no callback set 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb43a7fc640 10 monclient: dump: 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.161 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb44435d640 10 monclient: _renew_subs 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb44435d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.140+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43c131560 con 0x7fb43c09e910 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb44435d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb42c003020 con 0x7fb43c09e910 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb44435d640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 0x7fb43c132690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb44435d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c063740 con 0x7fb43c098480 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 0x7fb43c132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb43a7fc640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb428038440 0x7fb42803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 10 monclient: get_auth_request con 0x7fb43c098480 auth_method 0 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fb42c06c890 con 0x7fb43c09e910 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.141+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.142+0000 7fb4428fa640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb428038440 0x7fb42803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.142+0000 7fb4418f8640 10 monclient(hunting): handle_auth_done global_id 5221 payload 824 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.142+0000 7fb4418f8640 1 -- 172.21.15.116:0/405561417 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb43c09ad70 con 0x7fb43c098480 2024-09-18T12:29:30.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.142+0000 7fb4428fa640 10 monclient: get_auth_request con 0x7fb428038440 auth_method 0 2024-09-18T12:29:30.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb4418f8640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 msgr2=0x7fb43c132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb4418f8640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 0x7fb43c132690 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb430006fd0 tx=0x7fb430009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43000c9f0 con 0x7fb43c098480 2024-09-18T12:29:30.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:30.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 10 monclient: dump: 2024-09-18T12:29:30.166 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb4428fa640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb428038440 0x7fb42803a8f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb43c099780 tx=0x7fb434008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb43000cb50 con 0x7fb43c098480 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:30.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 msgr2=0x7fb43c132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 0x7fb43c132690 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb430006fd0 tx=0x7fb430009060 comp rx=0 tx=0).stop 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 msgr2=0x7fb43c132690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.144+0000 7fb43a7fc640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c098480 0x7fb43c132690 secure :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb430006fd0 tx=0x7fb430009060 comp rx=0 tx=0).stop 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb44435d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb44435d640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 0x7fb43c132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb44435d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c04bf60 con 0x7fb43c132bd0 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 0x7fb43c132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient: get_auth_request con 0x7fb43c132bd0 auth_method 0 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.150+0000 7fb4418f8640 10 monclient(hunting): handle_auth_done global_id 5224 payload 824 2024-09-18T12:29:30.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.151+0000 7fb4418f8640 1 -- 172.21.15.116:0/405561417 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb43c066280 con 0x7fb43c132bd0 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb4418f8640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 msgr2=0x7fb43c132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb4418f8640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 0x7fb43c132fd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb430006fd0 tx=0x7fb430015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430005dc0 con 0x7fb43c132bd0 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 10 monclient: dump: 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.170 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7fb430006030 con 0x7fb43c132bd0 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 msgr2=0x7fb43c132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 0x7fb43c132fd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb430006fd0 tx=0x7fb430015000 comp rx=0 tx=0).stop 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 msgr2=0x7fb43c132fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.153+0000 7fb43a7fc640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb43c132bd0 0x7fb43c132fd0 secure :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb430006fd0 tx=0x7fb430015000 comp rx=0 tx=0).stop 2024-09-18T12:29:30.171 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 46, 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:29:30.172 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:29:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:29:30.174 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:29:30.175 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:29:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:29:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:29:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:29:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:29:30.183 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:29:30.184 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb428038440 msgr2=0x7fb42803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb428038440 0x7fb42803a8f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb43c099780 tx=0x7fb434008040 comp rx=0 tx=0).stop 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 10 monclient: shutdown 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c09e910 msgr2=0x7fb43c0974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 --2- 172.21.15.116:0/405561417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb43c09e910 0x7fb43c0974c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f80 tx=0x7fb42c036000 comp rx=0 tx=0).stop 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb4428fa640 1 -- 172.21.15.116:0/405561417 reap_dead start 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb4428fa640 1 -- 172.21.15.116:0/405561417 reap_dead start 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 shutdown_connections 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 >> 172.21.15.116:0/405561417 conn(0x7fb43c067d30 msgr2=0x7fb43c0a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:30.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 shutdown_connections 2024-09-18T12:29:30.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:29:30.154+0000 7fb44435d640 1 -- 172.21.15.116:0/405561417 wait complete. 2024-09-18T12:29:30.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:29:30.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:29:30.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.671+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v84) v1 ==== 124+0+0 (secure 0 0 0) 0x7f369801a360 con 0x7f369c069a30 2024-09-18T12:29:30.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.671+0000 7f36927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T12:29:30.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.671+0000 7f36927fc640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-18T12:29:30.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.770+0000 7f36a4354640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T12:29:30.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.770+0000 7f36a4354640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T12:29:30.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.770+0000 7f36a4354640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:30.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.770+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f369c12ef10 con 0x7f369c069a30 2024-09-18T12:29:30.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.770+0000 7f36927fc640 1 -- 172.21.15.144:0/1313632578 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v84) v1 ==== 124+0+0 (secure 0 0 0) 0x7f369801a540 con 0x7f369c069a30 2024-09-18T12:29:30.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.770+0000 7f36927fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T12:29:30.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.770+0000 7f36927fc640 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-18T12:29:30.772 INFO:tasks.workunit.client.0.smithi144.stderr:device 'host1' does not appear in the crush map 2024-09-18T12:29:30.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3680038440 msgr2=0x7f368003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3680038440 0x7f368003a8f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f369c09e0d0 tx=0x7f368c032000 comp rx=0 tx=0).stop 2024-09-18T12:29:30.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 10 monclient: shutdown 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 msgr2=0x7f369c133490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 0x7f369c133490 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f369800c5e0 tx=0x7f3698004960 comp rx=0 tx=0).stop 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 shutdown_connections 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3680038440 0x7f368003a8f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f369c134b70 0x7f369c12e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f369c09baa0 0x7f369c096f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 --2- 172.21.15.144:0/1313632578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f369c069a30 0x7f369c133490 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 >> 172.21.15.144:0/1313632578 conn(0x7f369c09e9f0 msgr2=0x7f369c06ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 shutdown_connections 2024-09-18T12:29:30.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.772+0000 7f36a4354640 1 -- 172.21.15.144:0/1313632578 wait complete. 2024-09-18T12:29:30.782 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:30.783 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c host1 2024-09-18T12:29:30.783 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 0 2024-09-18T12:29:30.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.852+0000 7f959735d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:30.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.852+0000 7f959735d640 10 monclient: build_initial_monmap 2024-09-18T12:29:30.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.852+0000 7f959735d640 10 monclient: monmap: 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:30.853347+0000 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:30.853347+0000 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:30.854 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:30.855 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:30.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 1 Processor -- start 2024-09-18T12:29:30.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 1 -- start start 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient: init 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95900972c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f959009a7a0 0x7f959009be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient(hunting): picked mon.noname-a con 0x7f959009a7a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9590133680 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:30.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9590097690 con 0x7f9590133680 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9590098500 con 0x7f95900972c0 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f959009d6a0 con 0x7f959009a7a0 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f959735d640 10 monclient(hunting): authenticate will time out at 1006.422822s 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f95950f9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f95900972c0 0x7f959009b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49904/0 (socket says 172.21.15.144:49904) 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f95950f9640 1 -- 172.21.15.144:0/4028261009 learned_addr learned my addr 172.21.15.144:0/4028261009 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f95958fa640 1 -- 172.21.15.144:0/4028261009 >> v1:172.21.15.116:6789/0 conn(0x7f9590133680 legacy=0x7f9590135a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f95948f8640 1 -- 172.21.15.144:0/4028261009 >> v2:172.21.15.116:3300/0 conn(0x7f959009a7a0 msgr2=0x7f959009be80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:30.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.853+0000 7f95948f8640 1 --2- 172.21.15.144:0/4028261009 >> v2:172.21.15.116:3300/0 conn(0x7f959009a7a0 0x7f959009be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 -- 172.21.15.144:0/4028261009 >> v1:172.21.15.144:6789/0 conn(0x7f95900972c0 legacy=0x7f959009b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216231848 0 0) 0x7f9590098500 con 0x7f95900972c0 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): my global_id is 5227 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95780045f0 con 0x7f95900972c0 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3582988480 0 0) 0x7f95780045f0 con 0x7f95900972c0 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 >> v2:172.21.15.116:3300/0 conn(0x7f959009a7a0 msgr2=0x7f959009be80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 --2- 172.21.15.144:0/4028261009 >> v2:172.21.15.116:3300/0 conn(0x7f959009a7a0 0x7f959009be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 >> v1:172.21.15.116:6789/0 conn(0x7f9590133680 legacy=0x7f9590135a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:30.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900987e0 con 0x7f95900972c0 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: _finish_auth 0 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f95880017a0 con 0x7f95900972c0 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: dump: 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.871 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 >> v1:172.21.15.144:6789/0 conn(0x7f95900972c0 legacy=0x7f959009b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9578004ae0 0x7f9578004ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): picked mon.b con 0x7f9578004ae0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578005890 0x7f9578005d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): picked mon.a con 0x7f9578005890 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9578006490 0x7f957800cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): picked mon.c con 0x7f9578006490 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959009d6a0 con 0x7f9578005890 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957800d420 con 0x7f9578004ae0 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957800d5e0 con 0x7f9578006490 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f958ffff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:29:30.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9578004ae0 0x7f9578004ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): get_auth_request con 0x7f9578004ae0 auth_method 0 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95948f8640 1 -- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578005890 msgr2=0x7f9578005d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95948f8640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578005890 0x7f9578005d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95958fa640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9578006490 0x7f957800cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient(hunting): handle_auth_done global_id 5227 payload 293 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 -- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9578006490 msgr2=0x7f957800cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9578006490 0x7f957800cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 -- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578005890 msgr2=0x7f9578005d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578005890 0x7f9578005d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient: _finish_hunting 0 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 monclient: found mon.b 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 1 -- 172.21.15.144:0/4028261009 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780045f0 con 0x7f9578004ae0 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 10 monclient: _finish_auth 0 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95950f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95958fa640 10 monclient: get_auth_request con 0x7f9578006490 auth_method 0 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95958fa640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9578006490 0x7f957800cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f959735d640 5 monclient: authenticate success, global_id 5227 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f959735d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:30.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.856+0000 7f95958fa640 1 -- 172.21.15.144:0/4028261009 reap_dead start 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f95950f9640 1 -- 172.21.15.144:0/4028261009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9578004ae0 msgr2=0x7f9578004ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f95950f9640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9578004ae0 0x7f9578004ef0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f95880014d0 tx=0x7f9588007a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce68ec4b5b4212f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588005890 con 0x7f9578004ae0 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 10 monclient: dump: 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.876 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:30.877 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.877 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.877 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.877 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:30.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f959735d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:30.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9588005a10 con 0x7f9578004ae0 2024-09-18T12:29:30.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f959735d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f959735d640 4 set_mon_vals no callback set 2024-09-18T12:29:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 1 -- 172.21.15.144:0/4028261009 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588005dd0 con 0x7f9578004ae0 2024-09-18T12:29:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.859+0000 7f958ffff640 10 monclient: dump: 2024-09-18T12:29:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.879 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 10 monclient: shutdown 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 1 -- 172.21.15.144:0/4028261009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9578004ae0 msgr2=0x7f9578004ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9578004ae0 0x7f9578004ef0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f95880014d0 tx=0x7f9588007a90 comp rx=0 tx=0).stop 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 1 -- 172.21.15.144:0/4028261009 shutdown_connections 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 1 --2- 172.21.15.144:0/4028261009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9578004ae0 0x7f9578004ef0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 1 -- 172.21.15.144:0/4028261009 >> 172.21.15.144:0/4028261009 conn(0x7f95900714d0 msgr2=0x7f95900a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 1 -- 172.21.15.144:0/4028261009 shutdown_connections 2024-09-18T12:29:30.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 1 -- 172.21.15.144:0/4028261009 wait complete. 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 10 monclient: build_initial_monmap 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.860+0000 7f959735d640 10 monclient: monmap: 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:30.881 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 Processor -- start 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 -- start start 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient: init 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578004ae0 0x7f959013f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): picked mon.noname-a con 0x7f9578004ae0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 0x7f959011e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): picked mon.noname-b con 0x7f9590132c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9590133010 0x7f959011ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): picked mon.noname-c con 0x7f9590133010 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95901220f0 con 0x7f9578004ae0 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959009e480 con 0x7f9590132c20 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9590121e10 con 0x7f9590133010 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f959735d640 10 monclient(hunting): authenticate will time out at 1006.430881s 2024-09-18T12:29:30.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95950f9640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578004ae0 msgr2=0x7f959013f220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95948f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 0x7f959011e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95950f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578004ae0 0x7f959013f220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95948f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 0x7f959011e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53718/0 (socket says 172.21.15.144:53718) 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95948f8640 1 -- 172.21.15.144:0/27187435 learned_addr learned my addr 172.21.15.144:0/27187435 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95948f8640 10 monclient(hunting): get_auth_request con 0x7f9590132c20 auth_method 0 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95948f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95948f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.861+0000 7f95948f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9590133010 0x7f959011ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 10 monclient(hunting): get_auth_request con 0x7f9590133010 auth_method 0 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:30.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95958fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 10 monclient(hunting): handle_auth_done global_id 5230 payload 824 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 1 -- 172.21.15.144:0/27187435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9590133010 msgr2=0x7f959011ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9590133010 0x7f959011ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 1 -- 172.21.15.144:0/27187435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578004ae0 msgr2=0x7f959013f220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:30.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578004ae0 0x7f959013f220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 10 monclient: _finish_hunting 0 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 1 monclient: found mon.noname-b 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.862+0000 7f95948f8640 1 -- 172.21.15.144:0/27187435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590121b30 con 0x7f9590132c20 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f95948f8640 1 -- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 msgr2=0x7f959011e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f95948f8640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 0x7f959011e4c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f958400b840 tx=0x7f9584009270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584003e20 con 0x7f9590132c20 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: dump: 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:30.886 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95840050d0 con 0x7f9590132c20 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f959735d640 5 monclient: authenticate success, global_id 5230 2024-09-18T12:29:30.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f959735d640 10 monclient: _renew_subs 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f959735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9590120070 con 0x7f9590132c20 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584005490 con 0x7f9590132c20 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958d7fa640 10 monclient: dump: 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:30.888 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f959735d640 10 monclient: _renew_subs 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f959735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959011f0e0 con 0x7f9590132c20 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.864+0000 7f958f7fe640 4 set_mon_vals no callback set 2024-09-18T12:29:30.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9584012070 con 0x7f9590132c20 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f959735d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f959735d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f959735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95900636e0 con 0x7f9590132c20 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f958d7fa640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f957c038440 0x7f957c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f958404b2d0 con 0x7f9590132c20 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f95950f9640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f957c038440 0x7f957c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:30.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.865+0000 7f95950f9640 10 monclient: get_auth_request con 0x7f957c038440 auth_method 0 2024-09-18T12:29:30.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.868+0000 7f95950f9640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f957c038440 0x7f957c03a8f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9588035000 tx=0x7f9588036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:30.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.870+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9584012550 con 0x7f9590132c20 2024-09-18T12:29:30.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.870+0000 7f958d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:30.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:30.870+0000 7f958d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.025+0000 7f959735d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.025+0000 7f959735d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.025+0000 7f959735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.025+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f9590064b20 con 0x7f9590132c20 2024-09-18T12:29:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.025+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v84) v1 ==== 56+0+1273 (secure 0 0 0) 0x7f958401a070 con 0x7f9590132c20 2024-09-18T12:29:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.025+0000 7f958d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.025+0000 7f958d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:31.120 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.119+0000 7f959735d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:31.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.119+0000 7f959735d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:31.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.119+0000 7f959735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.119+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f95901333e0 con 0x7f9590132c20 2024-09-18T12:29:31.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.119+0000 7f958d7fa640 1 -- 172.21.15.144:0/27187435 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v84) v1 ==== 56+0+1273 (secure 0 0 0) 0x7f95840143d0 con 0x7f9590132c20 2024-09-18T12:29:31.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.119+0000 7f958d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:31.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.119+0000 7f958d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.121+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f957c038440 msgr2=0x7f957c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.121+0000 7f959735d640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f957c038440 0x7f957c03a8f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9588035000 tx=0x7f9588036000 comp rx=0 tx=0).stop 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.121+0000 7f959735d640 10 monclient: shutdown 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.121+0000 7f959735d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.121+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 msgr2=0x7f959011e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.121+0000 7f959735d640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 0x7f959011e4c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f958400b840 tx=0x7f9584009270 comp rx=0 tx=0).stop 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 shutdown_connections 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f957c038440 0x7f957c03a8f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9590133010 0x7f959011ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9590132c20 0x7f959011e4c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 --2- 172.21.15.144:0/27187435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9578004ae0 0x7f959013f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 >> 172.21.15.144:0/27187435 conn(0x7f95900714d0 msgr2=0x7f95900996d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:31.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 shutdown_connections 2024-09-18T12:29:31.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.122+0000 7f959735d640 1 -- 172.21.15.144:0/27187435 wait complete. 2024-09-18T12:29:31.132 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd tree-from host1 2024-09-18T12:29:31.132 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:29:31.132 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree-from host1 2024-09-18T12:29:31.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.202+0000 7fd95859d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:31.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.202+0000 7fd95859d640 10 monclient: build_initial_monmap 2024-09-18T12:29:31.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.202+0000 7fd95859d640 10 monclient: monmap: 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:31.203749+0000 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:31.203749+0000 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.203+0000 7fd95859d640 1 Processor -- start 2024-09-18T12:29:31.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.203+0000 7fd95859d640 1 -- start start 2024-09-18T12:29:31.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.203+0000 7fd95859d640 10 monclient: init 2024-09-18T12:29:31.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.203+0000 7fd95859d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:31.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T12:29:31.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9500972c0 0x7fd95009b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9500972c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:31.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd95009bcb0 0x7fd95009c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient(hunting): picked mon.noname-a con 0x7fd95009bcb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd950099790 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95009c5f0 con 0x7fd950099790 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95009d6a0 con 0x7fd95009bcb0 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95009c880 con 0x7fd9500972c0 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd95859d640 10 monclient(hunting): authenticate will time out at 1006.773093s 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd955b38640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fd95009bcb0 msgr2=0x7fd95009c0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956339640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9500972c0 0x7fd95009b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd955b38640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd95009bcb0 0x7fd95009c0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956339640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9500972c0 0x7fd95009b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53720/0 (socket says 172.21.15.144:53720) 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956339640 1 -- 172.21.15.144:0/637177090 learned_addr learned my addr 172.21.15.144:0/637177090 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956339640 10 monclient(hunting): get_auth_request con 0x7fd9500972c0 auth_method 0 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.204+0000 7fd956b3a640 1 --1- 172.21.15.144:0/637177090 >> v1:172.21.15.144:6789/0 conn(0x7fd950099790 0x7fd950099bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49918/0 (socket says 172.21.15.144:49918) 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956b3a640 1 -- 172.21.15.144:0/637177090 >> v1:172.21.15.144:6789/0 conn(0x7fd950099790 legacy=0x7fd950099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:31.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 10 monclient(hunting): handle_auth_done global_id 5233 payload 274 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 1 -- 172.21.15.144:0/637177090 >> v2:172.21.15.116:3300/0 conn(0x7fd95009bcb0 msgr2=0x7fd95009c0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 1 --2- 172.21.15.144:0/637177090 >> v2:172.21.15.116:3300/0 conn(0x7fd95009bcb0 0x7fd95009c0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 1 -- 172.21.15.144:0/637177090 >> v1:172.21.15.144:6789/0 conn(0x7fd950099790 legacy=0x7fd950099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 10 monclient: _finish_hunting 0 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 1 monclient: found mon.noname-c 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.206+0000 7fd956339640 1 -- 172.21.15.144:0/637177090 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950097690 con 0x7fd9500972c0 2024-09-18T12:29:31.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd956339640 1 -- 172.21.15.144:0/637177090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9500972c0 msgr2=0x7fd95009b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:31.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd956339640 1 --2- 172.21.15.144:0/637177090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9500972c0 0x7fd95009b770 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd95009e980 tx=0x7fd9480091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f26979940bbc07a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:31.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 1 -- 172.21.15.144:0/637177090 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94802f4b0 con 0x7fd9500972c0 2024-09-18T12:29:31.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:31.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: dump: 2024-09-18T12:29:31.223 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.223 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.224 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.224 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.224 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.224 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.224 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.224 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.224 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.225 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: _finish_auth 0 2024-09-18T12:29:31.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:31.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd95859d640 5 monclient: authenticate success, global_id 5233 2024-09-18T12:29:31.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd95859d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:31.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 1 -- 172.21.15.144:0/637177090 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94802f610 con 0x7fd9500972c0 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd95859d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 1 -- 172.21.15.144:0/637177090 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94802f9b0 con 0x7fd9500972c0 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd95859d640 4 set_mon_vals no callback set 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.209+0000 7fd955337640 10 monclient: dump: 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.226 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 10 monclient: shutdown 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 -- 172.21.15.144:0/637177090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9500972c0 msgr2=0x7fd95009b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:31.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 --2- 172.21.15.144:0/637177090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9500972c0 0x7fd95009b770 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd95009e980 tx=0x7fd9480091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 -- 172.21.15.144:0/637177090 shutdown_connections 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 --2- 172.21.15.144:0/637177090 >> v2:172.21.15.116:3300/0 conn(0x7fd95009bcb0 0x7fd95009c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 --2- 172.21.15.144:0/637177090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9500972c0 0x7fd95009b770 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 -- 172.21.15.144:0/637177090 >> 172.21.15.144:0/637177090 conn(0x7fd9500714d0 msgr2=0x7fd9500a0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 -- 172.21.15.144:0/637177090 shutdown_connections 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 1 -- 172.21.15.144:0/637177090 wait complete. 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 10 monclient: build_initial_monmap 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.210+0000 7fd95859d640 10 monclient: monmap: 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:31.228 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 Processor -- start 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 -- start start 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient: init 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:31.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500972c0 0x7fd95013ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9500972c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd950099790 0x7fd95013f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): picked mon.noname-a con 0x7fd950099790 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95009bcb0 0x7fd95013f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): picked mon.noname-b con 0x7fd95009bcb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95009abb0 con 0x7fd950099790 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950130710 con 0x7fd95009bcb0 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd950136470 con 0x7fd9500972c0 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:31.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd95859d640 10 monclient(hunting): authenticate will time out at 1006.780610s 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95009bcb0 0x7fd95013f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd955b38640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd950099790 msgr2=0x7fd95013f340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95009bcb0 0x7fd95013f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53728/0 (socket says 172.21.15.144:53728) 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 -- 172.21.15.144:0/2256283441 learned_addr learned my addr 172.21.15.144:0/2256283441 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): get_auth_request con 0x7fd95009bcb0 auth_method 0 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd955b38640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd950099790 0x7fd95013f340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:31.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956339640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500972c0 0x7fd95013ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956339640 10 monclient(hunting): get_auth_request con 0x7fd9500972c0 auth_method 0 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient(hunting): handle_auth_done global_id 5239 payload 824 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 -- 172.21.15.144:0/2256283441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500972c0 msgr2=0x7fd95013ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500972c0 0x7fd95013ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 -- 172.21.15.144:0/2256283441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd950099790 msgr2=0x7fd95013f340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd950099790 0x7fd95013f340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.211+0000 7fd956b3a640 1 monclient: found mon.noname-b 2024-09-18T12:29:31.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.212+0000 7fd956b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.212+0000 7fd956b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.212+0000 7fd956b3a640 1 -- 172.21.15.144:0/2256283441 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948009450 con 0x7fd95009bcb0 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.212+0000 7fd956339640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9500972c0 0x7fd95013ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd956b3a640 1 -- 172.21.15.144:0/2256283441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95009bcb0 msgr2=0x7fd95013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd956b3a640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd95009bcb0 0x7fd95013f880 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd94000c910 tx=0x7fd9400018a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 1 -- 172.21.15.144:0/2256283441 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000cea0 con 0x7fd95009bcb0 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: dump: 2024-09-18T12:29:31.233 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 1 -- 172.21.15.144:0/2256283441 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94000a640 con 0x7fd95009bcb0 2024-09-18T12:29:31.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd95859d640 5 monclient: authenticate success, global_id 5239 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd95859d640 10 monclient: _renew_subs 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd95859d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd95859d640 1 -- 172.21.15.144:0/2256283441 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd950134640 con 0x7fd95009bcb0 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 1 -- 172.21.15.144:0/2256283441 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000a9e0 con 0x7fd95009bcb0 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd94e7fc640 10 monclient: dump: 2024-09-18T12:29:31.235 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.236 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.236 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.236 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.236 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.236 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.236 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.236 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd954b36640 4 set_mon_vals no callback set 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd95859d640 10 monclient: _renew_subs 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd95859d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.214+0000 7fd95859d640 1 -- 172.21.15.144:0/2256283441 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9501370f0 con 0x7fd95009bcb0 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd94e7fc640 1 -- 172.21.15.144:0/2256283441 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd9400070f0 con 0x7fd95009bcb0 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd95859d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd95859d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd95859d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd95859d640 1 -- 172.21.15.144:0/2256283441 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950136600 con 0x7fd95009bcb0 2024-09-18T12:29:31.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd94e7fc640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd93c038440 0x7fd93c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd94e7fc640 1 -- 172.21.15.144:0/2256283441 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fd94004b3f0 con 0x7fd95009bcb0 2024-09-18T12:29:31.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd956339640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd93c038440 0x7fd93c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.215+0000 7fd956339640 10 monclient: get_auth_request con 0x7fd93c038440 auth_method 0 2024-09-18T12:29:31.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.217+0000 7fd956339640 1 --2- 172.21.15.144:0/2256283441 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd93c038440 0x7fd93c03a8f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd9500a0e20 tx=0x7fd948032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:31.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.219+0000 7fd94e7fc640 1 -- 172.21.15.144:0/2256283441 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd940018a40 con 0x7fd95009bcb0 2024-09-18T12:29:31.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.219+0000 7fd94e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:31.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.219+0000 7fd94e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:31.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.378+0000 7fd95859d640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-18T12:29:31.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.378+0000 7fd95859d640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-18T12:29:31.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.378+0000 7fd95859d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.378+0000 7fd95859d640 1 -- 172.21.15.144:0/2256283441 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) v1 -- 0x7fd9500636e0 con 0x7fd95009bcb0 2024-09-18T12:29:31.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.378+0000 7fd94e7fc640 1 -- 172.21.15.144:0/2256283441 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v84) v1 ==== 109+0+0 (secure 0 0 0) 0x7fd940018260 con 0x7fd95009bcb0 2024-09-18T12:29:31.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.378+0000 7fd94e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-18T12:29:31.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.378+0000 7fd94e7fc640 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-18T12:29:31.380 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-18T12:29:31.383 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:29:31.384 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree-from host2 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.454+0000 7f7057846640 10 monclient: get_monmap_and_config 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.454+0000 7f7057846640 10 monclient: build_initial_monmap 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.454+0000 7f7057846640 10 monclient: monmap: 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:31.455807+0000 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:31.455807+0000 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.456 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:31.457 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:31.457 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:31.457 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:31.457 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:31.457 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.455+0000 7f7057846640 1 Processor -- start 2024-09-18T12:29:31.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.455+0000 7f7057846640 1 -- start start 2024-09-18T12:29:31.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.455+0000 7f7057846640 10 monclient: init 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7050131b40 0x7f705009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient(hunting): picked mon.noname-a con 0x7f7050131b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f705009f240 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 0x7f7050139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient(hunting): picked mon.noname-b con 0x7f7050136e70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705006a230 con 0x7f705009f240 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70500a0300 con 0x7f7050131b40 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70500a00c0 con 0x7f7050136e70 2024-09-18T12:29:31.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7057846640 10 monclient(hunting): authenticate will time out at 1007.025175s 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7054de1640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f705009f240 legacy=0x7f7050131f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f70555e2640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f7050131b40 msgr2=0x7f705009ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f70555e2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7050131b40 0x7f705009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 0x7f7050139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 0x7f7050139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:52508/0 (socket says 172.21.15.144:52508) 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 -- 172.21.15.144:0/3745482662 learned_addr learned my addr 172.21.15.144:0/3745482662 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): get_auth_request con 0x7f7050136e70 auth_method 0 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:31.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient(hunting): handle_auth_done global_id 64193 payload 274 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 -- 172.21.15.144:0/3745482662 >> v2:172.21.15.116:3300/0 conn(0x7f7050131b40 msgr2=0x7f705009ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 --2- 172.21.15.144:0/3745482662 >> v2:172.21.15.116:3300/0 conn(0x7f7050131b40 0x7f705009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 -- 172.21.15.144:0/3745482662 >> v1:172.21.15.116:6789/0 conn(0x7f705009f240 legacy=0x7f7050131f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient: _finish_hunting 0 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 monclient: found mon.noname-b 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:31.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.456+0000 7f7055de3640 1 -- 172.21.15.144:0/3745482662 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705009bee0 con 0x7f7050136e70 2024-09-18T12:29:31.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.459+0000 7f7055de3640 1 -- 172.21.15.144:0/3745482662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 msgr2=0x7f7050139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:31.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.459+0000 7f7055de3640 1 --2- 172.21.15.144:0/3745482662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 0x7f7050139250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7040009f20 tx=0x7f7040009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91a4c33c698ac5ed server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 1 -- 172.21.15.144:0/3745482662 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040007610 con 0x7f7050136e70 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: dump: 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.467 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: _finish_auth 0 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 1 -- 172.21.15.144:0/3745482662 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704002d050 con 0x7f7050136e70 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 1 -- 172.21.15.144:0/3745482662 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70400079a0 con 0x7f7050136e70 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 5 monclient: authenticate success, global_id 64193 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 4 set_mon_vals no callback set 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:31.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f704ffff640 10 monclient: dump: 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 10 monclient: shutdown 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:31.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 -- 172.21.15.144:0/3745482662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 msgr2=0x7f7050139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 --2- 172.21.15.144:0/3745482662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 0x7f7050139250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7040009f20 tx=0x7f7040009840 comp rx=0 tx=0).stop 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 -- 172.21.15.144:0/3745482662 shutdown_connections 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 --2- 172.21.15.144:0/3745482662 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050136e70 0x7f7050139250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 --2- 172.21.15.144:0/3745482662 >> v2:172.21.15.116:3300/0 conn(0x7f7050131b40 0x7f705009ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 -- 172.21.15.144:0/3745482662 >> 172.21.15.144:0/3745482662 conn(0x7f70500a5100 msgr2=0x7f70500a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 -- 172.21.15.144:0/3745482662 shutdown_connections 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.460+0000 7f7057846640 1 -- 172.21.15.144:0/3745482662 wait complete. 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.461+0000 7f7057846640 10 monclient: build_initial_monmap 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.461+0000 7f7057846640 10 monclient: monmap: 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:31.470 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.461+0000 7f7057846640 1 Processor -- start 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.461+0000 7f7057846640 1 -- start start 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.461+0000 7f7057846640 10 monclient: init 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): picked mon.noname-b con 0x7f705009f240 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:31.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050131b40 0x7f7050141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): picked mon.noname-c con 0x7f7050131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7050136e70 0x7f7050098770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): picked mon.noname-a con 0x7f7050136e70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705011c110 con 0x7f7050136e70 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705011b870 con 0x7f705009f240 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705011bb50 con 0x7f7050131b40 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7057846640 10 monclient(hunting): authenticate will time out at 1007.031355s 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7055de3640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7050136e70 msgr2=0x7f7050098770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7055de3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7050136e70 0x7f7050098770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53734/0 (socket says 172.21.15.144:53734) 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 -- 172.21.15.144:0/2759005806 learned_addr learned my addr 172.21.15.144:0/2759005806 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): get_auth_request con 0x7f705009f240 auth_method 0 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050131b40 0x7f7050141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 10 monclient(hunting): get_auth_request con 0x7f7050131b40 auth_method 0 2024-09-18T12:29:31.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f7054de1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient(hunting): handle_auth_done global_id 5242 payload 824 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 -- 172.21.15.144:0/2759005806 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050131b40 msgr2=0x7f7050141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050131b40 0x7f7050141a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 -- 172.21.15.144:0/2759005806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7050136e70 msgr2=0x7f7050098770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7050136e70 0x7f7050098770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient: _finish_hunting 0 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 monclient: found mon.noname-b 2024-09-18T12:29:31.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.462+0000 7f70555e2640 1 -- 172.21.15.144:0/2759005806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040009ae0 con 0x7f705009f240 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f70555e2640 1 -- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 msgr2=0x7f7050141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f70555e2640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f704800b5c0 tx=0x7f7048009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048003cc0 con 0x7f705009f240 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f70555e2640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f704800b5c0 tx=0x7f7048009120 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f7048003e40 config(0 keys) v1 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 10 monclient: dump: 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.475 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7048003e40 con 0x7f705009f240 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f7057846640 5 monclient: authenticate success, global_id 5242 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f7057846640 10 monclient: _renew_subs 2024-09-18T12:29:31.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f7057846640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705009b9e0 con 0x7f705009f240 2024-09-18T12:29:31.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f704f7fe640 4 set_mon_vals no callback set 2024-09-18T12:29:31.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f7057846640 10 monclient: _renew_subs 2024-09-18T12:29:31.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f7057846640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.465+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7050141f90 con 0x7f705009f240 2024-09-18T12:29:31.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7048005340 con 0x7f705009f240 2024-09-18T12:29:31.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f704d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f704d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f704d7fa640 10 monclient: dump: 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:31.479 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f704801a460 con 0x7f705009f240 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f7057846640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f7057846640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f7057846640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70500636e0 con 0x7f705009f240 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f704d7fa640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f703c038440 0x7f703c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f7054de1640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f703c038440 0x7f703c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.466+0000 7f7054de1640 10 monclient: get_auth_request con 0x7f703c038440 auth_method 0 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.469+0000 7f7054de1640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f703c038440 0x7f703c03a8f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f70500a45f0 tx=0x7f7044006f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:31.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:31.470+0000 7f70555e2640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f704800b5c0 tx=0x7f7048009120 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f704801a710 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:29:32.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.088+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f704804b970 con 0x7f705009f240 2024-09-18T12:29:32.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.298+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f704801a710 con 0x7f705009f240 2024-09-18T12:29:32.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.298+0000 7f704d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:32.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.298+0000 7f704d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:32.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.455+0000 7f7057846640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T12:29:32.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.455+0000 7f7057846640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T12:29:32.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.455+0000 7f7057846640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.455+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f705009ad90 con 0x7f705009f240 2024-09-18T12:29:32.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.455+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v85) v1 ==== 80+0+177 (secure 0 0 0) 0x7f7048017780 con 0x7f705009f240 2024-09-18T12:29:32.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.455+0000 7f704d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T12:29:32.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.455+0000 7f704d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:32.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.548+0000 7f7057846640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T12:29:32.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.548+0000 7f7057846640 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T12:29:32.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.548+0000 7f7057846640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.548+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f705011be90 con 0x7f705009f240 2024-09-18T12:29:32.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.548+0000 7f704d7fa640 1 -- 172.21.15.144:0/2759005806 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v85) v1 ==== 80+0+177 (secure 0 0 0) 0x7f7048029090 con 0x7f705009f240 2024-09-18T12:29:32.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.548+0000 7f704d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T12:29:32.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.548+0000 7f704d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:32.550 INFO:tasks.workunit.client.0.smithi144.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T12:29:32.550 INFO:tasks.workunit.client.0.smithi144.stdout:-16 1.00000 host host2 2024-09-18T12:29:32.550 INFO:tasks.workunit.client.0.smithi144.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f703c038440 msgr2=0x7f703c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f703c038440 0x7f703c03a8f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f70500a45f0 tx=0x7f7044006f90 comp rx=0 tx=0).stop 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 10 monclient: shutdown 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 msgr2=0x7f7050141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f704800b5c0 tx=0x7f7048009120 comp rx=0 tx=0).stop 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 shutdown_connections 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f703c038440 0x7f703c03a8f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7050136e70 0x7f7050098770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7050131b40 0x7f7050141a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 --2- 172.21.15.144:0/2759005806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f705009f240 0x7f7050141510 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 >> 172.21.15.144:0/2759005806 conn(0x7f70500a5100 msgr2=0x7f70500a3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:32.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 shutdown_connections 2024-09-18T12:29:32.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.550+0000 7f7057846640 1 -- 172.21.15.144:0/2759005806 wait complete. 2024-09-18T12:29:32.560 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd tree-from osd.9 2024-09-18T12:29:32.560 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:29:32.560 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree-from osd.9 2024-09-18T12:29:32.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.631+0000 7fa42c0dd640 10 monclient: get_monmap_and_config 2024-09-18T12:29:32.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.631+0000 7fa42c0dd640 10 monclient: build_initial_monmap 2024-09-18T12:29:32.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.631+0000 7fa42c0dd640 10 monclient: monmap: 2024-09-18T12:29:32.634 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:32.634 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.634 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:32.632763+0000 2024-09-18T12:29:32.634 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:32.632763+0000 2024-09-18T12:29:32.634 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:32.634 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.635 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:32.635 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:32.635 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:32.635 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:32.635 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:32.635 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.632+0000 7fa42c0dd640 1 Processor -- start 2024-09-18T12:29:32.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.632+0000 7fa42c0dd640 1 -- start start 2024-09-18T12:29:32.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.632+0000 7fa42c0dd640 10 monclient: init 2024-09-18T12:29:32.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.632+0000 7fa42c0dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:32.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.632+0000 7fa42c0dd640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T12:29:32.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4240a3be0 0x7fa42406a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 10 monclient(hunting): picked mon.noname-a con 0x7fa4240a3be0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:32.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa424098f40 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:32.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa424136c50 0x7fa424139030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fa424136c50 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:32.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa424069920 con 0x7fa424098f40 2024-09-18T12:29:32.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa424069c00 con 0x7fa4240a3be0 2024-09-18T12:29:32.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42409f380 con 0x7fa424136c50 2024-09-18T12:29:32.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:32.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42c0dd640 10 monclient(hunting): authenticate will time out at 1008.202087s 2024-09-18T12:29:32.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa429e79640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fa4240a3be0 msgr2=0x7fa42406a5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:32.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa429678640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fa424098f40 legacy=0x7fa424099310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:32.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa429e79640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4240a3be0 0x7fa42406a5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:32.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa424136c50 0x7fa424139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa424136c50 0x7fa424139030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53742/0 (socket says 172.21.15.144:53742) 2024-09-18T12:29:32.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 -- 172.21.15.144:0/3945168635 learned_addr learned my addr 172.21.15.144:0/3945168635 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:32.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): get_auth_request con 0x7fa424136c50 auth_method 0 2024-09-18T12:29:32.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:32.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:32.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:32.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:32.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:32.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:32.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient(hunting): handle_auth_done global_id 5245 payload 274 2024-09-18T12:29:32.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 -- 172.21.15.144:0/3945168635 >> v2:172.21.15.116:3300/0 conn(0x7fa4240a3be0 msgr2=0x7fa42406a5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:32.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 --2- 172.21.15.144:0/3945168635 >> v2:172.21.15.116:3300/0 conn(0x7fa4240a3be0 0x7fa42406a5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 -- 172.21.15.144:0/3945168635 >> v1:172.21.15.116:6789/0 conn(0x7fa424098f40 legacy=0x7fa424099310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:32.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient: _finish_hunting 0 2024-09-18T12:29:32.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 monclient: found mon.noname-c 2024-09-18T12:29:32.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:32.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.633+0000 7fa42a67a640 1 -- 172.21.15.144:0/3945168635 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa424096a30 con 0x7fa424136c50 2024-09-18T12:29:32.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42a67a640 1 -- 172.21.15.144:0/3945168635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa424136c50 msgr2=0x7fa424139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:32.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42a67a640 1 --2- 172.21.15.144:0/3945168635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa424136c50 0x7fa424139030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa42409d680 tx=0x7fa414009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d762cd6a51b04db5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:32.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 1 -- 172.21.15.144:0/3945168635 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414007610 con 0x7fa424136c50 2024-09-18T12:29:32.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:32.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: dump: 2024-09-18T12:29:32.662 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.662 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.662 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.662 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.662 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.662 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.663 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.663 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.663 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.663 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: _finish_auth 0 2024-09-18T12:29:32.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:32.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 5 monclient: authenticate success, global_id 5245 2024-09-18T12:29:32.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 1 -- 172.21.15.144:0/3945168635 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41402d050 con 0x7fa424136c50 2024-09-18T12:29:32.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:32.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:32.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:32.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 1 -- 172.21.15.144:0/3945168635 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4140079e0 con 0x7fa424136c50 2024-09-18T12:29:32.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 4 set_mon_vals no callback set 2024-09-18T12:29:32.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:32.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa428e77640 10 monclient: dump: 2024-09-18T12:29:32.665 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.665 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.665 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.665 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 10 monclient: shutdown 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:32.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 1 -- 172.21.15.144:0/3945168635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa424136c50 msgr2=0x7fa424139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 1 --2- 172.21.15.144:0/3945168635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa424136c50 0x7fa424139030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa42409d680 tx=0x7fa414009840 comp rx=0 tx=0).stop 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 1 -- 172.21.15.144:0/3945168635 shutdown_connections 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 1 --2- 172.21.15.144:0/3945168635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa424136c50 0x7fa424139030 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 1 --2- 172.21.15.144:0/3945168635 >> v2:172.21.15.116:3300/0 conn(0x7fa4240a3be0 0x7fa42406a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.636+0000 7fa42c0dd640 1 -- 172.21.15.144:0/3945168635 >> 172.21.15.144:0/3945168635 conn(0x7fa424071350 msgr2=0x7fa4240a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.637+0000 7fa42c0dd640 1 -- 172.21.15.144:0/3945168635 shutdown_connections 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.637+0000 7fa42c0dd640 1 -- 172.21.15.144:0/3945168635 wait complete. 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.637+0000 7fa42c0dd640 10 monclient: build_initial_monmap 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.637+0000 7fa42c0dd640 10 monclient: monmap: 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:32.667 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.637+0000 7fa42c0dd640 1 Processor -- start 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.637+0000 7fa42c0dd640 1 -- start start 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.637+0000 7fa42c0dd640 10 monclient: init 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:32.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa424098f40 0x7fa424136620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): picked mon.noname-a con 0x7fa424098f40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4240a3be0 0x7fa42409bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): picked mon.noname-b con 0x7fa4240a3be0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa424136c50 0x7fa42409c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fa424136c50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42406ec10 con 0x7fa424098f40 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4240a3440 con 0x7fa4240a3be0 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa424120270 con 0x7fa424136c50 2024-09-18T12:29:32.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa42c0dd640 10 monclient(hunting): authenticate will time out at 1008.207193s 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4240a3be0 0x7fa42409bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4240a3be0 0x7fa42409bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53746/0 (socket says 172.21.15.144:53746) 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 -- 172.21.15.144:0/2735200930 learned_addr learned my addr 172.21.15.144:0/2735200930 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): get_auth_request con 0x7fa4240a3be0 auth_method 0 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429e79640 1 -- 172.21.15.144:0/2735200930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa424098f40 msgr2=0x7fa424136620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429e79640 1 --2- 172.21.15.144:0/2735200930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa424098f40 0x7fa424136620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:32.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient(hunting): handle_auth_done global_id 5248 payload 824 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 -- 172.21.15.144:0/2735200930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa424136c50 msgr2=0x7fa42409c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 --2- 172.21.15.144:0/2735200930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa424136c50 0x7fa42409c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 -- 172.21.15.144:0/2735200930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa424098f40 msgr2=0x7fa424136620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 --2- 172.21.15.144:0/2735200930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa424098f40 0x7fa424136620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient: _finish_hunting 0 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 monclient: found mon.noname-b 2024-09-18T12:29:32.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.638+0000 7fa429678640 1 -- 172.21.15.144:0/2735200930 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa414009ae0 con 0x7fa4240a3be0 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa429678640 1 -- 172.21.15.144:0/2735200930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4240a3be0 msgr2=0x7fa42409bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa429678640 1 --2- 172.21.15.144:0/2735200930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4240a3be0 0x7fa42409bbf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa41800b600 tx=0x7fa4180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 1 -- 172.21.15.144:0/2735200930 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa418015070 con 0x7fa4240a3be0 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: dump: 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.672 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: _finish_auth 0 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa42c0dd640 5 monclient: authenticate success, global_id 5248 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 1 -- 172.21.15.144:0/2735200930 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa418003d90 con 0x7fa4240a3be0 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:32.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 1 -- 172.21.15.144:0/2735200930 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa418006380 con 0x7fa4240a3be0 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa42c0dd640 10 monclient: _renew_subs 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa42c0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa42c0dd640 1 -- 172.21.15.144:0/2735200930 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42411df90 con 0x7fa4240a3be0 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa421ffb640 10 monclient: dump: 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.674 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.640+0000 7fa423fff640 4 set_mon_vals no callback set 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.641+0000 7fa42c0dd640 10 monclient: _renew_subs 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.641+0000 7fa42c0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.641+0000 7fa42c0dd640 1 -- 172.21.15.144:0/2735200930 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa424135ba0 con 0x7fa4240a3be0 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.641+0000 7fa421ffb640 1 -- 172.21.15.144:0/2735200930 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa418010440 con 0x7fa4240a3be0 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa421ffb640 1 --2- 172.21.15.144:0/2735200930 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa410038440 0x7fa41003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa42c0dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa42c0dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa42c0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa421ffb640 1 -- 172.21.15.144:0/2735200930 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fa41804b8d0 con 0x7fa4240a3be0 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa42c0dd640 1 -- 172.21.15.144:0/2735200930 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4240636e0 con 0x7fa4240a3be0 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa429e79640 1 --2- 172.21.15.144:0/2735200930 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa410038440 0x7fa41003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.642+0000 7fa429e79640 10 monclient: get_auth_request con 0x7fa410038440 auth_method 0 2024-09-18T12:29:32.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.644+0000 7fa429e79640 1 --2- 172.21.15.144:0/2735200930 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa410038440 0x7fa41003a8f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0038f0 tx=0x7fa41c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:32.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.646+0000 7fa421ffb640 1 -- 172.21.15.144:0/2735200930 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa418018ab0 con 0x7fa4240a3be0 2024-09-18T12:29:32.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.646+0000 7fa421ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:32.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.646+0000 7fa421ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.801+0000 7fa42c0dd640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-18T12:29:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.801+0000 7fa42c0dd640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-18T12:29:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.801+0000 7fa42c0dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.801+0000 7fa42c0dd640 1 -- 172.21.15.144:0/2735200930 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "osd.9"} v 0) v1 -- 0x7fa424064b20 con 0x7fa4240a3be0 2024-09-18T12:29:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.801+0000 7fa421ffb640 1 -- 172.21.15.144:0/2735200930 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.9"}]=-22 "osd.9" is not a bucket v85) v1 ==== 103+0+0 (secure 0 0 0) 0x7fa418018690 con 0x7fa4240a3be0 2024-09-18T12:29:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.801+0000 7fa421ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-18T12:29:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.801+0000 7fa421ffb640 10 monclient: _finish_command 2 = mon:22 "osd.9" is not a bucket 2024-09-18T12:29:32.804 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: "osd.9" is not a bucket 2024-09-18T12:29:32.806 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:29:32.806 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush rm bar 2024-09-18T12:29:32.806 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:29:32.806 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm bar 2024-09-18T12:29:32.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.878+0000 7f16f5f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:32.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.878+0000 7f16f5f9d640 10 monclient: build_initial_monmap 2024-09-18T12:29:32.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.878+0000 7f16f5f9d640 10 monclient: monmap: 2024-09-18T12:29:32.880 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:32.880 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.880 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:32.879500+0000 2024-09-18T12:29:32.880 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:32.879500+0000 2024-09-18T12:29:32.880 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:32.881 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.881 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:32.881 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:32.881 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:32.881 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:32.881 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:32.881 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 1 Processor -- start 2024-09-18T12:29:32.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 1 -- start start 2024-09-18T12:29:32.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient: init 2024-09-18T12:29:32.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16f0131c00 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16f01326e0 0x7f16f0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f16f01326e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f00a0b00 0x7f16f0138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f16f00a0b00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f009d230 con 0x7f16f0131c00 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f00a08c0 con 0x7f16f01326e0 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00a0200 con 0x7f16f00a0b00 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16f5f9d640 10 monclient(hunting): authenticate will time out at 1008.448850s 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16eeffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16f01326e0 0x7f16f0069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16eeffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f16f01326e0 0x7f16f0069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53758/0 (socket says 172.21.15.144:53758) 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16eeffd640 1 -- 172.21.15.144:0/2880644935 learned_addr learned my addr 172.21.15.144:0/2880644935 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16eeffd640 10 monclient(hunting): get_auth_request con 0x7f16f01326e0 auth_method 0 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.879+0000 7f16eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:32.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 1 --2- 172.21.15.144:0/2880644935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f00a0b00 0x7f16f0138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): get_auth_request con 0x7f16f00a0b00 auth_method 0 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient(hunting): handle_auth_done global_id 64199 payload 274 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 1 -- 172.21.15.144:0/2880644935 >> v2:172.21.15.144:3300/0 conn(0x7f16f01326e0 msgr2=0x7f16f0069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 1 --2- 172.21.15.144:0/2880644935 >> v2:172.21.15.144:3300/0 conn(0x7f16f01326e0 0x7f16f0069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 1 -- 172.21.15.144:0/2880644935 >> v1:172.21.15.144:6789/0 conn(0x7f16f0131c00 legacy=0x7f16f0131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:32.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:32.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 1 monclient: found mon.noname-b 2024-09-18T12:29:32.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:32.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:32.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.880+0000 7f16effff640 1 -- 172.21.15.144:0/2880644935 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00a1b70 con 0x7f16f00a0b00 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16effff640 1 -- 172.21.15.144:0/2880644935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f00a0b00 msgr2=0x7f16f0138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16effff640 1 --2- 172.21.15.144:0/2880644935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f00a0b00 0x7f16f0138fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009e60 tx=0x7f16e8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beca4a5daa8a6a11 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 1 -- 172.21.15.144:0/2880644935 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8007760 con 0x7f16f00a0b00 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: dump: 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.893 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 1 -- 172.21.15.144:0/2880644935 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e802d050 con 0x7f16f00a0b00 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 1 -- 172.21.15.144:0/2880644935 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8007af0 con 0x7f16f00a0b00 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16ee7fc640 10 monclient: dump: 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16f5f9d640 5 monclient: authenticate success, global_id 64199 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16f5f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:32.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.883+0000 7f16f5f9d640 4 set_mon_vals no callback set 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 10 monclient: shutdown 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 -- 172.21.15.144:0/2880644935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f00a0b00 msgr2=0x7f16f0138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 --2- 172.21.15.144:0/2880644935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f00a0b00 0x7f16f0138fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009e60 tx=0x7f16e8009780 comp rx=0 tx=0).stop 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 -- 172.21.15.144:0/2880644935 shutdown_connections 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 --2- 172.21.15.144:0/2880644935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f00a0b00 0x7f16f0138fb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 --2- 172.21.15.144:0/2880644935 >> v2:172.21.15.144:3300/0 conn(0x7f16f01326e0 0x7f16f0069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 -- 172.21.15.144:0/2880644935 >> 172.21.15.144:0/2880644935 conn(0x7f16f0071370 msgr2=0x7f16f00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 -- 172.21.15.144:0/2880644935 shutdown_connections 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 1 -- 172.21.15.144:0/2880644935 wait complete. 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 10 monclient: build_initial_monmap 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.884+0000 7f16f5f9d640 10 monclient: monmap: 2024-09-18T12:29:32.896 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 Processor -- start 2024-09-18T12:29:32.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 -- start start 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient: init 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f00a0b00 0x7f16f0096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f16f00a0b00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16f0131c00 0x7f16f0096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f16f0131c00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f01326e0 0x7f16f00974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f16f01326e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f009a0e0 con 0x7f16f00a0b00 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0099e00 con 0x7f16f0131c00 2024-09-18T12:29:32.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0098c50 con 0x7f16f01326e0 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16f5f9d640 10 monclient(hunting): authenticate will time out at 1008.454669s 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16f0131c00 0x7f16f0096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16f0131c00 0x7f16f0096f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53762/0 (socket says 172.21.15.144:53762) 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 1 -- 172.21.15.144:0/323154189 learned_addr learned my addr 172.21.15.144:0/323154189 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): get_auth_request con 0x7f16f0131c00 auth_method 0 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16ef7fe640 1 -- 172.21.15.144:0/323154189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f00a0b00 msgr2=0x7f16f0096a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16ef7fe640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f00a0b00 0x7f16f0096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:32.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16effff640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f01326e0 0x7f16f00974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16effff640 10 monclient(hunting): get_auth_request con 0x7f16f01326e0 auth_method 0 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.885+0000 7f16eeffd640 10 monclient(hunting): handle_auth_done global_id 5254 payload 824 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 1 -- 172.21.15.144:0/323154189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f01326e0 msgr2=0x7f16f00974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16f01326e0 0x7f16f00974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 1 -- 172.21.15.144:0/323154189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f00a0b00 msgr2=0x7f16f0096a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:32.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f00a0b00 0x7f16f0096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 1 monclient: found mon.noname-b 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.886+0000 7f16eeffd640 1 -- 172.21.15.144:0/323154189 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e8009a20 con 0x7f16f0131c00 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16eeffd640 1 -- 172.21.15.144:0/323154189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16f0131c00 msgr2=0x7f16f0096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16eeffd640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16f0131c00 0x7f16f0096f70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f16e400ea50 tx=0x7f16e40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 1 -- 172.21.15.144:0/323154189 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e40095a0 con 0x7f16f0131c00 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: dump: 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.902 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.903 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.903 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.903 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:32.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:32.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16f5f9d640 5 monclient: authenticate success, global_id 5254 2024-09-18T12:29:32.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 1 -- 172.21.15.144:0/323154189 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400cb50 con 0x7f16f0131c00 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 1 -- 172.21.15.144:0/323154189 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4013470 con 0x7f16f0131c00 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16f5f9d640 10 monclient: _renew_subs 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16f5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16f5f9d640 1 -- 172.21.15.144:0/323154189 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f0099b20 con 0x7f16f0131c00 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16edffb640 4 set_mon_vals no callback set 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:32.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:32.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16d77fe640 10 monclient: dump: 2024-09-18T12:29:32.905 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:32.905 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:32.905 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.906 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:32.906 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:32.906 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:32.906 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:32.906 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16f5f9d640 10 monclient: _renew_subs 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16f5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.888+0000 7f16f5f9d640 1 -- 172.21.15.144:0/323154189 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f0097e30 con 0x7f16f0131c00 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16f5f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16f5f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16f5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16d77fe640 1 -- 172.21.15.144:0/323154189 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f16e4013820 con 0x7f16f0131c00 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16f5f9d640 1 -- 172.21.15.144:0/323154189 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f00008d0 con 0x7f16f0131c00 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16d77fe640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16dc038440 0x7f16dc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:32.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16d77fe640 1 -- 172.21.15.144:0/323154189 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f16e404c320 con 0x7f16f0131c00 2024-09-18T12:29:32.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16ef7fe640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16dc038440 0x7f16dc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:32.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.889+0000 7f16ef7fe640 10 monclient: get_auth_request con 0x7f16dc038440 auth_method 0 2024-09-18T12:29:32.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.892+0000 7f16ef7fe640 1 --2- 172.21.15.144:0/323154189 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16dc038440 0x7f16dc03a8f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f16e0007fd0 tx=0x7f16e0007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:32.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.893+0000 7f16d77fe640 1 -- 172.21.15.144:0/323154189 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f16e401b770 con 0x7f16f0131c00 2024-09-18T12:29:32.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.894+0000 7f16d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:32.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:32.894+0000 7f16d77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:33.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.055+0000 7f16f5f9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:33.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.055+0000 7f16f5f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:33.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.055+0000 7f16f5f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:33.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.055+0000 7f16f5f9d640 1 -- 172.21.15.144:0/323154189 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f16f00a24c0 con 0x7f16f0131c00 2024-09-18T12:29:33.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.056+0000 7f16d77fe640 1 -- 172.21.15.144:0/323154189 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v85) v1 ==== 99+0+0 (secure 0 0 0) 0x7f16e401b3b0 con 0x7f16f0131c00 2024-09-18T12:29:33.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.056+0000 7f16d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:33.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.056+0000 7f16d77fe640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T12:29:33.058 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T12:29:33.060 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:29:33.061 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush unlink host2 2024-09-18T12:29:33.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.132+0000 7f03e64a8640 10 monclient: get_monmap_and_config 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.132+0000 7f03e64a8640 10 monclient: build_initial_monmap 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.132+0000 7f03e64a8640 10 monclient: monmap: 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:33.133697+0000 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:33.133697+0000 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:33.134 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:33.135 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:33.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.133+0000 7f03e64a8640 1 Processor -- start 2024-09-18T12:29:33.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.133+0000 7f03e64a8640 1 -- start start 2024-09-18T12:29:33.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.133+0000 7f03e64a8640 10 monclient: init 2024-09-18T12:29:33.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:33.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T12:29:33.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 0x7f03e0131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:33.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient(hunting): picked mon.noname-b con 0x7f03e0131c00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03e00a0b00 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03e0132510 0x7f03e0069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient(hunting): picked mon.noname-a con 0x7f03e0132510 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e00a1b70 con 0x7f03e00a0b00 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e00a08c0 con 0x7f03e0132510 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e0132910 con 0x7f03e0131c00 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e64a8640 10 monclient(hunting): authenticate will time out at 1008.703188s 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e4a45640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f03e0132510 msgr2=0x7f03e0069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03df7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f03e00a0b00 legacy=0x7f03e0068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03e4a45640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03e0132510 0x7f03e0069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 0x7f03e0131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:33.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 0x7f03e0131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:52534/0 (socket says 172.21.15.144:52534) 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 -- 172.21.15.144:0/3556929827 learned_addr learned my addr 172.21.15.144:0/3556929827 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): get_auth_request con 0x7f03e0131c00 auth_method 0 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient(hunting): handle_auth_done global_id 64205 payload 274 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 -- 172.21.15.144:0/3556929827 >> v2:172.21.15.116:3300/0 conn(0x7f03e0132510 msgr2=0x7f03e0069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 --2- 172.21.15.144:0/3556929827 >> v2:172.21.15.116:3300/0 conn(0x7f03e0132510 0x7f03e0069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 -- 172.21.15.144:0/3556929827 >> v1:172.21.15.116:6789/0 conn(0x7f03e00a0b00 legacy=0x7f03e0068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:33.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:33.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 monclient: found mon.noname-b 2024-09-18T12:29:33.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:33.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:33.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.134+0000 7f03dffff640 1 -- 172.21.15.144:0/3556929827 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e00a0200 con 0x7f03e0131c00 2024-09-18T12:29:33.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.137+0000 7f03dffff640 1 -- 172.21.15.144:0/3556929827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 msgr2=0x7f03e0131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:33.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.137+0000 7f03dffff640 1 --2- 172.21.15.144:0/3556929827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 0x7f03e0131fd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009910 tx=0x7f03d00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1412c6e46934c1cb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 1 -- 172.21.15.144:0/3556929827 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0032440 con 0x7f03e0131c00 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: dump: 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:33.146 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 5 monclient: authenticate success, global_id 64205 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 1 -- 172.21.15.144:0/3556929827 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d00325a0 con 0x7f03e0131c00 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 4 set_mon_vals no callback set 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 1 -- 172.21.15.144:0/3556929827 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0032940 con 0x7f03e0131c00 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:33.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03deffd640 10 monclient: dump: 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 10 monclient: shutdown 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 -- 172.21.15.144:0/3556929827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 msgr2=0x7f03e0131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:33.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3556929827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 0x7f03e0131fd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009910 tx=0x7f03d00091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 -- 172.21.15.144:0/3556929827 shutdown_connections 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3556929827 >> v2:172.21.15.116:3300/0 conn(0x7f03e0132510 0x7f03e0069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3556929827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0131c00 0x7f03e0131fd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 -- 172.21.15.144:0/3556929827 >> 172.21.15.144:0/3556929827 conn(0x7f03e0071370 msgr2=0x7f03e00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 -- 172.21.15.144:0/3556929827 shutdown_connections 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.138+0000 7f03e64a8640 1 -- 172.21.15.144:0/3556929827 wait complete. 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.139+0000 7f03e64a8640 10 monclient: build_initial_monmap 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.139+0000 7f03e64a8640 10 monclient: monmap: 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:33.149 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.139+0000 7f03e64a8640 1 Processor -- start 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.139+0000 7f03e64a8640 1 -- start start 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.139+0000 7f03e64a8640 10 monclient: init 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e00a0b00 0x7f03e009aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): picked mon.noname-a con 0x7f03e00a0b00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 0x7f03e0130bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): picked mon.noname-b con 0x7f03e0131c00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0132510 0x7f03e0141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:33.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): picked mon.noname-c con 0x7f03e0132510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e011dfd0 con 0x7f03e00a0b00 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e011dcf0 con 0x7f03e0131c00 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e011e2b0 con 0x7f03e0132510 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e64a8640 10 monclient(hunting): authenticate will time out at 1008.709209s 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03dffff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e00a0b00 msgr2=0x7f03e009aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e00a0b00 0x7f03e009aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 0x7f03e0130bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 0x7f03e0130bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53772/0 (socket says 172.21.15.144:53772) 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 -- 172.21.15.144:0/3101499304 learned_addr learned my addr 172.21.15.144:0/3101499304 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:33.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): get_auth_request con 0x7f03e0131c00 auth_method 0 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e4a45640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0132510 0x7f03e0141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e4a45640 10 monclient(hunting): get_auth_request con 0x7f03e0132510 auth_method 0 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e4a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e4a45640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03e4a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient(hunting): handle_auth_done global_id 5257 payload 824 2024-09-18T12:29:33.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 -- 172.21.15.144:0/3101499304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0132510 msgr2=0x7f03e0141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0132510 0x7f03e0141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 -- 172.21.15.144:0/3101499304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e00a0b00 msgr2=0x7f03e009aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e00a0b00 0x7f03e009aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 monclient: found mon.noname-b 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.140+0000 7f03df7fe640 1 -- 172.21.15.144:0/3101499304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0009450 con 0x7f03e0131c00 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.142+0000 7f03df7fe640 1 -- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 msgr2=0x7f03e0130bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.142+0000 7f03df7fe640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 0x7f03e0130bd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f03d400b5c0 tx=0x7f03d4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.142+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4003cc0 con 0x7f03e0131c00 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:33.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: dump: 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 5 monclient: authenticate success, global_id 5257 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 10 monclient: _renew_subs 2024-09-18T12:29:33.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e011c8b0 con 0x7f03e0131c00 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d4003e40 con 0x7f03e0131c00 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4005340 con 0x7f03e0131c00 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 10 monclient: dump: 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:33.155 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 10 monclient: _renew_subs 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03de7fc640 4 set_mon_vals no callback set 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e011cfa0 con 0x7f03e0131c00 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f03d401a460 con 0x7f03e0131c00 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.143+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e00008d0 con 0x7f03e0131c00 2024-09-18T12:29:33.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.144+0000 7f03c7fff640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f03cc038440 0x7f03cc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:33.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.144+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f03d404b8c0 con 0x7f03e0131c00 2024-09-18T12:29:33.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.144+0000 7f03dffff640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f03cc038440 0x7f03cc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:33.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.144+0000 7f03dffff640 10 monclient: get_auth_request con 0x7f03cc038440 auth_method 0 2024-09-18T12:29:33.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.146+0000 7f03dffff640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f03cc038440 0x7f03cc03a8f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009910 tx=0x7f03d0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:33.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.148+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f03d401a710 con 0x7f03e0131c00 2024-09-18T12:29:33.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.148+0000 7f03c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:33.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.148+0000 7f03c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:33.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.306+0000 7f03e64a8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T12:29:33.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.306+0000 7f03e64a8640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T12:29:33.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.306+0000 7f03e64a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:33.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.306+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f03e00a1890 con 0x7f03e0131c00 2024-09-18T12:29:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.680+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v86) v1 ==== 128+0+0 (secure 0 0 0) 0x7f03d4017310 con 0x7f03e0131c00 2024-09-18T12:29:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.680+0000 7f03c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T12:29:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.680+0000 7f03c7fff640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-18T12:29:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.777+0000 7f03e64a8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T12:29:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.777+0000 7f03e64a8640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T12:29:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.777+0000 7f03e64a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:33.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:33.777+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f03e0072640 con 0x7f03e0131c00 2024-09-18T12:29:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.689+0000 7f03c7fff640 1 -- 172.21.15.144:0/3101499304 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v87) v1 ==== 128+0+0 (secure 0 0 0) 0x7f03d4028090 con 0x7f03e0131c00 2024-09-18T12:29:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.689+0000 7f03c7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T12:29:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.689+0000 7f03c7fff640 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-18T12:29:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:removed item id -16 name 'host2' from crush map 2024-09-18T12:29:34.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f03cc038440 msgr2=0x7f03cc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:34.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f03cc038440 0x7f03cc03a8f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009910 tx=0x7f03d0032000 comp rx=0 tx=0).stop 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 10 monclient: shutdown 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 msgr2=0x7f03e0130bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 0x7f03e0130bd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f03d400b5c0 tx=0x7f03d4009120 comp rx=0 tx=0).stop 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 shutdown_connections 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f03cc038440 0x7f03cc03a8f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f03e0132510 0x7f03e0141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f03e0131c00 0x7f03e0130bd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 --2- 172.21.15.144:0/3101499304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e00a0b00 0x7f03e009aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 >> 172.21.15.144:0/3101499304 conn(0x7f03e0071370 msgr2=0x7f03e0134de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 shutdown_connections 2024-09-18T12:29:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.692+0000 7f03e64a8640 1 -- 172.21.15.144:0/3101499304 wait complete. 2024-09-18T12:29:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-18T12:29:34.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.782+0000 7ff322514640 10 monclient: get_monmap_and_config 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.782+0000 7ff322514640 10 monclient: build_initial_monmap 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.782+0000 7ff322514640 10 monclient: monmap: 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:34.783736+0000 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:34.783736+0000 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:34.784 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:34.785 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:34.785 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:34.785 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:34.785 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:34.785 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:34.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.783+0000 7ff322514640 1 Processor -- start 2024-09-18T12:29:34.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.783+0000 7ff322514640 1 -- start start 2024-09-18T12:29:34.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.783+0000 7ff322514640 10 monclient: init 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff31c09e560 0x7ff31c12e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient(hunting): picked mon.noname-c con 0x7ff31c09e560 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff31c09aeb0 0x7ff31c09b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient(hunting): picked mon.noname-a con 0x7ff31c09aeb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff31c131aa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c064c50 con 0x7ff31c131aa0 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c066090 con 0x7ff31c09aeb0 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c09b7c0 con 0x7ff31c09e560 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff322514640 10 monclient(hunting): authenticate will time out at 1010.353135s 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff31c09e560 0x7ff31c12e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff31c09e560 0x7ff31c12e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53786/0 (socket says 172.21.15.144:53786) 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7ff31c09aeb0 msgr2=0x7ff31c09b280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31bfff640 1 -- 172.21.15.144:0/3048465648 learned_addr learned my addr 172.21.15.144:0/3048465648 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31bfff640 10 monclient(hunting): get_auth_request con 0x7ff31c09e560 auth_method 0 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.784+0000 7ff31b7fe640 1 --2- 172.21.15.144:0/3048465648 >> v2:172.21.15.116:3300/0 conn(0x7ff31c09aeb0 0x7ff31c09b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff320ab1640 1 -- 172.21.15.144:0/3048465648 >> v1:172.21.15.144:6789/0 conn(0x7ff31c131aa0 legacy=0x7ff31c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 10 monclient(hunting): handle_auth_done global_id 5260 payload 274 2024-09-18T12:29:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 1 -- 172.21.15.144:0/3048465648 >> v2:172.21.15.116:3300/0 conn(0x7ff31c09aeb0 msgr2=0x7ff31c09b280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 1 --2- 172.21.15.144:0/3048465648 >> v2:172.21.15.116:3300/0 conn(0x7ff31c09aeb0 0x7ff31c09b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 1 -- 172.21.15.144:0/3048465648 >> v1:172.21.15.144:6789/0 conn(0x7ff31c131aa0 legacy=0x7ff31c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 1 monclient: found mon.noname-c 2024-09-18T12:29:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.786+0000 7ff31bfff640 1 -- 172.21.15.144:0/3048465648 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c12f0c0 con 0x7ff31c09e560 2024-09-18T12:29:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31bfff640 1 -- 172.21.15.144:0/3048465648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c09e560 msgr2=0x7ff31c12e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31bfff640 1 --2- 172.21.15.144:0/3048465648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c09e560 0x7ff31c12e7c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0684b0 tx=0x7ff30c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff8815ac8071619c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31affd640 1 -- 172.21.15.144:0/3048465648 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c02f4b0 con 0x7ff31c09e560 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31affd640 10 monclient: dump: 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31affd640 10 monclient: _finish_auth 0 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff31affd640 1 -- 172.21.15.144:0/3048465648 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c02f610 con 0x7ff31c09e560 2024-09-18T12:29:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.789+0000 7ff322514640 5 monclient: authenticate success, global_id 5260 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff31affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff31affd640 1 -- 172.21.15.144:0/3048465648 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c02f9f0 con 0x7ff31c09e560 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 4 set_mon_vals no callback set 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff31affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff31affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff31affd640 10 monclient: dump: 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 10 monclient: shutdown 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 -- 172.21.15.144:0/3048465648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c09e560 msgr2=0x7ff31c12e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 --2- 172.21.15.144:0/3048465648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c09e560 0x7ff31c12e7c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0684b0 tx=0x7ff30c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 -- 172.21.15.144:0/3048465648 shutdown_connections 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 --2- 172.21.15.144:0/3048465648 >> v2:172.21.15.116:3300/0 conn(0x7ff31c09aeb0 0x7ff31c09b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 --2- 172.21.15.144:0/3048465648 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c09e560 0x7ff31c12e7c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 -- 172.21.15.144:0/3048465648 >> 172.21.15.144:0/3048465648 conn(0x7ff31c0a1db0 msgr2=0x7ff31c09de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 -- 172.21.15.144:0/3048465648 shutdown_connections 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.790+0000 7ff322514640 1 -- 172.21.15.144:0/3048465648 wait complete. 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient: build_initial_monmap 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient: monmap: 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 Processor -- start 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 -- start start 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient: init 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:34.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff31c09aeb0 0x7ff31c12b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): picked mon.noname-c con 0x7ff31c09aeb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff31c09e560 0x7ff31c12b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): picked mon.noname-a con 0x7ff31c09e560 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 0x7ff31c12bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): picked mon.noname-b con 0x7ff31c131aa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c09eb80 con 0x7ff31c09e560 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c12f980 con 0x7ff31c131aa0 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c12d580 con 0x7ff31c09aeb0 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.791+0000 7ff322514640 10 monclient(hunting): authenticate will time out at 1010.360998s 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31b7fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff31c09e560 msgr2=0x7ff31c12b830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff31c09e560 0x7ff31c12b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 0x7ff31c12bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 0x7ff31c12bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53800/0 (socket says 172.21.15.144:53800) 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 -- 172.21.15.144:0/3390997457 learned_addr learned my addr 172.21.15.144:0/3390997457 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): get_auth_request con 0x7ff31c131aa0 auth_method 0 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff31c09aeb0 0x7ff31c12b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 10 monclient(hunting): get_auth_request con 0x7ff31c09aeb0 auth_method 0 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff31bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient(hunting): handle_auth_done global_id 5263 payload 824 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 -- 172.21.15.144:0/3390997457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff31c09aeb0 msgr2=0x7ff31c12b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff31c09aeb0 0x7ff31c12b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 -- 172.21.15.144:0/3390997457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff31c09e560 msgr2=0x7ff31c12b830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:34.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff31c09e560 0x7ff31c12b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:34.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient: _finish_hunting 0 2024-09-18T12:29:34.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 monclient: found mon.noname-b 2024-09-18T12:29:34.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:34.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:34.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.792+0000 7ff320ab1640 1 -- 172.21.15.144:0/3390997457 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c009450 con 0x7ff31c131aa0 2024-09-18T12:29:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff320ab1640 1 -- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 msgr2=0x7ff31c12bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff320ab1640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 0x7ff31c12bd70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff31400c940 tx=0x7ff3140018c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400a640 con 0x7ff31c131aa0 2024-09-18T12:29:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: dump: 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31400a7c0 con 0x7ff31c131aa0 2024-09-18T12:29:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff322514640 5 monclient: authenticate success, global_id 5263 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400ab80 con 0x7ff31c131aa0 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff322514640 10 monclient: _renew_subs 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff322514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c0889d0 con 0x7ff31c131aa0 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff303fff640 10 monclient: dump: 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:34.808 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff31a7fc640 4 set_mon_vals no callback set 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff322514640 10 monclient: _renew_subs 2024-09-18T12:29:34.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff322514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.794+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c12da00 con 0x7ff31c131aa0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.795+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff3140070f0 con 0x7ff31c131aa0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.795+0000 7ff322514640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.795+0000 7ff322514640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.795+0000 7ff322514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.795+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c0008d0 con 0x7ff31c131aa0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.795+0000 7ff303fff640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff308038440 0x7ff30803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.795+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5969+0+0 (secure 0 0 0) 0x7ff31404b3a0 con 0x7ff31c131aa0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.796+0000 7ff31bfff640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff308038440 0x7ff30803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.796+0000 7ff31bfff640 10 monclient: get_auth_request con 0x7ff308038440 auth_method 0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.798+0000 7ff31bfff640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff308038440 0x7ff30803a8f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff30c034000 tx=0x7ff30c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.800+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff314018cd0 con 0x7ff31c131aa0 2024-09-18T12:29:34.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.800+0000 7ff303fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:34.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.800+0000 7ff303fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:34.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.955+0000 7ff322514640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T12:29:34.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.955+0000 7ff322514640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T12:29:34.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.955+0000 7ff322514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:34.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:34.955+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7ff31c064900 con 0x7ff31c131aa0 2024-09-18T12:29:35.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.693+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v88) v1 ==== 251+0+0 (secure 0 0 0) 0x7ff31c064900 con 0x7ff31c131aa0 2024-09-18T12:29:35.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.693+0000 7ff303fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T12:29:35.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.693+0000 7ff303fff640 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.789+0000 7ff322514640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.789+0000 7ff322514640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.789+0000 7ff322514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.789+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7ff31c064900 con 0x7ff31c131aa0 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.790+0000 7ff303fff640 1 -- 172.21.15.144:0/3390997457 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v88) v1 ==== 200+0+0 (secure 0 0 0) 0x7ff31c064900 con 0x7ff31c131aa0 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.790+0000 7ff303fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.790+0000 7ff303fff640 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-18T12:29:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'host-for-test' already exists 2024-09-18T12:29:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff308038440 msgr2=0x7ff30803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff308038440 0x7ff30803a8f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff30c034000 tx=0x7ff30c032000 comp rx=0 tx=0).stop 2024-09-18T12:29:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 10 monclient: shutdown 2024-09-18T12:29:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 msgr2=0x7ff31c12bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 0x7ff31c12bd70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff31400c940 tx=0x7ff3140018c0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 shutdown_connections 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff308038440 0x7ff30803a8f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff31c131aa0 0x7ff31c12bd70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff31c09e560 0x7ff31c12b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 --2- 172.21.15.144:0/3390997457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff31c09aeb0 0x7ff31c12b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.791+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 >> 172.21.15.144:0/3390997457 conn(0x7ff31c0a1db0 msgr2=0x7ff31c09f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.792+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 shutdown_connections 2024-09-18T12:29:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.792+0000 7ff322514640 1 -- 172.21.15.144:0/3390997457 wait complete. 2024-09-18T12:29:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep host-for-test 2024-09-18T12:29:35.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.871+0000 7ff46bf5d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.871+0000 7ff46bf5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.871+0000 7ff46bf5d640 10 monclient: monmap: 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:35.872089+0000 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:35.872089+0000 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.871+0000 7ff46bf5d640 1 Processor -- start 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.871+0000 7ff46bf5d640 1 -- start start 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.871+0000 7ff46bf5d640 10 monclient: init 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff464131b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff464131f10 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff464132330 0x7ff46413a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff464132330 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4640a0300 con 0x7ff464131f10 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4640a00c0 con 0x7ff464131b40 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff46409bee0 con 0x7ff464132330 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46bf5d640 10 monclient(hunting): authenticate will time out at 1011.441477s 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff469cf9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff464131b40 0x7ff46409ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:49946/0 (socket says 172.21.15.144:49946) 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff469cf9640 1 -- 172.21.15.144:0/3518809092 learned_addr learned my addr 172.21.15.144:0/3518809092 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46a4fa640 1 -- 172.21.15.144:0/3518809092 >> v2:172.21.15.116:3300/0 conn(0x7ff464132330 msgr2=0x7ff46413a5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff4694f8640 1 -- 172.21.15.144:0/3518809092 >> v1:172.21.15.116:6789/0 conn(0x7ff464131f10 legacy=0x7ff464136e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.872+0000 7ff46a4fa640 1 --2- 172.21.15.144:0/3518809092 >> v2:172.21.15.116:3300/0 conn(0x7ff464132330 0x7ff46413a5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.874+0000 7ff469cf9640 1 -- 172.21.15.144:0/3518809092 >> v1:172.21.15.144:6789/0 conn(0x7ff464131b40 legacy=0x7ff46409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.874+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87514366 0 0) 0x7ff4640a00c0 con 0x7ff464131b40 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.874+0000 7ff468cf7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.874+0000 7ff468cf7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.874+0000 7ff468cf7640 10 monclient(hunting): my global_id is 5266 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.874+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4500034e0 con 0x7ff464131b40 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2452038295 0 0) 0x7ff4500034e0 con 0x7ff464131b40 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 >> v2:172.21.15.116:3300/0 conn(0x7ff464132330 msgr2=0x7ff46413a5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 --2- 172.21.15.144:0/3518809092 >> v2:172.21.15.116:3300/0 conn(0x7ff464132330 0x7ff46413a5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 >> v1:172.21.15.116:6789/0 conn(0x7ff464131f10 legacy=0x7ff464136e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: _finish_hunting 0 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640a14b0 con 0x7ff464131b40 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: _finish_auth 0 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7ff4600017a0 con 0x7ff464131b40 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: dump: 2024-09-18T12:29:35.887 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 >> v1:172.21.15.144:6789/0 conn(0x7ff464131b40 legacy=0x7ff46409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:35.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff450003aa0 0x7ff450003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): picked mon.c con 0x7ff450003aa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff450004780 0x7ff450004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): picked mon.a con 0x7ff450004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4500099c0 0x7ff45000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): picked mon.b con 0x7ff4500099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46409bee0 con 0x7ff450004780 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45000c3c0 con 0x7ff4500099c0 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:35.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45000c580 con 0x7ff450003aa0 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff468cf7640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff4694f8640 1 -- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff450004780 msgr2=0x7ff450004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff4694f8640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff450004780 0x7ff450004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4500099c0 0x7ff45000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): get_auth_request con 0x7ff4500099c0 auth_method 0 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff469cf9640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff450003aa0 0x7ff450003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:35.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff469cf9640 10 monclient(hunting): get_auth_request con 0x7ff450003aa0 auth_method 0 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff469cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff469cf9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff469cf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient(hunting): handle_auth_done global_id 5266 payload 293 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 1 -- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff450003aa0 msgr2=0x7ff450003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff450003aa0 0x7ff450003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 1 -- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff450004780 msgr2=0x7ff450004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff450004780 0x7ff450004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient: _finish_hunting 0 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 1 monclient: found mon.b 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:35.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 1 -- 172.21.15.144:0/3518809092 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4500034e0 con 0x7ff4500099c0 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46a4fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46bf5d640 5 monclient: authenticate success, global_id 5266 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.875+0000 7ff46bf5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.876+0000 7ff46a4fa640 1 -- 172.21.15.144:0/3518809092 reap_dead start 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46a4fa640 1 -- 172.21.15.144:0/3518809092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4500099c0 msgr2=0x7ff45000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46a4fa640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4500099c0 0x7ff45000bda0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff45c009f90 tx=0x7ff45c0095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afc6ba2b00b892ec server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c032430 con 0x7ff4500099c0 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 10 monclient: dump: 2024-09-18T12:29:35.892 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45c02d070 con 0x7ff4500099c0 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 1 -- 172.21.15.144:0/3518809092 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c032820 con 0x7ff4500099c0 2024-09-18T12:29:35.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46bf5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46bf5d640 4 set_mon_vals no callback set 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff468cf7640 10 monclient: dump: 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:35.894 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46bf5d640 10 monclient: shutdown 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46bf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46bf5d640 1 -- 172.21.15.144:0/3518809092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4500099c0 msgr2=0x7ff45000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.878+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4500099c0 0x7ff45000bda0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff45c009f90 tx=0x7ff45c0095d0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 1 -- 172.21.15.144:0/3518809092 shutdown_connections 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/3518809092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4500099c0 0x7ff45000bda0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 1 -- 172.21.15.144:0/3518809092 >> 172.21.15.144:0/3518809092 conn(0x7ff4640a5100 msgr2=0x7ff4640a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 1 -- 172.21.15.144:0/3518809092 shutdown_connections 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 1 -- 172.21.15.144:0/3518809092 wait complete. 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:35.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 10 monclient: monmap: 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 1 Processor -- start 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 1 -- start start 2024-09-18T12:29:35.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.879+0000 7ff46bf5d640 10 monclient: init 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4500099c0 0x7ff46412ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4500099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 0x7ff464099080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff464098c90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4640995c0 0x7ff464099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4640995c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46409ad80 con 0x7ff4500099c0 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46409e0c0 con 0x7ff464098c90 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:35.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff464141cc0 con 0x7ff4640995c0 2024-09-18T12:29:35.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:35.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46bf5d640 10 monclient(hunting): authenticate will time out at 1011.449326s 2024-09-18T12:29:35.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 0x7ff464099080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 0x7ff464099080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53812/0 (socket says 172.21.15.144:53812) 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 -- 172.21.15.144:0/937712379 learned_addr learned my addr 172.21.15.144:0/937712379 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): get_auth_request con 0x7ff464098c90 auth_method 0 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff469cf9640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4500099c0 msgr2=0x7ff46412ec80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:35.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff469cf9640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4500099c0 0x7ff46412ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46a4fa640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4640995c0 0x7ff464099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46a4fa640 10 monclient(hunting): get_auth_request con 0x7ff4640995c0 auth_method 0 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46a4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46a4fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff46a4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient(hunting): handle_auth_done global_id 5269 payload 824 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 -- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4640995c0 msgr2=0x7ff464099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4640995c0 0x7ff464099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 -- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4500099c0 msgr2=0x7ff46412ec80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4500099c0 0x7ff46412ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient: _finish_hunting 0 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 monclient: found mon.noname-b 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.880+0000 7ff4694f8640 1 -- 172.21.15.144:0/937712379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640a0a30 con 0x7ff464098c90 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.882+0000 7ff4694f8640 1 -- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 msgr2=0x7ff464099080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.882+0000 7ff4694f8640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 0x7ff464099080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff45400b810 tx=0x7ff4540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.882+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45400be00 con 0x7ff464098c90 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: dump: 2024-09-18T12:29:35.902 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: _finish_auth 0 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:35.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454003d50 con 0x7ff464098c90 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff46bf5d640 5 monclient: authenticate success, global_id 5269 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff46bf5d640 10 monclient: _renew_subs 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff46bf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46411c640 con 0x7ff464098c90 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454005340 con 0x7ff464098c90 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 10 monclient: dump: 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.904 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff45bfff640 4 set_mon_vals no callback set 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff46bf5d640 10 monclient: _renew_subs 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff46bf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46409a800 con 0x7ff464098c90 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.883+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff454012070 con 0x7ff464098c90 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff46bf5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:35.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff46bf5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:35.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff46bf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:35.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46409a060 con 0x7ff464098c90 2024-09-18T12:29:35.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff459ffb640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff448038440 0x7ff44803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:35.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6494+0+0 (secure 0 0 0) 0x7ff45404b3c0 con 0x7ff464098c90 2024-09-18T12:29:35.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff469cf9640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff448038440 0x7ff44803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:35.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.884+0000 7ff469cf9640 10 monclient: get_auth_request con 0x7ff448038440 auth_method 0 2024-09-18T12:29:35.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.886+0000 7ff469cf9640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff448038440 0x7ff44803a8f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff460001950 tx=0x7ff4640a0250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:35.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.888+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff454012550 con 0x7ff464098c90 2024-09-18T12:29:35.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.888+0000 7ff459ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:35.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:35.888+0000 7ff459ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:36.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.041+0000 7ff46bf5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:36.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.041+0000 7ff46bf5d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:36.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.041+0000 7ff46bf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:36.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.041+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff4640636e0 con 0x7ff464098c90 2024-09-18T12:29:36.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.042+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v88) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff45401a070 con 0x7ff464098c90 2024-09-18T12:29:36.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.042+0000 7ff459ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:36.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.042+0000 7ff459ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.133+0000 7ff46bf5d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.133+0000 7ff46bf5d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.133+0000 7ff46bf5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.133+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff46409b3f0 con 0x7ff464098c90 2024-09-18T12:29:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.133+0000 7ff459ffb640 1 -- 172.21.15.144:0/937712379 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v88) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff45401de60 con 0x7ff464098c90 2024-09-18T12:29:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.133+0000 7ff459ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.133+0000 7ff459ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff448038440 msgr2=0x7ff44803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff448038440 0x7ff44803a8f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff460001950 tx=0x7ff4640a0250 comp rx=0 tx=0).stop 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 10 monclient: shutdown 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 msgr2=0x7ff464099080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 0x7ff464099080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff45400b810 tx=0x7ff4540091a0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 shutdown_connections 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff448038440 0x7ff44803a8f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4640995c0 0x7ff464099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff464098c90 0x7ff464099080 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 --2- 172.21.15.144:0/937712379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4500099c0 0x7ff46412ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 >> 172.21.15.144:0/937712379 conn(0x7ff4640a5100 msgr2=0x7ff464130bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 shutdown_connections 2024-09-18T12:29:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.135+0000 7ff46bf5d640 1 -- 172.21.15.144:0/937712379 wait complete. 2024-09-18T12:29:36.145 INFO:tasks.workunit.client.0.smithi144.stdout:-11 0 host host-for-test 2024-09-18T12:29:36.146 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:36.146 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep rack-for-test 2024-09-18T12:29:36.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.217+0000 7ff07a24f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:36.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.217+0000 7ff07a24f640 10 monclient: build_initial_monmap 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.217+0000 7ff07a24f640 10 monclient: monmap: 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:36.218464+0000 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:36.218464+0000 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:36.219 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:36.220 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:36.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 1 Processor -- start 2024-09-18T12:29:36.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 1 -- start start 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 10 monclient: init 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0740a3c00 0x7ff07406a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 10 monclient(hunting): picked mon.noname-c con 0x7ff0740a3c00 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.218+0000 7ff07a24f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff0740991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 10 monclient(hunting): picked mon.noname-b con 0x7ff074098dd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff07409da60 0x7ff07409dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 10 monclient(hunting): picked mon.noname-a con 0x7ff07409da60 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:36.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff074069f00 con 0x7ff07409da60 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff074069c20 con 0x7ff0740a3c00 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff074069940 con 0x7ff074098dd0 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff07a24f640 10 monclient(hunting): authenticate will time out at 1011.788039s 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0740a3c00 0x7ff07406a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff073fff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7ff07409da60 msgr2=0x7ff07409dea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff073fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff07409da60 0x7ff07409dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0740a3c00 0x7ff07406a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53814/0 (socket says 172.21.15.144:53814) 2024-09-18T12:29:36.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 -- 172.21.15.144:0/277008610 learned_addr learned my addr 172.21.15.144:0/277008610 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): get_auth_request con 0x7ff0740a3c00 auth_method 0 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff06bfff640 1 --2- 172.21.15.144:0/277008610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff0740991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff06bfff640 10 monclient(hunting): get_auth_request con 0x7ff074098dd0 auth_method 0 2024-09-18T12:29:36.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff06bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff06bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff06bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient(hunting): handle_auth_done global_id 5272 payload 274 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 -- 172.21.15.144:0/277008610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 msgr2=0x7ff0740991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 --2- 172.21.15.144:0/277008610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff0740991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 -- 172.21.15.144:0/277008610 >> v2:172.21.15.116:3300/0 conn(0x7ff07409da60 msgr2=0x7ff07409dea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 --2- 172.21.15.144:0/277008610 >> v2:172.21.15.116:3300/0 conn(0x7ff07409da60 0x7ff07409dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 monclient: found mon.noname-c 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:36.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.219+0000 7ff0737fe640 1 -- 172.21.15.144:0/277008610 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07409f320 con 0x7ff0740a3c00 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff0737fe640 1 -- 172.21.15.144:0/277008610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 msgr2=0x7ff07406a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff0737fe640 1 --2- 172.21.15.144:0/277008610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07406a600 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff074064800 tx=0x7ff0640091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4954f957ae63e52 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 1 -- 172.21.15.144:0/277008610 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064032440 con 0x7ff0740a3c00 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: dump: 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:36.235 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 1 -- 172.21.15.144:0/277008610 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0640325a0 con 0x7ff0740a3c00 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff07a24f640 5 monclient: authenticate success, global_id 5272 2024-09-18T12:29:36.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff07a24f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 1 -- 172.21.15.144:0/277008610 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064032940 con 0x7ff0740a3c00 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff07a24f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff07a24f640 4 set_mon_vals no callback set 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff072ffd640 10 monclient: dump: 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:36.237 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.222+0000 7ff07a24f640 10 monclient: shutdown 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 -- 172.21.15.144:0/277008610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 msgr2=0x7ff07406a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 --2- 172.21.15.144:0/277008610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07406a600 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff074064800 tx=0x7ff0640091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 -- 172.21.15.144:0/277008610 shutdown_connections 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 --2- 172.21.15.144:0/277008610 >> v2:172.21.15.116:3300/0 conn(0x7ff07409da60 0x7ff07409dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 --2- 172.21.15.144:0/277008610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff0740991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 --2- 172.21.15.144:0/277008610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07406a600 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 -- 172.21.15.144:0/277008610 >> 172.21.15.144:0/277008610 conn(0x7ff074071350 msgr2=0x7ff0740a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 -- 172.21.15.144:0/277008610 shutdown_connections 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 -- 172.21.15.144:0/277008610 wait complete. 2024-09-18T12:29:36.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 10 monclient: build_initial_monmap 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 10 monclient: monmap: 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 Processor -- start 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 1 -- start start 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.223+0000 7ff07a24f640 10 monclient: init 2024-09-18T12:29:36.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff074141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): picked mon.noname-c con 0x7ff074098dd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07409da60 0x7ff074134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): picked mon.noname-a con 0x7ff07409da60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07411e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): picked mon.noname-b con 0x7ff0740a3c00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07406ec10 con 0x7ff07409da60 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074097e50 con 0x7ff0740a3c00 2024-09-18T12:29:36.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff074122250 con 0x7ff074098dd0 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff07a24f640 10 monclient(hunting): authenticate will time out at 1011.793446s 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07411e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff06bfff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07409da60 msgr2=0x7ff074134dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07411e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53824/0 (socket says 172.21.15.144:53824) 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 -- 172.21.15.144:0/545982649 learned_addr learned my addr 172.21.15.144:0/545982649 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): get_auth_request con 0x7ff0740a3c00 auth_method 0 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:36.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff06bfff640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07409da60 0x7ff074134dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff0737fe640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff074141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient(hunting): handle_auth_done global_id 5275 payload 824 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 -- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 msgr2=0x7ff074141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff074141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 -- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07409da60 msgr2=0x7ff074134dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:36.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07409da60 0x7ff074134dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 monclient: found mon.noname-b 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.224+0000 7ff073fff640 1 -- 172.21.15.144:0/545982649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064009450 con 0x7ff0740a3c00 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.226+0000 7ff073fff640 1 -- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 msgr2=0x7ff07411e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.226+0000 7ff073fff640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07411e920 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00bb40 tx=0x7ff06c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00cbf0 con 0x7ff0740a3c00 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:36.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: dump: 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:36.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c00cd70 con 0x7ff0740a3c00 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff07a24f640 5 monclient: authenticate success, global_id 5275 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff07a24f640 10 monclient: _renew_subs 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff07a24f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff074121ee0 con 0x7ff0740a3c00 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c006150 con 0x7ff0740a3c00 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff06b7fe640 10 monclient: dump: 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.245 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff0727fc640 4 set_mon_vals no callback set 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff07a24f640 10 monclient: _renew_subs 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff07a24f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.227+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff074141550 con 0x7ff0740a3c00 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff06c012020 con 0x7ff0740a3c00 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff07a24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:36.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff07a24f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff07a24f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0740636e0 con 0x7ff0740a3c00 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff06b7fe640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff054038490 0x7ff05403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6494+0+0 (secure 0 0 0) 0x7ff06c04bec0 con 0x7ff0740a3c00 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff0737fe640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff054038490 0x7ff05403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.228+0000 7ff0737fe640 10 monclient: get_auth_request con 0x7ff054038490 auth_method 0 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.230+0000 7ff0737fe640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff054038490 0x7ff05403a940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff074068c40 tx=0x7ff064032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:36.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:36.232+0000 7ff073fff640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07411e920 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00bb40 tx=0x7ff06c0097d0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7ff06c0132e0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:29:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.060+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff06c0132e0 con 0x7ff0740a3c00 2024-09-18T12:29:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.060+0000 7ff06b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.060+0000 7ff06b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:37.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.220+0000 7ff07a24f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:37.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.220+0000 7ff07a24f640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.220+0000 7ff07a24f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.220+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff074122390 con 0x7ff0740a3c00 2024-09-18T12:29:37.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.220+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v89) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff06c01d300 con 0x7ff0740a3c00 2024-09-18T12:29:37.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.220+0000 7ff06b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.220+0000 7ff06b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:37.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.314+0000 7ff07a24f640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:37.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.314+0000 7ff07a24f640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.314+0000 7ff07a24f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.314+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ff074072640 con 0x7ff0740a3c00 2024-09-18T12:29:37.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.315+0000 7ff06b7fe640 1 -- 172.21.15.144:0/545982649 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v89) v1 ==== 56+0+1500 (secure 0 0 0) 0x7ff06c01d950 con 0x7ff0740a3c00 2024-09-18T12:29:37.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.315+0000 7ff06b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.315+0000 7ff06b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:37.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.316+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff054038490 msgr2=0x7ff05403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.316+0000 7ff07a24f640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff054038490 0x7ff05403a940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff074068c40 tx=0x7ff064032000 comp rx=0 tx=0).stop 2024-09-18T12:29:37.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.316+0000 7ff07a24f640 10 monclient: shutdown 2024-09-18T12:29:37.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.316+0000 7ff07a24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:37.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.316+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 msgr2=0x7ff07411e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.316+0000 7ff07a24f640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07411e920 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00bb40 tx=0x7ff06c0097d0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 shutdown_connections 2024-09-18T12:29:37.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff054038490 0x7ff05403a940 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0740a3c00 0x7ff07411e920 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff07409da60 0x7ff074134dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 --2- 172.21.15.144:0/545982649 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff074098dd0 0x7ff074141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 >> 172.21.15.144:0/545982649 conn(0x7ff074071350 msgr2=0x7ff074098060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:37.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 shutdown_connections 2024-09-18T12:29:37.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.317+0000 7ff07a24f640 1 -- 172.21.15.144:0/545982649 wait complete. 2024-09-18T12:29:37.327 INFO:tasks.workunit.client.0.smithi144.stdout:-12 0 rack rack-for-test 2024-09-18T12:29:37.327 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:29:37.328 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep root-for-test 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.398+0000 7f442d99d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.398+0000 7f442d99d640 10 monclient: build_initial_monmap 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.398+0000 7f442d99d640 10 monclient: monmap: 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:37.399823+0000 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:37.399823+0000 2024-09-18T12:29:37.412 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 1 Processor -- start 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 1 -- start start 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 10 monclient: init 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:37.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f4428099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 10 monclient(hunting): picked mon.noname-b con 0x7f44280a3950 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f442809c7f0 0x7f442809cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.399+0000 7f442d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f442809c7f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f442809d960 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442809f520 con 0x7f442809d960 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442809b840 con 0x7f442809c7f0 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442809d570 con 0x7f44280a3950 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f442d99d640 10 monclient(hunting): authenticate will time out at 1012.969037s 2024-09-18T12:29:37.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f441e7fc640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f442809c7f0 msgr2=0x7f442809cbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f44277fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f442809d960 0x7f442809dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51648/0 (socket says 172.21.15.144:51648) 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f44277fe640 1 -- 172.21.15.144:0/243420570 learned_addr learned my addr 172.21.15.144:0/243420570 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f441e7fc640 1 --2- 172.21.15.144:0/243420570 >> v2:172.21.15.116:3300/0 conn(0x7f442809c7f0 0x7f442809cbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 1 --2- 172.21.15.144:0/243420570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f4428099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): get_auth_request con 0x7f44280a3950 auth_method 0 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:37.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient(hunting): handle_auth_done global_id 64220 payload 274 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 1 -- 172.21.15.144:0/243420570 >> v2:172.21.15.116:3300/0 conn(0x7f442809c7f0 msgr2=0x7f442809cbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 1 --2- 172.21.15.144:0/243420570 >> v2:172.21.15.116:3300/0 conn(0x7f442809c7f0 0x7f442809cbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 1 -- 172.21.15.144:0/243420570 >> v1:172.21.15.144:6789/0 conn(0x7f442809d960 legacy=0x7f442809dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 1 monclient: found mon.noname-b 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.400+0000 7f4426ffd640 1 -- 172.21.15.144:0/243420570 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4428099fa0 con 0x7f44280a3950 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f4426ffd640 1 -- 172.21.15.144:0/243420570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 msgr2=0x7f4428099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f4426ffd640 1 --2- 172.21.15.144:0/243420570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f4428099a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4418009850 tx=0x7f44180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f29f92c6ca400d88 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:37.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 1 -- 172.21.15.144:0/243420570 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418032440 con 0x7f44280a3950 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: dump: 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.417 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.418 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.418 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.418 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:37.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:37.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 1 -- 172.21.15.144:0/243420570 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44180325a0 con 0x7f44280a3950 2024-09-18T12:29:37.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f442d99d640 5 monclient: authenticate success, global_id 64220 2024-09-18T12:29:37.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f442d99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:37.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:37.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f442d99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:37.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 1 -- 172.21.15.144:0/243420570 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418032980 con 0x7f44280a3950 2024-09-18T12:29:37.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f442d99d640 4 set_mon_vals no callback set 2024-09-18T12:29:37.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:37.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.403+0000 7f44267fc640 10 monclient: dump: 2024-09-18T12:29:37.420 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.420 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.420 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 10 monclient: shutdown 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 -- 172.21.15.144:0/243420570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 msgr2=0x7f4428099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 --2- 172.21.15.144:0/243420570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f4428099a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4418009850 tx=0x7f44180090f0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 -- 172.21.15.144:0/243420570 shutdown_connections 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 --2- 172.21.15.144:0/243420570 >> v2:172.21.15.116:3300/0 conn(0x7f442809c7f0 0x7f442809cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 --2- 172.21.15.144:0/243420570 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f4428099a60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 -- 172.21.15.144:0/243420570 >> 172.21.15.144:0/243420570 conn(0x7f4428070fd0 msgr2=0x7f44280a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 -- 172.21.15.144:0/243420570 shutdown_connections 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 1 -- 172.21.15.144:0/243420570 wait complete. 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 10 monclient: build_initial_monmap 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.404+0000 7f442d99d640 10 monclient: monmap: 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:37.422 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 Processor -- start 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 -- start start 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient: init 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442809c7f0 0x7f442813ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f442809c7f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 0x7f4428132c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): picked mon.noname-b con 0x7f442809d960 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f442811c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f44280a3950 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44280a29d0 con 0x7f442809c7f0 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442811fd80 con 0x7f442809d960 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4428120060 con 0x7f44280a3950 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f442d99d640 10 monclient(hunting): authenticate will time out at 1012.974899s 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f4426ffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442809c7f0 msgr2=0x7f442813ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f4426ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442809c7f0 0x7f442813ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:37.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.405+0000 7f441e7fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 0x7f4428132c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 0x7f4428132c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42160/0 (socket says 172.21.15.144:42160) 2024-09-18T12:29:37.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 -- 172.21.15.144:0/1433864996 learned_addr learned my addr 172.21.15.144:0/1433864996 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:37.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): get_auth_request con 0x7f442809d960 auth_method 0 2024-09-18T12:29:37.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:37.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:37.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f442811c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 10 monclient(hunting): get_auth_request con 0x7f44280a3950 auth_method 0 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:37.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f44277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient(hunting): handle_auth_done global_id 5278 payload 824 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 -- 172.21.15.144:0/1433864996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 msgr2=0x7f442811c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f442811c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 -- 172.21.15.144:0/1433864996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442809c7f0 msgr2=0x7f442813ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442809c7f0 0x7f442813ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:29:37.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 monclient: found mon.noname-b 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.406+0000 7f441e7fc640 1 -- 172.21.15.144:0/1433864996 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418009390 con 0x7f442809d960 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441e7fc640 1 -- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 msgr2=0x7f4428132c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441e7fc640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 0x7f4428132c60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f441000b600 tx=0x7f44100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410015070 con 0x7f442809d960 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: dump: 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.428 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4410003d90 con 0x7f442809d960 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f442d99d640 5 monclient: authenticate success, global_id 5278 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410006380 con 0x7f442809d960 2024-09-18T12:29:37.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f442d99d640 10 monclient: _renew_subs 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f442d99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442813fc60 con 0x7f442809d960 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f441f7fe640 10 monclient: dump: 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.430 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f4425ffb640 4 set_mon_vals no callback set 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f442d99d640 10 monclient: _renew_subs 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f442d99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.408+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442811ee20 con 0x7f442809d960 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.409+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4410010440 con 0x7f442809d960 2024-09-18T12:29:37.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.409+0000 7f442d99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:37.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.409+0000 7f442d99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:37.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.409+0000 7f442d99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.409+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44280636e0 con 0x7f442809d960 2024-09-18T12:29:37.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.410+0000 7f441f7fe640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4408038440 0x7f440803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.410+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6494+0+0 (secure 0 0 0) 0x7f441004b8f0 con 0x7f442809d960 2024-09-18T12:29:37.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.410+0000 7f4426ffd640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4408038440 0x7f440803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.410+0000 7f4426ffd640 10 monclient: get_auth_request con 0x7f4408038440 auth_method 0 2024-09-18T12:29:37.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.412+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:29:37.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.412+0000 7f4426ffd640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4408038440 0x7f440803a8f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4418034000 tx=0x7f4418032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:37.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.414+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4410018ab0 con 0x7f442809d960 2024-09-18T12:29:37.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.414+0000 7f441f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:37.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.414+0000 7f441f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.574+0000 7f442d99d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.574+0000 7f442d99d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.574+0000 7f442d99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.574+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4428064b20 con 0x7f442809d960 2024-09-18T12:29:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.574+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v89) v1 ==== 56+0+1500 (secure 0 0 0) 0x7f4410018690 con 0x7f442809d960 2024-09-18T12:29:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.574+0000 7f441f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.574+0000 7f441f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:37.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.671+0000 7f442d99d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:29:37.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.671+0000 7f442d99d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.671+0000 7f442d99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.671+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f442813f370 con 0x7f442809d960 2024-09-18T12:29:37.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.671+0000 7f441f7fe640 1 -- 172.21.15.144:0/1433864996 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v89) v1 ==== 56+0+1500 (secure 0 0 0) 0x7f4410018690 con 0x7f442809d960 2024-09-18T12:29:37.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.671+0000 7f441f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:29:37.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.671+0000 7f441f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4408038440 msgr2=0x7f440803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4408038440 0x7f440803a8f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4418034000 tx=0x7f4418032000 comp rx=0 tx=0).stop 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 10 monclient: shutdown 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 msgr2=0x7f4428132c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 0x7f4428132c60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f441000b600 tx=0x7f44100091a0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 shutdown_connections 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4408038440 0x7f440803a8f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f44280a3950 0x7f442811c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f442809d960 0x7f4428132c60 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 --2- 172.21.15.144:0/1433864996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f442809c7f0 0x7f442813ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 >> 172.21.15.144:0/1433864996 conn(0x7f4428070fd0 msgr2=0x7f4428096a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:37.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 shutdown_connections 2024-09-18T12:29:37.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.673+0000 7f442d99d640 1 -- 172.21.15.144:0/1433864996 wait complete. 2024-09-18T12:29:37.683 INFO:tasks.workunit.client.0.smithi144.stdout:-14 0 root root-for-test 2024-09-18T12:29:37.684 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm host-for-test 2024-09-18T12:29:37.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.756+0000 7f9861890640 10 monclient: get_monmap_and_config 2024-09-18T12:29:37.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.756+0000 7f9861890640 10 monclient: build_initial_monmap 2024-09-18T12:29:37.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.756+0000 7f9861890640 10 monclient: monmap: 2024-09-18T12:29:37.761 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:37.761 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.761 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:37.757535+0000 2024-09-18T12:29:37.761 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:37.757535+0000 2024-09-18T12:29:37.761 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 1 Processor -- start 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 1 -- start start 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient: init 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:37.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f985c09ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f985c09f020 0x7f985c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient(hunting): picked mon.noname-a con 0x7f985c09f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c136d60 0x7f985c139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient(hunting): picked mon.noname-b con 0x7f985c136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c067420 con 0x7f985c09ec50 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985c0a0480 con 0x7f985c09f020 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0a0200 con 0x7f985c136d60 2024-09-18T12:29:37.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f9861890640 10 monclient(hunting): authenticate will time out at 1013.326727s 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f985affd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f985c09ec50 0x7f985c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51658/0 (socket says 172.21.15.144:51658) 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f985affd640 1 -- 172.21.15.144:0/3099330323 learned_addr learned my addr 172.21.15.144:0/3099330323 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f98527fc640 1 -- 172.21.15.144:0/3099330323 >> v2:172.21.15.116:3300/0 conn(0x7f985c09f020 msgr2=0x7f985c069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.757+0000 7f98527fc640 1 --2- 172.21.15.144:0/3099330323 >> v2:172.21.15.116:3300/0 conn(0x7f985c09f020 0x7f985c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 1 --2- 172.21.15.144:0/3099330323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c136d60 0x7f985c139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): get_auth_request con 0x7f985c136d60 auth_method 0 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:37.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient(hunting): handle_auth_done global_id 64226 payload 274 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 1 -- 172.21.15.144:0/3099330323 >> v2:172.21.15.116:3300/0 conn(0x7f985c09f020 msgr2=0x7f985c069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 1 --2- 172.21.15.144:0/3099330323 >> v2:172.21.15.116:3300/0 conn(0x7f985c09f020 0x7f985c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 1 -- 172.21.15.144:0/3099330323 >> v1:172.21.15.144:6789/0 conn(0x7f985c09ec50 legacy=0x7f985c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:37.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.758+0000 7f985b7fe640 1 -- 172.21.15.144:0/3099330323 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c09fd80 con 0x7f985c136d60 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985b7fe640 1 -- 172.21.15.144:0/3099330323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c136d60 msgr2=0x7f985c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985b7fe640 1 --2- 172.21.15.144:0/3099330323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c136d60 0x7f985c139140 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9854009e60 tx=0x7f9854009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c32709116307424 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 1 -- 172.21.15.144:0/3099330323 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854007840 con 0x7f985c136d60 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: dump: 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.773 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 1 -- 172.21.15.144:0/3099330323 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985402d050 con 0x7f985c136d60 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f9861890640 5 monclient: authenticate success, global_id 64226 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f9861890640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:37.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f9861890640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 1 -- 172.21.15.144:0/3099330323 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854007c10 con 0x7f985c136d60 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f9861890640 4 set_mon_vals no callback set 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.761+0000 7f985a7fc640 10 monclient: dump: 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.775 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.776 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.776 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.776 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.776 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 10 monclient: shutdown 2024-09-18T12:29:37.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:37.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 -- 172.21.15.144:0/3099330323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c136d60 msgr2=0x7f985c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 --2- 172.21.15.144:0/3099330323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c136d60 0x7f985c139140 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9854009e60 tx=0x7f9854009780 comp rx=0 tx=0).stop 2024-09-18T12:29:37.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 -- 172.21.15.144:0/3099330323 shutdown_connections 2024-09-18T12:29:37.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 --2- 172.21.15.144:0/3099330323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c136d60 0x7f985c139140 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 --2- 172.21.15.144:0/3099330323 >> v2:172.21.15.116:3300/0 conn(0x7f985c09f020 0x7f985c069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 -- 172.21.15.144:0/3099330323 >> 172.21.15.144:0/3099330323 conn(0x7f985c0a5cf0 msgr2=0x7f985c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:37.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 -- 172.21.15.144:0/3099330323 shutdown_connections 2024-09-18T12:29:37.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 -- 172.21.15.144:0/3099330323 wait complete. 2024-09-18T12:29:37.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 10 monclient: build_initial_monmap 2024-09-18T12:29:37.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 10 monclient: monmap: 2024-09-18T12:29:37.778 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:37.780 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.762+0000 7f9861890640 1 Processor -- start 2024-09-18T12:29:37.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 1 -- start start 2024-09-18T12:29:37.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient: init 2024-09-18T12:29:37.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 0x7f985c11dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): picked mon.noname-b con 0x7f985c09ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c09f020 0x7f985c1414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): picked mon.noname-c con 0x7f985c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f985c136d60 0x7f985c141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): picked mon.noname-a con 0x7f985c136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c11c830 con 0x7f985c136d60 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c09c020 con 0x7f985c09ec50 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0a4110 con 0x7f985c09f020 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f9861890640 10 monclient(hunting): authenticate will time out at 1013.332472s 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 0x7f985c11dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985b7fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f985c136d60 msgr2=0x7f985c141a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f985c136d60 0x7f985c141a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 0x7f985c11dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42164/0 (socket says 172.21.15.144:42164) 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 -- 172.21.15.144:0/1644155297 learned_addr learned my addr 172.21.15.144:0/1644155297 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): get_auth_request con 0x7f985c09ec50 auth_method 0 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:37.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f98527fc640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c09f020 0x7f985c1414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f98527fc640 10 monclient(hunting): get_auth_request con 0x7f985c09f020 auth_method 0 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f98527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f98527fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f98527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient(hunting): handle_auth_done global_id 5281 payload 824 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 -- 172.21.15.144:0/1644155297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c09f020 msgr2=0x7f985c1414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:37.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c09f020 0x7f985c1414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 -- 172.21.15.144:0/1644155297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f985c136d60 msgr2=0x7f985c141a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:37.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f985c136d60 0x7f985c141a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:37.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:37.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 monclient: found mon.noname-b 2024-09-18T12:29:37.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:37.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.763+0000 7f985affd640 1 -- 172.21.15.144:0/1644155297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9854009a20 con 0x7f985c09ec50 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f985affd640 1 -- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 msgr2=0x7f985c11dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f985affd640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 0x7f985c11dac0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f984c00c5e0 tx=0x7f984c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f985affd640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 0x7f985c11dac0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f984c00c5e0 tx=0x7f984c004960 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f984c005cb0 mon_map magic: 0 v1 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c005cb0 con 0x7f985c09ec50 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 10 monclient: dump: 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.785 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.786 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.786 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.786 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.786 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.786 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:37.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:37.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984c005e30 con 0x7f985c09ec50 2024-09-18T12:29:37.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9861890640 5 monclient: authenticate success, global_id 5281 2024-09-18T12:29:37.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9861890640 10 monclient: _renew_subs 2024-09-18T12:29:37.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9861890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c09b950 con 0x7f985c09ec50 2024-09-18T12:29:37.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f98537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:37.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9859ffb640 4 set_mon_vals no callback set 2024-09-18T12:29:37.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9861890640 10 monclient: _renew_subs 2024-09-18T12:29:37.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9861890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.766+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c141fe0 con 0x7f985c09ec50 2024-09-18T12:29:37.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c007340 con 0x7f985c09ec50 2024-09-18T12:29:37.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:37.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:37.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98537fe640 10 monclient: dump: 2024-09-18T12:29:37.789 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:37.789 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:37.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f984c01b400 con 0x7f985c09ec50 2024-09-18T12:29:37.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f9861890640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:37.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f9861890640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:37.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f9861890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:37.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c0008d0 con 0x7f985c09ec50 2024-09-18T12:29:37.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98537fe640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f983c038490 0x7f983c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:37.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98527fc640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f983c038490 0x7f983c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:37.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.767+0000 7f98527fc640 10 monclient: get_auth_request con 0x7f983c038490 auth_method 0 2024-09-18T12:29:37.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.769+0000 7f98527fc640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f983c038490 0x7f983c03a940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f985c11e000 tx=0x7f9844006ef0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:37.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:37.771+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6494+0+0 (secure 0 0 0) 0x7f984c04bb60 con 0x7f985c09ec50 2024-09-18T12:29:38.390 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.389+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f984c007730 con 0x7f985c09ec50 2024-09-18T12:29:38.390 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.389+0000 7f98537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:38.390 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.389+0000 7f98537fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:38.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.551+0000 7f9861890640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T12:29:38.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.551+0000 7f9861890640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T12:29:38.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.551+0000 7f9861890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:38.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.551+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f985c11b960 con 0x7f985c09ec50 2024-09-18T12:29:38.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.702+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v90) v1 ==== 140+0+0 (secure 0 0 0) 0x7f984c01a360 con 0x7f985c09ec50 2024-09-18T12:29:38.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.702+0000 7f98537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T12:29:38.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.702+0000 7f98537fe640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-18T12:29:38.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.800+0000 7f9861890640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T12:29:38.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.800+0000 7f9861890640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T12:29:38.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.800+0000 7f9861890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:38.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.800+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7f985c072640 con 0x7f985c09ec50 2024-09-18T12:29:38.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.800+0000 7f98537fe640 1 -- 172.21.15.144:0/1644155297 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v90) v1 ==== 140+0+0 (secure 0 0 0) 0x7f984c01a540 con 0x7f985c09ec50 2024-09-18T12:29:38.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.800+0000 7f98537fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T12:29:38.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.800+0000 7f98537fe640 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-18T12:29:38.802 INFO:tasks.workunit.client.0.smithi144.stderr:device 'host-for-test' does not appear in the crush map 2024-09-18T12:29:38.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f983c038490 msgr2=0x7f983c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:38.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f983c038490 0x7f983c03a940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f985c11e000 tx=0x7f9844006ef0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 10 monclient: shutdown 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 msgr2=0x7f985c11dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 0x7f985c11dac0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f984c00c5e0 tx=0x7f984c004960 comp rx=0 tx=0).stop 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 shutdown_connections 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f983c038490 0x7f983c03a940 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f985c136d60 0x7f985c141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f985c09f020 0x7f985c1414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 --2- 172.21.15.144:0/1644155297 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f985c09ec50 0x7f985c11dac0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 >> 172.21.15.144:0/1644155297 conn(0x7f985c0a5cf0 msgr2=0x7f985c09d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 shutdown_connections 2024-09-18T12:29:38.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.802+0000 7f9861890640 1 -- 172.21.15.144:0/1644155297 wait complete. 2024-09-18T12:29:38.812 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm rack-for-test 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.883+0000 7fd1588e8640 10 monclient: get_monmap_and_config 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.883+0000 7fd1588e8640 10 monclient: build_initial_monmap 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.884+0000 7fd1588e8640 10 monclient: monmap: 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:38.884909+0000 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:38.884909+0000 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:38.885 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:38.886 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:38.886 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:38.886 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:38.886 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:38.886 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.884+0000 7fd1588e8640 1 Processor -- start 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.884+0000 7fd1588e8640 1 -- start start 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.884+0000 7fd1588e8640 10 monclient: init 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd15406e130 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd15406a790 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd15406e500 0x7fd15409e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd15406e500 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15406a0b0 con 0x7fd15406e130 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd154068c30 con 0x7fd15406a790 2024-09-18T12:29:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd154069750 con 0x7fd15406e500 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd1588e8640 10 monclient(hunting): authenticate will time out at 1014.454313s 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd15406e500 0x7fd15409e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd15259c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fd15406e130 legacy=0x7fd15409da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd15406e500 0x7fd15409e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42166/0 (socket says 172.21.15.144:42166) 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 1 -- 172.21.15.144:0/1244898649 learned_addr learned my addr 172.21.15.144:0/1244898649 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 10 monclient(hunting): get_auth_request con 0x7fd15406e500 auth_method 0 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd15159a640 1 -- 172.21.15.144:0/1244898649 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:29:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.885+0000 7fd152d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd151d9b640 1 -- 172.21.15.144:0/1244898649 >> v1:172.21.15.144:6789/0 conn(0x7fd15406a790 legacy=0x7fd15409e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 10 monclient(hunting): handle_auth_done global_id 5284 payload 274 2024-09-18T12:29:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 1 -- 172.21.15.144:0/1244898649 >> v1:172.21.15.144:6789/0 conn(0x7fd15406a790 legacy=0x7fd15409e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 1 -- 172.21.15.144:0/1244898649 >> v1:172.21.15.116:6789/0 conn(0x7fd15406e130 legacy=0x7fd15409da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:29:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 1 monclient: found mon.noname-c 2024-09-18T12:29:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.887+0000 7fd152d9d640 1 -- 172.21.15.144:0/1244898649 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154069a60 con 0x7fd15406e500 2024-09-18T12:29:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.890+0000 7fd152d9d640 1 -- 172.21.15.144:0/1244898649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406e500 msgr2=0x7fd15409e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.890+0000 7fd152d9d640 1 --2- 172.21.15.144:0/1244898649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406e500 0x7fd15409e880 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009fa0 tx=0x7fd14c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be95b7643422413c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.890+0000 7fd15159a640 1 -- 172.21.15.144:0/1244898649 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c007610 con 0x7fd15406e500 2024-09-18T12:29:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.890+0000 7fd15159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 10 monclient: dump: 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 10 monclient: _finish_auth 0 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 1 -- 172.21.15.144:0/1244898649 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c02d050 con 0x7fd15406e500 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 5 monclient: authenticate success, global_id 5284 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:38.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 1 -- 172.21.15.144:0/1244898649 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0079c0 con 0x7fd15406e500 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 4 set_mon_vals no callback set 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd15159a640 10 monclient: dump: 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:38.898 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 10 monclient: shutdown 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 1 -- 172.21.15.144:0/1244898649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406e500 msgr2=0x7fd15409e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 1 --2- 172.21.15.144:0/1244898649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406e500 0x7fd15409e880 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009fa0 tx=0x7fd14c009840 comp rx=0 tx=0).stop 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 1 -- 172.21.15.144:0/1244898649 shutdown_connections 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 1 --2- 172.21.15.144:0/1244898649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406e500 0x7fd15409e880 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 1 -- 172.21.15.144:0/1244898649 >> 172.21.15.144:0/1244898649 conn(0x7fd154071340 msgr2=0x7fd1540a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 1 -- 172.21.15.144:0/1244898649 shutdown_connections 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.891+0000 7fd1588e8640 1 -- 172.21.15.144:0/1244898649 wait complete. 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient: build_initial_monmap 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient: monmap: 2024-09-18T12:29:38.899 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 Processor -- start 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 -- start start 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient: init 2024-09-18T12:29:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 0x7fd154134da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd15406a790 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd15406e130 0x7fd15411e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): picked mon.noname-a con 0x7fd15406e130 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd15406e500 0x7fd15411ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd15406e500 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15406e970 con 0x7fd15406e130 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15406de60 con 0x7fd15406a790 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541221d0 con 0x7fd15406e500 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd1588e8640 10 monclient(hunting): authenticate will time out at 1014.461897s 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd15259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 0x7fd154134da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd15259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 0x7fd154134da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42178/0 (socket says 172.21.15.144:42178) 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd15259c640 1 -- 172.21.15.144:0/952895942 learned_addr learned my addr 172.21.15.144:0/952895942 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd15259c640 10 monclient(hunting): get_auth_request con 0x7fd15406a790 auth_method 0 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd15259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd15259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.892+0000 7fd15259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd151d9b640 1 -- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd15406e130 msgr2=0x7fd15411e8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd151d9b640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd15406e130 0x7fd15411e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 10 monclient(hunting): handle_auth_done global_id 5287 payload 824 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 1 -- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd15406e500 msgr2=0x7fd15411ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd15406e500 0x7fd15411ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 1 -- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd15406e130 msgr2=0x7fd15411e8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd15406e130 0x7fd15411e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 1 monclient: found mon.noname-b 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.893+0000 7fd15259c640 1 -- 172.21.15.144:0/952895942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c009ae0 con 0x7fd15406a790 2024-09-18T12:29:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd15259c640 1 -- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 msgr2=0x7fd154134da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd15259c640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 0x7fd154134da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd14400b600 tx=0x7fd1440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144015070 con 0x7fd15406a790 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: dump: 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd144003d90 con 0x7fd15406a790 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1588e8640 5 monclient: authenticate success, global_id 5287 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144006380 con 0x7fd15406a790 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1588e8640 10 monclient: _renew_subs 2024-09-18T12:29:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1588e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd154121ec0 con 0x7fd15406a790 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd150d99640 4 set_mon_vals no callback set 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1427fc640 10 monclient: dump: 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:38.908 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1588e8640 10 monclient: _renew_subs 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1588e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.895+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd154120f60 con 0x7fd15406a790 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd144010440 con 0x7fd15406a790 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd1427fc640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd134038440 0x7fd13403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6311+0+0 (secure 0 0 0) 0x7fd14404b9b0 con 0x7fd15406a790 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd1588e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd1588e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd1588e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:38.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd151d9b640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd134038440 0x7fd13403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:38.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd154122310 con 0x7fd15406a790 2024-09-18T12:29:38.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.896+0000 7fd151d9b640 10 monclient: get_auth_request con 0x7fd134038440 auth_method 0 2024-09-18T12:29:38.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.899+0000 7fd151d9b640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd134038440 0x7fd13403a8f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd15406b8f0 tx=0x7fd148007a50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:38.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:38.901+0000 7fd15259c640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 0x7fd154134da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd14400b600 tx=0x7fd1440091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fd144018ab0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:29:39.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:39.729+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd144018ab0 con 0x7fd15406a790 2024-09-18T12:29:39.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:39.729+0000 7fd1427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:39.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:39.729+0000 7fd1427fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:39.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:39.892+0000 7fd1588e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T12:29:39.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:39.892+0000 7fd1588e8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T12:29:39.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:39.892+0000 7fd1588e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:39.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:39.892+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7fd1540008d0 con 0x7fd15406a790 2024-09-18T12:29:40.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.711+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v91) v1 ==== 140+0+0 (secure 0 0 0) 0x7fd144018690 con 0x7fd15406a790 2024-09-18T12:29:40.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.711+0000 7fd1427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T12:29:40.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.711+0000 7fd1427fc640 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-18T12:29:40.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.812+0000 7fd1588e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T12:29:40.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.812+0000 7fd1588e8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T12:29:40.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.812+0000 7fd1588e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:40.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.812+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7fd154072640 con 0x7fd15406a790 2024-09-18T12:29:40.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.812+0000 7fd1427fc640 1 -- 172.21.15.144:0/952895942 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v91) v1 ==== 140+0+0 (secure 0 0 0) 0x7fd144018690 con 0x7fd15406a790 2024-09-18T12:29:40.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.812+0000 7fd1427fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T12:29:40.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.812+0000 7fd1427fc640 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-18T12:29:40.814 INFO:tasks.workunit.client.0.smithi144.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-18T12:29:40.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd134038440 msgr2=0x7fd13403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd134038440 0x7fd13403a8f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd15406b8f0 tx=0x7fd148007a50 comp rx=0 tx=0).stop 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 10 monclient: shutdown 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 msgr2=0x7fd154134da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 0x7fd154134da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd14400b600 tx=0x7fd1440091a0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 shutdown_connections 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd134038440 0x7fd13403a8f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd15406e500 0x7fd15411ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd15406e130 0x7fd15411e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 --2- 172.21.15.144:0/952895942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd15406a790 0x7fd154134da0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 >> 172.21.15.144:0/952895942 conn(0x7fd154071340 msgr2=0x7fd15409c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:40.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 shutdown_connections 2024-09-18T12:29:40.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.814+0000 7fd1588e8640 1 -- 172.21.15.144:0/952895942 wait complete. 2024-09-18T12:29:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm root-for-test 2024-09-18T12:29:40.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.897+0000 7f158286f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:40.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.897+0000 7f158286f640 10 monclient: build_initial_monmap 2024-09-18T12:29:40.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.897+0000 7f158286f640 10 monclient: monmap: 2024-09-18T12:29:40.898 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:40.898 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:40.898161+0000 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:40.898161+0000 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:40.899 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.897+0000 7f158286f640 1 Processor -- start 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 1 -- start start 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient: init 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f157c09ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f157c09f020 0x7f157c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:40.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient(hunting): picked mon.noname-a con 0x7f157c09f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c136d60 0x7f157c139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient(hunting): picked mon.noname-b con 0x7f157c136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c067420 con 0x7f157c09ec50 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157c0a0480 con 0x7f157c09f020 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c0a0200 con 0x7f157c136d60 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f158286f640 10 monclient(hunting): authenticate will time out at 1016.467473s 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f157bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f157c09ec50 0x7f157c131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51680/0 (socket says 172.21.15.144:51680) 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f157bfff640 1 -- 172.21.15.144:0/2670374835 learned_addr learned my addr 172.21.15.144:0/2670374835 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:40.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f157b7fe640 1 -- 172.21.15.144:0/2670374835 >> v2:172.21.15.116:3300/0 conn(0x7f157c09f020 msgr2=0x7f157c069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f157b7fe640 1 --2- 172.21.15.144:0/2670374835 >> v2:172.21.15.116:3300/0 conn(0x7f157c09f020 0x7f157c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f1580e0c640 1 --2- 172.21.15.144:0/2670374835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c136d60 0x7f157c139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f1580e0c640 10 monclient(hunting): get_auth_request con 0x7f157c136d60 auth_method 0 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f1580e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f1580e0c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.898+0000 7f1580e0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 10 monclient(hunting): handle_auth_done global_id 64232 payload 274 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 1 -- 172.21.15.144:0/2670374835 >> v2:172.21.15.116:3300/0 conn(0x7f157c09f020 msgr2=0x7f157c069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 1 --2- 172.21.15.144:0/2670374835 >> v2:172.21.15.116:3300/0 conn(0x7f157c09f020 0x7f157c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 1 -- 172.21.15.144:0/2670374835 >> v1:172.21.15.144:6789/0 conn(0x7f157c09ec50 legacy=0x7f157c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:40.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:40.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 1 monclient: found mon.noname-b 2024-09-18T12:29:40.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:40.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:40.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.899+0000 7f1580e0c640 1 -- 172.21.15.144:0/2670374835 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c09fd80 con 0x7f157c136d60 2024-09-18T12:29:40.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f1580e0c640 1 -- 172.21.15.144:0/2670374835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c136d60 msgr2=0x7f157c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:40.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f1580e0c640 1 --2- 172.21.15.144:0/2670374835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c136d60 0x7f157c139140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1574009e60 tx=0x7f1574009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=238c2c33012594c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:40.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 1 -- 172.21.15.144:0/2670374835 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574007840 con 0x7f157c136d60 2024-09-18T12:29:40.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: dump: 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:40.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: _finish_auth 0 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 5 monclient: authenticate success, global_id 64232 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 1 -- 172.21.15.144:0/2670374835 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157402d050 con 0x7f157c136d60 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 4 set_mon_vals no callback set 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 1 -- 172.21.15.144:0/2670374835 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574007c10 con 0x7f157c136d60 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f157affd640 10 monclient: dump: 2024-09-18T12:29:40.912 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 10 monclient: shutdown 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 1 -- 172.21.15.144:0/2670374835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c136d60 msgr2=0x7f157c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 1 --2- 172.21.15.144:0/2670374835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c136d60 0x7f157c139140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1574009e60 tx=0x7f1574009780 comp rx=0 tx=0).stop 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 1 -- 172.21.15.144:0/2670374835 shutdown_connections 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 1 --2- 172.21.15.144:0/2670374835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c136d60 0x7f157c139140 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 1 --2- 172.21.15.144:0/2670374835 >> v2:172.21.15.116:3300/0 conn(0x7f157c09f020 0x7f157c069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 1 -- 172.21.15.144:0/2670374835 >> 172.21.15.144:0/2670374835 conn(0x7f157c0a5cf0 msgr2=0x7f157c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.902+0000 7f158286f640 1 -- 172.21.15.144:0/2670374835 shutdown_connections 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.903+0000 7f158286f640 1 -- 172.21.15.144:0/2670374835 wait complete. 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.903+0000 7f158286f640 10 monclient: build_initial_monmap 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.903+0000 7f158286f640 10 monclient: monmap: 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:40.914 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.903+0000 7f158286f640 1 Processor -- start 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.903+0000 7f158286f640 1 -- start start 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.903+0000 7f158286f640 10 monclient: init 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c09ec50 0x7f157c11dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:40.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): picked mon.noname-a con 0x7f157c09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c09f020 0x7f157c1414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): picked mon.noname-c con 0x7f157c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 0x7f157c141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): picked mon.noname-b con 0x7f157c136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c11c830 con 0x7f157c09ec50 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c09c020 con 0x7f157c136d60 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c0a4110 con 0x7f157c09f020 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:40.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f158286f640 10 monclient(hunting): authenticate will time out at 1016.473354s 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 0x7f157c141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157bfff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c09ec50 msgr2=0x7f157c11dac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c09ec50 0x7f157c11dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 0x7f157c141a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42186/0 (socket says 172.21.15.144:42186) 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 -- 172.21.15.144:0/2202524673 learned_addr learned my addr 172.21.15.144:0/2202524673 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): get_auth_request con 0x7f157c136d60 auth_method 0 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:40.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c09f020 0x7f157c1414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 10 monclient(hunting): get_auth_request con 0x7f157c09f020 auth_method 0 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f157b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient(hunting): handle_auth_done global_id 5290 payload 824 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 -- 172.21.15.144:0/2202524673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c09f020 msgr2=0x7f157c1414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c09f020 0x7f157c1414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 -- 172.21.15.144:0/2202524673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c09ec50 msgr2=0x7f157c11dac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:40.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c09ec50 0x7f157c11dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 monclient: found mon.noname-b 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.904+0000 7f1580e0c640 1 -- 172.21.15.144:0/2202524673 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1574009a20 con 0x7f157c136d60 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1580e0c640 1 -- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 msgr2=0x7f157c141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1580e0c640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 0x7f157c141a20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1574009e60 tx=0x7f1574037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574031cd0 con 0x7f157c136d60 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: dump: 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:40.919 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f158286f640 5 monclient: authenticate success, global_id 5290 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1574040070 con 0x7f157c136d60 2024-09-18T12:29:40.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574045640 con 0x7f157c136d60 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f158286f640 10 monclient: _renew_subs 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f158286f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157c09b950 con 0x7f157c136d60 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f157a7fc640 4 set_mon_vals no callback set 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f1563fff640 10 monclient: dump: 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.921 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f158286f640 10 monclient: _renew_subs 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f158286f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.907+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157c141f60 con 0x7f157c136d60 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1574042030 con 0x7f157c136d60 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f158286f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f158286f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:40.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f158286f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157c0008d0 con 0x7f157c136d60 2024-09-18T12:29:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f1563fff640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1568038440 0x7f156803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6128+0+0 (secure 0 0 0) 0x7f157406d740 con 0x7f157c136d60 2024-09-18T12:29:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f157bfff640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1568038440 0x7f156803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.908+0000 7f157bfff640 10 monclient: get_auth_request con 0x7f1568038440 auth_method 0 2024-09-18T12:29:40.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.910+0000 7f157bfff640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1568038440 0x7f156803a8f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f156c0047d0 tx=0x7f156c007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:40.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.912+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1574031e30 con 0x7f157c136d60 2024-09-18T12:29:40.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.912+0000 7f1563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:40.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:40.912+0000 7f1563fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:41.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.070+0000 7f158286f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T12:29:41.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.070+0000 7f158286f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T12:29:41.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.070+0000 7f158286f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:41.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.070+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f157c11b960 con 0x7f157c136d60 2024-09-18T12:29:41.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.728+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v92) v1 ==== 140+0+0 (secure 0 0 0) 0x7f15740330c0 con 0x7f157c136d60 2024-09-18T12:29:41.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.728+0000 7f1563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T12:29:41.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.728+0000 7f1563fff640 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-18T12:29:41.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.826+0000 7f158286f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T12:29:41.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.826+0000 7f158286f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T12:29:41.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.826+0000 7f158286f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:41.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.826+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f157c11b960 con 0x7f157c136d60 2024-09-18T12:29:41.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.826+0000 7f1563fff640 1 -- 172.21.15.144:0/2202524673 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v92) v1 ==== 140+0+0 (secure 0 0 0) 0x7f15740393c0 con 0x7f157c136d60 2024-09-18T12:29:41.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.826+0000 7f1563fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T12:29:41.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.826+0000 7f1563fff640 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-18T12:29:41.828 INFO:tasks.workunit.client.0.smithi144.stderr:device 'root-for-test' does not appear in the crush map 2024-09-18T12:29:41.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1568038440 msgr2=0x7f156803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:41.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1568038440 0x7f156803a8f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f156c0047d0 tx=0x7f156c007b80 comp rx=0 tx=0).stop 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 10 monclient: shutdown 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 msgr2=0x7f157c141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 0x7f157c141a20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1574009e60 tx=0x7f1574037000 comp rx=0 tx=0).stop 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 shutdown_connections 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1568038440 0x7f156803a8f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f157c136d60 0x7f157c141a20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f157c09f020 0x7f157c1414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 --2- 172.21.15.144:0/2202524673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c09ec50 0x7f157c11dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 >> 172.21.15.144:0/2202524673 conn(0x7f157c0a5cf0 msgr2=0x7f157c09d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 shutdown_connections 2024-09-18T12:29:41.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.828+0000 7f158286f640 1 -- 172.21.15.144:0/2202524673 wait complete. 2024-09-18T12:29:41.838 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.913+0000 7f96f026f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.913+0000 7f96f026f640 10 monclient: build_initial_monmap 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.913+0000 7f96f026f640 10 monclient: monmap: 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:41.914843+0000 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:41.914843+0000 2024-09-18T12:29:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:41.916 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.914+0000 7f96f026f640 1 Processor -- start 2024-09-18T12:29:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.914+0000 7f96f026f640 1 -- start start 2024-09-18T12:29:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.914+0000 7f96f026f640 10 monclient: init 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96e8131c90 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96e809d210 0x7f96e8069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient(hunting): picked mon.noname-c con 0x7f96e809d210 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96e80a0ab0 0x7f96e8138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient(hunting): picked mon.noname-a con 0x7f96e80a0ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e80a1800 con 0x7f96e8131c90 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e809fea0 con 0x7f96e80a0ab0 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e80a1520 con 0x7f96e809d210 2024-09-18T12:29:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96f026f640 10 monclient(hunting): authenticate will time out at 1017.484303s 2024-09-18T12:29:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ee00b640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f96e8131c90 legacy=0x7f96e8132060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ee80c640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f96e80a0ab0 msgr2=0x7f96e8138fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ee80c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96e80a0ab0 0x7f96e8138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96e809d210 0x7f96e8069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96e809d210 0x7f96e8069e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42200/0 (socket says 172.21.15.144:42200) 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 -- 172.21.15.144:0/3524225614 learned_addr learned my addr 172.21.15.144:0/3524225614 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): get_auth_request con 0x7f96e809d210 auth_method 0 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient(hunting): handle_auth_done global_id 5293 payload 274 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 -- 172.21.15.144:0/3524225614 >> v2:172.21.15.116:3300/0 conn(0x7f96e80a0ab0 msgr2=0x7f96e8138fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 --2- 172.21.15.144:0/3524225614 >> v2:172.21.15.116:3300/0 conn(0x7f96e80a0ab0 0x7f96e8138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 -- 172.21.15.144:0/3524225614 >> v1:172.21.15.116:6789/0 conn(0x7f96e8131c90 legacy=0x7f96e8132060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient: _finish_hunting 0 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 monclient: found mon.noname-c 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.915+0000 7f96ed80a640 1 -- 172.21.15.144:0/3524225614 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8132770 con 0x7f96e809d210 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed80a640 1 -- 172.21.15.144:0/3524225614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809d210 msgr2=0x7f96e8069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed80a640 1 --2- 172.21.15.144:0/3524225614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809d210 0x7f96e8069e10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96e80a1d20 tx=0x7f96dc0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eec1e9e03ed7e1b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 1 -- 172.21.15.144:0/3524225614 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc032440 con 0x7f96e809d210 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: dump: 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: _finish_auth 0 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:41.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 1 -- 172.21.15.144:0/3524225614 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc0325a0 con 0x7f96e809d210 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96f026f640 5 monclient: authenticate success, global_id 5293 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96f026f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96f026f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 1 -- 172.21.15.144:0/3524225614 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc032980 con 0x7f96e809d210 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96f026f640 4 set_mon_vals no callback set 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.918+0000 7f96ed009640 10 monclient: dump: 2024-09-18T12:29:41.928 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 10 monclient: shutdown 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 -- 172.21.15.144:0/3524225614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809d210 msgr2=0x7f96e8069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:41.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 --2- 172.21.15.144:0/3524225614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809d210 0x7f96e8069e10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96e80a1d20 tx=0x7f96dc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 -- 172.21.15.144:0/3524225614 shutdown_connections 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 --2- 172.21.15.144:0/3524225614 >> v2:172.21.15.116:3300/0 conn(0x7f96e80a0ab0 0x7f96e8138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 --2- 172.21.15.144:0/3524225614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809d210 0x7f96e8069e10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 -- 172.21.15.144:0/3524225614 >> 172.21.15.144:0/3524225614 conn(0x7f96e8071370 msgr2=0x7f96e80a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 -- 172.21.15.144:0/3524225614 shutdown_connections 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 1 -- 172.21.15.144:0/3524225614 wait complete. 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 10 monclient: build_initial_monmap 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.919+0000 7f96f026f640 10 monclient: monmap: 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:41.930 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 Processor -- start 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 -- start start 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient: init 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:41.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e80a0ab0 0x7f96e8096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): picked mon.noname-a con 0x7f96e80a0ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96e8131c90 0x7f96e8096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): picked mon.noname-c con 0x7f96e8131c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 0x7f96e80974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): picked mon.noname-b con 0x7f96e809aab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e8097e10 con 0x7f96e80a0ab0 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e80989d0 con 0x7f96e809aab0 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e809a080 con 0x7f96e8131c90 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:41.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96f026f640 10 monclient(hunting): authenticate will time out at 1017.489515s 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 0x7f96e80974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee00b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e80a0ab0 msgr2=0x7f96e8096a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 0x7f96e80974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42214/0 (socket says 172.21.15.144:42214) 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 -- 172.21.15.144:0/2717042585 learned_addr learned my addr 172.21.15.144:0/2717042585 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): get_auth_request con 0x7f96e809aab0 auth_method 0 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee00b640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e80a0ab0 0x7f96e8096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:41.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient(hunting): handle_auth_done global_id 5296 payload 824 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ed80a640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96e8131c90 0x7f96e8096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 -- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96e8131c90 msgr2=0x7f96e8096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96e8131c90 0x7f96e8096f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 -- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e80a0ab0 msgr2=0x7f96e8096a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e80a0ab0 0x7f96e8096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 monclient: found mon.noname-b 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:41.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:41.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.920+0000 7f96ee80c640 1 -- 172.21.15.144:0/2717042585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc009450 con 0x7f96e809aab0 2024-09-18T12:29:41.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96ee80c640 1 -- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 msgr2=0x7f96e80974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:41.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96ee80c640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 0x7f96e80974b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96d800bbb0 tx=0x7f96d80097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:41.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d800cd90 con 0x7f96e809aab0 2024-09-18T12:29:41.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:41.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: dump: 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:41.938 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d8006150 con 0x7f96e809aab0 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96f026f640 5 monclient: authenticate success, global_id 5296 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96f026f640 10 monclient: _renew_subs 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8006300 con 0x7f96e809aab0 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96f026f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e8099ac0 con 0x7f96e809aab0 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:41.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96e67fc640 10 monclient: dump: 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:41.940 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96ec808640 4 set_mon_vals no callback set 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96f026f640 10 monclient: _renew_subs 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96f026f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.923+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e80996a0 con 0x7f96e809aab0 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f96d8012020 con 0x7f96e809aab0 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96f026f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96f026f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:41.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96f026f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:41.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e80008d0 con 0x7f96e809aab0 2024-09-18T12:29:41.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96e67fc640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96d4038440 0x7f96d403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:41.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f96d804bd60 con 0x7f96e809aab0 2024-09-18T12:29:41.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96ee00b640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96d4038440 0x7f96d403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:41.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.924+0000 7f96ee00b640 10 monclient: get_auth_request con 0x7f96d4038440 auth_method 0 2024-09-18T12:29:41.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.926+0000 7f96ee00b640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96d4038440 0x7f96d403a8f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f96e00038f0 tx=0x7f96e0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:41.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.928+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f96d801dd30 con 0x7f96e809aab0 2024-09-18T12:29:41.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.928+0000 7f96e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:41.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:41.928+0000 7f96e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:42.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:42.084+0000 7f96f026f640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T12:29:42.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:42.084+0000 7f96f026f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T12:29:42.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:42.084+0000 7f96f026f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:42.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:42.084+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f96e8064a20 con 0x7f96e809aab0 2024-09-18T12:29:43.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.351+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v93) v1 ==== 145+0+0 (secure 0 0 0) 0x7f96d801d910 con 0x7f96e809aab0 2024-09-18T12:29:43.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.352+0000 7f96e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T12:29:43.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.352+0000 7f96e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:43.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.450+0000 7f96f026f640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T12:29:43.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.450+0000 7f96f026f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T12:29:43.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.450+0000 7f96f026f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.450+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f96e8098b60 con 0x7f96e809aab0 2024-09-18T12:29:43.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.450+0000 7f96e67fc640 1 -- 172.21.15.144:0/2717042585 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v93) v1 ==== 173+0+0 (secure 0 0 0) 0x7f96d801d910 con 0x7f96e809aab0 2024-09-18T12:29:43.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.450+0000 7f96e67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T12:29:43.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.450+0000 7f96e67fc640 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-18T12:29:43.452 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo-rule already exists 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96d4038440 msgr2=0x7f96d403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96d4038440 0x7f96d403a8f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f96e00038f0 tx=0x7f96e0006d80 comp rx=0 tx=0).stop 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 10 monclient: shutdown 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 msgr2=0x7f96e80974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 0x7f96e80974b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f96d800bbb0 tx=0x7f96d80097b0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 shutdown_connections 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96d4038440 0x7f96d403a8f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96e809aab0 0x7f96e80974b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96e8131c90 0x7f96e8096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 --2- 172.21.15.144:0/2717042585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e80a0ab0 0x7f96e8096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 >> 172.21.15.144:0/2717042585 conn(0x7f96e8071370 msgr2=0x7f96e80a3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:43.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.452+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 shutdown_connections 2024-09-18T12:29:43.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.453+0000 7f96f026f640 1 -- 172.21.15.144:0/2717042585 wait complete. 2024-09-18T12:29:43.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush rm foo 2024-09-18T12:29:43.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:29:43.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm foo 2024-09-18T12:29:43.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.535+0000 7fe5f9857640 10 monclient: get_monmap_and_config 2024-09-18T12:29:43.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.535+0000 7fe5f9857640 10 monclient: build_initial_monmap 2024-09-18T12:29:43.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.535+0000 7fe5f9857640 10 monclient: monmap: 2024-09-18T12:29:43.537 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:43.537 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.537 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:43.536630+0000 2024-09-18T12:29:43.537 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:43.536630+0000 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 1 Processor -- start 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 1 -- start start 2024-09-18T12:29:43.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient: init 2024-09-18T12:29:43.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5f409ec50 0x7fe5f4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient(hunting): picked mon.noname-a con 0x7fe5f409ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5f41321a0 0x7fe5f4068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient(hunting): picked mon.noname-c con 0x7fe5f41321a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5f409f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f4067420 con 0x7fe5f409f020 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f40a0480 con 0x7fe5f409ec50 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f40a0200 con 0x7fe5f41321a0 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:43.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f9857640 10 monclient(hunting): authenticate will time out at 1019.105848s 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5f41321a0 0x7fe5f4068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.536+0000 7fe5f2ffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fe5f409ec50 msgr2=0x7fe5f4131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5f41321a0 0x7fe5f4068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42222/0 (socket says 172.21.15.144:42222) 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f27fc640 1 -- 172.21.15.144:0/1065421380 learned_addr learned my addr 172.21.15.144:0/1065421380 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f27fc640 10 monclient(hunting): get_auth_request con 0x7fe5f41321a0 auth_method 0 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f2ffd640 1 --2- 172.21.15.144:0/1065421380 >> v2:172.21.15.116:3300/0 conn(0x7fe5f409ec50 0x7fe5f4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.537+0000 7fe5f37fe640 1 --1- 172.21.15.144:0/1065421380 >> v1:172.21.15.144:6789/0 conn(0x7fe5f409f020 0x7fe5f4069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51694/0 (socket says 172.21.15.144:51694) 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f37fe640 1 -- 172.21.15.144:0/1065421380 >> v1:172.21.15.144:6789/0 conn(0x7fe5f409f020 legacy=0x7fe5f4069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:43.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_done global_id 5299 payload 274 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 1 -- 172.21.15.144:0/1065421380 >> v2:172.21.15.116:3300/0 conn(0x7fe5f409ec50 msgr2=0x7fe5f4131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 1 --2- 172.21.15.144:0/1065421380 >> v2:172.21.15.116:3300/0 conn(0x7fe5f409ec50 0x7fe5f4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 1 -- 172.21.15.144:0/1065421380 >> v1:172.21.15.144:6789/0 conn(0x7fe5f409f020 legacy=0x7fe5f4069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 1 monclient: found mon.noname-c 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.539+0000 7fe5f27fc640 1 -- 172.21.15.144:0/1065421380 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f409fd80 con 0x7fe5f41321a0 2024-09-18T12:29:43.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f27fc640 1 -- 172.21.15.144:0/1065421380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f41321a0 msgr2=0x7fe5f4068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:43.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f27fc640 1 --2- 172.21.15.144:0/1065421380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f41321a0 0x7fe5f4068d00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe5f41325a0 tx=0x7fe5e80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b27b2c0e1f28aa50 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:43.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 1 -- 172.21.15.144:0/1065421380 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e802f4b0 con 0x7fe5f41321a0 2024-09-18T12:29:43.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:43.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: dump: 2024-09-18T12:29:43.550 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 5 monclient: authenticate success, global_id 5299 2024-09-18T12:29:43.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 1 -- 172.21.15.144:0/1065421380 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e802f610 con 0x7fe5f41321a0 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 4 set_mon_vals no callback set 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 1 -- 172.21.15.144:0/1065421380 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e802f9b0 con 0x7fe5f41321a0 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f1ffb640 10 monclient: dump: 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.552 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 10 monclient: shutdown 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 -- 172.21.15.144:0/1065421380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f41321a0 msgr2=0x7fe5f4068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 --2- 172.21.15.144:0/1065421380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f41321a0 0x7fe5f4068d00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe5f41325a0 tx=0x7fe5e80091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 -- 172.21.15.144:0/1065421380 shutdown_connections 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 --2- 172.21.15.144:0/1065421380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f41321a0 0x7fe5f4068d00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 --2- 172.21.15.144:0/1065421380 >> v2:172.21.15.116:3300/0 conn(0x7fe5f409ec50 0x7fe5f4131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 -- 172.21.15.144:0/1065421380 >> 172.21.15.144:0/1065421380 conn(0x7fe5f40a5cf0 msgr2=0x7fe5f40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 -- 172.21.15.144:0/1065421380 shutdown_connections 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.542+0000 7fe5f9857640 1 -- 172.21.15.144:0/1065421380 wait complete. 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.543+0000 7fe5f9857640 10 monclient: build_initial_monmap 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.543+0000 7fe5f9857640 10 monclient: monmap: 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:43.554 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.543+0000 7fe5f9857640 1 Processor -- start 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.543+0000 7fe5f9857640 1 -- start start 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.543+0000 7fe5f9857640 10 monclient: init 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5f409ec50 0x7fe5f409ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): picked mon.noname-c con 0x7fe5f409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f409f020 0x7fe5f4130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): picked mon.noname-b con 0x7fe5f409f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f41321a0 0x7fe5f41310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): picked mon.noname-a con 0x7fe5f41321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f409c020 con 0x7fe5f41321a0 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f409e140 con 0x7fe5f409f020 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f411c020 con 0x7fe5f409ec50 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f9857640 10 monclient(hunting): authenticate will time out at 1019.113311s 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f409f020 0x7fe5f4130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f409f020 0x7fe5f4130ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42224/0 (socket says 172.21.15.144:42224) 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f37fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f41321a0 msgr2=0x7fe5f41310e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 -- 172.21.15.144:0/2374810976 learned_addr learned my addr 172.21.15.144:0/2374810976 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): get_auth_request con 0x7fe5f409f020 auth_method 0 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f37fe640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f41321a0 0x7fe5f41310e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f2ffd640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5f409ec50 0x7fe5f409ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f2ffd640 10 monclient(hunting): get_auth_request con 0x7fe5f409ec50 auth_method 0 2024-09-18T12:29:43.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient(hunting): handle_auth_done global_id 5305 payload 824 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 -- 172.21.15.144:0/2374810976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5f409ec50 msgr2=0x7fe5f409ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5f409ec50 0x7fe5f409ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 -- 172.21.15.144:0/2374810976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f41321a0 msgr2=0x7fe5f41310e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f41321a0 0x7fe5f41310e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 monclient: found mon.noname-b 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.544+0000 7fe5f27fc640 1 -- 172.21.15.144:0/2374810976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8009450 con 0x7fe5f409f020 2024-09-18T12:29:43.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.546+0000 7fe5f27fc640 1 -- 172.21.15.144:0/2374810976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f409f020 msgr2=0x7fe5f4130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.546+0000 7fe5f27fc640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5f409f020 0x7fe5f4130ba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8035040 tx=0x7fe5e8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 1 -- 172.21.15.144:0/2374810976 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8032960 con 0x7fe5f409f020 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: dump: 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.559 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f9857640 5 monclient: authenticate success, global_id 5305 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 1 -- 172.21.15.144:0/2374810976 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e8032b00 con 0x7fe5f409f020 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 1 -- 172.21.15.144:0/2374810976 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e803b400 con 0x7fe5f409f020 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f9857640 10 monclient: _renew_subs 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f9857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f9857640 1 -- 172.21.15.144:0/2374810976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f411dcc0 con 0x7fe5f409f020 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f17fa640 4 set_mon_vals no callback set 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5daffd640 10 monclient: dump: 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f9857640 10 monclient: _renew_subs 2024-09-18T12:29:43.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f9857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.547+0000 7fe5f9857640 1 -- 172.21.15.144:0/2374810976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f411bc30 con 0x7fe5f409f020 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5f9857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5f9857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5f9857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5daffd640 1 -- 172.21.15.144:0/2374810976 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe5e8039040 con 0x7fe5f409f020 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5f9857640 1 -- 172.21.15.144:0/2374810976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f40008d0 con 0x7fe5f409f020 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5daffd640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe5e0038440 0x7fe5e003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5daffd640 1 -- 172.21.15.144:0/2374810976 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe5e806c030 con 0x7fe5f409f020 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5f2ffd640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe5e0038440 0x7fe5e003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.548+0000 7fe5f2ffd640 10 monclient: get_auth_request con 0x7fe5e0038440 auth_method 0 2024-09-18T12:29:43.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.550+0000 7fe5f2ffd640 1 --2- 172.21.15.144:0/2374810976 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe5e0038440 0x7fe5e003a8f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40038f0 tx=0x7fe5e4006ef0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:43.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.552+0000 7fe5daffd640 1 -- 172.21.15.144:0/2374810976 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe5e809e030 con 0x7fe5f409f020 2024-09-18T12:29:43.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.552+0000 7fe5daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:43.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.552+0000 7fe5daffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:43.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.709+0000 7fe5f9857640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:43.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.709+0000 7fe5f9857640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:43.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.709+0000 7fe5f9857640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.709+0000 7fe5f9857640 1 -- 172.21.15.144:0/2374810976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fe5f40a13e0 con 0x7fe5f409f020 2024-09-18T12:29:43.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.710+0000 7fe5daffd640 1 -- 172.21.15.144:0/2374810976 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v93) v1 ==== 103+0+0 (secure 0 0 0) 0x7fe5e8037020 con 0x7fe5f409f020 2024-09-18T12:29:43.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.710+0000 7fe5daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:43.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.710+0000 7fe5daffd640 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-18T12:29:43.711 INFO:tasks.workunit.client.0.smithi144.stderr:Error EBUSY: (16) Device or resource busy 2024-09-18T12:29:43.714 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:29:43.714 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rule rm foo-rule 2024-09-18T12:29:43.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.786+0000 7fddb4b5d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:43.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.786+0000 7fddb4b5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:43.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.786+0000 7fddb4b5d640 10 monclient: monmap: 2024-09-18T12:29:43.787 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:43.787338+0000 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:43.787338+0000 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:43.788 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.786+0000 7fddb4b5d640 1 Processor -- start 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 1 -- start start 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient: init 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddb0098c70 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fddb0099c70 0x7fddb0136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fddb0099c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb01395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fddb0137200 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb009f680 con 0x7fddb0098c70 2024-09-18T12:29:43.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb009f420 con 0x7fddb0099c70 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb0068a20 con 0x7fddb0137200 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddb4b5d640 10 monclient(hunting): authenticate will time out at 1019.356637s 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddadd9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fddb0099c70 0x7fddb0136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddadd9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fddb0099c70 0x7fddb0136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42230/0 (socket says 172.21.15.144:42230) 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddadd9b640 1 -- 172.21.15.144:0/1267898990 learned_addr learned my addr 172.21.15.144:0/1267898990 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddadd9b640 10 monclient(hunting): get_auth_request con 0x7fddb0099c70 auth_method 0 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddadd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddadd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.787+0000 7fddadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:43.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 1 --2- 172.21.15.144:0/1267898990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb01395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): get_auth_request con 0x7fddb0137200 auth_method 0 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient(hunting): handle_auth_done global_id 64241 payload 274 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 1 -- 172.21.15.144:0/1267898990 >> v2:172.21.15.144:3300/0 conn(0x7fddb0099c70 msgr2=0x7fddb0136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 1 --2- 172.21.15.144:0/1267898990 >> v2:172.21.15.144:3300/0 conn(0x7fddb0099c70 0x7fddb0136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 1 -- 172.21.15.144:0/1267898990 >> v1:172.21.15.144:6789/0 conn(0x7fddb0098c70 legacy=0x7fddb0099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:43.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:29:43.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 1 monclient: found mon.noname-b 2024-09-18T12:29:43.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:43.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:43.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.788+0000 7fddaed9d640 1 -- 172.21.15.144:0/1267898990 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb0096a30 con 0x7fddb0137200 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddaed9d640 1 -- 172.21.15.144:0/1267898990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 msgr2=0x7fddb01395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddaed9d640 1 --2- 172.21.15.144:0/1267898990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb01395e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009e90 tx=0x7fdda8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38581a8f5f0fd913 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 1 -- 172.21.15.144:0/1267898990 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8007760 con 0x7fddb0137200 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: dump: 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.798 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: _finish_auth 0 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 1 -- 172.21.15.144:0/1267898990 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda802d050 con 0x7fddb0137200 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 1 -- 172.21.15.144:0/1267898990 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8007af0 con 0x7fddb0137200 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddad59a640 10 monclient: dump: 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.800 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddb4b5d640 5 monclient: authenticate success, global_id 64241 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddb4b5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddb4b5d640 4 set_mon_vals no callback set 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddb4b5d640 10 monclient: shutdown 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddb4b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddb4b5d640 1 -- 172.21.15.144:0/1267898990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 msgr2=0x7fddb01395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.791+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/1267898990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb01395e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009e90 tx=0x7fdda8009780 comp rx=0 tx=0).stop 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 -- 172.21.15.144:0/1267898990 shutdown_connections 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/1267898990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb01395e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/1267898990 >> v2:172.21.15.144:3300/0 conn(0x7fddb0099c70 0x7fddb0136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 -- 172.21.15.144:0/1267898990 >> 172.21.15.144:0/1267898990 conn(0x7fddb0070e90 msgr2=0x7fddb00a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 -- 172.21.15.144:0/1267898990 shutdown_connections 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 -- 172.21.15.144:0/1267898990 wait complete. 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 10 monclient: build_initial_monmap 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 10 monclient: monmap: 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 Processor -- start 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 1 -- start start 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.792+0000 7fddb4b5d640 10 monclient: init 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 0x7fddb011e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fddb0098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddb0099c70 0x7fddb011ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fddb0099c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb011f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fddb0137200 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb0120230 con 0x7fddb0099c70 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb006eba0 con 0x7fddb0098c70 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb00a1ae0 con 0x7fddb0137200 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:43.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddb4b5d640 10 monclient(hunting): authenticate will time out at 1019.362347s 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddadd9b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddb0099c70 msgr2=0x7fddb011ec80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 0x7fddb011e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddadd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddb0099c70 0x7fddb011ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 0x7fddb011e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42238/0 (socket says 172.21.15.144:42238) 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 -- 172.21.15.144:0/331668322 learned_addr learned my addr 172.21.15.144:0/331668322 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): get_auth_request con 0x7fddb0098c70 auth_method 0 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:43.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddaed9d640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb011f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddaed9d640 10 monclient(hunting): get_auth_request con 0x7fddb0137200 auth_method 0 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddaed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddaed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddaed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient(hunting): handle_auth_done global_id 5311 payload 824 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 -- 172.21.15.144:0/331668322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 msgr2=0x7fddb011f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb011f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 -- 172.21.15.144:0/331668322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddb0099c70 msgr2=0x7fddb011ec80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddb0099c70 0x7fddb011ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient: _finish_hunting 0 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 monclient: found mon.noname-b 2024-09-18T12:29:43.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:43.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.793+0000 7fddae59c640 1 -- 172.21.15.144:0/331668322 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda8009a20 con 0x7fddb0098c70 2024-09-18T12:29:43.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.795+0000 7fddae59c640 1 -- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 msgr2=0x7fddb011e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:43.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.795+0000 7fddae59c640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 0x7fddb011e740 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdda000c610 tx=0x7fdda0007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:43.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.795+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.797+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0012840 con 0x7fddb0098c70 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.797+0000 7fdd9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 10 monclient: dump: 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.809 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda00129e0 con 0x7fddb0098c70 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0012da0 con 0x7fddb0098c70 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddb4b5d640 5 monclient: authenticate success, global_id 5311 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddb4b5d640 10 monclient: _renew_subs 2024-09-18T12:29:43.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddb4b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb0121cd0 con 0x7fddb0098c70 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddacd99640 4 set_mon_vals no callback set 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 10 monclient: dump: 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:43.811 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddb4b5d640 10 monclient: _renew_subs 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddb4b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb0120d70 con 0x7fddb0098c70 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.798+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdda0011590 con 0x7fddb0098c70 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fdd9e7fc640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd90038490 0x7fdd9003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdda004b7e0 con 0x7fddb0098c70 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fddb4b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fddb4b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fddb4b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb009a070 con 0x7fddb0098c70 2024-09-18T12:29:43.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fddadd9b640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd90038490 0x7fdd9003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:43.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.799+0000 7fddadd9b640 10 monclient: get_auth_request con 0x7fdd90038490 auth_method 0 2024-09-18T12:29:43.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.801+0000 7fddadd9b640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd90038490 0x7fdd9003a940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fddb0067690 tx=0x7fdda4006bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:43.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.803+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdda001a710 con 0x7fddb0098c70 2024-09-18T12:29:43.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.803+0000 7fdd9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:43.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.803+0000 7fdd9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:43.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.960+0000 7fddb4b5d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T12:29:43.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.960+0000 7fddb4b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T12:29:43.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.960+0000 7fddb4b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:43.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:43.960+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7fddb01220d0 con 0x7fddb0098c70 2024-09-18T12:29:44.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.353+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v94) v1 ==== 85+0+0 (secure 0 0 0) 0x7fdda0019310 con 0x7fddb0098c70 2024-09-18T12:29:44.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.353+0000 7fdd9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T12:29:44.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.353+0000 7fdd9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:29:44.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.450+0000 7fddb4b5d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T12:29:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.450+0000 7fddb4b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T12:29:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.450+0000 7fddb4b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.450+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7fddb01223b0 con 0x7fddb0098c70 2024-09-18T12:29:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.450+0000 7fdd9e7fc640 1 -- 172.21.15.144:0/331668322 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v94) v1 ==== 113+0+0 (secure 0 0 0) 0x7fdda00194f0 con 0x7fddb0098c70 2024-09-18T12:29:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.450+0000 7fdd9e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T12:29:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.450+0000 7fdd9e7fc640 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-18T12:29:44.452 INFO:tasks.workunit.client.0.smithi144.stderr:rule foo-rule does not exist 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd90038490 msgr2=0x7fdd9003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd90038490 0x7fdd9003a940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fddb0067690 tx=0x7fdda4006bc0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 10 monclient: shutdown 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 msgr2=0x7fddb011e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 0x7fddb011e740 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdda000c610 tx=0x7fdda0007ed0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 shutdown_connections 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd90038490 0x7fdd9003a940 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fddb0137200 0x7fddb011f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddb0099c70 0x7fddb011ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 --2- 172.21.15.144:0/331668322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddb0098c70 0x7fddb011e740 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 >> 172.21.15.144:0/331668322 conn(0x7fddb0070e90 msgr2=0x7fddb0097310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:44.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 shutdown_connections 2024-09-18T12:29:44.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.452+0000 7fddb4b5d640 1 -- 172.21.15.144:0/331668322 wait complete. 2024-09-18T12:29:44.462 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm bar 2024-09-18T12:29:44.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.535+0000 7f5d47868640 10 monclient: get_monmap_and_config 2024-09-18T12:29:44.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.535+0000 7f5d47868640 10 monclient: build_initial_monmap 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.535+0000 7f5d47868640 10 monclient: monmap: 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:44.536432+0000 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:44.536432+0000 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:44.537 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:44.538 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 1 Processor -- start 2024-09-18T12:29:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 1 -- start start 2024-09-18T12:29:44.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient: init 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d4009ead0 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5d40069e10 0x7f5d4009eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient(hunting): picked mon.noname-c con 0x7f5d40069e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009f3e0 0x7f5d40138d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4009f3e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4006a210 con 0x7f5d4009ead0 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4009d900 con 0x7f5d40069e10 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:44.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4009fef0 con 0x7f5d4009f3e0 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d47868640 10 monclient(hunting): authenticate will time out at 1020.105901s 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.536+0000 7f5d45604640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f5d4009ead0 legacy=0x7f5d4009d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5d40069e10 0x7f5d4009eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5d40069e10 0x7f5d4009eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42246/0 (socket says 172.21.15.144:42246) 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 -- 172.21.15.144:0/1446141899 learned_addr learned my addr 172.21.15.144:0/1446141899 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): get_auth_request con 0x7f5d40069e10 auth_method 0 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 1 --2- 172.21.15.144:0/1446141899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009f3e0 0x7f5d40138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 10 monclient(hunting): get_auth_request con 0x7f5d4009f3e0 auth_method 0 2024-09-18T12:29:44.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d45e05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient(hunting): handle_auth_done global_id 5314 payload 274 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 -- 172.21.15.144:0/1446141899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009f3e0 msgr2=0x7f5d40138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 --2- 172.21.15.144:0/1446141899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009f3e0 0x7f5d40138d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 -- 172.21.15.144:0/1446141899 >> v1:172.21.15.116:6789/0 conn(0x7f5d4009ead0 legacy=0x7f5d4009d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:44.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient: _finish_hunting 0 2024-09-18T12:29:44.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 monclient: found mon.noname-c 2024-09-18T12:29:44.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:44.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:44.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.537+0000 7f5d44e03640 1 -- 172.21.15.144:0/1446141899 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d400a0330 con 0x7f5d40069e10 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d44e03640 1 -- 172.21.15.144:0/1446141899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d40069e10 msgr2=0x7f5d4009eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d44e03640 1 --2- 172.21.15.144:0/1446141899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d40069e10 0x7f5d4009eea0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009940 tx=0x7f5d300091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1284048b687d0778 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 1 -- 172.21.15.144:0/1446141899 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3002ae80 con 0x7f5d40069e10 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: dump: 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:44.550 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 5 monclient: authenticate success, global_id 5314 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:44.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 1 -- 172.21.15.144:0/1446141899 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d30032440 con 0x7f5d40069e10 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 1 -- 172.21.15.144:0/1446141899 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d300327c0 con 0x7f5d40069e10 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 4 set_mon_vals no callback set 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d37fff640 10 monclient: dump: 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.552 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 10 monclient: shutdown 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 1 -- 172.21.15.144:0/1446141899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d40069e10 msgr2=0x7f5d4009eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:44.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 1 --2- 172.21.15.144:0/1446141899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d40069e10 0x7f5d4009eea0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009940 tx=0x7f5d300091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 1 -- 172.21.15.144:0/1446141899 shutdown_connections 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 1 --2- 172.21.15.144:0/1446141899 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009f3e0 0x7f5d40138d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 1 --2- 172.21.15.144:0/1446141899 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d40069e10 0x7f5d4009eea0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.540+0000 7f5d47868640 1 -- 172.21.15.144:0/1446141899 >> 172.21.15.144:0/1446141899 conn(0x7f5d400a5230 msgr2=0x7f5d400a5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.541+0000 7f5d47868640 1 -- 172.21.15.144:0/1446141899 shutdown_connections 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.541+0000 7f5d47868640 1 -- 172.21.15.144:0/1446141899 wait complete. 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.541+0000 7f5d47868640 10 monclient: build_initial_monmap 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.541+0000 7f5d47868640 10 monclient: monmap: 2024-09-18T12:29:44.554 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:44.555 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:44.556 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:44.556 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.541+0000 7f5d47868640 1 Processor -- start 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.541+0000 7f5d47868640 1 -- start start 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.541+0000 7f5d47868640 10 monclient: init 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d40069e10 0x7f5d4011e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): picked mon.noname-a con 0x7f5d40069e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009ead0 0x7f5d4011e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4009ead0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 0x7f5d4011ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4009f3e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:44.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d401202b0 con 0x7f5d40069e10 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4009bf70 con 0x7f5d4009f3e0 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d400a3130 con 0x7f5d4009ead0 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d47868640 10 monclient(hunting): authenticate will time out at 1020.111393s 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 0x7f5d4011ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45604640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d40069e10 msgr2=0x7f5d4011e400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45604640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d40069e10 0x7f5d4011e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 0x7f5d4011ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42260/0 (socket says 172.21.15.144:42260) 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 -- 172.21.15.144:0/2051185520 learned_addr learned my addr 172.21.15.144:0/2051185520 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): get_auth_request con 0x7f5d4009f3e0 auth_method 0 2024-09-18T12:29:44.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d44e03640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009ead0 0x7f5d4011e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d44e03640 10 monclient(hunting): get_auth_request con 0x7f5d4009ead0 auth_method 0 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d44e03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d44e03640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d44e03640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient(hunting): handle_auth_done global_id 5317 payload 824 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 -- 172.21.15.144:0/2051185520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009ead0 msgr2=0x7f5d4011e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009ead0 0x7f5d4011e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 -- 172.21.15.144:0/2051185520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d40069e10 msgr2=0x7f5d4011e400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d40069e10 0x7f5d4011e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient: _finish_hunting 0 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 monclient: found mon.noname-b 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.542+0000 7f5d45e05640 1 -- 172.21.15.144:0/2051185520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d30009450 con 0x7f5d4009f3e0 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.544+0000 7f5d45e05640 1 -- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 msgr2=0x7f5d4011ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.544+0000 7f5d45e05640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 0x7f5d4011ee80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009e90 tx=0x7f5d3800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38014480 con 0x7f5d4009f3e0 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:44.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: dump: 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:44.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3800f040 con 0x7f5d4009f3e0 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d47868640 5 monclient: authenticate success, global_id 5317 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d47868640 10 monclient: _renew_subs 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d47868640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d401388d0 con 0x7f5d4009f3e0 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38014830 con 0x7f5d4009f3e0 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d357fa640 10 monclient: dump: 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:44.562 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d47868640 10 monclient: _renew_subs 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d47868640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d40121840 con 0x7f5d4009f3e0 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.545+0000 7f5d377fe640 4 set_mon_vals no callback set 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d47868640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:44.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d47868640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d47868640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5d38014be0 con 0x7f5d4009f3e0 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d40121bb0 con 0x7f5d4009f3e0 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d357fa640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d24038440 0x7f5d2403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5969+0+0 (secure 0 0 0) 0x7f5d3804c830 con 0x7f5d4009f3e0 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d45604640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d24038440 0x7f5d2403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.546+0000 7f5d45604640 10 monclient: get_auth_request con 0x7f5d24038440 auth_method 0 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.549+0000 7f5d45604640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d24038440 0x7f5d2403a8f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0038f0 tx=0x7f5d3c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:44.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.549+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:29:44.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.551+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5d380118c0 con 0x7f5d4009f3e0 2024-09-18T12:29:44.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.551+0000 7f5d357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:44.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.551+0000 7f5d357fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.709+0000 7f5d47868640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.709+0000 7f5d47868640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.710+0000 7f5d47868640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:44.710+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f5d400008d0 con 0x7f5d4009f3e0 2024-09-18T12:29:45.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.358+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v95) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5d38018090 con 0x7f5d4009f3e0 2024-09-18T12:29:45.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.358+0000 7f5d357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:45.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.358+0000 7f5d357fa640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.456+0000 7f5d47868640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.456+0000 7f5d47868640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.456+0000 7f5d47868640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.456+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f5d40072640 con 0x7f5d4009f3e0 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.457+0000 7f5d357fa640 1 -- 172.21.15.144:0/2051185520 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v95) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5d3801a2c0 con 0x7f5d4009f3e0 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.457+0000 7f5d357fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.457+0000 7f5d357fa640 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-18T12:29:45.458 INFO:tasks.workunit.client.0.smithi144.stderr:device 'bar' does not appear in the crush map 2024-09-18T12:29:45.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d24038440 msgr2=0x7f5d2403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:45.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d24038440 0x7f5d2403a8f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0038f0 tx=0x7f5d3c006d80 comp rx=0 tx=0).stop 2024-09-18T12:29:45.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 10 monclient: shutdown 2024-09-18T12:29:45.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 msgr2=0x7f5d4011ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 0x7f5d4011ee80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009e90 tx=0x7f5d3800b040 comp rx=0 tx=0).stop 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 shutdown_connections 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d24038440 0x7f5d2403a8f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4009f3e0 0x7f5d4011ee80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4009ead0 0x7f5d4011e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 --2- 172.21.15.144:0/2051185520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d40069e10 0x7f5d4011e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 >> 172.21.15.144:0/2051185520 conn(0x7f5d400a5230 msgr2=0x7f5d40068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 shutdown_connections 2024-09-18T12:29:45.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.459+0000 7f5d47868640 1 -- 172.21.15.144:0/2051185520 wait complete. 2024-09-18T12:29:45.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm foo 2024-09-18T12:29:45.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.541+0000 7fb08135d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:45.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.541+0000 7fb08135d640 10 monclient: build_initial_monmap 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.541+0000 7fb08135d640 10 monclient: monmap: 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:45.542447+0000 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:45.542447+0000 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:45.543 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:45.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 1 Processor -- start 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 1 -- start start 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient: init 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb07c09ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb07c09f020 0x7fb07c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient(hunting): picked mon.noname-c con 0x7fb07c09f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb07c136d60 0x7fb07c139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient(hunting): picked mon.noname-a con 0x7fb07c136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07c067420 con 0x7fb07c09ec50 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:45.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07c0a0480 con 0x7fb07c136d60 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07c0a0200 con 0x7fb07c09f020 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb08135d640 10 monclient(hunting): authenticate will time out at 1021.111697s 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb07c09f020 0x7fb07c069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fb07c136d60 msgr2=0x7fb07c139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07affd640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fb07c09ec50 legacy=0x7fb07c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb07c09f020 0x7fb07c069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42266/0 (socket says 172.21.15.144:42266) 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 1 -- 172.21.15.144:0/2396397580 learned_addr learned my addr 172.21.15.144:0/2396397580 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 10 monclient(hunting): get_auth_request con 0x7fb07c09f020 auth_method 0 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:45.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07b7fe640 1 --2- 172.21.15.144:0/2396397580 >> v2:172.21.15.116:3300/0 conn(0x7fb07c136d60 0x7fb07c139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.542+0000 7fb07a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 10 monclient(hunting): handle_auth_done global_id 5320 payload 274 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 1 -- 172.21.15.144:0/2396397580 >> v2:172.21.15.116:3300/0 conn(0x7fb07c136d60 msgr2=0x7fb07c139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 1 --2- 172.21.15.144:0/2396397580 >> v2:172.21.15.116:3300/0 conn(0x7fb07c136d60 0x7fb07c139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 1 -- 172.21.15.144:0/2396397580 >> v1:172.21.15.116:6789/0 conn(0x7fb07c09ec50 legacy=0x7fb07c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:29:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 1 monclient: found mon.noname-c 2024-09-18T12:29:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:45.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.543+0000 7fb07a7fc640 1 -- 172.21.15.144:0/2396397580 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c09fd80 con 0x7fb07c09f020 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.545+0000 7fb07a7fc640 1 -- 172.21.15.144:0/2396397580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09f020 msgr2=0x7fb07c069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.545+0000 7fb07a7fc640 1 --2- 172.21.15.144:0/2396397580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09f020 0x7fb07c069d10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb07c06a250 tx=0x7fb0700091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9e675c8aae6b2fb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.545+0000 7fb079ffb640 1 -- 172.21.15.144:0/2396397580 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070032440 con 0x7fb07c09f020 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.545+0000 7fb079ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 10 monclient: dump: 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.555 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:45.556 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:45.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 10 monclient: _finish_auth 0 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 5 monclient: authenticate success, global_id 5320 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 1 -- 172.21.15.144:0/2396397580 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0700325a0 con 0x7fb07c09f020 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 4 set_mon_vals no callback set 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 1 -- 172.21.15.144:0/2396397580 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070032940 con 0x7fb07c09f020 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:45.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb079ffb640 10 monclient: dump: 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:45.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 10 monclient: shutdown 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 -- 172.21.15.144:0/2396397580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09f020 msgr2=0x7fb07c069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 --2- 172.21.15.144:0/2396397580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09f020 0x7fb07c069d10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb07c06a250 tx=0x7fb0700091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 -- 172.21.15.144:0/2396397580 shutdown_connections 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 --2- 172.21.15.144:0/2396397580 >> v2:172.21.15.116:3300/0 conn(0x7fb07c136d60 0x7fb07c139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 --2- 172.21.15.144:0/2396397580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09f020 0x7fb07c069d10 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 -- 172.21.15.144:0/2396397580 >> 172.21.15.144:0/2396397580 conn(0x7fb07c0a5cf0 msgr2=0x7fb07c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 -- 172.21.15.144:0/2396397580 shutdown_connections 2024-09-18T12:29:45.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.546+0000 7fb08135d640 1 -- 172.21.15.144:0/2396397580 wait complete. 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient: build_initial_monmap 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient: monmap: 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:45.560 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 1 Processor -- start 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 1 -- start start 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient: init 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 0x7fb07c141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient(hunting): picked mon.noname-b con 0x7fb07c09ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb07c09f020 0x7fb07c130d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient(hunting): picked mon.noname-c con 0x7fb07c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb07c136d60 0x7fb07c099c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.547+0000 7fb08135d640 10 monclient(hunting): picked mon.noname-a con 0x7fb07c136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:45.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c09c020 con 0x7fb07c136d60 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c11b880 con 0x7fb07c09ec50 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c0a46f0 con 0x7fb07c09f020 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb08135d640 10 monclient(hunting): authenticate will time out at 1021.117018s 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07b7fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb07c136d60 msgr2=0x7fb07c099c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 0x7fb07c141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb07c136d60 0x7fb07c099c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 0x7fb07c141340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42270/0 (socket says 172.21.15.144:42270) 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 -- 172.21.15.144:0/1118574039 learned_addr learned my addr 172.21.15.144:0/1118574039 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:45.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): get_auth_request con 0x7fb07c09ec50 auth_method 0 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07a7fc640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb07c09f020 0x7fb07c130d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07a7fc640 10 monclient(hunting): get_auth_request con 0x7fb07c09f020 auth_method 0 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient(hunting): handle_auth_done global_id 5323 payload 824 2024-09-18T12:29:45.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 -- 172.21.15.144:0/1118574039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb07c09f020 msgr2=0x7fb07c130d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb07c09f020 0x7fb07c130d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 -- 172.21.15.144:0/1118574039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb07c136d60 msgr2=0x7fb07c099c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb07c136d60 0x7fb07c099c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 monclient: found mon.noname-b 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:45.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.548+0000 7fb07affd640 1 -- 172.21.15.144:0/1118574039 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb070009450 con 0x7fb07c09ec50 2024-09-18T12:29:45.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb07affd640 1 -- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 msgr2=0x7fb07c141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb07affd640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 0x7fb07c141340 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00b5c0 tx=0x7fb06c009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c003cc0 con 0x7fb07c09ec50 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: dump: 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:45.567 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: _finish_auth 0 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c003e40 con 0x7fb07c09ec50 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb08135d640 5 monclient: authenticate success, global_id 5323 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb08135d640 10 monclient: _renew_subs 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb08135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07c11cf00 con 0x7fb07c09ec50 2024-09-18T12:29:45.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c005340 con 0x7fb07c09ec50 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb062ffd640 10 monclient: dump: 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:45.569 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb08135d640 10 monclient: _renew_subs 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb08135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c142100 con 0x7fb07c09ec50 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.550+0000 7fb0797fa640 4 set_mon_vals no callback set 2024-09-18T12:29:45.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.551+0000 7fb08135d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.551+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb06c01a460 con 0x7fb07c09ec50 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.551+0000 7fb08135d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.551+0000 7fb08135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.551+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07c0008d0 con 0x7fb07c09ec50 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.551+0000 7fb062ffd640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb068038440 0x7fb06803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.552+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5840+0+0 (secure 0 0 0) 0x7fb06c04b7e0 con 0x7fb07c09ec50 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.552+0000 7fb07a7fc640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb068038440 0x7fb06803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.552+0000 7fb07a7fc640 10 monclient: get_auth_request con 0x7fb068038440 auth_method 0 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.554+0000 7fb07a7fc640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb068038440 0x7fb06803a8f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb07c0719d0 tx=0x7fb070032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.556+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb06c01a710 con 0x7fb07c09ec50 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.556+0000 7fb062ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:45.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.556+0000 7fb062ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:45.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.715+0000 7fb08135d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:45.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.715+0000 7fb08135d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:45.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.715+0000 7fb08135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:45.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:45.715+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fb07c0a13e0 con 0x7fb07c09ec50 2024-09-18T12:29:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.364+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v96) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb06c017310 con 0x7fb07c09ec50 2024-09-18T12:29:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.364+0000 7fb062ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.364+0000 7fb062ffd640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.464+0000 7fb08135d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.464+0000 7fb08135d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.464+0000 7fb08135d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.464+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fb07c06a060 con 0x7fb07c09ec50 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.465+0000 7fb062ffd640 1 -- 172.21.15.144:0/1118574039 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v96) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb06c0174f0 con 0x7fb07c09ec50 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.465+0000 7fb062ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.465+0000 7fb062ffd640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-18T12:29:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:device 'foo' does not appear in the crush map 2024-09-18T12:29:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb068038440 msgr2=0x7fb06803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb068038440 0x7fb06803a8f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb07c0719d0 tx=0x7fb070032000 comp rx=0 tx=0).stop 2024-09-18T12:29:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 10 monclient: shutdown 2024-09-18T12:29:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 msgr2=0x7fb07c141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 0x7fb07c141340 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00b5c0 tx=0x7fb06c009120 comp rx=0 tx=0).stop 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 shutdown_connections 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb068038440 0x7fb06803a8f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb07c136d60 0x7fb07c099c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb07c09f020 0x7fb07c130d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 --2- 172.21.15.144:0/1118574039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb07c09ec50 0x7fb07c141340 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 >> 172.21.15.144:0/1118574039 conn(0x7fb07c0a5cf0 msgr2=0x7fb07c069780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 shutdown_connections 2024-09-18T12:29:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.467+0000 7fb08135d640 1 -- 172.21.15.144:0/1118574039 wait complete. 2024-09-18T12:29:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm osd.9 host2 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.550+0000 7f718d8e8640 10 monclient: get_monmap_and_config 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.550+0000 7f718d8e8640 10 monclient: build_initial_monmap 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.550+0000 7f718d8e8640 10 monclient: monmap: 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:46.550954+0000 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:46.550954+0000 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:46.551 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:46.552 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:46.552 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:46.552 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:46.552 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:46.552 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:46.552 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:46.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.550+0000 7f718d8e8640 1 Processor -- start 2024-09-18T12:29:46.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.550+0000 7f718d8e8640 1 -- start start 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.550+0000 7f718d8e8640 10 monclient: init 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f718809ec50 0x7f7188131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient(hunting): picked mon.noname-c con 0x7f718809ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71881321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f718809f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7188067420 con 0x7f71881321a0 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718809fdc0 con 0x7f718809f020 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71880a0510 con 0x7f718809ec50 2024-09-18T12:29:46.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f718d8e8640 10 monclient(hunting): authenticate will time out at 1022.120130s 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f71867fc640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f71881321a0 legacy=0x7f7188068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f718809ec50 0x7f7188131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f718809ec50 0x7f7188131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42284/0 (socket says 172.21.15.144:42284) 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 1 -- 172.21.15.144:0/3349176634 learned_addr learned my addr 172.21.15.144:0/3349176634 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 10 monclient(hunting): get_auth_request con 0x7f718809ec50 auth_method 0 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.551+0000 7f7186ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:46.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f71877fe640 1 -- 172.21.15.144:0/3349176634 >> v1:172.21.15.144:6789/0 conn(0x7f718809f020 legacy=0x7f7188069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 10 monclient(hunting): handle_auth_done global_id 5326 payload 274 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 1 -- 172.21.15.144:0/3349176634 >> v1:172.21.15.144:6789/0 conn(0x7f718809f020 legacy=0x7f7188069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 1 -- 172.21.15.144:0/3349176634 >> v1:172.21.15.116:6789/0 conn(0x7f71881321a0 legacy=0x7f7188068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 1 monclient: found mon.noname-c 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:46.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.553+0000 7f7186ffd640 1 -- 172.21.15.144:0/3349176634 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71880a0200 con 0x7f718809ec50 2024-09-18T12:29:46.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7186ffd640 1 -- 172.21.15.144:0/3349176634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 msgr2=0x7f7188131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7186ffd640 1 --2- 172.21.15.144:0/3349176634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f7188131c60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7180009990 tx=0x7f71800091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=528f8f7253e869e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 1 -- 172.21.15.144:0/3349176634 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718002ae90 con 0x7f718809ec50 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: dump: 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:46.568 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: _finish_auth 0 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 1 -- 172.21.15.144:0/3349176634 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718002f4b0 con 0x7f718809ec50 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f718d8e8640 5 monclient: authenticate success, global_id 5326 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f718d8e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 1 -- 172.21.15.144:0/3349176634 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718002f850 con 0x7f718809ec50 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:46.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f7185ffb640 10 monclient: dump: 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f718d8e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.556+0000 7f718d8e8640 4 set_mon_vals no callback set 2024-09-18T12:29:46.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 10 monclient: shutdown 2024-09-18T12:29:46.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:46.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 1 -- 172.21.15.144:0/3349176634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 msgr2=0x7f7188131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 1 --2- 172.21.15.144:0/3349176634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f7188131c60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7180009990 tx=0x7f71800091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 1 -- 172.21.15.144:0/3349176634 shutdown_connections 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 1 --2- 172.21.15.144:0/3349176634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f7188131c60 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 1 -- 172.21.15.144:0/3349176634 >> 172.21.15.144:0/3349176634 conn(0x7f71880a5d40 msgr2=0x7f71880a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 1 -- 172.21.15.144:0/3349176634 shutdown_connections 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 1 -- 172.21.15.144:0/3349176634 wait complete. 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 10 monclient: build_initial_monmap 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.557+0000 7f718d8e8640 10 monclient: monmap: 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:46.572 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 Processor -- start 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 -- start start 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient: init 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:29:46.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f718809ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): picked mon.noname-b con 0x7f718809ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f718809f020 0x7f7188130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): picked mon.noname-a con 0x7f718809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71881321a0 0x7f71881410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): picked mon.noname-c con 0x7f71881321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71880a42b0 con 0x7f718809f020 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718811dfa0 con 0x7f718809ec50 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718811cd60 con 0x7f71881321a0 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:46.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f718d8e8640 10 monclient(hunting): authenticate will time out at 1022.127562s 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f718809ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f718809ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42298/0 (socket says 172.21.15.144:42298) 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 -- 172.21.15.144:0/761993584 learned_addr learned my addr 172.21.15.144:0/761993584 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): get_auth_request con 0x7f718809ec50 auth_method 0 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f71867fc640 1 -- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f718809f020 msgr2=0x7f7188130ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f71867fc640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f718809f020 0x7f7188130ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:46.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient(hunting): handle_auth_done global_id 5329 payload 824 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 -- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71881321a0 msgr2=0x7f71881410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71881321a0 0x7f71881410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 -- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f718809f020 msgr2=0x7f7188130ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f718809f020 0x7f7188130ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 monclient: found mon.noname-b 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.558+0000 7f7186ffd640 1 -- 172.21.15.144:0/761993584 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7180009450 con 0x7f718809ec50 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f7186ffd640 1 -- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 msgr2=0x7f718809ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f7186ffd640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f718809ab00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7180009990 tx=0x7f7180032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180032a10 con 0x7f718809ec50 2024-09-18T12:29:46.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: dump: 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:46.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: _finish_auth 0 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718002ae90 con 0x7f718809ec50 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f718d8e8640 5 monclient: authenticate success, global_id 5329 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f718d8e8640 10 monclient: _renew_subs 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f718d8e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718811dcc0 con 0x7f718809ec50 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180032de0 con 0x7f718809ec50 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f716effd640 10 monclient: dump: 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:46.578 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f718d8e8640 10 monclient: _renew_subs 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f718d8e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718811e330 con 0x7f718809ec50 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.561+0000 7f71857fa640 4 set_mon_vals no callback set 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7180039040 con 0x7f718809ec50 2024-09-18T12:29:46.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f718d8e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f718d8e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f718d8e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718809b4c0 con 0x7f718809ec50 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f716effd640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7174038440 0x7f717403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5711+0+0 (secure 0 0 0) 0x7f7180041030 con 0x7f718809ec50 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f71867fc640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7174038440 0x7f717403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.562+0000 7f71867fc640 10 monclient: get_auth_request con 0x7f7174038440 auth_method 0 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.564+0000 7f71867fc640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7174038440 0x7f717403a8f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f717c006fd0 tx=0x7f717c006d70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.566+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7180030570 con 0x7f718809ec50 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.566+0000 7f716effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:46.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.566+0000 7f716effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:46.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.727+0000 7f718d8e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-18T12:29:46.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.727+0000 7f718d8e8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-18T12:29:46.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.727+0000 7f718d8e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:46.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:46.727+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f71880008d0 con 0x7f718809ec50 2024-09-18T12:29:47.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.583+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 removed item id 9 name 'osd.9' from crush map v98) v1 ==== 143+0+0 (secure 0 0 0) 0x7f718004b0a0 con 0x7f718809ec50 2024-09-18T12:29:47.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.583+0000 7f716effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-18T12:29:47.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.583+0000 7f716effd640 10 monclient: _finish_command 2 = system:0 removed item id 9 name 'osd.9' from crush map 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.682+0000 7f718d8e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.682+0000 7f718d8e8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.682+0000 7f718d8e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.682+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) v1 -- 0x7f71880008d0 con 0x7f718809ec50 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.682+0000 7f716effd640 1 -- 172.21.15.144:0/761993584 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 device 'osd.9' does not appear in the crush map v98) v1 ==== 145+0+0 (secure 0 0 0) 0x7f7180037020 con 0x7f718809ec50 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.683+0000 7f716effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.683+0000 7f716effd640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-18T12:29:47.684 INFO:tasks.workunit.client.0.smithi144.stderr:device 'osd.9' does not appear in the crush map 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7174038440 msgr2=0x7f717403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7174038440 0x7f717403a8f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f717c006fd0 tx=0x7f717c006d70 comp rx=0 tx=0).stop 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 10 monclient: shutdown 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 msgr2=0x7f718809ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f718809ab00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7180009990 tx=0x7f7180032000 comp rx=0 tx=0).stop 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 shutdown_connections 2024-09-18T12:29:47.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7174038440 0x7f717403a8f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f71881321a0 0x7f71881410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f718809f020 0x7f7188130ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 --2- 172.21.15.144:0/761993584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f718809ec50 0x7f718809ab00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 >> 172.21.15.144:0/761993584 conn(0x7f71880a5d40 msgr2=0x7f71880a39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:47.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 shutdown_connections 2024-09-18T12:29:47.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.684+0000 7f718d8e8640 1 -- 172.21.15.144:0/761993584 wait complete. 2024-09-18T12:29:47.694 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm host2 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.765+0000 7ffa46f1d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.765+0000 7ffa46f1d640 10 monclient: build_initial_monmap 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.766+0000 7ffa46f1d640 10 monclient: monmap: 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:47.766893+0000 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:47.766893+0000 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:47.767 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:47.768 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:47.768 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:47.768 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:47.768 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:47.768 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:47.768 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:47.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.766+0000 7ffa46f1d640 1 Processor -- start 2024-09-18T12:29:47.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.766+0000 7ffa46f1d640 1 -- start start 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.766+0000 7ffa46f1d640 10 monclient: init 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa40131c00 0x7ffa40131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient(hunting): picked mon.noname-c con 0x7ffa40131c00 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffa400a0b00 0x7ffa40132510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa400a0b00 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa40068d00 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4009d230 con 0x7ffa40068d00 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:47.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa400a08c0 con 0x7ffa400a0b00 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa400a0200 con 0x7ffa40131c00 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa46f1d640 10 monclient(hunting): authenticate will time out at 1023.336266s 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa40131c00 0x7ffa40131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffa40131c00 0x7ffa40131fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42542/0 (socket says 172.21.15.144:42542) 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 -- 172.21.15.144:0/1790374088 learned_addr learned my addr 172.21.15.144:0/1790374088 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): get_auth_request con 0x7ffa40131c00 auth_method 0 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:47.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa3ffff640 1 -- 172.21.15.144:0/1790374088 >> v2:172.21.15.116:3300/0 conn(0x7ffa400a0b00 msgr2=0x7ffa40132510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa454ba640 1 -- 172.21.15.144:0/1790374088 >> v1:172.21.15.116:6789/0 conn(0x7ffa40068d00 legacy=0x7ffa40069100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa3ffff640 1 --2- 172.21.15.144:0/1790374088 >> v2:172.21.15.116:3300/0 conn(0x7ffa400a0b00 0x7ffa40132510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa3f7fe640 1 -- 172.21.15.144:0/1790374088 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient(hunting): handle_auth_done global_id 5332 payload 274 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 -- 172.21.15.144:0/1790374088 >> v2:172.21.15.116:3300/0 conn(0x7ffa400a0b00 msgr2=0x7ffa40132510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 --2- 172.21.15.144:0/1790374088 >> v2:172.21.15.116:3300/0 conn(0x7ffa400a0b00 0x7ffa40132510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 -- 172.21.15.144:0/1790374088 >> v1:172.21.15.116:6789/0 conn(0x7ffa40068d00 legacy=0x7ffa40069100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient: _finish_hunting 0 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 monclient: found mon.noname-c 2024-09-18T12:29:47.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:47.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:47.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.767+0000 7ffa44cb9640 1 -- 172.21.15.144:0/1790374088 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa400a1b70 con 0x7ffa40131c00 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa44cb9640 1 -- 172.21.15.144:0/1790374088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa40131c00 msgr2=0x7ffa40131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa44cb9640 1 --2- 172.21.15.144:0/1790374088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa40131c00 0x7ffa40131fd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009f20 tx=0x7ffa30009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c06f5fbd9e32fb28 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 1 -- 172.21.15.144:0/1790374088 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30007610 con 0x7ffa40131c00 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: dump: 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.780 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa46f1d640 5 monclient: authenticate success, global_id 5332 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 1 -- 172.21.15.144:0/1790374088 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3002d050 con 0x7ffa40131c00 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa46f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa46f1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:47.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa46f1d640 4 set_mon_vals no callback set 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 1 -- 172.21.15.144:0/1790374088 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa300079a0 con 0x7ffa40131c00 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.770+0000 7ffa3f7fe640 10 monclient: dump: 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:47.782 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 10 monclient: shutdown 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 -- 172.21.15.144:0/1790374088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa40131c00 msgr2=0x7ffa40131fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/1790374088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa40131c00 0x7ffa40131fd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009f20 tx=0x7ffa30009840 comp rx=0 tx=0).stop 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 -- 172.21.15.144:0/1790374088 shutdown_connections 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/1790374088 >> v2:172.21.15.116:3300/0 conn(0x7ffa400a0b00 0x7ffa40132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/1790374088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa40131c00 0x7ffa40131fd0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 -- 172.21.15.144:0/1790374088 >> 172.21.15.144:0/1790374088 conn(0x7ffa40071370 msgr2=0x7ffa400a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 -- 172.21.15.144:0/1790374088 shutdown_connections 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 -- 172.21.15.144:0/1790374088 wait complete. 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 10 monclient: build_initial_monmap 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 10 monclient: monmap: 2024-09-18T12:29:47.783 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.771+0000 7ffa46f1d640 1 Processor -- start 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 1 -- start start 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient: init 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:47.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa40068d00 0x7ffa4009aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa40068d00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 0x7ffa40130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): picked mon.noname-b con 0x7ffa400a0b00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa40131c00 0x7ffa4011b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): picked mon.noname-c con 0x7ffa40131c00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4011dfa0 con 0x7ffa40068d00 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa400a1040 con 0x7ffa400a0b00 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:47.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa40134c50 con 0x7ffa40131c00 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa46f1d640 10 monclient(hunting): authenticate will time out at 1023.341591s 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 0x7ffa40130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 0x7ffa40130ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42544/0 (socket says 172.21.15.144:42544) 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 -- 172.21.15.144:0/695014868 learned_addr learned my addr 172.21.15.144:0/695014868 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request con 0x7ffa400a0b00 auth_method 0 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:47.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa44cb9640 1 -- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa40068d00 msgr2=0x7ffa4009aa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa44cb9640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa40068d00 0x7ffa4009aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient(hunting): handle_auth_done global_id 5335 payload 824 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 -- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa40131c00 msgr2=0x7ffa4011b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa40131c00 0x7ffa4011b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 -- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa40068d00 msgr2=0x7ffa4009aa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa40068d00 0x7ffa4009aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 monclient: found mon.noname-b 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:47.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.772+0000 7ffa3ffff640 1 -- 172.21.15.144:0/695014868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa30009ae0 con 0x7ffa400a0b00 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3ffff640 1 -- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 msgr2=0x7ffa40130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3ffff640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 0x7ffa40130ba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800b600 tx=0x7ffa280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28015070 con 0x7ffa400a0b00 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: dump: 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:47.788 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:47.789 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:47.789 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:47.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: _finish_auth 0 2024-09-18T12:29:47.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:47.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa28003d90 con 0x7ffa400a0b00 2024-09-18T12:29:47.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa46f1d640 5 monclient: authenticate success, global_id 5335 2024-09-18T12:29:47.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa46f1d640 10 monclient: _renew_subs 2024-09-18T12:29:47.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa46f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:47.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4011dcc0 con 0x7ffa400a0b00 2024-09-18T12:29:47.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:47.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28006380 con 0x7ffa400a0b00 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3cff9640 10 monclient: dump: 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:47.793 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa3effd640 4 set_mon_vals no callback set 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa46f1d640 10 monclient: _renew_subs 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa46f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.775+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4011ce50 con 0x7ffa400a0b00 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ffa28010440 con 0x7ffa400a0b00 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa46f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa46f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa46f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa400008d0 con 0x7ffa400a0b00 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa3cff9640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffa20038440 0x7ffa2003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5690+0+0 (secure 0 0 0) 0x7ffa2804b750 con 0x7ffa400a0b00 2024-09-18T12:29:47.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa44cb9640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffa20038440 0x7ffa2003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:47.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.776+0000 7ffa44cb9640 10 monclient: get_auth_request con 0x7ffa20038440 auth_method 0 2024-09-18T12:29:47.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.779+0000 7ffa44cb9640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffa20038440 0x7ffa2003a8f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffa400a0ad0 tx=0x7ffa30036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:47.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.781+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ffa28018910 con 0x7ffa400a0b00 2024-09-18T12:29:47.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.781+0000 7ffa3cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:47.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.781+0000 7ffa3cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:47.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.940+0000 7ffa46f1d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T12:29:47.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.940+0000 7ffa46f1d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T12:29:47.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.940+0000 7ffa46f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:47.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:47.940+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7ffa400a1890 con 0x7ffa400a0b00 2024-09-18T12:29:48.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.588+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v99) v1 ==== 124+0+0 (secure 0 0 0) 0x7ffa28018310 con 0x7ffa400a0b00 2024-09-18T12:29:48.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.588+0000 7ffa3cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T12:29:48.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.588+0000 7ffa3cff9640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-18T12:29:48.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.687+0000 7ffa46f1d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T12:29:48.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.687+0000 7ffa46f1d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T12:29:48.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.687+0000 7ffa46f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:48.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.687+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7ffa401365c0 con 0x7ffa400a0b00 2024-09-18T12:29:48.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.688+0000 7ffa3cff9640 1 -- 172.21.15.144:0/695014868 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v99) v1 ==== 124+0+0 (secure 0 0 0) 0x7ffa280184f0 con 0x7ffa400a0b00 2024-09-18T12:29:48.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.688+0000 7ffa3cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T12:29:48.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.688+0000 7ffa3cff9640 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-18T12:29:48.690 INFO:tasks.workunit.client.0.smithi144.stderr:device 'host2' does not appear in the crush map 2024-09-18T12:29:48.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffa20038440 msgr2=0x7ffa2003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:48.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffa20038440 0x7ffa2003a8f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffa400a0ad0 tx=0x7ffa30036000 comp rx=0 tx=0).stop 2024-09-18T12:29:48.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 10 monclient: shutdown 2024-09-18T12:29:48.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 msgr2=0x7ffa40130ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 0x7ffa40130ba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800b600 tx=0x7ffa280091a0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 shutdown_connections 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffa20038440 0x7ffa2003a8f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa40131c00 0x7ffa4011b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa400a0b00 0x7ffa40130ba0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 --2- 172.21.15.144:0/695014868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa40068d00 0x7ffa4009aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 >> 172.21.15.144:0/695014868 conn(0x7ffa40071370 msgr2=0x7ffa4009dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:48.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 shutdown_connections 2024-09-18T12:29:48.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.690+0000 7ffa46f1d640 1 -- 172.21.15.144:0/695014868 wait complete. 2024-09-18T12:29:48.700 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket foo host 2024-09-18T12:29:48.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.772+0000 7f4757a4f640 10 monclient: get_monmap_and_config 2024-09-18T12:29:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.772+0000 7f4757a4f640 10 monclient: build_initial_monmap 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.773+0000 7f4757a4f640 10 monclient: monmap: 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:48.773901+0000 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:48.773901+0000 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:48.787 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.773+0000 7f4757a4f640 1 Processor -- start 2024-09-18T12:29:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.773+0000 7f4757a4f640 1 -- start start 2024-09-18T12:29:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.773+0000 7f4757a4f640 10 monclient: init 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f475009cad0 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f475009c1a0 0x7f47501348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f475009c1a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4750134e10 0x7f47501371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f4750134e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475009d5b0 con 0x7f475009cad0 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f475009b710 con 0x7f475009c1a0 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f475009a650 con 0x7f4750134e10 2024-09-18T12:29:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4757a4f640 10 monclient(hunting): authenticate will time out at 1024.343274s 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f47557eb640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f475009cad0 0x7f475009cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:33170/0 (socket says 172.21.15.144:33170) 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f47557eb640 1 -- 172.21.15.144:0/3764813546 learned_addr learned my addr 172.21.15.144:0/3764813546 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4755fec640 1 --2- 172.21.15.144:0/3764813546 >> v2:172.21.15.144:3300/0 conn(0x7f4750134e10 0x7f47501371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4754fea640 1 -- 172.21.15.144:0/3764813546 >> v2:172.21.15.116:3300/0 conn(0x7f475009c1a0 msgr2=0x7f47501348d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4755fec640 10 monclient(hunting): get_auth_request con 0x7f4750134e10 auth_method 0 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4754fea640 1 --2- 172.21.15.144:0/3764813546 >> v2:172.21.15.116:3300/0 conn(0x7f475009c1a0 0x7f47501348d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4755fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4755fec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.774+0000 7f4755fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f47557eb640 1 -- 172.21.15.144:0/3764813546 >> v1:172.21.15.144:6789/0 conn(0x7f475009cad0 legacy=0x7f475009cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 10 monclient(hunting): handle_auth_done global_id 5338 payload 274 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 1 -- 172.21.15.144:0/3764813546 >> v2:172.21.15.116:3300/0 conn(0x7f475009c1a0 msgr2=0x7f47501348d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 1 --2- 172.21.15.144:0/3764813546 >> v2:172.21.15.116:3300/0 conn(0x7f475009c1a0 0x7f47501348d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 1 -- 172.21.15.144:0/3764813546 >> v1:172.21.15.144:6789/0 conn(0x7f475009cad0 legacy=0x7f475009cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 10 monclient: _finish_hunting 0 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 1 monclient: found mon.noname-c 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.776+0000 7f4755fec640 1 -- 172.21.15.144:0/3764813546 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475009f660 con 0x7f4750134e10 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4755fec640 1 -- 172.21.15.144:0/3764813546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 msgr2=0x7f47501371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4755fec640 1 --2- 172.21.15.144:0/3764813546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f47501371f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4748009ee0 tx=0x7f4748009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e19b4915ad50716 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 1 -- 172.21.15.144:0/3764813546 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748007840 con 0x7f4750134e10 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: dump: 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:48.798 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 1 -- 172.21.15.144:0/3764813546 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474802d050 con 0x7f4750134e10 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4757a4f640 5 monclient: authenticate success, global_id 5338 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4757a4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 1 -- 172.21.15.144:0/3764813546 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748007bd0 con 0x7f4750134e10 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4757a4f640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4757a4f640 4 set_mon_vals no callback set 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.779+0000 7f4747fff640 10 monclient: dump: 2024-09-18T12:29:48.799 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 10 monclient: shutdown 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 -- 172.21.15.144:0/3764813546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 msgr2=0x7f47501371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:48.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 --2- 172.21.15.144:0/3764813546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f47501371f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4748009ee0 tx=0x7f4748009780 comp rx=0 tx=0).stop 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 -- 172.21.15.144:0/3764813546 shutdown_connections 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 --2- 172.21.15.144:0/3764813546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f47501371f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 --2- 172.21.15.144:0/3764813546 >> v2:172.21.15.116:3300/0 conn(0x7f475009c1a0 0x7f47501348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 -- 172.21.15.144:0/3764813546 >> 172.21.15.144:0/3764813546 conn(0x7f4750070e90 msgr2=0x7f47500a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 -- 172.21.15.144:0/3764813546 shutdown_connections 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.780+0000 7f4757a4f640 1 -- 172.21.15.144:0/3764813546 wait complete. 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.781+0000 7f4757a4f640 10 monclient: build_initial_monmap 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.781+0000 7f4757a4f640 10 monclient: monmap: 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:48.801 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.781+0000 7f4757a4f640 1 Processor -- start 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.781+0000 7f4757a4f640 1 -- start start 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.781+0000 7f4757a4f640 10 monclient: init 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f475009c1a0 0x7f475011c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f475009c1a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:48.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f475009cad0 0x7f475011ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f475009cad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f475011d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f4750134e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47500a2200 con 0x7f475009cad0 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475011fb80 con 0x7f4750134e10 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475011fe60 con 0x7f475009c1a0 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4757a4f640 10 monclient(hunting): authenticate will time out at 1024.351163s 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f475011d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:48.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4754fea640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f475009cad0 msgr2=0x7f475011ccd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f475011d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42562/0 (socket says 172.21.15.144:42562) 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4754fea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f475009cad0 0x7f475011ccd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 -- 172.21.15.144:0/2400665549 learned_addr learned my addr 172.21.15.144:0/2400665549 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): get_auth_request con 0x7f4750134e10 auth_method 0 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f47557eb640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f475009c1a0 0x7f475011c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f47557eb640 10 monclient(hunting): get_auth_request con 0x7f475009c1a0 auth_method 0 2024-09-18T12:29:48.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f47557eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f47557eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f47557eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient(hunting): handle_auth_done global_id 5344 payload 824 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 -- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f475009c1a0 msgr2=0x7f475011c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f475009c1a0 0x7f475011c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 -- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f475009cad0 msgr2=0x7f475011ccd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f475009cad0 0x7f475011ccd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient: _finish_hunting 0 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 monclient: found mon.noname-b 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f4755fec640 1 -- 172.21.15.144:0/2400665549 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748009a20 con 0x7f4750134e10 2024-09-18T12:29:48.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.782+0000 7f47557eb640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f475009c1a0 0x7f475011c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4755fec640 1 -- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 msgr2=0x7f475011d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4755fec640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f475011d210 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f47480099f0 tx=0x7f4748037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748031cd0 con 0x7f4750134e10 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: dump: 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:48.806 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: _finish_auth 0 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4757a4f640 5 monclient: authenticate success, global_id 5344 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748040070 con 0x7f4750134e10 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4757a4f640 10 monclient: _renew_subs 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4757a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475011e050 con 0x7f4750134e10 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47480456d0 con 0x7f4750134e10 2024-09-18T12:29:48.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47457fa640 10 monclient: dump: 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f47477fe640 4 set_mon_vals no callback set 2024-09-18T12:29:48.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4757a4f640 10 monclient: _renew_subs 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4757a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.784+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475011ec20 con 0x7f4750134e10 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.785+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4748042030 con 0x7f4750134e10 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.785+0000 7f4757a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.785+0000 7f4757a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.785+0000 7f4757a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.785+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4750120020 con 0x7f4750134e10 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.785+0000 7f47457fa640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4734038440 0x7f473403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.786+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5555+0+0 (secure 0 0 0) 0x7f474806d340 con 0x7f4750134e10 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.786+0000 7f47557eb640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4734038440 0x7f473403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.786+0000 7f47557eb640 10 monclient: get_auth_request con 0x7f4734038440 auth_method 0 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.788+0000 7f47557eb640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4734038440 0x7f473403a8f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f474c005840 tx=0x7f474c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:48.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.790+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4748031e30 con 0x7f4750134e10 2024-09-18T12:29:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.790+0000 7f47457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:48.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.790+0000 7f47457fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:48.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.948+0000 7f4757a4f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T12:29:48.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.948+0000 7f4757a4f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T12:29:48.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.948+0000 7f4757a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:48.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:48.948+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7f47500008d0 con 0x7f4750134e10 2024-09-18T12:29:49.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.591+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v100) v1 ==== 138+0+0 (secure 0 0 0) 0x7f474803ab90 con 0x7f4750134e10 2024-09-18T12:29:49.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.591+0000 7f47457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T12:29:49.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.591+0000 7f47457fa640 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-18T12:29:49.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.690+0000 7f4757a4f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T12:29:49.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.690+0000 7f4757a4f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T12:29:49.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.690+0000 7f4757a4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:49.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.690+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7f475009b430 con 0x7f4750134e10 2024-09-18T12:29:49.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.690+0000 7f47457fa640 1 -- 172.21.15.144:0/2400665549 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v100) v1 ==== 126+0+0 (secure 0 0 0) 0x7f474803ad70 con 0x7f4750134e10 2024-09-18T12:29:49.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.690+0000 7f47457fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T12:29:49.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.690+0000 7f47457fa640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-18T12:29:49.692 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'foo' already exists 2024-09-18T12:29:49.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4734038440 msgr2=0x7f473403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:49.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4734038440 0x7f473403a8f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f474c005840 tx=0x7f474c009040 comp rx=0 tx=0).stop 2024-09-18T12:29:49.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 10 monclient: shutdown 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 msgr2=0x7f475011d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f475011d210 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f47480099f0 tx=0x7f4748037000 comp rx=0 tx=0).stop 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 shutdown_connections 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4734038440 0x7f473403a8f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4750134e10 0x7f475011d210 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f475009cad0 0x7f475011ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 --2- 172.21.15.144:0/2400665549 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f475009c1a0 0x7f475011c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 >> 172.21.15.144:0/2400665549 conn(0x7f4750070e90 msgr2=0x7f47500a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 shutdown_connections 2024-09-18T12:29:49.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.692+0000 7f4757a4f640 1 -- 172.21.15.144:0/2400665549 wait complete. 2024-09-18T12:29:49.702 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-18T12:29:49.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.776+0000 7f619659d640 10 monclient: get_monmap_and_config 2024-09-18T12:29:49.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.776+0000 7f619659d640 10 monclient: build_initial_monmap 2024-09-18T12:29:49.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.776+0000 7f619659d640 10 monclient: monmap: 2024-09-18T12:29:49.777 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:29:49.777352+0000 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:29:49.777352+0000 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:29:49.778 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 1 Processor -- start 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 1 -- start start 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient: init 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f619009d140 addr v1:172.21.15.116:6789/0 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 0x7f6190136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:49.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient(hunting): picked mon.noname-b con 0x7f619009dc20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f61901374e0 0x7f61901398c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient(hunting): picked mon.noname-a con 0x7f61901374e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619009ec20 con 0x7f619009d140 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619006f7a0 con 0x7f61901374e0 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6190098280 con 0x7f619009dc20 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f619659d640 10 monclient(hunting): authenticate will time out at 1025.346677s 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f618ffff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f619009d140 legacy=0x7f619009d510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f6194b3a640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f61901374e0 msgr2=0x7f61901398c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:49.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f6194b3a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f61901374e0 0x7f61901398c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.777+0000 7f618f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 0x7f6190136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 0x7f6190136fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:54622/0 (socket says 172.21.15.144:54622) 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 learned_addr learned my addr 172.21.15.144:0/2423907916 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): get_auth_request con 0x7f619009dc20 auth_method 0 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:49.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient(hunting): handle_auth_done global_id 64259 payload 274 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 >> v2:172.21.15.116:3300/0 conn(0x7f61901374e0 msgr2=0x7f61901398c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 1 --2- 172.21.15.144:0/2423907916 >> v2:172.21.15.116:3300/0 conn(0x7f61901374e0 0x7f61901398c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 >> v1:172.21.15.116:6789/0 conn(0x7f619009d140 legacy=0x7f619009d510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:29:49.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.778+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6190064cb0 con 0x7f619009dc20 2024-09-18T12:29:49.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 msgr2=0x7f6190136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:49.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618f7fe640 1 --2- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 0x7f6190136fa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6184009910 tx=0x7f61840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31296ad167a2cac9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 msgr2=0x7f6190136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 msgr2=0x7f6190136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618f7fe640 1 --2- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 0x7f6190136fa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6184009910 tx=0x7f61840091d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618f7fe640 1 --2- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 0x7f6190136fa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6184009910 tx=0x7f61840091d0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 1 -- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 msgr2=0x7f6190136fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 1 --2- 172.21.15.144:0/2423907916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619009dc20 0x7f6190136fa0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient: _add_conns ranks=[4,1] 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 1 --2- 172.21.15.144:0/2423907916 >> v2:172.21.15.144:3300/0 conn(0x7f61780034c0 0x7f6178005930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient(hunting): picked mon.noname-c con 0x7f61780034c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:29:49.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6178006260 addr v1:172.21.15.144:6789/0 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 1 -- 172.21.15.144:0/2423907916 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617800a1c0 con 0x7f6178006260 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 1 -- 172.21.15.144:0/2423907916 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617800b380 con 0x7f61780034c0 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618effd640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 1 --2- 172.21.15.144:0/2423907916 >> v2:172.21.15.144:3300/0 conn(0x7f61780034c0 0x7f6178005930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 10 monclient(hunting): get_auth_request con 0x7f61780034c0 auth_method 0 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:49.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.781+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618f7fe640 1 -- 172.21.15.144:0/2423907916 >> v1:172.21.15.144:6789/0 conn(0x7f6178006260 legacy=0x7f6178008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618ffff640 10 monclient(hunting): handle_auth_done global_id 64259 payload 293 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618ffff640 1 -- 172.21.15.144:0/2423907916 >> v1:172.21.15.144:6789/0 conn(0x7f6178006260 legacy=0x7f6178008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618ffff640 1 monclient: found mon.noname-c 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.783+0000 7f618ffff640 1 -- 172.21.15.144:0/2423907916 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617800b5b0 con 0x7f61780034c0 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618ffff640 1 -- 172.21.15.144:0/2423907916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 msgr2=0x7f6178005930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618ffff640 1 --2- 172.21.15.144:0/2423907916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f6178005930 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f61800038f0 tx=0x7f6180009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd66b1624c788409 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 1 -- 172.21.15.144:0/2423907916 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180003b80 con 0x7f61780034c0 2024-09-18T12:29:49.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: dump: 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:49.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: _finish_auth 0 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 1 -- 172.21.15.144:0/2423907916 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6180003ce0 con 0x7f61780034c0 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 5 monclient: authenticate success, global_id 64259 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 10 monclient: get_monmap_and_config success 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 1 -- 172.21.15.144:0/2423907916 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180005340 con 0x7f61780034c0 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 4 set_mon_vals no callback set 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f618effd640 10 monclient: dump: 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:49.793 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 10 monclient: shutdown 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 1 -- 172.21.15.144:0/2423907916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 msgr2=0x7f6178005930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 1 --2- 172.21.15.144:0/2423907916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f6178005930 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f61800038f0 tx=0x7f6180009060 comp rx=0 tx=0).stop 2024-09-18T12:29:49.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f6194b3a640 1 -- 172.21.15.144:0/2423907916 reap_dead start 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 1 -- 172.21.15.144:0/2423907916 shutdown_connections 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 1 --2- 172.21.15.144:0/2423907916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f6178005930 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 1 -- 172.21.15.144:0/2423907916 >> 172.21.15.144:0/2423907916 conn(0x7f61900a2190 msgr2=0x7f619009e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 1 -- 172.21.15.144:0/2423907916 shutdown_connections 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.786+0000 7f619659d640 1 -- 172.21.15.144:0/2423907916 wait complete. 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.787+0000 7f619659d640 10 monclient: build_initial_monmap 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.787+0000 7f619659d640 10 monclient: monmap: 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:49.795 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:29:49.796 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:29:49.796 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:29:49.796 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:49.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.787+0000 7f619659d640 1 Processor -- start 2024-09-18T12:29:49.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.787+0000 7f619659d640 1 -- start start 2024-09-18T12:29:49.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.787+0000 7f619659d640 10 monclient: init 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient: _reopen_session rank -1 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f619006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): picked mon.noname-b con 0x7f61780034c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619006af20 0x7f619006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): picked mon.noname-c con 0x7f619006af20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f619006b830 0x7f619014b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): picked mon.noname-a con 0x7f619006b830 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6190095090 con 0x7f619006b830 2024-09-18T12:29:49.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6190136ed0 con 0x7f61780034c0 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619006dc40 con 0x7f619006af20 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): _renew_subs 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f619659d640 10 monclient(hunting): authenticate will time out at 1025.357414s 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f619006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f619006a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42576/0 (socket says 172.21.15.144:42576) 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 -- 172.21.15.144:0/706652888 learned_addr learned my addr 172.21.15.144:0/706652888 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): get_auth_request con 0x7f61780034c0 auth_method 0 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:49.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f6194b3a640 1 -- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f619006b830 msgr2=0x7f619014b9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f6194b3a640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f619006b830 0x7f619014b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618f7fe640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619006af20 0x7f619006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618f7fe640 10 monclient(hunting): get_auth_request con 0x7f619006af20 auth_method 0 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:29:49.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient(hunting): handle_auth_done global_id 5347 payload 824 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 -- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619006af20 msgr2=0x7f619006b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619006af20 0x7f619006b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 -- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f619006b830 msgr2=0x7f619014b9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f619006b830 0x7f619014b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 monclient: found mon.noname-b 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.788+0000 7f618ffff640 1 -- 172.21.15.144:0/706652888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6190126e90 con 0x7f61780034c0 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.790+0000 7f618ffff640 1 -- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 msgr2=0x7f619006a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.790+0000 7f618ffff640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f619006a9e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f618000ba60 tx=0x7f6180015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180003e50 con 0x7f61780034c0 2024-09-18T12:29:49.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: dump: 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:49.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: _finish_auth 0 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6180005800 con 0x7f61780034c0 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f619659d640 5 monclient: authenticate success, global_id 5347 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f619659d640 10 monclient: _renew_subs 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f619659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619006e170 con 0x7f61780034c0 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618001c890 con 0x7f61780034c0 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f6177fff640 10 monclient: dump: 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:29:49.805 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f619659d640 10 monclient: _renew_subs 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f618e7fc640 4 set_mon_vals no callback set 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f619659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.791+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619006cc50 con 0x7f61780034c0 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6180023070 con 0x7f61780034c0 2024-09-18T12:29:49.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f619659d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f619659d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f619659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61900008d0 con 0x7f61780034c0 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f6177fff640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f617c0384c0 0x7f617c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5582+0+0 (secure 0 0 0) 0x7f618004d940 con 0x7f61780034c0 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f618f7fe640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f617c0384c0 0x7f617c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.792+0000 7f618f7fe640 10 monclient: get_auth_request con 0x7f617c0384c0 auth_method 0 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.794+0000 7f618f7fe640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f617c0384c0 0x7f617c03a970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f618402a630 tx=0x7f6184032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.796+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6180006b20 con 0x7f61780034c0 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.796+0000 7f6177fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:29:49.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.796+0000 7f6177fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:29:49.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.956+0000 7f619659d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T12:29:49.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.956+0000 7f619659d640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T12:29:49.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.956+0000 7f619659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:29:49.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:49.956+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7f619006c080 con 0x7f61780034c0 2024-09-18T12:29:50.188 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:29:50.188 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:29:50.188 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:29:50.189 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:29:50.192 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:29:50.192 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T12:29:50.192 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T12:29:50.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:29:52.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:52.788+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:52.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:52.788+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:52.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:52.788+0000 7f6176ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:29:53.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:53.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:53.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:53.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:54.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:54.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:54.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:54.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:55.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:55.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:55.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:55.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:56.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:56.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:56.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:56.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:57.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:57.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:57.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:57.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:58.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:58.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:58.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:58.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:29:59.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:59.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:29:59.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:29:59.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:00.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:00.790+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:30:00.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:00.790+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:01.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:01.790+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:30:01.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:01.790+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:02.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.789+0000 7f6176ffd640 10 monclient: tick 2024-09-18T12:30:02.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.789+0000 7f6176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:02.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.789+0000 7f6176ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:02.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.837+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v101) v1 ==== 204+0+0 (secure 0 0 0) 0x7f61800103a0 con 0x7f61780034c0 2024-09-18T12:30:02.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.837+0000 7f6177fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T12:30:02.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.837+0000 7f6177fff640 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-18T12:30:02.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.938+0000 7f619659d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T12:30:02.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.938+0000 7f619659d640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T12:30:02.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.938+0000 7f619659d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:02.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.938+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7f619006c080 con 0x7f61780034c0 2024-09-18T12:30:02.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.940+0000 7f6177fff640 1 -- 172.21.15.144:0/706652888 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v101) v1 ==== 214+0+0 (secure 0 0 0) 0x7f61800109f0 con 0x7f61780034c0 2024-09-18T12:30:02.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.940+0000 7f6177fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T12:30:02.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.940+0000 7f6177fff640 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-18T12:30:02.941 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-18T12:30:02.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f617c0384c0 msgr2=0x7f617c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:02.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f617c0384c0 0x7f617c03a970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f618402a630 tx=0x7f6184032000 comp rx=0 tx=0).stop 2024-09-18T12:30:02.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 10 monclient: shutdown 2024-09-18T12:30:02.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:02.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 msgr2=0x7f619006a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f619006a9e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f618000ba60 tx=0x7f6180015000 comp rx=0 tx=0).stop 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 shutdown_connections 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f617c0384c0 0x7f617c03a970 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f619006b830 0x7f619014b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f619006af20 0x7f619006b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 --2- 172.21.15.144:0/706652888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61780034c0 0x7f619006a9e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 >> 172.21.15.144:0/706652888 conn(0x7f61900a2190 msgr2=0x7f619009f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 shutdown_connections 2024-09-18T12:30:02.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:02.942+0000 7f619659d640 1 -- 172.21.15.144:0/706652888 wait complete. 2024-09-18T12:30:02.952 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2024-09-18T12:30:03.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.026+0000 7fc776660640 10 monclient: get_monmap_and_config 2024-09-18T12:30:03.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.026+0000 7fc776660640 10 monclient: build_initial_monmap 2024-09-18T12:30:03.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.026+0000 7fc776660640 10 monclient: monmap: 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:03.027827+0000 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:03.027827+0000 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:03.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.027+0000 7fc776660640 1 Processor -- start 2024-09-18T12:30:03.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.027+0000 7fc776660640 1 -- start start 2024-09-18T12:30:03.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.027+0000 7fc776660640 10 monclient: init 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc770098170 0x7fc770097a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient(hunting): picked mon.noname-a con 0x7fc770098170 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc770098540 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 0x7fc770069130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient(hunting): picked mon.noname-b con 0x7fc770098940 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77009d1c0 con 0x7fc770098540 2024-09-18T12:30:03.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc770096db0 con 0x7fc770098170 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc770097f60 con 0x7fc770098940 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc776660640 10 monclient(hunting): authenticate will time out at 1038.597250s 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc770098170 0x7fc770097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc770098540 0x7fc770068a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:36280/0 (socket says 172.21.15.144:36280) 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc770098170 0x7fc770097a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:35420/0 (socket says 172.21.15.144:35420) 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76f7fe640 1 -- 172.21.15.144:0/3276794218 learned_addr learned my addr 172.21.15.144:0/3276794218 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): get_auth_request con 0x7fc770098170 auth_method 0 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:03.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient(hunting): handle_auth_done global_id 74097 payload 274 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 1 -- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 msgr2=0x7fc770069130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 1 --2- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 0x7fc770069130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 1 -- 172.21.15.144:0/3276794218 >> v1:172.21.15.116:6789/0 conn(0x7fc770098540 legacy=0x7fc770068a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 1 monclient: found mon.noname-a 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:03.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.028+0000 7fc76ffff640 1 -- 172.21.15.144:0/3276794218 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc770098d10 con 0x7fc770098170 2024-09-18T12:30:03.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.031+0000 7fc76ffff640 1 -- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098170 msgr2=0x7fc770097a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:03.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.031+0000 7fc76ffff640 1 --2- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098170 0x7fc770097a20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc760009850 tx=0x7fc7600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36f28ca600504b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:03.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 1 -- 172.21.15.144:0/3276794218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760032440 con 0x7fc770098170 2024-09-18T12:30:03.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:03.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:03.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: dump: 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: _finish_auth 0 2024-09-18T12:30:03.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 1 -- 172.21.15.144:0/3276794218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7600325a0 con 0x7fc770098170 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 1 -- 172.21.15.144:0/3276794218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760032940 con 0x7fc770098170 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc76effd640 10 monclient: dump: 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:03.042 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc776660640 5 monclient: authenticate success, global_id 74097 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc776660640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.032+0000 7fc776660640 4 set_mon_vals no callback set 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 10 monclient: shutdown 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 -- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098170 msgr2=0x7fc770097a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 --2- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098170 0x7fc770097a20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc760009850 tx=0x7fc7600090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 -- 172.21.15.144:0/3276794218 shutdown_connections 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 --2- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 0x7fc770069130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 --2- 172.21.15.144:0/3276794218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098170 0x7fc770097a20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 -- 172.21.15.144:0/3276794218 >> 172.21.15.144:0/3276794218 conn(0x7fc7700a2b70 msgr2=0x7fc77009ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:03.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 -- 172.21.15.144:0/3276794218 shutdown_connections 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 -- 172.21.15.144:0/3276794218 wait complete. 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 10 monclient: build_initial_monmap 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 10 monclient: monmap: 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:03.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.033+0000 7fc776660640 1 Processor -- start 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 1 -- start start 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient: init 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 0x7fc77013ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): picked mon.noname-b con 0x7fc770098170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098540 0x7fc770132cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): picked mon.noname-a con 0x7fc770098540 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 0x7fc77011c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): picked mon.noname-c con 0x7fc770098940 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77013fe40 con 0x7fc770098540 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:03.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77006ed20 con 0x7fc770098170 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700a1870 con 0x7fc770098940 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc776660640 10 monclient(hunting): authenticate will time out at 1038.603531s 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 0x7fc77013ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 0x7fc77013ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41462/0 (socket says 172.21.15.144:41462) 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 -- 172.21.15.144:0/1651472473 learned_addr learned my addr 172.21.15.144:0/1651472473 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): get_auth_request con 0x7fc770098170 auth_method 0 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:03.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient(hunting): handle_auth_done global_id 5350 payload 824 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76f7fe640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098540 0x7fc770132cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 -- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 msgr2=0x7fc77011c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 0x7fc77011c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 -- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098540 msgr2=0x7fc770132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098540 0x7fc770132cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 monclient: found mon.noname-b 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76ffff640 1 -- 172.21.15.144:0/1651472473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760009390 con 0x7fc770098170 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76f7fe640 10 monclient: get_auth_request con 0x7fc770098540 auth_method 0 2024-09-18T12:30:03.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.034+0000 7fc76f7fe640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098540 0x7fc770132cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc76ffff640 1 -- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 msgr2=0x7fc77013ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc76ffff640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 0x7fc77013ef10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc760009850 tx=0x7fc760032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760031c50 con 0x7fc770098170 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: dump: 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:03.048 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc760031db0 con 0x7fc770098170 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc776660640 5 monclient: authenticate success, global_id 5350 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc776660640 10 monclient: _renew_subs 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc776660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7701200b0 con 0x7fc770098170 2024-09-18T12:30:03.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76003d690 con 0x7fc770098170 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc757fff640 10 monclient: dump: 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:03.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc776660640 10 monclient: _renew_subs 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc776660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77011eee0 con 0x7fc770098170 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.037+0000 7fc76e7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc760039040 con 0x7fc770098170 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc776660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc776660640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc776660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7700008d0 con 0x7fc770098170 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc757fff640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc75c038490 0x7fc75c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc76006bb60 con 0x7fc770098170 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc76f7fe640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc75c038490 0x7fc75c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:03.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.038+0000 7fc76f7fe640 10 monclient: get_auth_request con 0x7fc75c038490 auth_method 0 2024-09-18T12:30:03.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.040+0000 7fc76f7fe640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc75c038490 0x7fc75c03a940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc764007fd0 tx=0x7fc764009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:03.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.042+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc76003bc70 con 0x7fc770098170 2024-09-18T12:30:03.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.042+0000 7fc757fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:03.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.042+0000 7fc757fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:03.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.201+0000 7fc776660640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T12:30:03.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.201+0000 7fc776660640 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T12:30:03.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.201+0000 7fc776660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:03.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.201+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7fc770096ad0 con 0x7fc770098170 2024-09-18T12:30:03.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.848+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map v102) v1 ==== 212+0+0 (secure 0 0 0) 0x7fc7600370c0 con 0x7fc770098170 2024-09-18T12:30:03.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.848+0000 7fc757fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T12:30:03.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.848+0000 7fc757fff640 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-18T12:30:03.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.946+0000 7fc776660640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T12:30:03.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.946+0000 7fc776660640 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T12:30:03.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.946+0000 7fc776660640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:03.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.946+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7fc770120240 con 0x7fc770098170 2024-09-18T12:30:03.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.947+0000 7fc757fff640 1 -- 172.21.15.144:0/1651472473 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map v102) v1 ==== 211+0+0 (secure 0 0 0) 0x7fc760042030 con 0x7fc770098170 2024-09-18T12:30:03.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.947+0000 7fc757fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T12:30:03.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.947+0000 7fc757fff640 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-18T12:30:03.949 INFO:tasks.workunit.client.0.smithi144.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-18T12:30:03.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc75c038490 msgr2=0x7fc75c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc75c038490 0x7fc75c03a940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc764007fd0 tx=0x7fc764009040 comp rx=0 tx=0).stop 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 10 monclient: shutdown 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 msgr2=0x7fc77013ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 0x7fc77013ef10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc760009850 tx=0x7fc760032000 comp rx=0 tx=0).stop 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 shutdown_connections 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc75c038490 0x7fc75c03a940 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc770098940 0x7fc77011c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc770098540 0x7fc770132cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 --2- 172.21.15.144:0/1651472473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc770098170 0x7fc77013ef10 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 >> 172.21.15.144:0/1651472473 conn(0x7fc7700a2b70 msgr2=0x7fc77006a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:03.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 shutdown_connections 2024-09-18T12:30:03.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:03.949+0000 7fc776660640 1 -- 172.21.15.144:0/1651472473 wait complete. 2024-09-18T12:30:03.960 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush move osd.7 host=foo 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.032+0000 7fa4f4e34640 10 monclient: get_monmap_and_config 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.032+0000 7fa4f4e34640 10 monclient: build_initial_monmap 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.032+0000 7fa4f4e34640 10 monclient: monmap: 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:04.033589+0000 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:04.033589+0000 2024-09-18T12:30:04.034 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 1 Processor -- start 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 1 -- start start 2024-09-18T12:30:04.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient: init 2024-09-18T12:30:04.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:04.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T12:30:04.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 0x7fa4f0099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:04.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f0098c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4f0099580 0x7fa4f009dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f0099580 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4f0099c70 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f009f680 con 0x7fa4f0099c70 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f009f420 con 0x7fa4f0099580 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0068a20 con 0x7fa4f0098c70 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4f4e34640 10 monclient(hunting): authenticate will time out at 1039.602857s 2024-09-18T12:30:04.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.033+0000 7fa4eed9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa4f0099c70 0x7fa4f009e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:46402/0 (socket says 172.21.15.144:46402) 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4eed9d640 1 -- 172.21.15.144:0/2745093216 learned_addr learned my addr 172.21.15.144:0/2745093216 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 1 --2- 172.21.15.144:0/2745093216 >> v2:172.21.15.116:3300/0 conn(0x7fa4f0099580 0x7fa4f009dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request con 0x7fa4f0099580 auth_method 0 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_done global_id 74100 payload 274 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 1 -- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 msgr2=0x7fa4f0099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:04.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 1 --2- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 0x7fa4f0099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 1 -- 172.21.15.144:0/2745093216 >> v1:172.21.15.144:6789/0 conn(0x7fa4f0099c70 legacy=0x7fa4f009e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:04.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient: _finish_hunting 0 2024-09-18T12:30:04.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 1 monclient: found mon.noname-a 2024-09-18T12:30:04.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:04.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:04.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.034+0000 7fa4edd9b640 1 -- 172.21.15.144:0/2745093216 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f0096a30 con 0x7fa4f0099580 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.037+0000 7fa4edd9b640 1 -- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099580 msgr2=0x7fa4f009dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.037+0000 7fa4edd9b640 1 --2- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099580 0x7fa4f009dad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009850 tx=0x7fa4e40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a75ee529f3513a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 1 -- 172.21.15.144:0/2745093216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4032440 con 0x7fa4f0099580 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: dump: 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: _finish_auth 0 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 1 -- 172.21.15.144:0/2745093216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e40325a0 con 0x7fa4f0099580 2024-09-18T12:30:04.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 5 monclient: authenticate success, global_id 74100 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 1 -- 172.21.15.144:0/2745093216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4032940 con 0x7fa4f0099580 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 4 set_mon_vals no callback set 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4ed59a640 10 monclient: dump: 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:04.048 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 10 monclient: shutdown 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 -- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099580 msgr2=0x7fa4f009dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099580 0x7fa4f009dad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009850 tx=0x7fa4e40090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 -- 172.21.15.144:0/2745093216 shutdown_connections 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099580 0x7fa4f009dad0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/2745093216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 0x7fa4f0099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 -- 172.21.15.144:0/2745093216 >> 172.21.15.144:0/2745093216 conn(0x7fa4f0070e90 msgr2=0x7fa4f00a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 -- 172.21.15.144:0/2745093216 shutdown_connections 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.038+0000 7fa4f4e34640 1 -- 172.21.15.144:0/2745093216 wait complete. 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.039+0000 7fa4f4e34640 10 monclient: build_initial_monmap 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.039+0000 7fa4f4e34640 10 monclient: monmap: 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:04.050 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.039+0000 7fa4f4e34640 1 Processor -- start 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.039+0000 7fa4f4e34640 1 -- start start 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.039+0000 7fa4f4e34640 10 monclient: init 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 0x7fa4f0141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f0098c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 0x7fa4f0134120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f0099580 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099c70 0x7fa4f011ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f0099c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0122530 con 0x7fa4f0099c70 2024-09-18T12:30:04.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0120070 con 0x7fa4f0099580 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0120ca0 con 0x7fa4f0098c70 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4f4e34640 10 monclient(hunting): authenticate will time out at 1039.609222s 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 0x7fa4f0134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 0x7fa4f0134120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41470/0 (socket says 172.21.15.144:41470) 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 -- 172.21.15.144:0/853513108 learned_addr learned my addr 172.21.15.144:0/853513108 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request con 0x7fa4f0099580 auth_method 0 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:04.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4eed9d640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099c70 0x7fa4f011ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4eed9d640 10 monclient(hunting): get_auth_request con 0x7fa4f0099c70 auth_method 0 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_done global_id 5353 payload 824 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 -- 172.21.15.144:0/853513108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 msgr2=0x7fa4f0141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 0x7fa4f0141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 -- 172.21.15.144:0/853513108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099c70 msgr2=0x7fa4f011ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099c70 0x7fa4f011ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient: _finish_hunting 0 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 monclient: found mon.noname-b 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.040+0000 7fa4edd9b640 1 -- 172.21.15.144:0/853513108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e4009390 con 0x7fa4f0099580 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.042+0000 7fa4edd9b640 1 -- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 msgr2=0x7fa4f0134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.042+0000 7fa4edd9b640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 0x7fa4f0134120 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009850 tx=0x7fa4e4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e40318c0 con 0x7fa4f0099580 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: dump: 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:04.054 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 5 monclient: authenticate success, global_id 5353 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 10 monclient: _renew_subs 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e4031a20 con 0x7fa4f0099580 2024-09-18T12:30:04.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f006eba0 con 0x7fa4f0099580 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4031da0 con 0x7fa4f0099580 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 10 monclient: dump: 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:04.056 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4ecd99640 4 set_mon_vals no callback set 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 10 monclient: _renew_subs 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f0121360 con 0x7fa4f0099580 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa4e4039070 con 0x7fa4f0099580 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.043+0000 7fa4f4e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.044+0000 7fa4f4e34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.044+0000 7fa4f4e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.044+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f00008d0 con 0x7fa4f0099580 2024-09-18T12:30:04.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.044+0000 7fa4de7fc640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa4d0038490 0x7fa4d003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:04.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.044+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5900+0+0 (secure 0 0 0) 0x7fa4e406c360 con 0x7fa4f0099580 2024-09-18T12:30:04.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.044+0000 7fa4ee59c640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa4d0038490 0x7fa4d003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:04.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.044+0000 7fa4ee59c640 10 monclient: get_auth_request con 0x7fa4d0038490 auth_method 0 2024-09-18T12:30:04.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.046+0000 7fa4ee59c640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa4d0038490 0x7fa4d003a940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00097e0 tx=0x7fa4e0006d20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:04.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.048+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa4e403c620 con 0x7fa4f0099580 2024-09-18T12:30:04.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.048+0000 7fa4de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:04.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.048+0000 7fa4de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:04.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.207+0000 7fa4f4e34640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-18T12:30:04.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.207+0000 7fa4f4e34640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-18T12:30:04.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.207+0000 7fa4f4e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:04.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.207+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7fa4f00a01e0 con 0x7fa4f0099580 2024-09-18T12:30:04.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.849+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 moved item id 7 name 'osd.7' to location {host=foo} in crush map v103) v1 ==== 165+0+0 (secure 0 0 0) 0x7fa4e40400a0 con 0x7fa4f0099580 2024-09-18T12:30:04.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.849+0000 7fa4de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-18T12:30:04.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.849+0000 7fa4de7fc640 10 monclient: _finish_command 2 = system:0 moved item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-18T12:30:04.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.950+0000 7fa4f4e34640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-18T12:30:04.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.950+0000 7fa4f4e34640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-18T12:30:04.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.950+0000 7fa4f4e34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:04.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.950+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) v1 -- 0x7fa4f00a01e0 con 0x7fa4f0099580 2024-09-18T12:30:04.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.951+0000 7fa4de7fc640 1 -- 172.21.15.144:0/853513108 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map v103) v1 ==== 175+0+0 (secure 0 0 0) 0x7fa4e4037020 con 0x7fa4f0099580 2024-09-18T12:30:04.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.951+0000 7fa4de7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-18T12:30:04.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.951+0000 7fa4de7fc640 10 monclient: _finish_command 3 = system:0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-18T12:30:04.953 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.953+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa4d0038490 msgr2=0x7fa4d003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.953+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa4d0038490 0x7fa4d003a940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00097e0 tx=0x7fa4e0006d20 comp rx=0 tx=0).stop 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.953+0000 7fa4f4e34640 10 monclient: shutdown 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.953+0000 7fa4f4e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.953+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 msgr2=0x7fa4f0134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.953+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 0x7fa4f0134120 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009850 tx=0x7fa4e4032000 comp rx=0 tx=0).stop 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 shutdown_connections 2024-09-18T12:30:04.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa4d0038490 0x7fa4d003a940 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4f0099c70 0x7fa4f011ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4f0099580 0x7fa4f0134120 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 --2- 172.21.15.144:0/853513108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4f0098c70 0x7fa4f0141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:04.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 >> 172.21.15.144:0/853513108 conn(0x7fa4f0070e90 msgr2=0x7fa4f009c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:04.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 shutdown_connections 2024-09-18T12:30:04.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:04.954+0000 7fa4f4e34640 1 -- 172.21.15.144:0/853513108 wait complete. 2024-09-18T12:30:04.964 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd find osd.7 2024-09-18T12:30:04.964 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep host 2024-09-18T12:30:04.964 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep foo 2024-09-18T12:30:05.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.034+0000 7ffa1e71d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.034+0000 7ffa1e71d640 10 monclient: build_initial_monmap 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.034+0000 7ffa1e71d640 10 monclient: monmap: 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:05.035622+0000 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:05.035622+0000 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:05.036 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:05.037 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:05.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 1 Processor -- start 2024-09-18T12:30:05.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 1 -- start start 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 10 monclient: init 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa180a37b0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffa1806be40 0x7ffa18136c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.035+0000 7ffa1e71d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1806be40 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa1e71d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa1806a6f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa1e71d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa18099ff0 con 0x7ffa1806a6f0 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa1e71d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa180a3b80 con 0x7ffa180a37b0 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa1e71d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa18069d10 con 0x7ffa1806be40 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa1e71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:05.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa1e71d640 10 monclient(hunting): authenticate will time out at 1040.605084s 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa17fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ffa180a37b0 0x7ffa1806e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:46418/0 (socket says 172.21.15.144:46418) 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa17fff640 1 -- 172.21.15.144:0/2712588924 learned_addr learned my addr 172.21.15.144:0/2712588924 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 1 --2- 172.21.15.144:0/2712588924 >> v2:172.21.15.116:3300/0 conn(0x7ffa1806be40 0x7ffa18136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 10 monclient(hunting): get_auth_request con 0x7ffa1806be40 auth_method 0 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.036+0000 7ffa177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa17fff640 1 -- 172.21.15.144:0/2712588924 >> v1:172.21.15.144:6789/0 conn(0x7ffa180a37b0 legacy=0x7ffa1806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa16ffd640 1 -- 172.21.15.144:0/2712588924 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226196021 0 0) 0x7ffa180a3b80 con 0x7ffa180a37b0 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa16ffd640 10 monclient(hunting): my global_id is 5356 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa16ffd640 1 -- 172.21.15.144:0/2712588924 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa040034e0 con 0x7ffa180a37b0 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa1ccba640 1 -- 172.21.15.144:0/2712588924 >> v1:172.21.15.116:6789/0 conn(0x7ffa1806a6f0 legacy=0x7ffa181391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 10 monclient(hunting): handle_auth_done global_id 74106 payload 274 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa16ffd640 1 -- 172.21.15.144:0/2712588924 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253176001 0 0) 0x7ffa040034e0 con 0x7ffa180a37b0 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 1 -- 172.21.15.144:0/2712588924 >> v1:172.21.15.144:6789/0 conn(0x7ffa180a37b0 legacy=0x7ffa1806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 1 -- 172.21.15.144:0/2712588924 >> v1:172.21.15.116:6789/0 conn(0x7ffa1806a6f0 legacy=0x7ffa181391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 1 monclient: found mon.noname-a 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:05.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:05.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa177fe640 1 -- 172.21.15.144:0/2712588924 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1809f520 con 0x7ffa1806be40 2024-09-18T12:30:05.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.038+0000 7ffa16ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:30:05.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.041+0000 7ffa177fe640 1 -- 172.21.15.144:0/2712588924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa1806be40 msgr2=0x7ffa18136c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.041+0000 7ffa177fe640 1 --2- 172.21.15.144:0/2712588924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa1806be40 0x7ffa18136c50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa000098d0 tx=0x7ffa000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef23c2146cbddc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.041+0000 7ffa16ffd640 1 -- 172.21.15.144:0/2712588924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa080017a0 con 0x7ffa1806be40 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: dump: 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.047 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 5 monclient: authenticate success, global_id 74106 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 1 -- 172.21.15.144:0/2712588924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0002ad00 con 0x7ffa1806be40 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 1 -- 172.21.15.144:0/2712588924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00032650 con 0x7ffa1806be40 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 4 set_mon_vals no callback set 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa16ffd640 10 monclient: dump: 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 10 monclient: shutdown 2024-09-18T12:30:05.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 1 -- 172.21.15.144:0/2712588924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa1806be40 msgr2=0x7ffa18136c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/2712588924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa1806be40 0x7ffa18136c50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa000098d0 tx=0x7ffa000090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 1 -- 172.21.15.144:0/2712588924 shutdown_connections 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/2712588924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa1806be40 0x7ffa18136c50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 1 -- 172.21.15.144:0/2712588924 >> 172.21.15.144:0/2712588924 conn(0x7ffa18070fd0 msgr2=0x7ffa180a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 1 -- 172.21.15.144:0/2712588924 shutdown_connections 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.042+0000 7ffa1e71d640 1 -- 172.21.15.144:0/2712588924 wait complete. 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.043+0000 7ffa1e71d640 10 monclient: build_initial_monmap 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.043+0000 7ffa1e71d640 10 monclient: monmap: 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:05.050 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.043+0000 7ffa1e71d640 1 Processor -- start 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.043+0000 7ffa1e71d640 1 -- start start 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.043+0000 7ffa1e71d640 10 monclient: init 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 0x7ffa1813d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1806a6f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1806be40 0x7ffa1813cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1806be40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa180a37b0 0x7ffa18140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa180a37b0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181228f0 con 0x7ffa180a37b0 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18122330 con 0x7ffa1806a6f0 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1811fea0 con 0x7ffa1806be40 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1e71d640 10 monclient(hunting): authenticate will time out at 1040.613144s 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 0x7ffa1813d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 0x7ffa1813d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41482/0 (socket says 172.21.15.144:41482) 2024-09-18T12:30:05.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 -- 172.21.15.144:0/694118241 learned_addr learned my addr 172.21.15.144:0/694118241 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): get_auth_request con 0x7ffa1806a6f0 auth_method 0 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa180a37b0 0x7ffa18140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 10 monclient(hunting): get_auth_request con 0x7ffa180a37b0 auth_method 0 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa1ccba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient(hunting): handle_auth_done global_id 5359 payload 824 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 -- 172.21.15.144:0/694118241 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1806be40 msgr2=0x7ffa1813cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1806be40 0x7ffa1813cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 -- 172.21.15.144:0/694118241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa180a37b0 msgr2=0x7ffa18140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa180a37b0 0x7ffa18140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 monclient: found mon.noname-b 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:05.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.044+0000 7ffa17fff640 1 -- 172.21.15.144:0/694118241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00009390 con 0x7ffa1806a6f0 2024-09-18T12:30:05.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa17fff640 1 -- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 msgr2=0x7ffa1813d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa17fff640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 0x7ffa1813d900 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800c830 tx=0x7ffa08001720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0800ce20 con 0x7ffa1806a6f0 2024-09-18T12:30:05.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 10 monclient: dump: 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa17fff640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 0x7ffa1813d900 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800c830 tx=0x7ffa08001720 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7ffa080049f0 mon_map magic: 0 v1 2024-09-18T12:30:05.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 10 monclient: _finish_auth 0 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa1e71d640 5 monclient: authenticate success, global_id 5359 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa08007570 con 0x7ffa1806a6f0 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa0ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa1e71d640 10 monclient: _renew_subs 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa18122050 con 0x7ffa1806a6f0 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa167fc640 4 set_mon_vals no callback set 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa1e71d640 10 monclient: _renew_subs 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.046+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1813d690 con 0x7ffa1806a6f0 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa080049f0 con 0x7ffa1806a6f0 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa0ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa0ffff640 10 monclient: dump: 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ffa0800a640 con 0x7ffa1806a6f0 2024-09-18T12:30:05.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa1e71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa1e71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa18122730 con 0x7ffa1806a6f0 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.047+0000 7ffa0ffff640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff9f8038490 0x7ff9f803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.048+0000 7ffa177fe640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff9f8038490 0x7ff9f803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.048+0000 7ffa177fe640 10 monclient: get_auth_request con 0x7ff9f8038490 auth_method 0 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.050+0000 7ffa177fe640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff9f8038490 0x7ff9f803a940 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffa1809f2d0 tx=0x7ffa00032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.052+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6045+0+0 (secure 0 0 0) 0x7ffa0804bcb0 con 0x7ffa1806a6f0 2024-09-18T12:30:05.193 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T12:30:05.194 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T12:30:05.194 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:30:05.195 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.299+0000 7f07c9d5d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.299+0000 7f07c9d5d640 10 monclient: build_initial_monmap 2024-09-18T12:30:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.299+0000 7f07c9d5d640 10 monclient: monmap: 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:05.300173+0000 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:05.300173+0000 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:05.302 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 1 Processor -- start 2024-09-18T12:30:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 1 -- start start 2024-09-18T12:30:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient: init 2024-09-18T12:30:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T12:30:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07c4096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07c40971e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f07c4134b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c409d4a0 con 0x7f07c4096e10 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c4099620 con 0x7f07c40971e0 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c4099340 con 0x7f07c4134b40 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c9d5d640 10 monclient(hunting): authenticate will time out at 1032.621792s 2024-09-18T12:30:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f07c4096e10 0x7f07c409a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33336/0 (socket says 172.21.15.116:33336) 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c37fe640 1 -- 172.21.15.116:0/2453860288 learned_addr learned my addr 172.21.15.116:0/2453860288 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c3fff640 10 monclient(hunting): get_auth_request con 0x7f07c4134b40 auth_method 0 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.300+0000 7f07c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.301+0000 7f07c3fff640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 msgr2=0x7f07c4136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.301+0000 7f07c3fff640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 msgr2=0x7f07c4136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.301+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.301+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.303+0000 7f07c37fe640 1 -- 172.21.15.116:0/2453860288 >> v1:172.21.15.116:6789/0 conn(0x7f07c4096e10 legacy=0x7f07c409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.303+0000 7f07c2ffd640 1 -- 172.21.15.116:0/2453860288 >> v1:172.21.15.144:6789/0 conn(0x7f07c40971e0 legacy=0x7f07c40975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 10 monclient(hunting): get_auth_request con 0x7f07c4134b40 auth_method 0 2024-09-18T12:30:05.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:05.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 msgr2=0x7f07c4136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:05.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 msgr2=0x7f07c4136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:05.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:05.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.502+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440920486 0 0) 0x7f07c4099620 con 0x7f07c40971e0 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): my global_id is 5362 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac0034e0 con 0x7f07c40971e0 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104816598 0 0) 0x7f07ac0034e0 con 0x7f07c40971e0 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 msgr2=0x7f07c4136f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07c4136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 >> v1:172.21.15.116:6789/0 conn(0x7f07c4096e10 legacy=0x7f07c409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:30:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4099900 con 0x7f07c40971e0 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f07b80017f0 con 0x7f07c40971e0 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: dump: 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 >> v1:172.21.15.144:6789/0 conn(0x7f07c40971e0 legacy=0x7f07c40975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07ac003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): picked mon.c con 0x7f07c4134b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07ac0041f0 0x7f07ac004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): picked mon.a con 0x7f07ac0041f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ac004ec0 0x7f07ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): picked mon.b con 0x7f07ac004ec0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c4099340 con 0x7f07ac0041f0 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac00bf10 con 0x7f07ac004ec0 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac00c050 con 0x7f07c4134b40 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c27fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07ac0041f0 0x7f07ac004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 10 monclient(hunting): get_auth_request con 0x7f07ac0041f0 auth_method 0 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07ac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ac004ec0 0x7f07ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 10 monclient(hunting): get_auth_request con 0x7f07c4134b40 auth_method 0 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 10 monclient(hunting): get_auth_request con 0x7f07ac004ec0 auth_method 0 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.609+0000 7f07c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 10 monclient(hunting): handle_auth_done global_id 5362 payload 293 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 msgr2=0x7f07ac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07c4134b40 0x7f07ac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ac004ec0 msgr2=0x7f07ac00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07ac004ec0 0x7f07ac00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 1 monclient: found mon.a 2024-09-18T12:30:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 1 -- 172.21.15.116:0/2453860288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac0034e0 con 0x7f07ac0041f0 2024-09-18T12:30:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c9d5d640 5 monclient: authenticate success, global_id 5362 2024-09-18T12:30:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.610+0000 7f07c9d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.612+0000 7f07c2ffd640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07ac0041f0 msgr2=0x7f07ac004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.612+0000 7f07c2ffd640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07ac0041f0 0x7f07ac004780 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c410 tx=0x7f07b80079e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ef3971ab8c5a8b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b80070f0 con 0x7f07ac0041f0 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 10 monclient: dump: 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c9d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b8007290 con 0x7f07ac0041f0 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 1 -- 172.21.15.116:0/2453860288 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8007650 con 0x7f07ac0041f0 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c27fc640 10 monclient: dump: 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c9d5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c9d5d640 4 set_mon_vals no callback set 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c9d5d640 10 monclient: shutdown 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c9d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07ac0041f0 msgr2=0x7f07ac004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c9d5d640 1 --2- 172.21.15.116:0/2453860288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07ac0041f0 0x7f07ac004780 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c410 tx=0x7f07b80079e0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.613+0000 7f07c3fff640 1 -- 172.21.15.116:0/2453860288 reap_dead start 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.614+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2453860288 shutdown_connections 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.614+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2453860288 >> 172.21.15.116:0/2453860288 conn(0x7f07c40672b0 msgr2=0x7f07c40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.614+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2453860288 shutdown_connections 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.614+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2453860288 wait complete. 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.614+0000 7f07c9d5d640 10 monclient: build_initial_monmap 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.614+0000 7f07c9d5d640 10 monclient: monmap: 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 Processor -- start 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 -- start start 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient: init 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07ac0041f0 0x7f07c413d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f07ac0041f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07c41175a0 0x7f07c413d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f07c41175a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07c406d070 0x7f07c406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f07c406d070 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c406f680 con 0x7f07c406d070 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c409b050 con 0x7f07c41175a0 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c409fb00 con 0x7f07ac0041f0 2024-09-18T12:30:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c9d5d640 10 monclient(hunting): authenticate will time out at 1032.936744s 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07ac0041f0 0x7f07c413d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07ac0041f0 0x7f07c413d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35778/0 (socket says 172.21.15.116:35778) 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07c406d070 0x7f07c406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 1 -- 172.21.15.116:0/2662843414 learned_addr learned my addr 172.21.15.116:0/2662843414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 10 monclient(hunting): get_auth_request con 0x7f07ac0041f0 auth_method 0 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c2ffd640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07c41175a0 0x7f07c413d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c3fff640 10 monclient(hunting): get_auth_request con 0x7f07c406d070 auth_method 0 2024-09-18T12:30:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c2ffd640 10 monclient(hunting): get_auth_request con 0x7f07c41175a0 auth_method 0 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.615+0000 7f07c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 10 monclient(hunting): handle_auth_done global_id 74115 payload 824 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 1 -- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07ac0041f0 msgr2=0x7f07c413d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07ac0041f0 0x7f07c413d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 1 -- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07c41175a0 msgr2=0x7f07c413d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07c41175a0 0x7f07c413d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 1 monclient: found mon.noname-a 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c3fff640 1 -- 172.21.15.116:0/2662843414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4119620 con 0x7f07c406d070 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c37fe640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07ac0041f0 0x7f07c413d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.616+0000 7f07c2ffd640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07c41175a0 0x7f07c413d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c3fff640 1 -- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07c406d070 msgr2=0x7f07c406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c3fff640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07c406d070 0x7f07c406d510 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f07c4098070 tx=0x7f07bc0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 1 -- 172.21.15.116:0/2662843414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00f070 con 0x7f07c406d070 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: dump: 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.635 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 1 -- 172.21.15.116:0/2662843414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07bc0188a0 con 0x7f07c406d070 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c9d5d640 5 monclient: authenticate success, global_id 74115 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c9d5d640 10 monclient: _renew_subs 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c406ebf0 con 0x7f07c406d070 2024-09-18T12:30:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 1 -- 172.21.15.116:0/2662843414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc018c80 con 0x7f07c406d070 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07ab7fe640 10 monclient: dump: 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c9d5d640 10 monclient: _renew_subs 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c4118770 con 0x7f07c406d070 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.618+0000 7f07c1ffb640 4 set_mon_vals no callback set 2024-09-18T12:30:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07ab7fe640 1 -- 172.21.15.116:0/2662843414 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f07bc003040 con 0x7f07c406d070 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07ab7fe640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b0038490 0x7f07b003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07ab7fe640 1 -- 172.21.15.116:0/2662843414 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6045+0+0 (secure 0 0 0) 0x7f07bc04e100 con 0x7f07c406d070 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07c9d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07c9d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c404bf60 con 0x7f07c406d070 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07c37fe640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b0038490 0x7f07b003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.619+0000 7f07c37fe640 10 monclient: get_auth_request con 0x7f07b0038490 auth_method 0 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.622+0000 7f07c37fe640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b0038490 0x7f07b003a940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f07b4007ac0 tx=0x7f07b4007d70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.624+0000 7f07ab7fe640 1 -- 172.21.15.116:0/2662843414 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f07bc01c020 con 0x7f07c406d070 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.624+0000 7f07ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.624+0000 7f07ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.840+0000 7f07c9d5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.840+0000 7f07c9d5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.840+0000 7f07c9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.840+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f07c40638d0 con 0x7f07c406d070 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.841+0000 7f07ab7fe640 1 -- 172.21.15.116:0/2662843414 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f07bc0272a0 con 0x7f07c406d070 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.841+0000 7f07ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.841+0000 7f07ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:05.842 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b0038490 msgr2=0x7f07b003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b0038490 0x7f07b003a940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f07b4007ac0 tx=0x7f07b4007d70 comp rx=0 tx=0).stop 2024-09-18T12:30:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 10 monclient: shutdown 2024-09-18T12:30:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07c406d070 msgr2=0x7f07c406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07c406d070 0x7f07c406d510 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f07c4098070 tx=0x7f07bc0097d0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 shutdown_connections 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f07b0038490 0x7f07b003a940 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07c406d070 0x7f07c406d510 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07c41175a0 0x7f07c413d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 --2- 172.21.15.116:0/2662843414 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f07ac0041f0 0x7f07c413d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 >> 172.21.15.116:0/2662843414 conn(0x7f07c40672b0 msgr2=0x7f07c409d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 shutdown_connections 2024-09-18T12:30:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.843+0000 7f07c9d5d640 1 -- 172.21.15.116:0/2662843414 wait complete. 2024-09-18T12:30:05.857 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:30:05.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:30:05.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:30:05.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.874+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ffa08004e20 con 0x7ffa1806a6f0 2024-09-18T12:30:05.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.874+0000 7ffa0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:05.874+0000 7ffa0ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:05.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.950+0000 7fa29e19d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:05.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.950+0000 7fa29e19d640 10 monclient: build_initial_monmap 2024-09-18T12:30:05.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.950+0000 7fa29e19d640 10 monclient: monmap: 2024-09-18T12:30:05.951 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:05.951341+0000 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:05.951341+0000 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:05.952 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 1 Processor -- start 2024-09-18T12:30:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 1 -- start start 2024-09-18T12:30:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient: init 2024-09-18T12:30:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2980981c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa298131da0 0x7fa298136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient(hunting): picked mon.noname-c con 0x7fa298131da0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa298132170 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29809ca60 con 0x7fa298132170 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29809cca0 con 0x7fa2980981c0 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa298131bd0 con 0x7fa298131da0 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.951+0000 7fa29e19d640 10 monclient(hunting): authenticate will time out at 1033.272969s 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa297fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa298132170 0x7fa2981390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33404/0 (socket says 172.21.15.116:33404) 2024-09-18T12:30:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa297fff640 1 -- 172.21.15.116:0/3423225351 learned_addr learned my addr 172.21.15.116:0/3423225351 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 1 --2- 172.21.15.116:0/3423225351 >> v2:172.21.15.144:3300/0 conn(0x7fa298131da0 0x7fa298136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 10 monclient(hunting): get_auth_request con 0x7fa298131da0 auth_method 0 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.952+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa297fff640 1 -- 172.21.15.116:0/3423225351 >> v1:172.21.15.116:6789/0 conn(0x7fa298132170 legacy=0x7fa2981390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa2977fe640 1 -- 172.21.15.116:0/3423225351 >> v1:172.21.15.144:6789/0 conn(0x7fa2980981c0 legacy=0x7fa298098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa2967fc640 1 -- 172.21.15.116:0/3423225351 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727858983 0 0) 0x7fa29809ca60 con 0x7fa298132170 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa2967fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa2967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa2967fc640 10 monclient(hunting): my global_id is 74118 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa2967fc640 1 -- 172.21.15.116:0/3423225351 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2800034e0 con 0x7fa298132170 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 10 monclient(hunting): handle_auth_done global_id 5368 payload 274 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 1 -- 172.21.15.116:0/3423225351 >> v1:172.21.15.144:6789/0 conn(0x7fa2980981c0 legacy=0x7fa298098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 1 -- 172.21.15.116:0/3423225351 >> v1:172.21.15.116:6789/0 conn(0x7fa298132170 legacy=0x7fa2981390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 1 monclient: found mon.noname-c 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.954+0000 7fa296ffd640 1 -- 172.21.15.116:0/3423225351 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29809e010 con 0x7fa298131da0 2024-09-18T12:30:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa296ffd640 1 -- 172.21.15.116:0/3423225351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298131da0 msgr2=0x7fa298136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa296ffd640 1 --2- 172.21.15.116:0/3423225351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298131da0 0x7fa298136b70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009950 tx=0x7fa28c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89ddec4aea1307a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 1 -- 172.21.15.116:0/3423225351 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c02ae10 con 0x7fa298131da0 2024-09-18T12:30:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: dump: 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.964 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 1 -- 172.21.15.116:0/3423225351 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c02f4b0 con 0x7fa298131da0 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 1 -- 172.21.15.116:0/3423225351 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c02f850 con 0x7fa298131da0 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa2967fc640 10 monclient: dump: 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.965 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa29e19d640 5 monclient: authenticate success, global_id 5368 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa29e19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.957+0000 7fa29e19d640 4 set_mon_vals no callback set 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 10 monclient: shutdown 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 1 -- 172.21.15.116:0/3423225351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298131da0 msgr2=0x7fa298136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 1 --2- 172.21.15.116:0/3423225351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298131da0 0x7fa298136b70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009950 tx=0x7fa28c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 1 -- 172.21.15.116:0/3423225351 shutdown_connections 2024-09-18T12:30:05.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 1 --2- 172.21.15.116:0/3423225351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298131da0 0x7fa298136b70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 1 -- 172.21.15.116:0/3423225351 >> 172.21.15.116:0/3423225351 conn(0x7fa2980a1e30 msgr2=0x7fa2980a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 1 -- 172.21.15.116:0/3423225351 shutdown_connections 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 1 -- 172.21.15.116:0/3423225351 wait complete. 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 10 monclient: build_initial_monmap 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.958+0000 7fa29e19d640 10 monclient: monmap: 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:05.967 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 Processor -- start 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 -- start start 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient: init 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2980981c0 0x7fa2981313e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): picked mon.noname-c con 0x7fa2980981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298131da0 0x7fa298137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): picked mon.noname-a con 0x7fa298131da0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298132170 0x7fa298138510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): picked mon.noname-b con 0x7fa298132170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29809bbe0 con 0x7fa298131da0 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa298119340 con 0x7fa298132170 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa298119620 con 0x7fa2980981c0 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa29e19d640 10 monclient(hunting): authenticate will time out at 1033.280774s 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298131da0 0x7fa298137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2980981c0 0x7fa2981313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298131da0 0x7fa298137fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46282/0 (socket says 172.21.15.116:46282) 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 1 -- 172.21.15.116:0/925852577 learned_addr learned my addr 172.21.15.116:0/925852577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 10 monclient(hunting): get_auth_request con 0x7fa2980981c0 auth_method 0 2024-09-18T12:30:05.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 10 monclient(hunting): get_auth_request con 0x7fa298131da0 auth_method 0 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa297fff640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298132170 0x7fa298138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa2977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa297fff640 10 monclient(hunting): get_auth_request con 0x7fa298132170 auth_method 0 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa297fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.959+0000 7fa297fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa297fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa297fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa297fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 10 monclient(hunting): handle_auth_done global_id 64268 payload 824 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298132170 msgr2=0x7fa298138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa298132170 0x7fa298138510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298131da0 msgr2=0x7fa298137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298131da0 0x7fa298137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 1 monclient: found mon.noname-c 2024-09-18T12:30:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa2977fe640 1 -- 172.21.15.116:0/925852577 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c009390 con 0x7fa2980981c0 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.960+0000 7fa296ffd640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298131da0 0x7fa298137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa2977fe640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2980981c0 msgr2=0x7fa2981313e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa2977fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2980981c0 0x7fa2981313e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c5c0 tx=0x7fa288007cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2880074f0 con 0x7fa2980981c0 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 10 monclient: dump: 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa2977fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2980981c0 0x7fa2981313e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c5c0 tx=0x7fa288007cb0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fa288005780 config(0 keys) v1 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa29e19d640 5 monclient: authenticate success, global_id 64268 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa29e19d640 10 monclient: _renew_subs 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa29e19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa298139ba0 con 0x7fa2980981c0 2024-09-18T12:30:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa288005780 con 0x7fa2980981c0 2024-09-18T12:30:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa27f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa29e19d640 10 monclient: _renew_subs 2024-09-18T12:30:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa295ffb640 4 set_mon_vals no callback set 2024-09-18T12:30:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa29e19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.962+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa298130840 con 0x7fa2980981c0 2024-09-18T12:30:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa29e19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa29e19d640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 0x7fa29802aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa29e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa298065f90 con 0x7fa298118490 2024-09-18T12:30:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288005b40 con 0x7fa2980981c0 2024-09-18T12:30:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa27f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa27f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa27f7fe640 10 monclient: dump: 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa288012400 con 0x7fa2980981c0 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 0x7fa29802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient: get_auth_request con 0x7fa298118490 auth_method 0 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa27f7fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038490 0x7fa28403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 10 monclient(hunting): handle_auth_done global_id 74124 payload 824 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa296ffd640 1 -- 172.21.15.116:0/925852577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2981194b0 con 0x7fa298118490 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa297fff640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038490 0x7fa28403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.963+0000 7fa297fff640 10 monclient: get_auth_request con 0x7fa284038490 auth_method 0 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.965+0000 7fa296ffd640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 msgr2=0x7fa29802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.965+0000 7fa296ffd640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 0x7fa29802aa40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa28c035040 tx=0x7fa28c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c032d50 con 0x7fa298118490 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 10 monclient: dump: 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.981 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa28c03c440 con 0x7fa298118490 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 msgr2=0x7fa29802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 0x7fa29802aa40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa28c035040 tx=0x7fa28c032000 comp rx=0 tx=0).stop 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 msgr2=0x7fa29802aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa27f7fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298118490 0x7fa29802aa40 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa28c035040 tx=0x7fa28c032000 comp rx=0 tx=0).stop 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.966+0000 7fa297fff640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038490 0x7fa28403a940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa290005bc0 tx=0x7fa290009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa29e19d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa29e19d640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 0x7fa29805e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa29e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:05.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa298138e50 con 0x7fa298138a50 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 0x7fa29805e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient: get_auth_request con 0x7fa298138a50 auth_method 0 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 10 monclient(hunting): handle_auth_done global_id 74127 payload 824 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.972+0000 7fa296ffd640 1 -- 172.21.15.116:0/925852577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa29804bf60 con 0x7fa298138a50 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.974+0000 7fa296ffd640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 msgr2=0x7fa29805e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.974+0000 7fa296ffd640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 0x7fa29805e310 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa2980686b0 tx=0x7fa28c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.974+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c0469c0 con 0x7fa298138a50 2024-09-18T12:30:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.974+0000 7fa27f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.974+0000 7fa27f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.974+0000 7fa27f7fe640 10 monclient: dump: 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7fa28c046b20 con 0x7fa298138a50 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 msgr2=0x7fa29805e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 0x7fa29805e310 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa2980686b0 tx=0x7fa28c03d000 comp rx=0 tx=0).stop 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 msgr2=0x7fa29805e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa298138a50 0x7fa29805e310 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa2980686b0 tx=0x7fa28c03d000 comp rx=0 tx=0).stop 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:05.985 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 54, 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:05.986 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:05.988 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:05.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:05.990 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:05.991 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:05.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:05.994 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:05.995 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:05.996 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038490 msgr2=0x7fa28403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038490 0x7fa28403a940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa290005bc0 tx=0x7fa290009040 comp rx=0 tx=0).stop 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 10 monclient: shutdown 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2980981c0 msgr2=0x7fa2981313e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 --2- 172.21.15.116:0/925852577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2980981c0 0x7fa2981313e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa28800c5c0 tx=0x7fa288007cb0 comp rx=0 tx=0).stop 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa297fff640 1 -- 172.21.15.116:0/925852577 reap_dead start 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa27f7fe640 1 -- 172.21.15.116:0/925852577 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6132+0+0 (unknown 0 0 0) 0x7fa28804b1d0 con 0x7fa2980981c0 2024-09-18T12:30:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa297fff640 1 -- 172.21.15.116:0/925852577 reap_dead start 2024-09-18T12:30:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 shutdown_connections 2024-09-18T12:30:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 >> 172.21.15.116:0/925852577 conn(0x7fa2980a1e30 msgr2=0x7fa298132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 shutdown_connections 2024-09-18T12:30:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:05.975+0000 7fa29e19d640 1 -- 172.21.15.116:0/925852577 wait complete. 2024-09-18T12:30:05.999 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:05.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:30:05.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:30:06.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.033+0000 7ffa1e71d640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-18T12:30:06.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.033+0000 7ffa1e71d640 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 7}] 2024-09-18T12:30:06.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.033+0000 7ffa1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:06.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.033+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7ffa180636e0 con 0x7ffa1806a6f0 2024-09-18T12:30:06.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.033+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v104) v1 ==== 65+0+226 (secure 0 0 0) 0x7ffa0801b300 con 0x7ffa1806a6f0 2024-09-18T12:30:06.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.033+0000 7ffa0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 7}] 2024-09-18T12:30:06.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.033+0000 7ffa0ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.092+0000 7f6272834640 10 monclient: get_monmap_and_config 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.092+0000 7f6272834640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.092+0000 7f6272834640 10 monclient: monmap: 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:06.092840+0000 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:06.092840+0000 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.093 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:06.094 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.092+0000 7f6272834640 1 Processor -- start 2024-09-18T12:30:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.092+0000 7f6272834640 1 -- start start 2024-09-18T12:30:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.092+0000 7f6272834640 10 monclient: init 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f626c096e10 0x7f626c09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): picked mon.noname-a con 0x7f626c096e10 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c09aaa0 0x7f626c0971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): picked mon.noname-b con 0x7f626c09aaa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f626c097720 0x7f626c12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): picked mon.noname-c con 0x7f626c097720 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c099620 con 0x7f626c096e10 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c099340 con 0x7f626c097720 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c099900 con 0x7f626c09aaa0 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6272834640 10 monclient(hunting): authenticate will time out at 1033.414473s 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f626c096e10 0x7f626c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c09aaa0 0x7f626c0971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f626c096e10 0x7f626c09a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46312/0 (socket says 172.21.15.116:46312) 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 1 -- 172.21.15.116:0/3546421713 learned_addr learned my addr 172.21.15.116:0/3546421713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): get_auth_request con 0x7f626c09aaa0 auth_method 0 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 10 monclient(hunting): get_auth_request con 0x7f626c096e10 auth_method 0 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 1 --2- 172.21.15.116:0/3546421713 >> v2:172.21.15.144:3300/0 conn(0x7f626c097720 0x7f626c12e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 10 monclient(hunting): get_auth_request con 0x7f626c097720 auth_method 0 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f6270dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient(hunting): handle_auth_done global_id 64271 payload 274 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 1 -- 172.21.15.116:0/3546421713 >> v2:172.21.15.144:3300/0 conn(0x7f626c097720 msgr2=0x7f626c12e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3546421713 >> v2:172.21.15.144:3300/0 conn(0x7f626c097720 0x7f626c12e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 1 -- 172.21.15.116:0/3546421713 >> v2:172.21.15.116:3300/0 conn(0x7f626c096e10 msgr2=0x7f626c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3546421713 >> v2:172.21.15.116:3300/0 conn(0x7f626c096e10 0x7f626c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626b7fe640 1 -- 172.21.15.116:0/3546421713 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c097b10 con 0x7f626c09aaa0 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.093+0000 7f626bfff640 1 --2- 172.21.15.116:0/3546421713 >> v2:172.21.15.116:3300/0 conn(0x7f626c096e10 0x7f626c09a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626b7fe640 1 -- 172.21.15.116:0/3546421713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c09aaa0 msgr2=0x7f626c0971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3546421713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c09aaa0 0x7f626c0971e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f626c09aff0 tx=0x7f62600090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9997639cddbe49e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 1 -- 172.21.15.116:0/3546421713 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626002adf0 con 0x7f626c09aaa0 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: dump: 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.106 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 1 -- 172.21.15.116:0/3546421713 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6260032440 con 0x7f626c09aaa0 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f6272834640 5 monclient: authenticate success, global_id 64271 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f6272834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f6272834640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 1 -- 172.21.15.116:0/3546421713 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62600327e0 con 0x7f626c09aaa0 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f6272834640 4 set_mon_vals no callback set 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.096+0000 7f626affd640 10 monclient: dump: 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 10 monclient: shutdown 2024-09-18T12:30:06.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 -- 172.21.15.116:0/3546421713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c09aaa0 msgr2=0x7f626c0971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 --2- 172.21.15.116:0/3546421713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c09aaa0 0x7f626c0971e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f626c09aff0 tx=0x7f62600090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 -- 172.21.15.116:0/3546421713 shutdown_connections 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 --2- 172.21.15.116:0/3546421713 >> v2:172.21.15.144:3300/0 conn(0x7f626c097720 0x7f626c12e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 --2- 172.21.15.116:0/3546421713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c09aaa0 0x7f626c0971e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 --2- 172.21.15.116:0/3546421713 >> v2:172.21.15.116:3300/0 conn(0x7f626c096e10 0x7f626c09a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 -- 172.21.15.116:0/3546421713 >> 172.21.15.116:0/3546421713 conn(0x7f626c0672b0 msgr2=0x7f626c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 -- 172.21.15.116:0/3546421713 shutdown_connections 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 1 -- 172.21.15.116:0/3546421713 wait complete. 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.097+0000 7f6272834640 10 monclient: monmap: 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:06.109 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 Processor -- start 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 -- start start 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient: init 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c096e10 0x7f626c140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): picked mon.noname-c con 0x7f626c096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f626c097720 0x7f626c1298d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): picked mon.noname-b con 0x7f626c097720 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f626c09aaa0 0x7f626c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): picked mon.noname-a con 0x7f626c09aaa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c09f260 con 0x7f626c09aaa0 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c09b050 con 0x7f626c097720 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c132fe0 con 0x7f626c096e10 2024-09-18T12:30:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6272834640 10 monclient(hunting): authenticate will time out at 1033.419867s 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f626bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c096e10 0x7f626c140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f626bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c096e10 0x7f626c140d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35802/0 (socket says 172.21.15.116:35802) 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f626bfff640 1 -- 172.21.15.116:0/3772585093 learned_addr learned my addr 172.21.15.116:0/3772585093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6270dd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f626c09aaa0 0x7f626c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f626bfff640 10 monclient(hunting): get_auth_request con 0x7f626c096e10 auth_method 0 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f626bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f626bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f626bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6270dd1640 10 monclient(hunting): get_auth_request con 0x7f626c09aaa0 auth_method 0 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6270dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6270dd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.098+0000 7f6270dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f626c097720 0x7f626c1298d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f6270dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f6270dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f6270dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 10 monclient(hunting): get_auth_request con 0x7f626c097720 auth_method 0 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 10 monclient(hunting): handle_auth_done global_id 64274 payload 824 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f626c097720 msgr2=0x7f626c1298d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f626c097720 0x7f626c1298d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f626c09aaa0 msgr2=0x7f626c06db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f626c09aaa0 0x7f626c06db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 1 monclient: found mon.noname-c 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f626bfff640 1 -- 172.21.15.116:0/3772585093 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c009450 con 0x7f626c096e10 2024-09-18T12:30:06.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.099+0000 7f6270dd1640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f626c09aaa0 0x7f626c06db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f626bfff640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c096e10 msgr2=0x7f626c140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f626bfff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c096e10 0x7f626c140d60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f625c00ea30 tx=0x7f625c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c009690 con 0x7f626c096e10 2024-09-18T12:30:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: dump: 2024-09-18T12:30:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c00cc20 con 0x7f626c096e10 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6272834640 5 monclient: authenticate success, global_id 64274 2024-09-18T12:30:06.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c013470 con 0x7f626c096e10 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6272834640 10 monclient: _renew_subs 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6272834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6260009390 con 0x7f626c096e10 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6253fff640 10 monclient: dump: 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f626a7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6272834640 10 monclient: _renew_subs 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6272834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.101+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c134590 con 0x7f626c096e10 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f625c013820 con 0x7f626c096e10 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6272834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6272834640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 0x7f626c06e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c065f20 con 0x7f626c129e10 2024-09-18T12:30:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 0x7f626c06e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient: get_auth_request con 0x7f626c129e10 auth_method 0 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6253fff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62580384e0 0x7f625803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6132+0+0 (secure 0 0 0) 0x7f625c04c1a0 con 0x7f626c096e10 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6270dd1640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62580384e0 0x7f625803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 10 monclient(hunting): handle_auth_done global_id 64277 payload 824 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f626b7fe640 1 -- 172.21.15.116:0/3772585093 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f626c134a00 con 0x7f626c129e10 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.102+0000 7f6270dd1640 10 monclient: get_auth_request con 0x7f62580384e0 auth_method 0 2024-09-18T12:30:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f626b7fe640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 msgr2=0x7f626c06e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 0x7f626c06e0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6260034000 tx=0x7f6260032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626003d8f0 con 0x7f626c129e10 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 10 monclient: dump: 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6270dd1640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62580384e0 0x7f625803a990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f626400e950 tx=0x7f626400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f626003da50 con 0x7f626c129e10 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 msgr2=0x7f626c06e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 0x7f626c06e0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6260034000 tx=0x7f6260032000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 msgr2=0x7f626c06e0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.105+0000 7f6253fff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c129e10 0x7f626c06e0a0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6260034000 tx=0x7f6260032000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f6272834640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f6272834640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 0x7f626c12f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f6272834640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c06e9e0 con 0x7f626c06e5e0 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 0x7f626c12f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient: get_auth_request con 0x7f626c06e5e0 auth_method 0 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 10 monclient(hunting): handle_auth_done global_id 64280 payload 824 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.111+0000 7f626b7fe640 1 -- 172.21.15.116:0/3772585093 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f626c04bf60 con 0x7f626c06e5e0 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.113+0000 7f626b7fe640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 msgr2=0x7f626c12f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.113+0000 7f626b7fe640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 0x7f626c12f6f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f626c134800 tx=0x7f626003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.113+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626003de60 con 0x7f626c06e5e0 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.113+0000 7f6253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.113+0000 7f6253fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.113+0000 7f6253fff640 10 monclient: dump: 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f626003c7a0 con 0x7f626c06e5e0 2024-09-18T12:30:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6253fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6253fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 msgr2=0x7f626c12f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6253fff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 0x7f626c12f6f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f626c134800 tx=0x7f626003e000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6253fff640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 msgr2=0x7f626c12f6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6253fff640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c06e5e0 0x7f626c12f6f0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f626c134800 tx=0x7f626003e000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:06.128 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 54, 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:06.129 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:06.131 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:06.132 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:06.133 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:06.135 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:06.136 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:06.137 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:06.138 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:06.140 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62580384e0 msgr2=0x7f625803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62580384e0 0x7f625803a990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f626400e950 tx=0x7f626400b040 comp rx=0 tx=0).stop 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 10 monclient: shutdown 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c096e10 msgr2=0x7f626c140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 --2- 172.21.15.116:0/3772585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f626c096e10 0x7f626c140d60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f625c00ea30 tx=0x7f625c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6270dd1640 1 -- 172.21.15.116:0/3772585093 reap_dead start 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6270dd1640 1 -- 172.21.15.116:0/3772585093 reap_dead start 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 shutdown_connections 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 >> 172.21.15.116:0/3772585093 conn(0x7f626c0672b0 msgr2=0x7f626c09d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 shutdown_connections 2024-09-18T12:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.114+0000 7f6272834640 1 -- 172.21.15.116:0/3772585093 wait complete. 2024-09-18T12:30:06.143 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:06.143 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:30:06.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.128+0000 7ffa1e71d640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.128+0000 7ffa1e71d640 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 7}] 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.128+0000 7ffa1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.128+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd find", "id": 7} v 0) v1 -- 0x7ffa18072640 con 0x7ffa1806a6f0 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.129+0000 7ffa0ffff640 1 -- 172.21.15.144:0/694118241 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v104) v1 ==== 65+0+226 (secure 0 0 0) 0x7ffa0801b4e0 con 0x7ffa1806a6f0 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.129+0000 7ffa0ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 7}] 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.129+0000 7ffa0ffff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff9f8038490 msgr2=0x7ff9f803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff9f8038490 0x7ff9f803a940 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffa1809f2d0 tx=0x7ffa00032000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 10 monclient: shutdown 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 msgr2=0x7ffa1813d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 0x7ffa1813d900 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800c830 tx=0x7ffa08001720 comp rx=0 tx=0).stop 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 shutdown_connections 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff9f8038490 0x7ff9f803a940 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa180a37b0 0x7ffa18140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffa1806be40 0x7ffa1813cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 --2- 172.21.15.144:0/694118241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffa1806a6f0 0x7ffa1813d900 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 >> 172.21.15.144:0/694118241 conn(0x7ffa18070fd0 msgr2=0x7ffa180a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 shutdown_connections 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.131+0000 7ffa1e71d640 1 -- 172.21.15.144:0/694118241 wait complete. 2024-09-18T12:30:06.146 INFO:tasks.workunit.client.0.smithi144.stdout: "host": "foo", 2024-09-18T12:30:06.147 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm osd.7 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.213+0000 7f79ceb5d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.213+0000 7f79ceb5d640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.213+0000 7f79ceb5d640 10 monclient: monmap: 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:06.214775+0000 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:06.214775+0000 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:06.216 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:06.216 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:06.216 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:06.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.214+0000 7f79ceb5d640 1 Processor -- start 2024-09-18T12:30:06.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.214+0000 7f79ceb5d640 1 -- start start 2024-09-18T12:30:06.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.214+0000 7f79ceb5d640 10 monclient: init 2024-09-18T12:30:06.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79c8131c00 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79c81326e0 0x7f79c8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f79c81326e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79c80a0b00 0x7f79c8138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f79c80a0b00 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c80a1b70 con 0x7f79c8131c00 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80a08c0 con 0x7f79c81326e0 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80a0200 con 0x7f79c80a0b00 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79ceb5d640 10 monclient(hunting): authenticate will time out at 1041.784145s 2024-09-18T12:30:06.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79cc8f9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f79c8131c00 0x7f79c8131fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:46432/0 (socket says 172.21.15.144:46432) 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79cc8f9640 1 -- 172.21.15.144:0/895931519 learned_addr learned my addr 172.21.15.144:0/895931519 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79cd0fa640 1 --2- 172.21.15.144:0/895931519 >> v2:172.21.15.144:3300/0 conn(0x7f79c80a0b00 0x7f79c8138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79cd0fa640 10 monclient(hunting): get_auth_request con 0x7f79c80a0b00 auth_method 0 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79cd0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79cd0fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79cd0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 1 --2- 172.21.15.144:0/895931519 >> v2:172.21.15.116:3300/0 conn(0x7f79c81326e0 0x7f79c8069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): get_auth_request con 0x7f79c81326e0 auth_method 0 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient(hunting): handle_auth_done global_id 74136 payload 274 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 1 -- 172.21.15.144:0/895931519 >> v2:172.21.15.144:3300/0 conn(0x7f79c80a0b00 msgr2=0x7f79c8138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 1 --2- 172.21.15.144:0/895931519 >> v2:172.21.15.144:3300/0 conn(0x7f79c80a0b00 0x7f79c8138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 1 -- 172.21.15.144:0/895931519 >> v1:172.21.15.144:6789/0 conn(0x7f79c8131c00 legacy=0x7f79c8131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 1 monclient: found mon.noname-a 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.215+0000 7f79c7fff640 1 -- 172.21.15.144:0/895931519 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80a1890 con 0x7f79c81326e0 2024-09-18T12:30:06.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c7fff640 1 -- 172.21.15.144:0/895931519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c81326e0 msgr2=0x7f79c8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c7fff640 1 --2- 172.21.15.144:0/895931519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c81326e0 0x7f79c8069d10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f79c80a1310 tx=0x7f79b00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46b6289da7a8445 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c77fe640 1 -- 172.21.15.144:0/895931519 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b002adf0 con 0x7f79c81326e0 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c77fe640 10 monclient: dump: 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.228 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c77fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79ceb5d640 5 monclient: authenticate success, global_id 74136 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.218+0000 7f79c77fe640 1 -- 172.21.15.144:0/895931519 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b0032440 con 0x7f79c81326e0 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79c77fe640 1 -- 172.21.15.144:0/895931519 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b00327e0 con 0x7f79c81326e0 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 4 set_mon_vals no callback set 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79c77fe640 10 monclient: dump: 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:06.229 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 10 monclient: shutdown 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 -- 172.21.15.144:0/895931519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c81326e0 msgr2=0x7f79c8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/895931519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c81326e0 0x7f79c8069d10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f79c80a1310 tx=0x7f79b00090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 -- 172.21.15.144:0/895931519 shutdown_connections 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/895931519 >> v2:172.21.15.144:3300/0 conn(0x7f79c80a0b00 0x7f79c8138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/895931519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c81326e0 0x7f79c8069d10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 -- 172.21.15.144:0/895931519 >> 172.21.15.144:0/895931519 conn(0x7f79c8071370 msgr2=0x7f79c80a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 -- 172.21.15.144:0/895931519 shutdown_connections 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.219+0000 7f79ceb5d640 1 -- 172.21.15.144:0/895931519 wait complete. 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.220+0000 7f79ceb5d640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.220+0000 7f79ceb5d640 10 monclient: monmap: 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:06.231 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.220+0000 7f79ceb5d640 1 Processor -- start 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.220+0000 7f79ceb5d640 1 -- start start 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.220+0000 7f79ceb5d640 10 monclient: init 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c80a0b00 0x7f79c81313f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f79c80a0b00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79c8131c00 0x7f79c8096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f79c8131c00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:06.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c81326e0 0x7f79c8096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f79c81326e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c8097df0 con 0x7f79c80a0b00 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c809e840 con 0x7f79c8131c00 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c8099ac0 con 0x7f79c81326e0 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79ceb5d640 10 monclient(hunting): authenticate will time out at 1041.790129s 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79c8131c00 0x7f79c8096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79c8131c00 0x7f79c8096a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41498/0 (socket says 172.21.15.144:41498) 2024-09-18T12:30:06.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 1 -- 172.21.15.144:0/3366115964 learned_addr learned my addr 172.21.15.144:0/3366115964 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 10 monclient(hunting): get_auth_request con 0x7f79c8131c00 auth_method 0 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c80a0b00 0x7f79c81313f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): get_auth_request con 0x7f79c80a0b00 auth_method 0 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c81326e0 0x7f79c8096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 10 monclient(hunting): get_auth_request con 0x7f79c81326e0 auth_method 0 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cd0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient(hunting): handle_auth_done global_id 74139 payload 824 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 1 -- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c81326e0 msgr2=0x7f79c8096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c81326e0 0x7f79c8096f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 1 -- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79c8131c00 msgr2=0x7f79c8096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79c8131c00 0x7f79c8096a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 1 monclient: found mon.noname-a 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79cc8f9640 1 -- 172.21.15.144:0/3366115964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0009390 con 0x7f79c80a0b00 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.221+0000 7f79c7fff640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79c8131c00 0x7f79c8096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79cc8f9640 1 -- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c80a0b00 msgr2=0x7f79c81313f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79cc8f9640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c80a0b00 0x7f79c81313f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f79b800c4c0 tx=0x7f79b8007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8005cb0 con 0x7f79c80a0b00 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79c4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:06.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79c4ff9640 10 monclient: dump: 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79c4ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 5 monclient: authenticate success, global_id 74139 2024-09-18T12:30:06.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 10 monclient: _renew_subs 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.223+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b8005e30 con 0x7f79c80a0b00 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c80997e0 con 0x7f79c80a0b00 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8007320 con 0x7f79c80a0b00 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79c4ff9640 10 monclient: dump: 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.238 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79c6ffd640 4 set_mon_vals no callback set 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 10 monclient: _renew_subs 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c8098a20 con 0x7f79c80a0b00 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f79b801b3b0 con 0x7f79c80a0b00 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.224+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c80008d0 con 0x7f79c80a0b00 2024-09-18T12:30:06.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.225+0000 7f79c4ff9640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79a8038490 0x7f79a803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.225+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6132+0+0 (secure 0 0 0) 0x7f79b804b210 con 0x7f79c80a0b00 2024-09-18T12:30:06.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.225+0000 7f79c7fff640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79a8038490 0x7f79a803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.225+0000 7f79c7fff640 10 monclient: get_auth_request con 0x7f79a8038490 auth_method 0 2024-09-18T12:30:06.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.239+0000 7fa37e709640 10 monclient: get_monmap_and_config 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.239+0000 7fa37e709640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.239+0000 7fa37e709640 10 monclient: monmap: 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:06.240552+0000 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:06.240552+0000 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:06.241 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:06.242 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.227+0000 7f79c7fff640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79a8038490 0x7f79a803a940 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009130 tx=0x7f79b0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.229+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f79b8018680 con 0x7f79c80a0b00 2024-09-18T12:30:06.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.229+0000 7f79c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.229+0000 7f79c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:06.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.240+0000 7fa37e709640 1 Processor -- start 2024-09-18T12:30:06.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.240+0000 7fa37e709640 1 -- start start 2024-09-18T12:30:06.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.240+0000 7fa37e709640 10 monclient: init 2024-09-18T12:30:06.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3780a1ef0 0x7fa37809c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient(hunting): picked mon.noname-a con 0x7fa3780a1ef0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa37809abd0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa378136d00 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37806f010 con 0x7fa37809abd0 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37809dcc0 con 0x7fa378136d00 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37809da00 con 0x7fa3780a1ef0 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa37e709640 10 monclient(hunting): authenticate will time out at 1033.562178s 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3780a1ef0 0x7fa37809c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa3777fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa37809abd0 0x7fa37809afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33412/0 (socket says 172.21.15.116:33412) 2024-09-18T12:30:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3780a1ef0 0x7fa37809c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46320/0 (socket says 172.21.15.116:46320) 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa3777fe640 1 -- 172.21.15.116:0/3974127847 learned_addr learned my addr 172.21.15.116:0/3974127847 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): get_auth_request con 0x7fa3780a1ef0 auth_method 0 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient(hunting): handle_auth_done global_id 74142 payload 274 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 1 -- 172.21.15.116:0/3974127847 >> v1:172.21.15.144:6789/0 conn(0x7fa378136d00 legacy=0x7fa3781390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 1 -- 172.21.15.116:0/3974127847 >> v1:172.21.15.116:6789/0 conn(0x7fa37809abd0 legacy=0x7fa37809afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 1 monclient: found mon.noname-a 2024-09-18T12:30:06.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.241+0000 7fa377fff640 1 -- 172.21.15.116:0/3974127847 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3780a22c0 con 0x7fa3780a1ef0 2024-09-18T12:30:06.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa377fff640 1 -- 172.21.15.116:0/3974127847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 msgr2=0x7fa37809c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa377fff640 1 --2- 172.21.15.116:0/3974127847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa37809c6a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa368009850 tx=0x7fa3680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9041e0416908ab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 1 -- 172.21.15.116:0/3974127847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368032440 con 0x7fa3780a1ef0 2024-09-18T12:30:06.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: dump: 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 1 -- 172.21.15.116:0/3974127847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3680325a0 con 0x7fa3780a1ef0 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa37e709640 5 monclient: authenticate success, global_id 74142 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa37e709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa37e709640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 1 -- 172.21.15.116:0/3974127847 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368032940 con 0x7fa3780a1ef0 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa37e709640 4 set_mon_vals no callback set 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.244+0000 7fa376ffd640 10 monclient: dump: 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.256 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 10 monclient: shutdown 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 -- 172.21.15.116:0/3974127847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 msgr2=0x7fa37809c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 --2- 172.21.15.116:0/3974127847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa37809c6a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa368009850 tx=0x7fa3680090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 -- 172.21.15.116:0/3974127847 shutdown_connections 2024-09-18T12:30:06.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 --2- 172.21.15.116:0/3974127847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa37809c6a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 -- 172.21.15.116:0/3974127847 >> 172.21.15.116:0/3974127847 conn(0x7fa3780678a0 msgr2=0x7fa3780a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 -- 172.21.15.116:0/3974127847 shutdown_connections 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 -- 172.21.15.116:0/3974127847 wait complete. 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 10 monclient: monmap: 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:06.258 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.245+0000 7fa37e709640 1 Processor -- start 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 1 -- start start 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient: init 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa37809abd0 0x7fa3781355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): picked mon.noname-b con 0x7fa37809abd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa3781318c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): picked mon.noname-a con 0x7fa3780a1ef0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa378136d00 0x7fa378131e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): picked mon.noname-c con 0x7fa378136d00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378134910 con 0x7fa3780a1ef0 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378134bf0 con 0x7fa37809abd0 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa378134ed0 con 0x7fa378136d00 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37e709640 10 monclient(hunting): authenticate will time out at 1033.567644s 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa3781318c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa378136d00 0x7fa378131e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa3781318c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46336/0 (socket says 172.21.15.116:46336) 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 1 -- 172.21.15.116:0/21201567 learned_addr learned my addr 172.21.15.116:0/21201567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 10 monclient(hunting): get_auth_request con 0x7fa3780a1ef0 auth_method 0 2024-09-18T12:30:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 10 monclient(hunting): get_auth_request con 0x7fa378136d00 auth_method 0 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa37809abd0 0x7fa3781355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 10 monclient(hunting): get_auth_request con 0x7fa37809abd0 auth_method 0 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa3777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa37cca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.246+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 10 monclient(hunting): handle_auth_done global_id 64286 payload 824 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa37809abd0 msgr2=0x7fa3781355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa37809abd0 0x7fa3781355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 msgr2=0x7fa3781318c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa3781318c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 1 monclient: found mon.noname-c 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa37cca6640 1 -- 172.21.15.116:0/21201567 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368009390 con 0x7fa378136d00 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.247+0000 7fa3777fe640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780a1ef0 0x7fa3781318c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37cca6640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa378136d00 msgr2=0x7fa378131e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37cca6640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa378136d00 0x7fa378131e00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa37000c7e0 tx=0x7fa3700025b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37000a850 con 0x7fa378136d00 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: dump: 2024-09-18T12:30:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37000a9d0 con 0x7fa378136d00 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37e709640 5 monclient: authenticate success, global_id 64286 2024-09-18T12:30:06.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37e709640 10 monclient: _renew_subs 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37e709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa378133290 con 0x7fa378136d00 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37000ad90 con 0x7fa378136d00 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa35ffff640 10 monclient: dump: 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37e709640 10 monclient: _renew_subs 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37e709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa378135cb0 con 0x7fa378136d00 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.249+0000 7fa3767fc640 4 set_mon_vals no callback set 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa37e709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa37e709640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 0x7fa378095430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa370012630 con 0x7fa378136d00 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa37e709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378065e00 con 0x7fa378132340 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 0x7fa378095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa35ffff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa364038490 0x7fa36403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient: get_auth_request con 0x7fa378132340 auth_method 0 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6132+0+0 (secure 0 0 0) 0x7fa37004bfd0 con 0x7fa378136d00 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa3777fe640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa364038490 0x7fa36403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa3777fe640 10 monclient: get_auth_request con 0x7fa364038490 auth_method 0 2024-09-18T12:30:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 10 monclient(hunting): handle_auth_done global_id 5389 payload 824 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.250+0000 7fa377fff640 1 -- 172.21.15.116:0/21201567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa378134a80 con 0x7fa378132340 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa377fff640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 msgr2=0x7fa378095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa377fff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 0x7fa378095430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa368034000 tx=0x7fa368032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36803bab0 con 0x7fa378132340 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa3777fe640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa364038490 0x7fa36403a940 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa3780a1370 tx=0x7fa36c0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 10 monclient: dump: 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa368040070 con 0x7fa378132340 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 msgr2=0x7fa378095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 0x7fa378095430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa368034000 tx=0x7fa368032000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 msgr2=0x7fa378095430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.253+0000 7fa35ffff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378132340 0x7fa378095430 secure :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa368034000 tx=0x7fa368032000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa37e709640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:06.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa37e709640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 0x7fa378095d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa37e709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378133480 con 0x7fa378095970 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 0x7fa378095d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient: get_auth_request con 0x7fa378095970 auth_method 0 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 10 monclient(hunting): handle_auth_done global_id 5392 payload 824 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.259+0000 7fa377fff640 1 -- 172.21.15.116:0/21201567 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa3781339b0 con 0x7fa378095970 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.261+0000 7fa377fff640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 msgr2=0x7fa378095d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.261+0000 7fa377fff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 0x7fa378095d70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa368009130 tx=0x7fa36803e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368046820 con 0x7fa378095970 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 10 monclient: dump: 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.273 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fa3680453b0 con 0x7fa378095970 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 msgr2=0x7fa378095d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 0x7fa378095d70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa368009130 tx=0x7fa36803e070 comp rx=0 tx=0).stop 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 msgr2=0x7fa378095d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa35ffff640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa378095970 0x7fa378095d70 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:06.274 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 54, 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:06.275 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:06.276 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:06.277 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:06.278 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.279 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:06.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:06.282 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.284 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:06.285 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:06.286 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:06.287 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:06.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:06.287 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:06.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:06.287 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:06.287 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa364038490 msgr2=0x7fa36403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa364038490 0x7fa36403a940 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa3780a1370 tx=0x7fa36c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 10 monclient: shutdown 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa378136d00 msgr2=0x7fa378131e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 --2- 172.21.15.116:0/21201567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa378136d00 0x7fa378131e00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa37000c7e0 tx=0x7fa3700025b0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37cca6640 1 -- 172.21.15.116:0/21201567 reap_dead start 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37cca6640 1 -- 172.21.15.116:0/21201567 reap_dead start 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 shutdown_connections 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 >> 172.21.15.116:0/21201567 conn(0x7fa3780678a0 msgr2=0x7fa3780a02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 shutdown_connections 2024-09-18T12:30:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.262+0000 7fa37e709640 1 -- 172.21.15.116:0/21201567 wait complete. 2024-09-18T12:30:06.289 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:06.290 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:30:06.290 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:30:06.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.381+0000 7f3207d14640 10 monclient: get_monmap_and_config 2024-09-18T12:30:06.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.381+0000 7f3207d14640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.381+0000 7f3207d14640 10 monclient: monmap: 2024-09-18T12:30:06.382 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:06.382356+0000 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:06.382356+0000 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:06.383 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 1 Processor -- start 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 1 -- start start 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient: init 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32001318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f320009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient(hunting): picked mon.noname-b con 0x7f320009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3200136c20 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3200132640 con 0x7f3200136c20 2024-09-18T12:30:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3200131d00 con 0x7f32001318f0 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320009fc60 con 0x7f320009ea50 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.382+0000 7f3207d14640 10 monclient(hunting): authenticate will time out at 1033.703944s 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f320009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f320009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35850/0 (socket says 172.21.15.116:35850) 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 1 -- 172.21.15.116:0/3828427371 learned_addr learned my addr 172.21.15.116:0/3828427371 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32062b1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3200136c20 0x7f3200139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33428/0 (socket says 172.21.15.116:33428) 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): get_auth_request con 0x7f320009ea50 auth_method 0 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient(hunting): handle_auth_done global_id 64289 payload 274 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 1 -- 172.21.15.116:0/3828427371 >> v1:172.21.15.144:6789/0 conn(0x7f32001318f0 legacy=0x7f3200131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 1 -- 172.21.15.116:0/3828427371 >> v1:172.21.15.116:6789/0 conn(0x7f3200136c20 legacy=0x7f3200139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 1 monclient: found mon.noname-b 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.383+0000 7f32052af640 1 -- 172.21.15.116:0/3828427371 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320009fe30 con 0x7f320009ea50 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f32052af640 1 -- 172.21.15.116:0/3828427371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 msgr2=0x7f320009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f32052af640 1 --2- 172.21.15.116:0/3828427371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f320009ee20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009940 tx=0x7f31f00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e75366148b7042c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 1 -- 172.21.15.116:0/3828427371 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f0032440 con 0x7f320009ea50 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: dump: 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 1 -- 172.21.15.116:0/3828427371 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f00325a0 con 0x7f320009ea50 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 1 -- 172.21.15.116:0/3828427371 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f0032940 con 0x7f320009ea50 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3204aae640 10 monclient: dump: 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 5 monclient: authenticate success, global_id 64289 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 4 set_mon_vals no callback set 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 10 monclient: shutdown 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 1 -- 172.21.15.116:0/3828427371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 msgr2=0x7f320009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 1 --2- 172.21.15.116:0/3828427371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f320009ee20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009940 tx=0x7f31f00091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 1 -- 172.21.15.116:0/3828427371 shutdown_connections 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 1 --2- 172.21.15.116:0/3828427371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f320009ee20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 1 -- 172.21.15.116:0/3828427371 >> 172.21.15.116:0/3828427371 conn(0x7f3200067110 msgr2=0x7f3200067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 1 -- 172.21.15.116:0/3828427371 shutdown_connections 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.386+0000 7f3207d14640 1 -- 172.21.15.116:0/3828427371 wait complete. 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.387+0000 7f3207d14640 10 monclient: build_initial_monmap 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.387+0000 7f3207d14640 10 monclient: monmap: 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.387+0000 7f3207d14640 1 Processor -- start 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.387+0000 7f3207d14640 1 -- start start 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.387+0000 7f3207d14640 10 monclient: init 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f3200096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): picked mon.noname-c con 0x7f320009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32001318f0 0x7f3200096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): picked mon.noname-a con 0x7f32001318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3200136c20 0x7f32000974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): picked mon.noname-b con 0x7f3200136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3200099b10 con 0x7f32001318f0 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320009db90 con 0x7f3200136c20 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320009a0d0 con 0x7f320009ea50 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3207d14640 10 monclient(hunting): authenticate will time out at 1033.709162s 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32001318f0 0x7f3200096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f3200096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32001318f0 0x7f3200096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46346/0 (socket says 172.21.15.116:46346) 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 1 -- 172.21.15.116:0/1880125701 learned_addr learned my addr 172.21.15.116:0/1880125701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 10 monclient(hunting): get_auth_request con 0x7f32001318f0 auth_method 0 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): get_auth_request con 0x7f320009ea50 auth_method 0 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3200136c20 0x7f32000974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 10 monclient(hunting): get_auth_request con 0x7f3200136c20 auth_method 0 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32062b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient(hunting): handle_auth_done global_id 64292 payload 824 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 1 -- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3200136c20 msgr2=0x7f32000974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3200136c20 0x7f32000974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 1 -- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32001318f0 msgr2=0x7f3200096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32001318f0 0x7f3200096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient: _finish_hunting 0 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 1 monclient: found mon.noname-c 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f3205ab0640 1 -- 172.21.15.116:0/1880125701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0009450 con 0x7f320009ea50 2024-09-18T12:30:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.388+0000 7f32052af640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32001318f0 0x7f3200096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f3205ab0640 1 -- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 msgr2=0x7f3200096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f3205ab0640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f3200096a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00c4e0 tx=0x7f31fc007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 1 -- 172.21.15.116:0/1880125701 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0117a0 con 0x7f320009ea50 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: dump: 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 1 -- 172.21.15.116:0/1880125701 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc011920 con 0x7f320009ea50 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f3207d14640 5 monclient: authenticate success, global_id 64292 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f3207d14640 10 monclient: _renew_subs 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f3207d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3200098b10 con 0x7f320009ea50 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 1 -- 172.21.15.116:0/1880125701 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc011ce0 con 0x7f320009ea50 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f5ffb640 10 monclient: dump: 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.390+0000 7f31f7fff640 4 set_mon_vals no callback set 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.391+0000 7f3207d14640 10 monclient: _renew_subs 2024-09-18T12:30:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.391+0000 7f3207d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.391+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3200097d00 con 0x7f320009ea50 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.391+0000 7f31f5ffb640 1 -- 172.21.15.116:0/1880125701 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f31fc012630 con 0x7f320009ea50 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.391+0000 7f31f5ffb640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31e4038490 0x7f31e403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.391+0000 7f31f5ffb640 1 -- 172.21.15.116:0/1880125701 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6132+0+0 (secure 0 0 0) 0x7f31fc04b9e0 con 0x7f320009ea50 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.392+0000 7f3207d14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.392+0000 7f3207d14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.392+0000 7f3207d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.392+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320004bf60 con 0x7f320009ea50 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.392+0000 7f32052af640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31e4038490 0x7f31e403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.392+0000 7f32052af640 10 monclient: get_auth_request con 0x7f31e4038490 auth_method 0 2024-09-18T12:30:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.394+0000 7f32052af640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31e4038490 0x7f31e403a940 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3200136830 tx=0x7f31f0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.396+0000 7f31f5ffb640 1 -- 172.21.15.116:0/1880125701 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f31fc01c460 con 0x7f320009ea50 2024-09-18T12:30:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.396+0000 7f31f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.396+0000 7f31f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:06.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.391+0000 7f79ceb5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-18T12:30:06.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.391+0000 7f79ceb5d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-18T12:30:06.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.391+0000 7f79ceb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.391+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f79c80a0440 con 0x7f79c80a0b00 2024-09-18T12:30:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.610+0000 7f3207d14640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:30:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.610+0000 7f3207d14640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:30:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.610+0000 7f3207d14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.610+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f32000988f0 con 0x7f320009ea50 2024-09-18T12:30:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f31f5ffb640 1 -- 172.21.15.116:0/1880125701 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f31fc01beb0 con 0x7f320009ea50 2024-09-18T12:30:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f31f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:30:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f31f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:06.612 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.612 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1262536077,config=3140514009,health=3887269300,kv=2187923472,logm=2293443845}) 2024-09-18T12:30:06.613 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=1443923465,config=3140514009,health=3226445885,kv=2187923472,logm=3830918331}) 2024-09-18T12:30:06.613 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.613 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1262536077,config=3140514009,health=3887269300,kv=2187923472,logm=2293443845}) 2024-09-18T12:30:06.613 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.611+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1262536077,config=3140514009,health=3887269300,kv=1496160121,logm=965639518}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.612+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.612+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521559929}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.612+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260392209}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.612+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.612+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521559929}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.612+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2847318306}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708824141}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1907642216}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708824141}) 2024-09-18T12:30:06.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4140839846}) 2024-09-18T12:30:06.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31e4038490 msgr2=0x7f31e403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31e4038490 0x7f31e403a940 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3200136830 tx=0x7f31f0032000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 10 monclient: shutdown 2024-09-18T12:30:06.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 msgr2=0x7f3200096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f3200096a60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00c4e0 tx=0x7f31fc007e10 comp rx=0 tx=0).stop 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 shutdown_connections 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f31e4038490 0x7f31e403a940 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3200136c20 0x7f32000974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32001318f0 0x7f3200096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 --2- 172.21.15.116:0/1880125701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f320009ea50 0x7f3200096a60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 >> 172.21.15.116:0/1880125701 conn(0x7f3200067110 msgr2=0x7f32000a4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 shutdown_connections 2024-09-18T12:30:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f3207d14640 1 -- 172.21.15.116:0/1880125701 wait complete. 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202065246}) 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.613+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442094497}) 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202065246}) 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3848486618}) 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236656654}) 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=709281082}) 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.621 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236656654}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.614+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3069582457}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.615+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.615+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110472346}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.615+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1051594735}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.615+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.615+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110472346}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.615+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206507049}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.616+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.616+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966107744}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.616+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417552084}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.616+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.616+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966107744}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.616+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218941773}) 2024-09-18T12:30:06.622 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.617+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.617+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660370037}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.617+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074280680}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.617+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.617+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660370037}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.617+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1989945373}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.618+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.618+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853952469}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.618+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899692473}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.618+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.618+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853952469}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.618+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276753776}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168071306}) 2024-09-18T12:30:06.623 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113235125}) 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168071306}) 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1797235129}) 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.619+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945049444}) 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279393957}) 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945049444}) 2024-09-18T12:30:06.624 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925497129}) 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094334965}) 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315435770}) 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094334965}) 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.620+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507211233}) 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.621+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.621+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327159308}) 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.621+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1647983399}) 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.621+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.621+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327159308}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.621+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518386658}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.622+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.622+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568510199}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.622+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=681376989}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.622+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.622+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568510199}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.622+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1077338726}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.623+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.623+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3451491284}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.623+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52157482}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.623+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.623+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3451491284}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.623+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2741674048}) 2024-09-18T12:30:06.629 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.624+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.624+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=3372968584,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=10683881,mon_config_key=3655569570,monmap=148614106,osd_metadata=2372734516,osd_pg_creating=950391309,osd_snap=2033388227,osdmap=2174358851}) 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.624+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=37} crc {logm=3458329684,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1239352984,mon_config_key=892087436,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=950391309,osd_snap=2033388227,osdmap=3084870307}) 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.624+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.624+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=3372968584,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=10683881,mon_config_key=3655569570,monmap=148614106,osd_metadata=2372734516,osd_pg_creating=950391309,osd_snap=2033388227,osdmap=2174358851}) 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.624+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=39} crc {logm=3203504344,mdsmap=1531151339,mgr=3514226729,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3973247186,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=950391309,osd_snap=4249038086,osdmap=1217215992}) 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.625+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.625+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3646312077}) 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.625+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3296323838}) 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.625+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.625+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3646312077}) 2024-09-18T12:30:06.630 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.625+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2023063427}) 2024-09-18T12:30:06.631 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:30:06.632 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.627+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.632 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.627+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=6358174}) 2024-09-18T12:30:06.632 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.627+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=1498059144}) 2024-09-18T12:30:06.632 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.627+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:06.632 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.627+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=6358174}) 2024-09-18T12:30:06.632 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:06.627+0000 7f6d0edf1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=31} crc {osdmap=907296061}) 2024-09-18T12:30:06.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.868+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 removed item id 7 name 'osd.7' from crush map v105) v1 ==== 122+0+0 (secure 0 0 0) 0x7f79b8017070 con 0x7f79c80a0b00 2024-09-18T12:30:06.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.868+0000 7f79c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-18T12:30:06.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.868+0000 7f79c4ff9640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-18T12:30:06.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.963+0000 7f79ceb5d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-18T12:30:06.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.963+0000 7f79ceb5d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-18T12:30:06.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.963+0000 7f79ceb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:06.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.963+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) v1 -- 0x7f79c8072640 con 0x7f79c80a0b00 2024-09-18T12:30:06.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.964+0000 7f79c4ff9640 1 -- 172.21.15.144:0/3366115964 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 device 'osd.7' does not appear in the crush map v105) v1 ==== 124+0+0 (secure 0 0 0) 0x7f79b800e290 con 0x7f79c80a0b00 2024-09-18T12:30:06.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.964+0000 7f79c4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-18T12:30:06.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.964+0000 7f79c4ff9640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-18T12:30:06.965 INFO:tasks.workunit.client.0.smithi144.stderr:device 'osd.7' does not appear in the crush map 2024-09-18T12:30:06.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.965+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79a8038490 msgr2=0x7f79a803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.965+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79a8038490 0x7f79a803a940 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009130 tx=0x7f79b0032000 comp rx=0 tx=0).stop 2024-09-18T12:30:06.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.965+0000 7f79ceb5d640 10 monclient: shutdown 2024-09-18T12:30:06.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.965+0000 7f79ceb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:06.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.965+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c80a0b00 msgr2=0x7f79c81313f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:06.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.965+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c80a0b00 0x7f79c81313f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f79b800c4c0 tx=0x7f79b8007f00 comp rx=0 tx=0).stop 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 shutdown_connections 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79a8038490 0x7f79a803a940 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79c81326e0 0x7f79c8096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79c8131c00 0x7f79c8096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 --2- 172.21.15.144:0/3366115964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79c80a0b00 0x7f79c81313f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 >> 172.21.15.144:0/3366115964 conn(0x7f79c8071370 msgr2=0x7f79c809d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 shutdown_connections 2024-09-18T12:30:06.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:06.966+0000 7f79ceb5d640 1 -- 172.21.15.144:0/3366115964 wait complete. 2024-09-18T12:30:06.975 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm osd.9 2024-09-18T12:30:07.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.047+0000 7f5f9699d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:07.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.047+0000 7f5f9699d640 10 monclient: build_initial_monmap 2024-09-18T12:30:07.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.047+0000 7f5f9699d640 10 monclient: monmap: 2024-09-18T12:30:07.048 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:07.048219+0000 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:07.048219+0000 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:07.049 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:07.050 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.047+0000 7f5f9699d640 1 Processor -- start 2024-09-18T12:30:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 1 -- start start 2024-09-18T12:30:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient: init 2024-09-18T12:30:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T12:30:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f9006e950 0x7f5f9009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9006e950 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f9009be20 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f9009c220 0x7f5f90099e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9009c220 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f90097b40 con 0x7f5f9009be20 2024-09-18T12:30:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9009c6c0 con 0x7f5f9009c220 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f90098470 con 0x7f5f9006e950 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f9699d640 10 monclient(hunting): authenticate will time out at 1042.617565s 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f9006e950 0x7f5f9009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f9006e950 0x7f5f9009b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41506/0 (socket says 172.21.15.144:41506) 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 -- 172.21.15.144:0/4199087838 learned_addr learned my addr 172.21.15.144:0/4199087838 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): get_auth_request con 0x7f5f9006e950 auth_method 0 2024-09-18T12:30:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f94f3a640 1 --2- 172.21.15.144:0/4199087838 >> v2:172.21.15.116:3300/0 conn(0x7f5f9009c220 0x7f5f90099e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f94f3a640 10 monclient(hunting): get_auth_request con 0x7f5f9009c220 auth_method 0 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f94f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f94f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f94f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient(hunting): handle_auth_done global_id 5398 payload 274 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 -- 172.21.15.144:0/4199087838 >> v2:172.21.15.116:3300/0 conn(0x7f5f9009c220 msgr2=0x7f5f90099e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 --2- 172.21.15.144:0/4199087838 >> v2:172.21.15.116:3300/0 conn(0x7f5f9009c220 0x7f5f90099e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 -- 172.21.15.144:0/4199087838 >> v1:172.21.15.116:6789/0 conn(0x7f5f9009be20 legacy=0x7f5f900996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 monclient: found mon.noname-c 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.048+0000 7f5f8ffff640 1 -- 172.21.15.144:0/4199087838 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90098750 con 0x7f5f9006e950 2024-09-18T12:30:07.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8ffff640 1 -- 172.21.15.144:0/4199087838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9006e950 msgr2=0x7f5f9009b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8ffff640 1 --2- 172.21.15.144:0/4199087838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9006e950 0x7f5f9009b8e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5f80009910 tx=0x7f5f800091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c94b8734d60851a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 1 -- 172.21.15.144:0/4199087838 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8002aea0 con 0x7f5f9006e950 2024-09-18T12:30:07.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:07.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: dump: 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: _finish_auth 0 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:07.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f9699d640 5 monclient: authenticate success, global_id 5398 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 1 -- 172.21.15.144:0/4199087838 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f80032440 con 0x7f5f9006e950 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f9699d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f9699d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f9699d640 4 set_mon_vals no callback set 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 1 -- 172.21.15.144:0/4199087838 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f800327e0 con 0x7f5f9006e950 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.051+0000 7f5f8effd640 10 monclient: dump: 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.067 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 10 monclient: shutdown 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 -- 172.21.15.144:0/4199087838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9006e950 msgr2=0x7f5f9009b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 --2- 172.21.15.144:0/4199087838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9006e950 0x7f5f9009b8e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5f80009910 tx=0x7f5f800091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 -- 172.21.15.144:0/4199087838 shutdown_connections 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 --2- 172.21.15.144:0/4199087838 >> v2:172.21.15.116:3300/0 conn(0x7f5f9009c220 0x7f5f90099e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 --2- 172.21.15.144:0/4199087838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9006e950 0x7f5f9009b8e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 -- 172.21.15.144:0/4199087838 >> 172.21.15.144:0/4199087838 conn(0x7f5f90071100 msgr2=0x7f5f900a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 -- 172.21.15.144:0/4199087838 shutdown_connections 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 1 -- 172.21.15.144:0/4199087838 wait complete. 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 10 monclient: build_initial_monmap 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.052+0000 7f5f9699d640 10 monclient: monmap: 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:07.069 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 Processor -- start 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 -- start start 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient: init 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f9006e950 0x7f5f9012f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9006e950 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 0x7f5f90119320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9009be20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9009c220 0x7f5f90119860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9009c220 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:07.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9011b9a0 con 0x7f5f9009c220 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9011c900 con 0x7f5f9009be20 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.053+0000 7f5f9699d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9011cec0 con 0x7f5f9006e950 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f9699d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f9699d640 10 monclient(hunting): authenticate will time out at 1042.623041s 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 0x7f5f90119320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 0x7f5f90119320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:41514/0 (socket says 172.21.15.144:41514) 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 -- 172.21.15.144:0/3725485132 learned_addr learned my addr 172.21.15.144:0/3725485132 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:07.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): get_auth_request con 0x7f5f9009be20 auth_method 0 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f9006e950 0x7f5f9012f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 10 monclient(hunting): get_auth_request con 0x7f5f9006e950 auth_method 0 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9009c220 0x7f5f90119860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 10 monclient(hunting): get_auth_request con 0x7f5f9009c220 auth_method 0 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f94f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient(hunting): handle_auth_done global_id 5401 payload 824 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 -- 172.21.15.144:0/3725485132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f9006e950 msgr2=0x7f5f9012f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f9006e950 0x7f5f9012f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 -- 172.21.15.144:0/3725485132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9009c220 msgr2=0x7f5f90119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9009c220 0x7f5f90119860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.054+0000 7f5f8f7fe640 1 -- 172.21.15.144:0/3725485132 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f80009450 con 0x7f5f9009be20 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f8f7fe640 1 -- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 msgr2=0x7f5f90119320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f8f7fe640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 0x7f5f90119320 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c5e0 tx=0x7f5f84007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84012840 con 0x7f5f9009be20 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: dump: 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.074 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f840129a0 con 0x7f5f9009be20 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f9699d640 5 monclient: authenticate success, global_id 5401 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f9699d640 10 monclient: _renew_subs 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f9699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f90132150 con 0x7f5f9009be20 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:07.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84012da0 con 0x7f5f9009be20 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f77fff640 10 monclient: dump: 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.076 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f8e7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f9699d640 10 monclient: _renew_subs 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f9699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.056+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f90130580 con 0x7f5f9009be20 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5f84011590 con 0x7f5f9009be20 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f9699d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f9699d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f9699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9011ca40 con 0x7f5f9009be20 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f77fff640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f7c038490 0x7f5f7c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6024+0+0 (secure 0 0 0) 0x7f5f8404bdf0 con 0x7f5f9009be20 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f8ffff640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f7c038490 0x7f5f7c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.057+0000 7f5f8ffff640 10 monclient: get_auth_request con 0x7f5f7c038490 auth_method 0 2024-09-18T12:30:07.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.060+0000 7f5f8ffff640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f7c038490 0x7f5f7c03a940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5f90068e40 tx=0x7f5f80032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.062+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5f9011ca40 con 0x7f5f9009be20 2024-09-18T12:30:07.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.062+0000 7f5f77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.062+0000 7f5f77fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.221+0000 7f5f9699d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-18T12:30:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.221+0000 7f5f9699d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-18T12:30:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.221+0000 7f5f9699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.221+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7f5f900008d0 con 0x7f5f9009be20 2024-09-18T12:30:07.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.222+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v105) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5f84016030 con 0x7f5f9009be20 2024-09-18T12:30:07.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.222+0000 7f5f77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-18T12:30:07.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.222+0000 7f5f77fff640 10 monclient: _finish_command 2 = system:0 device 'osd.9' does not appear in the crush map 2024-09-18T12:30:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.318+0000 7f5f9699d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-18T12:30:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.318+0000 7f5f9699d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-18T12:30:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.318+0000 7f5f9699d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.318+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) v1 -- 0x7f5f9009d650 con 0x7f5f9009be20 2024-09-18T12:30:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.319+0000 7f5f77fff640 1 -- 172.21.15.144:0/3725485132 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v105) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5f84016020 con 0x7f5f9009be20 2024-09-18T12:30:07.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.319+0000 7f5f77fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-18T12:30:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.319+0000 7f5f77fff640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-18T12:30:07.321 INFO:tasks.workunit.client.0.smithi144.stderr:device 'osd.9' does not appear in the crush map 2024-09-18T12:30:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f7c038490 msgr2=0x7f5f7c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f7c038490 0x7f5f7c03a940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5f90068e40 tx=0x7f5f80032000 comp rx=0 tx=0).stop 2024-09-18T12:30:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 10 monclient: shutdown 2024-09-18T12:30:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 msgr2=0x7f5f90119320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 0x7f5f90119320 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c5e0 tx=0x7f5f84007ed0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 shutdown_connections 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f7c038490 0x7f5f7c03a940 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9009c220 0x7f5f90119860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f9009be20 0x7f5f90119320 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 --2- 172.21.15.144:0/3725485132 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f9006e950 0x7f5f9012f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 >> 172.21.15.144:0/3725485132 conn(0x7f5f90071100 msgr2=0x7f5f900995b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.321+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 shutdown_connections 2024-09-18T12:30:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.322+0000 7f5f9699d640 1 -- 172.21.15.144:0/3725485132 wait complete. 2024-09-18T12:30:07.331 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm foo 2024-09-18T12:30:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.403+0000 7f4f5911d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.403+0000 7f4f5911d640 10 monclient: build_initial_monmap 2024-09-18T12:30:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.403+0000 7f4f5911d640 10 monclient: monmap: 2024-09-18T12:30:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:07.404061+0000 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:07.404061+0000 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.403+0000 7f4f5911d640 1 Processor -- start 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.403+0000 7f4f5911d640 1 -- start start 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.403+0000 7f4f5911d640 10 monclient: init 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f5409ec50 0x7f4f54131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f5409ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f541321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f5409f020 0x7f4f54069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f5409f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f54067420 con 0x7f4f541321a0 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f540a0480 con 0x7f4f5409f020 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f540a0200 con 0x7f4f5409ec50 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5911d640 10 monclient(hunting): authenticate will time out at 1042.973264s 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f5409ec50 0x7f4f54131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f5409ec50 0x7f4f54131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38430/0 (socket says 172.21.15.144:38430) 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 -- 172.21.15.144:0/2362579489 learned_addr learned my addr 172.21.15.144:0/2362579489 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): get_auth_request con 0x7f4f5409ec50 auth_method 0 2024-09-18T12:30:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5359e640 1 --2- 172.21.15.144:0/2362579489 >> v2:172.21.15.116:3300/0 conn(0x7f4f5409f020 0x7f4f54069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5359e640 10 monclient(hunting): get_auth_request con 0x7f4f5409f020 auth_method 0 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f5359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient(hunting): handle_auth_done global_id 5404 payload 274 2024-09-18T12:30:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 -- 172.21.15.144:0/2362579489 >> v2:172.21.15.116:3300/0 conn(0x7f4f5409f020 msgr2=0x7f4f54069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 --2- 172.21.15.144:0/2362579489 >> v2:172.21.15.116:3300/0 conn(0x7f4f5409f020 0x7f4f54069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 -- 172.21.15.144:0/2362579489 >> v1:172.21.15.116:6789/0 conn(0x7f4f541321a0 legacy=0x7f4f54068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 monclient: found mon.noname-c 2024-09-18T12:30:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.404+0000 7f4f52d9d640 1 -- 172.21.15.144:0/2362579489 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5409fd80 con 0x7f4f5409ec50 2024-09-18T12:30:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f52d9d640 1 -- 172.21.15.144:0/2362579489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f5409ec50 msgr2=0x7f4f54131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f52d9d640 1 --2- 172.21.15.144:0/2362579489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f5409ec50 0x7f4f54131c60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4f540719d0 tx=0x7f4f4c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e4d848acd3cc819 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 1 -- 172.21.15.144:0/2362579489 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c032440 con 0x7f4f5409ec50 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: dump: 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.417 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: _finish_auth 0 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f5911d640 5 monclient: authenticate success, global_id 5404 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f5911d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 1 -- 172.21.15.144:0/2362579489 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c0325a0 con 0x7f4f5409ec50 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f5911d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f5911d640 4 set_mon_vals no callback set 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 1 -- 172.21.15.144:0/2362579489 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c032980 con 0x7f4f5409ec50 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.407+0000 7f4f51d9b640 10 monclient: dump: 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 10 monclient: shutdown 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 -- 172.21.15.144:0/2362579489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f5409ec50 msgr2=0x7f4f54131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 --2- 172.21.15.144:0/2362579489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f5409ec50 0x7f4f54131c60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4f540719d0 tx=0x7f4f4c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 -- 172.21.15.144:0/2362579489 shutdown_connections 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 --2- 172.21.15.144:0/2362579489 >> v2:172.21.15.116:3300/0 conn(0x7f4f5409f020 0x7f4f54069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 --2- 172.21.15.144:0/2362579489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f5409ec50 0x7f4f54131c60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 -- 172.21.15.144:0/2362579489 >> 172.21.15.144:0/2362579489 conn(0x7f4f540a5cf0 msgr2=0x7f4f540a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 -- 172.21.15.144:0/2362579489 shutdown_connections 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 1 -- 172.21.15.144:0/2362579489 wait complete. 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 10 monclient: build_initial_monmap 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.408+0000 7f4f5911d640 10 monclient: monmap: 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 Processor -- start 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 -- start start 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient: init 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5409ec50 0x7f4f5409ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f5409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f5409f020 0x7f4f54130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f5409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 0x7f4f54131170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): picked mon.noname-b con 0x7f4f541321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5409c020 con 0x7f4f5409f020 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5409e0d0 con 0x7f4f541321a0 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5411c6c0 con 0x7f4f5409ec50 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5911d640 10 monclient(hunting): authenticate will time out at 1042.978523s 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 0x7f4f54131170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 0x7f4f54131170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38446/0 (socket says 172.21.15.144:38446) 2024-09-18T12:30:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 -- 172.21.15.144:0/3143035663 learned_addr learned my addr 172.21.15.144:0/3143035663 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): get_auth_request con 0x7f4f541321a0 auth_method 0 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f52d9d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5409ec50 0x7f4f5409ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5259c640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f5409f020 0x7f4f54130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f52d9d640 10 monclient(hunting): get_auth_request con 0x7f4f5409ec50 auth_method 0 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5259c640 10 monclient(hunting): get_auth_request con 0x7f4f5409f020 auth_method 0 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient(hunting): handle_auth_done global_id 5407 payload 824 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 -- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5409ec50 msgr2=0x7f4f5409ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5409ec50 0x7f4f5409ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 -- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f5409f020 msgr2=0x7f4f54130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f5409f020 0x7f4f54130c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient: _finish_hunting 0 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 monclient: found mon.noname-b 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.409+0000 7f4f5359e640 1 -- 172.21.15.144:0/3143035663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c009450 con 0x7f4f541321a0 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5359e640 1 -- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 msgr2=0x7f4f54131170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5359e640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 0x7f4f54131170 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400d890 tx=0x7f4f44008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400b8d0 con 0x7f4f541321a0 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: dump: 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: _finish_auth 0 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400ba50 con 0x7f4f541321a0 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5911d640 5 monclient: authenticate success, global_id 5407 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5911d640 10 monclient: _renew_subs 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5411dd50 con 0x7f4f541321a0 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:07.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400be10 con 0x7f4f541321a0 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f3affd640 10 monclient: dump: 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:07.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5911d640 10 monclient: _renew_subs 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5159a640 4 set_mon_vals no callback set 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.412+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5411cdf0 con 0x7f4f541321a0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f5911d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f5911d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4f44014640 con 0x7f4f541321a0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f5911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f540008d0 con 0x7f4f541321a0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f3affd640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f40038490 0x7f4f4003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6024+0+0 (secure 0 0 0) 0x7f4f4404c3f0 con 0x7f4f541321a0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f52d9d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f40038490 0x7f4f4003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.413+0000 7f4f52d9d640 10 monclient: get_auth_request con 0x7f4f40038490 auth_method 0 2024-09-18T12:30:07.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.415+0000 7f4f52d9d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f40038490 0x7f4f4003a940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c034000 tx=0x7f4f4c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.417+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4f440148f0 con 0x7f4f541321a0 2024-09-18T12:30:07.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.417+0000 7f4f3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.417+0000 7f4f3affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:07.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.574+0000 7f4f5911d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:30:07.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.574+0000 7f4f5911d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:30:07.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.574+0000 7f4f5911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.574+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f4f540a13e0 con 0x7f4f541321a0 2024-09-18T12:30:07.632 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:30:07.632 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.724+0000 7f5d16a80640 10 monclient: get_monmap_and_config 2024-09-18T12:30:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.724+0000 7f5d16a80640 10 monclient: build_initial_monmap 2024-09-18T12:30:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.724+0000 7f5d16a80640 10 monclient: monmap: 2024-09-18T12:30:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:07.725175+0000 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:07.725175+0000 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:07.726 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:07.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 1 Processor -- start 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 1 -- start start 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient: init 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d100974f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d1009a8a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5d10134080 0x7f5d10136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient(hunting): picked mon.noname-c con 0x7f5d10134080 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1012b210 con 0x7f5d100974f0 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100978c0 con 0x7f5d1009a8a0 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1009ca20 con 0x7f5d10134080 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d16a80640 10 monclient(hunting): authenticate will time out at 1035.046770s 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d1481c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5d100974f0 0x7f5d1012ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33440/0 (socket says 172.21.15.116:33440) 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.725+0000 7f5d1481c640 1 -- 172.21.15.116:0/3423506257 learned_addr learned my addr 172.21.15.116:0/3423506257 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 1 --2- 172.21.15.116:0/3423506257 >> v2:172.21.15.144:3300/0 conn(0x7f5d10134080 0x7f5d10136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 10 monclient(hunting): get_auth_request con 0x7f5d10134080 auth_method 0 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.726+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 1 -- 172.21.15.116:0/3423506257 >> v1:172.21.15.116:6789/0 conn(0x7f5d100974f0 legacy=0x7f5d1012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630843281 0 0) 0x7f5d1012b210 con 0x7f5d100974f0 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): my global_id is 74163 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d000034e0 con 0x7f5d100974f0 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 1 -- 172.21.15.116:0/3423506257 >> v1:172.21.15.144:6789/0 conn(0x7f5d1009a8a0 legacy=0x7f5d1009ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 468859716 0 0) 0x7f5d000034e0 con 0x7f5d100974f0 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 >> v2:172.21.15.144:3300/0 conn(0x7f5d10134080 msgr2=0x7f5d10136460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 --2- 172.21.15.116:0/3423506257 >> v2:172.21.15.144:3300/0 conn(0x7f5d10134080 0x7f5d10136460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 >> v1:172.21.15.144:6789/0 conn(0x7f5d1009a8a0 legacy=0x7f5d1009ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10098ad0 con 0x7f5d100974f0 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d16a80640 5 monclient: authenticate success, global_id 74163 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d16a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f5d040017a0 con 0x7f5d100974f0 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:30:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: dump: 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 >> v1:172.21.15.116:6789/0 conn(0x7f5d100974f0 legacy=0x7f5d1012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00003bc0 0x7f5d00003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): picked mon.c con 0x7f5d00003bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d00004780 0x7f5d00004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): picked mon.a con 0x7f5d00004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d000059a0 0x7f5d00005e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): picked mon.b con 0x7f5d000059a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1009ca20 con 0x7f5d00004780 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d00006350 con 0x7f5d000059a0 2024-09-18T12:30:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d00006510 con 0x7f5d00003bc0 2024-09-18T12:30:07.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:07.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:07.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d16a80640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:07.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00003bc0 0x7f5d00003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d00004780 0x7f5d00004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 10 monclient(hunting): get_auth_request con 0x7f5d00003bc0 auth_method 0 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): get_auth_request con 0x7f5d00004780 auth_method 0 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d000059a0 0x7f5d00005e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1481c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 10 monclient(hunting): get_auth_request con 0x7f5d000059a0 auth_method 0 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 10 monclient(hunting): handle_auth_done global_id 74163 payload 293 2024-09-18T12:30:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 1 -- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00003bc0 msgr2=0x7f5d00003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00003bc0 0x7f5d00003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 1 -- 172.21.15.116:0/3423506257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d000059a0 msgr2=0x7f5d00005e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.728+0000 7f5d0ffff640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d000059a0 0x7f5d00005e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d0ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d0ffff640 1 monclient: found mon.a 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d1501d640 1 -- 172.21.15.116:0/3423506257 reap_dead start 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d0ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d0ffff640 1 -- 172.21.15.116:0/3423506257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d000034e0 con 0x7f5d00004780 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d0ffff640 10 monclient: _finish_auth 0 2024-09-18T12:30:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:07.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.729+0000 7f5d1481c640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00003bc0 0x7f5d00003fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0ffff640 1 -- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d00004780 msgr2=0x7f5d00004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0ffff640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d00004780 0x7f5d00004bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc001b10 tx=0x7f5cfc007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a361aa0c34578ac server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc02ab30 con 0x7f5d00004780 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 10 monclient: dump: 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.748 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d16a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc02ac90 con 0x7f5d00004780 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d16a80640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d16a80640 4 set_mon_vals no callback set 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 1 -- 172.21.15.116:0/3423506257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc007340 con 0x7f5d00004780 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.731+0000 7f5d0f7fe640 10 monclient: dump: 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:07.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 10 monclient: shutdown 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 1 -- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d00004780 msgr2=0x7f5d00004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d00004780 0x7f5d00004bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc001b10 tx=0x7f5cfc007c80 comp rx=0 tx=0).stop 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 1 -- 172.21.15.116:0/3423506257 shutdown_connections 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 1 --2- 172.21.15.116:0/3423506257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d00004780 0x7f5d00004bf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 1 -- 172.21.15.116:0/3423506257 >> 172.21.15.116:0/3423506257 conn(0x7f5d100672b0 msgr2=0x7f5d100676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 1 -- 172.21.15.116:0/3423506257 shutdown_connections 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.732+0000 7f5d16a80640 1 -- 172.21.15.116:0/3423506257 wait complete. 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 10 monclient: build_initial_monmap 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 10 monclient: monmap: 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 1 Processor -- start 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 1 -- start start 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 10 monclient: init 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 0x7f5d10124d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 10 monclient(hunting): picked mon.noname-c con 0x7f5d00004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d1006d070 0x7f5d100696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1006d070 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.733+0000 7f5d16a80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d10069be0 0x7f5d1006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 10 monclient(hunting): picked mon.noname-a con 0x7f5d10069be0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1006e8d0 con 0x7f5d10069be0 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10098300 con 0x7f5d1006d070 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1012a420 con 0x7f5d00004780 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d16a80640 10 monclient(hunting): authenticate will time out at 1035.055064s 2024-09-18T12:30:07.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 0x7f5d10124d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 0x7f5d10124d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35874/0 (socket says 172.21.15.116:35874) 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 -- 172.21.15.116:0/574477811 learned_addr learned my addr 172.21.15.116:0/574477811 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): get_auth_request con 0x7f5d00004780 auth_method 0 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d10069be0 0x7f5d1006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 10 monclient(hunting): get_auth_request con 0x7f5d10069be0 auth_method 0 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1501d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient(hunting): handle_auth_done global_id 64301 payload 824 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 -- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d1006d070 msgr2=0x7f5d100696a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d1006d070 0x7f5d100696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 -- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d10069be0 msgr2=0x7f5d1006a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d10069be0 0x7f5d1006a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 monclient: found mon.noname-c 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.734+0000 7f5d1481c640 1 -- 172.21.15.116:0/574477811 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1006c690 con 0x7f5d00004780 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d1481c640 1 -- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 msgr2=0x7f5d10124d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d1481c640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 0x7f5d10124d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5d04007140 tx=0x7f5d04007a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 1 -- 172.21.15.116:0/574477811 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040116d0 con 0x7f5d00004780 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: dump: 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 1 -- 172.21.15.116:0/574477811 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d04011830 con 0x7f5d00004780 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d16a80640 5 monclient: authenticate success, global_id 64301 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d16a80640 10 monclient: _renew_subs 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d16a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1006b450 con 0x7f5d00004780 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 1 -- 172.21.15.116:0/574477811 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04011bd0 con 0x7f5d00004780 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0cff9640 10 monclient: dump: 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d0effd640 4 set_mon_vals no callback set 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d16a80640 10 monclient: _renew_subs 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d16a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.736+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1006f530 con 0x7f5d00004780 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.737+0000 7f5d0cff9640 1 -- 172.21.15.116:0/574477811 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5d0401e3b0 con 0x7f5d00004780 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.737+0000 7f5d16a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.737+0000 7f5d16a80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.737+0000 7f5d16a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.737+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1006c820 con 0x7f5d00004780 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.737+0000 7f5d0cff9640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cf4038490 0x7f5cf403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.737+0000 7f5d0cff9640 1 -- 172.21.15.116:0/574477811 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6024+0+0 (secure 0 0 0) 0x7f5d0404b700 con 0x7f5d00004780 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.738+0000 7f5d0ffff640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cf4038490 0x7f5cf403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.738+0000 7f5d0ffff640 10 monclient: get_auth_request con 0x7f5cf4038490 auth_method 0 2024-09-18T12:30:07.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.740+0000 7f5d0ffff640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cf4038490 0x7f5cf403a940 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.742+0000 7f5d0cff9640 1 -- 172.21.15.116:0/574477811 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5d0401b680 con 0x7f5d00004780 2024-09-18T12:30:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.742+0000 7f5d0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.742+0000 7f5d0cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:07.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.867+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v106) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4f4401dbb0 con 0x7f4f541321a0 2024-09-18T12:30:07.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.867+0000 7f4f3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:30:07.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.867+0000 7f4f3affd640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.963+0000 7f5d16a80640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.963+0000 7f5d16a80640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.963+0000 7f5d16a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.963+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5d1004bf60 con 0x7f5d00004780 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.964+0000 7f5d0cff9640 1 -- 172.21.15.116:0/574477811 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f5d0400c290 con 0x7f5d00004780 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.964+0000 7f5d0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.964+0000 7f5d0cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:07.965 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.965+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cf4038490 msgr2=0x7f5cf403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.965+0000 7f5d16a80640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cf4038490 0x7f5cf403a940 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc030000 comp rx=0 tx=0).stop 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.965+0000 7f5d16a80640 10 monclient: shutdown 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.965+0000 7f5d16a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.965+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 msgr2=0x7f5d10124d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.965+0000 7f5d16a80640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 0x7f5d10124d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5d04007140 tx=0x7f5d04007a90 comp rx=0 tx=0).stop 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 shutdown_connections 2024-09-18T12:30:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cf4038490 0x7f5cf403a940 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d10069be0 0x7f5d1006a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d1006d070 0x7f5d100696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 --2- 172.21.15.116:0/574477811 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d00004780 0x7f5d10124d80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 >> 172.21.15.116:0/574477811 conn(0x7f5d100672b0 msgr2=0x7f5d1009b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 shutdown_connections 2024-09-18T12:30:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:07.966+0000 7f5d16a80640 1 -- 172.21.15.116:0/574477811 wait complete. 2024-09-18T12:30:07.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.967+0000 7f4f5911d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:30:07.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.967+0000 7f4f5911d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:30:07.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.967+0000 7f4f5911d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:07.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.967+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f4f54072640 con 0x7f4f541321a0 2024-09-18T12:30:07.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.968+0000 7f4f3affd640 1 -- 172.21.15.144:0/3143035663 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v106) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4f4401dd90 con 0x7f4f541321a0 2024-09-18T12:30:07.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.968+0000 7f4f3affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T12:30:07.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.968+0000 7f4f3affd640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-18T12:30:07.970 INFO:tasks.workunit.client.0.smithi144.stderr:device 'foo' does not appear in the crush map 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f40038490 msgr2=0x7f4f4003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f40038490 0x7f4f4003a940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c034000 tx=0x7f4f4c032000 comp rx=0 tx=0).stop 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 10 monclient: shutdown 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 msgr2=0x7f4f54131170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 0x7f4f54131170 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400d890 tx=0x7f4f44008040 comp rx=0 tx=0).stop 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 shutdown_connections 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f40038490 0x7f4f4003a940 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f541321a0 0x7f4f54131170 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f5409f020 0x7f4f54130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 --2- 172.21.15.144:0/3143035663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f5409ec50 0x7f4f5409ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:07.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 >> 172.21.15.144:0/3143035663 conn(0x7f4f540a5cf0 msgr2=0x7f4f5409e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:07.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 shutdown_connections 2024-09-18T12:30:07.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:07.970+0000 7f4f5911d640 1 -- 172.21.15.144:0/3143035663 wait complete. 2024-09-18T12:30:07.977 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:30:07.977 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:30:07.977 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:30:07.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:30:07.980 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 2024-09-18T12:30:08.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.050+0000 7f76d0f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:08.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.050+0000 7f76d0f9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.050+0000 7f76d0f9d640 10 monclient: monmap: 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:08.051036+0000 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:08.051036+0000 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:08.052 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:08.053 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:08.053 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:08.053 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:08.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.050+0000 7f76d0f9d640 1 Processor -- start 2024-09-18T12:30:08.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.050+0000 7f76d0f9d640 1 -- start start 2024-09-18T12:30:08.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.050+0000 7f76d0f9d640 10 monclient: init 2024-09-18T12:30:08.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T12:30:08.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76cc131b40 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:08.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f76cc131f10 0x7f76cc132330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f76cc131f10 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:08.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76cc136e70 0x7f76cc139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f76cc136e70 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc06a230 con 0x7f76cc131b40 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0a0300 con 0x7f76cc136e70 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0a00c0 con 0x7f76cc131f10 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76d0f9d640 10 monclient(hunting): authenticate will time out at 1043.620391s 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f76cc131f10 0x7f76cc132330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f76cc131f10 0x7f76cc132330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38460/0 (socket says 172.21.15.144:38460) 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 -- 172.21.15.144:0/792412594 learned_addr learned my addr 172.21.15.144:0/792412594 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): get_auth_request con 0x7f76cc131f10 auth_method 0 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76cad9d640 1 --2- 172.21.15.144:0/792412594 >> v2:172.21.15.116:3300/0 conn(0x7f76cc136e70 0x7f76cc139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76cad9d640 10 monclient(hunting): get_auth_request con 0x7f76cc136e70 auth_method 0 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient(hunting): handle_auth_done global_id 5416 payload 274 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 -- 172.21.15.144:0/792412594 >> v2:172.21.15.116:3300/0 conn(0x7f76cc136e70 msgr2=0x7f76cc139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 --2- 172.21.15.144:0/792412594 >> v2:172.21.15.116:3300/0 conn(0x7f76cc136e70 0x7f76cc139250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 -- 172.21.15.144:0/792412594 >> v1:172.21.15.116:6789/0 conn(0x7f76cc131b40 legacy=0x7f76cc09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 monclient: found mon.noname-c 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.051+0000 7f76c9d9b640 1 -- 172.21.15.144:0/792412594 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc09bee0 con 0x7f76cc131f10 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c9d9b640 1 -- 172.21.15.144:0/792412594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131f10 msgr2=0x7f76cc132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c9d9b640 1 --2- 172.21.15.144:0/792412594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131f10 0x7f76cc132330 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f76cc09cbe0 tx=0x7f76b40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3607e00ea5eeea69 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 1 -- 172.21.15.144:0/792412594 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4032440 con 0x7f76cc131f10 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: dump: 2024-09-18T12:30:08.065 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76d0f9d640 5 monclient: authenticate success, global_id 5416 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76d0f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:08.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 1 -- 172.21.15.144:0/792412594 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b40325a0 con 0x7f76cc131f10 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76d0f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76d0f9d640 4 set_mon_vals no callback set 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 1 -- 172.21.15.144:0/792412594 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4032980 con 0x7f76cc131f10 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.054+0000 7f76c959a640 10 monclient: dump: 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.067 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 10 monclient: shutdown 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 -- 172.21.15.144:0/792412594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131f10 msgr2=0x7f76cc132330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/792412594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131f10 0x7f76cc132330 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f76cc09cbe0 tx=0x7f76b40091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 -- 172.21.15.144:0/792412594 shutdown_connections 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/792412594 >> v2:172.21.15.116:3300/0 conn(0x7f76cc136e70 0x7f76cc139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/792412594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131f10 0x7f76cc132330 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 -- 172.21.15.144:0/792412594 >> 172.21.15.144:0/792412594 conn(0x7f76cc0a5100 msgr2=0x7f76cc0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 -- 172.21.15.144:0/792412594 shutdown_connections 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 1 -- 172.21.15.144:0/792412594 wait complete. 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.055+0000 7f76d0f9d640 10 monclient: monmap: 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:08.069 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 Processor -- start 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 -- start start 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient: init 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 0x7f76cc12e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f76cc131b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76cc131f10 0x7f76cc0976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f76cc131f10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76cc136e70 0x7f76cc097c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f76cc136e70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:08.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc09b440 con 0x7f76cc136e70 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc099840 con 0x7f76cc131b40 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0a0a30 con 0x7f76cc131f10 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76d0f9d640 10 monclient(hunting): authenticate will time out at 1043.625674s 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 0x7f76cc12e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 0x7f76cc12e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:38468/0 (socket says 172.21.15.144:38468) 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 1 -- 172.21.15.144:0/4057802241 learned_addr learned my addr 172.21.15.144:0/4057802241 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:08.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 10 monclient(hunting): get_auth_request con 0x7f76cc131b40 auth_method 0 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76cad9d640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76cc136e70 0x7f76cc097c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76c9d9b640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76cc131f10 0x7f76cc0976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76cad9d640 10 monclient(hunting): get_auth_request con 0x7f76cc136e70 auth_method 0 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76c9d9b640 10 monclient(hunting): get_auth_request con 0x7f76cc131f10 auth_method 0 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.056+0000 7f76cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 10 monclient(hunting): handle_auth_done global_id 5419 payload 824 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 1 -- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76cc131f10 msgr2=0x7f76cc0976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76cc131f10 0x7f76cc0976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 1 -- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76cc136e70 msgr2=0x7f76cc097c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76cc136e70 0x7f76cc097c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 1 monclient: found mon.noname-b 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.057+0000 7f76ca59c640 1 -- 172.21.15.144:0/4057802241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b4009450 con 0x7f76cc131b40 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76ca59c640 1 -- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 msgr2=0x7f76cc12e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76ca59c640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 0x7f76cc12e990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f76c400c8b0 tx=0x7f76c40026c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c400cea0 con 0x7f76cc131b40 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: dump: 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:08.076 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76d0f9d640 5 monclient: authenticate success, global_id 5419 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c400aa60 con 0x7f76cc131b40 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76d0f9d640 10 monclient: _renew_subs 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76d0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc1310b0 con 0x7f76cc131b40 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c400ae20 con 0x7f76cc131b40 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c27fc640 10 monclient: dump: 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:08.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76c8d99640 4 set_mon_vals no callback set 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76d0f9d640 10 monclient: _renew_subs 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76d0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.059+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc09aa90 con 0x7f76cc131b40 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f76c4012630 con 0x7f76cc131b40 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76d0f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76d0f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76d0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc0636e0 con 0x7f76cc131b40 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76c27fc640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f76ac038490 0x7f76ac03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7f76c404b8f0 con 0x7f76cc131b40 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76c9d9b640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f76ac038490 0x7f76ac03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.060+0000 7f76c9d9b640 10 monclient: get_auth_request con 0x7f76ac038490 auth_method 0 2024-09-18T12:30:08.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.062+0000 7f76c9d9b640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f76ac038490 0x7f76ac03a940 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f76b4034000 tx=0x7f76b4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.064+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f76c40188c0 con 0x7f76cc131b40 2024-09-18T12:30:08.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.064+0000 7f76c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.064+0000 7f76c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:08.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.069+0000 7f96a58dd640 10 monclient: get_monmap_and_config 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.069+0000 7f96a58dd640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.069+0000 7f96a58dd640 10 monclient: monmap: 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:08.070459+0000 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:08.070459+0000 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 1 Processor -- start 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 1 -- start start 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 10 monclient: init 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96a0131ab0 0x7f96a0131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 10 monclient(hunting): picked mon.noname-c con 0x7f96a0131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96a009e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0136bb0 0x7f96a0138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 10 monclient(hunting): picked mon.noname-b con 0x7f96a0136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a009faf0 con 0x7f96a009e910 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.070+0000 7f96a58dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f96a58dd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a009fd30 con 0x7f96a0131ab0 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f96a58dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f96a58dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00a1110 con 0x7f96a0136bb0 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f96a58dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f96a58dd640 10 monclient(hunting): authenticate will time out at 1035.392016s 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0136bb0 0x7f96a0138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f9697fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f96a009e910 0x7f96a009ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33442/0 (socket says 172.21.15.116:33442) 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f9697fff640 1 -- 172.21.15.116:0/3137317463 learned_addr learned my addr 172.21.15.116:0/3137317463 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): get_auth_request con 0x7f96a0136bb0 auth_method 0 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 1 --2- 172.21.15.116:0/3137317463 >> v2:172.21.15.144:3300/0 conn(0x7f96a0131ab0 0x7f96a0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 10 monclient(hunting): get_auth_request con 0x7f96a0131ab0 auth_method 0 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient(hunting): handle_auth_done global_id 64307 payload 274 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 1 -- 172.21.15.116:0/3137317463 >> v2:172.21.15.144:3300/0 conn(0x7f96a0131ab0 msgr2=0x7f96a0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 1 --2- 172.21.15.116:0/3137317463 >> v2:172.21.15.144:3300/0 conn(0x7f96a0131ab0 0x7f96a0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 1 -- 172.21.15.116:0/3137317463 >> v1:172.21.15.116:6789/0 conn(0x7f96a009e910 legacy=0x7f96a009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.071+0000 7f969f7fe640 1 -- 172.21.15.116:0/3137317463 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a00a13f0 con 0x7f96a0136bb0 2024-09-18T12:30:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969f7fe640 1 -- 172.21.15.116:0/3137317463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0136bb0 msgr2=0x7f96a0138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969f7fe640 1 --2- 172.21.15.116:0/3137317463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0136bb0 0x7f96a0138f90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9698009e60 tx=0x7f9698009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=984ad07a4e52615 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 1 -- 172.21.15.116:0/3137317463 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9698007760 con 0x7f96a0136bb0 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: dump: 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 1 -- 172.21.15.116:0/3137317463 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969802d050 con 0x7f96a0136bb0 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 5 monclient: authenticate success, global_id 64307 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 1 -- 172.21.15.116:0/3137317463 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9698007af0 con 0x7f96a0136bb0 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 4 set_mon_vals no callback set 2024-09-18T12:30:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f969e7fc640 10 monclient: dump: 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 10 monclient: shutdown 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 -- 172.21.15.116:0/3137317463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0136bb0 msgr2=0x7f96a0138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 --2- 172.21.15.116:0/3137317463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0136bb0 0x7f96a0138f90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9698009e60 tx=0x7f9698009780 comp rx=0 tx=0).stop 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 -- 172.21.15.116:0/3137317463 shutdown_connections 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 --2- 172.21.15.116:0/3137317463 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0136bb0 0x7f96a0138f90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 --2- 172.21.15.116:0/3137317463 >> v2:172.21.15.144:3300/0 conn(0x7f96a0131ab0 0x7f96a0131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 -- 172.21.15.116:0/3137317463 >> 172.21.15.116:0/3137317463 conn(0x7f96a0067d30 msgr2=0x7f96a00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 -- 172.21.15.116:0/3137317463 shutdown_connections 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.074+0000 7f96a58dd640 1 -- 172.21.15.116:0/3137317463 wait complete. 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.075+0000 7f96a58dd640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.075+0000 7f96a58dd640 10 monclient: monmap: 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.075+0000 7f96a58dd640 1 Processor -- start 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.075+0000 7f96a58dd640 1 -- start start 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.075+0000 7f96a58dd640 10 monclient: init 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96a009e910 0x7f96a012f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): picked mon.noname-b con 0x7f96a009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 0x7f96a012f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): picked mon.noname-c con 0x7f96a0131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0136bb0 0x7f96a012fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): picked mon.noname-a con 0x7f96a0136bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a01336d0 con 0x7f96a0136bb0 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a0131270 con 0x7f96a009e910 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a009bde0 con 0x7f96a0131ab0 2024-09-18T12:30:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f96a58dd640 10 monclient(hunting): authenticate will time out at 1035.397137s 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 0x7f96a012f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0136bb0 0x7f96a012fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 0x7f96a012f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35900/0 (socket says 172.21.15.116:35900) 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0136bb0 0x7f96a012fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46372/0 (socket says 172.21.15.116:46372) 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 -- 172.21.15.116:0/2510369693 learned_addr learned my addr 172.21.15.116:0/2510369693 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): get_auth_request con 0x7f96a0131ab0 auth_method 0 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 10 monclient(hunting): get_auth_request con 0x7f96a0136bb0 auth_method 0 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96a009e910 0x7f96a012f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 10 monclient(hunting): get_auth_request con 0x7f96a009e910 auth_method 0 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient(hunting): handle_auth_done global_id 64310 payload 824 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96a009e910 msgr2=0x7f96a012f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96a009e910 0x7f96a012f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0136bb0 msgr2=0x7f96a012fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0136bb0 0x7f96a012fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 monclient: found mon.noname-c 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.076+0000 7f9697fff640 1 -- 172.21.15.116:0/2510369693 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9698009a20 con 0x7f96a0131ab0 2024-09-18T12:30:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9697fff640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 msgr2=0x7f96a012f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9697fff640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 0x7f96a012f690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f968800b510 tx=0x7f96880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688011840 con 0x7f96a0131ab0 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: dump: 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f96a58dd640 5 monclient: authenticate success, global_id 64310 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96880119c0 con 0x7f96a0131ab0 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688011d80 con 0x7f96a0131ab0 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f96a58dd640 10 monclient: _renew_subs 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f96a58dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a0131610 con 0x7f96a0131ab0 2024-09-18T12:30:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f969dffb640 4 set_mon_vals no callback set 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f9696ffd640 10 monclient: dump: 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f96a58dd640 10 monclient: _renew_subs 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f96a58dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.078+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a0133c90 con 0x7f96a0131ab0 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9688010590 con 0x7f96a0131ab0 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f96a58dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f96a58dd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 0x7f96a0138320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f96a58dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a0063740 con 0x7f96a0130110 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 0x7f96a0138320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 10 monclient: get_auth_request con 0x7f96a0130110 auth_method 0 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f9696ffd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96800384c0 0x7f968003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.079+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.080+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7f968804b3b0 con 0x7f96a0131ab0 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.080+0000 7f969effd640 10 monclient(hunting): handle_auth_done global_id 74178 payload 824 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.080+0000 7f969effd640 1 -- 172.21.15.116:0/2510369693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96a004bf60 con 0x7f96a0130110 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.080+0000 7f969f7fe640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96800384c0 0x7f968003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.080+0000 7f969f7fe640 10 monclient: get_auth_request con 0x7f96800384c0 auth_method 0 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f969effd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 msgr2=0x7f96a0138320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f969effd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 0x7f96a0138320 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9690011c20 tx=0x7f96900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690011e60 con 0x7f96a0130110 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 10 monclient: dump: 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.110 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9690014650 con 0x7f96a0130110 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 msgr2=0x7f96a0138320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 0x7f96a0138320 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9690011c20 tx=0x7f96900090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 msgr2=0x7f96a0138320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f9696ffd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0130110 0x7f96a0138320 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9690011c20 tx=0x7f96900090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.082+0000 7f969f7fe640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96800384c0 0x7f968003a970 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f96a009bdb0 tx=0x7f9698037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f96a58dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f96a58dd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 0x7f96a0138c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f96a58dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a009db50 con 0x7f96a0138860 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 0x7f96a0138c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient: get_auth_request con 0x7f96a0138860 auth_method 0 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 10 monclient(hunting): handle_auth_done global_id 74181 payload 824 2024-09-18T12:30:08.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.088+0000 7f969effd640 1 -- 172.21.15.116:0/2510369693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f96a01304e0 con 0x7f96a0138860 2024-09-18T12:30:08.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f969effd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 msgr2=0x7f96a0138c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f969effd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 0x7f96a0138c60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9690011c20 tx=0x7f969001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9690028400 con 0x7f96a0138860 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 10 monclient: dump: 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.114 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f969001c3e0 con 0x7f96a0138860 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 msgr2=0x7f96a0138c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 0x7f96a0138c60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9690011c20 tx=0x7f969001c000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 msgr2=0x7f96a0138c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f9696ffd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a0138860 0x7f96a0138c60 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9690011c20 tx=0x7f969001c000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.115 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 54, 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:30:08.116 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.117 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.118 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:08.119 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.120 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.123 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.124 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.125 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:08.126 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.127 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.128 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:08.128 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96800384c0 msgr2=0x7f968003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96800384c0 0x7f968003a970 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f96a009bdb0 tx=0x7f9698037000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 10 monclient: shutdown 2024-09-18T12:30:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 msgr2=0x7f96a012f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 0x7f96a012f690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f968800b510 tx=0x7f96880090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f969f7fe640 1 -- 172.21.15.116:0/2510369693 reap_dead start 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 shutdown_connections 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 --2- 172.21.15.116:0/2510369693 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96a0131ab0 0x7f96a012f690 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 >> 172.21.15.116:0/2510369693 conn(0x7f96a0067d30 msgr2=0x7f96a012cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 shutdown_connections 2024-09-18T12:30:08.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.091+0000 7f96a58dd640 1 -- 172.21.15.116:0/2510369693 wait complete. 2024-09-18T12:30:08.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:08.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:30:08.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:30:08.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.224+0000 7fa3e659d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:08.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.224+0000 7fa3e659d640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.224+0000 7fa3e659d640 10 monclient: monmap: 2024-09-18T12:30:08.225 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.225 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.225 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:08.224795+0000 2024-09-18T12:30:08.225 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:08.224795+0000 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:08.226 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.224+0000 7fa3e659d640 1 Processor -- start 2024-09-18T12:30:08.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.224+0000 7fa3e659d640 1 -- start start 2024-09-18T12:30:08.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.224+0000 7fa3e659d640 10 monclient: init 2024-09-18T12:30:08.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3e0131ab0 0x7fa3e0131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e0131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3e009e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3e0136bb0 0x7fa3e0138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e0136bb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e009faf0 con 0x7fa3e009e910 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e009fd30 con 0x7fa3e0136bb0 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e00a1110 con 0x7fa3e0131ab0 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e659d640 10 monclient(hunting): authenticate will time out at 1035.546422s 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3e0136bb0 0x7fa3e0138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3e0136bb0 0x7fa3e0138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46410/0 (socket says 172.21.15.116:46410) 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 -- 172.21.15.116:0/1710810416 learned_addr learned my addr 172.21.15.116:0/1710810416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): get_auth_request con 0x7fa3e0136bb0 auth_method 0 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 1 --2- 172.21.15.116:0/1710810416 >> v2:172.21.15.144:3300/0 conn(0x7fa3e0131ab0 0x7fa3e0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 10 monclient(hunting): get_auth_request con 0x7fa3e0131ab0 auth_method 0 2024-09-18T12:30:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient(hunting): handle_auth_done global_id 74184 payload 274 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 -- 172.21.15.116:0/1710810416 >> v2:172.21.15.144:3300/0 conn(0x7fa3e0131ab0 msgr2=0x7fa3e0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 --2- 172.21.15.116:0/1710810416 >> v2:172.21.15.144:3300/0 conn(0x7fa3e0131ab0 0x7fa3e0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 -- 172.21.15.116:0/1710810416 >> v1:172.21.15.144:6789/0 conn(0x7fa3e009e910 legacy=0x7fa3e009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 monclient: found mon.noname-a 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.225+0000 7fa3e4b3a640 1 -- 172.21.15.116:0/1710810416 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e00a13f0 con 0x7fa3e0136bb0 2024-09-18T12:30:08.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.233+0000 7f76d0f9d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:30:08.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.233+0000 7f76d0f9d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T12:30:08.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.233+0000 7f76d0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:08.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:08.233+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f76cc099e90 con 0x7f76cc131b40 2024-09-18T12:30:08.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.228+0000 7fa3e4b3a640 1 -- 172.21.15.116:0/1710810416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0136bb0 msgr2=0x7fa3e0138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.228+0000 7fa3e4b3a640 1 --2- 172.21.15.116:0/1710810416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0136bb0 0x7fa3e0138f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8009f20 tx=0x7fa3d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8e8b3e5076000f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.228+0000 7fa3deffd640 1 -- 172.21.15.116:0/1710810416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8007610 con 0x7fa3e0136bb0 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.228+0000 7fa3deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.228+0000 7fa3deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.228+0000 7fa3deffd640 10 monclient: dump: 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 1 -- 172.21.15.116:0/1710810416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d802d050 con 0x7fa3e0136bb0 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 5 monclient: authenticate success, global_id 74184 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 1 -- 172.21.15.116:0/1710810416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80079a0 con 0x7fa3e0136bb0 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 4 set_mon_vals no callback set 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3deffd640 10 monclient: dump: 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 10 monclient: shutdown 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 -- 172.21.15.116:0/1710810416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0136bb0 msgr2=0x7fa3e0138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 --2- 172.21.15.116:0/1710810416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0136bb0 0x7fa3e0138f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8009f20 tx=0x7fa3d8009840 comp rx=0 tx=0).stop 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 -- 172.21.15.116:0/1710810416 shutdown_connections 2024-09-18T12:30:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 --2- 172.21.15.116:0/1710810416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0136bb0 0x7fa3e0138f90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 --2- 172.21.15.116:0/1710810416 >> v2:172.21.15.144:3300/0 conn(0x7fa3e0131ab0 0x7fa3e0131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 -- 172.21.15.116:0/1710810416 >> 172.21.15.116:0/1710810416 conn(0x7fa3e0067d30 msgr2=0x7fa3e00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 -- 172.21.15.116:0/1710810416 shutdown_connections 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.229+0000 7fa3e659d640 1 -- 172.21.15.116:0/1710810416 wait complete. 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient: monmap: 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 Processor -- start 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 -- start start 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient: init 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3e009e910 0x7fa3e0130f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0131ab0 0x7fa3e012f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e0131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e0136bb0 0x7fa3e012f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e0136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e0133bb0 con 0x7fa3e0131ab0 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e01333c0 con 0x7fa3e009e910 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e01335f0 con 0x7fa3e0136bb0 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.230+0000 7fa3e659d640 10 monclient(hunting): authenticate will time out at 1035.551966s 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0131ab0 0x7fa3e012f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e0136bb0 0x7fa3e012f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0131ab0 0x7fa3e012f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46416/0 (socket says 172.21.15.116:46416) 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 -- 172.21.15.116:0/2758744006 learned_addr learned my addr 172.21.15.116:0/2758744006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): get_auth_request con 0x7fa3e0131ab0 auth_method 0 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 10 monclient(hunting): get_auth_request con 0x7fa3e0136bb0 auth_method 0 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3e009e910 0x7fa3e0130f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 10 monclient(hunting): get_auth_request con 0x7fa3e009e910 auth_method 0 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient(hunting): handle_auth_done global_id 74187 payload 824 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e0136bb0 msgr2=0x7fa3e012f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e0136bb0 0x7fa3e012f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3e009e910 msgr2=0x7fa3e0130f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3e009e910 0x7fa3e0130f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 monclient: found mon.noname-a 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3df7fe640 1 -- 172.21.15.116:0/2758744006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8009ae0 con 0x7fa3e0131ab0 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.231+0000 7fa3e4b3a640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e0136bb0 0x7fa3e012f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3df7fe640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0131ab0 msgr2=0x7fa3e012f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3df7fe640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0131ab0 0x7fa3e012f2d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400c610 tx=0x7fa3d4007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4012840 con 0x7fa3e0131ab0 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: dump: 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3e659d640 5 monclient: authenticate success, global_id 74187 2024-09-18T12:30:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d40129c0 con 0x7fa3e0131ab0 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3e659d640 10 monclient: _renew_subs 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4012d80 con 0x7fa3e0131ab0 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3e659d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e0130cc0 con 0x7fa3e0131ab0 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3c7fff640 10 monclient: dump: 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.254 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3de7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3e659d640 10 monclient: _renew_subs 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3e659d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.233+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e01361d0 con 0x7fa3e0131ab0 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3e659d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3e659d640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 0x7fa3e009c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3e659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e0063740 con 0x7fa3e012fd50 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa3d4011590 con 0x7fa3e0131ab0 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 0x7fa3e009c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient: get_auth_request con 0x7fa3e012fd50 auth_method 0 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3cc0384c0 0x7fa3cc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 10 monclient(hunting): handle_auth_done global_id 64316 payload 824 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7fa3d404b710 con 0x7fa3e0131ab0 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.234+0000 7fa3dffff640 1 -- 172.21.15.116:0/2758744006 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3e0130120 con 0x7fa3e012fd50 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.235+0000 7fa3e4b3a640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3cc0384c0 0x7fa3cc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.235+0000 7fa3e4b3a640 10 monclient: get_auth_request con 0x7fa3cc0384c0 auth_method 0 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3dffff640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 msgr2=0x7fa3e009c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3dffff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 0x7fa3e009c8d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011c20 tx=0x7fa3d00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0011e60 con 0x7fa3e012fd50 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 10 monclient: dump: 2024-09-18T12:30:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa3d0014650 con 0x7fa3e012fd50 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:08.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 msgr2=0x7fa3e009c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 0x7fa3e009c8d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011c20 tx=0x7fa3d00090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 msgr2=0x7fa3e009c8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:08.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e012fd50 0x7fa3e009c8d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.237+0000 7fa3e4b3a640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3cc0384c0 0x7fa3cc03a970 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa3e009bdb0 tx=0x7fa3d8036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3e659d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3e659d640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 0x7fa3e009d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3e659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e0131610 con 0x7fa3e009ce10 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 0x7fa3e009d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient: get_auth_request con 0x7fa3e009ce10 auth_method 0 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 10 monclient(hunting): handle_auth_done global_id 64319 payload 824 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.243+0000 7fa3dffff640 1 -- 172.21.15.116:0/2758744006 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa3e004bf60 con 0x7fa3e009ce10 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.245+0000 7fa3dffff640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 msgr2=0x7fa3e009d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.245+0000 7fa3dffff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 0x7fa3e009d210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011c20 tx=0x7fa3d001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.245+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d001c5f0 con 0x7fa3e009ce10 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.245+0000 7fa3c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.245+0000 7fa3c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.245+0000 7fa3c7fff640 10 monclient: dump: 2024-09-18T12:30:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fa3d0014440 con 0x7fa3e009ce10 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 msgr2=0x7fa3e009d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 0x7fa3e009d210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011c20 tx=0x7fa3d001c000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3c7fff640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 msgr2=0x7fa3e009d210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3e009ce10 0x7fa3e009d210 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011c20 tx=0x7fa3d001c000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 54, 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:08.264 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.265 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.266 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:08.267 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.268 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:08.270 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.272 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.274 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:08.276 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3cc0384c0 msgr2=0x7fa3cc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3cc0384c0 0x7fa3cc03a970 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa3e009bdb0 tx=0x7fa3d8036000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 10 monclient: shutdown 2024-09-18T12:30:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0131ab0 msgr2=0x7fa3e012f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 --2- 172.21.15.116:0/2758744006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0131ab0 0x7fa3e012f2d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400c610 tx=0x7fa3d4007ed0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e4b3a640 1 -- 172.21.15.116:0/2758744006 reap_dead start 2024-09-18T12:30:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e4b3a640 1 -- 172.21.15.116:0/2758744006 reap_dead start 2024-09-18T12:30:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 shutdown_connections 2024-09-18T12:30:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 >> 172.21.15.116:0/2758744006 conn(0x7fa3e0067d30 msgr2=0x7fa3e012cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 shutdown_connections 2024-09-18T12:30:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.246+0000 7fa3e659d640 1 -- 172.21.15.116:0/2758744006 wait complete. 2024-09-18T12:30:08.280 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:08.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:30:08.281 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:30:08.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.373+0000 7ff5f6a34640 10 monclient: get_monmap_and_config 2024-09-18T12:30:08.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.373+0000 7ff5f6a34640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.373+0000 7ff5f6a34640 10 monclient: monmap: 2024-09-18T12:30:08.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.375 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.375 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:08.374408+0000 2024-09-18T12:30:08.375 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:08.374408+0000 2024-09-18T12:30:08.375 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.375 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:08.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 1 Processor -- start 2024-09-18T12:30:08.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 1 -- start start 2024-09-18T12:30:08.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 10 monclient: init 2024-09-18T12:30:08.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T12:30:08.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff5f00981c0 0x7ff5f0098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f00981c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5f0131da0 0x7ff5f0067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f0131da0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5f0132170 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.374+0000 7ff5f6a34640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f0098ad0 con 0x7ff5f0132170 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5f6a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5f6a34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f0131bd0 con 0x7ff5f0131da0 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5f6a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5f6a34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f009ca60 con 0x7ff5f00981c0 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5f6a34640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5f6a34640 10 monclient(hunting): authenticate will time out at 1035.696033s 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5f0131da0 0x7ff5f0067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff5f0131da0 0x7ff5f0067110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46422/0 (socket says 172.21.15.116:46422) 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/3988176150 learned_addr learned my addr 172.21.15.116:0/3988176150 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff5f0131da0 auth_method 0 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 1 --2- 172.21.15.116:0/3988176150 >> v2:172.21.15.144:3300/0 conn(0x7ff5f00981c0 0x7ff5f0098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 10 monclient(hunting): get_auth_request con 0x7ff5f00981c0 auth_method 0 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_done global_id 74190 payload 274 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/3988176150 >> v2:172.21.15.144:3300/0 conn(0x7ff5f00981c0 msgr2=0x7ff5f0098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/3988176150 >> v2:172.21.15.144:3300/0 conn(0x7ff5f00981c0 0x7ff5f0098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/3988176150 >> v1:172.21.15.144:6789/0 conn(0x7ff5f0132170 legacy=0x7ff5f0067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 monclient: found mon.noname-a 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.375+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/3988176150 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f009cca0 con 0x7ff5f0131da0 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/3988176150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f0131da0 msgr2=0x7ff5f0067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/3988176150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f0131da0 0x7ff5f0067110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0066560 tx=0x7ff5e40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f312bf1a5b3998a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 1 -- 172.21.15.116:0/3988176150 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4032440 con 0x7ff5f0131da0 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: dump: 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 1 -- 172.21.15.116:0/3988176150 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e40325a0 con 0x7ff5f0131da0 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 5 monclient: authenticate success, global_id 74190 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 1 -- 172.21.15.116:0/3988176150 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4032940 con 0x7ff5f0131da0 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 4 set_mon_vals no callback set 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5eeffd640 10 monclient: dump: 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.392 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 10 monclient: shutdown 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 -- 172.21.15.116:0/3988176150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f0131da0 msgr2=0x7ff5f0067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/3988176150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f0131da0 0x7ff5f0067110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0066560 tx=0x7ff5e40091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 -- 172.21.15.116:0/3988176150 shutdown_connections 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/3988176150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f0131da0 0x7ff5f0067110 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/3988176150 >> v2:172.21.15.144:3300/0 conn(0x7ff5f00981c0 0x7ff5f0098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 -- 172.21.15.116:0/3988176150 >> 172.21.15.116:0/3988176150 conn(0x7ff5f00a1e30 msgr2=0x7ff5f00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 -- 172.21.15.116:0/3988176150 shutdown_connections 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.378+0000 7ff5f6a34640 1 -- 172.21.15.116:0/3988176150 wait complete. 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.379+0000 7ff5f6a34640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.379+0000 7ff5f6a34640 10 monclient: monmap: 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.379+0000 7ff5f6a34640 1 Processor -- start 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 1 -- start start 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient: init 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 0x7ff5f011db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f00981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f0131da0 0x7ff5f011e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f0131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:08.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5f0132170 0x7ff5f011e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f0132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0120130 con 0x7ff5f00981c0 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0098cb0 con 0x7ff5f0131da0 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f00a08c0 con 0x7ff5f0132170 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f6a34640 10 monclient(hunting): authenticate will time out at 1035.701537s 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 0x7ff5f011db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 0x7ff5f011db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46430/0 (socket says 172.21.15.116:46430) 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 -- 172.21.15.116:0/1078165449 learned_addr learned my addr 172.21.15.116:0/1078165449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): get_auth_request con 0x7ff5f00981c0 auth_method 0 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5f0132170 0x7ff5f011e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 10 monclient(hunting): get_auth_request con 0x7ff5f0132170 auth_method 0 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f0131da0 0x7ff5f011e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff5f0131da0 auth_method 0 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient(hunting): handle_auth_done global_id 74193 payload 824 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5f0132170 msgr2=0x7ff5f011e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5f0132170 0x7ff5f011e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f0131da0 msgr2=0x7ff5f011e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f0131da0 0x7ff5f011e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 monclient: found mon.noname-a 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5effff640 1 -- 172.21.15.116:0/1078165449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e4009450 con 0x7ff5f00981c0 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.380+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f0131da0 0x7ff5f011e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5effff640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 msgr2=0x7ff5f011db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5effff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 0x7ff5f011db50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000ea20 tx=0x7ff5e00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000cc20 con 0x7ff5f00981c0 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: dump: 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5f6a34640 5 monclient: authenticate success, global_id 74193 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e000cd80 con 0x7ff5f00981c0 2024-09-18T12:30:08.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5f6a34640 10 monclient: _renew_subs 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5f6a34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f011f730 con 0x7ff5f00981c0 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0013680 con 0x7ff5f00981c0 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5d7fff640 10 monclient: dump: 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5f6a34640 10 monclient: _renew_subs 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5ee7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5f6a34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.383+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f01211f0 con 0x7ff5f00981c0 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff5e0013a30 con 0x7ff5f00981c0 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff5dc038490 0x7ff5dc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f6a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7ff5e004c340 con 0x7ff5f00981c0 2024-09-18T12:30:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 0x7ff5f011a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f6a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0065f90 con 0x7ff5f011eb10 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff5dc038490 0x7ff5dc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5ef7fe640 10 monclient: get_auth_request con 0x7ff5dc038490 auth_method 0 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 0x7ff5f011a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient: get_auth_request con 0x7ff5f011eb10 auth_method 0 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_done global_id 5440 payload 824 2024-09-18T12:30:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.384+0000 7ff5f4fd1640 1 -- 172.21.15.116:0/1078165449 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5f004bf60 con 0x7ff5f011eb10 2024-09-18T12:30:08.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.386+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff5dc038490 0x7ff5dc03a940 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff5f006f820 tx=0x7ff5e4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.386+0000 7ff5f4fd1640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 msgr2=0x7ff5f011a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.386+0000 7ff5f4fd1640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 0x7ff5f011a4a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800fb00 tx=0x7ff5e80025b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e800ddb0 con 0x7ff5f011eb10 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 10 monclient: dump: 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff5e8002850 con 0x7ff5f011eb10 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 msgr2=0x7ff5f011a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 0x7ff5f011a4a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800fb00 tx=0x7ff5e80025b0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 msgr2=0x7ff5f011a4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.387+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011eb10 0x7ff5f011a4a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f6a34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 0x7ff5f011ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f6a34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0098ee0 con 0x7ff5f011a9e0 2024-09-18T12:30:08.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 0x7ff5f011ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient: get_auth_request con 0x7ff5f011a9e0 auth_method 0 2024-09-18T12:30:08.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 10 monclient(hunting): handle_auth_done global_id 5443 payload 824 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.393+0000 7ff5f4fd1640 1 -- 172.21.15.116:0/1078165449 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff5f011eee0 con 0x7ff5f011a9e0 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.395+0000 7ff5f4fd1640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 msgr2=0x7ff5f011ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.395+0000 7ff5f4fd1640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 0x7ff5f011ade0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800fb00 tx=0x7ff5e8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.395+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8007030 con 0x7ff5f011a9e0 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 10 monclient: dump: 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7ff5e800dba0 con 0x7ff5f011a9e0 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 msgr2=0x7ff5f011ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 0x7ff5f011ade0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800fb00 tx=0x7ff5e8019000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 msgr2=0x7ff5f011ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5f011a9e0 0x7ff5f011ade0 secure :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800fb00 tx=0x7ff5e8019000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:08.413 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 54, 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.414 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.415 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:08.416 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:08.417 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.418 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.425 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:08.426 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:08.427 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff5dc038490 msgr2=0x7ff5dc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff5dc038490 0x7ff5dc03a940 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff5f006f820 tx=0x7ff5e4032000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 10 monclient: shutdown 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 msgr2=0x7ff5f011db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f4fd1640 1 -- 172.21.15.116:0/1078165449 reap_dead start 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 0x7ff5f011db50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000ea20 tx=0x7ff5e00090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 shutdown_connections 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 --2- 172.21.15.116:0/1078165449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f00981c0 0x7ff5f011db50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 >> 172.21.15.116:0/1078165449 conn(0x7ff5f00a1e30 msgr2=0x7ff5f0096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 shutdown_connections 2024-09-18T12:30:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.396+0000 7ff5f6a34640 1 -- 172.21.15.116:0/1078165449 wait complete. 2024-09-18T12:30:08.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:08.430 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:30:08.430 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:30:08.430 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:30:08.430 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:30:08.430 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:30:08.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:30:08.441 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:30:08.441 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:30:08.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.572+0000 7f166f257640 10 monclient: get_monmap_and_config 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.572+0000 7f166f257640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.572+0000 7f166f257640 10 monclient: monmap: 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:08.572746+0000 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:08.572746+0000 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:08.574 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.572+0000 7f166f257640 1 Processor -- start 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.572+0000 7f166f257640 1 -- start start 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.572+0000 7f166f257640 10 monclient: init 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 0x7f1668131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): picked mon.noname-b con 0x7f1668131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f166809e910 0x7f166809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): picked mon.noname-c con 0x7f166809e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f166809f220 0x7f1668068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): picked mon.noname-a con 0x7f166809f220 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166809faf0 con 0x7f166809f220 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166809fd30 con 0x7f166809e910 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16680a1110 con 0x7f1668131ab0 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166f257640 10 monclient(hunting): authenticate will time out at 1035.894382s 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166d7f4640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f166809f220 msgr2=0x7f1668068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166d7f4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f166809f220 0x7f1668068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 0x7f1668131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 0x7f1668131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35960/0 (socket says 172.21.15.116:35960) 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 -- 172.21.15.116:0/2189101711 learned_addr learned my addr 172.21.15.116:0/2189101711 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): get_auth_request con 0x7f1668131ab0 auth_method 0 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f1667fff640 1 --2- 172.21.15.116:0/2189101711 >> v2:172.21.15.144:3300/0 conn(0x7f166809e910 0x7f166809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f1667fff640 10 monclient(hunting): get_auth_request con 0x7f166809e910 auth_method 0 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f1667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f1667fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f1667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient(hunting): handle_auth_done global_id 64325 payload 274 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 -- 172.21.15.116:0/2189101711 >> v2:172.21.15.144:3300/0 conn(0x7f166809e910 msgr2=0x7f166809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 --2- 172.21.15.116:0/2189101711 >> v2:172.21.15.144:3300/0 conn(0x7f166809e910 0x7f166809ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 -- 172.21.15.116:0/2189101711 >> v2:172.21.15.116:3300/0 conn(0x7f166809f220 msgr2=0x7f1668068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 --2- 172.21.15.116:0/2189101711 >> v2:172.21.15.116:3300/0 conn(0x7f166809f220 0x7f1668068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 monclient: found mon.noname-b 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.573+0000 7f166cff3640 1 -- 172.21.15.116:0/2189101711 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16680a13f0 con 0x7f1668131ab0 2024-09-18T12:30:08.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f166cff3640 1 -- 172.21.15.116:0/2189101711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 msgr2=0x7f1668131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f166cff3640 1 --2- 172.21.15.116:0/2189101711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 0x7f1668131e80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16680a5ec0 tx=0x7f16600091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d86642ede1d29142 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 1 -- 172.21.15.116:0/2189101711 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660032440 con 0x7f1668131ab0 2024-09-18T12:30:08.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:08.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: dump: 2024-09-18T12:30:08.589 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.589 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.589 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f166f257640 5 monclient: authenticate success, global_id 64325 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f166f257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:08.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 1 -- 172.21.15.116:0/2189101711 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16600325a0 con 0x7f1668131ab0 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f166f257640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f166f257640 4 set_mon_vals no callback set 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 1 -- 172.21.15.116:0/2189101711 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660032940 con 0x7f1668131ab0 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.576+0000 7f16677fe640 10 monclient: dump: 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 10 monclient: shutdown 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 -- 172.21.15.116:0/2189101711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 msgr2=0x7f1668131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 --2- 172.21.15.116:0/2189101711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 0x7f1668131e80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16680a5ec0 tx=0x7f16600091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 -- 172.21.15.116:0/2189101711 shutdown_connections 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 --2- 172.21.15.116:0/2189101711 >> v2:172.21.15.116:3300/0 conn(0x7f166809f220 0x7f1668068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 --2- 172.21.15.116:0/2189101711 >> v2:172.21.15.144:3300/0 conn(0x7f166809e910 0x7f166809ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 --2- 172.21.15.116:0/2189101711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1668131ab0 0x7f1668131e80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 -- 172.21.15.116:0/2189101711 >> 172.21.15.116:0/2189101711 conn(0x7f1668067d30 msgr2=0x7f16680a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 -- 172.21.15.116:0/2189101711 shutdown_connections 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 1 -- 172.21.15.116:0/2189101711 wait complete. 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 10 monclient: build_initial_monmap 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.577+0000 7f166f257640 10 monclient: monmap: 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:08.593 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 Processor -- start 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 -- start start 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient: init 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166809e910 0x7f16681416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): picked mon.noname-a con 0x7f166809e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 0x7f1668141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): picked mon.noname-c con 0x7f166809f220 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1668131ab0 0x7f1668142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): picked mon.noname-b con 0x7f1668131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166809ddd0 con 0x7f166809e910 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166809bde0 con 0x7f1668131ab0 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16681207b0 con 0x7f166809f220 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166f257640 10 monclient(hunting): authenticate will time out at 1035.899868s 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166cff3640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166809e910 msgr2=0x7f16681416c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f166cff3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166809e910 0x7f16681416c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f1667fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 0x7f1668141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f1667fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 0x7f1668141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35972/0 (socket says 172.21.15.116:35972) 2024-09-18T12:30:08.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f1667fff640 1 -- 172.21.15.116:0/314913176 learned_addr learned my addr 172.21.15.116:0/314913176 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f1667fff640 10 monclient(hunting): get_auth_request con 0x7f166809f220 auth_method 0 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f1667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f1667fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.578+0000 7f1667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1668131ab0 0x7f1668142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 10 monclient(hunting): get_auth_request con 0x7f1668131ab0 auth_method 0 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:08.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f166d7f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 10 monclient(hunting): handle_auth_done global_id 64328 payload 824 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 1 -- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1668131ab0 msgr2=0x7f1668142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1668131ab0 0x7f1668142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 1 -- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166809e910 msgr2=0x7f16681416c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166809e910 0x7f16681416c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 1 monclient: found mon.noname-c 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.579+0000 7f1667fff640 1 -- 172.21.15.116:0/314913176 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1660009450 con 0x7f166809f220 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.581+0000 7f1667fff640 1 -- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 msgr2=0x7f1668141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.581+0000 7f1667fff640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 0x7f1668141c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f165800e190 tx=0x7f1658009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.581+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658013590 con 0x7f166809f220 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: dump: 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16580136f0 con 0x7f166809f220 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658013a90 con 0x7f166809f220 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f166f257640 5 monclient: authenticate success, global_id 64328 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f166f257640 10 monclient: _renew_subs 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f166f257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16681216a0 con 0x7f166809f220 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1666ffd640 4 set_mon_vals no callback set 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f1664ff9640 10 monclient: dump: 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f166f257640 10 monclient: _renew_subs 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f166f257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.583+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166811fbe0 con 0x7f166809f220 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.584+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f165801e460 con 0x7f166809f220 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.584+0000 7f166f257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.584+0000 7f166f257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.584+0000 7f166f257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.584+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166804bf60 con 0x7f166809f220 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.584+0000 7f1664ff9640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1648038490 0x7f164803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.584+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7f165804c2c0 con 0x7f166809f220 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.585+0000 7f166cff3640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1648038490 0x7f164803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.585+0000 7f166cff3640 10 monclient: get_auth_request con 0x7f1648038490 auth_method 0 2024-09-18T12:30:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.587+0000 7f166cff3640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1648038490 0x7f164803a940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f166809bd80 tx=0x7f1660032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.589+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1658013e40 con 0x7f166809f220 2024-09-18T12:30:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.589+0000 7f1664ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.589+0000 7f1664ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.803+0000 7f166f257640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.803+0000 7f166f257640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.803+0000 7f166f257640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.803+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1668066280 con 0x7f166809f220 2024-09-18T12:30:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.803+0000 7f1664ff9640 1 -- 172.21.15.116:0/314913176 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f165801b310 con 0x7f166809f220 2024-09-18T12:30:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.803+0000 7f1664ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.803+0000 7f1664ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:08.805 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1648038490 msgr2=0x7f164803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1648038490 0x7f164803a940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f166809bd80 tx=0x7f1660032000 comp rx=0 tx=0).stop 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 10 monclient: shutdown 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 msgr2=0x7f1668141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 0x7f1668141c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f165800e190 tx=0x7f1658009920 comp rx=0 tx=0).stop 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 shutdown_connections 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1648038490 0x7f164803a940 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1668131ab0 0x7f1668142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f166809f220 0x7f1668141c00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 --2- 172.21.15.116:0/314913176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f166809e910 0x7f16681416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:08.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 >> 172.21.15.116:0/314913176 conn(0x7f1668067d30 msgr2=0x7f16680a3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:08.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 shutdown_connections 2024-09-18T12:30:08.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:08.805+0000 7f166f257640 1 -- 172.21.15.116:0/314913176 wait complete. 2024-09-18T12:30:11.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:11.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:11.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:11.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:11.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:11.057+0000 7f76c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:11.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.920+0000 7fc2ecadd640 10 monclient: get_monmap_and_config 2024-09-18T12:30:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.920+0000 7fc2ecadd640 10 monclient: build_initial_monmap 2024-09-18T12:30:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.920+0000 7fc2ecadd640 10 monclient: monmap: 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:11.921365+0000 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:11.921365+0000 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:11.922 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:11.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 1 Processor -- start 2024-09-18T12:30:11.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 1 -- start start 2024-09-18T12:30:11.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient: init 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc2e809ad90 0x7fc2e809c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e809ad90 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2e809b160 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2e809b530 0x7fc2e8132070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e809b530 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e80692c0 con 0x7fc2e809b160 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e809b990 con 0x7fc2e809b530 2024-09-18T12:30:11.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e809db70 con 0x7fc2e809ad90 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2ecadd640 10 monclient(hunting): authenticate will time out at 1039.242799s 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2e6d9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fc2e809b530 msgr2=0x7fc2e8132070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.921+0000 7fc2e6d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2e809b530 0x7fc2e8132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.922+0000 7fc2e659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc2e809ad90 0x7fc2e809c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.922+0000 7fc2e659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc2e809ad90 0x7fc2e809c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:37024/0 (socket says 172.21.15.116:37024) 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.922+0000 7fc2e659c640 1 -- 172.21.15.116:0/2688230085 learned_addr learned my addr 172.21.15.116:0/2688230085 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.922+0000 7fc2e659c640 10 monclient(hunting): get_auth_request con 0x7fc2e809ad90 auth_method 0 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.922+0000 7fc2e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.922+0000 7fc2e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.922+0000 7fc2e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2ddd9b640 1 -- 172.21.15.116:0/2688230085 >> v1:172.21.15.144:6789/0 conn(0x7fc2e809b160 legacy=0x7fc2e8131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 10 monclient(hunting): handle_auth_done global_id 5452 payload 274 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 1 -- 172.21.15.116:0/2688230085 >> v2:172.21.15.116:3300/0 conn(0x7fc2e809b530 msgr2=0x7fc2e8132070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 1 --2- 172.21.15.116:0/2688230085 >> v2:172.21.15.116:3300/0 conn(0x7fc2e809b530 0x7fc2e8132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 1 -- 172.21.15.116:0/2688230085 >> v1:172.21.15.144:6789/0 conn(0x7fc2e809b160 legacy=0x7fc2e8131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 1 monclient: found mon.noname-c 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e659c640 1 -- 172.21.15.116:0/2688230085 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e809d5b0 con 0x7fc2e809ad90 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e5d9b640 1 -- 172.21.15.116:0/2688230085 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336756559 0 0) 0x7fc2e80692c0 con 0x7fc2e809b160 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.924+0000 7fc2e5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.927+0000 7fc2e659c640 1 -- 172.21.15.116:0/2688230085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809ad90 msgr2=0x7fc2e809c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.927+0000 7fc2e659c640 1 --2- 172.21.15.116:0/2688230085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809ad90 0x7fc2e809c5d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8068070 tx=0x7fc2d80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d865c0d19f18a12a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 1 -- 172.21.15.116:0/2688230085 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d802f4b0 con 0x7fc2e809ad90 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: dump: 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: _finish_auth 0 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 5 monclient: authenticate success, global_id 5452 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 1 -- 172.21.15.116:0/2688230085 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d802f610 con 0x7fc2e809ad90 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 1 -- 172.21.15.116:0/2688230085 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d802f9b0 con 0x7fc2e809ad90 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 4 set_mon_vals no callback set 2024-09-18T12:30:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2e5d9b640 10 monclient: dump: 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:11.933 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 10 monclient: shutdown 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 -- 172.21.15.116:0/2688230085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809ad90 msgr2=0x7fc2e809c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/2688230085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809ad90 0x7fc2e809c5d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8068070 tx=0x7fc2d80091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 -- 172.21.15.116:0/2688230085 shutdown_connections 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/2688230085 >> v2:172.21.15.116:3300/0 conn(0x7fc2e809b530 0x7fc2e8132070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/2688230085 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809ad90 0x7fc2e809c5d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 -- 172.21.15.116:0/2688230085 >> 172.21.15.116:0/2688230085 conn(0x7fc2e80678a0 msgr2=0x7fc2e80a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 -- 172.21.15.116:0/2688230085 shutdown_connections 2024-09-18T12:30:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.928+0000 7fc2ecadd640 1 -- 172.21.15.116:0/2688230085 wait complete. 2024-09-18T12:30:11.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.929+0000 7fc2ecadd640 10 monclient: build_initial_monmap 2024-09-18T12:30:11.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.929+0000 7fc2ecadd640 10 monclient: monmap: 2024-09-18T12:30:11.939 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:11.939 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:11.939 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:11.939 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.929+0000 7fc2ecadd640 1 Processor -- start 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.929+0000 7fc2ecadd640 1 -- start start 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.929+0000 7fc2ecadd640 10 monclient: init 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 0x7fc2e8132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e809ad90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809b160 0x7fc2e8132e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e809b160 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2e809b530 0x7fc2e808b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e809b530 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e808ec90 con 0x7fc2e809b530 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e808ea20 con 0x7fc2e809b160 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8134af0 con 0x7fc2e809ad90 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ecadd640 10 monclient(hunting): authenticate will time out at 1039.251164s 2024-09-18T12:30:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e6d9d640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2e809b530 msgr2=0x7fc2e808b240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2e809b530 0x7fc2e808b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 0x7fc2e8132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 0x7fc2e8132920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35980/0 (socket says 172.21.15.116:35980) 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 -- 172.21.15.116:0/379517768 learned_addr learned my addr 172.21.15.116:0/379517768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): get_auth_request con 0x7fc2e809ad90 auth_method 0 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ddd9b640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809b160 0x7fc2e8132e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:11.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ddd9b640 10 monclient(hunting): get_auth_request con 0x7fc2e809b160 auth_method 0 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient(hunting): handle_auth_done global_id 64331 payload 824 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 -- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809b160 msgr2=0x7fc2e8132e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809b160 0x7fc2e8132e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 -- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2e809b530 msgr2=0x7fc2e808b240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2e809b530 0x7fc2e808b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 monclient: found mon.noname-c 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:11.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.930+0000 7fc2e659c640 1 -- 172.21.15.116:0/379517768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8009450 con 0x7fc2e809ad90 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.932+0000 7fc2e659c640 1 -- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 msgr2=0x7fc2e8132920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.932+0000 7fc2e659c640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 0x7fc2e8132920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80099c0 tx=0x7fc2d8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 1 -- 172.21.15.116:0/379517768 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8032930 con 0x7fc2e809ad90 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: dump: 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:11.944 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 1 -- 172.21.15.116:0/379517768 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8032a90 con 0x7fc2e809ad90 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2ecadd640 5 monclient: authenticate success, global_id 64331 2024-09-18T12:30:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2ecadd640 10 monclient: _renew_subs 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2ecadd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e808cc60 con 0x7fc2e809ad90 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 1 -- 172.21.15.116:0/379517768 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8032e50 con 0x7fc2e809ad90 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 10 monclient: dump: 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2ecadd640 10 monclient: _renew_subs 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2ecadd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e81333a0 con 0x7fc2e809ad90 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2e559a640 4 set_mon_vals no callback set 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.933+0000 7fc2deffd640 1 -- 172.21.15.116:0/379517768 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc2d8039040 con 0x7fc2e809ad90 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2ecadd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2ecadd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2ecadd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e804bf60 con 0x7fc2e809ad90 2024-09-18T12:30:11.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2deffd640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2c8038440 0x7fc2c803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:11.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2deffd640 1 -- 172.21.15.116:0/379517768 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7fc2d806c380 con 0x7fc2e809ad90 2024-09-18T12:30:11.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2ddd9b640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2c8038440 0x7fc2c803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:11.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.934+0000 7fc2ddd9b640 10 monclient: get_auth_request con 0x7fc2c8038440 auth_method 0 2024-09-18T12:30:11.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.936+0000 7fc2ddd9b640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2c8038440 0x7fc2c803a8f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00014f0 tx=0x7fc2d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:11.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.938+0000 7fc2deffd640 1 -- 172.21.15.116:0/379517768 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc2d804c580 con 0x7fc2e809ad90 2024-09-18T12:30:11.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.938+0000 7fc2deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:11.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:11.938+0000 7fc2deffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:12.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:12.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:12.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:12.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.154+0000 7fc2ecadd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.154+0000 7fc2ecadd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.154+0000 7fc2ecadd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.154+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc2e80638d0 con 0x7fc2e809ad90 2024-09-18T12:30:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.155+0000 7fc2deffd640 1 -- 172.21.15.116:0/379517768 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fc2d803f620 con 0x7fc2e809ad90 2024-09-18T12:30:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.155+0000 7fc2deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.155+0000 7fc2deffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:12.156 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2c8038440 msgr2=0x7fc2c803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2c8038440 0x7fc2c803a8f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00014f0 tx=0x7fc2d0009040 comp rx=0 tx=0).stop 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 10 monclient: shutdown 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 msgr2=0x7fc2e8132920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 0x7fc2e8132920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80099c0 tx=0x7fc2d8032000 comp rx=0 tx=0).stop 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 shutdown_connections 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2c8038440 0x7fc2c803a8f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2e809b530 0x7fc2e808b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2e809b160 0x7fc2e8132e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 --2- 172.21.15.116:0/379517768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2e809ad90 0x7fc2e8132920 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:12.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.156+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 >> 172.21.15.116:0/379517768 conn(0x7fc2e80678a0 msgr2=0x7fc2e809f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.157+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 shutdown_connections 2024-09-18T12:30:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:12.157+0000 7fc2ecadd640 1 -- 172.21.15.116:0/379517768 wait complete. 2024-09-18T12:30:13.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:13.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:13.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:13.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:14.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:14.058+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:14.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:14.058+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:15.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:15.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:15.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:15.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:15.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.272+0000 7fe58ce80640 10 monclient: get_monmap_and_config 2024-09-18T12:30:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.272+0000 7fe58ce80640 10 monclient: build_initial_monmap 2024-09-18T12:30:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.272+0000 7fe58ce80640 10 monclient: monmap: 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:15.273266+0000 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:15.273266+0000 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:15.275 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 1 Processor -- start 2024-09-18T12:30:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 1 -- start start 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient: init 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe58809a510 0x7fe58812fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient(hunting): picked mon.noname-a con 0x7fe58809a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe58806e920 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5881349a0 0x7fe588136d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient(hunting): picked mon.noname-c con 0x7fe5881349a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58809cb70 con 0x7fe58806e920 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58809a8e0 con 0x7fe58809a510 2024-09-18T12:30:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58809bb40 con 0x7fe5881349a0 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58ce80640 10 monclient(hunting): authenticate will time out at 1042.594922s 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.273+0000 7fe58659c640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fe58809a510 msgr2=0x7fe58812fbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe58659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe58809a510 0x7fe58812fbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5881349a0 0x7fe588136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe5881349a0 0x7fe588136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:37042/0 (socket says 172.21.15.116:37042) 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 1 -- 172.21.15.116:0/2671138182 learned_addr learned my addr 172.21.15.116:0/2671138182 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 10 monclient(hunting): get_auth_request con 0x7fe5881349a0 auth_method 0 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.274+0000 7fe586d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe57ffff640 1 -- 172.21.15.116:0/2671138182 >> v1:172.21.15.144:6789/0 conn(0x7fe58806e920 legacy=0x7fe58806ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 10 monclient(hunting): handle_auth_done global_id 5461 payload 274 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 1 -- 172.21.15.116:0/2671138182 >> v2:172.21.15.116:3300/0 conn(0x7fe58809a510 msgr2=0x7fe58812fbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 1 --2- 172.21.15.116:0/2671138182 >> v2:172.21.15.116:3300/0 conn(0x7fe58809a510 0x7fe58812fbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 1 -- 172.21.15.116:0/2671138182 >> v1:172.21.15.144:6789/0 conn(0x7fe58806e920 legacy=0x7fe58806ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 1 monclient: found mon.noname-c 2024-09-18T12:30:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.276+0000 7fe586d9d640 1 -- 172.21.15.116:0/2671138182 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58809c890 con 0x7fe5881349a0 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.279+0000 7fe586d9d640 1 -- 172.21.15.116:0/2671138182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5881349a0 msgr2=0x7fe588136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.279+0000 7fe586d9d640 1 --2- 172.21.15.116:0/2671138182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5881349a0 0x7fe588136d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe58809b130 tx=0x7fe580009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcb5f6bda9df5baf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 1 -- 172.21.15.116:0/2671138182 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580007610 con 0x7fe5881349a0 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: dump: 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: _finish_auth 0 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 5 monclient: authenticate success, global_id 5461 2024-09-18T12:30:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 1 -- 172.21.15.116:0/2671138182 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58002d050 con 0x7fe5881349a0 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 1 -- 172.21.15.116:0/2671138182 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5800079e0 con 0x7fe5881349a0 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 4 set_mon_vals no callback set 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe585d9b640 10 monclient: dump: 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 10 monclient: shutdown 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:15.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 -- 172.21.15.116:0/2671138182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5881349a0 msgr2=0x7fe588136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 --2- 172.21.15.116:0/2671138182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5881349a0 0x7fe588136d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe58809b130 tx=0x7fe580009840 comp rx=0 tx=0).stop 2024-09-18T12:30:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 -- 172.21.15.116:0/2671138182 shutdown_connections 2024-09-18T12:30:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 --2- 172.21.15.116:0/2671138182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe5881349a0 0x7fe588136d80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 --2- 172.21.15.116:0/2671138182 >> v2:172.21.15.116:3300/0 conn(0x7fe58809a510 0x7fe58812fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 -- 172.21.15.116:0/2671138182 >> 172.21.15.116:0/2671138182 conn(0x7fe5880672b0 msgr2=0x7fe5880676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 -- 172.21.15.116:0/2671138182 shutdown_connections 2024-09-18T12:30:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.280+0000 7fe58ce80640 1 -- 172.21.15.116:0/2671138182 wait complete. 2024-09-18T12:30:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.281+0000 7fe58ce80640 10 monclient: build_initial_monmap 2024-09-18T12:30:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.281+0000 7fe58ce80640 10 monclient: monmap: 2024-09-18T12:30:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:15.291 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:15.291 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:15.291 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:15.291 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:15.291 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:15.291 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:15.292 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:15.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.281+0000 7fe58ce80640 1 Processor -- start 2024-09-18T12:30:15.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.281+0000 7fe58ce80640 1 -- start start 2024-09-18T12:30:15.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.281+0000 7fe58ce80640 10 monclient: init 2024-09-18T12:30:15.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:15.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:15.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe58806e920 0x7fe58806de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:15.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): picked mon.noname-a con 0x7fe58806e920 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:15.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58809a510 0x7fe5880983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:15.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): picked mon.noname-b con 0x7fe58809a510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 0x7fe5880988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): picked mon.noname-c con 0x7fe5881349a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58806f5f0 con 0x7fe58806e920 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58659c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe58806e920 msgr2=0x7fe58806de60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5880a0020 con 0x7fe58809a510 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58806d480 con 0x7fe5881349a0 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58ce80640 10 monclient(hunting): authenticate will time out at 1042.603290s 2024-09-18T12:30:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe58659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe58806e920 0x7fe58806de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 0x7fe5880988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 0x7fe5880988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35992/0 (socket says 172.21.15.116:35992) 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 -- 172.21.15.116:0/175045864 learned_addr learned my addr 172.21.15.116:0/175045864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): get_auth_request con 0x7fe5881349a0 auth_method 0 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58809a510 0x7fe5880983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 10 monclient(hunting): get_auth_request con 0x7fe58809a510 auth_method 0 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe57ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient(hunting): handle_auth_done global_id 64334 payload 824 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 -- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58809a510 msgr2=0x7fe5880983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58809a510 0x7fe5880983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 -- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe58806e920 msgr2=0x7fe58806de60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe58806e920 0x7fe58806de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 monclient: found mon.noname-c 2024-09-18T12:30:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.282+0000 7fe586d9d640 1 -- 172.21.15.116:0/175045864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe580009ae0 con 0x7fe5881349a0 2024-09-18T12:30:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe586d9d640 1 -- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 msgr2=0x7fe5880988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe586d9d640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 0x7fe5880988f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe580009880 tx=0x7fe580034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 1 -- 172.21.15.116:0/175045864 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580031c50 con 0x7fe5881349a0 2024-09-18T12:30:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: dump: 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 1 -- 172.21.15.116:0/175045864 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe580031db0 con 0x7fe5881349a0 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58ce80640 5 monclient: authenticate success, global_id 64334 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58ce80640 10 monclient: _renew_subs 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58ce80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58806d1a0 con 0x7fe5881349a0 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 1 -- 172.21.15.116:0/175045864 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58003c650 con 0x7fe5881349a0 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe57e7fc640 10 monclient: dump: 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58ce80640 10 monclient: _renew_subs 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58559a640 4 set_mon_vals no callback set 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58ce80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.285+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58806e3a0 con 0x7fe5881349a0 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe57e7fc640 1 -- 172.21.15.116:0/175045864 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe580003020 con 0x7fe5881349a0 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe57e7fc640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038440 0x7fe56803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe57e7fc640 1 -- 172.21.15.116:0/175045864 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7fe58006c980 con 0x7fe5881349a0 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe58ce80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe58ce80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe58ce80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58804bf60 con 0x7fe5881349a0 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe58659c640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038440 0x7fe56803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.286+0000 7fe58659c640 10 monclient: get_auth_request con 0x7fe568038440 auth_method 0 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.289+0000 7fe58659c640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038440 0x7fe56803a8f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe578003940 tx=0x7fe578006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.291+0000 7fe57e7fc640 1 -- 172.21.15.116:0/175045864 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe580040080 con 0x7fe5881349a0 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.291+0000 7fe57e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.291+0000 7fe57e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:15.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.505+0000 7fe58ce80640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:15.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.505+0000 7fe58ce80640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:15.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.505+0000 7fe58ce80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:15.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.505+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe5880638d0 con 0x7fe5881349a0 2024-09-18T12:30:15.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.505+0000 7fe57e7fc640 1 -- 172.21.15.116:0/175045864 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fe580036020 con 0x7fe5881349a0 2024-09-18T12:30:15.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.505+0000 7fe57e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:15.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.505+0000 7fe57e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:15.507 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:15.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038440 msgr2=0x7fe56803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038440 0x7fe56803a8f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe578003940 tx=0x7fe578006d80 comp rx=0 tx=0).stop 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 10 monclient: shutdown 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 msgr2=0x7fe5880988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 0x7fe5880988f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe580009880 tx=0x7fe580034000 comp rx=0 tx=0).stop 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 shutdown_connections 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038440 0x7fe56803a8f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5881349a0 0x7fe5880988f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe58809a510 0x7fe5880983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 --2- 172.21.15.116:0/175045864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe58806e920 0x7fe58806de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 >> 172.21.15.116:0/175045864 conn(0x7fe5880672b0 msgr2=0x7fe588134f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 shutdown_connections 2024-09-18T12:30:15.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:15.507+0000 7fe58ce80640 1 -- 172.21.15.116:0/175045864 wait complete. 2024-09-18T12:30:16.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:16.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:16.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:16.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:17.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:17.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:17.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:17.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:18.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:18.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:18.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:18.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:18.520 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.621+0000 7f1a3475d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.621+0000 7f1a3475d640 10 monclient: build_initial_monmap 2024-09-18T12:30:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.621+0000 7f1a3475d640 10 monclient: monmap: 2024-09-18T12:30:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:18.623105+0000 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:18.623105+0000 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:18.624 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:18.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 1 Processor -- start 2024-09-18T12:30:18.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 1 -- start start 2024-09-18T12:30:18.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient: init 2024-09-18T12:30:18.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:18.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T12:30:18.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a2c096c60 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 0x7f1a2c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2c097030 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2c133b40 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c09d2e0 con 0x7f1a2c096c60 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c09d6b0 con 0x7f1a2c097030 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0992f0 con 0x7f1a2c133b40 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a31cf8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 msgr2=0x7f1a2c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a324f9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1a2c096c60 legacy=0x7f1a2c09a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a3475d640 10 monclient(hunting): authenticate will time out at 1045.944865s 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.622+0000 7f1a31cf8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 0x7f1a2c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:43814/0 (socket says 172.21.15.116:43814) 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 learned_addr learned my addr 172.21.15.116:0/3628661875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request con 0x7f1a2c133b40 auth_method 0 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.623+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.823+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 msgr2=0x7f1a2c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.823+0000 7f1a324f9640 1 -- 172.21.15.116:0/3628661875 >> v1:172.21.15.116:6789/0 conn(0x7f1a2c096c60 legacy=0x7f1a2c09a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.823+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 0x7f1a2c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request con 0x7f1a2c133b40 auth_method 0 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:18.824+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:19.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:19.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.224+0000 7f1a324f9640 1 -- 172.21.15.116:0/3628661875 >> v1:172.21.15.116:6789/0 conn(0x7f1a2c096c60 legacy=0x7f1a2c09a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.224+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 msgr2=0x7f1a2c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.224+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 0x7f1a2c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request con 0x7f1a2c133b40 auth_method 0 2024-09-18T12:30:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:19.225+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.025+0000 7f1a324f9640 1 -- 172.21.15.116:0/3628661875 >> v1:172.21.15.116:6789/0 conn(0x7f1a2c096c60 legacy=0x7f1a2c09a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.025+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 msgr2=0x7f1a2c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.025+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 0x7f1a2c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:30:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request con 0x7f1a2c133b40 auth_method 0 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:20.027+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:30:20.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:20.057+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:20.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:20.057+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:21.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:21.058+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:21.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:21.058+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:21.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:21.058+0000 7f76c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 10 monclient(hunting): tick 2024-09-18T12:30:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 monclient(hunting): continuing hunt 2024-09-18T12:30:21.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 msgr2=0x7f1a2c135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.144:3300/0 conn(0x7f1a2c133b40 0x7f1a2c135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 -- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 msgr2=0x7f1a2c097430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 --2- 172.21.15.116:0/3628661875 >> v2:172.21.15.116:3300/0 conn(0x7f1a2c097030 0x7f1a2c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 -- 172.21.15.116:0/3628661875 >> v1:172.21.15.116:6789/0 conn(0x7f1a2c096c60 legacy=0x7f1a2c09a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 10 monclient: _add_conns ranks=[1,2] 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a10003630 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1a10007310 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 -- 172.21.15.116:0/3628661875 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0995d0 con 0x7f1a10003630 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:21.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 1 -- 172.21.15.116:0/3628661875 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c09d6b0 con 0x7f1a10007310 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a23fff640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request con 0x7f1a10007310 auth_method 0 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 msgr2=0x7f1a100097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 msgr2=0x7f1a100097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:21.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.623+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:21.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.625+0000 7f1a324f9640 1 -- 172.21.15.116:0/3628661875 >> v1:172.21.15.144:6789/0 conn(0x7f1a10003630 legacy=0x7f1a10005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request con 0x7f1a10007310 auth_method 0 2024-09-18T12:30:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:21.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 msgr2=0x7f1a100097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 msgr2=0x7f1a100097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:21.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:21.824+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:22.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:22.058+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:22.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:22.058+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request con 0x7f1a10007310 auth_method 0 2024-09-18T12:30:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 msgr2=0x7f1a100097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 msgr2=0x7f1a100097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.225+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.883+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742905950 0 0) 0x7f1a2c0995d0 con 0x7f1a10003630 2024-09-18T12:30:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.883+0000 7f1a314f7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.883+0000 7f1a314f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.883+0000 7f1a314f7640 10 monclient(hunting): my global_id is 5467 2024-09-18T12:30:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.883+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a180034e0 con 0x7f1a10003630 2024-09-18T12:30:22.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.884+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63357540 0 0) 0x7f1a180034e0 con 0x7f1a10003630 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 msgr2=0x7f1a100097c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a10007310 0x7f1a100097c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: _finish_hunting 0 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c09d2e0 con 0x7f1a10003630 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: _finish_auth 0 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f1a28004270 con 0x7f1a10003630 2024-09-18T12:30:22.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: dump: 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:22.891 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 >> v1:172.21.15.144:6789/0 conn(0x7f1a10003630 legacy=0x7f1a10005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 reap_dead start 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): picked mon.b con 0x7f1a180039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a18004780 0x7f1a18004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): picked mon.c con 0x7f1a18004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a180051a0 0x7f1a1800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:22.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): picked mon.a con 0x7f1a180051a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c09d6b0 con 0x7f1a180051a0 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1800c170 con 0x7f1a180039d0 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1800e540 con 0x7f1a18004780 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a314f7640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a180051a0 msgr2=0x7f1a1800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a180051a0 0x7f1a1800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a18004780 0x7f1a18004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:22.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request con 0x7f1a18004780 auth_method 0 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 10 monclient(hunting): get_auth_request con 0x7f1a180039d0 auth_method 0 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:22.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient(hunting): handle_auth_done global_id 5467 payload 293 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 msgr2=0x7f1a18003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a18003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a180051a0 msgr2=0x7f1a1800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a180051a0 0x7f1a1800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient: _finish_hunting 0 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 1 monclient: found mon.c 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180034e0 con 0x7f1a18004780 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 10 monclient: _finish_auth 0 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a31cf8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:22.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a3475d640 5 monclient: authenticate success, global_id 5467 2024-09-18T12:30:22.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a324f9640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a18003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:22.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.885+0000 7f1a3475d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a31cf8640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a18004780 msgr2=0x7f1a18004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a31cf8640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a18004780 0x7f1a18004bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c003940 tx=0x7f1a1c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af001899a2c7a5d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c02b950 con 0x7f1a18004780 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a314f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a314f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a314f7640 10 monclient: dump: 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a3475d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c02bab0 con 0x7f1a18004780 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.888+0000 7f1a314f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 4 set_mon_vals no callback set 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a314f7640 1 -- 172.21.15.116:0/3628661875 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c02be50 con 0x7f1a18004780 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a314f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a314f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a314f7640 10 monclient: dump: 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 10 monclient: shutdown 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 1 -- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a18004780 msgr2=0x7f1a18004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a18004780 0x7f1a18004bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c003940 tx=0x7f1a1c00b040 comp rx=0 tx=0).stop 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 1 -- 172.21.15.116:0/3628661875 shutdown_connections 2024-09-18T12:30:22.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a180051a0 0x7f1a1800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a18004780 0x7f1a18004bf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3628661875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a18003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.889+0000 7f1a3475d640 1 -- 172.21.15.116:0/3628661875 >> 172.21.15.116:0/3628661875 conn(0x7f1a2c0672b0 msgr2=0x7f1a2c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.890+0000 7f1a3475d640 1 -- 172.21.15.116:0/3628661875 shutdown_connections 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.890+0000 7f1a3475d640 1 -- 172.21.15.116:0/3628661875 wait complete. 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.890+0000 7f1a3475d640 10 monclient: build_initial_monmap 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.890+0000 7f1a3475d640 10 monclient: monmap: 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.890+0000 7f1a3475d640 1 Processor -- start 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 1 -- start start 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient: init 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a2c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a180039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a18004780 0x7f1a2c06e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a18004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 0x7f1a2c06eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a180051a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c09add0 con 0x7f1a18004780 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c132fa0 con 0x7f1a180039d0 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c119530 con 0x7f1a180051a0 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a3475d640 10 monclient(hunting): authenticate will time out at 1050.213526s 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a31cf8640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a18004780 msgr2=0x7f1a2c06e600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a31cf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a18004780 0x7f1a2c06e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 0x7f1a2c06eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 0x7f1a2c06eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39460/0 (socket says 172.21.15.116:39460) 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3835776242 learned_addr learned my addr 172.21.15.116:0/3835776242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request con 0x7f1a180051a0 auth_method 0 2024-09-18T12:30:22.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a2c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 10 monclient(hunting): get_auth_request con 0x7f1a180039d0 auth_method 0 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a32cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.891+0000 7f1a324f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 10 monclient(hunting): handle_auth_done global_id 64337 payload 824 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 msgr2=0x7f1a2c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a2c06e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a18004780 msgr2=0x7f1a2c06e600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a18004780 0x7f1a2c06e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 10 monclient: _finish_hunting 0 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 1 monclient: found mon.noname-c 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.892+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3835776242 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c1182f0 con 0x7f1a180051a0 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a32cfa640 1 -- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 msgr2=0x7f1a2c06eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a32cfa640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 0x7f1a2c06eb40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1a100072e0 tx=0x7f1a24009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 1 -- 172.21.15.116:0/3835776242 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400b2e0 con 0x7f1a180051a0 2024-09-18T12:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: dump: 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 1 -- 172.21.15.116:0/3835776242 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2400f040 con 0x7f1a180051a0 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a3475d640 5 monclient: authenticate success, global_id 64337 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 1 -- 172.21.15.116:0/3835776242 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24009720 con 0x7f1a180051a0 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a3475d640 10 monclient: _renew_subs 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a3475d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c119250 con 0x7f1a180051a0 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a227fc640 10 monclient: dump: 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a30cf6640 4 set_mon_vals no callback set 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a3475d640 10 monclient: _renew_subs 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a3475d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.894+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c12f2d0 con 0x7f1a180051a0 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.895+0000 7f1a227fc640 1 -- 172.21.15.116:0/3835776242 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1a24009ad0 con 0x7f1a180051a0 2024-09-18T12:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.895+0000 7f1a3475d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.895+0000 7f1a3475d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.895+0000 7f1a3475d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.895+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c1196c0 con 0x7f1a180051a0 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.896+0000 7f1a227fc640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1a10038a70 0x7f1a1003af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.896+0000 7f1a227fc640 1 -- 172.21.15.116:0/3835776242 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5726+0+0 (secure 0 0 0) 0x7f1a2404bba0 con 0x7f1a180051a0 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.896+0000 7f1a324f9640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1a10038a70 0x7f1a1003af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.896+0000 7f1a324f9640 10 monclient: get_auth_request con 0x7f1a10038a70 auth_method 0 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.898+0000 7f1a324f9640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1a10038a70 0x7f1a1003af20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1a280043a0 tx=0x7f1a28007b40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.900+0000 7f1a227fc640 1 -- 172.21.15.116:0/3835776242 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1a24026530 con 0x7f1a180051a0 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.900+0000 7f1a227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:22.900+0000 7f1a227fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:22.942+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v107) v1 ==== 58+0+3 (secure 0 0 0) 0x7f76c4016070 con 0x7f76cc131b40 2024-09-18T12:30:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:22.942+0000 7f76c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T12:30:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:22.942+0000 7f76c27fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:23.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.051+0000 7f76d0f9d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:30:23.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.051+0000 7f76d0f9d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T12:30:23.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.051+0000 7f76d0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:23.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.051+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f76cc072640 con 0x7f76cc131b40 2024-09-18T12:30:23.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.058+0000 7f76c17fa640 10 monclient: tick 2024-09-18T12:30:23.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.058+0000 7f76c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:23.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.125+0000 7f1a3475d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:23.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.126+0000 7f1a3475d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:23.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.126+0000 7f1a3475d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.126+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1a2c04bf60 con 0x7f1a180051a0 2024-09-18T12:30:23.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.126+0000 7f1a227fc640 1 -- 172.21.15.116:0/3835776242 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f1a2401de80 con 0x7f1a180051a0 2024-09-18T12:30:23.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.126+0000 7f1a227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:23.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.126+0000 7f1a227fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:23.128 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:23.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1a10038a70 msgr2=0x7f1a1003af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1a10038a70 0x7f1a1003af20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1a280043a0 tx=0x7f1a28007b40 comp rx=0 tx=0).stop 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 10 monclient: shutdown 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 msgr2=0x7f1a2c06eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 0x7f1a2c06eb40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1a100072e0 tx=0x7f1a24009040 comp rx=0 tx=0).stop 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 shutdown_connections 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1a10038a70 0x7f1a1003af20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1a180051a0 0x7f1a2c06eb40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a18004780 0x7f1a2c06e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 --2- 172.21.15.116:0/3835776242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1a180039d0 0x7f1a2c06e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 >> 172.21.15.116:0/3835776242 conn(0x7f1a2c0672b0 msgr2=0x7f1a2c1326a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 shutdown_connections 2024-09-18T12:30:23.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.128+0000 7f1a3475d640 1 -- 172.21.15.116:0/3835776242 wait complete. 2024-09-18T12:30:23.140 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:30:23.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:30:23.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:30:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.232+0000 7f32918ef640 10 monclient: get_monmap_and_config 2024-09-18T12:30:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.232+0000 7f32918ef640 10 monclient: build_initial_monmap 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.232+0000 7f32918ef640 10 monclient: monmap: 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:23.233892+0000 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:23.233892+0000 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:23.235 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.232+0000 7f32918ef640 1 Processor -- start 2024-09-18T12:30:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.232+0000 7f32918ef640 1 -- start start 2024-09-18T12:30:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.232+0000 7f32918ef640 10 monclient: init 2024-09-18T12:30:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f328c1318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f328c09ea50 0x7f328c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient(hunting): picked mon.noname-c con 0x7f328c09ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 0x7f328c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient(hunting): picked mon.noname-b con 0x7f328c136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c132640 con 0x7f328c1318f0 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c131cc0 con 0x7f328c09ea50 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c09fbf0 con 0x7f328c136c20 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32918ef640 10 monclient(hunting): authenticate will time out at 1050.555354s 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 0x7f328c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 0x7f328c139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39466/0 (socket says 172.21.15.116:39466) 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 -- 172.21.15.116:0/724180983 learned_addr learned my addr 172.21.15.116:0/724180983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): get_auth_request con 0x7f328c136c20 auth_method 0 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32827fc640 1 --2- 172.21.15.116:0/724180983 >> v2:172.21.15.144:3300/0 conn(0x7f328c09ea50 0x7f328c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32827fc640 10 monclient(hunting): get_auth_request con 0x7f328c09ea50 auth_method 0 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32827fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f32827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient(hunting): handle_auth_done global_id 64340 payload 274 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 -- 172.21.15.116:0/724180983 >> v2:172.21.15.144:3300/0 conn(0x7f328c09ea50 msgr2=0x7f328c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 --2- 172.21.15.116:0/724180983 >> v2:172.21.15.144:3300/0 conn(0x7f328c09ea50 0x7f328c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 -- 172.21.15.116:0/724180983 >> v1:172.21.15.144:6789/0 conn(0x7f328c1318f0 legacy=0x7f328c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.233+0000 7f328b7fe640 1 -- 172.21.15.116:0/724180983 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c09fe30 con 0x7f328c136c20 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328b7fe640 1 -- 172.21.15.116:0/724180983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 msgr2=0x7f328c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328b7fe640 1 --2- 172.21.15.116:0/724180983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 0x7f328c139000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3284009910 tx=0x7f32840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79072770a541fe0e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 1 -- 172.21.15.116:0/724180983 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328402ae90 con 0x7f328c136c20 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: dump: 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f32918ef640 5 monclient: authenticate success, global_id 64340 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f32918ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 1 -- 172.21.15.116:0/724180983 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3284032440 con 0x7f328c136c20 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f32918ef640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f32918ef640 4 set_mon_vals no callback set 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 1 -- 172.21.15.116:0/724180983 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840327e0 con 0x7f328c136c20 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.236+0000 7f328a7fc640 10 monclient: dump: 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 10 monclient: shutdown 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 -- 172.21.15.116:0/724180983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 msgr2=0x7f328c139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 --2- 172.21.15.116:0/724180983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 0x7f328c139000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3284009910 tx=0x7f32840091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 -- 172.21.15.116:0/724180983 shutdown_connections 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 --2- 172.21.15.116:0/724180983 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c136c20 0x7f328c139000 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 --2- 172.21.15.116:0/724180983 >> v2:172.21.15.144:3300/0 conn(0x7f328c09ea50 0x7f328c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 -- 172.21.15.116:0/724180983 >> 172.21.15.116:0/724180983 conn(0x7f328c067110 msgr2=0x7f328c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 -- 172.21.15.116:0/724180983 shutdown_connections 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 1 -- 172.21.15.116:0/724180983 wait complete. 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 10 monclient: build_initial_monmap 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.237+0000 7f32918ef640 10 monclient: monmap: 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 Processor -- start 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 -- start start 2024-09-18T12:30:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient: init 2024-09-18T12:30:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 0x7f328c1362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): picked mon.noname-c con 0x7f328c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f328c1318f0 0x7f328c12f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): picked mon.noname-a con 0x7f328c1318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328c136c20 0x7f328c12f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): picked mon.noname-b con 0x7f328c136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c133610 con 0x7f328c1318f0 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c133330 con 0x7f328c136c20 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c1338f0 con 0x7f328c09ea50 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32918ef640 10 monclient(hunting): authenticate will time out at 1050.560753s 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32827fc640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f328c1318f0 msgr2=0x7f328c12f180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f32827fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f328c1318f0 0x7f328c12f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 0x7f328c1362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 0x7f328c1362f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39478/0 (socket says 172.21.15.116:39478) 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 1 -- 172.21.15.116:0/401113406 learned_addr learned my addr 172.21.15.116:0/401113406 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 10 monclient(hunting): get_auth_request con 0x7f328c09ea50 auth_method 0 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.238+0000 7f328affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328b7fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328c136c20 0x7f328c12f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 10 monclient(hunting): handle_auth_done global_id 64343 payload 824 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328c136c20 msgr2=0x7f328c12f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f328c136c20 0x7f328c12f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f328c1318f0 msgr2=0x7f328c12f180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f328c1318f0 0x7f328c12f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 1 monclient: found mon.noname-c 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.239+0000 7f328affd640 1 -- 172.21.15.116:0/401113406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284009450 con 0x7f328c09ea50 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f328affd640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 msgr2=0x7f328c1362f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f328affd640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 0x7f328c1362f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c8b0 tx=0x7f327c0026c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00cea0 con 0x7f328c09ea50 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: dump: 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c00aa60 con 0x7f328c09ea50 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32918ef640 5 monclient: authenticate success, global_id 64343 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32918ef640 10 monclient: _renew_subs 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00ae00 con 0x7f328c09ea50 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32918ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c130ab0 con 0x7f328c09ea50 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32837fe640 10 monclient: dump: 2024-09-18T12:30:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f3289ffb640 4 set_mon_vals no callback set 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32918ef640 10 monclient: _renew_subs 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32918ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.241+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c135830 con 0x7f328c09ea50 2024-09-18T12:30:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32918ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f327c012670 con 0x7f328c09ea50 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32918ef640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 0x7f328c130030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32918ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c065e00 con 0x7f328c12fc60 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 0x7f328c130030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient: get_auth_request con 0x7f328c12fc60 auth_method 0 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32837fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f326c038490 0x7f326c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5841+0+0 (secure 0 0 0) 0x7f327c04c140 con 0x7f328c09ea50 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 10 monclient(hunting): handle_auth_done global_id 64346 payload 824 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f32827fc640 1 -- 172.21.15.116:0/401113406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f328c130570 con 0x7f328c12fc60 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f328b7fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f326c038490 0x7f326c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.242+0000 7f328b7fe640 10 monclient: get_auth_request con 0x7f326c038490 auth_method 0 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.244+0000 7f32827fc640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 msgr2=0x7f328c130030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.244+0000 7f32827fc640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 0x7f328c130030 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f328c132cd0 tx=0x7f3274008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400b810 con 0x7f328c12fc60 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 10 monclient: dump: 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.259 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f328b7fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f326c038490 0x7f326c03a940 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f328c076600 tx=0x7f3284032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f327400b970 con 0x7f328c12fc60 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 msgr2=0x7f328c130030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 0x7f328c130030 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f328c132cd0 tx=0x7f3274008040 comp rx=0 tx=0).stop 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 msgr2=0x7f328c130030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:23.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.245+0000 7f32837fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c12fc60 0x7f328c130030 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.250+0000 7f32918ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:23.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.250+0000 7f32918ef640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 0x7f328c09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.250+0000 7f32918ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.250+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c133780 con 0x7f328c09d5b0 2024-09-18T12:30:23.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 0x7f328c09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient: get_auth_request con 0x7f328c09d5b0 auth_method 0 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 10 monclient(hunting): handle_auth_done global_id 64349 payload 824 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.251+0000 7f32827fc640 1 -- 172.21.15.116:0/401113406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f328c1334a0 con 0x7f328c09d5b0 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32827fc640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 msgr2=0x7f328c09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32827fc640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 0x7f328c09d9b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f328c132cd0 tx=0x7f327401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274014ea0 con 0x7f328c09d5b0 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 10 monclient: dump: 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f327400b600 con 0x7f328c09d5b0 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 msgr2=0x7f328c09d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 0x7f328c09d9b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f328c132cd0 tx=0x7f327401a000 comp rx=0 tx=0).stop 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 msgr2=0x7f328c09d9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.253+0000 7f32837fe640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09d5b0 0x7f328c09d9b0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f328c132cd0 tx=0x7f327401a000 comp rx=0 tx=0).stop 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 58, 2024-09-18T12:30:23.265 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:23.266 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:23.269 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:23.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:23.271 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:23.277 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:23.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f326c038490 msgr2=0x7f326c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f326c038490 0x7f326c03a940 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f328c076600 tx=0x7f3284032000 comp rx=0 tx=0).stop 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 10 monclient: shutdown 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 msgr2=0x7f328c1362f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f328b7fe640 1 -- 172.21.15.116:0/401113406 reap_dead start 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 0x7f328c1362f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c8b0 tx=0x7f327c0026c0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 shutdown_connections 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 --2- 172.21.15.116:0/401113406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f328c09ea50 0x7f328c1362f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 >> 172.21.15.116:0/401113406 conn(0x7f328c067110 msgr2=0x7f328c12cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 shutdown_connections 2024-09-18T12:30:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.254+0000 7f32918ef640 1 -- 172.21.15.116:0/401113406 wait complete. 2024-09-18T12:30:23.279 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:30:23.280 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:30:23.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:30:23.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.372+0000 7f7afa79d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:23.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.372+0000 7f7afa79d640 10 monclient: build_initial_monmap 2024-09-18T12:30:23.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.372+0000 7f7afa79d640 10 monclient: monmap: 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:23.374150+0000 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:23.374150+0000 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:23.375 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:23.376 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:23.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 1 Processor -- start 2024-09-18T12:30:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 1 -- start start 2024-09-18T12:30:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient: init 2024-09-18T12:30:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:30:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7af409a510 0x7f7af4096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient(hunting): picked mon.noname-c con 0x7f7af409a510 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7af40971a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7af40975a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af409cb30 con 0x7f7af40971a0 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af409a900 con 0x7f7af40975a0 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af406ed80 con 0x7f7af409a510 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7afa79d640 10 monclient(hunting): authenticate will time out at 1050.695900s 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.373+0000 7f7aebfff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f7af40971a0 legacy=0x7f7af412f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7af409a510 0x7f7af4096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7af409a510 0x7f7af4096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:43900/0 (socket says 172.21.15.116:43900) 2024-09-18T12:30:23.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af8d3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7af40975a0 0x7f7af412fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:53760/0 (socket says 172.21.15.116:53760) 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 1 -- 172.21.15.116:0/2024579232 learned_addr learned my addr 172.21.15.116:0/2024579232 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 10 monclient(hunting): get_auth_request con 0x7f7af409a510 auth_method 0 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.374+0000 7f7af3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af8d3a640 1 -- 172.21.15.116:0/2024579232 >> v1:172.21.15.144:6789/0 conn(0x7f7af40975a0 legacy=0x7f7af412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 10 monclient(hunting): handle_auth_done global_id 5476 payload 274 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 1 -- 172.21.15.116:0/2024579232 >> v1:172.21.15.144:6789/0 conn(0x7f7af40975a0 legacy=0x7f7af412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 1 -- 172.21.15.116:0/2024579232 >> v1:172.21.15.116:6789/0 conn(0x7f7af40971a0 legacy=0x7f7af412f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:23.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:23.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 1 monclient: found mon.noname-c 2024-09-18T12:30:23.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:23.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:23.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.376+0000 7f7af3fff640 1 -- 172.21.15.116:0/2024579232 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af409c820 con 0x7f7af409a510 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af3fff640 1 -- 172.21.15.116:0/2024579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 msgr2=0x7f7af4096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af3fff640 1 --2- 172.21.15.116:0/2024579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 0x7f7af4096c60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40099c0 tx=0x7f7ae40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10123024f3e52bd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 1 -- 172.21.15.116:0/2024579232 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae402ae90 con 0x7f7af409a510 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: dump: 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.386 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 1 -- 172.21.15.116:0/2024579232 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae402f4b0 con 0x7f7af409a510 2024-09-18T12:30:23.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7afa79d640 5 monclient: authenticate success, global_id 5476 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7afa79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7afa79d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 1 -- 172.21.15.116:0/2024579232 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae402f890 con 0x7f7af409a510 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7afa79d640 4 set_mon_vals no callback set 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.379+0000 7f7af37fe640 10 monclient: dump: 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 10 monclient: shutdown 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 1 -- 172.21.15.116:0/2024579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 msgr2=0x7f7af4096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 1 --2- 172.21.15.116:0/2024579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 0x7f7af4096c60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40099c0 tx=0x7f7ae40091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 1 -- 172.21.15.116:0/2024579232 shutdown_connections 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 1 --2- 172.21.15.116:0/2024579232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 0x7f7af4096c60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 1 -- 172.21.15.116:0/2024579232 >> 172.21.15.116:0/2024579232 conn(0x7f7af40672b0 msgr2=0x7f7af40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 1 -- 172.21.15.116:0/2024579232 shutdown_connections 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 1 -- 172.21.15.116:0/2024579232 wait complete. 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 10 monclient: build_initial_monmap 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.380+0000 7f7afa79d640 10 monclient: monmap: 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 Processor -- start 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 -- start start 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient: init 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7af40971a0 0x7f7af4141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): picked mon.noname-c con 0x7f7af40971a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af40975a0 0x7f7af412eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): picked mon.noname-a con 0x7f7af40975a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 0x7f7af412f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): picked mon.noname-b con 0x7f7af409a510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af409c370 con 0x7f7af40975a0 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af412f7a0 con 0x7f7af409a510 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af411e4c0 con 0x7f7af40971a0 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7afa79d640 10 monclient(hunting): authenticate will time out at 1050.703769s 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7aebfff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af40975a0 msgr2=0x7f7af412eac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7aebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af40975a0 0x7f7af412eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7af40971a0 0x7f7af4141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7af40971a0 0x7f7af4141060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39516/0 (socket says 172.21.15.116:39516) 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 1 -- 172.21.15.116:0/2944416962 learned_addr learned my addr 172.21.15.116:0/2944416962 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 10 monclient(hunting): get_auth_request con 0x7f7af40971a0 auth_method 0 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.381+0000 7f7af3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af8d3a640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 0x7f7af412f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af8d3a640 10 monclient(hunting): get_auth_request con 0x7f7af409a510 auth_method 0 2024-09-18T12:30:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af8d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af8d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af8d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 10 monclient(hunting): handle_auth_done global_id 64352 payload 824 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 msgr2=0x7f7af412f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af409a510 0x7f7af412f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af40975a0 msgr2=0x7f7af412eac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af40975a0 0x7f7af412eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 1 monclient: found mon.noname-c 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.382+0000 7f7af3fff640 1 -- 172.21.15.116:0/2944416962 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae4009450 con 0x7f7af40971a0 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.384+0000 7f7af3fff640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7af40971a0 msgr2=0x7f7af4141060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.384+0000 7f7af3fff640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7af40971a0 0x7f7af4141060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4035040 tx=0x7f7ae4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.384+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:30:23.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae40329b0 con 0x7f7af40971a0 2024-09-18T12:30:23.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: dump: 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.398 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae402ae90 con 0x7f7af40971a0 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4032d80 con 0x7f7af40971a0 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7afa79d640 5 monclient: authenticate success, global_id 64352 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7afa79d640 10 monclient: _renew_subs 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7afa79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af411dc20 con 0x7f7af40971a0 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af2ffd640 4 set_mon_vals no callback set 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7af0ff9640 10 monclient: dump: 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7afa79d640 10 monclient: _renew_subs 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7afa79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.386+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af4099f40 con 0x7f7af40971a0 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7afa79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7ae4039040 con 0x7f7af40971a0 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7afa79d640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 0x7f7af4098380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7afa79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af4065f20 con 0x7f7af41415a0 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7af0ff9640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ad4038490 0x7f7ad403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5841+0+0 (secure 0 0 0) 0x7f7ae406bd60 con 0x7f7af40971a0 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7aebfff640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 0x7f7af4098380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7aebfff640 10 monclient: get_auth_request con 0x7f7af41415a0 auth_method 0 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7aebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7aebfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.387+0000 7f7af8d3a640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ad4038490 0x7f7ad403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.388+0000 7f7af8d3a640 10 monclient: get_auth_request con 0x7f7ad4038490 auth_method 0 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.388+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.388+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.388+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.388+0000 7f7aebfff640 10 monclient(hunting): handle_auth_done global_id 5485 payload 824 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.388+0000 7f7aebfff640 1 -- 172.21.15.116:0/2944416962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7af411c950 con 0x7f7af41415a0 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af8d3a640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ad4038490 0x7f7ad403a940 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7aec005fd0 tx=0x7f7aec007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7aebfff640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 msgr2=0x7f7af4098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7aebfff640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 0x7f7af4098380 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7adc003940 tx=0x7f7adc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0050f0 con 0x7f7af41415a0 2024-09-18T12:30:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 10 monclient: dump: 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7adc005250 con 0x7f7af41415a0 2024-09-18T12:30:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 msgr2=0x7f7af4098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 0x7f7af4098380 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7adc003940 tx=0x7f7adc0091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 msgr2=0x7f7af4098380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.390+0000 7f7af0ff9640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af41415a0 0x7f7af4098380 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7afa79d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7afa79d640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 0x7f7af4098cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7afa79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af4099200 con 0x7f7af40988c0 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 0x7f7af4098cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 10 monclient: get_auth_request con 0x7f7af40988c0 auth_method 0 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.396+0000 7f7aebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.397+0000 7f7aebfff640 10 monclient(hunting): handle_auth_done global_id 5488 payload 824 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.397+0000 7f7aebfff640 1 -- 172.21.15.116:0/2944416962 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7af404bf60 con 0x7f7af40988c0 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7aebfff640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 msgr2=0x7f7af4098cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7aebfff640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 0x7f7af4098cc0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7adc014000 tx=0x7f7adc015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc005a60 con 0x7f7af40988c0 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 10 monclient: dump: 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f7adc003cb0 con 0x7f7af40988c0 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 msgr2=0x7f7af4098cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 0x7f7af4098cc0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7adc014000 tx=0x7f7adc015000 comp rx=0 tx=0).stop 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 msgr2=0x7f7af4098cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af0ff9640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7af40988c0 0x7f7af4098cc0 secure :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7adc014000 tx=0x7f7adc015000 comp rx=0 tx=0).stop 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 58, 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:23.409 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:23.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:23.412 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:23.413 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:23.414 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:23.416 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:23.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:23.418 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:23.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:30:23.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:23.421 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ad4038490 msgr2=0x7f7ad403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ad4038490 0x7f7ad403a940 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7aec005fd0 tx=0x7f7aec007f00 comp rx=0 tx=0).stop 2024-09-18T12:30:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 10 monclient: shutdown 2024-09-18T12:30:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7af40971a0 msgr2=0x7f7af4141060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 --2- 172.21.15.116:0/2944416962 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7af40971a0 0x7f7af4141060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4035040 tx=0x7f7ae4032000 comp rx=0 tx=0).stop 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af8d3a640 1 -- 172.21.15.116:0/2944416962 reap_dead start 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7af8d3a640 1 -- 172.21.15.116:0/2944416962 reap_dead start 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 shutdown_connections 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 >> 172.21.15.116:0/2944416962 conn(0x7f7af40672b0 msgr2=0x7f7af409fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 shutdown_connections 2024-09-18T12:30:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:23.400+0000 7f7afa79d640 1 -- 172.21.15.116:0/2944416962 wait complete. 2024-09-18T12:30:23.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:30:23.423 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:30:23.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.945+0000 7f76c27fc640 1 -- 172.21.15.144:0/4057802241 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v108) v1 ==== 58+0+3 (secure 0 0 0) 0x7f76c4029070 con 0x7f76cc131b40 2024-09-18T12:30:23.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.945+0000 7f76c27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T12:30:23.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.945+0000 7f76c27fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f76ac038490 msgr2=0x7f76ac03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f76ac038490 0x7f76ac03a940 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f76b4034000 tx=0x7f76b4032000 comp rx=0 tx=0).stop 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 10 monclient: shutdown 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 msgr2=0x7f76cc12e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 0x7f76cc12e990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f76c400c8b0 tx=0x7f76c40026c0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 shutdown_connections 2024-09-18T12:30:23.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f76ac038490 0x7f76ac03a940 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76cc136e70 0x7f76cc097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f76cc131f10 0x7f76cc0976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 --2- 172.21.15.144:0/4057802241 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f76cc131b40 0x7f76cc12e990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:23.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 >> 172.21.15.144:0/4057802241 conn(0x7f76cc0a5100 msgr2=0x7f76cc130340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:23.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 shutdown_connections 2024-09-18T12:30:23.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:23.948+0000 7f76d0f9d640 1 -- 172.21.15.144:0/4057802241 wait complete. 2024-09-18T12:30:23.960 INFO:tasks.workunit.client.0.smithi144.stderr:+ o3=11 2024-09-18T12:30:23.960 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add 11 123 root=default 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.039+0000 7f8af555d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.039+0000 7f8af555d640 10 monclient: build_initial_monmap 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.039+0000 7f8af555d640 10 monclient: monmap: 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:24.040417+0000 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:24.040417+0000 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:24.041 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:24.042 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:24.042 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:24.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 Processor -- start 2024-09-18T12:30:24.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 -- start start 2024-09-18T12:30:24.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient: init 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af00980c0 0x7f8af0131fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): picked mon.noname-b con 0x7f8af00980c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8af0098490 0x7f8af0098860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): picked mon.noname-a con 0x7f8af0098490 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8af0068e00 0x7f8af0069230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): picked mon.noname-c con 0x7f8af0068e00 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:24.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af00a0200 con 0x7f8af0098490 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af00a15a0 con 0x7f8af0068e00 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af00a12c0 con 0x7f8af00980c0 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8af555d640 10 monclient(hunting): authenticate will time out at 1059.609617s 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8af0068e00 0x7f8af0069230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8af0068e00 0x7f8af0069230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60668/0 (socket says 172.21.15.144:60668) 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aee7fc640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f8af0098490 msgr2=0x7f8af0098860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 1 -- 172.21.15.144:0/852083629 learned_addr learned my addr 172.21.15.144:0/852083629 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 10 monclient(hunting): get_auth_request con 0x7f8af0068e00 auth_method 0 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:24.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aee7fc640 1 --2- 172.21.15.144:0/852083629 >> v2:172.21.15.116:3300/0 conn(0x7f8af0098490 0x7f8af0098860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aeeffd640 1 --2- 172.21.15.144:0/852083629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af00980c0 0x7f8af0131fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aeeffd640 10 monclient(hunting): get_auth_request con 0x7f8af00980c0 auth_method 0 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.040+0000 7f8aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:24.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 10 monclient(hunting): handle_auth_done global_id 5491 payload 274 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 1 -- 172.21.15.144:0/852083629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af00980c0 msgr2=0x7f8af0131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 1 --2- 172.21.15.144:0/852083629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af00980c0 0x7f8af0131fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 1 -- 172.21.15.144:0/852083629 >> v2:172.21.15.116:3300/0 conn(0x7f8af0098490 msgr2=0x7f8af0098860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 1 --2- 172.21.15.144:0/852083629 >> v2:172.21.15.116:3300/0 conn(0x7f8af0098490 0x7f8af0098860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 1 monclient: found mon.noname-c 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.041+0000 7f8aef7fe640 1 -- 172.21.15.144:0/852083629 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af009ff00 con 0x7f8af0068e00 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.043+0000 7f8aef7fe640 1 -- 172.21.15.144:0/852083629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 msgr2=0x7f8af0069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.043+0000 7f8aef7fe640 1 --2- 172.21.15.144:0/852083629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af0069230 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8af00a11c0 tx=0x7f8ae8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6f6fd6f2717c78a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 1 -- 172.21.15.144:0/852083629 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8007610 con 0x7f8af0068e00 2024-09-18T12:30:24.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: dump: 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:24.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 1 -- 172.21.15.144:0/852083629 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae802d050 con 0x7f8af0068e00 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 5 monclient: authenticate success, global_id 5491 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 1 -- 172.21.15.144:0/852083629 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae80079a0 con 0x7f8af0068e00 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8aedffb640 10 monclient: dump: 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:24.051 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 4 set_mon_vals no callback set 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 10 monclient: shutdown 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 -- 172.21.15.144:0/852083629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 msgr2=0x7f8af0069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 --2- 172.21.15.144:0/852083629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af0069230 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8af00a11c0 tx=0x7f8ae8009840 comp rx=0 tx=0).stop 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 -- 172.21.15.144:0/852083629 shutdown_connections 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 --2- 172.21.15.144:0/852083629 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af0069230 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 --2- 172.21.15.144:0/852083629 >> v2:172.21.15.116:3300/0 conn(0x7f8af0098490 0x7f8af0098860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:24.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 --2- 172.21.15.144:0/852083629 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af00980c0 0x7f8af0131fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 -- 172.21.15.144:0/852083629 >> 172.21.15.144:0/852083629 conn(0x7f8af0070fd0 msgr2=0x7f8af00a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 -- 172.21.15.144:0/852083629 shutdown_connections 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.044+0000 7f8af555d640 1 -- 172.21.15.144:0/852083629 wait complete. 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.045+0000 7f8af555d640 10 monclient: build_initial_monmap 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.045+0000 7f8af555d640 10 monclient: monmap: 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:24.053 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.045+0000 7f8af555d640 1 Processor -- start 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.045+0000 7f8af555d640 1 -- start start 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.045+0000 7f8af555d640 10 monclient: init 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): picked mon.noname-b con 0x7f8af0068e00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8af00980c0 0x7f8af0130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): picked mon.noname-a con 0x7f8af00980c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af0098490 0x7f8af0140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:24.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): picked mon.noname-c con 0x7f8af0098490 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af011dc70 con 0x7f8af00980c0 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af011df50 con 0x7f8af0068e00 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af011e230 con 0x7f8af0098490 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8af555d640 10 monclient(hunting): authenticate will time out at 1059.615241s 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af009aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af009aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60674/0 (socket says 172.21.15.144:60674) 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 -- 172.21.15.144:0/3208396898 learned_addr learned my addr 172.21.15.144:0/3208396898 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): get_auth_request con 0x7f8af0068e00 auth_method 0 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:24.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aee7fc640 1 -- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8af00980c0 msgr2=0x7f8af0130b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aee7fc640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8af00980c0 0x7f8af0130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aef7fe640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af0098490 0x7f8af0140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient(hunting): handle_auth_done global_id 5494 payload 824 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 -- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af0098490 msgr2=0x7f8af0140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af0098490 0x7f8af0140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 -- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8af00980c0 msgr2=0x7f8af0130b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8af00980c0 0x7f8af0130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:24.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 monclient: found mon.noname-b 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aeeffd640 1 -- 172.21.15.144:0/3208396898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae8009ae0 con 0x7f8af0068e00 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aef7fe640 10 monclient: get_auth_request con 0x7f8af0098490 auth_method 0 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.046+0000 7f8aef7fe640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af0098490 0x7f8af0140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8aeeffd640 1 -- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 msgr2=0x7f8af009aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8aeeffd640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af009aee0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8ae000ea20 tx=0x7f8ae00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0009600 con 0x7f8af0068e00 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 10 monclient: dump: 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.061 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae000cc20 con 0x7f8af0068e00 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0013470 con 0x7f8af0068e00 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8af555d640 5 monclient: authenticate success, global_id 5494 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8af555d640 10 monclient: _renew_subs 2024-09-18T12:30:24.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8af555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af011c8e0 con 0x7f8af0068e00 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8aed7fa640 4 set_mon_vals no callback set 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.048+0000 7f8ae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.049+0000 7f8ae6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.049+0000 7f8ae6ffd640 10 monclient: dump: 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:24.063 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.049+0000 7f8af555d640 10 monclient: _renew_subs 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.049+0000 7f8af555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.049+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af011cf20 con 0x7f8af0068e00 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8ae0013860 con 0x7f8af0068e00 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8af555d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8af555d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8af555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af00008d0 con 0x7f8af0068e00 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8ae6ffd640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ad0038440 0x7f8ad003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f8ae004bfd0 con 0x7f8af0068e00 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8aee7fc640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ad0038440 0x7f8ad003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.050+0000 7f8aee7fc640 10 monclient: get_auth_request con 0x7f8ad0038440 auth_method 0 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.052+0000 7f8aee7fc640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ad0038440 0x7f8ad003a8f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8af01319a0 tx=0x7f8ad8006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.052+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:30:24.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.054+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8ae00236b0 con 0x7f8af0068e00 2024-09-18T12:30:24.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.054+0000 7f8ae6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:24.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.054+0000 7f8ae6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:24.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.211+0000 7f8af555d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T12:30:24.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.211+0000 7f8af555d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T12:30:24.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.211+0000 7f8af555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:24.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.212+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f8af00a5b80 con 0x7f8af0068e00 2024-09-18T12:30:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.947+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map v109) v1 ==== 193+0+0 (secure 0 0 0) 0x7f8ae00232f0 con 0x7f8af0068e00 2024-09-18T12:30:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.947+0000 7f8ae6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T12:30:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:24.947+0000 7f8ae6ffd640 10 monclient: _finish_command 2 = system:0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map 2024-09-18T12:30:25.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.045+0000 7f8af555d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T12:30:25.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.045+0000 7f8af555d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T12:30:25.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.045+0000 7f8af555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.045+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f8af0064a20 con 0x7f8af0068e00 2024-09-18T12:30:25.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.045+0000 7f8ae6ffd640 1 -- 172.21.15.144:0/3208396898 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change v109) v1 ==== 191+0+0 (secure 0 0 0) 0x7f8ae00234d0 con 0x7f8af0068e00 2024-09-18T12:30:25.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.045+0000 7f8ae6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T12:30:25.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.045+0000 7f8ae6ffd640 10 monclient: _finish_command 3 = system:0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-18T12:30:25.047 INFO:tasks.workunit.client.0.smithi144.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-18T12:30:25.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ad0038440 msgr2=0x7f8ad003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ad0038440 0x7f8ad003a8f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8af01319a0 tx=0x7f8ad8006d80 comp rx=0 tx=0).stop 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 10 monclient: shutdown 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 msgr2=0x7f8af009aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af009aee0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8ae000ea20 tx=0x7f8ae00090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 shutdown_connections 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ad0038440 0x7f8ad003a8f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8af0098490 0x7f8af0140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8af00980c0 0x7f8af0130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 --2- 172.21.15.144:0/3208396898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8af0068e00 0x7f8af009aee0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 >> 172.21.15.144:0/3208396898 conn(0x7f8af0070fd0 msgr2=0x7f8af009e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:25.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 shutdown_connections 2024-09-18T12:30:25.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.047+0000 7f8af555d640 1 -- 172.21.15.144:0/3208396898 wait complete. 2024-09-18T12:30:25.058 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:30:25.058 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep osd.11 2024-09-18T12:30:25.058 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 123 2024-09-18T12:30:25.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.129+0000 7f5ae2434640 10 monclient: get_monmap_and_config 2024-09-18T12:30:25.130 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.129+0000 7f5ae2434640 10 monclient: build_initial_monmap 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.129+0000 7f5ae2434640 10 monclient: monmap: 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:25.130598+0000 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:25.130598+0000 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:25.131 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:25.132 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:25.132 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 1 Processor -- start 2024-09-18T12:30:25.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 1 -- start start 2024-09-18T12:30:25.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient: init 2024-09-18T12:30:25.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5adc131b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5adc131f10 0x7f5adc132330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient(hunting): picked mon.noname-a con 0x7f5adc131f10 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient(hunting): picked mon.noname-b con 0x7f5adc136e70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5adc06a230 con 0x7f5adc131b40 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5adc0a0300 con 0x7f5adc131f10 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0a00c0 con 0x7f5adc136e70 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.130+0000 7f5ae2434640 10 monclient(hunting): authenticate will time out at 1060.699951s 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5adb7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f5adc131f10 msgr2=0x7f5adc132330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5adbfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f5adc131b40 0x7f5adc09ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:60754/0 (socket says 172.21.15.144:60754) 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5adb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5adc131f10 0x7f5adc132330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5adbfff640 1 -- 172.21.15.144:0/3259908877 learned_addr learned my addr 172.21.15.144:0/3259908877 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 1 --2- 172.21.15.144:0/3259908877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): get_auth_request con 0x7f5adc136e70 auth_method 0 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:25.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient(hunting): handle_auth_done global_id 64358 payload 274 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 1 -- 172.21.15.144:0/3259908877 >> v2:172.21.15.116:3300/0 conn(0x7f5adc131f10 msgr2=0x7f5adc132330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 1 --2- 172.21.15.144:0/3259908877 >> v2:172.21.15.116:3300/0 conn(0x7f5adc131f10 0x7f5adc132330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 1 -- 172.21.15.144:0/3259908877 >> v1:172.21.15.144:6789/0 conn(0x7f5adc131b40 legacy=0x7f5adc09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient: _finish_hunting 0 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 1 monclient: found mon.noname-b 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:25.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.131+0000 7f5ae09d1640 1 -- 172.21.15.144:0/3259908877 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc09bee0 con 0x7f5adc136e70 2024-09-18T12:30:25.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5ae09d1640 1 -- 172.21.15.144:0/3259908877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 msgr2=0x7f5adc139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:25.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5ae09d1640 1 --2- 172.21.15.144:0/3259908877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc139250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009e60 tx=0x7f5ad4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=313d566f9ef6c089 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 1 -- 172.21.15.144:0/3259908877 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4007840 con 0x7f5adc136e70 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: dump: 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.144 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5ae2434640 5 monclient: authenticate success, global_id 64358 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5ae2434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 1 -- 172.21.15.144:0/3259908877 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad402d050 con 0x7f5adc136e70 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 1 -- 172.21.15.144:0/3259908877 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4007bd0 con 0x7f5adc136e70 2024-09-18T12:30:25.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5ae2434640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5ae2434640 4 set_mon_vals no callback set 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.134+0000 7f5adaffd640 10 monclient: dump: 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.146 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 10 monclient: shutdown 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 -- 172.21.15.144:0/3259908877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 msgr2=0x7f5adc139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 --2- 172.21.15.144:0/3259908877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc139250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009e60 tx=0x7f5ad4009780 comp rx=0 tx=0).stop 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 -- 172.21.15.144:0/3259908877 shutdown_connections 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 --2- 172.21.15.144:0/3259908877 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc139250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 --2- 172.21.15.144:0/3259908877 >> v2:172.21.15.116:3300/0 conn(0x7f5adc131f10 0x7f5adc132330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 -- 172.21.15.144:0/3259908877 >> 172.21.15.144:0/3259908877 conn(0x7f5adc0a5100 msgr2=0x7f5adc0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 -- 172.21.15.144:0/3259908877 shutdown_connections 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 1 -- 172.21.15.144:0/3259908877 wait complete. 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 10 monclient: build_initial_monmap 2024-09-18T12:30:25.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.135+0000 7f5ae2434640 10 monclient: monmap: 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 1 Processor -- start 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 1 -- start start 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 10 monclient: init 2024-09-18T12:30:25.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5adc131b40 0x7f5adc1418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 10 monclient(hunting): picked mon.noname-a con 0x7f5adc131b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 0x7f5adc0989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 10 monclient(hunting): picked mon.noname-b con 0x7f5adc131f10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc098f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.136+0000 7f5ae2434640 10 monclient(hunting): picked mon.noname-c con 0x7f5adc136e70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc130d60 con 0x7f5adc131b40 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0a0a30 con 0x7f5adc131f10 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc09a970 con 0x7f5adc136e70 2024-09-18T12:30:25.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae2434640 10 monclient(hunting): authenticate will time out at 1060.706025s 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adbfff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5adc131b40 msgr2=0x7f5adc1418d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5adc131b40 0x7f5adc1418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 0x7f5adc0989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 0x7f5adc0989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60688/0 (socket says 172.21.15.144:60688) 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 -- 172.21.15.144:0/2904598450 learned_addr learned my addr 172.21.15.144:0/2904598450 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): get_auth_request con 0x7f5adc131f10 auth_method 0 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:25.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc098f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 10 monclient(hunting): get_auth_request con 0x7f5adc136e70 auth_method 0 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5ae09d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:25.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient(hunting): handle_auth_done global_id 5497 payload 824 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 -- 172.21.15.144:0/2904598450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 msgr2=0x7f5adc098f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc098f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 -- 172.21.15.144:0/2904598450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5adc131b40 msgr2=0x7f5adc1418d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5adc131b40 0x7f5adc1418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.137+0000 7f5adb7fe640 1 -- 172.21.15.144:0/2904598450 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4009a20 con 0x7f5adc131f10 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5adb7fe640 1 -- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 msgr2=0x7f5adc0989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5adb7fe640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 0x7f5adc0989c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000b600 tx=0x7f5ad00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:25.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0015070 con 0x7f5adc131f10 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: dump: 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0003d90 con 0x7f5adc131f10 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ae2434640 5 monclient: authenticate success, global_id 5497 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0006380 con 0x7f5adc131f10 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ae2434640 10 monclient: _renew_subs 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ae2434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc11c230 con 0x7f5adc131f10 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ac3fff640 10 monclient: dump: 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.154 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.139+0000 7f5ada7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.140+0000 7f5ae2434640 10 monclient: _renew_subs 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.140+0000 7f5ae2434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.140+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc09a460 con 0x7f5adc131f10 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.140+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5ad0010440 con 0x7f5adc131f10 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.140+0000 7f5ac3fff640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ac8038440 0x7f5ac803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.140+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5978+0+0 (secure 0 0 0) 0x7f5ad004b6f0 con 0x7f5adc131f10 2024-09-18T12:30:25.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.141+0000 7f5adbfff640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ac8038440 0x7f5ac803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.141+0000 7f5adbfff640 10 monclient: get_auth_request con 0x7f5ac8038440 auth_method 0 2024-09-18T12:30:25.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.141+0000 7f5ae2434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:25.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.141+0000 7f5ae2434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:25.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.141+0000 7f5ae2434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.141+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc09ade0 con 0x7f5adc131f10 2024-09-18T12:30:25.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.143+0000 7f5adbfff640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ac8038440 0x7f5ac803a8f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0047d0 tx=0x7f5acc007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:25.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.143+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:30:25.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.145+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5ad0018ab0 con 0x7f5adc131f10 2024-09-18T12:30:25.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.145+0000 7f5ac3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:25.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.145+0000 7f5ac3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:25.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.300+0000 7f5ae2434640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:25.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.300+0000 7f5ae2434640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:25.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.300+0000 7f5ae2434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.300+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5adc0636e0 con 0x7f5adc131f10 2024-09-18T12:30:25.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.301+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v109) v1 ==== 56+0+1173 (secure 0 0 0) 0x7f5ad0018690 con 0x7f5adc131f10 2024-09-18T12:30:25.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.301+0000 7f5ac3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:25.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.301+0000 7f5ac3fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:25.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.392+0000 7f5ae2434640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:25.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.392+0000 7f5ae2434640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:25.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.392+0000 7f5ae2434640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.392+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f5adc1318a0 con 0x7f5adc131f10 2024-09-18T12:30:25.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.393+0000 7f5ac3fff640 1 -- 172.21.15.144:0/2904598450 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v109) v1 ==== 56+0+1173 (secure 0 0 0) 0x7f5ad0018ab0 con 0x7f5adc131f10 2024-09-18T12:30:25.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.393+0000 7f5ac3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:25.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.393+0000 7f5ac3fff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:25.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ac8038440 msgr2=0x7f5ac803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ac8038440 0x7f5ac803a8f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0047d0 tx=0x7f5acc007b80 comp rx=0 tx=0).stop 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 10 monclient: shutdown 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 msgr2=0x7f5adc0989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 0x7f5adc0989c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000b600 tx=0x7f5ad00091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 shutdown_connections 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ac8038440 0x7f5ac803a8f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5adc136e70 0x7f5adc098f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5adc131f10 0x7f5adc0989c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 --2- 172.21.15.144:0/2904598450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5adc131b40 0x7f5adc1418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 >> 172.21.15.144:0/2904598450 conn(0x7f5adc0a5100 msgr2=0x7f5adc0a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:25.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 shutdown_connections 2024-09-18T12:30:25.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.395+0000 7f5ae2434640 1 -- 172.21.15.144:0/2904598450 wait complete. 2024-09-18T12:30:25.405 INFO:tasks.workunit.client.0.smithi144.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2024-09-18T12:30:25.405 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush reweight osd.11 113 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.479+0000 7f09c1634640 10 monclient: get_monmap_and_config 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.479+0000 7f09c1634640 10 monclient: build_initial_monmap 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.479+0000 7f09c1634640 10 monclient: monmap: 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:25.480846+0000 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:25.480846+0000 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:25.481 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.482 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:25.482 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:25.482 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:25.482 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:25.482 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:25.482 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.480+0000 7f09c1634640 1 Processor -- start 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.480+0000 7f09c1634640 1 -- start start 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.480+0000 7f09c1634640 10 monclient: init 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.480+0000 7f09c1634640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.480+0000 7f09c1634640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09bc0a0ad0 0x7f09bc0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 10 monclient(hunting): picked mon.noname-c con 0x7f09bc0a0ad0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09bc09ee60 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09bc09f260 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc06a5b0 con 0x7f09bc09f260 2024-09-18T12:30:25.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc0a0460 con 0x7f09bc09ee60 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc0a0ea0 con 0x7f09bc0a0ad0 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09c1634640 10 monclient(hunting): authenticate will time out at 1061.050112s 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09bb7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f09bc09f260 legacy=0x7f09bc1321e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09bc0a0ad0 0x7f09bc0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09bc0a0ad0 0x7f09bc0691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60698/0 (socket says 172.21.15.144:60698) 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 1 -- 172.21.15.144:0/28167202 learned_addr learned my addr 172.21.15.144:0/28167202 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 10 monclient(hunting): get_auth_request con 0x7f09bc0a0ad0 auth_method 0 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09b9ffb640 1 -- 172.21.15.144:0/28167202 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:25.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:25.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:25.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09ba7fc640 1 --1- 172.21.15.144:0/28167202 >> v1:172.21.15.144:6789/0 conn(0x7f09bc09ee60 0x7f09bc131ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:60762/0 (socket says 172.21.15.144:60762) 2024-09-18T12:30:25.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:25.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:25.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.481+0000 7f09baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:25.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09ba7fc640 1 -- 172.21.15.144:0/28167202 >> v1:172.21.15.144:6789/0 conn(0x7f09bc09ee60 legacy=0x7f09bc131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 10 monclient(hunting): handle_auth_done global_id 5500 payload 274 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 1 -- 172.21.15.144:0/28167202 >> v1:172.21.15.144:6789/0 conn(0x7f09bc09ee60 legacy=0x7f09bc131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 1 -- 172.21.15.144:0/28167202 >> v1:172.21.15.116:6789/0 conn(0x7f09bc09f260 legacy=0x7f09bc1321e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 1 monclient: found mon.noname-c 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.483+0000 7f09baffd640 1 -- 172.21.15.144:0/28167202 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0a1bb0 con 0x7f09bc0a0ad0 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09baffd640 1 -- 172.21.15.144:0/28167202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc0a0ad0 msgr2=0x7f09bc0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09baffd640 1 --2- 172.21.15.144:0/28167202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc0a0ad0 0x7f09bc0691d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009990 tx=0x7f09ac0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c19dc2cbb26426 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:25.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 1 -- 172.21.15.144:0/28167202 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac02ae90 con 0x7f09bc0a0ad0 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: dump: 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09c1634640 5 monclient: authenticate success, global_id 5500 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 1 -- 172.21.15.144:0/28167202 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac02f4b0 con 0x7f09bc0a0ad0 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09c1634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09c1634640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09c1634640 4 set_mon_vals no callback set 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 1 -- 172.21.15.144:0/28167202 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac02f890 con 0x7f09bc0a0ad0 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.486+0000 7f09b9ffb640 10 monclient: dump: 2024-09-18T12:30:25.491 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 10 monclient: shutdown 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:25.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 1 -- 172.21.15.144:0/28167202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc0a0ad0 msgr2=0x7f09bc0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 1 --2- 172.21.15.144:0/28167202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc0a0ad0 0x7f09bc0691d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009990 tx=0x7f09ac0091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 1 -- 172.21.15.144:0/28167202 shutdown_connections 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 1 --2- 172.21.15.144:0/28167202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc0a0ad0 0x7f09bc0691d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 1 -- 172.21.15.144:0/28167202 >> 172.21.15.144:0/28167202 conn(0x7f09bc071370 msgr2=0x7f09bc0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 1 -- 172.21.15.144:0/28167202 shutdown_connections 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 1 -- 172.21.15.144:0/28167202 wait complete. 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 10 monclient: build_initial_monmap 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.487+0000 7f09c1634640 10 monclient: monmap: 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:25.493 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 Processor -- start 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 -- start start 2024-09-18T12:30:25.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient: init 2024-09-18T12:30:25.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:25.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:25.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ee60 0x7f09bc09aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): picked mon.noname-a con 0x7f09bc09ee60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:25.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 0x7f09bc130e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): picked mon.noname-b con 0x7f09bc09f260 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:25.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc0a0ad0 0x7f09bc11b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): picked mon.noname-c con 0x7f09bc0a0ad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0a1250 con 0x7f09bc09ee60 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc131a60 con 0x7f09bc09f260 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc11d010 con 0x7f09bc0a0ad0 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09c1634640 10 monclient(hunting): authenticate will time out at 1061.057645s 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 0x7f09bc130e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 0x7f09bc130e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60704/0 (socket says 172.21.15.144:60704) 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09baffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ee60 msgr2=0x7f09bc09aad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 1 -- 172.21.15.144:0/2023132936 learned_addr learned my addr 172.21.15.144:0/2023132936 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): get_auth_request con 0x7f09bc09f260 auth_method 0 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09baffd640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ee60 0x7f09bc09aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09bb7fe640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc0a0ad0 0x7f09bc11b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09bb7fe640 10 monclient(hunting): get_auth_request con 0x7f09bc0a0ad0 auth_method 0 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.488+0000 7f09ba7fc640 10 monclient(hunting): handle_auth_done global_id 5503 payload 824 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 1 -- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc0a0ad0 msgr2=0x7f09bc11b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc0a0ad0 0x7f09bc11b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 1 -- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ee60 msgr2=0x7f09bc09aad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ee60 0x7f09bc09aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 1 monclient: found mon.noname-b 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.489+0000 7f09ba7fc640 1 -- 172.21.15.144:0/2023132936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac009450 con 0x7f09bc09f260 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09ba7fc640 1 -- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 msgr2=0x7f09bc130e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09ba7fc640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 0x7f09bc130e50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c590 tx=0x7f09b0007dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0007510 con 0x7f09bc09f260 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: dump: 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.502 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09c1634640 5 monclient: authenticate success, global_id 5503 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b00047a0 con 0x7f09bc09f260 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09c1634640 10 monclient: _renew_subs 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09c1634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc11df70 con 0x7f09bc09f260 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0007690 con 0x7f09bc09f260 2024-09-18T12:30:25.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09a2ffd640 10 monclient: dump: 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:25.504 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09c1634640 10 monclient: _renew_subs 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09c1634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc1420d0 con 0x7f09bc09f260 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.491+0000 7f09b97fa640 4 set_mon_vals no callback set 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f09b0013400 con 0x7f09bc09f260 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09c1634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09c1634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09c1634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc0008d0 con 0x7f09bc09f260 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09a2ffd640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038440 0x7f09a803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5978+0+0 (secure 0 0 0) 0x7f09b004af30 con 0x7f09bc09f260 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09baffd640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038440 0x7f09a803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:25.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.492+0000 7f09baffd640 10 monclient: get_auth_request con 0x7f09a8038440 auth_method 0 2024-09-18T12:30:25.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.495+0000 7f09baffd640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038440 0x7f09a803a8f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f09ac034000 tx=0x7f09ac032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:25.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.496+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f09b000f660 con 0x7f09bc09f260 2024-09-18T12:30:25.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.496+0000 7f09a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:25.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.496+0000 7f09a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:25.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.655+0000 7f09c1634640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-18T12:30:25.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.655+0000 7f09c1634640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-18T12:30:25.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.655+0000 7f09c1634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:25.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.655+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7f09bc09b680 con 0x7f09bc09f260 2024-09-18T12:30:25.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.953+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v110) v1 ==== 156+0+0 (secure 0 0 0) 0x7f09b000e290 con 0x7f09bc09f260 2024-09-18T12:30:25.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.953+0000 7f09a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-18T12:30:25.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:25.953+0000 7f09a2ffd640 10 monclient: _finish_command 2 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-18T12:30:26.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.051+0000 7f09c1634640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-18T12:30:26.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.051+0000 7f09c1634640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-18T12:30:26.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.051+0000 7f09c1634640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:26.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.051+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) v1 -- 0x7f09bc072640 con 0x7f09bc09f260 2024-09-18T12:30:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.573+0000 7f09a2ffd640 1 -- 172.21.15.144:0/2023132936 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v111) v1 ==== 156+0+0 (secure 0 0 0) 0x7f09b000f660 con 0x7f09bc09f260 2024-09-18T12:30:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.573+0000 7f09a2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-18T12:30:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.573+0000 7f09a2ffd640 10 monclient: _finish_command 3 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-18T12:30:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-18T12:30:26.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038440 msgr2=0x7f09a803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:26.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038440 0x7f09a803a8f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f09ac034000 tx=0x7f09ac032000 comp rx=0 tx=0).stop 2024-09-18T12:30:26.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 10 monclient: shutdown 2024-09-18T12:30:26.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:26.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 msgr2=0x7f09bc130e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:26.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 0x7f09bc130e50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c590 tx=0x7f09b0007dc0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 shutdown_connections 2024-09-18T12:30:26.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038440 0x7f09a803a8f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc0a0ad0 0x7f09bc11b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc09f260 0x7f09bc130e50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 --2- 172.21.15.144:0/2023132936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ee60 0x7f09bc09aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 >> 172.21.15.144:0/2023132936 conn(0x7f09bc071370 msgr2=0x7f09bc131c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:26.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 shutdown_connections 2024-09-18T12:30:26.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.576+0000 7f09c1634640 1 -- 172.21.15.144:0/2023132936 wait complete. 2024-09-18T12:30:26.588 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2024-09-18T12:30:26.588 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:30:26.588 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush reweight osd.11 123456 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.662+0000 7effda6c9640 10 monclient: get_monmap_and_config 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.662+0000 7effda6c9640 10 monclient: build_initial_monmap 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.662+0000 7effda6c9640 10 monclient: monmap: 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:26.662970+0000 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:26.662970+0000 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:26.663 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:26.664 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:26.664 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:26.664 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:26.664 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:26.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.662+0000 7effda6c9640 1 Processor -- start 2024-09-18T12:30:26.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.662+0000 7effda6c9640 1 -- start start 2024-09-18T12:30:26.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.662+0000 7effda6c9640 10 monclient: init 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7effd409ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7effd409f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7effd4069d10 0x7effd413a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient(hunting): picked mon.noname-c con 0x7effd4069d10 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd406a120 con 0x7effd409f020 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd4067420 con 0x7effd409ec50 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd409fdc0 con 0x7effd4069d10 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effda6c9640 10 monclient(hunting): authenticate will time out at 1062.232335s 2024-09-18T12:30:26.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd3fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7effd409ec50 0x7effd4131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:60770/0 (socket says 172.21.15.144:60770) 2024-09-18T12:30:26.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd3fff640 1 -- 172.21.15.144:0/3777318238 learned_addr learned my addr 172.21.15.144:0/3777318238 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:26.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd37fe640 1 -- 172.21.15.144:0/3777318238 >> v1:172.21.15.116:6789/0 conn(0x7effd409f020 legacy=0x7effd4136d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:26.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd8c66640 1 --2- 172.21.15.144:0/3777318238 >> v2:172.21.15.144:3300/0 conn(0x7effd4069d10 0x7effd413a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:26.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd8c66640 10 monclient(hunting): get_auth_request con 0x7effd4069d10 auth_method 0 2024-09-18T12:30:26.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd8c66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:26.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd8c66640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:26.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.663+0000 7effd8c66640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:26.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd3fff640 1 -- 172.21.15.144:0/3777318238 >> v1:172.21.15.144:6789/0 conn(0x7effd409ec50 legacy=0x7effd4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:26.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:26.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:26.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:26.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 10 monclient(hunting): handle_auth_done global_id 5506 payload 274 2024-09-18T12:30:26.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 1 -- 172.21.15.144:0/3777318238 >> v1:172.21.15.144:6789/0 conn(0x7effd409ec50 legacy=0x7effd4131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:26.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 1 -- 172.21.15.144:0/3777318238 >> v1:172.21.15.116:6789/0 conn(0x7effd409f020 legacy=0x7effd4136d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:26.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 10 monclient: _finish_hunting 0 2024-09-18T12:30:26.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 1 monclient: found mon.noname-c 2024-09-18T12:30:26.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:26.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:26.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.665+0000 7effd8c66640 1 -- 172.21.15.144:0/3777318238 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40a0510 con 0x7effd4069d10 2024-09-18T12:30:26.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd8c66640 1 -- 172.21.15.144:0/3777318238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4069d10 msgr2=0x7effd413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:26.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd8c66640 1 --2- 172.21.15.144:0/3777318238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4069d10 0x7effd413a490 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7effcc009ee0 tx=0x7effcc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd1cf167ba1d406 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:26.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 1 -- 172.21.15.144:0/3777318238 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc02aea0 con 0x7effd4069d10 2024-09-18T12:30:26.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:26.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: dump: 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:26.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 1 -- 172.21.15.144:0/3777318238 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc02d050 con 0x7effd4069d10 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effda6c9640 5 monclient: authenticate success, global_id 5506 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effda6c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 1 -- 172.21.15.144:0/3777318238 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc007a70 con 0x7effd4069d10 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effd2ffd640 10 monclient: dump: 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.675 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effda6c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.668+0000 7effda6c9640 4 set_mon_vals no callback set 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 10 monclient: shutdown 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 1 -- 172.21.15.144:0/3777318238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4069d10 msgr2=0x7effd413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 1 --2- 172.21.15.144:0/3777318238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4069d10 0x7effd413a490 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7effcc009ee0 tx=0x7effcc009780 comp rx=0 tx=0).stop 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 1 -- 172.21.15.144:0/3777318238 shutdown_connections 2024-09-18T12:30:26.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 1 --2- 172.21.15.144:0/3777318238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4069d10 0x7effd413a490 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 1 -- 172.21.15.144:0/3777318238 >> 172.21.15.144:0/3777318238 conn(0x7effd40a5d40 msgr2=0x7effd40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 1 -- 172.21.15.144:0/3777318238 shutdown_connections 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 1 -- 172.21.15.144:0/3777318238 wait complete. 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 10 monclient: build_initial_monmap 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.669+0000 7effda6c9640 10 monclient: monmap: 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:26.677 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 Processor -- start 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 -- start start 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient: init 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd4069d10 0x7effd409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): picked mon.noname-c con 0x7effd4069d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd409ec50 0x7effd4130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): picked mon.noname-b con 0x7effd409ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd409f020 0x7effd4140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:26.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): picked mon.noname-a con 0x7effd409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40a4070 con 0x7effd409f020 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd411dce0 con 0x7effd409ec50 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd411dfc0 con 0x7effd4069d10 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effda6c9640 10 monclient(hunting): authenticate will time out at 1062.239909s 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effd37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd409ec50 0x7effd4130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.670+0000 7effd37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd409ec50 0x7effd4130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:60718/0 (socket says 172.21.15.144:60718) 2024-09-18T12:30:26.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 1 -- 172.21.15.144:0/4177943748 learned_addr learned my addr 172.21.15.144:0/4177943748 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): get_auth_request con 0x7effd409ec50 auth_method 0 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd8c66640 1 -- 172.21.15.144:0/4177943748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd409f020 msgr2=0x7effd4140e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd8c66640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd409f020 0x7effd4140e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient(hunting): handle_auth_done global_id 5512 payload 824 2024-09-18T12:30:26.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 1 -- 172.21.15.144:0/4177943748 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd4069d10 msgr2=0x7effd409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd4069d10 0x7effd409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 1 -- 172.21.15.144:0/4177943748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd409f020 msgr2=0x7effd4140e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd409f020 0x7effd4140e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:26.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.671+0000 7effd37fe640 1 -- 172.21.15.144:0/4177943748 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc009a20 con 0x7effd409ec50 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effd37fe640 1 -- 172.21.15.144:0/4177943748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd409ec50 msgr2=0x7effd4130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effd37fe640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd409ec50 0x7effd4130bc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7effc800b600 tx=0x7effc80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effd37fe640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd409ec50 0x7effd4130bc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7effc800b600 tx=0x7effc80091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7effc8015070 mon_map magic: 0 v1 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 1 -- 172.21.15.144:0/4177943748 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8015070 con 0x7effd409ec50 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 10 monclient: dump: 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:26.684 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 10 monclient: _finish_auth 0 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 1 -- 172.21.15.144:0/4177943748 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc8003d90 con 0x7effd409ec50 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effda6c9640 5 monclient: authenticate success, global_id 5512 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effda6c9640 10 monclient: _renew_subs 2024-09-18T12:30:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effda6c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effda6c9640 1 -- 172.21.15.144:0/4177943748 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd411e2a0 con 0x7effd409ec50 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effda6c9640 10 monclient: _renew_subs 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effd27fc640 4 set_mon_vals no callback set 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effda6c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.673+0000 7effda6c9640 1 -- 172.21.15.144:0/4177943748 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd411c5c0 con 0x7effd409ec50 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effbbfff640 1 -- 172.21.15.144:0/4177943748 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8006380 con 0x7effd409ec50 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effbbfff640 10 monclient: dump: 2024-09-18T12:30:26.686 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effbbfff640 1 -- 172.21.15.144:0/4177943748 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7effc8010440 con 0x7effd409ec50 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effda6c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effda6c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effda6c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effda6c9640 1 -- 172.21.15.144:0/4177943748 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd40008d0 con 0x7effd409ec50 2024-09-18T12:30:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.674+0000 7effbbfff640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effc0038490 0x7effc003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.675+0000 7effd3fff640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effc0038490 0x7effc003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.675+0000 7effd3fff640 10 monclient: get_auth_request con 0x7effc0038490 auth_method 0 2024-09-18T12:30:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.677+0000 7effd3fff640 1 --2- 172.21.15.144:0/4177943748 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effc0038490 0x7effc003a940 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7effc4007fd0 tx=0x7effc4007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:26.679+0000 7effbbfff640 1 -- 172.21.15.144:0/4177943748 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6007+0+0 (secure 0 0 0) 0x7effc804b850 con 0x7effd409ec50 2024-09-18T12:30:27.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.296+0000 7effbbfff640 1 -- 172.21.15.144:0/4177943748 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7effc8018d60 con 0x7effd409ec50 2024-09-18T12:30:27.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.296+0000 7effbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:27.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.296+0000 7effbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:27.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.457+0000 7effda6c9640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-18T12:30:27.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.457+0000 7effda6c9640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-18T12:30:27.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.457+0000 7effda6c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.457+0000 7effda6c9640 1 -- 172.21.15.144:0/4177943748 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0} v 0) v1 -- 0x7effd4064900 con 0x7effd409ec50 2024-09-18T12:30:27.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.457+0000 7effbbfff640 1 -- 172.21.15.144:0/4177943748 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}]=-75 (75) Value too large for defined data type v111) v1 ==== 146+0+0 (secure 0 0 0) 0x7effc8018780 con 0x7effd409ec50 2024-09-18T12:30:27.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.457+0000 7effbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-18T12:30:27.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.457+0000 7effbbfff640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-18T12:30:27.459 INFO:tasks.workunit.client.0.smithi144.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-18T12:30:27.462 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:30:27.462 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:30:27.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep osd.11 2024-09-18T12:30:27.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 113 2024-09-18T12:30:27.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.533+0000 7f702459d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:27.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.533+0000 7f702459d640 10 monclient: build_initial_monmap 2024-09-18T12:30:27.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.533+0000 7f702459d640 10 monclient: monmap: 2024-09-18T12:30:27.534 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:27.534197+0000 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:27.534197+0000 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:27.535 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.533+0000 7f702459d640 1 Processor -- start 2024-09-18T12:30:27.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.533+0000 7f702459d640 1 -- start start 2024-09-18T12:30:27.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.533+0000 7f702459d640 10 monclient: init 2024-09-18T12:30:27.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:27.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T12:30:27.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f701c131b40 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:27.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f701c131f10 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f701c132330 0x7f701c13a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient(hunting): picked mon.noname-a con 0x7f701c132330 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0a0300 con 0x7f701c131b40 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0a00c0 con 0x7f701c131f10 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c09bee0 con 0x7f701c132330 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f702459d640 10 monclient(hunting): authenticate will time out at 1063.103706s 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f7021b38640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f701c131f10 0x7f701c136e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39018/0 (socket says 172.21.15.144:39018) 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f7022b3a640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f701c132330 msgr2=0x7f701c13a5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f7022339640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f701c131b40 legacy=0x7f701c09ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:27.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f7021b38640 1 -- 172.21.15.144:0/1863125043 learned_addr learned my addr 172.21.15.144:0/1863125043 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:27.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.534+0000 7f7022b3a640 1 --2- 172.21.15.144:0/1863125043 >> v2:172.21.15.116:3300/0 conn(0x7f701c132330 0x7f701c13a5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:27.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.536+0000 7f7021b38640 1 -- 172.21.15.144:0/1863125043 >> v1:172.21.15.144:6789/0 conn(0x7f701c131f10 legacy=0x7f701c136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:27.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288976662 0 0) 0x7f701c0a00c0 con 0x7f701c131f10 2024-09-18T12:30:27.547 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): my global_id is 5515 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70080034e0 con 0x7f701c131f10 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2135286801 0 0) 0x7f70080034e0 con 0x7f701c131f10 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 >> v2:172.21.15.116:3300/0 conn(0x7f701c132330 msgr2=0x7f701c13a5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 --2- 172.21.15.144:0/1863125043 >> v2:172.21.15.116:3300/0 conn(0x7f701c132330 0x7f701c13a5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 >> v1:172.21.15.116:6789/0 conn(0x7f701c131b40 legacy=0x7f701c09ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: _finish_hunting 0 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:30:27.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c0a14b0 con 0x7f701c131f10 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: _finish_auth 0 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f702459d640 5 monclient: authenticate success, global_id 5515 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f702459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f7018001860 con 0x7f701c131f10 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: dump: 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.549 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 >> v1:172.21.15.144:6789/0 conn(0x7f701c131f10 legacy=0x7f701c136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70080039d0 0x7f7008003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): picked mon.c con 0x7f70080039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7008004780 0x7f7008004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): picked mon.b con 0x7f7008004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7008005330 0x7f700800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): picked mon.a con 0x7f7008005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c09bee0 con 0x7f7008005330 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700800c2f0 con 0x7f7008004780 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700800c4b0 con 0x7f70080039d0 2024-09-18T12:30:27.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021337640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f702459d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7008004780 0x7f7008004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): get_auth_request con 0x7f7008004780 auth_method 0 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022b3a640 1 -- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7008005330 msgr2=0x7f700800ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022b3a640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7008005330 0x7f700800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022339640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70080039d0 0x7f7008003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022339640 10 monclient(hunting): get_auth_request con 0x7f70080039d0 auth_method 0 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022339640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient(hunting): handle_auth_done global_id 5515 payload 293 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 1 -- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70080039d0 msgr2=0x7f7008003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70080039d0 0x7f7008003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 1 -- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7008005330 msgr2=0x7f700800ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7008005330 0x7f700800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient: _finish_hunting 0 2024-09-18T12:30:27.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 1 monclient: found mon.b 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 1 -- 172.21.15.144:0/1863125043 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70080034e0 con 0x7f7008004780 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7022b3a640 1 -- 172.21.15.144:0/1863125043 reap_dead start 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 10 monclient: _finish_auth 0 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.537+0000 7f7021b38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021b38640 1 -- 172.21.15.144:0/1863125043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7008004780 msgr2=0x7f7008004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021b38640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7008004780 0x7f7008004bf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f70180014d0 tx=0x7f701c0a0250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2324aee7ee1bacc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018005890 con 0x7f7008004780 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:27.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 10 monclient: dump: 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f702459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7018005a30 con 0x7f7008004780 2024-09-18T12:30:27.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f702459d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f702459d640 4 set_mon_vals no callback set 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 1 -- 172.21.15.144:0/1863125043 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018005df0 con 0x7f7008004780 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.540+0000 7f7021337640 10 monclient: dump: 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.556 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 10 monclient: shutdown 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 1 -- 172.21.15.144:0/1863125043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7008004780 msgr2=0x7f7008004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7008004780 0x7f7008004bf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f70180014d0 tx=0x7f701c0a0250 comp rx=0 tx=0).stop 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 1 -- 172.21.15.144:0/1863125043 shutdown_connections 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 1 --2- 172.21.15.144:0/1863125043 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7008004780 0x7f7008004bf0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 1 -- 172.21.15.144:0/1863125043 >> 172.21.15.144:0/1863125043 conn(0x7f701c0a5100 msgr2=0x7f701c0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 1 -- 172.21.15.144:0/1863125043 shutdown_connections 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 1 -- 172.21.15.144:0/1863125043 wait complete. 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 10 monclient: build_initial_monmap 2024-09-18T12:30:27.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.541+0000 7f702459d640 10 monclient: monmap: 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 Processor -- start 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 -- start start 2024-09-18T12:30:27.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient: init 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7008004780 0x7f701c141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): picked mon.noname-c con 0x7f7008004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 0x7f701c12f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): picked mon.noname-b con 0x7f701c12ed30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c098c30 0x7f701c0990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): picked mon.noname-a con 0x7f701c098c30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c11c130 con 0x7f701c098c30 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c0a0a30 con 0x7f701c12ed30 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c09e310 con 0x7f7008004780 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f702459d640 10 monclient(hunting): authenticate will time out at 1063.111751s 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 0x7f701c12f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7022b3a640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c098c30 msgr2=0x7f701c0990a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 0x7f701c12f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42240/0 (socket says 172.21.15.144:42240) 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 1 -- 172.21.15.144:0/2301179705 learned_addr learned my addr 172.21.15.144:0/2301179705 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 10 monclient(hunting): get_auth_request con 0x7f701c12ed30 auth_method 0 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:27.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7022b3a640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c098c30 0x7f701c0990a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.542+0000 7f7021b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7022339640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7008004780 0x7f701c141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7022339640 10 monclient(hunting): get_auth_request con 0x7f7008004780 auth_method 0 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7022339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7022339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7022339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 10 monclient(hunting): handle_auth_done global_id 5518 payload 824 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 1 -- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7008004780 msgr2=0x7f701c141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7008004780 0x7f701c141010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 1 -- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c098c30 msgr2=0x7f701c0990a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:27.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c098c30 0x7f701c0990a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 10 monclient: _finish_hunting 0 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 1 monclient: found mon.noname-b 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.543+0000 7f7021b38640 1 -- 172.21.15.144:0/2301179705 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c141e60 con 0x7f701c12ed30 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f7021b38640 1 -- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 msgr2=0x7f701c12f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f7021b38640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 0x7f701c12f1a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f701800c5f0 tx=0x7f7018034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018007720 con 0x7f701c12ed30 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: dump: 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.562 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70180047a0 con 0x7f701c12ed30 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f702459d640 5 monclient: authenticate success, global_id 5518 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:27.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f702459d640 10 monclient: _renew_subs 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018004b20 con 0x7f701c12ed30 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f702459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c11be50 con 0x7f701c12ed30 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f70127fc640 10 monclient: dump: 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f7020b36640 4 set_mon_vals no callback set 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f702459d640 10 monclient: _renew_subs 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f702459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.545+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c09b1e0 con 0x7f701c12ed30 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7018005050 con 0x7f701c12ed30 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f702459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f702459d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f702459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c0636e0 con 0x7f701c12ed30 2024-09-18T12:30:27.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f70127fc640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7000038440 0x7f700003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f701806c520 con 0x7f701c12ed30 2024-09-18T12:30:27.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f7022339640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7000038440 0x7f700003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.546+0000 7f7022339640 10 monclient: get_auth_request con 0x7f7000038440 auth_method 0 2024-09-18T12:30:27.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.549+0000 7f7022339640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7000038440 0x7f700003a8f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:27.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.550+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7018042340 con 0x7f701c12ed30 2024-09-18T12:30:27.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.550+0000 7f70127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.550+0000 7f70127fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:27.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.706+0000 7f702459d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:27.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.706+0000 7f702459d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:27.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.706+0000 7f702459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.706+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f701c064a70 con 0x7f701c12ed30 2024-09-18T12:30:27.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.706+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) v1 ==== 56+0+1173 (secure 0 0 0) 0x7f701803d0b0 con 0x7f701c12ed30 2024-09-18T12:30:27.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.706+0000 7f70127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:27.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.706+0000 7f70127fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:27.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.798+0000 7f702459d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:27.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.798+0000 7f702459d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:27.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.798+0000 7f702459d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.798+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f701c11c370 con 0x7f701c12ed30 2024-09-18T12:30:27.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.799+0000 7f70127fc640 1 -- 172.21.15.144:0/2301179705 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) v1 ==== 56+0+1173 (secure 0 0 0) 0x7f701804c350 con 0x7f701c12ed30 2024-09-18T12:30:27.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.799+0000 7f70127fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:27.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.799+0000 7f70127fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7000038440 msgr2=0x7f700003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7000038440 0x7f700003a8f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c008040 comp rx=0 tx=0).stop 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 10 monclient: shutdown 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 msgr2=0x7f701c12f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 0x7f701c12f1a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f701800c5f0 tx=0x7f7018034000 comp rx=0 tx=0).stop 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 shutdown_connections 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7000038440 0x7f700003a8f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c098c30 0x7f701c0990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f701c12ed30 0x7f701c12f1a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 --2- 172.21.15.144:0/2301179705 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7008004780 0x7f701c141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.800+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 >> 172.21.15.144:0/2301179705 conn(0x7f701c0a5100 msgr2=0x7f701c130150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:27.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.801+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 shutdown_connections 2024-09-18T12:30:27.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.801+0000 7f702459d640 1 -- 172.21.15.144:0/2301179705 wait complete. 2024-09-18T12:30:27.810 INFO:tasks.workunit.client.0.smithi144.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2024-09-18T12:30:27.810 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm osd.11 2024-09-18T12:30:27.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.883+0000 7f0f21109640 10 monclient: get_monmap_and_config 2024-09-18T12:30:27.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.883+0000 7f0f21109640 10 monclient: build_initial_monmap 2024-09-18T12:30:27.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.883+0000 7f0f21109640 10 monclient: monmap: 2024-09-18T12:30:27.885 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:27.885 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.885 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:27.883979+0000 2024-09-18T12:30:27.886 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:27.883979+0000 2024-09-18T12:30:27.886 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:27.886 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.886 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:27.886 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:27.886 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:27.887 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:27.887 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:27.887 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.883+0000 7f0f21109640 1 Processor -- start 2024-09-18T12:30:27.887 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.883+0000 7f0f21109640 1 -- start start 2024-09-18T12:30:27.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.883+0000 7f0f21109640 10 monclient: init 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f1c069bd0 0x7f0f1c1320a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient(hunting): picked mon.noname-a con 0x7f0f1c069bd0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f1c098330 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f1c136d50 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c096bd0 con 0x7f0f1c098330 2024-09-18T12:30:27.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c098e80 con 0x7f0f1c136d50 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1c131db0 con 0x7f0f1c069bd0 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f21109640 10 monclient(hunting): authenticate will time out at 1063.453337s 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f1ad9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f0f1c069bd0 msgr2=0x7f0f1c1320a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f1a59c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f0f1c098330 legacy=0x7f0f1c098770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f1ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f1c069bd0 0x7f0f1c1320a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f1b59e640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f0f1c136d50 0x7f0f1c139130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39034/0 (socket says 172.21.15.144:39034) 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.884+0000 7f0f1b59e640 1 -- 172.21.15.144:0/398484124 learned_addr learned my addr 172.21.15.144:0/398484124 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.886+0000 7f0f1b59e640 1 -- 172.21.15.144:0/398484124 >> v1:172.21.15.144:6789/0 conn(0x7f0f1c136d50 legacy=0x7f0f1c139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.886+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328121549 0 0) 0x7f0f1c098e80 con 0x7f0f1c136d50 2024-09-18T12:30:27.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.886+0000 7f0f19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.886+0000 7f0f19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.886+0000 7f0f19d9b640 10 monclient(hunting): my global_id is 5521 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.886+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f040034e0 con 0x7f0f1c136d50 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2989380111 0 0) 0x7f0f040034e0 con 0x7f0f1c136d50 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 >> v2:172.21.15.116:3300/0 conn(0x7f0f1c069bd0 msgr2=0x7f0f1c1320a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 --2- 172.21.15.144:0/398484124 >> v2:172.21.15.116:3300/0 conn(0x7f0f1c069bd0 0x7f0f1c1320a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 >> v1:172.21.15.116:6789/0 conn(0x7f0f1c098330 legacy=0x7f0f1c098770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c09d7f0 con 0x7f0f1c136d50 2024-09-18T12:30:27.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: _finish_auth 0 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f0f14001b20 con 0x7f0f1c136d50 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f21109640 5 monclient: authenticate success, global_id 5521 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f21109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: dump: 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.911 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 >> v1:172.21.15.144:6789/0 conn(0x7f0f1c136d50 legacy=0x7f0f1c139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04003aa0 0x7f0f04003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): picked mon.a con 0x7f0f04003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f04004780 0x7f0f04004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): picked mon.c con 0x7f0f04004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:27.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f040099c0 0x7f0f0400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): picked mon.b con 0x7f0f040099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c131db0 con 0x7f0f04003aa0 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0400c3c0 con 0x7f0f040099c0 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0400c580 con 0x7f0f04004780 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f19d9b640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:27.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f21109640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f040099c0 0x7f0f0400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1ad9d640 1 -- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04003aa0 msgr2=0x7f0f04003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): get_auth_request con 0x7f0f040099c0 auth_method 0 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1ad9d640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04003aa0 0x7f0f04003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:27.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient(hunting): handle_auth_done global_id 5521 payload 293 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 1 -- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f04004780 msgr2=0x7f0f04004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f04004780 0x7f0f04004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 1 -- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04003aa0 msgr2=0x7f0f04003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04003aa0 0x7f0f04003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient: _finish_hunting 0 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 1 monclient: found mon.b 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 1 -- 172.21.15.144:0/398484124 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f040034e0 con 0x7f0f040099c0 2024-09-18T12:30:27.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 10 monclient: _finish_auth 0 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.887+0000 7f0f1b59e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.888+0000 7f0f1b59e640 1 -- 172.21.15.144:0/398484124 reap_dead start 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f1b59e640 1 -- 172.21.15.144:0/398484124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f040099c0 msgr2=0x7f0f0400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f1b59e640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f040099c0 0x7f0f0400bda0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c09f950 tx=0x7f0f1400a240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68d7a97caa13eaa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1402b5d0 con 0x7f0f040099c0 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 10 monclient: dump: 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.916 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f21109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1402b750 con 0x7f0f040099c0 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f21109640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:27.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 1 -- 172.21.15.144:0/398484124 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1402bb10 con 0x7f0f040099c0 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f21109640 4 set_mon_vals no callback set 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.890+0000 7f0f19d9b640 10 monclient: dump: 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.918 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.919 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.919 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.919 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 10 monclient: shutdown 2024-09-18T12:30:27.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:27.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 1 -- 172.21.15.144:0/398484124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f040099c0 msgr2=0x7f0f0400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f040099c0 0x7f0f0400bda0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c09f950 tx=0x7f0f1400a240 comp rx=0 tx=0).stop 2024-09-18T12:30:27.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 1 -- 172.21.15.144:0/398484124 shutdown_connections 2024-09-18T12:30:27.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 1 --2- 172.21.15.144:0/398484124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f040099c0 0x7f0f0400bda0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 1 -- 172.21.15.144:0/398484124 >> 172.21.15.144:0/398484124 conn(0x7f0f1c09e9f0 msgr2=0x7f0f1c09edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:27.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 1 -- 172.21.15.144:0/398484124 shutdown_connections 2024-09-18T12:30:27.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 1 -- 172.21.15.144:0/398484124 wait complete. 2024-09-18T12:30:27.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 10 monclient: build_initial_monmap 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.891+0000 7f0f21109640 10 monclient: monmap: 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 Processor -- start 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 -- start start 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient: init 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f040099c0 0x7f0f1c1308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): picked mon.noname-a con 0x7f0f040099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 0x7f0f1c1193e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): picked mon.noname-b con 0x7f0f1c130df0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f1c119920 0x7f0f1c119d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): picked mon.noname-c con 0x7f0f1c119920 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1375f0 con 0x7f0f040099c0 2024-09-18T12:30:27.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c1311c0 con 0x7f0f1c130df0 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c09d5e0 con 0x7f0f1c119920 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f21109640 10 monclient(hunting): authenticate will time out at 1063.461661s 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 0x7f0f1c1193e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 0x7f0f1c1193e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42264/0 (socket says 172.21.15.144:42264) 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 -- 172.21.15.144:0/639491787 learned_addr learned my addr 172.21.15.144:0/639491787 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): get_auth_request con 0x7f0f1c130df0 auth_method 0 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:27.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1ad9d640 1 -- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f040099c0 msgr2=0x7f0f1c1308b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1ad9d640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f040099c0 0x7f0f1c1308b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1b59e640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f1c119920 0x7f0f1c119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1b59e640 10 monclient(hunting): get_auth_request con 0x7f0f1c119920 auth_method 0 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1b59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:27.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient(hunting): handle_auth_done global_id 5524 payload 824 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 -- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f1c119920 msgr2=0x7f0f1c119d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f1c119920 0x7f0f1c119d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 -- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f040099c0 msgr2=0x7f0f1c1308b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f040099c0 0x7f0f1c1308b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 monclient: found mon.noname-b 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.892+0000 7f0f1a59c640 1 -- 172.21.15.144:0/639491787 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c11b3a0 con 0x7f0f1c130df0 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f1a59c640 1 -- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 msgr2=0x7f0f1c1193e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f1a59c640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 0x7f0f1c1193e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000b5c0 tx=0x7f0f10009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10003cc0 con 0x7f0f1c130df0 2024-09-18T12:30:27.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: dump: 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:27.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f10003e40 con 0x7f0f1c130df0 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f21109640 5 monclient: authenticate success, global_id 5524 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f21109640 10 monclient: _renew_subs 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f21109640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c11cf80 con 0x7f0f1c130df0 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10005340 con 0x7f0f1c130df0 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f02ffd640 10 monclient: dump: 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:27.927 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f21109640 10 monclient: _renew_subs 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f21109640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c11ca70 con 0x7f0f1c130df0 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.895+0000 7f0f1959a640 4 set_mon_vals no callback set 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0f1001a460 con 0x7f0f1c130df0 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f21109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f21109640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f21109640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:27.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c11cc00 con 0x7f0f1c130df0 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f02ffd640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f08038440 0x7f0f0803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f0f1004b8e0 con 0x7f0f1c130df0 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f1ad9d640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f08038440 0x7f0f0803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.896+0000 7f0f1ad9d640 10 monclient: get_auth_request con 0x7f0f08038440 auth_method 0 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.898+0000 7f0f1ad9d640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f08038440 0x7f0f0803a8f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c006fd0 tx=0x7f0f0c006dd0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.900+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0f1001a710 con 0x7f0f1c130df0 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.900+0000 7f0f02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:27.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:27.900+0000 7f0f02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:28.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:28.060+0000 7f0f21109640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-18T12:30:28.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:28.060+0000 7f0f21109640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-18T12:30:28.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:28.060+0000 7f0f21109640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:28.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:28.060+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7f0f1c11aef0 con 0x7f0f1c130df0 2024-09-18T12:30:28.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:28.966+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 removed item id 11 name 'osd.11' from crush map v112) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0f10017310 con 0x7f0f1c130df0 2024-09-18T12:30:28.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:28.966+0000 7f0f02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-18T12:30:28.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:28.966+0000 7f0f02ffd640 10 monclient: _finish_command 2 = system:0 removed item id 11 name 'osd.11' from crush map 2024-09-18T12:30:29.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.065+0000 7f0f21109640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-18T12:30:29.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.065+0000 7f0f21109640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-18T12:30:29.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.065+0000 7f0f21109640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:29.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.065+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) v1 -- 0x7f0f1c0008d0 con 0x7f0f1c130df0 2024-09-18T12:30:29.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.065+0000 7f0f02ffd640 1 -- 172.21.15.144:0/639491787 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 device 'osd.11' does not appear in the crush map v112) v1 ==== 126+0+0 (secure 0 0 0) 0x7f0f100174f0 con 0x7f0f1c130df0 2024-09-18T12:30:29.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.065+0000 7f0f02ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-18T12:30:29.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.065+0000 7f0f02ffd640 10 monclient: _finish_command 3 = system:0 device 'osd.11' does not appear in the crush map 2024-09-18T12:30:29.067 INFO:tasks.workunit.client.0.smithi144.stderr:device 'osd.11' does not appear in the crush map 2024-09-18T12:30:29.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f08038440 msgr2=0x7f0f0803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f08038440 0x7f0f0803a8f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c006fd0 tx=0x7f0f0c006dd0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 10 monclient: shutdown 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 msgr2=0x7f0f1c1193e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 0x7f0f1c1193e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000b5c0 tx=0x7f0f10009120 comp rx=0 tx=0).stop 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 shutdown_connections 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f08038440 0x7f0f0803a8f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f1c119920 0x7f0f1c119d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f1c130df0 0x7f0f1c1193e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 --2- 172.21.15.144:0/639491787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f040099c0 0x7f0f1c1308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 >> 172.21.15.144:0/639491787 conn(0x7f0f1c09e9f0 msgr2=0x7f0f1c097940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:29.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 shutdown_connections 2024-09-18T12:30:29.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.067+0000 7f0f21109640 1 -- 172.21.15.144:0/639491787 wait complete. 2024-09-18T12:30:29.078 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd rm osd.11 2024-09-18T12:30:29.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.150+0000 7f232b42b640 10 monclient: get_monmap_and_config 2024-09-18T12:30:29.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.150+0000 7f232b42b640 10 monclient: build_initial_monmap 2024-09-18T12:30:29.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.150+0000 7f232b42b640 10 monclient: monmap: 2024-09-18T12:30:29.151 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:29.151 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:29.151 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:29.151215+0000 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:29.151215+0000 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:29.152 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.150+0000 7f232b42b640 1 Processor -- start 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 1 -- start start 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient: init 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23241319c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f232409eb30 0x7f232409ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient(hunting): picked mon.noname-c con 0x7f232409eb30 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2324136cf0 0x7f23241390d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient(hunting): picked mon.noname-a con 0x7f2324136cf0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:29.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2324131d90 con 0x7f23241319c0 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232406a260 con 0x7f2324136cf0 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23241326f0 con 0x7f232409eb30 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f232b42b640 10 monclient(hunting): authenticate will time out at 1064.720588s 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23289c6640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f232409eb30 0x7f232409ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23299c8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f2324136cf0 msgr2=0x7f23241390d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:29.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23289c6640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f232409eb30 0x7f232409ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42268/0 (socket says 172.21.15.144:42268) 2024-09-18T12:30:29.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23289c6640 1 -- 172.21.15.144:0/3230026593 learned_addr learned my addr 172.21.15.144:0/3230026593 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:29.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23289c6640 10 monclient(hunting): get_auth_request con 0x7f232409eb30 auth_method 0 2024-09-18T12:30:29.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23299c8640 1 --2- 172.21.15.144:0/3230026593 >> v2:172.21.15.116:3300/0 conn(0x7f2324136cf0 0x7f23241390d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:29.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23289c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:29.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23289c6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:29.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f23289c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:29.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.151+0000 7f231bfff640 1 -- 172.21.15.144:0/3230026593 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:30:29.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.153+0000 7f23291c7640 1 -- 172.21.15.144:0/3230026593 >> v1:172.21.15.144:6789/0 conn(0x7f23241319c0 legacy=0x7f2324131fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:29.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:29.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:29.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:29.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 10 monclient(hunting): handle_auth_done global_id 5527 payload 274 2024-09-18T12:30:29.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 1 -- 172.21.15.144:0/3230026593 >> v2:172.21.15.116:3300/0 conn(0x7f2324136cf0 msgr2=0x7f23241390d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:29.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 1 --2- 172.21.15.144:0/3230026593 >> v2:172.21.15.116:3300/0 conn(0x7f2324136cf0 0x7f23241390d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 1 -- 172.21.15.144:0/3230026593 >> v1:172.21.15.144:6789/0 conn(0x7f23241319c0 legacy=0x7f2324131fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:29.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 10 monclient: _finish_hunting 0 2024-09-18T12:30:29.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 1 monclient: found mon.noname-c 2024-09-18T12:30:29.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:29.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:29.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.154+0000 7f23289c6640 1 -- 172.21.15.144:0/3230026593 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240a0050 con 0x7f232409eb30 2024-09-18T12:30:29.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f23289c6640 1 -- 172.21.15.144:0/3230026593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232409eb30 msgr2=0x7f232409ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:29.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f23289c6640 1 --2- 172.21.15.144:0/3230026593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232409eb30 0x7f232409ef30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2314009990 tx=0x7f23140091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf197d41e87a6000 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:29.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 1 -- 172.21.15.144:0/3230026593 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231402f4b0 con 0x7f232409eb30 2024-09-18T12:30:29.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:29.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:29.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: dump: 2024-09-18T12:30:29.165 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:29.165 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:29.168 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: _finish_auth 0 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 1 -- 172.21.15.144:0/3230026593 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231402f610 con 0x7f232409eb30 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 5 monclient: authenticate success, global_id 5527 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:29.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 1 -- 172.21.15.144:0/3230026593 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231402f9b0 con 0x7f232409eb30 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 4 set_mon_vals no callback set 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f231bfff640 10 monclient: dump: 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:29.170 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 10 monclient: shutdown 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 -- 172.21.15.144:0/3230026593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232409eb30 msgr2=0x7f232409ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 --2- 172.21.15.144:0/3230026593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232409eb30 0x7f232409ef30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2314009990 tx=0x7f23140091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 -- 172.21.15.144:0/3230026593 shutdown_connections 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 --2- 172.21.15.144:0/3230026593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f232409eb30 0x7f232409ef30 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 --2- 172.21.15.144:0/3230026593 >> v2:172.21.15.116:3300/0 conn(0x7f2324136cf0 0x7f23241390d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 -- 172.21.15.144:0/3230026593 >> 172.21.15.144:0/3230026593 conn(0x7f2324071370 msgr2=0x7f23240a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 -- 172.21.15.144:0/3230026593 shutdown_connections 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.157+0000 7f232b42b640 1 -- 172.21.15.144:0/3230026593 wait complete. 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.158+0000 7f232b42b640 10 monclient: build_initial_monmap 2024-09-18T12:30:29.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.158+0000 7f232b42b640 10 monclient: monmap: 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.158+0000 7f232b42b640 1 Processor -- start 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.158+0000 7f232b42b640 1 -- start start 2024-09-18T12:30:29.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.158+0000 7f232b42b640 10 monclient: init 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232409eb30 0x7f2324130d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): picked mon.noname-a con 0x7f232409eb30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 0x7f2324141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): picked mon.noname-b con 0x7f23241319c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2324136cf0 0x7f2324141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): picked mon.noname-c con 0x7f2324136cf0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232409bee0 con 0x7f232409eb30 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232411b880 con 0x7f23241319c0 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:29.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232409e750 con 0x7f2324136cf0 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f232b42b640 10 monclient(hunting): authenticate will time out at 1064.728467s 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 0x7f2324141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 0x7f2324141340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42276/0 (socket says 172.21.15.144:42276) 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 -- 172.21.15.144:0/2508713216 learned_addr learned my addr 172.21.15.144:0/2508713216 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): get_auth_request con 0x7f23241319c0 auth_method 0 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23291c7640 1 -- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232409eb30 msgr2=0x7f2324130d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23291c7640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232409eb30 0x7f2324130d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:29.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23299c8640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2324136cf0 0x7f2324141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23299c8640 10 monclient(hunting): get_auth_request con 0x7f2324136cf0 auth_method 0 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23299c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23299c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23299c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient(hunting): handle_auth_done global_id 5533 payload 824 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 -- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2324136cf0 msgr2=0x7f2324141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2324136cf0 0x7f2324141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 -- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232409eb30 msgr2=0x7f2324130d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232409eb30 0x7f2324130d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient: _finish_hunting 0 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 monclient: found mon.noname-b 2024-09-18T12:30:29.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.159+0000 7f23289c6640 1 -- 172.21.15.144:0/2508713216 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2314009450 con 0x7f23241319c0 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23289c6640 1 -- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 msgr2=0x7f2324141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23289c6640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 0x7f2324141340 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2314035040 tx=0x7f2314032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314032a30 con 0x7f23241319c0 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: dump: 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.176 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: _finish_auth 0 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f232b42b640 5 monclient: authenticate success, global_id 5533 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2314032bb0 con 0x7f23241319c0 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f232b42b640 10 monclient: _renew_subs 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f232b42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:29.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232411ce30 con 0x7f23241319c0 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314044410 con 0x7f23241319c0 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f23197fa640 10 monclient: dump: 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:29.178 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f231b7fe640 4 set_mon_vals no callback set 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f232b42b640 10 monclient: _renew_subs 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f232b42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.162+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2324131260 con 0x7f23241319c0 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2314039040 con 0x7f23241319c0 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f232b42b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f232b42b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f232b42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232404bf60 con 0x7f23241319c0 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f23197fa640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2308038440 0x7f230803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:29.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f231406bc20 con 0x7f23241319c0 2024-09-18T12:30:29.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f23291c7640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2308038440 0x7f230803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:29.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.163+0000 7f23291c7640 10 monclient: get_auth_request con 0x7f2308038440 auth_method 0 2024-09-18T12:30:29.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.165+0000 7f23291c7640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2308038440 0x7f230803a8f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f23200014b0 tx=0x7f2320007a50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:29.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.167+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f231403aeb0 con 0x7f23241319c0 2024-09-18T12:30:29.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.167+0000 7f23197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:29.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.167+0000 7f23197fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:29.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.330+0000 7f232b42b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-18T12:30:29.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.330+0000 7f232b42b640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-18T12:30:29.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.330+0000 7f232b42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:29.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.330+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7f23240a1320 con 0x7f23241319c0 2024-09-18T12:30:29.331 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-09-18T12:30:29.330+0000 7fd9503f1640 -1 mon.b@1(leader).osd e112 definitely_dead 0 2024-09-18T12:30:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.967+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 removed osd.11 v113) v1 ==== 87+0+0 (secure 0 0 0) 0x7f231403f090 con 0x7f23241319c0 2024-09-18T12:30:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.967+0000 7f23197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-18T12:30:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:29.967+0000 7f23197fa640 10 monclient: _finish_command 2 = system:0 removed osd.11 2024-09-18T12:30:30.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.070+0000 7f232b42b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-18T12:30:30.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.070+0000 7f232b42b640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-18T12:30:30.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.070+0000 7f232b42b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:30.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.070+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) v1 -- 0x7f23240a39a0 con 0x7f23241319c0 2024-09-18T12:30:30.072 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-09-18T12:30:30.070+0000 7fd9503f1640 -1 mon.b@1(leader).osd e113 definitely_dead 0 2024-09-18T12:30:30.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.070+0000 7f23197fa640 1 -- 172.21.15.144:0/2508713216 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 osd.11 does not exist. v113) v1 ==== 96+0+0 (secure 0 0 0) 0x7f2314037020 con 0x7f23241319c0 2024-09-18T12:30:30.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.070+0000 7f23197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-18T12:30:30.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.070+0000 7f23197fa640 10 monclient: _finish_command 3 = system:0 osd.11 does not exist. 2024-09-18T12:30:30.073 INFO:tasks.workunit.client.0.smithi144.stderr:osd.11 does not exist. 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2308038440 msgr2=0x7f230803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2308038440 0x7f230803a8f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f23200014b0 tx=0x7f2320007a50 comp rx=0 tx=0).stop 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 10 monclient: shutdown 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 msgr2=0x7f2324141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 0x7f2324141340 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2314035040 tx=0x7f2314032000 comp rx=0 tx=0).stop 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 shutdown_connections 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2308038440 0x7f230803a8f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2324136cf0 0x7f2324141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23241319c0 0x7f2324141340 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 --2- 172.21.15.144:0/2508713216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232409eb30 0x7f2324130d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 >> 172.21.15.144:0/2508713216 conn(0x7f2324071370 msgr2=0x7f23240a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:30.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 shutdown_connections 2024-09-18T12:30:30.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.072+0000 7f232b42b640 1 -- 172.21.15.144:0/2508713216 wait complete. 2024-09-18T12:30:30.082 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 2024-09-18T12:30:30.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.152+0000 7f3abaf09640 10 monclient: get_monmap_and_config 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.152+0000 7f3abaf09640 10 monclient: build_initial_monmap 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.152+0000 7f3abaf09640 10 monclient: monmap: 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:30.153608+0000 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:30.153608+0000 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:30.154 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:30.155 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:30.155 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:30.155 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:30.155 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:30.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 1 Processor -- start 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 1 -- start start 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient: init 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ab409c230 0x7f3ab409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient(hunting): picked mon.noname-a con 0x7f3ab409c230 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ab409c600 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab409e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:30.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient(hunting): picked mon.noname-b con 0x7f3ab409d170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab409b1d0 con 0x7f3ab409c600 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab409d570 con 0x7f3ab409c230 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab409f5b0 con 0x7f3ab409d170 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.153+0000 7f3abaf09640 10 monclient(hunting): authenticate will time out at 1065.722965s 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab8ca5640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f3ab409c230 msgr2=0x7f3ab409cc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3aabfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f3ab409c600 0x7f3ab409da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39040/0 (socket says 172.21.15.144:39040) 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3aabfff640 1 -- 172.21.15.144:0/3517812376 learned_addr learned my addr 172.21.15.144:0/3517812376 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:30.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab8ca5640 1 --2- 172.21.15.144:0/3517812376 >> v2:172.21.15.116:3300/0 conn(0x7f3ab409c230 0x7f3ab409cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:30.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 1 --2- 172.21.15.144:0/3517812376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab409e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:30.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): get_auth_request con 0x7f3ab409d170 auth_method 0 2024-09-18T12:30:30.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:30.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:30.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient(hunting): handle_auth_done global_id 64376 payload 274 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 1 -- 172.21.15.144:0/3517812376 >> v2:172.21.15.116:3300/0 conn(0x7f3ab409c230 msgr2=0x7f3ab409cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 1 --2- 172.21.15.144:0/3517812376 >> v2:172.21.15.116:3300/0 conn(0x7f3ab409c230 0x7f3ab409cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 1 -- 172.21.15.144:0/3517812376 >> v1:172.21.15.144:6789/0 conn(0x7f3ab409c600 legacy=0x7f3ab409da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient: _finish_hunting 0 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 1 monclient: found mon.noname-b 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:30.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.154+0000 7f3ab94a6640 1 -- 172.21.15.144:0/3517812376 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab409b470 con 0x7f3ab409d170 2024-09-18T12:30:30.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3ab94a6640 1 -- 172.21.15.144:0/3517812376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 msgr2=0x7f3ab409e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:30.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3ab94a6640 1 --2- 172.21.15.144:0/3517812376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab409e170 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009e60 tx=0x7f3aac009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=454e4d409a2ba3d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:30.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 1 -- 172.21.15.144:0/3517812376 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac007840 con 0x7f3ab409d170 2024-09-18T12:30:30.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:30.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: dump: 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:30.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 1 -- 172.21.15.144:0/3517812376 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac02d050 con 0x7f3ab409d170 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 1 -- 172.21.15.144:0/3517812376 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac007bd0 con 0x7f3ab409d170 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3abaf09640 5 monclient: authenticate success, global_id 64376 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3abaf09640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3abaf09640 4 set_mon_vals no callback set 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.157+0000 7f3aab7fe640 10 monclient: dump: 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.170 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 10 monclient: shutdown 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 -- 172.21.15.144:0/3517812376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 msgr2=0x7f3ab409e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 --2- 172.21.15.144:0/3517812376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab409e170 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009e60 tx=0x7f3aac009780 comp rx=0 tx=0).stop 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 -- 172.21.15.144:0/3517812376 shutdown_connections 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 --2- 172.21.15.144:0/3517812376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab409e170 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 --2- 172.21.15.144:0/3517812376 >> v2:172.21.15.116:3300/0 conn(0x7f3ab409c230 0x7f3ab409cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 -- 172.21.15.144:0/3517812376 >> 172.21.15.144:0/3517812376 conn(0x7f3ab4071350 msgr2=0x7f3ab40a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:30.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 -- 172.21.15.144:0/3517812376 shutdown_connections 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 1 -- 172.21.15.144:0/3517812376 wait complete. 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 10 monclient: build_initial_monmap 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.158+0000 7f3abaf09640 10 monclient: monmap: 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:30.172 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 Processor -- start 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 -- start start 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient: init 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab409c230 0x7f3ab413d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): picked mon.noname-a con 0x7f3ab409c230 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 0x7f3ab413ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): picked mon.noname-b con 0x7f3ab409c600 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab4140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): picked mon.noname-c con 0x7f3ab409d170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:30.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab41204c0 con 0x7f3ab409c230 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab406ec10 con 0x7f3ab409c600 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40a29a0 con 0x7f3ab409d170 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3abaf09640 10 monclient(hunting): authenticate will time out at 1065.728881s 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3ab8ca5640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab409c230 msgr2=0x7f3ab413d7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3aabfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 0x7f3ab413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.159+0000 7f3ab8ca5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab409c230 0x7f3ab413d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 0x7f3ab413ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42288/0 (socket says 172.21.15.144:42288) 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 -- 172.21.15.144:0/1025416635 learned_addr learned my addr 172.21.15.144:0/1025416635 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): get_auth_request con 0x7f3ab409c600 auth_method 0 2024-09-18T12:30:30.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab4140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 10 monclient(hunting): get_auth_request con 0x7f3ab409d170 auth_method 0 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:30.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3ab94a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient(hunting): handle_auth_done global_id 5536 payload 824 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 -- 172.21.15.144:0/1025416635 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 msgr2=0x7f3ab4140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab4140d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 -- 172.21.15.144:0/1025416635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab409c230 msgr2=0x7f3ab413d7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab409c230 0x7f3ab413d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 monclient: found mon.noname-b 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.160+0000 7f3aabfff640 1 -- 172.21.15.144:0/1025416635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac009a20 con 0x7f3ab409c600 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aabfff640 1 -- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 msgr2=0x7f3ab413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aabfff640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 0x7f3ab413ce80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400c610 tx=0x7f3aa4004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4005cb0 con 0x7f3ab409c600 2024-09-18T12:30:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: dump: 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:30.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa4005e30 con 0x7f3ab409c600 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3abaf09640 5 monclient: authenticate success, global_id 5536 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3abaf09640 10 monclient: _renew_subs 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4007340 con 0x7f3ab409c600 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3abaf09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab4122200 con 0x7f3ab409c600 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aa8ff9640 10 monclient: dump: 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:30.178 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.162+0000 7f3aaaffd640 4 set_mon_vals no callback set 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3abaf09640 10 monclient: _renew_subs 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3abaf09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab413d3c0 con 0x7f3ab409c600 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3aa401b400 con 0x7f3ab409c600 2024-09-18T12:30:30.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3aa8ff9640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3aa0038440 0x7f3aa003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6004+0+0 (secure 0 0 0) 0x7f3aa404b880 con 0x7f3ab409c600 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3abaf09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3abaf09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3abaf09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3ab8ca5640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3aa0038440 0x7f3aa003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.163+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab40636e0 con 0x7f3ab409c600 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.164+0000 7f3ab8ca5640 10 monclient: get_auth_request con 0x7f3aa0038440 auth_method 0 2024-09-18T12:30:30.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.166+0000 7f3ab8ca5640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3aa0038440 0x7f3aa003a8f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3ab00038f0 tx=0x7f3ab0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:30.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.168+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3aa4012760 con 0x7f3ab409c600 2024-09-18T12:30:30.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.168+0000 7f3aa8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:30.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.168+0000 7f3aa8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:30.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.335+0000 7f3abaf09640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:30:30.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.335+0000 7f3abaf09640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T12:30:30.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.335+0000 7f3abaf09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:30.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.335+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f3ab4121f80 con 0x7f3ab409c600 2024-09-18T12:30:30.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.986+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v114) v1 ==== 58+0+3 (secure 0 0 0) 0x7f3aa401a360 con 0x7f3ab409c600 2024-09-18T12:30:30.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.986+0000 7f3aa8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T12:30:30.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:30.986+0000 7f3aa8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:31.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.089+0000 7f3abaf09640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:30:31.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.089+0000 7f3abaf09640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T12:30:31.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.089+0000 7f3abaf09640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:31.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.089+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f3ab4072640 con 0x7f3ab409c600 2024-09-18T12:30:31.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.576+0000 7f3aa8ff9640 1 -- 172.21.15.144:0/1025416635 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v115) v1 ==== 58+0+3 (secure 0 0 0) 0x7f3aa401a940 con 0x7f3ab409c600 2024-09-18T12:30:31.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.576+0000 7f3aa8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T12:30:31.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.576+0000 7f3aa8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3aa0038440 msgr2=0x7f3aa003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3aa0038440 0x7f3aa003a8f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3ab00038f0 tx=0x7f3ab0006d80 comp rx=0 tx=0).stop 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 10 monclient: shutdown 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 msgr2=0x7f3ab413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 0x7f3ab413ce80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400c610 tx=0x7f3aa4004960 comp rx=0 tx=0).stop 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 shutdown_connections 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3aa0038440 0x7f3aa003a8f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ab409d170 0x7f3ab4140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ab409c600 0x7f3ab413ce80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 --2- 172.21.15.144:0/1025416635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ab409c230 0x7f3ab413d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 >> 172.21.15.144:0/1025416635 conn(0x7f3ab4071350 msgr2=0x7f3ab4068a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:31.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 shutdown_connections 2024-09-18T12:30:31.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.579+0000 7f3abaf09640 1 -- 172.21.15.144:0/1025416635 wait complete. 2024-09-18T12:30:31.590 INFO:tasks.workunit.client.0.smithi144.stderr:+ o4=12 2024-09-18T12:30:31.590 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 2024-09-18T12:30:31.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.661+0000 7f56d0294640 10 monclient: get_monmap_and_config 2024-09-18T12:30:31.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.661+0000 7f56d0294640 10 monclient: build_initial_monmap 2024-09-18T12:30:31.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.661+0000 7f56d0294640 10 monclient: monmap: 2024-09-18T12:30:31.663 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:31.663 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:31.663 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:31.662845+0000 2024-09-18T12:30:31.663 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:31.662845+0000 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:31.664 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:31.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.662+0000 7f56d0294640 1 Processor -- start 2024-09-18T12:30:31.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.662+0000 7f56d0294640 1 -- start start 2024-09-18T12:30:31.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.662+0000 7f56d0294640 10 monclient: init 2024-09-18T12:30:31.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:31.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T12:30:31.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f56c8069bd0 0x7f56c8132190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:31.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient(hunting): picked mon.noname-c con 0x7f56c8069bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:31.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56c8070920 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:31.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f56c8071430 0x7f56c8139040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:31.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient(hunting): picked mon.noname-a con 0x7f56c8071430 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:31.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c809e560 con 0x7f56c8070920 2024-09-18T12:30:31.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c809cea0 con 0x7f56c8071430 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c809e280 con 0x7f56c8069bd0 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56d0294640 10 monclient(hunting): authenticate will time out at 1067.232305s 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f56c8069bd0 0x7f56c8132190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce831640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f56c8071430 msgr2=0x7f56c8139040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f56c8069bd0 0x7f56c8132190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42300/0 (socket says 172.21.15.144:42300) 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce831640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f56c8071430 0x7f56c8139040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 1 -- 172.21.15.144:0/3029586101 learned_addr learned my addr 172.21.15.144:0/3029586101 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 10 monclient(hunting): get_auth_request con 0x7f56c8069bd0 auth_method 0 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:31.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.663+0000 7f56ce030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56cd82f640 1 -- 172.21.15.144:0/3029586101 >> v1:172.21.15.144:6789/0 conn(0x7f56c8070920 legacy=0x7f56c8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 10 monclient(hunting): handle_auth_done global_id 5539 payload 274 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 1 -- 172.21.15.144:0/3029586101 >> v2:172.21.15.116:3300/0 conn(0x7f56c8071430 msgr2=0x7f56c8139040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 1 --2- 172.21.15.144:0/3029586101 >> v2:172.21.15.116:3300/0 conn(0x7f56c8071430 0x7f56c8139040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 1 -- 172.21.15.144:0/3029586101 >> v1:172.21.15.144:6789/0 conn(0x7f56c8070920 legacy=0x7f56c8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:31.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 10 monclient: _finish_hunting 0 2024-09-18T12:30:31.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 1 monclient: found mon.noname-c 2024-09-18T12:30:31.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:31.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:31.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.665+0000 7f56ce030640 1 -- 172.21.15.144:0/3029586101 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c81326d0 con 0x7f56c8069bd0 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56ce030640 1 -- 172.21.15.144:0/3029586101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8069bd0 msgr2=0x7f56c8132190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56ce030640 1 --2- 172.21.15.144:0/3029586101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8069bd0 0x7f56c8132190 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f56c804ca20 tx=0x7f56c40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=939939de92e9f8c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 1 -- 172.21.15.144:0/3029586101 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c402f4b0 con 0x7f56c8069bd0 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: dump: 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:31.677 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: _finish_auth 0 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 1 -- 172.21.15.144:0/3029586101 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c402f610 con 0x7f56c8069bd0 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56d0294640 5 monclient: authenticate success, global_id 5539 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56d0294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:31.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 1 -- 172.21.15.144:0/3029586101 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c402f9f0 con 0x7f56c8069bd0 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56d0294640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56d0294640 4 set_mon_vals no callback set 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.668+0000 7f56cd02e640 10 monclient: dump: 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:31.679 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 10 monclient: shutdown 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 -- 172.21.15.144:0/3029586101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8069bd0 msgr2=0x7f56c8132190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 --2- 172.21.15.144:0/3029586101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8069bd0 0x7f56c8132190 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f56c804ca20 tx=0x7f56c40091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 -- 172.21.15.144:0/3029586101 shutdown_connections 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 --2- 172.21.15.144:0/3029586101 >> v2:172.21.15.116:3300/0 conn(0x7f56c8071430 0x7f56c8139040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 --2- 172.21.15.144:0/3029586101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8069bd0 0x7f56c8132190 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 -- 172.21.15.144:0/3029586101 >> 172.21.15.144:0/3029586101 conn(0x7f56c809ea70 msgr2=0x7f56c809ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 -- 172.21.15.144:0/3029586101 shutdown_connections 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.669+0000 7f56d0294640 1 -- 172.21.15.144:0/3029586101 wait complete. 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient: build_initial_monmap 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient: monmap: 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:31.681 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 Processor -- start 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 -- start start 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient: init 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56c8069bd0 0x7f56c8136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): picked mon.noname-c con 0x7f56c8069bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 0x7f56c8136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): picked mon.noname-b con 0x7f56c8070920 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c8071430 0x7f56c81374f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): picked mon.noname-a con 0x7f56c8071430 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:31.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c80a0120 con 0x7f56c8071430 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8119db0 con 0x7f56c8070920 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8118be0 con 0x7f56c8069bd0 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56d0294640 10 monclient(hunting): authenticate will time out at 1067.239884s 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56cd82f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 0x7f56c8136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56cd82f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 0x7f56c8136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42310/0 (socket says 172.21.15.144:42310) 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56ce831640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c8071430 msgr2=0x7f56c81374f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56cd82f640 1 -- 172.21.15.144:0/1876931472 learned_addr learned my addr 172.21.15.144:0/1876931472 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.670+0000 7f56cd82f640 10 monclient(hunting): get_auth_request con 0x7f56c8070920 auth_method 0 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:31.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56ce831640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c8071430 0x7f56c81374f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56ce030640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56c8069bd0 0x7f56c8136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56ce030640 10 monclient(hunting): get_auth_request con 0x7f56c8069bd0 auth_method 0 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56ce030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56ce030640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56ce030640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient(hunting): handle_auth_done global_id 5542 payload 824 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 1 -- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56c8069bd0 msgr2=0x7f56c8136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:31.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56c8069bd0 0x7f56c8136a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 1 -- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c8071430 msgr2=0x7f56c81374f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c8071430 0x7f56c81374f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient: _finish_hunting 0 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 1 monclient: found mon.noname-b 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.671+0000 7f56cd82f640 1 -- 172.21.15.144:0/1876931472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4009450 con 0x7f56c8070920 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56cd82f640 1 -- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 msgr2=0x7f56c8136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56cd82f640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 0x7f56c8136fb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c590 tx=0x7f56b8007d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8007510 con 0x7f56c8070920 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:31.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: dump: 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:31.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56d0294640 5 monclient: authenticate success, global_id 5542 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b8005780 con 0x7f56c8070920 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56d0294640 10 monclient: _renew_subs 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56d0294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c811a060 con 0x7f56c8070920 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8005b40 con 0x7f56c8070920 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56be7fc640 10 monclient: dump: 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:31.687 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56cc82d640 4 set_mon_vals no callback set 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56d0294640 10 monclient: _renew_subs 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56d0294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.673+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c8130ba0 con 0x7f56c8070920 2024-09-18T12:30:31.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f56b801b3b0 con 0x7f56c8070920 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56d0294640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56d0294640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56d0294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c80636e0 con 0x7f56c8070920 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56be7fc640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56ac038440 0x7f56ac03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6123+0+0 (secure 0 0 0) 0x7f56b804b0a0 con 0x7f56c8070920 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56ce030640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56ac038440 0x7f56ac03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:31.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.674+0000 7f56ce030640 10 monclient: get_auth_request con 0x7f56ac038440 auth_method 0 2024-09-18T12:30:31.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.677+0000 7f56ce030640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56ac038440 0x7f56ac03a8f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f56c4034000 tx=0x7f56c4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:31.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.679+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f56b8018820 con 0x7f56c8070920 2024-09-18T12:30:31.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.679+0000 7f56be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:31.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.679+0000 7f56be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:31.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.844+0000 7f56d0294640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:30:31.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.844+0000 7f56d0294640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T12:30:31.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.844+0000 7f56d0294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:31.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:31.844+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f56c8064c00 con 0x7f56c8070920 2024-09-18T12:30:32.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:32.578+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v116) v1 ==== 58+0+3 (secure 0 0 0) 0x7f56b8017080 con 0x7f56c8070920 2024-09-18T12:30:32.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:32.578+0000 7f56be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T12:30:32.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:32.578+0000 7f56be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:32.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:32.683+0000 7f56d0294640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T12:30:32.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:32.683+0000 7f56d0294640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T12:30:32.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:32.683+0000 7f56d0294640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:32.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:32.683+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f56c8137a30 con 0x7f56c8070920 2024-09-18T12:30:33.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.578+0000 7f56be7fc640 1 -- 172.21.15.144:0/1876931472 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v117) v1 ==== 58+0+3 (secure 0 0 0) 0x7f56b8018260 con 0x7f56c8070920 2024-09-18T12:30:33.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.578+0000 7f56be7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T12:30:33.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.578+0000 7f56be7fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56ac038440 msgr2=0x7f56ac03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56ac038440 0x7f56ac03a8f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f56c4034000 tx=0x7f56c4032000 comp rx=0 tx=0).stop 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 10 monclient: shutdown 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 msgr2=0x7f56c8136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 0x7f56c8136fb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c590 tx=0x7f56b8007d60 comp rx=0 tx=0).stop 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 shutdown_connections 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f56ac038440 0x7f56ac03a8f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c8071430 0x7f56c81374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f56c8070920 0x7f56c8136fb0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 --2- 172.21.15.144:0/1876931472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56c8069bd0 0x7f56c8136a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.580+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 >> 172.21.15.144:0/1876931472 conn(0x7f56c809ea70 msgr2=0x7f56c8131590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:33.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.581+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 shutdown_connections 2024-09-18T12:30:33.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.581+0000 7f56d0294640 1 -- 172.21.15.144:0/1876931472 wait complete. 2024-09-18T12:30:33.593 INFO:tasks.workunit.client.0.smithi144.stderr:+ o5=14 2024-09-18T12:30:33.593 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2024-09-18T12:30:33.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.673+0000 7f175a39d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:33.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.673+0000 7f175a39d640 10 monclient: build_initial_monmap 2024-09-18T12:30:33.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.673+0000 7f175a39d640 10 monclient: monmap: 2024-09-18T12:30:33.674 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:33.674 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:33.674026+0000 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:33.674026+0000 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:33.675 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:33.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.673+0000 7f175a39d640 1 Processor -- start 2024-09-18T12:30:33.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.673+0000 7f175a39d640 1 -- start start 2024-09-18T12:30:33.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.673+0000 7f175a39d640 10 monclient: init 2024-09-18T12:30:33.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f175409ec50 0x7f1754131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient(hunting): picked mon.noname-a con 0x7f175409ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17541321a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f175409f020 0x7f1754069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient(hunting): picked mon.noname-b con 0x7f175409f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1754067420 con 0x7f17541321a0 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175409fdc0 con 0x7f175409ec50 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17540a0510 con 0x7f175409f020 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:33.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175a39d640 10 monclient(hunting): authenticate will time out at 1069.243451s 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f1753fff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f175409ec50 msgr2=0x7f1754131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f1753fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f175409ec50 0x7f1754131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f17537fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f17541321a0 0x7f1754068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39068/0 (socket says 172.21.15.144:39068) 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f17537fe640 1 -- 172.21.15.144:0/3692997234 learned_addr learned my addr 172.21.15.144:0/3692997234 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175893a640 1 --2- 172.21.15.144:0/3692997234 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f175409f020 0x7f1754069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175893a640 10 monclient(hunting): get_auth_request con 0x7f175409f020 auth_method 0 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175893a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:33.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175893a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175893a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175893a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.674+0000 7f175893a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 10 monclient(hunting): handle_auth_done global_id 64385 payload 274 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 1 -- 172.21.15.144:0/3692997234 >> v2:172.21.15.116:3300/0 conn(0x7f175409ec50 msgr2=0x7f1754131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 1 --2- 172.21.15.144:0/3692997234 >> v2:172.21.15.116:3300/0 conn(0x7f175409ec50 0x7f1754131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 1 -- 172.21.15.144:0/3692997234 >> v1:172.21.15.144:6789/0 conn(0x7f17541321a0 legacy=0x7f1754068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 10 monclient: _finish_hunting 0 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 1 monclient: found mon.noname-b 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:33.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.675+0000 7f175893a640 1 -- 172.21.15.144:0/3692997234 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17540a0200 con 0x7f175409f020 2024-09-18T12:30:33.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175893a640 1 -- 172.21.15.144:0/3692997234 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f175409f020 msgr2=0x7f1754069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:33.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175893a640 1 --2- 172.21.15.144:0/3692997234 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f175409f020 0x7f1754069410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f174c009e60 tx=0x7f174c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14515b2b812bf391 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 1 -- 172.21.15.144:0/3692997234 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c007840 con 0x7f175409f020 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: dump: 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:33.689 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 1 -- 172.21.15.144:0/3692997234 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c02d050 con 0x7f175409f020 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 5 monclient: authenticate success, global_id 64385 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:33.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 4 set_mon_vals no callback set 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 1 -- 172.21.15.144:0/3692997234 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c007c10 con 0x7f175409f020 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f1752ffd640 10 monclient: dump: 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:33.691 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 10 monclient: shutdown 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 -- 172.21.15.144:0/3692997234 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f175409f020 msgr2=0x7f1754069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 --2- 172.21.15.144:0/3692997234 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f175409f020 0x7f1754069410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f174c009e60 tx=0x7f174c009780 comp rx=0 tx=0).stop 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 -- 172.21.15.144:0/3692997234 shutdown_connections 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 --2- 172.21.15.144:0/3692997234 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f175409f020 0x7f1754069410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 --2- 172.21.15.144:0/3692997234 >> v2:172.21.15.116:3300/0 conn(0x7f175409ec50 0x7f1754131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 -- 172.21.15.144:0/3692997234 >> 172.21.15.144:0/3692997234 conn(0x7f17540a5d40 msgr2=0x7f17540a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 -- 172.21.15.144:0/3692997234 shutdown_connections 2024-09-18T12:30:33.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.678+0000 7f175a39d640 1 -- 172.21.15.144:0/3692997234 wait complete. 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.679+0000 7f175a39d640 10 monclient: build_initial_monmap 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.679+0000 7f175a39d640 10 monclient: monmap: 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:33.693 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.679+0000 7f175a39d640 1 Processor -- start 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.679+0000 7f175a39d640 1 -- start start 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.679+0000 7f175a39d640 10 monclient: init 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 0x7f175409aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): picked mon.noname-b con 0x7f175409ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175409f020 0x7f1754130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): picked mon.noname-a con 0x7f175409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17541321a0 0x7f1754140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): picked mon.noname-c con 0x7f17541321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175409e240 con 0x7f175409f020 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175411df50 con 0x7f175409ec50 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175409b4c0 con 0x7f17541321a0 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175a39d640 10 monclient(hunting): authenticate will time out at 1069.249368s 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 0x7f175409aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 0x7f175409aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42312/0 (socket says 172.21.15.144:42312) 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f17537fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175409f020 msgr2=0x7f1754130b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 -- 172.21.15.144:0/608730035 learned_addr learned my addr 172.21.15.144:0/608730035 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): get_auth_request con 0x7f175409ec50 auth_method 0 2024-09-18T12:30:33.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:33.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:33.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:33.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f17537fe640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175409f020 0x7f1754130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17541321a0 0x7f1754140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 10 monclient(hunting): get_auth_request con 0x7f17541321a0 auth_method 0 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f175893a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:33.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient(hunting): handle_auth_done global_id 5545 payload 824 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 -- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17541321a0 msgr2=0x7f1754140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17541321a0 0x7f1754140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 -- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175409f020 msgr2=0x7f1754130b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175409f020 0x7f1754130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 monclient: found mon.noname-b 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.680+0000 7f1753fff640 1 -- 172.21.15.144:0/608730035 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c009a20 con 0x7f175409ec50 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f1753fff640 1 -- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 msgr2=0x7f175409aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:33.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f1753fff640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 0x7f175409aa20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f174400b600 tx=0x7f17440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744003d90 con 0x7f175409ec50 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: dump: 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:33.699 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: _finish_auth 0 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1744006150 con 0x7f175409ec50 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f175a39d640 5 monclient: authenticate success, global_id 5545 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f175a39d640 10 monclient: _renew_subs 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f175a39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:33.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175411dc70 con 0x7f175409ec50 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744006510 con 0x7f175409ec50 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f173bfff640 10 monclient: dump: 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.701 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f17527fc640 4 set_mon_vals no callback set 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f175a39d640 10 monclient: _renew_subs 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f175a39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.683+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175411bc30 con 0x7f175409ec50 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f17440103c0 con 0x7f175409ec50 2024-09-18T12:30:33.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f175a39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f175a39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f175a39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f173bfff640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1740038490 0x7f174003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175411e110 con 0x7f175409ec50 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6361+0+0 (secure 0 0 0) 0x7f174404b510 con 0x7f175409ec50 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f17537fe640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1740038490 0x7f174003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.684+0000 7f17537fe640 10 monclient: get_auth_request con 0x7f1740038490 auth_method 0 2024-09-18T12:30:33.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.687+0000 7f17537fe640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1740038490 0x7f174003a940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f17480047d0 tx=0x7f1748007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:33.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.689+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1744017070 con 0x7f175409ec50 2024-09-18T12:30:33.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.689+0000 7f173bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:33.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.689+0000 7f173bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:33.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.848+0000 7f175a39d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:33.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.848+0000 7f175a39d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:33.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.848+0000 7f175a39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:33.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:33.848+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f17540008d0 con 0x7f175409ec50 2024-09-18T12:30:34.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.585+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map v118) v1 ==== 220+0+0 (secure 0 0 0) 0x7f174400eaa0 con 0x7f175409ec50 2024-09-18T12:30:34.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.585+0000 7f173bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:34.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.585+0000 7f173bfff640 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-18T12:30:34.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.686+0000 7f175a39d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:34.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.686+0000 7f175a39d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:34.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.686+0000 7f175a39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:34.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.686+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f17540a13e0 con 0x7f175409ec50 2024-09-18T12:30:34.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.686+0000 7f173bfff640 1 -- 172.21.15.144:0/608730035 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change v118) v1 ==== 218+0+0 (secure 0 0 0) 0x7f174400e2c0 con 0x7f175409ec50 2024-09-18T12:30:34.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.686+0000 7f173bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:34.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.686+0000 7f173bfff640 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T12:30:34.689 INFO:tasks.workunit.client.0.smithi144.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T12:30:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1740038490 msgr2=0x7f174003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1740038490 0x7f174003a940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f17480047d0 tx=0x7f1748007b80 comp rx=0 tx=0).stop 2024-09-18T12:30:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 10 monclient: shutdown 2024-09-18T12:30:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 msgr2=0x7f175409aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 0x7f175409aa20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f174400b600 tx=0x7f17440091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 shutdown_connections 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1740038490 0x7f174003a940 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17541321a0 0x7f1754140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f175409f020 0x7f1754130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 --2- 172.21.15.144:0/608730035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f175409ec50 0x7f175409aa20 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 >> 172.21.15.144:0/608730035 conn(0x7f17540a5d40 msgr2=0x7f1754134a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 shutdown_connections 2024-09-18T12:30:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.688+0000 7f175a39d640 1 -- 172.21.15.144:0/608730035 wait complete. 2024-09-18T12:30:34.698 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.772+0000 7ff43735d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.772+0000 7ff43735d640 10 monclient: build_initial_monmap 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.772+0000 7ff43735d640 10 monclient: monmap: 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:34.773737+0000 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:34.773737+0000 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:34.774 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:34.775 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:34.775 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:34.775 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:34.775 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:34.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 1 Processor -- start 2024-09-18T12:30:34.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 1 -- start start 2024-09-18T12:30:34.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 10 monclient: init 2024-09-18T12:30:34.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:34.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff43009d140 0x7ff43009d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 10 monclient(hunting): picked mon.noname-a con 0x7ff43009d140 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff43009da50 0x7ff430094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 10 monclient(hunting): picked mon.noname-c con 0x7ff43009da50 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.773+0000 7ff43735d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4300945c0 0x7ff430094a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4300945c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4300670b0 con 0x7ff43009d140 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff430097fb0 con 0x7ff43009da50 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:34.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff430094f40 con 0x7ff4300945c0 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff43735d640 10 monclient(hunting): authenticate will time out at 1070.343024s 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff43009da50 0x7ff430094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4350f9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7ff43009d140 msgr2=0x7ff43009d510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4350f9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff43009d140 0x7ff43009d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff43009da50 0x7ff430094080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42362/0 (socket says 172.21.15.144:42362) 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 -- 172.21.15.144:0/455092135 learned_addr learned my addr 172.21.15.144:0/455092135 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): get_auth_request con 0x7ff43009da50 auth_method 0 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:34.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4358fa640 1 --2- 172.21.15.144:0/455092135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4300945c0 0x7ff430094a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4358fa640 10 monclient(hunting): get_auth_request con 0x7ff4300945c0 auth_method 0 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4358fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4358fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4358fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient(hunting): handle_auth_done global_id 5548 payload 274 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 -- 172.21.15.144:0/455092135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4300945c0 msgr2=0x7ff430094a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 --2- 172.21.15.144:0/455092135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4300945c0 0x7ff430094a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 -- 172.21.15.144:0/455092135 >> v2:172.21.15.116:3300/0 conn(0x7ff43009d140 msgr2=0x7ff43009d510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 --2- 172.21.15.144:0/455092135 >> v2:172.21.15.116:3300/0 conn(0x7ff43009d140 0x7ff43009d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient: _finish_hunting 0 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 monclient: found mon.noname-c 2024-09-18T12:30:34.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:34.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:34.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.774+0000 7ff4348f8640 1 -- 172.21.15.144:0/455092135 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430098570 con 0x7ff43009da50 2024-09-18T12:30:34.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff4348f8640 1 -- 172.21.15.144:0/455092135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff43009da50 msgr2=0x7ff430094080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff4348f8640 1 --2- 172.21.15.144:0/455092135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff43009da50 0x7ff430094080 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff430063900 tx=0x7ff4240091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ad465ca117d8037 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 1 -- 172.21.15.144:0/455092135 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff424032440 con 0x7ff43009da50 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: dump: 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:34.787 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: _finish_auth 0 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 5 monclient: authenticate success, global_id 5548 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 1 -- 172.21.15.144:0/455092135 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4240325a0 con 0x7ff43009da50 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 1 -- 172.21.15.144:0/455092135 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff424032940 con 0x7ff43009da50 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 4 set_mon_vals no callback set 2024-09-18T12:30:34.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff42ffff640 10 monclient: dump: 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 10 monclient: shutdown 2024-09-18T12:30:34.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 1 -- 172.21.15.144:0/455092135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff43009da50 msgr2=0x7ff430094080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 1 --2- 172.21.15.144:0/455092135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff43009da50 0x7ff430094080 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff430063900 tx=0x7ff4240091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 1 -- 172.21.15.144:0/455092135 shutdown_connections 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 1 --2- 172.21.15.144:0/455092135 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff4300945c0 0x7ff430094a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 1 --2- 172.21.15.144:0/455092135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff43009da50 0x7ff430094080 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 1 --2- 172.21.15.144:0/455092135 >> v2:172.21.15.116:3300/0 conn(0x7ff43009d140 0x7ff43009d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.777+0000 7ff43735d640 1 -- 172.21.15.144:0/455092135 >> 172.21.15.144:0/455092135 conn(0x7ff4300a2190 msgr2=0x7ff43009e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.778+0000 7ff43735d640 1 -- 172.21.15.144:0/455092135 shutdown_connections 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.778+0000 7ff43735d640 1 -- 172.21.15.144:0/455092135 wait complete. 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.778+0000 7ff43735d640 10 monclient: build_initial_monmap 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.778+0000 7ff43735d640 10 monclient: monmap: 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:34.790 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.778+0000 7ff43735d640 1 Processor -- start 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.778+0000 7ff43735d640 1 -- start start 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.778+0000 7ff43735d640 10 monclient: init 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 0x7ff4300934c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:34.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4300945c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff43009d140 0x7ff43013ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): picked mon.noname-c con 0x7ff43009d140 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff43009da50 0x7ff43008a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): picked mon.noname-a con 0x7ff43009da50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43009baf0 con 0x7ff43009da50 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300a1260 con 0x7ff4300945c0 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43008d9c0 con 0x7ff43009d140 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff43735d640 10 monclient(hunting): authenticate will time out at 1070.348531s 2024-09-18T12:30:34.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 0x7ff4300934c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4358fa640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff43009da50 msgr2=0x7ff43008a120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 0x7ff4300934c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42368/0 (socket says 172.21.15.144:42368) 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 -- 172.21.15.144:0/3782725069 learned_addr learned my addr 172.21.15.144:0/3782725069 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): get_auth_request con 0x7ff4300945c0 auth_method 0 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4358fa640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff43009da50 0x7ff43008a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff43009d140 0x7ff43013ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:34.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 10 monclient(hunting): get_auth_request con 0x7ff43009d140 auth_method 0 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4348f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient(hunting): handle_auth_done global_id 5551 payload 824 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 -- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff43009d140 msgr2=0x7ff43013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff43009d140 0x7ff43013ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 -- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff43009da50 msgr2=0x7ff43008a120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff43009da50 0x7ff43008a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient: _finish_hunting 0 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 monclient: found mon.noname-b 2024-09-18T12:30:34.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.779+0000 7ff4350f9640 1 -- 172.21.15.144:0/3782725069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff424009450 con 0x7ff4300945c0 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff4350f9640 1 -- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 msgr2=0x7ff4300934c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff4350f9640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 0x7ff4300934c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff42800b600 tx=0x7ff4280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428015070 con 0x7ff4300945c0 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: dump: 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:34.795 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff428003d90 con 0x7ff4300945c0 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff43735d640 5 monclient: authenticate success, global_id 5551 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff43735d640 10 monclient: _renew_subs 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff43735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43008d6b0 con 0x7ff4300945c0 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:34.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428006380 con 0x7ff4300945c0 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42d7fa640 10 monclient: dump: 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:34.797 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff43735d640 10 monclient: _renew_subs 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff43735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43008bc30 con 0x7ff4300945c0 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.782+0000 7ff42f7fe640 4 set_mon_vals no callback set 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff43735d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff428010440 con 0x7ff4300945c0 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff43735d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff43735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43008db50 con 0x7ff4300945c0 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff42d7fa640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6545+0+0 (secure 0 0 0) 0x7ff42804b930 con 0x7ff4300945c0 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff4348f8640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:34.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff4348f8640 1 -- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 msgr2=0x7ff41c03a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:30:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff4348f8640 1 -- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 msgr2=0x7ff41c03a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff4348f8640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.783+0000 7ff4348f8640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.787+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff428018ab0 con 0x7ff4300945c0 2024-09-18T12:30:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.787+0000 7ff42d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:34.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.787+0000 7ff42d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:34.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.945+0000 7ff43735d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:34.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.945+0000 7ff43735d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:34.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.945+0000 7ff43735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:34.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.945+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7ff4300008d0 con 0x7ff4300945c0 2024-09-18T12:30:34.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.984+0000 7ff4348f8640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:34.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.984+0000 7ff4348f8640 10 monclient: get_auth_request con 0x7ff41c038440 auth_method 0 2024-09-18T12:30:34.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:34.986+0000 7ff4348f8640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff430063900 tx=0x7ff424032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:35.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.590+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map v119) v1 ==== 220+0+0 (secure 0 0 0) 0x7ff428018690 con 0x7ff4300945c0 2024-09-18T12:30:35.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.590+0000 7ff42d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:35.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.590+0000 7ff42d7fa640 10 monclient: _finish_command 2 = system:0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-18T12:30:35.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.687+0000 7ff43735d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:35.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.687+0000 7ff43735d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:35.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.687+0000 7ff43735d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:35.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.687+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7ff430093a00 con 0x7ff4300945c0 2024-09-18T12:30:35.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.688+0000 7ff42d7fa640 1 -- 172.21.15.144:0/3782725069 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change v119) v1 ==== 218+0+0 (secure 0 0 0) 0x7ff428018ab0 con 0x7ff4300945c0 2024-09-18T12:30:35.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.688+0000 7ff42d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T12:30:35.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.688+0000 7ff42d7fa640 10 monclient: _finish_command 3 = system:0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T12:30:35.690 INFO:tasks.workunit.client.0.smithi144.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T12:30:35.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 msgr2=0x7ff41c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:35.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff430063900 tx=0x7ff424032000 comp rx=0 tx=0).stop 2024-09-18T12:30:35.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 10 monclient: shutdown 2024-09-18T12:30:35.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:35.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 msgr2=0x7ff4300934c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:35.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 0x7ff4300934c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff42800b600 tx=0x7ff4280091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 shutdown_connections 2024-09-18T12:30:35.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff41c038440 0x7ff41c03a8f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff43009da50 0x7ff43008a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff43009d140 0x7ff43013ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 --2- 172.21.15.144:0/3782725069 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4300945c0 0x7ff4300934c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 >> 172.21.15.144:0/3782725069 conn(0x7ff4300a2190 msgr2=0x7ff43009f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:35.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 shutdown_connections 2024-09-18T12:30:35.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.690+0000 7ff43735d640 1 -- 172.21.15.144:0/3782725069 wait complete. 2024-09-18T12:30:35.700 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:30:35.700 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep osd.12 2024-09-18T12:30:35.700 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 123 2024-09-18T12:30:35.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.770+0000 7f7b89494640 10 monclient: get_monmap_and_config 2024-09-18T12:30:35.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.770+0000 7f7b89494640 10 monclient: build_initial_monmap 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.770+0000 7f7b89494640 10 monclient: monmap: 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:35.771375+0000 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:35.771375+0000 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:35.772 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 1 Processor -- start 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 1 -- start start 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient: init 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b84131b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b84131f10 0x7f7b84132330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient(hunting): picked mon.noname-a con 0x7f7b84131f10 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b84136e70 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:35.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8406a230 con 0x7f7b84136e70 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b840a0300 con 0x7f7b84131b40 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b840a00c0 con 0x7f7b84131f10 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b89494640 10 monclient(hunting): authenticate will time out at 1071.340597s 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b82ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7b84131b40 0x7f7b8409ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39078/0 (socket says 172.21.15.144:39078) 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b82ffd640 1 -- 172.21.15.144:0/1503056930 learned_addr learned my addr 172.21.15.144:0/1503056930 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1503056930 >> v2:172.21.15.116:3300/0 conn(0x7f7b84131f10 msgr2=0x7f7b84132330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b837fe640 1 -- 172.21.15.144:0/1503056930 >> v1:172.21.15.116:6789/0 conn(0x7f7b84136e70 legacy=0x7f7b84139250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.771+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1503056930 >> v2:172.21.15.116:3300/0 conn(0x7f7b84131f10 0x7f7b84132330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.773+0000 7f7b82ffd640 1 -- 172.21.15.144:0/1503056930 >> v1:172.21.15.144:6789/0 conn(0x7f7b84131b40 legacy=0x7f7b8409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.773+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276992135 0 0) 0x7f7b840a0300 con 0x7f7b84131b40 2024-09-18T12:30:35.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.773+0000 7f7b827fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.773+0000 7f7b827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.773+0000 7f7b827fc640 10 monclient(hunting): my global_id is 5554 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.773+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b700034e0 con 0x7f7b84131b40 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1274628365 0 0) 0x7f7b700034e0 con 0x7f7b84131b40 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 >> v2:172.21.15.116:3300/0 conn(0x7f7b84131f10 msgr2=0x7f7b84132330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 --2- 172.21.15.144:0/1503056930 >> v2:172.21.15.116:3300/0 conn(0x7f7b84131f10 0x7f7b84132330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 >> v1:172.21.15.116:6789/0 conn(0x7f7b84136e70 legacy=0x7f7b84139250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8409bee0 con 0x7f7b84131b40 2024-09-18T12:30:35.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f7b740017a0 con 0x7f7b84131b40 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: dump: 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:35.780 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 >> v1:172.21.15.144:6789/0 conn(0x7f7b84131b40 legacy=0x7f7b8409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70003aa0 0x7f7b70003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): picked mon.a con 0x7f7b70003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b70004780 0x7f7b70004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): picked mon.b con 0x7f7b70004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b700099c0 0x7f7b7000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): picked mon.c con 0x7f7b700099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:35.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b840a00c0 con 0x7f7b70003aa0 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7000c3c0 con 0x7f7b70004780 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7000c580 con 0x7f7b700099c0 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b827fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b70004780 0x7f7b70004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): get_auth_request con 0x7f7b70004780 auth_method 0 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:35.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b82ffd640 1 -- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70003aa0 msgr2=0x7f7b70003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b82ffd640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70003aa0 0x7f7b70003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_done global_id 5554 payload 293 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b700099c0 msgr2=0x7f7b7000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b700099c0 0x7f7b7000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70003aa0 msgr2=0x7f7b70003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70003aa0 0x7f7b70003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:35.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 1 monclient: found mon.b 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1503056930 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b700034e0 con 0x7f7b70004780 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b837fe640 1 -- 172.21.15.144:0/1503056930 reap_dead start 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b89494640 5 monclient: authenticate success, global_id 5554 2024-09-18T12:30:35.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.774+0000 7f7b89494640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:35.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1503056930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b70004780 msgr2=0x7f7b70004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:35.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b70004780 0x7f7b70004bf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0099c0 tx=0x7f7b6c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38ac1299ef42ebf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:35.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c003b80 con 0x7f7b70004780 2024-09-18T12:30:35.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:35.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:35.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 10 monclient: dump: 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c003d20 con 0x7f7b70004780 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b89494640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:35.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 1 -- 172.21.15.144:0/1503056930 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c02ab80 con 0x7f7b70004780 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b89494640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b89494640 4 set_mon_vals no callback set 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.777+0000 7f7b827fc640 10 monclient: dump: 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:35.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 10 monclient: shutdown 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 1 -- 172.21.15.144:0/1503056930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b70004780 msgr2=0x7f7b70004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b70004780 0x7f7b70004bf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c0099c0 tx=0x7f7b6c009060 comp rx=0 tx=0).stop 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 1 -- 172.21.15.144:0/1503056930 shutdown_connections 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 1 --2- 172.21.15.144:0/1503056930 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b70004780 0x7f7b70004bf0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 1 -- 172.21.15.144:0/1503056930 >> 172.21.15.144:0/1503056930 conn(0x7f7b840a5100 msgr2=0x7f7b840a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 1 -- 172.21.15.144:0/1503056930 shutdown_connections 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 1 -- 172.21.15.144:0/1503056930 wait complete. 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 10 monclient: build_initial_monmap 2024-09-18T12:30:35.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.778+0000 7f7b89494640 10 monclient: monmap: 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 Processor -- start 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 -- start start 2024-09-18T12:30:35.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient: init 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b70004780 0x7f7b8412f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): picked mon.noname-c con 0x7f7b70004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 0x7f7b840986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): picked mon.noname-b con 0x7f7b8412f6d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8412faa0 0x7f7b84098c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): picked mon.noname-a con 0x7f7b8412faa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8411bd70 con 0x7f7b8412faa0 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8411bfe0 con 0x7f7b8412f6d0 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:35.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b840a0a30 con 0x7f7b70004780 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b89494640 10 monclient(hunting): authenticate will time out at 1071.348500s 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 0x7f7b840986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 0x7f7b840986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42380/0 (socket says 172.21.15.144:42380) 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b837fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8412faa0 msgr2=0x7f7b84098c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1199302971 learned_addr learned my addr 172.21.15.144:0/1199302971 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): get_auth_request con 0x7f7b8412f6d0 auth_method 0 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b837fe640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8412faa0 0x7f7b84098c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:35.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b70004780 0x7f7b8412f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 10 monclient(hunting): get_auth_request con 0x7f7b70004780 auth_method 0 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient(hunting): handle_auth_done global_id 5557 payload 824 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b70004780 msgr2=0x7f7b8412f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b70004780 0x7f7b8412f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8412faa0 msgr2=0x7f7b84098c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:35.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8412faa0 0x7f7b84098c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 monclient: found mon.noname-b 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.779+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1199302971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84130fa0 con 0x7f7b8412f6d0 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7a7fc640 1 -- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 msgr2=0x7f7b840986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7a7fc640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 0x7f7b840986f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009a20 tx=0x7f7b6c030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c005690 con 0x7f7b8412f6d0 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: dump: 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:35.798 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b89494640 5 monclient: authenticate success, global_id 5557 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c0057f0 con 0x7f7b8412f6d0 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b89494640 10 monclient: _renew_subs 2024-09-18T12:30:35.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b89494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8409a7f0 con 0x7f7b8412f6d0 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c033680 con 0x7f7b8412f6d0 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 10 monclient: dump: 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:35.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b89494640 10 monclient: _renew_subs 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b89494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b81ffb640 4 set_mon_vals no callback set 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8409a280 con 0x7f7b8412f6d0 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.782+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7b6c030070 con 0x7f7b8412f6d0 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b89494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b89494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b89494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8411c120 con 0x7f7b8412f6d0 2024-09-18T12:30:35.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b7b7fe640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b64038470 0x7f7b6403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:35.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f7b6c03a070 con 0x7f7b8412f6d0 2024-09-18T12:30:35.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b82ffd640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b64038470 0x7f7b6403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:35.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.783+0000 7f7b82ffd640 10 monclient: get_auth_request con 0x7f7b64038470 auth_method 0 2024-09-18T12:30:35.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.785+0000 7f7b82ffd640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b64038470 0x7f7b6403a920 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7b74001ed0 tx=0x7f7b740010c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:35.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.787+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7b8411c120 con 0x7f7b8412f6d0 2024-09-18T12:30:35.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.787+0000 7f7b7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:35.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.787+0000 7f7b7b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:35.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.943+0000 7f7b89494640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:35.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.943+0000 7f7b89494640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:35.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.943+0000 7f7b89494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:35.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.943+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f7b8411c120 con 0x7f7b8412f6d0 2024-09-18T12:30:35.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.943+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v119) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f7b8411c120 con 0x7f7b8412f6d0 2024-09-18T12:30:35.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.943+0000 7f7b7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:35.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:35.943+0000 7f7b7b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:36.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.035+0000 7f7b89494640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:36.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.035+0000 7f7b89494640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:36.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.035+0000 7f7b89494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.035+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f7b8411c120 con 0x7f7b8412f6d0 2024-09-18T12:30:36.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.035+0000 7f7b7b7fe640 1 -- 172.21.15.144:0/1199302971 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v119) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f7b8411c120 con 0x7f7b8412f6d0 2024-09-18T12:30:36.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.036+0000 7f7b7b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:36.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.036+0000 7f7b7b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b64038470 msgr2=0x7f7b6403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b64038470 0x7f7b6403a920 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7b74001ed0 tx=0x7f7b740010c0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 10 monclient: shutdown 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 msgr2=0x7f7b840986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 0x7f7b840986f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009a20 tx=0x7f7b6c030000 comp rx=0 tx=0).stop 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 shutdown_connections 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b64038470 0x7f7b6403a920 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8412faa0 0x7f7b84098c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8412f6d0 0x7f7b840986f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 --2- 172.21.15.144:0/1199302971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b70004780 0x7f7b8412f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 >> 172.21.15.144:0/1199302971 conn(0x7f7b840a5100 msgr2=0x7f7b840a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:36.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 shutdown_connections 2024-09-18T12:30:36.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.037+0000 7f7b89494640 1 -- 172.21.15.144:0/1199302971 wait complete. 2024-09-18T12:30:36.047 INFO:tasks.workunit.client.0.smithi144.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-18T12:30:36.048 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:30:36.048 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep osd.14 2024-09-18T12:30:36.048 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 123 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.117+0000 7f1d8395d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.117+0000 7f1d8395d640 10 monclient: build_initial_monmap 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.117+0000 7f1d8395d640 10 monclient: monmap: 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:36.118755+0000 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:36.118755+0000 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:36.119 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.120 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:36.120 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:36.120 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:36.120 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:36.120 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:36.120 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.118+0000 7f1d8395d640 1 Processor -- start 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.118+0000 7f1d8395d640 1 -- start start 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.118+0000 7f1d8395d640 10 monclient: init 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d7c0a3950 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c09c7f0 0x7f1d7c09cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7c09c7f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d7c134860 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c09f520 con 0x7f1d7c0a3950 2024-09-18T12:30:36.123 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c09f760 con 0x7f1d7c134860 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c09b840 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d8395d640 10 monclient(hunting): authenticate will time out at 1071.688231s 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d816f9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1d7c0a3950 legacy=0x7f1d7c099a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d81efa640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1d7c134860 0x7f1d7c136c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39094/0 (socket says 172.21.15.144:39094) 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d81efa640 1 -- 172.21.15.144:0/1744754694 learned_addr learned my addr 172.21.15.144:0/1744754694 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 1 --2- 172.21.15.144:0/1744754694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c09c7f0 0x7f1d7c09cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): get_auth_request con 0x7f1d7c09c7f0 auth_method 0 2024-09-18T12:30:36.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient(hunting): handle_auth_done global_id 64400 payload 274 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 1 -- 172.21.15.144:0/1744754694 >> v1:172.21.15.144:6789/0 conn(0x7f1d7c134860 legacy=0x7f1d7c136c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 1 -- 172.21.15.144:0/1744754694 >> v1:172.21.15.116:6789/0 conn(0x7f1d7c0a3950 legacy=0x7f1d7c099a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient: _finish_hunting 0 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 1 monclient: found mon.noname-b 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:36.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:36.126 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.119+0000 7f1d80ef8640 1 -- 172.21.15.144:0/1744754694 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c09b560 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.122+0000 7f1d80ef8640 1 -- 172.21.15.144:0/1744754694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c09c7f0 msgr2=0x7f1d7c09cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:36.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.122+0000 7f1d80ef8640 1 --2- 172.21.15.144:0/1744754694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c09c7f0 0x7f1d7c09cbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009880 tx=0x7f1d700090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7ea7c96878bc06f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:36.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 1 -- 172.21.15.144:0/1744754694 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7002aea0 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: dump: 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.132 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: _finish_auth 0 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 1 -- 172.21.15.144:0/1744754694 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d70032440 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 1 -- 172.21.15.144:0/1744754694 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70032820 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 5 monclient: authenticate success, global_id 64400 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 4 set_mon_vals no callback set 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d7bfff640 10 monclient: dump: 2024-09-18T12:30:36.133 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 10 monclient: shutdown 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 1 -- 172.21.15.144:0/1744754694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c09c7f0 msgr2=0x7f1d7c09cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.134 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1744754694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c09c7f0 0x7f1d7c09cbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009880 tx=0x7f1d700090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 1 -- 172.21.15.144:0/1744754694 shutdown_connections 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1744754694 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c09c7f0 0x7f1d7c09cbc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 1 -- 172.21.15.144:0/1744754694 >> 172.21.15.144:0/1744754694 conn(0x7f1d7c070fd0 msgr2=0x7f1d7c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 1 -- 172.21.15.144:0/1744754694 shutdown_connections 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.123+0000 7f1d8395d640 1 -- 172.21.15.144:0/1744754694 wait complete. 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.124+0000 7f1d8395d640 10 monclient: build_initial_monmap 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.124+0000 7f1d8395d640 10 monclient: monmap: 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:36.135 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.124+0000 7f1d8395d640 1 Processor -- start 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.124+0000 7f1d8395d640 1 -- start start 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.124+0000 7f1d8395d640 10 monclient: init 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 0x7f1d7c136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7c09c7f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d7c0a3950 0x7f1d7c096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d7c0a3950 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:36.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c134860 0x7f1d7c096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d7c134860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c099260 con 0x7f1d7c0a3950 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c134480 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c06eaa0 con 0x7f1d7c134860 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d8395d640 10 monclient(hunting): authenticate will time out at 1071.694336s 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 0x7f1d7c136da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 0x7f1d7c136da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42390/0 (socket says 172.21.15.144:42390) 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 -- 172.21.15.144:0/1050956014 learned_addr learned my addr 172.21.15.144:0/1050956014 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:36.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): get_auth_request con 0x7f1d7c09c7f0 auth_method 0 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d80ef8640 1 -- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d7c0a3950 msgr2=0x7f1d7c096a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d80ef8640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d7c0a3950 0x7f1d7c096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c134860 0x7f1d7c096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 10 monclient(hunting): get_auth_request con 0x7f1d7c134860 auth_method 0 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:36.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d81efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient(hunting): handle_auth_done global_id 5560 payload 824 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 -- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c134860 msgr2=0x7f1d7c096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c134860 0x7f1d7c096f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 -- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d7c0a3950 msgr2=0x7f1d7c096a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d7c0a3950 0x7f1d7c096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient: _finish_hunting 0 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 monclient: found mon.noname-b 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.125+0000 7f1d816f9640 1 -- 172.21.15.144:0/1050956014 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d70009390 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.127+0000 7f1d816f9640 1 -- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 msgr2=0x7f1d7c136da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.127+0000 7f1d816f9640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 0x7f1d7c136da0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00bbb0 tx=0x7f1d6c0097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00cc80 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: dump: 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.140 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: _finish_auth 0 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d8395d640 5 monclient: authenticate success, global_id 5560 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c00cde0 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d8395d640 10 monclient: _renew_subs 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d8395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c098300 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c016430 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d797fa640 10 monclient: dump: 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d7b7fe640 4 set_mon_vals no callback set 2024-09-18T12:30:36.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d8395d640 10 monclient: _renew_subs 2024-09-18T12:30:36.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d8395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.128+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c097be0 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1d6c012020 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d8395d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:36.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d8395d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:36.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d8395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c09ce70 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d797fa640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d68038440 0x7f1d6803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f1d6c04be80 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d80ef8640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d68038440 0x7f1d6803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.129+0000 7f1d80ef8640 10 monclient: get_auth_request con 0x7f1d68038440 auth_method 0 2024-09-18T12:30:36.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.131+0000 7f1d80ef8640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d68038440 0x7f1d6803a8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1d70034000 tx=0x7f1d70032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:36.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.133+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1d7c09ce70 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.133+0000 7f1d797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:36.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.133+0000 7f1d797fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:36.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.288+0000 7f1d8395d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:36.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.288+0000 7f1d8395d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:36.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.288+0000 7f1d8395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.288+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1d7c09ce70 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.289+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v119) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f1d7c09ce70 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.289+0000 7f1d797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:36.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.289+0000 7f1d797fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:36.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.382+0000 7f1d8395d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:36.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.382+0000 7f1d8395d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:36.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.382+0000 7f1d8395d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.382+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1d7c09ce70 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.382+0000 7f1d797fa640 1 -- 172.21.15.144:0/1050956014 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v119) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f1d7c09ce70 con 0x7f1d7c09c7f0 2024-09-18T12:30:36.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.382+0000 7f1d797fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:36.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.382+0000 7f1d797fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d68038440 msgr2=0x7f1d6803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d68038440 0x7f1d6803a8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1d70034000 tx=0x7f1d70032000 comp rx=0 tx=0).stop 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 10 monclient: shutdown 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 msgr2=0x7f1d7c136da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 0x7f1d7c136da0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00bbb0 tx=0x7f1d6c0097f0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 shutdown_connections 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d68038440 0x7f1d6803a8f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d7c134860 0x7f1d7c096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d7c0a3950 0x7f1d7c096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 --2- 172.21.15.144:0/1050956014 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d7c09c7f0 0x7f1d7c136da0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 >> 172.21.15.144:0/1050956014 conn(0x7f1d7c070fd0 msgr2=0x7f1d7c0a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:36.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 shutdown_connections 2024-09-18T12:30:36.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.384+0000 7f1d8395d640 1 -- 172.21.15.144:0/1050956014 wait complete. 2024-09-18T12:30:36.394 INFO:tasks.workunit.client.0.smithi144.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2024-09-18T12:30:36.394 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.467+0000 7f41294e8640 10 monclient: get_monmap_and_config 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.467+0000 7f41294e8640 10 monclient: build_initial_monmap 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.467+0000 7f41294e8640 10 monclient: monmap: 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:36.468762+0000 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:36.468762+0000 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.469 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:36.470 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:36.470 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:36.470 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:36.470 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:36.470 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 1 Processor -- start 2024-09-18T12:30:36.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 1 -- start start 2024-09-18T12:30:36.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 10 monclient: init 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f412409ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f412409f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4124069d10 0x7f412413a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4124069d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.468+0000 7f41294e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4124067420 con 0x7f412409ec50 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41294e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412409fdc0 con 0x7f412409f020 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41294e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41294e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412406a170 con 0x7f4124069d10 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41294e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:36.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41294e8640 10 monclient(hunting): authenticate will time out at 1072.038027s 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41227fc640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f412409f020 0x7f4124136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39096/0 (socket says 172.21.15.144:39096) 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41227fc640 1 -- 172.21.15.144:0/590583747 learned_addr learned my addr 172.21.15.144:0/590583747 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f4122ffd640 1 -- 172.21.15.144:0/590583747 >> v1:172.21.15.116:6789/0 conn(0x7f412409ec50 legacy=0x7f4124131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 1 --2- 172.21.15.144:0/590583747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4124069d10 0x7f412413a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): get_auth_request con 0x7f4124069d10 auth_method 0 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient(hunting): handle_auth_done global_id 64406 payload 274 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 1 -- 172.21.15.144:0/590583747 >> v1:172.21.15.144:6789/0 conn(0x7f412409f020 legacy=0x7f4124136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:36.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 1 -- 172.21.15.144:0/590583747 >> v1:172.21.15.116:6789/0 conn(0x7f412409ec50 legacy=0x7f4124131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:36.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:36.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:36.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:36.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:36.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.469+0000 7f41237fe640 1 -- 172.21.15.144:0/590583747 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240a0510 con 0x7f4124069d10 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f41237fe640 1 -- 172.21.15.144:0/590583747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4124069d10 msgr2=0x7f412413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f41237fe640 1 --2- 172.21.15.144:0/590583747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4124069d10 0x7f412413a490 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f411c009e90 tx=0x7f411c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=828d2d12fae211f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 1 -- 172.21.15.144:0/590583747 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c007840 con 0x7f4124069d10 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: dump: 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.481 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 1 -- 172.21.15.144:0/590583747 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c02d050 con 0x7f4124069d10 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f41294e8640 5 monclient: authenticate success, global_id 64406 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f41294e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f41294e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:36.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f41294e8640 4 set_mon_vals no callback set 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 1 -- 172.21.15.144:0/590583747 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c007bd0 con 0x7f4124069d10 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.472+0000 7f4121ffb640 10 monclient: dump: 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.483 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 10 monclient: shutdown 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 1 -- 172.21.15.144:0/590583747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4124069d10 msgr2=0x7f412413a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 1 --2- 172.21.15.144:0/590583747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4124069d10 0x7f412413a490 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f411c009e90 tx=0x7f411c009780 comp rx=0 tx=0).stop 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 1 -- 172.21.15.144:0/590583747 shutdown_connections 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 1 --2- 172.21.15.144:0/590583747 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4124069d10 0x7f412413a490 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 1 -- 172.21.15.144:0/590583747 >> 172.21.15.144:0/590583747 conn(0x7f41240a5d40 msgr2=0x7f41240a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 1 -- 172.21.15.144:0/590583747 shutdown_connections 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 1 -- 172.21.15.144:0/590583747 wait complete. 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 10 monclient: build_initial_monmap 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.473+0000 7f41294e8640 10 monclient: monmap: 2024-09-18T12:30:36.484 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 Processor -- start 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 -- start start 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient: init 2024-09-18T12:30:36.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 0x7f412409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4124069d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f412409ec50 0x7f412411bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): picked mon.noname-a con 0x7f412409ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f412409f020 0x7f4124130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): picked mon.noname-c con 0x7f412409f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412409b630 con 0x7f412409ec50 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412411c170 con 0x7f4124069d10 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:36.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412411df10 con 0x7f412409f020 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41294e8640 10 monclient(hunting): authenticate will time out at 1072.043661s 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41227fc640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f412409ec50 msgr2=0x7f412411bc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41227fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f412409ec50 0x7f412411bc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 0x7f412409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 0x7f412409a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42396/0 (socket says 172.21.15.144:42396) 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 1 -- 172.21.15.144:0/216906929 learned_addr learned my addr 172.21.15.144:0/216906929 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 10 monclient(hunting): get_auth_request con 0x7f4124069d10 auth_method 0 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:36.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f412409f020 0x7f4124130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f4122ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 10 monclient(hunting): get_auth_request con 0x7f412409f020 auth_method 0 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.474+0000 7f41237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 10 monclient(hunting): handle_auth_done global_id 5563 payload 824 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 1 -- 172.21.15.144:0/216906929 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f412409f020 msgr2=0x7f4124130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:36.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f412409f020 0x7f4124130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 1 -- 172.21.15.144:0/216906929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f412409ec50 msgr2=0x7f412411bc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f412409ec50 0x7f412411bc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 1 monclient: found mon.noname-b 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.475+0000 7f4122ffd640 1 -- 172.21.15.144:0/216906929 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c009a20 con 0x7f4124069d10 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f4122ffd640 1 -- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 msgr2=0x7f412409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f4122ffd640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 0x7f412409a9d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f411400b600 tx=0x7f41140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114015070 con 0x7f4124069d10 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:36.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: dump: 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: _finish_auth 0 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:36.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4114003d90 con 0x7f4124069d10 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41294e8640 5 monclient: authenticate success, global_id 5563 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41294e8640 10 monclient: _renew_subs 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41294e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412411cf80 con 0x7f4124069d10 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114006380 con 0x7f4124069d10 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f410affd640 10 monclient: dump: 2024-09-18T12:30:36.491 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41294e8640 10 monclient: _renew_subs 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41294e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412411dce0 con 0x7f4124069d10 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.477+0000 7f41217fa640 4 set_mon_vals no callback set 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4114010440 con 0x7f4124069d10 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f41294e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f41294e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f41294e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41240008d0 con 0x7f4124069d10 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f410affd640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4110038440 0x7f411003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f411404b9e0 con 0x7f4124069d10 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f41227fc640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4110038440 0x7f411003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:36.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.478+0000 7f41227fc640 10 monclient: get_auth_request con 0x7f4110038440 auth_method 0 2024-09-18T12:30:36.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.481+0000 7f41227fc640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4110038440 0x7f411003a8f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f412409b1a0 tx=0x7f4118007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:36.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.482+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4114018ab0 con 0x7f4124069d10 2024-09-18T12:30:36.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.482+0000 7f410affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:36.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.482+0000 7f410affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:36.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.646+0000 7f41294e8640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T12:30:36.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.646+0000 7f41294e8640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T12:30:36.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.646+0000 7f41294e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:36.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:36.646+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f41240a13e0 con 0x7f4124069d10 2024-09-18T12:30:37.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:37.600+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v120) v1 ==== 168+0+0 (secure 0 0 0) 0x7f4114018690 con 0x7f4124069d10 2024-09-18T12:30:37.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:37.600+0000 7f410affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T12:30:37.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:37.600+0000 7f410affd640 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-18T12:30:37.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:37.700+0000 7f41294e8640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T12:30:37.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:37.701+0000 7f41294e8640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T12:30:37.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:37.701+0000 7f41294e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:37.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:37.701+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f412409d510 con 0x7f4124069d10 2024-09-18T12:30:38.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.608+0000 7f410affd640 1 -- 172.21.15.144:0/216906929 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v121) v1 ==== 168+0+0 (secure 0 0 0) 0x7f41140280b0 con 0x7f4124069d10 2024-09-18T12:30:38.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.608+0000 7f410affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T12:30:38.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.608+0000 7f410affd640 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-18T12:30:38.610 INFO:tasks.workunit.client.0.smithi144.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4110038440 msgr2=0x7f411003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4110038440 0x7f411003a8f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f412409b1a0 tx=0x7f4118007b80 comp rx=0 tx=0).stop 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 10 monclient: shutdown 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 msgr2=0x7f412409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 0x7f412409a9d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f411400b600 tx=0x7f41140091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 shutdown_connections 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4110038440 0x7f411003a8f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f412409f020 0x7f4124130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f412409ec50 0x7f412411bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 --2- 172.21.15.144:0/216906929 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4124069d10 0x7f412409a9d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 >> 172.21.15.144:0/216906929 conn(0x7f41240a5d40 msgr2=0x7f4124134ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:38.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 shutdown_connections 2024-09-18T12:30:38.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.611+0000 7f41294e8640 1 -- 172.21.15.144:0/216906929 wait complete. 2024-09-18T12:30:38.624 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-18T12:30:38.624 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:30:38.624 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-18T12:30:38.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.702+0000 7fdcbfd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.702+0000 7fdcbfd9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.702+0000 7fdcbfd9d640 10 monclient: monmap: 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:38.703584+0000 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:38.703584+0000 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:38.704 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:38.705 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:38.705 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 1 Processor -- start 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 1 -- start start 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient: init 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcb8067640 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdcb809d030 0x7fdcb8094790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdcb809d030 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcb8094cd0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb809d930 con 0x7fdcb8094cd0 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb8067a10 con 0x7fdcb8067640 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb8099370 con 0x7fdcb809d030 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.703+0000 7fdcbfd9d640 10 monclient(hunting): authenticate will time out at 1074.272958s 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbd338640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdcb809d030 0x7fdcb8094790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbe33a640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fdcb8094cd0 legacy=0x7fdcb8139240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbd338640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdcb809d030 0x7fdcb8094790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44416/0 (socket says 172.21.15.144:44416) 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbd338640 1 -- 172.21.15.144:0/3127947663 learned_addr learned my addr 172.21.15.144:0/3127947663 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbd338640 10 monclient(hunting): get_auth_request con 0x7fdcb809d030 auth_method 0 2024-09-18T12:30:38.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbd338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:38.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbd338640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:38.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.704+0000 7fdcbd338640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbdb39640 1 -- 172.21.15.144:0/3127947663 >> v1:172.21.15.144:6789/0 conn(0x7fdcb8067640 legacy=0x7fdcb8094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 10 monclient(hunting): handle_auth_done global_id 5566 payload 274 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 1 -- 172.21.15.144:0/3127947663 >> v1:172.21.15.144:6789/0 conn(0x7fdcb8067640 legacy=0x7fdcb8094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 1 -- 172.21.15.144:0/3127947663 >> v1:172.21.15.116:6789/0 conn(0x7fdcb8094cd0 legacy=0x7fdcb8139240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 10 monclient: _finish_hunting 0 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 1 monclient: found mon.noname-c 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.706+0000 7fdcbd338640 1 -- 172.21.15.144:0/3127947663 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb809d650 con 0x7fdcb809d030 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbd338640 1 -- 172.21.15.144:0/3127947663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb809d030 msgr2=0x7fdcb8094790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbd338640 1 --2- 172.21.15.144:0/3127947663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb809d030 0x7fdcb8094790 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009990 tx=0x7fdca80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=594da08c09996eab server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 1 -- 172.21.15.144:0/3127947663 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca802ae90 con 0x7fdcb809d030 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: dump: 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.714 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: _finish_auth 0 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 5 monclient: authenticate success, global_id 5566 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 1 -- 172.21.15.144:0/3127947663 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca802f4b0 con 0x7fdcb809d030 2024-09-18T12:30:38.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 4 set_mon_vals no callback set 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 1 -- 172.21.15.144:0/3127947663 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca802f850 con 0x7fdcb809d030 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbcb37640 10 monclient: dump: 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.716 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 10 monclient: shutdown 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/3127947663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb809d030 msgr2=0x7fdcb8094790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 1 --2- 172.21.15.144:0/3127947663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb809d030 0x7fdcb8094790 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009990 tx=0x7fdca80091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/3127947663 shutdown_connections 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 1 --2- 172.21.15.144:0/3127947663 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb809d030 0x7fdcb8094790 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.709+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/3127947663 >> 172.21.15.144:0/3127947663 conn(0x7fdcb80a1630 msgr2=0x7fdcb80a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.710+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/3127947663 shutdown_connections 2024-09-18T12:30:38.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.710+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/3127947663 wait complete. 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.710+0000 7fdcbfd9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.710+0000 7fdcbfd9d640 10 monclient: monmap: 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.710+0000 7fdcbfd9d640 1 Processor -- start 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.710+0000 7fdcbfd9d640 1 -- start start 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.710+0000 7fdcbfd9d640 10 monclient: init 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb8067640 0x7fdcb81412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdcb8067640 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcb8094cd0 0x7fdcb81340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdcb8094cd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb809d030 0x7fdcb812acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdcb809d030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb812e590 con 0x7fdcb809d030 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb812cf40 con 0x7fdcb8067640 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb812c8c0 con 0x7fdcb8094cd0 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbfd9d640 10 monclient(hunting): authenticate will time out at 1074.280328s 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb8067640 0x7fdcb81412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb8067640 0x7fdcb81412c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44430/0 (socket says 172.21.15.144:44430) 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 -- 172.21.15.144:0/867035773 learned_addr learned my addr 172.21.15.144:0/867035773 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): get_auth_request con 0x7fdcb8067640 auth_method 0 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:38.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbe33a640 1 -- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb809d030 msgr2=0x7fdcb812acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbe33a640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb809d030 0x7fdcb812acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient(hunting): handle_auth_done global_id 5572 payload 824 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 -- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcb8094cd0 msgr2=0x7fdcb81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbd338640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcb8094cd0 0x7fdcb81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcb8094cd0 0x7fdcb81340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 -- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb809d030 msgr2=0x7fdcb812acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb809d030 0x7fdcb812acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient: _finish_hunting 0 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 monclient: found mon.noname-b 2024-09-18T12:30:38.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:38.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbdb39640 1 -- 172.21.15.144:0/867035773 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca8009450 con 0x7fdcb8067640 2024-09-18T12:30:38.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbd338640 10 monclient: get_auth_request con 0x7fdcb8094cd0 auth_method 0 2024-09-18T12:30:38.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.711+0000 7fdcbd338640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdcb8094cd0 0x7fdcb81340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.713+0000 7fdcbdb39640 1 -- 172.21.15.144:0/867035773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb8067640 msgr2=0x7fdcb81412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.713+0000 7fdcbdb39640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdcb8067640 0x7fdcb81412c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400c590 tx=0x7fdcb4007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 1 -- 172.21.15.144:0/867035773 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb40075f0 con 0x7fdcb8067640 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: dump: 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.724 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcbfd9d640 5 monclient: authenticate success, global_id 5572 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 1 -- 172.21.15.144:0/867035773 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb4005780 con 0x7fdcb8067640 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcbfd9d640 10 monclient: _renew_subs 2024-09-18T12:30:38.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcbfd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/867035773 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb8142120 con 0x7fdcb8067640 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 1 -- 172.21.15.144:0/867035773 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4005b40 con 0x7fdcb8067640 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcadffb640 10 monclient: dump: 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.726 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcbfd9d640 10 monclient: _renew_subs 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcbfd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcaffff640 4 set_mon_vals no callback set 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.714+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/867035773 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb812e2b0 con 0x7fdcb8067640 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcadffb640 1 -- 172.21.15.144:0/867035773 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdcb401b3b0 con 0x7fdcb8067640 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcbfd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcbfd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcbfd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/867035773 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb812ea20 con 0x7fdcb8067640 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcadffb640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdc9c038440 0x7fdc9c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcadffb640 1 -- 172.21.15.144:0/867035773 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(121..121 src has 13..121) v4 ==== 6567+0+0 (secure 0 0 0) 0x7fdcb404b360 con 0x7fdcb8067640 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcbd338640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdc9c038440 0x7fdc9c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.715+0000 7fdcbd338640 10 monclient: get_auth_request con 0x7fdc9c038440 auth_method 0 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.717+0000 7fdcbd338640 1 --2- 172.21.15.144:0/867035773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdc9c038440 0x7fdc9c03a8f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdcb80649b0 tx=0x7fdca8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.719+0000 7fdcadffb640 1 -- 172.21.15.144:0/867035773 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdcb812ea20 con 0x7fdcb8067640 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.719+0000 7fdcadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:38.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.719+0000 7fdcadffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:38.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.877+0000 7fdcbfd9d640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-18T12:30:38.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.877+0000 7fdcbfd9d640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-18T12:30:38.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.877+0000 7fdcbfd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.877+0000 7fdcbfd9d640 1 -- 172.21.15.144:0/867035773 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) v1 -- 0x7fdcb812ea20 con 0x7fdcb8067640 2024-09-18T12:30:38.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.878+0000 7fdcadffb640 1 -- 172.21.15.144:0/867035773 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v121) v1 ==== 154+0+0 (secure 0 0 0) 0x7fdcb40170b0 con 0x7fdcb8067640 2024-09-18T12:30:38.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.878+0000 7fdcadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-18T12:30:38.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.878+0000 7fdcadffb640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-18T12:30:38.880 INFO:tasks.workunit.client.0.smithi144.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-18T12:30:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:30:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:30:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep osd.12 2024-09-18T12:30:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 155 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.953+0000 7f1df20ef640 10 monclient: get_monmap_and_config 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.953+0000 7f1df20ef640 10 monclient: build_initial_monmap 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.953+0000 7f1df20ef640 10 monclient: monmap: 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:38.954844+0000 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:38.954844+0000 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:38.955 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:38.956 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:38.956 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:38.956 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:38.956 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.954+0000 7f1df20ef640 1 Processor -- start 2024-09-18T12:30:38.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.954+0000 7f1df20ef640 1 -- start start 2024-09-18T12:30:38.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.954+0000 7f1df20ef640 10 monclient: init 2024-09-18T12:30:38.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:38.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1dec09c230 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec09ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient(hunting): picked mon.noname-b con 0x7f1dec09c600 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1dec134860 0x7f1dec136c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient(hunting): picked mon.noname-a con 0x7f1dec134860 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec09b1d0 con 0x7f1dec09c230 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dec09f5b0 con 0x7f1dec134860 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec09b470 con 0x7f1dec09c600 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1df20ef640 10 monclient(hunting): authenticate will time out at 1074.524398s 2024-09-18T12:30:38.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1deb7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1dec09c230 legacy=0x7f1dec09cc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1debfff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1dec134860 msgr2=0x7f1dec136c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1debfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1dec134860 0x7f1dec136c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec09ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec09ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:58244/0 (socket says 172.21.15.144:58244) 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 -- 172.21.15.144:0/3441234742 learned_addr learned my addr 172.21.15.144:0/3441234742 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): get_auth_request con 0x7f1dec09c600 auth_method 0 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient(hunting): handle_auth_done global_id 64412 payload 274 2024-09-18T12:30:38.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 -- 172.21.15.144:0/3441234742 >> v2:172.21.15.116:3300/0 conn(0x7f1dec134860 msgr2=0x7f1dec136c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:38.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 --2- 172.21.15.144:0/3441234742 >> v2:172.21.15.116:3300/0 conn(0x7f1dec134860 0x7f1dec136c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 -- 172.21.15.144:0/3441234742 >> v1:172.21.15.116:6789/0 conn(0x7f1dec09c230 legacy=0x7f1dec09cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:38.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:38.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 monclient: found mon.noname-b 2024-09-18T12:30:38.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:38.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:38.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.955+0000 7f1de3fff640 1 -- 172.21.15.144:0/3441234742 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec09d340 con 0x7f1dec09c600 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.958+0000 7f1de3fff640 1 -- 172.21.15.144:0/3441234742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 msgr2=0x7f1dec09ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.958+0000 7f1de3fff640 1 --2- 172.21.15.144:0/3441234742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec09ea70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009910 tx=0x7f1dd40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd08320055f784bc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 1 -- 172.21.15.144:0/3441234742 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4032440 con 0x7f1dec09c600 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: dump: 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.967 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 1 -- 172.21.15.144:0/3441234742 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd40325a0 con 0x7f1dec09c600 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1df20ef640 5 monclient: authenticate success, global_id 64412 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1df20ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 1 -- 172.21.15.144:0/3441234742 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4032940 con 0x7f1dec09c600 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1df20ef640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:38.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1df20ef640 4 set_mon_vals no callback set 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.959+0000 7f1deaffd640 10 monclient: dump: 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.969 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 10 monclient: shutdown 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 -- 172.21.15.144:0/3441234742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 msgr2=0x7f1dec09ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 --2- 172.21.15.144:0/3441234742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec09ea70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009910 tx=0x7f1dd40091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 -- 172.21.15.144:0/3441234742 shutdown_connections 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 --2- 172.21.15.144:0/3441234742 >> v2:172.21.15.116:3300/0 conn(0x7f1dec134860 0x7f1dec136c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 --2- 172.21.15.144:0/3441234742 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec09ea70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 -- 172.21.15.144:0/3441234742 >> 172.21.15.144:0/3441234742 conn(0x7f1dec071350 msgr2=0x7f1dec0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 -- 172.21.15.144:0/3441234742 shutdown_connections 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 1 -- 172.21.15.144:0/3441234742 wait complete. 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 10 monclient: build_initial_monmap 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.960+0000 7f1df20ef640 10 monclient: monmap: 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:38.970 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 Processor -- start 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 -- start start 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient: init 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:38.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 0x7f1dec11a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): picked mon.noname-b con 0x7f1dec09c230 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec11a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): picked mon.noname-c con 0x7f1dec09c600 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dec134860 0x7f1dec11adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): picked mon.noname-a con 0x7f1dec134860 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec06ec10 con 0x7f1dec134860 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0a1f80 con 0x7f1dec09c230 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec11c040 con 0x7f1dec09c600 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1df20ef640 10 monclient(hunting): authenticate will time out at 1074.530636s 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 0x7f1dec11a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1debfff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dec134860 msgr2=0x7f1dec11adf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 0x7f1dec11a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44446/0 (socket says 172.21.15.144:44446) 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1debfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dec134860 0x7f1dec11adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 1 -- 172.21.15.144:0/4054378025 learned_addr learned my addr 172.21.15.144:0/4054378025 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): get_auth_request con 0x7f1dec09c230 auth_method 0 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:38.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec11a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 10 monclient(hunting): get_auth_request con 0x7f1dec09c600 auth_method 0 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1de3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.961+0000 7f1deb7fe640 10 monclient(hunting): handle_auth_done global_id 5575 payload 824 2024-09-18T12:30:38.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 1 -- 172.21.15.144:0/4054378025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 msgr2=0x7f1dec11a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec11a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 1 -- 172.21.15.144:0/4054378025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dec134860 msgr2=0x7f1dec11adf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dec134860 0x7f1dec11adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.962+0000 7f1deb7fe640 1 -- 172.21.15.144:0/4054378025 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd4009450 con 0x7f1dec09c230 2024-09-18T12:30:38.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1deb7fe640 1 -- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 msgr2=0x7f1dec11a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1deb7fe640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 0x7f1dec11a370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00b600 tx=0x7f1ddc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc015070 con 0x7f1dec09c230 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: dump: 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.976 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc003d90 con 0x7f1dec09c230 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1df20ef640 5 monclient: authenticate success, global_id 5575 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1df20ef640 10 monclient: _renew_subs 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1df20ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec11d660 con 0x7f1dec09c230 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc006380 con 0x7f1dec09c230 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:38.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1de37fe640 10 monclient: dump: 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1dea7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:38.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1df20ef640 10 monclient: _renew_subs 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1df20ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.964+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec11c700 con 0x7f1dec09c230 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1ddc010440 con 0x7f1dec09c230 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1de37fe640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dcc038440 0x7f1dcc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1df20ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1df20ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1df20ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(121..121 src has 13..121) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f1ddc04b970 con 0x7f1dec09c230 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec11d8a0 con 0x7f1dec09c230 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1de3fff640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dcc038440 0x7f1dcc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:38.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.965+0000 7f1de3fff640 10 monclient: get_auth_request con 0x7f1dcc038440 auth_method 0 2024-09-18T12:30:38.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.968+0000 7f1de3fff640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dcc038440 0x7f1dcc03a8f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1dec09fac0 tx=0x7f1dd4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:38.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.970+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1ddc018910 con 0x7f1dec09c230 2024-09-18T12:30:38.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.970+0000 7f1de37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:38.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:38.970+0000 7f1de37fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:39.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.123+0000 7f1df20ef640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:39.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.123+0000 7f1df20ef640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.123+0000 7f1df20ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.123+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1dec0636e0 con 0x7f1dec09c230 2024-09-18T12:30:39.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.123+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v121) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f1ddc029070 con 0x7f1dec09c230 2024-09-18T12:30:39.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.123+0000 7f1de37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.123+0000 7f1de37fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:39.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.215+0000 7f1df20ef640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:39.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.215+0000 7f1df20ef640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.215+0000 7f1df20ef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.215+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1dec072640 con 0x7f1dec09c230 2024-09-18T12:30:39.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.216+0000 7f1de37fe640 1 -- 172.21.15.144:0/4054378025 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v121) v1 ==== 56+0+1449 (secure 0 0 0) 0x7f1ddc018310 con 0x7f1dec09c230 2024-09-18T12:30:39.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.216+0000 7f1de37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.216+0000 7f1de37fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:39.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dcc038440 msgr2=0x7f1dcc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dcc038440 0x7f1dcc03a8f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1dec09fac0 tx=0x7f1dd4032000 comp rx=0 tx=0).stop 2024-09-18T12:30:39.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 10 monclient: shutdown 2024-09-18T12:30:39.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:39.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 msgr2=0x7f1dec11a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 0x7f1dec11a370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00b600 tx=0x7f1ddc0091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 shutdown_connections 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1dcc038440 0x7f1dcc03a8f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dec134860 0x7f1dec11adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1dec09c600 0x7f1dec11a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 --2- 172.21.15.144:0/4054378025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1dec09c230 0x7f1dec11a370 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 >> 172.21.15.144:0/4054378025 conn(0x7f1dec071350 msgr2=0x7f1dec135610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 shutdown_connections 2024-09-18T12:30:39.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.218+0000 7f1df20ef640 1 -- 172.21.15.144:0/4054378025 wait complete. 2024-09-18T12:30:39.228 INFO:tasks.workunit.client.0.smithi144.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2024-09-18T12:30:39.228 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd tree 2024-09-18T12:30:39.228 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep osd.14 2024-09-18T12:30:39.228 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 155 2024-09-18T12:30:39.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.298+0000 7fb4a7b1d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:39.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.298+0000 7fb4a7b1d640 10 monclient: build_initial_monmap 2024-09-18T12:30:39.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.298+0000 7fb4a7b1d640 10 monclient: monmap: 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:39.299401+0000 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:39.299401+0000 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:39.300 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 1 Processor -- start 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 1 -- start start 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient: init 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4a009c230 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4a009c600 0x7fb4a009ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a009c600 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb4a0134860 0x7fb4a0136c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a0134860 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a009b1d0 con 0x7fb4a009c230 2024-09-18T12:30:39.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a009f5b0 con 0x7fb4a009c600 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a009b470 con 0x7fb4a0134860 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a7b1d640 10 monclient(hunting): authenticate will time out at 1074.868862s 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a60ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb4a0134860 0x7fb4a0136c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a50b8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fb4a009c600 msgr2=0x7fb4a009ea70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.299+0000 7fb4a58b9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fb4a009c230 legacy=0x7fb4a009cc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb4a0134860 0x7fb4a0136c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44458/0 (socket says 172.21.15.144:44458) 2024-09-18T12:30:39.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a50b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb4a009c600 0x7fb4a009ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 1 -- 172.21.15.144:0/2967974463 learned_addr learned my addr 172.21.15.144:0/2967974463 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): get_auth_request con 0x7fb4a0134860 auth_method 0 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient(hunting): handle_auth_done global_id 5578 payload 274 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 1 -- 172.21.15.144:0/2967974463 >> v2:172.21.15.116:3300/0 conn(0x7fb4a009c600 msgr2=0x7fb4a009ea70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 1 --2- 172.21.15.144:0/2967974463 >> v2:172.21.15.116:3300/0 conn(0x7fb4a009c600 0x7fb4a009ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 1 -- 172.21.15.144:0/2967974463 >> v1:172.21.15.116:6789/0 conn(0x7fb4a009c230 legacy=0x7fb4a009cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:39.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient: _finish_hunting 0 2024-09-18T12:30:39.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 1 monclient: found mon.noname-c 2024-09-18T12:30:39.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:39.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.300+0000 7fb4a60ba640 1 -- 172.21.15.144:0/2967974463 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a009d340 con 0x7fb4a0134860 2024-09-18T12:30:39.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.302+0000 7fb4a60ba640 1 -- 172.21.15.144:0/2967974463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a0134860 msgr2=0x7fb4a0136c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:39.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.302+0000 7fb4a60ba640 1 --2- 172.21.15.144:0/2967974463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a0134860 0x7fb4a0136c40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb4a009b850 tx=0x7fb498009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ea232409a9a8d56 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:39.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 1 -- 172.21.15.144:0/2967974463 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498007610 con 0x7fb4a0134860 2024-09-18T12:30:39.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: dump: 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: _finish_auth 0 2024-09-18T12:30:39.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 5 monclient: authenticate success, global_id 5578 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 1 -- 172.21.15.144:0/2967974463 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49802d050 con 0x7fb4a0134860 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 4 set_mon_vals no callback set 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 1 -- 172.21.15.144:0/2967974463 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4980079a0 con 0x7fb4a0134860 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a48b7640 10 monclient: dump: 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.313 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 10 monclient: shutdown 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/2967974463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a0134860 msgr2=0x7fb4a0136c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/2967974463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a0134860 0x7fb4a0136c40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb4a009b850 tx=0x7fb498009840 comp rx=0 tx=0).stop 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/2967974463 shutdown_connections 2024-09-18T12:30:39.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/2967974463 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a0134860 0x7fb4a0136c40 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/2967974463 >> v2:172.21.15.116:3300/0 conn(0x7fb4a009c600 0x7fb4a009ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/2967974463 >> 172.21.15.144:0/2967974463 conn(0x7fb4a0071350 msgr2=0x7fb4a00a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/2967974463 shutdown_connections 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.303+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/2967974463 wait complete. 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.304+0000 7fb4a7b1d640 10 monclient: build_initial_monmap 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.304+0000 7fb4a7b1d640 10 monclient: monmap: 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.315 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.304+0000 7fb4a7b1d640 1 Processor -- start 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.304+0000 7fb4a7b1d640 1 -- start start 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.304+0000 7fb4a7b1d640 10 monclient: init 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.304+0000 7fb4a7b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4a009c230 0x7fb4a0068a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a009c230 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 0x7fb4a0068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a009c600 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4a0134860 0x7fb4a00694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a0134860 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0134420 con 0x7fb4a0134860 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00a1ad0 con 0x7fb4a009c600 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a006aa10 con 0x7fb4a009c230 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a7b1d640 10 monclient(hunting): authenticate will time out at 1074.874074s 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a60ba640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4a0134860 msgr2=0x7fb4a00694a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 0x7fb4a0068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a60ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4a0134860 0x7fb4a00694a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 0x7fb4a0068f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44472/0 (socket says 172.21.15.144:44472) 2024-09-18T12:30:39.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 -- 172.21.15.144:0/4016943501 learned_addr learned my addr 172.21.15.144:0/4016943501 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): get_auth_request con 0x7fb4a009c600 auth_method 0 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a58b9640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4a009c230 0x7fb4a0068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a58b9640 10 monclient(hunting): get_auth_request con 0x7fb4a009c230 auth_method 0 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a58b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a58b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a58b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:39.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient(hunting): handle_auth_done global_id 5581 payload 824 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 -- 172.21.15.144:0/4016943501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4a009c230 msgr2=0x7fb4a0068a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4a009c230 0x7fb4a0068a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 -- 172.21.15.144:0/4016943501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4a0134860 msgr2=0x7fb4a00694a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4a0134860 0x7fb4a00694a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient: _finish_hunting 0 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 monclient: found mon.noname-b 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.305+0000 7fb4a50b8640 1 -- 172.21.15.144:0/4016943501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb498009ae0 con 0x7fb4a009c600 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a50b8640 1 -- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 msgr2=0x7fb4a0068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a50b8640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 0x7fb4a0068f60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb49000b600 tx=0x7fb4900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490015070 con 0x7fb4a009c600 2024-09-18T12:30:39.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: dump: 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:39.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb490003d90 con 0x7fb4a009c600 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a7b1d640 5 monclient: authenticate success, global_id 5581 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a7b1d640 10 monclient: _renew_subs 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a011f3b0 con 0x7fb4a009c600 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490006380 con 0x7fb4a009c600 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb495ffb640 10 monclient: dump: 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.321 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a7b1d640 10 monclient: _renew_subs 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb497fff640 4 set_mon_vals no callback set 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.307+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.308+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a011e450 con 0x7fb4a009c600 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.308+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb490010440 con 0x7fb4a009c600 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.308+0000 7fb495ffb640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb484038440 0x7fb48403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.308+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(121..121 src has 13..121) v4 ==== 6567+0+0 (secure 0 0 0) 0x7fb49004b9b0 con 0x7fb4a009c600 2024-09-18T12:30:39.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.309+0000 7fb4a7b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:39.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.309+0000 7fb4a7b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:39.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.309+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.309+0000 7fb4a58b9640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb484038440 0x7fb48403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.309+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a00636e0 con 0x7fb4a009c600 2024-09-18T12:30:39.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.309+0000 7fb4a58b9640 10 monclient: get_auth_request con 0x7fb484038440 auth_method 0 2024-09-18T12:30:39.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.311+0000 7fb4a58b9640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb484038440 0x7fb48403a8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb49c006fd0 tx=0x7fb49c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:39.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.313+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb490018ab0 con 0x7fb4a009c600 2024-09-18T12:30:39.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.313+0000 7fb495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:39.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.313+0000 7fb495ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:39.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.466+0000 7fb4a7b1d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:39.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.466+0000 7fb4a7b1d640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.466+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.466+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb4a0064b20 con 0x7fb4a009c600 2024-09-18T12:30:39.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.467+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v121) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fb490018690 con 0x7fb4a009c600 2024-09-18T12:30:39.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.467+0000 7fb495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.467+0000 7fb495ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:39.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.559+0000 7fb4a7b1d640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T12:30:39.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.559+0000 7fb4a7b1d640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.559+0000 7fb4a7b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.559+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fb4a011f770 con 0x7fb4a009c600 2024-09-18T12:30:39.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.560+0000 7fb495ffb640 1 -- 172.21.15.144:0/4016943501 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v121) v1 ==== 56+0+1449 (secure 0 0 0) 0x7fb490018ab0 con 0x7fb4a009c600 2024-09-18T12:30:39.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.560+0000 7fb495ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T12:30:39.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.560+0000 7fb495ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:39.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb484038440 msgr2=0x7fb48403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb484038440 0x7fb48403a8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb49c006fd0 tx=0x7fb49c008040 comp rx=0 tx=0).stop 2024-09-18T12:30:39.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 10 monclient: shutdown 2024-09-18T12:30:39.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:39.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 msgr2=0x7fb4a0068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 0x7fb4a0068f60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb49000b600 tx=0x7fb4900091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 shutdown_connections 2024-09-18T12:30:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb484038440 0x7fb48403a8f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4a0134860 0x7fb4a00694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb4a009c600 0x7fb4a0068f60 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 --2- 172.21.15.144:0/4016943501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb4a009c230 0x7fb4a0068a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 >> 172.21.15.144:0/4016943501 conn(0x7fb4a0071350 msgr2=0x7fb4a0134fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 shutdown_connections 2024-09-18T12:30:39.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.562+0000 7fb4a7b1d640 1 -- 172.21.15.144:0/4016943501 wait complete. 2024-09-18T12:30:39.572 INFO:tasks.workunit.client.0.smithi144.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2024-09-18T12:30:39.572 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm osd.12 2024-09-18T12:30:39.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.645+0000 7ff16f100640 10 monclient: get_monmap_and_config 2024-09-18T12:30:39.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.645+0000 7ff16f100640 10 monclient: build_initial_monmap 2024-09-18T12:30:39.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.645+0000 7ff16f100640 10 monclient: monmap: 2024-09-18T12:30:39.646 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:39.646 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.646 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:39.646275+0000 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:39.646275+0000 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:39.647 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 1 Processor -- start 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 1 -- start start 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient: init 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff168069bd0 0x7ff1681320a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient(hunting): picked mon.noname-c con 0x7ff168069bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:39.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff168098330 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff168136d50 0x7ff168139130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient(hunting): picked mon.noname-a con 0x7ff168136d50 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff168096bd0 con 0x7ff168098330 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff168131db0 con 0x7ff168136d50 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff168098e80 con 0x7ff168069bd0 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16f100640 10 monclient(hunting): authenticate will time out at 1075.215605s 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff168069bd0 0x7ff1681320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff168069bd0 0x7ff1681320a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44488/0 (socket says 172.21.15.144:44488) 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 1 -- 172.21.15.144:0/1981455720 learned_addr learned my addr 172.21.15.144:0/1981455720 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 10 monclient(hunting): get_auth_request con 0x7ff168069bd0 auth_method 0 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16d69d640 1 -- 172.21.15.144:0/1981455720 >> v2:172.21.15.116:3300/0 conn(0x7ff168136d50 msgr2=0x7ff168139130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16d69d640 1 --2- 172.21.15.144:0/1981455720 >> v2:172.21.15.116:3300/0 conn(0x7ff168136d50 0x7ff168139130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.646+0000 7ff16ce9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.648+0000 7ff167fff640 1 -- 172.21.15.144:0/1981455720 >> v1:172.21.15.144:6789/0 conn(0x7ff168098330 legacy=0x7ff168098770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 10 monclient(hunting): handle_auth_done global_id 5584 payload 274 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 1 -- 172.21.15.144:0/1981455720 >> v2:172.21.15.116:3300/0 conn(0x7ff168136d50 msgr2=0x7ff168139130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 1 --2- 172.21.15.144:0/1981455720 >> v2:172.21.15.116:3300/0 conn(0x7ff168136d50 0x7ff168139130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 1 -- 172.21.15.144:0/1981455720 >> v1:172.21.15.144:6789/0 conn(0x7ff168098330 legacy=0x7ff168098770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 1 monclient: found mon.noname-c 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.649+0000 7ff16ce9c640 1 -- 172.21.15.144:0/1981455720 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16809d7f0 con 0x7ff168069bd0 2024-09-18T12:30:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.651+0000 7ff16ce9c640 1 -- 172.21.15.144:0/1981455720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168069bd0 msgr2=0x7ff1681320a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.651+0000 7ff16ce9c640 1 --2- 172.21.15.144:0/1981455720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168069bd0 0x7ff1681320a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ff16809c530 tx=0x7ff15c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1e9a5827d3f6980 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.651+0000 7ff1677fe640 1 -- 172.21.15.144:0/1981455720 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c02f4b0 con 0x7ff168069bd0 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.651+0000 7ff1677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 10 monclient: dump: 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.657 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 1 -- 172.21.15.144:0/1981455720 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15c02f610 con 0x7ff168069bd0 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 5 monclient: authenticate success, global_id 5584 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 1 -- 172.21.15.144:0/1981455720 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c02f9b0 con 0x7ff168069bd0 2024-09-18T12:30:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 4 set_mon_vals no callback set 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff1677fe640 10 monclient: dump: 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.659 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 10 monclient: shutdown 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 -- 172.21.15.144:0/1981455720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168069bd0 msgr2=0x7ff1681320a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 --2- 172.21.15.144:0/1981455720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168069bd0 0x7ff1681320a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ff16809c530 tx=0x7ff15c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 -- 172.21.15.144:0/1981455720 shutdown_connections 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 --2- 172.21.15.144:0/1981455720 >> v2:172.21.15.116:3300/0 conn(0x7ff168136d50 0x7ff168139130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 --2- 172.21.15.144:0/1981455720 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168069bd0 0x7ff1681320a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 -- 172.21.15.144:0/1981455720 >> 172.21.15.144:0/1981455720 conn(0x7ff16809e9f0 msgr2=0x7ff16809edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 -- 172.21.15.144:0/1981455720 shutdown_connections 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 1 -- 172.21.15.144:0/1981455720 wait complete. 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.652+0000 7ff16f100640 10 monclient: build_initial_monmap 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient: monmap: 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 Processor -- start 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 -- start start 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient: init 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:30:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff168069bd0 0x7ff168135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): picked mon.noname-a con 0x7ff168069bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff168098330 0x7ff16812eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): picked mon.noname-c con 0x7ff168098330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 0x7ff168141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): picked mon.noname-b con 0x7ff168136d50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16809d5e0 con 0x7ff168069bd0 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16811b470 con 0x7ff168136d50 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff168119890 con 0x7ff168098330 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.653+0000 7ff16f100640 10 monclient(hunting): authenticate will time out at 1075.222942s 2024-09-18T12:30:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 0x7ff168141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 0x7ff168141330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44504/0 (socket says 172.21.15.144:44504) 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 -- 172.21.15.144:0/889343849 learned_addr learned my addr 172.21.15.144:0/889343849 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): get_auth_request con 0x7ff168136d50 auth_method 0 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16ce9c640 1 -- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff168069bd0 msgr2=0x7ff168135cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16ce9c640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff168069bd0 0x7ff168135cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff167fff640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff168098330 0x7ff16812eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient(hunting): handle_auth_done global_id 5587 payload 824 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 -- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff168098330 msgr2=0x7ff16812eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff168098330 0x7ff16812eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 -- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff168069bd0 msgr2=0x7ff168135cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff168069bd0 0x7ff168135cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 monclient: found mon.noname-b 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff16d69d640 1 -- 172.21.15.144:0/889343849 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c009450 con 0x7ff168136d50 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff167fff640 10 monclient: get_auth_request con 0x7ff168098330 auth_method 0 2024-09-18T12:30:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.654+0000 7ff167fff640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff168098330 0x7ff16812eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16d69d640 1 -- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 msgr2=0x7ff168141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16d69d640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 0x7ff168141330 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff16000bb60 tx=0x7ff1600097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16000cc00 con 0x7ff168136d50 2024-09-18T12:30:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: dump: 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16000cd60 con 0x7ff168136d50 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16f100640 5 monclient: authenticate success, global_id 5587 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16f100640 10 monclient: _renew_subs 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16f100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff160006150 con 0x7ff168136d50 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1680a05e0 con 0x7ff168136d50 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff166ffd640 4 set_mon_vals no callback set 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff164ff9640 10 monclient: dump: 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:39.669 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16f100640 10 monclient: _renew_subs 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16f100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.656+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff168119f50 con 0x7ff168136d50 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.657+0000 7ff16f100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:39.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.657+0000 7ff16f100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.657+0000 7ff16f100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.657+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff160012020 con 0x7ff168136d50 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.657+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff168119d30 con 0x7ff168136d50 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.657+0000 7ff164ff9640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff14c038440 0x7ff14c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.658+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(121..121 src has 13..121) v4 ==== 6567+0+0 (secure 0 0 0) 0x7ff16004bf00 con 0x7ff168136d50 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.658+0000 7ff16ce9c640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff14c038440 0x7ff14c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.658+0000 7ff16ce9c640 10 monclient: get_auth_request con 0x7ff14c038440 auth_method 0 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.660+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.660+0000 7ff16ce9c640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff14c038440 0x7ff14c03a8f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff16806a000 tx=0x7ff15c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.662+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff16001da80 con 0x7ff168136d50 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.662+0000 7ff164ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:39.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.662+0000 7ff164ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:39.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.821+0000 7ff16f100640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T12:30:39.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.821+0000 7ff16f100640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T12:30:39.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.821+0000 7ff16f100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:39.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:39.821+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7ff1680008d0 con 0x7ff168136d50 2024-09-18T12:30:40.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.620+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v122) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff160006640 con 0x7ff168136d50 2024-09-18T12:30:40.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.620+0000 7ff164ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T12:30:40.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.620+0000 7ff164ff9640 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-18T12:30:40.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.720+0000 7ff16f100640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T12:30:40.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.720+0000 7ff16f100640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T12:30:40.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.720+0000 7ff16f100640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:40.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.720+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7ff168072640 con 0x7ff168136d50 2024-09-18T12:30:40.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.720+0000 7ff164ff9640 1 -- 172.21.15.144:0/889343849 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v122) v1 ==== 126+0+0 (secure 0 0 0) 0x7ff160006640 con 0x7ff168136d50 2024-09-18T12:30:40.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.720+0000 7ff164ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T12:30:40.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.720+0000 7ff164ff9640 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-18T12:30:40.722 INFO:tasks.workunit.client.0.smithi144.stderr:device 'osd.12' does not appear in the crush map 2024-09-18T12:30:40.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff14c038440 msgr2=0x7ff14c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff14c038440 0x7ff14c03a8f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff16806a000 tx=0x7ff15c032000 comp rx=0 tx=0).stop 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 10 monclient: shutdown 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 msgr2=0x7ff168141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 0x7ff168141330 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff16000bb60 tx=0x7ff1600097a0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 shutdown_connections 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff14c038440 0x7ff14c03a8f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff168136d50 0x7ff168141330 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff168098330 0x7ff16812eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 --2- 172.21.15.144:0/889343849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff168069bd0 0x7ff168135cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 >> 172.21.15.144:0/889343849 conn(0x7ff16809e9f0 msgr2=0x7ff1680970b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:40.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 shutdown_connections 2024-09-18T12:30:40.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.722+0000 7ff16f100640 1 -- 172.21.15.144:0/889343849 wait complete. 2024-09-18T12:30:40.733 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm osd.14 2024-09-18T12:30:40.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.804+0000 7feca1ee8640 10 monclient: get_monmap_and_config 2024-09-18T12:30:40.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.804+0000 7feca1ee8640 10 monclient: build_initial_monmap 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.804+0000 7feca1ee8640 10 monclient: monmap: 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:40.805559+0000 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:40.805559+0000 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:40.807 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:40.807 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:40.807 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:40.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.805+0000 7feca1ee8640 1 Processor -- start 2024-09-18T12:30:40.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.805+0000 7feca1ee8640 1 -- start start 2024-09-18T12:30:40.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.805+0000 7feca1ee8640 10 monclient: init 2024-09-18T12:30:40.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.805+0000 7feca1ee8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.805+0000 7feca1ee8640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.805+0000 7feca1ee8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fec9c096a30 0x7fec9c09d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 10 monclient(hunting): picked mon.noname-a con 0x7fec9c096a30 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec9c09b0b0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fec9c097510 0x7fec9c136e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 10 monclient(hunting): picked mon.noname-c con 0x7fec9c097510 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c09b970 con 0x7fec9c09b0b0 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9c09c470 con 0x7fec9c096a30 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9c09c190 con 0x7fec9c097510 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:40.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7feca1ee8640 10 monclient(hunting): authenticate will time out at 1076.375065s 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fec9c097510 0x7fec9c136e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fec9c096a30 msgr2=0x7fec9c09d320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fec9c096a30 0x7fec9c09d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fec9c097510 0x7fec9c136e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44520/0 (socket says 172.21.15.144:44520) 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 1 -- 172.21.15.144:0/254211280 learned_addr learned my addr 172.21.15.144:0/254211280 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 10 monclient(hunting): get_auth_request con 0x7fec9c097510 auth_method 0 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.806+0000 7fec9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:40.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec93fff640 1 -- 172.21.15.144:0/254211280 >> v1:172.21.15.144:6789/0 conn(0x7fec9c09b0b0 legacy=0x7fec9c096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 10 monclient(hunting): handle_auth_done global_id 5590 payload 274 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 1 -- 172.21.15.144:0/254211280 >> v2:172.21.15.116:3300/0 conn(0x7fec9c096a30 msgr2=0x7fec9c09d320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 1 --2- 172.21.15.144:0/254211280 >> v2:172.21.15.116:3300/0 conn(0x7fec9c096a30 0x7fec9c09d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 1 -- 172.21.15.144:0/254211280 >> v1:172.21.15.144:6789/0 conn(0x7fec9c09b0b0 legacy=0x7fec9c096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 1 monclient: found mon.noname-c 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:40.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.808+0000 7fec9bfff640 1 -- 172.21.15.144:0/254211280 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c09c750 con 0x7fec9c097510 2024-09-18T12:30:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9bfff640 1 -- 172.21.15.144:0/254211280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c097510 msgr2=0x7fec9c136e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9bfff640 1 --2- 172.21.15.144:0/254211280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c097510 0x7fec9c136e00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fec9c09ac00 tx=0x7fec94009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f841d3277231324 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:40.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 1 -- 172.21.15.144:0/254211280 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9402e050 con 0x7fec9c097510 2024-09-18T12:30:40.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:40.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: dump: 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:40.818 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: _finish_auth 0 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 1 -- 172.21.15.144:0/254211280 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec94007610 con 0x7fec9c097510 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7feca1ee8640 5 monclient: authenticate success, global_id 5590 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7feca1ee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7feca1ee8640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 1 -- 172.21.15.144:0/254211280 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec940079f0 con 0x7fec9c097510 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:40.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7feca1ee8640 4 set_mon_vals no callback set 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.811+0000 7fec9affd640 10 monclient: dump: 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:40.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 10 monclient: shutdown 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 -- 172.21.15.144:0/254211280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c097510 msgr2=0x7fec9c136e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 --2- 172.21.15.144:0/254211280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c097510 0x7fec9c136e00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fec9c09ac00 tx=0x7fec94009840 comp rx=0 tx=0).stop 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 -- 172.21.15.144:0/254211280 shutdown_connections 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 --2- 172.21.15.144:0/254211280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c097510 0x7fec9c136e00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 --2- 172.21.15.144:0/254211280 >> v2:172.21.15.116:3300/0 conn(0x7fec9c096a30 0x7fec9c09d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 -- 172.21.15.144:0/254211280 >> 172.21.15.144:0/254211280 conn(0x7fec9c071100 msgr2=0x7fec9c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 -- 172.21.15.144:0/254211280 shutdown_connections 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 1 -- 172.21.15.144:0/254211280 wait complete. 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 10 monclient: build_initial_monmap 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.812+0000 7feca1ee8640 10 monclient: monmap: 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:40.821 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 Processor -- start 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 -- start start 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient: init 2024-09-18T12:30:40.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 0x7fec9c132120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): picked mon.noname-b con 0x7fec9c096a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec9c097510 0x7fec9c1179d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): picked mon.noname-a con 0x7fec9c097510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec9c09b0b0 0x7fec9c117f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): picked mon.noname-c con 0x7fec9c09b0b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c11b400 con 0x7fec9c097510 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c1316a0 con 0x7fec9c096a30 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:40.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c132740 con 0x7fec9c09b0b0 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7feca1ee8640 10 monclient(hunting): authenticate will time out at 1076.382655s 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 0x7fec9c132120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 0x7fec9c132120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44530/0 (socket says 172.21.15.144:44530) 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 -- 172.21.15.144:0/707733107 learned_addr learned my addr 172.21.15.144:0/707733107 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): get_auth_request con 0x7fec9c096a30 auth_method 0 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec93fff640 1 -- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec9c097510 msgr2=0x7fec9c1179d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:40.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec93fff640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec9c097510 0x7fec9c1179d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient(hunting): handle_auth_done global_id 5593 payload 824 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 -- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec9c09b0b0 msgr2=0x7fec9c117f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec9c09b0b0 0x7fec9c117f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 -- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec9c097510 msgr2=0x7fec9c1179d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec9c097510 0x7fec9c1179d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:40.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.813+0000 7fec9b7fe640 1 -- 172.21.15.144:0/707733107 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94009ae0 con 0x7fec9c096a30 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec9b7fe640 1 -- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 msgr2=0x7fec9c132120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec9b7fe640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 0x7fec9c132120 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00b630 tx=0x7fec8c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c015070 con 0x7fec9c096a30 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: dump: 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:40.828 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c003d90 con 0x7fec9c096a30 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7feca1ee8640 5 monclient: authenticate success, global_id 5593 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7feca1ee8640 10 monclient: _renew_subs 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7feca1ee8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c11bce0 con 0x7fec9c096a30 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c006380 con 0x7fec9c096a30 2024-09-18T12:30:40.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec937fe640 10 monclient: dump: 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:40.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7fec9a7fc640 4 set_mon_vals no callback set 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7feca1ee8640 10 monclient: _renew_subs 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7feca1ee8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.816+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9c11b020 con 0x7fec9c096a30 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fec8c010440 con 0x7fec9c096a30 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7feca1ee8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7feca1ee8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7feca1ee8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c0008d0 con 0x7fec9c096a30 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7fec937fe640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fec7c038470 0x7fec7c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(122..122 src has 13..122) v4 ==== 6545+0+0 (secure 0 0 0) 0x7fec8c04ba60 con 0x7fec9c096a30 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7fec93fff640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fec7c038470 0x7fec7c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.817+0000 7fec93fff640 10 monclient: get_auth_request con 0x7fec7c038470 auth_method 0 2024-09-18T12:30:40.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.819+0000 7fec93fff640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fec7c038470 0x7fec7c03a920 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fec9c118fd0 tx=0x7fec84007a50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:40.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.821+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fec8c018ab0 con 0x7fec9c096a30 2024-09-18T12:30:40.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.821+0000 7fec937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:40.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.821+0000 7fec937fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:40.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.981+0000 7feca1ee8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-18T12:30:40.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.981+0000 7feca1ee8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-18T12:30:40.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.981+0000 7feca1ee8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:40.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:40.981+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7fec9c0a1a30 con 0x7fec9c096a30 2024-09-18T12:30:41.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.576+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 removed item id 14 name 'osd.14' from crush map v123) v1 ==== 125+0+0 (secure 0 0 0) 0x7fec8c018690 con 0x7fec9c096a30 2024-09-18T12:30:41.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.576+0000 7fec937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-18T12:30:41.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.576+0000 7fec937fe640 10 monclient: _finish_command 2 = system:0 removed item id 14 name 'osd.14' from crush map 2024-09-18T12:30:41.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.678+0000 7feca1ee8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-18T12:30:41.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.678+0000 7feca1ee8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-18T12:30:41.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.678+0000 7feca1ee8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:41.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.678+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) v1 -- 0x7fec9c133fe0 con 0x7fec9c096a30 2024-09-18T12:30:41.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.678+0000 7fec937fe640 1 -- 172.21.15.144:0/707733107 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 device 'osd.14' does not appear in the crush map v123) v1 ==== 126+0+0 (secure 0 0 0) 0x7fec8c028090 con 0x7fec9c096a30 2024-09-18T12:30:41.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.678+0000 7fec937fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-18T12:30:41.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.678+0000 7fec937fe640 10 monclient: _finish_command 3 = system:0 device 'osd.14' does not appear in the crush map 2024-09-18T12:30:41.680 INFO:tasks.workunit.client.0.smithi144.stderr:device 'osd.14' does not appear in the crush map 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fec7c038470 msgr2=0x7fec7c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fec7c038470 0x7fec7c03a920 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fec9c118fd0 tx=0x7fec84007a50 comp rx=0 tx=0).stop 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 10 monclient: shutdown 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 msgr2=0x7fec9c132120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 0x7fec9c132120 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00b630 tx=0x7fec8c0091a0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 shutdown_connections 2024-09-18T12:30:41.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fec7c038470 0x7fec7c03a920 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fec9c09b0b0 0x7fec9c117f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec9c097510 0x7fec9c1179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 --2- 172.21.15.144:0/707733107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fec9c096a30 0x7fec9c132120 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 >> 172.21.15.144:0/707733107 conn(0x7fec9c071100 msgr2=0x7fec9c06d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:41.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 shutdown_connections 2024-09-18T12:30:41.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.680+0000 7feca1ee8640 1 -- 172.21.15.144:0/707733107 wait complete. 2024-09-18T12:30:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd rm osd.12 2024-09-18T12:30:41.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.762+0000 7f272df9d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:41.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.762+0000 7f272df9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:41.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.762+0000 7f272df9d640 10 monclient: monmap: 2024-09-18T12:30:41.763 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:41.764 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:41.764 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:41.763078+0000 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:41.763078+0000 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:41.765 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:41.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.762+0000 7f272df9d640 1 Processor -- start 2024-09-18T12:30:41.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.762+0000 7f272df9d640 1 -- start start 2024-09-18T12:30:41.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.762+0000 7f272df9d640 10 monclient: init 2024-09-18T12:30:41.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27280677e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2728094080 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2728094b90 0x7f272813a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2728094b90 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272809d950 con 0x7f2728094080 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2728094fd0 con 0x7f27280677e0 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272809d670 con 0x7f2728094b90 2024-09-18T12:30:41.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f272df9d640 10 monclient(hunting): authenticate will time out at 1077.332423s 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f27277fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f27280677e0 0x7f2728098d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58250/0 (socket says 172.21.15.144:58250) 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f27277fe640 1 -- 172.21.15.144:0/3718967431 learned_addr learned my addr 172.21.15.144:0/3718967431 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f2727fff640 1 -- 172.21.15.144:0/3718967431 >> v2:172.21.15.116:3300/0 conn(0x7f2728094b90 msgr2=0x7f272813a2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f2726ffd640 1 -- 172.21.15.144:0/3718967431 >> v1:172.21.15.116:6789/0 conn(0x7f2728094080 legacy=0x7f2728094480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.763+0000 7f2727fff640 1 --2- 172.21.15.144:0/3718967431 >> v2:172.21.15.116:3300/0 conn(0x7f2728094b90 0x7f272813a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.765+0000 7f27277fe640 1 -- 172.21.15.144:0/3718967431 >> v1:172.21.15.144:6789/0 conn(0x7f27280677e0 legacy=0x7f2728098d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.765+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439116188 0 0) 0x7f2728094fd0 con 0x7f27280677e0 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.765+0000 7f27267fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.765+0000 7f27267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:41.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.765+0000 7f27267fc640 10 monclient(hunting): my global_id is 5596 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.765+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27140034e0 con 0x7f27280677e0 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 534128051 0 0) 0x7f27140034e0 con 0x7f27280677e0 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 >> v2:172.21.15.116:3300/0 conn(0x7f2728094b90 msgr2=0x7f272813a2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 --2- 172.21.15.144:0/3718967431 >> v2:172.21.15.116:3300/0 conn(0x7f2728094b90 0x7f272813a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 >> v1:172.21.15.116:6789/0 conn(0x7f2728094080 legacy=0x7f2728094480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272809dc30 con 0x7f27280677e0 2024-09-18T12:30:41.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f27180017a0 con 0x7f27280677e0 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f272df9d640 5 monclient: authenticate success, global_id 5596 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: dump: 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.781 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 >> v1:172.21.15.144:6789/0 conn(0x7f27280677e0 legacy=0x7f2728098d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2714003aa0 0x7f2714003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): picked mon.a con 0x7f2714003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2714004780 0x7f2714004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): picked mon.b con 0x7f2714004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27140099c0 0x7f271400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): picked mon.c con 0x7f27140099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272809d670 con 0x7f2714003aa0 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271400c3f0 con 0x7f2714004780 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271400c5b0 con 0x7f27140099c0 2024-09-18T12:30:41.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27267fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f272df9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2714004780 0x7f2714004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): get_auth_request con 0x7f2714004780 auth_method 0 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27277fe640 1 -- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2714003aa0 msgr2=0x7f2714003eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f27277fe640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2714003aa0 0x7f2714003eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:41.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2727fff640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27140099c0 0x7f271400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient(hunting): handle_auth_done global_id 5596 payload 293 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 1 -- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27140099c0 msgr2=0x7f271400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27140099c0 0x7f271400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 1 -- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2714003aa0 msgr2=0x7f2714003eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2714003aa0 0x7f2714003eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 1 monclient: found mon.b 2024-09-18T12:30:41.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 1 -- 172.21.15.144:0/3718967431 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27140034e0 con 0x7f2714004780 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2727fff640 10 monclient: get_auth_request con 0x7f27140099c0 auth_method 0 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2727fff640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27140099c0 0x7f271400bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.766+0000 7f2727fff640 1 -- 172.21.15.144:0/3718967431 reap_dead start 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f2726ffd640 1 -- 172.21.15.144:0/3718967431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2714004780 msgr2=0x7f2714004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f2726ffd640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2714004780 0x7f2714004bf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2710003bf0 tx=0x7f27100095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e378d39897901 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100050d0 con 0x7f2714004780 2024-09-18T12:30:41.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 10 monclient: dump: 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:41.787 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271002d070 con 0x7f2714004780 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 1 -- 172.21.15.144:0/3718967431 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100054c0 con 0x7f2714004780 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 4 set_mon_vals no callback set 2024-09-18T12:30:41.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f27267fc640 10 monclient: dump: 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:41.789 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 10 monclient: shutdown 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 1 -- 172.21.15.144:0/3718967431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2714004780 msgr2=0x7f2714004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2714004780 0x7f2714004bf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2710003bf0 tx=0x7f27100095d0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 1 -- 172.21.15.144:0/3718967431 shutdown_connections 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 1 --2- 172.21.15.144:0/3718967431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2714004780 0x7f2714004bf0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.769+0000 7f272df9d640 1 -- 172.21.15.144:0/3718967431 >> 172.21.15.144:0/3718967431 conn(0x7f27280a15a0 msgr2=0x7f27280a1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.770+0000 7f272df9d640 1 -- 172.21.15.144:0/3718967431 shutdown_connections 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.770+0000 7f272df9d640 1 -- 172.21.15.144:0/3718967431 wait complete. 2024-09-18T12:30:41.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.770+0000 7f272df9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.770+0000 7f272df9d640 10 monclient: monmap: 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:41.791 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.770+0000 7f272df9d640 1 Processor -- start 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 1 -- start start 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient: init 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2714004780 0x7f2728140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2714004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f272812f080 0x7f2728141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): picked mon.noname-a con 0x7f272812f080 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 0x7f272812acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): picked mon.noname-b con 0x7f272812a880 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280a0d10 con 0x7f272812f080 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272812e0e0 con 0x7f272812a880 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272812de00 con 0x7f2714004780 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f272df9d640 10 monclient(hunting): authenticate will time out at 1077.340455s 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2726ffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f272812f080 msgr2=0x7f2728141530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:41.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2726ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f272812f080 0x7f2728141530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 0x7f272812acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 0x7f272812acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44546/0 (socket says 172.21.15.144:44546) 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 -- 172.21.15.144:0/4007904872 learned_addr learned my addr 172.21.15.144:0/4007904872 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): get_auth_request con 0x7f272812a880 auth_method 0 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f27277fe640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2714004780 0x7f2728140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:41.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f27277fe640 10 monclient(hunting): get_auth_request con 0x7f2714004780 auth_method 0 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f27277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f27277fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f27277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient(hunting): handle_auth_done global_id 5599 payload 824 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 -- 172.21.15.144:0/4007904872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2714004780 msgr2=0x7f2728140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2714004780 0x7f2728140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 -- 172.21.15.144:0/4007904872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f272812f080 msgr2=0x7f2728141530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f272812f080 0x7f2728141530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 monclient: found mon.noname-b 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:41.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.771+0000 7f2727fff640 1 -- 172.21.15.144:0/4007904872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272812e3c0 con 0x7f272812a880 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f2727fff640 1 -- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 msgr2=0x7f272812acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f2727fff640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 0x7f272812acb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f27200038f0 tx=0x7f2720008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720003d90 con 0x7f272812a880 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: dump: 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.796 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f272df9d640 5 monclient: authenticate success, global_id 5599 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272000c970 con 0x7f272812a880 2024-09-18T12:30:41.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272000cd30 con 0x7f272812a880 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f272df9d640 10 monclient: _renew_subs 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f272df9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272812cea0 con 0x7f272812a880 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f271f7fe640 10 monclient: dump: 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.798 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f27257fa640 4 set_mon_vals no callback set 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f272df9d640 10 monclient: _renew_subs 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f272df9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.774+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2728141a70 con 0x7f272812a880 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2720005300 con 0x7f272812a880 2024-09-18T12:30:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f272df9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f272df9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f272df9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272804bf60 con 0x7f272812a880 2024-09-18T12:30:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f271f7fe640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2708038440 0x7f270803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(123..123 src has 13..123) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f272004aca0 con 0x7f272812a880 2024-09-18T12:30:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f27277fe640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2708038440 0x7f270803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.775+0000 7f27277fe640 10 monclient: get_auth_request con 0x7f2708038440 auth_method 0 2024-09-18T12:30:41.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.777+0000 7f27277fe640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2708038440 0x7f270803a8f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f27180013f0 tx=0x7f27180018f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:41.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.779+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f272001ada0 con 0x7f272812a880 2024-09-18T12:30:41.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.779+0000 7f271f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:41.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.779+0000 7f271f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:41.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.944+0000 7f272df9d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T12:30:41.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.944+0000 7f272df9d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T12:30:41.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.944+0000 7f272df9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:41.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:41.944+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f2728064a70 con 0x7f272812a880 2024-09-18T12:30:41.946 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-09-18T12:30:41.944+0000 7fd9503f1640 -1 mon.b@1(leader).osd e123 definitely_dead 0 2024-09-18T12:30:42.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.578+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v124) v1 ==== 87+0+0 (secure 0 0 0) 0x7f272001a7d0 con 0x7f272812a880 2024-09-18T12:30:42.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.578+0000 7f271f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T12:30:42.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.578+0000 7f271f7fe640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-18T12:30:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.683+0000 7f272df9d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T12:30:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.683+0000 7f272df9d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T12:30:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.683+0000 7f272df9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.683+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7f272812b1f0 con 0x7f272812a880 2024-09-18T12:30:42.686 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-09-18T12:30:42.684+0000 7fd9503f1640 -1 mon.b@1(leader).osd e124 definitely_dead 0 2024-09-18T12:30:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.684+0000 7f271f7fe640 1 -- 172.21.15.144:0/4007904872 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v124) v1 ==== 96+0+0 (secure 0 0 0) 0x7f272001a9b0 con 0x7f272812a880 2024-09-18T12:30:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.684+0000 7f271f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T12:30:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.684+0000 7f271f7fe640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-18T12:30:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:osd.12 does not exist. 2024-09-18T12:30:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2708038440 msgr2=0x7f270803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2708038440 0x7f270803a8f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f27180013f0 tx=0x7f27180018f0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 10 monclient: shutdown 2024-09-18T12:30:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 msgr2=0x7f272812acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 0x7f272812acb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f27200038f0 tx=0x7f2720008040 comp rx=0 tx=0).stop 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 shutdown_connections 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2708038440 0x7f270803a8f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f272812a880 0x7f272812acb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f272812f080 0x7f2728141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 --2- 172.21.15.144:0/4007904872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2714004780 0x7f2728140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 >> 172.21.15.144:0/4007904872 conn(0x7f27280a15a0 msgr2=0x7f2728128680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 shutdown_connections 2024-09-18T12:30:42.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.686+0000 7f272df9d640 1 -- 172.21.15.144:0/4007904872 wait complete. 2024-09-18T12:30:42.696 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd rm osd.14 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.768+0000 7fc8c999d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.768+0000 7fc8c999d640 10 monclient: build_initial_monmap 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.768+0000 7fc8c999d640 10 monclient: monmap: 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:42.769445+0000 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:42.769445+0000 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:42.770 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 1 Processor -- start 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 1 -- start start 2024-09-18T12:30:42.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient: init 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8c4131ad0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc8c409ec50 0x7fc8c409f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8c409ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8c4136e00 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c409c010 con 0x7fc8c4136e00 2024-09-18T12:30:42.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c409ffe0 con 0x7fc8c4131ad0 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c409f5b0 con 0x7fc8c409ec50 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c999d640 10 monclient(hunting): authenticate will time out at 1078.338685s 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc8c409ec50 0x7fc8c409f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc8c409ec50 0x7fc8c409f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44548/0 (socket says 172.21.15.144:44548) 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 1 -- 172.21.15.144:0/2500201323 learned_addr learned my addr 172.21.15.144:0/2500201323 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 10 monclient(hunting): get_auth_request con 0x7fc8c409ec50 auth_method 0 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c37fe640 1 -- 172.21.15.144:0/2500201323 >> v1:172.21.15.116:6789/0 conn(0x7fc8c4136e00 legacy=0x7fc8c41391e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:42.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.769+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c2ffd640 1 -- 172.21.15.144:0/2500201323 >> v1:172.21.15.144:6789/0 conn(0x7fc8c4131ad0 legacy=0x7fc8c4132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_done global_id 5602 payload 274 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 1 -- 172.21.15.144:0/2500201323 >> v1:172.21.15.144:6789/0 conn(0x7fc8c4131ad0 legacy=0x7fc8c4132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 1 -- 172.21.15.144:0/2500201323 >> v1:172.21.15.116:6789/0 conn(0x7fc8c4136e00 legacy=0x7fc8c41391e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 1 monclient: found mon.noname-c 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:42.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.772+0000 7fc8c27fc640 1 -- 172.21.15.144:0/2500201323 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c4132890 con 0x7fc8c409ec50 2024-09-18T12:30:42.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c27fc640 1 -- 172.21.15.144:0/2500201323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c409ec50 msgr2=0x7fc8c409f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:42.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c27fc640 1 --2- 172.21.15.144:0/2500201323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c409ec50 0x7fc8c409f070 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009990 tx=0x7fc8b80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=115390d345446b10 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:42.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 1 -- 172.21.15.144:0/2500201323 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b802ae90 con 0x7fc8c409ec50 2024-09-18T12:30:42.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:42.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: dump: 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:42.781 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 5 monclient: authenticate success, global_id 5602 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 1 -- 172.21.15.144:0/2500201323 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b802f4b0 con 0x7fc8c409ec50 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:42.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 1 -- 172.21.15.144:0/2500201323 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b802f850 con 0x7fc8c409ec50 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 4 set_mon_vals no callback set 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c1ffb640 10 monclient: dump: 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:42.783 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 10 monclient: shutdown 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 1 -- 172.21.15.144:0/2500201323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c409ec50 msgr2=0x7fc8c409f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 1 --2- 172.21.15.144:0/2500201323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c409ec50 0x7fc8c409f070 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009990 tx=0x7fc8b80091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 1 -- 172.21.15.144:0/2500201323 shutdown_connections 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 1 --2- 172.21.15.144:0/2500201323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c409ec50 0x7fc8c409f070 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 1 -- 172.21.15.144:0/2500201323 >> 172.21.15.144:0/2500201323 conn(0x7fc8c40a5090 msgr2=0x7fc8c40a5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 1 -- 172.21.15.144:0/2500201323 shutdown_connections 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.775+0000 7fc8c999d640 1 -- 172.21.15.144:0/2500201323 wait complete. 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.776+0000 7fc8c999d640 10 monclient: build_initial_monmap 2024-09-18T12:30:42.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.776+0000 7fc8c999d640 10 monclient: monmap: 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.776+0000 7fc8c999d640 1 Processor -- start 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.776+0000 7fc8c999d640 1 -- start start 2024-09-18T12:30:42.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.776+0000 7fc8c999d640 10 monclient: init 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8c409ec50 0x7fc8c412eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): picked mon.noname-a con 0x7fc8c409ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 0x7fc8c4097a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8c4131ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8c4136e00 0x7fc8c4097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8c4136e00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4099a80 con 0x7fc8c409ec50 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40a0710 con 0x7fc8c4131ad0 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:42.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40a3e70 con 0x7fc8c4136e00 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c999d640 10 monclient(hunting): authenticate will time out at 1078.346368s 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 0x7fc8c4097a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 0x7fc8c4097a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44556/0 (socket says 172.21.15.144:44556) 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 -- 172.21.15.144:0/1373077059 learned_addr learned my addr 172.21.15.144:0/1373077059 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): get_auth_request con 0x7fc8c4131ad0 auth_method 0 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c2ffd640 1 -- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8c409ec50 msgr2=0x7fc8c412eb00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c2ffd640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8c409ec50 0x7fc8c412eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:42.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c37fe640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8c4136e00 0x7fc8c4097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c37fe640 10 monclient(hunting): get_auth_request con 0x7fc8c4136e00 auth_method 0 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient(hunting): handle_auth_done global_id 5605 payload 824 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 -- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8c4136e00 msgr2=0x7fc8c4097f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8c4136e00 0x7fc8c4097f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 -- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8c409ec50 msgr2=0x7fc8c412eb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8c409ec50 0x7fc8c412eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:42.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:42.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 monclient: found mon.noname-b 2024-09-18T12:30:42.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:42.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:42.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.777+0000 7fc8c27fc640 1 -- 172.21.15.144:0/1373077059 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b8009450 con 0x7fc8c4131ad0 2024-09-18T12:30:42.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.779+0000 7fc8c27fc640 1 -- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 msgr2=0x7fc8c4097a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:42.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.779+0000 7fc8c27fc640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 0x7fc8c4097a50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009990 tx=0x7fc8b8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:42.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80328e0 con 0x7fc8c4131ad0 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: dump: 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:42.792 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c999d640 5 monclient: authenticate success, global_id 5605 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c999d640 10 monclient: _renew_subs 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b802ae90 con 0x7fc8c4131ad0 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c4142040 con 0x7fc8c4131ad0 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8032cd0 con 0x7fc8c4131ad0 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8aaffd640 10 monclient: dump: 2024-09-18T12:30:42.793 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c999d640 10 monclient: _renew_subs 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:42.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c4099580 con 0x7fc8c4131ad0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.780+0000 7fc8c17fa640 4 set_mon_vals no callback set 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc8b8039070 con 0x7fc8c4131ad0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8c999d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8c999d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8c999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c409a190 con 0x7fc8c4131ad0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8aaffd640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc8b0038440 0x7fc8b003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(124..124 src has 13..124) v4 ==== 6523+0+0 (secure 0 0 0) 0x7fc8b806c3d0 con 0x7fc8c4131ad0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8c2ffd640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc8b0038440 0x7fc8b003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.781+0000 7fc8c2ffd640 10 monclient: get_auth_request con 0x7fc8b0038440 auth_method 0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.783+0000 7fc8c2ffd640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc8b0038440 0x7fc8b003a8f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40014b0 tx=0x7fc8b4007a10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.785+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc8b8030570 con 0x7fc8c4131ad0 2024-09-18T12:30:42.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.785+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:42.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.785+0000 7fc8aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:42.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.950+0000 7fc8c999d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-18T12:30:42.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.950+0000 7fc8c999d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-18T12:30:42.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.950+0000 7fc8c999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:42.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:42.950+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7fc8c404bf60 con 0x7fc8c4131ad0 2024-09-18T12:30:42.952 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-09-18T12:30:42.950+0000 7fd9503f1640 -1 mon.b@1(leader).osd e124 definitely_dead 0 2024-09-18T12:30:43.424 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:30:43.424 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:30:43.424 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:30:43.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:30:43.427 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:30:43.427 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:30:43.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.531+0000 7feddaa6f640 10 monclient: get_monmap_and_config 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.531+0000 7feddaa6f640 10 monclient: build_initial_monmap 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.531+0000 7feddaa6f640 10 monclient: monmap: 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:43.531875+0000 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:43.531875+0000 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:43.532 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:43.533 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.531+0000 7feddaa6f640 1 Processor -- start 2024-09-18T12:30:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 1 -- start start 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient: init 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fedd4131ab0 0x7fedd4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient(hunting): picked mon.noname-c con 0x7fedd4131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedd409e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd4138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient(hunting): picked mon.noname-b con 0x7fedd4136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd409faf0 con 0x7fedd409e910 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd409fd30 con 0x7fedd4131ab0 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40a1110 con 0x7fedd4136bb0 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7feddaa6f640 10 monclient(hunting): authenticate will time out at 1070.853542s 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd4138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd4138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54110/0 (socket says 172.21.15.116:54110) 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 -- 172.21.15.116:0/2303102600 learned_addr learned my addr 172.21.15.116:0/2303102600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): get_auth_request con 0x7fedd4136bb0 auth_method 0 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 1 --2- 172.21.15.116:0/2303102600 >> v2:172.21.15.144:3300/0 conn(0x7fedd4131ab0 0x7fedd4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 10 monclient(hunting): get_auth_request con 0x7fedd4131ab0 auth_method 0 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd880b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient(hunting): handle_auth_done global_id 64427 payload 274 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 -- 172.21.15.116:0/2303102600 >> v2:172.21.15.144:3300/0 conn(0x7fedd4131ab0 msgr2=0x7fedd4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 --2- 172.21.15.116:0/2303102600 >> v2:172.21.15.144:3300/0 conn(0x7fedd4131ab0 0x7fedd4131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 -- 172.21.15.116:0/2303102600 >> v1:172.21.15.144:6789/0 conn(0x7fedd409e910 legacy=0x7fedd409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 monclient: found mon.noname-b 2024-09-18T12:30:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:43.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:43.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.532+0000 7fedd900c640 1 -- 172.21.15.116:0/2303102600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40a13f0 con 0x7fedd4136bb0 2024-09-18T12:30:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd900c640 1 -- 172.21.15.116:0/2303102600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 msgr2=0x7fedd4138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd900c640 1 --2- 172.21.15.116:0/2303102600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd4138f90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009910 tx=0x7fedbc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b3e37a356a3568 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 1 -- 172.21.15.116:0/2303102600 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc032440 con 0x7fedd4136bb0 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: dump: 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.545 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 1 -- 172.21.15.116:0/2303102600 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc0325a0 con 0x7fedd4136bb0 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7feddaa6f640 5 monclient: authenticate success, global_id 64427 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7feddaa6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7feddaa6f640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 1 -- 172.21.15.116:0/2303102600 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc032940 con 0x7fedd4136bb0 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7feddaa6f640 4 set_mon_vals no callback set 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.535+0000 7fedd37fe640 10 monclient: dump: 2024-09-18T12:30:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 10 monclient: shutdown 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 -- 172.21.15.116:0/2303102600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 msgr2=0x7fedd4138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 --2- 172.21.15.116:0/2303102600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd4138f90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009910 tx=0x7fedbc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 -- 172.21.15.116:0/2303102600 shutdown_connections 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 --2- 172.21.15.116:0/2303102600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd4138f90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 --2- 172.21.15.116:0/2303102600 >> v2:172.21.15.144:3300/0 conn(0x7fedd4131ab0 0x7fedd4131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 -- 172.21.15.116:0/2303102600 >> 172.21.15.116:0/2303102600 conn(0x7fedd4067d30 msgr2=0x7fedd40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 -- 172.21.15.116:0/2303102600 shutdown_connections 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 1 -- 172.21.15.116:0/2303102600 wait complete. 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 10 monclient: build_initial_monmap 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.536+0000 7feddaa6f640 10 monclient: monmap: 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 Processor -- start 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 -- start start 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient: init 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 0x7fedd4136140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient(hunting): picked mon.noname-a con 0x7fedd409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd4131ab0 0x7fedd412ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient(hunting): picked mon.noname-b con 0x7fedd4131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd412f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient(hunting): picked mon.noname-c con 0x7fedd4136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4130e20 con 0x7fedd409e910 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4133a20 con 0x7fedd4131ab0 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.537+0000 7feddaa6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4130900 con 0x7fedd4136bb0 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7feddaa6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7feddaa6f640 10 monclient(hunting): authenticate will time out at 1070.859012s 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 0x7fedd4136140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd412f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 0x7fedd4136140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40680/0 (socket says 172.21.15.116:40680) 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 1 -- 172.21.15.116:0/1165534347 learned_addr learned my addr 172.21.15.116:0/1165534347 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 10 monclient(hunting): get_auth_request con 0x7fedd409e910 auth_method 0 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): get_auth_request con 0x7fedd4136bb0 auth_method 0 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 1 -- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 msgr2=0x7fedd4136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 1 -- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 msgr2=0x7fedd4136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 0x7fedd4136140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd880b640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 0x7fedd4136140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd4131ab0 0x7fedd412ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 10 monclient(hunting): get_auth_request con 0x7fedd4131ab0 auth_method 0 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient(hunting): handle_auth_done global_id 64430 payload 824 2024-09-18T12:30:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 1 -- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd4131ab0 msgr2=0x7fedd412ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd4131ab0 0x7fedd412ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 1 -- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 msgr2=0x7fedd4136140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 0x7fedd4136140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 1 monclient: found mon.noname-c 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.538+0000 7fedd900c640 1 -- 172.21.15.116:0/1165534347 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc009450 con 0x7fedd4136bb0 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd900c640 1 -- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 msgr2=0x7fedd412f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd900c640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd412f470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fedd40a12a0 tx=0x7fedbc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 1 -- 172.21.15.116:0/1165534347 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc031c70 con 0x7fedd4136bb0 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: dump: 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7feddaa6f640 5 monclient: authenticate success, global_id 64430 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 1 -- 172.21.15.116:0/1165534347 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc031dd0 con 0x7fedd4136bb0 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 1 -- 172.21.15.116:0/1165534347 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc03d670 con 0x7fedd4136bb0 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd0ff9640 10 monclient: dump: 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7feddaa6f640 10 monclient: _renew_subs 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7fedd2ffd640 4 set_mon_vals no callback set 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7feddaa6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd409bde0 con 0x7fedd4136bb0 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7feddaa6f640 10 monclient: _renew_subs 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7feddaa6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.540+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd4133740 con 0x7fedd4136bb0 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.541+0000 7fedd0ff9640 1 -- 172.21.15.116:0/1165534347 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fedbc039040 con 0x7fedd4136bb0 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.541+0000 7feddaa6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.541+0000 7feddaa6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.541+0000 7feddaa6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.541+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd404bf60 con 0x7fedd4136bb0 2024-09-18T12:30:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.541+0000 7fedd0ff9640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedb4038490 0x7fedb403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.542+0000 7fedd0ff9640 1 -- 172.21.15.116:0/1165534347 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(124..124 src has 13..124) v4 ==== 6523+0+0 (secure 0 0 0) 0x7fedbc06bc90 con 0x7fedd4136bb0 2024-09-18T12:30:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.542+0000 7fedd880b640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedb4038490 0x7fedb403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.542+0000 7fedd880b640 10 monclient: get_auth_request con 0x7fedb4038490 auth_method 0 2024-09-18T12:30:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.544+0000 7fedd880b640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedb4038490 0x7fedb403a940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009550 tx=0x7fedcc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.546+0000 7fedd0ff9640 1 -- 172.21.15.116:0/1165534347 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fedbc0432c0 con 0x7fedd4136bb0 2024-09-18T12:30:43.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.546+0000 7fedd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:43.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.546+0000 7fedd0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:43.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.629+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 removed osd.14 v125) v1 ==== 87+0+0 (secure 0 0 0) 0x7fc8b8044ed0 con 0x7fc8c4131ad0 2024-09-18T12:30:43.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.629+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-18T12:30:43.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.629+0000 7fc8aaffd640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-18T12:30:43.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.732+0000 7fc8c999d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-18T12:30:43.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.732+0000 7fc8c999d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-18T12:30:43.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.732+0000 7fc8c999d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:43.734 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.732+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) v1 -- 0x7fc8c404bf60 con 0x7fc8c4131ad0 2024-09-18T12:30:43.735 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-09-18T12:30:43.733+0000 7fd9503f1640 -1 mon.b@1(leader).osd e125 definitely_dead 0 2024-09-18T12:30:43.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.733+0000 7fc8aaffd640 1 -- 172.21.15.144:0/1373077059 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 osd.14 does not exist. v125) v1 ==== 96+0+0 (secure 0 0 0) 0x7fc8b8037070 con 0x7fc8c4131ad0 2024-09-18T12:30:43.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.733+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-18T12:30:43.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.733+0000 7fc8aaffd640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-18T12:30:43.735 INFO:tasks.workunit.client.0.smithi144.stderr:osd.14 does not exist. 2024-09-18T12:30:43.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc8b0038440 msgr2=0x7fc8b003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc8b0038440 0x7fc8b003a8f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40014b0 tx=0x7fc8b4007a10 comp rx=0 tx=0).stop 2024-09-18T12:30:43.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 10 monclient: shutdown 2024-09-18T12:30:43.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 msgr2=0x7fc8c4097a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 0x7fc8c4097a50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009990 tx=0x7fc8b8032000 comp rx=0 tx=0).stop 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 shutdown_connections 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc8b0038440 0x7fc8b003a8f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc8c4136e00 0x7fc8c4097f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8c4131ad0 0x7fc8c4097a50 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 --2- 172.21.15.144:0/1373077059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8c409ec50 0x7fc8c412eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 >> 172.21.15.144:0/1373077059 conn(0x7fc8c40a5090 msgr2=0x7fc8c4130690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 shutdown_connections 2024-09-18T12:30:43.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.735+0000 7fc8c999d640 1 -- 172.21.15.144:0/1373077059 wait complete. 2024-09-18T12:30:43.745 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-18T12:30:43.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.763+0000 7feddaa6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.763+0000 7feddaa6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.763+0000 7feddaa6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.763+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fedd404bf60 con 0x7fedd4136bb0 2024-09-18T12:30:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.764+0000 7fedd0ff9640 1 -- 172.21.15.116:0/1165534347 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fedbc037070 con 0x7fedd4136bb0 2024-09-18T12:30:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.764+0000 7fedd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:43.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.764+0000 7fedd0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:43.765 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:43.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedb4038490 msgr2=0x7fedb403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedb4038490 0x7fedb403a940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009550 tx=0x7fedcc008040 comp rx=0 tx=0).stop 2024-09-18T12:30:43.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 10 monclient: shutdown 2024-09-18T12:30:43.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:43.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 msgr2=0x7fedd412f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd412f470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fedd40a12a0 tx=0x7fedbc032000 comp rx=0 tx=0).stop 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 shutdown_connections 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fedb4038490 0x7fedb403a940 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fedd4136bb0 0x7fedd412f470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fedd4131ab0 0x7fedd412ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 --2- 172.21.15.116:0/1165534347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fedd409e910 0x7fedd4136140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 >> 172.21.15.116:0/1165534347 conn(0x7fedd4067d30 msgr2=0x7fedd412cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 shutdown_connections 2024-09-18T12:30:43.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:43.766+0000 7feddaa6f640 1 -- 172.21.15.116:0/1165534347 wait complete. 2024-09-18T12:30:43.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.816+0000 7f1d332c0640 10 monclient: get_monmap_and_config 2024-09-18T12:30:43.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.816+0000 7f1d332c0640 10 monclient: build_initial_monmap 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.816+0000 7f1d332c0640 10 monclient: monmap: 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:43.817568+0000 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:43.817568+0000 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:43.818 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 1 Processor -- start 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 1 -- start start 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient: init 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d2c0a3c00 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d2c0994e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 0x7f1d2c13a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1d2c06a600 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c069f00 con 0x7f1d2c0a3c00 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c069c20 con 0x7f1d2c0994e0 2024-09-18T12:30:43.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d2c06aa40 con 0x7f1d2c06a600 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.817+0000 7f1d332c0640 10 monclient(hunting): authenticate will time out at 1079.386906s 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3085b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1d2c0994e0 0x7f1d2c136c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58264/0 (socket says 172.21.15.144:58264) 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3085b640 1 -- 172.21.15.144:0/995692748 learned_addr learned my addr 172.21.15.144:0/995692748 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 1 --2- 172.21.15.144:0/995692748 >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 0x7f1d2c13a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 10 monclient(hunting): get_auth_request con 0x7f1d2c06a600 auth_method 0 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 1 -- 172.21.15.144:0/995692748 >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 msgr2=0x7f1d2c13a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 1 -- 172.21.15.144:0/995692748 >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 msgr2=0x7f1d2c13a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:43.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 1 --2- 172.21.15.144:0/995692748 >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 0x7f1d2c13a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.818+0000 7f1d3185d640 1 --2- 172.21.15.144:0/995692748 >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 0x7f1d2c13a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 1 -- 172.21.15.144:0/995692748 >> v1:172.21.15.144:6789/0 conn(0x7f1d2c0994e0 legacy=0x7f1d2c136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706905714 0 0) 0x7f1d2c069c20 con 0x7f1d2c0994e0 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): my global_id is 5614 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 1 -- 172.21.15.144:0/995692748 >> v1:172.21.15.116:6789/0 conn(0x7f1d2c0a3c00 legacy=0x7f1d2c098dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d180034e0 con 0x7f1d2c0994e0 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3792259021 0 0) 0x7f1d180034e0 con 0x7f1d2c0994e0 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 msgr2=0x7f1d2c13a380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 --2- 172.21.15.144:0/995692748 >> v2:172.21.15.116:3300/0 conn(0x7f1d2c06a600 0x7f1d2c13a380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 >> v1:172.21.15.116:6789/0 conn(0x7f1d2c0a3c00 legacy=0x7f1d2c098dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:43.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c069940 con 0x7f1d2c0994e0 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f1d1c0017a0 con 0x7f1d2c0994e0 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d332c0640 5 monclient: authenticate success, global_id 5614 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d332c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: dump: 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:43.827 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 >> v1:172.21.15.144:6789/0 conn(0x7f1d2c0994e0 legacy=0x7f1d2c136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18003bc0 0x7f1d18003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): picked mon.c con 0x7f1d18003bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d18004780 0x7f1d18004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): picked mon.b con 0x7f1d18004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d180099c0 0x7f1d1800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): picked mon.a con 0x7f1d180099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c06aa40 con 0x7f1d180099c0 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1800c420 con 0x7f1d18004780 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1800c5e0 con 0x7f1d18003bc0 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d23fff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d332c0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:43.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d18004780 0x7f1d18004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): get_auth_request con 0x7f1d18004780 auth_method 0 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18003bc0 0x7f1d18003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d180099c0 0x7f1d1800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 10 monclient(hunting): get_auth_request con 0x7f1d18003bc0 auth_method 0 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 10 monclient(hunting): get_auth_request con 0x7f1d180099c0 auth_method 0 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3185d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3105c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 10 monclient(hunting): handle_auth_done global_id 5614 payload 293 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 1 -- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18003bc0 msgr2=0x7f1d18003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18003bc0 0x7f1d18003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 1 -- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d180099c0 msgr2=0x7f1d1800bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.820+0000 7f1d3085b640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d180099c0 0x7f1d1800bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3085b640 10 monclient: _finish_hunting 0 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3085b640 1 monclient: found mon.b 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3085b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3185d640 1 -- 172.21.15.144:0/995692748 reap_dead start 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3085b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3085b640 1 -- 172.21.15.144:0/995692748 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d180034e0 con 0x7f1d18004780 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3085b640 10 monclient: _finish_auth 0 2024-09-18T12:30:43.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.821+0000 7f1d3085b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d3085b640 1 -- 172.21.15.144:0/995692748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d18004780 msgr2=0x7f1d18004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d3085b640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d18004780 0x7f1d18004bf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0014d0 tx=0x7f1d1c007a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a17041069bbbd22a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c005890 con 0x7f1d18004780 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 10 monclient: dump: 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.838 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1c005a10 con 0x7f1d18004780 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d332c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 1 -- 172.21.15.144:0/995692748 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c005dd0 con 0x7f1d18004780 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d332c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d332c0640 4 set_mon_vals no callback set 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.823+0000 7f1d23fff640 10 monclient: dump: 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 10 monclient: shutdown 2024-09-18T12:30:43.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 1 -- 172.21.15.144:0/995692748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d18004780 msgr2=0x7f1d18004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d18004780 0x7f1d18004bf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0014d0 tx=0x7f1d1c007a90 comp rx=0 tx=0).stop 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 1 -- 172.21.15.144:0/995692748 shutdown_connections 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 1 --2- 172.21.15.144:0/995692748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d18004780 0x7f1d18004bf0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 1 -- 172.21.15.144:0/995692748 >> 172.21.15.144:0/995692748 conn(0x7f1d2c071350 msgr2=0x7f1d2c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 1 -- 172.21.15.144:0/995692748 shutdown_connections 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.824+0000 7f1d332c0640 1 -- 172.21.15.144:0/995692748 wait complete. 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 10 monclient: build_initial_monmap 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 10 monclient: monmap: 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:43.841 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 1 Processor -- start 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 1 -- start start 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 10 monclient: init 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18004780 0x7f1d2c13d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.825+0000 7f1d332c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1d18004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:43.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 0x7f1d2c13cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1d2c121180 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 0x7f1d2c140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1d2c13e280 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c1228f0 con 0x7f1d2c13e280 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c122330 con 0x7f1d2c121180 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c11fdc0 con 0x7f1d18004780 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d332c0640 10 monclient(hunting): authenticate will time out at 1079.395057s 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 0x7f1d2c13cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 0x7f1d2c13cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44570/0 (socket says 172.21.15.144:44570) 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 -- 172.21.15.144:0/2466572622 learned_addr learned my addr 172.21.15.144:0/2466572622 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): get_auth_request con 0x7f1d2c121180 auth_method 0 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 0x7f1d2c140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18004780 0x7f1d2c13d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 10 monclient(hunting): get_auth_request con 0x7f1d18004780 auth_method 0 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 10 monclient(hunting): get_auth_request con 0x7f1d2c13e280 auth_method 0 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 1 -- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 msgr2=0x7f1d2c140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3105c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 1 -- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 msgr2=0x7f1d2c140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 0x7f1d2c140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3185d640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 0x7f1d2c140e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:43.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient(hunting): handle_auth_done global_id 5617 payload 824 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 -- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18004780 msgr2=0x7f1d2c13d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18004780 0x7f1d2c13d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 -- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 msgr2=0x7f1d2c140e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 0x7f1d2c140e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient: _finish_hunting 0 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 monclient: found mon.noname-b 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.826+0000 7f1d3085b640 1 -- 172.21.15.144:0/2466572622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c122050 con 0x7f1d2c121180 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d3085b640 1 -- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 msgr2=0x7f1d2c13cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d3085b640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 0x7f1d2c13cfb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c007af0 tx=0x7f1d1c034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c007510 con 0x7f1d2c121180 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: dump: 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: _finish_auth 0 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:43.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1c007690 con 0x7f1d2c121180 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d332c0640 5 monclient: authenticate success, global_id 5617 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d332c0640 10 monclient: _renew_subs 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d332c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c122610 con 0x7f1d2c121180 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c044680 con 0x7f1d2c121180 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d217fa640 10 monclient: dump: 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.848 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d237fe640 4 set_mon_vals no callback set 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d332c0640 10 monclient: _renew_subs 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d332c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.828+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c13d690 con 0x7f1d2c121180 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.829+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1d1c038420 con 0x7f1d2c121180 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.829+0000 7f1d332c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.829+0000 7f1d332c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:43.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.829+0000 7f1d332c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.829+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c0636e0 con 0x7f1d2c121180 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.830+0000 7f1d217fa640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d100384c0 0x7f1d1003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.830+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(125..125 src has 13..125) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f1d1c06c5d0 con 0x7f1d2c121180 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.830+0000 7f1d3105c640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d100384c0 0x7f1d1003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.830+0000 7f1d3105c640 10 monclient: get_auth_request con 0x7f1d100384c0 auth_method 0 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.832+0000 7f1d3105c640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d100384c0 0x7f1d1003a970 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d28007b10 tx=0x7f1d28007c80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.834+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1d1c007800 con 0x7f1d2c121180 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.834+0000 7f1d217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:43.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.834+0000 7f1d217fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:43.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.995+0000 7f1d332c0640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T12:30:43.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.995+0000 7f1d332c0640 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T12:30:43.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.995+0000 7f1d332c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:43.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:43.995+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7f1d2c09f320 con 0x7f1d2c121180 2024-09-18T12:30:44.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:44.634+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v126) v1 ==== 145+0+0 (secure 0 0 0) 0x7f1d1c02e370 con 0x7f1d2c121180 2024-09-18T12:30:44.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:44.634+0000 7f1d217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T12:30:44.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:44.634+0000 7f1d217fa640 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-18T12:30:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:44.735+0000 7f1d332c0640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T12:30:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:44.735+0000 7f1d332c0640 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T12:30:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:44.735+0000 7f1d332c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:44.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:44.735+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7f1d2c072640 con 0x7f1d2c121180 2024-09-18T12:30:45.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.637+0000 7f1d217fa640 1 -- 172.21.15.144:0/2466572622 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v127) v1 ==== 145+0+0 (secure 0 0 0) 0x7f1d1c049030 con 0x7f1d2c121180 2024-09-18T12:30:45.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.637+0000 7f1d217fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T12:30:45.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.637+0000 7f1d217fa640 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-18T12:30:45.638 INFO:tasks.workunit.client.0.smithi144.stderr:set require_min_compat_client to luminous 2024-09-18T12:30:45.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d100384c0 msgr2=0x7f1d1003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:45.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d100384c0 0x7f1d1003a970 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1d28007b10 tx=0x7f1d28007c80 comp rx=0 tx=0).stop 2024-09-18T12:30:45.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 10 monclient: shutdown 2024-09-18T12:30:45.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:45.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 msgr2=0x7f1d2c13cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:45.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 0x7f1d2c13cfb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c007af0 tx=0x7f1d1c034000 comp rx=0 tx=0).stop 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 shutdown_connections 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d100384c0 0x7f1d1003a970 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d2c13e280 0x7f1d2c140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d2c121180 0x7f1d2c13cfb0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 --2- 172.21.15.144:0/2466572622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d18004780 0x7f1d2c13d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 >> 172.21.15.144:0/2466572622 conn(0x7f1d2c071350 msgr2=0x7f1d2c09c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 shutdown_connections 2024-09-18T12:30:45.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.640+0000 7f1d332c0640 1 -- 172.21.15.144:0/2466572622 wait complete. 2024-09-18T12:30:45.651 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set dump 2024-09-18T12:30:45.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.723+0000 7f29930a8640 10 monclient: get_monmap_and_config 2024-09-18T12:30:45.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.723+0000 7f29930a8640 10 monclient: build_initial_monmap 2024-09-18T12:30:45.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.723+0000 7f29930a8640 10 monclient: monmap: 2024-09-18T12:30:45.724 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:45.724245+0000 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:45.724245+0000 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:45.725 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:45.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 1 Processor -- start 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 1 -- start start 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient: init 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f298c131ad0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c09f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient(hunting): picked mon.noname-b con 0x7f298c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f298c136e00 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c09c010 con 0x7f298c136e00 2024-09-18T12:30:45.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c09ffe0 con 0x7f298c131ad0 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c09f5b0 con 0x7f298c09ec50 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f29930a8640 10 monclient(hunting): authenticate will time out at 1081.293578s 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f2990e44640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f298c131ad0 0x7f298c132180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58276/0 (socket says 172.21.15.144:58276) 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f2990e44640 1 -- 172.21.15.144:0/3379476173 learned_addr learned my addr 172.21.15.144:0/3379476173 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f298bfff640 1 --2- 172.21.15.144:0/3379476173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c09f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f298bfff640 10 monclient(hunting): get_auth_request con 0x7f298c09ec50 auth_method 0 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f298bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f298bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.724+0000 7f298bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:45.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 10 monclient(hunting): handle_auth_done global_id 64436 payload 274 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 1 -- 172.21.15.144:0/3379476173 >> v1:172.21.15.144:6789/0 conn(0x7f298c131ad0 legacy=0x7f298c132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 1 -- 172.21.15.144:0/3379476173 >> v1:172.21.15.116:6789/0 conn(0x7f298c136e00 legacy=0x7f298c1391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 1 monclient: found mon.noname-b 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:45.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.725+0000 7f298bfff640 1 -- 172.21.15.144:0/3379476173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c132890 con 0x7f298c09ec50 2024-09-18T12:30:45.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298bfff640 1 -- 172.21.15.144:0/3379476173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 msgr2=0x7f298c09f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:45.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298bfff640 1 --2- 172.21.15.144:0/3379476173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c09f070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2978009880 tx=0x7f29780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e54de2fa23ad479c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:45.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 1 -- 172.21.15.144:0/3379476173 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297802aea0 con 0x7f298c09ec50 2024-09-18T12:30:45.736 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: dump: 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:45.737 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 1 -- 172.21.15.144:0/3379476173 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2978032440 con 0x7f298c09ec50 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 1 -- 172.21.15.144:0/3379476173 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29780327e0 con 0x7f298c09ec50 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f298b7fe640 10 monclient: dump: 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.738 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f29930a8640 5 monclient: authenticate success, global_id 64436 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f29930a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f29930a8640 4 set_mon_vals no callback set 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.728+0000 7f29930a8640 10 monclient: shutdown 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 -- 172.21.15.144:0/3379476173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 msgr2=0x7f298c09f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 --2- 172.21.15.144:0/3379476173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c09f070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2978009880 tx=0x7f29780090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 -- 172.21.15.144:0/3379476173 shutdown_connections 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 --2- 172.21.15.144:0/3379476173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c09f070 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 -- 172.21.15.144:0/3379476173 >> 172.21.15.144:0/3379476173 conn(0x7f298c0a5090 msgr2=0x7f298c0a5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 -- 172.21.15.144:0/3379476173 shutdown_connections 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 -- 172.21.15.144:0/3379476173 wait complete. 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 10 monclient: build_initial_monmap 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 10 monclient: monmap: 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:45.740 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.729+0000 7f29930a8640 1 Processor -- start 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 1 -- start start 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient: init 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): picked mon.noname-c con 0x7f298c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 0x7f298c0999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:45.741 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): picked mon.noname-b con 0x7f298c131ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298c136e00 0x7f298c099f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): picked mon.noname-a con 0x7f298c136e00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0a0710 con 0x7f298c136e00 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0a25f0 con 0x7f298c131ad0 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c11b810 con 0x7f298c09ec50 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f29930a8640 10 monclient(hunting): authenticate will time out at 1081.299449s 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 0x7f298c0999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:45.742 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 0x7f298c0999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44582/0 (socket says 172.21.15.144:44582) 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 -- 172.21.15.144:0/4234039625 learned_addr learned my addr 172.21.15.144:0/4234039625 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): get_auth_request con 0x7f298c131ad0 auth_method 0 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2990e44640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2991645640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298c136e00 0x7f298c099f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2990e44640 10 monclient(hunting): get_auth_request con 0x7f298c09ec50 auth_method 0 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2990e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:45.743 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2990e44640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2990e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2991645640 10 monclient(hunting): get_auth_request con 0x7f298c136e00 auth_method 0 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2991645640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2991645640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f2991645640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient(hunting): handle_auth_done global_id 5620 payload 824 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 -- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 msgr2=0x7f298c130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c130a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 -- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298c136e00 msgr2=0x7f298c099f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298c136e00 0x7f298c099f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 monclient: found mon.noname-b 2024-09-18T12:30:45.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.730+0000 7f298bfff640 1 -- 172.21.15.144:0/4234039625 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978009390 con 0x7f298c131ad0 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f298bfff640 1 -- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 msgr2=0x7f298c0999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f298bfff640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 0x7f298c0999f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2978009880 tx=0x7f2978032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297802aea0 con 0x7f298c131ad0 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: dump: 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.745 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2978031990 con 0x7f298c131ad0 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f29930a8640 5 monclient: authenticate success, global_id 5620 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f29930a8640 10 monclient: _renew_subs 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f29930a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:45.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c142040 con 0x7f298c131ad0 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978031d50 con 0x7f298c131ad0 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f2988ff9640 10 monclient: dump: 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:45.747 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f298affd640 4 set_mon_vals no callback set 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f29930a8640 10 monclient: _renew_subs 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f29930a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.733+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c09b4f0 con 0x7f298c131ad0 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2978039040 con 0x7f298c131ad0 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f29930a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f29930a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f29930a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c04bf60 con 0x7f298c131ad0 2024-09-18T12:30:45.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f2988ff9640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2970038490 0x7f297003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:45.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(127..127 src has 13..127) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f297806c100 con 0x7f298c131ad0 2024-09-18T12:30:45.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f2990e44640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2970038490 0x7f297003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:45.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.734+0000 7f2990e44640 10 monclient: get_auth_request con 0x7f2970038490 auth_method 0 2024-09-18T12:30:45.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.736+0000 7f2990e44640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2970038490 0x7f297003a940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2980007fd0 tx=0x7f2980009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:45.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.738+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f297804bd50 con 0x7f298c131ad0 2024-09-18T12:30:45.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.738+0000 7f2988ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:45.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.738+0000 7f2988ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:45.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.898+0000 7f29930a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:30:45.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.898+0000 7f29930a8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:30:45.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.898+0000 7f29930a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:45.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.898+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f298c04bf60 con 0x7f298c131ad0 2024-09-18T12:30:45.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.898+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v127) v1 ==== 73+0+3 (secure 0 0 0) 0x7f2978037090 con 0x7f298c131ad0 2024-09-18T12:30:45.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.898+0000 7f2988ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:30:45.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.898+0000 7f2988ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:45.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.998+0000 7f29930a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:30:45.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.998+0000 7f29930a8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:30:45.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.998+0000 7f29930a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:45.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.998+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f298c04bf60 con 0x7f298c131ad0 2024-09-18T12:30:46.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.998+0000 7f2988ff9640 1 -- 172.21.15.144:0/4234039625 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v127) v1 ==== 73+0+3 (secure 0 0 0) 0x7f2978049030 con 0x7f298c131ad0 2024-09-18T12:30:46.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.998+0000 7f2988ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:30:46.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:45.998+0000 7f2988ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:46.000 INFO:tasks.workunit.client.0.smithi144.stdout:{} 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2970038490 msgr2=0x7f297003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2970038490 0x7f297003a940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2980007fd0 tx=0x7f2980009040 comp rx=0 tx=0).stop 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 10 monclient: shutdown 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 msgr2=0x7f298c0999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 0x7f298c0999f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2978009880 tx=0x7f2978032000 comp rx=0 tx=0).stop 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 shutdown_connections 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2970038490 0x7f297003a940 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f298c136e00 0x7f298c099f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f298c131ad0 0x7f298c0999f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 --2- 172.21.15.144:0/4234039625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f298c09ec50 0x7f298c130a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 >> 172.21.15.144:0/4234039625 conn(0x7f298c0a5090 msgr2=0x7f298c09e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:46.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 shutdown_connections 2024-09-18T12:30:46.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.000+0000 7f29930a8640 1 -- 172.21.15.144:0/4234039625 wait complete. 2024-09-18T12:30:46.010 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:30:46.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.083+0000 7f5bc58e8640 10 monclient: get_monmap_and_config 2024-09-18T12:30:46.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.083+0000 7f5bc58e8640 10 monclient: build_initial_monmap 2024-09-18T12:30:46.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.083+0000 7f5bc58e8640 10 monclient: monmap: 2024-09-18T12:30:46.084 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:46.084 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:46.084178+0000 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:46.084178+0000 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:46.085 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.083+0000 7f5bc58e8640 1 Processor -- start 2024-09-18T12:30:46.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.083+0000 7f5bc58e8640 1 -- start start 2024-09-18T12:30:46.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.083+0000 7f5bc58e8640 10 monclient: init 2024-09-18T12:30:46.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:46.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T12:30:46.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc0131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc01319c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 0x7f5bc009ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5bc009eb30 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bc0068e00 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc0131d90 con 0x7f5bc0068e00 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc006a260 con 0x7f5bc009eb30 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00a0050 con 0x7f5bc01319c0 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bc58e8640 10 monclient(hunting): authenticate will time out at 1081.653464s 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbf7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f5bc0068e00 0x7f5bc0069280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58282/0 (socket says 172.21.15.144:58282) 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbf7fe640 1 -- 172.21.15.144:0/2920416118 learned_addr learned my addr 172.21.15.144:0/2920416118 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:46.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 1 --2- 172.21.15.144:0/2920416118 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc0131fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 1 --2- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 0x7f5bc009ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 10 monclient(hunting): get_auth_request con 0x7f5bc01319c0 auth_method 0 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 10 monclient(hunting): get_auth_request con 0x7f5bc009eb30 auth_method 0 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 1 -- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 msgr2=0x7f5bc009ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 1 -- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 msgr2=0x7f5bc009ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 1 --2- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 0x7f5bc009ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.084+0000 7f5bbe7fc640 1 --2- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 0x7f5bc009ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_done global_id 64442 payload 274 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 1 -- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 msgr2=0x7f5bc009ef00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:46.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 1 --2- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 0x7f5bc009ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 1 -- 172.21.15.144:0/2920416118 >> v1:172.21.15.144:6789/0 conn(0x7f5bc0068e00 legacy=0x7f5bc0069280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 1 monclient: found mon.noname-b 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.085+0000 7f5bbeffd640 1 -- 172.21.15.144:0/2920416118 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc009fe10 con 0x7f5bc01319c0 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.087+0000 7f5bbeffd640 1 -- 172.21.15.144:0/2920416118 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 msgr2=0x7f5bc0131fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.087+0000 7f5bbeffd640 1 --2- 172.21.15.144:0/2920416118 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc0131fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009e60 tx=0x7f5bb8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12d5b43d4d01e740 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 1 -- 172.21.15.144:0/2920416118 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8007760 con 0x7f5bc01319c0 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: dump: 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.094 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 1 -- 172.21.15.144:0/2920416118 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb802d050 con 0x7f5bc01319c0 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 5 monclient: authenticate success, global_id 64442 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 1 -- 172.21.15.144:0/2920416118 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8007af0 con 0x7f5bc01319c0 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 4 set_mon_vals no callback set 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bbdffb640 10 monclient: dump: 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 10 monclient: shutdown 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 -- 172.21.15.144:0/2920416118 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 msgr2=0x7f5bc0131fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/2920416118 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc0131fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009e60 tx=0x7f5bb8009780 comp rx=0 tx=0).stop 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 -- 172.21.15.144:0/2920416118 shutdown_connections 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/2920416118 >> v2:172.21.15.116:3300/0 conn(0x7f5bc009eb30 0x7f5bc009ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/2920416118 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc0131fe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 -- 172.21.15.144:0/2920416118 >> 172.21.15.144:0/2920416118 conn(0x7f5bc0071370 msgr2=0x7f5bc00a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 -- 172.21.15.144:0/2920416118 shutdown_connections 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.088+0000 7f5bc58e8640 1 -- 172.21.15.144:0/2920416118 wait complete. 2024-09-18T12:30:46.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.089+0000 7f5bc58e8640 10 monclient: build_initial_monmap 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.089+0000 7f5bc58e8640 10 monclient: monmap: 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:46.098 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.089+0000 7f5bc58e8640 1 Processor -- start 2024-09-18T12:30:46.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.089+0000 7f5bc58e8640 1 -- start start 2024-09-18T12:30:46.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.089+0000 7f5bc58e8640 10 monclient: init 2024-09-18T12:30:46.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:46.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:46.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 0x7f5bc009ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc0068e00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc009eb30 0x7f5bc0130cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5bc009eb30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc011b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5bc01319c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc0069e90 con 0x7f5bc009eb30 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00a3380 con 0x7f5bc0068e00 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc011e140 con 0x7f5bc01319c0 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bc58e8640 10 monclient(hunting): authenticate will time out at 1081.659368s 2024-09-18T12:30:46.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 0x7f5bc009ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 0x7f5bc009ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44596/0 (socket says 172.21.15.144:44596) 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 -- 172.21.15.144:0/3115737522 learned_addr learned my addr 172.21.15.144:0/3115737522 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): get_auth_request con 0x7f5bc0068e00 auth_method 0 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbe7fc640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc009eb30 0x7f5bc0130cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbf7fe640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc011b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient(hunting): handle_auth_done global_id 5623 payload 824 2024-09-18T12:30:46.105 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 -- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 msgr2=0x7f5bc011b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc011b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 -- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc009eb30 msgr2=0x7f5bc0130cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc009eb30 0x7f5bc0130cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 monclient: found mon.noname-b 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbeffd640 1 -- 172.21.15.144:0/3115737522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8009a20 con 0x7f5bc0068e00 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbe7fc640 10 monclient: get_auth_request con 0x7f5bc009eb30 auth_method 0 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.090+0000 7f5bbe7fc640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc009eb30 0x7f5bc0130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.092+0000 7f5bbeffd640 1 -- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 msgr2=0x7f5bc009ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.092+0000 7f5bbeffd640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 0x7f5bc009ab40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009ec0 tx=0x7f5bb8037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:46.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8031b20 con 0x7f5bc0068e00 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: dump: 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb8040070 con 0x7f5bc0068e00 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bc58e8640 5 monclient: authenticate success, global_id 5623 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8045430 con 0x7f5bc0068e00 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bc58e8640 10 monclient: _renew_subs 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bc58e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc0142030 con 0x7f5bc0068e00 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bb6ffd640 10 monclient: dump: 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.108 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.109 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.112 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.112 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.112 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.112 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.112 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bbd7fa640 4 set_mon_vals no callback set 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bc58e8640 10 monclient: _renew_subs 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bc58e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.093+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc011ce70 con 0x7f5bc0068e00 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5bb8042030 con 0x7f5bc0068e00 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bb6ffd640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ba0038490 0x7f5ba003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(127..127 src has 13..127) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f5bb806d810 con 0x7f5bc0068e00 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bc58e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bbe7fc640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ba0038490 0x7f5ba003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bc58e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bc58e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc011df70 con 0x7f5bc0068e00 2024-09-18T12:30:46.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.094+0000 7f5bbe7fc640 10 monclient: get_auth_request con 0x7f5ba0038490 auth_method 0 2024-09-18T12:30:46.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.096+0000 7f5bbe7fc640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ba0038490 0x7f5ba003a940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00a3250 tx=0x7f5ba8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:46.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.098+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5bb804ee30 con 0x7f5bc0068e00 2024-09-18T12:30:46.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.098+0000 7f5bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:46.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.098+0000 7f5bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:46.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.261+0000 7f5bc58e8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:30:46.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.261+0000 7f5bc58e8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:30:46.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.261+0000 7f5bc58e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.261+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f5bc004bf60 con 0x7f5bc0068e00 2024-09-18T12:30:46.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.261+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v127) v1 ==== 71+0+0 (secure 0 0 0) 0x7f5bb803a3a0 con 0x7f5bc0068e00 2024-09-18T12:30:46.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.261+0000 7f5bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:30:46.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.261+0000 7f5bb6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.361+0000 7f5bc58e8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.361+0000 7f5bc58e8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.361+0000 7f5bc58e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.361+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f5bc004bf60 con 0x7f5bc0068e00 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.361+0000 7f5bb6ffd640 1 -- 172.21.15.144:0/3115737522 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v127) v1 ==== 71+0+0 (secure 0 0 0) 0x7f5bb803a580 con 0x7f5bc0068e00 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.361+0000 7f5bb6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.361+0000 7f5bb6ffd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:30:46.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.363+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ba0038490 msgr2=0x7f5ba003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ba0038490 0x7f5ba003a940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00a3250 tx=0x7f5ba8008040 comp rx=0 tx=0).stop 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 10 monclient: shutdown 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 msgr2=0x7f5bc009ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 0x7f5bc009ab40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009ec0 tx=0x7f5bb8037000 comp rx=0 tx=0).stop 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 shutdown_connections 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ba0038490 0x7f5ba003a940 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5bc01319c0 0x7f5bc011b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bc009eb30 0x7f5bc0130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 --2- 172.21.15.144:0/3115737522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc0068e00 0x7f5bc009ab40 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 >> 172.21.15.144:0/3115737522 conn(0x7f5bc0071370 msgr2=0x7f5bc00a4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 shutdown_connections 2024-09-18T12:30:46.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.364+0000 7f5bc58e8640 1 -- 172.21.15.144:0/3115737522 wait complete. 2024-09-18T12:30:46.374 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T12:30:46.374 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:30:46.374 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.448+0000 7f0dfdf9d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.449+0000 7f0dfdf9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.449+0000 7f0dfdf9d640 10 monclient: monmap: 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:46.449922+0000 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:46.449922+0000 2024-09-18T12:30:46.450 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:46.451 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.451 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:46.451 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:46.451 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:46.451 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:46.451 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:46.451 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.449+0000 7f0dfdf9d640 1 Processor -- start 2024-09-18T12:30:46.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.449+0000 7f0dfdf9d640 1 -- start start 2024-09-18T12:30:46.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.449+0000 7f0dfdf9d640 10 monclient: init 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0df809ec50 0x7f0df8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0df809ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0df81321a0 0x7f0df8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0df81321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df809f020 0x7f0df8069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0df809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df8067420 con 0x7f0df81321a0 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df809fdc0 con 0x7f0df809ec50 2024-09-18T12:30:46.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df80a0510 con 0x7f0df809f020 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0dfdf9d640 10 monclient(hunting): authenticate will time out at 1082.019435s 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0df809ec50 0x7f0df8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0df809ec50 0x7f0df8131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44604/0 (socket says 172.21.15.144:44604) 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 -- 172.21.15.144:0/3180816280 learned_addr learned my addr 172.21.15.144:0/3180816280 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): get_auth_request con 0x7f0df809ec50 auth_method 0 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:46.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df7fff640 1 --2- 172.21.15.144:0/3180816280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df809f020 0x7f0df8069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0deffff640 1 --2- 172.21.15.144:0/3180816280 >> v2:172.21.15.116:3300/0 conn(0x7f0df81321a0 0x7f0df8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df7fff640 10 monclient(hunting): get_auth_request con 0x7f0df809f020 auth_method 0 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0deffff640 10 monclient(hunting): get_auth_request con 0x7f0df81321a0 auth_method 0 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0deffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient(hunting): handle_auth_done global_id 5626 payload 274 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 -- 172.21.15.144:0/3180816280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df809f020 msgr2=0x7f0df8069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:46.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3180816280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df809f020 0x7f0df8069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 -- 172.21.15.144:0/3180816280 >> v2:172.21.15.116:3300/0 conn(0x7f0df81321a0 msgr2=0x7f0df8068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:46.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3180816280 >> v2:172.21.15.116:3300/0 conn(0x7f0df81321a0 0x7f0df8068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:46.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 monclient: found mon.noname-c 2024-09-18T12:30:46.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:46.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.450+0000 7f0df77fe640 1 -- 172.21.15.144:0/3180816280 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df80a0200 con 0x7f0df809ec50 2024-09-18T12:30:46.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df77fe640 1 -- 172.21.15.144:0/3180816280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809ec50 msgr2=0x7f0df8131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:46.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3180816280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809ec50 0x7f0df8131c60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0df80719d0 tx=0x7f0de80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5895da00c9e0e8c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:46.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 1 -- 172.21.15.144:0/3180816280 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8032440 con 0x7f0df809ec50 2024-09-18T12:30:46.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: dump: 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 1 -- 172.21.15.144:0/3180816280 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80325a0 con 0x7f0df809ec50 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0dfdf9d640 5 monclient: authenticate success, global_id 5626 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0dfdf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0dfdf9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0dfdf9d640 4 set_mon_vals no callback set 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 1 -- 172.21.15.144:0/3180816280 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8032980 con 0x7f0df809ec50 2024-09-18T12:30:46.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.453+0000 7f0df6ffd640 10 monclient: dump: 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 10 monclient: shutdown 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3180816280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809ec50 msgr2=0x7f0df8131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 --2- 172.21.15.144:0/3180816280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809ec50 0x7f0df8131c60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0df80719d0 tx=0x7f0de80091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3180816280 shutdown_connections 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 --2- 172.21.15.144:0/3180816280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df809f020 0x7f0df8069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 --2- 172.21.15.144:0/3180816280 >> v2:172.21.15.116:3300/0 conn(0x7f0df81321a0 0x7f0df8068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 --2- 172.21.15.144:0/3180816280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809ec50 0x7f0df8131c60 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3180816280 >> 172.21.15.144:0/3180816280 conn(0x7f0df80a5d40 msgr2=0x7f0df80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3180816280 shutdown_connections 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3180816280 wait complete. 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.454+0000 7f0dfdf9d640 10 monclient: monmap: 2024-09-18T12:30:46.467 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 Processor -- start 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 -- start start 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient: init 2024-09-18T12:30:46.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 0x7f0df809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0df809ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809f020 0x7f0df8130ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0df809f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df81321a0 0x7f0df8140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0df81321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df811c8f0 con 0x7f0df809ec50 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df811dec0 con 0x7f0df809f020 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.469 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df811dbe0 con 0x7f0df81321a0 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0dfdf9d640 10 monclient(hunting): authenticate will time out at 1082.024632s 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809f020 0x7f0df8130ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809f020 0x7f0df8130ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44618/0 (socket says 172.21.15.144:44618) 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 -- 172.21.15.144:0/3776408389 learned_addr learned my addr 172.21.15.144:0/3776408389 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): get_auth_request con 0x7f0df809f020 auth_method 0 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:46.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df81321a0 0x7f0df8140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 10 monclient(hunting): get_auth_request con 0x7f0df81321a0 auth_method 0 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 0x7f0df809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 10 monclient(hunting): get_auth_request con 0x7f0df809ec50 auth_method 0 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:46.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 1 -- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 msgr2=0x7f0df809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient(hunting): handle_auth_done global_id 5629 payload 824 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 1 -- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 msgr2=0x7f0df809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 0x7f0df809a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 -- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df81321a0 msgr2=0x7f0df8140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 0x7f0df809a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0df81321a0 0x7f0df8140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 -- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 msgr2=0x7f0df809a9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0df809ec50 0x7f0df809a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:46.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 monclient: found mon.noname-b 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.455+0000 7f0deffff640 1 -- 172.21.15.144:0/3776408389 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8009450 con 0x7f0df809f020 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0deffff640 1 -- 172.21.15.144:0/3776408389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809f020 msgr2=0x7f0df8130ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0deffff640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0df809f020 0x7f0df8130ac0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0de000d890 tx=0x7f0de0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 1 -- 172.21.15.144:0/3776408389 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de000de20 con 0x7f0df809f020 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: dump: 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.473 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0dfdf9d640 5 monclient: authenticate success, global_id 5629 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 1 -- 172.21.15.144:0/3776408389 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de000bae0 con 0x7f0df809f020 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 1 -- 172.21.15.144:0/3776408389 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de000be80 con 0x7f0df809f020 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:46.474 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0def7fe640 10 monclient: dump: 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0dfdf9d640 10 monclient: _renew_subs 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0df67fc640 4 set_mon_vals no callback set 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0dfdf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.475 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3776408389 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df811e1a0 con 0x7f0df809f020 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0dfdf9d640 10 monclient: _renew_subs 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0dfdf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.458+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3776408389 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df811cef0 con 0x7f0df809f020 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0def7fe640 1 -- 172.21.15.144:0/3776408389 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0de0014640 con 0x7f0df809f020 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0dfdf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0dfdf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0dfdf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3776408389 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df80008d0 con 0x7f0df809f020 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0def7fe640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dd8038490 0x7f0dd803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0def7fe640 1 -- 172.21.15.144:0/3776408389 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(127..127 src has 13..127) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f0de004c530 con 0x7f0df809f020 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dd8038490 0x7f0dd803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.459+0000 7f0df77fe640 10 monclient: get_auth_request con 0x7f0dd8038490 auth_method 0 2024-09-18T12:30:46.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.461+0000 7f0df77fe640 1 --2- 172.21.15.144:0/3776408389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dd8038490 0x7f0dd803a940 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0de8034000 tx=0x7f0de8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:46.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.463+0000 7f0def7fe640 1 -- 172.21.15.144:0/3776408389 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0de0011570 con 0x7f0df809f020 2024-09-18T12:30:46.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.463+0000 7f0def7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:46.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.463+0000 7f0def7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:46.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.627+0000 7f0dfdf9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:30:46.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.627+0000 7f0dfdf9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:30:46.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.627+0000 7f0dfdf9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:46.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:46.627+0000 7f0dfdf9d640 1 -- 172.21.15.144:0/3776408389 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f0df80a13e0 con 0x7f0df809f020 2024-09-18T12:30:46.778 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.881+0000 7f75330c0640 10 monclient: get_monmap_and_config 2024-09-18T12:30:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.881+0000 7f75330c0640 10 monclient: build_initial_monmap 2024-09-18T12:30:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.881+0000 7f75330c0640 10 monclient: monmap: 2024-09-18T12:30:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:46.882198+0000 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:46.882198+0000 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:46.883 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 1 Processor -- start 2024-09-18T12:30:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 1 -- start start 2024-09-18T12:30:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient: init 2024-09-18T12:30:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient(hunting): picked mon.noname-b con 0x7f752c0981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient(hunting): picked mon.noname-a con 0x7f752c131da0 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f752c132170 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c098ad0 con 0x7f752c132170 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752c131bd0 con 0x7f752c131da0 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752c09ca60 con 0x7f752c0981c0 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.882+0000 7f75330c0640 10 monclient(hunting): authenticate will time out at 1074.203900s 2024-09-18T12:30:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57678/0 (socket says 172.21.15.116:57678) 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 learned_addr learned my addr 172.21.15.116:0/1808819542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f752c0981c0 auth_method 0 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f752c131da0 auth_method 0 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.883+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:46.885+0000 7f753165d640 1 -- 172.21.15.116:0/1808819542 >> v1:172.21.15.144:6789/0 conn(0x7f752c132170 legacy=0x7f752c067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:47.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f752c0981c0 auth_method 0 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f752c131da0 auth_method 0 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.084+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:47.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f752c0981c0 auth_method 0 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f752c131da0 auth_method 0 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:47.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:47.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:47.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:47.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:47.485+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:48.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f752c0981c0 auth_method 0 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f752c131da0 auth_method 0 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:48.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:48.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:48.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:48.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:48.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:30:48.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:48.286+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:30:49.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:49.456+0000 7f0dee7fc640 10 monclient: tick 2024-09-18T12:30:49.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:49.456+0000 7f0dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:49.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:49.456+0000 7f0dee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:49.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 10 monclient(hunting): tick 2024-09-18T12:30:49.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 monclient(hunting): continuing hunt 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 msgr2=0x7f752c098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f752c0981c0 0x7f752c098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 msgr2=0x7f752c067110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.116:3300/0 conn(0x7f752c131da0 0x7f752c067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 -- 172.21.15.116:0/1808819542 >> v1:172.21.15.144:6789/0 conn(0x7f752c132170 legacy=0x7f752c067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 10 monclient: _add_conns ranks=[0,4] 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7510003720 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f7510007420 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 -- 172.21.15.116:0/1808819542 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c09cca0 con 0x7f7510003720 2024-09-18T12:30:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 1 -- 172.21.15.116:0/1808819542 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752c131bd0 con 0x7f7510007420 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752a7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f7510007420 auth_method 0 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:49.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.883+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:49.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:49.885+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> v1:172.21.15.116:6789/0 conn(0x7f7510003720 legacy=0x7f7510005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f7510007420 auth_method 0 2024-09-18T12:30:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.084+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:50.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:50.456+0000 7f0dee7fc640 10 monclient: tick 2024-09-18T12:30:50.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:50.456+0000 7f0dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f7510007420 auth_method 0 2024-09-18T12:30:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:50.485+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f7510007420 auth_method 0 2024-09-18T12:30:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.286+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:30:51.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.456+0000 7f0dee7fc640 10 monclient: tick 2024-09-18T12:30:51.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.456+0000 7f0dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:51.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.566+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492685026 0 0) 0x7f752c09cca0 con 0x7f7510003720 2024-09-18T12:30:51.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.566+0000 7f752b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.566+0000 7f752b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.566+0000 7f752b7fe640 10 monclient(hunting): my global_id is 84099 2024-09-18T12:30:51.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.566+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f751c0034e0 con 0x7f7510003720 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 764353784 0 0) 0x7f751c0034e0 con 0x7f7510003720 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 msgr2=0x7f75100098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 --2- 172.21.15.116:0/1808819542 >> v2:172.21.15.144:3300/0 conn(0x7f7510007420 0x7f75100098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751000b000 con 0x7f7510003720 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:51.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f752000c680 con 0x7f7510003720 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f75330c0640 5 monclient: authenticate success, global_id 84099 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f75330c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: dump: 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 >> v1:172.21.15.116:6789/0 conn(0x7f7510003720 legacy=0x7f7510005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f753165d640 1 -- 172.21.15.116:0/1808819542 reap_dead start 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c003ac0 0x7f751c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): picked mon.a con 0x7f751c003ac0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c004780 0x7f751c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): picked mon.c con 0x7f751c004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c005330 0x7f751c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): picked mon.b con 0x7f751c005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c131bd0 con 0x7f751c003ac0 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c00c2f0 con 0x7f751c005330 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c00c4b0 con 0x7f751c004780 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f75330c0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c003ac0 0x7f751c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c004780 0x7f751c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f751c003ac0 auth_method 0 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f7530e5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f751c004780 auth_method 0 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.567+0000 7f753165d640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c005330 0x7f751c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f753165d640 10 monclient(hunting): get_auth_request con 0x7f751c005330 auth_method 0 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f753165d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f753165d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f753165d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 10 monclient(hunting): handle_auth_done global_id 84099 payload 293 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c004780 msgr2=0x7f751c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c004780 0x7f751c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c005330 msgr2=0x7f751c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c005330 0x7f751c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:51.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 1 monclient: found mon.a 2024-09-18T12:30:51.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:51.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0034e0 con 0x7f751c003ac0 2024-09-18T12:30:51.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 10 monclient: _finish_auth 0 2024-09-18T12:30:51.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f7530e5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:51.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.568+0000 7f752bfff640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c004780 0x7f751c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:51.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.569+0000 7f0def7fe640 1 -- 172.21.15.144:0/3776408389 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v127) v1 ==== 152+0+0 (secure 0 0 0) 0x7f0de001dbb0 con 0x7f0df809f020 2024-09-18T12:30:51.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.569+0000 7f0def7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:30:51.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.569+0000 7f0def7fe640 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-18T12:30:51.579 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOENT: pool 'fooset' not found 2024-09-18T12:30:51.579 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:30:51.579 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create fooset 8 2024-09-18T12:30:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f7530e5c640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c003ac0 msgr2=0x7f751c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c003ac0 0x7f751c003eb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7520006fd0 tx=0x7f7520008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317bb4609b103b73 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520036440 con 0x7f751c003ac0 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 10 monclient: dump: 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752000c040 con 0x7f751c003ac0 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f75330c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f75330c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 1 -- 172.21.15.116:0/1808819542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75200365e0 con 0x7f751c003ac0 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f75330c0640 4 set_mon_vals no callback set 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.571+0000 7f752b7fe640 10 monclient: dump: 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 10 monclient: shutdown 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 -- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c003ac0 msgr2=0x7f751c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c003ac0 0x7f751c003eb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7520006fd0 tx=0x7f7520008040 comp rx=0 tx=0).stop 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 -- 172.21.15.116:0/1808819542 shutdown_connections 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c005330 0x7f751c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c004780 0x7f751c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 --2- 172.21.15.116:0/1808819542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c003ac0 0x7f751c003eb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 -- 172.21.15.116:0/1808819542 >> 172.21.15.116:0/1808819542 conn(0x7f752c0a1e30 msgr2=0x7f752c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 -- 172.21.15.116:0/1808819542 shutdown_connections 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.572+0000 7f75330c0640 1 -- 172.21.15.116:0/1808819542 wait complete. 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.573+0000 7f75330c0640 10 monclient: build_initial_monmap 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.573+0000 7f75330c0640 10 monclient: monmap: 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.573+0000 7f75330c0640 1 Processor -- start 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.573+0000 7f75330c0640 1 -- start start 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.573+0000 7f75330c0640 10 monclient: init 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f752c140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): picked mon.noname-b con 0x7f751c003ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 0x7f752c130950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): picked mon.noname-a con 0x7f751c004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f752c11a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): picked mon.noname-c con 0x7f751c005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c11dd50 con 0x7f751c004780 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c09b5b0 con 0x7f751c003ac0 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752c141890 con 0x7f751c005330 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f75330c0640 10 monclient(hunting): authenticate will time out at 1078.895452s 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f752c11a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 0x7f752c130950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f752c11a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57744/0 (socket says 172.21.15.116:57744) 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 1 -- 172.21.15.116:0/1199941859 learned_addr learned my addr 172.21.15.116:0/1199941859 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 10 monclient(hunting): get_auth_request con 0x7f751c005330 auth_method 0 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f751c004780 auth_method 0 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f752c140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f751c003ac0 auth_method 0 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 msgr2=0x7f752c11a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 msgr2=0x7f752c11a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f752c11a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f753165d640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f752c11a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:51.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient(hunting): handle_auth_done global_id 84102 payload 824 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 msgr2=0x7f752c11a400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f752c11a400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 msgr2=0x7f752c140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f752c140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 1 monclient: found mon.noname-a 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.574+0000 7f752bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.575+0000 7f752bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.575+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c11da70 con 0x7f751c004780 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 msgr2=0x7f752c130950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 0x7f752c130950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7518006e70 tx=0x7f7518008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 msgr2=0x7f752c130950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 msgr2=0x7f752c130950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 0x7f752c130950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7518006e70 tx=0x7f7518008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 0x7f752c130950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7518006e70 tx=0x7f7518008040 comp rx=0 tx=0).stop 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 msgr2=0x7f752c130950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f751c004780 0x7f752c130950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f7510005680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f751c003ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 0x7f7510009330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f7510008ec0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:51.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510006300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f751c005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7510006ad0 con 0x7f7510008ec0 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7510006c10 con 0x7f751c003ac0 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7510006dd0 con 0x7f751c005330 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7528ff9640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 0x7f7510009330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510006300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f7510005680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f7510008ec0 auth_method 0 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 10 monclient(hunting): get_auth_request con 0x7f751c005330 auth_method 0 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f751c003ac0 auth_method 0 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.577+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 10 monclient(hunting): handle_auth_done global_id 84102 payload 293 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 msgr2=0x7f7510006300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510006300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 msgr2=0x7f7510005680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f7510005680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 1 monclient: found mon.noname-a 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.578+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510007050 con 0x7f7510008ec0 2024-09-18T12:30:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 msgr2=0x7f7510009330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 0x7f7510009330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7518025000 tx=0x7f7518029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751801e050 con 0x7f7510008ec0 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: dump: 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.600 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751801f040 con 0x7f7510008ec0 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f75330c0640 5 monclient: authenticate success, global_id 84102 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f75330c0640 10 monclient: _renew_subs 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f75330c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c11cb10 con 0x7f7510008ec0 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518005c20 con 0x7f7510008ec0 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f7528ff9640 10 monclient: dump: 2024-09-18T12:30:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f75330c0640 10 monclient: _renew_subs 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f75330c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c140bc0 con 0x7f7510008ec0 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.580+0000 7f752affd640 4 set_mon_vals no callback set 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7518035070 con 0x7f7510008ec0 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f751c004780 0x7f7510043470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f75330c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f75330c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f75330c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 13..127) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f751804f7b0 con 0x7f7510008ec0 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752c098d20 con 0x7f7510008ec0 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f751c004780 0x7f7510043470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.581+0000 7f7530e5c640 10 monclient: get_auth_request con 0x7f751c004780 auth_method 0 2024-09-18T12:30:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.584+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f751c004780 0x7f7510043470 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7520033000 tx=0x7f7520039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.586+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f752c098d20 con 0x7f7510008ec0 2024-09-18T12:30:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.586+0000 7f7528ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.586+0000 7f7528ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:51.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.655+0000 7f09c122b640 10 monclient: get_monmap_and_config 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.655+0000 7f09c122b640 10 monclient: build_initial_monmap 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.655+0000 7f09c122b640 10 monclient: monmap: 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:51.656629+0000 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:51.656629+0000 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:51.657 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:51.658 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 1 Processor -- start 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 1 -- start start 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient: init 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09bc09ec50 0x7f09bc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient(hunting): picked mon.noname-a con 0x7f09bc09ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:51.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09bc1321a0 0x7f09bc068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient(hunting): picked mon.noname-c con 0x7f09bc1321a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09bc09f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc067420 con 0x7f09bc09f020 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc09fdc0 con 0x7f09bc09ec50 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc0a0510 con 0x7f09bc1321a0 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09c122b640 10 monclient(hunting): authenticate will time out at 1087.225799s 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09ba59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09bc1321a0 0x7f09bc068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09ba59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f09bc1321a0 0x7f09bc068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50212/0 (socket says 172.21.15.144:50212) 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.656+0000 7f09ba59c640 1 -- 172.21.15.144:0/3544100730 learned_addr learned my addr 172.21.15.144:0/3544100730 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): get_auth_request con 0x7f09bc1321a0 auth_method 0 2024-09-18T12:30:51.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 1 --2- 172.21.15.144:0/3544100730 >> v2:172.21.15.116:3300/0 conn(0x7f09bc09ec50 0x7f09bc131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 10 monclient(hunting): get_auth_request con 0x7f09bc09ec50 auth_method 0 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient(hunting): handle_auth_done global_id 5635 payload 274 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 1 -- 172.21.15.144:0/3544100730 >> v2:172.21.15.116:3300/0 conn(0x7f09bc09ec50 msgr2=0x7f09bc131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 1 --2- 172.21.15.144:0/3544100730 >> v2:172.21.15.116:3300/0 conn(0x7f09bc09ec50 0x7f09bc131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 1 -- 172.21.15.144:0/3544100730 >> v1:172.21.15.116:6789/0 conn(0x7f09bc09f020 legacy=0x7f09bc069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 1 monclient: found mon.noname-c 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:51.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.657+0000 7f09ba59c640 1 -- 172.21.15.144:0/3544100730 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0a0200 con 0x7f09bc1321a0 2024-09-18T12:30:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09ba59c640 1 -- 172.21.15.144:0/3544100730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 msgr2=0x7f09bc068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09ba59c640 1 --2- 172.21.15.144:0/3544100730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc068d00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0a2b00 tx=0x7f09b00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4f51b725be89f78 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 1 -- 172.21.15.144:0/3544100730 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0032440 con 0x7f09bc1321a0 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: dump: 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.670 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: _finish_auth 0 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 1 -- 172.21.15.144:0/3544100730 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b00325a0 con 0x7f09bc1321a0 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 5 monclient: authenticate success, global_id 5635 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 1 -- 172.21.15.144:0/3544100730 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0032980 con 0x7f09bc1321a0 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 4 set_mon_vals no callback set 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:51.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09b9d9b640 10 monclient: dump: 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 10 monclient: shutdown 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:51.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 -- 172.21.15.144:0/3544100730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 msgr2=0x7f09bc068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 --2- 172.21.15.144:0/3544100730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc068d00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0a2b00 tx=0x7f09b00091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 -- 172.21.15.144:0/3544100730 shutdown_connections 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 --2- 172.21.15.144:0/3544100730 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc068d00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 --2- 172.21.15.144:0/3544100730 >> v2:172.21.15.116:3300/0 conn(0x7f09bc09ec50 0x7f09bc131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 -- 172.21.15.144:0/3544100730 >> 172.21.15.144:0/3544100730 conn(0x7f09bc0a5d40 msgr2=0x7f09bc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 -- 172.21.15.144:0/3544100730 shutdown_connections 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.660+0000 7f09c122b640 1 -- 172.21.15.144:0/3544100730 wait complete. 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.661+0000 7f09c122b640 10 monclient: build_initial_monmap 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.661+0000 7f09c122b640 10 monclient: monmap: 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:51.673 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.661+0000 7f09c122b640 1 Processor -- start 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.661+0000 7f09c122b640 1 -- start start 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.661+0000 7f09c122b640 10 monclient: init 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ec50 0x7f09bc09a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): picked mon.noname-a con 0x7f09bc09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:51.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 0x7f09bc130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): picked mon.noname-c con 0x7f09bc09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): picked mon.noname-b con 0x7f09bc1321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc09dd10 con 0x7f09bc09ec50 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc11de70 con 0x7f09bc1321a0 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc09b480 con 0x7f09bc09f020 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09c122b640 10 monclient(hunting): authenticate will time out at 1087.231259s 2024-09-18T12:30:51.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc11b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50214/0 (socket says 172.21.15.144:50214) 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 -- 172.21.15.144:0/323925024 learned_addr learned my addr 172.21.15.144:0/323925024 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): get_auth_request con 0x7f09bc1321a0 auth_method 0 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 0x7f09bc130a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 10 monclient(hunting): get_auth_request con 0x7f09bc09f020 auth_method 0 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ec50 0x7f09bc09a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 10 monclient(hunting): get_auth_request con 0x7f09bc09ec50 auth_method 0 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 1 -- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 msgr2=0x7f09bc130a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 1 -- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 msgr2=0x7f09bc130a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:51.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 0x7f09bc130a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09ba59c640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 0x7f09bc130a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient(hunting): handle_auth_done global_id 5638 payload 824 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 -- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 msgr2=0x7f09bc130a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 0x7f09bc130a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 -- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ec50 msgr2=0x7f09bc09a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ec50 0x7f09bc09a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient: _finish_hunting 0 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 monclient: found mon.noname-b 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.662+0000 7f09bb59e640 1 -- 172.21.15.144:0/323925024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0009450 con 0x7f09bc1321a0 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.664+0000 7f09bb59e640 1 -- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 msgr2=0x7f09bc11b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:51.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.664+0000 7f09bb59e640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc11b760 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f09b400c8c0 tx=0x7f09b4002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.664+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b400aa00 con 0x7f09bc1321a0 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: dump: 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.679 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b400ab80 con 0x7f09bc1321a0 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 5 monclient: authenticate success, global_id 5638 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 10 monclient: _renew_subs 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc11db90 con 0x7f09bc1321a0 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4012420 con 0x7f09bc1321a0 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 10 monclient: dump: 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:51.680 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09b959a640 4 set_mon_vals no callback set 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 10 monclient: _renew_subs 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:51.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc11cca0 con 0x7f09bc1321a0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f09b4012890 con 0x7f09bc1321a0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.665+0000 7f09c122b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.666+0000 7f09c122b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.666+0000 7f09c122b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.666+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc11e030 con 0x7f09bc1321a0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.666+0000 7f09a2ffd640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038490 0x7f09a803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.666+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(127..127 src has 13..127) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f09b404c1f0 con 0x7f09bc1321a0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.666+0000 7f09bad9d640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038490 0x7f09a803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.666+0000 7f09bad9d640 10 monclient: get_auth_request con 0x7f09a8038490 auth_method 0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.668+0000 7f09bad9d640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038490 0x7f09a803a940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009d20 tx=0x7f09ac0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.670+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f09b4010510 con 0x7f09bc1321a0 2024-09-18T12:30:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.670+0000 7f09a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:51.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.670+0000 7f09a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:51.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f75330c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:51.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f75330c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:51.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f75330c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f752c098d20 con 0x7f7510008ec0 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 msgr2=0x7f7510009330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f752bfff640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 msgr2=0x7f7510009330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 0x7f7510009330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7518025000 tx=0x7f7518029000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 0x7f7510009330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7518025000 tx=0x7f7518029000 comp rx=0 tx=0).stop 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f7528ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f7528ff9640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 msgr2=0x7f7510009330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.803+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510008ec0 0x7f7510009330 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f7510005680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient(hunting): picked mon.b con 0x7f751c003ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510048f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient(hunting): picked mon.c con 0x7f751c005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510049480 0x7f751004b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient(hunting): picked mon.a con 0x7f7510049480 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751004c240 con 0x7f7510049480 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751004c380 con 0x7f751c003ac0 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751004c510 con 0x7f751c005330 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7528ff9640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510049480 0x7f751004b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510048f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): get_auth_request con 0x7f7510049480 auth_method 0 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 10 monclient(hunting): get_auth_request con 0x7f751c005330 auth_method 0 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f7510005680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 10 monclient(hunting): get_auth_request con 0x7f751c003ac0 auth_method 0 2024-09-18T12:30:51.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f752bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient(hunting): handle_auth_done global_id 84102 payload 293 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 msgr2=0x7f7510048f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510048f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 msgr2=0x7f7510005680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f7510005680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 monclient: found mon.a 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f751004c790 con 0x7f7510049480 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 1 -- 172.21.15.116:0/1199941859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f752003a040 con 0x7f7510049480 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 10 monclient: _finish_auth 0 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f7530e5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.804+0000 7f753165d640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510048f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7530e5c640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510049480 msgr2=0x7f751004b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7530e5c640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510049480 0x7f751004b860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f75200148a0 tx=0x7f7520034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:51.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520053430 con 0x7f7510049480 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7528ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7528ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7528ff9640 10 monclient: dump: 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:51.814 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:51.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752004e070 con 0x7f7510049480 2024-09-18T12:30:51.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f7528ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:51.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.806+0000 7f752affd640 4 set_mon_vals no callback set 2024-09-18T12:30:51.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.807+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7520050030 con 0x7f7510049480 2024-09-18T12:30:51.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.807+0000 7f7528ff9640 1 -- 172.21.15.116:0/1199941859 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f752006e5f0 con 0x7f7510049480 2024-09-18T12:30:51.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.807+0000 7f7528ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:51.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.807+0000 7f7528ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:51.816 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":62,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:51.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f751c004780 msgr2=0x7f7510043470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f751c004780 0x7f7510043470 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7520033000 tx=0x7f7520039000 comp rx=0 tx=0).stop 2024-09-18T12:30:51.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 10 monclient: shutdown 2024-09-18T12:30:51.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:51.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510049480 msgr2=0x7f751004b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:51.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510049480 0x7f751004b860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f75200148a0 tx=0x7f7520034000 comp rx=0 tx=0).stop 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 shutdown_connections 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7510049480 0x7f751004b860 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f751c005330 0x7f7510048f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f751c003ac0 0x7f7510005680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 --2- 172.21.15.116:0/1199941859 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f751c004780 0x7f7510043470 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 >> 172.21.15.116:0/1199941859 conn(0x7f752c0a1e30 msgr2=0x7f752c096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 shutdown_connections 2024-09-18T12:30:51.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:51.809+0000 7f75330c0640 1 -- 172.21.15.116:0/1199941859 wait complete. 2024-09-18T12:30:51.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.827+0000 7f09c122b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T12:30:51.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.827+0000 7f09c122b640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T12:30:51.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.827+0000 7f09c122b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:51.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:51.827+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7f09bc0008d0 con 0x7f09bc1321a0 2024-09-18T12:30:54.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:54.662+0000 7f09a1ffb640 10 monclient: tick 2024-09-18T12:30:54.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:54.662+0000 7f09a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:54.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:54.662+0000 7f09a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:54.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:54.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.924+0000 7f184d03f640 10 monclient: get_monmap_and_config 2024-09-18T12:30:54.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.924+0000 7f184d03f640 10 monclient: build_initial_monmap 2024-09-18T12:30:54.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.924+0000 7f184d03f640 10 monclient: monmap: 2024-09-18T12:30:54.925 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:54.925 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:54.925 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:54.925301+0000 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:54.925301+0000 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:54.926 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 1 Processor -- start 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 1 -- start start 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient: init 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient(hunting): picked mon.noname-b con 0x7f18480975c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient(hunting): picked mon.noname-c con 0x7f184812ea30 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f184812ee90 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:54.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184806e9e0 con 0x7f184812ee90 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1848063950 con 0x7f184812ea30 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184809d1c0 con 0x7f18480975c0 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.925+0000 7f184d03f640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f184d03f640 10 monclient(hunting): authenticate will time out at 1082.247006s 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f184759e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f184812ee90 0x7f1848131e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49966/0 (socket says 172.21.15.116:49966) 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57776/0 (socket says 172.21.15.116:57776) 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f184759e640 1 -- 172.21.15.116:0/3905816060 learned_addr learned my addr 172.21.15.116:0/3905816060 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 10 monclient(hunting): get_auth_request con 0x7f18480975c0 auth_method 0 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1848097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1848097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 10 monclient(hunting): get_auth_request con 0x7f184812ea30 auth_method 0 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 msgr2=0x7f1848131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 msgr2=0x7f1848131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.926+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:54.928+0000 7f184759e640 1 -- 172.21.15.116:0/3905816060 >> v1:172.21.15.116:6789/0 conn(0x7f184812ee90 legacy=0x7f1848131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 10 monclient(hunting): get_auth_request con 0x7f18480975c0 auth_method 0 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1848097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1848097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 10 monclient(hunting): get_auth_request con 0x7f184812ea30 auth_method 0 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 msgr2=0x7f1848131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 msgr2=0x7f1848131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.127+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:30:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 10 monclient(hunting): get_auth_request con 0x7f18480975c0 auth_method 0 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1848097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1848097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 10 monclient(hunting): get_auth_request con 0x7f184812ea30 auth_method 0 2024-09-18T12:30:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 msgr2=0x7f1848131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:30:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 msgr2=0x7f1848131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:30:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:55.528+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:30:55.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:55.663+0000 7f09a1ffb640 10 monclient: tick 2024-09-18T12:30:55.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:55.663+0000 7f09a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.158+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872302655 0 0) 0x7f184806e9e0 con 0x7f184812ee90 2024-09-18T12:30:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.158+0000 7f184659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.158+0000 7f184659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.158+0000 7f184659c640 10 monclient(hunting): my global_id is 84111 2024-09-18T12:30:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.158+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18340034e0 con 0x7f184812ee90 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170999819 0 0) 0x7f18340034e0 con 0x7f184812ee90 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1848097990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1848097990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 msgr2=0x7f1848131950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 --2- 172.21.15.116:0/3905816060 >> v2:172.21.15.144:3300/0 conn(0x7f184812ea30 0x7f1848131950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848097ed0 con 0x7f184812ee90 2024-09-18T12:30:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f1840001b20 con 0x7f184812ee90 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: dump: 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 >> v1:172.21.15.116:6789/0 conn(0x7f184812ee90 legacy=0x7f1848131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1834003ba0 0x7f1834003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): picked mon.b con 0x7f1834003ba0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1834004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): picked mon.c con 0x7f18480975c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1834004e00 0x7f183400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): picked mon.a con 0x7f1834004e00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848063950 con 0x7f1834004e00 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184809d1c0 con 0x7f1834003ba0 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1834005630 con 0x7f18480975c0 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184659c640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1834004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1834004e00 0x7f183400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): get_auth_request con 0x7f18480975c0 auth_method 0 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 10 monclient(hunting): get_auth_request con 0x7f1834004e00 auth_method 0 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1834003ba0 0x7f1834003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 10 monclient(hunting): get_auth_request con 0x7f1834003ba0 auth_method 0 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f184759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f1846d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 10 monclient(hunting): handle_auth_done global_id 84111 payload 293 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1834003ba0 msgr2=0x7f1834003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1834003ba0 0x7f1834003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.179+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1834004e00 msgr2=0x7f183400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1834004e00 0x7f183400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 1 monclient: found mon.c 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18340034e0 con 0x7f18480975c0 2024-09-18T12:30:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f183ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f184d03f640 5 monclient: authenticate success, global_id 84111 2024-09-18T12:30:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f184d03f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.180+0000 7f184759e640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1834004e00 0x7f183400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.182+0000 7f183ffff640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1834004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.182+0000 7f183ffff640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1834004780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1830006ea0 tx=0x7f1830008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=548cecc40de619fb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830032480 con 0x7f18480975c0 2024-09-18T12:30:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 10 monclient: dump: 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1830032620 con 0x7f18480975c0 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 1 -- 172.21.15.116:0/3905816060 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18300329e0 con 0x7f18480975c0 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 4 set_mon_vals no callback set 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184659c640 10 monclient: dump: 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.200 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 10 monclient: shutdown 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 1 -- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 msgr2=0x7f1834004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 1 --2- 172.21.15.116:0/3905816060 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f18480975c0 0x7f1834004780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1830006ea0 tx=0x7f1830008040 comp rx=0 tx=0).stop 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184759e640 1 -- 172.21.15.116:0/3905816060 reap_dead start 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 1 -- 172.21.15.116:0/3905816060 shutdown_connections 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.183+0000 7f184d03f640 1 -- 172.21.15.116:0/3905816060 >> 172.21.15.116:0/3905816060 conn(0x7f1848067b50 msgr2=0x7f18480a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.184+0000 7f184d03f640 1 -- 172.21.15.116:0/3905816060 shutdown_connections 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.184+0000 7f184d03f640 1 -- 172.21.15.116:0/3905816060 wait complete. 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.184+0000 7f184d03f640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.184+0000 7f184d03f640 10 monclient: monmap: 2024-09-18T12:30:56.201 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.184+0000 7f184d03f640 1 Processor -- start 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.184+0000 7f184d03f640 1 -- start start 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.184+0000 7f184d03f640 10 monclient: init 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 0x7f184809a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): picked mon.noname-a con 0x7f18480975c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184812b580 0x7f184812b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): picked mon.noname-c con 0x7f184812b580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184812f540 0x7f184812f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): picked mon.noname-b con 0x7f184812f540 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848131da0 con 0x7f18480975c0 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184812be90 con 0x7f184812f540 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1848131fa0 con 0x7f184812b580 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184d03f640 10 monclient(hunting): authenticate will time out at 1083.506457s 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 0x7f184809a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184812b580 0x7f184812b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 0x7f184809a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51076/0 (socket says 172.21.15.116:51076) 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184812b580 0x7f184812b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46828/0 (socket says 172.21.15.116:46828) 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 -- 172.21.15.116:0/570895157 learned_addr learned my addr 172.21.15.116:0/570895157 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): get_auth_request con 0x7f18480975c0 auth_method 0 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 10 monclient(hunting): get_auth_request con 0x7f184812b580 auth_method 0 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184812f540 0x7f184812f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 10 monclient(hunting): get_auth_request con 0x7f184812f540 auth_method 0 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f184759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient(hunting): handle_auth_done global_id 84114 payload 824 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 -- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184812b580 msgr2=0x7f184812b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184812b580 0x7f184812b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 -- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184812f540 msgr2=0x7f184812f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184812f540 0x7f184812f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 monclient: found mon.noname-a 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f1846d9d640 1 -- 172.21.15.116:0/570895157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848141ee0 con 0x7f18480975c0 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.185+0000 7f183ffff640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184812b580 0x7f184812b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f1846d9d640 1 -- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 msgr2=0x7f184809a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f1846d9d640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 0x7f184809a940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1838014a50 tx=0x7f1838008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 1 -- 172.21.15.116:0/570895157 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183800f730 con 0x7f18480975c0 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: dump: 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.207 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 1 -- 172.21.15.116:0/570895157 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1838012da0 con 0x7f18480975c0 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f184d03f640 5 monclient: authenticate success, global_id 84114 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f184d03f640 10 monclient: _renew_subs 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f184d03f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1848130e50 con 0x7f18480975c0 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 1 -- 172.21.15.116:0/570895157 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838016680 con 0x7f18480975c0 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f183effd640 10 monclient: dump: 2024-09-18T12:30:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f1845d9b640 4 set_mon_vals no callback set 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f184d03f640 10 monclient: _renew_subs 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f184d03f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.188+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1848141520 con 0x7f18480975c0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f183effd640 1 -- 172.21.15.116:0/570895157 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1838016a70 con 0x7f18480975c0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f183effd640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1828038490 0x7f182803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f183effd640 1 -- 172.21.15.116:0/570895157 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 13..127) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f183804ccb0 con 0x7f18480975c0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f184d03f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f184d03f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f184d03f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184804bf60 con 0x7f18480975c0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f183ffff640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1828038490 0x7f182803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.189+0000 7f183ffff640 10 monclient: get_auth_request con 0x7f1828038490 auth_method 0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.192+0000 7f183ffff640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1828038490 0x7f182803a940 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1830006fd0 tx=0x7f1830038000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.194+0000 7f183effd640 1 -- 172.21.15.116:0/570895157 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1838028580 con 0x7f18480975c0 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.194+0000 7f183effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.194+0000 7f183effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:56.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.293+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v128) v1 ==== 115+0+0 (secure 0 0 0) 0x7f09b4016070 con 0x7f09bc1321a0 2024-09-18T12:30:56.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.293+0000 7f09a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T12:30:56.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.293+0000 7f09a2ffd640 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-18T12:30:56.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.393+0000 7f09c122b640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T12:30:56.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.393+0000 7f09c122b640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T12:30:56.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.393+0000 7f09c122b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:56.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.393+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7f09bc0a13e0 con 0x7f09bc1321a0 2024-09-18T12:30:56.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.394+0000 7f09a2ffd640 1 -- 172.21.15.144:0/323925024 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v128) v1 ==== 122+0+0 (secure 0 0 0) 0x7f09b401ae20 con 0x7f09bc1321a0 2024-09-18T12:30:56.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.394+0000 7f09a2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T12:30:56.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.394+0000 7f09a2ffd640 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-18T12:30:56.396 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'fooset' already exists 2024-09-18T12:30:56.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.396+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038490 msgr2=0x7f09a803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.396+0000 7f09c122b640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038490 0x7f09a803a940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009d20 tx=0x7f09ac0090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.396+0000 7f09c122b640 10 monclient: shutdown 2024-09-18T12:30:56.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.396+0000 7f09c122b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.396+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 msgr2=0x7f09bc11b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.396+0000 7f09c122b640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc11b760 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f09b400c8c0 tx=0x7f09b4002640 comp rx=0 tx=0).stop 2024-09-18T12:30:56.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 shutdown_connections 2024-09-18T12:30:56.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09a8038490 0x7f09a803a940 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09bc1321a0 0x7f09bc11b760 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09bc09f020 0x7f09bc130a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 --2- 172.21.15.144:0/323925024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc09ec50 0x7f09bc09a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 >> 172.21.15.144:0/323925024 conn(0x7f09bc0a5d40 msgr2=0x7f09bc09e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 shutdown_connections 2024-09-18T12:30:56.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.397+0000 7f09c122b640 1 -- 172.21.15.144:0/323925024 wait complete. 2024-09-18T12:30:56.407 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create barset 8 2024-09-18T12:30:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.417+0000 7f184d03f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.417+0000 7f184d03f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.417+0000 7f184d03f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.417+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f18480980b0 con 0x7f18480975c0 2024-09-18T12:30:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.418+0000 7f183effd640 1 -- 172.21.15.116:0/570895157 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f1838029e40 con 0x7f18480975c0 2024-09-18T12:30:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.418+0000 7f183effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.418+0000 7f183effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:56.419 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1828038490 msgr2=0x7f182803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1828038490 0x7f182803a940 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1830006fd0 tx=0x7f1830038000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 10 monclient: shutdown 2024-09-18T12:30:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 msgr2=0x7f184809a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 0x7f184809a940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1838014a50 tx=0x7f1838008040 comp rx=0 tx=0).stop 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 shutdown_connections 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1828038490 0x7f182803a940 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184812f540 0x7f184812f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184812b580 0x7f184812b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 --2- 172.21.15.116:0/570895157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18480975c0 0x7f184809a940 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 >> 172.21.15.116:0/570895157 conn(0x7f1848067b50 msgr2=0x7f184812c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 shutdown_connections 2024-09-18T12:30:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.420+0000 7f184d03f640 1 -- 172.21.15.116:0/570895157 wait complete. 2024-09-18T12:30:56.431 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:30:56.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:30:56.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:30:56.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.481+0000 7f671a99d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:56.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.481+0000 7f671a99d640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.481+0000 7f671a99d640 10 monclient: monmap: 2024-09-18T12:30:56.482 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:56.481935+0000 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:56.481935+0000 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:56.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.481+0000 7f671a99d640 1 Processor -- start 2024-09-18T12:30:56.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.481+0000 7f671a99d640 1 -- start start 2024-09-18T12:30:56.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.481+0000 7f671a99d640 10 monclient: init 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6714131c90 0x7f6714132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient(hunting): picked mon.noname-b con 0x7f6714131c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f671409d210 0x7f67141325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient(hunting): picked mon.noname-c con 0x7f671409d210 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67140a0ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67140a1800 con 0x7f67140a0ab0 2024-09-18T12:30:56.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671409fea0 con 0x7f671409d210 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67140a1520 con 0x7f6714131c90 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f671a99d640 10 monclient(hunting): authenticate will time out at 1092.051389s 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f671409d210 0x7f67141325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f671409d210 0x7f67141325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50216/0 (socket says 172.21.15.144:50216) 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6718f3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f67140a0ab0 0x7f6714068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51056/0 (socket says 172.21.15.144:51056) 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 -- 172.21.15.144:0/917483827 learned_addr learned my addr 172.21.15.144:0/917483827 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): get_auth_request con 0x7f671409d210 auth_method 0 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 1 --2- 172.21.15.144:0/917483827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6714131c90 0x7f6714132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 10 monclient(hunting): get_auth_request con 0x7f6714131c90 auth_method 0 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient(hunting): handle_auth_done global_id 5644 payload 274 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 -- 172.21.15.144:0/917483827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6714131c90 msgr2=0x7f6714132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 --2- 172.21.15.144:0/917483827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6714131c90 0x7f6714132060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.490 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 -- 172.21.15.144:0/917483827 >> v1:172.21.15.144:6789/0 conn(0x7f67140a0ab0 legacy=0x7f6714068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 monclient: found mon.noname-c 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.482+0000 7f67137fe640 1 -- 172.21.15.144:0/917483827 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714069510 con 0x7f671409d210 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.485+0000 7f67137fe640 1 -- 172.21.15.144:0/917483827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f671409d210 msgr2=0x7f67141325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.485+0000 7f67137fe640 1 --2- 172.21.15.144:0/917483827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f671409d210 0x7f67141325a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6708009910 tx=0x7f67080091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53eb4fa2abc68926 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.485+0000 7f6712ffd640 1 -- 172.21.15.144:0/917483827 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670802ae90 con 0x7f671409d210 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.485+0000 7f6712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 10 monclient: dump: 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.491 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 1 -- 172.21.15.144:0/917483827 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6708032440 con 0x7f671409d210 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 5 monclient: authenticate success, global_id 5644 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 1 -- 172.21.15.144:0/917483827 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080327e0 con 0x7f671409d210 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 4 set_mon_vals no callback set 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f6712ffd640 10 monclient: dump: 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.493 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 10 monclient: shutdown 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 -- 172.21.15.144:0/917483827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f671409d210 msgr2=0x7f67141325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 --2- 172.21.15.144:0/917483827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f671409d210 0x7f67141325a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6708009910 tx=0x7f67080091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 -- 172.21.15.144:0/917483827 shutdown_connections 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 --2- 172.21.15.144:0/917483827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f671409d210 0x7f67141325a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 --2- 172.21.15.144:0/917483827 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6714131c90 0x7f6714132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 -- 172.21.15.144:0/917483827 >> 172.21.15.144:0/917483827 conn(0x7f6714071370 msgr2=0x7f67140a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 -- 172.21.15.144:0/917483827 shutdown_connections 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.486+0000 7f671a99d640 1 -- 172.21.15.144:0/917483827 wait complete. 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.487+0000 7f671a99d640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.487+0000 7f671a99d640 10 monclient: monmap: 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:56.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.487+0000 7f671a99d640 1 Processor -- start 2024-09-18T12:30:56.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.487+0000 7f671a99d640 1 -- start start 2024-09-18T12:30:56.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.487+0000 7f671a99d640 10 monclient: init 2024-09-18T12:30:56.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:56.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67140a0ab0 0x7f6714141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): picked mon.noname-c con 0x7f67140a0ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 0x7f6714141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): picked mon.noname-b con 0x7f6714131c90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714141bd0 0x7f671411e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): picked mon.noname-a con 0x7f6714141bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6714142000 con 0x7f6714141bd0 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6714120d30 con 0x7f6714131c90 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6714120ac0 con 0x7f67140a0ab0 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f671a99d640 10 monclient(hunting): authenticate will time out at 1092.057108s 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 0x7f6714141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 0x7f6714141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:50228/0 (socket says 172.21.15.144:50228) 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 -- 172.21.15.144:0/4050579228 learned_addr learned my addr 172.21.15.144:0/4050579228 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): get_auth_request con 0x7f6714131c90 auth_method 0 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67140a0ab0 0x7f6714141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 10 monclient(hunting): get_auth_request con 0x7f67140a0ab0 auth_method 0 2024-09-18T12:30:56.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6718f3a640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714141bd0 0x7f671411e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6718f3a640 10 monclient(hunting): get_auth_request con 0x7f6714141bd0 auth_method 0 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6718f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6718f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6718f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient(hunting): handle_auth_done global_id 5647 payload 824 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 -- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67140a0ab0 msgr2=0x7f6714141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67140a0ab0 0x7f6714141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 -- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714141bd0 msgr2=0x7f671411e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714141bd0 0x7f671411e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 monclient: found mon.noname-b 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f67137fe640 1 -- 172.21.15.144:0/4050579228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009450 con 0x7f6714131c90 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.488+0000 7f6718f3a640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714141bd0 0x7f671411e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f67137fe640 1 -- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 msgr2=0x7f6714141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f67137fe640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 0x7f6714141690 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6708009910 tx=0x7f6708032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708000c40 con 0x7f6714131c90 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: dump: 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.505 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670802ae90 con 0x7f6714131c90 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f671a99d640 5 monclient: authenticate success, global_id 5647 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670803c400 con 0x7f6714131c90 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f671a99d640 10 monclient: _renew_subs 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f671a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671409f080 con 0x7f6714131c90 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f66fbfff640 10 monclient: dump: 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.490+0000 7f67127fc640 4 set_mon_vals no callback set 2024-09-18T12:30:56.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f671a99d640 10 monclient: _renew_subs 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f671a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6714122e80 con 0x7f6714131c90 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f670803c770 con 0x7f6714131c90 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f671a99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f671a99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f671a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.491+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67140008d0 con 0x7f6714131c90 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.492+0000 7f66fbfff640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67000384c0 0x7f670003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.492+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(128..128 src has 13..128) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f670806c570 con 0x7f6714131c90 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.492+0000 7f6713fff640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67000384c0 0x7f670003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.492+0000 7f6713fff640 10 monclient: get_auth_request con 0x7f67000384c0 auth_method 0 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.494+0000 7f6713fff640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67000384c0 0x7f670003a970 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6704009d20 tx=0x7f67040090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.496+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f670803b3a0 con 0x7f6714131c90 2024-09-18T12:30:56.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.496+0000 7f66fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.496+0000 7f66fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.524+0000 7f0b7dedd640 10 monclient: get_monmap_and_config 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.524+0000 7f0b7dedd640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.524+0000 7f0b7dedd640 10 monclient: monmap: 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:56.525093+0000 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:56.525093+0000 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:56.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 1 Processor -- start 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 1 -- start start 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient: init 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b780974f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b7809a8a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b78134080 0x7f0b78136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient(hunting): picked mon.noname-a con 0x7f0b78134080 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7812b210 con 0x7f0b780974f0 2024-09-18T12:30:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780978c0 con 0x7f0b7809a8a0 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7809ca20 con 0x7f0b78134080 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b7dedd640 10 monclient(hunting): authenticate will time out at 1083.846678s 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b78134080 0x7f0b78136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b777fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0b780974f0 0x7f0b7812ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55488/0 (socket says 172.21.15.116:55488) 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b777fe640 1 -- 172.21.15.116:0/3053867212 learned_addr learned my addr 172.21.15.116:0/3053867212 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 10 monclient(hunting): get_auth_request con 0x7f0b78134080 auth_method 0 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.525+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 10 monclient(hunting): handle_auth_done global_id 84120 payload 274 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 1 -- 172.21.15.116:0/3053867212 >> v1:172.21.15.144:6789/0 conn(0x7f0b7809a8a0 legacy=0x7f0b7809ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 1 -- 172.21.15.116:0/3053867212 >> v1:172.21.15.116:6789/0 conn(0x7f0b780974f0 legacy=0x7f0b7812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 1 monclient: found mon.noname-a 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.526+0000 7f0b77fff640 1 -- 172.21.15.116:0/3053867212 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78098ad0 con 0x7f0b78134080 2024-09-18T12:30:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b77fff640 1 -- 172.21.15.116:0/3053867212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78134080 msgr2=0x7f0b78136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b77fff640 1 --2- 172.21.15.116:0/3053867212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78134080 0x7f0b78136460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009e60 tx=0x7f0b70009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac466cdbd617ea8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b767fc640 1 -- 172.21.15.116:0/3053867212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70007840 con 0x7f0b78134080 2024-09-18T12:30:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b767fc640 10 monclient: dump: 2024-09-18T12:30:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b767fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b767fc640 1 -- 172.21.15.116:0/3053867212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7002d050 con 0x7f0b78134080 2024-09-18T12:30:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b7dedd640 5 monclient: authenticate success, global_id 84120 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.528+0000 7f0b7dedd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b767fc640 1 -- 172.21.15.116:0/3053867212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70007bd0 con 0x7f0b78134080 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 4 set_mon_vals no callback set 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b767fc640 10 monclient: dump: 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 10 monclient: shutdown 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 1 -- 172.21.15.116:0/3053867212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78134080 msgr2=0x7f0b78136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 1 --2- 172.21.15.116:0/3053867212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78134080 0x7f0b78136460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009e60 tx=0x7f0b70009780 comp rx=0 tx=0).stop 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 1 -- 172.21.15.116:0/3053867212 shutdown_connections 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 1 --2- 172.21.15.116:0/3053867212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78134080 0x7f0b78136460 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 1 -- 172.21.15.116:0/3053867212 >> 172.21.15.116:0/3053867212 conn(0x7f0b780672b0 msgr2=0x7f0b780676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 1 -- 172.21.15.116:0/3053867212 shutdown_connections 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.529+0000 7f0b7dedd640 1 -- 172.21.15.116:0/3053867212 wait complete. 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 10 monclient: monmap: 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 1 Processor -- start 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 1 -- start start 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 10 monclient: init 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780974f0 0x7f0b7806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 10 monclient(hunting): picked mon.noname-a con 0x7f0b780974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b7809a8a0 0x7f0b7806d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.530+0000 7f0b7dedd640 10 monclient(hunting): picked mon.noname-b con 0x7f0b7809a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b78134080 0x7f0b7806ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 10 monclient(hunting): picked mon.noname-c con 0x7f0b78134080 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78098300 con 0x7f0b780974f0 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7809e490 con 0x7f0b7809a8a0 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7812cda0 con 0x7f0b78134080 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b7dedd640 10 monclient(hunting): authenticate will time out at 1083.852034s 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780974f0 0x7f0b7806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780974f0 0x7f0b7806d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51096/0 (socket says 172.21.15.116:51096) 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 -- 172.21.15.116:0/853177806 learned_addr learned my addr 172.21.15.116:0/853177806 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): get_auth_request con 0x7f0b780974f0 auth_method 0 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b78134080 0x7f0b7806ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 10 monclient(hunting): get_auth_request con 0x7f0b78134080 auth_method 0 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b7809a8a0 0x7f0b7806d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request con 0x7f0b7809a8a0 auth_method 0 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient(hunting): handle_auth_done global_id 84123 payload 824 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b78134080 msgr2=0x7f0b7806ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b78134080 0x7f0b7806ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b7809a8a0 msgr2=0x7f0b7806d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b7809a8a0 0x7f0b7806d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 monclient: found mon.noname-a 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.531+0000 7f0b777fe640 1 -- 172.21.15.116:0/853177806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b70009a20 con 0x7f0b780974f0 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b777fe640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780974f0 msgr2=0x7f0b7806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b777fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780974f0 0x7f0b7806d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0b6800c5e0 tx=0x7f0b68004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68005cb0 con 0x7f0b780974f0 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: dump: 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b68005e30 con 0x7f0b780974f0 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b7dedd640 5 monclient: authenticate success, global_id 84123 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68007340 con 0x7f0b780974f0 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b7dedd640 10 monclient: _renew_subs 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b7dedd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7806f110 con 0x7f0b780974f0 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b5f7fe640 10 monclient: dump: 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b75ffb640 4 set_mon_vals no callback set 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b7dedd640 10 monclient: _renew_subs 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b7dedd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.533+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7806e310 con 0x7f0b780974f0 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b7dedd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b7dedd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 0x7f0b78125980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b7dedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7812cc60 con 0x7f0b78133770 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0b6801b400 con 0x7f0b780974f0 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 0x7f0b78125980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 10 monclient: get_auth_request con 0x7f0b78133770 auth_method 0 2024-09-18T12:30:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b5f7fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b64038490 0x7f0b6403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.534+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(128..128 src has 13..128) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f0b6804ba50 con 0x7f0b780974f0 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.535+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_done global_id 84126 payload 824 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.535+0000 7f0b76ffd640 1 -- 172.21.15.116:0/853177806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b7804bf60 con 0x7f0b78133770 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.535+0000 7f0b77fff640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b64038490 0x7f0b6403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.535+0000 7f0b77fff640 10 monclient: get_auth_request con 0x7f0b64038490 auth_method 0 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b76ffd640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 msgr2=0x7f0b78125980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b76ffd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 0x7f0b78125980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00f780 tx=0x7f0b6c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c00d8b0 con 0x7f0b78133770 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 10 monclient: dump: 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b77fff640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b64038490 0x7f0b6403a940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0b780768c0 tx=0x7f0b70037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0b6c00da10 con 0x7f0b78133770 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 msgr2=0x7f0b78125980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 0x7f0b78125980 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00f780 tx=0x7f0b6c007e10 comp rx=0 tx=0).stop 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 msgr2=0x7f0b78125980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.537+0000 7f0b5f7fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78133770 0x7f0b78125980 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00f780 tx=0x7f0b6c007e10 comp rx=0 tx=0).stop 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b7dedd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b7dedd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 0x7f0b781262c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b7dedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78065f20 con 0x7f0b78125ec0 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 0x7f0b781262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient: get_auth_request con 0x7f0b78125ec0 auth_method 0 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_done global_id 84129 payload 824 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.543+0000 7f0b76ffd640 1 -- 172.21.15.116:0/853177806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0b7812c7e0 con 0x7f0b78125ec0 2024-09-18T12:30:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.545+0000 7f0b76ffd640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 msgr2=0x7f0b781262c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.545+0000 7f0b76ffd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 0x7f0b781262c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00f780 tx=0x7f0b6c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.545+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c014e20 con 0x7f0b78125ec0 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.545+0000 7f0b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.545+0000 7f0b5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.545+0000 7f0b5f7fe640 10 monclient: dump: 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f0b6c01d610 con 0x7f0b78125ec0 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b5f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b5f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 msgr2=0x7f0b781262c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b5f7fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 0x7f0b781262c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00f780 tx=0x7f0b6c019000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b5f7fe640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 msgr2=0x7f0b781262c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b5f7fe640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78125ec0 0x7f0b781262c0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00f780 tx=0x7f0b6c019000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.557 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 66, 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:56.558 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:56.559 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:56.560 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:56.561 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.562 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.566 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:56.567 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.568 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:56.569 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b64038490 msgr2=0x7f0b6403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b64038490 0x7f0b6403a940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0b780768c0 tx=0x7f0b70037000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 10 monclient: shutdown 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780974f0 msgr2=0x7f0b7806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 --2- 172.21.15.116:0/853177806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b780974f0 0x7f0b7806d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0b6800c5e0 tx=0x7f0b68004960 comp rx=0 tx=0).stop 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b77fff640 1 -- 172.21.15.116:0/853177806 reap_dead start 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b77fff640 1 -- 172.21.15.116:0/853177806 reap_dead start 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 shutdown_connections 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 >> 172.21.15.116:0/853177806 conn(0x7f0b780672b0 msgr2=0x7f0b78124630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 shutdown_connections 2024-09-18T12:30:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.546+0000 7f0b7dedd640 1 -- 172.21.15.116:0/853177806 wait complete. 2024-09-18T12:30:56.572 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:56.572 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:30:56.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:30:56.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.656+0000 7f671a99d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T12:30:56.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.656+0000 7f671a99d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T12:30:56.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.656+0000 7f671a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:56.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:56.656+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f67140697b0 con 0x7f6714131c90 2024-09-18T12:30:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.665+0000 7f2ee459d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.665+0000 7f2ee459d640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.665+0000 7f2ee459d640 10 monclient: monmap: 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:56.665790+0000 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:56.665790+0000 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:56.666 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.665+0000 7f2ee459d640 1 Processor -- start 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.665+0000 7f2ee459d640 1 -- start start 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.665+0000 7f2ee459d640 10 monclient: init 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2edc1318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient(hunting): picked mon.noname-b con 0x7f2edc09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2edc136c20 0x7f2edc139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient(hunting): picked mon.noname-c con 0x7f2edc136c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc132640 con 0x7f2edc1318f0 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc131d00 con 0x7f2edc136c20 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc09fbf0 con 0x7f2edc09ea50 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee459d640 10 monclient(hunting): authenticate will time out at 1083.987357s 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46840/0 (socket says 172.21.15.116:46840) 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 -- 172.21.15.116:0/1955815344 learned_addr learned my addr 172.21.15.116:0/1955815344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): get_auth_request con 0x7f2edc09ea50 auth_method 0 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee2b3a640 1 --2- 172.21.15.116:0/1955815344 >> v2:172.21.15.144:3300/0 conn(0x7f2edc136c20 0x7f2edc139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_done global_id 64463 payload 274 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 -- 172.21.15.116:0/1955815344 >> v2:172.21.15.144:3300/0 conn(0x7f2edc136c20 msgr2=0x7f2edc139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 --2- 172.21.15.116:0/1955815344 >> v2:172.21.15.144:3300/0 conn(0x7f2edc136c20 0x7f2edc139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 -- 172.21.15.116:0/1955815344 >> v1:172.21.15.116:6789/0 conn(0x7f2edc1318f0 legacy=0x7f2edc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 monclient: found mon.noname-b 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee1b38640 1 -- 172.21.15.116:0/1955815344 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc09fe30 con 0x7f2edc09ea50 2024-09-18T12:30:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee2b3a640 10 monclient: get_auth_request con 0x7f2edc136c20 auth_method 0 2024-09-18T12:30:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.666+0000 7f2ee2b3a640 1 --2- 172.21.15.116:0/1955815344 >> v2:172.21.15.144:3300/0 conn(0x7f2edc136c20 0x7f2edc139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1b38640 1 -- 172.21.15.116:0/1955815344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 msgr2=0x7f2edc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1b38640 1 --2- 172.21.15.116:0/1955815344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc09ee20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009910 tx=0x7f2ed80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b2030df5050dc95 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 1 -- 172.21.15.116:0/1955815344 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8032440 con 0x7f2edc09ea50 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: dump: 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 1 -- 172.21.15.116:0/1955815344 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed80325a0 con 0x7f2edc09ea50 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee459d640 5 monclient: authenticate success, global_id 64463 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee459d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 1 -- 172.21.15.116:0/1955815344 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8032940 con 0x7f2edc09ea50 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee459d640 4 set_mon_vals no callback set 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.669+0000 7f2ee1337640 10 monclient: dump: 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 10 monclient: shutdown 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 -- 172.21.15.116:0/1955815344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 msgr2=0x7f2edc09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 --2- 172.21.15.116:0/1955815344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc09ee20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009910 tx=0x7f2ed80091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 -- 172.21.15.116:0/1955815344 shutdown_connections 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 --2- 172.21.15.116:0/1955815344 >> v2:172.21.15.144:3300/0 conn(0x7f2edc136c20 0x7f2edc139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 --2- 172.21.15.116:0/1955815344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc09ee20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 -- 172.21.15.116:0/1955815344 >> 172.21.15.116:0/1955815344 conn(0x7f2edc067110 msgr2=0x7f2edc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 -- 172.21.15.116:0/1955815344 shutdown_connections 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 -- 172.21.15.116:0/1955815344 wait complete. 2024-09-18T12:30:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 10 monclient: monmap: 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.670+0000 7f2ee459d640 1 Processor -- start 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 1 -- start start 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient: init 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc12f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): picked mon.noname-c con 0x7f2edc09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2edc1318f0 0x7f2edc12f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): picked mon.noname-b con 0x7f2edc1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc136c20 0x7f2edc12fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): picked mon.noname-a con 0x7f2edc136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc135a30 con 0x7f2edc136c20 2024-09-18T12:30:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc133d50 con 0x7f2edc1318f0 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc09db00 con 0x7f2edc09ea50 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee459d640 10 monclient(hunting): authenticate will time out at 1083.992482s 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc12f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc12f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46846/0 (socket says 172.21.15.116:46846) 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 -- 172.21.15.116:0/3914926933 learned_addr learned my addr 172.21.15.116:0/3914926933 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): get_auth_request con 0x7f2edc09ea50 auth_method 0 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc136c20 0x7f2edc12fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2edc1318f0 0x7f2edc12f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 10 monclient(hunting): get_auth_request con 0x7f2edc1318f0 auth_method 0 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 10 monclient(hunting): get_auth_request con 0x7f2edc136c20 auth_method 0 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient(hunting): handle_auth_done global_id 64466 payload 824 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2edc1318f0 msgr2=0x7f2edc12f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2edc1318f0 0x7f2edc12f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc136c20 msgr2=0x7f2edc12fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc136c20 0x7f2edc12fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 monclient: found mon.noname-c 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.671+0000 7f2ee2339640 1 -- 172.21.15.116:0/3914926933 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed8009450 con 0x7f2edc09ea50 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee2339640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 msgr2=0x7f2edc12f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee2339640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc12f100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00c5e0 tx=0x7f2ecc007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc012840 con 0x7f2edc09ea50 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: dump: 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.688 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc0129c0 con 0x7f2edc09ea50 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee459d640 5 monclient: authenticate success, global_id 64466 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee459d640 10 monclient: _renew_subs 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc130a30 con 0x7f2edc09ea50 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc012d80 con 0x7f2edc09ea50 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ed27fc640 10 monclient: dump: 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.690 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee0b36640 4 set_mon_vals no callback set 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee459d640 10 monclient: _renew_subs 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.674+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc131600 con 0x7f2edc09ea50 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee459d640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 0x7f2edc12ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc065e00 con 0x7f2edc1300c0 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2ecc011590 con 0x7f2edc09ea50 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 0x7f2edc12ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient: get_auth_request con 0x7f2edc1300c0 auth_method 0 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ed27fc640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ec0038490 0x7f2ec003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(128..128 src has 13..128) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f2ecc04c1d0 con 0x7f2edc09ea50 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_done global_id 64469 payload 824 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee1b38640 1 -- 172.21.15.116:0/3914926933 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2edc133b00 con 0x7f2edc1300c0 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee2b3a640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ec0038490 0x7f2ec003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.675+0000 7f2ee2b3a640 10 monclient: get_auth_request con 0x7f2ec0038490 auth_method 0 2024-09-18T12:30:56.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.677+0000 7f2ee1b38640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 msgr2=0x7f2edc12ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.677+0000 7f2ee1b38640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 0x7f2edc12ab20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8034000 tx=0x7f2ed8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.677+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8031bc0 con 0x7f2edc1300c0 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.677+0000 7f2ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.677+0000 7f2ed27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.677+0000 7f2ed27fc640 10 monclient: dump: 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ee2b3a640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ec0038490 0x7f2ec003a940 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2edc133a40 tx=0x7f2ed4007860 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2ed8031d20 con 0x7f2edc1300c0 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ed27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ed27fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 msgr2=0x7f2edc12ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ed27fc640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 0x7f2edc12ab20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8034000 tx=0x7f2ed8032000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 msgr2=0x7f2edc12ab20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:56.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.678+0000 7f2ed27fc640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc1300c0 0x7f2edc12ab20 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8034000 tx=0x7f2ed8032000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.683+0000 7f2ee459d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.683+0000 7f2ee459d640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 0x7f2edc12b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.683+0000 7f2ee459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.683+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc09bef0 con 0x7f2edc12b060 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 0x7f2edc12b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient: get_auth_request con 0x7f2edc12b060 auth_method 0 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 10 monclient(hunting): handle_auth_done global_id 64472 payload 824 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.684+0000 7f2ee1b38640 1 -- 172.21.15.116:0/3914926933 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2edc04bf60 con 0x7f2edc12b060 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ee1b38640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 msgr2=0x7f2edc12b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ee1b38640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 0x7f2edc12b460 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009910 tx=0x7f2ed803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed803ce90 con 0x7f2edc12b060 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 10 monclient: dump: 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f2ed803b3d0 con 0x7f2edc12b060 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 msgr2=0x7f2edc12b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 0x7f2edc12b460 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009910 tx=0x7f2ed803d000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 msgr2=0x7f2edc12b460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.686+0000 7f2ed27fc640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc12b060 0x7f2edc12b460 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009910 tx=0x7f2ed803d000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:56.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 66, 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:56.700 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.701 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:56.702 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:56.703 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.704 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:56.705 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:56.706 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:56.709 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.710 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.711 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.712 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:56.712 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ec0038490 msgr2=0x7f2ec003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ec0038490 0x7f2ec003a940 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2edc133a40 tx=0x7f2ed4007860 comp rx=0 tx=0).stop 2024-09-18T12:30:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 10 monclient: shutdown 2024-09-18T12:30:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 msgr2=0x7f2edc12f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 --2- 172.21.15.116:0/3914926933 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2edc09ea50 0x7f2edc12f100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00c5e0 tx=0x7f2ecc007ed0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee2b3a640 1 -- 172.21.15.116:0/3914926933 reap_dead start 2024-09-18T12:30:56.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee2b3a640 1 -- 172.21.15.116:0/3914926933 reap_dead start 2024-09-18T12:30:56.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 shutdown_connections 2024-09-18T12:30:56.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 >> 172.21.15.116:0/3914926933 conn(0x7f2edc067110 msgr2=0x7f2edc12cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 shutdown_connections 2024-09-18T12:30:56.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.687+0000 7f2ee459d640 1 -- 172.21.15.116:0/3914926933 wait complete. 2024-09-18T12:30:56.714 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:56.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:30:56.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.807+0000 7f3506e20640 10 monclient: get_monmap_and_config 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.807+0000 7f3506e20640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.807+0000 7f3506e20640 10 monclient: monmap: 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:56.807962+0000 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:56.807962+0000 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:56.809 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.807+0000 7f3506e20640 1 Processor -- start 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 1 -- start start 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient: init 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35000a57e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3500131e90 0x7f350009f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient(hunting): picked mon.noname-c con 0x7f3500131e90 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 0x7f3500138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient(hunting): picked mon.noname-b con 0x7f3500136bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350009fbd0 con 0x7f35000a57e0 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35000a13f0 con 0x7f3500131e90 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35000a5bb0 con 0x7f3500136bd0 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f3506e20640 10 monclient(hunting): authenticate will time out at 1084.129515s 2024-09-18T12:30:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 0x7f3500138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 0x7f3500138fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46870/0 (socket says 172.21.15.116:46870) 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 -- 172.21.15.116:0/58931258 learned_addr learned my addr 172.21.15.116:0/58931258 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): get_auth_request con 0x7f3500136bd0 auth_method 0 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient(hunting): handle_auth_done global_id 64475 payload 274 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f34fffff640 1 --2- 172.21.15.116:0/58931258 >> v2:172.21.15.144:3300/0 conn(0x7f3500131e90 0x7f350009f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 -- 172.21.15.116:0/58931258 >> v2:172.21.15.144:3300/0 conn(0x7f3500131e90 msgr2=0x7f350009f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 --2- 172.21.15.116:0/58931258 >> v2:172.21.15.144:3300/0 conn(0x7f3500131e90 0x7f350009f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 -- 172.21.15.116:0/58931258 >> v1:172.21.15.144:6789/0 conn(0x7f35000a57e0 legacy=0x7f350009e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 monclient: found mon.noname-b 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f35053bd640 1 -- 172.21.15.116:0/58931258 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35000a1110 con 0x7f3500136bd0 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f34fffff640 10 monclient: get_auth_request con 0x7f3500131e90 auth_method 0 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.808+0000 7f34fffff640 1 --2- 172.21.15.116:0/58931258 >> v2:172.21.15.144:3300/0 conn(0x7f3500131e90 0x7f350009f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f35053bd640 1 -- 172.21.15.116:0/58931258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 msgr2=0x7f3500138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f35053bd640 1 --2- 172.21.15.116:0/58931258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 0x7f3500138fb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009f20 tx=0x7f34f8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc71e49bc0eab795 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 1 -- 172.21.15.116:0/58931258 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8007610 con 0x7f3500136bd0 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: dump: 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 1 -- 172.21.15.116:0/58931258 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f802d050 con 0x7f3500136bd0 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f3506e20640 5 monclient: authenticate success, global_id 64475 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f3506e20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f3506e20640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 1 -- 172.21.15.116:0/58931258 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80079a0 con 0x7f3500136bd0 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f3506e20640 4 set_mon_vals no callback set 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.811+0000 7f34ff7fe640 10 monclient: dump: 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 10 monclient: shutdown 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 -- 172.21.15.116:0/58931258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 msgr2=0x7f3500138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 --2- 172.21.15.116:0/58931258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 0x7f3500138fb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009f20 tx=0x7f34f8009840 comp rx=0 tx=0).stop 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 -- 172.21.15.116:0/58931258 shutdown_connections 2024-09-18T12:30:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 --2- 172.21.15.116:0/58931258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3500136bd0 0x7f3500138fb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 --2- 172.21.15.116:0/58931258 >> v2:172.21.15.144:3300/0 conn(0x7f3500131e90 0x7f350009f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 -- 172.21.15.116:0/58931258 >> 172.21.15.116:0/58931258 conn(0x7f3500067760 msgr2=0x7f35000a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 -- 172.21.15.116:0/58931258 shutdown_connections 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 1 -- 172.21.15.116:0/58931258 wait complete. 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.812+0000 7f3506e20640 10 monclient: monmap: 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 Processor -- start 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 -- start start 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient: init 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35000a57e0 0x7f3500097150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): picked mon.noname-c con 0x7f35000a57e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500131e90 0x7f3500097690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): picked mon.noname-b con 0x7f3500131e90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3500136bd0 0x7f3500097bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): picked mon.noname-a con 0x7f3500136bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350009be50 con 0x7f3500136bd0 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35000a4480 con 0x7f3500131e90 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350009b8f0 con 0x7f35000a57e0 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3506e20640 10 monclient(hunting): authenticate will time out at 1084.134917s 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.813+0000 7f3504bbc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35000a57e0 0x7f3500097150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3500136bd0 0x7f3500097bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35000a57e0 0x7f3500097150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46876/0 (socket says 172.21.15.116:46876) 2024-09-18T12:30:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 -- 172.21.15.116:0/3505542455 learned_addr learned my addr 172.21.15.116:0/3505542455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): get_auth_request con 0x7f35000a57e0 auth_method 0 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 10 monclient(hunting): get_auth_request con 0x7f3500136bd0 auth_method 0 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f34fffff640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500131e90 0x7f3500097690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f35053bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient(hunting): handle_auth_done global_id 64478 payload 824 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500131e90 msgr2=0x7f3500097690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500131e90 0x7f3500097690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3500136bd0 msgr2=0x7f3500097bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3500136bd0 0x7f3500097bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 monclient: found mon.noname-c 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f3504bbc640 1 -- 172.21.15.116:0/3505542455 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8009ae0 con 0x7f35000a57e0 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f34fffff640 10 monclient: get_auth_request con 0x7f3500131e90 auth_method 0 2024-09-18T12:30:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.814+0000 7f34fffff640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500131e90 0x7f3500097690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3504bbc640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35000a57e0 msgr2=0x7f3500097150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3504bbc640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35000a57e0 0x7f3500097150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f34f400b4f0 tx=0x7f34f40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f400f720 con 0x7f35000a57e0 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: dump: 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3506e20640 5 monclient: authenticate success, global_id 64478 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3506e20640 10 monclient: _renew_subs 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f400f8a0 con 0x7f35000a57e0 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3506e20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35000991b0 con 0x7f35000a57e0 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f400fc60 con 0x7f35000a57e0 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34fcff9640 10 monclient: dump: 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.833 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f34feffd640 4 set_mon_vals no callback set 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3506e20640 10 monclient: _renew_subs 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3506e20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.816+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3500098ca0 con 0x7f35000a57e0 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f3506e20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f34f4011610 con 0x7f35000a57e0 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f3506e20640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 0x7f3500132770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f3506e20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3500065e00 con 0x7f3500098110 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fcff9640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34e40384e0 0x7f34e403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(128..128 src has 13..128) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f34f404bb00 con 0x7f35000a57e0 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fffff640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 0x7f3500132770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fffff640 10 monclient: get_auth_request con 0x7f3500098110 auth_method 0 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.817+0000 7f34fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.818+0000 7f35053bd640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34e40384e0 0x7f34e403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.818+0000 7f35053bd640 10 monclient: get_auth_request con 0x7f34e40384e0 auth_method 0 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.818+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.818+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.818+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.818+0000 7f34fffff640 10 monclient(hunting): handle_auth_done global_id 5656 payload 824 2024-09-18T12:30:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.818+0000 7f34fffff640 1 -- 172.21.15.116:0/3505542455 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f350004bf60 con 0x7f3500098110 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fffff640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 msgr2=0x7f3500132770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fffff640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 0x7f3500132770 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00d7a0 tx=0x7f34ec008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f35053bd640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34e40384e0 0x7f34e403a990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3500069e50 tx=0x7f34f8036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00de10 con 0x7f3500098110 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 10 monclient: dump: 2024-09-18T12:30:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f34ec00c040 con 0x7f3500098110 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 msgr2=0x7f3500132770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 0x7f3500132770 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00d7a0 tx=0x7f34ec008040 comp rx=0 tx=0).stop 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 msgr2=0x7f3500132770 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.820+0000 7f34fcff9640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500098110 0x7f3500132770 secure :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00d7a0 tx=0x7f34ec008040 comp rx=0 tx=0).stop 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f3506e20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f3506e20640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 0x7f35001330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f3506e20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35000984e0 con 0x7f3500132cb0 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 0x7f35001330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 10 monclient: get_auth_request con 0x7f3500132cb0 auth_method 0 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.826+0000 7f34fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.827+0000 7f34fffff640 10 monclient(hunting): handle_auth_done global_id 5659 payload 824 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.827+0000 7f34fffff640 1 -- 172.21.15.116:0/3505542455 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f350009aa90 con 0x7f3500132cb0 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fffff640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 msgr2=0x7f35001330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fffff640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 0x7f35001330b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00b630 tx=0x7f34ec019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec020070 con 0x7f3500132cb0 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 10 monclient: dump: 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f34ec027480 con 0x7f3500132cb0 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 msgr2=0x7f35001330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 0x7f35001330b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00b630 tx=0x7f34ec019000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 msgr2=0x7f35001330b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.829+0000 7f34fcff9640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3500132cb0 0x7f35001330b0 secure :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00b630 tx=0x7f34ec019000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 66, 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:56.843 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.844 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.845 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:56.846 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.848 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.849 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:56.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.854 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:56.855 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34e40384e0 msgr2=0x7f34e403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f34e40384e0 0x7f34e403a990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3500069e50 tx=0x7f34f8036000 comp rx=0 tx=0).stop 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 10 monclient: shutdown 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35000a57e0 msgr2=0x7f3500097150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 --2- 172.21.15.116:0/3505542455 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35000a57e0 0x7f3500097150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f34f400b4f0 tx=0x7f34f40090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f35053bd640 1 -- 172.21.15.116:0/3505542455 reap_dead start 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f35053bd640 1 -- 172.21.15.116:0/3505542455 reap_dead start 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 shutdown_connections 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 >> 172.21.15.116:0/3505542455 conn(0x7f3500067760 msgr2=0x7f35000a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 shutdown_connections 2024-09-18T12:30:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.830+0000 7f3506e20640 1 -- 172.21.15.116:0/3505542455 wait complete. 2024-09-18T12:30:56.858 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:56.858 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:30:56.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:30:56.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.949+0000 7f6640c9f640 10 monclient: get_monmap_and_config 2024-09-18T12:30:56.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.949+0000 7f6640c9f640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.949+0000 7f6640c9f640 10 monclient: monmap: 2024-09-18T12:30:56.950 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.950 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.950 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:56.950196+0000 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:56.950196+0000 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:56.951 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 Processor -- start 2024-09-18T12:30:56.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 -- start start 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient: init 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f663c131ab0 0x7f663c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f663c131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c09e910 0x7f663c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f663c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f663c09f220 0x7f663c068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f663c09f220 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663c09faf0 con 0x7f663c09f220 2024-09-18T12:30:56.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663c09fd30 con 0x7f663c131ab0 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0a1110 con 0x7f663c09e910 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6640c9f640 10 monclient(hunting): authenticate will time out at 1084.271761s 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f663ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f663c09f220 0x7f663c068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6639d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c09e910 0x7f663c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f663ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f663c09f220 0x7f663c068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51144/0 (socket says 172.21.15.116:51144) 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f663ad9d640 1 -- 172.21.15.116:0/1960430378 learned_addr learned my addr 172.21.15.116:0/1960430378 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f663ad9d640 10 monclient(hunting): get_auth_request con 0x7f663c09f220 auth_method 0 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f663ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f663ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f663ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6639d9b640 10 monclient(hunting): get_auth_request con 0x7f663c09e910 auth_method 0 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6639d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6639d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.950+0000 7f6639d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f6639d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f6639d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f6639d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663a59c640 1 --2- 172.21.15.116:0/1960430378 >> v2:172.21.15.144:3300/0 conn(0x7f663c131ab0 0x7f663c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663a59c640 10 monclient(hunting): get_auth_request con 0x7f663c131ab0 auth_method 0 2024-09-18T12:30:56.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 10 monclient(hunting): handle_auth_done global_id 84138 payload 274 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 1 -- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c09e910 msgr2=0x7f663c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 1 --2- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c09e910 0x7f663c09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 1 -- 172.21.15.116:0/1960430378 >> v2:172.21.15.144:3300/0 conn(0x7f663c131ab0 msgr2=0x7f663c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 1 --2- 172.21.15.116:0/1960430378 >> v2:172.21.15.144:3300/0 conn(0x7f663c131ab0 0x7f663c131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 1 monclient: found mon.noname-a 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:56.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663ad9d640 1 -- 172.21.15.116:0/1960430378 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0a13f0 con 0x7f663c09f220 2024-09-18T12:30:56.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f6639d9b640 1 --2- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c09e910 0x7f663c09ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:56.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.951+0000 7f663a59c640 1 --2- 172.21.15.116:0/1960430378 >> v2:172.21.15.144:3300/0 conn(0x7f663c131ab0 0x7f663c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.953+0000 7f663ad9d640 1 -- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09f220 msgr2=0x7f663c068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.953+0000 7f663ad9d640 1 --2- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09f220 0x7f663c068ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f663c076760 tx=0x7f6634009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7635ccdb33634b73 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 1 -- 172.21.15.116:0/1960430378 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634007610 con 0x7f663c09f220 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: dump: 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.964 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 1 -- 172.21.15.116:0/1960430378 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663402d050 con 0x7f663c09f220 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 5 monclient: authenticate success, global_id 84138 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 4 set_mon_vals no callback set 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 1 -- 172.21.15.116:0/1960430378 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66340079e0 con 0x7f663c09f220 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f663959a640 10 monclient: dump: 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 10 monclient: shutdown 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 -- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09f220 msgr2=0x7f663c068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 --2- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09f220 0x7f663c068ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f663c076760 tx=0x7f6634009840 comp rx=0 tx=0).stop 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 -- 172.21.15.116:0/1960430378 shutdown_connections 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 --2- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09f220 0x7f663c068ce0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 --2- 172.21.15.116:0/1960430378 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c09e910 0x7f663c09ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 --2- 172.21.15.116:0/1960430378 >> v2:172.21.15.144:3300/0 conn(0x7f663c131ab0 0x7f663c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 -- 172.21.15.116:0/1960430378 >> 172.21.15.116:0/1960430378 conn(0x7f663c067d30 msgr2=0x7f663c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 -- 172.21.15.116:0/1960430378 shutdown_connections 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.954+0000 7f6640c9f640 1 -- 172.21.15.116:0/1960430378 wait complete. 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient: build_initial_monmap 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient: monmap: 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 Processor -- start 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 -- start start 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient: init 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09e910 0x7f663c09a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f663c09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f663c09f220 0x7f663c130860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f663c09f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c131ab0 0x7f663c11b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f663c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c131600 con 0x7f663c09e910 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c11dc60 con 0x7f663c09f220 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c11ca90 con 0x7f663c131ab0 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.955+0000 7f6640c9f640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f6640c9f640 10 monclient(hunting): authenticate will time out at 1084.277010s 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09e910 0x7f663c09a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09e910 0x7f663c09a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51146/0 (socket says 172.21.15.116:51146) 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c131ab0 0x7f663c11b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 1 -- 172.21.15.116:0/3932356675 learned_addr learned my addr 172.21.15.116:0/3932356675 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 10 monclient(hunting): get_auth_request con 0x7f663c09e910 auth_method 0 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): get_auth_request con 0x7f663c131ab0 auth_method 0 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f6639d9b640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f663c09f220 0x7f663c130860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f6639d9b640 10 monclient(hunting): get_auth_request con 0x7f663c09f220 auth_method 0 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f6639d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f6639d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f6639d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient(hunting): handle_auth_done global_id 64484 payload 824 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 1 -- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f663c09f220 msgr2=0x7f663c130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f663c09f220 0x7f663c130860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 1 -- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09e910 msgr2=0x7f663c09a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09e910 0x7f663c09a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 1 monclient: found mon.noname-c 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663ad9d640 1 -- 172.21.15.116:0/3932356675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c009390 con 0x7f663c131ab0 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f663a59c640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09e910 0x7f663c09a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.956+0000 7f6639d9b640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f663c09f220 0x7f663c130860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f663ad9d640 1 -- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c131ab0 msgr2=0x7f663c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f663ad9d640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c131ab0 0x7f663c11b6b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6634030000 tx=0x7f6634036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 1 -- 172.21.15.116:0/3932356675 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634007170 con 0x7f663c131ab0 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: dump: 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 1 -- 172.21.15.116:0/3932356675 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6634032ce0 con 0x7f663c131ab0 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6640c9f640 5 monclient: authenticate success, global_id 64484 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 1 -- 172.21.15.116:0/3932356675 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663403b660 con 0x7f663c131ab0 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f66327fc640 10 monclient: dump: 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6640c9f640 10 monclient: _renew_subs 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6640c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6634009ae0 con 0x7f663c131ab0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6638d99640 4 set_mon_vals no callback set 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6640c9f640 10 monclient: _renew_subs 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6640c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.958+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c11d980 con 0x7f663c131ab0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.959+0000 7f66327fc640 1 -- 172.21.15.116:0/3932356675 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6634038050 con 0x7f663c131ab0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.959+0000 7f6640c9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.959+0000 7f6640c9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.959+0000 7f6640c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.959+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c04bf60 con 0x7f663c131ab0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.959+0000 7f66327fc640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f661c038490 0x7f661c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.959+0000 7f66327fc640 1 -- 172.21.15.116:0/3932356675 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(128..128 src has 13..128) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f663406c300 con 0x7f663c131ab0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.960+0000 7f663a59c640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f661c038490 0x7f661c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.960+0000 7f663a59c640 10 monclient: get_auth_request con 0x7f661c038490 auth_method 0 2024-09-18T12:30:56.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.962+0000 7f663a59c640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f661c038490 0x7f661c03a940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6624010370 tx=0x7f6624008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:56.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.964+0000 7f66327fc640 1 -- 172.21.15.116:0/3932356675 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f663403ba50 con 0x7f663c131ab0 2024-09-18T12:30:56.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.964+0000 7f66327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:56.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:56.964+0000 7f66327fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.177+0000 7f6640c9f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:30:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.177+0000 7f6640c9f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:30:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.177+0000 7f6640c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.177+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f663c0008d0 con 0x7f663c131ab0 2024-09-18T12:30:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f66327fc640 1 -- 172.21.15.116:0/3932356675 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f663402f300 con 0x7f663c131ab0 2024-09-18T12:30:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f66327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:30:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f66327fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:57.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.180 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=8,logm=63} crc {auth=1338667438,config=3140514009,health=2301855438,kv=3604100458,logm=250466073}) 2024-09-18T12:30:57.180 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=10,kv=8,logm=65} crc {auth=3626649433,config=3140514009,health=1210771477,kv=3080278282,logm=3366212424}) 2024-09-18T12:30:57.180 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.180 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=8,logm=63} crc {auth=1338667438,config=3140514009,health=2301855438,kv=3604100458,logm=250466073}) 2024-09-18T12:30:57.180 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.178+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=2308288659,config=3140514009,health=2301855438,kv=2187923472,logm=2451263190}) 2024-09-18T12:30:57.181 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.179+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.181 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.179+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602616771}) 2024-09-18T12:30:57.181 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.179+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1129721175}) 2024-09-18T12:30:57.181 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.179+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.181 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.179+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602616771}) 2024-09-18T12:30:57.182 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.179+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=235313638}) 2024-09-18T12:30:57.182 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.180+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.182 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.180+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490866250}) 2024-09-18T12:30:57.182 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.180+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1991501373}) 2024-09-18T12:30:57.182 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.180+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.182 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.180+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490866250}) 2024-09-18T12:30:57.182 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.180+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503675590}) 2024-09-18T12:30:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f661c038490 msgr2=0x7f661c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f661c038490 0x7f661c03a940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6624010370 tx=0x7f6624008040 comp rx=0 tx=0).stop 2024-09-18T12:30:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 10 monclient: shutdown 2024-09-18T12:30:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c131ab0 msgr2=0x7f663c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c131ab0 0x7f663c11b6b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6634030000 tx=0x7f6634036000 comp rx=0 tx=0).stop 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 shutdown_connections 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f661c038490 0x7f661c03a940 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f663c131ab0 0x7f663c11b6b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f663c09f220 0x7f663c130860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 --2- 172.21.15.116:0/3932356675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f663c09e910 0x7f663c09a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 >> 172.21.15.116:0/3932356675 conn(0x7f663c067d30 msgr2=0x7f663c1348c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 shutdown_connections 2024-09-18T12:30:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f6640c9f640 1 -- 172.21.15.116:0/3932356675 wait complete. 2024-09-18T12:30:57.190 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105785544}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1732490681}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105785544}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.181+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3315124627}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.182+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.182+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454255253}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.182+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1597499173}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.182+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.182+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454255253}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.182+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708212293}) 2024-09-18T12:30:57.191 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.183+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.183+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835540834}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.183+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727729963}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.183+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.183+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835540834}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.183+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3685292519}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511791894}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129093871}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511791894}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214965923}) 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.192 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433521987}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=510144615}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.184+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433521987}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298686866}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168091920}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510171047}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168091920}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.185+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4153881111}) 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.186+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.193 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.186+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053632681}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.186+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3415846510}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.186+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.186+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053632681}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.186+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3842715173}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226638473}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478740416}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226638473}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3644988880}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354287500}) 2024-09-18T12:30:57.194 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=115030179}) 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354287500}) 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.187+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907930790}) 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.188+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.188+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216045499}) 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.188+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=322721807}) 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.188+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.188+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216045499}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.188+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3527248662}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.189+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.189+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952834819}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.189+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2619201476}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.189+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.189+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952834819}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.189+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439969302}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321659019}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181232293}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321659019}) 2024-09-18T12:30:57.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2572866874}) 2024-09-18T12:30:57.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316291477}) 2024-09-18T12:30:57.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1137514021}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316291477}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.190+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259674550}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.191+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.191+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958661610}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.191+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573466839}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.191+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.191+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958661610}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.191+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639304020}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.192+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.192+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541938745}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.192+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1539683254}) 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.192+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.192+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541938745}) 2024-09-18T12:30:57.200 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.192+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1706825758}) 2024-09-18T12:30:57.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:30:57.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.193+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.193+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2838004115}) 2024-09-18T12:30:57.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.193+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3814573500}) 2024-09-18T12:30:57.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.193+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.193+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2838004115}) 2024-09-18T12:30:57.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.193+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567554679}) 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.194+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.194+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=1265116475,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4154487906}) 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.194+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=85,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=2790919448,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=12609547}) 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.194+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.194+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=1265116475,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4154487906}) 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.194+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=3} crc {logm=206241370,mdsmap=1531151339,mgr=3426601819,mgr_metadata=3239289319,mgrstat=12609547}) 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.195+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.195+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mgrstat=2426558143,mon_config_key=1355063917,monmap=148614106,osd_metadata=2372734516,osd_pg_creating=2918661036,osd_snap=2033388227,osdmap=4162958290}) 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.195+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=78} crc {mgrstat=440178278,mon_config_key=3655569570,monmap=148614106,osd_metadata=4175318930,osd_pg_creating=2918661036,osd_snap=3823785925,osdmap=1074225007}) 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.195+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.195+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=77} crc {mgrstat=2426558143,mon_config_key=1355063917,monmap=148614106,osd_metadata=2372734516,osd_pg_creating=2918661036,osd_snap=2033388227,osdmap=4162958290}) 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.195+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=75} crc {mgrstat=2426558143,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2918661036,osd_snap=2033388227,osdmap=1677301184}) 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.196+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.196+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1506042211}) 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.196+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3828592673}) 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.196+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.196+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1506042211}) 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.196+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2426871277}) 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.199+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.199+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=3} crc {osdmap=3073728680}) 2024-09-18T12:30:57.205 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:30:57.199+0000 7f46be3f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=2} crc {osdmap=2064384180}) 2024-09-18T12:30:57.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.305+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v129) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6708030e20 con 0x7f6714131c90 2024-09-18T12:30:57.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.305+0000 7f66fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T12:30:57.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.305+0000 7f66fbfff640 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-18T12:30:57.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.402+0000 7f671a99d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T12:30:57.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.402+0000 7f671a99d640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T12:30:57.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.402+0000 7f671a99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:57.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.402+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f67140697b0 con 0x7f6714131c90 2024-09-18T12:30:57.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.403+0000 7f66fbfff640 1 -- 172.21.15.144:0/4050579228 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v129) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6708037030 con 0x7f6714131c90 2024-09-18T12:30:57.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.403+0000 7f66fbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T12:30:57.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.403+0000 7f66fbfff640 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-18T12:30:57.404 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'barset' already exists 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67000384c0 msgr2=0x7f670003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67000384c0 0x7f670003a970 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6704009d20 tx=0x7f67040090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 10 monclient: shutdown 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 msgr2=0x7f6714141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 0x7f6714141690 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6708009910 tx=0x7f6708032000 comp rx=0 tx=0).stop 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 shutdown_connections 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67000384c0 0x7f670003a970 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714141bd0 0x7f671411e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131c90 0x7f6714141690 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 --2- 172.21.15.144:0/4050579228 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67140a0ab0 0x7f6714141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.405+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 >> 172.21.15.144:0/4050579228 conn(0x7f6714071370 msgr2=0x7f67140a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:57.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.406+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 shutdown_connections 2024-09-18T12:30:57.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.406+0000 7f671a99d640 1 -- 172.21.15.144:0/4050579228 wait complete. 2024-09-18T12:30:57.416 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set barset size 3 2024-09-18T12:30:57.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.490+0000 7fcc3d59d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:57.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.490+0000 7fcc3d59d640 10 monclient: build_initial_monmap 2024-09-18T12:30:57.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.490+0000 7fcc3d59d640 10 monclient: monmap: 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:57.491357+0000 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:57.491357+0000 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:57.492 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 1 Processor -- start 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 1 -- start start 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient: init 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc3809ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient(hunting): picked mon.noname-b con 0x7fcc3809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc38136d60 0x7fcc38139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc38136d60 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc38067420 con 0x7fcc3809ec50 2024-09-18T12:30:57.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3809fdc0 con 0x7fcc38136d60 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380a0510 con 0x7fcc3809f020 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc3d59d640 10 monclient(hunting): authenticate will time out at 1093.060586s 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc38136d60 0x7fcc38139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc36ffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fcc3809ec50 0x7fcc38131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39710/0 (socket says 172.21.15.144:39710) 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc36ffd640 1 -- 172.21.15.144:0/1960902218 learned_addr learned my addr 172.21.15.144:0/1960902218 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): get_auth_request con 0x7fcc38136d60 auth_method 0 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:57.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc367fc640 1 --2- 172.21.15.144:0/1960902218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc367fc640 10 monclient(hunting): get_auth_request con 0x7fcc3809f020 auth_method 0 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc367fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.491+0000 7fcc377fe640 10 monclient(hunting): handle_auth_done global_id 5668 payload 274 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 1 -- 172.21.15.144:0/1960902218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 msgr2=0x7fcc38069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 1 --2- 172.21.15.144:0/1960902218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 1 -- 172.21.15.144:0/1960902218 >> v1:172.21.15.144:6789/0 conn(0x7fcc3809ec50 legacy=0x7fcc38131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:57.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 10 monclient: _finish_hunting 0 2024-09-18T12:30:57.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 1 monclient: found mon.noname-c 2024-09-18T12:30:57.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:57.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:57.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.492+0000 7fcc377fe640 1 -- 172.21.15.144:0/1960902218 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc380a0200 con 0x7fcc38136d60 2024-09-18T12:30:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.494+0000 7fcc377fe640 1 -- 172.21.15.144:0/1960902218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc38136d60 msgr2=0x7fcc38139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.494+0000 7fcc377fe640 1 --2- 172.21.15.144:0/1960902218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc38136d60 0x7fcc38139140 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009e60 tx=0x7fcc30009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c77f5889d521515e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 1 -- 172.21.15.144:0/1960902218 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30007760 con 0x7fcc38136d60 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: dump: 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 1 -- 172.21.15.144:0/1960902218 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3002d050 con 0x7fcc38136d60 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 5 monclient: authenticate success, global_id 5668 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 1 -- 172.21.15.144:0/1960902218 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30007af0 con 0x7fcc38136d60 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 4 set_mon_vals no callback set 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc35ffb640 10 monclient: dump: 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:57.506 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 10 monclient: shutdown 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1960902218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc38136d60 msgr2=0x7fcc38139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1960902218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc38136d60 0x7fcc38139140 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009e60 tx=0x7fcc30009780 comp rx=0 tx=0).stop 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1960902218 shutdown_connections 2024-09-18T12:30:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1960902218 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc38136d60 0x7fcc38139140 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1960902218 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1960902218 >> 172.21.15.144:0/1960902218 conn(0x7fcc380a5d40 msgr2=0x7fcc380a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1960902218 shutdown_connections 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.495+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1960902218 wait complete. 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.496+0000 7fcc3d59d640 10 monclient: build_initial_monmap 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.496+0000 7fcc3d59d640 10 monclient: monmap: 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.496+0000 7fcc3d59d640 1 Processor -- start 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.496+0000 7fcc3d59d640 1 -- start start 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.496+0000 7fcc3d59d640 10 monclient: init 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.496+0000 7fcc3d59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 0x7fcc3809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): picked mon.noname-b con 0x7fcc3809ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38097260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc3809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc38136d60 0x7fcc380977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc38136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc381318f0 con 0x7fcc38136d60 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3809bab0 con 0x7fcc3809ec50 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3809c000 con 0x7fcc3809f020 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc3d59d640 10 monclient(hunting): authenticate will time out at 1093.066069s 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 0x7fcc3809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 0x7fcc3809b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42040/0 (socket says 172.21.15.144:42040) 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 -- 172.21.15.144:0/1484369370 learned_addr learned my addr 172.21.15.144:0/1484369370 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): get_auth_request con 0x7fcc3809ec50 auth_method 0 2024-09-18T12:30:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38097260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 10 monclient(hunting): get_auth_request con 0x7fcc3809f020 auth_method 0 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc377fe640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc38136d60 0x7fcc380977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc377fe640 10 monclient(hunting): get_auth_request con 0x7fcc38136d60 auth_method 0 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc377fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient(hunting): handle_auth_done global_id 5671 payload 824 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 -- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 msgr2=0x7fcc38097260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38097260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 -- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc38136d60 msgr2=0x7fcc380977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc38136d60 0x7fcc380977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 monclient: found mon.noname-b 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc36ffd640 1 -- 172.21.15.144:0/1484369370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30009a20 con 0x7fcc3809ec50 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.497+0000 7fcc377fe640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc38136d60 0x7fcc380977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc36ffd640 1 -- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 msgr2=0x7fcc3809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc36ffd640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 0x7fcc3809b540 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800b510 tx=0x7fcc280090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28011840 con 0x7fcc3809ec50 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: dump: 2024-09-18T12:30:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: _finish_auth 0 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc280119c0 con 0x7fcc3809ec50 2024-09-18T12:30:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc3d59d640 5 monclient: authenticate success, global_id 5671 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc3d59d640 10 monclient: _renew_subs 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc3d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc380986d0 con 0x7fcc3809ec50 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28011d80 con 0x7fcc3809ec50 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc1effd640 10 monclient: dump: 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:57.515 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc3d59d640 10 monclient: _renew_subs 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc3d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc38098d50 con 0x7fcc3809ec50 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.499+0000 7fcc357fa640 4 set_mon_vals no callback set 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fcc28010590 con 0x7fcc3809ec50 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc3d59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc3d59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc3d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc380008d0 con 0x7fcc3809ec50 2024-09-18T12:30:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc1effd640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc24038490 0x7fcc2403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(129..129 src has 13..129) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fcc2804c220 con 0x7fcc3809ec50 2024-09-18T12:30:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc367fc640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc24038490 0x7fcc2403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.500+0000 7fcc367fc640 10 monclient: get_auth_request con 0x7fcc24038490 auth_method 0 2024-09-18T12:30:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.503+0000 7fcc367fc640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc24038490 0x7fcc2403a940 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c007ba0 tx=0x7fcc2c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.505+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fcc2800f8c0 con 0x7fcc3809ec50 2024-09-18T12:30:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.505+0000 7fcc1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.505+0000 7fcc1effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:57.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.661+0000 7fcc3d59d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T12:30:57.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.661+0000 7fcc3d59d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T12:30:57.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.661+0000 7fcc3d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:57.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:57.661+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fcc3809ea30 con 0x7fcc3809ec50 2024-09-18T12:30:58.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:30:58.202 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.305+0000 7fa33399d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.305+0000 7fa33399d640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.305+0000 7fa33399d640 10 monclient: monmap: 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:58.305852+0000 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:58.305852+0000 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.306 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:58.307 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.305+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v130) v1 ==== 126+0+0 (secure 0 0 0) 0x7fcc28015070 con 0x7fcc3809ec50 2024-09-18T12:30:58.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.305+0000 7fcc1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T12:30:58.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.305+0000 7fcc1effd640 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-18T12:30:58.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.305+0000 7fa33399d640 1 Processor -- start 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.306+0000 7fa33399d640 1 -- start start 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.306+0000 7fa33399d640 10 monclient: init 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.306+0000 7fa33399d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.306+0000 7fa33399d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.306+0000 7fa33399d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa32c1318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa32c09ea50 0x7fa32c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 10 monclient(hunting): picked mon.noname-a con 0x7fa32c09ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa32c136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c131cc0 con 0x7fa32c1318f0 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c09fbf0 con 0x7fa32c136c20 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa32c09fe30 con 0x7fa32c09ea50 2024-09-18T12:30:58.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa33399d640 10 monclient(hunting): authenticate will time out at 1085.628085s 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa330f38640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa32c09ea50 0x7fa32c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa330f38640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa32c09ea50 0x7fa32c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51156/0 (socket says 172.21.15.116:51156) 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa331739640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa32c1318f0 0x7fa32c131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55500/0 (socket says 172.21.15.116:55500) 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa331739640 1 -- 172.21.15.116:0/1477080133 learned_addr learned my addr 172.21.15.116:0/1477080133 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa330f38640 10 monclient(hunting): get_auth_request con 0x7fa32c09ea50 auth_method 0 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa330f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa330f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.307+0000 7fa330f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 10 monclient(hunting): handle_auth_done global_id 84147 payload 274 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 1 -- 172.21.15.116:0/1477080133 >> v1:172.21.15.144:6789/0 conn(0x7fa32c136c20 legacy=0x7fa32c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 1 -- 172.21.15.116:0/1477080133 >> v1:172.21.15.116:6789/0 conn(0x7fa32c1318f0 legacy=0x7fa32c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 1 monclient: found mon.noname-a 2024-09-18T12:30:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.308+0000 7fa330f38640 1 -- 172.21.15.116:0/1477080133 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c0a1210 con 0x7fa32c09ea50 2024-09-18T12:30:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.311+0000 7fa330f38640 1 -- 172.21.15.116:0/1477080133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c09ea50 msgr2=0x7fa32c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.311+0000 7fa330f38640 1 --2- 172.21.15.116:0/1477080133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c09ea50 0x7fa32c09ee20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009850 tx=0x7fa31c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2ae492cb6111ee server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 1 -- 172.21.15.116:0/1477080133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c032440 con 0x7fa32c09ea50 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: dump: 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.314 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa33399d640 5 monclient: authenticate success, global_id 84147 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa33399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 1 -- 172.21.15.116:0/1477080133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31c0325a0 con 0x7fa32c09ea50 2024-09-18T12:30:58.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 1 -- 172.21.15.116:0/1477080133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c032940 con 0x7fa32c09ea50 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa33399d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa33399d640 4 set_mon_vals no callback set 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.312+0000 7fa323fff640 10 monclient: dump: 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.316 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 10 monclient: shutdown 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 1 -- 172.21.15.116:0/1477080133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c09ea50 msgr2=0x7fa32c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 1 --2- 172.21.15.116:0/1477080133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c09ea50 0x7fa32c09ee20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009850 tx=0x7fa31c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 1 -- 172.21.15.116:0/1477080133 shutdown_connections 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 1 --2- 172.21.15.116:0/1477080133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c09ea50 0x7fa32c09ee20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 1 -- 172.21.15.116:0/1477080133 >> 172.21.15.116:0/1477080133 conn(0x7fa32c067110 msgr2=0x7fa32c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 1 -- 172.21.15.116:0/1477080133 shutdown_connections 2024-09-18T12:30:58.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 1 -- 172.21.15.116:0/1477080133 wait complete. 2024-09-18T12:30:58.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.313+0000 7fa33399d640 10 monclient: monmap: 2024-09-18T12:30:58.322 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.322 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.314+0000 7fa33399d640 1 Processor -- start 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.314+0000 7fa33399d640 1 -- start start 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.314+0000 7fa33399d640 10 monclient: init 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 0x7fa32c1327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): picked mon.noname-c con 0x7fa32c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa32c1318f0 0x7fa32c132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): picked mon.noname-b con 0x7fa32c1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c136c20 0x7fa32c133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): picked mon.noname-a con 0x7fa32c136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c13a460 con 0x7fa32c136c20 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c1360c0 con 0x7fa32c1318f0 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c135de0 con 0x7fa32c09ea50 2024-09-18T12:30:58.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa33399d640 10 monclient(hunting): authenticate will time out at 1085.636168s 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c136c20 0x7fa32c133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 0x7fa32c1327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c136c20 0x7fa32c133250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51160/0 (socket says 172.21.15.116:51160) 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 0x7fa32c1327d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46922/0 (socket says 172.21.15.116:46922) 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 1 -- 172.21.15.116:0/3365401341 learned_addr learned my addr 172.21.15.116:0/3365401341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 10 monclient(hunting): get_auth_request con 0x7fa32c136c20 auth_method 0 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): get_auth_request con 0x7fa32c09ea50 auth_method 0 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa32c1318f0 0x7fa32c132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 10 monclient(hunting): get_auth_request con 0x7fa32c1318f0 auth_method 0 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa330f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient(hunting): handle_auth_done global_id 64493 payload 824 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 -- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa32c1318f0 msgr2=0x7fa32c132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa32c1318f0 0x7fa32c132d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 -- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c136c20 msgr2=0x7fa32c133250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c136c20 0x7fa32c133250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 monclient: found mon.noname-c 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331739640 1 -- 172.21.15.116:0/3365401341 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c009390 con 0x7fa32c09ea50 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.315+0000 7fa331f3a640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c136c20 0x7fa32c133250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.317+0000 7fa331739640 1 -- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 msgr2=0x7fa32c1327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.317+0000 7fa331739640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 0x7fa32c1327d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa32800c4c0 tx=0x7fa328007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.317+0000 7fa3217fa640 1 -- 172.21.15.116:0/3365401341 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328005cb0 con 0x7fa32c09ea50 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.317+0000 7fa3217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 10 monclient: dump: 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.328 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 1 -- 172.21.15.116:0/3365401341 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa328005e30 con 0x7fa32c09ea50 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa33399d640 5 monclient: authenticate success, global_id 64493 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa33399d640 10 monclient: _renew_subs 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa33399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c134320 con 0x7fa32c09ea50 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 1 -- 172.21.15.116:0/3365401341 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328007320 con 0x7fa32c09ea50 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 10 monclient: dump: 2024-09-18T12:30:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa33399d640 10 monclient: _renew_subs 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3237fe640 4 set_mon_vals no callback set 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa33399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c1366f0 con 0x7fa32c09ea50 2024-09-18T12:30:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.318+0000 7fa3217fa640 1 -- 172.21.15.116:0/3365401341 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa32801b3b0 con 0x7fa32c09ea50 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa33399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa33399d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa33399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa3217fa640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32c04bf60 con 0x7fa32c09ea50 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa3217fa640 1 -- 172.21.15.116:0/3365401341 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(130..130 src has 13..130) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fa32804b630 con 0x7fa32c09ea50 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa330f38640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa330f38640 1 -- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 msgr2=0x7fa31003a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa330f38640 1 -- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 msgr2=0x7fa31003a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa330f38640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:30:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.319+0000 7fa330f38640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.323+0000 7fa3217fa640 1 -- 172.21.15.116:0/3365401341 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa32801a360 con 0x7fa32c09ea50 2024-09-18T12:30:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.323+0000 7fa3217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.323+0000 7fa3217fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:58.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.402+0000 7fcc3d59d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T12:30:58.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.402+0000 7fcc3d59d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T12:30:58.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.402+0000 7fcc3d59d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.402+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fcc3809abf0 con 0x7fcc3809ec50 2024-09-18T12:30:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.520+0000 7fa330f38640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.520+0000 7fa330f38640 10 monclient: get_auth_request con 0x7fa310038490 auth_method 0 2024-09-18T12:30:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.522+0000 7fa330f38640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa32c076600 tx=0x7fa31c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.526+0000 7fcc1effd640 1 -- 172.21.15.144:0/1484369370 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v131) v1 ==== 126+0+0 (secure 0 0 0) 0x7fcc2800f2e0 con 0x7fcc3809ec50 2024-09-18T12:30:58.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.527+0000 7fcc1effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T12:30:58.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.527+0000 7fcc1effd640 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-18T12:30:58.528 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 10 size to 3 2024-09-18T12:30:58.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc24038490 msgr2=0x7fcc2403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc24038490 0x7fcc2403a940 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c007ba0 tx=0x7fcc2c008040 comp rx=0 tx=0).stop 2024-09-18T12:30:58.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 10 monclient: shutdown 2024-09-18T12:30:58.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 msgr2=0x7fcc3809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 0x7fcc3809b540 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800b510 tx=0x7fcc280090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 shutdown_connections 2024-09-18T12:30:58.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc24038490 0x7fcc2403a940 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc38136d60 0x7fcc380977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc3809f020 0x7fcc38097260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 --2- 172.21.15.144:0/1484369370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc3809ec50 0x7fcc3809b540 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 >> 172.21.15.144:0/1484369370 conn(0x7fcc380a5d40 msgr2=0x7fcc38069490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.530+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 shutdown_connections 2024-09-18T12:30:58.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.531+0000 7fcc3d59d640 1 -- 172.21.15.144:0/1484369370 wait complete. 2024-09-18T12:30:58.544 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T12:30:58.544 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:30:58.544 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T12:30:58.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.544+0000 7fa33399d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:30:58.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.544+0000 7fa33399d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:58.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.544+0000 7fa33399d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.544+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa32c0638d0 con 0x7fa32c09ea50 2024-09-18T12:30:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.544+0000 7fa3217fa640 1 -- 172.21.15.116:0/3365401341 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fa328017080 con 0x7fa32c09ea50 2024-09-18T12:30:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.545+0000 7fa3217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:30:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.545+0000 7fa3217fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:58.546 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.546+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 msgr2=0x7fa31003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa32c076600 tx=0x7fa31c032000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 10 monclient: shutdown 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 msgr2=0x7fa32c1327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 0x7fa32c1327d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa32800c4c0 tx=0x7fa328007f00 comp rx=0 tx=0).stop 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 shutdown_connections 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa310038490 0x7fa31003a940 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa32c136c20 0x7fa32c133250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa32c1318f0 0x7fa32c132d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 --2- 172.21.15.116:0/3365401341 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa32c09ea50 0x7fa32c1327d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 >> 172.21.15.116:0/3365401341 conn(0x7fa32c067110 msgr2=0x7fa32c12bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 shutdown_connections 2024-09-18T12:30:58.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.547+0000 7fa33399d640 1 -- 172.21.15.116:0/3365401341 wait complete. 2024-09-18T12:30:58.558 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:30:58.558 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:30:58.558 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:30:58.558 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.619+0000 7fe51d0d4640 10 monclient: get_monmap_and_config 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.619+0000 7fe51d0d4640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.619+0000 7fe51d0d4640 10 monclient: monmap: 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:58.620873+0000 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:58.620873+0000 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.621 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:58.622 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:58.622 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:58.622 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:58.622 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:58.622 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.620+0000 7fe51d0d4640 1 Processor -- start 2024-09-18T12:30:58.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.620+0000 7fe51d0d4640 1 -- start start 2024-09-18T12:30:58.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.620+0000 7fe51d0d4640 10 monclient: init 2024-09-18T12:30:58.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T12:30:58.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5180a0ad0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe51809ee60 0x7fe51809f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient(hunting): picked mon.noname-a con 0x7fe51809ee60 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 0x7fe5181391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient(hunting): picked mon.noname-b con 0x7fe518136e00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51806a5b0 con 0x7fe5180a0ad0 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51809f7a0 con 0x7fe51809ee60 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5180a0ea0 con 0x7fe518136e00 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51d0d4640 10 monclient(hunting): authenticate will time out at 1094.190243s 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe516d9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe5180a0ad0 0x7fe5180691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39712/0 (socket says 172.21.15.144:39712) 2024-09-18T12:30:58.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe516d9d640 1 -- 172.21.15.144:0/3403722106 learned_addr learned my addr 172.21.15.144:0/3403722106 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 1 --2- 172.21.15.144:0/3403722106 >> v2:172.21.15.116:3300/0 conn(0x7fe51809ee60 0x7fe51809f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): get_auth_request con 0x7fe51809ee60 auth_method 0 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 1 --2- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 0x7fe5181391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 10 monclient(hunting): get_auth_request con 0x7fe518136e00 auth_method 0 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient(hunting): handle_auth_done global_id 84153 payload 274 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 1 -- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 msgr2=0x7fe5181391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 1 --2- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 0x7fe5181391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 1 -- 172.21.15.144:0/3403722106 >> v1:172.21.15.144:6789/0 conn(0x7fe5180a0ad0 legacy=0x7fe5180691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 1 monclient: found mon.noname-a 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.621+0000 7fe51659c640 1 -- 172.21.15.144:0/3403722106 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5180a0460 con 0x7fe51809ee60 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.624+0000 7fe51659c640 1 -- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51809ee60 msgr2=0x7fe51809f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.624+0000 7fe51659c640 1 --2- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51809ee60 0x7fe51809f260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009850 tx=0x7fe50c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b0bf810318b0ea server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.624+0000 7fe515d9b640 1 -- 172.21.15.144:0/3403722106 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c02ae60 con 0x7fe51809ee60 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: dump: 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.634 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 1 -- 172.21.15.144:0/3403722106 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50c032440 con 0x7fe51809ee60 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 1 -- 172.21.15.144:0/3403722106 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0327e0 con 0x7fe51809ee60 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 5 monclient: authenticate success, global_id 84153 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 4 set_mon_vals no callback set 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe515d9b640 10 monclient: dump: 2024-09-18T12:30:58.635 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 10 monclient: shutdown 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 -- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51809ee60 msgr2=0x7fe51809f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 --2- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51809ee60 0x7fe51809f260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009850 tx=0x7fe50c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 -- 172.21.15.144:0/3403722106 shutdown_connections 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 --2- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 0x7fe5181391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 --2- 172.21.15.144:0/3403722106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51809ee60 0x7fe51809f260 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 -- 172.21.15.144:0/3403722106 >> 172.21.15.144:0/3403722106 conn(0x7fe518071370 msgr2=0x7fe5180a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 -- 172.21.15.144:0/3403722106 shutdown_connections 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.625+0000 7fe51d0d4640 1 -- 172.21.15.144:0/3403722106 wait complete. 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.626+0000 7fe51d0d4640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.626+0000 7fe51d0d4640 10 monclient: monmap: 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:58.637 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.626+0000 7fe51d0d4640 1 Processor -- start 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.626+0000 7fe51d0d4640 1 -- start start 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.626+0000 7fe51d0d4640 10 monclient: init 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe51809ee60 0x7fe5181414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): picked mon.noname-b con 0x7fe51809ee60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5180a0ad0 0x7fe5181419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): picked mon.noname-a con 0x7fe5180a0ad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 0x7fe51811e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): picked mon.noname-c con 0x7fe518136e00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5180a11e0 con 0x7fe5180a0ad0 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5180a4af0 con 0x7fe51809ee60 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe518121d60 con 0x7fe518136e00 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51d0d4640 10 monclient(hunting): authenticate will time out at 1094.196117s 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe51809ee60 0x7fe5181414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe51809ee60 0x7fe5181414a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42056/0 (socket says 172.21.15.144:42056) 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 -- 172.21.15.144:0/1294889590 learned_addr learned my addr 172.21.15.144:0/1294889590 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:58.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): get_auth_request con 0x7fe51809ee60 auth_method 0 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5180a0ad0 0x7fe5181419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 10 monclient(hunting): get_auth_request con 0x7fe5180a0ad0 auth_method 0 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 0x7fe51811e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 10 monclient(hunting): get_auth_request con 0x7fe518136e00 auth_method 0 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe51759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient(hunting): handle_auth_done global_id 5677 payload 824 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 -- 172.21.15.144:0/1294889590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 msgr2=0x7fe51811e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe518136e00 0x7fe51811e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 -- 172.21.15.144:0/1294889590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5180a0ad0 msgr2=0x7fe5181419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5180a0ad0 0x7fe5181419e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 monclient: found mon.noname-b 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.627+0000 7fe516d9d640 1 -- 172.21.15.144:0/1294889590 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c009390 con 0x7fe51809ee60 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe516d9d640 1 -- 172.21.15.144:0/1294889590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe51809ee60 msgr2=0x7fe5181414a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe516d9d640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe51809ee60 0x7fe5181414a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe50800c610 tx=0x7fe508004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 1 -- 172.21.15.144:0/1294889590 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508005cb0 con 0x7fe51809ee60 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: dump: 2024-09-18T12:30:58.642 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51d0d4640 5 monclient: authenticate success, global_id 5677 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 1 -- 172.21.15.144:0/1294889590 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe508005e30 con 0x7fe51809ee60 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51d0d4640 10 monclient: _renew_subs 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51d0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51d0d4640 1 -- 172.21.15.144:0/1294889590 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe518120230 con 0x7fe51809ee60 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 1 -- 172.21.15.144:0/1294889590 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508007340 con 0x7fe51809ee60 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe4feffd640 10 monclient: dump: 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.644 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51d0d4640 10 monclient: _renew_subs 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51559a640 4 set_mon_vals no callback set 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51d0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.629+0000 7fe51d0d4640 1 -- 172.21.15.144:0/1294889590 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe518141f20 con 0x7fe51809ee60 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.630+0000 7fe4feffd640 1 -- 172.21.15.144:0/1294889590 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe50801b400 con 0x7fe51809ee60 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.630+0000 7fe51d0d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.630+0000 7fe51d0d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.630+0000 7fe51d0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.630+0000 7fe51d0d4640 1 -- 172.21.15.144:0/1294889590 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5180008d0 con 0x7fe51809ee60 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.630+0000 7fe4feffd640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe504038490 0x7fe50403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.631+0000 7fe4feffd640 1 -- 172.21.15.144:0/1294889590 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fe50804bd30 con 0x7fe51809ee60 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.631+0000 7fe51659c640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe504038490 0x7fe50403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.631+0000 7fe51659c640 10 monclient: get_auth_request con 0x7fe504038490 auth_method 0 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.633+0000 7fe51659c640 1 --2- 172.21.15.144:0/1294889590 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe504038490 0x7fe50403a940 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe50c031000 tx=0x7fe50c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.635+0000 7fe4feffd640 1 -- 172.21.15.144:0/1294889590 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe50801a940 con 0x7fe51809ee60 2024-09-18T12:30:58.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.635+0000 7fe4feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.635+0000 7fe4feffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.651+0000 7f0cbb22b640 10 monclient: get_monmap_and_config 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.651+0000 7f0cbb22b640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.651+0000 7f0cbb22b640 10 monclient: monmap: 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:58.652236+0000 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:58.652236+0000 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.653 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:58.653 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:58.653 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:58.653 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:58.653 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:58.653 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 Processor -- start 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 -- start start 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient: init 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cb409c860 0x7f0cb409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): picked mon.noname-a con 0x7f0cb409c860 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0cb409aba0 0x7f0cb409d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): picked mon.noname-c con 0x7f0cb409aba0 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 0x7f0cb409dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): picked mon.noname-b con 0x7f0cb409afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb409a9d0 con 0x7f0cb409c860 2024-09-18T12:30:58.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb406a320 con 0x7f0cb409aba0 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4068a20 con 0x7f0cb409afa0 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cbb22b640 10 monclient(hunting): authenticate will time out at 1085.973784s 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb97c8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 0x7f0cb409dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb8fc7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cb409c860 0x7f0cb409cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb97c8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 0x7f0cb409dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46930/0 (socket says 172.21.15.116:46930) 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb8fc7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cb409c860 0x7f0cb409cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51164/0 (socket says 172.21.15.116:51164) 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb97c8640 1 -- 172.21.15.116:0/3447846353 learned_addr learned my addr 172.21.15.116:0/3447846353 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb97c8640 10 monclient(hunting): get_auth_request con 0x7f0cb409afa0 auth_method 0 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb97c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb97c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb97c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb8fc7640 10 monclient(hunting): get_auth_request con 0x7f0cb409c860 auth_method 0 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb8fc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.652+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 1 --2- 172.21.15.116:0/3447846353 >> v2:172.21.15.144:3300/0 conn(0x7f0cb409aba0 0x7f0cb409d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 10 monclient(hunting): get_auth_request con 0x7f0cb409aba0 auth_method 0 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_done global_id 64502 payload 274 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 1 -- 172.21.15.116:0/3447846353 >> v2:172.21.15.144:3300/0 conn(0x7f0cb409aba0 msgr2=0x7f0cb409d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 1 --2- 172.21.15.116:0/3447846353 >> v2:172.21.15.144:3300/0 conn(0x7f0cb409aba0 0x7f0cb409d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 1 -- 172.21.15.116:0/3447846353 >> v2:172.21.15.116:3300/0 conn(0x7f0cb409c860 msgr2=0x7f0cb409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 1 --2- 172.21.15.116:0/3447846353 >> v2:172.21.15.116:3300/0 conn(0x7f0cb409c860 0x7f0cb409cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 1 monclient: found mon.noname-b 2024-09-18T12:30:58.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb97c8640 1 -- 172.21.15.116:0/3447846353 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4068e60 con 0x7f0cb409afa0 2024-09-18T12:30:58.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.653+0000 7f0cb8fc7640 1 --2- 172.21.15.116:0/3447846353 >> v2:172.21.15.116:3300/0 conn(0x7f0cb409c860 0x7f0cb409cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.655+0000 7f0cb97c8640 1 -- 172.21.15.116:0/3447846353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 msgr2=0x7f0cb409dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.655+0000 7f0cb97c8640 1 --2- 172.21.15.116:0/3447846353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 0x7f0cb409dd70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0cb406b700 tx=0x7f0cac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b29033d5e8384236 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 1 -- 172.21.15.116:0/3447846353 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac007610 con 0x7f0cb409afa0 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: dump: 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 1 -- 172.21.15.116:0/3447846353 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac02d050 con 0x7f0cb409afa0 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 5 monclient: authenticate success, global_id 64502 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 1 -- 172.21.15.116:0/3447846353 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0079a0 con 0x7f0cb409afa0 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 4 set_mon_vals no callback set 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cab7fe640 10 monclient: dump: 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 10 monclient: shutdown 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 -- 172.21.15.116:0/3447846353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 msgr2=0x7f0cb409dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/3447846353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 0x7f0cb409dd70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0cb406b700 tx=0x7f0cac009840 comp rx=0 tx=0).stop 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 -- 172.21.15.116:0/3447846353 shutdown_connections 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/3447846353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409afa0 0x7f0cb409dd70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/3447846353 >> v2:172.21.15.144:3300/0 conn(0x7f0cb409aba0 0x7f0cb409d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/3447846353 >> v2:172.21.15.116:3300/0 conn(0x7f0cb409c860 0x7f0cb409cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 -- 172.21.15.116:0/3447846353 >> 172.21.15.116:0/3447846353 conn(0x7f0cb40a2e40 msgr2=0x7f0cb40a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 -- 172.21.15.116:0/3447846353 shutdown_connections 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.656+0000 7f0cbb22b640 1 -- 172.21.15.116:0/3447846353 wait complete. 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.657+0000 7f0cbb22b640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.657+0000 7f0cbb22b640 10 monclient: monmap: 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.657+0000 7f0cbb22b640 1 Processor -- start 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.657+0000 7f0cbb22b640 1 -- start start 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.657+0000 7f0cbb22b640 10 monclient: init 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb409aba0 0x7f0cb413d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): picked mon.noname-b con 0x7f0cb409aba0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb409afa0 0x7f0cb413cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): picked mon.noname-a con 0x7f0cb409afa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409c860 0x7f0cb4140c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): picked mon.noname-c con 0x7f0cb409c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4122680 con 0x7f0cb409afa0 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb41220c0 con 0x7f0cb409aba0 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb411fb50 con 0x7f0cb409c860 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cbb22b640 10 monclient(hunting): authenticate will time out at 1085.979239s 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409c860 0x7f0cb4140c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb409afa0 0x7f0cb413cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409c860 0x7f0cb4140c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46942/0 (socket says 172.21.15.116:46942) 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 -- 172.21.15.116:0/4088601291 learned_addr learned my addr 172.21.15.116:0/4088601291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb409afa0 0x7f0cb413cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51180/0 (socket says 172.21.15.116:51180) 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): get_auth_request con 0x7f0cb409c860 auth_method 0 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 10 monclient(hunting): get_auth_request con 0x7f0cb409afa0 auth_method 0 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb409aba0 0x7f0cb413d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 10 monclient(hunting): get_auth_request con 0x7f0cb409aba0 auth_method 0 2024-09-18T12:30:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient(hunting): handle_auth_done global_id 64505 payload 824 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb409aba0 msgr2=0x7f0cb413d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cb409aba0 0x7f0cb413d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb409afa0 msgr2=0x7f0cb413cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb409afa0 0x7f0cb413cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 monclient: found mon.noname-c 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cb97c8640 1 -- 172.21.15.116:0/4088601291 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0009390 con 0x7f0cb409c860 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.658+0000 7f0cabfff640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb409afa0 0x7f0cb413cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0cb97c8640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409c860 msgr2=0x7f0cb4140c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0cb97c8640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409c860 0x7f0cb4140c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009f20 tx=0x7f0cac036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac007170 con 0x7f0cb409c860 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 10 monclient: dump: 2024-09-18T12:30:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0cbb22b640 5 monclient: authenticate success, global_id 64505 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac032c90 con 0x7f0cb409c860 2024-09-18T12:30:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0cbb22b640 10 monclient: _renew_subs 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac03b640 con 0x7f0cb409c860 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0cbb22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cac009ae0 con 0x7f0cb409c860 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0ca8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0ca8ff9640 10 monclient: dump: 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.660+0000 7f0caaffd640 4 set_mon_vals no callback set 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cbb22b640 10 monclient: _renew_subs 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cbb22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb413d280 con 0x7f0cb409c860 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cbb22b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 0x7f0cb40298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4122540 con 0x7f0cb4120f10 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0cac038050 con 0x7f0cb409c860 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cb8fc7640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 0x7f0cb40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cb8fc7640 10 monclient: get_auth_request con 0x7f0cb4120f10 auth_method 0 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cb8fc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.661+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0ca8ff9640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca0038490 0x7f0ca003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f0cac06c420 con 0x7f0cb409c860 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_done global_id 84165 payload 824 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0cb8fc7640 1 -- 172.21.15.116:0/4088601291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cb41227f0 con 0x7f0cb4120f10 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0cabfff640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca0038490 0x7f0ca003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.662+0000 7f0cabfff640 10 monclient: get_auth_request con 0x7f0ca0038490 auth_method 0 2024-09-18T12:30:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0cb8fc7640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 msgr2=0x7f0cb40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0cb8fc7640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 0x7f0cb40298c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0011bf0 tx=0x7f0cb00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb000fe90 con 0x7f0cb4120f10 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 10 monclient: dump: 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0cabfff640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca0038490 0x7f0ca003a940 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40718e0 tx=0x7f0ca40090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0cb0014650 con 0x7f0cb4120f10 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 msgr2=0x7f0cb40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 0x7f0cb40298c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0011bf0 tx=0x7f0cb00090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 msgr2=0x7f0cb40298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.664+0000 7f0ca8ff9640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4120f10 0x7f0cb40298c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cbb22b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 0x7f0cb4029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cbb22b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4065e00 con 0x7f0cb413dff0 2024-09-18T12:30:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 0x7f0cb4029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient: get_auth_request con 0x7f0cb413dff0 auth_method 0 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 10 monclient(hunting): handle_auth_done global_id 84168 payload 824 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.670+0000 7f0cb8fc7640 1 -- 172.21.15.116:0/4088601291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0cb4121ef0 con 0x7f0cb413dff0 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cb8fc7640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 msgr2=0x7f0cb4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cb8fc7640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 0x7f0cb4029e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0011bf0 tx=0x7f0cb001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0025040 con 0x7f0cb413dff0 2024-09-18T12:30:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 10 monclient: dump: 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f0cb0014440 con 0x7f0cb413dff0 2024-09-18T12:30:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 msgr2=0x7f0cb4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 0x7f0cb4029e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0011bf0 tx=0x7f0cb001d000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 msgr2=0x7f0cb4029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0ca8ff9640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb413dff0 0x7f0cb4029e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:30:58.686 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 66, 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.687 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.688 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:58.689 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.690 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.691 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.692 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.693 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.694 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.696 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.697 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:58.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.698 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.698 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:58.698 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca0038490 msgr2=0x7f0ca003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca0038490 0x7f0ca003a940 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40718e0 tx=0x7f0ca40090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 10 monclient: shutdown 2024-09-18T12:30:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409c860 msgr2=0x7f0cb4140c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 --2- 172.21.15.116:0/4088601291 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cb409c860 0x7f0cb4140c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009f20 tx=0x7f0cac036000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cb97c8640 1 -- 172.21.15.116:0/4088601291 reap_dead start 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cb97c8640 1 -- 172.21.15.116:0/4088601291 reap_dead start 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 shutdown_connections 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 >> 172.21.15.116:0/4088601291 conn(0x7f0cb40a2e40 msgr2=0x7f0cb4098a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 shutdown_connections 2024-09-18T12:30:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.673+0000 7f0cbb22b640 1 -- 172.21.15.116:0/4088601291 wait complete. 2024-09-18T12:30:58.700 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:58.700 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:30:58.701 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:30:58.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.792+0000 7f3671d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:58.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.792+0000 7f3671d9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.792+0000 7f3671d9d640 10 monclient: monmap: 2024-09-18T12:30:58.793 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:58.793324+0000 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:58.793324+0000 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:58.794 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 1 Processor -- start 2024-09-18T12:30:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 1 -- start start 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient: init 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f366c09c860 0x7f366c09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f366c09c860 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f366c09aba0 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c09df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f366c09afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c06a320 con 0x7f366c09aba0 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c09a9d0 con 0x7f366c09c860 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c068a20 con 0x7f366c09afa0 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.793+0000 7f3671d9d640 10 monclient(hunting): authenticate will time out at 1086.114961s 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c09df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f366c09c860 0x7f366c09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f366c09c860 0x7f366c09cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51198/0 (socket says 172.21.15.116:51198) 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c09df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46946/0 (socket says 172.21.15.116:46946) 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 1 -- 172.21.15.116:0/2026092424 learned_addr learned my addr 172.21.15.116:0/2026092424 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 10 monclient(hunting): get_auth_request con 0x7f366c09c860 auth_method 0 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): get_auth_request con 0x7f366c09afa0 auth_method 0 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient(hunting): handle_auth_done global_id 64508 payload 274 2024-09-18T12:30:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 1 -- 172.21.15.116:0/2026092424 >> v2:172.21.15.116:3300/0 conn(0x7f366c09c860 msgr2=0x7f366c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 1 --2- 172.21.15.116:0/2026092424 >> v2:172.21.15.116:3300/0 conn(0x7f366c09c860 0x7f366c09cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 1 -- 172.21.15.116:0/2026092424 >> v1:172.21.15.144:6789/0 conn(0x7f366c09aba0 legacy=0x7f366c09d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 1 monclient: found mon.noname-b 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366bfff640 1 -- 172.21.15.116:0/2026092424 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c068e60 con 0x7f366c09afa0 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.794+0000 7f366b7fe640 1 --2- 172.21.15.116:0/2026092424 >> v2:172.21.15.116:3300/0 conn(0x7f366c09c860 0x7f366c09cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366bfff640 1 -- 172.21.15.116:0/2026092424 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 msgr2=0x7f366c09df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366bfff640 1 --2- 172.21.15.116:0/2026092424 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c09df40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3664009e60 tx=0x7f3664009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=779c273f7b3a99ca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.797+0000 7fe51d0d4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:30:58.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.797+0000 7fe51d0d4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:30:58.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.797+0000 7fe51d0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.797+0000 7fe51d0d4640 1 -- 172.21.15.144:0/1294889590 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7fe518122210 con 0x7fe51809ee60 2024-09-18T12:30:58.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.798+0000 7fe4feffd640 1 -- 172.21.15.144:0/1294889590 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v131) v1 ==== 160+0+0 (secure 0 0 0) 0x7fe50801a360 con 0x7fe51809ee60 2024-09-18T12:30:58.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.798+0000 7fe4feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:30:58.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.798+0000 7fe4feffd640 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-18T12:30:58.803 INFO:tasks.workunit.client.0.smithi144.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-18T12:30:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:30:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-18T12:30:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 1 -- 172.21.15.116:0/2026092424 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664007760 con 0x7f366c09afa0 2024-09-18T12:30:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: dump: 2024-09-18T12:30:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f3671d9d640 5 monclient: authenticate success, global_id 64508 2024-09-18T12:30:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 1 -- 172.21.15.116:0/2026092424 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366402d050 con 0x7f366c09afa0 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f3671d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f3671d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f3671d9d640 4 set_mon_vals no callback set 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 1 -- 172.21.15.116:0/2026092424 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664007af0 con 0x7f366c09afa0 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.797+0000 7f366a7fc640 10 monclient: dump: 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 10 monclient: shutdown 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 -- 172.21.15.116:0/2026092424 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 msgr2=0x7f366c09df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 --2- 172.21.15.116:0/2026092424 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c09df40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3664009e60 tx=0x7f3664009780 comp rx=0 tx=0).stop 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 -- 172.21.15.116:0/2026092424 shutdown_connections 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 --2- 172.21.15.116:0/2026092424 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c09df40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 --2- 172.21.15.116:0/2026092424 >> v2:172.21.15.116:3300/0 conn(0x7f366c09c860 0x7f366c09cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 -- 172.21.15.116:0/2026092424 >> 172.21.15.116:0/2026092424 conn(0x7f366c0a2e40 msgr2=0x7f366c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 -- 172.21.15.116:0/2026092424 shutdown_connections 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 1 -- 172.21.15.116:0/2026092424 wait complete. 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.798+0000 7f3671d9d640 10 monclient: monmap: 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 Processor -- start 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 -- start start 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient: init 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c09aba0 0x7f366c140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f366c09aba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f366c09afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c09c860 0x7f366c11e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f366c09c860 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0a1280 con 0x7f366c09aba0 2024-09-18T12:30:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c122060 con 0x7f366c09c860 2024-09-18T12:30:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c099c90 con 0x7f366c09afa0 2024-09-18T12:30:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f3671d9d640 10 monclient(hunting): authenticate will time out at 1086.120692s 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c09aba0 0x7f366c140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c09aba0 0x7f366c140e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51214/0 (socket says 172.21.15.116:51214) 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 1 -- 172.21.15.116:0/1072358942 learned_addr learned my addr 172.21.15.116:0/1072358942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 10 monclient(hunting): get_auth_request con 0x7f366c09aba0 auth_method 0 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 10 monclient(hunting): get_auth_request con 0x7f366c09afa0 auth_method 0 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366bfff640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c09c860 0x7f366c11e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366bfff640 10 monclient(hunting): get_auth_request con 0x7f366c09c860 auth_method 0 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.799+0000 7f366bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 10 monclient(hunting): handle_auth_done global_id 64511 payload 824 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c09c860 msgr2=0x7f366c11e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c09c860 0x7f366c11e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c09aba0 msgr2=0x7f366c140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c09aba0 0x7f366c140e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 1 monclient: found mon.noname-c 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366affd640 1 -- 172.21.15.116:0/1072358942 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009450 con 0x7f366c09afa0 2024-09-18T12:30:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.800+0000 7f366b7fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c09aba0 0x7f366c140e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f366affd640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 msgr2=0x7f366c134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f366affd640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c134c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f366000c510 tx=0x7f3660007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600117a0 con 0x7f366c09afa0 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: dump: 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3671d9d640 5 monclient: authenticate success, global_id 64511 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3660011920 con 0x7f366c09afa0 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660011ce0 con 0x7f366c09afa0 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3671d9d640 10 monclient: _renew_subs 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3671d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3664009a20 con 0x7f366c09afa0 2024-09-18T12:30:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f36537fe640 10 monclient: dump: 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3669ffb640 4 set_mon_vals no callback set 2024-09-18T12:30:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3671d9d640 10 monclient: _renew_subs 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3671d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.802+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c141c60 con 0x7f366c09afa0 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f3671d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366affd640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c134c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f366000c510 tx=0x7f3660007e10 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f366004be50 osd_map(131..131 src has 13..131) v4 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f3671d9d640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 0x7f366c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f3671d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c1224e0 con 0x7f366c120e90 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3660012630 con 0x7f366c09afa0 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 0x7f366c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient: get_auth_request con 0x7f366c120e90 auth_method 0 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f36537fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3658038490 0x7f365803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 10 monclient(hunting): handle_auth_done global_id 64514 payload 824 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366b7fe640 1 -- 172.21.15.116:0/1072358942 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f366c121d80 con 0x7f366c120e90 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366bfff640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3658038490 0x7f365803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.803+0000 7f366bfff640 10 monclient: get_auth_request con 0x7f3658038490 auth_method 0 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.805+0000 7f366b7fe640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 msgr2=0x7f366c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.805+0000 7f366b7fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 0x7f366c0298c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f365c00f4a0 tx=0x7f365c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c013a40 con 0x7f366c120e90 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 10 monclient: dump: 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f365c013ba0 con 0x7f366c120e90 2024-09-18T12:30:58.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 msgr2=0x7f366c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 0x7f366c0298c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f365c00f4a0 tx=0x7f365c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 msgr2=0x7f366c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f36537fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c120e90 0x7f366c0298c0 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f365c00f4a0 tx=0x7f365c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.806+0000 7f366bfff640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3658038490 0x7f365803a940 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f366c06b700 tx=0x7f3664037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.811+0000 7f3671d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.811+0000 7f3671d9d640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 0x7f366c029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.811+0000 7f3671d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.811+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c065e00 con 0x7f366c1351a0 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 0x7f366c029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient: get_auth_request con 0x7f366c1351a0 auth_method 0 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 10 monclient(hunting): handle_auth_done global_id 64517 payload 824 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.812+0000 7f366b7fe640 1 -- 172.21.15.116:0/1072358942 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f366c120350 con 0x7f366c1351a0 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f366b7fe640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 msgr2=0x7f366c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f366b7fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 0x7f366c029e00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f365c01b000 tx=0x7f365c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c01da00 con 0x7f366c1351a0 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 10 monclient: dump: 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f365c01c3e0 con 0x7f366c1351a0 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 msgr2=0x7f366c029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 0x7f366c029e00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f365c01b000 tx=0x7f365c01c000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 msgr2=0x7f366c029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.814+0000 7f36537fe640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c1351a0 0x7f366c029e00 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f365c01b000 tx=0x7f365c01c000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.832 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:58.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:58.832 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 66, 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.833 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.834 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:58.835 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:58.836 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:58.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.838 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.840 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.842 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.843 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.844 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:30:58.845 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.846 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.846 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:58.846 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3658038490 msgr2=0x7f365803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3658038490 0x7f365803a940 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f366c06b700 tx=0x7f3664037000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 10 monclient: shutdown 2024-09-18T12:30:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 msgr2=0x7f366c134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 --2- 172.21.15.116:0/1072358942 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09afa0 0x7f366c134c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f366000c510 tx=0x7f3660007e10 comp rx=0 tx=0).stop 2024-09-18T12:30:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f366bfff640 1 -- 172.21.15.116:0/1072358942 reap_dead start 2024-09-18T12:30:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f366bfff640 1 -- 172.21.15.116:0/1072358942 reap_dead start 2024-09-18T12:30:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f36537fe640 1 -- 172.21.15.116:0/1072358942 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (unknown 0 0 0) 0x7f366004be50 con 0x7f366c09afa0 2024-09-18T12:30:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 shutdown_connections 2024-09-18T12:30:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 >> 172.21.15.116:0/1072358942 conn(0x7f366c0a2e40 msgr2=0x7f366c098db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 shutdown_connections 2024-09-18T12:30:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.815+0000 7f3671d9d640 1 -- 172.21.15.116:0/1072358942 wait complete. 2024-09-18T12:30:58.849 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:58.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:30:58.849 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.877+0000 7f5538a78640 10 monclient: get_monmap_and_config 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.877+0000 7f5538a78640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.877+0000 7f5538a78640 10 monclient: monmap: 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:30:58.878101+0000 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:30:58.878101+0000 2024-09-18T12:30:58.878 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.879 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.879 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:58.879 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:58.879 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:58.879 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:58.879 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:58.879 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.877+0000 7f5538a78640 1 Processor -- start 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.877+0000 7f5538a78640 1 -- start start 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.877+0000 7f5538a78640 10 monclient: init 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5534098c70 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f5534136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient(hunting): picked mon.noname-b con 0x7f5534099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5534137200 0x7f55341395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient(hunting): picked mon.noname-c con 0x7f5534137200 addr v2:172.21.15.144:3300/0 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553409f680 con 0x7f5534098c70 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553409f420 con 0x7f5534137200 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5534068a20 con 0x7f5534099c70 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5538a78640 10 monclient(hunting): authenticate will time out at 1094.447488s 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5534137200 0x7f55341395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5534137200 0x7f55341395e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42072/0 (socket says 172.21.15.144:42072) 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 -- 172.21.15.144:0/1180722459 learned_addr learned my addr 172.21.15.144:0/1180722459 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:58.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): get_auth_request con 0x7f5534137200 auth_method 0 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f553259c640 1 --1- 172.21.15.144:0/1180722459 >> v1:172.21.15.144:6789/0 conn(0x7f5534098c70 0x7f5534099040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:39718/0 (socket says 172.21.15.144:39718) 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5531d9b640 1 --2- 172.21.15.144:0/1180722459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f5534136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5531d9b640 10 monclient(hunting): get_auth_request con 0x7f5534099c70 auth_method 0 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5531d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5531d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5531d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient(hunting): handle_auth_done global_id 5689 payload 274 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 -- 172.21.15.144:0/1180722459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 msgr2=0x7f5534136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 --2- 172.21.15.144:0/1180722459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f5534136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 -- 172.21.15.144:0/1180722459 >> v1:172.21.15.144:6789/0 conn(0x7f5534098c70 legacy=0x7f5534099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 monclient: found mon.noname-c 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5532d9d640 1 -- 172.21.15.144:0/1180722459 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5534096a30 con 0x7f5534137200 2024-09-18T12:30:58.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.878+0000 7f5531d9b640 1 --2- 172.21.15.144:0/1180722459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f5534136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:30:58.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f5532d9d640 1 -- 172.21.15.144:0/1180722459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534137200 msgr2=0x7f55341395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f5532d9d640 1 --2- 172.21.15.144:0/1180722459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534137200 0x7f55341395e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f552c009f20 tx=0x7f552c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6160f6d5c4584658 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 1 -- 172.21.15.144:0/1180722459 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c007610 con 0x7f5534137200 2024-09-18T12:30:58.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: dump: 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.892 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 1 -- 172.21.15.144:0/1180722459 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c02d050 con 0x7f5534137200 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f5538a78640 5 monclient: authenticate success, global_id 5689 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f5538a78640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 1 -- 172.21.15.144:0/1180722459 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c0079a0 con 0x7f5534137200 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f5538a78640 4 set_mon_vals no callback set 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.881+0000 7f553159a640 10 monclient: dump: 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.893 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 10 monclient: shutdown 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 -- 172.21.15.144:0/1180722459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534137200 msgr2=0x7f55341395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 --2- 172.21.15.144:0/1180722459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534137200 0x7f55341395e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f552c009f20 tx=0x7f552c009840 comp rx=0 tx=0).stop 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 -- 172.21.15.144:0/1180722459 shutdown_connections 2024-09-18T12:30:58.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 --2- 172.21.15.144:0/1180722459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534137200 0x7f55341395e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 --2- 172.21.15.144:0/1180722459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f5534136cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 -- 172.21.15.144:0/1180722459 >> 172.21.15.144:0/1180722459 conn(0x7f5534070e90 msgr2=0x7f55340a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 -- 172.21.15.144:0/1180722459 shutdown_connections 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 1 -- 172.21.15.144:0/1180722459 wait complete. 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.882+0000 7f5538a78640 10 monclient: monmap: 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.895 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 Processor -- start 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 -- start start 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient: init 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 0x7f553411e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): picked mon.noname-b con 0x7f5534098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f553411ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): picked mon.noname-c con 0x7f5534099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534137200 0x7f553411f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): picked mon.noname-a con 0x7f5534137200 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534136350 con 0x7f5534137200 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534136da0 con 0x7f5534098c70 2024-09-18T12:30:58.897 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55340a1b70 con 0x7f5534099c70 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5538a78640 10 monclient(hunting): authenticate will time out at 1094.452712s 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 0x7f553411e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 0x7f553411e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42076/0 (socket says 172.21.15.144:42076) 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 1 -- 172.21.15.144:0/2695433451 learned_addr learned my addr 172.21.15.144:0/2695433451 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 10 monclient(hunting): get_auth_request con 0x7f5534098c70 auth_method 0 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f553259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5531d9b640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f553411ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5531d9b640 10 monclient(hunting): get_auth_request con 0x7f5534099c70 auth_method 0 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5531d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5531d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5531d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5532d9d640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534137200 0x7f553411f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5532d9d640 10 monclient(hunting): get_auth_request con 0x7f5534137200 auth_method 0 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5532d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5532d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5532d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5531d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.883+0000 7f5531d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f5531d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f5532d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f5532d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f5532d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 10 monclient(hunting): handle_auth_done global_id 5692 payload 824 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 1 -- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 msgr2=0x7f553411ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f553411ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 1 -- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534137200 msgr2=0x7f553411f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534137200 0x7f553411f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 1 monclient: found mon.noname-b 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.884+0000 7f553259c640 1 -- 172.21.15.144:0/2695433451 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c009ae0 con 0x7f5534098c70 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f553259c640 1 -- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 msgr2=0x7f553411e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f553259c640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 0x7f553411e700 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c5b0 tx=0x7f551c007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c007510 con 0x7f5534098c70 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: dump: 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.901 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c0057d0 con 0x7f5534098c70 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5538a78640 5 monclient: authenticate success, global_id 5692 2024-09-18T12:30:58.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c007690 con 0x7f5534098c70 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5538a78640 10 monclient: _renew_subs 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5538a78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5534121cc0 con 0x7f5534098c70 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5530d99640 4 set_mon_vals no callback set 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f552a7fc640 10 monclient: dump: 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.903 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5538a78640 10 monclient: _renew_subs 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5538a78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.886+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5534120200 con 0x7f5534098c70 2024-09-18T12:30:58.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f551c01b400 con 0x7f5534098c70 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f5538a78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f5538a78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f5538a78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553409a070 con 0x7f5534098c70 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f552a7fc640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5514038490 0x7f551403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f551c04bc70 con 0x7f5534098c70 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f5531d9b640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5514038490 0x7f551403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.887+0000 7f5531d9b640 10 monclient: get_auth_request con 0x7f5514038490 auth_method 0 2024-09-18T12:30:58.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.890+0000 7f5531d9b640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5514038490 0x7f551403a940 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f553411fe90 tx=0x7f5524008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.891+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f551c01b6b0 con 0x7f5534098c70 2024-09-18T12:30:58.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.891+0000 7f552a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:58.891+0000 7f552a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.941+0000 7fb187d5d640 10 monclient: get_monmap_and_config 2024-09-18T12:30:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.941+0000 7fb187d5d640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.941+0000 7fb187d5d640 10 monclient: monmap: 2024-09-18T12:30:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:58.942303+0000 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:58.942303+0000 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:58.943 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 1 Processor -- start 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 1 -- start start 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient: init 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb180067110 addr v1:172.21.15.144:6789/0 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb18009b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb18009b010 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1801349a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb180130540 con 0x7fb1801349a0 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb180099800 con 0x7fb180067110 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18009b940 con 0x7fb18009b010 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.942+0000 7fb187d5d640 10 monclient(hunting): authenticate will time out at 1086.263895s 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb18009b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1862fa640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb1801349a0 0x7fb180136d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55504/0 (socket says 172.21.15.116:55504) 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1862fa640 1 -- 172.21.15.116:0/2506378183 learned_addr learned my addr 172.21.15.116:0/2506378183 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): get_auth_request con 0x7fb18009b010 auth_method 0 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient(hunting): handle_auth_done global_id 64526 payload 274 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 1 -- 172.21.15.116:0/2506378183 >> v1:172.21.15.144:6789/0 conn(0x7fb180067110 legacy=0x7fb18012f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 1 -- 172.21.15.116:0/2506378183 >> v1:172.21.15.116:6789/0 conn(0x7fb1801349a0 legacy=0x7fb180136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 1 monclient: found mon.noname-b 2024-09-18T12:30:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:58.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.943+0000 7fb1852f8640 1 -- 172.21.15.116:0/2506378183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800674e0 con 0x7fb18009b010 2024-09-18T12:30:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb1852f8640 1 -- 172.21.15.116:0/2506378183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 msgr2=0x7fb18009b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb1852f8640 1 --2- 172.21.15.116:0/2506378183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb18009b400 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb170009880 tx=0x7fb1700090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0e0c70606e9118 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 1 -- 172.21.15.116:0/2506378183 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170032440 con 0x7fb18009b010 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: dump: 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.955 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 1 -- 172.21.15.116:0/2506378183 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1700325a0 con 0x7fb18009b010 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 5 monclient: authenticate success, global_id 64526 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 1 -- 172.21.15.116:0/2506378183 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170032940 con 0x7fb18009b010 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 4 set_mon_vals no callback set 2024-09-18T12:30:58.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb184af7640 10 monclient: dump: 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 10 monclient: shutdown 2024-09-18T12:30:58.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 1 -- 172.21.15.116:0/2506378183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 msgr2=0x7fb18009b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 1 --2- 172.21.15.116:0/2506378183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb18009b400 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb170009880 tx=0x7fb1700090f0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 1 -- 172.21.15.116:0/2506378183 shutdown_connections 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 1 --2- 172.21.15.116:0/2506378183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb18009b400 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.946+0000 7fb187d5d640 1 -- 172.21.15.116:0/2506378183 >> 172.21.15.116:0/2506378183 conn(0x7fb1800a1e30 msgr2=0x7fb1800a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.947+0000 7fb187d5d640 1 -- 172.21.15.116:0/2506378183 shutdown_connections 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.947+0000 7fb187d5d640 1 -- 172.21.15.116:0/2506378183 wait complete. 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.947+0000 7fb187d5d640 10 monclient: build_initial_monmap 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.947+0000 7fb187d5d640 10 monclient: monmap: 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:58.958 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.947+0000 7fb187d5d640 1 Processor -- start 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.947+0000 7fb187d5d640 1 -- start start 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.947+0000 7fb187d5d640 10 monclient: init 2024-09-18T12:30:58.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180067110 0x7fb180096cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb180067110 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb180097200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb18009b010 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1801349a0 0x7fb180097740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb1801349a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18006dbc0 con 0x7fb1801349a0 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800a0820 con 0x7fb180067110 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18006d080 con 0x7fb18009b010 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb187d5d640 10 monclient(hunting): authenticate will time out at 1086.269207s 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1801349a0 0x7fb180097740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb180097200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1801349a0 0x7fb180097740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51228/0 (socket says 172.21.15.116:51228) 2024-09-18T12:30:58.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb180097200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46990/0 (socket says 172.21.15.116:46990) 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 -- 172.21.15.116:0/1996821439 learned_addr learned my addr 172.21.15.116:0/1996821439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): get_auth_request con 0x7fb1801349a0 auth_method 0 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 10 monclient(hunting): get_auth_request con 0x7fb18009b010 auth_method 0 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180067110 0x7fb180096cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 10 monclient(hunting): get_auth_request con 0x7fb180067110 auth_method 0 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient(hunting): handle_auth_done global_id 84180 payload 824 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 msgr2=0x7fb180097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb180097200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180067110 msgr2=0x7fb180096cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180067110 0x7fb180096cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient: _finish_hunting 0 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 monclient: found mon.noname-a 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1862fa640 1 -- 172.21.15.116:0/1996821439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb170009390 con 0x7fb1801349a0 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.948+0000 7fb1852f8640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb18009b010 0x7fb180097200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:30:58.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.950+0000 7fb1862fa640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1801349a0 msgr2=0x7fb180097740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.950+0000 7fb1862fa640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1801349a0 0x7fb180097740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c570 tx=0x7fb178007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.950+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178005cb0 con 0x7fb1801349a0 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: dump: 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.965 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: _finish_auth 0 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb178005e30 con 0x7fb1801349a0 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 5 monclient: authenticate success, global_id 84180 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 10 monclient: _renew_subs 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18006ca30 con 0x7fb1801349a0 2024-09-18T12:30:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178007320 con 0x7fb1801349a0 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb175ffb640 10 monclient: dump: 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 10 monclient: _renew_subs 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb177fff640 4 set_mon_vals no callback set 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18006bb40 con 0x7fb1801349a0 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.951+0000 7fb187d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb187d5d640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 0x7fb18002aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb187d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18006ceb0 con 0x7fb180097c80 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb17801b3b0 con 0x7fb1801349a0 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 0x7fb18002aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb175ffb640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb164038490 0x7fb16403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient: get_auth_request con 0x7fb180097c80 auth_method 0 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fb17804b5e0 con 0x7fb1801349a0 2024-09-18T12:30:58.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 10 monclient(hunting): handle_auth_done global_id 5698 payload 824 2024-09-18T12:30:58.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb1852f8640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb164038490 0x7fb16403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb185af9640 1 -- 172.21.15.116:0/1996821439 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb180098750 con 0x7fb180097c80 2024-09-18T12:30:58.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.952+0000 7fb1852f8640 10 monclient: get_auth_request con 0x7fb164038490 auth_method 0 2024-09-18T12:30:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.954+0000 7fb185af9640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 msgr2=0x7fb18002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.954+0000 7fb185af9640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 0x7fb18002aa40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007fd0 tx=0x7fb17c002630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.954+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00dcd0 con 0x7fb180097c80 2024-09-18T12:30:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.954+0000 7fb175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.954+0000 7fb175ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.954+0000 7fb175ffb640 10 monclient: dump: 2024-09-18T12:30:58.973 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.954+0000 7fb1852f8640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb164038490 0x7fb16403a940 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb180099430 tx=0x7fb170032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.955+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb17c00de30 con 0x7fb180097c80 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.955+0000 7fb175ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:58.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.955+0000 7fb175ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.955+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 msgr2=0x7fb18002aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.955+0000 7fb175ffb640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 0x7fb18002aa40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007fd0 tx=0x7fb17c002630 comp rx=0 tx=0).stop 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.955+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 msgr2=0x7fb18002aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.955+0000 7fb175ffb640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb180097c80 0x7fb18002aa40 secure :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007fd0 tx=0x7fb17c002630 comp rx=0 tx=0).stop 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.960+0000 7fb187d5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.960+0000 7fb187d5d640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 0x7fb18005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.960+0000 7fb187d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.960+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb180065f90 con 0x7fb18005e310 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 0x7fb18005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient: get_auth_request con 0x7fb18005e310 auth_method 0 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:58.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 10 monclient(hunting): handle_auth_done global_id 5701 payload 824 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.961+0000 7fb185af9640 1 -- 172.21.15.116:0/1996821439 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb18004bf60 con 0x7fb18005e310 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb185af9640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 msgr2=0x7fb18005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb185af9640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 0x7fb18005e710 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007fd0 tx=0x7fb17c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c01ed20 con 0x7fb18005e310 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:30:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 10 monclient: dump: 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fb17c029410 con 0x7fb18005e310 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:30:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 msgr2=0x7fb18005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 0x7fb18005e710 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007fd0 tx=0x7fb17c017000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 msgr2=0x7fb18005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.963+0000 7fb175ffb640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb18005e310 0x7fb18005e710 secure :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb17c007fd0 tx=0x7fb17c017000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 66, 2024-09-18T12:30:58.978 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.979 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.981 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:30:58.982 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:30:58.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:30:58.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:30:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:30:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:30:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:30:58.993 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:30:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb164038490 msgr2=0x7fb16403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb164038490 0x7fb16403a940 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb180099430 tx=0x7fb170032000 comp rx=0 tx=0).stop 2024-09-18T12:30:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 10 monclient: shutdown 2024-09-18T12:30:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1801349a0 msgr2=0x7fb180097740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 --2- 172.21.15.116:0/1996821439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1801349a0 0x7fb180097740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c570 tx=0x7fb178007f70 comp rx=0 tx=0).stop 2024-09-18T12:30:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb1862fa640 1 -- 172.21.15.116:0/1996821439 reap_dead start 2024-09-18T12:30:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb1862fa640 1 -- 172.21.15.116:0/1996821439 reap_dead start 2024-09-18T12:30:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 shutdown_connections 2024-09-18T12:30:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 >> 172.21.15.116:0/1996821439 conn(0x7fb1800a1e30 msgr2=0x7fb180137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 shutdown_connections 2024-09-18T12:30:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:58.964+0000 7fb187d5d640 1 -- 172.21.15.116:0/1996821439 wait complete. 2024-09-18T12:30:58.996 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:30:58.996 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:30:58.997 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:30:58.997 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:30:58.997 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:30:58.997 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:30:58.997 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:30:59.007 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:30:59.007 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:30:59.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:30:59.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:59.053+0000 7f5538a78640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T12:30:59.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:59.053+0000 7f5538a78640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T12:30:59.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:59.053+0000 7f5538a78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:30:59.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:30:59.054+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f55340008d0 con 0x7f5534098c70 2024-09-18T12:30:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.139+0000 7f9e4aedd640 10 monclient: get_monmap_and_config 2024-09-18T12:30:59.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.139+0000 7f9e4aedd640 10 monclient: build_initial_monmap 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.139+0000 7f9e4aedd640 10 monclient: monmap: 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:30:59.140429+0000 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:30:59.140429+0000 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:30:59.141 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:59.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 1 Processor -- start 2024-09-18T12:30:59.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 1 -- start start 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient: init 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 0x7f9e44096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient(hunting): picked mon.noname-b con 0x7f9e4409a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e440971a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e44097570 0x7f9e4412fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient(hunting): picked mon.noname-a con 0x7f9e44097570 addr v2:172.21.15.116:3300/0 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4409cb00 con 0x7f9e440971a0 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e4409a8e0 con 0x7f9e44097570 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:59.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4406ed80 con 0x7f9e4409a510 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4aedd640 10 monclient(hunting): authenticate will time out at 1086.461961s 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e43fff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f9e440971a0 legacy=0x7f9e4412f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.140+0000 7f9e4947a640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f9e44097570 msgr2=0x7f9e4412fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e4947a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e44097570 0x7f9e4412fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 0x7f9e44096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 0x7f9e44096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47006/0 (socket says 172.21.15.116:47006) 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 -- 172.21.15.116:0/1251434566 learned_addr learned my addr 172.21.15.116:0/1251434566 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): get_auth_request con 0x7f9e4409a510 auth_method 0 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:59.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient(hunting): handle_auth_done global_id 64532 payload 274 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 -- 172.21.15.116:0/1251434566 >> v2:172.21.15.116:3300/0 conn(0x7f9e44097570 msgr2=0x7f9e4412fd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 --2- 172.21.15.116:0/1251434566 >> v2:172.21.15.116:3300/0 conn(0x7f9e44097570 0x7f9e4412fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 -- 172.21.15.116:0/1251434566 >> v1:172.21.15.116:6789/0 conn(0x7f9e440971a0 legacy=0x7f9e4412f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient: _finish_hunting 0 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 monclient: found mon.noname-b 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.141+0000 7f9e48c79640 1 -- 172.21.15.116:0/1251434566 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4409c820 con 0x7f9e4409a510 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e48c79640 1 -- 172.21.15.116:0/1251434566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 msgr2=0x7f9e44096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e48c79640 1 --2- 172.21.15.116:0/1251434566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 0x7f9e44096c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e4406ef70 tx=0x7f9e380091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52ad7bf12c938211 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:59.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 1 -- 172.21.15.116:0/1251434566 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38032440 con 0x7f9e4409a510 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: dump: 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:59.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: _finish_auth 0 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 1 -- 172.21.15.116:0/1251434566 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e380325a0 con 0x7f9e4409a510 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 5 monclient: authenticate success, global_id 64532 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 10 monclient: get_monmap_and_config success 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 1 -- 172.21.15.116:0/1251434566 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38032980 con 0x7f9e4409a510 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 4 set_mon_vals no callback set 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e437fe640 10 monclient: dump: 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:59.153 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 10 monclient: shutdown 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 -- 172.21.15.116:0/1251434566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 msgr2=0x7f9e44096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/1251434566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 0x7f9e44096c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e4406ef70 tx=0x7f9e380091d0 comp rx=0 tx=0).stop 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 -- 172.21.15.116:0/1251434566 shutdown_connections 2024-09-18T12:30:59.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/1251434566 >> v2:172.21.15.116:3300/0 conn(0x7f9e44097570 0x7f9e4412fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/1251434566 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e4409a510 0x7f9e44096c60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 -- 172.21.15.116:0/1251434566 >> 172.21.15.116:0/1251434566 conn(0x7f9e440672b0 msgr2=0x7f9e440676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 -- 172.21.15.116:0/1251434566 shutdown_connections 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.144+0000 7f9e4aedd640 1 -- 172.21.15.116:0/1251434566 wait complete. 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.145+0000 7f9e4aedd640 10 monclient: build_initial_monmap 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.145+0000 7f9e4aedd640 10 monclient: monmap: 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:30:59.155 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.145+0000 7f9e4aedd640 1 Processor -- start 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.145+0000 7f9e4aedd640 1 -- start start 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.145+0000 7f9e4aedd640 10 monclient: init 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient: _reopen_session rank -1 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 0x7f9e4413ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): picked mon.noname-c con 0x7f9e440971a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e44097570 0x7f9e4412f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): picked mon.noname-b con 0x7f9e44097570 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e4409a510 0x7f9e44119390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): picked mon.noname-a con 0x7f9e4409a510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:30:59.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4409c370 con 0x7f9e4409a510 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4406dba0 con 0x7f9e44097570 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4411cfc0 con 0x7f9e440971a0 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): _renew_subs 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4aedd640 10 monclient(hunting): authenticate will time out at 1086.467280s 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4947a640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e4409a510 msgr2=0x7f9e44119390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e4947a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e4409a510 0x7f9e44119390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 0x7f9e4413ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 0x7f9e4413ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47016/0 (socket says 172.21.15.116:47016) 2024-09-18T12:30:59.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 -- 172.21.15.116:0/2393052147 learned_addr learned my addr 172.21.15.116:0/2393052147 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): get_auth_request con 0x7f9e440971a0 auth_method 0 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e44097570 0x7f9e4412f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 10 monclient(hunting): get_auth_request con 0x7f9e44097570 auth_method 0 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:30:59.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient(hunting): handle_auth_done global_id 64535 payload 824 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 -- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e44097570 msgr2=0x7f9e4412f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e44097570 0x7f9e4412f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 -- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e4409a510 msgr2=0x7f9e44119390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e4409a510 0x7f9e44119390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient: _finish_hunting 0 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 monclient: found mon.noname-c 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:30:59.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.146+0000 7f9e48c79640 1 -- 172.21.15.116:0/2393052147 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e38009450 con 0x7f9e440971a0 2024-09-18T12:30:59.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.148+0000 7f9e48c79640 1 -- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 msgr2=0x7f9e4413ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.148+0000 7f9e48c79640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 0x7f9e4413ef60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9e38035040 tx=0x7f9e38032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 1 -- 172.21.15.116:0/2393052147 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3803b8f0 con 0x7f9e440971a0 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: dump: 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:59.162 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: _finish_auth 0 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 1 -- 172.21.15.116:0/2393052147 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3803ba50 con 0x7f9e440971a0 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 5 monclient: authenticate success, global_id 64535 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 10 monclient: _renew_subs 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4411b440 con 0x7f9e440971a0 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 1 -- 172.21.15.116:0/2393052147 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3803bdd0 con 0x7f9e440971a0 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:30:59.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 10 monclient: dump: 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 10 monclient: _renew_subs 2024-09-18T12:30:59.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4413fda0 con 0x7f9e440971a0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e42ffd640 4 set_mon_vals no callback set 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e40ff9640 1 -- 172.21.15.116:0/2393052147 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9e38039040 con 0x7f9e440971a0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.149+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4404bf60 con 0x7f9e440971a0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.150+0000 7f9e40ff9640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e28038440 0x7f9e2803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.150+0000 7f9e40ff9640 1 -- 172.21.15.116:0/2393052147 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f9e3806c260 con 0x7f9e440971a0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.150+0000 7f9e43fff640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e28038440 0x7f9e2803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.150+0000 7f9e43fff640 10 monclient: get_auth_request con 0x7f9e28038440 auth_method 0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.152+0000 7f9e43fff640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e28038440 0x7f9e2803a8f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9e30003940 tx=0x7f9e300090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:30:59.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.154+0000 7f9e48c79640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 0x7f9e4413ef60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9e38035040 tx=0x7f9e38032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f9e38030b70 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:30:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.982+0000 7f9e40ff9640 1 -- 172.21.15.116:0/2393052147 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9e38030b70 con 0x7f9e440971a0 2024-09-18T12:30:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.982+0000 7f9e40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:30:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:30:59.982+0000 7f9e40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.204+0000 7f9e4aedd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.204+0000 7f9e4aedd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.204+0000 7f9e4aedd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.204+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e4411b3e0 con 0x7f9e440971a0 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.204+0000 7f9e40ff9640 1 -- 172.21.15.116:0/2393052147 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f9e3803f080 con 0x7f9e440971a0 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.204+0000 7f9e40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.204+0000 7f9e40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:00.205 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:00.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e28038440 msgr2=0x7f9e2803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:00.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e28038440 0x7f9e2803a8f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9e30003940 tx=0x7f9e300090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:00.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 10 monclient: shutdown 2024-09-18T12:31:00.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:00.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 msgr2=0x7f9e4413ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 0x7f9e4413ef60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9e38035040 tx=0x7f9e38032000 comp rx=0 tx=0).stop 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 shutdown_connections 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e28038440 0x7f9e2803a8f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e4409a510 0x7f9e44119390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e44097570 0x7f9e4412f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 --2- 172.21.15.116:0/2393052147 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e440971a0 0x7f9e4413ef60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 >> 172.21.15.116:0/2393052147 conn(0x7f9e440672b0 msgr2=0x7f9e4409fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 shutdown_connections 2024-09-18T12:31:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:00.206+0000 7f9e4aedd640 1 -- 172.21.15.116:0/2393052147 wait complete. 2024-09-18T12:31:01.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:01.883+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:01.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:01.883+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:01.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:01.883+0000 7f55297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:02.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:02.884+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:02.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:02.884+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:03.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.320+0000 7ff33935d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.320+0000 7ff33935d640 10 monclient: build_initial_monmap 2024-09-18T12:31:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.320+0000 7ff33935d640 10 monclient: monmap: 2024-09-18T12:31:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:03.321074+0000 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:03.321074+0000 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:03.322 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 1 Processor -- start 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 1 -- start start 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient: init 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3341318f0 0x7ff334131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3341318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff33409ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:03.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 0x7ff334139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:03.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient(hunting): picked mon.noname-b con 0x7ff334136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:03.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff334131cc0 con 0x7ff33409ea50 2024-09-18T12:31:03.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:03.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33409f530 con 0x7ff3341318f0 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33409fbf0 con 0x7ff334136c20 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff33935d640 10 monclient(hunting): authenticate will time out at 1090.642848s 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff32bfff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7ff33409ea50 legacy=0x7ff33409ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.321+0000 7ff3337fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 0x7ff334139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 0x7ff334139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47028/0 (socket says 172.21.15.116:47028) 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 1 -- 172.21.15.116:0/2787151852 learned_addr learned my addr 172.21.15.116:0/2787151852 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): get_auth_request con 0x7ff334136c20 auth_method 0 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:03.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 1 --2- 172.21.15.116:0/2787151852 >> v2:172.21.15.144:3300/0 conn(0x7ff3341318f0 0x7ff334131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 10 monclient(hunting): get_auth_request con 0x7ff3341318f0 auth_method 0 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient(hunting): handle_auth_done global_id 64538 payload 274 2024-09-18T12:31:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 1 -- 172.21.15.116:0/2787151852 >> v2:172.21.15.144:3300/0 conn(0x7ff3341318f0 msgr2=0x7ff334131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 1 --2- 172.21.15.116:0/2787151852 >> v2:172.21.15.144:3300/0 conn(0x7ff3341318f0 0x7ff334131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 1 -- 172.21.15.116:0/2787151852 >> v1:172.21.15.116:6789/0 conn(0x7ff33409ea50 legacy=0x7ff33409ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.322+0000 7ff3337fe640 1 -- 172.21.15.116:0/2787151852 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33409fe30 con 0x7ff334136c20 2024-09-18T12:31:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3337fe640 1 -- 172.21.15.116:0/2787151852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 msgr2=0x7ff334139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3337fe640 1 --2- 172.21.15.116:0/2787151852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 0x7ff334139000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009f20 tx=0x7ff32c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f955d5e87f49a099 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 1 -- 172.21.15.116:0/2787151852 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c007610 con 0x7ff334136c20 2024-09-18T12:31:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: dump: 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 5 monclient: authenticate success, global_id 64538 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 1 -- 172.21.15.116:0/2787151852 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c02d050 con 0x7ff334136c20 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 1 -- 172.21.15.116:0/2787151852 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0079e0 con 0x7ff334136c20 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 4 set_mon_vals no callback set 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff3327fc640 10 monclient: dump: 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 10 monclient: shutdown 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 -- 172.21.15.116:0/2787151852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 msgr2=0x7ff334139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 --2- 172.21.15.116:0/2787151852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 0x7ff334139000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009f20 tx=0x7ff32c009840 comp rx=0 tx=0).stop 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 -- 172.21.15.116:0/2787151852 shutdown_connections 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 --2- 172.21.15.116:0/2787151852 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff334136c20 0x7ff334139000 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 --2- 172.21.15.116:0/2787151852 >> v2:172.21.15.144:3300/0 conn(0x7ff3341318f0 0x7ff334131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 -- 172.21.15.116:0/2787151852 >> 172.21.15.116:0/2787151852 conn(0x7ff334067110 msgr2=0x7ff334067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 -- 172.21.15.116:0/2787151852 shutdown_connections 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.325+0000 7ff33935d640 1 -- 172.21.15.116:0/2787151852 wait complete. 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.326+0000 7ff33935d640 10 monclient: build_initial_monmap 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.326+0000 7ff33935d640 10 monclient: monmap: 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:03.340 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.326+0000 7ff33935d640 1 Processor -- start 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.326+0000 7ff33935d640 1 -- start start 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.326+0000 7ff33935d640 10 monclient: init 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 0x7ff33411da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): picked mon.noname-c con 0x7ff33409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3341318f0 0x7ff3341412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:03.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3341318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334136c20 0x7ff334141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): picked mon.noname-b con 0x7ff334136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33411ce90 con 0x7ff3341318f0 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33409bf40 con 0x7ff334136c20 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33409b5c0 con 0x7ff33409ea50 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff33935d640 10 monclient(hunting): authenticate will time out at 1090.648317s 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff32bfff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3341318f0 msgr2=0x7ff3341412c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:03.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff32bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3341318f0 0x7ff3341412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 0x7ff33411da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 0x7ff33411da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47030/0 (socket says 172.21.15.116:47030) 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 -- 172.21.15.116:0/959372372 learned_addr learned my addr 172.21.15.116:0/959372372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): get_auth_request con 0x7ff33409ea50 auth_method 0 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff3337fe640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334136c20 0x7ff334141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff3337fe640 10 monclient(hunting): get_auth_request con 0x7ff334136c20 auth_method 0 2024-09-18T12:31:03.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff3337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff3337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff3337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient(hunting): handle_auth_done global_id 64541 payload 824 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 -- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334136c20 msgr2=0x7ff334141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334136c20 0x7ff334141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 -- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3341318f0 msgr2=0x7ff3341412c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3341318f0 0x7ff3341412c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 monclient: found mon.noname-c 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff332ffd640 1 -- 172.21.15.116:0/959372372 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c009ae0 con 0x7ff33409ea50 2024-09-18T12:31:03.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.327+0000 7ff3337fe640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334136c20 0x7ff334141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.329+0000 7ff332ffd640 1 -- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 msgr2=0x7ff33411da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff332ffd640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 0x7ff33411da90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff32400ea20 tx=0x7ff3240090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 1 -- 172.21.15.116:0/959372372 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32400cc20 con 0x7ff33409ea50 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: dump: 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:03.345 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: _finish_auth 0 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 1 -- 172.21.15.116:0/959372372 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32400cd80 con 0x7ff33409ea50 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff33935d640 5 monclient: authenticate success, global_id 64541 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 1 -- 172.21.15.116:0/959372372 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324013680 con 0x7ff33409ea50 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff32affd640 10 monclient: dump: 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:03.346 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff33935d640 10 monclient: _renew_subs 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff33935d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33409b110 con 0x7ff33409ea50 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff331ffb640 4 set_mon_vals no callback set 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff33935d640 10 monclient: _renew_subs 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff33935d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:03.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.330+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff334141ee0 con 0x7ff33409ea50 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff32affd640 1 -- 172.21.15.116:0/959372372 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff324013a70 con 0x7ff33409ea50 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff32affd640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff314038440 0x7ff31403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff32affd640 1 -- 172.21.15.116:0/959372372 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7ff32404c570 con 0x7ff33409ea50 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff33935d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff33935d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff33935d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33404bf60 con 0x7ff33409ea50 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff32bfff640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff314038440 0x7ff31403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.331+0000 7ff32bfff640 10 monclient: get_auth_request con 0x7ff314038440 auth_method 0 2024-09-18T12:31:03.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.334+0000 7ff32bfff640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff314038440 0x7ff31403a8f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff31c003940 tx=0x7ff31c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:03.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.336+0000 7ff32affd640 1 -- 172.21.15.116:0/959372372 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff3240123d0 con 0x7ff33409ea50 2024-09-18T12:31:03.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.336+0000 7ff32affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:03.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.336+0000 7ff32affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.551+0000 7ff33935d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.551+0000 7ff33935d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.551+0000 7ff33935d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.551+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff3341316d0 con 0x7ff33409ea50 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.552+0000 7ff32affd640 1 -- 172.21.15.116:0/959372372 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7ff324023760 con 0x7ff33409ea50 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.552+0000 7ff32affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.552+0000 7ff32affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:03.553 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff314038440 msgr2=0x7ff31403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff314038440 0x7ff31403a8f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff31c003940 tx=0x7ff31c006d80 comp rx=0 tx=0).stop 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 10 monclient: shutdown 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 msgr2=0x7ff33411da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 0x7ff33411da90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff32400ea20 tx=0x7ff3240090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 shutdown_connections 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff314038440 0x7ff31403a8f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff334136c20 0x7ff334141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3341318f0 0x7ff3341412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 --2- 172.21.15.116:0/959372372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff33409ea50 0x7ff33411da90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:03.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 >> 172.21.15.116:0/959372372 conn(0x7ff334067110 msgr2=0x7ff3340a3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:03.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 shutdown_connections 2024-09-18T12:31:03.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:03.554+0000 7ff33935d640 1 -- 172.21.15.116:0/959372372 wait complete. 2024-09-18T12:31:03.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:03.884+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:03.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:03.884+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:04.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:04.884+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:04.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:04.884+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:05.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:05.885+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:05.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:05.885+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:06.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.670+0000 7fc591d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.670+0000 7fc591d1d640 10 monclient: build_initial_monmap 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.670+0000 7fc591d1d640 10 monclient: monmap: 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:06.670761+0000 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:06.670761+0000 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:06.672 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:06.672 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:06.672 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:06.672 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:06.672 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:06.672 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:06.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 1 Processor -- start 2024-09-18T12:31:06.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 1 -- start start 2024-09-18T12:31:06.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient: init 2024-09-18T12:31:06.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T12:31:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc58c1318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc58c136c20 0x7fc58c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fc58c136c20 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c132640 con 0x7fc58c1318f0 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c131d00 con 0x7fc58c136c20 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c09fbf0 con 0x7fc58c09ea50 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc591d1d640 10 monclient(hunting): authenticate will time out at 1093.992636s 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc58b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fc58c1318f0 legacy=0x7fc58c131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc58bfff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fc58c136c20 msgr2=0x7fc58c139000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc58bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc58c136c20 0x7fc58c139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57976/0 (socket says 172.21.15.116:57976) 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 1 -- 172.21.15.116:0/2361192369 learned_addr learned my addr 172.21.15.116:0/2361192369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): get_auth_request con 0x7fc58c09ea50 auth_method 0 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 10 monclient(hunting): handle_auth_done global_id 64544 payload 274 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 1 -- 172.21.15.116:0/2361192369 >> v2:172.21.15.116:3300/0 conn(0x7fc58c136c20 msgr2=0x7fc58c139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.671+0000 7fc583fff640 1 --2- 172.21.15.116:0/2361192369 >> v2:172.21.15.116:3300/0 conn(0x7fc58c136c20 0x7fc58c139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.672+0000 7fc583fff640 1 -- 172.21.15.116:0/2361192369 >> v1:172.21.15.116:6789/0 conn(0x7fc58c1318f0 legacy=0x7fc58c131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.672+0000 7fc583fff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.672+0000 7fc583fff640 1 monclient: found mon.noname-b 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.672+0000 7fc583fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.672+0000 7fc583fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.672+0000 7fc583fff640 1 -- 172.21.15.116:0/2361192369 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c09fe30 con 0x7fc58c09ea50 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.674+0000 7fc583fff640 1 -- 172.21.15.116:0/2361192369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 msgr2=0x7fc58c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.674+0000 7fc583fff640 1 --2- 172.21.15.116:0/2361192369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c09ee20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc574009910 tx=0x7fc5740091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1139282aba13af1a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.674+0000 7fc58affd640 1 -- 172.21.15.116:0/2361192369 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574032440 con 0x7fc58c09ea50 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.674+0000 7fc58affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.674+0000 7fc58affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.674+0000 7fc58affd640 10 monclient: dump: 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.674+0000 7fc58affd640 10 monclient: _finish_auth 0 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc58affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc58affd640 1 -- 172.21.15.116:0/2361192369 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5740325a0 con 0x7fc58c09ea50 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 5 monclient: authenticate success, global_id 64544 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc58affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc58affd640 1 -- 172.21.15.116:0/2361192369 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574032940 con 0x7fc58c09ea50 2024-09-18T12:31:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc58affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 4 set_mon_vals no callback set 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc58affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc58affd640 10 monclient: dump: 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.685 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 10 monclient: shutdown 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 -- 172.21.15.116:0/2361192369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 msgr2=0x7fc58c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 --2- 172.21.15.116:0/2361192369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c09ee20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc574009910 tx=0x7fc5740091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 -- 172.21.15.116:0/2361192369 shutdown_connections 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 --2- 172.21.15.116:0/2361192369 >> v2:172.21.15.116:3300/0 conn(0x7fc58c136c20 0x7fc58c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 --2- 172.21.15.116:0/2361192369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c09ee20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 -- 172.21.15.116:0/2361192369 >> 172.21.15.116:0/2361192369 conn(0x7fc58c067110 msgr2=0x7fc58c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 -- 172.21.15.116:0/2361192369 shutdown_connections 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.675+0000 7fc591d1d640 1 -- 172.21.15.116:0/2361192369 wait complete. 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient: build_initial_monmap 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient: monmap: 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 Processor -- start 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 -- start start 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient: init 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c0974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fc58c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc58c1318f0 0x7fc58c0979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fc58c1318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc58c136c20 0x7fc58c097f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c099990 con 0x7fc58c1318f0 2024-09-18T12:31:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c09bf10 con 0x7fc58c136c20 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c09db10 con 0x7fc58c09ea50 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc591d1d640 10 monclient(hunting): authenticate will time out at 1093.997973s 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.676+0000 7fc583fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc58c1318f0 msgr2=0x7fc58c0979e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc583fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc58c1318f0 0x7fc58c0979e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c0974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c0974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57982/0 (socket says 172.21.15.116:57982) 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 -- 172.21.15.116:0/3169434222 learned_addr learned my addr 172.21.15.116:0/3169434222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): get_auth_request con 0x7fc58c09ea50 auth_method 0 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58bfff640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc58c136c20 0x7fc58c097f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58bfff640 10 monclient(hunting): get_auth_request con 0x7fc58c136c20 auth_method 0 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_done global_id 64547 payload 824 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 -- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc58c136c20 msgr2=0x7fc58c097f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc58c136c20 0x7fc58c097f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 -- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc58c1318f0 msgr2=0x7fc58c0979e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc58c1318f0 0x7fc58c0979e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:31:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:06.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.677+0000 7fc58b7fe640 1 -- 172.21.15.116:0/3169434222 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574009450 con 0x7fc58c09ea50 2024-09-18T12:31:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc58b7fe640 1 -- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 msgr2=0x7fc58c0974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc58b7fe640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c0974a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00b600 tx=0x7fc57c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 1 -- 172.21.15.116:0/3169434222 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c015070 con 0x7fc58c09ea50 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: dump: 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 1 -- 172.21.15.116:0/3169434222 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c003d90 con 0x7fc58c09ea50 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc591d1d640 5 monclient: authenticate success, global_id 64547 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc591d1d640 10 monclient: _renew_subs 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc591d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c0992d0 con 0x7fc58c09ea50 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 1 -- 172.21.15.116:0/3169434222 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c006380 con 0x7fc58c09ea50 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc5837fe640 10 monclient: dump: 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc58a7fc640 4 set_mon_vals no callback set 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc591d1d640 10 monclient: _renew_subs 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc591d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.679+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c09ae40 con 0x7fc58c09ea50 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.680+0000 7fc5837fe640 1 -- 172.21.15.116:0/3169434222 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc57c010440 con 0x7fc58c09ea50 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.680+0000 7fc591d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.680+0000 7fc591d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.680+0000 7fc591d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.680+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c04bf60 con 0x7fc58c09ea50 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.680+0000 7fc5837fe640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc56c038490 0x7fc56c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.680+0000 7fc5837fe640 1 -- 172.21.15.116:0/3169434222 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fc57c04bac0 con 0x7fc58c09ea50 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.681+0000 7fc583fff640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc56c038490 0x7fc56c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.681+0000 7fc583fff640 10 monclient: get_auth_request con 0x7fc56c038490 auth_method 0 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.683+0000 7fc583fff640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc56c038490 0x7fc56c03a940 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc58c076600 tx=0x7fc574032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:06.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.685+0000 7fc5837fe640 1 -- 172.21.15.116:0/3169434222 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc57c006770 con 0x7fc58c09ea50 2024-09-18T12:31:06.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.685+0000 7fc5837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:06.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.685+0000 7fc5837fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:06.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:06.884+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:06.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:06.884+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.897+0000 7fc591d1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.897+0000 7fc591d1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.897+0000 7fc591d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.897+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc58c0638d0 con 0x7fc58c09ea50 2024-09-18T12:31:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.897+0000 7fc5837fe640 1 -- 172.21.15.116:0/3169434222 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fc57c018780 con 0x7fc58c09ea50 2024-09-18T12:31:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.897+0000 7fc5837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.897+0000 7fc5837fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:06.899 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc56c038490 msgr2=0x7fc56c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc56c038490 0x7fc56c03a940 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc58c076600 tx=0x7fc574032000 comp rx=0 tx=0).stop 2024-09-18T12:31:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 10 monclient: shutdown 2024-09-18T12:31:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 msgr2=0x7fc58c0974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c0974a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00b600 tx=0x7fc57c0091a0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 shutdown_connections 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc56c038490 0x7fc56c03a940 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc58c136c20 0x7fc58c097f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc58c1318f0 0x7fc58c0979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 --2- 172.21.15.116:0/3169434222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc58c09ea50 0x7fc58c0974a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 >> 172.21.15.116:0/3169434222 conn(0x7fc58c067110 msgr2=0x7fc58c0a1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 shutdown_connections 2024-09-18T12:31:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:06.899+0000 7fc591d1d640 1 -- 172.21.15.116:0/3169434222 wait complete. 2024-09-18T12:31:07.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:07.885+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:07.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:07.885+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:08.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:08.884+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:08.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:08.884+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:09.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:09.884+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:09.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:09.884+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:09.911 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.014+0000 7fc1f36f8640 10 monclient: get_monmap_and_config 2024-09-18T12:31:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.014+0000 7fc1f36f8640 10 monclient: build_initial_monmap 2024-09-18T12:31:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.014+0000 7fc1f36f8640 10 monclient: monmap: 2024-09-18T12:31:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:10.014903+0000 2024-09-18T12:31:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:10.014903+0000 2024-09-18T12:31:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:10.017 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:10.017 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:10.017 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:10.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.014+0000 7fc1f36f8640 1 Processor -- start 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 1 -- start start 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient: init 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient(hunting): picked mon.noname-c con 0x7fc1ec09ad90 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1ec09b160 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1ec09b530 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ec0692c0 con 0x7fc1ec09b160 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ec09b990 con 0x7fc1ec09b530 2024-09-18T12:31:10.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec09db70 con 0x7fc1ec09ad90 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f36f8640 10 monclient(hunting): authenticate will time out at 1097.336559s 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f0c93640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fc1ec09b160 legacy=0x7fc1ec131960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.015+0000 7fc1f1494640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:34300/0 (socket says 172.21.15.116:34300) 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 learned_addr learned my addr 172.21.15.116:0/774658593 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1ec09ad90 auth_method 0 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.016+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.018+0000 7fc1f1c95640 1 -- 172.21.15.116:0/774658593 >> v1:172.21.15.144:6789/0 conn(0x7fc1ec09b530 legacy=0x7fc1ec132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.216+0000 7fc1f0c93640 1 -- 172.21.15.116:0/774658593 >> v1:172.21.15.116:6789/0 conn(0x7fc1ec09b160 legacy=0x7fc1ec131960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1ec09ad90 auth_method 0 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.217+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:31:10.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.617+0000 7fc1f0c93640 1 -- 172.21.15.116:0/774658593 >> v1:172.21.15.116:6789/0 conn(0x7fc1ec09b160 legacy=0x7fc1ec131960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1ec09ad90 auth_method 0 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:10.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:10.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:10.618+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:31:10.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:10.885+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:10.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:10.885+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.418+0000 7fc1f0c93640 1 -- 172.21.15.116:0/774658593 >> v1:172.21.15.116:6789/0 conn(0x7fc1ec09b160 legacy=0x7fc1ec131960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:11.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:11.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1ec09ad90 auth_method 0 2024-09-18T12:31:11.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:11.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:11.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:11.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:31:11.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:11.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:11.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:11.419+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:31:11.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:11.886+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:11.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:11.886+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:11.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:11.886+0000 7f55297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:12.886+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:12.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:12.886+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:13.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient(hunting): tick 2024-09-18T12:31:13.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 monclient(hunting): continuing hunt 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 msgr2=0x7fc1ec09c5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.144:3300/0 conn(0x7fc1ec09ad90 0x7fc1ec09c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 -- 172.21.15.116:0/774658593 >> v1:172.21.15.144:6789/0 conn(0x7fc1ec09b530 legacy=0x7fc1ec132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 -- 172.21.15.116:0/774658593 >> v1:172.21.15.116:6789/0 conn(0x7fc1ec09b160 legacy=0x7fc1ec131960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient: _add_conns ranks=[2,3] 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fc1d8003700 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 0x7fc1d8008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d8006200 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 -- 172.21.15.116:0/774658593 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ec09db70 con 0x7fc1d8006200 2024-09-18T12:31:13.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 1 -- 172.21.15.116:0/774658593 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d8008d50 con 0x7fc1d8003700 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1eaffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f0c93640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 msgr2=0x7fc1d8008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f0c93640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 0x7fc1d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1d8003700 auth_method 0 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 msgr2=0x7fc1d8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 msgr2=0x7fc1d8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.016+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:13.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f0c93640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 msgr2=0x7fc1d8008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:13.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f0c93640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 0x7fc1d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:31:13.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1d8003700 auth_method 0 2024-09-18T12:31:13.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:13.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 msgr2=0x7fc1d8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:31:13.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 msgr2=0x7fc1d8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:13.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:13.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.217+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:31:13.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f0c93640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 msgr2=0x7fc1d8008690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f0c93640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 0x7fc1d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1d8003700 auth_method 0 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient(hunting): handle_auth_done global_id 64550 payload 274 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 msgr2=0x7fc1d8008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> v2:172.21.15.116:3300/0 conn(0x7fc1d8006200 0x7fc1d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient: _finish_hunting 0 2024-09-18T12:31:13.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 1 monclient: found mon.noname-b 2024-09-18T12:31:13.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:13.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.618+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec09d5b0 con 0x7fc1d8003700 2024-09-18T12:31:13.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1f1494640 1 -- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 msgr2=0x7fc1d8005b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:13.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1f1494640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4006fd0 tx=0x7fc1e4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84c28743eb74f7bb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 1 -- 172.21.15.116:0/774658593 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4031730 con 0x7fc1d8003700 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: dump: 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.623 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: _finish_auth 0 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 1 -- 172.21.15.116:0/774658593 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e400c040 con 0x7fc1d8003700 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1f36f8640 5 monclient: authenticate success, global_id 64550 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1f36f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1f36f8640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 1 -- 172.21.15.116:0/774658593 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4031ae0 con 0x7fc1d8003700 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1f36f8640 4 set_mon_vals no callback set 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.621+0000 7fc1ebfff640 10 monclient: dump: 2024-09-18T12:31:13.624 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.625 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 10 monclient: shutdown 2024-09-18T12:31:13.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:13.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 1 -- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 msgr2=0x7fc1d8005b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 1 --2- 172.21.15.116:0/774658593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1d8003700 0x7fc1d8005b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4006fd0 tx=0x7fc1e4008040 comp rx=0 tx=0).stop 2024-09-18T12:31:13.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f1c95640 1 -- 172.21.15.116:0/774658593 reap_dead start 2024-09-18T12:31:13.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 1 -- 172.21.15.116:0/774658593 shutdown_connections 2024-09-18T12:31:13.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 1 -- 172.21.15.116:0/774658593 >> 172.21.15.116:0/774658593 conn(0x7fc1ec0678a0 msgr2=0x7fc1ec0a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:13.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 1 -- 172.21.15.116:0/774658593 shutdown_connections 2024-09-18T12:31:13.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.622+0000 7fc1f36f8640 1 -- 172.21.15.116:0/774658593 wait complete. 2024-09-18T12:31:13.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.623+0000 7fc1f36f8640 10 monclient: build_initial_monmap 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.623+0000 7fc1f36f8640 10 monclient: monmap: 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:13.634 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.623+0000 7fc1f36f8640 1 Processor -- start 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.623+0000 7fc1f36f8640 1 -- start start 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.623+0000 7fc1f36f8640 10 monclient: init 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:31:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1ec141010 0x7fc1ec130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): picked mon.noname-b con 0x7fc1ec141010 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 0x7fc1ec127810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): picked mon.noname-c con 0x7fc1ec1413e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec131180 0x7fc1ec127d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): picked mon.noname-a con 0x7fc1ec131180 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec12b480 con 0x7fc1ec131180 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec12ae30 con 0x7fc1ec141010 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec06f020 con 0x7fc1ec1413e0 2024-09-18T12:31:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f36f8640 10 monclient(hunting): authenticate will time out at 1100.945548s 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1c95640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec131180 msgr2=0x7fc1ec127d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1c95640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec131180 0x7fc1ec127d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 0x7fc1ec127810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 0x7fc1ec127810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58016/0 (socket says 172.21.15.116:58016) 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 1 -- 172.21.15.116:0/2097854237 learned_addr learned my addr 172.21.15.116:0/2097854237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): get_auth_request con 0x7fc1ec1413e0 auth_method 0 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1ec141010 0x7fc1ec130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 10 monclient(hunting): get_auth_request con 0x7fc1ec141010 auth_method 0 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f1494640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.624+0000 7fc1f0c93640 10 monclient(hunting): handle_auth_done global_id 64553 payload 824 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 1 -- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1ec141010 msgr2=0x7fc1ec130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1ec141010 0x7fc1ec130c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 1 -- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec131180 msgr2=0x7fc1ec127d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec131180 0x7fc1ec127d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 10 monclient: _finish_hunting 0 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 1 monclient: found mon.noname-c 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.625+0000 7fc1f0c93640 1 -- 172.21.15.116:0/2097854237 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4010780 con 0x7fc1ec1413e0 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f0c93640 1 -- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 msgr2=0x7fc1ec127810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f0c93640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 0x7fc1ec127810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000ce60 tx=0x7fc1e00076f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 1 -- 172.21.15.116:0/2097854237 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000aea0 con 0x7fc1ec1413e0 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: dump: 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.640 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: _finish_auth 0 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 1 -- 172.21.15.116:0/2097854237 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e000d600 con 0x7fc1ec1413e0 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f36f8640 5 monclient: authenticate success, global_id 64553 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 1 -- 172.21.15.116:0/2097854237 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000d9a0 con 0x7fc1ec1413e0 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f36f8640 10 monclient: _renew_subs 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f36f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec1418d0 con 0x7fc1ec1413e0 2024-09-18T12:31:13.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1e97fa640 10 monclient: dump: 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.642 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1eb7fe640 4 set_mon_vals no callback set 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f36f8640 10 monclient: _renew_subs 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f36f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.627+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec129370 con 0x7fc1ec1413e0 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.628+0000 7fc1e97fa640 1 -- 172.21.15.116:0/2097854237 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc1e0013070 con 0x7fc1ec1413e0 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1f36f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1f36f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1f36f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1e97fa640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc1d8039210 0x7fc1d803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec131920 con 0x7fc1ec1413e0 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1e97fa640 1 -- 172.21.15.116:0/2097854237 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(131..131 src has 13..131) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fc1e004ce10 con 0x7fc1ec1413e0 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1f1494640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc1d8039210 0x7fc1d803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.629+0000 7fc1f1494640 10 monclient: get_auth_request con 0x7fc1d8039210 auth_method 0 2024-09-18T12:31:13.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.632+0000 7fc1f1494640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc1d8039210 0x7fc1d803b6c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4034000 tx=0x7fc1e403a000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:13.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.633+0000 7fc1e97fa640 1 -- 172.21.15.116:0/2097854237 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc1e0013530 con 0x7fc1ec1413e0 2024-09-18T12:31:13.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.633+0000 7fc1e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:13.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.633+0000 7fc1e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:13.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.791+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v132) v1 ==== 109+0+0 (secure 0 0 0) 0x7f551c01a360 con 0x7f5534098c70 2024-09-18T12:31:13.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.791+0000 7f552a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T12:31:13.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.791+0000 7f552a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:13.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.858+0000 7fc1f36f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:13.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.858+0000 7fc1f36f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:13.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.858+0000 7fc1f36f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.858+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc1ec04bf60 con 0x7fc1ec1413e0 2024-09-18T12:31:13.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.858+0000 7fc1e97fa640 1 -- 172.21.15.116:0/2097854237 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fc1e001c020 con 0x7fc1ec1413e0 2024-09-18T12:31:13.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.858+0000 7fc1e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:13.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.858+0000 7fc1e97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:13.860 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":68,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:13.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc1d8039210 msgr2=0x7fc1d803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc1d8039210 0x7fc1d803b6c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4034000 tx=0x7fc1e403a000 comp rx=0 tx=0).stop 2024-09-18T12:31:13.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 10 monclient: shutdown 2024-09-18T12:31:13.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:13.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 msgr2=0x7fc1ec127810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 0x7fc1ec127810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000ce60 tx=0x7fc1e00076f0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 shutdown_connections 2024-09-18T12:31:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc1d8039210 0x7fc1d803b6c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.860+0000 7fc1f36f8640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec131180 0x7fc1ec127d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.861+0000 7fc1f36f8640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc1ec1413e0 0x7fc1ec127810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.861+0000 7fc1f36f8640 1 --2- 172.21.15.116:0/2097854237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc1ec141010 0x7fc1ec130c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.861+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 >> 172.21.15.116:0/2097854237 conn(0x7fc1ec0678a0 msgr2=0x7fc1ec131b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.861+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 shutdown_connections 2024-09-18T12:31:13.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.861+0000 7fc1f36f8640 1 -- 172.21.15.116:0/2097854237 wait complete. 2024-09-18T12:31:13.872 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:31:13.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:31:13.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:31:13.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.884+0000 7f55297fa640 10 monclient: tick 2024-09-18T12:31:13.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.884+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:13.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.896+0000 7f5538a78640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T12:31:13.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.896+0000 7f5538a78640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T12:31:13.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.896+0000 7f5538a78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:13.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.896+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f5534064a20 con 0x7f5534098c70 2024-09-18T12:31:13.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.896+0000 7f552a7fc640 1 -- 172.21.15.144:0/2695433451 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v132) v1 ==== 153+0+0 (secure 0 0 0) 0x7f551c01a540 con 0x7f5534098c70 2024-09-18T12:31:13.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.896+0000 7f552a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T12:31:13.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.896+0000 7f552a7fc640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-18T12:31:13.899 INFO:tasks.workunit.client.0.smithi144.stderr:weight-set for pool 'fooset' already created 2024-09-18T12:31:13.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5514038490 msgr2=0x7f551403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5514038490 0x7f551403a940 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f553411fe90 tx=0x7f5524008040 comp rx=0 tx=0).stop 2024-09-18T12:31:13.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 10 monclient: shutdown 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 msgr2=0x7f553411e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 0x7f553411e700 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c5b0 tx=0x7f551c007f00 comp rx=0 tx=0).stop 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 shutdown_connections 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5514038490 0x7f551403a940 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534137200 0x7f553411f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534099c70 0x7f553411ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 --2- 172.21.15.144:0/2695433451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534098c70 0x7f553411e700 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 >> 172.21.15.144:0/2695433451 conn(0x7f5534070e90 msgr2=0x7f5534097380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 shutdown_connections 2024-09-18T12:31:13.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.898+0000 7f5538a78640 1 -- 172.21.15.144:0/2695433451 wait complete. 2024-09-18T12:31:13.910 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create barset positional 2024-09-18T12:31:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.966+0000 7f633c08b640 10 monclient: get_monmap_and_config 2024-09-18T12:31:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.966+0000 7f633c08b640 10 monclient: build_initial_monmap 2024-09-18T12:31:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.966+0000 7f633c08b640 10 monclient: monmap: 2024-09-18T12:31:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:13.967232+0000 2024-09-18T12:31:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:13.967232+0000 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:13.968 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 1 Processor -- start 2024-09-18T12:31:13.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 1 -- start start 2024-09-18T12:31:13.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient: init 2024-09-18T12:31:13.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:13.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T12:31:13.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f633409c8b0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:13.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 0x7f6334133920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient(hunting): picked mon.noname-b con 0x7f633409d440 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6334096f20 0x7f6334135e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient(hunting): picked mon.noname-a con 0x7f6334096f20 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6334098300 con 0x7f633409c8b0 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633409cc80 con 0x7f6334096f20 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6334099220 con 0x7f633409d440 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633c08b640 10 monclient(hunting): authenticate will time out at 1101.288793s 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633a628640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f6334096f20 msgr2=0x7f6334135e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f633a628640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6334096f20 0x7f6334135e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:13.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f6339626640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 0x7f6334133920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f6339626640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 0x7f6334133920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58028/0 (socket says 172.21.15.116:58028) 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f6339626640 1 -- 172.21.15.116:0/552440293 learned_addr learned my addr 172.21.15.116:0/552440293 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f6339626640 10 monclient(hunting): get_auth_request con 0x7f633409d440 auth_method 0 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f6339626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f6339626640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.967+0000 7f6339626640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 10 monclient(hunting): handle_auth_done global_id 64556 payload 274 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 1 -- 172.21.15.116:0/552440293 >> v2:172.21.15.116:3300/0 conn(0x7f6334096f20 msgr2=0x7f6334135e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 1 --2- 172.21.15.116:0/552440293 >> v2:172.21.15.116:3300/0 conn(0x7f6334096f20 0x7f6334135e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 1 -- 172.21.15.116:0/552440293 >> v1:172.21.15.144:6789/0 conn(0x7f633409c8b0 legacy=0x7f633409aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 10 monclient: _finish_hunting 0 2024-09-18T12:31:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 1 monclient: found mon.noname-b 2024-09-18T12:31:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.968+0000 7f6339626640 1 -- 172.21.15.116:0/552440293 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334098c60 con 0x7f633409d440 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.970+0000 7f6339626640 1 -- 172.21.15.116:0/552440293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 msgr2=0x7f6334133920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.970+0000 7f6339626640 1 --2- 172.21.15.116:0/552440293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 0x7f6334133920 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6324009910 tx=0x7f63240091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b811d7fa56b067e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 1 -- 172.21.15.116:0/552440293 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324032440 con 0x7f633409d440 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: dump: 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: _finish_auth 0 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 1 -- 172.21.15.116:0/552440293 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63240325a0 con 0x7f633409d440 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 5 monclient: authenticate success, global_id 64556 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:13.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 1 -- 172.21.15.116:0/552440293 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324032940 con 0x7f633409d440 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 4 set_mon_vals no callback set 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f6338e25640 10 monclient: dump: 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.983 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 10 monclient: shutdown 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 -- 172.21.15.116:0/552440293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 msgr2=0x7f6334133920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 --2- 172.21.15.116:0/552440293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 0x7f6334133920 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6324009910 tx=0x7f63240091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 -- 172.21.15.116:0/552440293 shutdown_connections 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 --2- 172.21.15.116:0/552440293 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409d440 0x7f6334133920 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 --2- 172.21.15.116:0/552440293 >> v2:172.21.15.116:3300/0 conn(0x7f6334096f20 0x7f6334135e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 -- 172.21.15.116:0/552440293 >> 172.21.15.116:0/552440293 conn(0x7f6334067d00 msgr2=0x7f63340a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 -- 172.21.15.116:0/552440293 shutdown_connections 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.971+0000 7f633c08b640 1 -- 172.21.15.116:0/552440293 wait complete. 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient: build_initial_monmap 2024-09-18T12:31:13.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient: monmap: 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 1 Processor -- start 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 1 -- start start 2024-09-18T12:31:13.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient: init 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6334096f20 0x7f633409b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient(hunting): picked mon.noname-b con 0x7f6334096f20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409c8b0 0x7f633412fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient(hunting): picked mon.noname-c con 0x7f633409c8b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f633409d440 0x7f633412ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient(hunting): picked mon.noname-a con 0x7f633409d440 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.972+0000 7f633c08b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633c08b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633411fcb0 con 0x7f633409d440 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633c08b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633c08b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633411feb0 con 0x7f6334096f20 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633c08b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633c08b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633411a250 con 0x7f633409c8b0 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633c08b640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633c08b640 10 monclient(hunting): authenticate will time out at 1101.294019s 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633a628640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f633409d440 msgr2=0x7f633412ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f633a628640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f633409d440 0x7f633412ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409c8b0 0x7f633412fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409c8b0 0x7f633412fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58040/0 (socket says 172.21.15.116:58040) 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 -- 172.21.15.116:0/15909991 learned_addr learned my addr 172.21.15.116:0/15909991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): get_auth_request con 0x7f633409c8b0 auth_method 0 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6334096f20 0x7f633409b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 10 monclient(hunting): get_auth_request con 0x7f6334096f20 auth_method 0 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:13.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient(hunting): handle_auth_done global_id 64559 payload 824 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6334096f20 msgr2=0x7f633409b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6334096f20 0x7f633409b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f633409d440 msgr2=0x7f633412ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f633409d440 0x7f633412ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient: _finish_hunting 0 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 monclient: found mon.noname-c 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.973+0000 7f6339626640 1 -- 172.21.15.116:0/15909991 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324009450 con 0x7f633409c8b0 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6339626640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409c8b0 msgr2=0x7f633412fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6339626640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409c8b0 0x7f633412fa60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6324009210 tx=0x7f6324032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324031990 con 0x7f633409c8b0 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: dump: 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.990 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: _finish_auth 0 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6324031af0 con 0x7f633409c8b0 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f633c08b640 5 monclient: authenticate success, global_id 64559 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f633c08b640 10 monclient: _renew_subs 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f633c08b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633411a810 con 0x7f633409c8b0 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632403c470 con 0x7f633409c8b0 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f6329ffb640 10 monclient: dump: 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.975+0000 7f632bfff640 4 set_mon_vals no callback set 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f633c08b640 10 monclient: _renew_subs 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f633c08b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6334120470 con 0x7f633409c8b0 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f633c08b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f633c08b640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 0x7f6334140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f633c08b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6334065f90 con 0x7f63341197a0 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6324039040 con 0x7f633409c8b0 2024-09-18T12:31:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 0x7f6334140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 10 monclient: get_auth_request con 0x7f63341197a0 auth_method 0 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.976+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.977+0000 7f6329ffb640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6318038440 0x7f631803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.977+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(132..132 src has 13..132) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f632406c990 con 0x7f633409c8b0 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.977+0000 7f6339e27640 10 monclient(hunting): handle_auth_done global_id 64562 payload 824 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.977+0000 7f6339e27640 1 -- 172.21.15.116:0/15909991 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6334120070 con 0x7f63341197a0 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.977+0000 7f633a628640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6318038440 0x7f631803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.977+0000 7f633a628640 10 monclient: get_auth_request con 0x7f6318038440 auth_method 0 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6339e27640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 msgr2=0x7f6334140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6339e27640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 0x7f6334140ff0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f633000f780 tx=0x7f6330007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633000d8b0 con 0x7f63341197a0 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 10 monclient: dump: 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f633a628640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6318038440 0x7f631803a8f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f633409bd20 tx=0x7f632c009370 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f633000da10 con 0x7f63341197a0 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 msgr2=0x7f6334140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 0x7f6334140ff0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f633000f780 tx=0x7f6330007e10 comp rx=0 tx=0).stop 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 msgr2=0x7f6334140ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:13.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.979+0000 7f6329ffb640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63341197a0 0x7f6334140ff0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f633000f780 tx=0x7f6330007e10 comp rx=0 tx=0).stop 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f633c08b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f633c08b640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 0x7f6334141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f633c08b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6334141e70 con 0x7f6334141530 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 0x7f6334141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient: get_auth_request con 0x7f6334141530 auth_method 0 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 10 monclient(hunting): handle_auth_done global_id 64565 payload 824 2024-09-18T12:31:14.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.985+0000 7f6339e27640 1 -- 172.21.15.116:0/15909991 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f633404bf60 con 0x7f6334141530 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6339e27640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 msgr2=0x7f6334141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6339e27640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 0x7f6334141930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f633000f780 tx=0x7f6330019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6330014e20 con 0x7f6334141530 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 10 monclient: dump: 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.002 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f633001d610 con 0x7f6334141530 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 msgr2=0x7f6334141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 0x7f6334141930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f633000f780 tx=0x7f6330019000 comp rx=0 tx=0).stop 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 msgr2=0x7f6334141930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:14.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f6329ffb640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6334141530 0x7f6334141930 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f633000f780 tx=0x7f6330019000 comp rx=0 tx=0).stop 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 68, 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:14.004 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:14.005 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:14.006 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:14.007 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:14.008 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:14.009 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:14.010 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.012 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.014 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:14.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:14.016 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6318038440 msgr2=0x7f631803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6318038440 0x7f631803a8f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f633409bd20 tx=0x7f632c009370 comp rx=0 tx=0).stop 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 10 monclient: shutdown 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409c8b0 msgr2=0x7f633412fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 --2- 172.21.15.116:0/15909991 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f633409c8b0 0x7f633412fa60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6324009210 tx=0x7f6324032000 comp rx=0 tx=0).stop 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633a628640 1 -- 172.21.15.116:0/15909991 reap_dead start 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633a628640 1 -- 172.21.15.116:0/15909991 reap_dead start 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 shutdown_connections 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 >> 172.21.15.116:0/15909991 conn(0x7f6334067d00 msgr2=0x7f633409ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 shutdown_connections 2024-09-18T12:31:14.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:13.988+0000 7f633c08b640 1 -- 172.21.15.116:0/15909991 wait complete. 2024-09-18T12:31:14.020 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:31:14.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:31:14.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:31:14.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.985+0000 7f0315a6f640 10 monclient: get_monmap_and_config 2024-09-18T12:31:14.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.985+0000 7f0315a6f640 10 monclient: build_initial_monmap 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.985+0000 7f0315a6f640 10 monclient: monmap: 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:13.987177+0000 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:13.987177+0000 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:14.028 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.985+0000 7f0315a6f640 1 Processor -- start 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 1 -- start start 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient: init 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f031009ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f031009f020 0x7f0310069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f031009f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0310136d60 0x7f0310139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f0310136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0310067420 con 0x7f031009ec50 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031009fdc0 con 0x7f031009f020 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03100a0510 con 0x7f0310136d60 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f0315a6f640 10 monclient(hunting): authenticate will time out at 1109.556510s 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f031009f020 0x7f0310069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030f7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f031009ec50 0x7f0310131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57672/0 (socket says 172.21.15.144:57672) 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): get_auth_request con 0x7f031009f020 auth_method 0 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030f7fe640 1 -- 172.21.15.144:0/3465004389 learned_addr learned my addr 172.21.15.144:0/3465004389 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030ffff640 1 --2- 172.21.15.144:0/3465004389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0310136d60 0x7f0310139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030ffff640 10 monclient(hunting): get_auth_request con 0x7f0310136d60 auth_method 0 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient(hunting): handle_auth_done global_id 5722 payload 274 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 1 -- 172.21.15.144:0/3465004389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0310136d60 msgr2=0x7f0310139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 1 --2- 172.21.15.144:0/3465004389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0310136d60 0x7f0310139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 1 -- 172.21.15.144:0/3465004389 >> v1:172.21.15.144:6789/0 conn(0x7f031009ec50 legacy=0x7f0310131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 1 monclient: found mon.noname-c 2024-09-18T12:31:14.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030effd640 1 -- 172.21.15.144:0/3465004389 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03100a0200 con 0x7f031009f020 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.986+0000 7f030ffff640 1 --2- 172.21.15.144:0/3465004389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0310136d60 0x7f0310139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.989+0000 7f030effd640 1 -- 172.21.15.144:0/3465004389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 msgr2=0x7f0310069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.989+0000 7f030effd640 1 --2- 172.21.15.144:0/3465004389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310069d10 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0304009850 tx=0x7f03040090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc041c79f3c67de8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 1 -- 172.21.15.144:0/3465004389 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030402add0 con 0x7f031009f020 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: dump: 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.032 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 1 -- 172.21.15.144:0/3465004389 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0304032440 con 0x7f031009f020 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 5 monclient: authenticate success, global_id 5722 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:14.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 1 -- 172.21.15.144:0/3465004389 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304032820 con 0x7f031009f020 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 4 set_mon_vals no callback set 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f030e7fc640 10 monclient: dump: 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.034 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 10 monclient: shutdown 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 -- 172.21.15.144:0/3465004389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 msgr2=0x7f0310069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 --2- 172.21.15.144:0/3465004389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310069d10 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0304009850 tx=0x7f03040090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 -- 172.21.15.144:0/3465004389 shutdown_connections 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 --2- 172.21.15.144:0/3465004389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0310136d60 0x7f0310139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 --2- 172.21.15.144:0/3465004389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310069d10 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 -- 172.21.15.144:0/3465004389 >> 172.21.15.144:0/3465004389 conn(0x7f03100a5d40 msgr2=0x7f03100a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 -- 172.21.15.144:0/3465004389 shutdown_connections 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.990+0000 7f0315a6f640 1 -- 172.21.15.144:0/3465004389 wait complete. 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.991+0000 7f0315a6f640 10 monclient: build_initial_monmap 2024-09-18T12:31:14.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.991+0000 7f0315a6f640 10 monclient: monmap: 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.991+0000 7f0315a6f640 1 Processor -- start 2024-09-18T12:31:14.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.991+0000 7f0315a6f640 1 -- start start 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.991+0000 7f0315a6f640 10 monclient: init 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f031009ec50 0x7f0310097200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f031009ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310097740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f031009f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0310136d60 0x7f0310097c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): picked mon.noname-a con 0x7f0310136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101315a0 con 0x7f0310136d60 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310099230 con 0x7f031009f020 2024-09-18T12:31:14.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03100a3a50 con 0x7f031009ec50 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f0315a6f640 10 monclient(hunting): authenticate will time out at 1109.562242s 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310097740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030ffff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0310136d60 msgr2=0x7f0310097c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0310136d60 0x7f0310097c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310097740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37816/0 (socket says 172.21.15.144:37816) 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 -- 172.21.15.144:0/2532043715 learned_addr learned my addr 172.21.15.144:0/2532043715 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): get_auth_request con 0x7f031009f020 auth_method 0 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f031009ec50 0x7f0310097200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 10 monclient(hunting): get_auth_request con 0x7f031009ec50 auth_method 0 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient(hunting): handle_auth_done global_id 5725 payload 824 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 -- 172.21.15.144:0/2532043715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f031009ec50 msgr2=0x7f0310097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f031009ec50 0x7f0310097200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 -- 172.21.15.144:0/2532043715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0310136d60 msgr2=0x7f0310097c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0310136d60 0x7f0310097c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 monclient: found mon.noname-b 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.992+0000 7f030effd640 1 -- 172.21.15.144:0/2532043715 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0304009390 con 0x7f031009f020 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f030effd640 1 -- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 msgr2=0x7f0310097740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f030effd640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310097740 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0304009360 tx=0x7f0304032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304031c70 con 0x7f031009f020 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:14.040 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: dump: 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:14.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f0315a6f640 5 monclient: authenticate success, global_id 5725 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0304031dd0 con 0x7f031009f020 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030403d690 con 0x7f031009f020 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f0315a6f640 10 monclient: _renew_subs 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f0315a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031009c000 con 0x7f031009f020 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f02f77fe640 10 monclient: dump: 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.042 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.994+0000 7f030dffb640 4 set_mon_vals no callback set 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f0315a6f640 10 monclient: _renew_subs 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f0315a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0310130ae0 con 0x7f031009f020 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0304039040 con 0x7f031009f020 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f0315a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f0315a6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f0315a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.995+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03100008d0 con 0x7f031009f020 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.996+0000 7f02f77fe640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f02fc038440 0x7f02fc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.996+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(132..132 src has 13..132) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f030406c330 con 0x7f031009f020 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.996+0000 7f030f7fe640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f02fc038440 0x7f02fc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.996+0000 7f030f7fe640 10 monclient: get_auth_request con 0x7f02fc038440 auth_method 0 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:13.998+0000 7f030f7fe640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f02fc038440 0x7f02fc03a8f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0300007fd0 tx=0x7f0300007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.000+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f030402f2a0 con 0x7f031009f020 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.000+0000 7f02f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:14.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.000+0000 7f02f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:14.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.117+0000 7f7481220640 10 monclient: get_monmap_and_config 2024-09-18T12:31:14.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.117+0000 7f7481220640 10 monclient: build_initial_monmap 2024-09-18T12:31:14.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.117+0000 7f7481220640 10 monclient: monmap: 2024-09-18T12:31:14.118 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:14.118 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.118 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:14.118082+0000 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:14.118082+0000 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:14.119 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 1 Processor -- start 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 1 -- start start 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient: init 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f747c131980 0x7f747c131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient(hunting): picked mon.noname-a con 0x7f747c131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f747c09fda0 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f747c132290 0x7f747c09f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient(hunting): picked mon.noname-c con 0x7f747c132290 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c0a1180 con 0x7f747c09fda0 2024-09-18T12:31:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c0a1460 con 0x7f747c131980 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c139cd0 con 0x7f747c132290 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f7481220640 10 monclient(hunting): authenticate will time out at 1101.439643s 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747a59c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f747c09fda0 legacy=0x7f747c09e9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747ad9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f747c131980 msgr2=0x7f747c131d50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f747c131980 0x7f747c131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747b59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f747c132290 0x7f747c09f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747b59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f747c132290 0x7f747c09f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:34346/0 (socket says 172.21.15.116:34346) 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747b59e640 1 -- 172.21.15.116:0/2692004219 learned_addr learned my addr 172.21.15.116:0/2692004219 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747b59e640 10 monclient(hunting): get_auth_request con 0x7f747c132290 auth_method 0 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747b59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.118+0000 7f747b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 10 monclient(hunting): handle_auth_done global_id 5728 payload 274 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 1 -- 172.21.15.116:0/2692004219 >> v2:172.21.15.116:3300/0 conn(0x7f747c131980 msgr2=0x7f747c131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 1 --2- 172.21.15.116:0/2692004219 >> v2:172.21.15.116:3300/0 conn(0x7f747c131980 0x7f747c131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 1 -- 172.21.15.116:0/2692004219 >> v1:172.21.15.116:6789/0 conn(0x7f747c09fda0 legacy=0x7f747c09e9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 10 monclient: _finish_hunting 0 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 1 monclient: found mon.noname-c 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.119+0000 7f747b59e640 1 -- 172.21.15.116:0/2692004219 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c139e10 con 0x7f747c132290 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f747b59e640 1 -- 172.21.15.116:0/2692004219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 msgr2=0x7f747c09f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f747b59e640 1 --2- 172.21.15.116:0/2692004219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 0x7f747c09f0d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f747c068aa0 tx=0x7f7474009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4343ad9813664188 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 1 -- 172.21.15.116:0/2692004219 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474007610 con 0x7f747c132290 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: dump: 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: _finish_auth 0 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 5 monclient: authenticate success, global_id 5728 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 1 -- 172.21.15.116:0/2692004219 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747402d050 con 0x7f747c132290 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 1 -- 172.21.15.116:0/2692004219 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74740079e0 con 0x7f747c132290 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 4 set_mon_vals no callback set 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7479d9b640 10 monclient: dump: 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 10 monclient: shutdown 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 -- 172.21.15.116:0/2692004219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 msgr2=0x7f747c09f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 --2- 172.21.15.116:0/2692004219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 0x7f747c09f0d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f747c068aa0 tx=0x7f7474009840 comp rx=0 tx=0).stop 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 -- 172.21.15.116:0/2692004219 shutdown_connections 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 --2- 172.21.15.116:0/2692004219 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 0x7f747c09f0d0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 --2- 172.21.15.116:0/2692004219 >> v2:172.21.15.116:3300/0 conn(0x7f747c131980 0x7f747c131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 -- 172.21.15.116:0/2692004219 >> 172.21.15.116:0/2692004219 conn(0x7f747c067bd0 msgr2=0x7f747c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 -- 172.21.15.116:0/2692004219 shutdown_connections 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.122+0000 7f7481220640 1 -- 172.21.15.116:0/2692004219 wait complete. 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.123+0000 7f7481220640 10 monclient: build_initial_monmap 2024-09-18T12:31:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.123+0000 7f7481220640 10 monclient: monmap: 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.123+0000 7f7481220640 1 Processor -- start 2024-09-18T12:31:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.123+0000 7f7481220640 1 -- start start 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.123+0000 7f7481220640 10 monclient: init 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747c09fda0 0x7f747c137a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): picked mon.noname-a con 0x7f747c09fda0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747c131980 0x7f747c133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): picked mon.noname-c con 0x7f747c131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 0x7f747c133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): picked mon.noname-b con 0x7f747c132290 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c136d30 con 0x7f747c09fda0 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c09bf10 con 0x7f747c132290 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0a46c0 con 0x7f747c131980 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f7481220640 10 monclient(hunting): authenticate will time out at 1101.445367s 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747ad9d640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747c09fda0 msgr2=0x7f747c137a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747c09fda0 0x7f747c137a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747c131980 0x7f747c133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747c131980 0x7f747c133440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58084/0 (socket says 172.21.15.116:58084) 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 -- 172.21.15.116:0/4254310762 learned_addr learned my addr 172.21.15.116:0/4254310762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): get_auth_request con 0x7f747c131980 auth_method 0 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 0x7f747c133980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 10 monclient(hunting): get_auth_request con 0x7f747c132290 auth_method 0 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient(hunting): handle_auth_done global_id 64574 payload 824 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 msgr2=0x7f747c133980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c132290 0x7f747c133980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747c09fda0 msgr2=0x7f747c137a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747c09fda0 0x7f747c137a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 monclient: found mon.noname-c 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.124+0000 7f747a59c640 1 -- 172.21.15.116:0/4254310762 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7474009ae0 con 0x7f747c131980 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f747a59c640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747c131980 msgr2=0x7f747c133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f747a59c640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747c131980 0x7f747c133440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f747000b5f0 tx=0x7f7470009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7470015070 con 0x7f747c131980 2024-09-18T12:31:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: dump: 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7470003d90 con 0x7f747c131980 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 5 monclient: authenticate success, global_id 64574 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 10 monclient: _renew_subs 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c136a50 con 0x7f747c131980 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74700063a0 con 0x7f747c131980 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7462ffd640 10 monclient: dump: 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 10 monclient: _renew_subs 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f747959a640 4 set_mon_vals no callback set 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c137f80 con 0x7f747c131980 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 0x7f747c141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.127+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c065e00 con 0x7f747c133ec0 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 0x7f747c141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient: get_auth_request con 0x7f747c133ec0 auth_method 0 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7470010440 con 0x7f747c131980 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f7462ffd640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 10 monclient(hunting): handle_auth_done global_id 5734 payload 824 2024-09-18T12:31:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(132..132 src has 13..132) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f747004be90 con 0x7f747c131980 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747ad9d640 1 -- 172.21.15.116:0/4254310762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f747c136f70 con 0x7f747c133ec0 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747b59e640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.128+0000 7f747b59e640 10 monclient: get_auth_request con 0x7f7468038440 auth_method 0 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.130+0000 7f747ad9d640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 msgr2=0x7f747c141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.130+0000 7f747ad9d640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 0x7f747c141080 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f746c00b440 tx=0x7f746c009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.130+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c003df0 con 0x7f747c133ec0 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.130+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.130+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.130+0000 7f7462ffd640 10 monclient: dump: 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f747b59e640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7474033000 tx=0x7f7474036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f746c0050f0 con 0x7f747c133ec0 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f7462ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f7462ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 msgr2=0x7f747c141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f7462ffd640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 0x7f747c141080 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f746c00b440 tx=0x7f746c009120 comp rx=0 tx=0).stop 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 msgr2=0x7f747c141080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.131+0000 7f7462ffd640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c133ec0 0x7f747c141080 secure :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f746c00b440 tx=0x7f746c009120 comp rx=0 tx=0).stop 2024-09-18T12:31:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.136+0000 7f7481220640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.136+0000 7f7481220640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 0x7f747c1419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.136+0000 7f7481220640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.136+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c141f00 con 0x7f747c1415c0 2024-09-18T12:31:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 0x7f747c1419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient: get_auth_request con 0x7f747c1415c0 auth_method 0 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 10 monclient(hunting): handle_auth_done global_id 5737 payload 824 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.137+0000 7f747ad9d640 1 -- 172.21.15.116:0/4254310762 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f747c04bf60 con 0x7f747c1415c0 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.139+0000 7f747ad9d640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 msgr2=0x7f747c1419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.139+0000 7f747ad9d640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 0x7f747c1419c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f746c014000 tx=0x7f746c015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.139+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c01e620 con 0x7f747c1415c0 2024-09-18T12:31:14.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.139+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.139+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.139+0000 7f7462ffd640 10 monclient: dump: 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:14.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f746c01e780 con 0x7f747c1415c0 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7462ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7462ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 msgr2=0x7f747c1419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7462ffd640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 0x7f747c1419c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f746c014000 tx=0x7f746c015000 comp rx=0 tx=0).stop 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7462ffd640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 msgr2=0x7f747c1419c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7462ffd640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f747c1415c0 0x7f747c1419c0 secure :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f746c014000 tx=0x7f746c015000 comp rx=0 tx=0).stop 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:31:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 68, 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:14.152 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:14.153 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:14.154 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:14.155 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:14.157 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:14.160 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:14.161 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.162 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:14.163 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:14.164 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 msgr2=0x7f746803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7474033000 tx=0x7f7474036000 comp rx=0 tx=0).stop 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 10 monclient: shutdown 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747c131980 msgr2=0x7f747c133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 --2- 172.21.15.116:0/4254310762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747c131980 0x7f747c133440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f747000b5f0 tx=0x7f7470009120 comp rx=0 tx=0).stop 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f747b59e640 1 -- 172.21.15.116:0/4254310762 reap_dead start 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f747b59e640 1 -- 172.21.15.116:0/4254310762 reap_dead start 2024-09-18T12:31:14.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 shutdown_connections 2024-09-18T12:31:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 >> 172.21.15.116:0/4254310762 conn(0x7f747c067bd0 msgr2=0x7f747c0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 shutdown_connections 2024-09-18T12:31:14.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:14.140+0000 7f7481220640 1 -- 172.21.15.116:0/4254310762 wait complete. 2024-09-18T12:31:14.166 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:31:14.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:31:14.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.165+0000 7f0315a6f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T12:31:14.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.165+0000 7f0315a6f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T12:31:14.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.165+0000 7f0315a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.165+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f0310064900 con 0x7f031009f020 2024-09-18T12:31:14.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.799+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v133) v1 ==== 115+0+0 (secure 0 0 0) 0x7f0304030b30 con 0x7f031009f020 2024-09-18T12:31:14.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.799+0000 7f02f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T12:31:14.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.799+0000 7f02f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:14.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.901+0000 7f0315a6f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T12:31:14.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.901+0000 7f0315a6f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T12:31:14.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.901+0000 7f0315a6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.901+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f0310064900 con 0x7f031009f020 2024-09-18T12:31:14.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.902+0000 7f02f77fe640 1 -- 172.21.15.144:0/2532043715 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v133) v1 ==== 159+0+0 (secure 0 0 0) 0x7f0304037070 con 0x7f031009f020 2024-09-18T12:31:14.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.902+0000 7f02f77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T12:31:14.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.902+0000 7f02f77fe640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-18T12:31:14.904 INFO:tasks.workunit.client.0.smithi144.stderr:weight-set for pool 'barset' already created 2024-09-18T12:31:14.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f02fc038440 msgr2=0x7f02fc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f02fc038440 0x7f02fc03a8f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0300007fd0 tx=0x7f0300007f00 comp rx=0 tx=0).stop 2024-09-18T12:31:14.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 10 monclient: shutdown 2024-09-18T12:31:14.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:14.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 msgr2=0x7f0310097740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310097740 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0304009360 tx=0x7f0304032000 comp rx=0 tx=0).stop 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 shutdown_connections 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f02fc038440 0x7f02fc03a8f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0310136d60 0x7f0310097c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f031009f020 0x7f0310097740 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 --2- 172.21.15.144:0/2532043715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f031009ec50 0x7f0310097200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 >> 172.21.15.144:0/2532043715 conn(0x7f03100a5d40 msgr2=0x7f031009df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 shutdown_connections 2024-09-18T12:31:14.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.904+0000 7f0315a6f640 1 -- 172.21.15.144:0/2532043715 wait complete. 2024-09-18T12:31:14.915 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:31:14.915 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep fooset 2024-09-18T12:31:14.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.986+0000 7fb30215d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.986+0000 7fb30215d640 10 monclient: build_initial_monmap 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.986+0000 7fb30215d640 10 monclient: monmap: 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:14.988751+0000 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:14.988751+0000 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:14.989 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:14.990 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:14.990 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:14.990 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:14.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.987+0000 7fb30215d640 1 Processor -- start 2024-09-18T12:31:14.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.987+0000 7fb30215d640 1 -- start start 2024-09-18T12:31:14.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.987+0000 7fb30215d640 10 monclient: init 2024-09-18T12:31:14.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:14.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2fc069a30 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb2fc099c80 0x7fb2fc071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc099c80 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2fc135710 0x7fb2fc137af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc135710 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc09e7e0 con 0x7fb2fc069a30 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2fc069e20 con 0x7fb2fc099c80 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2fc09cef0 con 0x7fb2fc135710 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:14.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb30215d640 10 monclient(hunting): authenticate will time out at 1110.558219s 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2fc135710 0x7fb2fc137af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb2fc135710 0x7fb2fc137af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37832/0 (socket says 172.21.15.144:37832) 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2f2ffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fb2fc099c80 msgr2=0x7fb2fc071030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 -- 172.21.15.144:0/754696567 learned_addr learned my addr 172.21.15.144:0/754696567 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): get_auth_request con 0x7fb2fc135710 auth_method 0 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fb7fe640 1 --1- 172.21.15.144:0/754696567 >> v1:172.21.15.144:6789/0 conn(0x7fb2fc069a30 0x7fb2fc070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57686/0 (socket says 172.21.15.144:57686) 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2f2ffd640 1 --2- 172.21.15.144:0/754696567 >> v2:172.21.15.116:3300/0 conn(0x7fb2fc099c80 0x7fb2fc071030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:14.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient(hunting): handle_auth_done global_id 5740 payload 274 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 -- 172.21.15.144:0/754696567 >> v2:172.21.15.116:3300/0 conn(0x7fb2fc099c80 msgr2=0x7fb2fc071030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 --2- 172.21.15.144:0/754696567 >> v2:172.21.15.116:3300/0 conn(0x7fb2fc099c80 0x7fb2fc071030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 -- 172.21.15.144:0/754696567 >> v1:172.21.15.144:6789/0 conn(0x7fb2fc069a30 legacy=0x7fb2fc070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 monclient: found mon.noname-c 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:14.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.988+0000 7fb2fbfff640 1 -- 172.21.15.144:0/754696567 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc09d390 con 0x7fb2fc135710 2024-09-18T12:31:15.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2fbfff640 1 -- 172.21.15.144:0/754696567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc135710 msgr2=0x7fb2fc137af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2fbfff640 1 --2- 172.21.15.144:0/754696567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc135710 0x7fb2fc137af0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc09e430 tx=0x7fb2f4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=624c4d67f5ac037d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 1 -- 172.21.15.144:0/754696567 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4007610 con 0x7fb2fc135710 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: dump: 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.001 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 1 -- 172.21.15.144:0/754696567 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f402d050 con 0x7fb2fc135710 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb30215d640 5 monclient: authenticate success, global_id 5740 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb30215d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb30215d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb30215d640 4 set_mon_vals no callback set 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 1 -- 172.21.15.144:0/754696567 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f40079e0 con 0x7fb2fc135710 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:15.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.991+0000 7fb2faffd640 10 monclient: dump: 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 10 monclient: shutdown 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:15.003 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 -- 172.21.15.144:0/754696567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc135710 msgr2=0x7fb2fc137af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 --2- 172.21.15.144:0/754696567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc135710 0x7fb2fc137af0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc09e430 tx=0x7fb2f4009840 comp rx=0 tx=0).stop 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 -- 172.21.15.144:0/754696567 shutdown_connections 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 --2- 172.21.15.144:0/754696567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc135710 0x7fb2fc137af0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 --2- 172.21.15.144:0/754696567 >> v2:172.21.15.116:3300/0 conn(0x7fb2fc099c80 0x7fb2fc071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 -- 172.21.15.144:0/754696567 >> 172.21.15.144:0/754696567 conn(0x7fb2fc09e9f0 msgr2=0x7fb2fc09edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 -- 172.21.15.144:0/754696567 shutdown_connections 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 -- 172.21.15.144:0/754696567 wait complete. 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 10 monclient: build_initial_monmap 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 10 monclient: monmap: 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:15.004 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:15.005 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:15.005 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.005 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:15.005 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:15.005 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.992+0000 7fb30215d640 1 Processor -- start 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 1 -- start start 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient: init 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 0x7fb2fc13f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2fc069a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc099c80 0x7fb2fc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc099c80 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb2fc135710 0x7fb2fc12fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc135710 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc11ab70 con 0x7fb2fc099c80 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc09d8c0 con 0x7fb2fc069a30 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc132700 con 0x7fb2fc135710 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb30215d640 10 monclient(hunting): authenticate will time out at 1110.563611s 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 0x7fb2fc13f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 0x7fb2fc13f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37838/0 (socket says 172.21.15.144:37838) 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 -- 172.21.15.144:0/239374121 learned_addr learned my addr 172.21.15.144:0/239374121 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:15.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): get_auth_request con 0x7fb2fc069a30 auth_method 0 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2f2ffd640 1 -- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc099c80 msgr2=0x7fb2fc06d820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2f2ffd640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc099c80 0x7fb2fc06d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient(hunting): handle_auth_done global_id 5743 payload 824 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 -- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb2fc135710 msgr2=0x7fb2fc12fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.008 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb2fc135710 0x7fb2fc12fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 -- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc099c80 msgr2=0x7fb2fc06d820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc099c80 0x7fb2fc06d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.993+0000 7fb2fb7fe640 1 -- 172.21.15.144:0/239374121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f4009ae0 con 0x7fb2fc069a30 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2fb7fe640 1 -- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 msgr2=0x7fb2fc13f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2fb7fe640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 0x7fb2fc13f220 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00c5b0 tx=0x7fb2ec007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec0075f0 con 0x7fb2fc069a30 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.009 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: dump: 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.010 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: _finish_auth 0 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec0057d0 con 0x7fb2fc069a30 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb30215d640 5 monclient: authenticate success, global_id 5743 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec005b90 con 0x7fb2fc069a30 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb30215d640 10 monclient: _renew_subs 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb30215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc119790 con 0x7fb2fc069a30 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2f3fff640 10 monclient: dump: 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.011 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb2fa7fc640 4 set_mon_vals no callback set 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb30215d640 10 monclient: _renew_subs 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb30215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.996+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc12ffb0 con 0x7fb2fc069a30 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb2ec01b400 con 0x7fb2fc069a30 2024-09-18T12:31:15.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb30215d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb30215d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb30215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc04bf60 con 0x7fb2fc069a30 2024-09-18T12:31:15.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb2f3fff640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb2dc038440 0x7fb2dc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(133..133 src has 13..133) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fb2ec04c100 con 0x7fb2fc069a30 2024-09-18T12:31:15.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb2f2ffd640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb2dc038440 0x7fb2dc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.997+0000 7fb2f2ffd640 10 monclient: get_auth_request con 0x7fb2dc038440 auth_method 0 2024-09-18T12:31:15.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:14.999+0000 7fb2f2ffd640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb2dc038440 0x7fb2dc03a8f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc119f20 tx=0x7fb2e4006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.001+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb2ec01a720 con 0x7fb2fc069a30 2024-09-18T12:31:15.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.001+0000 7fb2f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.001+0000 7fb2f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:15.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.161+0000 7fb30215d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.161+0000 7fb30215d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.161+0000 7fb30215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.161+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb2fc130230 con 0x7fb2fc069a30 2024-09-18T12:31:15.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.162+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+14 (secure 0 0 0) 0x7fb2ec01a360 con 0x7fb2fc069a30 2024-09-18T12:31:15.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.162+0000 7fb2f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.162+0000 7fb2f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:15.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.260+0000 7fb30215d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.260+0000 7fb30215d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.260+0000 7fb30215d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.260+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb2fc11a7e0 con 0x7fb2fc069a30 2024-09-18T12:31:15.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.261+0000 7fb2f3fff640 1 -- 172.21.15.144:0/239374121 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+14 (secure 0 0 0) 0x7fb2ec01a540 con 0x7fb2fc069a30 2024-09-18T12:31:15.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.261+0000 7fb2f3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.261+0000 7fb2f3fff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb2dc038440 msgr2=0x7fb2dc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb2dc038440 0x7fb2dc03a8f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc119f20 tx=0x7fb2e4006d80 comp rx=0 tx=0).stop 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 10 monclient: shutdown 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 msgr2=0x7fb2fc13f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 0x7fb2fc13f220 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00c5b0 tx=0x7fb2ec007f00 comp rx=0 tx=0).stop 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 shutdown_connections 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb2dc038440 0x7fb2dc03a8f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb2fc135710 0x7fb2fc12fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc099c80 0x7fb2fc06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 --2- 172.21.15.144:0/239374121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb2fc069a30 0x7fb2fc13f220 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.262+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 >> 172.21.15.144:0/239374121 conn(0x7fb2fc09e9f0 msgr2=0x7fb2fc134920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:15.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.263+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 shutdown_connections 2024-09-18T12:31:15.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.263+0000 7fb30215d640 1 -- 172.21.15.144:0/239374121 wait complete. 2024-09-18T12:31:15.274 INFO:tasks.workunit.client.0.smithi144.stdout:fooset 2024-09-18T12:31:15.274 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:31:15.274 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep barset 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.345+0000 7fc2fc75d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.345+0000 7fc2fc75d640 10 monclient: build_initial_monmap 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.345+0000 7fc2fc75d640 10 monclient: monmap: 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:15.347593+0000 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:15.347593+0000 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:15.348 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 1 Processor -- start 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 1 -- start start 2024-09-18T12:31:15.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient: init 2024-09-18T12:31:15.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2f409c2b0 0x7fc2f409c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f409c2b0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2f409a7a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f4136bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f41347f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f40a0a30 con 0x7fc2f409a7a0 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f409b7d0 con 0x7fc2f409c2b0 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f409b280 con 0x7fc2f41347f0 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.346+0000 7fc2fc75d640 10 monclient(hunting): authenticate will time out at 1110.916998s 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2fa4f9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fc2f409c2b0 msgr2=0x7fc2f409c680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2fa4f9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2f409c2b0 0x7fc2f409c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2f9cf8640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc2f409a7a0 0x7fc2f409ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57694/0 (socket says 172.21.15.144:57694) 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2f9cf8640 1 -- 172.21.15.144:0/1275952120 learned_addr learned my addr 172.21.15.144:0/1275952120 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 1 --2- 172.21.15.144:0/1275952120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f4136bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): get_auth_request con 0x7fc2f41347f0 auth_method 0 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient(hunting): handle_auth_done global_id 64577 payload 274 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 1 -- 172.21.15.144:0/1275952120 >> v2:172.21.15.116:3300/0 conn(0x7fc2f409c2b0 msgr2=0x7fc2f409c680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 1 --2- 172.21.15.144:0/1275952120 >> v2:172.21.15.116:3300/0 conn(0x7fc2f409c2b0 0x7fc2f409c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 1 -- 172.21.15.144:0/1275952120 >> v1:172.21.15.144:6789/0 conn(0x7fc2f409a7a0 legacy=0x7fc2f409ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 1 monclient: found mon.noname-b 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:15.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.347+0000 7fc2facfa640 1 -- 172.21.15.144:0/1275952120 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f409b4f0 con 0x7fc2f41347f0 2024-09-18T12:31:15.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.350+0000 7fc2facfa640 1 -- 172.21.15.144:0/1275952120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 msgr2=0x7fc2f4136bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:15.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.350+0000 7fc2facfa640 1 --2- 172.21.15.144:0/1275952120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f4136bd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009e60 tx=0x7fc2ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5274e4fda720813 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 1 -- 172.21.15.144:0/1275952120 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec007840 con 0x7fc2f41347f0 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: dump: 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.361 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: _finish_auth 0 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 1 -- 172.21.15.144:0/1275952120 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec02d050 con 0x7fc2f41347f0 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 5 monclient: authenticate success, global_id 64577 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 4 set_mon_vals no callback set 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 1 -- 172.21.15.144:0/1275952120 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec007bd0 con 0x7fc2f41347f0 2024-09-18T12:31:15.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2f94f7640 10 monclient: dump: 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 10 monclient: shutdown 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 -- 172.21.15.144:0/1275952120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 msgr2=0x7fc2f4136bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/1275952120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f4136bd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009e60 tx=0x7fc2ec009780 comp rx=0 tx=0).stop 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 -- 172.21.15.144:0/1275952120 shutdown_connections 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/1275952120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f4136bd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/1275952120 >> v2:172.21.15.116:3300/0 conn(0x7fc2f409c2b0 0x7fc2f409c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 -- 172.21.15.144:0/1275952120 >> 172.21.15.144:0/1275952120 conn(0x7fc2f40a3a50 msgr2=0x7fc2f409faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 -- 172.21.15.144:0/1275952120 shutdown_connections 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.351+0000 7fc2fc75d640 1 -- 172.21.15.144:0/1275952120 wait complete. 2024-09-18T12:31:15.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 10 monclient: build_initial_monmap 2024-09-18T12:31:15.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 10 monclient: monmap: 2024-09-18T12:31:15.365 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:15.365 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:15.365 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:15.365 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:15.365 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:15.365 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.366 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:15.366 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:15.366 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:15.366 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 1 Processor -- start 2024-09-18T12:31:15.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 1 -- start start 2024-09-18T12:31:15.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 10 monclient: init 2024-09-18T12:31:15.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:15.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:15.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 0x7fc2f412f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.352+0000 7fc2fc75d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f409a7a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:15.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f409c2b0 0x7fc2f412f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f409c2b0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:15.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f412fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f41347f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:15.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f406e950 con 0x7fc2f409c2b0 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4132c70 con 0x7fc2f409a7a0 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4133330 con 0x7fc2f41347f0 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fc75d640 10 monclient(hunting): authenticate will time out at 1110.923085s 2024-09-18T12:31:15.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 0x7fc2f412f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 0x7fc2f412f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37846/0 (socket says 172.21.15.144:37846) 2024-09-18T12:31:15.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 -- 172.21.15.144:0/3159420333 learned_addr learned my addr 172.21.15.144:0/3159420333 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:15.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): get_auth_request con 0x7fc2f409a7a0 auth_method 0 2024-09-18T12:31:15.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2f9cf8640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f409c2b0 msgr2=0x7fc2f412f910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:15.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2f9cf8640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f409c2b0 0x7fc2f412f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:15.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:15.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:15.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f412fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 10 monclient(hunting): get_auth_request con 0x7fc2f41347f0 auth_method 0 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2facfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient(hunting): handle_auth_done global_id 5746 payload 824 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 -- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 msgr2=0x7fc2f412fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f412fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 -- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f409c2b0 msgr2=0x7fc2f412f910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f409c2b0 0x7fc2f412f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient: _finish_hunting 0 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 monclient: found mon.noname-b 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.353+0000 7fc2fa4f9640 1 -- 172.21.15.144:0/3159420333 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec009a20 con 0x7fc2f409a7a0 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fa4f9640 1 -- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 msgr2=0x7fc2f412f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fa4f9640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 0x7fc2f412f250 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000b600 tx=0x7fc2f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0015070 con 0x7fc2f409a7a0 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: dump: 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.373 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fc75d640 5 monclient: authenticate success, global_id 5746 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f0003d90 con 0x7fc2f409a7a0 2024-09-18T12:31:15.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fc75d640 10 monclient: _renew_subs 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fc75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f4130bc0 con 0x7fc2f409a7a0 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0006380 con 0x7fc2f409a7a0 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2ea7fc640 10 monclient: dump: 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.375 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2f8cf6640 4 set_mon_vals no callback set 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fc75d640 10 monclient: _renew_subs 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fc75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.355+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f40994a0 con 0x7fc2f409a7a0 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.356+0000 7fc2fc75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.356+0000 7fc2fc75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.356+0000 7fc2fc75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.356+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc2f0010440 con 0x7fc2f409a7a0 2024-09-18T12:31:15.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.356+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f404bf60 con 0x7fc2f409a7a0 2024-09-18T12:31:15.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.357+0000 7fc2ea7fc640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2d8038440 0x7fc2d803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.357+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(133..133 src has 13..133) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fc2f004c010 con 0x7fc2f409a7a0 2024-09-18T12:31:15.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.357+0000 7fc2f9cf8640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2d8038440 0x7fc2d803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.357+0000 7fc2f9cf8640 10 monclient: get_auth_request con 0x7fc2d8038440 auth_method 0 2024-09-18T12:31:15.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.359+0000 7fc2f9cf8640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2d8038440 0x7fc2d803a8f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4099790 tx=0x7fc2e4007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.361+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc2f007e030 con 0x7fc2f409a7a0 2024-09-18T12:31:15.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.361+0000 7fc2ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.361+0000 7fc2ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:15.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.520+0000 7fc2fc75d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.520+0000 7fc2fc75d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.520+0000 7fc2fc75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.520+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fc2f4064a90 con 0x7fc2f409a7a0 2024-09-18T12:31:15.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.520+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+14 (secure 0 0 0) 0x7fc2f0018ab0 con 0x7fc2f409a7a0 2024-09-18T12:31:15.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.520+0000 7fc2ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.520+0000 7fc2ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:15.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.618+0000 7fc2fc75d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.618+0000 7fc2fc75d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.618+0000 7fc2fc75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.618+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fc2f4131750 con 0x7fc2f409a7a0 2024-09-18T12:31:15.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.619+0000 7fc2ea7fc640 1 -- 172.21.15.144:0/3159420333 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) v1 ==== 71+0+14 (secure 0 0 0) 0x7fc2f0018690 con 0x7fc2f409a7a0 2024-09-18T12:31:15.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.619+0000 7fc2ea7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:15.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.619+0000 7fc2ea7fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:15.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2d8038440 msgr2=0x7fc2d803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2d8038440 0x7fc2d803a8f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4099790 tx=0x7fc2e4007b80 comp rx=0 tx=0).stop 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 10 monclient: shutdown 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 msgr2=0x7fc2f412f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 0x7fc2f412f250 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000b600 tx=0x7fc2f00091a0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 shutdown_connections 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc2d8038440 0x7fc2d803a8f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2f41347f0 0x7fc2f412fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f409c2b0 0x7fc2f412f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 --2- 172.21.15.144:0/3159420333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2f409a7a0 0x7fc2f412f250 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 >> 172.21.15.144:0/3159420333 conn(0x7fc2f40a3a50 msgr2=0x7fc2f409e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 shutdown_connections 2024-09-18T12:31:15.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.621+0000 7fc2fc75d640 1 -- 172.21.15.144:0/3159420333 wait complete. 2024-09-18T12:31:15.632 INFO:tasks.workunit.client.0.smithi144.stdout:barset 2024-09-18T12:31:15.632 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set dump 2024-09-18T12:31:15.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.704+0000 7f7026e78640 10 monclient: get_monmap_and_config 2024-09-18T12:31:15.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.704+0000 7f7026e78640 10 monclient: build_initial_monmap 2024-09-18T12:31:15.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.704+0000 7f7026e78640 10 monclient: monmap: 2024-09-18T12:31:15.706 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:15.706 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:15.706146+0000 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:15.706146+0000 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:15.707 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.704+0000 7f7026e78640 1 Processor -- start 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 1 -- start start 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient: init 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7020096a30 0x7f702009c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): picked mon.noname-a con 0x7f7020096a30 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f7020096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): picked mon.noname-b con 0x7f702009caa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:15.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7020097340 0x7f702012fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): picked mon.noname-c con 0x7f7020097340 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702009b520 con 0x7f7020096a30 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702009dbc0 con 0x7f7020097340 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70200981d0 con 0x7f702009caa0 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7026e78640 10 monclient(hunting): authenticate will time out at 1111.275500s 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7020097340 0x7f702012fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7024c14640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f7020096a30 msgr2=0x7f702009c560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7020097340 0x7f702012fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37860/0 (socket says 172.21.15.144:37860) 2024-09-18T12:31:15.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 -- 172.21.15.144:0/2253090584 learned_addr learned my addr 172.21.15.144:0/2253090584 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): get_auth_request con 0x7f7020097340 auth_method 0 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7024c14640 1 --2- 172.21.15.144:0/2253090584 >> v2:172.21.15.116:3300/0 conn(0x7f7020096a30 0x7f702009c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f701ffff640 1 --2- 172.21.15.144:0/2253090584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f7020096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f701ffff640 10 monclient(hunting): get_auth_request con 0x7f702009caa0 auth_method 0 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f701ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f701ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f701ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient(hunting): handle_auth_done global_id 5749 payload 274 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 -- 172.21.15.144:0/2253090584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 msgr2=0x7f7020096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 --2- 172.21.15.144:0/2253090584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f7020096e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 -- 172.21.15.144:0/2253090584 >> v2:172.21.15.116:3300/0 conn(0x7f7020096a30 msgr2=0x7f702009c560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 --2- 172.21.15.144:0/2253090584 >> v2:172.21.15.116:3300/0 conn(0x7f7020096a30 0x7f702009c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient: _finish_hunting 0 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 monclient: found mon.noname-c 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.705+0000 7f7025415640 1 -- 172.21.15.144:0/2253090584 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020097d40 con 0x7f7020097340 2024-09-18T12:31:15.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f7025415640 1 -- 172.21.15.144:0/2253090584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 msgr2=0x7f702012fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f7025415640 1 --2- 172.21.15.144:0/2253090584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702012fab0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f70200a2050 tx=0x7f7018009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f43672d2749947 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 1 -- 172.21.15.144:0/2253090584 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018007610 con 0x7f7020097340 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: dump: 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.720 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 1 -- 172.21.15.144:0/2253090584 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701802d050 con 0x7f7020097340 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f7026e78640 5 monclient: authenticate success, global_id 5749 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f7026e78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f7026e78640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 1 -- 172.21.15.144:0/2253090584 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70180079e0 con 0x7f7020097340 2024-09-18T12:31:15.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f7026e78640 4 set_mon_vals no callback set 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.708+0000 7f701f7fe640 10 monclient: dump: 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.722 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 10 monclient: shutdown 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 -- 172.21.15.144:0/2253090584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 msgr2=0x7f702012fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 --2- 172.21.15.144:0/2253090584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702012fab0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f70200a2050 tx=0x7f7018009840 comp rx=0 tx=0).stop 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 -- 172.21.15.144:0/2253090584 shutdown_connections 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 --2- 172.21.15.144:0/2253090584 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702012fab0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 --2- 172.21.15.144:0/2253090584 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f7020096e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 --2- 172.21.15.144:0/2253090584 >> v2:172.21.15.116:3300/0 conn(0x7f7020096a30 0x7f702009c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 -- 172.21.15.144:0/2253090584 >> 172.21.15.144:0/2253090584 conn(0x7f7020068c30 msgr2=0x7f70200a12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 -- 172.21.15.144:0/2253090584 shutdown_connections 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 1 -- 172.21.15.144:0/2253090584 wait complete. 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 10 monclient: build_initial_monmap 2024-09-18T12:31:15.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.709+0000 7f7026e78640 10 monclient: monmap: 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 Processor -- start 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 -- start start 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient: init 2024-09-18T12:31:15.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7020096a30 0x7f702013b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): picked mon.noname-a con 0x7f7020096a30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702013abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): picked mon.noname-b con 0x7f7020097340 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f702013ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): picked mon.noname-c con 0x7f702009caa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70201204f0 con 0x7f7020096a30 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702011ff30 con 0x7f7020097340 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:15.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702011d9c0 con 0x7f702009caa0 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7026e78640 10 monclient(hunting): authenticate will time out at 1111.280945s 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.710+0000 7f7024c14640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7020096a30 msgr2=0x7f702013b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702013abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7024c14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7020096a30 0x7f702013b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702013abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37876/0 (socket says 172.21.15.144:37876) 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 -- 172.21.15.144:0/2899760603 learned_addr learned my addr 172.21.15.144:0/2899760603 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): get_auth_request con 0x7f7020097340 auth_method 0 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:15.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f702013ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 10 monclient(hunting): get_auth_request con 0x7f702009caa0 auth_method 0 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f7025415640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient(hunting): handle_auth_done global_id 5752 payload 824 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 -- 172.21.15.144:0/2899760603 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 msgr2=0x7f702013ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f702013ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 -- 172.21.15.144:0/2899760603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7020096a30 msgr2=0x7f702013b500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7020096a30 0x7f702013b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 monclient: found mon.noname-b 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.711+0000 7f701ffff640 1 -- 172.21.15.144:0/2899760603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7018009ae0 con 0x7f7020097340 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701ffff640 1 -- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 msgr2=0x7f702013abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701ffff640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702013abb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f700c00d890 tx=0x7f700c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c00de20 con 0x7f7020097340 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: dump: 2024-09-18T12:31:15.728 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: _finish_auth 0 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c00bae0 con 0x7f7020097340 2024-09-18T12:31:15.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f7026e78640 5 monclient: authenticate success, global_id 5752 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f7026e78640 10 monclient: _renew_subs 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f7026e78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702011fc50 con 0x7f7020097340 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c00be80 con 0x7f7020097340 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701cff9640 10 monclient: dump: 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:15.730 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f701effd640 4 set_mon_vals no callback set 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f7026e78640 10 monclient: _renew_subs 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f7026e78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.713+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702013b290 con 0x7f7020097340 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f700c014640 con 0x7f7020097340 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f7026e78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f7026e78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f7026e78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702004bf60 con 0x7f7020097340 2024-09-18T12:31:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f701cff9640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(133..133 src has 13..133) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f700c04cad0 con 0x7f7020097340 2024-09-18T12:31:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f7024c14640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.714+0000 7f7024c14640 10 monclient: get_auth_request con 0x7f7004038440 auth_method 0 2024-09-18T12:31:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.717+0000 7f7024c14640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f70140038f0 tx=0x7f7014006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.719+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f700c01e460 con 0x7f7020097340 2024-09-18T12:31:15.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.719+0000 7f701cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:15.733 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.719+0000 7f701cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:15.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.882+0000 7f7026e78640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:31:15.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.882+0000 7f7026e78640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:31:15.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.882+0000 7f7026e78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.882+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f70200977c0 con 0x7f7020097340 2024-09-18T12:31:15.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.882+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v133) v1 ==== 73+0+6748 (secure 0 0 0) 0x7f700c01dbb0 con 0x7f7020097340 2024-09-18T12:31:15.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.882+0000 7f701cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:31:15.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.882+0000 7f701cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:15.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.983+0000 7f7026e78640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:31:15.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.983+0000 7f7026e78640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:31:15.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.983+0000 7f7026e78640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:15.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.983+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f7020072640 con 0x7f7020097340 2024-09-18T12:31:15.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.983+0000 7f701cff9640 1 -- 172.21.15.144:0/2899760603 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v133) v1 ==== 73+0+6748 (secure 0 0 0) 0x7f700c011340 con 0x7f7020097340 2024-09-18T12:31:15.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.983+0000 7f701cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T12:31:15.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.983+0000 7f701cff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:15.986 INFO:tasks.workunit.client.0.smithi144.stdout:{ 2024-09-18T12:31:15.986 INFO:tasks.workunit.client.0.smithi144.stdout: "9": [ 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -1, 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.987 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -2, 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -3, 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:15.988 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -4, 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.989 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -5, 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -6, 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.990 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -7, 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.991 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -8, 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -9, 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.992 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -10, 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.993 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -11, 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -12, 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.994 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -13, 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -14, 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.995 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -15, 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: "10": [ 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -1, 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.996 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:15.997 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: 0.263671875, 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -2, 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:15.998 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -3, 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:15.999 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.000 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -4, 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.001 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -5, 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.002 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -6, 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.003 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.004 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -7, 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.005 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -8, 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.006 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -9, 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: ], 2024-09-18T12:31:16.007 INFO:tasks.workunit.client.0.smithi144.stdout: [ 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: 0.087890625, 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: 0, 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: 0 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -10, 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.008 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -11, 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -12, 2024-09-18T12:31:16.009 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -13, 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.010 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -14, 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: }, 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: { 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: "bucket_id": -15, 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: "weight_set": [ 2024-09-18T12:31:16.011 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.012 INFO:tasks.workunit.client.0.smithi144.stdout: [], 2024-09-18T12:31:16.012 INFO:tasks.workunit.client.0.smithi144.stdout: [] 2024-09-18T12:31:16.012 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.012 INFO:tasks.workunit.client.0.smithi144.stdout: } 2024-09-18T12:31:16.012 INFO:tasks.workunit.client.0.smithi144.stdout: ] 2024-09-18T12:31:16.012 INFO:tasks.workunit.client.0.smithi144.stdout:} 2024-09-18T12:31:16.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 msgr2=0x7f700403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:16.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f70140038f0 tx=0x7f7014006d80 comp rx=0 tx=0).stop 2024-09-18T12:31:16.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 10 monclient: shutdown 2024-09-18T12:31:16.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 msgr2=0x7f702013abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702013abb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f700c00d890 tx=0x7f700c008040 comp rx=0 tx=0).stop 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 shutdown_connections 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f702009caa0 0x7f702013ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020097340 0x7f702013abb0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 --2- 172.21.15.144:0/2899760603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7020096a30 0x7f702013b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 >> 172.21.15.144:0/2899760603 conn(0x7f7020068c30 msgr2=0x7f702006a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 shutdown_connections 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:15.985+0000 7f7026e78640 1 -- 172.21.15.144:0/2899760603 wait complete. 2024-09-18T12:31:16.014 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T12:31:16.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.070+0000 7fa2a6b5d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:16.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.070+0000 7fa2a6b5d640 10 monclient: build_initial_monmap 2024-09-18T12:31:16.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.070+0000 7fa2a6b5d640 10 monclient: monmap: 2024-09-18T12:31:16.072 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:16.072 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:16.072063+0000 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:16.072063+0000 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:16.073 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:16.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.070+0000 7fa2a6b5d640 1 Processor -- start 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.070+0000 7fa2a6b5d640 1 -- start start 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.070+0000 7fa2a6b5d640 10 monclient: init 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2a009ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2a009f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2a0069d10 0x7fa2a013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa2a0069d10 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a006a120 con 0x7fa2a009ec50 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a0067420 con 0x7fa2a009f020 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a009fdc0 con 0x7fa2a0069d10 2024-09-18T12:31:16.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:16.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a6b5d640 10 monclient(hunting): authenticate will time out at 1111.641533s 2024-09-18T12:31:16.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a50fa640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fa2a0069d10 msgr2=0x7fa2a013a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:16.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a48f9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fa2a009ec50 legacy=0x7fa2a0131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:16.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa2a50fa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa2a0069d10 0x7fa2a013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:16.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa29ffff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa2a009f020 0x7fa2a0136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57704/0 (socket says 172.21.15.144:57704) 2024-09-18T12:31:16.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.071+0000 7fa29ffff640 1 -- 172.21.15.144:0/2679616140 learned_addr learned my addr 172.21.15.144:0/2679616140 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.073+0000 7fa29ffff640 1 -- 172.21.15.144:0/2679616140 >> v1:172.21.15.144:6789/0 conn(0x7fa2a009f020 legacy=0x7fa2a0136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.073+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242115718 0 0) 0x7fa2a0067420 con 0x7fa2a009f020 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.073+0000 7fa29f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.073+0000 7fa29f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.073+0000 7fa29f7fe640 10 monclient(hunting): my global_id is 5755 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.073+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2900034e0 con 0x7fa2a009f020 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 182971879 0 0) 0x7fa2900034e0 con 0x7fa2a009f020 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 >> v2:172.21.15.116:3300/0 conn(0x7fa2a0069d10 msgr2=0x7fa2a013a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 --2- 172.21.15.144:0/2679616140 >> v2:172.21.15.116:3300/0 conn(0x7fa2a0069d10 0x7fa2a013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 >> v1:172.21.15.116:6789/0 conn(0x7fa2a009ec50 legacy=0x7fa2a0131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:16.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a00a0510 con 0x7fa2a009f020 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fa28c0017a0 con 0x7fa2a009f020 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: dump: 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:16.084 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 >> v1:172.21.15.144:6789/0 conn(0x7fa2a009f020 legacy=0x7fa2a0136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:16.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa290003aa0 0x7fa290003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): picked mon.a con 0x7fa290003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa290004780 0x7fa290004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): picked mon.b con 0x7fa290004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900099c0 0x7fa29000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): picked mon.c con 0x7fa2900099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a009fdc0 con 0x7fa290003aa0 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29000c3c0 con 0x7fa290004780 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29000c580 con 0x7fa2900099c0 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:16.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa290004780 0x7fa290004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): get_auth_request con 0x7fa290004780 auth_method 0 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a48f9640 1 -- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa290003aa0 msgr2=0x7fa290003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a48f9640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa290003aa0 0x7fa290003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a50fa640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900099c0 0x7fa29000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:16.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient(hunting): handle_auth_done global_id 5755 payload 293 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 1 -- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900099c0 msgr2=0x7fa29000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900099c0 0x7fa29000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 1 -- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa290003aa0 msgr2=0x7fa290003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa290003aa0 0x7fa290003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 1 monclient: found mon.b 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 1 -- 172.21.15.144:0/2679616140 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900034e0 con 0x7fa290004780 2024-09-18T12:31:16.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 10 monclient: _finish_auth 0 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa29ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a50fa640 10 monclient: get_auth_request con 0x7fa2900099c0 auth_method 0 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a50fa640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa2900099c0 0x7fa29000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a6b5d640 5 monclient: authenticate success, global_id 5755 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a6b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.074+0000 7fa2a50fa640 1 -- 172.21.15.144:0/2679616140 reap_dead start 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29ffff640 1 -- 172.21.15.144:0/2679616140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa290004780 msgr2=0x7fa290004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29ffff640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa290004780 0x7fa290004bf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0014d0 tx=0x7fa2a009fca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467c113cdcb03fe2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c005890 con 0x7fa290004780 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 10 monclient: dump: 2024-09-18T12:31:16.089 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa2a6b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c005a10 con 0x7fa290004780 2024-09-18T12:31:16.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 1 -- 172.21.15.144:0/2679616140 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c005dd0 con 0x7fa290004780 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa2a6b5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa2a6b5d640 4 set_mon_vals no callback set 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.077+0000 7fa29f7fe640 10 monclient: dump: 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:16.091 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 10 monclient: shutdown 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/2679616140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa290004780 msgr2=0x7fa290004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa290004780 0x7fa290004bf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0014d0 tx=0x7fa2a009fca0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/2679616140 shutdown_connections 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/2679616140 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa290004780 0x7fa290004bf0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/2679616140 >> 172.21.15.144:0/2679616140 conn(0x7fa2a00a5d40 msgr2=0x7fa2a00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/2679616140 shutdown_connections 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/2679616140 wait complete. 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 10 monclient: build_initial_monmap 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 10 monclient: monmap: 2024-09-18T12:31:16.092 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.078+0000 7fa2a6b5d640 1 Processor -- start 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 1 -- start start 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient: init 2024-09-18T12:31:16.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa290004780 0x7fa2a009aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa290004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2a011cca0 0x7fa2a0130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa2a011cca0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 0x7fa2a011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa2a0131020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a0141dc0 con 0x7fa2a011cca0 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00a4920 con 0x7fa2a0131020 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:16.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a011e1c0 con 0x7fa290004780 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a6b5d640 10 monclient(hunting): authenticate will time out at 1111.649640s 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 0x7fa2a011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 0x7fa2a011b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:37884/0 (socket says 172.21.15.144:37884) 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa29ffff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2a011cca0 msgr2=0x7fa2a0130ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 -- 172.21.15.144:0/218595647 learned_addr learned my addr 172.21.15.144:0/218595647 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): get_auth_request con 0x7fa2a0131020 auth_method 0 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa29ffff640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2a011cca0 0x7fa2a0130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient(hunting): handle_auth_done global_id 5758 payload 824 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 -- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa290004780 msgr2=0x7fa2a009aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a48f9640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa290004780 0x7fa2a009aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa290004780 0x7fa2a009aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 -- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2a011cca0 msgr2=0x7fa2a0130ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2a011cca0 0x7fa2a0130ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 1 monclient: found mon.noname-b 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.079+0000 7fa2a50fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:16.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.080+0000 7fa2a50fa640 1 -- 172.21.15.144:0/218595647 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a011dc00 con 0x7fa2a0131020 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a50fa640 1 -- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 msgr2=0x7fa2a011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a50fa640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 0x7fa2a011b760 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00a1b50 tx=0x7fa298007600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29800f680 con 0x7fa2a0131020 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: dump: 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:16.099 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: _finish_auth 0 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a6b5d640 5 monclient: authenticate success, global_id 5758 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29800f820 con 0x7fa2a0131020 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29800fbe0 con 0x7fa2a0131020 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:16.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29cff9640 10 monclient: dump: 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa29effd640 4 set_mon_vals no callback set 2024-09-18T12:31:16.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a6b5d640 10 monclient: _renew_subs 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a6b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a011dee0 con 0x7fa2a0131020 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a6b5d640 10 monclient: _renew_subs 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a6b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.082+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a0141530 con 0x7fa2a0131020 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa298018400 con 0x7fa2a0131020 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa2a6b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa2a6b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa2a6b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a00008d0 con 0x7fa2a0131020 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa29cff9640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038470 0x7fa28403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(133..133 src has 13..133) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fa298006e40 con 0x7fa2a0131020 2024-09-18T12:31:16.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa2a48f9640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038470 0x7fa28403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:16.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.083+0000 7fa2a48f9640 10 monclient: get_auth_request con 0x7fa284038470 auth_method 0 2024-09-18T12:31:16.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.086+0000 7fa2a48f9640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038470 0x7fa28403a920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa294006fd0 tx=0x7fa294008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:16.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.087+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa298020dc0 con 0x7fa2a0131020 2024-09-18T12:31:16.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.087+0000 7fa29cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:16.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.087+0000 7fa29cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:16.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.249+0000 7fa2a6b5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:16.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.249+0000 7fa2a6b5d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:16.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.249+0000 7fa2a6b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:16.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.249+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7fa2a00a0200 con 0x7fa2a0131020 2024-09-18T12:31:16.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.802+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v134) v1 ==== 129+0+0 (secure 0 0 0) 0x7fa2980207c0 con 0x7fa2a0131020 2024-09-18T12:31:16.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.802+0000 7fa29cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:16.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.802+0000 7fa29cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:16.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.906+0000 7fa2a6b5d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:16.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.906+0000 7fa2a6b5d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:16.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.906+0000 7fa2a6b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:16.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:16.906+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7fa2a0064900 con 0x7fa2a0131020 2024-09-18T12:31:17.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.802+0000 7fa29cff9640 1 -- 172.21.15.144:0/218595647 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v135) v1 ==== 129+0+0 (secure 0 0 0) 0x7fa2980209a0 con 0x7fa2a0131020 2024-09-18T12:31:17.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.802+0000 7fa29cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:17.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.802+0000 7fa29cff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:17.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038470 msgr2=0x7fa28403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:17.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038470 0x7fa28403a920 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa294006fd0 tx=0x7fa294008040 comp rx=0 tx=0).stop 2024-09-18T12:31:17.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 10 monclient: shutdown 2024-09-18T12:31:17.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 msgr2=0x7fa2a011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 0x7fa2a011b760 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00a1b50 tx=0x7fa298007600 comp rx=0 tx=0).stop 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 shutdown_connections 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa284038470 0x7fa28403a920 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa2a0131020 0x7fa2a011b760 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2a011cca0 0x7fa2a0130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 --2- 172.21.15.144:0/218595647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa290004780 0x7fa2a009aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 >> 172.21.15.144:0/218595647 conn(0x7fa2a00a5d40 msgr2=0x7fa2a0134ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 shutdown_connections 2024-09-18T12:31:17.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.806+0000 7fa2a6b5d640 1 -- 172.21.15.144:0/218595647 wait complete. 2024-09-18T12:31:17.820 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-18T12:31:17.820 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:31:17.820 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.899+0000 7f955e8c9640 10 monclient: get_monmap_and_config 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.899+0000 7f955e8c9640 10 monclient: build_initial_monmap 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.900+0000 7f955e8c9640 10 monclient: monmap: 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:17.901906+0000 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:17.901906+0000 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:17.902 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:17.903 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:17.903 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:17.903 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:17.903 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:17.903 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:17.903 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.900+0000 7f955e8c9640 1 Processor -- start 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.900+0000 7f955e8c9640 1 -- start start 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.900+0000 7f955e8c9640 10 monclient: init 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9558098c70 0x7f9558099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f9558098c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9558099580 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9558099c70 0x7f955809e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f9558099c70 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955809f680 con 0x7f9558099580 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:17.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955809f420 con 0x7f9558099c70 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9558068a20 con 0x7f9558098c70 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955e8c9640 10 monclient(hunting): authenticate will time out at 1113.471412s 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9558098c70 0x7f9558099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f95577fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f9558099580 legacy=0x7f955809dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955ce66640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f9558099c70 msgr2=0x7f955809e1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9558098c70 0x7f9558099040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58026/0 (socket says 172.21.15.144:58026) 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 -- 172.21.15.144:0/322244701 learned_addr learned my addr 172.21.15.144:0/322244701 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): get_auth_request con 0x7f9558098c70 auth_method 0 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:17.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f955ce66640 1 --2- 172.21.15.144:0/322244701 >> v2:172.21.15.116:3300/0 conn(0x7f9558099c70 0x7f955809e1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient(hunting): handle_auth_done global_id 5761 payload 274 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 -- 172.21.15.144:0/322244701 >> v2:172.21.15.116:3300/0 conn(0x7f9558099c70 msgr2=0x7f955809e1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 --2- 172.21.15.144:0/322244701 >> v2:172.21.15.116:3300/0 conn(0x7f9558099c70 0x7f955809e1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 -- 172.21.15.144:0/322244701 >> v1:172.21.15.116:6789/0 conn(0x7f9558099580 legacy=0x7f955809dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 monclient: found mon.noname-c 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:17.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.901+0000 7f9557fff640 1 -- 172.21.15.144:0/322244701 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9558096a30 con 0x7f9558098c70 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9557fff640 1 -- 172.21.15.144:0/322244701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558098c70 msgr2=0x7f9558099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9557fff640 1 --2- 172.21.15.144:0/322244701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558098c70 0x7f9558099040 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9548009910 tx=0x7f95480091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed111e28496c64b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 1 -- 172.21.15.144:0/322244701 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9548032440 con 0x7f9558098c70 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: dump: 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:17.913 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f955e8c9640 5 monclient: authenticate success, global_id 5761 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f955e8c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 1 -- 172.21.15.144:0/322244701 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95480325a0 con 0x7f9558098c70 2024-09-18T12:31:17.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f955e8c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f955e8c9640 4 set_mon_vals no callback set 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 1 -- 172.21.15.144:0/322244701 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9548032940 con 0x7f9558098c70 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.904+0000 7f9556ffd640 10 monclient: dump: 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:17.915 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 10 monclient: shutdown 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 -- 172.21.15.144:0/322244701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558098c70 msgr2=0x7f9558099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 --2- 172.21.15.144:0/322244701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558098c70 0x7f9558099040 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9548009910 tx=0x7f95480091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 -- 172.21.15.144:0/322244701 shutdown_connections 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 --2- 172.21.15.144:0/322244701 >> v2:172.21.15.116:3300/0 conn(0x7f9558099c70 0x7f955809e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 --2- 172.21.15.144:0/322244701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558098c70 0x7f9558099040 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 -- 172.21.15.144:0/322244701 >> 172.21.15.144:0/322244701 conn(0x7f9558070e90 msgr2=0x7f95580a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 -- 172.21.15.144:0/322244701 shutdown_connections 2024-09-18T12:31:17.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.905+0000 7f955e8c9640 1 -- 172.21.15.144:0/322244701 wait complete. 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.906+0000 7f955e8c9640 10 monclient: build_initial_monmap 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.906+0000 7f955e8c9640 10 monclient: monmap: 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:17.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.906+0000 7f955e8c9640 1 Processor -- start 2024-09-18T12:31:17.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.906+0000 7f955e8c9640 1 -- start start 2024-09-18T12:31:17.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.906+0000 7f955e8c9640 10 monclient: init 2024-09-18T12:31:17.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:17.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:17.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558098c70 0x7f9558141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:17.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f9558098c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:17.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558099580 0x7f9558131e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:17.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f9558099580 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:17.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9558099c70 0x7f955809b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:17.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f9558099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:17.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955811f810 con 0x7f9558098c70 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955811f250 con 0x7f9558099580 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95581418b0 con 0x7f9558099c70 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955e8c9640 10 monclient(hunting): authenticate will time out at 1113.477098s 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558099580 0x7f9558131e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558099580 0x7f9558131e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58034/0 (socket says 172.21.15.144:58034) 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 -- 172.21.15.144:0/744576684 learned_addr learned my addr 172.21.15.144:0/744576684 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): get_auth_request con 0x7f9558099580 auth_method 0 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:17.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f9557fff640 1 -- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558098c70 msgr2=0x7f9558141370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f9557fff640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558098c70 0x7f9558141370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955ce66640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9558099c70 0x7f955809b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient(hunting): handle_auth_done global_id 5764 payload 824 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 -- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9558099c70 msgr2=0x7f955809b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9558099c70 0x7f955809b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 -- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558098c70 msgr2=0x7f9558141370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558098c70 0x7f9558141370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:17.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f95577fe640 1 -- 172.21.15.144:0/744576684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9548009450 con 0x7f9558099580 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955ce66640 10 monclient: get_auth_request con 0x7f9558099c70 auth_method 0 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.907+0000 7f955ce66640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9558099c70 0x7f955809b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f95577fe640 1 -- 172.21.15.144:0/744576684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558099580 msgr2=0x7f9558131e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f95577fe640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558099580 0x7f9558131e50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f954c00b630 tx=0x7f954c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 1 -- 172.21.15.144:0/744576684 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c015070 con 0x7f9558099580 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 10 monclient: dump: 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:17.922 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 10 monclient: _finish_auth 0 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f95577fe640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9558099580 0x7f9558131e50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f954c00b630 tx=0x7f954c0091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f954c006380 mon_map magic: 0 v1 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f955e8c9640 5 monclient: authenticate success, global_id 5764 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f955e8c9640 10 monclient: _renew_subs 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f955e8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f955e8c9640 1 -- 172.21.15.144:0/744576684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955811f530 con 0x7f9558099580 2024-09-18T12:31:17.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 1 -- 172.21.15.144:0/744576684 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c003d90 con 0x7f9558099580 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f953ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f955e8c9640 10 monclient: _renew_subs 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.909+0000 7f95567fc640 4 set_mon_vals no callback set 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f955e8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f955e8c9640 1 -- 172.21.15.144:0/744576684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95581420b0 con 0x7f9558099580 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f953ffff640 1 -- 172.21.15.144:0/744576684 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c006380 con 0x7f9558099580 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f953ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f953ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f953ffff640 10 monclient: dump: 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:17.924 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f953ffff640 1 -- 172.21.15.144:0/744576684 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f954c010440 con 0x7f9558099580 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f955e8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f955e8c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f955e8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.910+0000 7f955e8c9640 1 -- 172.21.15.144:0/744576684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95580008d0 con 0x7f9558099580 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.911+0000 7f953ffff640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9544038440 0x7f954403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.911+0000 7f9557fff640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9544038440 0x7f954403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:17.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.911+0000 7f9557fff640 10 monclient: get_auth_request con 0x7f9544038440 auth_method 0 2024-09-18T12:31:17.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.913+0000 7f9557fff640 1 --2- 172.21.15.144:0/744576684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9544038440 0x7f954403a8f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f95580a0c20 tx=0x7f9548032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:17.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:17.915+0000 7f953ffff640 1 -- 172.21.15.144:0/744576684 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(135..135 src has 13..135) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f954c04c1e0 con 0x7f9558099580 2024-09-18T12:31:18.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.532+0000 7f953ffff640 1 -- 172.21.15.144:0/744576684 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f954c018d60 con 0x7f9558099580 2024-09-18T12:31:18.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.532+0000 7f953ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:18.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.532+0000 7f953ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:18.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.699+0000 7f955e8c9640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-18T12:31:18.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.699+0000 7f955e8c9640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-18T12:31:18.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.699+0000 7f955e8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:18.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.699+0000 7f955e8c9640 1 -- 172.21.15.144:0/744576684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) v1 -- 0x7f955811efd0 con 0x7f9558099580 2024-09-18T12:31:18.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.699+0000 7f953ffff640 1 -- 172.21.15.144:0/744576684 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v135) v1 ==== 168+0+0 (secure 0 0 0) 0x7f954c018780 con 0x7f9558099580 2024-09-18T12:31:18.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.699+0000 7f953ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-18T12:31:18.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.699+0000 7f953ffff640 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-18T12:31:18.703 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-18T12:31:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:31:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-18T12:31:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -x 2024-09-18T12:31:18.705 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.779+0000 7f4be939d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.779+0000 7f4be939d640 10 monclient: build_initial_monmap 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.779+0000 7f4be939d640 10 monclient: monmap: 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:18.780996+0000 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:18.780996+0000 2024-09-18T12:31:18.781 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:18.782 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:18.782 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:18.782 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:18.782 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:18.782 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:18.782 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:18.782 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:18.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.779+0000 7f4be939d640 1 Processor -- start 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.779+0000 7f4be939d640 1 -- start start 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.779+0000 7f4be939d640 10 monclient: init 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4be409ec50 0x7f4be4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient(hunting): picked mon.noname-a con 0x7f4be409ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4be41321a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4be409f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:18.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be4067420 con 0x7f4be409f020 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be409fdc0 con 0x7f4be41321a0 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be40a0510 con 0x7f4be409ec50 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be939d640 10 monclient(hunting): authenticate will time out at 1114.350247s 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be27fc640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f4be41321a0 0x7f4be4068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51304/0 (socket says 172.21.15.144:51304) 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be2ffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f4be409ec50 msgr2=0x7f4be4131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be27fc640 1 -- 172.21.15.144:0/3654447546 learned_addr learned my addr 172.21.15.144:0/3654447546 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be37fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f4be409f020 legacy=0x7f4be4069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:18.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.780+0000 7f4be2ffd640 1 --2- 172.21.15.144:0/3654447546 >> v2:172.21.15.116:3300/0 conn(0x7f4be409ec50 0x7f4be4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:18.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be27fc640 1 -- 172.21.15.144:0/3654447546 >> v1:172.21.15.144:6789/0 conn(0x7f4be41321a0 legacy=0x7f4be4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:18.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117564880 0 0) 0x7f4be409fdc0 con 0x7f4be41321a0 2024-09-18T12:31:18.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:18.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:18.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient(hunting): my global_id is 5767 2024-09-18T12:31:18.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bcc0034e0 con 0x7f4be41321a0 2024-09-18T12:31:18.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667661020 0 0) 0x7f4bcc0034e0 con 0x7f4be41321a0 2024-09-18T12:31:18.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 >> v2:172.21.15.116:3300/0 conn(0x7f4be409ec50 msgr2=0x7f4be4131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:18.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 --2- 172.21.15.144:0/3654447546 >> v2:172.21.15.116:3300/0 conn(0x7f4be409ec50 0x7f4be4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:18.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 >> v1:172.21.15.116:6789/0 conn(0x7f4be409f020 legacy=0x7f4be4069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:18.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:31:18.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:31:18.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:18.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:31:18.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be40a0200 con 0x7f4be41321a0 2024-09-18T12:31:18.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:31:18.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:18.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be939d640 5 monclient: authenticate success, global_id 5767 2024-09-18T12:31:18.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f4bd80017a0 con 0x7f4be41321a0 2024-09-18T12:31:18.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:18.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:18.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:31:18.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: dump: 2024-09-18T12:31:18.795 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:18.796 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:18.796 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.796 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.796 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:18.796 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:18.797 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:18.797 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:18.797 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:18.797 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:18.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:31:18.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:18.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 >> v1:172.21.15.144:6789/0 conn(0x7f4be41321a0 legacy=0x7f4be4068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:18.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.782+0000 7f4be1ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:18.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bcc003aa0 0x7f4bcc003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): picked mon.b con 0x7f4bcc003aa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:18.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc004780 0x7f4bcc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): picked mon.a con 0x7f4bcc004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bcc0099c0 0x7f4bcc00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): picked mon.c con 0x7f4bcc0099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40a0510 con 0x7f4bcc004780 2024-09-18T12:31:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc00c3c0 con 0x7f4bcc003aa0 2024-09-18T12:31:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc00c580 con 0x7f4bcc0099c0 2024-09-18T12:31:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:31:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be939d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bcc003aa0 0x7f4bcc003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:18.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be27fc640 1 -- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc004780 msgr2=0x7f4bcc004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:18.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): get_auth_request con 0x7f4bcc003aa0 auth_method 0 2024-09-18T12:31:18.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:18.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:18.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:18.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be27fc640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc004780 0x7f4bcc004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:18.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:18.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:18.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:18.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be37fe640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bcc0099c0 0x7f4bcc00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:18.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be37fe640 10 monclient(hunting): get_auth_request con 0x7f4bcc0099c0 auth_method 0 2024-09-18T12:31:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_done global_id 5767 payload 293 2024-09-18T12:31:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 1 -- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bcc0099c0 msgr2=0x7f4bcc00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4bcc0099c0 0x7f4bcc00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 1 -- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc004780 msgr2=0x7f4bcc004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc004780 0x7f4bcc004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 1 monclient: found mon.b 2024-09-18T12:31:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 1 -- 172.21.15.144:0/3654447546 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc0034e0 con 0x7f4bcc003aa0 2024-09-18T12:31:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.783+0000 7f4be37fe640 1 -- 172.21.15.144:0/3654447546 reap_dead start 2024-09-18T12:31:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be2ffd640 1 -- 172.21.15.144:0/3654447546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bcc003aa0 msgr2=0x7f4bcc003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be2ffd640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bcc003aa0 0x7f4bcc003e90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009f90 tx=0x7f4bd40095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bc76e485519109 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4032430 con 0x7f4bcc003aa0 2024-09-18T12:31:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 10 monclient: dump: 2024-09-18T12:31:18.807 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:18.807 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:18.807 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.807 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.807 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:18.807 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:18.808 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:18.808 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:18.808 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:18.808 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:18.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:18.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd402d070 con 0x7f4bcc003aa0 2024-09-18T12:31:18.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:18.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 1 -- 172.21.15.144:0/3654447546 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4032820 con 0x7f4bcc003aa0 2024-09-18T12:31:18.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:18.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 4 set_mon_vals no callback set 2024-09-18T12:31:18.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:18.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:18.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be1ffb640 10 monclient: dump: 2024-09-18T12:31:18.810 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:18.810 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:18.810 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.810 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.811 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:18.811 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:18.811 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:18.811 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:18.811 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:18.811 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 10 monclient: shutdown 2024-09-18T12:31:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 1 -- 172.21.15.144:0/3654447546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bcc003aa0 msgr2=0x7f4bcc003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bcc003aa0 0x7f4bcc003e90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009f90 tx=0x7f4bd40095d0 comp rx=0 tx=0).stop 2024-09-18T12:31:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 1 -- 172.21.15.144:0/3654447546 shutdown_connections 2024-09-18T12:31:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 1 --2- 172.21.15.144:0/3654447546 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4bcc003aa0 0x7f4bcc003e90 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 1 -- 172.21.15.144:0/3654447546 >> 172.21.15.144:0/3654447546 conn(0x7f4be40a5d40 msgr2=0x7f4be40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 1 -- 172.21.15.144:0/3654447546 shutdown_connections 2024-09-18T12:31:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.786+0000 7f4be939d640 1 -- 172.21.15.144:0/3654447546 wait complete. 2024-09-18T12:31:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.787+0000 7f4be939d640 10 monclient: build_initial_monmap 2024-09-18T12:31:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.787+0000 7f4be939d640 10 monclient: monmap: 2024-09-18T12:31:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:18.815 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.787+0000 7f4be939d640 1 Processor -- start 2024-09-18T12:31:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.787+0000 7f4be939d640 1 -- start start 2024-09-18T12:31:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.787+0000 7f4be939d640 10 monclient: init 2024-09-18T12:31:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc003aa0 0x7f4be409a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): picked mon.noname-a con 0x7f4bcc003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4be4130a50 0x7f4be4130ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): picked mon.noname-c con 0x7f4be4130a50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4be411b760 0x7f4be411bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): picked mon.noname-b con 0x7f4be411b760 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41349e0 con 0x7f4bcc003aa0 2024-09-18T12:31:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be409b320 con 0x7f4be411b760 2024-09-18T12:31:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be411db90 con 0x7f4be4130a50 2024-09-18T12:31:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be939d640 10 monclient(hunting): authenticate will time out at 1114.358287s 2024-09-18T12:31:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4be411b760 0x7f4be411bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be2ffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc003aa0 msgr2=0x7f4be409a7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc003aa0 0x7f4be409a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4be411b760 0x7f4be411bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58048/0 (socket says 172.21.15.144:58048) 2024-09-18T12:31:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 -- 172.21.15.144:0/2519776878 learned_addr learned my addr 172.21.15.144:0/2519776878 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): get_auth_request con 0x7f4be411b760 auth_method 0 2024-09-18T12:31:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 1 --2- 172.21.15.144:0/2519776878 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4be4130a50 0x7f4be4130ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 10 monclient(hunting): get_auth_request con 0x7f4be4130a50 auth_method 0 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient(hunting): handle_auth_done global_id 5770 payload 824 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 -- 172.21.15.144:0/2519776878 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4be4130a50 msgr2=0x7f4be4130ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 --2- 172.21.15.144:0/2519776878 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4be4130a50 0x7f4be4130ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 -- 172.21.15.144:0/2519776878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc003aa0 msgr2=0x7f4be409a7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 --2- 172.21.15.144:0/2519776878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bcc003aa0 0x7f4be409a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.788+0000 7f4be37fe640 1 -- 172.21.15.144:0/2519776878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be411de70 con 0x7f4be411b760 2024-09-18T12:31:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.790+0000 7f4be37fe640 1 -- 172.21.15.144:0/2519776878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4be411b760 msgr2=0x7f4be411bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.790+0000 7f4be37fe640 1 --2- 172.21.15.144:0/2519776878 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4be411b760 0x7f4be411bc00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4be40a1b50 tx=0x7f4bdc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.790+0000 7f4bcaffd640 1 -- 172.21.15.144:0/2519776878 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc003cc0 con 0x7f4be411b760 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.790+0000 7f4bcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.790+0000 7f4bcaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.790+0000 7f4bcaffd640 10 monclient: dump: 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 1 -- 172.21.15.144:0/2519776878 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc003e20 con 0x7f4be411b760 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 5 monclient: authenticate success, global_id 5770 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 10 monclient: _renew_subs 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 1 -- 172.21.15.144:0/2519776878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be411e150 con 0x7f4be411b760 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 1 -- 172.21.15.144:0/2519776878 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc00dad0 con 0x7f4be411b760 2024-09-18T12:31:18.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 10 monclient: dump: 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:18.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be17fa640 4 set_mon_vals no callback set 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 10 monclient: _renew_subs 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 1 -- 172.21.15.144:0/2519776878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be411cc30 con 0x7f4be411b760 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4bcaffd640 1 -- 172.21.15.144:0/2519776878 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4bdc0050d0 con 0x7f4be411b760 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.791+0000 7f4be939d640 1 -- 172.21.15.144:0/2519776878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be411e390 con 0x7f4be411b760 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.792+0000 7f4bcaffd640 1 --2- 172.21.15.144:0/2519776878 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4bd0038440 0x7f4bd003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.792+0000 7f4bcaffd640 1 -- 172.21.15.144:0/2519776878 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(135..135 src has 13..135) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f4bdc04b550 con 0x7f4be411b760 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.792+0000 7f4be2ffd640 1 --2- 172.21.15.144:0/2519776878 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4bd0038440 0x7f4bd003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.792+0000 7f4be2ffd640 10 monclient: get_auth_request con 0x7f4bd0038440 auth_method 0 2024-09-18T12:31:18.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.794+0000 7f4be2ffd640 1 --2- 172.21.15.144:0/2519776878 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4bd0038440 0x7f4bd003a8f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4003c70 tx=0x7f4bd4035000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:18.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.796+0000 7f4bcaffd640 1 -- 172.21.15.144:0/2519776878 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4be411e390 con 0x7f4be411b760 2024-09-18T12:31:18.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.796+0000 7f4bcaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:18.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.796+0000 7f4bcaffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:18.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.958+0000 7f4be939d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:18.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.958+0000 7f4be939d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:18.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.958+0000 7f4be939d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:18.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.958+0000 7f4be939d640 1 -- 172.21.15.144:0/2519776878 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f4be411e340 con 0x7f4be411b760 2024-09-18T12:31:18.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.958+0000 7f4bcaffd640 1 -- 172.21.15.144:0/2519776878 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v135) v1 ==== 163+0+0 (secure 0 0 0) 0x7f4bdc019540 con 0x7f4be411b760 2024-09-18T12:31:18.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.958+0000 7f4bcaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T12:31:18.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:18.958+0000 7f4bcaffd640 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-18T12:31:18.961 INFO:tasks.workunit.client.0.smithi144.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-18T12:31:18.964 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:31:18.964 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-18T12:31:19.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.040+0000 7fb5d2f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:19.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.040+0000 7fb5d2f9d640 10 monclient: build_initial_monmap 2024-09-18T12:31:19.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.040+0000 7fb5d2f9d640 10 monclient: monmap: 2024-09-18T12:31:19.042 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:19.042 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:19.042089+0000 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:19.042089+0000 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:19.043 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:19.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.040+0000 7fb5d2f9d640 1 Processor -- start 2024-09-18T12:31:19.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.040+0000 7fb5d2f9d640 1 -- start start 2024-09-18T12:31:19.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.040+0000 7fb5d2f9d640 10 monclient: init 2024-09-18T12:31:19.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:19.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc09ec50 0x7fb5cc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb5cc09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb5cc1321a0 0x7fb5cc068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5cc1321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5cc09f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc067420 con 0x7fb5cc09f020 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:19.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5cc09fdc0 con 0x7fb5cc1321a0 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc0a0510 con 0x7fb5cc09ec50 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d2f9d640 10 monclient(hunting): authenticate will time out at 1114.611495s 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5cbfff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fb5cc1321a0 msgr2=0x7fb5cc068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb5cc1321a0 0x7fb5cc068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d153a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb5cc09f020 0x7fb5cc069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51308/0 (socket says 172.21.15.144:51308) 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d153a640 1 -- 172.21.15.144:0/701513724 learned_addr learned my addr 172.21.15.144:0/701513724 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 1 --2- 172.21.15.144:0/701513724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc09ec50 0x7fb5cc131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): get_auth_request con 0x7fb5cc09ec50 auth_method 0 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:19.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.041+0000 7fb5d0d39640 10 monclient(hunting): handle_auth_done global_id 64592 payload 274 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 1 -- 172.21.15.144:0/701513724 >> v2:172.21.15.116:3300/0 conn(0x7fb5cc1321a0 msgr2=0x7fb5cc068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 1 --2- 172.21.15.144:0/701513724 >> v2:172.21.15.116:3300/0 conn(0x7fb5cc1321a0 0x7fb5cc068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 1 -- 172.21.15.144:0/701513724 >> v1:172.21.15.144:6789/0 conn(0x7fb5cc09f020 legacy=0x7fb5cc069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 10 monclient: _finish_hunting 0 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 1 monclient: found mon.noname-b 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:19.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.042+0000 7fb5d0d39640 1 -- 172.21.15.144:0/701513724 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0a0200 con 0x7fb5cc09ec50 2024-09-18T12:31:19.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.044+0000 7fb5d0d39640 1 -- 172.21.15.144:0/701513724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc09ec50 msgr2=0x7fb5cc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.044+0000 7fb5d0d39640 1 --2- 172.21.15.144:0/701513724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc09ec50 0x7fb5cc131c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009850 tx=0x7fb5c00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=334312aca3f2633e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 1 -- 172.21.15.144:0/701513724 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0032440 con 0x7fb5cc09ec50 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: dump: 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:19.056 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 1 -- 172.21.15.144:0/701513724 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c00325a0 con 0x7fb5cc09ec50 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 1 -- 172.21.15.144:0/701513724 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0032940 con 0x7fb5cc09ec50 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 5 monclient: authenticate success, global_id 64592 2024-09-18T12:31:19.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 4 set_mon_vals no callback set 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5cb7fe640 10 monclient: dump: 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:19.058 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 10 monclient: shutdown 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/701513724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc09ec50 msgr2=0x7fb5cc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/701513724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc09ec50 0x7fb5cc131c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009850 tx=0x7fb5c00090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/701513724 shutdown_connections 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/701513724 >> v2:172.21.15.116:3300/0 conn(0x7fb5cc1321a0 0x7fb5cc068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/701513724 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc09ec50 0x7fb5cc131c60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/701513724 >> 172.21.15.144:0/701513724 conn(0x7fb5cc0a5d40 msgr2=0x7fb5cc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/701513724 shutdown_connections 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.045+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/701513724 wait complete. 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.046+0000 7fb5d2f9d640 10 monclient: build_initial_monmap 2024-09-18T12:31:19.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.046+0000 7fb5d2f9d640 10 monclient: monmap: 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.046+0000 7fb5d2f9d640 1 Processor -- start 2024-09-18T12:31:19.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.046+0000 7fb5d2f9d640 1 -- start start 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.046+0000 7fb5d2f9d640 10 monclient: init 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc09ec50 0x7fb5cc141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5cc09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 0x7fb5cc130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb5cc09f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc1321a0 0x7fb5cc131180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb5cc1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc11e040 con 0x7fb5cc09ec50 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:19.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc141f90 con 0x7fb5cc09f020 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc09c000 con 0x7fb5cc1321a0 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d2f9d640 10 monclient(hunting): authenticate will time out at 1114.617150s 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d0d39640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc09ec50 msgr2=0x7fb5cc141580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d0d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc09ec50 0x7fb5cc141580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 0x7fb5cc130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 0x7fb5cc130c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58062/0 (socket says 172.21.15.144:58062) 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 -- 172.21.15.144:0/339346145 learned_addr learned my addr 172.21.15.144:0/339346145 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): get_auth_request con 0x7fb5cc09f020 auth_method 0 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:19.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc1321a0 0x7fb5cc131180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 10 monclient(hunting): get_auth_request con 0x7fb5cc1321a0 auth_method 0 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5d153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient(hunting): handle_auth_done global_id 5773 payload 824 2024-09-18T12:31:19.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 -- 172.21.15.144:0/339346145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc1321a0 msgr2=0x7fb5cc131180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc1321a0 0x7fb5cc131180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 -- 172.21.15.144:0/339346145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc09ec50 msgr2=0x7fb5cc141580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc09ec50 0x7fb5cc141580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 monclient: found mon.noname-b 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.047+0000 7fb5cbfff640 1 -- 172.21.15.144:0/339346145 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0009390 con 0x7fb5cc09f020 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5cbfff640 1 -- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 msgr2=0x7fb5cc130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5cbfff640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 0x7fb5cc130c40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800b600 tx=0x7fb5b80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8015070 con 0x7fb5cc09f020 2024-09-18T12:31:19.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: dump: 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: _finish_auth 0 2024-09-18T12:31:19.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5d2f9d640 5 monclient: authenticate success, global_id 5773 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b8003d90 con 0x7fb5cc09f020 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8006380 con 0x7fb5cc09f020 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5c8ff9640 10 monclient: dump: 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:19.066 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5caffd640 4 set_mon_vals no callback set 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5d2f9d640 10 monclient: _renew_subs 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5d2f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.049+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc09de10 con 0x7fb5cc09f020 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5d2f9d640 10 monclient: _renew_subs 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5d2f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:19.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5cc09adc0 con 0x7fb5cc09f020 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb5b8010440 con 0x7fb5cc09f020 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5d2f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5d2f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5d2f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.050+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5cc11e180 con 0x7fb5cc09f020 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.051+0000 7fb5c8ff9640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5b0038470 0x7fb5b003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.051+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(135..135 src has 13..135) v4 ==== 8157+0+0 (secure 0 0 0) 0x7fb5b804bf60 con 0x7fb5cc09f020 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.051+0000 7fb5d0d39640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5b0038470 0x7fb5b003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.051+0000 7fb5d0d39640 10 monclient: get_auth_request con 0x7fb5b0038470 auth_method 0 2024-09-18T12:31:19.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.053+0000 7fb5d0d39640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5b0038470 0x7fb5b003a920 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc0a0150 tx=0x7fb5c0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:19.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.056+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb5cc11e180 con 0x7fb5cc09f020 2024-09-18T12:31:19.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.056+0000 7fb5c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:19.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.056+0000 7fb5c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:19.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.216+0000 7fb5d2f9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T12:31:19.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.216+0000 7fb5d2f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T12:31:19.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.216+0000 7fb5d2f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:19.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.216+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7fb5cc11e180 con 0x7fb5cc09f020 2024-09-18T12:31:19.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.814+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v136) v1 ==== 139+0+0 (secure 0 0 0) 0x7fb5cc11e180 con 0x7fb5cc09f020 2024-09-18T12:31:19.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.814+0000 7fb5c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T12:31:19.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.814+0000 7fb5c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:19.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.917+0000 7fb5d2f9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T12:31:19.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.917+0000 7fb5d2f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T12:31:19.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.917+0000 7fb5d2f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:19.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:19.917+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7fb5cc11e180 con 0x7fb5cc09f020 2024-09-18T12:31:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.818+0000 7fb5c8ff9640 1 -- 172.21.15.144:0/339346145 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v137) v1 ==== 139+0+0 (secure 0 0 0) 0x7fb5cc11e180 con 0x7fb5cc09f020 2024-09-18T12:31:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.818+0000 7fb5c8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T12:31:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.818+0000 7fb5c8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:20.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5b0038470 msgr2=0x7fb5b003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:20.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5b0038470 0x7fb5b003a920 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc0a0150 tx=0x7fb5c0032000 comp rx=0 tx=0).stop 2024-09-18T12:31:20.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 10 monclient: shutdown 2024-09-18T12:31:20.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:20.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 msgr2=0x7fb5cc130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:20.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 0x7fb5cc130c40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800b600 tx=0x7fb5b80091a0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 shutdown_connections 2024-09-18T12:31:20.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb5b0038470 0x7fb5b003a920 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb5cc1321a0 0x7fb5cc131180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5cc09f020 0x7fb5cc130c40 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 --2- 172.21.15.144:0/339346145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc09ec50 0x7fb5cc141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.821+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 >> 172.21.15.144:0/339346145 conn(0x7fb5cc0a5d40 msgr2=0x7fb5cc09e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:20.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.822+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 shutdown_connections 2024-09-18T12:31:20.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.822+0000 7fb5d2f9d640 1 -- 172.21.15.144:0/339346145 wait complete. 2024-09-18T12:31:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:31:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c fooset 2024-09-18T12:31:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 1 2024-09-18T12:31:20.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.912+0000 7ff7f62dd640 10 monclient: get_monmap_and_config 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.912+0000 7ff7f62dd640 10 monclient: build_initial_monmap 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.912+0000 7ff7f62dd640 10 monclient: monmap: 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:20.913161+0000 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:20.913161+0000 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:20.942 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:20.943 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:20.943 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:20.943 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:20.943 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:20.943 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:20.943 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:20.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 Processor -- start 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 -- start start 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient: init 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f01319c0 0x7ff7f0131fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f01319c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff7f009eb30 0x7ff7f009ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f009eb30 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7f0068e00 0x7ff7f0069280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f0068e00 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:20.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f006a260 con 0x7ff7f009eb30 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f0131d90 con 0x7ff7f0068e00 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00a0050 con 0x7ff7f01319c0 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f62dd640 10 monclient(hunting): authenticate will time out at 1116.482859s 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7e7fff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7ff7f009eb30 msgr2=0x7ff7f009ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:20.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.913+0000 7ff7f487a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7f0068e00 0x7ff7f0069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7e7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff7f009eb30 0x7ff7f009ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7f0068e00 0x7ff7f0069280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58078/0 (socket says 172.21.15.144:58078) 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 -- 172.21.15.144:0/2326157440 learned_addr learned my addr 172.21.15.144:0/2326157440 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): get_auth_request con 0x7ff7f0068e00 auth_method 0 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:20.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7effff640 1 --2- 172.21.15.144:0/2326157440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f01319c0 0x7ff7f0131fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7effff640 10 monclient(hunting): get_auth_request con 0x7ff7f01319c0 auth_method 0 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient(hunting): handle_auth_done global_id 5776 payload 274 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 -- 172.21.15.144:0/2326157440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f01319c0 msgr2=0x7ff7f0131fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 --2- 172.21.15.144:0/2326157440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f01319c0 0x7ff7f0131fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 -- 172.21.15.144:0/2326157440 >> v2:172.21.15.116:3300/0 conn(0x7ff7f009eb30 msgr2=0x7ff7f009ef00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 --2- 172.21.15.144:0/2326157440 >> v2:172.21.15.116:3300/0 conn(0x7ff7f009eb30 0x7ff7f009ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient: _finish_hunting 0 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 monclient: found mon.noname-c 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:20.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.914+0000 7ff7f487a640 1 -- 172.21.15.144:0/2326157440 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f009fe10 con 0x7ff7f0068e00 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f487a640 1 -- 172.21.15.144:0/2326157440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 msgr2=0x7ff7f0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f487a640 1 --2- 172.21.15.144:0/2326157440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f0069280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00a2e90 tx=0x7ff7e8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a3ed8011eb54ca3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 1 -- 172.21.15.144:0/2326157440 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8007610 con 0x7ff7f0068e00 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: dump: 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:20.956 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 5 monclient: authenticate success, global_id 5776 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 1 -- 172.21.15.144:0/2326157440 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e802d050 con 0x7ff7f0068e00 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:20.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 4 set_mon_vals no callback set 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 1 -- 172.21.15.144:0/2326157440 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80079a0 con 0x7ff7f0068e00 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7ef7fe640 10 monclient: dump: 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:20.958 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 10 monclient: shutdown 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 -- 172.21.15.144:0/2326157440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 msgr2=0x7ff7f0069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/2326157440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f0069280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00a2e90 tx=0x7ff7e8009840 comp rx=0 tx=0).stop 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 -- 172.21.15.144:0/2326157440 shutdown_connections 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/2326157440 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f0069280 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/2326157440 >> v2:172.21.15.116:3300/0 conn(0x7ff7f009eb30 0x7ff7f009ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/2326157440 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f01319c0 0x7ff7f0131fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 -- 172.21.15.144:0/2326157440 >> 172.21.15.144:0/2326157440 conn(0x7ff7f0071370 msgr2=0x7ff7f00a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 -- 172.21.15.144:0/2326157440 shutdown_connections 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.917+0000 7ff7f62dd640 1 -- 172.21.15.144:0/2326157440 wait complete. 2024-09-18T12:31:20.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.918+0000 7ff7f62dd640 10 monclient: build_initial_monmap 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.918+0000 7ff7f62dd640 10 monclient: monmap: 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:20.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.918+0000 7ff7f62dd640 1 Processor -- start 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.918+0000 7ff7f62dd640 1 -- start start 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.918+0000 7ff7f62dd640 10 monclient: init 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f009a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f0068e00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f009eb30 0x7ff7f0130830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f009eb30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7f01319c0 0x7ff7f011b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f01319c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f009bee0 con 0x7ff7f01319c0 2024-09-18T12:31:20.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00a4610 con 0x7ff7f0068e00 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f0136150 con 0x7ff7f009eb30 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f62dd640 10 monclient(hunting): authenticate will time out at 1116.488098s 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f009a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f009a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58080/0 (socket says 172.21.15.144:58080) 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f487a640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7f01319c0 msgr2=0x7ff7f011b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 -- 172.21.15.144:0/845983456 learned_addr learned my addr 172.21.15.144:0/845983456 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): get_auth_request con 0x7ff7f0068e00 auth_method 0 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:20.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7f487a640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7f01319c0 0x7ff7f011b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f009eb30 0x7ff7f0130830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 10 monclient(hunting): get_auth_request con 0x7ff7f009eb30 auth_method 0 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:20.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient(hunting): handle_auth_done global_id 5779 payload 824 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 -- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f009eb30 msgr2=0x7ff7f0130830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f009eb30 0x7ff7f0130830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 -- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7f01319c0 msgr2=0x7ff7f011b6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7f01319c0 0x7ff7f011b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 monclient: found mon.noname-b 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.919+0000 7ff7effff640 1 -- 172.21.15.144:0/845983456 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e8009ae0 con 0x7ff7f0068e00 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7effff640 1 -- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 msgr2=0x7ff7f009a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7effff640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f009a840 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000d8c0 tx=0x7ff7e0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e000de50 con 0x7ff7f0068e00 2024-09-18T12:31:20.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: dump: 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:20.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: _finish_auth 0 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e000bae0 con 0x7ff7f0068e00 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7f62dd640 5 monclient: authenticate success, global_id 5779 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e000be80 con 0x7ff7f0068e00 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7f62dd640 10 monclient: _renew_subs 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7f62dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f011df10 con 0x7ff7f0068e00 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7eeffd640 4 set_mon_vals no callback set 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7ecff9640 10 monclient: dump: 2024-09-18T12:31:20.966 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7f62dd640 10 monclient: _renew_subs 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7f62dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.921+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f011d950 con 0x7ff7f0068e00 2024-09-18T12:31:20.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.922+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff7e0014640 con 0x7ff7f0068e00 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.922+0000 7ff7ecff9640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff7d0038440 0x7ff7d003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.922+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(137..137 src has 13..137) v4 ==== 8157+0+0 (secure 0 0 0) 0x7ff7e004cbd0 con 0x7ff7f0068e00 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.922+0000 7ff7f62dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.922+0000 7ff7f62dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.922+0000 7ff7f62dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.922+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f004bf60 con 0x7ff7f0068e00 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.923+0000 7ff7e7fff640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff7d0038440 0x7ff7d003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.923+0000 7ff7e7fff640 10 monclient: get_auth_request con 0x7ff7d0038440 auth_method 0 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.925+0000 7ff7e7fff640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff7d0038440 0x7ff7d003a8f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff7f009eb00 tx=0x7ff7d80090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.927+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff7e004ce60 con 0x7ff7f0068e00 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.927+0000 7ff7ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:20.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:20.927+0000 7ff7ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.087+0000 7ff7f62dd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.087+0000 7ff7f62dd640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.087+0000 7ff7f62dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.087+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7ff7f00a1320 con 0x7ff7f0068e00 2024-09-18T12:31:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.088+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v137) v1 ==== 71+0+14 (secure 0 0 0) 0x7ff7e002c080 con 0x7ff7f0068e00 2024-09-18T12:31:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.088+0000 7ff7ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.088+0000 7ff7ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.186+0000 7ff7f62dd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.186+0000 7ff7f62dd640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.186+0000 7ff7f62dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.186+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7ff7f0141af0 con 0x7ff7f0068e00 2024-09-18T12:31:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.186+0000 7ff7ecff9640 1 -- 172.21.15.144:0/845983456 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v137) v1 ==== 71+0+14 (secure 0 0 0) 0x7ff7e0011340 con 0x7ff7f0068e00 2024-09-18T12:31:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.186+0000 7ff7ecff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.186+0000 7ff7ecff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff7d0038440 msgr2=0x7ff7d003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff7d0038440 0x7ff7d003a8f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff7f009eb00 tx=0x7ff7d80090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 10 monclient: shutdown 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 msgr2=0x7ff7f009a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f009a840 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000d8c0 tx=0x7ff7e0008040 comp rx=0 tx=0).stop 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 shutdown_connections 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff7d0038440 0x7ff7d003a8f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7f01319c0 0x7ff7f011b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7f009eb30 0x7ff7f0130830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 --2- 172.21.15.144:0/845983456 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7f0068e00 0x7ff7f009a840 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 >> 172.21.15.144:0/845983456 conn(0x7ff7f0071370 msgr2=0x7ff7f00a4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 shutdown_connections 2024-09-18T12:31:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.188+0000 7ff7f62dd640 1 -- 172.21.15.144:0/845983456 wait complete. 2024-09-18T12:31:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set rm fooset 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.273+0000 7f8f4379d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.273+0000 7f8f4379d640 10 monclient: build_initial_monmap 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.273+0000 7f8f4379d640 10 monclient: monmap: 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:21.274845+0000 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:21.274845+0000 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:21.275 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:21.276 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:21.276 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:21.276 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:21.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.274+0000 7f8f4379d640 1 Processor -- start 2024-09-18T12:31:21.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.274+0000 7f8f4379d640 1 -- start start 2024-09-18T12:31:21.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.274+0000 7f8f4379d640 10 monclient: init 2024-09-18T12:31:21.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:21.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-18T12:31:21.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8f3c0a0ad0 0x7f8f3c0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:21.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c0a0ad0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c09ee60 0x7f8f3c09f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3c09ee60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f3c131ad0 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c06a5b0 con 0x7f8f3c131ad0 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3c09f7a0 con 0x7f8f3c0a0ad0 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c0a0ea0 con 0x7f8f3c09ee60 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f4379d640 10 monclient(hunting): authenticate will time out at 1116.844220s 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8f3c0a0ad0 0x7f8f3c0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41d3a640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f8f3c131ad0 legacy=0x7f8f3c131f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8f3c0a0ad0 0x7f8f3c0691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58092/0 (socket says 172.21.15.144:58092) 2024-09-18T12:31:21.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 -- 172.21.15.144:0/3959422762 learned_addr learned my addr 172.21.15.144:0/3959422762 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): get_auth_request con 0x7f8f3c0a0ad0 auth_method 0 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f40d38640 1 --2- 172.21.15.144:0/3959422762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c09ee60 0x7f8f3c09f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f40d38640 10 monclient(hunting): get_auth_request con 0x7f8f3c09ee60 auth_method 0 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f40d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f40d38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:21.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f40d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient(hunting): handle_auth_done global_id 5782 payload 274 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 -- 172.21.15.144:0/3959422762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c09ee60 msgr2=0x7f8f3c09f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 --2- 172.21.15.144:0/3959422762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c09ee60 0x7f8f3c09f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 -- 172.21.15.144:0/3959422762 >> v1:172.21.15.116:6789/0 conn(0x7f8f3c131ad0 legacy=0x7f8f3c131f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient: _finish_hunting 0 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 monclient: found mon.noname-c 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.275+0000 7f8f41539640 1 -- 172.21.15.144:0/3959422762 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c0a0460 con 0x7f8f3c0a0ad0 2024-09-18T12:31:21.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f41539640 1 -- 172.21.15.144:0/3959422762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c0a0ad0 msgr2=0x7f8f3c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:21.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f41539640 1 --2- 172.21.15.144:0/3959422762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c0a0ad0 0x7f8f3c0691d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009910 tx=0x7f8f2c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74f9fade3ba4200d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:21.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 1 -- 172.21.15.144:0/3959422762 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c02ae90 con 0x7f8f3c0a0ad0 2024-09-18T12:31:21.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:21.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: dump: 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:21.288 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: _finish_auth 0 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f4379d640 5 monclient: authenticate success, global_id 5782 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f4379d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 1 -- 172.21.15.144:0/3959422762 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c032440 con 0x7f8f3c0a0ad0 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f4379d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f4379d640 4 set_mon_vals no callback set 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 1 -- 172.21.15.144:0/3959422762 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0327e0 con 0x7f8f3c0a0ad0 2024-09-18T12:31:21.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.278+0000 7f8f33fff640 10 monclient: dump: 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:21.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 10 monclient: shutdown 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 -- 172.21.15.144:0/3959422762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c0a0ad0 msgr2=0x7f8f3c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 --2- 172.21.15.144:0/3959422762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c0a0ad0 0x7f8f3c0691d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009910 tx=0x7f8f2c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 -- 172.21.15.144:0/3959422762 shutdown_connections 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 --2- 172.21.15.144:0/3959422762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c09ee60 0x7f8f3c09f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 --2- 172.21.15.144:0/3959422762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c0a0ad0 0x7f8f3c0691d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 -- 172.21.15.144:0/3959422762 >> 172.21.15.144:0/3959422762 conn(0x7f8f3c071370 msgr2=0x7f8f3c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 -- 172.21.15.144:0/3959422762 shutdown_connections 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 -- 172.21.15.144:0/3959422762 wait complete. 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 10 monclient: build_initial_monmap 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 10 monclient: monmap: 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:21.291 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.279+0000 7f8f4379d640 1 Processor -- start 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 1 -- start start 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient: init 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:21.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3c09ee60 0x7f8f3c140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f3c09ee60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c0a0ad0 0x7f8f3c130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c0a0ad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 0x7f8f3c11b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3c131ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c11f100 con 0x7f8f3c09ee60 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c11d870 con 0x7f8f3c131ad0 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:21.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c141e10 con 0x7f8f3c0a0ad0 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f4379d640 10 monclient(hunting): authenticate will time out at 1116.849608s 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41539640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3c09ee60 msgr2=0x7f8f3c140fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3c09ee60 0x7f8f3c140fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 0x7f8f3c11b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 0x7f8f3c11b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58094/0 (socket says 172.21.15.144:58094) 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 1 -- 172.21.15.144:0/1513235622 learned_addr learned my addr 172.21.15.144:0/1513235622 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): get_auth_request con 0x7f8f3c131ad0 auth_method 0 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c0a0ad0 0x7f8f3c130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:21.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 10 monclient(hunting): get_auth_request con 0x7f8f3c0a0ad0 auth_method 0 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f40d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 10 monclient(hunting): handle_auth_done global_id 5785 payload 824 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.280+0000 7f8f41d3a640 1 -- 172.21.15.144:0/1513235622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c0a0ad0 msgr2=0x7f8f3c130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:21.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c0a0ad0 0x7f8f3c130cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 1 -- 172.21.15.144:0/1513235622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3c09ee60 msgr2=0x7f8f3c140fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3c09ee60 0x7f8f3c140fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 10 monclient: _finish_hunting 0 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 1 monclient: found mon.noname-b 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.281+0000 7f8f41d3a640 1 -- 172.21.15.144:0/1513235622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009450 con 0x7f8f3c131ad0 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f41d3a640 1 -- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 msgr2=0x7f8f3c11b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f41d3a640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 0x7f8f3c11b900 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400bb10 tx=0x7f8f340097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3400cbc0 con 0x7f8f3c131ad0 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:21.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: dump: 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: _finish_auth 0 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:21.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3400cd40 con 0x7f8f3c131ad0 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f4379d640 5 monclient: authenticate success, global_id 5785 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f4379d640 10 monclient: _renew_subs 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34006150 con 0x7f8f3c131ad0 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f4379d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3c11d360 con 0x7f8f3c131ad0 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f317fa640 10 monclient: dump: 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.298 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f337fe640 4 set_mon_vals no callback set 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f4379d640 10 monclient: _renew_subs 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f4379d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.283+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3c11eed0 con 0x7f8f3c131ad0 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8f34012020 con 0x7f8f3c131ad0 2024-09-18T12:31:21.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f4379d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:21.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f4379d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:21.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f4379d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3c0008d0 con 0x7f8f3c131ad0 2024-09-18T12:31:21.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f317fa640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f20038440 0x7f8f2003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:21.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(137..137 src has 13..137) v4 ==== 8157+0+0 (secure 0 0 0) 0x7f8f3404c3a0 con 0x7f8f3c131ad0 2024-09-18T12:31:21.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f41539640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f20038440 0x7f8f2003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:21.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.284+0000 7f8f41539640 10 monclient: get_auth_request con 0x7f8f20038440 auth_method 0 2024-09-18T12:31:21.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.287+0000 7f8f41539640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f20038440 0x7f8f2003a8f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c0a00e0 tx=0x7f8f2c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:21.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.288+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8f3401d900 con 0x7f8f3c131ad0 2024-09-18T12:31:21.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.288+0000 7f8f317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:21.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.288+0000 7f8f317fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:21.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.449+0000 7f8f4379d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T12:31:21.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.450+0000 7f8f4379d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T12:31:21.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.450+0000 7f8f4379d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.450+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f8f3c11f2a0 con 0x7f8f3c131ad0 2024-09-18T12:31:21.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.828+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v138) v1 ==== 89+0+0 (secure 0 0 0) 0x7f8f3401d300 con 0x7f8f3c131ad0 2024-09-18T12:31:21.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.828+0000 7f8f317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T12:31:21.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.828+0000 7f8f317fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:21.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.930+0000 7f8f4379d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T12:31:21.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.930+0000 7f8f4379d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T12:31:21.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.930+0000 7f8f4379d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:21.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:21.930+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f8f3c072640 con 0x7f8f3c131ad0 2024-09-18T12:31:22.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.830+0000 7f8f317fa640 1 -- 172.21.15.144:0/1513235622 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v139) v1 ==== 89+0+0 (secure 0 0 0) 0x7f8f3401d4e0 con 0x7f8f3c131ad0 2024-09-18T12:31:22.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.830+0000 7f8f317fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T12:31:22.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.830+0000 7f8f317fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:22.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f20038440 msgr2=0x7f8f2003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:22.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f20038440 0x7f8f2003a8f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c0a00e0 tx=0x7f8f2c032000 comp rx=0 tx=0).stop 2024-09-18T12:31:22.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 10 monclient: shutdown 2024-09-18T12:31:22.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:22.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 msgr2=0x7f8f3c11b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:22.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 0x7f8f3c11b900 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400bb10 tx=0x7f8f340097d0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 shutdown_connections 2024-09-18T12:31:22.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8f20038440 0x7f8f2003a8f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f3c131ad0 0x7f8f3c11b900 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f3c0a0ad0 0x7f8f3c130cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 --2- 172.21.15.144:0/1513235622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3c09ee60 0x7f8f3c140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 >> 172.21.15.144:0/1513235622 conn(0x7f8f3c071370 msgr2=0x7f8f3c09d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:22.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 shutdown_connections 2024-09-18T12:31:22.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.834+0000 7f8f4379d640 1 -- 172.21.15.144:0/1513235622 wait complete. 2024-09-18T12:31:22.848 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:31:22.849 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c fooset 2024-09-18T12:31:22.849 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 0 2024-09-18T12:31:22.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.924+0000 7f015aa2b640 10 monclient: get_monmap_and_config 2024-09-18T12:31:22.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.924+0000 7f015aa2b640 10 monclient: build_initial_monmap 2024-09-18T12:31:22.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.924+0000 7f015aa2b640 10 monclient: monmap: 2024-09-18T12:31:22.925 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:22.925 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:22.925334+0000 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:22.925334+0000 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:22.926 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 1 Processor -- start 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 1 -- start start 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient: init 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f015406a6e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0154099c70 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f015413a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient(hunting): picked mon.noname-b con 0x7f015409d1c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:22.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015409ea00 con 0x7f0154099c70 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154069f70 con 0x7f015406a6e0 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015406aab0 con 0x7f015409d1c0 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f015aa2b640 10 monclient(hunting): authenticate will time out at 1118.494741s 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f0153fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f015406a6e0 0x7f015409cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51324/0 (socket says 172.21.15.144:51324) 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f0153fff640 1 -- 172.21.15.144:0/526846352 learned_addr learned my addr 172.21.15.144:0/526846352 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.925+0000 7f01537fe640 1 -- 172.21.15.144:0/526846352 >> v1:172.21.15.116:6789/0 conn(0x7f0154099c70 legacy=0x7f0154136be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 1 --2- 172.21.15.144:0/526846352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f015413a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): get_auth_request con 0x7f015409d1c0 auth_method 0 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:22.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient(hunting): handle_auth_done global_id 64610 payload 274 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 1 -- 172.21.15.144:0/526846352 >> v1:172.21.15.144:6789/0 conn(0x7f015406a6e0 legacy=0x7f015409cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 1 -- 172.21.15.144:0/526846352 >> v1:172.21.15.116:6789/0 conn(0x7f0154099c70 legacy=0x7f0154136be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient: _finish_hunting 0 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 1 monclient: found mon.noname-b 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:22.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.926+0000 7f0158fc8640 1 -- 172.21.15.144:0/526846352 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154069c00 con 0x7f015409d1c0 2024-09-18T12:31:22.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0158fc8640 1 -- 172.21.15.144:0/526846352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 msgr2=0x7f015413a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:22.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0158fc8640 1 --2- 172.21.15.144:0/526846352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f015413a310 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f014c009e90 tx=0x7f014c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f986dac622f2a2f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:22.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 1 -- 172.21.15.144:0/526846352 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c007840 con 0x7f015409d1c0 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: dump: 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:22.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 1 -- 172.21.15.144:0/526846352 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c02d050 con 0x7f015409d1c0 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 1 -- 172.21.15.144:0/526846352 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c007bd0 con 0x7f015409d1c0 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f0152ffd640 10 monclient: dump: 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:22.939 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f015aa2b640 5 monclient: authenticate success, global_id 64610 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f015aa2b640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.929+0000 7f015aa2b640 4 set_mon_vals no callback set 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 10 monclient: shutdown 2024-09-18T12:31:22.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 1 -- 172.21.15.144:0/526846352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 msgr2=0x7f015413a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 1 --2- 172.21.15.144:0/526846352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f015413a310 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f014c009e90 tx=0x7f014c009780 comp rx=0 tx=0).stop 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 1 -- 172.21.15.144:0/526846352 shutdown_connections 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 1 --2- 172.21.15.144:0/526846352 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f015413a310 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 1 -- 172.21.15.144:0/526846352 >> 172.21.15.144:0/526846352 conn(0x7f01540a3a50 msgr2=0x7f015409faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 1 -- 172.21.15.144:0/526846352 shutdown_connections 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 1 -- 172.21.15.144:0/526846352 wait complete. 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 10 monclient: build_initial_monmap 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.930+0000 7f015aa2b640 10 monclient: monmap: 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:22.941 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 Processor -- start 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 -- start start 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient: init 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015406a6e0 0x7f01541354c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): picked mon.noname-a con 0x7f015406a6e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 0x7f0154136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:22.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): picked mon.noname-b con 0x7f0154099c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f0154137310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): picked mon.noname-c con 0x7f015409d1c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015406eb80 con 0x7f015406a6e0 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01540a2880 con 0x7f0154099c70 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015411e0e0 con 0x7f015409d1c0 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f015aa2b640 10 monclient(hunting): authenticate will time out at 1118.500923s 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f01537fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 0x7f0154136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.931+0000 7f0153fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015406a6e0 msgr2=0x7f01541354c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:22.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0153fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015406a6e0 0x7f01541354c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 0x7f0154136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58108/0 (socket says 172.21.15.144:58108) 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 -- 172.21.15.144:0/3709496038 learned_addr learned my addr 172.21.15.144:0/3709496038 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): get_auth_request con 0x7f0154099c70 auth_method 0 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f0154137310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 10 monclient(hunting): get_auth_request con 0x7f015409d1c0 auth_method 0 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:22.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f0158fc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient(hunting): handle_auth_done global_id 5788 payload 824 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 -- 172.21.15.144:0/3709496038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 msgr2=0x7f0154137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f0154137310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 -- 172.21.15.144:0/3709496038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015406a6e0 msgr2=0x7f01541354c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015406a6e0 0x7f01541354c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:22.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.932+0000 7f01537fe640 1 -- 172.21.15.144:0/3709496038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c009a20 con 0x7f0154099c70 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f01537fe640 1 -- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 msgr2=0x7f0154136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f01537fe640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 0x7f0154136dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f014800b630 tx=0x7f01480091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148015070 con 0x7f0154099c70 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: dump: 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:22.946 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: _finish_auth 0 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f015aa2b640 5 monclient: authenticate success, global_id 5788 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f015aa2b640 10 monclient: _renew_subs 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0148003d90 con 0x7f0154099c70 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f015aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0154138bd0 con 0x7f0154099c70 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148006380 con 0x7f0154099c70 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:22.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f013bfff640 10 monclient: dump: 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f015aa2b640 10 monclient: _renew_subs 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f015aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:22.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01541349b0 con 0x7f0154099c70 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.934+0000 7f01527fc640 4 set_mon_vals no callback set 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.935+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0148010440 con 0x7f0154099c70 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.935+0000 7f015aa2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.935+0000 7f015aa2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.935+0000 7f015aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.935+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015404bf60 con 0x7f0154099c70 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.935+0000 7f013bfff640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0140038440 0x7f014003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.935+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(139..139 src has 13..139) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f014804bee0 con 0x7f0154099c70 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.936+0000 7f0153fff640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0140038440 0x7f014003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:22.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.936+0000 7f0153fff640 10 monclient: get_auth_request con 0x7f0140038440 auth_method 0 2024-09-18T12:31:22.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.938+0000 7f0153fff640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0140038440 0x7f014003a8f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01440047d0 tx=0x7f0144007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:22.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.940+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0148018a70 con 0x7f0154099c70 2024-09-18T12:31:22.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.940+0000 7f013bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:22.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:22.940+0000 7f013bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:23.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.102+0000 7f015aa2b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.102+0000 7f015aa2b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.102+0000 7f015aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.102+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f0154139010 con 0x7f0154099c70 2024-09-18T12:31:23.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.103+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v139) v1 ==== 71+0+7 (secure 0 0 0) 0x7f0148018a70 con 0x7f0154099c70 2024-09-18T12:31:23.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.103+0000 7f013bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.103+0000 7f013bfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:23.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.202+0000 7f015aa2b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.202+0000 7f015aa2b640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.202+0000 7f015aa2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.202+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f0154072640 con 0x7f0154099c70 2024-09-18T12:31:23.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.202+0000 7f013bfff640 1 -- 172.21.15.144:0/3709496038 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v139) v1 ==== 71+0+7 (secure 0 0 0) 0x7f0148018a70 con 0x7f0154099c70 2024-09-18T12:31:23.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.202+0000 7f013bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.202+0000 7f013bfff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0140038440 msgr2=0x7f014003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0140038440 0x7f014003a8f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01440047d0 tx=0x7f0144007b80 comp rx=0 tx=0).stop 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 10 monclient: shutdown 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 msgr2=0x7f0154136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 0x7f0154136dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f014800b630 tx=0x7f01480091a0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 shutdown_connections 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0140038440 0x7f014003a8f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015409d1c0 0x7f0154137310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0154099c70 0x7f0154136dd0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 --2- 172.21.15.144:0/3709496038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015406a6e0 0x7f01541354c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 >> 172.21.15.144:0/3709496038 conn(0x7f01540a3a50 msgr2=0x7f01540a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:23.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.204+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 shutdown_connections 2024-09-18T12:31:23.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.205+0000 7f015aa2b640 1 -- 172.21.15.144:0/3709496038 wait complete. 2024-09-18T12:31:23.215 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:31:23.215 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep barset 2024-09-18T12:31:23.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.287+0000 7f5f62b1d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:23.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.287+0000 7f5f62b1d640 10 monclient: build_initial_monmap 2024-09-18T12:31:23.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.287+0000 7f5f62b1d640 10 monclient: monmap: 2024-09-18T12:31:23.289 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:23.289 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.289 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:23.288625+0000 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:23.288625+0000 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:23.290 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:23.291 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 Processor -- start 2024-09-18T12:31:23.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 -- start start 2024-09-18T12:31:23.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient: init 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f5c06a6e0 0x7f5f5c09cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c06a6e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f5c099c70 0x7f5f5c09cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c099c70 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c09d9f0 0x7f5f5c09de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c09d9f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5c06aab0 con 0x7f5f5c099c70 2024-09-18T12:31:23.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5c069f70 con 0x7f5f5c06a6e0 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c09d530 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.288+0000 7f5f62b1d640 10 monclient(hunting): authenticate will time out at 1118.857934s 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f5bfff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f5f5c099c70 msgr2=0x7f5f5c09cff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f5c06a6e0 0x7f5f5c09cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f5bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f5c099c70 0x7f5f5c09cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f5c06a6e0 0x7f5f5c09cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58116/0 (socket says 172.21.15.144:58116) 2024-09-18T12:31:23.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 -- 172.21.15.144:0/1568650893 learned_addr learned my addr 172.21.15.144:0/1568650893 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): get_auth_request con 0x7f5f5c06a6e0 auth_method 0 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f610ba640 1 --2- 172.21.15.144:0/1568650893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c09d9f0 0x7f5f5c09de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f610ba640 10 monclient(hunting): get_auth_request con 0x7f5f5c09d9f0 auth_method 0 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f610ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:23.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f610ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f610ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient(hunting): handle_auth_done global_id 5791 payload 274 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 -- 172.21.15.144:0/1568650893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c09d9f0 msgr2=0x7f5f5c09de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 --2- 172.21.15.144:0/1568650893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c09d9f0 0x7f5f5c09de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 -- 172.21.15.144:0/1568650893 >> v2:172.21.15.116:3300/0 conn(0x7f5f5c099c70 msgr2=0x7f5f5c09cff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 --2- 172.21.15.144:0/1568650893 >> v2:172.21.15.116:3300/0 conn(0x7f5f5c099c70 0x7f5f5c09cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient: _finish_hunting 0 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 monclient: found mon.noname-c 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.289+0000 7f5f608b9640 1 -- 172.21.15.144:0/1568650893 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c069c00 con 0x7f5f5c06a6e0 2024-09-18T12:31:23.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f608b9640 1 -- 172.21.15.144:0/1568650893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c06a6e0 msgr2=0x7f5f5c09cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f608b9640 1 --2- 172.21.15.144:0/1568650893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c06a6e0 0x7f5f5c09cab0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c06ac00 tx=0x7f5f500091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db6ecd9e2142fce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 1 -- 172.21.15.144:0/1568650893 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50032440 con 0x7f5f5c06a6e0 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: dump: 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.307 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 5 monclient: authenticate success, global_id 5791 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 1 -- 172.21.15.144:0/1568650893 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f500325a0 con 0x7f5f5c06a6e0 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 4 set_mon_vals no callback set 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 1 -- 172.21.15.144:0/1568650893 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50032940 con 0x7f5f5c06a6e0 2024-09-18T12:31:23.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f5b7fe640 10 monclient: dump: 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.309 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 10 monclient: shutdown 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 -- 172.21.15.144:0/1568650893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c06a6e0 msgr2=0x7f5f5c09cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/1568650893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c06a6e0 0x7f5f5c09cab0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c06ac00 tx=0x7f5f500091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 -- 172.21.15.144:0/1568650893 shutdown_connections 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/1568650893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c09d9f0 0x7f5f5c09de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/1568650893 >> v2:172.21.15.116:3300/0 conn(0x7f5f5c099c70 0x7f5f5c09cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/1568650893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c06a6e0 0x7f5f5c09cab0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 -- 172.21.15.144:0/1568650893 >> 172.21.15.144:0/1568650893 conn(0x7f5f5c0a3a50 msgr2=0x7f5f5c09faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 -- 172.21.15.144:0/1568650893 shutdown_connections 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.292+0000 7f5f62b1d640 1 -- 172.21.15.144:0/1568650893 wait complete. 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.293+0000 7f5f62b1d640 10 monclient: build_initial_monmap 2024-09-18T12:31:23.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.293+0000 7f5f62b1d640 10 monclient: monmap: 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.293+0000 7f5f62b1d640 1 Processor -- start 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.293+0000 7f5f62b1d640 1 -- start start 2024-09-18T12:31:23.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.293+0000 7f5f62b1d640 10 monclient: init 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c06a6e0 0x7f5f5c141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c06a6e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c099c70 0x7f5f5c133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 0x7f5f5c11da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c09d9f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c141ff0 con 0x7f5f5c06a6e0 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c120f80 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c1214d0 con 0x7f5f5c099c70 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f62b1d640 10 monclient(hunting): authenticate will time out at 1118.863457s 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f608b9640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c06a6e0 msgr2=0x7f5f5c141130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f608b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c06a6e0 0x7f5f5c141130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 0x7f5f5c11da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 0x7f5f5c11da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58126/0 (socket says 172.21.15.144:58126) 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 -- 172.21.15.144:0/4092203935 learned_addr learned my addr 172.21.15.144:0/4092203935 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): get_auth_request con 0x7f5f5c09d9f0 auth_method 0 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:23.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c099c70 0x7f5f5c133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 10 monclient(hunting): get_auth_request con 0x7f5f5c099c70 auth_method 0 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient(hunting): handle_auth_done global_id 5794 payload 824 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 -- 172.21.15.144:0/4092203935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c099c70 msgr2=0x7f5f5c133df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c099c70 0x7f5f5c133df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 -- 172.21.15.144:0/4092203935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c06a6e0 msgr2=0x7f5f5c141130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c06a6e0 0x7f5f5c141130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient: _finish_hunting 0 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 monclient: found mon.noname-b 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.294+0000 7f5f610ba640 1 -- 172.21.15.144:0/4092203935 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50009450 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f610ba640 1 -- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 msgr2=0x7f5f5c11da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f610ba640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 0x7f5f5c11da70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009fd0 tx=0x7f5f54009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400ee90 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:23.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: dump: 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: _finish_auth 0 2024-09-18T12:31:23.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f62b1d640 5 monclient: authenticate success, global_id 5794 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f62b1d640 10 monclient: _renew_subs 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f54013590 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f62b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c06eb40 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54013950 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f58ff9640 10 monclient: dump: 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.317 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f62b1d640 10 monclient: _renew_subs 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f62b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c11ffb0 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.297+0000 7f5f5a7fc640 4 set_mon_vals no callback set 2024-09-18T12:31:23.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5f54013cc0 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f62b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f62b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f62b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c04bf60 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f58ff9640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f40038440 0x7f5f4003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(139..139 src has 13..139) v4 ==== 7825+0+0 (secure 0 0 0) 0x7f5f5404cc20 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f608b9640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f40038440 0x7f5f4003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.298+0000 7f5f608b9640 10 monclient: get_auth_request con 0x7f5f40038440 auth_method 0 2024-09-18T12:31:23.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.300+0000 7f5f608b9640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f40038440 0x7f5f4003a8f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c09f1c0 tx=0x7f5f50032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:23.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.302+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5f54014420 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.302+0000 7f5f58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:23.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.302+0000 7f5f58ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:23.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.463+0000 7f5f62b1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.463+0000 7f5f62b1d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.463+0000 7f5f62b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.463+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f5f5c09a900 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.463+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v139) v1 ==== 71+0+7 (secure 0 0 0) 0x7f5f540122e0 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.463+0000 7f5f58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.463+0000 7f5f58ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:23.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.560+0000 7f5f62b1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.560+0000 7f5f62b1d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.560+0000 7f5f62b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.560+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f5f5c072640 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.561+0000 7f5f58ff9640 1 -- 172.21.15.144:0/4092203935 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v139) v1 ==== 71+0+7 (secure 0 0 0) 0x7f5f540124c0 con 0x7f5f5c09d9f0 2024-09-18T12:31:23.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.561+0000 7f5f58ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:23.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.561+0000 7f5f58ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:23.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f40038440 msgr2=0x7f5f4003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f40038440 0x7f5f4003a8f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c09f1c0 tx=0x7f5f50032000 comp rx=0 tx=0).stop 2024-09-18T12:31:23.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 10 monclient: shutdown 2024-09-18T12:31:23.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:23.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 msgr2=0x7f5f5c11da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 0x7f5f5c11da70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009fd0 tx=0x7f5f54009900 comp rx=0 tx=0).stop 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 shutdown_connections 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5f40038440 0x7f5f4003a8f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f5c09d9f0 0x7f5f5c11da70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f5c099c70 0x7f5f5c133df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 --2- 172.21.15.144:0/4092203935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c06a6e0 0x7f5f5c141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 >> 172.21.15.144:0/4092203935 conn(0x7f5f5c0a3a50 msgr2=0x7f5f5c134df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 shutdown_connections 2024-09-18T12:31:23.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.563+0000 7f5f62b1d640 1 -- 172.21.15.144:0/4092203935 wait complete. 2024-09-18T12:31:23.572 INFO:tasks.workunit.client.0.smithi144.stdout:barset 2024-09-18T12:31:23.573 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set rm barset 2024-09-18T12:31:23.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.647+0000 7fddad080640 10 monclient: get_monmap_and_config 2024-09-18T12:31:23.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.647+0000 7fddad080640 10 monclient: build_initial_monmap 2024-09-18T12:31:23.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.647+0000 7fddad080640 10 monclient: monmap: 2024-09-18T12:31:23.648 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:23.648126+0000 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:23.648126+0000 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:23.649 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.647+0000 7fddad080640 1 Processor -- start 2024-09-18T12:31:23.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.647+0000 7fddad080640 1 -- start start 2024-09-18T12:31:23.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.647+0000 7fddad080640 10 monclient: init 2024-09-18T12:31:23.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:23.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:31:23.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdda809ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdda809f020 0x7fdda8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient(hunting): picked mon.noname-a con 0x7fdda809f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda8136d60 0x7fdda8139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient(hunting): picked mon.noname-b con 0x7fdda8136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8067420 con 0x7fdda809ec50 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda809fdc0 con 0x7fdda809f020 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda80a0510 con 0x7fdda8136d60 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fddad080640 10 monclient(hunting): authenticate will time out at 1119.217269s 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda659c640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fdda809f020 msgr2=0x7fdda8069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:23.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda6d9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fdda809ec50 0x7fdda8131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51334/0 (socket says 172.21.15.144:51334) 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda6d9d640 1 -- 172.21.15.144:0/2222405018 learned_addr learned my addr 172.21.15.144:0/2222405018 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda659c640 1 --2- 172.21.15.144:0/2222405018 >> v2:172.21.15.116:3300/0 conn(0x7fdda809f020 0x7fdda8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 1 --2- 172.21.15.144:0/2222405018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda8136d60 0x7fdda8139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): get_auth_request con 0x7fdda8136d60 auth_method 0 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient(hunting): handle_auth_done global_id 64622 payload 274 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 1 -- 172.21.15.144:0/2222405018 >> v2:172.21.15.116:3300/0 conn(0x7fdda809f020 msgr2=0x7fdda8069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 1 --2- 172.21.15.144:0/2222405018 >> v2:172.21.15.116:3300/0 conn(0x7fdda809f020 0x7fdda8069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 1 -- 172.21.15.144:0/2222405018 >> v1:172.21.15.144:6789/0 conn(0x7fdda809ec50 legacy=0x7fdda8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:23.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient: _finish_hunting 0 2024-09-18T12:31:23.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 1 monclient: found mon.noname-b 2024-09-18T12:31:23.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:23.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:23.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.648+0000 7fdda759e640 1 -- 172.21.15.144:0/2222405018 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda80a0200 con 0x7fdda8136d60 2024-09-18T12:31:23.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.651+0000 7fdda759e640 1 -- 172.21.15.144:0/2222405018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda8136d60 msgr2=0x7fdda8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:23.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.651+0000 7fdda759e640 1 --2- 172.21.15.144:0/2222405018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda8136d60 0x7fdda8139140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009e60 tx=0x7fdda0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c414fc696b3e0af8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:23.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 1 -- 172.21.15.144:0/2222405018 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0007840 con 0x7fdda8136d60 2024-09-18T12:31:23.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:23.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: dump: 2024-09-18T12:31:23.662 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: _finish_auth 0 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:23.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 1 -- 172.21.15.144:0/2222405018 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda002d050 con 0x7fdda8136d60 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 5 monclient: authenticate success, global_id 64622 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 1 -- 172.21.15.144:0/2222405018 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0007bd0 con 0x7fdda8136d60 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 4 set_mon_vals no callback set 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:23.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fdda5d9b640 10 monclient: dump: 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.665 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 10 monclient: shutdown 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 -- 172.21.15.144:0/2222405018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda8136d60 msgr2=0x7fdda8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 --2- 172.21.15.144:0/2222405018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda8136d60 0x7fdda8139140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009e60 tx=0x7fdda0009780 comp rx=0 tx=0).stop 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 -- 172.21.15.144:0/2222405018 shutdown_connections 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 --2- 172.21.15.144:0/2222405018 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda8136d60 0x7fdda8139140 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 --2- 172.21.15.144:0/2222405018 >> v2:172.21.15.116:3300/0 conn(0x7fdda809f020 0x7fdda8069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 -- 172.21.15.144:0/2222405018 >> 172.21.15.144:0/2222405018 conn(0x7fdda80a5d40 msgr2=0x7fdda80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 -- 172.21.15.144:0/2222405018 shutdown_connections 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.652+0000 7fddad080640 1 -- 172.21.15.144:0/2222405018 wait complete. 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.653+0000 7fddad080640 10 monclient: build_initial_monmap 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.653+0000 7fddad080640 10 monclient: monmap: 2024-09-18T12:31:23.666 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.653+0000 7fddad080640 1 Processor -- start 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.653+0000 7fddad080640 1 -- start start 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.653+0000 7fddad080640 10 monclient: init 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:23.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda809ec50 0x7fdda81312e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): picked mon.noname-c con 0x7fdda809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda809f020 0x7fdda8097020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): picked mon.noname-a con 0x7fdda809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 0x7fdda8097560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): picked mon.noname-b con 0x7fdda8136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda809ac40 con 0x7fdda809f020 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda809c000 con 0x7fdda8136d60 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda80a4e00 con 0x7fdda809ec50 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:23.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fddad080640 10 monclient(hunting): authenticate will time out at 1119.223386s 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 0x7fdda8097560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda659c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda809f020 msgr2=0x7fdda8097020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda809f020 0x7fdda8097020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 0x7fdda8097560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58128/0 (socket says 172.21.15.144:58128) 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 -- 172.21.15.144:0/2973215520 learned_addr learned my addr 172.21.15.144:0/2973215520 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): get_auth_request con 0x7fdda8136d60 auth_method 0 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:23.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda809ec50 0x7fdda81312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 10 monclient(hunting): get_auth_request con 0x7fdda809ec50 auth_method 0 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient(hunting): handle_auth_done global_id 5797 payload 824 2024-09-18T12:31:23.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 -- 172.21.15.144:0/2973215520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda809ec50 msgr2=0x7fdda81312e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda809ec50 0x7fdda81312e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 -- 172.21.15.144:0/2973215520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda809f020 msgr2=0x7fdda8097020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda809f020 0x7fdda8097020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient: _finish_hunting 0 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 monclient: found mon.noname-b 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.654+0000 7fdda759e640 1 -- 172.21.15.144:0/2973215520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda0009a20 con 0x7fdda8136d60 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdda759e640 1 -- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 msgr2=0x7fdda8097560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:23.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdda759e640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 0x7fdda8097560 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdda002aae0 tx=0x7fdda0037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0031cd0 con 0x7fdda8136d60 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: dump: 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.672 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: _finish_auth 0 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fddad080640 5 monclient: authenticate success, global_id 5797 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda0040070 con 0x7fdda8136d60 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fddad080640 10 monclient: _renew_subs 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fddad080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda8099690 con 0x7fdda8136d60 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:23.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0045640 con 0x7fdda8136d60 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdd8effd640 10 monclient: dump: 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:23.674 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fddad080640 10 monclient: _renew_subs 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fdda4d99640 4 set_mon_vals no callback set 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fddad080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.657+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda809a5f0 con 0x7fdda8136d60 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdda0003020 con 0x7fdda8136d60 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fddad080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fddad080640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fddad080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda80008d0 con 0x7fdda8136d60 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fdd8effd640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd94038470 0x7fdd9403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(139..139 src has 13..139) v4 ==== 7825+0+0 (secure 0 0 0) 0x7fdda006dcb0 con 0x7fdda8136d60 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fdda6d9d640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd94038470 0x7fdd9403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.658+0000 7fdda6d9d640 10 monclient: get_auth_request con 0x7fdd94038470 auth_method 0 2024-09-18T12:31:23.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.660+0000 7fdda6d9d640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd94038470 0x7fdd9403a920 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009a40 tx=0x7fdd98006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:23.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.662+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdda0031e30 con 0x7fdda8136d60 2024-09-18T12:31:23.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.662+0000 7fdd8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:23.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.662+0000 7fdd8effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:23.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.825+0000 7fddad080640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T12:31:23.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.825+0000 7fddad080640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T12:31:23.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.825+0000 7fddad080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.825+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7fdda8098ac0 con 0x7fdda8136d60 2024-09-18T12:31:23.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.844+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v140) v1 ==== 89+0+0 (secure 0 0 0) 0x7fdda00330c0 con 0x7fdda8136d60 2024-09-18T12:31:23.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.844+0000 7fdd8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T12:31:23.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.844+0000 7fdd8effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:23.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.947+0000 7fddad080640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T12:31:23.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.947+0000 7fddad080640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T12:31:23.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.947+0000 7fddad080640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:23.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:23.947+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7fdda8098ac0 con 0x7fdda8136d60 2024-09-18T12:31:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.846+0000 7fdd8effd640 1 -- 172.21.15.144:0/2973215520 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v141) v1 ==== 89+0+0 (secure 0 0 0) 0x7fdda003a3a0 con 0x7fdda8136d60 2024-09-18T12:31:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.846+0000 7fdd8effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T12:31:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.846+0000 7fdd8effd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd94038470 msgr2=0x7fdd9403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd94038470 0x7fdd9403a920 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009a40 tx=0x7fdd98006f50 comp rx=0 tx=0).stop 2024-09-18T12:31:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 10 monclient: shutdown 2024-09-18T12:31:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 msgr2=0x7fdda8097560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 0x7fdda8097560 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdda002aae0 tx=0x7fdda0037000 comp rx=0 tx=0).stop 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 shutdown_connections 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd94038470 0x7fdd9403a920 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdda8136d60 0x7fdda8097560 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda809f020 0x7fdda8097020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 --2- 172.21.15.144:0/2973215520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdda809ec50 0x7fdda81312e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 >> 172.21.15.144:0/2973215520 conn(0x7fdda80a5d40 msgr2=0x7fdda80a31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.851+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 shutdown_connections 2024-09-18T12:31:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.852+0000 7fddad080640 1 -- 172.21.15.144:0/2973215520 wait complete. 2024-09-18T12:31:24.867 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:31:24.867 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -c barset 2024-09-18T12:31:24.867 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep -q 0 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.942+0000 7f3ecb95d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.942+0000 7f3ecb95d640 10 monclient: build_initial_monmap 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.942+0000 7f3ecb95d640 10 monclient: monmap: 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:24.943772+0000 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:24.943772+0000 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:24.944 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:24.945 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:24.945 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:24.945 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.943+0000 7f3ecb95d640 1 Processor -- start 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.943+0000 7f3ecb95d640 1 -- start start 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.943+0000 7f3ecb95d640 10 monclient: init 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec409ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ec409c9f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ec4096f00 0x7f3ec412e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:24.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec4096f00 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ec409d580 0x7f3ec412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ec409d580 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40983d0 con 0x7f3ec4096f00 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec409cdc0 con 0x7f3ec409d580 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40992f0 con 0x7f3ec409c9f0 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ecb95d640 10 monclient(hunting): authenticate will time out at 1120.513164s 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ec409d580 0x7f3ec412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec8ef8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f3ec4096f00 msgr2=0x7f3ec412e8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:24.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec8ef8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ec4096f00 0x7f3ec412e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ec409d580 0x7f3ec412ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58138/0 (socket says 172.21.15.144:58138) 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 -- 172.21.15.144:0/4013050792 learned_addr learned my addr 172.21.15.144:0/4013050792 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): get_auth_request con 0x7f3ec409d580 auth_method 0 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec96f9640 1 --2- 172.21.15.144:0/4013050792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec409ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_done global_id 5800 payload 274 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 -- 172.21.15.144:0/4013050792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 msgr2=0x7f3ec409ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:24.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 --2- 172.21.15.144:0/4013050792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec409ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 -- 172.21.15.144:0/4013050792 >> v2:172.21.15.116:3300/0 conn(0x7f3ec4096f00 msgr2=0x7f3ec412e8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 --2- 172.21.15.144:0/4013050792 >> v2:172.21.15.116:3300/0 conn(0x7f3ec4096f00 0x7f3ec412e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 monclient: found mon.noname-c 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec9efa640 1 -- 172.21.15.144:0/4013050792 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4098d30 con 0x7f3ec409d580 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec96f9640 10 monclient: get_auth_request con 0x7f3ec409c9f0 auth_method 0 2024-09-18T12:31:24.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.944+0000 7f3ec96f9640 1 --2- 172.21.15.144:0/4013050792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec409ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:24.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec9efa640 1 -- 172.21.15.144:0/4013050792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 msgr2=0x7f3ec412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:24.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec9efa640 1 --2- 172.21.15.144:0/4013050792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec412ede0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4069750 tx=0x7f3eb4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4030a78873cc4dc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:24.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 1 -- 172.21.15.144:0/4013050792 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4007610 con 0x7f3ec409d580 2024-09-18T12:31:24.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: dump: 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:24.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: _finish_auth 0 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 1 -- 172.21.15.144:0/4013050792 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb402d050 con 0x7f3ec409d580 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ecb95d640 5 monclient: authenticate success, global_id 5800 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ecb95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 1 -- 172.21.15.144:0/4013050792 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb40079a0 con 0x7f3ec409d580 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ecb95d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ecb95d640 4 set_mon_vals no callback set 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.947+0000 7f3ec3fff640 10 monclient: dump: 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:24.958 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 10 monclient: shutdown 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 -- 172.21.15.144:0/4013050792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 msgr2=0x7f3ec412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/4013050792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec412ede0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4069750 tx=0x7f3eb4009840 comp rx=0 tx=0).stop 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 -- 172.21.15.144:0/4013050792 shutdown_connections 2024-09-18T12:31:24.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/4013050792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec412ede0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/4013050792 >> v2:172.21.15.116:3300/0 conn(0x7f3ec4096f00 0x7f3ec412e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/4013050792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec409ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 -- 172.21.15.144:0/4013050792 >> 172.21.15.144:0/4013050792 conn(0x7f3ec4071510 msgr2=0x7f3ec40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 -- 172.21.15.144:0/4013050792 shutdown_connections 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 -- 172.21.15.144:0/4013050792 wait complete. 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 10 monclient: build_initial_monmap 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 10 monclient: monmap: 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:24.960 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.948+0000 7f3ecb95d640 1 Processor -- start 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 1 -- start start 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient: init 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec4096f00 0x7f3ec406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec4096f00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec4130800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:24.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ec409c9f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec4117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ec409d580 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec413fd10 con 0x7f3ec4096f00 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec409f8d0 con 0x7f3ec409d580 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec41198e0 con 0x7f3ec409c9f0 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ecb95d640 10 monclient(hunting): authenticate will time out at 1120.518675s 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec4117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec4117550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58154/0 (socket says 172.21.15.144:58154) 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 1 -- 172.21.15.144:0/1303370978 learned_addr learned my addr 172.21.15.144:0/1303370978 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:24.962 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): get_auth_request con 0x7f3ec409d580 auth_method 0 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec96f9640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec4096f00 msgr2=0x7f3ec406cef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec96f9640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec4096f00 0x7f3ec406cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec8ef8640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec4130800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec8ef8640 10 monclient(hunting): get_auth_request con 0x7f3ec409c9f0 auth_method 0 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec8ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec8ef8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:24.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec8ef8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.949+0000 7f3ec9efa640 10 monclient(hunting): handle_auth_done global_id 5803 payload 824 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 1 -- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 msgr2=0x7f3ec4130800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec4130800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 1 -- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec4096f00 msgr2=0x7f3ec406cef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec4096f00 0x7f3ec406cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 1 monclient: found mon.noname-b 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.950+0000 7f3ec9efa640 1 -- 172.21.15.144:0/1303370978 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb4009ae0 con 0x7f3ec409d580 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec9efa640 1 -- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 msgr2=0x7f3ec4117550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec9efa640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec4117550 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009ab0 tx=0x7f3eb4036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:24.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb40094e0 con 0x7f3ec409d580 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: dump: 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:24.965 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: _finish_auth 0 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb4032cc0 con 0x7f3ec409d580 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ecb95d640 5 monclient: authenticate success, global_id 5803 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ecb95d640 10 monclient: _renew_subs 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ecb95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec4119bc0 con 0x7f3ec409d580 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb403b660 con 0x7f3ec409d580 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec17fa640 10 monclient: dump: 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:24.966 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ecb95d640 10 monclient: _renew_subs 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ecb95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ec37fe640 4 set_mon_vals no callback set 2024-09-18T12:31:24.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.952+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec4138460 con 0x7f3ec409d580 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3eb4003020 con 0x7f3ec409d580 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ecb95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ecb95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ecb95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec404bf60 con 0x7f3ec409d580 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ec17fa640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3eb0038440 0x7f3eb003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(141..141 src has 13..141) v4 ==== 7213+0+0 (secure 0 0 0) 0x7f3eb406cab0 con 0x7f3ec409d580 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ec96f9640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3eb0038440 0x7f3eb003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.953+0000 7f3ec96f9640 10 monclient: get_auth_request con 0x7f3eb0038440 auth_method 0 2024-09-18T12:31:24.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.955+0000 7f3ec96f9640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3eb0038440 0x7f3eb003a8f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc006ca0 tx=0x7f3ebc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:24.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.957+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3eb402ae60 con 0x7f3ec409d580 2024-09-18T12:31:24.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.957+0000 7f3ec17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:24.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:24.957+0000 7f3ec17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:25.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.120+0000 7f3ecb95d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:25.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.120+0000 7f3ecb95d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:25.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.120+0000 7f3ecb95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:25.121 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.120+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f3ec4064b40 con 0x7f3ec409d580 2024-09-18T12:31:25.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.120+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v141) v1 ==== 71+0+0 (secure 0 0 0) 0x7f3eb403f020 con 0x7f3ec409d580 2024-09-18T12:31:25.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.120+0000 7f3ec17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:25.122 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.120+0000 7f3ec17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:25.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.219+0000 7f3ecb95d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:25.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.219+0000 7f3ecb95d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:25.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.219+0000 7f3ecb95d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:25.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.219+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f3ec4119a20 con 0x7f3ec409d580 2024-09-18T12:31:25.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.219+0000 7f3ec17fa640 1 -- 172.21.15.144:0/1303370978 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v141) v1 ==== 71+0+0 (secure 0 0 0) 0x7f3ec4119a20 con 0x7f3ec409d580 2024-09-18T12:31:25.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.219+0000 7f3ec17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:25.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.219+0000 7f3ec17fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:25.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3eb0038440 msgr2=0x7f3eb003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3eb0038440 0x7f3eb003a8f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc006ca0 tx=0x7f3ebc008040 comp rx=0 tx=0).stop 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 10 monclient: shutdown 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 msgr2=0x7f3ec4117550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec4117550 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009ab0 tx=0x7f3eb4036000 comp rx=0 tx=0).stop 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 shutdown_connections 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3eb0038440 0x7f3eb003a8f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ec409d580 0x7f3ec4117550 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ec409c9f0 0x7f3ec4130800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 --2- 172.21.15.144:0/1303370978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec4096f00 0x7f3ec406cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 >> 172.21.15.144:0/1303370978 conn(0x7f3ec4071510 msgr2=0x7f3ec4099ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 shutdown_connections 2024-09-18T12:31:25.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.221+0000 7f3ecb95d640 1 -- 172.21.15.144:0/1303370978 wait complete. 2024-09-18T12:31:25.231 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.304+0000 7ff86835d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.304+0000 7ff86835d640 10 monclient: build_initial_monmap 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.304+0000 7ff86835d640 10 monclient: monmap: 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:25.305030+0000 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:25.305030+0000 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:25.305 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:25.306 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:25.306 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:25.306 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:25.306 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:25.306 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:25.306 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:25.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.304+0000 7ff86835d640 1 Processor -- start 2024-09-18T12:31:25.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.304+0000 7ff86835d640 1 -- start start 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.304+0000 7ff86835d640 10 monclient: init 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff86009ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 0x7ff860069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient(hunting): picked mon.noname-b con 0x7ff86009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff860136d60 0x7ff860139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient(hunting): picked mon.noname-a con 0x7ff860136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff860067420 con 0x7ff86009ec50 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86009f3f0 con 0x7ff860136d60 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:25.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600a0480 con 0x7ff86009f020 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff86835d640 10 monclient(hunting): authenticate will time out at 1120.874437s 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8668fa640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7ff860136d60 msgr2=0x7ff860139140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8660f9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7ff86009ec50 legacy=0x7ff860131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8668fa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff860136d60 0x7ff860139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 0x7ff860069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 0x7ff860069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:58446/0 (socket says 172.21.15.144:58446) 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 1 -- 172.21.15.144:0/1459865103 learned_addr learned my addr 172.21.15.144:0/1459865103 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 10 monclient(hunting): get_auth_request con 0x7ff86009f020 auth_method 0 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:25.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.305+0000 7ff8658f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 10 monclient(hunting): handle_auth_done global_id 64631 payload 274 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 1 -- 172.21.15.144:0/1459865103 >> v2:172.21.15.116:3300/0 conn(0x7ff860136d60 msgr2=0x7ff860139140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 1 --2- 172.21.15.144:0/1459865103 >> v2:172.21.15.116:3300/0 conn(0x7ff860136d60 0x7ff860139140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 1 -- 172.21.15.144:0/1459865103 >> v1:172.21.15.116:6789/0 conn(0x7ff86009ec50 legacy=0x7ff860131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 10 monclient: _finish_hunting 0 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 1 monclient: found mon.noname-b 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:25.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.306+0000 7ff8658f8640 1 -- 172.21.15.144:0/1459865103 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8600a0200 con 0x7ff86009f020 2024-09-18T12:31:25.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.308+0000 7ff8658f8640 1 -- 172.21.15.144:0/1459865103 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 msgr2=0x7ff860069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.308+0000 7ff8658f8640 1 --2- 172.21.15.144:0/1459865103 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 0x7ff860069d10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff850009910 tx=0x7ff8500091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f094a2839a3914a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 1 -- 172.21.15.144:0/1459865103 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850032440 con 0x7ff86009f020 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: dump: 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:25.317 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: _finish_auth 0 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 1 -- 172.21.15.144:0/1459865103 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8500325a0 con 0x7ff86009f020 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 1 -- 172.21.15.144:0/1459865103 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850032940 con 0x7ff86009f020 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff86835d640 5 monclient: authenticate success, global_id 64631 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff86835d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff86835d640 4 set_mon_vals no callback set 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:25.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.309+0000 7ff8650f7640 10 monclient: dump: 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 10 monclient: shutdown 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 -- 172.21.15.144:0/1459865103 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 msgr2=0x7ff860069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:25.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 --2- 172.21.15.144:0/1459865103 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 0x7ff860069d10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff850009910 tx=0x7ff8500091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 -- 172.21.15.144:0/1459865103 shutdown_connections 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 --2- 172.21.15.144:0/1459865103 >> v2:172.21.15.116:3300/0 conn(0x7ff860136d60 0x7ff860139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 --2- 172.21.15.144:0/1459865103 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff86009f020 0x7ff860069d10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 -- 172.21.15.144:0/1459865103 >> 172.21.15.144:0/1459865103 conn(0x7ff8600a5cf0 msgr2=0x7ff8600a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 -- 172.21.15.144:0/1459865103 shutdown_connections 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 1 -- 172.21.15.144:0/1459865103 wait complete. 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 10 monclient: build_initial_monmap 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.310+0000 7ff86835d640 10 monclient: monmap: 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:25.320 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 Processor -- start 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 -- start start 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient: init 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff86009ec50 0x7ff860097690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): picked mon.noname-a con 0x7ff86009ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 0x7ff860097bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:25.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): picked mon.noname-b con 0x7ff86009f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff860136d60 0x7ff860098110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): picked mon.noname-c con 0x7ff860136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff860099570 con 0x7ff86009ec50 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86009c020 con 0x7ff86009f020 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600a40b0 con 0x7ff860136d60 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff86835d640 10 monclient(hunting): authenticate will time out at 1120.880544s 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 0x7ff860097bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8660f9640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff86009ec50 msgr2=0x7ff860097690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:25.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8660f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff86009ec50 0x7ff860097690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 0x7ff860097bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58162/0 (socket says 172.21.15.144:58162) 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 1 -- 172.21.15.144:0/481816129 learned_addr learned my addr 172.21.15.144:0/481816129 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): get_auth_request con 0x7ff86009f020 auth_method 0 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8668fa640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff860136d60 0x7ff860098110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8668fa640 10 monclient(hunting): get_auth_request con 0x7ff860136d60 auth_method 0 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8668fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8668fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:25.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8668fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 10 monclient(hunting): handle_auth_done global_id 5806 payload 824 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 1 -- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff860136d60 msgr2=0x7ff860098110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff860136d60 0x7ff860098110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 1 -- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff86009ec50 msgr2=0x7ff860097690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.311+0000 7ff8658f8640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff86009ec50 0x7ff860097690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.312+0000 7ff8658f8640 10 monclient: _finish_hunting 0 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.312+0000 7ff8658f8640 1 monclient: found mon.noname-b 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.312+0000 7ff8658f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.312+0000 7ff8658f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.312+0000 7ff8658f8640 1 -- 172.21.15.144:0/481816129 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850009450 con 0x7ff86009f020 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.312+0000 7ff8668fa640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff860136d60 0x7ff860098110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8658f8640 1 -- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 msgr2=0x7ff860097bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:25.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8658f8640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 0x7ff860097bd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff850009910 tx=0x7ff850032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85003c960 con 0x7ff86009f020 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 10 monclient: dump: 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:25.325 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8658f8640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 0x7ff860097bd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff850009910 tx=0x7ff850032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7ff85003cac0 config(0 keys) v1 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff86835d640 5 monclient: authenticate success, global_id 5806 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff86835d640 10 monclient: _renew_subs 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff86835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff860098ff0 con 0x7ff86009f020 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85003cac0 con 0x7ff86009f020 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff8648f6640 4 set_mon_vals no callback set 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff86835d640 10 monclient: _renew_subs 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff86835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.314+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86009b850 con 0x7ff86009f020 2024-09-18T12:31:25.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85003ce60 con 0x7ff86009f020 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8567fc640 10 monclient: dump: 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff850039040 con 0x7ff86009f020 2024-09-18T12:31:25.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff86835d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:25.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff86835d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:25.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff86835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:25.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8600008d0 con 0x7ff86009f020 2024-09-18T12:31:25.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8567fc640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff844038440 0x7ff84403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:25.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8660f9640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff844038440 0x7ff84403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:25.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.315+0000 7ff8660f9640 10 monclient: get_auth_request con 0x7ff844038440 auth_method 0 2024-09-18T12:31:25.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.318+0000 7ff8660f9640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff844038440 0x7ff84403a8f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0038f0 tx=0x7ff85c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:25.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.320+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(141..141 src has 13..141) v4 ==== 7213+0+0 (secure 0 0 0) 0x7ff85006c4f0 con 0x7ff86009f020 2024-09-18T12:31:25.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.937+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff85004b300 con 0x7ff86009f020 2024-09-18T12:31:25.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.937+0000 7ff8567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:25.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:25.937+0000 7ff8567fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:26.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.098+0000 7ff86835d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:26.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.098+0000 7ff86835d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:26.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.098+0000 7ff86835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:26.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.098+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7ff8600a13e0 con 0x7ff86009f020 2024-09-18T12:31:26.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.583+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v143) v1 ==== 82+0+0 (secure 0 0 0) 0x7ff850030340 con 0x7ff86009f020 2024-09-18T12:31:26.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.583+0000 7ff8567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:26.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.583+0000 7ff8567fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.683+0000 7ff86835d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.683+0000 7ff86835d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.683+0000 7ff86835d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.683+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7ff8600a13e0 con 0x7ff86009f020 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.684+0000 7ff8567fc640 1 -- 172.21.15.144:0/481816129 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v143) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff850040030 con 0x7ff86009f020 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.684+0000 7ff8567fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.684+0000 7ff8567fc640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T12:31:26.685 INFO:tasks.workunit.client.0.smithi144.stderr:compat weight-set already created 2024-09-18T12:31:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff844038440 msgr2=0x7ff84403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff844038440 0x7ff84403a8f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0038f0 tx=0x7ff85c006d80 comp rx=0 tx=0).stop 2024-09-18T12:31:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 10 monclient: shutdown 2024-09-18T12:31:26.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 msgr2=0x7ff860097bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 0x7ff860097bd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff850009910 tx=0x7ff850032000 comp rx=0 tx=0).stop 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 shutdown_connections 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff844038440 0x7ff84403a8f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff860136d60 0x7ff860098110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff86009f020 0x7ff860097bd0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 --2- 172.21.15.144:0/481816129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff86009ec50 0x7ff860097690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 >> 172.21.15.144:0/481816129 conn(0x7ff8600a5cf0 msgr2=0x7ff86009dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 shutdown_connections 2024-09-18T12:31:26.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.686+0000 7ff86835d640 1 -- 172.21.15.144:0/481816129 wait complete. 2024-09-18T12:31:26.696 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set ls 2024-09-18T12:31:26.696 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '(compat)' 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.768+0000 7f086e39d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.768+0000 7f086e39d640 10 monclient: build_initial_monmap 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.768+0000 7f086e39d640 10 monclient: monmap: 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:26.769792+0000 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:26.769792+0000 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:26.770 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:26.771 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:26.771 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:26.771 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:26.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.769+0000 7f086e39d640 1 Processor -- start 2024-09-18T12:31:26.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.769+0000 7f086e39d640 1 -- start start 2024-09-18T12:31:26.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.769+0000 7f086e39d640 10 monclient: init 2024-09-18T12:31:26.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:26.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:31:26.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08680972c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f086809a7a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0868134700 0x7f0868136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient(hunting): picked mon.noname-c con 0x7f0868134700 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08680987e0 con 0x7f08680972c0 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0868097690 con 0x7f086809a7a0 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086809c700 con 0x7f0868134700 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086e39d640 10 monclient(hunting): authenticate will time out at 1122.339217s 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0868134700 0x7f0868136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f0867fff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f08680972c0 legacy=0x7f086809b8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f08677fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f086809a7a0 0x7f086809bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:51340/0 (socket says 172.21.15.144:51340) 2024-09-18T12:31:26.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f08677fe640 1 -- 172.21.15.144:0/3801679595 learned_addr learned my addr 172.21.15.144:0/3801679595 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): get_auth_request con 0x7f0868134700 auth_method 0 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient(hunting): handle_auth_done global_id 5809 payload 274 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 1 -- 172.21.15.144:0/3801679595 >> v1:172.21.15.144:6789/0 conn(0x7f086809a7a0 legacy=0x7f086809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 1 -- 172.21.15.144:0/3801679595 >> v1:172.21.15.116:6789/0 conn(0x7f08680972c0 legacy=0x7f086809b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient: _finish_hunting 0 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 1 monclient: found mon.noname-c 2024-09-18T12:31:26.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:26.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:26.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.770+0000 7f086c93a640 1 -- 172.21.15.144:0/3801679595 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0868097e30 con 0x7f0868134700 2024-09-18T12:31:26.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f086c93a640 1 -- 172.21.15.144:0/3801679595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0868134700 msgr2=0x7f0868136ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:26.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f086c93a640 1 --2- 172.21.15.144:0/3801679595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0868134700 0x7f0868136ae0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f08680a1fa0 tx=0x7f086000b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20c4f7b3c3b1194c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:26.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 1 -- 172.21.15.144:0/3801679595 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086000b3c0 con 0x7f0868134700 2024-09-18T12:31:26.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:26.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:26.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: dump: 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:26.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 1 -- 172.21.15.144:0/3801679595 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086002f050 con 0x7f0868134700 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f086e39d640 5 monclient: authenticate success, global_id 5809 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f086e39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f086e39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f086e39d640 4 set_mon_vals no callback set 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 1 -- 172.21.15.144:0/3801679595 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860009990 con 0x7f0868134700 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.773+0000 7f0866ffd640 10 monclient: dump: 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.783 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 10 monclient: shutdown 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 -- 172.21.15.144:0/3801679595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0868134700 msgr2=0x7f0868136ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 --2- 172.21.15.144:0/3801679595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0868134700 0x7f0868136ae0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f08680a1fa0 tx=0x7f086000b990 comp rx=0 tx=0).stop 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 -- 172.21.15.144:0/3801679595 shutdown_connections 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 --2- 172.21.15.144:0/3801679595 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0868134700 0x7f0868136ae0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 -- 172.21.15.144:0/3801679595 >> 172.21.15.144:0/3801679595 conn(0x7f0868071510 msgr2=0x7f08680a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:26.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 -- 172.21.15.144:0/3801679595 shutdown_connections 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 -- 172.21.15.144:0/3801679595 wait complete. 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 10 monclient: build_initial_monmap 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 10 monclient: monmap: 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:26.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.774+0000 7f086e39d640 1 Processor -- start 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 1 -- start start 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient: init 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 0x7f086813cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): picked mon.noname-b con 0x7f08680972c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f086809a7a0 0x7f086813d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): picked mon.noname-a con 0x7f086809a7a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0868134700 0x7f086813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): picked mon.noname-c con 0x7f0868134700 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086809f3c0 con 0x7f086809a7a0 2024-09-18T12:31:26.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086811e090 con 0x7f08680972c0 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086811ddb0 con 0x7f0868134700 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f086e39d640 10 monclient(hunting): authenticate will time out at 1122.344515s 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 0x7f086813cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 0x7f086813cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:58174/0 (socket says 172.21.15.144:58174) 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 -- 172.21.15.144:0/370598292 learned_addr learned my addr 172.21.15.144:0/370598292 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): get_auth_request con 0x7f08680972c0 auth_method 0 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:26.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f08677fe640 1 -- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f086809a7a0 msgr2=0x7f086813d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f08677fe640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f086809a7a0 0x7f086813d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient(hunting): handle_auth_done global_id 5812 payload 824 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 -- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0868134700 msgr2=0x7f086813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0868134700 0x7f086813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 -- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f086809a7a0 msgr2=0x7f086813d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f086809a7a0 0x7f086813d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 monclient: found mon.noname-b 2024-09-18T12:31:26.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.775+0000 7f0867fff640 1 -- 172.21.15.144:0/370598292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086000bc30 con 0x7f08680972c0 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f0867fff640 1 -- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 msgr2=0x7f086813cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f0867fff640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 0x7f086813cd30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f085800b600 tx=0x7f08580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0858015070 con 0x7f08680972c0 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f0867fff640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 0x7f086813cd30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f085800b600 tx=0x7f08580091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f0858006380 mon_map magic: 0 v1 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 10 monclient: dump: 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.789 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 10 monclient: _finish_auth 0 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0858003d90 con 0x7f08680972c0 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f086e39d640 5 monclient: authenticate success, global_id 5812 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f086e39d640 10 monclient: _renew_subs 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f086e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086811dad0 con 0x7f08680972c0 2024-09-18T12:31:26.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f084ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f086e39d640 10 monclient: _renew_subs 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f086e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f08667fc640 4 set_mon_vals no callback set 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.778+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086811a510 con 0x7f08680972c0 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0858006380 con 0x7f08680972c0 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f084ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f084ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f084ffff640 10 monclient: dump: 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:26.791 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0858010440 con 0x7f08680972c0 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f086e39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f086e39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f086e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086804bf60 con 0x7f08680972c0 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f084ffff640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0854038440 0x7f085403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f08677fe640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0854038440 0x7f085403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:26.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.779+0000 7f08677fe640 10 monclient: get_auth_request con 0x7f0854038440 auth_method 0 2024-09-18T12:31:26.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.781+0000 7f08677fe640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0854038440 0x7f085403a8f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f086811bd40 tx=0x7f085c007af0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:26.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:26.783+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(143..143 src has 13..143) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f085804bec0 con 0x7f08680972c0 2024-09-18T12:31:27.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.401+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f085804f030 con 0x7f08680972c0 2024-09-18T12:31:27.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.401+0000 7f084ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:27.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.401+0000 7f084ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.563+0000 7f086e39d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.563+0000 7f086e39d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:27.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.563+0000 7f086e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.563+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f0868064b40 con 0x7f08680972c0 2024-09-18T12:31:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.563+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v143) v1 ==== 71+0+9 (secure 0 0 0) 0x7f0858018780 con 0x7f08680972c0 2024-09-18T12:31:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.563+0000 7f084ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:27.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.563+0000 7f084ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:27.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.663+0000 7f086e39d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:27.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.663+0000 7f086e39d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:27.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.663+0000 7f086e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.663+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f086811b910 con 0x7f08680972c0 2024-09-18T12:31:27.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.663+0000 7f084ffff640 1 -- 172.21.15.144:0/370598292 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v143) v1 ==== 71+0+9 (secure 0 0 0) 0x7f0858018780 con 0x7f08680972c0 2024-09-18T12:31:27.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.663+0000 7f084ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T12:31:27.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.663+0000 7f084ffff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0854038440 msgr2=0x7f085403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0854038440 0x7f085403a8f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f086811bd40 tx=0x7f085c007af0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 10 monclient: shutdown 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 msgr2=0x7f086813cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 0x7f086813cd30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f085800b600 tx=0x7f08580091a0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 shutdown_connections 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0854038440 0x7f085403a8f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0868134700 0x7f086813d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f086809a7a0 0x7f086813d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 --2- 172.21.15.144:0/370598292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08680972c0 0x7f086813cd30 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 >> 172.21.15.144:0/370598292 conn(0x7f0868071510 msgr2=0x7f086809b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:27.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 shutdown_connections 2024-09-18T12:31:27.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.665+0000 7f086e39d640 1 -- 172.21.15.144:0/370598292 wait complete. 2024-09-18T12:31:27.675 INFO:tasks.workunit.client.0.smithi144.stdout:(compat) 2024-09-18T12:31:27.676 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.748+0000 7f22a0b1d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.748+0000 7f22a0b1d640 10 monclient: build_initial_monmap 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.748+0000 7f22a0b1d640 10 monclient: monmap: 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:27.749045+0000 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:27.749045+0000 2024-09-18T12:31:27.749 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:27.750 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:27.750 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:27.750 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:27.750 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:27.750 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:27.750 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:27.750 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:27.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.748+0000 7f22a0b1d640 1 Processor -- start 2024-09-18T12:31:27.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.748+0000 7f22a0b1d640 1 -- start start 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.748+0000 7f22a0b1d640 10 monclient: init 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f229c0a1800 0x7f229c09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f229c0a1800 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f229c0983d0 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f229c12ec70 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0a1bd0 con 0x7f229c0983d0 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c097850 con 0x7f229c12ec70 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c09cbf0 con 0x7f229c0a1800 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:27.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f22a0b1d640 10 monclient(hunting): authenticate will time out at 1123.318443s 2024-09-18T12:31:27.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f229ad9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f229c12ec70 0x7f229c097140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:50462/0 (socket says 172.21.15.144:50462) 2024-09-18T12:31:27.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f229ad9d640 1 -- 172.21.15.144:0/2830087844 learned_addr learned my addr 172.21.15.144:0/2830087844 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:27.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f229a59c640 1 -- 172.21.15.144:0/2830087844 >> v2:172.21.15.116:3300/0 conn(0x7f229c0a1800 msgr2=0x7f229c09aa90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:27.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f2299d9b640 1 -- 172.21.15.144:0/2830087844 >> v1:172.21.15.116:6789/0 conn(0x7f229c0983d0 legacy=0x7f229c096a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:27.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.749+0000 7f229a59c640 1 --2- 172.21.15.144:0/2830087844 >> v2:172.21.15.116:3300/0 conn(0x7f229c0a1800 0x7f229c09aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.751+0000 7f229ad9d640 1 -- 172.21.15.144:0/2830087844 >> v1:172.21.15.144:6789/0 conn(0x7f229c12ec70 legacy=0x7f229c097140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.751+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235031646 0 0) 0x7f229c097850 con 0x7f229c12ec70 2024-09-18T12:31:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.751+0000 7f229959a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.751+0000 7f229959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.751+0000 7f229959a640 10 monclient(hunting): my global_id is 5815 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.751+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22880034e0 con 0x7f229c12ec70 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692143599 0 0) 0x7f22880034e0 con 0x7f229c12ec70 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 >> v2:172.21.15.116:3300/0 conn(0x7f229c0a1800 msgr2=0x7f229c09aa90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 --2- 172.21.15.144:0/2830087844 >> v2:172.21.15.116:3300/0 conn(0x7f229c0a1800 0x7f229c09aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 >> v1:172.21.15.116:6789/0 conn(0x7f229c0983d0 legacy=0x7f229c096a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: _finish_hunting 0 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c12f600 con 0x7f229c12ec70 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: _finish_auth 0 2024-09-18T12:31:27.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f228c0017a0 con 0x7f229c12ec70 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f22a0b1d640 5 monclient: authenticate success, global_id 5815 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f22a0b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: dump: 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:27.762 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 >> v1:172.21.15.144:6789/0 conn(0x7f229c12ec70 legacy=0x7f229c097140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2288003aa0 0x7f2288003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): picked mon.a con 0x7f2288003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2288004780 0x7f2288004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): picked mon.c con 0x7f2288004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f228800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): picked mon.b con 0x7f22880099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:27.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c09cbf0 con 0x7f2288003aa0 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228800c3c0 con 0x7f22880099c0 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228800c580 con 0x7f2288004780 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229959a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f22a0b1d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f228800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229a59c640 1 -- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2288003aa0 msgr2=0x7f2288003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): get_auth_request con 0x7f22880099c0 auth_method 0 2024-09-18T12:31:27.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229a59c640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2288003aa0 0x7f2288003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient(hunting): handle_auth_done global_id 5815 payload 293 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 1 -- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2288004780 msgr2=0x7f2288004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f2299d9b640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2288004780 0x7f2288004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:27.765 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2288004780 0x7f2288004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 1 -- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2288003aa0 msgr2=0x7f2288003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2288003aa0 0x7f2288003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 1 monclient: found mon.b 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 1 -- 172.21.15.144:0/2830087844 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880034e0 con 0x7f22880099c0 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 10 monclient: _finish_auth 0 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f229ad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f2299d9b640 10 monclient: get_auth_request con 0x7f2288004780 auth_method 0 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.752+0000 7f2299d9b640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2288004780 0x7f2288004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.753+0000 7f229ad9d640 1 -- 172.21.15.144:0/2830087844 reap_dead start 2024-09-18T12:31:27.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229ad9d640 1 -- 172.21.15.144:0/2830087844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 msgr2=0x7f228800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229ad9d640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f228800bda0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f228c001980 tx=0x7f228c007a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f29d3b089a737f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c005890 con 0x7f22880099c0 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 10 monclient: dump: 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c005a10 con 0x7f22880099c0 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f22a0b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f22a0b1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 1 -- 172.21.15.144:0/2830087844 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c005dd0 con 0x7f22880099c0 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f22a0b1d640 4 set_mon_vals no callback set 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:27.768 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f229959a640 10 monclient: dump: 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f22a0b1d640 10 monclient: shutdown 2024-09-18T12:31:27.769 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f22a0b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f22a0b1d640 1 -- 172.21.15.144:0/2830087844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 msgr2=0x7f228800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.755+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f228800bda0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f228c001980 tx=0x7f228c007a90 comp rx=0 tx=0).stop 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 1 -- 172.21.15.144:0/2830087844 shutdown_connections 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/2830087844 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f228800bda0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 1 -- 172.21.15.144:0/2830087844 >> 172.21.15.144:0/2830087844 conn(0x7f229c071100 msgr2=0x7f229c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 1 -- 172.21.15.144:0/2830087844 shutdown_connections 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 1 -- 172.21.15.144:0/2830087844 wait complete. 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 10 monclient: build_initial_monmap 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 10 monclient: monmap: 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:27.770 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 1 Processor -- start 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 1 -- start start 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.756+0000 7f22a0b1d640 10 monclient: init 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f229c12d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f22880099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f229c12dc90 0x7f229c117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f229c12dc90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f229c117a90 0x7f229c117f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f229c117a90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c11b1a0 con 0x7f229c12dc90 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c09f970 con 0x7f22880099c0 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c098830 con 0x7f229c117a90 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:27.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f22a0b1d640 10 monclient(hunting): authenticate will time out at 1123.326408s 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f2299d9b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f229c12dc90 msgr2=0x7f229c117550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f2299d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f229c12dc90 0x7f229c117550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f229c12d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f229c12d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40594/0 (socket says 172.21.15.144:40594) 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 -- 172.21.15.144:0/1549513876 learned_addr learned my addr 172.21.15.144:0/1549513876 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): get_auth_request con 0x7f22880099c0 auth_method 0 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f229c117a90 0x7f229c117f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:27.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 10 monclient(hunting): get_auth_request con 0x7f229c117a90 auth_method 0 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient(hunting): handle_auth_done global_id 5818 payload 824 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 -- 172.21.15.144:0/1549513876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f229c117a90 msgr2=0x7f229c117f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f229c117a90 0x7f229c117f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 -- 172.21.15.144:0/1549513876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f229c12dc90 msgr2=0x7f229c117550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f229c12dc90 0x7f229c117550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:31:27.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 monclient: found mon.noname-b 2024-09-18T12:31:27.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:27.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.757+0000 7f229a59c640 1 -- 172.21.15.144:0/1549513876 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c133250 con 0x7f22880099c0 2024-09-18T12:31:27.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f229a59c640 1 -- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 msgr2=0x7f229c12d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:27.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f229a59c640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f229c12d750 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f229400bb30 tx=0x7f22940097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:27.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229400cbf0 con 0x7f22880099c0 2024-09-18T12:31:27.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:27.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:27.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: dump: 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:27.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22a0b1d640 5 monclient: authenticate success, global_id 5818 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2294006e90 con 0x7f22880099c0 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22a0b1d640 10 monclient: _renew_subs 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22a0b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c118d40 con 0x7f22880099c0 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2294015470 con 0x7f22880099c0 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22927fc640 10 monclient: dump: 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:27.779 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f2298d99640 4 set_mon_vals no callback set 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22a0b1d640 10 monclient: _renew_subs 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22a0b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.760+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c11ab50 con 0x7f22880099c0 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2294012070 con 0x7f22880099c0 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f22a0b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f22a0b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:27.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f22a0b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c0008d0 con 0x7f22880099c0 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f22927fc640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f227c038470 0x7f227c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(143..143 src has 13..143) v4 ==== 7545+0+0 (secure 0 0 0) 0x7f229404c020 con 0x7f22880099c0 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f2299d9b640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f227c038470 0x7f227c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.761+0000 7f2299d9b640 10 monclient: get_auth_request con 0x7f227c038470 auth_method 0 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.763+0000 7f2299d9b640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f227c038470 0x7f227c03a920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f229c09e9c0 tx=0x7f2284008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.765+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f229407e030 con 0x7f22880099c0 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.765+0000 7f22927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:27.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.765+0000 7f22927fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:27.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.929+0000 7f22a0b1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:31:27.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.929+0000 7f22a0b1d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:31:27.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.929+0000 7f22a0b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:27.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:27.929+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f229c119930 con 0x7f22880099c0 2024-09-18T12:31:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:28.866+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v144) v1 ==== 78+0+0 (secure 0 0 0) 0x7f22940058d0 con 0x7f22880099c0 2024-09-18T12:31:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:28.866+0000 7f22927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:31:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:28.866+0000 7f22927fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:28.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:28.968+0000 7f22a0b1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:31:28.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:28.968+0000 7f22a0b1d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:31:28.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:28.968+0000 7f22a0b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:28.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:28.968+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f229c132f50 con 0x7f22880099c0 2024-09-18T12:31:29.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.868+0000 7f22927fc640 1 -- 172.21.15.144:0/1549513876 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v145) v1 ==== 78+0+0 (secure 0 0 0) 0x7f2294022030 con 0x7f22880099c0 2024-09-18T12:31:29.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.868+0000 7f22927fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:31:29.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.868+0000 7f22927fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f227c038470 msgr2=0x7f227c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f227c038470 0x7f227c03a920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f229c09e9c0 tx=0x7f2284008040 comp rx=0 tx=0).stop 2024-09-18T12:31:29.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 10 monclient: shutdown 2024-09-18T12:31:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 msgr2=0x7f229c12d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f229c12d750 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f229400bb30 tx=0x7f22940097f0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 shutdown_connections 2024-09-18T12:31:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f227c038470 0x7f227c03a920 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f229c117a90 0x7f229c117f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f229c12dc90 0x7f229c117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 --2- 172.21.15.144:0/1549513876 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22880099c0 0x7f229c12d750 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 >> 172.21.15.144:0/1549513876 conn(0x7f229c071100 msgr2=0x7f229c09d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 shutdown_connections 2024-09-18T12:31:29.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.871+0000 7f22a0b1d640 1 -- 172.21.15.144:0/1549513876 wait complete. 2024-09-18T12:31:29.884 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create cool 2 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.962+0000 7fd3c66e8640 10 monclient: get_monmap_and_config 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.962+0000 7fd3c66e8640 10 monclient: build_initial_monmap 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.962+0000 7fd3c66e8640 10 monclient: monmap: 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:29.962992+0000 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:29.962992+0000 2024-09-18T12:31:29.963 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:29.964 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:29.964 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:29.964 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:29.964 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:29.964 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:29.964 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:29.964 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:29.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.962+0000 7fd3c66e8640 1 Processor -- start 2024-09-18T12:31:29.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.962+0000 7fd3c66e8640 1 -- start start 2024-09-18T12:31:29.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.962+0000 7fd3c66e8640 10 monclient: init 2024-09-18T12:31:29.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3c009ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd3c009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3c0136d60 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c0067420 con 0x7fd3c009ec50 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c009fdc0 con 0x7fd3c0136d60 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c00a0510 con 0x7fd3c009f020 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c66e8640 10 monclient(hunting): authenticate will time out at 1125.532379s 2024-09-18T12:31:29.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bffff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fd3c009ec50 legacy=0x7fd3c0131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c4c85640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd3c0136d60 0x7fd3c0139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:50468/0 (socket says 172.21.15.144:50468) 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3c4c85640 1 -- 172.21.15.144:0/1965672322 learned_addr learned my addr 172.21.15.144:0/1965672322 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 1 --2- 172.21.15.144:0/1965672322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd3c009f020 auth_method 0 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.963+0000 7fd3bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:29.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 10 monclient(hunting): handle_auth_done global_id 64640 payload 274 2024-09-18T12:31:29.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 1 -- 172.21.15.144:0/1965672322 >> v1:172.21.15.144:6789/0 conn(0x7fd3c0136d60 legacy=0x7fd3c0139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:29.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 1 -- 172.21.15.144:0/1965672322 >> v1:172.21.15.116:6789/0 conn(0x7fd3c009ec50 legacy=0x7fd3c0131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:29.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:29.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:29.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:29.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:29.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.964+0000 7fd3bf7fe640 1 -- 172.21.15.144:0/1965672322 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c00a0200 con 0x7fd3c009f020 2024-09-18T12:31:29.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.966+0000 7fd3bf7fe640 1 -- 172.21.15.144:0/1965672322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 msgr2=0x7fd3c0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:29.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3bf7fe640 1 --2- 172.21.15.144:0/1965672322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0069d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009880 tx=0x7fd3b40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8bc255fad85b86f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:29.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 1 -- 172.21.15.144:0/1965672322 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b402ae90 con 0x7fd3c009f020 2024-09-18T12:31:29.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:29.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:29.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: dump: 2024-09-18T12:31:29.976 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 1 -- 172.21.15.144:0/1965672322 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b4032440 con 0x7fd3c009f020 2024-09-18T12:31:29.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3c66e8640 5 monclient: authenticate success, global_id 64640 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3c66e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3c66e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 1 -- 172.21.15.144:0/1965672322 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b40327e0 con 0x7fd3c009f020 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3c66e8640 4 set_mon_vals no callback set 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.967+0000 7fd3beffd640 10 monclient: dump: 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.978 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 10 monclient: shutdown 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 -- 172.21.15.144:0/1965672322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 msgr2=0x7fd3c0069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/1965672322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0069d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009880 tx=0x7fd3b40090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 -- 172.21.15.144:0/1965672322 shutdown_connections 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/1965672322 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0069d10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 -- 172.21.15.144:0/1965672322 >> 172.21.15.144:0/1965672322 conn(0x7fd3c00a5d40 msgr2=0x7fd3c00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 -- 172.21.15.144:0/1965672322 shutdown_connections 2024-09-18T12:31:29.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 -- 172.21.15.144:0/1965672322 wait complete. 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 10 monclient: build_initial_monmap 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 10 monclient: monmap: 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:29.980 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.968+0000 7fd3c66e8640 1 Processor -- start 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 1 -- start start 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient: init 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c009ec50 0x7fd3c009b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): picked mon.noname-a con 0x7fd3c009ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd3c009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 0x7fd3c0096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd3c0136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:29.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c009c000 con 0x7fd3c009ec50 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c013a5c0 con 0x7fd3c0136d60 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c009a350 con 0x7fd3c009f020 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c66e8640 10 monclient(hunting): authenticate will time out at 1125.538532s 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bffff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c009ec50 msgr2=0x7fd3c009b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c009ec50 0x7fd3c009b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 0x7fd3c0096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:29.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 0x7fd3c0096f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40596/0 (socket says 172.21.15.144:40596) 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 -- 172.21.15.144:0/2557143258 learned_addr learned my addr 172.21.15.144:0/2557143258 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): get_auth_request con 0x7fd3c0136d60 auth_method 0 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0096a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd3c009f020 auth_method 0 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:29.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient(hunting): handle_auth_done global_id 5821 payload 824 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 -- 172.21.15.144:0/2557143258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 msgr2=0x7fd3c0096a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0096a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 -- 172.21.15.144:0/2557143258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c009ec50 msgr2=0x7fd3c009b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c009ec50 0x7fd3c009b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient: _finish_hunting 0 2024-09-18T12:31:29.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 monclient: found mon.noname-b 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.969+0000 7fd3c4c85640 1 -- 172.21.15.144:0/2557143258 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b4009390 con 0x7fd3c0136d60 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c4c85640 1 -- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 msgr2=0x7fd3c0096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c4c85640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 0x7fd3c0096f70 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800cbf0 tx=0x7fd3b8005340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b800ce10 con 0x7fd3c0136d60 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: dump: 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:29.985 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: _finish_auth 0 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c66e8640 5 monclient: authenticate success, global_id 5821 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b800aa60 con 0x7fd3c0136d60 2024-09-18T12:31:29.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c66e8640 10 monclient: _renew_subs 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c66e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c009ab30 con 0x7fd3c0136d60 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b800ae20 con 0x7fd3c0136d60 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3a7fff640 10 monclient: dump: 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:29.987 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c66e8640 10 monclient: _renew_subs 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c66e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c0097b60 con 0x7fd3c0136d60 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.972+0000 7fd3be7fc640 4 set_mon_vals no callback set 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd3b8007300 con 0x7fd3c0136d60 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3c66e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:29.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3c66e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:29.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3c66e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:29.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c00008d0 con 0x7fd3c0136d60 2024-09-18T12:31:29.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3a7fff640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd3ac038440 0x7fd3ac03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:29.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(145..145 src has 13..145) v4 ==== 7213+0+0 (secure 0 0 0) 0x7fd3b804c2a0 con 0x7fd3c0136d60 2024-09-18T12:31:29.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3bffff640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd3ac038440 0x7fd3ac03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:29.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.973+0000 7fd3bffff640 10 monclient: get_auth_request con 0x7fd3ac038440 auth_method 0 2024-09-18T12:31:29.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.976+0000 7fd3bffff640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd3ac038440 0x7fd3ac03a8f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00038f0 tx=0x7fd3b0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:29.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.977+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd3b80075b0 con 0x7fd3c0136d60 2024-09-18T12:31:29.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.977+0000 7fd3a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:29.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:29.977+0000 7fd3a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:30.135 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.134+0000 7fd3c66e8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T12:31:30.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.134+0000 7fd3c66e8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T12:31:30.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.134+0000 7fd3c66e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:30.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.134+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7fd3c0064900 con 0x7fd3c0136d60 2024-09-18T12:31:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.877+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v146) v1 ==== 111+0+0 (secure 0 0 0) 0x7fd3b8010af0 con 0x7fd3c0136d60 2024-09-18T12:31:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.877+0000 7fd3a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T12:31:30.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.877+0000 7fd3a7fff640 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-18T12:31:30.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.974+0000 7fd3c66e8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T12:31:30.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.974+0000 7fd3c66e8640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T12:31:30.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.974+0000 7fd3c66e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:30.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.974+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7fd3c0099aa0 con 0x7fd3c0136d60 2024-09-18T12:31:30.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.974+0000 7fd3a7fff640 1 -- 172.21.15.144:0/2557143258 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v146) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd3b801a360 con 0x7fd3c0136d60 2024-09-18T12:31:30.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.974+0000 7fd3a7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T12:31:30.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.974+0000 7fd3a7fff640 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-18T12:31:30.976 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'cool' already exists 2024-09-18T12:31:30.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd3ac038440 msgr2=0x7fd3ac03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:30.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd3ac038440 0x7fd3ac03a8f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00038f0 tx=0x7fd3b0006d80 comp rx=0 tx=0).stop 2024-09-18T12:31:30.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 10 monclient: shutdown 2024-09-18T12:31:30.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:30.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 msgr2=0x7fd3c0096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:30.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 0x7fd3c0096f70 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800cbf0 tx=0x7fd3b8005340 comp rx=0 tx=0).stop 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 shutdown_connections 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd3ac038440 0x7fd3ac03a8f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3c0136d60 0x7fd3c0096f70 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd3c009f020 0x7fd3c0096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 --2- 172.21.15.144:0/2557143258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c009ec50 0x7fd3c009b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 >> 172.21.15.144:0/2557143258 conn(0x7fd3c00a5d40 msgr2=0x7fd3c009e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 shutdown_connections 2024-09-18T12:31:30.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:30.976+0000 7fd3c66e8640 1 -- 172.21.15.144:0/2557143258 wait complete. 2024-09-18T12:31:30.986 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool create cold 2 2024-09-18T12:31:31.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.059+0000 7f748ce4f640 10 monclient: get_monmap_and_config 2024-09-18T12:31:31.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.059+0000 7f748ce4f640 10 monclient: build_initial_monmap 2024-09-18T12:31:31.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.059+0000 7f748ce4f640 10 monclient: monmap: 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:31.060420+0000 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:31.060420+0000 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:31.061 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:31.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 1 Processor -- start 2024-09-18T12:31:31.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 1 -- start start 2024-09-18T12:31:31.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient: init 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f748809f320 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f748812e740 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f748812f250 0x7f748813a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient(hunting): picked mon.noname-c con 0x7f748812f250 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7488132880 con 0x7f748809f320 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748809e1f0 con 0x7f748812e740 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:31.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748809e4d0 con 0x7f748812f250 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748ce4f640 10 monclient(hunting): authenticate will time out at 1126.629803s 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f748659c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f748809f320 legacy=0x7f7488132170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f7486d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f748812f250 0x7f748813a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f7486d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f748812f250 0x7f748813a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40606/0 (socket says 172.21.15.144:40606) 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f7486d9d640 1 -- 172.21.15.144:0/395647718 learned_addr learned my addr 172.21.15.144:0/395647718 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f7486d9d640 10 monclient(hunting): get_auth_request con 0x7f748812f250 auth_method 0 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f7486d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f7486d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.060+0000 7f7486d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:31.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 10 monclient(hunting): handle_auth_done global_id 5824 payload 274 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 1 -- 172.21.15.144:0/395647718 >> v1:172.21.15.144:6789/0 conn(0x7f748812e740 legacy=0x7f748812eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 1 -- 172.21.15.144:0/395647718 >> v1:172.21.15.116:6789/0 conn(0x7f748809f320 legacy=0x7f7488132170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 1 monclient: found mon.noname-c 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.061+0000 7f7486d9d640 1 -- 172.21.15.144:0/395647718 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748809e7b0 con 0x7f748812f250 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.063+0000 7f7486d9d640 1 -- 172.21.15.144:0/395647718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 msgr2=0x7f748813a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f7486d9d640 1 --2- 172.21.15.144:0/395647718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748813a050 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7480009f50 tx=0x7f7480009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a8ad8e57921e8c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 1 -- 172.21.15.144:0/395647718 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480007610 con 0x7f748812f250 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:31.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: dump: 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: _finish_auth 0 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 1 -- 172.21.15.144:0/395647718 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748002d050 con 0x7f748812f250 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:31.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 1 -- 172.21.15.144:0/395647718 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800079a0 con 0x7f748812f250 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 5 monclient: authenticate success, global_id 5824 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 4 set_mon_vals no callback set 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748559a640 10 monclient: dump: 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:31.071 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 10 monclient: shutdown 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 1 -- 172.21.15.144:0/395647718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 msgr2=0x7f748813a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 1 --2- 172.21.15.144:0/395647718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748813a050 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7480009f50 tx=0x7f7480009840 comp rx=0 tx=0).stop 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 1 -- 172.21.15.144:0/395647718 shutdown_connections 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 1 --2- 172.21.15.144:0/395647718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748813a050 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.064+0000 7f748ce4f640 1 -- 172.21.15.144:0/395647718 >> 172.21.15.144:0/395647718 conn(0x7f74880a2b70 msgr2=0x7f748809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.065+0000 7f748ce4f640 1 -- 172.21.15.144:0/395647718 shutdown_connections 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.065+0000 7f748ce4f640 1 -- 172.21.15.144:0/395647718 wait complete. 2024-09-18T12:31:31.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.065+0000 7f748ce4f640 10 monclient: build_initial_monmap 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.065+0000 7f748ce4f640 10 monclient: monmap: 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.065+0000 7f748ce4f640 1 Processor -- start 2024-09-18T12:31:31.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.065+0000 7f748ce4f640 1 -- start start 2024-09-18T12:31:31.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.065+0000 7f748ce4f640 10 monclient: init 2024-09-18T12:31:31.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:31.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:31.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f748809f320 0x7f748812e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:31.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): picked mon.noname-c con 0x7f748809f320 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:31.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f748812e740 0x7f748809a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:31.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): picked mon.noname-a con 0x7f748812e740 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:31.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): picked mon.noname-b con 0x7f748812f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748812b420 con 0x7f748812e740 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748809d5b0 con 0x7f748812f250 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748813a220 con 0x7f748809f320 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748ce4f640 10 monclient(hunting): authenticate will time out at 1126.635305s 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748809aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40618/0 (socket says 172.21.15.144:40618) 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 -- 172.21.15.144:0/3558580212 learned_addr learned my addr 172.21.15.144:0/3558580212 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:31.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): get_auth_request con 0x7f748812f250 auth_method 0 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7485d9b640 1 -- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f748812e740 msgr2=0x7f748809a550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7485d9b640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f748812e740 0x7f748809a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient(hunting): handle_auth_done global_id 5827 payload 824 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 -- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f748809f320 msgr2=0x7f748812e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f748659c640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f748809f320 0x7f748812e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:31.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f748809f320 0x7f748812e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 -- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f748812e740 msgr2=0x7f748809a550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f748812e740 0x7f748809a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 monclient: found mon.noname-b 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.066+0000 7f7486d9d640 1 -- 172.21.15.144:0/3558580212 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7480009ae0 con 0x7f748812f250 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.068+0000 7f7486d9d640 1 -- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 msgr2=0x7f748809aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.068+0000 7f7486d9d640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748809aa90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7480009f50 tx=0x7f7480036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480031d90 con 0x7f748812f250 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:31.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f7486d9d640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748809aa90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7480009f50 tx=0x7f7480036000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f748003c420 config(0 keys) v1 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: dump: 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:31.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 5 monclient: authenticate success, global_id 5827 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748003c420 con 0x7f748812f250 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 10 monclient: _renew_subs 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748812b140 con 0x7f748812f250 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f7484d99640 4 set_mon_vals no callback set 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 10 monclient: _renew_subs 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748809bfd0 con 0x7f748812f250 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748003c7c0 con 0x7f748812f250 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:31.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 10 monclient: dump: 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:31.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f748003cb30 con 0x7f748812f250 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.069+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748812be00 con 0x7f748812f250 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.070+0000 7f74767fc640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.070+0000 7f748659c640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.070+0000 7f748659c640 10 monclient: get_auth_request con 0x7f7468038440 auth_method 0 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.072+0000 7f748659c640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7478006fd0 tx=0x7f7478008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:31.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.074+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(146..146 src has 13..146) v4 ==== 7556+0+0 (secure 0 0 0) 0x7f748006cf30 con 0x7f748812f250 2024-09-18T12:31:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.692+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f748002fae0 con 0x7f748812f250 2024-09-18T12:31:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.692+0000 7f74767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.692+0000 7f74767fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:31.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.854+0000 7f748ce4f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T12:31:31.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.854+0000 7f748ce4f640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T12:31:31.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.854+0000 7f748ce4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:31.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:31.854+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f74880008d0 con 0x7f748812f250 2024-09-18T12:31:32.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.592+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v148) v1 ==== 111+0+0 (secure 0 0 0) 0x7f7480040020 con 0x7f748812f250 2024-09-18T12:31:32.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.592+0000 7f74767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T12:31:32.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.592+0000 7f74767fc640 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-18T12:31:32.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.689+0000 7f748ce4f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T12:31:32.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.689+0000 7f748ce4f640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T12:31:32.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.689+0000 7f748ce4f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:32.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.689+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f74880008d0 con 0x7f748812f250 2024-09-18T12:31:32.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.690+0000 7f74767fc640 1 -- 172.21.15.144:0/3558580212 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v148) v1 ==== 118+0+0 (secure 0 0 0) 0x7f748002f300 con 0x7f748812f250 2024-09-18T12:31:32.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.690+0000 7f74767fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T12:31:32.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.690+0000 7f74767fc640 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-18T12:31:32.692 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'cold' already exists 2024-09-18T12:31:32.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 msgr2=0x7f746803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:32.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7478006fd0 tx=0x7f7478008040 comp rx=0 tx=0).stop 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 10 monclient: shutdown 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 msgr2=0x7f748809aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748809aa90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7480009f50 tx=0x7f7480036000 comp rx=0 tx=0).stop 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 shutdown_connections 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7468038440 0x7f746803a8f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f748812f250 0x7f748809aa90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f748812e740 0x7f748809a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 --2- 172.21.15.144:0/3558580212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f748809f320 0x7f748812e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 >> 172.21.15.144:0/3558580212 conn(0x7f74880a2b70 msgr2=0x7f74880a0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 shutdown_connections 2024-09-18T12:31:32.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.692+0000 7f748ce4f640 1 -- 172.21.15.144:0/3558580212 wait complete. 2024-09-18T12:31:32.704 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool set cold size 2 2024-09-18T12:31:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.778+0000 7f4ca8b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.778+0000 7f4ca8b9d640 10 monclient: build_initial_monmap 2024-09-18T12:31:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.778+0000 7f4ca8b9d640 10 monclient: monmap: 2024-09-18T12:31:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:32.779223+0000 2024-09-18T12:31:32.779 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:32.779223+0000 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:32.780 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.778+0000 7f4ca8b9d640 1 Processor -- start 2024-09-18T12:31:32.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.778+0000 7f4ca8b9d640 1 -- start start 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.778+0000 7f4ca8b9d640 10 monclient: init 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ca409d140 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ca406ebc0 0x7f4ca4136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca406ebc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409dc20 0x7f4ca4139470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca409dc20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca409eb90 con 0x7f4ca409d140 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca409ef50 con 0x7f4ca406ebc0 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:32.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca4067420 con 0x7f4ca409dc20 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca8b9d640 10 monclient(hunting): authenticate will time out at 1128.348577s 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca259c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f4ca409d140 0x7f4ca409d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:50472/0 (socket says 172.21.15.144:50472) 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca259c640 1 -- 172.21.15.144:0/2114901859 learned_addr learned my addr 172.21.15.144:0/2114901859 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca1d9b640 1 -- 172.21.15.144:0/2114901859 >> v2:172.21.15.116:3300/0 conn(0x7f4ca406ebc0 msgr2=0x7f4ca4136f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca1d9b640 1 --2- 172.21.15.144:0/2114901859 >> v2:172.21.15.116:3300/0 conn(0x7f4ca406ebc0 0x7f4ca4136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca2d9d640 1 --2- 172.21.15.144:0/2114901859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409dc20 0x7f4ca4139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca2d9d640 10 monclient(hunting): get_auth_request con 0x7f4ca409dc20 auth_method 0 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.779+0000 7f4ca2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:32.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_done global_id 64646 payload 274 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2114901859 >> v2:172.21.15.116:3300/0 conn(0x7f4ca406ebc0 msgr2=0x7f4ca4136f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 1 --2- 172.21.15.144:0/2114901859 >> v2:172.21.15.116:3300/0 conn(0x7f4ca406ebc0 0x7f4ca4136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2114901859 >> v1:172.21.15.144:6789/0 conn(0x7f4ca409d140 legacy=0x7f4ca409d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 1 monclient: found mon.noname-b 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:32.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.780+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2114901859 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca406f1e0 con 0x7f4ca409dc20 2024-09-18T12:31:32.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2114901859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409dc20 msgr2=0x7f4ca4139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:32.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca2d9d640 1 --2- 172.21.15.144:0/2114901859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409dc20 0x7f4ca4139470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e60 tx=0x7f4c9c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a243da88207376f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:32.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 1 -- 172.21.15.144:0/2114901859 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c007840 con 0x7f4ca409dc20 2024-09-18T12:31:32.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:32.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: dump: 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: _finish_auth 0 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:32.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 1 -- 172.21.15.144:0/2114901859 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c02d050 con 0x7f4ca409dc20 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 1 -- 172.21.15.144:0/2114901859 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c007bd0 con 0x7f4ca409dc20 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca8b9d640 5 monclient: authenticate success, global_id 64646 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca8b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca8b9d640 4 set_mon_vals no callback set 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.783+0000 7f4ca159a640 10 monclient: dump: 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:32.793 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 10 monclient: shutdown 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2114901859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409dc20 msgr2=0x7f4ca4139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2114901859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409dc20 0x7f4ca4139470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e60 tx=0x7f4c9c009780 comp rx=0 tx=0).stop 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2114901859 shutdown_connections 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2114901859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409dc20 0x7f4ca4139470 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2114901859 >> v2:172.21.15.116:3300/0 conn(0x7f4ca406ebc0 0x7f4ca4136f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2114901859 >> 172.21.15.144:0/2114901859 conn(0x7f4ca40a2190 msgr2=0x7f4ca409e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:32.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2114901859 shutdown_connections 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2114901859 wait complete. 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 10 monclient: build_initial_monmap 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.784+0000 7f4ca8b9d640 10 monclient: monmap: 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:32.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 Processor -- start 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 -- start start 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient: init 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ca406ebc0 0x7f4ca4094a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca406ebc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409d140 0x7f4ca4129250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca409d140 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 0x7f4ca4129790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca409dc20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca412cb50 con 0x7f4ca406ebc0 2024-09-18T12:31:32.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40950b0 con 0x7f4ca409dc20 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca4095e70 con 0x7f4ca409d140 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca8b9d640 10 monclient(hunting): authenticate will time out at 1128.354618s 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca259c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ca406ebc0 msgr2=0x7f4ca4094a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 0x7f4ca4129790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ca406ebc0 0x7f4ca4094a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 0x7f4ca4129790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40624/0 (socket says 172.21.15.144:40624) 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2151983108 learned_addr learned my addr 172.21.15.144:0/2151983108 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): get_auth_request con 0x7f4ca409dc20 auth_method 0 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:32.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409d140 0x7f4ca4129250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 10 monclient(hunting): get_auth_request con 0x7f4ca409d140 auth_method 0 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:32.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.785+0000 7f4ca2d9d640 10 monclient(hunting): handle_auth_done global_id 5830 payload 824 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2151983108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409d140 msgr2=0x7f4ca4129250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409d140 0x7f4ca4129250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2151983108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ca406ebc0 msgr2=0x7f4ca4094a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ca406ebc0 0x7f4ca4094a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 1 monclient: found mon.noname-b 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.786+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2151983108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c009a20 con 0x7f4ca409dc20 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca2d9d640 1 -- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 msgr2=0x7f4ca4129790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca2d9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 0x7f4ca4129790 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e60 tx=0x7f4c9c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c031cd0 con 0x7f4ca409dc20 2024-09-18T12:31:32.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: dump: 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:32.800 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c040070 con 0x7f4ca409dc20 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca8b9d640 5 monclient: authenticate success, global_id 5830 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca8b9d640 10 monclient: _renew_subs 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca412c870 con 0x7f4ca409dc20 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c045640 con 0x7f4ca409dc20 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4c927fc640 10 monclient: dump: 2024-09-18T12:31:32.801 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca0d99640 4 set_mon_vals no callback set 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca8b9d640 10 monclient: _renew_subs 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.788+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca4094010 con 0x7f4ca409dc20 2024-09-18T12:31:32.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4c9c003020 con 0x7f4ca409dc20 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4ca8b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4ca8b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4ca8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca40671b0 con 0x7f4ca409dc20 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4c927fc640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c84038440 0x7f4c8403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(148..148 src has 13..148) v4 ==== 7899+0+0 (secure 0 0 0) 0x7f4c9c06e120 con 0x7f4ca409dc20 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4ca259c640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c84038440 0x7f4c8403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:32.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.789+0000 7f4ca259c640 10 monclient: get_auth_request con 0x7f4c84038440 auth_method 0 2024-09-18T12:31:32.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.792+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:31:32.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.792+0000 7f4ca259c640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c84038440 0x7f4c8403a8f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4c940047d0 tx=0x7f4c94007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:32.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.794+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4c9c03a9a0 con 0x7f4ca409dc20 2024-09-18T12:31:32.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.794+0000 7f4c927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:32.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.794+0000 7f4c927fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:32.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.949+0000 7f4ca8b9d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T12:31:32.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.950+0000 7f4ca8b9d640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T12:31:32.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.950+0000 7f4ca8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:32.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:32.950+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7f4ca40008d0 con 0x7f4ca409dc20 2024-09-18T12:31:33.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:33.597+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v149) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4c9c06d070 con 0x7f4ca409dc20 2024-09-18T12:31:33.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:33.597+0000 7f4c927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T12:31:33.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:33.597+0000 7f4c927fc640 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-18T12:31:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:33.692+0000 7f4ca8b9d640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T12:31:33.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:33.692+0000 7f4ca8b9d640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T12:31:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:33.692+0000 7f4ca8b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:33.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:33.692+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7f4ca40008d0 con 0x7f4ca409dc20 2024-09-18T12:31:34.167 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:31:34.168 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:31:34.168 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:31:34.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:31:34.171 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:31:34.171 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:31:34.171 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:34.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.273+0000 7fb27675d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:34.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.273+0000 7fb27675d640 10 monclient: build_initial_monmap 2024-09-18T12:31:34.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.273+0000 7fb27675d640 10 monclient: monmap: 2024-09-18T12:31:34.276 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:34.276 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:34.275317+0000 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:34.275317+0000 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.274+0000 7fb27675d640 1 Processor -- start 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.274+0000 7fb27675d640 1 -- start start 2024-09-18T12:31:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.274+0000 7fb27675d640 10 monclient: init 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb270131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 0x7fb27009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient(hunting): picked mon.noname-a con 0x7fb27009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb27009f250 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27009faf0 con 0x7fb27009f250 2024-09-18T12:31:34.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27009fd30 con 0x7fb270131ab0 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb270132590 con 0x7fb27009e910 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb27675d640 10 monclient(hunting): authenticate will time out at 1121.597145s 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 0x7fb27009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb274cfa640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb27009f250 0x7fb270138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54772/0 (socket says 172.21.15.116:54772) 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 0x7fb27009ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54742/0 (socket says 172.21.15.116:54742) 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb274cfa640 1 -- 172.21.15.116:0/3200140968 learned_addr learned my addr 172.21.15.116:0/3200140968 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 10 monclient(hunting): get_auth_request con 0x7fb27009e910 auth_method 0 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3200140968 >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 msgr2=0x7fb27009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3200140968 >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 msgr2=0x7fb27009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3200140968 >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 0x7fb27009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.275+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3200140968 >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 0x7fb27009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.277+0000 7fb274cfa640 1 -- 172.21.15.116:0/3200140968 >> v1:172.21.15.116:6789/0 conn(0x7fb27009f250 legacy=0x7fb270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.277+0000 7fb26ffff640 1 -- 172.21.15.116:0/3200140968 >> v1:172.21.15.144:6789/0 conn(0x7fb270131ab0 legacy=0x7fb270131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.277+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517696627 0 0) 0x7fb27009fd30 con 0x7fb270131ab0 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.277+0000 7fb26effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.277+0000 7fb26effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.277+0000 7fb26effd640 10 monclient(hunting): my global_id is 5833 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.277+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2580034e0 con 0x7fb270131ab0 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3440028546 0 0) 0x7fb2580034e0 con 0x7fb270131ab0 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 msgr2=0x7fb27009ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:34.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 --2- 172.21.15.116:0/3200140968 >> v2:172.21.15.116:3300/0 conn(0x7fb27009e910 0x7fb27009ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 >> v1:172.21.15.116:6789/0 conn(0x7fb27009f250 legacy=0x7fb270138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2700a1110 con 0x7fb270131ab0 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: _finish_auth 0 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb27675d640 5 monclient: authenticate success, global_id 5833 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fb2600018b0 con 0x7fb270131ab0 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb27675d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: dump: 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 >> v1:172.21.15.144:6789/0 conn(0x7fb270131ab0 legacy=0x7fb270131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb258003af0 0x7fb258003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): picked mon.c con 0x7fb258003af0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb258004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): picked mon.a con 0x7fb258004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb258005380 0x7fb25800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): picked mon.b con 0x7fb258005380 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb270132590 con 0x7fb258004780 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25800c350 con 0x7fb258005380 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25800c510 con 0x7fb258003af0 2024-09-18T12:31:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26effd640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb27675d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb258004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): get_auth_request con 0x7fb258004780 auth_method 0 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb258003af0 0x7fb258003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb258005380 0x7fb25800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 10 monclient(hunting): get_auth_request con 0x7fb258003af0 auth_method 0 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 10 monclient(hunting): get_auth_request con 0x7fb258005380 auth_method 0 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_done global_id 5833 payload 293 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb258003af0 msgr2=0x7fb258003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb258003af0 0x7fb258003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3200140968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb258005380 msgr2=0x7fb25800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb258005380 0x7fb25800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 1 monclient: found mon.a 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb274cfa640 1 -- 172.21.15.116:0/3200140968 reap_dead start 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3200140968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2580034e0 con 0x7fb258004780 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.278+0000 7fb26ffff640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb258003af0 0x7fb258003f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 msgr2=0x7fb258004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb258004bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb264006fd0 tx=0x7fb264008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ea97e7158dd734 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26402add0 con 0x7fb258004780 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 10 monclient: dump: 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26402e070 con 0x7fb258004780 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb27675d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb27675d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 1 -- 172.21.15.116:0/3200140968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264034660 con 0x7fb258004780 2024-09-18T12:31:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb27675d640 4 set_mon_vals no callback set 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.281+0000 7fb26effd640 10 monclient: dump: 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 10 monclient: shutdown 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 1 -- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 msgr2=0x7fb258004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb258004bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb264006fd0 tx=0x7fb264008040 comp rx=0 tx=0).stop 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 1 -- 172.21.15.116:0/3200140968 shutdown_connections 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 1 --2- 172.21.15.116:0/3200140968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb258004bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 1 -- 172.21.15.116:0/3200140968 >> 172.21.15.116:0/3200140968 conn(0x7fb270067d30 msgr2=0x7fb2700a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 1 -- 172.21.15.116:0/3200140968 shutdown_connections 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 1 -- 172.21.15.116:0/3200140968 wait complete. 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 10 monclient: build_initial_monmap 2024-09-18T12:31:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.282+0000 7fb27675d640 10 monclient: monmap: 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 Processor -- start 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 -- start start 2024-09-18T12:31:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient: init 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb27009aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): picked mon.noname-a con 0x7fb258004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 0x7fb270130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): picked mon.noname-c con 0x7fb27011bbc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb270131090 0x7fb270140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): picked mon.noname-b con 0x7fb270131090 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27011e510 con 0x7fb258004780 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27011cd80 con 0x7fb270131090 2024-09-18T12:31:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27011dc70 con 0x7fb27011bbc0 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb27675d640 10 monclient(hunting): authenticate will time out at 1121.605762s 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 0x7fb270130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb27009aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 0x7fb270130b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53540/0 (socket says 172.21.15.116:53540) 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3996651083 learned_addr learned my addr 172.21.15.116:0/3996651083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 10 monclient(hunting): get_auth_request con 0x7fb258004780 auth_method 0 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 10 monclient(hunting): get_auth_request con 0x7fb27011bbc0 auth_method 0 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 1 -- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 msgr2=0x7fb27009aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 1 -- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 msgr2=0x7fb27009aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb27009aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb274cfa640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb270131090 0x7fb270140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb274cfa640 10 monclient(hunting): get_auth_request con 0x7fb270131090 auth_method 0 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb274cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb274cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb274cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.283+0000 7fb26ffff640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb27009aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb274cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb274cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb274cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 10 monclient(hunting): handle_auth_done global_id 64652 payload 824 2024-09-18T12:31:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb270131090 msgr2=0x7fb270140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb270131090 0x7fb270140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 msgr2=0x7fb27009aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb27009aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.284+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3996651083 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27011df50 con 0x7fb27011bbc0 2024-09-18T12:31:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb26f7fe640 1 -- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 msgr2=0x7fb270130b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb26f7fe640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 0x7fb270130b50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb26402a7c0 tx=0x7fb264035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 1 -- 172.21.15.116:0/3996651083 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264043070 con 0x7fb27011bbc0 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: dump: 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: _finish_auth 0 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb27675d640 5 monclient: authenticate success, global_id 64652 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 1 -- 172.21.15.116:0/3996651083 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2640335c0 con 0x7fb27011bbc0 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb27675d640 10 monclient: _renew_subs 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb27675d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27011e230 con 0x7fb27011bbc0 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 1 -- 172.21.15.116:0/3996651083 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264033960 con 0x7fb27011bbc0 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb257fff640 10 monclient: dump: 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:34.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb27675d640 10 monclient: _renew_subs 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb27675d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2701414e0 con 0x7fb27011bbc0 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.286+0000 7fb26e7fc640 4 set_mon_vals no callback set 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb257fff640 1 -- 172.21.15.116:0/3996651083 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb264033d50 con 0x7fb27011bbc0 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb27675d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb27675d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb27675d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27004bf60 con 0x7fb27011bbc0 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb257fff640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb25c038490 0x7fb25c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb257fff640 1 -- 172.21.15.116:0/3996651083 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(149..149 src has 13..149) v4 ==== 7899+0+0 (secure 0 0 0) 0x7fb26406c5f0 con 0x7fb27011bbc0 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb26ffff640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb25c038490 0x7fb25c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.287+0000 7fb26ffff640 10 monclient: get_auth_request con 0x7fb25c038490 auth_method 0 2024-09-18T12:31:34.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.290+0000 7fb26ffff640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb25c038490 0x7fb25c03a940 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2600019e0 tx=0x7fb260009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:34.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.292+0000 7fb257fff640 1 -- 172.21.15.116:0/3996651083 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb264032e60 con 0x7fb27011bbc0 2024-09-18T12:31:34.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.292+0000 7fb257fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:34.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.292+0000 7fb257fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.506+0000 7fb27675d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.507+0000 7fb27675d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.507+0000 7fb27675d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.507+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb270066280 con 0x7fb27011bbc0 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.507+0000 7fb257fff640 1 -- 172.21.15.116:0/3996651083 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fb264032ac0 con 0x7fb27011bbc0 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.507+0000 7fb257fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.507+0000 7fb257fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:34.509 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":68,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb25c038490 msgr2=0x7fb25c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb25c038490 0x7fb25c03a940 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2600019e0 tx=0x7fb260009040 comp rx=0 tx=0).stop 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 10 monclient: shutdown 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 msgr2=0x7fb270130b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 0x7fb270130b50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb26402a7c0 tx=0x7fb264035000 comp rx=0 tx=0).stop 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 shutdown_connections 2024-09-18T12:31:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb25c038490 0x7fb25c03a940 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb270131090 0x7fb270140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27011bbc0 0x7fb270130b50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 --2- 172.21.15.116:0/3996651083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb258004780 0x7fb27009aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 >> 172.21.15.116:0/3996651083 conn(0x7fb270067d30 msgr2=0x7fb2700a2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 shutdown_connections 2024-09-18T12:31:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:34.509+0000 7fb27675d640 1 -- 172.21.15.116:0/3996651083 wait complete. 2024-09-18T12:31:34.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.599+0000 7f4c927fc640 1 -- 172.21.15.144:0/2151983108 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v150) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4c9c0393c0 con 0x7f4ca409dc20 2024-09-18T12:31:34.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.599+0000 7f4c927fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T12:31:34.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.599+0000 7f4c927fc640 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-18T12:31:34.600 INFO:tasks.workunit.client.0.smithi144.stderr:set pool 12 size to 2 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c84038440 msgr2=0x7f4c8403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c84038440 0x7f4c8403a8f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4c940047d0 tx=0x7f4c94007b80 comp rx=0 tx=0).stop 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 10 monclient: shutdown 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 msgr2=0x7f4ca4129790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 0x7f4ca4129790 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009e60 tx=0x7f4c9c037000 comp rx=0 tx=0).stop 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 shutdown_connections 2024-09-18T12:31:34.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c84038440 0x7f4c8403a8f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ca409dc20 0x7f4ca4129790 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4ca409d140 0x7f4ca4129250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 --2- 172.21.15.144:0/2151983108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ca406ebc0 0x7f4ca4094a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.601+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 >> 172.21.15.144:0/2151983108 conn(0x7f4ca40a2190 msgr2=0x7f4ca409f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:34.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.602+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 shutdown_connections 2024-09-18T12:31:34.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.602+0000 7f4ca8b9d640 1 -- 172.21.15.144:0/2151983108 wait complete. 2024-09-18T12:31:34.613 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T12:31:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.689+0000 7f1158f54640 10 monclient: get_monmap_and_config 2024-09-18T12:31:34.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.689+0000 7f1158f54640 10 monclient: build_initial_monmap 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.689+0000 7f1158f54640 10 monclient: monmap: 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:34.690149+0000 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:34.690149+0000 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:34.691 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:34.692 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:34.692 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:34.692 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:34.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.689+0000 7f1158f54640 1 Processor -- start 2024-09-18T12:31:34.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.689+0000 7f1158f54640 1 -- start start 2024-09-18T12:31:34.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.689+0000 7f1158f54640 10 monclient: init 2024-09-18T12:31:34.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f115406e000 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f115406a710 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406e3d0 0x7f115409e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient(hunting): picked mon.noname-b con 0x7f115406e3d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11540699c0 con 0x7f115406e000 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115406be20 con 0x7f115406a710 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115406a360 con 0x7f115406e3d0 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1158f54640 10 monclient(hunting): authenticate will time out at 1130.259536s 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1151d9b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f115406a710 0x7f115409e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:50474/0 (socket says 172.21.15.144:50474) 2024-09-18T12:31:34.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1151d9b640 1 -- 172.21.15.144:0/1865392286 learned_addr learned my addr 172.21.15.144:0/1865392286 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 1 --2- 172.21.15.144:0/1865392286 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406e3d0 0x7f115409e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 10 monclient(hunting): get_auth_request con 0x7f115406e3d0 auth_method 0 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.690+0000 7f1152d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 10 monclient(hunting): handle_auth_done global_id 64655 payload 274 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 1 -- 172.21.15.144:0/1865392286 >> v1:172.21.15.144:6789/0 conn(0x7f115406a710 legacy=0x7f115409e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 1 -- 172.21.15.144:0/1865392286 >> v1:172.21.15.116:6789/0 conn(0x7f115406e000 legacy=0x7f115409da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:34.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 1 monclient: found mon.noname-b 2024-09-18T12:31:34.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:34.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:34.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.691+0000 7f1152d9d640 1 -- 172.21.15.144:0/1865392286 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1154069f80 con 0x7f115406e3d0 2024-09-18T12:31:34.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.693+0000 7f1152d9d640 1 -- 172.21.15.144:0/1865392286 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406e3d0 msgr2=0x7f115409e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:34.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.693+0000 7f1152d9d640 1 --2- 172.21.15.144:0/1865392286 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406e3d0 0x7f115409e880 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f114c009e90 tx=0x7f114c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98de27715e68e3a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:34.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 1 -- 172.21.15.144:0/1865392286 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c007840 con 0x7f115406e3d0 2024-09-18T12:31:34.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: dump: 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:34.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: _finish_auth 0 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 1 -- 172.21.15.144:0/1865392286 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c02d050 con 0x7f115406e3d0 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 5 monclient: authenticate success, global_id 64655 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 1 -- 172.21.15.144:0/1865392286 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c007bd0 con 0x7f115406e3d0 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f115159a640 10 monclient: dump: 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.703 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 4 set_mon_vals no callback set 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 10 monclient: shutdown 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 1 -- 172.21.15.144:0/1865392286 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406e3d0 msgr2=0x7f115409e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.694+0000 7f1158f54640 1 --2- 172.21.15.144:0/1865392286 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406e3d0 0x7f115409e880 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f114c009e90 tx=0x7f114c009780 comp rx=0 tx=0).stop 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 1 -- 172.21.15.144:0/1865392286 shutdown_connections 2024-09-18T12:31:34.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 1 --2- 172.21.15.144:0/1865392286 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406e3d0 0x7f115409e880 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 1 -- 172.21.15.144:0/1865392286 >> 172.21.15.144:0/1865392286 conn(0x7f1154071340 msgr2=0x7f11540a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 1 -- 172.21.15.144:0/1865392286 shutdown_connections 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 1 -- 172.21.15.144:0/1865392286 wait complete. 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 10 monclient: build_initial_monmap 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 10 monclient: monmap: 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:34.705 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 1 Processor -- start 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 1 -- start start 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.695+0000 7f1158f54640 10 monclient: init 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406a710 0x7f115413d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): picked mon.noname-c con 0x7f115406a710 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f115406e000 0x7f115413ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): picked mon.noname-a con 0x7f115406e000 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 0x7f1154140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): picked mon.noname-b con 0x7f115406e3d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:34.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1154122810 con 0x7f115406e000 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1154122250 con 0x7f115406e3d0 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115411fce0 con 0x7f115406a710 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1158f54640 10 monclient(hunting): authenticate will time out at 1130.265425s 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 0x7f1154140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 0x7f1154140d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40628/0 (socket says 172.21.15.144:40628) 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 -- 172.21.15.144:0/3099784734 learned_addr learned my addr 172.21.15.144:0/3099784734 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): get_auth_request con 0x7f115406e3d0 auth_method 0 2024-09-18T12:31:34.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:34.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:34.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:34.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f115259c640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406a710 0x7f115413d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1151d9b640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f115406e000 0x7f115413ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f115259c640 10 monclient(hunting): get_auth_request con 0x7f115406a710 auth_method 0 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f115259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f115259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f115259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1151d9b640 10 monclient(hunting): get_auth_request con 0x7f115406e000 auth_method 0 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1151d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1151d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1151d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient(hunting): handle_auth_done global_id 5839 payload 824 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 -- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406a710 msgr2=0x7f115413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406a710 0x7f115413d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 -- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f115406e000 msgr2=0x7f115413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f115406e000 0x7f115413ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 monclient: found mon.noname-b 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.696+0000 7f1152d9d640 1 -- 172.21.15.144:0/3099784734 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c009a20 con 0x7f115406e3d0 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1152d9d640 1 -- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 msgr2=0x7f1154140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1152d9d640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 0x7f1154140d90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f114c009de0 tx=0x7f114c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c031cd0 con 0x7f115406e3d0 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:34.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: dump: 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:34.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1158f54640 5 monclient: authenticate success, global_id 5839 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1158f54640 10 monclient: _renew_subs 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1158f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c040070 con 0x7f115406e3d0 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1154121f70 con 0x7f115406e3d0 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c045690 con 0x7f115406e3d0 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f114a7fc640 10 monclient: dump: 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:34.712 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1150d99640 4 set_mon_vals no callback set 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1158f54640 10 monclient: _renew_subs 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1158f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.699+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115413d5b0 con 0x7f115406e3d0 2024-09-18T12:31:34.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f114c042030 con 0x7f115406e3d0 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f1158f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f1158f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f1158f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1154122640 con 0x7f115406e3d0 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f114a7fc640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1134038490 0x7f113403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(150..150 src has 13..150) v4 ==== 7899+0+0 (secure 0 0 0) 0x7f114c06dce0 con 0x7f115406e3d0 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f115259c640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1134038490 0x7f113403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:34.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.700+0000 7f115259c640 10 monclient: get_auth_request con 0x7f1134038490 auth_method 0 2024-09-18T12:31:34.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.702+0000 7f115259c640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1134038490 0x7f113403a940 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f11440037d0 tx=0x7f1144009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:34.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.704+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f114c073020 con 0x7f115406e3d0 2024-09-18T12:31:34.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.704+0000 7f114a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:34.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.704+0000 7f114a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:34.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.866+0000 7f1158f54640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:34.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.866+0000 7f1158f54640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:34.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.866+0000 7f1158f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:34.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:34.866+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f11540008d0 con 0x7f115406e3d0 2024-09-18T12:31:35.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.771+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v151) v1 ==== 82+0+0 (secure 0 0 0) 0x7f114c0309a0 con 0x7f115406e3d0 2024-09-18T12:31:35.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.771+0000 7f114a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:35.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.771+0000 7f114a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.873+0000 7f1158f54640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.873+0000 7f1158f54640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.873+0000 7f1158f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.873+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f11540008d0 con 0x7f115406e3d0 2024-09-18T12:31:35.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.873+0000 7f114a7fc640 1 -- 172.21.15.144:0/3099784734 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v151) v1 ==== 115+0+0 (secure 0 0 0) 0x7f114c03a3a0 con 0x7f115406e3d0 2024-09-18T12:31:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.873+0000 7f114a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:31:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.873+0000 7f114a7fc640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T12:31:35.875 INFO:tasks.workunit.client.0.smithi144.stderr:compat weight-set already created 2024-09-18T12:31:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1134038490 msgr2=0x7f113403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1134038490 0x7f113403a940 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f11440037d0 tx=0x7f1144009040 comp rx=0 tx=0).stop 2024-09-18T12:31:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 10 monclient: shutdown 2024-09-18T12:31:35.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 msgr2=0x7f1154140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 0x7f1154140d90 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f114c009de0 tx=0x7f114c037000 comp rx=0 tx=0).stop 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 shutdown_connections 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1134038490 0x7f113403a940 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f115406e3d0 0x7f1154140d90 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f115406e000 0x7f115413ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 --2- 172.21.15.144:0/3099784734 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f115406a710 0x7f115413d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 >> 172.21.15.144:0/3099784734 conn(0x7f1154071340 msgr2=0x7f115409e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 shutdown_connections 2024-09-18T12:31:35.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.875+0000 7f1158f54640 1 -- 172.21.15.144:0/3099784734 wait complete. 2024-09-18T12:31:35.885 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create cool flat 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.960+0000 7f8c0415d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.960+0000 7f8c0415d640 10 monclient: build_initial_monmap 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.960+0000 7f8c0415d640 10 monclient: monmap: 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:35.960974+0000 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:35.960974+0000 2024-09-18T12:31:35.961 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:35.962 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:35.962 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:35.962 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:35.962 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:35.962 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:35.962 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:35.962 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.960+0000 7f8c0415d640 1 Processor -- start 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.960+0000 7f8c0415d640 1 -- start start 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.960+0000 7f8c0415d640 10 monclient: init 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bfc131bd0 0x7f8bfc131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): picked mon.noname-a con 0x7f8bfc131bd0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8bfc09ebe0 0x7f8bfc068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): picked mon.noname-c con 0x7f8bfc09ebe0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc09efe0 0x7f8bfc069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:35.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): picked mon.noname-b con 0x7f8bfc09efe0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc0a06d0 con 0x7f8bfc131bd0 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc071780 con 0x7f8bfc09ebe0 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0a0010 con 0x7f8bfc09efe0 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c0415d640 10 monclient(hunting): authenticate will time out at 1131.530488s 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8bfc09ebe0 0x7f8bfc068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8bfc09ebe0 0x7f8bfc068e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40636/0 (socket says 172.21.15.144:40636) 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 -- 172.21.15.144:0/2654393550 learned_addr learned my addr 172.21.15.144:0/2654393550 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): get_auth_request con 0x7f8bfc09ebe0 auth_method 0 2024-09-18T12:31:35.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 1 --2- 172.21.15.144:0/2654393550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc09efe0 0x7f8bfc069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 10 monclient(hunting): get_auth_request con 0x7f8bfc09efe0 auth_method 0 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c01ef9640 1 --2- 172.21.15.144:0/2654393550 >> v2:172.21.15.116:3300/0 conn(0x7f8bfc131bd0 0x7f8bfc131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c01ef9640 10 monclient(hunting): get_auth_request con 0x7f8bfc131bd0 auth_method 0 2024-09-18T12:31:35.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c01ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c01ef9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c01ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c026fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient(hunting): handle_auth_done global_id 5842 payload 274 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 -- 172.21.15.144:0/2654393550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc09efe0 msgr2=0x7f8bfc069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 --2- 172.21.15.144:0/2654393550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc09efe0 0x7f8bfc069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 -- 172.21.15.144:0/2654393550 >> v2:172.21.15.116:3300/0 conn(0x7f8bfc131bd0 msgr2=0x7f8bfc131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 --2- 172.21.15.144:0/2654393550 >> v2:172.21.15.116:3300/0 conn(0x7f8bfc131bd0 0x7f8bfc131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient: _finish_hunting 0 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 monclient: found mon.noname-c 2024-09-18T12:31:35.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:35.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:35.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.961+0000 7f8c016f8640 1 -- 172.21.15.144:0/2654393550 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0a1910 con 0x7f8bfc09ebe0 2024-09-18T12:31:35.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c016f8640 1 -- 172.21.15.144:0/2654393550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09ebe0 msgr2=0x7f8bfc068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c016f8640 1 --2- 172.21.15.144:0/2654393550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc068e00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc068d90 tx=0x7f8bf00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c27a13d0af889e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 1 -- 172.21.15.144:0/2654393550 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0032440 con 0x7f8bfc09ebe0 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: dump: 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:35.975 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: _finish_auth 0 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 1 -- 172.21.15.144:0/2654393550 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf00325a0 con 0x7f8bfc09ebe0 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c0415d640 5 monclient: authenticate success, global_id 5842 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c0415d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c0415d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 1 -- 172.21.15.144:0/2654393550 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0032940 con 0x7f8bfc09ebe0 2024-09-18T12:31:35.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c0415d640 4 set_mon_vals no callback set 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.964+0000 7f8c00ef7640 10 monclient: dump: 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:35.977 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 10 monclient: shutdown 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 -- 172.21.15.144:0/2654393550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09ebe0 msgr2=0x7f8bfc068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 --2- 172.21.15.144:0/2654393550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc068e00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc068d90 tx=0x7f8bf00091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 -- 172.21.15.144:0/2654393550 shutdown_connections 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 --2- 172.21.15.144:0/2654393550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc09efe0 0x7f8bfc069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 --2- 172.21.15.144:0/2654393550 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc068e00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 --2- 172.21.15.144:0/2654393550 >> v2:172.21.15.116:3300/0 conn(0x7f8bfc131bd0 0x7f8bfc131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 -- 172.21.15.144:0/2654393550 >> 172.21.15.144:0/2654393550 conn(0x7f8bfc0713b0 msgr2=0x7f8bfc0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 -- 172.21.15.144:0/2654393550 shutdown_connections 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.965+0000 7f8c0415d640 1 -- 172.21.15.144:0/2654393550 wait complete. 2024-09-18T12:31:35.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient: build_initial_monmap 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient: monmap: 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 Processor -- start 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 -- start start 2024-09-18T12:31:35.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient: init 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc09a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): picked mon.noname-a con 0x7f8bfc09ebe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 0x7f8bfc130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): picked mon.noname-b con 0x7f8bfc09efe0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc131bd0 0x7f8bfc140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): picked mon.noname-c con 0x7f8bfc131bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc11dfc0 con 0x7f8bfc09ebe0 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:35.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc11e2a0 con 0x7f8bfc09efe0 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc11e580 con 0x7f8bfc131bd0 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.966+0000 7f8c0415d640 10 monclient(hunting): authenticate will time out at 1131.535947s 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 0x7f8bfc130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 0x7f8bfc130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40646/0 (socket says 172.21.15.144:40646) 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 -- 172.21.15.144:0/1388717781 learned_addr learned my addr 172.21.15.144:0/1388717781 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): get_auth_request con 0x7f8bfc09efe0 auth_method 0 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:35.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc09a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 10 monclient(hunting): get_auth_request con 0x7f8bfc09ebe0 auth_method 0 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c026fa640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc131bd0 0x7f8bfc140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c026fa640 10 monclient(hunting): get_auth_request con 0x7f8bfc131bd0 auth_method 0 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c026fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c026fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c026fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient(hunting): handle_auth_done global_id 5845 payload 824 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 1 -- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 msgr2=0x7f8bfc09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 1 -- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 msgr2=0x7f8bfc09a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc09a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 -- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc131bd0 msgr2=0x7f8bfc140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc131bd0 0x7f8bfc140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c01ef9640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 -- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 msgr2=0x7f8bfc09a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc09a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient: _finish_hunting 0 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 monclient: found mon.noname-b 2024-09-18T12:31:35.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c016f8640 1 -- 172.21.15.144:0/1388717781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf0009450 con 0x7f8bfc09efe0 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.967+0000 7f8c026fa640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc131bd0 0x7f8bfc140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c016f8640 1 -- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 msgr2=0x7f8bfc130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c016f8640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 0x7f8bfc130bc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009210 tx=0x7f8bf0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf00319c0 con 0x7f8bfc09efe0 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: dump: 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:35.984 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c0415d640 5 monclient: authenticate success, global_id 5845 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0031b20 con 0x7f8bfc09efe0 2024-09-18T12:31:35.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c0415d640 10 monclient: _renew_subs 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c0415d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc11c630 con 0x7f8bfc09efe0 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf003d440 con 0x7f8bfc09efe0 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bf9ffb640 10 monclient: dump: 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:35.986 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c0415d640 10 monclient: _renew_subs 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8bfbfff640 4 set_mon_vals no callback set 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c0415d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.969+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc11cf20 con 0x7f8bfc09efe0 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.970+0000 7f8c0415d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.970+0000 7f8c0415d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.970+0000 7f8c0415d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.970+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8bf0039040 con 0x7f8bfc09efe0 2024-09-18T12:31:35.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.970+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc04bf60 con 0x7f8bfc09efe0 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.970+0000 7f8bf9ffb640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8be80384e0 0x7f8be803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.971+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(151..151 src has 13..151) v4 ==== 8231+0+0 (secure 0 0 0) 0x7f8bf006c320 con 0x7f8bfc09efe0 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.971+0000 7f8c01ef9640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8be80384e0 0x7f8be803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.971+0000 7f8c01ef9640 10 monclient: get_auth_request con 0x7f8be80384e0 auth_method 0 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.973+0000 7f8c01ef9640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8be80384e0 0x7f8be803a990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009710 tx=0x7f8bec0097d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.975+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8bf002f2a0 con 0x7f8bfc09efe0 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.975+0000 7f8bf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:35.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:35.975+0000 7f8bf9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.138+0000 7f8c0415d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T12:31:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.138+0000 7f8c0415d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T12:31:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.138+0000 7f8c0415d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:36.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.138+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f8bfc064990 con 0x7f8bfc09efe0 2024-09-18T12:31:36.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.777+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v152) v1 ==== 107+0+0 (secure 0 0 0) 0x7f8bf00370b0 con 0x7f8bfc09efe0 2024-09-18T12:31:36.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.777+0000 7f8bf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T12:31:36.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.777+0000 7f8bf9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.878+0000 7f8c0415d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.878+0000 7f8c0415d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.878+0000 7f8c0415d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.878+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f8bfc11bc00 con 0x7f8bfc09efe0 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.879+0000 7f8bf9ffb640 1 -- 172.21.15.144:0/1388717781 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v152) v1 ==== 149+0+0 (secure 0 0 0) 0x7f8bf0042030 con 0x7f8bfc09efe0 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.879+0000 7f8bf9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.879+0000 7f8bf9ffb640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-18T12:31:36.880 INFO:tasks.workunit.client.0.smithi144.stderr:weight-set for pool 'cool' already created 2024-09-18T12:31:36.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8be80384e0 msgr2=0x7f8be803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:36.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8be80384e0 0x7f8be803a990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009710 tx=0x7f8bec0097d0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 10 monclient: shutdown 2024-09-18T12:31:36.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:36.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 msgr2=0x7f8bfc130bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:36.882 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 0x7f8bfc130bc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009210 tx=0x7f8bf0032000 comp rx=0 tx=0).stop 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 shutdown_connections 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8be80384e0 0x7f8be803a990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8bfc131bd0 0x7f8bfc140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8bfc09efe0 0x7f8bfc130bc0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 --2- 172.21.15.144:0/1388717781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc09ebe0 0x7f8bfc09a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 >> 172.21.15.144:0/1388717781 conn(0x7f8bfc0713b0 msgr2=0x7f8bfc0a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 shutdown_connections 2024-09-18T12:31:36.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.881+0000 7f8c0415d640 1 -- 172.21.15.144:0/1388717781 wait complete. 2024-09-18T12:31:36.891 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create cold positional 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.965+0000 7fa7c411d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.965+0000 7fa7c411d640 10 monclient: build_initial_monmap 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.965+0000 7fa7c411d640 10 monclient: monmap: 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:36.966438+0000 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:36.966438+0000 2024-09-18T12:31:36.967 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 1 Processor -- start 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 1 -- start start 2024-09-18T12:31:36.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient: init 2024-09-18T12:31:36.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:36.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:31:36.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7bc0a0ad0 0x7fa7bc0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:36.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient(hunting): picked mon.noname-c con 0x7fa7bc0a0ad0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:36.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7bc09ee60 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7bc09f260 0x7fa7bc1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient(hunting): picked mon.noname-a con 0x7fa7bc09f260 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc06a5b0 con 0x7fa7bc09ee60 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc0a0ea0 con 0x7fa7bc09f260 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc0a1bb0 con 0x7fa7bc0a0ad0 2024-09-18T12:31:36.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c411d640 10 monclient(hunting): authenticate will time out at 1132.535819s 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7bc0a0ad0 0x7fa7bc0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7bc0a0ad0 0x7fa7bc0691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40656/0 (socket says 172.21.15.144:40656) 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 1 -- 172.21.15.144:0/941197965 learned_addr learned my addr 172.21.15.144:0/941197965 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 10 monclient(hunting): get_auth_request con 0x7fa7bc0a0ad0 auth_method 0 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.966+0000 7fa7c1eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:36.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:36.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 10 monclient(hunting): handle_auth_done global_id 5848 payload 274 2024-09-18T12:31:36.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 1 -- 172.21.15.144:0/941197965 >> v2:172.21.15.116:3300/0 conn(0x7fa7bc09f260 msgr2=0x7fa7bc1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:36.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 1 --2- 172.21.15.144:0/941197965 >> v2:172.21.15.116:3300/0 conn(0x7fa7bc09f260 0x7fa7bc1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 1 -- 172.21.15.144:0/941197965 >> v1:172.21.15.116:6789/0 conn(0x7fa7bc09ee60 legacy=0x7fa7bc131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:36.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 10 monclient: _finish_hunting 0 2024-09-18T12:31:36.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 1 monclient: found mon.noname-c 2024-09-18T12:31:36.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:36.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:36.974 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.967+0000 7fa7c1eb9640 1 -- 172.21.15.144:0/941197965 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc0a0460 con 0x7fa7bc0a0ad0 2024-09-18T12:31:36.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.969+0000 7fa7c1eb9640 1 -- 172.21.15.144:0/941197965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc0a0ad0 msgr2=0x7fa7bc0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.969+0000 7fa7c1eb9640 1 --2- 172.21.15.144:0/941197965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc0a0ad0 0x7fa7bc0691d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc06abc0 tx=0x7fa7b8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133facff25eed41b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.969+0000 7fa7c0eb7640 1 -- 172.21.15.144:0/941197965 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8007610 con 0x7fa7bc0a0ad0 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: dump: 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:36.981 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: _finish_auth 0 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 5 monclient: authenticate success, global_id 5848 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 1 -- 172.21.15.144:0/941197965 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b802d050 con 0x7fa7bc0a0ad0 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 4 set_mon_vals no callback set 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 1 -- 172.21.15.144:0/941197965 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80079e0 con 0x7fa7bc0a0ad0 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:36.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c0eb7640 10 monclient: dump: 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:36.983 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 10 monclient: shutdown 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 -- 172.21.15.144:0/941197965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc0a0ad0 msgr2=0x7fa7bc0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 --2- 172.21.15.144:0/941197965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc0a0ad0 0x7fa7bc0691d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc06abc0 tx=0x7fa7b8009840 comp rx=0 tx=0).stop 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 -- 172.21.15.144:0/941197965 shutdown_connections 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 --2- 172.21.15.144:0/941197965 >> v2:172.21.15.116:3300/0 conn(0x7fa7bc09f260 0x7fa7bc1321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 --2- 172.21.15.144:0/941197965 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc0a0ad0 0x7fa7bc0691d0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 -- 172.21.15.144:0/941197965 >> 172.21.15.144:0/941197965 conn(0x7fa7bc071370 msgr2=0x7fa7bc0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 -- 172.21.15.144:0/941197965 shutdown_connections 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.970+0000 7fa7c411d640 1 -- 172.21.15.144:0/941197965 wait complete. 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient: build_initial_monmap 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient: monmap: 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:36.985 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 Processor -- start 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 -- start start 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient: init 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7bc09ee60 0x7fa7bc1411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): picked mon.noname-c con 0x7fa7bc09ee60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 0x7fa7bc133f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): picked mon.noname-b con 0x7fa7bc09f260 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc0a0ad0 0x7fa7bc1344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): picked mon.noname-a con 0x7fa7bc0a0ad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:36.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0a11e0 con 0x7fa7bc0a0ad0 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc122700 con 0x7fa7bc09f260 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc141c00 con 0x7fa7bc09ee60 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c411d640 10 monclient(hunting): authenticate will time out at 1132.540929s 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.971+0000 7fa7c16b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 0x7fa7bc133f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 0x7fa7bc133f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:40658/0 (socket says 172.21.15.144:40658) 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 -- 172.21.15.144:0/2227115316 learned_addr learned my addr 172.21.15.144:0/2227115316 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): get_auth_request con 0x7fa7bc09f260 auth_method 0 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:36.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c1eb9640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7bc09ee60 0x7fa7bc1411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c1eb9640 10 monclient(hunting): get_auth_request con 0x7fa7bc09ee60 auth_method 0 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c26ba640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc0a0ad0 0x7fa7bc1344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c1eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c1eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c1eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c26ba640 10 monclient(hunting): get_auth_request con 0x7fa7bc0a0ad0 auth_method 0 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c26ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c26ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c26ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient(hunting): handle_auth_done global_id 5851 payload 824 2024-09-18T12:31:36.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 -- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7bc09ee60 msgr2=0x7fa7bc1411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7bc09ee60 0x7fa7bc1411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 -- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc0a0ad0 msgr2=0x7fa7bc1344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc0a0ad0 0x7fa7bc1344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient: _finish_hunting 0 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 monclient: found mon.noname-b 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.972+0000 7fa7c16b8640 1 -- 172.21.15.144:0/2227115316 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8009ae0 con 0x7fa7bc09f260 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c16b8640 1 -- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 msgr2=0x7fa7bc133f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c16b8640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 0x7fa7bc133f80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00b610 tx=0x7fa7ac0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00f7a0 con 0x7fa7bc09f260 2024-09-18T12:31:36.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: dump: 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:36.990 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c411d640 5 monclient: authenticate success, global_id 5851 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac00f920 con 0x7fa7bc09f260 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00fce0 con 0x7fa7bc09f260 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c411d640 10 monclient: _renew_subs 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c411d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc120580 con 0x7fa7bc09f260 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:36.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b3fff640 4 set_mon_vals no callback set 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7b1ffb640 10 monclient: dump: 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:36.992 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c411d640 10 monclient: _renew_subs 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c411d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.974+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc122390 con 0x7fa7bc09f260 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa7ac010630 con 0x7fa7bc09f260 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7b1ffb640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7a0038490 0x7fa7a003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7c411d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(152..152 src has 13..152) v4 ==== 8563+0+0 (secure 0 0 0) 0x7fa7ac04c2b0 con 0x7fa7bc09f260 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7c411d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7c411d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7bc1228b0 con 0x7fa7bc09f260 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7c1eb9640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7a0038490 0x7fa7a003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.975+0000 7fa7c1eb9640 10 monclient: get_auth_request con 0x7fa7a0038490 auth_method 0 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.978+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:31:36.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.978+0000 7fa7c1eb9640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7a0038490 0x7fa7a003a940 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8033000 tx=0x7fa7b8036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:36.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.980+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa7bc1228b0 con 0x7fa7bc09f260 2024-09-18T12:31:36.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.980+0000 7fa7b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:36.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:36.980+0000 7fa7b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:37.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.142+0000 7fa7c411d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T12:31:37.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.142+0000 7fa7c411d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T12:31:37.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.142+0000 7fa7c411d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.142+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7fa7bc1228b0 con 0x7fa7bc09f260 2024-09-18T12:31:37.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.624+0000 7fb6586dd640 10 monclient: get_monmap_and_config 2024-09-18T12:31:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.624+0000 7fb6586dd640 10 monclient: build_initial_monmap 2024-09-18T12:31:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.624+0000 7fb6586dd640 10 monclient: monmap: 2024-09-18T12:31:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:37.625909+0000 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:37.625909+0000 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.624+0000 7fb6586dd640 1 Processor -- start 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.624+0000 7fb6586dd640 1 -- start start 2024-09-18T12:31:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.624+0000 7fb6586dd640 10 monclient: init 2024-09-18T12:31:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb650096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6500971e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 0x7fb650136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient(hunting): picked mon.noname-a con 0x7fb650134b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65009d4a0 con 0x7fb650096e10 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb650099620 con 0x7fb6500971e0 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb650099340 con 0x7fb650134b40 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb6586dd640 10 monclient(hunting): authenticate will time out at 1124.947598s 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 0x7fb650136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656479640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb650096e10 0x7fb65009a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58116/0 (socket says 172.21.15.116:58116) 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656479640 1 -- 172.21.15.116:0/2493154599 learned_addr learned my addr 172.21.15.116:0/2493154599 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 10 monclient(hunting): get_auth_request con 0x7fb650134b40 auth_method 0 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 1 -- 172.21.15.116:0/2493154599 >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 msgr2=0x7fb650136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 1 -- 172.21.15.116:0/2493154599 >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 msgr2=0x7fb650136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 1 --2- 172.21.15.116:0/2493154599 >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 0x7fb650136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.625+0000 7fb656c7a640 1 --2- 172.21.15.116:0/2493154599 >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 0x7fb650136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.627+0000 7fb656479640 1 -- 172.21.15.116:0/2493154599 >> v1:172.21.15.116:6789/0 conn(0x7fb650096e10 legacy=0x7fb65009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655c78640 1 -- 172.21.15.116:0/2493154599 >> v1:172.21.15.144:6789/0 conn(0x7fb6500971e0 legacy=0x7fb6500975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603053766 0 0) 0x7fb650099620 con 0x7fb6500971e0 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): my global_id is 5854 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb63c0034e0 con 0x7fb6500971e0 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 72745788 0 0) 0x7fb63c0034e0 con 0x7fb6500971e0 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 msgr2=0x7fb650136f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 --2- 172.21.15.116:0/2493154599 >> v2:172.21.15.116:3300/0 conn(0x7fb650134b40 0x7fb650136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 >> v1:172.21.15.116:6789/0 conn(0x7fb650096e10 legacy=0x7fb65009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: _finish_hunting 0 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:31:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650099900 con 0x7fb6500971e0 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: _finish_auth 0 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fb6440018b0 con 0x7fb6500971e0 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: dump: 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.639 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 >> v1:172.21.15.144:6789/0 conn(0x7fb6500971e0 legacy=0x7fb6500975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c003af0 0x7fb63c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): picked mon.a con 0x7fb63c003af0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63c004780 0x7fb63c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): picked mon.b con 0x7fb63c004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb63c005380 0x7fb63c00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): picked mon.c con 0x7fb63c005380 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb650099340 con 0x7fb63c003af0 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c00c350 con 0x7fb63c004780 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c00c510 con 0x7fb63c005380 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655477640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c003af0 0x7fb63c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 10 monclient(hunting): get_auth_request con 0x7fb63c003af0 auth_method 0 2024-09-18T12:31:37.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656c7a640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb63c005380 0x7fb63c00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655c78640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63c004780 0x7fb63c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656c7a640 10 monclient(hunting): get_auth_request con 0x7fb63c005380 auth_method 0 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656c7a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656c7a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655c78640 10 monclient(hunting): get_auth_request con 0x7fb63c004780 auth_method 0 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655c78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655c78640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb655c78640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656479640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.628+0000 7fb656c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb655c78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb655c78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb655c78640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 10 monclient(hunting): handle_auth_done global_id 5854 payload 293 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 1 -- 172.21.15.116:0/2493154599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63c004780 msgr2=0x7fb63c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb63c004780 0x7fb63c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 1 -- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c003af0 msgr2=0x7fb63c003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c003af0 0x7fb63c003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 10 monclient: _finish_hunting 0 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 1 monclient: found mon.c 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 1 -- 172.21.15.116:0/2493154599 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c0034e0 con 0x7fb63c005380 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 10 monclient: _finish_auth 0 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb6586dd640 5 monclient: authenticate success, global_id 5854 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656479640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c003af0 0x7fb63c003f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb6586dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.629+0000 7fb656c7a640 1 -- 172.21.15.116:0/2493154599 reap_dead start 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.631+0000 7fb656c7a640 1 -- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb63c005380 msgr2=0x7fb63c00ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.631+0000 7fb656c7a640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb63c005380 0x7fb63c00ba70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb64c006fd0 tx=0x7fb64c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f87dd0b83631968 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c02ade0 con 0x7fb63c005380 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 10 monclient: dump: 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c02e070 con 0x7fb63c005380 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 4 set_mon_vals no callback set 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 1 -- 172.21.15.116:0/2493154599 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c034660 con 0x7fb63c005380 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb655477640 10 monclient: dump: 2024-09-18T12:31:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 10 monclient: shutdown 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 1 -- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb63c005380 msgr2=0x7fb63c00ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb63c005380 0x7fb63c00ba70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb64c006fd0 tx=0x7fb64c008040 comp rx=0 tx=0).stop 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 1 -- 172.21.15.116:0/2493154599 shutdown_connections 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 1 --2- 172.21.15.116:0/2493154599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb63c005380 0x7fb63c00ba70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 1 -- 172.21.15.116:0/2493154599 >> 172.21.15.116:0/2493154599 conn(0x7fb6500672b0 msgr2=0x7fb6500676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 1 -- 172.21.15.116:0/2493154599 shutdown_connections 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.632+0000 7fb6586dd640 1 -- 172.21.15.116:0/2493154599 wait complete. 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.633+0000 7fb6586dd640 10 monclient: build_initial_monmap 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.633+0000 7fb6586dd640 10 monclient: monmap: 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.633+0000 7fb6586dd640 1 Processor -- start 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.633+0000 7fb6586dd640 1 -- start start 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.633+0000 7fb6586dd640 10 monclient: init 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 0x7fb65012e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): picked mon.noname-a con 0x7fb63c005380 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb65012eb40 0x7fb6501190c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): picked mon.noname-c con 0x7fb65012eb40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb650119600 0x7fb650119aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): picked mon.noname-b con 0x7fb650119600 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65012ef30 con 0x7fb63c005380 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65011cd80 con 0x7fb650119600 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65009b050 con 0x7fb65012eb40 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb6586dd640 10 monclient(hunting): authenticate will time out at 1124.956128s 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 0x7fb65012e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb65012eb40 0x7fb6501190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 0x7fb65012e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50154/0 (socket says 172.21.15.116:50154) 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 1 -- 172.21.15.116:0/3276613229 learned_addr learned my addr 172.21.15.116:0/3276613229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 10 monclient(hunting): get_auth_request con 0x7fb63c005380 auth_method 0 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): get_auth_request con 0x7fb65012eb40 auth_method 0 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:37.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 1 -- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 msgr2=0x7fb65012e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 1 -- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 msgr2=0x7fb65012e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 0x7fb65012e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656479640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 0x7fb65012e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb656c7a640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb650119600 0x7fb650119aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient(hunting): handle_auth_done global_id 64670 payload 824 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 1 -- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb650119600 msgr2=0x7fb650119aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb650119600 0x7fb650119aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 1 -- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 msgr2=0x7fb65012e600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 0x7fb65012e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient: _finish_hunting 0 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 1 monclient: found mon.noname-c 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.634+0000 7fb655c78640 1 -- 172.21.15.116:0/3276613229 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6501342a0 con 0x7fb65012eb40 2024-09-18T12:31:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb655c78640 1 -- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb65012eb40 msgr2=0x7fb6501190c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb655c78640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb65012eb40 0x7fb6501190c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb644007170 tx=0x7fb644007a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 1 -- 172.21.15.116:0/3276613229 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644007d90 con 0x7fb65012eb40 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 10 monclient: dump: 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 1 -- 172.21.15.116:0/3276613229 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb644007760 con 0x7fb65012eb40 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6586dd640 5 monclient: authenticate success, global_id 64670 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6427fc640 1 -- 172.21.15.116:0/3276613229 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64400fe20 con 0x7fb65012eb40 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb6586dd640 10 monclient: _renew_subs 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65011b7d0 con 0x7fb65012eb40 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.636+0000 7fb654c76640 4 set_mon_vals no callback set 2024-09-18T12:31:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6427fc640 10 monclient: dump: 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 10 monclient: _renew_subs 2024-09-18T12:31:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65011c730 con 0x7fb65012eb40 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6427fc640 1 -- 172.21.15.116:0/3276613229 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb644013600 con 0x7fb65012eb40 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.637+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65004bf60 con 0x7fb65012eb40 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.638+0000 7fb6427fc640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb6340384e0 0x7fb63403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.638+0000 7fb6427fc640 1 -- 172.21.15.116:0/3276613229 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(152..152 src has 13..152) v4 ==== 8563+0+0 (secure 0 0 0) 0x7fb64404cb90 con 0x7fb65012eb40 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.638+0000 7fb656479640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb6340384e0 0x7fb63403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.638+0000 7fb656479640 10 monclient: get_auth_request con 0x7fb6340384e0 auth_method 0 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.640+0000 7fb656479640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb6340384e0 0x7fb63403a990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb648002250 tx=0x7fb648009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.642+0000 7fb6427fc640 1 -- 172.21.15.116:0/3276613229 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb644023ae0 con 0x7fb65012eb40 2024-09-18T12:31:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.642+0000 7fb6427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.642+0000 7fb6427fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.778+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v153) v1 ==== 113+0+0 (secure 0 0 0) 0x7fa7bc1228b0 con 0x7fa7bc09f260 2024-09-18T12:31:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.778+0000 7fa7b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T12:31:37.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.778+0000 7fa7b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:37.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.860+0000 7fb6586dd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.860+0000 7fb6586dd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.860+0000 7fb6586dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.860+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb650141e20 con 0x7fb65012eb40 2024-09-18T12:31:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.861+0000 7fb6427fc640 1 -- 172.21.15.116:0/3276613229 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fb644017020 con 0x7fb65012eb40 2024-09-18T12:31:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.861+0000 7fb6427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.861+0000 7fb6427fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:37.863 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":68,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.862+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb6340384e0 msgr2=0x7fb63403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.862+0000 7fb6586dd640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb6340384e0 0x7fb63403a990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb648002250 tx=0x7fb648009040 comp rx=0 tx=0).stop 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.862+0000 7fb6586dd640 10 monclient: shutdown 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.862+0000 7fb6586dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.862+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb65012eb40 msgr2=0x7fb6501190c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.862+0000 7fb6586dd640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb65012eb40 0x7fb6501190c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb644007170 tx=0x7fb644007a60 comp rx=0 tx=0).stop 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 shutdown_connections 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb6340384e0 0x7fb63403a990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb650119600 0x7fb650119aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb65012eb40 0x7fb6501190c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 --2- 172.21.15.116:0/3276613229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb63c005380 0x7fb65012e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 >> 172.21.15.116:0/3276613229 conn(0x7fb6500672b0 msgr2=0x7fb650132900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:37.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 shutdown_connections 2024-09-18T12:31:37.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:37.863+0000 7fb6586dd640 1 -- 172.21.15.116:0/3276613229 wait complete. 2024-09-18T12:31:37.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.879+0000 7fa7c411d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T12:31:37.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.879+0000 7fa7c411d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T12:31:37.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.879+0000 7fa7c411d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.879+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7fa7bc1228b0 con 0x7fa7bc09f260 2024-09-18T12:31:37.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.879+0000 7fa7b1ffb640 1 -- 172.21.15.144:0/2227115316 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v153) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa7bc1228b0 con 0x7fa7bc09f260 2024-09-18T12:31:37.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.879+0000 7fa7b1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T12:31:37.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.879+0000 7fa7b1ffb640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-18T12:31:37.881 INFO:tasks.workunit.client.0.smithi144.stderr:weight-set for pool 'cold' already created 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7a0038490 msgr2=0x7fa7a003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7a0038490 0x7fa7a003a940 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8033000 tx=0x7fa7b8036000 comp rx=0 tx=0).stop 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 10 monclient: shutdown 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 msgr2=0x7fa7bc133f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 0x7fa7bc133f80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00b610 tx=0x7fa7ac0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 shutdown_connections 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7a0038490 0x7fa7a003a940 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7bc0a0ad0 0x7fa7bc1344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7bc09f260 0x7fa7bc133f80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 --2- 172.21.15.144:0/2227115316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7bc09ee60 0x7fa7bc1411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 >> 172.21.15.144:0/2227115316 conn(0x7fa7bc071370 msgr2=0x7fa7bc09e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:37.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 shutdown_connections 2024-09-18T12:31:37.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.881+0000 7fa7c411d640 1 -- 172.21.15.144:0/2227115316 wait complete. 2024-09-18T12:31:37.891 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-18T12:31:37.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.964+0000 7f7d8e2e8640 10 monclient: get_monmap_and_config 2024-09-18T12:31:37.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.964+0000 7f7d8e2e8640 10 monclient: build_initial_monmap 2024-09-18T12:31:37.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.964+0000 7f7d8e2e8640 10 monclient: monmap: 2024-09-18T12:31:37.965 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:37.965 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:37.965205+0000 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:37.965205+0000 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:37.966 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:37.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 1 Processor -- start 2024-09-18T12:31:37.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 1 -- start start 2024-09-18T12:31:37.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient: init 2024-09-18T12:31:37.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:37.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:31:37.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d8809ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:37.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7d8809f020 0x7f7d88069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient(hunting): picked mon.noname-c con 0x7f7d8809f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d88136d60 0x7f7d88139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f7d88136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d88067420 con 0x7f7d8809ec50 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d880a0480 con 0x7f7d8809f020 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880a0200 con 0x7f7d88136d60 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8e2e8640 10 monclient(hunting): authenticate will time out at 1133.534601s 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7d8809f020 0x7f7d88069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7d8809f020 0x7f7d88069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53502/0 (socket says 172.21.15.144:53502) 2024-09-18T12:31:37.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 -- 172.21.15.144:0/3939493714 learned_addr learned my addr 172.21.15.144:0/3939493714 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): get_auth_request con 0x7f7d8809f020 auth_method 0 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 1 --2- 172.21.15.144:0/3939493714 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d88136d60 0x7f7d88139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 10 monclient(hunting): get_auth_request con 0x7f7d88136d60 auth_method 0 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d8c885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient(hunting): handle_auth_done global_id 5857 payload 274 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 -- 172.21.15.144:0/3939493714 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d88136d60 msgr2=0x7f7d88139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 --2- 172.21.15.144:0/3939493714 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d88136d60 0x7f7d88139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 -- 172.21.15.144:0/3939493714 >> v1:172.21.15.116:6789/0 conn(0x7f7d8809ec50 legacy=0x7f7d88131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 monclient: found mon.noname-c 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.965+0000 7f7d877fe640 1 -- 172.21.15.144:0/3939493714 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8809fd80 con 0x7f7d8809f020 2024-09-18T12:31:37.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.968+0000 7f7d877fe640 1 -- 172.21.15.144:0/3939493714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 msgr2=0x7f7d88069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:37.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.968+0000 7f7d877fe640 1 --2- 172.21.15.144:0/3939493714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88069d10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009910 tx=0x7f7d7c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f224714aa65f90 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:37.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 1 -- 172.21.15.144:0/3939493714 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c02ae90 con 0x7f7d8809f020 2024-09-18T12:31:37.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: dump: 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:37.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 1 -- 172.21.15.144:0/3939493714 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c032440 con 0x7f7d8809f020 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 5 monclient: authenticate success, global_id 5857 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 1 -- 172.21.15.144:0/3939493714 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c032820 con 0x7f7d8809f020 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 4 set_mon_vals no callback set 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d86ffd640 10 monclient: dump: 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:37.983 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 10 monclient: shutdown 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/3939493714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 msgr2=0x7f7d88069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/3939493714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88069d10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009910 tx=0x7f7d7c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/3939493714 shutdown_connections 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/3939493714 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d88136d60 0x7f7d88139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/3939493714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88069d10 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/3939493714 >> 172.21.15.144:0/3939493714 conn(0x7f7d880a5cf0 msgr2=0x7f7d880a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/3939493714 shutdown_connections 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.969+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/3939493714 wait complete. 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 10 monclient: build_initial_monmap 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 10 monclient: monmap: 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:37.985 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 1 Processor -- start 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 1 -- start start 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 10 monclient: init 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:37.986 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 0x7f7d88097420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 10 monclient(hunting): picked mon.noname-c con 0x7f7d8809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88097960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.970+0000 7f7d8e2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f7d8809f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d88097ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 10 monclient(hunting): picked mon.noname-a con 0x7f7d88136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8809ace0 con 0x7f7d88136d60 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8809c020 con 0x7f7d8809f020 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880a4060 con 0x7f7d8809ec50 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8e2e8640 10 monclient(hunting): authenticate will time out at 1133.540031s 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88097960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88097960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53508/0 (socket says 172.21.15.144:53508) 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 -- 172.21.15.144:0/15967623 learned_addr learned my addr 172.21.15.144:0/15967623 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): get_auth_request con 0x7f7d8809f020 auth_method 0 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 0x7f7d88097420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 10 monclient(hunting): get_auth_request con 0x7f7d8809ec50 auth_method 0 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d88097ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 10 monclient(hunting): get_auth_request con 0x7f7d88136d60 auth_method 0 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 msgr2=0x7f7d88097ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 msgr2=0x7f7d88097ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d88097ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient(hunting): handle_auth_done global_id 5860 payload 824 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d8c885640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d88097ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 msgr2=0x7f7d88097420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 0x7f7d88097420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 msgr2=0x7f7d88097ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d88097ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:37.990 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.971+0000 7f7d877fe640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c009450 con 0x7f7d8809f020 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d877fe640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 msgr2=0x7f7d88097960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d877fe640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88097960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c035040 tx=0x7f7d7c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c000d10 con 0x7f7d8809f020 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: dump: 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.991 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: _finish_auth 0 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c02ae90 con 0x7f7d8809f020 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d8e2e8640 5 monclient: authenticate success, global_id 5860 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d8e2e8640 10 monclient: _renew_subs 2024-09-18T12:31:37.992 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d8e2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d88098ed0 con 0x7f7d8809f020 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c03d440 con 0x7f7d8809f020 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: dump: 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:37.993 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d8e2e8640 10 monclient: _renew_subs 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d8e2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d867fc640 4 set_mon_vals no callback set 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8809aab0 con 0x7f7d8809f020 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d877fe640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 msgr2=0x7f7d88097960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d877fe640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 msgr2=0x7f7d88097960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d877fe640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88097960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c035040 tx=0x7f7d7c032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:37.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d877fe640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88097960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c035040 tx=0x7f7d7c032000 comp rx=0 tx=0).stop 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 msgr2=0x7f7d88097960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.973+0000 7f7d6ffff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d8809f020 0x7f7d88097960 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d740057a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient(hunting): picked mon.a con 0x7f7d88136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:37.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 0x7f7d74008110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient(hunting): picked mon.c con 0x7f7d8809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d74008a90 0x7f7d7400af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient(hunting): picked mon.b con 0x7f7d74008a90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7400b9c0 con 0x7f7d88136d60 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7400bb00 con 0x7f7d74008a90 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7400bc40 con 0x7f7d8809ec50 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d6ffff640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d74008a90 0x7f7d7400af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): get_auth_request con 0x7f7d74008a90 auth_method 0 2024-09-18T12:31:37.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 0x7f7d74008110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 10 monclient(hunting): get_auth_request con 0x7f7d8809ec50 auth_method 0 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d740057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 10 monclient(hunting): get_auth_request con 0x7f7d88136d60 auth_method 0 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.997 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d877fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d87fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient(hunting): handle_auth_done global_id 5860 payload 293 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 msgr2=0x7f7d74008110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 0x7f7d74008110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 msgr2=0x7f7d740057a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d740057a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:37.998 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient: _finish_hunting 0 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 1 monclient: found mon.b 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f7d7400bf30 con 0x7f7d74008a90 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 10 monclient: _finish_auth 0 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8c885640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8e2e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8e2e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8e2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.974+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8809af20 con 0x7f7d74008a90 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d8c885640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d74008a90 msgr2=0x7f7d7400af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d8c885640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d74008a90 0x7f7d7400af10 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009b00 tx=0x7f7d8000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:37.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000e030 con 0x7f7d74008a90 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d6ffff640 10 monclient: dump: 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:38.000 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d80015600 con 0x7f7d74008a90 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.976+0000 7f7d85ffb640 4 set_mon_vals no callback set 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.977+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7d80015770 con 0x7f7d74008a90 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.977+0000 7f7d6ffff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d74040f40 0x7f7d740433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.977+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 4 ==== osd_map(153..153 src has 13..153) v4 ==== 9035+0+0 (secure 0 0 0) 0x7f7d8000f070 con 0x7f7d74008a90 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.977+0000 7f7d87fff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d74040f40 0x7f7d740433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.977+0000 7f7d87fff640 10 monclient: get_auth_request con 0x7f7d74040f40 auth_method 0 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.980+0000 7f7d87fff640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d74040f40 0x7f7d740433f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009b80 tx=0x7f7d78006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:38.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.981+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7d800132b0 con 0x7f7d74008a90 2024-09-18T12:31:38.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.981+0000 7f7d6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:38.002 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:37.981+0000 7f7d6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:38.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.139+0000 7f7d8e2e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:38.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.139+0000 7f7d8e2e8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:38.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.139+0000 7f7d8e2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:38.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.139+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f7d880008d0 con 0x7f7d74008a90 2024-09-18T12:31:38.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.785+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v154) v1 ==== 123+0+0 (secure 0 0 0) 0x7f7d80019030 con 0x7f7d74008a90 2024-09-18T12:31:38.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.785+0000 7f7d6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:38.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.785+0000 7f7d6ffff640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-18T12:31:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.882+0000 7f7d8e2e8640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.882+0000 7f7d8e2e8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.882+0000 7f7d8e2e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:38.882+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f7d88072640 con 0x7f7d74008a90 2024-09-18T12:31:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.791+0000 7f7d6ffff640 1 -- 172.21.15.144:0/15967623 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v155) v1 ==== 149+0+0 (secure 0 0 0) 0x7f7d80019020 con 0x7f7d74008a90 2024-09-18T12:31:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.791+0000 7f7d6ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.791+0000 7f7d6ffff640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T12:31:39.792 INFO:tasks.workunit.client.0.smithi144.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d74040f40 msgr2=0x7f7d740433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d74040f40 0x7f7d740433f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009b80 tx=0x7f7d78006f50 comp rx=0 tx=0).stop 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 10 monclient: shutdown 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d74008a90 msgr2=0x7f7d7400af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d74008a90 0x7f7d7400af10 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009b00 tx=0x7f7d8000b040 comp rx=0 tx=0).stop 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 shutdown_connections 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d74040f40 0x7f7d740433f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d74008a90 0x7f7d7400af10 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d8809ec50 0x7f7d74008110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 --2- 172.21.15.144:0/15967623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d88136d60 0x7f7d740057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 >> 172.21.15.144:0/15967623 conn(0x7f7d880a5cf0 msgr2=0x7f7d88068cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 shutdown_connections 2024-09-18T12:31:39.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.793+0000 7f7d8e2e8640 1 -- 172.21.15.144:0/15967623 wait complete. 2024-09-18T12:31:39.805 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.884+0000 7eff39e94640 10 monclient: get_monmap_and_config 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.884+0000 7eff39e94640 10 monclient: build_initial_monmap 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.884+0000 7eff39e94640 10 monclient: monmap: 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:39.885023+0000 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:39.885023+0000 2024-09-18T12:31:39.885 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:39.886 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:39.886 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:39.886 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:39.886 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:39.886 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:39.886 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:39.886 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:39.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.884+0000 7eff39e94640 1 Processor -- start 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.884+0000 7eff39e94640 1 -- start start 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.884+0000 7eff39e94640 10 monclient: init 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3409ec50 0x7eff34131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): picked mon.noname-b con 0x7eff3409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff341321a0 0x7eff34068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): picked mon.noname-a con 0x7eff341321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7eff3409f020 0x7eff34069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): picked mon.noname-c con 0x7eff3409f020 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff34067420 con 0x7eff341321a0 2024-09-18T12:31:39.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3409fdc0 con 0x7eff3409f020 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340a0510 con 0x7eff3409ec50 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff39e94640 10 monclient(hunting): authenticate will time out at 1135.454408s 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7eff3409f020 0x7eff34069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7eff3409f020 0x7eff34069240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53528/0 (socket says 172.21.15.144:53528) 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 -- 172.21.15.144:0/2727007856 learned_addr learned my addr 172.21.15.144:0/2727007856 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): get_auth_request con 0x7eff3409f020 auth_method 0 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:39.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff337fe640 1 --2- 172.21.15.144:0/2727007856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3409ec50 0x7eff34131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff337fe640 10 monclient(hunting): get_auth_request con 0x7eff3409ec50 auth_method 0 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient(hunting): handle_auth_done global_id 5863 payload 274 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 -- 172.21.15.144:0/2727007856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3409ec50 msgr2=0x7eff34131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 --2- 172.21.15.144:0/2727007856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3409ec50 0x7eff34131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 -- 172.21.15.144:0/2727007856 >> v2:172.21.15.116:3300/0 conn(0x7eff341321a0 msgr2=0x7eff34068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 --2- 172.21.15.144:0/2727007856 >> v2:172.21.15.116:3300/0 conn(0x7eff341321a0 0x7eff34068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:39.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 monclient: found mon.noname-c 2024-09-18T12:31:39.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:39.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:39.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.885+0000 7eff33fff640 1 -- 172.21.15.144:0/2727007856 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340a0200 con 0x7eff3409f020 2024-09-18T12:31:39.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff33fff640 1 -- 172.21.15.144:0/2727007856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 msgr2=0x7eff34069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:39.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff33fff640 1 --2- 172.21.15.144:0/2727007856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff34069240 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7eff340a0150 tx=0x7eff2c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbed8d6f020e10c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:39.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 1 -- 172.21.15.144:0/2727007856 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c007610 con 0x7eff3409f020 2024-09-18T12:31:39.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:39.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: dump: 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:39.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 1 -- 172.21.15.144:0/2727007856 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c02d050 con 0x7eff3409f020 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff39e94640 5 monclient: authenticate success, global_id 5863 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff39e94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff39e94640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 1 -- 172.21.15.144:0/2727007856 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0079a0 con 0x7eff3409f020 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff39e94640 4 set_mon_vals no callback set 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.888+0000 7eff327fc640 10 monclient: dump: 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:39.900 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 10 monclient: shutdown 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 -- 172.21.15.144:0/2727007856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 msgr2=0x7eff34069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 --2- 172.21.15.144:0/2727007856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff34069240 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7eff340a0150 tx=0x7eff2c009840 comp rx=0 tx=0).stop 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 -- 172.21.15.144:0/2727007856 shutdown_connections 2024-09-18T12:31:39.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 --2- 172.21.15.144:0/2727007856 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff34069240 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 --2- 172.21.15.144:0/2727007856 >> v2:172.21.15.116:3300/0 conn(0x7eff341321a0 0x7eff34068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 --2- 172.21.15.144:0/2727007856 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff3409ec50 0x7eff34131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 -- 172.21.15.144:0/2727007856 >> 172.21.15.144:0/2727007856 conn(0x7eff340a5d40 msgr2=0x7eff340a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 -- 172.21.15.144:0/2727007856 shutdown_connections 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 1 -- 172.21.15.144:0/2727007856 wait complete. 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 10 monclient: build_initial_monmap 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.889+0000 7eff39e94640 10 monclient: monmap: 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:39.902 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 Processor -- start 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 -- start start 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient: init 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:39.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 0x7eff3409a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): picked mon.noname-a con 0x7eff3409ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff341309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): picked mon.noname-b con 0x7eff3409f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff341321a0 0x7eff34130f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): picked mon.noname-c con 0x7eff341321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3409b320 con 0x7eff3409ec50 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3409e100 con 0x7eff3409f020 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3411db10 con 0x7eff341321a0 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:39.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff39e94640 10 monclient(hunting): authenticate will time out at 1135.459913s 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.890+0000 7eff32ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff341309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff341309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53538/0 (socket says 172.21.15.144:53538) 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 1 -- 172.21.15.144:0/225944393 learned_addr learned my addr 172.21.15.144:0/225944393 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 10 monclient(hunting): get_auth_request con 0x7eff3409f020 auth_method 0 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 msgr2=0x7eff341309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 msgr2=0x7eff341309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff341309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff32ffd640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff341309f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff341321a0 0x7eff34130f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:39.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 0x7eff3409a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): get_auth_request con 0x7eff341321a0 auth_method 0 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 10 monclient(hunting): get_auth_request con 0x7eff3409ec50 auth_method 0 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 msgr2=0x7eff3409a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:31:39.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 msgr2=0x7eff3409a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 0x7eff3409a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff337fe640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 0x7eff3409a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient(hunting): handle_auth_done global_id 64682 payload 824 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 msgr2=0x7eff341309f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff341309f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 msgr2=0x7eff3409a810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 0x7eff3409a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 1 monclient: found mon.noname-c 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.891+0000 7eff33fff640 1 -- 172.21.15.144:0/225944393 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c009ae0 con 0x7eff341321a0 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.893+0000 7eff33fff640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff341321a0 msgr2=0x7eff34130f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:39.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.893+0000 7eff33fff640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff341321a0 0x7eff34130f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff3409f500 tx=0x7eff2c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.893+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c032cc0 con 0x7eff341321a0 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.893+0000 7eff1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.893+0000 7eff1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.893+0000 7eff1b7fe640 10 monclient: dump: 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:39.908 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c032e20 con 0x7eff341321a0 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 5 monclient: authenticate success, global_id 64682 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 10 monclient: _renew_subs 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3411ddf0 con 0x7eff341321a0 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c03b640 con 0x7eff341321a0 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:39.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 10 monclient: dump: 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff31ffb640 4 set_mon_vals no callback set 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 10 monclient: _renew_subs 2024-09-18T12:31:39.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff341361c0 con 0x7eff341321a0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7eff2c038050 con 0x7eff341321a0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.894+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3411e1f0 con 0x7eff341321a0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.895+0000 7eff1b7fe640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff20038490 0x7eff2003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.895+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(155..155 src has 13..155) v4 ==== 9003+0+0 (secure 0 0 0) 0x7eff2c06d600 con 0x7eff341321a0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.895+0000 7eff337fe640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff20038490 0x7eff2003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.895+0000 7eff337fe640 10 monclient: get_auth_request con 0x7eff20038490 auth_method 0 2024-09-18T12:31:39.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.897+0000 7eff337fe640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff20038490 0x7eff2003a940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7eff24006fd0 tx=0x7eff24008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:39.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.900+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7eff2c03b9f0 con 0x7eff341321a0 2024-09-18T12:31:39.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.900+0000 7eff1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:39.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:39.900+0000 7eff1b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:40.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.061+0000 7eff39e94640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T12:31:40.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.062+0000 7eff39e94640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T12:31:40.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.062+0000 7eff39e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:40.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.062+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7eff3411dc50 con 0x7eff341321a0 2024-09-18T12:31:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.805+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v156) v1 ==== 119+0+0 (secure 0 0 0) 0x7eff3411dc50 con 0x7eff341321a0 2024-09-18T12:31:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.805+0000 7eff1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T12:31:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.805+0000 7eff1b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:40.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.911+0000 7eff39e94640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T12:31:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.911+0000 7eff39e94640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T12:31:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.911+0000 7eff39e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:40.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:40.911+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7eff3411dc50 con 0x7eff341321a0 2024-09-18T12:31:40.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.976+0000 7f7e9275d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.976+0000 7f7e9275d640 10 monclient: build_initial_monmap 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.976+0000 7f7e9275d640 10 monclient: monmap: 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:40.977749+0000 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:40.977749+0000 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:40.978 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:40.979 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:40.979 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:40.979 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:40.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.976+0000 7f7e9275d640 1 Processor -- start 2024-09-18T12:31:40.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.976+0000 7f7e9275d640 1 -- start start 2024-09-18T12:31:40.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.976+0000 7f7e9275d640 10 monclient: init 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 0x7f7e8c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e8c131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e8c09e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7e8c136bb0 0x7f7e8c138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e8c136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8c09faf0 con 0x7f7e8c09e910 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8c09fd30 con 0x7f7e8c131ab0 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8c0a1110 con 0x7f7e8c136bb0 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:40.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e9275d640 10 monclient(hunting): authenticate will time out at 1128.299409s 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 0x7f7e8c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 0x7f7e8c131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50168/0 (socket says 172.21.15.116:50168) 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7e8c09e910 0x7f7e8c09ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58126/0 (socket says 172.21.15.116:58126) 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 1 -- 172.21.15.116:0/1953745292 learned_addr learned my addr 172.21.15.116:0/1953745292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request con 0x7f7e8c131ab0 auth_method 0 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 1 -- 172.21.15.116:0/1953745292 >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 msgr2=0x7f7e8c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 1 -- 172.21.15.116:0/1953745292 >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 msgr2=0x7f7e8c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 1 --2- 172.21.15.116:0/1953745292 >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 0x7f7e8c131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e8bfff640 1 --2- 172.21.15.116:0/1953745292 >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 0x7f7e8c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:40.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 1 --2- 172.21.15.116:0/1953745292 >> v2:172.21.15.144:3300/0 conn(0x7f7e8c136bb0 0x7f7e8c138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): get_auth_request con 0x7f7e8c136bb0 auth_method 0 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient(hunting): handle_auth_done global_id 5866 payload 274 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 1 -- 172.21.15.116:0/1953745292 >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 msgr2=0x7f7e8c131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 1 --2- 172.21.15.116:0/1953745292 >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 0x7f7e8c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 1 -- 172.21.15.116:0/1953745292 >> v1:172.21.15.116:6789/0 conn(0x7f7e8c09e910 legacy=0x7f7e8c09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:40.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 1 monclient: found mon.noname-c 2024-09-18T12:31:40.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:40.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:40.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.977+0000 7f7e90cfa640 1 -- 172.21.15.116:0/1953745292 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c0a13f0 con 0x7f7e8c136bb0 2024-09-18T12:31:40.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.980+0000 7f7e90cfa640 1 -- 172.21.15.116:0/1953745292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 msgr2=0x7f7e8c138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:40.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.980+0000 7f7e90cfa640 1 --2- 172.21.15.116:0/1953745292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c138f90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009e60 tx=0x7f7e84009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d89633cf780e0f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:40.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 1 -- 172.21.15.116:0/1953745292 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84007760 con 0x7f7e8c136bb0 2024-09-18T12:31:40.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: dump: 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:40.992 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: _finish_auth 0 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 5 monclient: authenticate success, global_id 5866 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 1 -- 172.21.15.116:0/1953745292 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8402d050 con 0x7f7e8c136bb0 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 4 set_mon_vals no callback set 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 1 -- 172.21.15.116:0/1953745292 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84007b30 con 0x7f7e8c136bb0 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e8affd640 10 monclient: dump: 2024-09-18T12:31:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 10 monclient: shutdown 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 -- 172.21.15.116:0/1953745292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 msgr2=0x7f7e8c138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1953745292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c138f90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009e60 tx=0x7f7e84009780 comp rx=0 tx=0).stop 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 -- 172.21.15.116:0/1953745292 shutdown_connections 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1953745292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c138f90 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1953745292 >> v2:172.21.15.116:3300/0 conn(0x7f7e8c131ab0 0x7f7e8c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 -- 172.21.15.116:0/1953745292 >> 172.21.15.116:0/1953745292 conn(0x7f7e8c067d30 msgr2=0x7f7e8c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 -- 172.21.15.116:0/1953745292 shutdown_connections 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.981+0000 7f7e9275d640 1 -- 172.21.15.116:0/1953745292 wait complete. 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.982+0000 7f7e9275d640 10 monclient: build_initial_monmap 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.982+0000 7f7e9275d640 10 monclient: monmap: 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.982+0000 7f7e9275d640 1 Processor -- start 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.982+0000 7f7e9275d640 1 -- start start 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.982+0000 7f7e9275d640 10 monclient: init 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 0x7f7e8c1361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e8c09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e8c131ab0 0x7f7e8c12f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e8c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c12f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e8c136bb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c1331e0 con 0x7f7e8c09e910 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c133830 con 0x7f7e8c136bb0 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c1305e0 con 0x7f7e8c131ab0 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e9275d640 10 monclient(hunting): authenticate will time out at 1128.305356s 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 0x7f7e8c1361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 0x7f7e8c1361a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50180/0 (socket says 172.21.15.116:50180) 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 1 -- 172.21.15.116:0/1249563422 learned_addr learned my addr 172.21.15.116:0/1249563422 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request con 0x7f7e8c09e910 auth_method 0 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e8c131ab0 0x7f7e8c12f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f7e8c131ab0 auth_method 0 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 1 -- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 msgr2=0x7f7e8c1361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 1 -- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 msgr2=0x7f7e8c1361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 0x7f7e8c1361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8bfff640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 0x7f7e8c1361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e90cfa640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c12f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e90cfa640 10 monclient(hunting): get_auth_request con 0x7f7e8c136bb0 auth_method 0 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e90cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e90cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e90cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient(hunting): handle_auth_done global_id 64685 payload 824 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 1 -- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 msgr2=0x7f7e8c12f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c12f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 1 -- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 msgr2=0x7f7e8c1361a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 0x7f7e8c1361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e8b7fe640 1 -- 172.21.15.116:0/1249563422 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84009a20 con 0x7f7e8c131ab0 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.983+0000 7f7e90cfa640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c12f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e8b7fe640 1 -- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e8c131ab0 msgr2=0x7f7e8c12f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e8b7fe640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e8c131ab0 0x7f7e8c12f100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400c490 tx=0x7f7e74007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 1 -- 172.21.15.116:0/1249563422 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e740074f0 con 0x7f7e8c131ab0 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: dump: 2024-09-18T12:31:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: _finish_auth 0 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e9275d640 5 monclient: authenticate success, global_id 64685 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 1 -- 172.21.15.116:0/1249563422 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e74007670 con 0x7f7e8c131ab0 2024-09-18T12:31:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 1 -- 172.21.15.116:0/1249563422 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74005a20 con 0x7f7e8c131ab0 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e83fff640 10 monclient: dump: 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:41.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e9275d640 10 monclient: _renew_subs 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e8a7fc640 4 set_mon_vals no callback set 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e9275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c09be40 con 0x7f7e8c131ab0 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e9275d640 10 monclient: _renew_subs 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e9275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.986+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c130970 con 0x7f7e8c131ab0 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e83fff640 1 -- 172.21.15.116:0/1249563422 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7e7401b3b0 con 0x7f7e8c131ab0 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e9275d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e9275d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e9275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:41.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c130b60 con 0x7f7e8c131ab0 2024-09-18T12:31:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e83fff640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7e6c038490 0x7f7e6c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e83fff640 1 -- 172.21.15.116:0/1249563422 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(156..156 src has 13..156) v4 ==== 9003+0+0 (secure 0 0 0) 0x7f7e7404bd70 con 0x7f7e8c131ab0 2024-09-18T12:31:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e8bfff640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7e6c038490 0x7f7e6c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.987+0000 7f7e8bfff640 10 monclient: get_auth_request con 0x7f7e6c038490 auth_method 0 2024-09-18T12:31:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.990+0000 7f7e8bfff640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7e6c038490 0x7f7e6c03a940 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006f30 tx=0x7f7e7c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.991+0000 7f7e83fff640 1 -- 172.21.15.116:0/1249563422 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7e74018840 con 0x7f7e8c131ab0 2024-09-18T12:31:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.991+0000 7f7e83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:40.991+0000 7f7e83fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:41.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.204+0000 7f7e9275d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:41.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.204+0000 7f7e9275d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:41.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.204+0000 7f7e9275d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:41.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.204+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7e8c09db20 con 0x7f7e8c131ab0 2024-09-18T12:31:41.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.204+0000 7f7e83fff640 1 -- 172.21.15.116:0/1249563422 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f7e74017070 con 0x7f7e8c131ab0 2024-09-18T12:31:41.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.204+0000 7f7e83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:41.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.204+0000 7f7e83fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:41.207 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":68,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:41.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7e6c038490 msgr2=0x7f7e6c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7e6c038490 0x7f7e6c03a940 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006f30 tx=0x7f7e7c008040 comp rx=0 tx=0).stop 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 10 monclient: shutdown 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e8c131ab0 msgr2=0x7f7e8c12f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e8c131ab0 0x7f7e8c12f100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400c490 tx=0x7f7e74007e40 comp rx=0 tx=0).stop 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 shutdown_connections 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7e6c038490 0x7f7e6c03a940 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e8c136bb0 0x7f7e8c12f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e8c131ab0 0x7f7e8c12f100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 --2- 172.21.15.116:0/1249563422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e8c09e910 0x7f7e8c1361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 >> 172.21.15.116:0/1249563422 conn(0x7f7e8c067d30 msgr2=0x7f7e8c12cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:41.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 shutdown_connections 2024-09-18T12:31:41.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:41.206+0000 7f7e9275d640 1 -- 172.21.15.116:0/1249563422 wait complete. 2024-09-18T12:31:41.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.798+0000 7eff1b7fe640 1 -- 172.21.15.144:0/225944393 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v157) v1 ==== 119+0+0 (secure 0 0 0) 0x7eff3411dc50 con 0x7eff341321a0 2024-09-18T12:31:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.798+0000 7eff1b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T12:31:41.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.798+0000 7eff1b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:41.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.802+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff20038490 msgr2=0x7eff2003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:41.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.802+0000 7eff39e94640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff20038490 0x7eff2003a940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7eff24006fd0 tx=0x7eff24008040 comp rx=0 tx=0).stop 2024-09-18T12:31:41.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.802+0000 7eff39e94640 10 monclient: shutdown 2024-09-18T12:31:41.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.802+0000 7eff39e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:41.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.802+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff341321a0 msgr2=0x7eff34130f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:41.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.802+0000 7eff39e94640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff341321a0 0x7eff34130f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff3409f500 tx=0x7eff2c036000 comp rx=0 tx=0).stop 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 shutdown_connections 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff20038490 0x7eff2003a940 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff341321a0 0x7eff34130f30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff3409f020 0x7eff341309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 --2- 172.21.15.144:0/225944393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3409ec50 0x7eff3409a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 >> 172.21.15.144:0/225944393 conn(0x7eff340a5d40 msgr2=0x7eff340a41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 shutdown_connections 2024-09-18T12:31:41.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.803+0000 7eff39e94640 1 -- 172.21.15.144:0/225944393 wait complete. 2024-09-18T12:31:41.818 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-18T12:31:41.898 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.897+0000 7f965e0a8640 10 monclient: get_monmap_and_config 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.897+0000 7f965e0a8640 10 monclient: build_initial_monmap 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.897+0000 7f965e0a8640 10 monclient: monmap: 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:41.898660+0000 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:41.898660+0000 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:41.899 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:41.900 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:41.900 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:41.900 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.898+0000 7f965e0a8640 1 Processor -- start 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.898+0000 7f965e0a8640 1 -- start start 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.898+0000 7f965e0a8640 10 monclient: init 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.898+0000 7f965e0a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.898+0000 7f965e0a8640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.898+0000 7f965e0a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 0x7f96581321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.898+0000 7f965e0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f965809f320 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:41.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9658070920 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f9658138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9658071430 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9658096b60 con 0x7f9658070920 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965809e4d0 con 0x7f965809f320 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965809e7b0 con 0x7f9658071430 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f965e0a8640 10 monclient(hunting): authenticate will time out at 1137.468047s 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 0x7f96581321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f9658138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9656ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9658070920 0x7f9658070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:53250/0 (socket says 172.21.15.144:53250) 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9656ffd640 1 -- 172.21.15.144:0/3457265129 learned_addr learned my addr 172.21.15.144:0/3457265129 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:41.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 10 monclient(hunting): get_auth_request con 0x7f965809f320 auth_method 0 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): get_auth_request con 0x7f9658071430 auth_method 0 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 1 -- 172.21.15.144:0/3457265129 >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 msgr2=0x7f96581321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 1 -- 172.21.15.144:0/3457265129 >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 msgr2=0x7f96581321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:41.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 1 --2- 172.21.15.144:0/3457265129 >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 0x7f96581321b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f96577fe640 1 --2- 172.21.15.144:0/3457265129 >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 0x7f96581321b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient(hunting): handle_auth_done global_id 64688 payload 274 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 1 -- 172.21.15.144:0/3457265129 >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 msgr2=0x7f96581321b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 1 --2- 172.21.15.144:0/3457265129 >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 0x7f96581321b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 1 -- 172.21.15.144:0/3457265129 >> v1:172.21.15.116:6789/0 conn(0x7f9658070920 legacy=0x7f9658070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 1 monclient: found mon.noname-b 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:41.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.899+0000 7f9657fff640 1 -- 172.21.15.144:0/3457265129 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965809e1f0 con 0x7f9658071430 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f9657fff640 1 -- 172.21.15.144:0/3457265129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 msgr2=0x7f9658138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f9657fff640 1 --2- 172.21.15.144:0/3457265129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f9658138fb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9650009e60 tx=0x7f9650009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c706edbc148128 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 1 -- 172.21.15.144:0/3457265129 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650007760 con 0x7f9658071430 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: dump: 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:41.912 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 1 -- 172.21.15.144:0/3457265129 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965002d050 con 0x7f9658071430 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f965e0a8640 5 monclient: authenticate success, global_id 64688 2024-09-18T12:31:41.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f965e0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f965e0a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f965e0a8640 4 set_mon_vals no callback set 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 1 -- 172.21.15.144:0/3457265129 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650007b30 con 0x7f9658071430 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.902+0000 7f96567fc640 10 monclient: dump: 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:41.914 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 10 monclient: shutdown 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 -- 172.21.15.144:0/3457265129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 msgr2=0x7f9658138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 --2- 172.21.15.144:0/3457265129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f9658138fb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9650009e60 tx=0x7f9650009780 comp rx=0 tx=0).stop 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 -- 172.21.15.144:0/3457265129 shutdown_connections 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 --2- 172.21.15.144:0/3457265129 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f9658138fb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 --2- 172.21.15.144:0/3457265129 >> v2:172.21.15.116:3300/0 conn(0x7f965809f320 0x7f96581321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 -- 172.21.15.144:0/3457265129 >> 172.21.15.144:0/3457265129 conn(0x7f96580a2b70 msgr2=0x7f965809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:41.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 -- 172.21.15.144:0/3457265129 shutdown_connections 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 1 -- 172.21.15.144:0/3457265129 wait complete. 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 10 monclient: build_initial_monmap 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.903+0000 7f965e0a8640 10 monclient: monmap: 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:41.916 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 Processor -- start 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 -- start start 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient: init 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 0x7f9658137590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9658070920 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f965812edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9658071430 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 0x7f965812f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): picked mon.noname-a con 0x7f965809f320 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:41.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96581334d0 con 0x7f965809f320 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9658130e00 con 0x7f9658070920 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965809d5b0 con 0x7f9658071430 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f965e0a8640 10 monclient(hunting): authenticate will time out at 1137.473567s 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 0x7f9658137590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 0x7f9658137590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53544/0 (socket says 172.21.15.144:53544) 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 1 -- 172.21.15.144:0/810929088 learned_addr learned my addr 172.21.15.144:0/810929088 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): get_auth_request con 0x7f9658070920 auth_method 0 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:41.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9656ffd640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f965812edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 0x7f965812f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 10 monclient(hunting): get_auth_request con 0x7f965809f320 auth_method 0 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9656ffd640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 msgr2=0x7f965812edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:41.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9656ffd640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 msgr2=0x7f965812edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9656ffd640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f965812edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9656ffd640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f965812edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 msgr2=0x7f965812f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 msgr2=0x7f965812f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 0x7f965812f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 10 monclient(hunting): handle_auth_done global_id 5872 payload 824 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f9657fff640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 0x7f965812f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 msgr2=0x7f965812edd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:41.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f965812edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 msgr2=0x7f965812f310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.904+0000 7f96577fe640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 0x7f965812f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.905+0000 7f96577fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.905+0000 7f96577fe640 1 monclient: found mon.noname-b 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.905+0000 7f96577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.905+0000 7f96577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.905+0000 7f96577fe640 1 -- 172.21.15.144:0/810929088 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650009a20 con 0x7f9658070920 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f96577fe640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 msgr2=0x7f9658137590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f96577fe640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 0x7f9658137590 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f964800d890 tx=0x7f9648008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964800de20 con 0x7f9658070920 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:41.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: dump: 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:41.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f965e0a8640 5 monclient: authenticate success, global_id 5872 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f965e0a8640 10 monclient: _renew_subs 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964800bae0 con 0x7f9658070920 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f965e0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9658139140 con 0x7f9658070920 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964800bea0 con 0x7f9658070920 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f963f7fe640 10 monclient: dump: 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:41.923 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f965e0a8640 10 monclient: _renew_subs 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f965e0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9658138760 con 0x7f9658070920 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.907+0000 7f9655ffb640 4 set_mon_vals no callback set 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9648014640 con 0x7f9658070920 2024-09-18T12:31:41.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f965e0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f965e0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f965e0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96580008d0 con 0x7f9658070920 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f963f7fe640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96440384c0 0x7f964403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(157..157 src has 13..157) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f964804cff0 con 0x7f9658070920 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f9656ffd640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96440384c0 0x7f964403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.908+0000 7f9656ffd640 10 monclient: get_auth_request con 0x7f96440384c0 auth_method 0 2024-09-18T12:31:41.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.910+0000 7f9656ffd640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96440384c0 0x7f964403a970 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f965806ec70 tx=0x7f964c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.912+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f96480148f0 con 0x7f9658070920 2024-09-18T12:31:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.912+0000 7f963f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:41.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:41.912+0000 7f963f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.073+0000 7f965e0a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T12:31:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.073+0000 7f965e0a8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T12:31:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.073+0000 7f965e0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.073+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f9658133710 con 0x7f9658070920 2024-09-18T12:31:42.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.798+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v158) v1 ==== 128+0+0 (secure 0 0 0) 0x7f964801dbb0 con 0x7f9658070920 2024-09-18T12:31:42.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.798+0000 7f963f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T12:31:42.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.798+0000 7f963f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:42.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.903+0000 7f965e0a8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T12:31:42.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.903+0000 7f965e0a8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T12:31:42.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.903+0000 7f965e0a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:42.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:42.903+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7f96581326f0 con 0x7f9658070920 2024-09-18T12:31:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.798+0000 7f963f7fe640 1 -- 172.21.15.144:0/810929088 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v159) v1 ==== 128+0+0 (secure 0 0 0) 0x7f964802c070 con 0x7f9658070920 2024-09-18T12:31:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.798+0000 7f963f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T12:31:43.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.798+0000 7f963f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96440384c0 msgr2=0x7f964403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96440384c0 0x7f964403a970 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f965806ec70 tx=0x7f964c009040 comp rx=0 tx=0).stop 2024-09-18T12:31:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 10 monclient: shutdown 2024-09-18T12:31:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:43.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 msgr2=0x7f9658137590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 0x7f9658137590 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f964800d890 tx=0x7f9648008040 comp rx=0 tx=0).stop 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 shutdown_connections 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f96440384c0 0x7f964403a970 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f965809f320 0x7f965812f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9658071430 0x7f965812edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 --2- 172.21.15.144:0/810929088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9658070920 0x7f9658137590 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 >> 172.21.15.144:0/810929088 conn(0x7f96580a2b70 msgr2=0x7f96580a02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 shutdown_connections 2024-09-18T12:31:43.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.801+0000 7f965e0a8640 1 -- 172.21.15.144:0/810929088 wait complete. 2024-09-18T12:31:43.813 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-18T12:31:43.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.894+0000 7fccb7e94640 10 monclient: get_monmap_and_config 2024-09-18T12:31:43.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.894+0000 7fccb7e94640 10 monclient: build_initial_monmap 2024-09-18T12:31:43.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.894+0000 7fccb7e94640 10 monclient: monmap: 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:43.895488+0000 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:43.895488+0000 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:43.896 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:43.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 1 Processor -- start 2024-09-18T12:31:43.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 1 -- start start 2024-09-18T12:31:43.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient: init 2024-09-18T12:31:43.899 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb0096a30 0x7fccb0096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient(hunting): picked mon.noname-b con 0x7fccb0096a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccb0099f00 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fccb009d4e0 0x7fccb0097340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient(hunting): picked mon.noname-c con 0x7fccb009d4e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb0097880 con 0x7fccb0099f00 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb009cb80 con 0x7fccb009d4e0 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0098c50 con 0x7fccb0096a30 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:43.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb7e94640 10 monclient(hunting): authenticate will time out at 1139.464838s 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb6431640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fccb009d4e0 0x7fccb0097340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.895+0000 7fccb6431640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fccb009d4e0 0x7fccb0097340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53546/0 (socket says 172.21.15.144:53546) 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 1 -- 172.21.15.144:0/1827266239 learned_addr learned my addr 172.21.15.144:0/1827266239 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 10 monclient(hunting): get_auth_request con 0x7fccb009d4e0 auth_method 0 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb6431640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 1 --2- 172.21.15.144:0/1827266239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb0096a30 0x7fccb0096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): get_auth_request con 0x7fccb0096a30 auth_method 0 2024-09-18T12:31:43.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient(hunting): handle_auth_done global_id 64691 payload 274 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 1 -- 172.21.15.144:0/1827266239 >> v2:172.21.15.144:3300/0 conn(0x7fccb009d4e0 msgr2=0x7fccb0097340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 1 --2- 172.21.15.144:0/1827266239 >> v2:172.21.15.144:3300/0 conn(0x7fccb009d4e0 0x7fccb0097340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 1 -- 172.21.15.144:0/1827266239 >> v1:172.21.15.144:6789/0 conn(0x7fccb0099f00 legacy=0x7fccb012e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:43.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient: _finish_hunting 0 2024-09-18T12:31:43.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 1 monclient: found mon.noname-b 2024-09-18T12:31:43.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:43.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:43.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.896+0000 7fccb5c30640 1 -- 172.21.15.144:0/1827266239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0098f30 con 0x7fccb0096a30 2024-09-18T12:31:43.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb5c30640 1 -- 172.21.15.144:0/1827266239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb0096a30 msgr2=0x7fccb0096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:43.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb5c30640 1 --2- 172.21.15.144:0/1827266239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb0096a30 0x7fccb0096e00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fccac009850 tx=0x7fccac0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e4bf651010303eb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:43.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 1 -- 172.21.15.144:0/1827266239 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac02add0 con 0x7fccb0096a30 2024-09-18T12:31:43.908 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: dump: 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: _finish_auth 0 2024-09-18T12:31:43.909 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb7e94640 5 monclient: authenticate success, global_id 64691 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb7e94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 1 -- 172.21.15.144:0/1827266239 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac032440 con 0x7fccb0096a30 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb7e94640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 1 -- 172.21.15.144:0/1827266239 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac0327e0 con 0x7fccb0096a30 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb7e94640 4 set_mon_vals no callback set 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.899+0000 7fccb4c2e640 10 monclient: dump: 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:43.910 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 10 monclient: shutdown 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 -- 172.21.15.144:0/1827266239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb0096a30 msgr2=0x7fccb0096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 --2- 172.21.15.144:0/1827266239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb0096a30 0x7fccb0096e00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fccac009850 tx=0x7fccac0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 -- 172.21.15.144:0/1827266239 shutdown_connections 2024-09-18T12:31:43.911 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 --2- 172.21.15.144:0/1827266239 >> v2:172.21.15.144:3300/0 conn(0x7fccb009d4e0 0x7fccb0097340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 --2- 172.21.15.144:0/1827266239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb0096a30 0x7fccb0096e00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 -- 172.21.15.144:0/1827266239 >> 172.21.15.144:0/1827266239 conn(0x7fccb00711a0 msgr2=0x7fccb00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 -- 172.21.15.144:0/1827266239 shutdown_connections 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 1 -- 172.21.15.144:0/1827266239 wait complete. 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 10 monclient: build_initial_monmap 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.900+0000 7fccb7e94640 10 monclient: monmap: 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:43.912 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 Processor -- start 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 -- start start 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient: init 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 0x7fccb012d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): picked mon.noname-b con 0x7fccb0096a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb0099f00 0x7fccb011a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): picked mon.noname-a con 0x7fccb0099f00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb009d4e0 0x7fccb011a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): picked mon.noname-c con 0x7fccb009d4e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:43.913 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb011ddd0 con 0x7fccb0099f00 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0098460 con 0x7fccb0096a30 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb009c390 con 0x7fccb009d4e0 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb7e94640 10 monclient(hunting): authenticate will time out at 1139.470796s 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 0x7fccb012d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 0x7fccb012d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53562/0 (socket says 172.21.15.144:53562) 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 1 -- 172.21.15.144:0/929500516 learned_addr learned my addr 172.21.15.144:0/929500516 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 10 monclient(hunting): get_auth_request con 0x7fccb0096a30 auth_method 0 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:43.914 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb5c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.901+0000 7fccb6431640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb009d4e0 0x7fccb011a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb6431640 10 monclient(hunting): get_auth_request con 0x7fccb009d4e0 auth_method 0 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb6431640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb6431640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb6431640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb542f640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb0099f00 0x7fccb011a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb542f640 10 monclient(hunting): get_auth_request con 0x7fccb0099f00 auth_method 0 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb542f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:43.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb542f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb542f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 10 monclient(hunting): handle_auth_done global_id 5878 payload 824 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 1 -- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb009d4e0 msgr2=0x7fccb011a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb009d4e0 0x7fccb011a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 1 -- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb0099f00 msgr2=0x7fccb011a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb0099f00 0x7fccb011a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 10 monclient: _finish_hunting 0 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 1 monclient: found mon.noname-b 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb5c30640 1 -- 172.21.15.144:0/929500516 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac009390 con 0x7fccb0096a30 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.902+0000 7fccb6431640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb009d4e0 0x7fccb011a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:43.916 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb5c30640 1 -- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 msgr2=0x7fccb012d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb5c30640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 0x7fccb012d630 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fccac009360 tx=0x7fccac032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac031c70 con 0x7fccb0096a30 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: dump: 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:43.917 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:43.918 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:31:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:43.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb7e94640 5 monclient: authenticate success, global_id 5878 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac031dd0 con 0x7fccb0096a30 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb7e94640 10 monclient: _renew_subs 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb7e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb011bc50 con 0x7fccb0096a30 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac03d670 con 0x7fccb0096a30 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca5ffb640 10 monclient: dump: 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.919 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb7e94640 10 monclient: _renew_subs 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb7e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fcca7fff640 4 set_mon_vals no callback set 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.904+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb012f3b0 con 0x7fccb0096a30 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fccb7e94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fccb7e94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:43.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fccb7e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:43.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fccac039040 con 0x7fccb0096a30 2024-09-18T12:31:43.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb011d8a0 con 0x7fccb0096a30 2024-09-18T12:31:43.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fcca5ffb640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc94038490 0x7fcc9403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:43.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(159..159 src has 13..159) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fccac06c810 con 0x7fccb0096a30 2024-09-18T12:31:43.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fccb542f640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc94038490 0x7fcc9403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:43.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.905+0000 7fccb542f640 10 monclient: get_auth_request con 0x7fcc94038490 auth_method 0 2024-09-18T12:31:43.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.908+0000 7fccb542f640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc94038490 0x7fcc9403a940 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fccb011d6b0 tx=0x7fcca0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:43.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.910+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fccac02f2a0 con 0x7fccb0096a30 2024-09-18T12:31:43.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.910+0000 7fcca5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:43.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:43.910+0000 7fcca5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:44.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.071+0000 7fccb7e94640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T12:31:44.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.071+0000 7fccb7e94640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T12:31:44.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.071+0000 7fccb7e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:44.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.071+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7fccb011db80 con 0x7fccb0096a30 2024-09-18T12:31:44.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.323+0000 7f6719e4f640 10 monclient: get_monmap_and_config 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.323+0000 7f6719e4f640 10 monclient: build_initial_monmap 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.323+0000 7f6719e4f640 10 monclient: monmap: 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:44.323914+0000 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:44.323914+0000 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:44.324 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:44.325 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:44.325 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:44.325 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:44.325 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:44.325 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:44.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 1 Processor -- start 2024-09-18T12:31:44.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 1 -- start start 2024-09-18T12:31:44.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient: init 2024-09-18T12:31:44.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:44.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 0x7f6714131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient(hunting): picked mon.noname-a con 0x7f6714131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f671409e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6714136bb0 0x7f6714138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient(hunting): picked mon.noname-c con 0x7f6714136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671409faf0 con 0x7f671409e910 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671409fd30 con 0x7f6714131ab0 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:44.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67140a1110 con 0x7f6714136bb0 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.324+0000 7f6719e4f640 10 monclient(hunting): authenticate will time out at 1131.645902s 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 0x7f6714131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 0x7f6714131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50196/0 (socket says 172.21.15.116:50196) 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 1 -- 172.21.15.116:0/3533131912 learned_addr learned my addr 172.21.15.116:0/3533131912 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 10 monclient(hunting): get_auth_request con 0x7f6714131ab0 auth_method 0 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:44.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 1 -- 172.21.15.116:0/3533131912 >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 msgr2=0x7f6714131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 1 -- 172.21.15.116:0/3533131912 >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 msgr2=0x7f6714131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 1 --2- 172.21.15.116:0/3533131912 >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 0x7f6714131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f67137fe640 1 --2- 172.21.15.116:0/3533131912 >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 0x7f6714131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 1 --2- 172.21.15.116:0/3533131912 >> v2:172.21.15.144:3300/0 conn(0x7f6714136bb0 0x7f6714138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): get_auth_request con 0x7f6714136bb0 auth_method 0 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:44.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient(hunting): handle_auth_done global_id 5881 payload 274 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 1 -- 172.21.15.116:0/3533131912 >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 msgr2=0x7f6714131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 1 --2- 172.21.15.116:0/3533131912 >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 0x7f6714131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 1 -- 172.21.15.116:0/3533131912 >> v1:172.21.15.144:6789/0 conn(0x7f671409e910 legacy=0x7f671409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 1 monclient: found mon.noname-c 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:44.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.325+0000 7f6713fff640 1 -- 172.21.15.116:0/3533131912 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67140a13f0 con 0x7f6714136bb0 2024-09-18T12:31:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6713fff640 1 -- 172.21.15.116:0/3533131912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714136bb0 msgr2=0x7f6714138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6713fff640 1 --2- 172.21.15.116:0/3533131912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714136bb0 0x7f6714138f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f670c009e60 tx=0x7f670c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8fa672edb65996 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 1 -- 172.21.15.116:0/3533131912 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c007760 con 0x7f6714136bb0 2024-09-18T12:31:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: dump: 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:44.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 1 -- 172.21.15.116:0/3533131912 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670c02d050 con 0x7f6714136bb0 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 1 -- 172.21.15.116:0/3533131912 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c007b30 con 0x7f6714136bb0 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6719e4f640 5 monclient: authenticate success, global_id 5881 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6719e4f640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6719e4f640 4 set_mon_vals no callback set 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.328+0000 7f6712ffd640 10 monclient: dump: 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:44.340 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 10 monclient: shutdown 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 -- 172.21.15.116:0/3533131912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714136bb0 msgr2=0x7f6714138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 --2- 172.21.15.116:0/3533131912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714136bb0 0x7f6714138f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f670c009e60 tx=0x7f670c009780 comp rx=0 tx=0).stop 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 -- 172.21.15.116:0/3533131912 shutdown_connections 2024-09-18T12:31:44.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 --2- 172.21.15.116:0/3533131912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714136bb0 0x7f6714138f90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 --2- 172.21.15.116:0/3533131912 >> v2:172.21.15.116:3300/0 conn(0x7f6714131ab0 0x7f6714131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 -- 172.21.15.116:0/3533131912 >> 172.21.15.116:0/3533131912 conn(0x7f6714067d30 msgr2=0x7f67140a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 -- 172.21.15.116:0/3533131912 shutdown_connections 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.329+0000 7f6719e4f640 1 -- 172.21.15.116:0/3533131912 wait complete. 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient: build_initial_monmap 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient: monmap: 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:44.342 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 Processor -- start 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 -- start start 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient: init 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 0x7f67141361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): picked mon.noname-c con 0x7f671409e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131ab0 0x7f671412f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): picked mon.noname-b con 0x7f6714131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 0x7f671412f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): picked mon.noname-a con 0x7f6714136bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67141331e0 con 0x7f6714136bb0 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6714133830 con 0x7f6714131ab0 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67141305e0 con 0x7f671409e910 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6719e4f640 10 monclient(hunting): authenticate will time out at 1131.651856s 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f6713fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 0x7f671412f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f67137fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 0x7f67141361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f67137fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 0x7f67141361a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39316/0 (socket says 172.21.15.116:39316) 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f67137fe640 1 -- 172.21.15.116:0/1362854614 learned_addr learned my addr 172.21.15.116:0/1362854614 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.330+0000 7f67137fe640 10 monclient(hunting): get_auth_request con 0x7f671409e910 auth_method 0 2024-09-18T12:31:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 10 monclient(hunting): get_auth_request con 0x7f6714136bb0 auth_method 0 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 1 -- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 msgr2=0x7f671412f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 1 -- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 msgr2=0x7f671412f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 0x7f671412f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131ab0 0x7f671412f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 10 monclient(hunting): get_auth_request con 0x7f6714131ab0 auth_method 0 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f6713fff640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 0x7f671412f640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f670bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient(hunting): handle_auth_done global_id 64697 payload 824 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 1 -- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131ab0 msgr2=0x7f671412f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131ab0 0x7f671412f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 1 -- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 msgr2=0x7f671412f640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 0x7f671412f640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 1 monclient: found mon.noname-c 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.331+0000 7f67137fe640 1 -- 172.21.15.116:0/1362854614 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c009a20 con 0x7f671409e910 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f67137fe640 1 -- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 msgr2=0x7f67141361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f67137fe640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 0x7f67141361a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f670400e870 tx=0x7f6704008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 1 -- 172.21.15.116:0/1362854614 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400c6d0 con 0x7f671409e910 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: dump: 2024-09-18T12:31:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 1 -- 172.21.15.116:0/1362854614 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400c830 con 0x7f671409e910 2024-09-18T12:31:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 1 -- 172.21.15.116:0/1362854614 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400cbf0 con 0x7f671409e910 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f670b7fe640 10 monclient: dump: 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:44.349 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f6719e4f640 5 monclient: authenticate success, global_id 64697 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f67127fc640 4 set_mon_vals no callback set 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f6719e4f640 10 monclient: _renew_subs 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f6719e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671409be40 con 0x7f671409e910 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f6719e4f640 10 monclient: _renew_subs 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f6719e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.333+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6714130970 con 0x7f671409e910 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.334+0000 7f670b7fe640 1 -- 172.21.15.116:0/1362854614 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6704013410 con 0x7f671409e910 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.334+0000 7f6719e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.334+0000 7f6719e4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.334+0000 7f6719e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.334+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671409db20 con 0x7f671409e910 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.334+0000 7f670b7fe640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66f4038490 0x7f66f403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.335+0000 7f670b7fe640 1 -- 172.21.15.116:0/1362854614 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(159..159 src has 13..159) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f670404c790 con 0x7f671409e910 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.335+0000 7f670bfff640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66f4038490 0x7f66f403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.335+0000 7f670bfff640 10 monclient: get_auth_request con 0x7f66f4038490 auth_method 0 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.337+0000 7f670bfff640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66f4038490 0x7f66f403a940 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f66fc007fd0 tx=0x7f66fc007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.339+0000 7f670b7fe640 1 -- 172.21.15.116:0/1362854614 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6704023300 con 0x7f671409e910 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.339+0000 7f670b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.339+0000 7f670b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.553+0000 7f6719e4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.553+0000 7f6719e4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.553+0000 7f6719e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.553+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f671404bf60 con 0x7f671409e910 2024-09-18T12:31:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.554+0000 7f670b7fe640 1 -- 172.21.15.116:0/1362854614 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f6704024350 con 0x7f671409e910 2024-09-18T12:31:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.554+0000 7f670b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.554+0000 7f670b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:44.555 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":68,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66f4038490 msgr2=0x7f66f403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66f4038490 0x7f66f403a940 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f66fc007fd0 tx=0x7f66fc007e10 comp rx=0 tx=0).stop 2024-09-18T12:31:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 10 monclient: shutdown 2024-09-18T12:31:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 msgr2=0x7f67141361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 0x7f67141361a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f670400e870 tx=0x7f6704008040 comp rx=0 tx=0).stop 2024-09-18T12:31:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 shutdown_connections 2024-09-18T12:31:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66f4038490 0x7f66f403a940 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6714136bb0 0x7f671412f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6714131ab0 0x7f671412f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 --2- 172.21.15.116:0/1362854614 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f671409e910 0x7f67141361a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 >> 172.21.15.116:0/1362854614 conn(0x7f6714067d30 msgr2=0x7f671412cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 shutdown_connections 2024-09-18T12:31:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:44.556+0000 7f6719e4f640 1 -- 172.21.15.116:0/1362854614 wait complete. 2024-09-18T12:31:44.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.823+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v160) v1 ==== 134+0+0 (secure 0 0 0) 0x7fccac037020 con 0x7fccb0096a30 2024-09-18T12:31:44.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.823+0000 7fcca5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T12:31:44.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.823+0000 7fcca5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:44.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.926+0000 7fccb7e94640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T12:31:44.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.926+0000 7fccb7e94640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T12:31:44.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.926+0000 7fccb7e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:44.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:44.926+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7fccb011db80 con 0x7fccb0096a30 2024-09-18T12:31:45.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.828+0000 7fcca5ffb640 1 -- 172.21.15.144:0/929500516 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v161) v1 ==== 134+0+0 (secure 0 0 0) 0x7fccac037070 con 0x7fccb0096a30 2024-09-18T12:31:45.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.828+0000 7fcca5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T12:31:45.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.828+0000 7fcca5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:45.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc94038490 msgr2=0x7fcc9403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:45.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc94038490 0x7fcc9403a940 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fccb011d6b0 tx=0x7fcca0009040 comp rx=0 tx=0).stop 2024-09-18T12:31:45.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 10 monclient: shutdown 2024-09-18T12:31:45.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 msgr2=0x7fccb012d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 0x7fccb012d630 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fccac009360 tx=0x7fccac032000 comp rx=0 tx=0).stop 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 shutdown_connections 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc94038490 0x7fcc9403a940 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb009d4e0 0x7fccb011a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb0099f00 0x7fccb011a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 --2- 172.21.15.144:0/929500516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb0096a30 0x7fccb012d630 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 >> 172.21.15.144:0/929500516 conn(0x7fccb00711a0 msgr2=0x7fccb009d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 shutdown_connections 2024-09-18T12:31:45.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.831+0000 7fccb7e94640 1 -- 172.21.15.144:0/929500516 wait complete. 2024-09-18T12:31:45.843 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-18T12:31:45.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.920+0000 7fe836f5d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:45.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.920+0000 7fe836f5d640 10 monclient: build_initial_monmap 2024-09-18T12:31:45.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.920+0000 7fe836f5d640 10 monclient: monmap: 2024-09-18T12:31:45.921 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:45.921 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:45.921 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:45.920967+0000 2024-09-18T12:31:45.921 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:45.920967+0000 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:45.922 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.920+0000 7fe836f5d640 1 Processor -- start 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.920+0000 7fe836f5d640 1 -- start start 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.920+0000 7fe836f5d640 10 monclient: init 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe83009ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe83009f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe830069d10 0x7fe83013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe830069d10 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:45.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83006a120 con 0x7fe83009ec50 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe830067420 con 0x7fe83009f020 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe83009fdc0 con 0x7fe830069d10 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe836f5d640 10 monclient(hunting): authenticate will time out at 1141.490354s 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe82ffff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe83009f020 0x7fe830136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:37446/0 (socket says 172.21.15.144:37446) 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe82ffff640 1 -- 172.21.15.144:0/3305030048 learned_addr learned my addr 172.21.15.144:0/3305030048 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe8354fa640 1 --2- 172.21.15.144:0/3305030048 >> v2:172.21.15.144:3300/0 conn(0x7fe830069d10 0x7fe83013a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe8354fa640 10 monclient(hunting): get_auth_request con 0x7fe830069d10 auth_method 0 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe8354fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:45.926 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe8354fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.921+0000 7fe8354fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe82ffff640 1 -- 172.21.15.144:0/3305030048 >> v1:172.21.15.144:6789/0 conn(0x7fe83009f020 legacy=0x7fe830136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 10 monclient(hunting): handle_auth_done global_id 5887 payload 274 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 1 -- 172.21.15.144:0/3305030048 >> v1:172.21.15.144:6789/0 conn(0x7fe83009f020 legacy=0x7fe830136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:45.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 1 -- 172.21.15.144:0/3305030048 >> v1:172.21.15.116:6789/0 conn(0x7fe83009ec50 legacy=0x7fe830131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:45.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:45.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 1 monclient: found mon.noname-c 2024-09-18T12:31:45.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:45.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:45.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.923+0000 7fe8354fa640 1 -- 172.21.15.144:0/3305030048 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8300a0510 con 0x7fe830069d10 2024-09-18T12:31:45.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe8354fa640 1 -- 172.21.15.144:0/3305030048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe830069d10 msgr2=0x7fe83013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:45.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe8354fa640 1 --2- 172.21.15.144:0/3305030048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe830069d10 0x7fe83013a490 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe8180098d0 tx=0x7fe8180090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15019649625d64f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:45.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 1 -- 172.21.15.144:0/3305030048 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe818032440 con 0x7fe830069d10 2024-09-18T12:31:45.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: dump: 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:45.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 1 -- 172.21.15.144:0/3305030048 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8180325a0 con 0x7fe830069d10 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe836f5d640 5 monclient: authenticate success, global_id 5887 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe836f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe836f5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe836f5d640 4 set_mon_vals no callback set 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 1 -- 172.21.15.144:0/3305030048 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe818032940 con 0x7fe830069d10 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.926+0000 7fe82f7fe640 10 monclient: dump: 2024-09-18T12:31:45.936 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 10 monclient: shutdown 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:45.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 1 -- 172.21.15.144:0/3305030048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe830069d10 msgr2=0x7fe83013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 1 --2- 172.21.15.144:0/3305030048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe830069d10 0x7fe83013a490 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe8180098d0 tx=0x7fe8180090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 1 -- 172.21.15.144:0/3305030048 shutdown_connections 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 1 --2- 172.21.15.144:0/3305030048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe830069d10 0x7fe83013a490 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 1 -- 172.21.15.144:0/3305030048 >> 172.21.15.144:0/3305030048 conn(0x7fe8300a5d40 msgr2=0x7fe8300a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 1 -- 172.21.15.144:0/3305030048 shutdown_connections 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 1 -- 172.21.15.144:0/3305030048 wait complete. 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 10 monclient: build_initial_monmap 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.927+0000 7fe836f5d640 10 monclient: monmap: 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:45.938 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 Processor -- start 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 -- start start 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient: init 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe830069d10 0x7fe83009acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe830069d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:45.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 0x7fe830130d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe83009ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe83009f020 0x7fe83011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe83009f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83009c000 con 0x7fe83009f020 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300a4190 con 0x7fe83009ec50 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83011c900 con 0x7fe830069d10 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.928+0000 7fe836f5d640 10 monclient(hunting): authenticate will time out at 1141.497993s 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 0x7fe830130d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:45.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 0x7fe830130d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:53582/0 (socket says 172.21.15.144:53582) 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 -- 172.21.15.144:0/1513614396 learned_addr learned my addr 172.21.15.144:0/1513614396 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): get_auth_request con 0x7fe83009ec50 auth_method 0 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe834cf9640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe830069d10 0x7fe83009acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient(hunting): handle_auth_done global_id 5893 payload 824 2024-09-18T12:31:45.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe8354fa640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe83009f020 0x7fe83011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 -- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe830069d10 msgr2=0x7fe83009acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe830069d10 0x7fe83009acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 -- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe83009f020 msgr2=0x7fe83011b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe83009f020 0x7fe83011b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 monclient: found mon.noname-b 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe82ffff640 1 -- 172.21.15.144:0/1513614396 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe818009390 con 0x7fe83009ec50 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe834cf9640 10 monclient: get_auth_request con 0x7fe830069d10 auth_method 0 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.929+0000 7fe834cf9640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe830069d10 0x7fe83009acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82ffff640 1 -- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 msgr2=0x7fe830130d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:45.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82ffff640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 0x7fe830130d90 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe82000c590 tx=0x7fe820007dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820005d90 con 0x7fe83009ec50 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: dump: 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:45.943 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: _finish_auth 0 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8200070f0 con 0x7fe83009ec50 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe836f5d640 5 monclient: authenticate success, global_id 5893 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200074b0 con 0x7fe83009ec50 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe836f5d640 10 monclient: _renew_subs 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe836f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83011cfc0 con 0x7fe83009ec50 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:45.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82cff9640 10 monclient: dump: 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe82effd640 4 set_mon_vals no callback set 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe836f5d640 10 monclient: _renew_subs 2024-09-18T12:31:45.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe836f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.931+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe830141f20 con 0x7fe83009ec50 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.932+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe82001b3b0 con 0x7fe83009ec50 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.932+0000 7fe836f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.932+0000 7fe836f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.932+0000 7fe836f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.932+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8300008d0 con 0x7fe83009ec50 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.933+0000 7fe82cff9640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe810038490 0x7fe81003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.933+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(161..161 src has 13..161) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fe82004bdf0 con 0x7fe83009ec50 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.933+0000 7fe834cf9640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe810038490 0x7fe81003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:45.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.933+0000 7fe834cf9640 10 monclient: get_auth_request con 0x7fe810038490 auth_method 0 2024-09-18T12:31:45.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.935+0000 7fe834cf9640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe810038490 0x7fe81003a940 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe828007fd0 tx=0x7fe828009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:45.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.937+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe82004f030 con 0x7fe83009ec50 2024-09-18T12:31:45.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.937+0000 7fe82cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:45.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:45.937+0000 7fe82cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.094+0000 7fe836f5d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T12:31:46.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.094+0000 7fe836f5d640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T12:31:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.094+0000 7fe836f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:46.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.094+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fe8300a30b0 con 0x7fe83009ec50 2024-09-18T12:31:46.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.835+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v162) v1 ==== 137+0+0 (secure 0 0 0) 0x7fe820018650 con 0x7fe83009ec50 2024-09-18T12:31:46.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.836+0000 7fe82cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T12:31:46.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.836+0000 7fe82cff9640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-18T12:31:46.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.938+0000 7fe836f5d640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T12:31:46.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.938+0000 7fe836f5d640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T12:31:46.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.938+0000 7fe836f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:46.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:46.938+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fe830072640 con 0x7fe83009ec50 2024-09-18T12:31:47.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.671+0000 7f60a18c9640 10 monclient: get_monmap_and_config 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.671+0000 7f60a18c9640 10 monclient: build_initial_monmap 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.671+0000 7f60a18c9640 10 monclient: monmap: 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:47.671846+0000 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:47.671846+0000 2024-09-18T12:31:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:47.673 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:47.673 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:47.673 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:47.673 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:47.673 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:47.673 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:47.673 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.671+0000 7f60a18c9640 1 Processor -- start 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.671+0000 7f60a18c9640 1 -- start start 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.671+0000 7f60a18c9640 10 monclient: init 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f609c1318f0 0x7f609c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient(hunting): picked mon.noname-a con 0x7f609c1318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c09ea50 0x7f609c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient(hunting): picked mon.noname-b con 0x7f609c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f609c068d90 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609c131cc0 con 0x7f609c068d90 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609c09fbf0 con 0x7f609c1318f0 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c09fe30 con 0x7f609c09ea50 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f60a18c9640 10 monclient(hunting): authenticate will time out at 1134.993608s 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c09ea50 0x7f609c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f609c1318f0 0x7f609c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f609c068d90 0x7f609c0691f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54670/0 (socket says 172.21.15.116:54670) 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609b7fe640 1 -- 172.21.15.116:0/2401592388 learned_addr learned my addr 172.21.15.116:0/2401592388 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 10 monclient(hunting): get_auth_request con 0x7f609c09ea50 auth_method 0 2024-09-18T12:31:47.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 10 monclient(hunting): get_auth_request con 0x7f609c1318f0 auth_method 0 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f609affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.672+0000 7f6093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 10 monclient(hunting): handle_auth_done global_id 94098 payload 274 2024-09-18T12:31:47.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 1 -- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c09ea50 msgr2=0x7f609c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 1 --2- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c09ea50 0x7f609c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 1 -- 172.21.15.116:0/2401592388 >> v1:172.21.15.116:6789/0 conn(0x7f609c068d90 legacy=0x7f609c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 1 monclient: found mon.noname-a 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f609affd640 1 -- 172.21.15.116:0/2401592388 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c0a1210 con 0x7f609c1318f0 2024-09-18T12:31:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.673+0000 7f6093fff640 1 --2- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c09ea50 0x7f609c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:47.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.675+0000 7f609affd640 1 -- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 msgr2=0x7f609c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:47.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.675+0000 7f609affd640 1 --2- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c131f30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f608c009850 tx=0x7f608c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4671825672af194 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:47.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 1 -- 172.21.15.116:0/2401592388 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c02add0 con 0x7f609c1318f0 2024-09-18T12:31:47.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:47.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: dump: 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 1 -- 172.21.15.116:0/2401592388 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c032440 con 0x7f609c1318f0 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 5 monclient: authenticate success, global_id 94098 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 1 -- 172.21.15.116:0/2401592388 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c032820 con 0x7f609c1318f0 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 4 set_mon_vals no callback set 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f609a7fc640 10 monclient: dump: 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.687 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 10 monclient: shutdown 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 -- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 msgr2=0x7f609c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 --2- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c131f30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f608c009850 tx=0x7f608c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 -- 172.21.15.116:0/2401592388 shutdown_connections 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 --2- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c09ea50 0x7f609c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 --2- 172.21.15.116:0/2401592388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c131f30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 -- 172.21.15.116:0/2401592388 >> 172.21.15.116:0/2401592388 conn(0x7f609c067110 msgr2=0x7f609c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 -- 172.21.15.116:0/2401592388 shutdown_connections 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.676+0000 7f60a18c9640 1 -- 172.21.15.116:0/2401592388 wait complete. 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.677+0000 7f60a18c9640 10 monclient: build_initial_monmap 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.677+0000 7f60a18c9640 10 monclient: monmap: 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:47.689 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.677+0000 7f60a18c9640 1 Processor -- start 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.677+0000 7f60a18c9640 1 -- start start 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.677+0000 7f60a18c9640 10 monclient: init 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 0x7f609c09a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): picked mon.noname-c con 0x7f609c068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f609c09ea50 0x7f609c1309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): picked mon.noname-b con 0x7f609c09ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c11b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): picked mon.noname-a con 0x7f609c1318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c11dc10 con 0x7f609c1318f0 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c11ddf0 con 0x7f609c09ea50 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c141db0 con 0x7f609c068d90 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f60a18c9640 10 monclient(hunting): authenticate will time out at 1134.999118s 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 0x7f609c09a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 0x7f609c09a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:32950/0 (socket says 172.21.15.116:32950) 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c11b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 -- 172.21.15.116:0/3445999600 learned_addr learned my addr 172.21.15.116:0/3445999600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): get_auth_request con 0x7f609c068d90 auth_method 0 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 10 monclient(hunting): get_auth_request con 0x7f609c1318f0 auth_method 0 2024-09-18T12:31:47.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient(hunting): handle_auth_done global_id 64703 payload 824 2024-09-18T12:31:47.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f6093fff640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f609c09ea50 0x7f609c1309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 -- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f609c09ea50 msgr2=0x7f609c1309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f609c09ea50 0x7f609c1309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 -- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 msgr2=0x7f609c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c11b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 monclient: found mon.noname-c 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609affd640 1 -- 172.21.15.116:0/3445999600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6084009390 con 0x7f609c068d90 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.678+0000 7f609b7fe640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f609affd640 1 -- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 msgr2=0x7f609c09a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f609affd640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 0x7f609c09a850 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f608c009850 tx=0x7f608c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:47.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 1 -- 172.21.15.116:0/3445999600 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c031c70 con 0x7f609c068d90 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: dump: 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:47.695 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:47.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:47.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:47.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 1 -- 172.21.15.116:0/3445999600 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c031dd0 con 0x7f609c068d90 2024-09-18T12:31:47.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f60a18c9640 5 monclient: authenticate success, global_id 64703 2024-09-18T12:31:47.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f60a18c9640 10 monclient: _renew_subs 2024-09-18T12:31:47.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f60a18c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:47.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c009390 con 0x7f609c068d90 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 1 -- 172.21.15.116:0/3445999600 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c03d670 con 0x7f609c068d90 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6092ffd640 10 monclient: dump: 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:47.697 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.680+0000 7f6099ffb640 4 set_mon_vals no callback set 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.681+0000 7f60a18c9640 10 monclient: _renew_subs 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.681+0000 7f60a18c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.681+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c136320 con 0x7f609c068d90 2024-09-18T12:31:47.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.681+0000 7f6092ffd640 1 -- 172.21.15.116:0/3445999600 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f608c039040 con 0x7f609c068d90 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.681+0000 7f6092ffd640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f607c038490 0x7f607c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.682+0000 7f6092ffd640 1 -- 172.21.15.116:0/3445999600 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(162..162 src has 13..162) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f608c06c860 con 0x7f609c068d90 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.682+0000 7f60a18c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.682+0000 7f60a18c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.682+0000 7f60a18c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.682+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c09bf50 con 0x7f609c068d90 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.682+0000 7f6093fff640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f607c038490 0x7f607c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.682+0000 7f6093fff640 10 monclient: get_auth_request con 0x7f607c038490 auth_method 0 2024-09-18T12:31:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.684+0000 7f6093fff640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f607c038490 0x7f607c03a940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6084006e70 tx=0x7f6084008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:47.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.686+0000 7f6092ffd640 1 -- 172.21.15.116:0/3445999600 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f608c042070 con 0x7f609c068d90 2024-09-18T12:31:47.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.686+0000 7f6092ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:47.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.686+0000 7f6092ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.905+0000 7f60a18c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.905+0000 7f60a18c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.905+0000 7f60a18c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.905+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f609c04bf60 con 0x7f609c068d90 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.905+0000 7f6092ffd640 1 -- 172.21.15.116:0/3445999600 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f608c02f4b0 con 0x7f609c068d90 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.905+0000 7f6092ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.905+0000 7f6092ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:47.906 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:47.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f607c038490 msgr2=0x7f607c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:47.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f607c038490 0x7f607c03a940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6084006e70 tx=0x7f6084008040 comp rx=0 tx=0).stop 2024-09-18T12:31:47.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 10 monclient: shutdown 2024-09-18T12:31:47.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 msgr2=0x7f609c09a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 0x7f609c09a850 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f608c009850 tx=0x7f608c032000 comp rx=0 tx=0).stop 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 shutdown_connections 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f607c038490 0x7f607c03a940 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f609c1318f0 0x7f609c11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f609c09ea50 0x7f609c1309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 --2- 172.21.15.116:0/3445999600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f609c068d90 0x7f609c09a850 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 >> 172.21.15.116:0/3445999600 conn(0x7f609c067110 msgr2=0x7f609c0a46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 shutdown_connections 2024-09-18T12:31:47.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:47.907+0000 7f60a18c9640 1 -- 172.21.15.116:0/3445999600 wait complete. 2024-09-18T12:31:47.918 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:31:47.918 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:31:47.918 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:31:47.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.948+0000 7fe82cff9640 1 -- 172.21.15.144:0/1513614396 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v163) v1 ==== 241+0+0 (secure 0 0 0) 0x7fe82000e290 con 0x7fe83009ec50 2024-09-18T12:31:47.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.948+0000 7fe82cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T12:31:47.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.948+0000 7fe82cff9640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-18T12:31:47.950 INFO:tasks.workunit.client.0.smithi144.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-18T12:31:47.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe810038490 msgr2=0x7fe81003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:47.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe810038490 0x7fe81003a940 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe828007fd0 tx=0x7fe828009040 comp rx=0 tx=0).stop 2024-09-18T12:31:47.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 10 monclient: shutdown 2024-09-18T12:31:47.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:47.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 msgr2=0x7fe830130d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:47.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 0x7fe830130d90 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe82000c590 tx=0x7fe820007dc0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 shutdown_connections 2024-09-18T12:31:47.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe810038490 0x7fe81003a940 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe83009f020 0x7fe83011b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe83009ec50 0x7fe830130d90 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 --2- 172.21.15.144:0/1513614396 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe830069d10 0x7fe83009acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:47.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.951+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 >> 172.21.15.144:0/1513614396 conn(0x7fe8300a5d40 msgr2=0x7fe8300a46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:47.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.952+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 shutdown_connections 2024-09-18T12:31:47.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:47.952+0000 7fe836f5d640 1 -- 172.21.15.144:0/1513614396 wait complete. 2024-09-18T12:31:47.965 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:31:47.965 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:31:47.965 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep fish 2024-09-18T12:31:47.965 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '10\.' 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.013+0000 7f2d5619d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.013+0000 7f2d5619d640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.013+0000 7f2d5619d640 10 monclient: monmap: 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:48.013899+0000 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:48.013899+0000 2024-09-18T12:31:48.014 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.015 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.015 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:48.015 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:48.015 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:48.015 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:48.015 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:48.015 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 1 Processor -- start 2024-09-18T12:31:48.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 1 -- start start 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient: init 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d5006eb80 0x7f2d5006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d5006eb80 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d50068a20 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2d50136ab0 0x7f2d50138e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d50136ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5006a0e0 con 0x7f2d50068a20 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5006a7a0 con 0x7f2d5006eb80 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d50069e00 con 0x7f2d50136ab0 2024-09-18T12:31:48.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d5619d640 10 monclient(hunting): authenticate will time out at 1135.335692s 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d5006eb80 0x7f2d5006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d5006eb80 0x7f2d5006ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52544/0 (socket says 172.21.15.116:52544) 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/1432800286 learned_addr learned my addr 172.21.15.116:0/1432800286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d5006eb80 auth_method 0 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.014+0000 7f2d4ffff640 1 --2- 172.21.15.116:0/1432800286 >> v2:172.21.15.144:3300/0 conn(0x7f2d50136ab0 0x7f2d50138e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request con 0x7f2d50136ab0 auth_method 0 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_done global_id 94104 payload 274 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/1432800286 >> v2:172.21.15.144:3300/0 conn(0x7f2d50136ab0 msgr2=0x7f2d50138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 1 --2- 172.21.15.116:0/1432800286 >> v2:172.21.15.144:3300/0 conn(0x7f2d50136ab0 0x7f2d50138e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/1432800286 >> v1:172.21.15.144:6789/0 conn(0x7f2d50068a20 legacy=0x7f2d5009c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 1 monclient: found mon.noname-a 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.015+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/1432800286 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5006f490 con 0x7f2d5006eb80 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.017+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/1432800286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006eb80 msgr2=0x7f2d5006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.017+0000 7f2d4f7fe640 1 --2- 172.21.15.116:0/1432800286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006eb80 0x7f2d5006ef50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009910 tx=0x7f2d400091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13fdd84a2d0eb98c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 1 -- 172.21.15.116:0/1432800286 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4002aea0 con 0x7f2d5006eb80 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: dump: 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.022 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 1 -- 172.21.15.116:0/1432800286 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d40032440 con 0x7f2d5006eb80 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 5 monclient: authenticate success, global_id 94104 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 1 -- 172.21.15.116:0/1432800286 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d400327e0 con 0x7f2d5006eb80 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 4 set_mon_vals no callback set 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d4effd640 10 monclient: dump: 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.024 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 10 monclient: shutdown 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 -- 172.21.15.116:0/1432800286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006eb80 msgr2=0x7f2d5006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 --2- 172.21.15.116:0/1432800286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006eb80 0x7f2d5006ef50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009910 tx=0x7f2d400091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 -- 172.21.15.116:0/1432800286 shutdown_connections 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 --2- 172.21.15.116:0/1432800286 >> v2:172.21.15.144:3300/0 conn(0x7f2d50136ab0 0x7f2d50138e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 --2- 172.21.15.116:0/1432800286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006eb80 0x7f2d5006ef50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 -- 172.21.15.116:0/1432800286 >> 172.21.15.116:0/1432800286 conn(0x7f2d50067cd0 msgr2=0x7f2d500a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 -- 172.21.15.116:0/1432800286 shutdown_connections 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.018+0000 7f2d5619d640 1 -- 172.21.15.116:0/1432800286 wait complete. 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.019+0000 7f2d5619d640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.019+0000 7f2d5619d640 10 monclient: monmap: 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:48.032 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.019+0000 7f2d5619d640 1 Processor -- start 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.019+0000 7f2d5619d640 1 -- start start 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.019+0000 7f2d5619d640 10 monclient: init 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d50068a20 0x7f2d50098cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d50068a20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d5006eb80 0x7f2d5006c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d5006eb80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d50136ab0 0x7f2d5006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d50136ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:48.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5009d050 con 0x7f2d50136ab0 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d501387a0 con 0x7f2d50068a20 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d50098320 con 0x7f2d5006eb80 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d5619d640 10 monclient(hunting): authenticate will time out at 1135.341102s 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d50136ab0 0x7f2d5006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d5006eb80 0x7f2d5006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d50136ab0 0x7f2d5006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52554/0 (socket says 172.21.15.116:52554) 2024-09-18T12:31:48.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 10 monclient(hunting): get_auth_request con 0x7f2d5006eb80 auth_method 0 2024-09-18T12:31:48.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 -- 172.21.15.116:0/2738355939 learned_addr learned my addr 172.21.15.116:0/2738355939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request con 0x7f2d50136ab0 auth_method 0 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d50068a20 0x7f2d50098cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d50068a20 auth_method 0 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient(hunting): handle_auth_done global_id 94107 payload 824 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d5006eb80 msgr2=0x7f2d5006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d5006eb80 0x7f2d5006c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d50068a20 msgr2=0x7f2d50098cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d50068a20 0x7f2d50098cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 monclient: found mon.noname-a 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d4ffff640 1 -- 172.21.15.116:0/2738355939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40009450 con 0x7f2d50136ab0 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.020+0000 7f2d47fff640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d5006eb80 0x7f2d5006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d4ffff640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d50136ab0 msgr2=0x7f2d5006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d4ffff640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d50136ab0 0x7f2d5006cda0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009fa0 tx=0x7f2d48009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4800ee60 con 0x7f2d50136ab0 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: dump: 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.038 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d5619d640 5 monclient: authenticate success, global_id 94107 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d5619d640 10 monclient: _renew_subs 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d48013590 con 0x7f2d50136ab0 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d5619d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5006e360 con 0x7f2d50136ab0 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48013950 con 0x7f2d50136ab0 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d477fe640 10 monclient: dump: 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.040 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d4e7fc640 4 set_mon_vals no callback set 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d5619d640 10 monclient: _renew_subs 2024-09-18T12:31:48.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d5619d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.022+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d50097120 con 0x7f2d50136ab0 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2d48013cc0 con 0x7f2d50136ab0 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d5619d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d5619d640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 0x7f2d5009b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d5619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d50065e00 con 0x7f2d5006d2e0 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 0x7f2d5009b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 10 monclient: get_auth_request con 0x7f2d5006d2e0 auth_method 0 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d477fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d30038490 0x7f2d3003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.023+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f2d4804d3f0 con 0x7f2d50136ab0 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.024+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_done global_id 94110 payload 824 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.024+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/2738355939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d5004bf60 con 0x7f2d5006d2e0 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.024+0000 7f2d47fff640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d30038490 0x7f2d3003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.024+0000 7f2d47fff640 10 monclient: get_auth_request con 0x7f2d30038490 auth_method 0 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 msgr2=0x7f2d5009b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d4f7fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 0x7f2d5009b330 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d40034000 tx=0x7f2d40032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4002aea0 con 0x7f2d5006d2e0 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 10 monclient: dump: 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.044 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d47fff640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d30038490 0x7f2d3003a940 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009e40 tx=0x7f2d38006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2d40031d30 con 0x7f2d5006d2e0 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 msgr2=0x7f2d5009b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 0x7f2d5009b330 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d40034000 tx=0x7f2d40032000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 msgr2=0x7f2d5009b330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.026+0000 7f2d477fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5006d2e0 0x7f2d5009b330 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d40034000 tx=0x7f2d40032000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d5619d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d5619d640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 0x7f2d5009bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d5619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5006d6b0 con 0x7f2d5009b870 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 0x7f2d5009bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient: get_auth_request con 0x7f2d5009b870 auth_method 0 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 10 monclient(hunting): handle_auth_done global_id 94113 payload 824 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.032+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/2738355939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2d50098460 con 0x7f2d5009b870 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.034+0000 7f2d4f7fe640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 msgr2=0x7f2d5009bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.034+0000 7f2d4f7fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 0x7f2d5009bc70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2d50098840 tx=0x7f2d4003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4004d070 con 0x7f2d5009b870 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 10 monclient: dump: 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.050 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f2d4003cc10 con 0x7f2d5009b870 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:48.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:48.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 msgr2=0x7f2d5009bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 0x7f2d5009bc70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2d50098840 tx=0x7f2d4003d000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 msgr2=0x7f2d5009bc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:48.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d477fe640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d5009b870 0x7f2d5009bc70 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2d50098840 tx=0x7f2d4003d000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.052 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:48.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 72, 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:48.053 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.054 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:48.055 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:48.056 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.057 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:48.058 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:48.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.063 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.064 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:48.065 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d30038490 msgr2=0x7f2d3003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d30038490 0x7f2d3003a940 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009e40 tx=0x7f2d38006f50 comp rx=0 tx=0).stop 2024-09-18T12:31:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 10 monclient: shutdown 2024-09-18T12:31:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d50136ab0 msgr2=0x7f2d5006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 --2- 172.21.15.116:0/2738355939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d50136ab0 0x7f2d5006cda0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009fa0 tx=0x7f2d48009900 comp rx=0 tx=0).stop 2024-09-18T12:31:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d4ffff640 1 -- 172.21.15.116:0/2738355939 reap_dead start 2024-09-18T12:31:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d4ffff640 1 -- 172.21.15.116:0/2738355939 reap_dead start 2024-09-18T12:31:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 shutdown_connections 2024-09-18T12:31:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 >> 172.21.15.116:0/2738355939 conn(0x7f2d50067cd0 msgr2=0x7f2d50136e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 shutdown_connections 2024-09-18T12:31:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.035+0000 7f2d5619d640 1 -- 172.21.15.116:0/2738355939 wait complete. 2024-09-18T12:31:48.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:31:48.070 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:31:48.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:31:48.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.043+0000 7fa235d00640 10 monclient: get_monmap_and_config 2024-09-18T12:31:48.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.043+0000 7fa235d00640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.043+0000 7fa235d00640 10 monclient: monmap: 2024-09-18T12:31:48.077 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:48.077 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.077 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:48.043944+0000 2024-09-18T12:31:48.077 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:48.043944+0000 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.043+0000 7fa235d00640 1 Processor -- start 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.043+0000 7fa235d00640 1 -- start start 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.043+0000 7fa235d00640 10 monclient: init 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:31:48.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa23009ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009f020 0x7fa230069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient(hunting): picked mon.noname-b con 0x7fa23009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa230136d60 0x7fa230139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient(hunting): picked mon.noname-c con 0x7fa230136d60 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa230067420 con 0x7fa23009ec50 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa23009f3f0 con 0x7fa230136d60 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2300a0480 con 0x7fa23009f020 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa235d00640 10 monclient(hunting): authenticate will time out at 1143.613412s 2024-09-18T12:31:48.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa230136d60 0x7fa230139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22f7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa23009ec50 0x7fa230131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59300/0 (socket says 172.21.15.144:59300) 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22f7fe640 1 -- 172.21.15.144:0/365958779 learned_addr learned my addr 172.21.15.144:0/365958779 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): get_auth_request con 0x7fa230136d60 auth_method 0 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 1 --2- 172.21.15.144:0/365958779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009f020 0x7fa230069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 10 monclient(hunting): get_auth_request con 0x7fa23009f020 auth_method 0 2024-09-18T12:31:48.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient(hunting): handle_auth_done global_id 5902 payload 274 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 1 -- 172.21.15.144:0/365958779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009f020 msgr2=0x7fa230069d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 1 --2- 172.21.15.144:0/365958779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009f020 0x7fa230069d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 1 -- 172.21.15.144:0/365958779 >> v1:172.21.15.144:6789/0 conn(0x7fa23009ec50 legacy=0x7fa230131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 1 monclient: found mon.noname-c 2024-09-18T12:31:48.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.044+0000 7fa22ffff640 1 -- 172.21.15.144:0/365958779 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2300a0200 con 0x7fa230136d60 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.047+0000 7fa22ffff640 1 -- 172.21.15.144:0/365958779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 msgr2=0x7fa230139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.047+0000 7fa22ffff640 1 --2- 172.21.15.144:0/365958779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230139140 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa228009e60 tx=0x7fa228009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997fc819c2578e83 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 1 -- 172.21.15.144:0/365958779 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228007760 con 0x7fa230136d60 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: dump: 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.082 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 1 -- 172.21.15.144:0/365958779 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22802d050 con 0x7fa230136d60 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa235d00640 5 monclient: authenticate success, global_id 5902 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa235d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:48.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa235d00640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 1 -- 172.21.15.144:0/365958779 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228007af0 con 0x7fa230136d60 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa235d00640 4 set_mon_vals no callback set 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.049+0000 7fa22e7fc640 10 monclient: dump: 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.084 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 10 monclient: shutdown 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 -- 172.21.15.144:0/365958779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 msgr2=0x7fa230139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 --2- 172.21.15.144:0/365958779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230139140 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa228009e60 tx=0x7fa228009780 comp rx=0 tx=0).stop 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 -- 172.21.15.144:0/365958779 shutdown_connections 2024-09-18T12:31:48.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 --2- 172.21.15.144:0/365958779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230139140 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 --2- 172.21.15.144:0/365958779 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009f020 0x7fa230069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 -- 172.21.15.144:0/365958779 >> 172.21.15.144:0/365958779 conn(0x7fa2300a5cf0 msgr2=0x7fa2300a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 -- 172.21.15.144:0/365958779 shutdown_connections 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 1 -- 172.21.15.144:0/365958779 wait complete. 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.050+0000 7fa235d00640 10 monclient: monmap: 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.086 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 Processor -- start 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 -- start start 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient: init 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009ec50 0x7fa2301411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): picked mon.noname-c con 0x7fa23009ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23009f020 0x7fa230141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): picked mon.noname-a con 0x7fa23009f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): picked mon.noname-b con 0x7fa230136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23011ce30 con 0x7fa23009f020 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23009c020 con 0x7fa230136d60 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2300a4140 con 0x7fa23009ec50 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa235d00640 10 monclient(hunting): authenticate will time out at 1143.620877s 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.051+0000 7fa22ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230141c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35416/0 (socket says 172.21.15.144:35416) 2024-09-18T12:31:48.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009ec50 0x7fa2301411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 -- 172.21.15.144:0/1203326872 learned_addr learned my addr 172.21.15.144:0/1203326872 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23009f020 0x7fa230141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): get_auth_request con 0x7fa230136d60 auth_method 0 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 10 monclient(hunting): get_auth_request con 0x7fa23009ec50 auth_method 0 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 10 monclient(hunting): get_auth_request con 0x7fa23009f020 auth_method 0 2024-09-18T12:31:48.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient(hunting): handle_auth_done global_id 5905 payload 824 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 -- 172.21.15.144:0/1203326872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009ec50 msgr2=0x7fa2301411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009ec50 0x7fa2301411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 -- 172.21.15.144:0/1203326872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23009f020 msgr2=0x7fa230141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23009f020 0x7fa230141710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 monclient: found mon.noname-b 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.052+0000 7fa22ffff640 1 -- 172.21.15.144:0/1203326872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228009a20 con 0x7fa230136d60 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa22ffff640 1 -- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 msgr2=0x7fa230141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa22ffff640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230141c50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa228009ec0 tx=0x7fa228037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228031ac0 con 0x7fa230136d60 2024-09-18T12:31:48.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: dump: 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.092 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa228040070 con 0x7fa230136d60 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa235d00640 5 monclient: authenticate success, global_id 5905 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa235d00640 10 monclient: _renew_subs 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa235d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23009b0d0 con 0x7fa230136d60 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228031e70 con 0x7fa230136d60 2024-09-18T12:31:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa2177fe640 10 monclient: dump: 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa235d00640 10 monclient: _renew_subs 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa235d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23011da70 con 0x7fa230136d60 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.054+0000 7fa22dffb640 4 set_mon_vals no callback set 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa228042030 con 0x7fa230136d60 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa235d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa235d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa235d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2300008d0 con 0x7fa230136d60 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa2177fe640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa21c038490 0x7fa21c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7fa22806e390 con 0x7fa230136d60 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa22f7fe640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa21c038490 0x7fa21c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.055+0000 7fa22f7fe640 10 monclient: get_auth_request con 0x7fa21c038490 auth_method 0 2024-09-18T12:31:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.058+0000 7fa22f7fe640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa21c038490 0x7fa21c03a940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa220007fd0 tx=0x7fa220007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.060+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa228030b80 con 0x7fa230136d60 2024-09-18T12:31:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.060+0000 7fa2177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.060+0000 7fa2177fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.165+0000 7f011e31d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.165+0000 7f011e31d640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.165+0000 7f011e31d640 10 monclient: monmap: 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:48.165895+0000 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:48.165895+0000 2024-09-18T12:31:48.166 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:48.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:48.167 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:48.167 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:48.167 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:48.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.165+0000 7f011e31d640 1 Processor -- start 2024-09-18T12:31:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.165+0000 7f011e31d640 1 -- start start 2024-09-18T12:31:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.165+0000 7f011e31d640 10 monclient: init 2024-09-18T12:31:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0118131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f011809e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0118137bc0 0x7f0118139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient(hunting): picked mon.noname-a con 0x7f0118137bc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011809faf0 con 0x7f0118131ab0 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011809fd30 con 0x7f011809e910 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0118132590 con 0x7f0118137bc0 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011e31d640 10 monclient(hunting): authenticate will time out at 1135.487458s 2024-09-18T12:31:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0118137bc0 0x7f0118139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0118137bc0 0x7f0118139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52580/0 (socket says 172.21.15.116:52580) 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f0117fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0118131ab0 0x7f0118131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54686/0 (socket says 172.21.15.116:54686) 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 1 -- 172.21.15.116:0/3679882287 learned_addr learned my addr 172.21.15.116:0/3679882287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): get_auth_request con 0x7f0118137bc0 auth_method 0 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient(hunting): handle_auth_done global_id 94119 payload 274 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 1 -- 172.21.15.116:0/3679882287 >> v1:172.21.15.144:6789/0 conn(0x7f011809e910 legacy=0x7f011809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 1 -- 172.21.15.116:0/3679882287 >> v1:172.21.15.116:6789/0 conn(0x7f0118131ab0 legacy=0x7f0118131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 1 monclient: found mon.noname-a 2024-09-18T12:31:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.166+0000 7f011c8ba640 1 -- 172.21.15.116:0/3679882287 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01180a1110 con 0x7f0118137bc0 2024-09-18T12:31:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f011c8ba640 1 -- 172.21.15.116:0/3679882287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118137bc0 msgr2=0x7f0118139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f011c8ba640 1 --2- 172.21.15.116:0/3679882287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118137bc0 0x7f0118139fa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0110009f50 tx=0x7f0110009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f03846f03c9d905 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 1 -- 172.21.15.116:0/3679882287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110007610 con 0x7f0118137bc0 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: dump: 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 1 -- 172.21.15.116:0/3679882287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011002d050 con 0x7f0118137bc0 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f011e31d640 5 monclient: authenticate success, global_id 94119 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f011e31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f011e31d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 1 -- 172.21.15.116:0/3679882287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100079a0 con 0x7f0118137bc0 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f011e31d640 4 set_mon_vals no callback set 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.169+0000 7f0116ffd640 10 monclient: dump: 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 10 monclient: shutdown 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 1 -- 172.21.15.116:0/3679882287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118137bc0 msgr2=0x7f0118139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 1 --2- 172.21.15.116:0/3679882287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118137bc0 0x7f0118139fa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0110009f50 tx=0x7f0110009840 comp rx=0 tx=0).stop 2024-09-18T12:31:48.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 1 -- 172.21.15.116:0/3679882287 shutdown_connections 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 1 --2- 172.21.15.116:0/3679882287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118137bc0 0x7f0118139fa0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 1 -- 172.21.15.116:0/3679882287 >> 172.21.15.116:0/3679882287 conn(0x7f0118067d30 msgr2=0x7f01180a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 1 -- 172.21.15.116:0/3679882287 shutdown_connections 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 1 -- 172.21.15.116:0/3679882287 wait complete. 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.170+0000 7f011e31d640 10 monclient: monmap: 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:48.183 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 Processor -- start 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 -- start start 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient: init 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f011809e910 0x7f0118137070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): picked mon.noname-b con 0x7f011809e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118131ab0 0x7f011812df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): picked mon.noname-a con 0x7f0118131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118137bc0 0x7f011812e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): picked mon.noname-c con 0x7f0118137bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01180a5550 con 0x7f0118131ab0 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0118132e70 con 0x7f011809e910 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0118132770 con 0x7f0118137bc0 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011e31d640 10 monclient(hunting): authenticate will time out at 1135.492669s 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118131ab0 0x7f011812df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118137bc0 0x7f011812e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118131ab0 0x7f011812df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52596/0 (socket says 172.21.15.116:52596) 2024-09-18T12:31:48.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 1 -- 172.21.15.116:0/3954837601 learned_addr learned my addr 172.21.15.116:0/3954837601 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 10 monclient(hunting): get_auth_request con 0x7f0118131ab0 auth_method 0 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 10 monclient(hunting): get_auth_request con 0x7f0118137bc0 auth_method 0 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f011809e910 0x7f0118137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 10 monclient(hunting): get_auth_request con 0x7f011809e910 auth_method 0 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f01177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f011c8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.171+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 10 monclient(hunting): handle_auth_done global_id 94122 payload 824 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118137bc0 msgr2=0x7f011812e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118137bc0 0x7f011812e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f011809e910 msgr2=0x7f0118137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f011809e910 0x7f0118137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 1 monclient: found mon.noname-a 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f01177fe640 1 -- 172.21.15.116:0/3954837601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0110009ae0 con 0x7f0118131ab0 2024-09-18T12:31:48.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.172+0000 7f011c8ba640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118137bc0 0x7f011812e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f01177fe640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118131ab0 msgr2=0x7f011812df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f01177fe640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118131ab0 0x7f011812df10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f010000c5e0 tx=0x7f0100007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100012840 con 0x7f0118131ab0 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: dump: 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.189 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01000129c0 con 0x7f0118131ab0 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f011e31d640 5 monclient: authenticate success, global_id 94122 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f011e31d640 10 monclient: _renew_subs 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f011e31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01181313d0 con 0x7f0118131ab0 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100012d80 con 0x7f0118131ab0 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f010ffff640 10 monclient: dump: 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f011e31d640 10 monclient: _renew_subs 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f01167fc640 4 set_mon_vals no callback set 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f011e31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.174+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011812fa90 con 0x7f0118131ab0 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f011e31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0100011590 con 0x7f0118131ab0 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f011e31d640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 0x7f011812e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f011e31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0118063740 con 0x7f0118130000 2024-09-18T12:31:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 0x7f011812e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient: get_auth_request con 0x7f0118130000 auth_method 0 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f010ffff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00f80384e0 0x7f00f803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f010004cd60 con 0x7f0118131ab0 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 10 monclient(hunting): handle_auth_done global_id 64718 payload 824 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f0117fff640 1 -- 172.21.15.116:0/3954837601 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f011804bf60 con 0x7f0118130000 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f011c8ba640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00f80384e0 0x7f00f803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.175+0000 7f011c8ba640 10 monclient: get_auth_request con 0x7f00f80384e0 auth_method 0 2024-09-18T12:31:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.177+0000 7f0117fff640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 msgr2=0x7f011812e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.177+0000 7f0117fff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 0x7f011812e990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f010800e740 tx=0x7f0108007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.177+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108005050 con 0x7f0118130000 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.177+0000 7f010ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.177+0000 7f010ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.177+0000 7f010ffff640 10 monclient: dump: 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f01080051b0 con 0x7f0118130000 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f010ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f010ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 msgr2=0x7f011812e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f010ffff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 0x7f011812e990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f010800e740 tx=0x7f0108007e40 comp rx=0 tx=0).stop 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 msgr2=0x7f011812e990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f010ffff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118130000 0x7f011812e990 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f010800e740 tx=0x7f0108007e40 comp rx=0 tx=0).stop 2024-09-18T12:31:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.178+0000 7f011c8ba640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00f80384e0 0x7f00f803a990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0118076760 tx=0x7f0110036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f011e31d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f011e31d640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 0x7f0118129c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f011e31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0118066280 con 0x7f0118129890 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f0117fff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 0x7f0118129c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f0117fff640 10 monclient: get_auth_request con 0x7f0118129890 auth_method 0 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f0117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f0117fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.183+0000 7f0117fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.184+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.184+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.184+0000 7f0117fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.184+0000 7f0117fff640 10 monclient(hunting): handle_auth_done global_id 64721 payload 824 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.184+0000 7f0117fff640 1 -- 172.21.15.116:0/3954837601 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0118132930 con 0x7f0118129890 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f0117fff640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 msgr2=0x7f0118129c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f0117fff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 0x7f0118129c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f010800e740 tx=0x7f0108019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010801d610 con 0x7f0118129890 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 10 monclient: dump: 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.200 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.201 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.201 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f0108025070 con 0x7f0118129890 2024-09-18T12:31:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 msgr2=0x7f0118129c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 0x7f0118129c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f010800e740 tx=0x7f0108019000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 msgr2=0x7f0118129c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:48.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.186+0000 7f010ffff640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0118129890 0x7f0118129c90 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f010800e740 tx=0x7f0108019000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.203 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:48.203 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:48.203 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:48.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:31:48.204 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 72, 2024-09-18T12:31:48.204 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:48.204 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:31:48.204 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:48.204 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:48.204 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:31:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.205 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.206 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.206 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.206 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.206 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.206 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.206 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.207 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:48.208 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.209 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:48.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.212 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.214 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:48.215 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.217 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.219 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:48.219 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00f80384e0 msgr2=0x7f00f803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00f80384e0 0x7f00f803a990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0118076760 tx=0x7f0110036000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 10 monclient: shutdown 2024-09-18T12:31:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118131ab0 msgr2=0x7f011812df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 --2- 172.21.15.116:0/3954837601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0118131ab0 0x7f011812df10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f010000c5e0 tx=0x7f0100007ed0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011c8ba640 1 -- 172.21.15.116:0/3954837601 reap_dead start 2024-09-18T12:31:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011c8ba640 1 -- 172.21.15.116:0/3954837601 reap_dead start 2024-09-18T12:31:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 shutdown_connections 2024-09-18T12:31:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 >> 172.21.15.116:0/3954837601 conn(0x7f0118067d30 msgr2=0x7f0118139120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 shutdown_connections 2024-09-18T12:31:48.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.187+0000 7f011e31d640 1 -- 172.21.15.116:0/3954837601 wait complete. 2024-09-18T12:31:48.221 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:31:48.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:31:48.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:31:48.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.221+0000 7fa235d00640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.221+0000 7fa235d00640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.221+0000 7fa235d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.221+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fa2300a13e0 con 0x7fa230136d60 2024-09-18T12:31:48.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.221+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fa2280303a0 con 0x7fa230136d60 2024-09-18T12:31:48.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.221+0000 7fa2177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.221+0000 7fa2177fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.316+0000 7f290144f640 10 monclient: get_monmap_and_config 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.316+0000 7f290144f640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.316+0000 7f290144f640 10 monclient: monmap: 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:48.316853+0000 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:48.316853+0000 2024-09-18T12:31:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:48.318 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.316+0000 7f290144f640 1 Processor -- start 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.316+0000 7f290144f640 1 -- start start 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.316+0000 7f290144f640 10 monclient: init 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f28fc1318f0 0x7f28fc131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): picked mon.noname-c con 0x7f28fc1318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc09ea50 0x7f28fc09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): picked mon.noname-b con 0x7f28fc09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28fc068d90 0x7f28fc0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): picked mon.noname-a con 0x7f28fc068d90 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc131cc0 con 0x7f28fc068d90 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc09fbf0 con 0x7f28fc1318f0 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc09fe30 con 0x7f28fc09ea50 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f290144f640 10 monclient(hunting): authenticate will time out at 1135.638502s 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc09ea50 0x7f28fc09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28fc068d90 0x7f28fc0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc09ea50 0x7f28fc09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33014/0 (socket says 172.21.15.116:33014) 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 1 -- 172.21.15.116:0/3394919934 learned_addr learned my addr 172.21.15.116:0/3394919934 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): get_auth_request con 0x7f28fc068d90 auth_method 0 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 10 monclient(hunting): get_auth_request con 0x7f28fc09ea50 auth_method 0 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 1 --2- 172.21.15.116:0/3394919934 >> v2:172.21.15.144:3300/0 conn(0x7f28fc1318f0 0x7f28fc131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 10 monclient(hunting): get_auth_request con 0x7f28fc1318f0 auth_method 0 2024-09-18T12:31:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient(hunting): handle_auth_done global_id 94125 payload 274 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 1 -- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc09ea50 msgr2=0x7f28fc09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 1 --2- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc09ea50 0x7f28fc09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 1 -- 172.21.15.116:0/3394919934 >> v2:172.21.15.144:3300/0 conn(0x7f28fc1318f0 msgr2=0x7f28fc131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 1 --2- 172.21.15.116:0/3394919934 >> v2:172.21.15.144:3300/0 conn(0x7f28fc1318f0 0x7f28fc131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 1 monclient: found mon.noname-a 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28fb7fe640 1 -- 172.21.15.116:0/3394919934 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0a1210 con 0x7f28fc068d90 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.317+0000 7f28f27fc640 1 --2- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc09ea50 0x7f28fc09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fb7fe640 1 -- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc068d90 msgr2=0x7f28fc0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fb7fe640 1 --2- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc068d90 0x7f28fc0691f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28fc076600 tx=0x7f28f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c04c925ec890e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 1 -- 172.21.15.116:0/3394919934 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4007760 con 0x7f28fc068d90 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: dump: 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f290144f640 5 monclient: authenticate success, global_id 94125 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 1 -- 172.21.15.116:0/3394919934 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f402d050 con 0x7f28fc068d90 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f290144f640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f290144f640 4 set_mon_vals no callback set 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 1 -- 172.21.15.116:0/3394919934 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4007af0 con 0x7f28fc068d90 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.320+0000 7f28fa7fc640 10 monclient: dump: 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.330 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 10 monclient: shutdown 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 -- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc068d90 msgr2=0x7f28fc0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 --2- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc068d90 0x7f28fc0691f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28fc076600 tx=0x7f28f4009780 comp rx=0 tx=0).stop 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 -- 172.21.15.116:0/3394919934 shutdown_connections 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 --2- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc068d90 0x7f28fc0691f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 --2- 172.21.15.116:0/3394919934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc09ea50 0x7f28fc09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 --2- 172.21.15.116:0/3394919934 >> v2:172.21.15.144:3300/0 conn(0x7f28fc1318f0 0x7f28fc131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 -- 172.21.15.116:0/3394919934 >> 172.21.15.116:0/3394919934 conn(0x7f28fc067110 msgr2=0x7f28fc067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 -- 172.21.15.116:0/3394919934 shutdown_connections 2024-09-18T12:31:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 1 -- 172.21.15.116:0/3394919934 wait complete. 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.321+0000 7f290144f640 10 monclient: monmap: 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 Processor -- start 2024-09-18T12:31:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 -- start start 2024-09-18T12:31:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient: init 2024-09-18T12:31:48.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.323+0000 7fa235d00640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.323+0000 7fa235d00640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.323+0000 7fa235d00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.323+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fa2300a13e0 con 0x7fa230136d60 2024-09-18T12:31:48.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.323+0000 7fa2177fe640 1 -- 172.21.15.144:0/1203326872 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fa228030580 con 0x7fa230136d60 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.323+0000 7fa2177fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.323+0000 7fa2177fe640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa21c038490 msgr2=0x7fa21c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa21c038490 0x7fa21c03a940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa220007fd0 tx=0x7fa220007f00 comp rx=0 tx=0).stop 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 10 monclient: shutdown 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 msgr2=0x7fa230141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230141c50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa228009ec0 tx=0x7fa228037000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 shutdown_connections 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa21c038490 0x7fa21c03a940 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa230136d60 0x7fa230141c50 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23009f020 0x7fa230141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 --2- 172.21.15.144:0/1203326872 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa23009ec50 0x7fa2301411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 >> 172.21.15.144:0/1203326872 conn(0x7fa2300a5cf0 msgr2=0x7fa23009d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 shutdown_connections 2024-09-18T12:31:48.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.325+0000 7fa235d00640 1 -- 172.21.15.144:0/1203326872 wait complete. 2024-09-18T12:31:48.341 INFO:tasks.workunit.client.0.smithi144.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:31:48.341 INFO:tasks.workunit.client.0.smithi144.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:31:48.343 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:31:48.343 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:31:48.343 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep fish 2024-09-18T12:31:48.343 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '11\.' 2024-09-18T12:31:48.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:48.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc068d90 0x7f28fc09a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): picked mon.noname-c con 0x7f28fc068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc09ea50 0x7f28fc11bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): picked mon.noname-a con 0x7f28fc09ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:48.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc1318f0 0x7f28fc130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): picked mon.noname-b con 0x7f28fc1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc09b4c0 con 0x7f28fc09ea50 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc11dc30 con 0x7f28fc1318f0 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc11df10 con 0x7f28fc068d90 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.322+0000 7f290144f640 10 monclient(hunting): authenticate will time out at 1135.643983s 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc09ea50 0x7f28fc11bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc068d90 0x7f28fc09a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc09ea50 0x7f28fc11bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52610/0 (socket says 172.21.15.116:52610) 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 1 -- 172.21.15.116:0/2624936076 learned_addr learned my addr 172.21.15.116:0/2624936076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): get_auth_request con 0x7f28fc068d90 auth_method 0 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 10 monclient(hunting): get_auth_request con 0x7f28fc09ea50 auth_method 0 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc1318f0 0x7f28fc130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 10 monclient(hunting): get_auth_request con 0x7f28fc1318f0 auth_method 0 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient(hunting): handle_auth_done global_id 64727 payload 824 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc1318f0 msgr2=0x7f28fc130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc1318f0 0x7f28fc130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc09ea50 msgr2=0x7f28fc11bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc09ea50 0x7f28fc11bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 1 monclient: found mon.noname-c 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28faffd640 1 -- 172.21.15.116:0/2624936076 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4009450 con 0x7f28fc068d90 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.323+0000 7f28f27fc640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28fc09ea50 0x7f28fc11bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28faffd640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc068d90 msgr2=0x7f28fc09a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28faffd640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc068d90 0x7f28fc09a940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00e990 tx=0x7f28ec0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00cb50 con 0x7f28fc068d90 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: dump: 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec00ccb0 con 0x7f28fc068d90 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f290144f640 5 monclient: authenticate success, global_id 64727 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f290144f640 10 monclient: _renew_subs 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f290144f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f4009a20 con 0x7f28fc068d90 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec013470 con 0x7f28fc068d90 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f37fe640 10 monclient: dump: 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f290144f640 10 monclient: _renew_subs 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f28f9ffb640 4 set_mon_vals no callback set 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f290144f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.325+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc11e2a0 con 0x7f28fc068d90 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f28ec013860 con 0x7f28fc068d90 2024-09-18T12:31:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f290144f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f290144f640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 0x7f28fc140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc065e00 con 0x7f28fc131050 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f27fc640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 0x7f28fc140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f37fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28dc038490 0x7f28dc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f27fc640 10 monclient: get_auth_request con 0x7f28fc131050 auth_method 0 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f28ec04ce80 con 0x7f28fc068d90 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.326+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.327+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.327+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.327+0000 7f28fb7fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28dc038490 0x7f28dc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.327+0000 7f28fb7fe640 10 monclient: get_auth_request con 0x7f28dc038490 auth_method 0 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.327+0000 7f28f27fc640 10 monclient(hunting): handle_auth_done global_id 5917 payload 824 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.327+0000 7f28f27fc640 1 -- 172.21.15.116:0/2624936076 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28fc131420 con 0x7f28fc131050 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f27fc640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 msgr2=0x7f28fc140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f27fc640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 0x7f28fc140b50 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f28fc09ba80 tx=0x7f28e40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28fb7fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28dc038490 0x7f28dc03a940 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f28fc064050 tx=0x7f28f4037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4014a10 con 0x7f28fc131050 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 10 monclient: dump: 2024-09-18T12:31:48.355 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f28e4014b70 con 0x7f28fc131050 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 msgr2=0x7f28fc140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.329+0000 7f28f37fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 0x7f28fc140b50 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f28fc09ba80 tx=0x7f28e40091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.330+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 msgr2=0x7f28fc140b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.330+0000 7f28f37fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc131050 0x7f28fc140b50 secure :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f28fc09ba80 tx=0x7f28e40091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.335+0000 7f290144f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.335+0000 7f290144f640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 0x7f28fc141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.335+0000 7f290144f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.335+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc11dda0 con 0x7f28fc141090 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 0x7f28fc141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient: get_auth_request con 0x7f28fc141090 auth_method 0 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 10 monclient(hunting): handle_auth_done global_id 5920 payload 824 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.336+0000 7f28f27fc640 1 -- 172.21.15.116:0/2624936076 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f28fc1419d0 con 0x7f28fc141090 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.338+0000 7f28f27fc640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 msgr2=0x7f28fc141490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.338+0000 7f28f27fc640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 0x7f28fc141490 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f28fc09ba80 tx=0x7f28e401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.338+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e401c800 con 0x7f28fc141090 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.338+0000 7f28f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.338+0000 7f28f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.338+0000 7f28f37fe640 10 monclient: dump: 2024-09-18T12:31:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f28e401c960 con 0x7f28fc141090 2024-09-18T12:31:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28f37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 msgr2=0x7f28fc141490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28f37fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 0x7f28fc141490 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f28fc09ba80 tx=0x7f28e401d000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28f37fe640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 msgr2=0x7f28fc141490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28f37fe640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f28fc141090 0x7f28fc141490 secure :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f28fc09ba80 tx=0x7f28e401d000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.360 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 72, 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:48.361 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.362 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:48.363 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:48.364 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.365 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.366 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.368 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.371 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:48.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:48.373 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:48.373 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28dc038490 msgr2=0x7f28dc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28dc038490 0x7f28dc03a940 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f28fc064050 tx=0x7f28f4037000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 10 monclient: shutdown 2024-09-18T12:31:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc068d90 msgr2=0x7f28fc09a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 --2- 172.21.15.116:0/2624936076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f28fc068d90 0x7f28fc09a940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00e990 tx=0x7f28ec0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28fb7fe640 1 -- 172.21.15.116:0/2624936076 reap_dead start 2024-09-18T12:31:48.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f28fb7fe640 1 -- 172.21.15.116:0/2624936076 reap_dead start 2024-09-18T12:31:48.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 shutdown_connections 2024-09-18T12:31:48.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 >> 172.21.15.116:0/2624936076 conn(0x7f28fc067110 msgr2=0x7f28fc0a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 shutdown_connections 2024-09-18T12:31:48.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.339+0000 7f290144f640 1 -- 172.21.15.116:0/2624936076 wait complete. 2024-09-18T12:31:48.375 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:31:48.375 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:31:48.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.409+0000 7fdec0d5d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.409+0000 7fdec0d5d640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.409+0000 7fdec0d5d640 10 monclient: monmap: 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:48.410115+0000 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:48.410115+0000 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.410 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:48.411 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:48.411 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:48.411 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:48.411 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:48.411 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.409+0000 7fdec0d5d640 1 Processor -- start 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.409+0000 7fdec0d5d640 1 -- start start 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.409+0000 7fdec0d5d640 10 monclient: init 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdebc09ec50 0x7fdebc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc09ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdebc1321a0 0x7fdebc068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc1321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc09f020 0x7fdebc069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdebc067420 con 0x7fdebc1321a0 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdebc0a0480 con 0x7fdebc09ec50 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc0a0200 con 0x7fdebc09f020 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdec0d5d640 10 monclient(hunting): authenticate will time out at 1143.979511s 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdebc09ec50 0x7fdebc131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdebc09ec50 0x7fdebc131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35428/0 (socket says 172.21.15.144:35428) 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 -- 172.21.15.144:0/4071605857 learned_addr learned my addr 172.21.15.144:0/4071605857 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): get_auth_request con 0x7fdebc09ec50 auth_method 0 2024-09-18T12:31:48.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 1 --2- 172.21.15.144:0/4071605857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc09f020 0x7fdebc069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 1 --2- 172.21.15.144:0/4071605857 >> v2:172.21.15.116:3300/0 conn(0x7fdebc1321a0 0x7fdebc068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 10 monclient(hunting): get_auth_request con 0x7fdebc09f020 auth_method 0 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 10 monclient(hunting): get_auth_request con 0x7fdebc1321a0 auth_method 0 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdebad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient(hunting): handle_auth_done global_id 5923 payload 274 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 -- 172.21.15.144:0/4071605857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc09f020 msgr2=0x7fdebc069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 --2- 172.21.15.144:0/4071605857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc09f020 0x7fdebc069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 -- 172.21.15.144:0/4071605857 >> v2:172.21.15.116:3300/0 conn(0x7fdebc1321a0 msgr2=0x7fdebc068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 --2- 172.21.15.144:0/4071605857 >> v2:172.21.15.116:3300/0 conn(0x7fdebc1321a0 0x7fdebc068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 monclient: found mon.noname-c 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.410+0000 7fdeba59c640 1 -- 172.21.15.144:0/4071605857 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc09fd80 con 0x7fdebc09ec50 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeba59c640 1 -- 172.21.15.144:0/4071605857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09ec50 msgr2=0x7fdebc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeba59c640 1 --2- 172.21.15.144:0/4071605857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09ec50 0x7fdebc131c60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0719d0 tx=0x7fdeac0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94952bfa48a249dd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 1 -- 172.21.15.144:0/4071605857 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac032440 con 0x7fdebc09ec50 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: dump: 2024-09-18T12:31:48.425 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 1 -- 172.21.15.144:0/4071605857 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeac0325a0 con 0x7fdebc09ec50 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdec0d5d640 5 monclient: authenticate success, global_id 5923 2024-09-18T12:31:48.426 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdec0d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdec0d5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 1 -- 172.21.15.144:0/4071605857 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac032940 con 0x7fdebc09ec50 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdec0d5d640 4 set_mon_vals no callback set 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.413+0000 7fdeb959a640 10 monclient: dump: 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.427 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 10 monclient: shutdown 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 -- 172.21.15.144:0/4071605857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09ec50 msgr2=0x7fdebc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/4071605857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09ec50 0x7fdebc131c60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0719d0 tx=0x7fdeac0091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 -- 172.21.15.144:0/4071605857 shutdown_connections 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/4071605857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc09f020 0x7fdebc069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/4071605857 >> v2:172.21.15.116:3300/0 conn(0x7fdebc1321a0 0x7fdebc068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/4071605857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09ec50 0x7fdebc131c60 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 -- 172.21.15.144:0/4071605857 >> 172.21.15.144:0/4071605857 conn(0x7fdebc0a5cf0 msgr2=0x7fdebc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 -- 172.21.15.144:0/4071605857 shutdown_connections 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 1 -- 172.21.15.144:0/4071605857 wait complete. 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.414+0000 7fdec0d5d640 10 monclient: monmap: 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:48.429 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 Processor -- start 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 -- start start 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient: init 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdebc09ec50 0x7fdebc130390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 0x7fdebc11ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc09f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc1321a0 0x7fdebc11d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.430 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc120300 con 0x7fdebc09ec50 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc11f240 con 0x7fdebc09f020 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc09e0e0 con 0x7fdebc1321a0 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdec0d5d640 10 monclient(hunting): authenticate will time out at 1143.984779s 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 0x7fdebc11ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 0x7fdebc11ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35444/0 (socket says 172.21.15.144:35444) 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 1 -- 172.21.15.144:0/3022042275 learned_addr learned my addr 172.21.15.144:0/3022042275 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 10 monclient(hunting): get_auth_request con 0x7fdebc09f020 auth_method 0 2024-09-18T12:31:48.431 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeba59c640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdebc09ec50 0x7fdebc130390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeba59c640 10 monclient(hunting): get_auth_request con 0x7fdebc09ec50 auth_method 0 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdebad9d640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc1321a0 0x7fdebc11d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdebad9d640 10 monclient(hunting): get_auth_request con 0x7fdebc1321a0 auth_method 0 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdebad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdebad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.432 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdebad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.415+0000 7fdeb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdebad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdebad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdebad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 10 monclient(hunting): handle_auth_done global_id 5926 payload 824 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 1 -- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc1321a0 msgr2=0x7fdebc11d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc1321a0 0x7fdebc11d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 1 -- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdebc09ec50 msgr2=0x7fdebc130390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdebc09ec50 0x7fdebc130390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 1 monclient: found mon.noname-b 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeb9d9b640 1 -- 172.21.15.144:0/3022042275 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac009450 con 0x7fdebc09f020 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.416+0000 7fdeba59c640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdebc09ec50 0x7fdebc130390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeb9d9b640 1 -- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 msgr2=0x7fdebc11ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeb9d9b640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 0x7fdebc11ccb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000ea20 tx=0x7fdeb00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0009600 con 0x7fdebc09f020 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: dump: 2024-09-18T12:31:48.434 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdec0d5d640 5 monclient: authenticate success, global_id 5926 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb000cc20 con 0x7fdebc09f020 2024-09-18T12:31:48.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdec0d5d640 10 monclient: _renew_subs 2024-09-18T12:31:48.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdec0d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdebc120fc0 con 0x7fdebc09f020 2024-09-18T12:31:48.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0013470 con 0x7fdebc09f020 2024-09-18T12:31:48.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:48.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeaa7fc640 10 monclient: dump: 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.437 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdec0d5d640 10 monclient: _renew_subs 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdeb8d99640 4 set_mon_vals no callback set 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdec0d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.418+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdebc11e840 con 0x7fdebc09f020 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdeb0013860 con 0x7fdebc09f020 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdec0d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdec0d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdec0d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdebc0008d0 con 0x7fdebc09f020 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdeaa7fc640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fde9c038490 0x7fde9c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7fdeb004cf20 con 0x7fdebc09f020 2024-09-18T12:31:48.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdeba59c640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fde9c038490 0x7fde9c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.419+0000 7fdeba59c640 10 monclient: get_auth_request con 0x7fde9c038490 auth_method 0 2024-09-18T12:31:48.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.422+0000 7fdeba59c640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fde9c038490 0x7fde9c03a940 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdebc1325a0 tx=0x7fdeac032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.423+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdeb0023ad0 con 0x7fdebc09f020 2024-09-18T12:31:48.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.424+0000 7fdeaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.424+0000 7fdeaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.467+0000 7f3c44a2b640 10 monclient: get_monmap_and_config 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.467+0000 7f3c44a2b640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.467+0000 7f3c44a2b640 10 monclient: monmap: 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:48.468243+0000 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:48.468243+0000 2024-09-18T12:31:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:48.469 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 1 Processor -- start 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 1 -- start start 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient: init 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3c4009c9c0 0x7f3c4009d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f3c4009c9c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c4009e340 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:48.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009e750 0x7f3c4012fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f3c4009e750 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4009bdd0 con 0x7f3c4009e340 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4009a880 con 0x7f3c4009c9c0 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4009a440 con 0x7f3c4009e750 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c44a2b640 10 monclient(hunting): authenticate will time out at 1135.789814s 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3c4009c9c0 0x7f3c4009d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009e750 0x7f3c4012fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3c4009c9c0 0x7f3c4009d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52616/0 (socket says 172.21.15.116:52616) 2024-09-18T12:31:48.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3e59c640 1 -- 172.21.15.116:0/455159897 learned_addr learned my addr 172.21.15.116:0/455159897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3e59c640 10 monclient(hunting): get_auth_request con 0x7f3c4009c9c0 auth_method 0 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3ed9d640 10 monclient(hunting): get_auth_request con 0x7f3c4009e750 auth_method 0 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.468+0000 7f3c3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 10 monclient(hunting): handle_auth_done global_id 64736 payload 274 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 1 -- 172.21.15.116:0/455159897 >> v2:172.21.15.116:3300/0 conn(0x7f3c4009c9c0 msgr2=0x7f3c4009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 1 --2- 172.21.15.116:0/455159897 >> v2:172.21.15.116:3300/0 conn(0x7f3c4009c9c0 0x7f3c4009d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 1 -- 172.21.15.116:0/455159897 >> v1:172.21.15.144:6789/0 conn(0x7f3c4009e340 legacy=0x7f3c4012f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 1 monclient: found mon.noname-b 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3ed9d640 1 -- 172.21.15.116:0/455159897 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4009ba60 con 0x7f3c4009e750 2024-09-18T12:31:48.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.469+0000 7f3c3e59c640 1 --2- 172.21.15.116:0/455159897 >> v2:172.21.15.116:3300/0 conn(0x7f3c4009c9c0 0x7f3c4009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.471+0000 7f3c3ed9d640 1 -- 172.21.15.116:0/455159897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009e750 msgr2=0x7f3c4012fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.471+0000 7f3c3ed9d640 1 --2- 172.21.15.116:0/455159897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009e750 0x7f3c4012fe80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009e60 tx=0x7f3c38009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=954f7fa0574c6a98 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 1 -- 172.21.15.116:0/455159897 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38007760 con 0x7f3c4009e750 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: dump: 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.481 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 5 monclient: authenticate success, global_id 64736 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 1 -- 172.21.15.116:0/455159897 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3802d050 con 0x7f3c4009e750 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 4 set_mon_vals no callback set 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 1 -- 172.21.15.116:0/455159897 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38007af0 con 0x7f3c4009e750 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c3dd9b640 10 monclient: dump: 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 10 monclient: shutdown 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 -- 172.21.15.116:0/455159897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009e750 msgr2=0x7f3c4012fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/455159897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009e750 0x7f3c4012fe80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009e60 tx=0x7f3c38009780 comp rx=0 tx=0).stop 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 -- 172.21.15.116:0/455159897 shutdown_connections 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/455159897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009e750 0x7f3c4012fe80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/455159897 >> v2:172.21.15.116:3300/0 conn(0x7f3c4009c9c0 0x7f3c4009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 -- 172.21.15.116:0/455159897 >> 172.21.15.116:0/455159897 conn(0x7f3c400a2e40 msgr2=0x7f3c400a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 -- 172.21.15.116:0/455159897 shutdown_connections 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.472+0000 7f3c44a2b640 1 -- 172.21.15.116:0/455159897 wait complete. 2024-09-18T12:31:48.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient: monmap: 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 1 Processor -- start 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 1 -- start start 2024-09-18T12:31:48.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient: init 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 0x7f3c4013ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f3c4009c9c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c4009e340 0x7f3c40131a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f3c4009e340 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c4009e750 0x7f3c4011c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f3c4009e750 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4011e0e0 con 0x7f3c4009e340 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.473+0000 7f3c44a2b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4011e5d0 con 0x7f3c4009e750 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c44a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c44a2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4011fe60 con 0x7f3c4009c9c0 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c44a2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c44a2b640 10 monclient(hunting): authenticate will time out at 1135.795012s 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 0x7f3c4013ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c4009e340 0x7f3c40131a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 0x7f3c4013ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33046/0 (socket says 172.21.15.116:33046) 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c4009e340 0x7f3c40131a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52628/0 (socket says 172.21.15.116:52628) 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 -- 172.21.15.116:0/1803806843 learned_addr learned my addr 172.21.15.116:0/1803806843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): get_auth_request con 0x7f3c4009c9c0 auth_method 0 2024-09-18T12:31:48.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 10 monclient(hunting): get_auth_request con 0x7f3c4009e340 auth_method 0 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3ed9d640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c4009e750 0x7f3c4011c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3ed9d640 10 monclient(hunting): get_auth_request con 0x7f3c4009e750 auth_method 0 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient(hunting): handle_auth_done global_id 64739 payload 824 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 -- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c4009e750 msgr2=0x7f3c4011c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c4009e750 0x7f3c4011c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 -- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c4009e340 msgr2=0x7f3c40131a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c4009e340 0x7f3c40131a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 monclient: found mon.noname-c 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3e59c640 1 -- 172.21.15.116:0/1803806843 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30009450 con 0x7f3c4009c9c0 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.474+0000 7f3c3ed9d640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c4009e750 0x7f3c4011c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c3e59c640 1 -- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 msgr2=0x7f3c4013ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c3e59c640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 0x7f3c4013ebf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000ecb0 tx=0x7f3c300091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 1 -- 172.21.15.116:0/1803806843 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30009730 con 0x7f3c4009c9c0 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:48.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: dump: 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c44a2b640 5 monclient: authenticate success, global_id 64739 2024-09-18T12:31:48.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 1 -- 172.21.15.116:0/1803806843 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3000cdd0 con 0x7f3c4009c9c0 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c44a2b640 10 monclient: _renew_subs 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 1 -- 172.21.15.116:0/1803806843 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c300136a0 con 0x7f3c4009c9c0 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c44a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c38009a20 con 0x7f3c4009c9c0 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c3d59a640 4 set_mon_vals no callback set 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c367fc640 10 monclient: dump: 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.492 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c44a2b640 10 monclient: _renew_subs 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c44a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.476+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4011fc30 con 0x7f3c4009c9c0 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.477+0000 7f3c367fc640 1 -- 172.21.15.116:0/1803806843 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3c30013a50 con 0x7f3c4009c9c0 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.477+0000 7f3c44a2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.477+0000 7f3c44a2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.477+0000 7f3c44a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.477+0000 7f3c367fc640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c20038490 0x7f3c2003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.477+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4011ec80 con 0x7f3c4009c9c0 2024-09-18T12:31:48.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.477+0000 7f3c367fc640 1 -- 172.21.15.116:0/1803806843 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f3c3004d0b0 con 0x7f3c4009c9c0 2024-09-18T12:31:48.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.478+0000 7f3c37fff640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c20038490 0x7f3c2003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.478+0000 7f3c37fff640 10 monclient: get_auth_request con 0x7f3c20038490 auth_method 0 2024-09-18T12:31:48.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.480+0000 7f3c37fff640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c20038490 0x7f3c2003a940 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3c28007fd0 tx=0x7f3c28007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.482+0000 7f3c367fc640 1 -- 172.21.15.116:0/1803806843 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3c300123d0 con 0x7f3c4009c9c0 2024-09-18T12:31:48.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.482+0000 7f3c367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.482+0000 7f3c367fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:48.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.584+0000 7fdec0d5d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.584+0000 7fdec0d5d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.584+0000 7fdec0d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.584+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fdebc0a13e0 con 0x7fdebc09f020 2024-09-18T12:31:48.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.585+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fdeb00232f0 con 0x7fdebc09f020 2024-09-18T12:31:48.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.585+0000 7fdeaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.585+0000 7fdeaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:48.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.683+0000 7fdec0d5d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.683+0000 7fdec0d5d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.683+0000 7fdec0d5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.683+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fdebc072640 con 0x7fdebc09f020 2024-09-18T12:31:48.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.684+0000 7fdeaa7fc640 1 -- 172.21.15.144:0/3022042275 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fdeb00234d0 con 0x7fdebc09f020 2024-09-18T12:31:48.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.684+0000 7fdeaa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.684+0000 7fdeaa7fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:48.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fde9c038490 msgr2=0x7fde9c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fde9c038490 0x7fde9c03a940 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdebc1325a0 tx=0x7fdeac032000 comp rx=0 tx=0).stop 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 10 monclient: shutdown 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 msgr2=0x7fdebc11ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 0x7fdebc11ccb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000ea20 tx=0x7fdeb00090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 shutdown_connections 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fde9c038490 0x7fde9c03a940 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdebc1321a0 0x7fdebc11d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdebc09f020 0x7fdebc11ccb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 --2- 172.21.15.144:0/3022042275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdebc09ec50 0x7fdebc130390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 >> 172.21.15.144:0/3022042275 conn(0x7fdebc0a5cf0 msgr2=0x7fdebc131150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 shutdown_connections 2024-09-18T12:31:48.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.686+0000 7fdec0d5d640 1 -- 172.21.15.144:0/3022042275 wait complete. 2024-09-18T12:31:48.696 INFO:tasks.workunit.client.0.smithi144.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:31:48.696 INFO:tasks.workunit.client.0.smithi144.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:31:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.695+0000 7f3c44a2b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:31:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.696+0000 7f3c44a2b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:31:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.696+0000 7f3c44a2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.696+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3c4004bf60 con 0x7f3c4009c9c0 2024-09-18T12:31:48.698 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:31:48.698 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:31:48.698 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep fish 2024-09-18T12:31:48.698 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '12\.' 2024-09-18T12:31:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f3c367fc640 1 -- 172.21.15.116:0/1803806843 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3c30011e00 con 0x7f3c4009c9c0 2024-09-18T12:31:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f3c367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:31:48.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f3c367fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:48.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=2592169029,config=3140514009,health=1032194209,kv=2187923472,logm=2513913032}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=9,logm=62} crc {auth=2921565801,config=3140514009,health=1032194209,kv=2187923472,logm=3309520787}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=2592169029,config=3140514009,health=1032194209,kv=2187923472,logm=2513913032}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.697+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1331010701,config=3140514009,health=1032194209,kv=2187923472,logm=1296489187}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.698+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.698+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354133509}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.698+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489106003}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.698+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.698+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354133509}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.698+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2154134612}) 2024-09-18T12:31:48.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609105093}) 2024-09-18T12:31:48.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161467901}) 2024-09-18T12:31:48.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609105093}) 2024-09-18T12:31:48.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438837567}) 2024-09-18T12:31:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c20038490 msgr2=0x7f3c2003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c20038490 0x7f3c2003a940 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3c28007fd0 tx=0x7f3c28007e10 comp rx=0 tx=0).stop 2024-09-18T12:31:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 10 monclient: shutdown 2024-09-18T12:31:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 msgr2=0x7f3c4013ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 0x7f3c4013ebf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000ecb0 tx=0x7f3c300091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 shutdown_connections 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c20038490 0x7f3c2003a940 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3c4009e750 0x7f3c4011c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c4009e340 0x7f3c40131a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 --2- 172.21.15.116:0/1803806843 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3c4009c9c0 0x7f3c4013ebf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 >> 172.21.15.116:0/1803806843 conn(0x7f3c400a2e40 msgr2=0x7f3c400a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 shutdown_connections 2024-09-18T12:31:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f3c44a2b640 1 -- 172.21.15.116:0/1803806843 wait complete. 2024-09-18T12:31:48.712 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.712 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586592520}) 2024-09-18T12:31:48.712 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.699+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583497999}) 2024-09-18T12:31:48.712 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.712 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586592520}) 2024-09-18T12:31:48.712 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=725220668}) 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206046211}) 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1559285096}) 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206046211}) 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.700+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626189032}) 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.713 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174844129}) 2024-09-18T12:31:48.714 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162540813}) 2024-09-18T12:31:48.714 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.714 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174844129}) 2024-09-18T12:31:48.714 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949227856}) 2024-09-18T12:31:48.714 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.714 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238364672}) 2024-09-18T12:31:48.715 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1585158577}) 2024-09-18T12:31:48.715 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.715 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238364672}) 2024-09-18T12:31:48.715 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.701+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3464502761}) 2024-09-18T12:31:48.715 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.702+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.715 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.702+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521994625}) 2024-09-18T12:31:48.716 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.702+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413920875}) 2024-09-18T12:31:48.716 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.702+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.716 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.702+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521994625}) 2024-09-18T12:31:48.716 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.702+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3272341163}) 2024-09-18T12:31:48.716 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.703+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.703+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016549674}) 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.703+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670529283}) 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.703+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.703+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016549674}) 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.703+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2360935818}) 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114005448}) 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4261646631}) 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114005448}) 2024-09-18T12:31:48.717 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454186293}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404775373}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839841791}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404775373}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.704+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887520103}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.705+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.705+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039850335}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.705+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864786805}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.705+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.705+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039850335}) 2024-09-18T12:31:48.718 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.705+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1135430813}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.706+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.706+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72699601}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.706+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1776293877}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.706+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.706+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72699601}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.706+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3833753826}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2566645184}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189244628}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2566645184}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3369032857}) 2024-09-18T12:31:48.719 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334627335}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074643939}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334627335}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.707+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814731901}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938509050}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2160073488}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938509050}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3789465172}) 2024-09-18T12:31:48.720 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2975352382}) 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107203890}) 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2975352382}) 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.708+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2702143717}) 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.709+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.709+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811752897}) 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.709+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341585100}) 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.709+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.709+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811752897}) 2024-09-18T12:31:48.721 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.709+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2026798208}) 2024-09-18T12:31:48.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:31:48.727 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.710+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.727 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.710+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937123616}) 2024-09-18T12:31:48.727 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.710+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817621853}) 2024-09-18T12:31:48.727 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.710+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.727 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.710+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937123616}) 2024-09-18T12:31:48.727 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.710+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3879785932}) 2024-09-18T12:31:48.727 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079349483}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199325614}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079349483}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3513648764}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283425236}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310119541}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283425236}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.711+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2476776081}) 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.728 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301160005}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1658703709}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301160005}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=724847051}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056780477}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633753134}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056780477}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.712+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2474316607}) 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.729 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955382799}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4031614268}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955382799}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1070965211}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552948669}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=857067678}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552948669}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.713+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1165894377}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.714+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.714+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=4215406953,mdsmap=1531151339,mgr=3608348820}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.714+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=930776034,mdsmap=1531151339,mgr=1624230820}) 2024-09-18T12:31:48.730 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.714+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.714+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=4215406953,mdsmap=1531151339,mgr=3608348820}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.714+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=1301700111,mdsmap=1531151339,mgr=2494729330}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.724+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.724+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=232604827,mon_config_key=565504783,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=1459412894,osd_snap=2033388227,osdmap=2544211877}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.724+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=264798453,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=1459412894,osd_snap=2033388227,osdmap=481945804}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.724+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.724+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=232604827,mon_config_key=565504783,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=1459412894,osd_snap=2033388227,osdmap=2544211877}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.724+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=69} crc {mgr=2586702516,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=154515914,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=1459412894,osd_snap=553062916,osdmap=103685985}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.725+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.725+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3642661060}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.725+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2345104745}) 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.725+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.731 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.725+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3642661060}) 2024-09-18T12:31:48.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.725+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2727888413}) 2024-09-18T12:31:48.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.728+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.728+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=3207843889}) 2024-09-18T12:31:48.733 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.728+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=3101657908}) 2024-09-18T12:31:48.733 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.728+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:31:48.733 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.728+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=3207843889}) 2024-09-18T12:31:48.733 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:31:48.728+0000 7f71091f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=335014913}) 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.770+0000 7fbc73a57640 10 monclient: get_monmap_and_config 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.770+0000 7fbc73a57640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.770+0000 7fbc73a57640 10 monclient: monmap: 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:48.771486+0000 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:48.771486+0000 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:48.772 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:48.773 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:48.773 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:48.773 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 1 Processor -- start 2024-09-18T12:31:48.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 1 -- start start 2024-09-18T12:31:48.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient: init 2024-09-18T12:31:48.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:31:48.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc6c06e1f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:48.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbc6c06e5c0 0x7fbc6c06ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient(hunting): picked mon.noname-a con 0x7fbc6c06e5c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc6c09e170 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c06a0d0 con 0x7fbc6c09e170 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c069e30 con 0x7fbc6c06e1f0 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6c099e30 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc73a57640 10 monclient(hunting): authenticate will time out at 1144.340835s 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc717f3640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fbc6c06e1f0 0x7fbc6c09da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:59304/0 (socket says 172.21.15.144:59304) 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.771+0000 7fbc717f3640 1 -- 172.21.15.144:0/1785286265 learned_addr learned my addr 172.21.15.144:0/1785286265 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 1 --2- 172.21.15.144:0/1785286265 >> v2:172.21.15.116:3300/0 conn(0x7fbc6c06e5c0 0x7fbc6c06ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): get_auth_request con 0x7fbc6c06e5c0 auth_method 0 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_done global_id 94143 payload 274 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 1 -- 172.21.15.144:0/1785286265 >> v1:172.21.15.144:6789/0 conn(0x7fbc6c06e1f0 legacy=0x7fbc6c09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 1 -- 172.21.15.144:0/1785286265 >> v1:172.21.15.116:6789/0 conn(0x7fbc6c09e170 legacy=0x7fbc6c09e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 1 monclient: found mon.noname-a 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:48.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.772+0000 7fbc70ff2640 1 -- 172.21.15.144:0/1785286265 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c099f70 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc70ff2640 1 -- 172.21.15.144:0/1785286265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c06e5c0 msgr2=0x7fbc6c06ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc70ff2640 1 --2- 172.21.15.144:0/1785286265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c06ad90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009850 tx=0x7fbc5c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860db0b9c166572b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 1 -- 172.21.15.144:0/1785286265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c032440 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:48.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: dump: 2024-09-18T12:31:48.783 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.784 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 1 -- 172.21.15.144:0/1785286265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c0325a0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc73a57640 5 monclient: authenticate success, global_id 94143 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc73a57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc73a57640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 1 -- 172.21.15.144:0/1785286265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c032940 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc73a57640 4 set_mon_vals no callback set 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.775+0000 7fbc63fff640 10 monclient: dump: 2024-09-18T12:31:48.785 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 10 monclient: shutdown 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 -- 172.21.15.144:0/1785286265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c06e5c0 msgr2=0x7fbc6c06ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 --2- 172.21.15.144:0/1785286265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c06ad90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009850 tx=0x7fbc5c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.786 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 -- 172.21.15.144:0/1785286265 shutdown_connections 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 --2- 172.21.15.144:0/1785286265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c06ad90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 -- 172.21.15.144:0/1785286265 >> 172.21.15.144:0/1785286265 conn(0x7fbc6c0a3070 msgr2=0x7fbc6c0a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 -- 172.21.15.144:0/1785286265 shutdown_connections 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 -- 172.21.15.144:0/1785286265 wait complete. 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 10 monclient: build_initial_monmap 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 10 monclient: monmap: 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:48.787 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 Processor -- start 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 1 -- start start 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.776+0000 7fbc73a57640 10 monclient: init 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc6c06e1f0 0x7fbc6c1411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): picked mon.noname-c con 0x7fbc6c06e1f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c133ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): picked mon.noname-b con 0x7fbc6c06e5c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c09e170 0x7fbc6c11dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): picked mon.noname-a con 0x7fbc6c09e170 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c1215a0 con 0x7fbc6c09e170 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c06d2b0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c120fe0 con 0x7fbc6c06e1f0 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc73a57640 10 monclient(hunting): authenticate will time out at 1144.346583s 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c133ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c133ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35454/0 (socket says 172.21.15.144:35454) 2024-09-18T12:31:48.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 1 -- 172.21.15.144:0/304110831 learned_addr learned my addr 172.21.15.144:0/304110831 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): get_auth_request con 0x7fbc6c06e5c0 auth_method 0 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc6c06e1f0 0x7fbc6c1411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 10 monclient(hunting): get_auth_request con 0x7fbc6c06e1f0 auth_method 0 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c09e170 0x7fbc6c11dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 10 monclient(hunting): get_auth_request con 0x7fbc6c09e170 auth_method 0 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc717f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc71ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 10 monclient(hunting): handle_auth_done global_id 5932 payload 824 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 1 -- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc6c06e1f0 msgr2=0x7fbc6c1411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.777+0000 7fbc70ff2640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc6c06e1f0 0x7fbc6c1411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.778+0000 7fbc70ff2640 1 -- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c09e170 msgr2=0x7fbc6c11dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:48.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.778+0000 7fbc70ff2640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c09e170 0x7fbc6c11dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.778+0000 7fbc70ff2640 10 monclient: _finish_hunting 0 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.778+0000 7fbc70ff2640 1 monclient: found mon.noname-b 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.778+0000 7fbc70ff2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.778+0000 7fbc70ff2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.778+0000 7fbc70ff2640 1 -- 172.21.15.144:0/304110831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c009390 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc70ff2640 1 -- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 msgr2=0x7fbc6c133ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc70ff2640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c133ec0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009850 tx=0x7fbc5c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c031940 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: dump: 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.792 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: _finish_auth 0 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc73a57640 5 monclient: authenticate success, global_id 5932 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c031aa0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:48.793 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c031e60 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc73a57640 10 monclient: _renew_subs 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc73a57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6c1212c0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc637fe640 4 set_mon_vals no callback set 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc617fa640 10 monclient: dump: 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.794 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc73a57640 10 monclient: _renew_subs 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc73a57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.780+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6c1202e0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbc5c039050 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc73a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc73a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc73a57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6c0008d0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc617fa640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc50038490 0x7fbc5003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7fbc5c06d2e0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc717f3640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc50038490 0x7fbc5003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.781+0000 7fbc717f3640 10 monclient: get_auth_request con 0x7fbc50038490 auth_method 0 2024-09-18T12:31:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.783+0000 7fbc717f3640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc50038490 0x7fbc5003a940 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc68007fd0 tx=0x7fbc68007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:48.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.786+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbc5c030340 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.786+0000 7fbc617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:48.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.786+0000 7fbc617fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:48.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.945+0000 7fbc73a57640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.945+0000 7fbc73a57640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.945+0000 7fbc73a57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:48.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.945+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fbc6c0008d0 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.946+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fbc5c040070 con 0x7fbc6c06e5c0 2024-09-18T12:31:48.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.946+0000 7fbc617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:48.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:48.946+0000 7fbc617fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:49.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.046+0000 7fbc73a57640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:49.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.047+0000 7fbc73a57640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:49.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.047+0000 7fbc73a57640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.047+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fbc6c0008d0 con 0x7fbc6c06e5c0 2024-09-18T12:31:49.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.047+0000 7fbc617fa640 1 -- 172.21.15.144:0/304110831 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fbc5c037030 con 0x7fbc6c06e5c0 2024-09-18T12:31:49.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.047+0000 7fbc617fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:31:49.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.047+0000 7fbc617fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:31:49.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc50038490 msgr2=0x7fbc5003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc50038490 0x7fbc5003a940 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc68007fd0 tx=0x7fbc68007f00 comp rx=0 tx=0).stop 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 10 monclient: shutdown 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 msgr2=0x7fbc6c133ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c133ec0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009850 tx=0x7fbc5c032000 comp rx=0 tx=0).stop 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 shutdown_connections 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbc50038490 0x7fbc5003a940 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6c09e170 0x7fbc6c11dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc6c06e5c0 0x7fbc6c133ec0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 --2- 172.21.15.144:0/304110831 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc6c06e1f0 0x7fbc6c1411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 >> 172.21.15.144:0/304110831 conn(0x7fbc6c0a3070 msgr2=0x7fbc6c06d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 shutdown_connections 2024-09-18T12:31:49.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.049+0000 7fbc73a57640 1 -- 172.21.15.144:0/304110831 wait complete. 2024-09-18T12:31:49.059 INFO:tasks.workunit.client.0.smithi144.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:31:49.059 INFO:tasks.workunit.client.0.smithi144.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:31:49.059 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-18T12:31:49.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.131+0000 7f84a7ac9640 10 monclient: get_monmap_and_config 2024-09-18T12:31:49.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.131+0000 7f84a7ac9640 10 monclient: build_initial_monmap 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.131+0000 7f84a7ac9640 10 monclient: monmap: 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:31:49.132452+0000 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:31:49.132452+0000 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:49.133 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 1 Processor -- start 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 1 -- start start 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient: init 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84a0069a30 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a009baa0 0x7f84a009bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f84a009baa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f84a0134b70 0x7f84a0136f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f84a0134b70 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a009e550 con 0x7f84a0069a30 2024-09-18T12:31:49.136 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a012ff50 con 0x7f84a0134b70 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0069e00 con 0x7f84a009baa0 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a7ac9640 10 monclient(hunting): authenticate will time out at 1144.701822s 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a6066640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f84a0134b70 0x7f84a0136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a6066640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f84a0134b70 0x7f84a0136f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35468/0 (socket says 172.21.15.144:35468) 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a6066640 1 -- 172.21.15.144:0/785777304 learned_addr learned my addr 172.21.15.144:0/785777304 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a6066640 10 monclient(hunting): get_auth_request con 0x7f84a0134b70 auth_method 0 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a6066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a6066640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:49.137 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.132+0000 7f84a6066640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a5064640 1 --2- 172.21.15.144:0/785777304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a009baa0 0x7f84a009bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a5064640 10 monclient(hunting): get_auth_request con 0x7f84a009baa0 auth_method 0 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a5064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a5064640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a5064640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 10 monclient(hunting): handle_auth_done global_id 5935 payload 274 2024-09-18T12:31:49.138 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 1 -- 172.21.15.144:0/785777304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a009baa0 msgr2=0x7f84a009bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:49.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 1 --2- 172.21.15.144:0/785777304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a009baa0 0x7f84a009bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 1 -- 172.21.15.144:0/785777304 >> v1:172.21.15.116:6789/0 conn(0x7f84a0069a30 legacy=0x7f84a012f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:49.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 10 monclient: _finish_hunting 0 2024-09-18T12:31:49.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 1 monclient: found mon.noname-c 2024-09-18T12:31:49.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:49.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.133+0000 7f84a6066640 1 -- 172.21.15.144:0/785777304 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a009d100 con 0x7f84a0134b70 2024-09-18T12:31:49.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a6066640 1 -- 172.21.15.144:0/785777304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0134b70 msgr2=0x7f84a0136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a6066640 1 --2- 172.21.15.144:0/785777304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0134b70 0x7f84a0136f50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f84a0099060 tx=0x7f8498009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f0edb4e1485f84c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 1 -- 172.21.15.144:0/785777304 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498007610 con 0x7f84a0134b70 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: dump: 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.147 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: _finish_auth 0 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 5 monclient: authenticate success, global_id 5935 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 1 -- 172.21.15.144:0/785777304 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849802d050 con 0x7f84a0134b70 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 4 set_mon_vals no callback set 2024-09-18T12:31:49.148 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 1 -- 172.21.15.144:0/785777304 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84980079a0 con 0x7f84a0134b70 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a4863640 10 monclient: dump: 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.149 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 10 monclient: shutdown 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 -- 172.21.15.144:0/785777304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0134b70 msgr2=0x7f84a0136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/785777304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0134b70 0x7f84a0136f50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f84a0099060 tx=0x7f8498009840 comp rx=0 tx=0).stop 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 -- 172.21.15.144:0/785777304 shutdown_connections 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/785777304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0134b70 0x7f84a0136f50 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/785777304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a009baa0 0x7f84a009bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 -- 172.21.15.144:0/785777304 >> 172.21.15.144:0/785777304 conn(0x7f84a009e9f0 msgr2=0x7f84a009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 -- 172.21.15.144:0/785777304 shutdown_connections 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.136+0000 7f84a7ac9640 1 -- 172.21.15.144:0/785777304 wait complete. 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.137+0000 7f84a7ac9640 10 monclient: build_initial_monmap 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.137+0000 7f84a7ac9640 10 monclient: monmap: 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:49.151 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.137+0000 7f84a7ac9640 1 Processor -- start 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.137+0000 7f84a7ac9640 1 -- start start 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.137+0000 7f84a7ac9640 10 monclient: init 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 0x7f84a012d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f84a0069a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a009baa0 0x7f84a012d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f84a009baa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a0134b70 0x7f84a0096ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f84a0134b70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:49.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0133ea0 con 0x7f84a009baa0 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a009dea0 con 0x7f84a0069a30 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0133bc0 con 0x7f84a0134b70 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a7ac9640 10 monclient(hunting): authenticate will time out at 1144.707114s 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 0x7f84a012d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 0x7f84a012d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:35476/0 (socket says 172.21.15.144:35476) 2024-09-18T12:31:49.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 -- 172.21.15.144:0/824612391 learned_addr learned my addr 172.21.15.144:0/824612391 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:31:49.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): get_auth_request con 0x7f84a0069a30 auth_method 0 2024-09-18T12:31:49.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:49.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:49.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:49.154 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:49.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:49.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:49.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5064640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a009baa0 0x7f84a012d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient(hunting): handle_auth_done global_id 5938 payload 824 2024-09-18T12:31:49.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 -- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a0134b70 msgr2=0x7f84a0096ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a6066640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a0134b70 0x7f84a0096ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a0134b70 0x7f84a0096ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 -- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a009baa0 msgr2=0x7f84a012d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a009baa0 0x7f84a012d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient: _finish_hunting 0 2024-09-18T12:31:49.156 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 monclient: found mon.noname-b 2024-09-18T12:31:49.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:49.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5865640 1 -- 172.21.15.144:0/824612391 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8498009ae0 con 0x7f84a0069a30 2024-09-18T12:31:49.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5064640 10 monclient: get_auth_request con 0x7f84a009baa0 auth_method 0 2024-09-18T12:31:49.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.138+0000 7f84a5064640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a009baa0 0x7f84a012d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:49.157 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f84a5865640 1 -- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 msgr2=0x7f84a012d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:49.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f84a5865640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 0x7f84a012d200 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f849c00c5e0 tx=0x7f849c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:49.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c012920 con 0x7f84a0069a30 2024-09-18T12:31:49.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:49.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: dump: 2024-09-18T12:31:49.158 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:49.159 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.159 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.159 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.159 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.159 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:49.159 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.160 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.160 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.160 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:49.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: _finish_auth 0 2024-09-18T12:31:49.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:49.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849c012a80 con 0x7f84a0069a30 2024-09-18T12:31:49.160 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f84a7ac9640 5 monclient: authenticate success, global_id 5938 2024-09-18T12:31:49.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f84a7ac9640 10 monclient: _renew_subs 2024-09-18T12:31:49.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f84a7ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a0132a50 con 0x7f84a0069a30 2024-09-18T12:31:49.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:49.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849c012e80 con 0x7f84a0069a30 2024-09-18T12:31:49.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:49.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8495ffb640 10 monclient: dump: 2024-09-18T12:31:49.162 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:31:49.162 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.162 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.162 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:31:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.163 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:31:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.141+0000 7f84a7ac9640 10 monclient: _renew_subs 2024-09-18T12:31:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.140+0000 7f8497fff640 4 set_mon_vals no callback set 2024-09-18T12:31:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.141+0000 7f84a7ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.141+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a01345a0 con 0x7f84a0069a30 2024-09-18T12:31:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.141+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f849c011590 con 0x7f84a0069a30 2024-09-18T12:31:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f84a7ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f84a7ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f84a7ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a00008d0 con 0x7f84a0069a30 2024-09-18T12:31:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f8495ffb640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8484038490 0x7f848403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f849c04cf40 con 0x7f84a0069a30 2024-09-18T12:31:49.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f84a5064640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8484038490 0x7f848403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.142+0000 7f84a5064640 10 monclient: get_auth_request con 0x7f8484038490 auth_method 0 2024-09-18T12:31:49.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.144+0000 7f84a5064640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8484038490 0x7f848403a940 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f84900099b0 tx=0x7f8490008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:49.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.146+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f849c010920 con 0x7f84a0069a30 2024-09-18T12:31:49.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.146+0000 7f8495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:49.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.146+0000 7f8495ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:49.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.304+0000 7f84a7ac9640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:49.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.304+0000 7f84a7ac9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:49.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.304+0000 7f84a7ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:49.304+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f84a009fe40 con 0x7f84a0069a30 2024-09-18T12:31:49.724 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:31:49.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:49.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.827+0000 7f6b05100640 10 monclient: get_monmap_and_config 2024-09-18T12:31:49.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.827+0000 7f6b05100640 10 monclient: build_initial_monmap 2024-09-18T12:31:49.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.827+0000 7f6b05100640 10 monclient: monmap: 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:49.827960+0000 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:49.827960+0000 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:49.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.827+0000 7f6b05100640 1 Processor -- start 2024-09-18T12:31:49.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.827+0000 7f6b05100640 1 -- start start 2024-09-18T12:31:49.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.827+0000 7f6b05100640 10 monclient: init 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b00131980 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b0009fe10 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b0009f140 0x7f6b00139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient(hunting): picked mon.noname-c con 0x7f6b0009f140 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000a14d0 con 0x7f6b0009fe10 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b001325b0 con 0x7f6b00131980 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0009f560 con 0x7f6b0009f140 2024-09-18T12:31:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6b05100640 10 monclient(hunting): authenticate will time out at 1137.149492s 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b0009f140 0x7f6b00139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6afe59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6b0009fe10 0x7f6b0009ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54688/0 (socket says 172.21.15.116:54688) 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6afe59c640 1 -- 172.21.15.116:0/1007095116 learned_addr learned my addr 172.21.15.116:0/1007095116 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): get_auth_request con 0x7f6b0009f140 auth_method 0 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient(hunting): handle_auth_done global_id 5941 payload 274 2024-09-18T12:31:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 1 -- 172.21.15.116:0/1007095116 >> v1:172.21.15.144:6789/0 conn(0x7f6b00131980 legacy=0x7f6b00131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 1 -- 172.21.15.116:0/1007095116 >> v1:172.21.15.116:6789/0 conn(0x7f6b0009fe10 legacy=0x7f6b0009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient: _finish_hunting 0 2024-09-18T12:31:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 1 monclient: found mon.noname-c 2024-09-18T12:31:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.828+0000 7f6aff59e640 1 -- 172.21.15.116:0/1007095116 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000a11f0 con 0x7f6b0009f140 2024-09-18T12:31:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6aff59e640 1 -- 172.21.15.116:0/1007095116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 msgr2=0x7f6b00139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6aff59e640 1 --2- 172.21.15.116:0/1007095116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 0x7f6b00139cd0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009e60 tx=0x7f6af8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab6c855c959f9f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 1 -- 172.21.15.116:0/1007095116 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8007840 con 0x7f6b0009f140 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: dump: 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: _finish_auth 0 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 1 -- 172.21.15.116:0/1007095116 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af802d050 con 0x7f6b0009f140 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 5 monclient: authenticate success, global_id 5941 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 4 set_mon_vals no callback set 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 1 -- 172.21.15.116:0/1007095116 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8007bd0 con 0x7f6b0009f140 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6afdd9b640 10 monclient: dump: 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 10 monclient: shutdown 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 1 -- 172.21.15.116:0/1007095116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 msgr2=0x7f6b00139cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 1 --2- 172.21.15.116:0/1007095116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 0x7f6b00139cd0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009e60 tx=0x7f6af8009780 comp rx=0 tx=0).stop 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 1 -- 172.21.15.116:0/1007095116 shutdown_connections 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 1 --2- 172.21.15.116:0/1007095116 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 0x7f6b00139cd0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 1 -- 172.21.15.116:0/1007095116 >> 172.21.15.116:0/1007095116 conn(0x7f6b000a5050 msgr2=0x7f6b000a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 1 -- 172.21.15.116:0/1007095116 shutdown_connections 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.832+0000 7f6b05100640 1 -- 172.21.15.116:0/1007095116 wait complete. 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.833+0000 7f6b05100640 10 monclient: build_initial_monmap 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.833+0000 7f6b05100640 10 monclient: monmap: 2024-09-18T12:31:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.833+0000 7f6b05100640 1 Processor -- start 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.833+0000 7f6b05100640 1 -- start start 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.833+0000 7f6b05100640 10 monclient: init 2024-09-18T12:31:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 0x7f6b001384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): picked mon.noname-b con 0x7f6b0009f140 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 0x7f6b001341a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): picked mon.noname-a con 0x7f6b0009fe10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b00131980 0x7f6b001346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): picked mon.noname-c con 0x7f6b00131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00135610 con 0x7f6b0009fe10 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000a4130 con 0x7f6b0009f140 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00137af0 con 0x7f6b00131980 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6b05100640 10 monclient(hunting): authenticate will time out at 1137.155249s 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 0x7f6b001341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b00131980 0x7f6b001346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b00131980 0x7f6b001346e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33062/0 (socket says 172.21.15.116:33062) 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 0x7f6b001341a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52652/0 (socket says 172.21.15.116:52652) 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 1 -- 172.21.15.116:0/352845289 learned_addr learned my addr 172.21.15.116:0/352845289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 10 monclient(hunting): get_auth_request con 0x7f6b00131980 auth_method 0 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): get_auth_request con 0x7f6b0009fe10 auth_method 0 2024-09-18T12:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 0x7f6b001384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 10 monclient(hunting): get_auth_request con 0x7f6b0009f140 auth_method 0 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient(hunting): handle_auth_done global_id 94149 payload 824 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 -- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b00131980 msgr2=0x7f6b001346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b00131980 0x7f6b001346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 -- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 msgr2=0x7f6b001384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 0x7f6b001384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient: _finish_hunting 0 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 monclient: found mon.noname-a 2024-09-18T12:31:49.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6afe59c640 1 -- 172.21.15.116:0/352845289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af8009a20 con 0x7f6b0009fe10 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.834+0000 7f6aff59e640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b00131980 0x7f6b001346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.836+0000 7f6afe59c640 1 -- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 msgr2=0x7f6b001341a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.836+0000 7f6afe59c640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 0x7f6b001341a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800c490 tx=0x7f6ae8007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 1 -- 172.21.15.116:0/352845289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8005cb0 con 0x7f6b0009fe10 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: dump: 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 1 -- 172.21.15.116:0/352845289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8005e30 con 0x7f6b0009fe10 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 5 monclient: authenticate success, global_id 94149 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 10 monclient: _renew_subs 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:49.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b00135d40 con 0x7f6b0009fe10 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 1 -- 172.21.15.116:0/352845289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8007320 con 0x7f6b0009fe10 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 10 monclient: dump: 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:49.852 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 10 monclient: _renew_subs 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b00136a50 con 0x7f6b0009fe10 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6afd59a640 4 set_mon_vals no callback set 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6af6ffd640 1 -- 172.21.15.116:0/352845289 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6ae801b3b0 con 0x7f6b0009fe10 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.837+0000 7f6b05100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.838+0000 7f6b05100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.838+0000 7f6b05100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.838+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0004bf60 con 0x7f6b0009fe10 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.838+0000 7f6af6ffd640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ae0038490 0x7f6ae003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:49.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.838+0000 7f6af6ffd640 1 -- 172.21.15.116:0/352845289 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f6ae804c0d0 con 0x7f6b0009fe10 2024-09-18T12:31:49.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.838+0000 7f6afed9d640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ae0038490 0x7f6ae003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:49.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.838+0000 7f6afed9d640 10 monclient: get_auth_request con 0x7f6ae0038490 auth_method 0 2024-09-18T12:31:49.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.840+0000 7f6afed9d640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ae0038490 0x7f6ae003a940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6af0007fd0 tx=0x7f6af0007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:49.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.842+0000 7f6af6ffd640 1 -- 172.21.15.116:0/352845289 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6ae80187f0 con 0x7f6b0009fe10 2024-09-18T12:31:49.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.842+0000 7f6af6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:49.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:49.842+0000 7f6af6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:50.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.059+0000 7f6b05100640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:50.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.059+0000 7f6b05100640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:50.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.059+0000 7f6b05100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.059+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6b00067f00 con 0x7f6b0009fe10 2024-09-18T12:31:50.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.059+0000 7f6af6ffd640 1 -- 172.21.15.116:0/352845289 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f6ae8017020 con 0x7f6b0009fe10 2024-09-18T12:31:50.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.059+0000 7f6af6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:50.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.059+0000 7f6af6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:50.061 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:50.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ae0038490 msgr2=0x7f6ae003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ae0038490 0x7f6ae003a940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6af0007fd0 tx=0x7f6af0007e10 comp rx=0 tx=0).stop 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 10 monclient: shutdown 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 msgr2=0x7f6b001341a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 0x7f6b001341a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800c490 tx=0x7f6ae8007f00 comp rx=0 tx=0).stop 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 shutdown_connections 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ae0038490 0x7f6ae003a940 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b00131980 0x7f6b001346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b0009fe10 0x7f6b001341a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 --2- 172.21.15.116:0/352845289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b0009f140 0x7f6b001384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 >> 172.21.15.116:0/352845289 conn(0x7f6b000a5050 msgr2=0x7f6b0009e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 shutdown_connections 2024-09-18T12:31:50.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.061+0000 7f6b05100640 1 -- 172.21.15.116:0/352845289 wait complete. 2024-09-18T12:31:50.073 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:31:50.073 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:31:50.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:31:50.073 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:31:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.165+0000 7f5cdaa60640 10 monclient: get_monmap_and_config 2024-09-18T12:31:50.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.165+0000 7f5cdaa60640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.165+0000 7f5cdaa60640 10 monclient: monmap: 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:50.166235+0000 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:50.166235+0000 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:50.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 1 Processor -- start 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 1 -- start start 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient: init 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5cd409c820 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5cd4097ff0 0x7f5cd4134c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient(hunting): picked mon.noname-a con 0x7f5cd4097ff0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5cd409d300 0x7f5cd4137190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd409d300 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd409e760 con 0x7f5cd409c820 2024-09-18T12:31:50.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd4097500 con 0x7f5cd4097ff0 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd4097120 con 0x7f5cd409d300 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cdaa60640 10 monclient(hunting): authenticate will time out at 1137.487875s 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.166+0000 7f5cd37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5cd4097ff0 0x7f5cd4134c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5cd4097ff0 0x7f5cd4134c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52660/0 (socket says 172.21.15.116:52660) 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 1 -- 172.21.15.116:0/4251166075 learned_addr learned my addr 172.21.15.116:0/4251166075 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): get_auth_request con 0x7f5cd4097ff0 auth_method 0 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 1 --2- 172.21.15.116:0/4251166075 >> v2:172.21.15.144:3300/0 conn(0x7f5cd409d300 0x7f5cd4137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 10 monclient(hunting): get_auth_request con 0x7f5cd409d300 auth_method 0 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd8ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_done global_id 94152 payload 274 2024-09-18T12:31:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 1 -- 172.21.15.116:0/4251166075 >> v2:172.21.15.144:3300/0 conn(0x7f5cd409d300 msgr2=0x7f5cd4137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 1 --2- 172.21.15.116:0/4251166075 >> v2:172.21.15.144:3300/0 conn(0x7f5cd409d300 0x7f5cd4137190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 1 -- 172.21.15.116:0/4251166075 >> v1:172.21.15.144:6789/0 conn(0x7f5cd409c820 legacy=0x7f5cd409cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 1 monclient: found mon.noname-a 2024-09-18T12:31:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.167+0000 7f5cd37fe640 1 -- 172.21.15.116:0/4251166075 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4096e40 con 0x7f5cd4097ff0 2024-09-18T12:31:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd37fe640 1 -- 172.21.15.116:0/4251166075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd4097ff0 msgr2=0x7f5cd4134c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd37fe640 1 --2- 172.21.15.116:0/4251166075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd4097ff0 0x7f5cd4134c40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009910 tx=0x7f5cc80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95646c4f1276209e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 1 -- 172.21.15.116:0/4251166075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc802aea0 con 0x7f5cd4097ff0 2024-09-18T12:31:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: dump: 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 1 -- 172.21.15.116:0/4251166075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc8032440 con 0x7f5cd4097ff0 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 5 monclient: authenticate success, global_id 94152 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 4 set_mon_vals no callback set 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 1 -- 172.21.15.116:0/4251166075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8032820 con 0x7f5cd4097ff0 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cd2ffd640 10 monclient: dump: 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.182 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 10 monclient: shutdown 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 -- 172.21.15.116:0/4251166075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd4097ff0 msgr2=0x7f5cd4134c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 --2- 172.21.15.116:0/4251166075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd4097ff0 0x7f5cd4134c40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009910 tx=0x7f5cc80091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 -- 172.21.15.116:0/4251166075 shutdown_connections 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 --2- 172.21.15.116:0/4251166075 >> v2:172.21.15.144:3300/0 conn(0x7f5cd409d300 0x7f5cd4137190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 --2- 172.21.15.116:0/4251166075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd4097ff0 0x7f5cd4134c40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 -- 172.21.15.116:0/4251166075 >> 172.21.15.116:0/4251166075 conn(0x7f5cd40a1e30 msgr2=0x7f5cd40a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 -- 172.21.15.116:0/4251166075 shutdown_connections 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.170+0000 7f5cdaa60640 1 -- 172.21.15.116:0/4251166075 wait complete. 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.171+0000 7f5cdaa60640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.171+0000 7f5cdaa60640 10 monclient: monmap: 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:50.184 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.171+0000 7f5cdaa60640 1 Processor -- start 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.171+0000 7f5cdaa60640 1 -- start start 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.171+0000 7f5cdaa60640 10 monclient: init 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cd4097ff0 0x7f5cd413f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): picked mon.noname-b con 0x7f5cd4097ff0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd409c820 0x7f5cd412ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): picked mon.noname-a con 0x7f5cd409c820 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cd409d300 0x7f5cd406d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd409d300 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd409b870 con 0x7f5cd409c820 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd411be10 con 0x7f5cd4097ff0 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd413ff60 con 0x7f5cd409d300 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cdaa60640 10 monclient(hunting): authenticate will time out at 1137.493274s 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd409c820 0x7f5cd412ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd409c820 0x7f5cd412ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52676/0 (socket says 172.21.15.116:52676) 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 -- 172.21.15.116:0/3255010841 learned_addr learned my addr 172.21.15.116:0/3255010841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): get_auth_request con 0x7f5cd409c820 auth_method 0 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cd409d300 0x7f5cd406d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 10 monclient(hunting): get_auth_request con 0x7f5cd409d300 auth_method 0 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd8ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd3fff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cd4097ff0 0x7f5cd413f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd3fff640 10 monclient(hunting): get_auth_request con 0x7f5cd4097ff0 auth_method 0 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient(hunting): handle_auth_done global_id 94155 payload 824 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cd409d300 msgr2=0x7f5cd406d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cd409d300 0x7f5cd406d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cd4097ff0 msgr2=0x7f5cd413f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cd4097ff0 0x7f5cd413f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 monclient: found mon.noname-a 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd37fe640 1 -- 172.21.15.116:0/3255010841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc8009450 con 0x7f5cd409c820 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.172+0000 7f5cd3fff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cd4097ff0 0x7f5cd413f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.174+0000 7f5cd37fe640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd409c820 msgr2=0x7f5cd412ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.174+0000 7f5cd37fe640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd409c820 0x7f5cd412ea10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009970 tx=0x7f5cc8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc802aea0 con 0x7f5cd409c820 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: dump: 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.189 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc8000d10 con 0x7f5cd409c820 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 5 monclient: authenticate success, global_id 94155 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc803d440 con 0x7f5cd409c820 2024-09-18T12:31:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cbbfff640 10 monclient: dump: 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 10 monclient: _renew_subs 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cd27fc640 4 set_mon_vals no callback set 2024-09-18T12:31:50.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd411bb30 con 0x7f5cd409c820 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 10 monclient: _renew_subs 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd413f650 con 0x7f5cd409c820 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 0x7f5cd406dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.175+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd411c290 con 0x7f5cd411abd0 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 0x7f5cd406dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient: get_auth_request con 0x7f5cd411abd0 auth_method 0 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5cc8039040 con 0x7f5cd409c820 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cbbfff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cc0038490 0x7f5cc003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_done global_id 94158 payload 824 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd3fff640 1 -- 172.21.15.116:0/3255010841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5cd406f100 con 0x7f5cd411abd0 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (secure 0 0 0) 0x7f5cc806c630 con 0x7f5cd409c820 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd8ffd640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cc0038490 0x7f5cc003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.176+0000 7f5cd8ffd640 10 monclient: get_auth_request con 0x7f5cc0038490 auth_method 0 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cd3fff640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 msgr2=0x7f5cd406dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cd3fff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 0x7f5cd406dc70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e710 tx=0x7f5cc4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc400c820 con 0x7f5cd411abd0 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 10 monclient: dump: 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.194 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5cc400c980 con 0x7f5cd411abd0 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 msgr2=0x7f5cd406dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 0x7f5cd406dc70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e710 tx=0x7f5cc4009040 comp rx=0 tx=0).stop 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 msgr2=0x7f5cd406dc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.178+0000 7f5cbbfff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd411abd0 0x7f5cd406dc70 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e710 tx=0x7f5cc4009040 comp rx=0 tx=0).stop 2024-09-18T12:31:50.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.179+0000 7f5cd8ffd640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cc0038490 0x7f5cc003a940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009fd0 tx=0x7f5ccc00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cdaa60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cdaa60640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 0x7f5cd406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cdaa60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd4065f90 con 0x7f5cd412ef50 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 0x7f5cd406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient: get_auth_request con 0x7f5cd412ef50 auth_method 0 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 10 monclient(hunting): handle_auth_done global_id 94161 payload 824 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.184+0000 7f5cd3fff640 1 -- 172.21.15.116:0/3255010841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5cd404bf60 con 0x7f5cd412ef50 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cd3fff640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 msgr2=0x7f5cd406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cd3fff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 0x7f5cd406e1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4002380 tx=0x7f5cc401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4026070 con 0x7f5cd412ef50 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 10 monclient: dump: 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.198 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f5cc4006d10 con 0x7f5cd412ef50 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 msgr2=0x7f5cd406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 0x7f5cd406e1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4002380 tx=0x7f5cc401a000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 msgr2=0x7f5cd406e1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cbbfff640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd412ef50 0x7f5cd406e1b0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4002380 tx=0x7f5cc401a000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 72, 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:50.207 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.208 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.209 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:50.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:50.211 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:50.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cc0038490 msgr2=0x7f5cc003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5cc0038490 0x7f5cc003a940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009fd0 tx=0x7f5ccc00b040 comp rx=0 tx=0).stop 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 10 monclient: shutdown 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd409c820 msgr2=0x7f5cd412ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 --2- 172.21.15.116:0/3255010841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cd409c820 0x7f5cd412ea10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009970 tx=0x7f5cc8032000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cd8ffd640 1 -- 172.21.15.116:0/3255010841 reap_dead start 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cd8ffd640 1 -- 172.21.15.116:0/3255010841 reap_dead start 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 shutdown_connections 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 >> 172.21.15.116:0/3255010841 conn(0x7f5cd40a1e30 msgr2=0x7f5cd409bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 shutdown_connections 2024-09-18T12:31:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.187+0000 7f5cdaa60640 1 -- 172.21.15.116:0/3255010841 wait complete. 2024-09-18T12:31:50.214 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:31:50.214 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:31:50.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:31:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.306+0000 7f3670c80640 10 monclient: get_monmap_and_config 2024-09-18T12:31:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.306+0000 7f3670c80640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.306+0000 7f3670c80640 10 monclient: monmap: 2024-09-18T12:31:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:50.307307+0000 2024-09-18T12:31:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:50.307307+0000 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:50.308 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 Processor -- start 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 -- start start 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient: init 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f366c0975f0 0x7f366c095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): picked mon.noname-a con 0x7f366c0975f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f366c0979c0 0x7f366c097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): picked mon.noname-c con 0x7f366c0979c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c09c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): picked mon.noname-b con 0x7f366c09bf50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c09d700 con 0x7f366c0975f0 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c09d1b0 con 0x7f366c0979c0 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c09d420 con 0x7f366c09bf50 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f3670c80640 10 monclient(hunting): authenticate will time out at 1137.628691s 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f366c0975f0 0x7f366c095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c09c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c09c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33072/0 (socket says 172.21.15.116:33072) 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 1 -- 172.21.15.116:0/3405288969 learned_addr learned my addr 172.21.15.116:0/3405288969 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f366c0975f0 0x7f366c095ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52708/0 (socket says 172.21.15.116:52708) 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 10 monclient(hunting): get_auth_request con 0x7f366c09bf50 auth_method 0 2024-09-18T12:31:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 10 monclient(hunting): get_auth_request con 0x7f366c0975f0 auth_method 0 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.307+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f3661d9b640 1 --2- 172.21.15.116:0/3405288969 >> v2:172.21.15.144:3300/0 conn(0x7f366c0979c0 0x7f366c097dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 10 monclient(hunting): handle_auth_done global_id 64754 payload 274 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 1 -- 172.21.15.116:0/3405288969 >> v2:172.21.15.144:3300/0 conn(0x7f366c0979c0 msgr2=0x7f366c097dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 1 --2- 172.21.15.116:0/3405288969 >> v2:172.21.15.144:3300/0 conn(0x7f366c0979c0 0x7f366c097dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 1 -- 172.21.15.116:0/3405288969 >> v2:172.21.15.116:3300/0 conn(0x7f366c0975f0 msgr2=0x7f366c095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 1 --2- 172.21.15.116:0/3405288969 >> v2:172.21.15.116:3300/0 conn(0x7f366c0975f0 0x7f366c095ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 1 monclient: found mon.noname-b 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366ad9d640 1 -- 172.21.15.116:0/3405288969 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c09d9e0 con 0x7f366c09bf50 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f3661d9b640 10 monclient: get_auth_request con 0x7f366c0979c0 auth_method 0 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f366a59c640 1 --2- 172.21.15.116:0/3405288969 >> v2:172.21.15.116:3300/0 conn(0x7f366c0975f0 0x7f366c095ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.308+0000 7f3661d9b640 1 --2- 172.21.15.116:0/3405288969 >> v2:172.21.15.144:3300/0 conn(0x7f366c0979c0 0x7f366c097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.310+0000 7f366ad9d640 1 -- 172.21.15.116:0/3405288969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 msgr2=0x7f366c09c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.310+0000 7f366ad9d640 1 --2- 172.21.15.116:0/3405288969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c09c360 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f366c0959e0 tx=0x7f3664009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=710e771815217126 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.310+0000 7f3669d9b640 1 -- 172.21.15.116:0/3405288969 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664007610 con 0x7f366c09bf50 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.310+0000 7f3669d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 10 monclient: dump: 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 1 -- 172.21.15.116:0/3405288969 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366402d050 con 0x7f366c09bf50 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 5 monclient: authenticate success, global_id 64754 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 1 -- 172.21.15.116:0/3405288969 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640079a0 con 0x7f366c09bf50 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 4 set_mon_vals no callback set 2024-09-18T12:31:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3669d9b640 10 monclient: dump: 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 10 monclient: shutdown 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 -- 172.21.15.116:0/3405288969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 msgr2=0x7f366c09c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 --2- 172.21.15.116:0/3405288969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c09c360 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f366c0959e0 tx=0x7f3664009840 comp rx=0 tx=0).stop 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 -- 172.21.15.116:0/3405288969 shutdown_connections 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 --2- 172.21.15.116:0/3405288969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c09c360 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 --2- 172.21.15.116:0/3405288969 >> v2:172.21.15.144:3300/0 conn(0x7f366c0979c0 0x7f366c097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 --2- 172.21.15.116:0/3405288969 >> v2:172.21.15.116:3300/0 conn(0x7f366c0975f0 0x7f366c095ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 -- 172.21.15.116:0/3405288969 >> 172.21.15.116:0/3405288969 conn(0x7f366c0678a0 msgr2=0x7f366c0a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 -- 172.21.15.116:0/3405288969 shutdown_connections 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.311+0000 7f3670c80640 1 -- 172.21.15.116:0/3405288969 wait complete. 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient: monmap: 2024-09-18T12:31:50.325 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 Processor -- start 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 -- start start 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient: init 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c0975f0 0x7f366c12dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): picked mon.noname-b con 0x7f366c0975f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c0979c0 0x7f366c131620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): picked mon.noname-a con 0x7f366c0979c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c131b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): picked mon.noname-c con 0x7f366c09bf50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c12e0e0 con 0x7f366c0979c0 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c13f840 con 0x7f366c0975f0 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0983f0 con 0x7f366c09bf50 2024-09-18T12:31:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.312+0000 7f3670c80640 10 monclient(hunting): authenticate will time out at 1137.633992s 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c131b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c0979c0 0x7f366c131620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c131b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33088/0 (socket says 172.21.15.116:33088) 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 1 -- 172.21.15.116:0/2790011314 learned_addr learned my addr 172.21.15.116:0/2790011314 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): get_auth_request con 0x7f366c0979c0 auth_method 0 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 10 monclient(hunting): get_auth_request con 0x7f366c09bf50 auth_method 0 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c0975f0 0x7f366c12dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 10 monclient(hunting): get_auth_request con 0x7f366c0975f0 auth_method 0 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient(hunting): handle_auth_done global_id 94167 payload 824 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 msgr2=0x7f366c131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c09bf50 0x7f366c131b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c0975f0 msgr2=0x7f366c12dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f366c0975f0 0x7f366c12dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 1 monclient: found mon.noname-a 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.313+0000 7f3661d9b640 1 -- 172.21.15.116:0/2790011314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009390 con 0x7f366c0979c0 2024-09-18T12:31:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3661d9b640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c0979c0 msgr2=0x7f366c131620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3661d9b640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c0979c0 0x7f366c131620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f365400c880 tx=0x7f3654008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365400ce10 con 0x7f366c0979c0 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: dump: 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3654007a40 con 0x7f366c0979c0 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3670c80640 5 monclient: authenticate success, global_id 94167 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3670c80640 10 monclient: _renew_subs 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3670c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3664009ae0 con 0x7f366c0979c0 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654007e00 con 0x7f366c0979c0 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3662ffd640 10 monclient: dump: 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.333 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f366959a640 4 set_mon_vals no callback set 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3670c80640 10 monclient: _renew_subs 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3670c80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.315+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c12ced0 con 0x7f366c0979c0 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f3670c80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f3670c80640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 0x7f366c06e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f3661d9b640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c0979c0 0x7f366c131620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f365400c880 tx=0x7f3654008040 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f3654011600 mgrmap(e 5) v1 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c065e00 con 0x7f366c06de00 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3654011600 con 0x7f366c0979c0 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 0x7f366c06e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient: get_auth_request con 0x7f366c06de00 auth_method 0 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f3662ffd640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f364c0384c0 0x7f364c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 10 monclient(hunting): handle_auth_done global_id 64760 payload 824 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366a59c640 1 -- 172.21.15.116:0/2790011314 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f366c04bf60 con 0x7f366c06de00 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366ad9d640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f364c0384c0 0x7f364c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.316+0000 7f366ad9d640 10 monclient: get_auth_request con 0x7f364c0384c0 auth_method 0 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.319+0000 7f366a59c640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 msgr2=0x7f366c06e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.319+0000 7f366a59c640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 0x7f366c06e1d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f365c011bf0 tx=0x7f365c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.319+0000 7f366ad9d640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f364c0384c0 0x7f364c03a970 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f36640072b0 tx=0x7f3664036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c00fe90 con 0x7f366c06de00 2024-09-18T12:31:50.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 10 monclient: dump: 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f365c014650 con 0x7f366c06de00 2024-09-18T12:31:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 msgr2=0x7f366c06e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 0x7f366c06e1d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f365c011bf0 tx=0x7f365c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 msgr2=0x7f366c06e1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.320+0000 7f3662ffd640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06de00 0x7f366c06e1d0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f365c011bf0 tx=0x7f365c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:50.328+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v164) v1 ==== 123+0+0 (secure 0 0 0) 0x7f849c016030 con 0x7f84a0069a30 2024-09-18T12:31:50.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:50.328+0000 7f8495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:50.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:50.328+0000 7f8495ffb640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-18T12:31:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.326+0000 7f3670c80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.326+0000 7f3670c80640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 0x7f366c129400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.326+0000 7f3670c80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.326+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0638d0 con 0x7f366c06e710 2024-09-18T12:31:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 0x7f366c129400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient: get_auth_request con 0x7f366c06e710 auth_method 0 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 10 monclient(hunting): handle_auth_done global_id 64763 payload 824 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.327+0000 7f366a59c640 1 -- 172.21.15.116:0/2790011314 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f366c06eb10 con 0x7f366c06e710 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f366a59c640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 msgr2=0x7f366c129400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f366a59c640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 0x7f366c129400 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f365c011bf0 tx=0x7f365c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c025040 con 0x7f366c06e710 2024-09-18T12:31:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 10 monclient: dump: 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f365c01c3e0 con 0x7f366c06e710 2024-09-18T12:31:50.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 msgr2=0x7f366c129400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 0x7f366c129400 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f365c011bf0 tx=0x7f365c01d000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 msgr2=0x7f366c129400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.329+0000 7f3662ffd640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f366c06e710 0x7f366c129400 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f365c011bf0 tx=0x7f365c01d000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:31:50.342 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 72, 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:50.343 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.344 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:50.345 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:50.346 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:50.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:50.355 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f364c0384c0 msgr2=0x7f364c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f364c0384c0 0x7f364c03a970 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f36640072b0 tx=0x7f3664036000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 10 monclient: shutdown 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c0979c0 msgr2=0x7f366c131620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 --2- 172.21.15.116:0/2790011314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f366c0979c0 0x7f366c131620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f365400c880 tx=0x7f3654008040 comp rx=0 tx=0).stop 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f366ad9d640 1 -- 172.21.15.116:0/2790011314 reap_dead start 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f366ad9d640 1 -- 172.21.15.116:0/2790011314 reap_dead start 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 shutdown_connections 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 >> 172.21.15.116:0/2790011314 conn(0x7f366c0678a0 msgr2=0x7f366c0962f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3662ffd640 1 -- 172.21.15.116:0/2790011314 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(163..163 src has 13..163) v4 ==== 9825+0+0 (unknown 0 0 0) 0x7f365404cb80 con 0x7f366c0979c0 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 shutdown_connections 2024-09-18T12:31:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.330+0000 7f3670c80640 1 -- 172.21.15.116:0/2790011314 wait complete. 2024-09-18T12:31:50.357 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:31:50.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:31:50.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:31:50.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:50.427+0000 7f84a7ac9640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:50.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:50.427+0000 7f84a7ac9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:31:50.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:50.427+0000 7f84a7ac9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:31:50.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:50.427+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f84a0064b00 con 0x7f84a0069a30 2024-09-18T12:31:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.450+0000 7f3b6cd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.450+0000 7f3b6cd9d640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.450+0000 7f3b6cd9d640 10 monclient: monmap: 2024-09-18T12:31:50.451 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:50.451450+0000 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:50.451450+0000 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:50.452 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 1 Processor -- start 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 1 -- start start 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 10 monclient: init 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b6809a510 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b681349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b680973b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b68097780 0x7f3b68136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b68097780 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.451+0000 7f3b6cd9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6806ee10 con 0x7f3b6809a510 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b6cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b6cd9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6809cb00 con 0x7f3b68097780 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b6cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b6cd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6809a8e0 con 0x7f3b680973b0 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b6cd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b6cd9d640 10 monclient(hunting): authenticate will time out at 1137.773020s 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b681349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b681349a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33134/0 (socket says 172.21.15.116:33134) 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 -- 172.21.15.116:0/3514906435 learned_addr learned my addr 172.21.15.116:0/3514906435 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): get_auth_request con 0x7f3b680973b0 auth_method 0 2024-09-18T12:31:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3514906435 >> v2:172.21.15.144:3300/0 conn(0x7f3b68097780 0x7f3b68136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_done global_id 64766 payload 274 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 -- 172.21.15.116:0/3514906435 >> v2:172.21.15.144:3300/0 conn(0x7f3b68097780 msgr2=0x7f3b68136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 --2- 172.21.15.116:0/3514906435 >> v2:172.21.15.144:3300/0 conn(0x7f3b68097780 0x7f3b68136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 -- 172.21.15.116:0/3514906435 >> v1:172.21.15.144:6789/0 conn(0x7f3b6809a510 legacy=0x7f3b68096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 monclient: found mon.noname-b 2024-09-18T12:31:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b65d9b640 1 -- 172.21.15.116:0/3514906435 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6809d590 con 0x7f3b680973b0 2024-09-18T12:31:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b66d9d640 10 monclient: get_auth_request con 0x7f3b68097780 auth_method 0 2024-09-18T12:31:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.452+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3514906435 >> v2:172.21.15.144:3300/0 conn(0x7f3b68097780 0x7f3b68136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b65d9b640 1 -- 172.21.15.116:0/3514906435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 msgr2=0x7f3b681349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b65d9b640 1 --2- 172.21.15.116:0/3514906435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b681349a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009910 tx=0x7f3b5c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90d9ccc352bf9b23 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 1 -- 172.21.15.116:0/3514906435 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c02ae00 con 0x7f3b680973b0 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: dump: 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 1 -- 172.21.15.116:0/3514906435 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c02f4b0 con 0x7f3b680973b0 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 5 monclient: authenticate success, global_id 64766 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 1 -- 172.21.15.116:0/3514906435 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c02f850 con 0x7f3b680973b0 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 4 set_mon_vals no callback set 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6559a640 10 monclient: dump: 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 10 monclient: shutdown 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/3514906435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 msgr2=0x7f3b681349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 1 --2- 172.21.15.116:0/3514906435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b681349a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009910 tx=0x7f3b5c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/3514906435 shutdown_connections 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 1 --2- 172.21.15.116:0/3514906435 >> v2:172.21.15.144:3300/0 conn(0x7f3b68097780 0x7f3b68136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.455+0000 7f3b6cd9d640 1 --2- 172.21.15.116:0/3514906435 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b681349a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/3514906435 >> 172.21.15.116:0/3514906435 conn(0x7f3b680672b0 msgr2=0x7f3b680676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/3514906435 shutdown_connections 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/3514906435 wait complete. 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 10 monclient: monmap: 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 1 Processor -- start 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 1 -- start start 2024-09-18T12:31:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.456+0000 7f3b6cd9d640 10 monclient: init 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b68140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b680973b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68097780 0x7f3b6812fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b68097780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b6809a510 0x7f3b68098330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b6809a510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68099f90 con 0x7f3b6809a510 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6806d8b0 con 0x7f3b68097780 2024-09-18T12:31:50.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68141e30 con 0x7f3b680973b0 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6cd9d640 10 monclient(hunting): authenticate will time out at 1137.778242s 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b68140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b68140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33150/0 (socket says 172.21.15.116:33150) 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 -- 172.21.15.116:0/2226413660 learned_addr learned my addr 172.21.15.116:0/2226413660 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b6809a510 0x7f3b68098330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): get_auth_request con 0x7f3b680973b0 auth_method 0 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 10 monclient(hunting): get_auth_request con 0x7f3b6809a510 auth_method 0 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68097780 0x7f3b6812fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 10 monclient(hunting): get_auth_request con 0x7f3b68097780 auth_method 0 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient(hunting): handle_auth_done global_id 64769 payload 824 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68097780 msgr2=0x7f3b6812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68097780 0x7f3b6812fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b6809a510 msgr2=0x7f3b68098330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b6809a510 0x7f3b68098330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 monclient: found mon.noname-c 2024-09-18T12:31:50.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b6659c640 1 -- 172.21.15.116:0/2226413660 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c009450 con 0x7f3b680973b0 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.457+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b6809a510 0x7f3b68098330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.459+0000 7f3b6659c640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 msgr2=0x7f3b68140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.459+0000 7f3b6659c640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b68140fa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800c5e0 tx=0x7f3b58007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b58012840 con 0x7f3b680973b0 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: dump: 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.474 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b580129a0 con 0x7f3b680973b0 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 5 monclient: authenticate success, global_id 64769 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 10 monclient: _renew_subs 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6811ee30 con 0x7f3b680973b0 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b58012da0 con 0x7f3b680973b0 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 10 monclient: dump: 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.476 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b57fff640 4 set_mon_vals no callback set 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 10 monclient: _renew_subs 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6811df40 con 0x7f3b680973b0 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 0x7f3b68098870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6811f2b0 con 0x7f3b68130230 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.460+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3b58011590 con 0x7f3b680973b0 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b567fc640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b48038490 0x7f3b4803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 0x7f3b68098870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(164..164 src has 13..164) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f3b5804c9f0 con 0x7f3b680973b0 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient: get_auth_request con 0x7f3b68130230 auth_method 0 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b48038490 0x7f3b4803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b66d9d640 10 monclient: get_auth_request con 0x7f3b48038490 auth_method 0 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_done global_id 5959 payload 824 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.461+0000 7f3b65d9b640 1 -- 172.21.15.116:0/2226413660 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b6804bf60 con 0x7f3b68130230 2024-09-18T12:31:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.463+0000 7f3b65d9b640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 msgr2=0x7f3b68098870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.463+0000 7f3b65d9b640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 0x7f3b68098870 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c034000 tx=0x7f3b5c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.463+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b48038490 0x7f3b4803a940 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000dab0 tx=0x7f3b60007840 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.463+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c032aa0 con 0x7f3b68130230 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.463+0000 7f3b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.463+0000 7f3b567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.463+0000 7f3b567fc640 10 monclient: dump: 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.464+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3b5c046440 con 0x7f3b68130230 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.464+0000 7f3b567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.464+0000 7f3b567fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.464+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 msgr2=0x7f3b68098870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.464+0000 7f3b567fc640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 0x7f3b68098870 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c034000 tx=0x7f3b5c032000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.464+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 msgr2=0x7f3b68098870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.464+0000 7f3b567fc640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68130230 0x7f3b68098870 secure :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c034000 tx=0x7f3b5c032000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.469+0000 7f3b6cd9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:31:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.469+0000 7f3b6cd9d640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 0x7f3b681316e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.469+0000 7f3b6cd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.469+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68065f20 con 0x7f3b68098db0 2024-09-18T12:31:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 0x7f3b681316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient: get_auth_request con 0x7f3b68098db0 auth_method 0 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 10 monclient(hunting): handle_auth_done global_id 5962 payload 824 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.470+0000 7f3b65d9b640 1 -- 172.21.15.116:0/2226413660 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b680991b0 con 0x7f3b68098db0 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.472+0000 7f3b65d9b640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 msgr2=0x7f3b681316e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.472+0000 7f3b65d9b640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 0x7f3b681316e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3b68130600 tx=0x7f3b5c03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.472+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c04d070 con 0x7f3b68098db0 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.472+0000 7f3b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.472+0000 7f3b567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.472+0000 7f3b567fc640 10 monclient: dump: 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f3b5c03cd30 con 0x7f3b68098db0 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b567fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 msgr2=0x7f3b681316e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b567fc640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 0x7f3b681316e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3b68130600 tx=0x7f3b5c03d000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b567fc640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 msgr2=0x7f3b681316e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b567fc640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b68098db0 0x7f3b681316e0 secure :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3b68130600 tx=0x7f3b5c03d000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 72, 2024-09-18T12:31:50.485 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.486 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:31:50.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:31:50.489 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:31:50.490 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:31:50.493 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:31:50.494 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:31:50.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:31:50.497 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:31:50.498 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:31:50.498 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:31:50.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:31:50.498 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:31:50.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:31:50.498 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:31:50.498 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:31:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b48038490 msgr2=0x7f3b4803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b48038490 0x7f3b4803a940 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000dab0 tx=0x7f3b60007840 comp rx=0 tx=0).stop 2024-09-18T12:31:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 10 monclient: shutdown 2024-09-18T12:31:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 msgr2=0x7f3b68140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 --2- 172.21.15.116:0/2226413660 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b680973b0 0x7f3b68140fa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800c5e0 tx=0x7f3b58007ed0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b66d9d640 1 -- 172.21.15.116:0/2226413660 reap_dead start 2024-09-18T12:31:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b66d9d640 1 -- 172.21.15.116:0/2226413660 reap_dead start 2024-09-18T12:31:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 shutdown_connections 2024-09-18T12:31:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 >> 172.21.15.116:0/2226413660 conn(0x7f3b680672b0 msgr2=0x7f3b6809faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 shutdown_connections 2024-09-18T12:31:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.473+0000 7f3b6cd9d640 1 -- 172.21.15.116:0/2226413660 wait complete. 2024-09-18T12:31:50.500 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:31:50.501 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:31:50.501 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:31:50.501 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:31:50.501 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:31:50.501 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:31:50.501 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:31:50.510 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:31:50.511 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:31:50.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.641+0000 7f75db55d640 10 monclient: get_monmap_and_config 2024-09-18T12:31:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.641+0000 7f75db55d640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.641+0000 7f75db55d640 10 monclient: monmap: 2024-09-18T12:31:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:50.642178+0000 2024-09-18T12:31:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:50.642178+0000 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:50.643 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 1 Processor -- start 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 1 -- start start 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient: init 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f75d4096e10 0x7f75d409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient(hunting): picked mon.noname-c con 0x7f75d4096e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75d409aaa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 0x7f75d4135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40978f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d409d4a0 con 0x7f75d409aaa0 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d4099620 con 0x7f75d4096e10 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d4099340 con 0x7f75d40978f0 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75db55d640 10 monclient(hunting): authenticate will time out at 1137.963758s 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 0x7f75d4135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 0x7f75d4135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33166/0 (socket says 172.21.15.116:33166) 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 1 -- 172.21.15.116:0/2670995844 learned_addr learned my addr 172.21.15.116:0/2670995844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 10 monclient(hunting): get_auth_request con 0x7f75d40978f0 auth_method 0 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d9afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.642+0000 7f75d92f9640 1 --2- 172.21.15.116:0/2670995844 >> v2:172.21.15.144:3300/0 conn(0x7f75d4096e10 0x7f75d409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d92f9640 10 monclient(hunting): get_auth_request con 0x7f75d4096e10 auth_method 0 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d92f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d92f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d92f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 10 monclient(hunting): handle_auth_done global_id 64772 payload 274 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 1 -- 172.21.15.116:0/2670995844 >> v2:172.21.15.144:3300/0 conn(0x7f75d4096e10 msgr2=0x7f75d409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 1 --2- 172.21.15.116:0/2670995844 >> v2:172.21.15.144:3300/0 conn(0x7f75d4096e10 0x7f75d409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 1 -- 172.21.15.116:0/2670995844 >> v1:172.21.15.144:6789/0 conn(0x7f75d409aaa0 legacy=0x7f75d40971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 1 monclient: found mon.noname-b 2024-09-18T12:31:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d9afa640 1 -- 172.21.15.116:0/2670995844 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d4099900 con 0x7f75d40978f0 2024-09-18T12:31:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.643+0000 7f75d92f9640 1 --2- 172.21.15.116:0/2670995844 >> v2:172.21.15.144:3300/0 conn(0x7f75d4096e10 0x7f75d409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.645+0000 7f75d9afa640 1 -- 172.21.15.116:0/2670995844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 msgr2=0x7f75d4135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.645+0000 7f75d9afa640 1 --2- 172.21.15.116:0/2670995844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 0x7f75d4135b40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009f20 tx=0x7f75cc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb3af10bc89d447 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.645+0000 7f75cbfff640 1 -- 172.21.15.116:0/2670995844 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc007610 con 0x7f75d40978f0 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.645+0000 7f75cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 10 monclient: dump: 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.654 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 5 monclient: authenticate success, global_id 64772 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 1 -- 172.21.15.116:0/2670995844 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc02d050 con 0x7f75d40978f0 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 1 -- 172.21.15.116:0/2670995844 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc0079e0 con 0x7f75d40978f0 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 4 set_mon_vals no callback set 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75cbfff640 10 monclient: dump: 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.656 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 10 monclient: shutdown 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 -- 172.21.15.116:0/2670995844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 msgr2=0x7f75d4135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 --2- 172.21.15.116:0/2670995844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 0x7f75d4135b40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009f20 tx=0x7f75cc009840 comp rx=0 tx=0).stop 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 -- 172.21.15.116:0/2670995844 shutdown_connections 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 --2- 172.21.15.116:0/2670995844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d40978f0 0x7f75d4135b40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 --2- 172.21.15.116:0/2670995844 >> v2:172.21.15.144:3300/0 conn(0x7f75d4096e10 0x7f75d409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 -- 172.21.15.116:0/2670995844 >> 172.21.15.116:0/2670995844 conn(0x7f75d40672b0 msgr2=0x7f75d40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 -- 172.21.15.116:0/2670995844 shutdown_connections 2024-09-18T12:31:50.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.646+0000 7f75db55d640 1 -- 172.21.15.116:0/2670995844 wait complete. 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 10 monclient: build_initial_monmap 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 10 monclient: monmap: 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:50.658 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 1 Processor -- start 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 1 -- start start 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 10 monclient: init 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75d4096e10 0x7f75d413d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.647+0000 7f75db55d640 10 monclient(hunting): picked mon.noname-b con 0x7f75d4096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d40978f0 0x7f75d413cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 10 monclient(hunting): picked mon.noname-a con 0x7f75d40978f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 0x7f75d4140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 10 monclient(hunting): picked mon.noname-c con 0x7f75d409aaa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d4122390 con 0x7f75d40978f0 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d41203a0 con 0x7f75d4096e10 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d411fb40 con 0x7f75d409aaa0 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75db55d640 10 monclient(hunting): authenticate will time out at 1137.969038s 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d8af8640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d40978f0 msgr2=0x7f75d413cd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d8af8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d40978f0 0x7f75d413cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 0x7f75d4140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 0x7f75d4140bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33168/0 (socket says 172.21.15.116:33168) 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 -- 172.21.15.116:0/4290623920 learned_addr learned my addr 172.21.15.116:0/4290623920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): get_auth_request con 0x7f75d409aaa0 auth_method 0 2024-09-18T12:31:50.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d92f9640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75d4096e10 0x7f75d413d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient(hunting): handle_auth_done global_id 64775 payload 824 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 -- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75d4096e10 msgr2=0x7f75d413d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75d4096e10 0x7f75d413d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 -- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d40978f0 msgr2=0x7f75d413cd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d40978f0 0x7f75d413cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient: _finish_hunting 0 2024-09-18T12:31:50.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 monclient: found mon.noname-c 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d9afa640 1 -- 172.21.15.116:0/4290623920 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc009ae0 con 0x7f75d409aaa0 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d92f9640 10 monclient: get_auth_request con 0x7f75d4096e10 auth_method 0 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.648+0000 7f75d92f9640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75d4096e10 0x7f75d413d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.650+0000 7f75d9afa640 1 -- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 msgr2=0x7f75d4140bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.650+0000 7f75d9afa640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 0x7f75d4140bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009ab0 tx=0x7f75cc036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.650+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc032ce0 con 0x7f75d409aaa0 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: dump: 2024-09-18T12:31:50.662 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: _finish_auth 0 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc032e40 con 0x7f75d409aaa0 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:50.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc03b660 con 0x7f75d409aaa0 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75c97fa640 10 monclient: dump: 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:50.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75db55d640 5 monclient: authenticate success, global_id 64775 2024-09-18T12:31:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75db55d640 10 monclient: _renew_subs 2024-09-18T12:31:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75db55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d4121dd0 con 0x7f75d409aaa0 2024-09-18T12:31:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75cb7fe640 4 set_mon_vals no callback set 2024-09-18T12:31:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75db55d640 10 monclient: _renew_subs 2024-09-18T12:31:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75db55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.652+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d413e090 con 0x7f75d409aaa0 2024-09-18T12:31:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.653+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f75cc003020 con 0x7f75d409aaa0 2024-09-18T12:31:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.653+0000 7f75db55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.653+0000 7f75db55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.653+0000 7f75db55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.653+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d41224d0 con 0x7f75d409aaa0 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.654+0000 7f75c97fa640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f75b8038490 0x7f75b803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.654+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(164..164 src has 13..164) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f75cc06d330 con 0x7f75d409aaa0 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.654+0000 7f75d92f9640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f75b8038490 0x7f75b803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.654+0000 7f75d92f9640 10 monclient: get_auth_request con 0x7f75b8038490 auth_method 0 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.656+0000 7f75d92f9640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f75b8038490 0x7f75b803a940 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006e70 tx=0x7f75d0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.658+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f75d41224d0 con 0x7f75d409aaa0 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.658+0000 7f75c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.658+0000 7f75c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.874+0000 7f75db55d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.874+0000 7f75db55d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.874+0000 7f75db55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.874+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f75d41224d0 con 0x7f75d409aaa0 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.875+0000 7f75c97fa640 1 -- 172.21.15.116:0/4290623920 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f75d41224d0 con 0x7f75d409aaa0 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.875+0000 7f75c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.875+0000 7f75c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:50.876 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f75b8038490 msgr2=0x7f75b803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f75b8038490 0x7f75b803a940 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006e70 tx=0x7f75d0008040 comp rx=0 tx=0).stop 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 10 monclient: shutdown 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 msgr2=0x7f75d4140bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 0x7f75d4140bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009ab0 tx=0x7f75cc036000 comp rx=0 tx=0).stop 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 shutdown_connections 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f75b8038490 0x7f75b803a940 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f75d409aaa0 0x7f75d4140bf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d40978f0 0x7f75d413cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 --2- 172.21.15.116:0/4290623920 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75d4096e10 0x7f75d413d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 >> 172.21.15.116:0/4290623920 conn(0x7f75d40672b0 msgr2=0x7f75d409ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 shutdown_connections 2024-09-18T12:31:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:50.877+0000 7f75db55d640 1 -- 172.21.15.116:0/4290623920 wait complete. 2024-09-18T12:31:52.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:52.138+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:52.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:52.138+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:52.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:52.138+0000 7f8494ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:53.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:53.138+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:53.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:53.138+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:53.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.992+0000 7f30f0cc0640 10 monclient: get_monmap_and_config 2024-09-18T12:31:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.992+0000 7f30f0cc0640 10 monclient: build_initial_monmap 2024-09-18T12:31:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.992+0000 7f30f0cc0640 10 monclient: monmap: 2024-09-18T12:31:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:53.993219+0000 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:53.993219+0000 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:53.994 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:53.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 1 Processor -- start 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 1 -- start start 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient: init 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30ec09d2f0 0x7f30ec099360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f30ec09d2f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f30ec09b7a0 0x7f30ec09bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f30ec09b7a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30ec09c120 addr v1:172.21.15.144:6789/0 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ec0977f0 con 0x7f30ec09c120 2024-09-18T12:31:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ec0989f0 con 0x7f30ec09d2f0 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ec09d6c0 con 0x7f30ec09b7a0 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30f0cc0640 10 monclient(hunting): authenticate will time out at 1141.314898s 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30ea59c640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f30ec09d2f0 msgr2=0x7f30ec099360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.993+0000 7f30ea59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30ec09d2f0 0x7f30ec099360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f30ec09b7a0 0x7f30ec09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30ead9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f30ec09c120 0x7f30ec1345b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:52330/0 (socket says 172.21.15.116:52330) 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30ead9d640 1 -- 172.21.15.116:0/1972203799 learned_addr learned my addr 172.21.15.116:0/1972203799 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:31:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): get_auth_request con 0x7f30ec09b7a0 auth_method 0 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_done global_id 5968 payload 274 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 1 -- 172.21.15.116:0/1972203799 >> v2:172.21.15.116:3300/0 conn(0x7f30ec09d2f0 msgr2=0x7f30ec099360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 1 --2- 172.21.15.116:0/1972203799 >> v2:172.21.15.116:3300/0 conn(0x7f30ec09d2f0 0x7f30ec099360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 1 -- 172.21.15.116:0/1972203799 >> v1:172.21.15.144:6789/0 conn(0x7f30ec09c120 legacy=0x7f30ec1345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 1 monclient: found mon.noname-c 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:53.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.994+0000 7f30e9d9b640 1 -- 172.21.15.116:0/1972203799 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec098710 con 0x7f30ec09b7a0 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.997+0000 7f30e9d9b640 1 -- 172.21.15.116:0/1972203799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 msgr2=0x7f30ec09bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.997+0000 7f30e9d9b640 1 --2- 172.21.15.116:0/1972203799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 0x7f30ec09bbe0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f30ec09ec60 tx=0x7f30e000a2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c28e2f01008941 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 1 -- 172.21.15.116:0/1972203799 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0030800 con 0x7f30ec09b7a0 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: dump: 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: _finish_auth 0 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 1 -- 172.21.15.116:0/1972203799 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e002e050 con 0x7f30ec09b7a0 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 1 -- 172.21.15.116:0/1972203799 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0030b90 con 0x7f30ec09b7a0 2024-09-18T12:31:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 5 monclient: authenticate success, global_id 5968 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30e959a640 10 monclient: dump: 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 4 set_mon_vals no callback set 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 10 monclient: shutdown 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 1 -- 172.21.15.116:0/1972203799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 msgr2=0x7f30ec09bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/1972203799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 0x7f30ec09bbe0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f30ec09ec60 tx=0x7f30e000a2b0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 1 -- 172.21.15.116:0/1972203799 shutdown_connections 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/1972203799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 0x7f30ec09bbe0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/1972203799 >> v2:172.21.15.116:3300/0 conn(0x7f30ec09d2f0 0x7f30ec099360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 1 -- 172.21.15.116:0/1972203799 >> 172.21.15.116:0/1972203799 conn(0x7f30ec067d00 msgr2=0x7f30ec0a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.998+0000 7f30f0cc0640 1 -- 172.21.15.116:0/1972203799 shutdown_connections 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.999+0000 7f30f0cc0640 1 -- 172.21.15.116:0/1972203799 wait complete. 2024-09-18T12:31:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.999+0000 7f30f0cc0640 10 monclient: build_initial_monmap 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.999+0000 7f30f0cc0640 10 monclient: monmap: 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.999+0000 7f30f0cc0640 1 Processor -- start 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.999+0000 7f30f0cc0640 1 -- start start 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:53.999+0000 7f30f0cc0640 10 monclient: init 2024-09-18T12:31:54.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 0x7f30ec13ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f30ec09b7a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 0x7f30ec12f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f30ec09c120 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30ec09d2f0 0x7f30ec1192c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f30ec09d2f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec11aef0 con 0x7f30ec09d2f0 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec06db30 con 0x7f30ec09b7a0 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec11c8d0 con 0x7f30ec09c120 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30f0cc0640 10 monclient(hunting): authenticate will time out at 1141.321334s 2024-09-18T12:31:54.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ead9d640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30ec09d2f0 msgr2=0x7f30ec1192c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30ec09d2f0 0x7f30ec1192c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 0x7f30ec12f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 0x7f30ec12f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33182/0 (socket says 172.21.15.116:33182) 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 -- 172.21.15.116:0/3674017217 learned_addr learned my addr 172.21.15.116:0/3674017217 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): get_auth_request con 0x7f30ec09c120 auth_method 0 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 0x7f30ec13ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 10 monclient(hunting): get_auth_request con 0x7f30ec09b7a0 auth_method 0 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:54.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient(hunting): handle_auth_done global_id 64778 payload 824 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 -- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 msgr2=0x7f30ec13ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 0x7f30ec13ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 -- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30ec09d2f0 msgr2=0x7f30ec1192c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30ec09d2f0 0x7f30ec1192c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 monclient: found mon.noname-c 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.000+0000 7f30e9d9b640 1 -- 172.21.15.116:0/3674017217 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e000a530 con 0x7f30ec09c120 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.002+0000 7f30e9d9b640 1 -- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 msgr2=0x7f30ec12f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30e9d9b640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 0x7f30ec12f7b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30e000aa20 tx=0x7f30e0038000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 1 -- 172.21.15.116:0/3674017217 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0032c50 con 0x7f30ec09c120 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: dump: 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:54.014 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: _finish_auth 0 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30f0cc0640 5 monclient: authenticate success, global_id 64778 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 1 -- 172.21.15.116:0/3674017217 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e0032dd0 con 0x7f30ec09c120 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 1 -- 172.21.15.116:0/3674017217 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e003c660 con 0x7f30ec09c120 2024-09-18T12:31:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30da7fc640 10 monclient: dump: 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30e8d99640 4 set_mon_vals no callback set 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30f0cc0640 10 monclient: _renew_subs 2024-09-18T12:31:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30f0cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec11cbb0 con 0x7f30ec09c120 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30f0cc0640 10 monclient: _renew_subs 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30f0cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.003+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec13f430 con 0x7f30ec09c120 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30da7fc640 1 -- 172.21.15.116:0/3674017217 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f30e003ca90 con 0x7f30ec09c120 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30f0cc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30f0cc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30f0cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec06e8b0 con 0x7f30ec09c120 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30da7fc640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f30cc038440 0x7f30cc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30da7fc640 1 -- 172.21.15.116:0/3674017217 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(164..164 src has 13..164) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f30e006d570 con 0x7f30ec09c120 2024-09-18T12:31:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30ea59c640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f30cc038440 0x7f30cc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.004+0000 7f30ea59c640 10 monclient: get_auth_request con 0x7f30cc038440 auth_method 0 2024-09-18T12:31:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.007+0000 7f30ea59c640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f30cc038440 0x7f30cc03a8f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f30dc003850 tx=0x7f30dc0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.008+0000 7f30da7fc640 1 -- 172.21.15.116:0/3674017217 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f30ec06e8b0 con 0x7f30ec09c120 2024-09-18T12:31:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.008+0000 7f30da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.008+0000 7f30da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:54.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:54.138+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:54.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:54.138+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.222+0000 7f30f0cc0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.222+0000 7f30f0cc0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.222+0000 7f30f0cc0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.222+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f30ec06e8b0 con 0x7f30ec09c120 2024-09-18T12:31:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.222+0000 7f30da7fc640 1 -- 172.21.15.116:0/3674017217 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f30ec06e8b0 con 0x7f30ec09c120 2024-09-18T12:31:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.222+0000 7f30da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.222+0000 7f30da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:54.224 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:54.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f30cc038440 msgr2=0x7f30cc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f30cc038440 0x7f30cc03a8f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f30dc003850 tx=0x7f30dc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 10 monclient: shutdown 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 msgr2=0x7f30ec12f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 0x7f30ec12f7b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30e000aa20 tx=0x7f30e0038000 comp rx=0 tx=0).stop 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 shutdown_connections 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f30cc038440 0x7f30cc03a8f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30ec09d2f0 0x7f30ec1192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f30ec09c120 0x7f30ec12f7b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 --2- 172.21.15.116:0/3674017217 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f30ec09b7a0 0x7f30ec13ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 >> 172.21.15.116:0/3674017217 conn(0x7f30ec067d00 msgr2=0x7f30ec06ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:54.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 shutdown_connections 2024-09-18T12:31:54.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:54.224+0000 7f30f0cc0640 1 -- 172.21.15.116:0/3674017217 wait complete. 2024-09-18T12:31:55.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:55.139+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:55.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:55.139+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:56.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:56.139+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:56.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:56.139+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:57.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:57.139+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:57.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:57.139+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:57.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:31:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.340+0000 7fb57e754640 10 monclient: get_monmap_and_config 2024-09-18T12:31:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.340+0000 7fb57e754640 10 monclient: build_initial_monmap 2024-09-18T12:31:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.340+0000 7fb57e754640 10 monclient: monmap: 2024-09-18T12:31:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:57.341 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:31:57.340930+0000 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:31:57.340930+0000 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:31:57.342 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 1 Processor -- start 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 1 -- start start 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient: init 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb578131ab0 0x7fb578131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient(hunting): picked mon.noname-a con 0x7fb578131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb57809e910 0x7fb57809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient(hunting): picked mon.noname-c con 0x7fb57809e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:31:57.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb57809f250 addr v1:172.21.15.116:6789/0 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57809faf0 con 0x7fb57809f250 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57809fd30 con 0x7fb578131ab0 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5780a1110 con 0x7fb57809e910 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57e754640 10 monclient(hunting): authenticate will time out at 1144.662617s 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb577fff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fb578131ab0 msgr2=0x7fb578131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb57ccf1640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fb57809f250 legacy=0x7fb578068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb577fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb578131ab0 0x7fb578131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.341+0000 7fb5777fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb57809e910 0x7fb57809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:57.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb57809e910 0x7fb57809ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:59040/0 (socket says 172.21.15.116:59040) 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 1 -- 172.21.15.116:0/4237702120 learned_addr learned my addr 172.21.15.116:0/4237702120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): get_auth_request con 0x7fb57809e910 auth_method 0 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient(hunting): handle_auth_done global_id 5974 payload 274 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 1 -- 172.21.15.116:0/4237702120 >> v2:172.21.15.116:3300/0 conn(0x7fb578131ab0 msgr2=0x7fb578131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 1 --2- 172.21.15.116:0/4237702120 >> v2:172.21.15.116:3300/0 conn(0x7fb578131ab0 0x7fb578131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 1 -- 172.21.15.116:0/4237702120 >> v1:172.21.15.116:6789/0 conn(0x7fb57809f250 legacy=0x7fb578068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:31:57.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:57.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 1 monclient: found mon.noname-c 2024-09-18T12:31:57.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:57.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:31:57.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.342+0000 7fb5777fe640 1 -- 172.21.15.116:0/4237702120 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5780a13f0 con 0x7fb57809e910 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb5777fe640 1 -- 172.21.15.116:0/4237702120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb57809e910 msgr2=0x7fb57809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb5777fe640 1 --2- 172.21.15.116:0/4237702120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb57809e910 0x7fb57809ed10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb57809bdb0 tx=0x7fb5600091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6512d34fd9947366 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 1 -- 172.21.15.116:0/4237702120 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560032440 con 0x7fb57809e910 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: dump: 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: _finish_auth 0 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 1 -- 172.21.15.116:0/4237702120 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5600325a0 con 0x7fb57809e910 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb57e754640 5 monclient: authenticate success, global_id 5974 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb57e754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 1 -- 172.21.15.116:0/4237702120 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560032940 con 0x7fb57809e910 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb57e754640 10 monclient: get_monmap_and_config success 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb57e754640 4 set_mon_vals no callback set 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.345+0000 7fb576ffd640 10 monclient: dump: 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:57.357 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 10 monclient: shutdown 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 -- 172.21.15.116:0/4237702120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb57809e910 msgr2=0x7fb57809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 --2- 172.21.15.116:0/4237702120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb57809e910 0x7fb57809ed10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb57809bdb0 tx=0x7fb5600091d0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 -- 172.21.15.116:0/4237702120 shutdown_connections 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 --2- 172.21.15.116:0/4237702120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb57809e910 0x7fb57809ed10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 --2- 172.21.15.116:0/4237702120 >> v2:172.21.15.116:3300/0 conn(0x7fb578131ab0 0x7fb578131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 -- 172.21.15.116:0/4237702120 >> 172.21.15.116:0/4237702120 conn(0x7fb578067d30 msgr2=0x7fb5780a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 -- 172.21.15.116:0/4237702120 shutdown_connections 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.346+0000 7fb57e754640 1 -- 172.21.15.116:0/4237702120 wait complete. 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.347+0000 7fb57e754640 10 monclient: build_initial_monmap 2024-09-18T12:31:57.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.347+0000 7fb57e754640 10 monclient: monmap: 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.347+0000 7fb57e754640 1 Processor -- start 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.347+0000 7fb57e754640 1 -- start start 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.347+0000 7fb57e754640 10 monclient: init 2024-09-18T12:31:57.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient: _reopen_session rank -1 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57809e910 0x7fb57811e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): picked mon.noname-a con 0x7fb57809e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 0x7fb57811e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): picked mon.noname-c con 0x7fb57809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb578131ab0 0x7fb57811eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): picked mon.noname-b con 0x7fb578131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb578121740 con 0x7fb57809e910 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57809bde0 con 0x7fb578131ab0 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): start opening mon connection 2024-09-18T12:31:57.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb577fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57809e910 msgr2=0x7fb57811e040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780a44a0 con 0x7fb57809f250 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): _renew_subs 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57e754640 10 monclient(hunting): authenticate will time out at 1144.669139s 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb577fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57809e910 0x7fb57811e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 0x7fb57811e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 0x7fb57811e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57458/0 (socket says 172.21.15.116:57458) 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 -- 172.21.15.116:0/1035858912 learned_addr learned my addr 172.21.15.116:0/1035858912 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): get_auth_request con 0x7fb57809f250 auth_method 0 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:57.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb578131ab0 0x7fb57811eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 10 monclient(hunting): get_auth_request con 0x7fb578131ab0 auth_method 0 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb57ccf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient(hunting): handle_auth_done global_id 64781 payload 824 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 -- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb578131ab0 msgr2=0x7fb57811eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb578131ab0 0x7fb57811eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 -- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57809e910 msgr2=0x7fb57811e040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57809e910 0x7fb57811e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient: _finish_hunting 0 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 monclient: found mon.noname-c 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.348+0000 7fb5777fe640 1 -- 172.21.15.116:0/1035858912 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb560009450 con 0x7fb57809f250 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.350+0000 7fb5777fe640 1 -- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 msgr2=0x7fb57811e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.350+0000 7fb5777fe640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 0x7fb57811e580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb56002a6c0 tx=0x7fb560032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.350+0000 7fb56ffff640 1 -- 172.21.15.116:0/1035858912 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560031a10 con 0x7fb57809f250 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.350+0000 7fb56ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.350+0000 7fb56ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.350+0000 7fb56ffff640 10 monclient: dump: 2024-09-18T12:31:57.363 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 10 monclient: _finish_auth 0 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 1 -- 172.21.15.116:0/1035858912 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb560031b90 con 0x7fb57809f250 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 5 monclient: authenticate success, global_id 64781 2024-09-18T12:31:57.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 1 -- 172.21.15.116:0/1035858912 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56003c410 con 0x7fb57809f250 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 10 monclient: _renew_subs 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57811f9a0 con 0x7fb57809f250 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb5767fc640 4 set_mon_vals no callback set 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 10 monclient: dump: 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:31:57.365 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 10 monclient: _renew_subs 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5781314f0 con 0x7fb57809f250 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb56ffff640 1 -- 172.21.15.116:0/1035858912 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb560039070 con 0x7fb57809f250 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:57.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.351+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb578121980 con 0x7fb57809f250 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb56ffff640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb56ffff640 1 -- 172.21.15.116:0/1035858912 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(164..164 src has 13..164) v4 ==== 9160+0+0 (secure 0 0 0) 0x7fb56006c970 con 0x7fb57809f250 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb577fff640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb577fff640 10 monclient: get_auth_request con 0x7fb558038440 auth_method 0 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb577fff640 1 -- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 msgr2=0x7fb55803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb577fff640 1 -- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 msgr2=0x7fb55803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb577fff640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fb568006fd0 tx=0x7fb568006d30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:31:57.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.352+0000 7fb577fff640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:31:57.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.356+0000 7fb56ffff640 1 -- 172.21.15.116:0/1035858912 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb560042af0 con 0x7fb57809f250 2024-09-18T12:31:57.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.356+0000 7fb56ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:31:57.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.356+0000 7fb56ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:31:57.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.553+0000 7fb577fff640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:31:57.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.553+0000 7fb577fff640 10 monclient: get_auth_request con 0x7fb558038440 auth_method 0 2024-09-18T12:31:57.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.556+0000 7fb577fff640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb5680092b0 tx=0x7fb568006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:31:57.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.571+0000 7fb57e754640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:31:57.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.571+0000 7fb57e754640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:57.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.571+0000 7fb57e754640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:31:57.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.571+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb57804bf60 con 0x7fb57809f250 2024-09-18T12:31:57.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.571+0000 7fb56ffff640 1 -- 172.21.15.116:0/1035858912 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fb560040070 con 0x7fb57809f250 2024-09-18T12:31:57.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.571+0000 7fb56ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:31:57.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.571+0000 7fb56ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:31:57.573 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:31:57.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 msgr2=0x7fb55803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb5680092b0 tx=0x7fb568006d30 comp rx=0 tx=0).stop 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 10 monclient: shutdown 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 msgr2=0x7fb57811e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 0x7fb57811e580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb56002a6c0 tx=0x7fb560032000 comp rx=0 tx=0).stop 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 shutdown_connections 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb558038440 0x7fb55803a8f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb578131ab0 0x7fb57811eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb57809f250 0x7fb57811e580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 --2- 172.21.15.116:0/1035858912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57809e910 0x7fb57811e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 >> 172.21.15.116:0/1035858912 conn(0x7fb578067d30 msgr2=0x7fb57809b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:31:57.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 shutdown_connections 2024-09-18T12:31:57.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:31:57.573+0000 7fb57e754640 1 -- 172.21.15.116:0/1035858912 wait complete. 2024-09-18T12:31:58.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:58.139+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:58.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:58.139+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:31:59.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:59.139+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:31:59.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:31:59.139+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:00.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:00.139+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:32:00.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:00.139+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:00.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:00.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.690+0000 7f49c46b8640 10 monclient: get_monmap_and_config 2024-09-18T12:32:00.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.690+0000 7f49c46b8640 10 monclient: build_initial_monmap 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.690+0000 7f49c46b8640 10 monclient: monmap: 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:00.691475+0000 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:00.691475+0000 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:00.693 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.691+0000 7f49c46b8640 1 Processor -- start 2024-09-18T12:32:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.691+0000 7f49c46b8640 1 -- start start 2024-09-18T12:32:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.691+0000 7f49c46b8640 10 monclient: init 2024-09-18T12:32:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.691+0000 7f49c46b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): picked mon.noname-b con 0x7f49bc0974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 0x7f49bc09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): picked mon.noname-a con 0x7f49bc09a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): picked mon.noname-c con 0x7f49bc12dd40 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49bc0978c0 con 0x7f49bc09a8a0 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49bc09ca20 con 0x7f49bc12dd40 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc09b1f0 con 0x7f49bc0974f0 2024-09-18T12:32:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c46b8640 10 monclient(hunting): authenticate will time out at 1148.013083s 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c1c53640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 msgr2=0x7f49bc09acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c1c53640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 0x7f49bc09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57470/0 (socket says 172.21.15.116:57470) 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 learned_addr learned my addr 172.21.15.116:0/4254732439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 10 monclient(hunting): get_auth_request con 0x7f49bc0974f0 auth_method 0 2024-09-18T12:32:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 10 monclient(hunting): get_auth_request con 0x7f49bc12dd40 auth_method 0 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:00.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:00.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.692+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:00.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 msgr2=0x7f49bc09acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c1c53640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 0x7f49bc09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 10 monclient(hunting): get_auth_request con 0x7f49bc0974f0 auth_method 0 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:00.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 10 monclient(hunting): get_auth_request con 0x7f49bc12dd40 auth_method 0 2024-09-18T12:32:00.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:00.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:00.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:00.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:00.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:00.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:00.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:00.893+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:32:01.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:01.140+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:32:01.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:01.140+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 msgr2=0x7f49bc09acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c1c53640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 0x7f49bc09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 10 monclient(hunting): get_auth_request con 0x7f49bc0974f0 auth_method 0 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 10 monclient(hunting): get_auth_request con 0x7f49bc12dd40 auth_method 0 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:01.294+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 msgr2=0x7f49bc09acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c1c53640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 0x7f49bc09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 10 monclient(hunting): get_auth_request con 0x7f49bc0974f0 auth_method 0 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2c55640 10 monclient(hunting): get_auth_request con 0x7f49bc12dd40 auth_method 0 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2c55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.095+0000 7f49c2c55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.096+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.096+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.096+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:02.096+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:32:02.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:02.140+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:32:02.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:02.140+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:02.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:02.140+0000 7f8494ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:03.140 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:03.140+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:32:03.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:03.140+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:03.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 10 monclient(hunting): tick 2024-09-18T12:32:03.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 monclient(hunting): continuing hunt 2024-09-18T12:32:03.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:32:03.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 msgr2=0x7f49bc12ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:03.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49bc0974f0 0x7f49bc12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 msgr2=0x7f49bc12e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.144:3300/0 conn(0x7f49bc12dd40 0x7f49bc12e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 -- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 msgr2=0x7f49bc09acb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 --2- 172.21.15.116:0/4254732439 >> v2:172.21.15.116:3300/0 conn(0x7f49bc09a8a0 0x7f49bc09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 10 monclient: _add_conns ranks=[1,0] 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49a0003630 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49a0007370 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 -- 172.21.15.116:0/4254732439 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc098ad0 con 0x7f49a0007370 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 1 -- 172.21.15.116:0/4254732439 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a000c0a0 con 0x7f49a0003630 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49b3fff640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:03.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.692+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> v1:172.21.15.116:6789/0 conn(0x7f49a0007370 legacy=0x7f49a0009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:03.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.694+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> v1:172.21.15.144:6789/0 conn(0x7f49a0003630 legacy=0x7f49a0005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:03.893+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> v1:172.21.15.116:6789/0 conn(0x7f49a0007370 legacy=0x7f49a0009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:04.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:04.140+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:32:04.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:04.140+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:04.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:04.294+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> v1:172.21.15.116:6789/0 conn(0x7f49a0007370 legacy=0x7f49a0009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.095+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> v1:172.21.15.116:6789/0 conn(0x7f49a0007370 legacy=0x7f49a0009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:05.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.140+0000 7f8494ff9640 10 monclient: tick 2024-09-18T12:32:05.141 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.140+0000 7f8494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.334+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624800129 0 0) 0x7f49a000c0a0 con 0x7f49a0003630 2024-09-18T12:32:05.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.334+0000 7f49c1452640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.334+0000 7f49c1452640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.334+0000 7f49c1452640 10 monclient(hunting): my global_id is 5980 2024-09-18T12:32:05.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.334+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a80034e0 con 0x7f49a0003630 2024-09-18T12:32:05.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429243233 0 0) 0x7f49a80034e0 con 0x7f49a0003630 2024-09-18T12:32:05.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 >> v1:172.21.15.116:6789/0 conn(0x7f49a0007370 legacy=0x7f49a0009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:05.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:32:05.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:32:05.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a000d260 con 0x7f49a0003630 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f49b800c680 con 0x7f49a0003630 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c46b8640 5 monclient: authenticate success, global_id 5980 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c46b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: dump: 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.341 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 >> v1:172.21.15.144:6789/0 conn(0x7f49a0003630 legacy=0x7f49a0005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:05.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c2c55640 1 -- 172.21.15.116:0/4254732439 reap_dead start 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a80039d0 0x7f49a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): picked mon.a con 0x7f49a80039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8004780 0x7f49a8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): picked mon.c con 0x7f49a8004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8005330 0x7f49a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): picked mon.b con 0x7f49a8005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a800e420 con 0x7f49a80039d0 2024-09-18T12:32:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a800e590 con 0x7f49a8005330 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a800e750 con 0x7f49a8004780 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c1452640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c2454640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a80039d0 msgr2=0x7f49a8003da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c46b8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.336+0000 7f49c2454640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a80039d0 0x7f49a8003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8004780 0x7f49a8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): get_auth_request con 0x7f49a8004780 auth_method 0 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8005330 0x7f49a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 10 monclient(hunting): get_auth_request con 0x7f49a8005330 auth_method 0 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient(hunting): handle_auth_done global_id 5980 payload 293 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8005330 msgr2=0x7f49a800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8005330 0x7f49a800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a80039d0 msgr2=0x7f49a8003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a80039d0 0x7f49a8003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 1 monclient: found mon.c 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a80034e0 con 0x7f49a8004780 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c1c53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.337+0000 7f49c2c55640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8005330 0x7f49a800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1c53640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8004780 msgr2=0x7f49a8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1c53640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8004780 0x7f49a8004bf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003940 tx=0x7f49ac009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f54261915d217645 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac02e420 con 0x7f49a8004780 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 10 monclient: dump: 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.352 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c46b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ac02e5c0 con 0x7f49a8004780 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c46b8640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c46b8640 4 set_mon_vals no callback set 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 1 -- 172.21.15.116:0/4254732439 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac02e980 con 0x7f49a8004780 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.340+0000 7f49c1452640 10 monclient: dump: 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 10 monclient: shutdown 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 -- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8004780 msgr2=0x7f49a8004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8004780 0x7f49a8004bf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003940 tx=0x7f49ac009590 comp rx=0 tx=0).stop 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 -- 172.21.15.116:0/4254732439 shutdown_connections 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a8005330 0x7f49a800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8004780 0x7f49a8004bf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 --2- 172.21.15.116:0/4254732439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a80039d0 0x7f49a8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 -- 172.21.15.116:0/4254732439 >> 172.21.15.116:0/4254732439 conn(0x7f49bc0672b0 msgr2=0x7f49bc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 -- 172.21.15.116:0/4254732439 shutdown_connections 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.341+0000 7f49c46b8640 1 -- 172.21.15.116:0/4254732439 wait complete. 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.342+0000 7f49c46b8640 10 monclient: build_initial_monmap 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.342+0000 7f49c46b8640 10 monclient: monmap: 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:05.355 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.342+0000 7f49c46b8640 1 Processor -- start 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.342+0000 7f49c46b8640 1 -- start start 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.342+0000 7f49c46b8640 10 monclient: init 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a80039d0 0x7f49bc13c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): picked mon.noname-b con 0x7f49a80039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8004780 0x7f49bc06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): picked mon.noname-a con 0x7f49a8004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8005330 0x7f49bc06cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): picked mon.noname-c con 0x7f49a8005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc099b40 con 0x7f49a8004780 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc06eb20 con 0x7f49a80039d0 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc098300 con 0x7f49a8005330 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c46b8640 10 monclient(hunting): authenticate will time out at 1152.664288s 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c1c53640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8004780 msgr2=0x7f49bc06c6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c1c53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8004780 0x7f49bc06c6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8005330 0x7f49bc06cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a80039d0 0x7f49bc13c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a80039d0 0x7f49bc13c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:59096/0 (socket says 172.21.15.116:59096) 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 1 -- 172.21.15.116:0/338096810 learned_addr learned my addr 172.21.15.116:0/338096810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 10 monclient(hunting): get_auth_request con 0x7f49a80039d0 auth_method 0 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): get_auth_request con 0x7f49a8005330 auth_method 0 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2454640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient(hunting): handle_auth_done global_id 64784 payload 824 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 1 -- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a80039d0 msgr2=0x7f49bc13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a80039d0 0x7f49bc13c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 1 -- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8004780 msgr2=0x7f49bc06c6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8004780 0x7f49bc06c6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 1 monclient: found mon.noname-c 2024-09-18T12:32:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.343+0000 7f49c2c55640 1 -- 172.21.15.116:0/338096810 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc117f00 con 0x7f49a8005330 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c2c55640 1 -- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8005330 msgr2=0x7f49bc06cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c2c55640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8005330 0x7f49bc06cc10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0987b0 tx=0x7f49b400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 1 -- 172.21.15.116:0/338096810 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4014cb0 con 0x7f49a8005330 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: dump: 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.360 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 1 -- 172.21.15.116:0/338096810 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b400f040 con 0x7f49a8005330 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c46b8640 5 monclient: authenticate success, global_id 64784 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c46b8640 10 monclient: _renew_subs 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c46b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49bc06e870 con 0x7f49a8005330 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 1 -- 172.21.15.116:0/338096810 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4021460 con 0x7f49a8005330 2024-09-18T12:32:05.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49b27fc640 10 monclient: dump: 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c0c51640 4 set_mon_vals no callback set 2024-09-18T12:32:05.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c46b8640 10 monclient: _renew_subs 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c46b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.346+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc06e120 con 0x7f49a8005330 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.347+0000 7f49b27fc640 1 -- 172.21.15.116:0/338096810 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f49b4003040 con 0x7f49a8005330 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.347+0000 7f49c46b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.347+0000 7f49c46b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.347+0000 7f49c46b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.347+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49bc04bf60 con 0x7f49a8005330 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.347+0000 7f49b27fc640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f49a0038cc0 0x7f49a003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.347+0000 7f49b27fc640 1 -- 172.21.15.116:0/338096810 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(164..164 src has 13..164) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f49b404d660 con 0x7f49a8005330 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.348+0000 7f49c2454640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f49a0038cc0 0x7f49a003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.348+0000 7f49c2454640 10 monclient: get_auth_request con 0x7f49a0038cc0 auth_method 0 2024-09-18T12:32:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.350+0000 7f49c2454640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f49a0038cc0 0x7f49a003b170 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c510 tx=0x7f49b8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.352+0000 7f49b27fc640 1 -- 172.21.15.116:0/338096810 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f49b4021850 con 0x7f49a8005330 2024-09-18T12:32:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.352+0000 7f49b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.352+0000 7f49b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:05.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.399+0000 7f8495ffb640 1 -- 172.21.15.144:0/824612391 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v165) v1 ==== 149+0+0 (secure 0 0 0) 0x7f849c016020 con 0x7f84a0069a30 2024-09-18T12:32:05.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.399+0000 7f8495ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:32:05.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.399+0000 7f8495ffb640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T12:32:05.401 INFO:tasks.workunit.client.0.smithi144.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8484038490 msgr2=0x7f848403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8484038490 0x7f848403a940 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f84900099b0 tx=0x7f8490008040 comp rx=0 tx=0).stop 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 10 monclient: shutdown 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 msgr2=0x7f84a012d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 0x7f84a012d200 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f849c00c5e0 tx=0x7f849c007ed0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 shutdown_connections 2024-09-18T12:32:05.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8484038490 0x7f848403a940 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f84a0134b70 0x7f84a0096ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a009baa0 0x7f84a012d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 --2- 172.21.15.144:0/824612391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84a0069a30 0x7f84a012d200 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 >> 172.21.15.144:0/824612391 conn(0x7f84a009e9f0 msgr2=0x7f84a00a1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 shutdown_connections 2024-09-18T12:32:05.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.402+0000 7f84a7ac9640 1 -- 172.21.15.144:0/824612391 wait complete. 2024-09-18T12:32:05.415 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-18T12:32:05.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.492+0000 7f4b6409f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.492+0000 7f4b6409f640 10 monclient: build_initial_monmap 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.492+0000 7f4b6409f640 10 monclient: monmap: 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:05.493616+0000 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:05.493616+0000 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:05.494 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:05.495 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 1 Processor -- start 2024-09-18T12:32:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 1 -- start start 2024-09-18T12:32:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient: init 2024-09-18T12:32:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T12:32:05.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b5c096a30 0x7f4b5c096e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient(hunting): picked mon.noname-c con 0x7f4b5c096a30 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c097340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient(hunting): picked mon.noname-b con 0x7f4b5c099f00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b5c09d4e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c097880 con 0x7f4b5c09d4e0 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5c09cb80 con 0x7f4b5c096a30 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c098c50 con 0x7f4b5c099f00 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.493+0000 7f4b6409f640 10 monclient(hunting): authenticate will time out at 1161.062967s 2024-09-18T12:32:05.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6263c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f4b5c09d4e0 legacy=0x7f4b5c12e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b5c096a30 0x7f4b5c096e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4b5c096a30 0x7f4b5c096e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43898/0 (socket says 172.21.15.144:43898) 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 -- 172.21.15.144:0/4010524258 learned_addr learned my addr 172.21.15.144:0/4010524258 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): get_auth_request con 0x7f4b5c096a30 auth_method 0 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 1 --2- 172.21.15.144:0/4010524258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c097340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 10 monclient(hunting): get_auth_request con 0x7f4b5c099f00 auth_method 0 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b6163a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_done global_id 5986 payload 274 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 -- 172.21.15.144:0/4010524258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 msgr2=0x7f4b5c097340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 --2- 172.21.15.144:0/4010524258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c097340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 -- 172.21.15.144:0/4010524258 >> v1:172.21.15.116:6789/0 conn(0x7f4b5c09d4e0 legacy=0x7f4b5c12e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 monclient: found mon.noname-c 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:05.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.494+0000 7f4b61e3b640 1 -- 172.21.15.144:0/4010524258 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c098f30 con 0x7f4b5c096a30 2024-09-18T12:32:05.506 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b61e3b640 1 -- 172.21.15.144:0/4010524258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 msgr2=0x7f4b5c096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b61e3b640 1 --2- 172.21.15.144:0/4010524258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c096e00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009910 tx=0x7f4b580091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aefdb2e667e46ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 1 -- 172.21.15.144:0/4010524258 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5802ae90 con 0x7f4b5c096a30 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: dump: 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.507 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 1 -- 172.21.15.144:0/4010524258 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b58032440 con 0x7f4b5c096a30 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 5 monclient: authenticate success, global_id 5986 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 1 -- 172.21.15.144:0/4010524258 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b580327e0 con 0x7f4b5c096a30 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b60e39640 10 monclient: dump: 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:05.508 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 4 set_mon_vals no callback set 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 10 monclient: shutdown 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 1 -- 172.21.15.144:0/4010524258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 msgr2=0x7f4b5c096e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 1 --2- 172.21.15.144:0/4010524258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c096e00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009910 tx=0x7f4b580091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 1 -- 172.21.15.144:0/4010524258 shutdown_connections 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 1 --2- 172.21.15.144:0/4010524258 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c097340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 1 --2- 172.21.15.144:0/4010524258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c096e00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 1 -- 172.21.15.144:0/4010524258 >> 172.21.15.144:0/4010524258 conn(0x7f4b5c0711a0 msgr2=0x7f4b5c0a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.497+0000 7f4b6409f640 1 -- 172.21.15.144:0/4010524258 shutdown_connections 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.498+0000 7f4b6409f640 1 -- 172.21.15.144:0/4010524258 wait complete. 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.498+0000 7f4b6409f640 10 monclient: build_initial_monmap 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.498+0000 7f4b6409f640 10 monclient: monmap: 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:05.510 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.498+0000 7f4b6409f640 1 Processor -- start 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.498+0000 7f4b6409f640 1 -- start start 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.498+0000 7f4b6409f640 10 monclient: init 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c12d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): picked mon.noname-b con 0x7f4b5c096a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): picked mon.noname-c con 0x7f4b5c099f00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b5c09d4e0 0x7f4b5c069a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): picked mon.noname-a con 0x7f4b5c09d4e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c09c5f0 con 0x7f4b5c09d4e0 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c11cf30 con 0x7f4b5c096a30 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c11cc50 con 0x7f4b5c099f00 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6409f640 10 monclient(hunting): authenticate will time out at 1161.068248s 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c12d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6263c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b5c09d4e0 msgr2=0x7f4b5c069a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6263c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b5c09d4e0 0x7f4b5c069a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c12d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43910/0 (socket says 172.21.15.144:43910) 2024-09-18T12:32:05.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 -- 172.21.15.144:0/3968226182 learned_addr learned my addr 172.21.15.144:0/3968226182 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): get_auth_request con 0x7f4b5c096a30 auth_method 0 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 10 monclient(hunting): get_auth_request con 0x7f4b5c099f00 auth_method 0 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b6163a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient(hunting): handle_auth_done global_id 5989 payload 824 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 -- 172.21.15.144:0/3968226182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 msgr2=0x7f4b5c0694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c0694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 -- 172.21.15.144:0/3968226182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b5c09d4e0 msgr2=0x7f4b5c069a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b5c09d4e0 0x7f4b5c069a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 monclient: found mon.noname-b 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.499+0000 7f4b61e3b640 1 -- 172.21.15.144:0/3968226182 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b58009450 con 0x7f4b5c096a30 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b61e3b640 1 -- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 msgr2=0x7f4b5c12d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b61e3b640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c12d830 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4b5802a680 tx=0x7f4b58032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b58000d10 con 0x7f4b5c096a30 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b51ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b51ffb640 10 monclient: dump: 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b51ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.515 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5802ae90 con 0x7f4b5c096a30 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b6409f640 5 monclient: authenticate success, global_id 5989 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b6409f640 10 monclient: _renew_subs 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.501+0000 7f4b6409f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c11c970 con 0x7f4b5c096a30 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b51ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b58031ea0 con 0x7f4b5c096a30 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b51ffb640 10 monclient: dump: 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.516 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b6409f640 10 monclient: _renew_subs 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b6409f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c141f90 con 0x7f4b5c096a30 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b53fff640 4 set_mon_vals no callback set 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.502+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4b58039040 con 0x7f4b5c096a30 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b51ffb640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4b40038440 0x7f4b4003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(165..165 src has 13..165) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f4b5806c440 con 0x7f4b5c096a30 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b6409f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b6409f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b6409f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c11d170 con 0x7f4b5c096a30 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b6163a640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4b40038440 0x7f4b4003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.503+0000 7f4b6163a640 10 monclient: get_auth_request con 0x7f4b40038440 auth_method 0 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.505+0000 7f4b6163a640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4b40038440 0x7f4b4003a8f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c133d30 tx=0x7f4b4c0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.507+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4b58037080 con 0x7f4b5c096a30 2024-09-18T12:32:05.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.507+0000 7f4b51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.507+0000 7f4b51ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.571+0000 7f49c46b8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.572+0000 7f49c46b8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.572+0000 7f49c46b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.572+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f49bc0638d0 con 0x7f49a8005330 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.572+0000 7f49b27fc640 1 -- 172.21.15.116:0/338096810 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f49b401f360 con 0x7f49a8005330 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.572+0000 7f49b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.572+0000 7f49b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:05.573 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":76,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f49a0038cc0 msgr2=0x7f49a003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f49a0038cc0 0x7f49a003b170 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c510 tx=0x7f49b8008040 comp rx=0 tx=0).stop 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 10 monclient: shutdown 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8005330 msgr2=0x7f49bc06cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8005330 0x7f49bc06cc10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0987b0 tx=0x7f49b400b040 comp rx=0 tx=0).stop 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 shutdown_connections 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f49a0038cc0 0x7f49a003b170 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a8005330 0x7f49bc06cc10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8004780 0x7f49bc06c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 --2- 172.21.15.116:0/338096810 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a80039d0 0x7f49bc13c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 >> 172.21.15.116:0/338096810 conn(0x7f49bc0672b0 msgr2=0x7f49bc1296c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 shutdown_connections 2024-09-18T12:32:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.574+0000 7f49c46b8640 1 -- 172.21.15.116:0/338096810 wait complete. 2024-09-18T12:32:05.585 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:32:05.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:32:05.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:32:05.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.665+0000 7f4b6409f640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T12:32:05.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.665+0000 7f4b6409f640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T12:32:05.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.665+0000 7f4b6409f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:05.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:05.665+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f4b5c0008d0 con 0x7f4b5c096a30 2024-09-18T12:32:05.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.678+0000 7f0b3a8dd640 10 monclient: get_monmap_and_config 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.678+0000 7f0b3a8dd640 10 monclient: build_initial_monmap 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.678+0000 7f0b3a8dd640 10 monclient: monmap: 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:05.678830+0000 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:05.678830+0000 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:05.679 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:05.680 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:05.680 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.678+0000 7f0b3a8dd640 1 Processor -- start 2024-09-18T12:32:05.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.678+0000 7f0b3a8dd640 1 -- start start 2024-09-18T12:32:05.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.678+0000 7f0b3a8dd640 10 monclient: init 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0b340974f0 0x7f0b3412ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f0b340974f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b3409a8a0 0x7f0b3409ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f0b3409a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3412e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f0b3412dd40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b340978c0 con 0x7f0b3409a8a0 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3409b220 con 0x7f0b340974f0 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3409ca20 con 0x7f0b3412dd40 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b3a8dd640 10 monclient(hunting): authenticate will time out at 1153.000446s 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b337fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f0b3409a8a0 msgr2=0x7f0b3409ace0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b337fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b3409a8a0 0x7f0b3409ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3412e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3412e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57990/0 (socket says 172.21.15.116:57990) 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 -- 172.21.15.116:0/368552058 learned_addr learned my addr 172.21.15.116:0/368552058 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): get_auth_request con 0x7f0b3412dd40 auth_method 0 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 1 --2- 172.21.15.116:0/368552058 >> v2:172.21.15.144:3300/0 conn(0x7f0b340974f0 0x7f0b3412ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 10 monclient(hunting): get_auth_request con 0x7f0b340974f0 auth_method 0 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_done global_id 64793 payload 274 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 -- 172.21.15.116:0/368552058 >> v2:172.21.15.144:3300/0 conn(0x7f0b340974f0 msgr2=0x7f0b3412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 --2- 172.21.15.116:0/368552058 >> v2:172.21.15.144:3300/0 conn(0x7f0b340974f0 0x7f0b3412ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 -- 172.21.15.116:0/368552058 >> v2:172.21.15.116:3300/0 conn(0x7f0b3409a8a0 msgr2=0x7f0b3409ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 --2- 172.21.15.116:0/368552058 >> v2:172.21.15.116:3300/0 conn(0x7f0b3409a8a0 0x7f0b3409ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 monclient: found mon.noname-b 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.679+0000 7f0b38e7a640 1 -- 172.21.15.116:0/368552058 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b34098ad0 con 0x7f0b3412dd40 2024-09-18T12:32:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b38e7a640 1 -- 172.21.15.116:0/368552058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 msgr2=0x7f0b3412e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b38e7a640 1 --2- 172.21.15.116:0/368552058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3412e170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b340768c0 tx=0x7f0b2c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=493bcad444146eff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 1 -- 172.21.15.116:0/368552058 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c007610 con 0x7f0b3412dd40 2024-09-18T12:32:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: dump: 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 1 -- 172.21.15.116:0/368552058 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c02d050 con 0x7f0b3412dd40 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 1 -- 172.21.15.116:0/368552058 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c0079e0 con 0x7f0b3412dd40 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b3a8dd640 5 monclient: authenticate success, global_id 64793 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b3a8dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b3a8dd640 4 set_mon_vals no callback set 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.682+0000 7f0b32ffd640 10 monclient: dump: 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 10 monclient: shutdown 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/368552058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 msgr2=0x7f0b3412e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/368552058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3412e170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b340768c0 tx=0x7f0b2c009840 comp rx=0 tx=0).stop 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/368552058 shutdown_connections 2024-09-18T12:32:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/368552058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3412e170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/368552058 >> v2:172.21.15.116:3300/0 conn(0x7f0b3409a8a0 0x7f0b3409ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/368552058 >> v2:172.21.15.144:3300/0 conn(0x7f0b340974f0 0x7f0b3412ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/368552058 >> 172.21.15.116:0/368552058 conn(0x7f0b340672b0 msgr2=0x7f0b340676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/368552058 shutdown_connections 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/368552058 wait complete. 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 10 monclient: build_initial_monmap 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.683+0000 7f0b3a8dd640 10 monclient: monmap: 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 Processor -- start 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 -- start start 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient: init 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b340974f0 0x7f0b34128680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f0b340974f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b3409a8a0 0x7f0b3406d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f0b3409a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f0b3412dd40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341185f0 con 0x7f0b3409a8a0 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3406f5d0 con 0x7f0b340974f0 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34129c70 con 0x7f0b3412dd40 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b3a8dd640 10 monclient(hunting): authenticate will time out at 1153.005883s 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b337fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b3409a8a0 msgr2=0x7f0b3406d3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b3409a8a0 0x7f0b3406d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b38e7a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.684+0000 7f0b38e7a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3406d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:57998/0 (socket says 172.21.15.116:57998) 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 1 -- 172.21.15.116:0/3715758052 learned_addr learned my addr 172.21.15.116:0/3715758052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:05.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): get_auth_request con 0x7f0b3412dd40 auth_method 0 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b33fff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b340974f0 0x7f0b34128680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b33fff640 10 monclient(hunting): get_auth_request con 0x7f0b340974f0 auth_method 0 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient(hunting): handle_auth_done global_id 64796 payload 824 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b340974f0 msgr2=0x7f0b34128680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b340974f0 0x7f0b34128680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b3409a8a0 msgr2=0x7f0b3406d3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b3409a8a0 0x7f0b3406d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 1 monclient: found mon.noname-c 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b38e7a640 1 -- 172.21.15.116:0/3715758052 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c009ae0 con 0x7f0b3412dd40 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.685+0000 7f0b33fff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0b340974f0 0x7f0b34128680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.687+0000 7f0b38e7a640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 msgr2=0x7f0b3406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.687+0000 7f0b38e7a640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3406d8e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009f50 tx=0x7f0b2c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.687+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c007170 con 0x7f0b3412dd40 2024-09-18T12:32:05.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: dump: 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.701 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c007e10 con 0x7f0b3412dd40 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b3a8dd640 5 monclient: authenticate success, global_id 64796 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c03c420 con 0x7f0b3412dd40 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b3a8dd640 10 monclient: _renew_subs 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b3a8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b34118e90 con 0x7f0b3412dd40 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b1bfff640 10 monclient: dump: 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b327fc640 4 set_mon_vals no callback set 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b3a8dd640 10 monclient: _renew_subs 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b3a8dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.689+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b34117700 con 0x7f0b3412dd40 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b3a8dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 0x7f0b340696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34065f20 con 0x7f0b3406de20 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 0x7f0b340696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0b2c003020 con 0x7f0b3412dd40 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient: get_auth_request con 0x7f0b3406de20 auth_method 0 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b1bfff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b200384c0 0x7f0b2003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 10 monclient(hunting): handle_auth_done global_id 64799 payload 824 2024-09-18T12:32:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(165..165 src has 13..165) v4 ==== 9160+0+0 (secure 0 0 0) 0x7f0b2c06d3a0 con 0x7f0b3412dd40 2024-09-18T12:32:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.690+0000 7f0b33fff640 1 -- 172.21.15.116:0/3715758052 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b34118af0 con 0x7f0b3406de20 2024-09-18T12:32:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.691+0000 7f0b337fe640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b200384c0 0x7f0b2003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.691+0000 7f0b337fe640 10 monclient: get_auth_request con 0x7f0b200384c0 auth_method 0 2024-09-18T12:32:05.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b33fff640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 msgr2=0x7f0b340696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b33fff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 0x7f0b340696a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0b24011840 tx=0x7f0b24008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b24011eb0 con 0x7f0b3406de20 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 10 monclient: dump: 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0b2400fd60 con 0x7f0b3406de20 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 msgr2=0x7f0b340696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 0x7f0b340696a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0b24011840 tx=0x7f0b24008040 comp rx=0 tx=0).stop 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 msgr2=0x7f0b340696a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b1bfff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3406de20 0x7f0b340696a0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0b24011840 tx=0x7f0b24008040 comp rx=0 tx=0).stop 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.693+0000 7f0b337fe640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b200384c0 0x7f0b2003a970 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0b3406ed40 tx=0x7f0b28006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.698+0000 7f0b3a8dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.698+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 0x7f0b34069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.698+0000 7f0b3a8dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.698+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34119020 con 0x7f0b34069be0 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 0x7f0b34069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient: get_auth_request con 0x7f0b34069be0 auth_method 0 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 10 monclient(hunting): handle_auth_done global_id 64802 payload 824 2024-09-18T12:32:05.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.699+0000 7f0b33fff640 1 -- 172.21.15.116:0/3715758052 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0b3404bf60 con 0x7f0b34069be0 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b33fff640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 msgr2=0x7f0b34069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b33fff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 0x7f0b34069fe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0b2401c000 tx=0x7f0b2401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2401dbd0 con 0x7f0b34069be0 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 10 monclient: dump: 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f0b2401dd30 con 0x7f0b34069be0 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 msgr2=0x7f0b34069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 0x7f0b34069fe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0b2401c000 tx=0x7f0b2401d000 comp rx=0 tx=0).stop 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 msgr2=0x7f0b34069fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.701+0000 7f0b1bfff640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b34069be0 0x7f0b34069fe0 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0b2401c000 tx=0x7f0b2401d000 comp rx=0 tx=0).stop 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 76, 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:05.713 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:05.715 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:05.716 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:05.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:05.718 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.720 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:05.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:05.722 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:05.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.724 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:05.725 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b200384c0 msgr2=0x7f0b2003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0b200384c0 0x7f0b2003a970 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0b3406ed40 tx=0x7f0b28006d80 comp rx=0 tx=0).stop 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 10 monclient: shutdown 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 msgr2=0x7f0b3406d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3406d8e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009f50 tx=0x7f0b2c036000 comp rx=0 tx=0).stop 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b38e7a640 1 -- 172.21.15.116:0/3715758052 reap_dead start 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 shutdown_connections 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 --2- 172.21.15.116:0/3715758052 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0b3412dd40 0x7f0b3406d8e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 >> 172.21.15.116:0/3715758052 conn(0x7f0b340672b0 msgr2=0x7f0b3412d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 shutdown_connections 2024-09-18T12:32:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.702+0000 7f0b3a8dd640 1 -- 172.21.15.116:0/3715758052 wait complete. 2024-09-18T12:32:05.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:32:05.728 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:32:05.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:32:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.821+0000 7ff22911d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.821+0000 7ff22911d640 10 monclient: build_initial_monmap 2024-09-18T12:32:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.821+0000 7ff22911d640 10 monclient: monmap: 2024-09-18T12:32:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:05.821918+0000 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:05.821918+0000 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.821+0000 7ff22911d640 1 Processor -- start 2024-09-18T12:32:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.821+0000 7ff22911d640 1 -- start start 2024-09-18T12:32:05.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.821+0000 7ff22911d640 10 monclient: init 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff224096e10 0x7ff22409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient(hunting): picked mon.noname-a con 0x7ff224096e10 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff22409aaa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2240978f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22409d4a0 con 0x7ff2240978f0 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff224099620 con 0x7ff22409aaa0 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff224099340 con 0x7ff224096e10 2024-09-18T12:32:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22911d640 10 monclient(hunting): authenticate will time out at 1153.143345s 2024-09-18T12:32:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22359e640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7ff2240978f0 legacy=0x7ff224135b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff222d9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7ff224096e10 msgr2=0x7ff22409a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff222d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff224096e10 0x7ff22409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22259c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff22409aaa0 0x7ff2240971e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:49936/0 (socket says 172.21.15.116:49936) 2024-09-18T12:32:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.822+0000 7ff22259c640 1 -- 172.21.15.116:0/1693188047 learned_addr learned my addr 172.21.15.116:0/1693188047 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:32:05.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.824+0000 7ff22259c640 1 -- 172.21.15.116:0/1693188047 >> v1:172.21.15.144:6789/0 conn(0x7ff22409aaa0 legacy=0x7ff2240971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290414745 0 0) 0x7ff224099620 con 0x7ff22409aaa0 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): my global_id is 5998 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2100034e0 con 0x7ff22409aaa0 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753318872 0 0) 0x7ff2100034e0 con 0x7ff22409aaa0 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 >> v2:172.21.15.116:3300/0 conn(0x7ff224096e10 msgr2=0x7ff22409a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 --2- 172.21.15.116:0/1693188047 >> v2:172.21.15.116:3300/0 conn(0x7ff224096e10 0x7ff22409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 >> v1:172.21.15.116:6789/0 conn(0x7ff2240978f0 legacy=0x7ff224135b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:32:05.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff224099900 con 0x7ff22409aaa0 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7ff20c001790 con 0x7ff22409aaa0 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22911d640 5 monclient: authenticate success, global_id 5998 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22911d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: dump: 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.830 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 >> v1:172.21.15.144:6789/0 conn(0x7ff22409aaa0 legacy=0x7ff2240971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210003aa0 0x7ff210003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): picked mon.a con 0x7ff210003aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff210004780 0x7ff210004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): picked mon.c con 0x7ff210004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2100099c0 0x7ff21000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): picked mon.b con 0x7ff2100099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff224099340 con 0x7ff210003aa0 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21000e5d0 con 0x7ff2100099c0 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21000e790 con 0x7ff210004780 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff222d9d640 1 -- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210003aa0 msgr2=0x7ff210003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff221d9b640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22911d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff222d9d640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210003aa0 0x7ff210003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff210004780 0x7ff210004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 10 monclient(hunting): get_auth_request con 0x7ff210004780 auth_method 0 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2100099c0 0x7ff21000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 10 monclient(hunting): get_auth_request con 0x7ff2100099c0 auth_method 0 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.825+0000 7ff22359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 10 monclient(hunting): handle_auth_done global_id 5998 payload 293 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 1 -- 172.21.15.116:0/1693188047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2100099c0 msgr2=0x7ff21000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2100099c0 0x7ff21000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 1 -- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210003aa0 msgr2=0x7ff210003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210003aa0 0x7ff210003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 1 monclient: found mon.c 2024-09-18T12:32:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22359e640 1 -- 172.21.15.116:0/1693188047 reap_dead start 2024-09-18T12:32:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 1 -- 172.21.15.116:0/1693188047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2100034e0 con 0x7ff210004780 2024-09-18T12:32:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.826+0000 7ff22259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff22259c640 1 -- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff210004780 msgr2=0x7ff210004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff22259c640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff210004780 0x7ff210004bf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff20c001970 tx=0x7ff20c007a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf278a707d6b05e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c005890 con 0x7ff210004780 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 10 monclient: dump: 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff22911d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c005a10 con 0x7ff210004780 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff22911d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff22911d640 4 set_mon_vals no callback set 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 1 -- 172.21.15.116:0/1693188047 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c005dd0 con 0x7ff210004780 2024-09-18T12:32:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.828+0000 7ff221d9b640 10 monclient: dump: 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.844 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 10 monclient: shutdown 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 1 -- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff210004780 msgr2=0x7ff210004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff210004780 0x7ff210004bf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff20c001970 tx=0x7ff20c007a90 comp rx=0 tx=0).stop 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 1 -- 172.21.15.116:0/1693188047 shutdown_connections 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 1 --2- 172.21.15.116:0/1693188047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff210004780 0x7ff210004bf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 1 -- 172.21.15.116:0/1693188047 >> 172.21.15.116:0/1693188047 conn(0x7ff2240672b0 msgr2=0x7ff2240676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 1 -- 172.21.15.116:0/1693188047 shutdown_connections 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.829+0000 7ff22911d640 1 -- 172.21.15.116:0/1693188047 wait complete. 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient: build_initial_monmap 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient: monmap: 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 Processor -- start 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 -- start start 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient: init 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210004780 0x7ff22413cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient(hunting): picked mon.noname-a con 0x7ff210004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22413d080 0x7ff224139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient(hunting): picked mon.noname-b con 0x7ff22413d080 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 0x7ff22411a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient(hunting): picked mon.noname-c con 0x7ff22413a180 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240a0190 con 0x7ff210004780 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22413a600 con 0x7ff22413d080 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22411e290 con 0x7ff22413a180 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.830+0000 7ff22911d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22911d640 10 monclient(hunting): authenticate will time out at 1153.152008s 2024-09-18T12:32:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff222d9d640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210004780 msgr2=0x7ff22413cb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff222d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210004780 0x7ff22413cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 0x7ff22411a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 0x7ff22411a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58024/0 (socket says 172.21.15.116:58024) 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 -- 172.21.15.116:0/74321707 learned_addr learned my addr 172.21.15.116:0/74321707 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): get_auth_request con 0x7ff22413a180 auth_method 0 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22413d080 0x7ff224139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 10 monclient(hunting): get_auth_request con 0x7ff22413d080 auth_method 0 2024-09-18T12:32:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient(hunting): handle_auth_done global_id 64805 payload 824 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22413d080 msgr2=0x7ff224139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22413d080 0x7ff224139c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210004780 msgr2=0x7ff22413cb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff210004780 0x7ff22413cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient: _finish_hunting 0 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 monclient: found mon.noname-c 2024-09-18T12:32:05.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.831+0000 7ff22359e640 1 -- 172.21.15.116:0/74321707 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22411dfb0 con 0x7ff22413a180 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22359e640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 msgr2=0x7ff22411a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22359e640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 0x7ff22411a640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff2240980c0 tx=0x7ff21c0095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c011070 con 0x7ff22413a180 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22359e640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 0x7ff22411a640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff2240980c0 tx=0x7ff21c0095d0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7ff21c00b820 config(0 keys) v1 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 10 monclient: dump: 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 10 monclient: _finish_auth 0 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22911d640 5 monclient: authenticate success, global_id 64805 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22911d640 10 monclient: _renew_subs 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22911d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c00b820 con 0x7ff22413a180 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22411c1a0 con 0x7ff22413a180 2024-09-18T12:32:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff21affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22911d640 10 monclient: _renew_subs 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22911d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22159a640 4 set_mon_vals no callback set 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.833+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22411cd70 con 0x7ff22413a180 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff22911d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff22911d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 0x7ff224029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff22911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22411de70 con 0x7ff2240298c0 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c00bbe0 con 0x7ff22413a180 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff21affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff21affd640 10 monclient: dump: 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.852 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff21c016420 con 0x7ff22413a180 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff21affd640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff204038440 0x7ff20403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff222d9d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 0x7ff224029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff222d9d640 10 monclient: get_auth_request con 0x7ff2240298c0 auth_method 0 2024-09-18T12:32:05.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff222d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff222d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.834+0000 7ff222d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.835+0000 7ff222d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.835+0000 7ff222d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.835+0000 7ff22259c640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff204038440 0x7ff20403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.835+0000 7ff222d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.835+0000 7ff22259c640 10 monclient: get_auth_request con 0x7ff204038440 auth_method 0 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.835+0000 7ff222d9d640 10 monclient(hunting): handle_auth_done global_id 6004 payload 824 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.835+0000 7ff222d9d640 1 -- 172.21.15.116:0/74321707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff22404bf60 con 0x7ff2240298c0 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff22259c640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff204038440 0x7ff20403a8f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff20c033000 tx=0x7ff20c034000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff222d9d640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 msgr2=0x7ff224029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff222d9d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 0x7ff224029c90 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b480 tx=0x7ff2140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214003e80 con 0x7ff2240298c0 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 10 monclient: dump: 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff2140050f0 con 0x7ff2240298c0 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 msgr2=0x7ff224029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 0x7ff224029c90 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b480 tx=0x7ff2140091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 msgr2=0x7ff224029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.837+0000 7ff21affd640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2240298c0 0x7ff224029c90 secure :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b480 tx=0x7ff2140091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff22911d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff22911d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 0x7ff22402a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff22911d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff224065f20 con 0x7ff22402a1d0 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 0x7ff22402a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 10 monclient: get_auth_request con 0x7ff22402a1d0 auth_method 0 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.843+0000 7ff222d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.844+0000 7ff222d9d640 10 monclient(hunting): handle_auth_done global_id 6007 payload 824 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.844+0000 7ff222d9d640 1 -- 172.21.15.116:0/74321707 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff22406f3d0 con 0x7ff22402a1d0 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.846+0000 7ff222d9d640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 msgr2=0x7ff22402a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.846+0000 7ff222d9d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 0x7ff22402a5a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b480 tx=0x7ff214015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21401b440 con 0x7ff22402a1d0 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 10 monclient: dump: 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7ff21401b5a0 con 0x7ff22402a1d0 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 msgr2=0x7ff22402a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 0x7ff22402a5a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b480 tx=0x7ff214015000 comp rx=0 tx=0).stop 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 msgr2=0x7ff22402a5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff22402a1d0 0x7ff22402a5a0 secure :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff21400b480 tx=0x7ff214015000 comp rx=0 tx=0).stop 2024-09-18T12:32:05.860 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 76, 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:05.861 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:05.862 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:05.863 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:05.864 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:05.865 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:05.866 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:05.867 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.869 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:05.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.871 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.872 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:05.873 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:05.874 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff204038440 msgr2=0x7ff20403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff204038440 0x7ff20403a8f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff20c033000 tx=0x7ff20c034000 comp rx=0 tx=0).stop 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 10 monclient: shutdown 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 msgr2=0x7ff22411a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 --2- 172.21.15.116:0/74321707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff22413a180 0x7ff22411a640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff2240980c0 tx=0x7ff21c0095d0 comp rx=0 tx=0).stop 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22359e640 1 -- 172.21.15.116:0/74321707 reap_dead start 2024-09-18T12:32:05.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22359e640 1 -- 172.21.15.116:0/74321707 reap_dead start 2024-09-18T12:32:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff21affd640 1 -- 172.21.15.116:0/74321707 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(165..165 src has 13..165) v4 ==== 9160+0+0 (unknown 0 0 0) 0x7ff21c006e40 con 0x7ff22413a180 2024-09-18T12:32:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 shutdown_connections 2024-09-18T12:32:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 >> 172.21.15.116:0/74321707 conn(0x7ff2240672b0 msgr2=0x7ff2240a0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 shutdown_connections 2024-09-18T12:32:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:05.847+0000 7ff22911d640 1 -- 172.21.15.116:0/74321707 wait complete. 2024-09-18T12:32:05.876 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:32:05.876 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:32:06.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.403+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v166) v1 ==== 145+0+0 (secure 0 0 0) 0x7f4b58042020 con 0x7f4b5c096a30 2024-09-18T12:32:06.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.403+0000 7f4b51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T12:32:06.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.403+0000 7f4b51ffb640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-18T12:32:06.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.500+0000 7f4b6409f640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T12:32:06.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.500+0000 7f4b6409f640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T12:32:06.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.500+0000 7f4b6409f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:06.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.500+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f4b5c0008d0 con 0x7f4b5c096a30 2024-09-18T12:32:06.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.589+0000 7f4b51ffb640 1 -- 172.21.15.144:0/3968226182 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v167) v1 ==== 257+0+0 (secure 0 0 0) 0x7f4b58044540 con 0x7f4b5c096a30 2024-09-18T12:32:06.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.589+0000 7f4b51ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T12:32:06.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.589+0000 7f4b51ffb640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-18T12:32:06.591 INFO:tasks.workunit.client.0.smithi144.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4b40038440 msgr2=0x7f4b4003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4b40038440 0x7f4b4003a8f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c133d30 tx=0x7f4b4c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 10 monclient: shutdown 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 msgr2=0x7f4b5c12d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c12d830 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4b5802a680 tx=0x7f4b58032000 comp rx=0 tx=0).stop 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 shutdown_connections 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4b40038440 0x7f4b4003a8f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b5c09d4e0 0x7f4b5c069a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4b5c099f00 0x7f4b5c0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 --2- 172.21.15.144:0/3968226182 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b5c096a30 0x7f4b5c12d830 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.591+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 >> 172.21.15.144:0/3968226182 conn(0x7f4b5c0711a0 msgr2=0x7f4b5c06d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:06.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.592+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 shutdown_connections 2024-09-18T12:32:06.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.592+0000 7f4b6409f640 1 -- 172.21.15.144:0/3968226182 wait complete. 2024-09-18T12:32:06.603 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:32:06.603 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:32:06.603 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep globster 2024-09-18T12:32:06.603 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '10\.' 2024-09-18T12:32:06.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.675+0000 7fe2af446640 10 monclient: get_monmap_and_config 2024-09-18T12:32:06.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.675+0000 7fe2af446640 10 monclient: build_initial_monmap 2024-09-18T12:32:06.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.675+0000 7fe2af446640 10 monclient: monmap: 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:06.676377+0000 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:06.676377+0000 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:06.677 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 1 Processor -- start 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 1 -- start start 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient: init 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a809c250 0x7fe2a809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient(hunting): picked mon.noname-a con 0x7fe2a809c250 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 0x7fe2a809e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient(hunting): picked mon.noname-b con 0x7fe2a809dfa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2a809c620 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:06.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a809b790 con 0x7fe2a809c620 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a809b230 con 0x7fe2a809c250 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a809e8b0 con 0x7fe2a809dfa0 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2af446640 10 monclient(hunting): authenticate will time out at 1162.245841s 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2ad1e2640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fe2a809c250 msgr2=0x7fe2a809da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2ad9e3640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fe2a809c620 legacy=0x7fe2a809ca80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.676+0000 7fe2ad1e2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a809c250 0x7fe2a809da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 0x7fe2a809e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 0x7fe2a809e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:40324/0 (socket says 172.21.15.144:40324) 2024-09-18T12:32:06.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/2933468217 learned_addr learned my addr 172.21.15.144:0/2933468217 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): get_auth_request con 0x7fe2a809dfa0 auth_method 0 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_done global_id 64808 payload 274 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/2933468217 >> v2:172.21.15.116:3300/0 conn(0x7fe2a809c250 msgr2=0x7fe2a809da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 --2- 172.21.15.144:0/2933468217 >> v2:172.21.15.116:3300/0 conn(0x7fe2a809c250 0x7fe2a809da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/2933468217 >> v1:172.21.15.116:6789/0 conn(0x7fe2a809c620 legacy=0x7fe2a809ca80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient: _finish_hunting 0 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 monclient: found mon.noname-b 2024-09-18T12:32:06.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:06.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:06.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.677+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/2933468217 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a809b4b0 con 0x7fe2a809dfa0 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/2933468217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 msgr2=0x7fe2a809e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2ac9e1640 1 --2- 172.21.15.144:0/2933468217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 0x7fe2a809e370 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80a3fd0 tx=0x7fe29c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f3ddd913044e8da server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 1 -- 172.21.15.144:0/2933468217 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c032440 con 0x7fe2a809dfa0 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: dump: 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.688 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: _finish_auth 0 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 5 monclient: authenticate success, global_id 64808 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 1 -- 172.21.15.144:0/2933468217 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29c0325a0 con 0x7fe2a809dfa0 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 4 set_mon_vals no callback set 2024-09-18T12:32:06.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 1 -- 172.21.15.144:0/2933468217 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c032980 con 0x7fe2a809dfa0 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2a7fff640 10 monclient: dump: 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 10 monclient: shutdown 2024-09-18T12:32:06.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 1 -- 172.21.15.144:0/2933468217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 msgr2=0x7fe2a809e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.680+0000 7fe2af446640 1 --2- 172.21.15.144:0/2933468217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 0x7fe2a809e370 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80a3fd0 tx=0x7fe29c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 -- 172.21.15.144:0/2933468217 shutdown_connections 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 --2- 172.21.15.144:0/2933468217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809dfa0 0x7fe2a809e370 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 --2- 172.21.15.144:0/2933468217 >> v2:172.21.15.116:3300/0 conn(0x7fe2a809c250 0x7fe2a809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 -- 172.21.15.144:0/2933468217 >> 172.21.15.144:0/2933468217 conn(0x7fe2a8071340 msgr2=0x7fe2a80a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 -- 172.21.15.144:0/2933468217 shutdown_connections 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 -- 172.21.15.144:0/2933468217 wait complete. 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 10 monclient: build_initial_monmap 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 10 monclient: monmap: 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:06.691 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 Processor -- start 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 1 -- start start 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.681+0000 7fe2af446640 10 monclient: init 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809c250 0x7fe2a8141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): picked mon.noname-c con 0x7fe2a809c250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:06.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 0x7fe2a8131bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): picked mon.noname-b con 0x7fe2a809c620 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a809dfa0 0x7fe2a81320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): picked mon.noname-a con 0x7fe2a809dfa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a8132950 con 0x7fe2a809dfa0 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a81212b0 con 0x7fe2a809c620 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a8120fd0 con 0x7fe2a809c250 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2af446640 10 monclient(hunting): authenticate will time out at 1162.251559s 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 0x7fe2a8131bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ad9e3640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a809dfa0 msgr2=0x7fe2a81320f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:06.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ad9e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a809dfa0 0x7fe2a81320f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 0x7fe2a8131bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43926/0 (socket says 172.21.15.144:43926) 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/3745134183 learned_addr learned my addr 172.21.15.144:0/3745134183 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): get_auth_request con 0x7fe2a809c620 auth_method 0 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ad1e2640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809c250 0x7fe2a8141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient(hunting): handle_auth_done global_id 6010 payload 824 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809c250 msgr2=0x7fe2a8141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809c250 0x7fe2a8141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a809dfa0 msgr2=0x7fe2a81320f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a809dfa0 0x7fe2a81320f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient: _finish_hunting 0 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 monclient: found mon.noname-b 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/3745134183 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c009450 con 0x7fe2a809c620 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ad1e2640 10 monclient: get_auth_request con 0x7fe2a809c250 auth_method 0 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.682+0000 7fe2ad1e2640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809c250 0x7fe2a8141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2ac9e1640 1 -- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 msgr2=0x7fe2a8131bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2ac9e1640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 0x7fe2a8131bb0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009210 tx=0x7fe29c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c03b9d0 con 0x7fe2a809c620 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: dump: 2024-09-18T12:32:06.695 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: _finish_auth 0 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29c03bb30 con 0x7fe2a809c620 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2af446640 5 monclient: authenticate success, global_id 6010 2024-09-18T12:32:06.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c044410 con 0x7fe2a809c620 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2af446640 10 monclient: _renew_subs 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2af446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a8141a60 con 0x7fe2a809c620 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a57fa640 10 monclient: dump: 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:06.697 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2a77fe640 4 set_mon_vals no callback set 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2af446640 10 monclient: _renew_subs 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2af446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.685+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a8132630 con 0x7fe2a809c620 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe29c039040 con 0x7fe2a809c620 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2af446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2af446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2af446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a80008d0 con 0x7fe2a809c620 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2a57fa640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe294038440 0x7fe29403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:06.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(167..167 src has 13..167) v4 ==== 9849+0+0 (secure 0 0 0) 0x7fe29c06c860 con 0x7fe2a809c620 2024-09-18T12:32:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2ad1e2640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe294038440 0x7fe29403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.686+0000 7fe2ad1e2640 10 monclient: get_auth_request con 0x7fe294038440 auth_method 0 2024-09-18T12:32:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.688+0000 7fe2ad1e2640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe294038440 0x7fe29403a8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe298003c70 tx=0x7fe2980074d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.690+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe29c042320 con 0x7fe2a809c620 2024-09-18T12:32:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.690+0000 7fe2a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:06.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.690+0000 7fe2a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:06.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.852+0000 7fe2af446640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:06.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.852+0000 7fe2af446640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:06.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.852+0000 7fe2af446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:06.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.852+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fe2a8141a60 con 0x7fe2a809c620 2024-09-18T12:32:06.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.853+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v167) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fe2a8141a60 con 0x7fe2a809c620 2024-09-18T12:32:06.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.853+0000 7fe2a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:06.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.853+0000 7fe2a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:06.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.953+0000 7fe2af446640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:06.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.953+0000 7fe2af446640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:06.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.953+0000 7fe2af446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:06.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.953+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fe2a8141a60 con 0x7fe2a809c620 2024-09-18T12:32:06.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.953+0000 7fe2a57fa640 1 -- 172.21.15.144:0/3745134183 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v167) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fe2a8141a60 con 0x7fe2a809c620 2024-09-18T12:32:06.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.953+0000 7fe2a57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:06.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.953+0000 7fe2a57fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.955+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe294038440 msgr2=0x7fe29403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.955+0000 7fe2af446640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe294038440 0x7fe29403a8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe298003c70 tx=0x7fe2980074d0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.955+0000 7fe2af446640 10 monclient: shutdown 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.955+0000 7fe2af446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.955+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 msgr2=0x7fe2a8131bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.955+0000 7fe2af446640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 0x7fe2a8131bb0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009210 tx=0x7fe29c032000 comp rx=0 tx=0).stop 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 shutdown_connections 2024-09-18T12:32:06.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe294038440 0x7fe29403a8f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a809dfa0 0x7fe2a81320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2a809c620 0x7fe2a8131bb0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 --2- 172.21.15.144:0/3745134183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe2a809c250 0x7fe2a8141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:06.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 >> 172.21.15.144:0/3745134183 conn(0x7fe2a8071340 msgr2=0x7fe2a80a1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:06.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 shutdown_connections 2024-09-18T12:32:06.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:06.956+0000 7fe2af446640 1 -- 172.21.15.144:0/3745134183 wait complete. 2024-09-18T12:32:06.965 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:32:06.965 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:32:06.966 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:32:06.966 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:32:06.966 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep globster 2024-09-18T12:32:06.966 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '11\.' 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.038+0000 7fcc7fd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.038+0000 7fcc7fd9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.038+0000 7fcc7fd9d640 10 monclient: monmap: 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:07.038992+0000 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:07.038992+0000 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:07.039 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.040 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:07.040 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:07.040 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:07.040 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:07.040 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:07.040 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.038+0000 7fcc7fd9d640 1 Processor -- start 2024-09-18T12:32:07.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.038+0000 7fcc7fd9d640 1 -- start start 2024-09-18T12:32:07.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.038+0000 7fcc7fd9d640 10 monclient: init 2024-09-18T12:32:07.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:07.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T12:32:07.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc780a1a70 0x7fcc7809c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc780a1a70 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcc7809ac40 0x7fcc7809b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7809ac40 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc7812f840 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780a1e40 con 0x7fcc7812f840 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc780a2230 con 0x7fcc7809ac40 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7809d3a0 con 0x7fcc780a1a70 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7fd9d640 10 monclient(hunting): authenticate will time out at 1162.608340s 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc780a1a70 0x7fcc7809c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7d338640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fcc7809ac40 msgr2=0x7fcc7809b0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7e33a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fcc7812f840 0x7fcc7812fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:47794/0 (socket says 172.21.15.144:47794) 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc780a1a70 0x7fcc7809c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43942/0 (socket says 172.21.15.144:43942) 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7d338640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcc7809ac40 0x7fcc7809b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7e33a640 1 -- 172.21.15.144:0/1089197526 learned_addr learned my addr 172.21.15.144:0/1089197526 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): get_auth_request con 0x7fcc780a1a70 auth_method 0 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient(hunting): handle_auth_done global_id 6013 payload 274 2024-09-18T12:32:07.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 1 -- 172.21.15.144:0/1089197526 >> v2:172.21.15.116:3300/0 conn(0x7fcc7809ac40 msgr2=0x7fcc7809b0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:07.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 1 --2- 172.21.15.144:0/1089197526 >> v2:172.21.15.116:3300/0 conn(0x7fcc7809ac40 0x7fcc7809b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 1 -- 172.21.15.144:0/1089197526 >> v1:172.21.15.144:6789/0 conn(0x7fcc7812f840 legacy=0x7fcc7812fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:07.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient: _finish_hunting 0 2024-09-18T12:32:07.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 1 monclient: found mon.noname-c 2024-09-18T12:32:07.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:07.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.039+0000 7fcc7db39640 1 -- 172.21.15.144:0/1089197526 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7809d680 con 0x7fcc780a1a70 2024-09-18T12:32:07.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7db39640 1 -- 172.21.15.144:0/1089197526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc780a1a70 msgr2=0x7fcc7809c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:07.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7db39640 1 --2- 172.21.15.144:0/1089197526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc780a1a70 0x7fcc7809c2f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcc780a12c0 tx=0x7fcc74009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a76a891168f24c7a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 1 -- 172.21.15.144:0/1089197526 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74007840 con 0x7fcc780a1a70 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: dump: 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.051 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: _finish_auth 0 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7fd9d640 5 monclient: authenticate success, global_id 6013 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 1 -- 172.21.15.144:0/1089197526 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7402d050 con 0x7fcc780a1a70 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7fd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7fd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7fd9d640 4 set_mon_vals no callback set 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 1 -- 172.21.15.144:0/1089197526 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74007c10 con 0x7fcc780a1a70 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.042+0000 7fcc7cb37640 10 monclient: dump: 2024-09-18T12:32:07.052 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 10 monclient: shutdown 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/1089197526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc780a1a70 msgr2=0x7fcc7809c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/1089197526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc780a1a70 0x7fcc7809c2f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcc780a12c0 tx=0x7fcc74009780 comp rx=0 tx=0).stop 2024-09-18T12:32:07.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/1089197526 shutdown_connections 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/1089197526 >> v2:172.21.15.116:3300/0 conn(0x7fcc7809ac40 0x7fcc7809b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/1089197526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc780a1a70 0x7fcc7809c2f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/1089197526 >> 172.21.15.144:0/1089197526 conn(0x7fcc78067e90 msgr2=0x7fcc780a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/1089197526 shutdown_connections 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/1089197526 wait complete. 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.043+0000 7fcc7fd9d640 10 monclient: monmap: 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.054 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 Processor -- start 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 -- start start 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient: init 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7809ac40 0x7fcc78094080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7809ac40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc780a1a70 0x7fcc780945c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc780a1a70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 0x7fcc78094b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcc7812f840 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc78128db0 con 0x7fcc7809ac40 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7806f000 con 0x7fcc7812f840 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc78128ad0 con 0x7fcc780a1a70 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7fd9d640 10 monclient(hunting): authenticate will time out at 1162.613557s 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 0x7fcc78094b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 0x7fcc78094b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43948/0 (socket says 172.21.15.144:43948) 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 -- 172.21.15.144:0/2219706000 learned_addr learned my addr 172.21.15.144:0/2219706000 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:07.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): get_auth_request con 0x7fcc7812f840 auth_method 0 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7db39640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7809ac40 msgr2=0x7fcc78094080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7db39640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7809ac40 0x7fcc78094080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient(hunting): handle_auth_done global_id 6016 payload 824 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 -- 172.21.15.144:0/2219706000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc780a1a70 msgr2=0x7fcc780945c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc780a1a70 0x7fcc780945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 -- 172.21.15.144:0/2219706000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7809ac40 msgr2=0x7fcc78094080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:07.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7809ac40 0x7fcc78094080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 monclient: found mon.noname-b 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.044+0000 7fcc7e33a640 1 -- 172.21.15.144:0/2219706000 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc74009a20 con 0x7fcc7812f840 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7e33a640 1 -- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 msgr2=0x7fcc78094b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7e33a640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 0x7fcc78094b00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c510 tx=0x7fcc70004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70005cb0 con 0x7fcc7812f840 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: dump: 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.058 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc70005e30 con 0x7fcc7812f840 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7fd9d640 5 monclient: authenticate success, global_id 6016 2024-09-18T12:32:07.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7fd9d640 10 monclient: _renew_subs 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7fd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc78129090 con 0x7fcc7812f840 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70007320 con 0x7fcc7812f840 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6dffb640 10 monclient: dump: 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.060 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7fd9d640 10 monclient: _renew_subs 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7fd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc6ffff640 4 set_mon_vals no callback set 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.047+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7806e910 con 0x7fcc7812f840 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fcc7001b3b0 con 0x7fcc7812f840 2024-09-18T12:32:07.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc7fd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc7fd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc7fd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc780008d0 con 0x7fcc7812f840 2024-09-18T12:32:07.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc6dffb640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc5c038440 0x7fcc5c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(167..167 src has 13..167) v4 ==== 9849+0+0 (secure 0 0 0) 0x7fcc7004c150 con 0x7fcc7812f840 2024-09-18T12:32:07.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc7db39640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc5c038440 0x7fcc5c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.048+0000 7fcc7db39640 10 monclient: get_auth_request con 0x7fcc5c038440 auth_method 0 2024-09-18T12:32:07.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.050+0000 7fcc7db39640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc5c038440 0x7fcc5c03a8f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcc74034000 tx=0x7fcc74037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.052+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fcc70007750 con 0x7fcc7812f840 2024-09-18T12:32:07.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.052+0000 7fcc6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.052+0000 7fcc6dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:07.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.214+0000 7fcc7fd9d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.214+0000 7fcc7fd9d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.214+0000 7fcc7fd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.214+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcc7809d960 con 0x7fcc7812f840 2024-09-18T12:32:07.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.215+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v167) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcc70017070 con 0x7fcc7812f840 2024-09-18T12:32:07.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.215+0000 7fcc6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.215+0000 7fcc6dffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:07.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.312+0000 7fcc7fd9d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.312+0000 7fcc7fd9d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.312+0000 7fcc7fd9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.312+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcc78072640 con 0x7fcc7812f840 2024-09-18T12:32:07.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.312+0000 7fcc6dffb640 1 -- 172.21.15.144:0/2219706000 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v167) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fcc7000e290 con 0x7fcc7812f840 2024-09-18T12:32:07.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.312+0000 7fcc6dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.312+0000 7fcc6dffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc5c038440 msgr2=0x7fcc5c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc5c038440 0x7fcc5c03a8f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcc74034000 tx=0x7fcc74037000 comp rx=0 tx=0).stop 2024-09-18T12:32:07.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 10 monclient: shutdown 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 msgr2=0x7fcc78094b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 0x7fcc78094b00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c510 tx=0x7fcc70004900 comp rx=0 tx=0).stop 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 shutdown_connections 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc5c038440 0x7fcc5c03a8f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc7812f840 0x7fcc78094b00 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc780a1a70 0x7fcc780945c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 --2- 172.21.15.144:0/2219706000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc7809ac40 0x7fcc78094080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 >> 172.21.15.144:0/2219706000 conn(0x7fcc78067e90 msgr2=0x7fcc78131c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 shutdown_connections 2024-09-18T12:32:07.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.314+0000 7fcc7fd9d640 1 -- 172.21.15.144:0/2219706000 wait complete. 2024-09-18T12:32:07.324 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:32:07.324 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:32:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:32:07.324 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:32:07.325 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep globster 2024-09-18T12:32:07.325 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '12\.' 2024-09-18T12:32:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.396+0000 7f8b412dd640 10 monclient: get_monmap_and_config 2024-09-18T12:32:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.397+0000 7f8b412dd640 10 monclient: build_initial_monmap 2024-09-18T12:32:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.397+0000 7f8b412dd640 10 monclient: monmap: 2024-09-18T12:32:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:07.406 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:07.397927+0000 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:07.397927+0000 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.397+0000 7f8b412dd640 1 Processor -- start 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.397+0000 7f8b412dd640 1 -- start start 2024-09-18T12:32:07.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.397+0000 7f8b412dd640 10 monclient: init 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09ead0 0x7f8b3c09d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): picked mon.noname-b con 0x7f8b3c09ead0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b3c09eea0 0x7f8b3c09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): picked mon.noname-a con 0x7f8b3c09eea0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8b3c068e00 0x7f8b3c069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): picked mon.noname-c con 0x7f8b3c068e00 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c09fef0 con 0x7f8b3c09eea0 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0a0330 con 0x7f8b3c068e00 2024-09-18T12:32:07.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0a1a60 con 0x7f8b3c09ead0 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b412dd640 10 monclient(hunting): authenticate will time out at 1162.967358s 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b33fff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f8b3c09eea0 msgr2=0x7f8b3c09f2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b33fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b3c09eea0 0x7f8b3c09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8b3c068e00 0x7f8b3c069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8b3c068e00 0x7f8b3c069200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43672/0 (socket says 172.21.15.144:43672) 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 -- 172.21.15.144:0/3403956193 learned_addr learned my addr 172.21.15.144:0/3403956193 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): get_auth_request con 0x7f8b3c068e00 auth_method 0 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 1 --2- 172.21.15.144:0/3403956193 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09ead0 0x7f8b3c09d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 10 monclient(hunting): get_auth_request con 0x7f8b3c09ead0 auth_method 0 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient(hunting): handle_auth_done global_id 6019 payload 274 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 -- 172.21.15.144:0/3403956193 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09ead0 msgr2=0x7f8b3c09d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 --2- 172.21.15.144:0/3403956193 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09ead0 0x7f8b3c09d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 -- 172.21.15.144:0/3403956193 >> v2:172.21.15.116:3300/0 conn(0x7f8b3c09eea0 msgr2=0x7f8b3c09f2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 --2- 172.21.15.144:0/3403956193 >> v2:172.21.15.116:3300/0 conn(0x7f8b3c09eea0 0x7f8b3c09f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.398+0000 7f8b3b7fe640 1 -- 172.21.15.144:0/3403956193 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c09d730 con 0x7f8b3c068e00 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3b7fe640 1 -- 172.21.15.144:0/3403956193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 msgr2=0x7f8b3c069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3b7fe640 1 --2- 172.21.15.144:0/3403956193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c069200 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c09fcd0 tx=0x7f8b34009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d86c278f67854c77 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 1 -- 172.21.15.144:0/3403956193 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34007610 con 0x7f8b3c068e00 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: dump: 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.412 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 1 -- 172.21.15.144:0/3403956193 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3402d050 con 0x7f8b3c068e00 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b412dd640 5 monclient: authenticate success, global_id 6019 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b412dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b412dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 1 -- 172.21.15.144:0/3403956193 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b340079a0 con 0x7f8b3c068e00 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b412dd640 4 set_mon_vals no callback set 2024-09-18T12:32:07.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.401+0000 7f8b3a7fc640 10 monclient: dump: 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 10 monclient: shutdown 2024-09-18T12:32:07.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 -- 172.21.15.144:0/3403956193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 msgr2=0x7f8b3c069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 --2- 172.21.15.144:0/3403956193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c069200 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c09fcd0 tx=0x7f8b34009840 comp rx=0 tx=0).stop 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 -- 172.21.15.144:0/3403956193 shutdown_connections 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 --2- 172.21.15.144:0/3403956193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c069200 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 --2- 172.21.15.144:0/3403956193 >> v2:172.21.15.116:3300/0 conn(0x7f8b3c09eea0 0x7f8b3c09f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 --2- 172.21.15.144:0/3403956193 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09ead0 0x7f8b3c09d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 -- 172.21.15.144:0/3403956193 >> 172.21.15.144:0/3403956193 conn(0x7f8b3c0a5230 msgr2=0x7f8b3c0a5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 -- 172.21.15.144:0/3403956193 shutdown_connections 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 1 -- 172.21.15.144:0/3403956193 wait complete. 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 10 monclient: build_initial_monmap 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.402+0000 7f8b412dd640 10 monclient: monmap: 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:07.415 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 Processor -- start 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 -- start start 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient: init 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): picked mon.noname-b con 0x7f8b3c068e00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b3c09ead0 0x7f8b3c11e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): picked mon.noname-a con 0x7f8b3c09ead0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09eea0 0x7f8b3c11ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): picked mon.noname-c con 0x7f8b3c09eea0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0a5a60 con 0x7f8b3c09ead0 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c121e60 con 0x7f8b3c068e00 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c121b80 con 0x7f8b3c09eea0 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b412dd640 10 monclient(hunting): authenticate will time out at 1162.972741s 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b33fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b3c09ead0 msgr2=0x7f8b3c11e530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:07.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b3c09ead0 0x7f8b3c11e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c134a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43676/0 (socket says 172.21.15.144:43676) 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 1 -- 172.21.15.144:0/2992498041 learned_addr learned my addr 172.21.15.144:0/2992498041 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 10 monclient(hunting): get_auth_request con 0x7f8b3c068e00 auth_method 0 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3b7fe640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09eea0 0x7f8b3c11ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3b7fe640 10 monclient(hunting): get_auth_request con 0x7f8b3c09eea0 auth_method 0 2024-09-18T12:32:07.418 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.403+0000 7f8b3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 10 monclient(hunting): handle_auth_done global_id 6022 payload 824 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 1 -- 172.21.15.144:0/2992498041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09eea0 msgr2=0x7f8b3c11ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09eea0 0x7f8b3c11ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 1 -- 172.21.15.144:0/2992498041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b3c09ead0 msgr2=0x7f8b3c11e530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b3c09ead0 0x7f8b3c11e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:07.419 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 1 monclient: found mon.noname-b 2024-09-18T12:32:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.420 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.404+0000 7f8b3affd640 1 -- 172.21.15.144:0/2992498041 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b34009ae0 con 0x7f8b3c068e00 2024-09-18T12:32:07.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b3affd640 1 -- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 msgr2=0x7f8b3c134a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b3affd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c134a60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00ea50 tx=0x7f8b2c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c009600 con 0x7f8b3c068e00 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: dump: 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.422 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c00cc20 con 0x7f8b3c068e00 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b412dd640 5 monclient: authenticate success, global_id 6022 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b412dd640 10 monclient: _renew_subs 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b412dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c120c90 con 0x7f8b3c068e00 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c013470 con 0x7f8b3c068e00 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:07.423 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b32ffd640 10 monclient: dump: 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b412dd640 10 monclient: _renew_subs 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b412dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c1200c0 con 0x7f8b3c068e00 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.406+0000 7f8b39ffb640 4 set_mon_vals no callback set 2024-09-18T12:32:07.424 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8b2c013860 con 0x7f8b3c068e00 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b412dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b32ffd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8b1c038440 0x7f8b1c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b412dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b412dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(167..167 src has 13..167) v4 ==== 9849+0+0 (secure 0 0 0) 0x7f8b2c04ced0 con 0x7f8b3c068e00 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c0008d0 con 0x7f8b3c068e00 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b33fff640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8b1c038440 0x7f8b1c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.407+0000 7f8b33fff640 10 monclient: get_auth_request con 0x7f8b1c038440 auth_method 0 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.410+0000 7f8b33fff640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8b1c038440 0x7f8b1c03a8f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b240038f0 tx=0x7f8b24006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.411+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8b2c023ad0 con 0x7f8b3c068e00 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.411+0000 7f8b32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.425 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.411+0000 7f8b32ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:07.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.572+0000 7f8b412dd640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.572+0000 7f8b412dd640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.572+0000 7f8b412dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.572+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f8b3c069710 con 0x7f8b3c068e00 2024-09-18T12:32:07.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.573+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v167) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f8b2c0232f0 con 0x7f8b3c068e00 2024-09-18T12:32:07.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.573+0000 7f8b32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.573+0000 7f8b32ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:07.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.670+0000 7f8b412dd640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.671+0000 7f8b412dd640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.671+0000 7f8b412dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.671+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f8b3c072640 con 0x7f8b3c068e00 2024-09-18T12:32:07.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.671+0000 7f8b32ffd640 1 -- 172.21.15.144:0/2992498041 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v167) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f8b2c0234d0 con 0x7f8b3c068e00 2024-09-18T12:32:07.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.671+0000 7f8b32ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:07.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.671+0000 7f8b32ffd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:07.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8b1c038440 msgr2=0x7f8b1c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8b1c038440 0x7f8b1c03a8f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b240038f0 tx=0x7f8b24006d80 comp rx=0 tx=0).stop 2024-09-18T12:32:07.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 10 monclient: shutdown 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 msgr2=0x7f8b3c134a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c134a60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00ea50 tx=0x7f8b2c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 shutdown_connections 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8b1c038440 0x7f8b1c03a8f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b3c09eea0 0x7f8b3c11ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b3c09ead0 0x7f8b3c11e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 --2- 172.21.15.144:0/2992498041 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8b3c068e00 0x7f8b3c134a60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 >> 172.21.15.144:0/2992498041 conn(0x7f8b3c0a5230 msgr2=0x7f8b3c0a3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 shutdown_connections 2024-09-18T12:32:07.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.673+0000 7f8b412dd640 1 -- 172.21.15.144:0/2992498041 wait complete. 2024-09-18T12:32:07.682 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:32:07.683 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T12:32:07.683 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.757+0000 7f72e6d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.757+0000 7f72e6d9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.757+0000 7f72e6d9d640 10 monclient: monmap: 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:07.758083+0000 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:07.758083+0000 2024-09-18T12:32:07.758 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:07.759 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.759 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:07.759 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:07.759 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:07.759 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:07.759 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:07.759 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.761 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.757+0000 7f72e6d9d640 1 Processor -- start 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.757+0000 7f72e6d9d640 1 -- start start 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.757+0000 7f72e6d9d640 10 monclient: init 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72e009ec50 0x7f72e0131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f72e009ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e0068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f72e01321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72e009f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e0067420 con 0x7f72e009f020 2024-09-18T12:32:07.762 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e009fdc0 con 0x7f72e009ec50 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00a0510 con 0x7f72e01321a0 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e6d9d640 10 monclient(hunting): authenticate will time out at 1163.327469s 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e533a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f72e009f020 0x7f72e0069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58638/0 (socket says 172.21.15.144:58638) 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e533a640 1 -- 172.21.15.144:0/1026155969 learned_addr learned my addr 172.21.15.144:0/1026155969 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e4b39640 1 -- 172.21.15.144:0/1026155969 >> v2:172.21.15.116:3300/0 conn(0x7f72e009ec50 msgr2=0x7f72e0131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72e4b39640 1 --2- 172.21.15.144:0/1026155969 >> v2:172.21.15.116:3300/0 conn(0x7f72e009ec50 0x7f72e0131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72df7fe640 1 -- 172.21.15.144:0/1026155969 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 1 --2- 172.21.15.144:0/1026155969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e0068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 10 monclient(hunting): get_auth_request con 0x7f72e01321a0 auth_method 0 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.763 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.758+0000 7f72dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 10 monclient(hunting): handle_auth_done global_id 64817 payload 274 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 1 -- 172.21.15.144:0/1026155969 >> v2:172.21.15.116:3300/0 conn(0x7f72e009ec50 msgr2=0x7f72e0131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 1 --2- 172.21.15.144:0/1026155969 >> v2:172.21.15.116:3300/0 conn(0x7f72e009ec50 0x7f72e0131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 1 -- 172.21.15.144:0/1026155969 >> v1:172.21.15.144:6789/0 conn(0x7f72e009f020 legacy=0x7f72e0069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 1 monclient: found mon.noname-b 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:07.764 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.759+0000 7f72dffff640 1 -- 172.21.15.144:0/1026155969 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00a0200 con 0x7f72e01321a0 2024-09-18T12:32:07.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72dffff640 1 -- 172.21.15.144:0/1026155969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 msgr2=0x7f72e0068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:07.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72dffff640 1 --2- 172.21.15.144:0/1026155969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e0068d00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009850 tx=0x7f72cc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac72b3000ac37c14 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 1 -- 172.21.15.144:0/1026155969 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc032440 con 0x7f72e01321a0 2024-09-18T12:32:07.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:07.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: dump: 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 1 -- 172.21.15.144:0/1026155969 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc0325a0 con 0x7f72e01321a0 2024-09-18T12:32:07.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 5 monclient: authenticate success, global_id 64817 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 1 -- 172.21.15.144:0/1026155969 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc032980 con 0x7f72e01321a0 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72df7fe640 10 monclient: dump: 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.772 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 4 set_mon_vals no callback set 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 10 monclient: shutdown 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 1 -- 172.21.15.144:0/1026155969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 msgr2=0x7f72e0068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/1026155969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e0068d00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009850 tx=0x7f72cc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 1 -- 172.21.15.144:0/1026155969 shutdown_connections 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.762+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/1026155969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e0068d00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/1026155969 >> v2:172.21.15.116:3300/0 conn(0x7f72e009ec50 0x7f72e0131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 1 -- 172.21.15.144:0/1026155969 >> 172.21.15.144:0/1026155969 conn(0x7f72e00a5d40 msgr2=0x7f72e00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 1 -- 172.21.15.144:0/1026155969 shutdown_connections 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 1 -- 172.21.15.144:0/1026155969 wait complete. 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 10 monclient: monmap: 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:07.774 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 1 Processor -- start 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 1 -- start start 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.763+0000 7f72e6d9d640 10 monclient: init 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e009ec50 0x7f72e009a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f72e009ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 0x7f72e0130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f72e009f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f72e01321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00a4140 con 0x7f72e009ec50 2024-09-18T12:32:07.775 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e011de50 con 0x7f72e009f020 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e011db70 con 0x7f72e01321a0 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e6d9d640 10 monclient(hunting): authenticate will time out at 1163.333443s 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 0x7f72e0130a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 0x7f72e0130a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43690/0 (socket says 172.21.15.144:43690) 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 -- 172.21.15.144:0/621405054 learned_addr learned my addr 172.21.15.144:0/621405054 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): get_auth_request con 0x7f72e009f020 auth_method 0 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e4b39640 1 -- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e009ec50 msgr2=0x7f72e009a9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e4b39640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e009ec50 0x7f72e009a9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 10 monclient(hunting): get_auth_request con 0x7f72e01321a0 auth_method 0 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72e533a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:07.777 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient(hunting): handle_auth_done global_id 6025 payload 824 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 -- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 msgr2=0x7f72e011b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e011b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 -- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e009ec50 msgr2=0x7f72e009a9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e009ec50 0x7f72e009a9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 monclient: found mon.noname-b 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.764+0000 7f72dffff640 1 -- 172.21.15.144:0/621405054 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc009390 con 0x7f72e009f020 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dffff640 1 -- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 msgr2=0x7f72e0130a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dffff640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 0x7f72e0130a50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009850 tx=0x7f72cc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0319c0 con 0x7f72e009f020 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.778 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: dump: 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:07.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc031b20 con 0x7f72e009f020 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72e6d9d640 5 monclient: authenticate success, global_id 6025 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72e6d9d640 10 monclient: _renew_subs 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72e6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e009b4e0 con 0x7f72e009f020 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc031ea0 con 0x7f72e009f020 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72dcff9640 10 monclient: dump: 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.780 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72deffd640 4 set_mon_vals no callback set 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72e6d9d640 10 monclient: _renew_subs 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72e6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.767+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e0141580 con 0x7f72e009f020 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f72cc03d630 con 0x7f72e009f020 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72e6d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72e6d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72e6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e00008d0 con 0x7f72e009f020 2024-09-18T12:32:07.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72dcff9640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f72c4038440 0x7f72c403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:07.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(167..167 src has 13..167) v4 ==== 9849+0+0 (secure 0 0 0) 0x7f72cc06d740 con 0x7f72e009f020 2024-09-18T12:32:07.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72e4b39640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f72c4038440 0x7f72c403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:07.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.768+0000 7f72e4b39640 10 monclient: get_auth_request con 0x7f72c4038440 auth_method 0 2024-09-18T12:32:07.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.771+0000 7f72e4b39640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f72c4038440 0x7f72c403a8f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f72d40038f0 tx=0x7f72d4006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:07.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.772+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f72cc03baf0 con 0x7f72e009f020 2024-09-18T12:32:07.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.772+0000 7f72dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:07.783 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.772+0000 7f72dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:07.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.937+0000 7f72e6d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T12:32:07.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.937+0000 7f72e6d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T12:32:07.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.937+0000 7f72e6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:07.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:07.937+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f72e00a13e0 con 0x7f72e009f020 2024-09-18T12:32:08.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:08.596+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v168) v1 ==== 118+0+0 (secure 0 0 0) 0x7f72cc037030 con 0x7f72e009f020 2024-09-18T12:32:08.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:08.596+0000 7f72dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T12:32:08.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:08.596+0000 7f72dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:08.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:08.703+0000 7f72e6d9d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T12:32:08.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:08.703+0000 7f72e6d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T12:32:08.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:08.703+0000 7f72e6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:08.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:08.703+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f72e00a13e0 con 0x7f72e009f020 2024-09-18T12:32:09.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.601+0000 7f72dcff9640 1 -- 172.21.15.144:0/621405054 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v169) v1 ==== 118+0+0 (secure 0 0 0) 0x7f72cc037020 con 0x7f72e009f020 2024-09-18T12:32:09.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.601+0000 7f72dcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T12:32:09.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.601+0000 7f72dcff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:09.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f72c4038440 msgr2=0x7f72c403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:09.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f72c4038440 0x7f72c403a8f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f72d40038f0 tx=0x7f72d4006d80 comp rx=0 tx=0).stop 2024-09-18T12:32:09.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 10 monclient: shutdown 2024-09-18T12:32:09.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 msgr2=0x7f72e0130a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 0x7f72e0130a50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009850 tx=0x7f72cc032000 comp rx=0 tx=0).stop 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 shutdown_connections 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f72c4038440 0x7f72c403a8f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72e01321a0 0x7f72e011b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72e009f020 0x7f72e0130a50 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 --2- 172.21.15.144:0/621405054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e009ec50 0x7f72e009a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 >> 172.21.15.144:0/621405054 conn(0x7f72e00a5d40 msgr2=0x7f72e00a4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 shutdown_connections 2024-09-18T12:32:09.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.604+0000 7f72e6d9d640 1 -- 172.21.15.144:0/621405054 wait complete. 2024-09-18T12:32:09.618 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-18T12:32:09.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.697+0000 7f9c68a2b640 10 monclient: get_monmap_and_config 2024-09-18T12:32:09.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.697+0000 7f9c68a2b640 10 monclient: build_initial_monmap 2024-09-18T12:32:09.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.697+0000 7f9c68a2b640 10 monclient: monmap: 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:09.698867+0000 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:09.698867+0000 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:09.700 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:09.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.698+0000 7f9c68a2b640 1 Processor -- start 2024-09-18T12:32:09.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.698+0000 7f9c68a2b640 1 -- start start 2024-09-18T12:32:09.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.698+0000 7f9c68a2b640 10 monclient: init 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9c6409f180 0x7f9c64130040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f9c6409f180 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c6409cee0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c64134c60 0x7f9c64137040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f9c64134c60 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:09.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6409e610 con 0x7f9c6409cee0 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6409f550 con 0x7f9c64134c60 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6409e2a0 con 0x7f9c6409f180 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c68a2b640 10 monclient(hunting): authenticate will time out at 1165.268231s 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c59d9b640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f9c6409cee0 legacy=0x7f9c6409bb40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9c6409f180 0x7f9c64130040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c62d9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f9c64134c60 msgr2=0x7f9c64137040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c62d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c64134c60 0x7f9c64137040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9c6409f180 0x7f9c64130040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43706/0 (socket says 172.21.15.144:43706) 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 -- 172.21.15.144:0/237086249 learned_addr learned my addr 172.21.15.144:0/237086249 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:09.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): get_auth_request con 0x7f9c6409f180 auth_method 0 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient(hunting): handle_auth_done global_id 6028 payload 274 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 -- 172.21.15.144:0/237086249 >> v2:172.21.15.116:3300/0 conn(0x7f9c64134c60 msgr2=0x7f9c64137040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 --2- 172.21.15.144:0/237086249 >> v2:172.21.15.116:3300/0 conn(0x7f9c64134c60 0x7f9c64137040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 -- 172.21.15.144:0/237086249 >> v1:172.21.15.116:6789/0 conn(0x7f9c6409cee0 legacy=0x7f9c6409bb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 monclient: found mon.noname-c 2024-09-18T12:32:09.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.699+0000 7f9c6259c640 1 -- 172.21.15.144:0/237086249 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6409c250 con 0x7f9c6409f180 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c6259c640 1 -- 172.21.15.144:0/237086249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409f180 msgr2=0x7f9c64130040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c6259c640 1 --2- 172.21.15.144:0/237086249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409f180 0x7f9c64130040 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9c6409f9d0 tx=0x7f9c540091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3483ee26b101858 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 1 -- 172.21.15.144:0/237086249 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54032440 con 0x7f9c6409f180 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: dump: 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.712 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: _finish_auth 0 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 1 -- 172.21.15.144:0/237086249 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c540325a0 con 0x7f9c6409f180 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 1 -- 172.21.15.144:0/237086249 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54032940 con 0x7f9c6409f180 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c68a2b640 5 monclient: authenticate success, global_id 6028 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c68a2b640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:09.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c68a2b640 4 set_mon_vals no callback set 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c61d9b640 10 monclient: dump: 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:09.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c68a2b640 10 monclient: shutdown 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c68a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c68a2b640 1 -- 172.21.15.144:0/237086249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409f180 msgr2=0x7f9c64130040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.702+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/237086249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409f180 0x7f9c64130040 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9c6409f9d0 tx=0x7f9c540091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 1 -- 172.21.15.144:0/237086249 shutdown_connections 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/237086249 >> v2:172.21.15.116:3300/0 conn(0x7f9c64134c60 0x7f9c64137040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/237086249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409f180 0x7f9c64130040 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 1 -- 172.21.15.144:0/237086249 >> 172.21.15.144:0/237086249 conn(0x7f9c640a2b70 msgr2=0x7f9c6409ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 1 -- 172.21.15.144:0/237086249 shutdown_connections 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 1 -- 172.21.15.144:0/237086249 wait complete. 2024-09-18T12:32:09.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 10 monclient: build_initial_monmap 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.703+0000 7f9c68a2b640 10 monclient: monmap: 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 Processor -- start 2024-09-18T12:32:09.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 -- start start 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient: init 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 0x7f9c6406b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): picked mon.noname-b con 0x7f9c6409cee0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c6409f180 0x7f9c6406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): picked mon.noname-a con 0x7f9c6409f180 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c64134c60 0x7f9c6406c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): picked mon.noname-c con 0x7f9c64134c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c641187c0 con 0x7f9c6409f180 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:09.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6406dac0 con 0x7f9c6409cee0 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c640a1560 con 0x7f9c64134c60 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c68a2b640 10 monclient(hunting): authenticate will time out at 1165.273601s 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 0x7f9c6406b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 0x7f9c6406b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43712/0 (socket says 172.21.15.144:43712) 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 -- 172.21.15.144:0/2163812377 learned_addr learned my addr 172.21.15.144:0/2163812377 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): get_auth_request con 0x7f9c6409cee0 auth_method 0 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:09.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c59d9b640 1 -- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c6409f180 msgr2=0x7f9c6406bd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c59d9b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c6409f180 0x7f9c6406bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient(hunting): handle_auth_done global_id 6031 payload 824 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c62d9d640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c64134c60 0x7f9c6406c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 -- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c64134c60 msgr2=0x7f9c6406c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c64134c60 0x7f9c6406c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 -- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c6409f180 msgr2=0x7f9c6406bd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c6409f180 0x7f9c6406bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 monclient: found mon.noname-b 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:09.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c6259c640 1 -- 172.21.15.144:0/2163812377 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54009450 con 0x7f9c6409cee0 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c62d9d640 10 monclient: get_auth_request con 0x7f9c64134c60 auth_method 0 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.704+0000 7f9c62d9d640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c64134c60 0x7f9c6406c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c6259c640 1 -- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 msgr2=0x7f9c6406b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c6259c640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 0x7f9c6406b810 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9c54035040 tx=0x7f9c54032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5403ca40 con 0x7f9c6409cee0 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: dump: 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:09.720 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: _finish_auth 0 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5403cba0 con 0x7f9c6409cee0 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c68a2b640 5 monclient: authenticate success, global_id 6031 2024-09-18T12:32:09.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c68a2b640 10 monclient: _renew_subs 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c68a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c64119a50 con 0x7f9c6409cee0 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c540313a0 con 0x7f9c6409cee0 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c5affd640 10 monclient: dump: 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:09.722 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:09.723 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:09.723 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:09.723 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:09.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c6159a640 4 set_mon_vals no callback set 2024-09-18T12:32:09.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c68a2b640 10 monclient: _renew_subs 2024-09-18T12:32:09.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c68a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:09.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.707+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6406dc90 con 0x7f9c6409cee0 2024-09-18T12:32:09.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c68a2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:09.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c68a2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c68a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9c54039040 con 0x7f9c6409cee0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c640008d0 con 0x7f9c6409cee0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c5affd640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c44038440 0x7f9c4403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(169..169 src has 13..169) v4 ==== 9787+0+0 (secure 0 0 0) 0x7f9c5406cd40 con 0x7f9c6409cee0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c59d9b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c44038440 0x7f9c4403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.708+0000 7f9c59d9b640 10 monclient: get_auth_request con 0x7f9c44038440 auth_method 0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.711+0000 7f9c59d9b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c44038440 0x7f9c4403a8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9c6406f080 tx=0x7f9c4c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.712+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9c54043b40 con 0x7f9c6409cee0 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.712+0000 7f9c5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:09.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.712+0000 7f9c5affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:09.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.873+0000 7f9c68a2b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T12:32:09.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.873+0000 7f9c68a2b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T12:32:09.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.873+0000 7f9c68a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:09.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:09.873+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f9c64130580 con 0x7f9c6409cee0 2024-09-18T12:32:10.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:10.610+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v170) v1 ==== 127+0+0 (secure 0 0 0) 0x7f9c54040020 con 0x7f9c6409cee0 2024-09-18T12:32:10.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:10.610+0000 7f9c5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T12:32:10.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:10.610+0000 7f9c5affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:10.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:10.712+0000 7f9c68a2b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T12:32:10.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:10.712+0000 7f9c68a2b640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T12:32:10.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:10.712+0000 7f9c68a2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:10.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:10.713+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f9c64130580 con 0x7f9c6409cee0 2024-09-18T12:32:11.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.592+0000 7f9c5affd640 1 -- 172.21.15.144:0/2163812377 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v171) v1 ==== 127+0+0 (secure 0 0 0) 0x7f9c54037070 con 0x7f9c6409cee0 2024-09-18T12:32:11.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.592+0000 7f9c5affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T12:32:11.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.592+0000 7f9c5affd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:11.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c44038440 msgr2=0x7f9c4403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c44038440 0x7f9c4403a8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9c6406f080 tx=0x7f9c4c006d80 comp rx=0 tx=0).stop 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 10 monclient: shutdown 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 msgr2=0x7f9c6406b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 0x7f9c6406b810 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9c54035040 tx=0x7f9c54032000 comp rx=0 tx=0).stop 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 shutdown_connections 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c44038440 0x7f9c4403a8f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c64134c60 0x7f9c6406c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c6409f180 0x7f9c6406bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 --2- 172.21.15.144:0/2163812377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c6409cee0 0x7f9c6406b810 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 >> 172.21.15.144:0/2163812377 conn(0x7f9c640a2b70 msgr2=0x7f9c64135d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 shutdown_connections 2024-09-18T12:32:11.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.595+0000 7f9c68a2b640 1 -- 172.21.15.144:0/2163812377 wait complete. 2024-09-18T12:32:11.607 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-18T12:32:11.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.691+0000 7f7b7c39d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:11.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.691+0000 7f7b7c39d640 10 monclient: build_initial_monmap 2024-09-18T12:32:11.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.691+0000 7f7b7c39d640 10 monclient: monmap: 2024-09-18T12:32:11.693 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:11.693 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:11.693 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:11.693087+0000 2024-09-18T12:32:11.693 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:11.693087+0000 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:11.694 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:11.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.691+0000 7f7b7c39d640 1 Processor -- start 2024-09-18T12:32:11.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.691+0000 7f7b7c39d640 1 -- start start 2024-09-18T12:32:11.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.691+0000 7f7b7c39d640 10 monclient: init 2024-09-18T12:32:11.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b7409ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b7409f020 0x7f7b74069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7409f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b74136d60 0x7f7b74139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b74136d60 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b74067420 con 0x7f7b7409ec50 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7409fdc0 con 0x7f7b7409f020 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b740a0510 con 0x7f7b74136d60 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:11.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7c39d640 10 monclient(hunting): authenticate will time out at 1167.262604s 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a139640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f7b7409ec50 legacy=0x7f7b74131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b79938640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f7b7409f020 msgr2=0x7f7b74069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b79938640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b7409f020 0x7f7b74069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b74136d60 0x7f7b74139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b74136d60 0x7f7b74139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43724/0 (socket says 172.21.15.144:43724) 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 1 -- 172.21.15.144:0/2417706956 learned_addr learned my addr 172.21.15.144:0/2417706956 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): get_auth_request con 0x7f7b74136d60 auth_method 0 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:11.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.692+0000 7f7b7a93a640 10 monclient(hunting): handle_auth_done global_id 6034 payload 274 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 1 -- 172.21.15.144:0/2417706956 >> v2:172.21.15.116:3300/0 conn(0x7f7b7409f020 msgr2=0x7f7b74069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 1 --2- 172.21.15.144:0/2417706956 >> v2:172.21.15.116:3300/0 conn(0x7f7b7409f020 0x7f7b74069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 1 -- 172.21.15.144:0/2417706956 >> v1:172.21.15.116:6789/0 conn(0x7f7b7409ec50 legacy=0x7f7b74131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 1 monclient: found mon.noname-c 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:11.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.693+0000 7f7b7a93a640 1 -- 172.21.15.144:0/2417706956 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b740a0200 con 0x7f7b74136d60 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b7a93a640 1 -- 172.21.15.144:0/2417706956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b74136d60 msgr2=0x7f7b74139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b7a93a640 1 --2- 172.21.15.144:0/2417706956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b74136d60 0x7f7b74139140 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7b74132a50 tx=0x7f7b6c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56dd1be6c132a579 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 1 -- 172.21.15.144:0/2417706956 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c007610 con 0x7f7b74136d60 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: dump: 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:11.705 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: _finish_auth 0 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 1 -- 172.21.15.144:0/2417706956 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c02d050 con 0x7f7b74136d60 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b7c39d640 5 monclient: authenticate success, global_id 6034 2024-09-18T12:32:11.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b7c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b7c39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b7c39d640 4 set_mon_vals no callback set 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 1 -- 172.21.15.144:0/2417706956 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0079a0 con 0x7f7b74136d60 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.695+0000 7f7b79137640 10 monclient: dump: 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:11.707 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 10 monclient: shutdown 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2417706956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b74136d60 msgr2=0x7f7b74139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2417706956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b74136d60 0x7f7b74139140 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7b74132a50 tx=0x7f7b6c009840 comp rx=0 tx=0).stop 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2417706956 shutdown_connections 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2417706956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b74136d60 0x7f7b74139140 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2417706956 >> v2:172.21.15.116:3300/0 conn(0x7f7b7409f020 0x7f7b74069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2417706956 >> 172.21.15.144:0/2417706956 conn(0x7f7b740a5d40 msgr2=0x7f7b740a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:11.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2417706956 shutdown_connections 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.696+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2417706956 wait complete. 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient: build_initial_monmap 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient: monmap: 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:11.709 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 Processor -- start 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 -- start start 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient: init 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b7409ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7409f020 0x7f7b74097110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b7409f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b74136d60 0x7f7b74097650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b74136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7409b890 con 0x7f7b74136d60 2024-09-18T12:32:11.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7409ace0 con 0x7f7b7409ec50 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7409c000 con 0x7f7b7409f020 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7c39d640 10 monclient(hunting): authenticate will time out at 1167.267874s 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7a139640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7a139640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43734/0 (socket says 172.21.15.144:43734) 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7a139640 1 -- 172.21.15.144:0/2469379594 learned_addr learned my addr 172.21.15.144:0/2469379594 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7a139640 10 monclient(hunting): get_auth_request con 0x7f7b7409ec50 auth_method 0 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7a139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7a139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.697+0000 7f7b7a139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:11.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a93a640 1 -- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b74136d60 msgr2=0x7f7b74097650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a93a640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b74136d60 0x7f7b74097650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 10 monclient(hunting): handle_auth_done global_id 6037 payload 824 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 1 -- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7409f020 msgr2=0x7f7b74097110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7409f020 0x7f7b74097110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 1 -- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b74136d60 msgr2=0x7f7b74097650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b74136d60 0x7f7b74097650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 10 monclient: _finish_hunting 0 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 1 monclient: found mon.noname-b 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:11.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.698+0000 7f7b7a139640 1 -- 172.21.15.144:0/2469379594 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c009ae0 con 0x7f7b7409ec50 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7a139640 1 -- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 msgr2=0x7f7b7409b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7a139640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b630 tx=0x7f7b700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70015070 con 0x7f7b7409ec50 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: dump: 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:11.713 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b70003d90 con 0x7f7b7409ec50 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7c39d640 5 monclient: authenticate success, global_id 6037 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7c39d640 10 monclient: _renew_subs 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b740a38f0 con 0x7f7b7409ec50 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:11.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70006380 con 0x7f7b7409ec50 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b6a7fc640 10 monclient: dump: 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:11.715 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7c39d640 10 monclient: _renew_subs 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b74098b60 con 0x7f7b7409ec50 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.700+0000 7f7b78936640 4 set_mon_vals no callback set 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b7a139640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b630 tx=0x7f7b700091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f7b70010440 mgrmap(e 5) v1 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7b70010440 con 0x7f7b7409ec50 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b7c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b7c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b7c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b740990d0 con 0x7f7b7409ec50 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b6a7fc640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b58038440 0x7f7b5803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b79938640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b58038440 0x7f7b5803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:11.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.701+0000 7f7b79938640 10 monclient: get_auth_request con 0x7f7b58038440 auth_method 0 2024-09-18T12:32:11.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.704+0000 7f7b79938640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b58038440 0x7f7b5803a8f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7b740a0d10 tx=0x7f7b64006ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:11.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:11.706+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(171..171 src has 13..171) v4 ==== 9787+0+0 (secure 0 0 0) 0x7f7b7004c740 con 0x7f7b7409ec50 2024-09-18T12:32:12.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.324+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7b740990d0 con 0x7f7b7409ec50 2024-09-18T12:32:12.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.324+0000 7f7b6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:12.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.324+0000 7f7b6a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:12.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.490+0000 7f7b7c39d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T12:32:12.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.490+0000 7f7b7c39d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T12:32:12.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.490+0000 7f7b7c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:12.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.490+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7f7b740990d0 con 0x7f7b7409ec50 2024-09-18T12:32:12.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.646+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v172) v1 ==== 132+0+0 (secure 0 0 0) 0x7f7b740990d0 con 0x7f7b7409ec50 2024-09-18T12:32:12.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.646+0000 7f7b6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T12:32:12.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.646+0000 7f7b6a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:12.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.746+0000 7f7b7c39d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T12:32:12.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.746+0000 7f7b7c39d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T12:32:12.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.746+0000 7f7b7c39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:12.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:12.746+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7f7b740990d0 con 0x7f7b7409ec50 2024-09-18T12:32:13.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:13.646+0000 7f7b7a139640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b630 tx=0x7f7b700091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f7b740990d0 mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v173) v1 2024-09-18T12:32:14.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.475+0000 7f7b6a7fc640 1 -- 172.21.15.144:0/2469379594 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v173) v1 ==== 132+0+0 (secure 0 0 0) 0x7f7b740990d0 con 0x7f7b7409ec50 2024-09-18T12:32:14.476 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.475+0000 7f7b6a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T12:32:14.477 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.475+0000 7f7b6a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:14.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b58038440 msgr2=0x7f7b5803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b58038440 0x7f7b5803a8f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7b740a0d10 tx=0x7f7b64006ca0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 10 monclient: shutdown 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 msgr2=0x7f7b7409b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b630 tx=0x7f7b700091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 shutdown_connections 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b58038440 0x7f7b5803a8f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b74136d60 0x7f7b74097650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b7409f020 0x7f7b74097110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 --2- 172.21.15.144:0/2469379594 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b7409ec50 0x7f7b7409b350 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 >> 172.21.15.144:0/2469379594 conn(0x7f7b740a5d40 msgr2=0x7f7b7409ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 shutdown_connections 2024-09-18T12:32:14.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.477+0000 7f7b7c39d640 1 -- 172.21.15.144:0/2469379594 wait complete. 2024-09-18T12:32:14.491 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:32:14.491 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:32:14.492 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep globster 2024-09-18T12:32:14.492 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '7\.' 2024-09-18T12:32:14.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.567+0000 7f800e99d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:14.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.567+0000 7f800e99d640 10 monclient: build_initial_monmap 2024-09-18T12:32:14.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.567+0000 7f800e99d640 10 monclient: monmap: 2024-09-18T12:32:14.569 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:14.569 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.569 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:14.569199+0000 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:14.569199+0000 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:14.570 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 1 Processor -- start 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 1 -- start start 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient: init 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809ec50 0x7f8008131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient(hunting): picked mon.noname-b con 0x7f800809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80081321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f800809f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8008067420 con 0x7f80081321a0 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080a0480 con 0x7f800809f020 2024-09-18T12:32:14.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80080a0200 con 0x7f800809ec50 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800e99d640 10 monclient(hunting): authenticate will time out at 1170.138676s 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f80077fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f80081321a0 legacy=0x7f8008068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800cf3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f800809f020 0x7f8008069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58654/0 (socket says 172.21.15.144:58654) 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.568+0000 7f800cf3a640 1 -- 172.21.15.144:0/2060814501 learned_addr learned my addr 172.21.15.144:0/2060814501 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 1 --2- 172.21.15.144:0/2060814501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809ec50 0x7f8008131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): get_auth_request con 0x7f800809ec50 auth_method 0 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:14.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient(hunting): handle_auth_done global_id 64823 payload 274 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 1 -- 172.21.15.144:0/2060814501 >> v1:172.21.15.144:6789/0 conn(0x7f800809f020 legacy=0x7f8008069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 1 -- 172.21.15.144:0/2060814501 >> v1:172.21.15.116:6789/0 conn(0x7f80081321a0 legacy=0x7f8008068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.569+0000 7f8007fff640 1 -- 172.21.15.144:0/2060814501 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800809fd80 con 0x7f800809ec50 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8007fff640 1 -- 172.21.15.144:0/2060814501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809ec50 msgr2=0x7f8008131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8007fff640 1 --2- 172.21.15.144:0/2060814501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809ec50 0x7f8008131c60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009880 tx=0x7f7ff80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=414d1dc6568836d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 1 -- 172.21.15.144:0/2060814501 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8032440 con 0x7f800809ec50 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: dump: 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:14.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 1 -- 172.21.15.144:0/2060814501 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff80325a0 con 0x7f800809ec50 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 1 -- 172.21.15.144:0/2060814501 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8032940 con 0x7f800809ec50 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f8006ffd640 10 monclient: dump: 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.582 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f800e99d640 5 monclient: authenticate success, global_id 64823 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f800e99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.572+0000 7f800e99d640 4 set_mon_vals no callback set 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 10 monclient: shutdown 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 -- 172.21.15.144:0/2060814501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809ec50 msgr2=0x7f8008131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 --2- 172.21.15.144:0/2060814501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809ec50 0x7f8008131c60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009880 tx=0x7f7ff80090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 -- 172.21.15.144:0/2060814501 shutdown_connections 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 --2- 172.21.15.144:0/2060814501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809ec50 0x7f8008131c60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 -- 172.21.15.144:0/2060814501 >> 172.21.15.144:0/2060814501 conn(0x7f80080a5cf0 msgr2=0x7f80080a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 -- 172.21.15.144:0/2060814501 shutdown_connections 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 -- 172.21.15.144:0/2060814501 wait complete. 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 10 monclient: build_initial_monmap 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 10 monclient: monmap: 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.584 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.573+0000 7f800e99d640 1 Processor -- start 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 1 -- start start 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient: init 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800809ec50 0x7f800809ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): picked mon.noname-a con 0x7f800809ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809f020 0x7f8008130e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): picked mon.noname-c con 0x7f800809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:14.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 0x7f8008131360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): picked mon.noname-b con 0x7f80081321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800809b990 con 0x7f800809ec50 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800809c020 con 0x7f80081321a0 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800809dd30 con 0x7f800809f020 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800e99d640 10 monclient(hunting): authenticate will time out at 1170.144524s 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f8007fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800809ec50 msgr2=0x7f800809ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f8007fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800809ec50 0x7f800809ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:14.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 0x7f8008131360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 0x7f8008131360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43756/0 (socket says 172.21.15.144:43756) 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 -- 172.21.15.144:0/3748401911 learned_addr learned my addr 172.21.15.144:0/3748401911 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): get_auth_request con 0x7f80081321a0 auth_method 0 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809f020 0x7f8008130e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 10 monclient(hunting): get_auth_request con 0x7f800809f020 auth_method 0 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:14.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f80077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient(hunting): handle_auth_done global_id 6040 payload 824 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 -- 172.21.15.144:0/3748401911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809f020 msgr2=0x7f8008130e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809f020 0x7f8008130e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 -- 172.21.15.144:0/3748401911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800809ec50 msgr2=0x7f800809ad30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800809ec50 0x7f800809ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 1 monclient: found mon.noname-b 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.574+0000 7f800cf3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:14.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.575+0000 7f800cf3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.575+0000 7f800cf3a640 1 -- 172.21.15.144:0/3748401911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8009390 con 0x7f80081321a0 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800cf3a640 1 -- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 msgr2=0x7f8008131360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800cf3a640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 0x7f8008131360 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f800000ccf0 tx=0x7f8000005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800000aa60 con 0x7f80081321a0 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: dump: 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.591 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: _finish_auth 0 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800e99d640 5 monclient: authenticate success, global_id 6040 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800000abe0 con 0x7f80081321a0 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:14.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000070f0 con 0x7f80081321a0 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800e99d640 10 monclient: _renew_subs 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800811df40 con 0x7f80081321a0 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f7feffff640 10 monclient: dump: 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.593 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f80067fc640 4 set_mon_vals no callback set 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800e99d640 10 monclient: _renew_subs 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.577+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800811e2d0 con 0x7f80081321a0 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f80000074e0 con 0x7f80081321a0 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f7feffff640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ff4038470 0x7f7ff403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(173..173 src has 13..173) v4 ==== 9787+0+0 (secure 0 0 0) 0x7f800004ccc0 con 0x7f80081321a0 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f800e99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f800e99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:14.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f800e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f8007fff640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ff4038470 0x7f7ff403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80080008d0 con 0x7f80081321a0 2024-09-18T12:32:14.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.578+0000 7f8007fff640 10 monclient: get_auth_request con 0x7f7ff4038470 auth_method 0 2024-09-18T12:32:14.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.580+0000 7f8007fff640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ff4038470 0x7f7ff403a920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8008071930 tx=0x7f7ff8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:14.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.583+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f800001a590 con 0x7f80081321a0 2024-09-18T12:32:14.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.583+0000 7f7feffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:14.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.583+0000 7f7feffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:14.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.746+0000 7f800e99d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:14.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.746+0000 7f800e99d640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:14.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.746+0000 7f800e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.746+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f80080a13e0 con 0x7f80081321a0 2024-09-18T12:32:14.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.746+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v173) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f8000010af0 con 0x7f80081321a0 2024-09-18T12:32:14.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.746+0000 7f7feffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:14.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.746+0000 7f7feffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:14.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.846+0000 7f800e99d640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:14.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.846+0000 7f800e99d640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:14.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.846+0000 7f800e99d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.846+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f800809ea30 con 0x7f80081321a0 2024-09-18T12:32:14.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.846+0000 7f7feffff640 1 -- 172.21.15.144:0/3748401911 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v173) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f800001a360 con 0x7f80081321a0 2024-09-18T12:32:14.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.846+0000 7f7feffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:14.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.846+0000 7f7feffff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ff4038470 msgr2=0x7f7ff403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ff4038470 0x7f7ff403a920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8008071930 tx=0x7f7ff8032000 comp rx=0 tx=0).stop 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 10 monclient: shutdown 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 msgr2=0x7f8008131360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 0x7f8008131360 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f800000ccf0 tx=0x7f8000005370 comp rx=0 tx=0).stop 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 shutdown_connections 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ff4038470 0x7f7ff403a920 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80081321a0 0x7f8008131360 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f800809f020 0x7f8008130e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 --2- 172.21.15.144:0/3748401911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800809ec50 0x7f800809ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 >> 172.21.15.144:0/3748401911 conn(0x7f80080a5cf0 msgr2=0x7f80080a4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:14.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 shutdown_connections 2024-09-18T12:32:14.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.848+0000 7f800e99d640 1 -- 172.21.15.144:0/3748401911 wait complete. 2024-09-18T12:32:14.859 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T12:32:14.859 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T12:32:14.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:32:14.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:32:14.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep globster 2024-09-18T12:32:14.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '8\.' 2024-09-18T12:32:14.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.931+0000 7f2ae54a8640 10 monclient: get_monmap_and_config 2024-09-18T12:32:14.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.931+0000 7f2ae54a8640 10 monclient: build_initial_monmap 2024-09-18T12:32:14.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.931+0000 7f2ae54a8640 10 monclient: monmap: 2024-09-18T12:32:14.933 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:14.933437+0000 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:14.933437+0000 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:14.934 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 1 Processor -- start 2024-09-18T12:32:14.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 1 -- start start 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient: init 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ae009ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ae009f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ae0069d10 0x7f2ae013a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae0069d10 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae0067420 con 0x7f2ae009f020 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae00a0480 con 0x7f2ae009ec50 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae006a170 con 0x7f2ae0069d10 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:14.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ae54a8640 10 monclient(hunting): authenticate will time out at 1170.502809s 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2adeffd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f2ae009ec50 0x7f2ae0131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58660/0 (socket says 172.21.15.144:58660) 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2adeffd640 1 -- 172.21.15.144:0/476117666 learned_addr learned my addr 172.21.15.144:0/476117666 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2adf7fe640 1 --2- 172.21.15.144:0/476117666 >> v2:172.21.15.144:3300/0 conn(0x7f2ae0069d10 0x7f2ae013a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2ad7fff640 1 -- 172.21.15.144:0/476117666 >> v1:172.21.15.116:6789/0 conn(0x7f2ae009f020 legacy=0x7f2ae0136d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2adf7fe640 10 monclient(hunting): get_auth_request con 0x7f2ae0069d10 auth_method 0 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2adf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2adf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:14.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.932+0000 7f2adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 10 monclient(hunting): handle_auth_done global_id 6043 payload 274 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 1 -- 172.21.15.144:0/476117666 >> v1:172.21.15.144:6789/0 conn(0x7f2ae009ec50 legacy=0x7f2ae0131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 1 -- 172.21.15.144:0/476117666 >> v1:172.21.15.116:6789/0 conn(0x7f2ae009f020 legacy=0x7f2ae0136d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 1 monclient: found mon.noname-c 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.933+0000 7f2adf7fe640 1 -- 172.21.15.144:0/476117666 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae00a0200 con 0x7f2ae0069d10 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.935+0000 7f2adf7fe640 1 -- 172.21.15.144:0/476117666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae0069d10 msgr2=0x7f2ae013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2adf7fe640 1 --2- 172.21.15.144:0/476117666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae0069d10 0x7f2ae013a490 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009e60 tx=0x7f2ad8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53b35c487a1322e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 1 -- 172.21.15.144:0/476117666 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8007840 con 0x7f2ae0069d10 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: dump: 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.945 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 1 -- 172.21.15.144:0/476117666 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad802d050 con 0x7f2ae0069d10 2024-09-18T12:32:14.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 5 monclient: authenticate success, global_id 6043 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 1 -- 172.21.15.144:0/476117666 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8007bd0 con 0x7f2ae0069d10 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 4 set_mon_vals no callback set 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ade7fc640 10 monclient: dump: 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.947 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 10 monclient: shutdown 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 1 -- 172.21.15.144:0/476117666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae0069d10 msgr2=0x7f2ae013a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/476117666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae0069d10 0x7f2ae013a490 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009e60 tx=0x7f2ad8009780 comp rx=0 tx=0).stop 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 1 -- 172.21.15.144:0/476117666 shutdown_connections 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/476117666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae0069d10 0x7f2ae013a490 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 1 -- 172.21.15.144:0/476117666 >> 172.21.15.144:0/476117666 conn(0x7f2ae00a5cf0 msgr2=0x7f2ae00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:14.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 1 -- 172.21.15.144:0/476117666 shutdown_connections 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.936+0000 7f2ae54a8640 1 -- 172.21.15.144:0/476117666 wait complete. 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient: build_initial_monmap 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient: monmap: 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:14.949 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 Processor -- start 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 -- start start 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient: init 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ae0069d10 0x7f2ae011cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae0069d10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 0x7f2ae011d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae009ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ae009f020 0x7f2ae011d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae011f3c0 con 0x7f2ae0069d10 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae009c050 con 0x7f2ae009ec50 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.937+0000 7f2ae54a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0120720 con 0x7f2ae009f020 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ae54a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ae54a8640 10 monclient(hunting): authenticate will time out at 1170.508011s 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 0x7f2ae011d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 0x7f2ae011d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43780/0 (socket says 172.21.15.144:43780) 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 -- 172.21.15.144:0/282265173 learned_addr learned my addr 172.21.15.144:0/282265173 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): get_auth_request con 0x7f2ae009ec50 auth_method 0 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:14.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2adeffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ae0069d10 msgr2=0x7f2ae011cda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2adeffd640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ae0069d10 0x7f2ae011cda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient(hunting): handle_auth_done global_id 6046 payload 824 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2adf7fe640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ae009f020 0x7f2ae011d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 -- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ae009f020 msgr2=0x7f2ae011d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ae009f020 0x7f2ae011d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 -- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ae0069d10 msgr2=0x7f2ae011cda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ae0069d10 0x7f2ae011cda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:14.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.938+0000 7f2ad7fff640 1 -- 172.21.15.144:0/282265173 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad8009a20 con 0x7f2ae009ec50 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad7fff640 1 -- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 msgr2=0x7f2ae011d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad7fff640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 0x7f2ae011d2e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800b600 tx=0x7f2ac80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8015070 con 0x7f2ae009ec50 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad7fff640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 0x7f2ae011d2e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800b600 tx=0x7f2ac80091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f2ac8003d90 config(0 keys) v1 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 10 monclient: dump: 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.953 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac8003d90 con 0x7f2ae009ec50 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ae54a8640 5 monclient: authenticate success, global_id 6046 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ad6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ae54a8640 10 monclient: _renew_subs 2024-09-18T12:32:14.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ae54a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae01304d0 con 0x7f2ae009ec50 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2addffb640 4 set_mon_vals no callback set 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ae54a8640 10 monclient: _renew_subs 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ae54a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.940+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae011f550 con 0x7f2ae009ec50 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8006380 con 0x7f2ae009ec50 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ad6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ad6ffd640 10 monclient: dump: 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.955 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2ac8010440 con 0x7f2ae009ec50 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ae54a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ae54a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ae54a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae0120470 con 0x7f2ae009ec50 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2ad6ffd640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ac0038470 0x7f2ac003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2adeffd640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ac0038470 0x7f2ac003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:14.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.941+0000 7f2adeffd640 10 monclient: get_auth_request con 0x7f2ac0038470 auth_method 0 2024-09-18T12:32:14.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.944+0000 7f2adeffd640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ac0038470 0x7f2ac003a920 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00047d0 tx=0x7f2ad0007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:14.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:14.946+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(173..173 src has 13..173) v4 ==== 9787+0+0 (secure 0 0 0) 0x7f2ac804c810 con 0x7f2ae009ec50 2024-09-18T12:32:15.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.563+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2ac8018d60 con 0x7f2ae009ec50 2024-09-18T12:32:15.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.563+0000 7f2ad6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:15.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.563+0000 7f2ad6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.729+0000 7f2ae54a8640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.729+0000 7f2ae54a8640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.729+0000 7f2ae54a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.729+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f2ae00008d0 con 0x7f2ae009ec50 2024-09-18T12:32:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.730+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v173) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f2ac8018780 con 0x7f2ae009ec50 2024-09-18T12:32:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.730+0000 7f2ad6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:15.732 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.730+0000 7f2ad6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:15.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.830+0000 7f2ae54a8640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:15.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.830+0000 7f2ae54a8640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:15.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.830+0000 7f2ae54a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:15.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.830+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f2ae009fd80 con 0x7f2ae009ec50 2024-09-18T12:32:15.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.831+0000 7f2ad6ffd640 1 -- 172.21.15.144:0/282265173 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v173) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f2ac8018780 con 0x7f2ae009ec50 2024-09-18T12:32:15.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.831+0000 7f2ad6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:15.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.831+0000 7f2ad6ffd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:15.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ac0038470 msgr2=0x7f2ac003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:15.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ac0038470 0x7f2ac003a920 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00047d0 tx=0x7f2ad0007b80 comp rx=0 tx=0).stop 2024-09-18T12:32:15.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 10 monclient: shutdown 2024-09-18T12:32:15.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:15.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 msgr2=0x7f2ae011d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:15.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 0x7f2ae011d2e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800b600 tx=0x7f2ac80091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 shutdown_connections 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2ac0038470 0x7f2ac003a920 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ae009f020 0x7f2ae011d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ae009ec50 0x7f2ae011d2e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 --2- 172.21.15.144:0/282265173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ae0069d10 0x7f2ae011cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 >> 172.21.15.144:0/282265173 conn(0x7f2ae00a5cf0 msgr2=0x7f2ae009d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 shutdown_connections 2024-09-18T12:32:15.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.833+0000 7f2ae54a8640 1 -- 172.21.15.144:0/282265173 wait complete. 2024-09-18T12:32:15.843 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T12:32:15.843 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T12:32:15.844 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T12:32:15.844 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep 'osd\.0' 2024-09-18T12:32:15.844 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep globster 2024-09-18T12:32:15.844 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep '6\.' 2024-09-18T12:32:15.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.915+0000 7fcf90017640 10 monclient: get_monmap_and_config 2024-09-18T12:32:15.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.915+0000 7fcf90017640 10 monclient: build_initial_monmap 2024-09-18T12:32:15.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.915+0000 7fcf90017640 10 monclient: monmap: 2024-09-18T12:32:15.917 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:15.917 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:15.917 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:15.917026+0000 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:15.917026+0000 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:15.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.915+0000 7fcf90017640 1 Processor -- start 2024-09-18T12:32:15.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.915+0000 7fcf90017640 1 -- start start 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.915+0000 7fcf90017640 10 monclient: init 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf8806e950 0x7fcf88096a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): picked mon.noname-a con 0x7fcf8806e950 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcf8809b8e0 0x7fcf8809bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): picked mon.noname-c con 0x7fcf8809b8e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88135700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): picked mon.noname-b con 0x7fcf8809c220 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf88097b40 con 0x7fcf8806e950 2024-09-18T12:32:15.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8809c6c0 con 0x7fcf8809b8e0 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88098470 con 0x7fcf8809c220 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf90017640 10 monclient(hunting): authenticate will time out at 1171.486415s 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcf8809b8e0 0x7fcf8809bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcf8809b8e0 0x7fcf8809bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43792/0 (socket says 172.21.15.144:43792) 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/1037769428 learned_addr learned my addr 172.21.15.144:0/1037769428 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): get_auth_request con 0x7fcf8809b8e0 auth_method 0 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:15.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8ddb3640 1 -- 172.21.15.144:0/1037769428 >> v2:172.21.15.116:3300/0 conn(0x7fcf8806e950 msgr2=0x7fcf88096a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8ddb3640 1 --2- 172.21.15.144:0/1037769428 >> v2:172.21.15.116:3300/0 conn(0x7fcf8806e950 0x7fcf88096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8e5b4640 1 --2- 172.21.15.144:0/1037769428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_done global_id 6049 payload 274 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/1037769428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 msgr2=0x7fcf88135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 --2- 172.21.15.144:0/1037769428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/1037769428 >> v2:172.21.15.116:3300/0 conn(0x7fcf8806e950 msgr2=0x7fcf88096a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 --2- 172.21.15.144:0/1037769428 >> v2:172.21.15.116:3300/0 conn(0x7fcf8806e950 0x7fcf88096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient: _finish_hunting 0 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 monclient: found mon.noname-c 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:15.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:15.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/1037769428 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88098750 con 0x7fcf8809b8e0 2024-09-18T12:32:15.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8e5b4640 10 monclient: get_auth_request con 0x7fcf8809c220 auth_method 0 2024-09-18T12:32:15.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.916+0000 7fcf8e5b4640 1 --2- 172.21.15.144:0/1037769428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/1037769428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 msgr2=0x7fcf8809bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8d5b2640 1 --2- 172.21.15.144:0/1037769428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf8809bce0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009910 tx=0x7fcf780091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3550530083b1f9a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 1 -- 172.21.15.144:0/1037769428 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78032440 con 0x7fcf8809b8e0 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: dump: 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:15.930 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: _finish_auth 0 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf90017640 5 monclient: authenticate success, global_id 6049 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf90017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 1 -- 172.21.15.144:0/1037769428 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf780325a0 con 0x7fcf8809b8e0 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf90017640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf90017640 4 set_mon_vals no callback set 2024-09-18T12:32:15.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 1 -- 172.21.15.144:0/1037769428 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78032940 con 0x7fcf8809b8e0 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.919+0000 7fcf8cdb1640 10 monclient: dump: 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:15.932 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 10 monclient: shutdown 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 -- 172.21.15.144:0/1037769428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 msgr2=0x7fcf8809bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 --2- 172.21.15.144:0/1037769428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf8809bce0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009910 tx=0x7fcf780091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 -- 172.21.15.144:0/1037769428 shutdown_connections 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 --2- 172.21.15.144:0/1037769428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 --2- 172.21.15.144:0/1037769428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf8809bce0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 --2- 172.21.15.144:0/1037769428 >> v2:172.21.15.116:3300/0 conn(0x7fcf8806e950 0x7fcf88096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 -- 172.21.15.144:0/1037769428 >> 172.21.15.144:0/1037769428 conn(0x7fcf88071100 msgr2=0x7fcf880a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 -- 172.21.15.144:0/1037769428 shutdown_connections 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 1 -- 172.21.15.144:0/1037769428 wait complete. 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 10 monclient: build_initial_monmap 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.920+0000 7fcf90017640 10 monclient: monmap: 2024-09-18T12:32:15.933 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 Processor -- start 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 -- start start 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient: init 2024-09-18T12:32:15.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8806e950 0x7fcf8812f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): picked mon.noname-a con 0x7fcf8806e950 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf88133860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): picked mon.noname-b con 0x7fcf8809b8e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): picked mon.noname-c con 0x7fcf8809c220 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8809ab90 con 0x7fcf8806e950 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88118ef0 con 0x7fcf8809b8e0 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:15.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88118c10 con 0x7fcf8809c220 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf90017640 10 monclient(hunting): authenticate will time out at 1171.491577s 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf88133860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf88133860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43804/0 (socket says 172.21.15.144:43804) 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/3118236698 learned_addr learned my addr 172.21.15.144:0/3118236698 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): get_auth_request con 0x7fcf8809b8e0 auth_method 0 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8ddb3640 1 -- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8806e950 msgr2=0x7fcf8812f850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8ddb3640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8806e950 0x7fcf8812f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:15.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient(hunting): handle_auth_done global_id 6052 payload 824 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8e5b4640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 msgr2=0x7fcf88133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88133da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8806e950 msgr2=0x7fcf8812f850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8806e950 0x7fcf8812f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient: _finish_hunting 0 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 monclient: found mon.noname-b 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/3118236698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78009450 con 0x7fcf8809b8e0 2024-09-18T12:32:15.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8e5b4640 10 monclient: get_auth_request con 0x7fcf8809c220 auth_method 0 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.921+0000 7fcf8e5b4640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf8d5b2640 1 -- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 msgr2=0x7fcf88133860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf8d5b2640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf88133860 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcf78035040 tx=0x7fcf78032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78031bf0 con 0x7fcf8809b8e0 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: dump: 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:15.938 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf78031d90 con 0x7fcf8809b8e0 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf90017640 5 monclient: authenticate success, global_id 6052 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf90017640 10 monclient: _renew_subs 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf90017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8813fd20 con 0x7fcf8809b8e0 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:15.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7803b650 con 0x7fcf8809b8e0 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7dffb640 10 monclient: dump: 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:15.940 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf7ffff640 4 set_mon_vals no callback set 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf90017640 10 monclient: _renew_subs 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf90017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.924+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf88138320 con 0x7fcf8809b8e0 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fcf78039040 con 0x7fcf8809b8e0 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf90017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf90017640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf90017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf880008d0 con 0x7fcf8809b8e0 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf7dffb640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcf6c038440 0x7fcf6c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(173..173 src has 13..173) v4 ==== 9787+0+0 (secure 0 0 0) 0x7fcf7806cbc0 con 0x7fcf8809b8e0 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf8ddb3640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcf6c038440 0x7fcf6c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:15.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.925+0000 7fcf8ddb3640 10 monclient: get_auth_request con 0x7fcf6c038440 auth_method 0 2024-09-18T12:32:15.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.928+0000 7fcf8ddb3640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcf6c038440 0x7fcf6c03a8f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcf840038f0 tx=0x7fcf84006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:15.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.929+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fcf78039ad0 con 0x7fcf8809b8e0 2024-09-18T12:32:15.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.930+0000 7fcf7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:15.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:15.930+0000 7fcf7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.092+0000 7fcf90017640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.092+0000 7fcf90017640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.092+0000 7fcf90017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.092+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcf8809d650 con 0x7fcf8809b8e0 2024-09-18T12:32:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.093+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v173) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fcf7803f070 con 0x7fcf8809b8e0 2024-09-18T12:32:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.093+0000 7fcf7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:16.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.093+0000 7fcf7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:16.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.193+0000 7fcf90017640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:16.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.193+0000 7fcf90017640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:16.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.193+0000 7fcf90017640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.193+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fcf8809fdf0 con 0x7fcf8809b8e0 2024-09-18T12:32:16.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.193+0000 7fcf7dffb640 1 -- 172.21.15.144:0/3118236698 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v173) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fcf78037030 con 0x7fcf8809b8e0 2024-09-18T12:32:16.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.193+0000 7fcf7dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T12:32:16.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.193+0000 7fcf7dffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:16.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.195+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcf6c038440 msgr2=0x7fcf6c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:16.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.195+0000 7fcf90017640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcf6c038440 0x7fcf6c03a8f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcf840038f0 tx=0x7fcf84006d80 comp rx=0 tx=0).stop 2024-09-18T12:32:16.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.195+0000 7fcf90017640 10 monclient: shutdown 2024-09-18T12:32:16.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.195+0000 7fcf90017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:16.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.195+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 msgr2=0x7fcf88133860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.195+0000 7fcf90017640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf88133860 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcf78035040 tx=0x7fcf78032000 comp rx=0 tx=0).stop 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 shutdown_connections 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcf6c038440 0x7fcf6c03a8f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcf8809c220 0x7fcf88133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf8809b8e0 0x7fcf88133860 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 --2- 172.21.15.144:0/3118236698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8806e950 0x7fcf8812f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 >> 172.21.15.144:0/3118236698 conn(0x7fcf88071100 msgr2=0x7fcf880996f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 shutdown_connections 2024-09-18T12:32:16.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.196+0000 7fcf90017640 1 -- 172.21.15.144:0/3118236698 wait complete. 2024-09-18T12:32:16.208 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T12:32:16.208 INFO:tasks.workunit.client.0.smithi144.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T12:32:16.208 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-18T12:32:16.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.278+0000 7f79f8d54640 10 monclient: get_monmap_and_config 2024-09-18T12:32:16.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.278+0000 7f79f8d54640 10 monclient: build_initial_monmap 2024-09-18T12:32:16.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.278+0000 7f79f8d54640 10 monclient: monmap: 2024-09-18T12:32:16.280 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:16.280 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:16.280 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:16.280277+0000 2024-09-18T12:32:16.280 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:16.280277+0000 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:16.281 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:16.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 1 Processor -- start 2024-09-18T12:32:16.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 1 -- start start 2024-09-18T12:32:16.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient: init 2024-09-18T12:32:16.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:16.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79f409c250 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4134860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient(hunting): picked mon.noname-b con 0x7f79f409e170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79f409e540 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f409b790 con 0x7f79f409c250 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f409b4b0 con 0x7f79f409e540 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f409b230 con 0x7f79f409e170 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f8d54640 10 monclient(hunting): authenticate will time out at 1171.849687s 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f259c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f79f409c250 legacy=0x7f79f409da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:16.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f2d9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f79f409e540 0x7f79f4136db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58676/0 (socket says 172.21.15.144:58676) 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79f2d9d640 1 -- 172.21.15.144:0/3735938473 learned_addr learned my addr 172.21.15.144:0/3735938473 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 1 --2- 172.21.15.144:0/3735938473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4134860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 10 monclient(hunting): get_auth_request con 0x7f79f409e170 auth_method 0 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.279+0000 7f79ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 10 monclient(hunting): handle_auth_done global_id 64829 payload 274 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 1 -- 172.21.15.144:0/3735938473 >> v1:172.21.15.144:6789/0 conn(0x7f79f409e540 legacy=0x7f79f4136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 1 -- 172.21.15.144:0/3735938473 >> v1:172.21.15.116:6789/0 conn(0x7f79f409c250 legacy=0x7f79f409da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:16.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:16.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 1 monclient: found mon.noname-b 2024-09-18T12:32:16.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:16.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:16.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.280+0000 7f79ebfff640 1 -- 172.21.15.144:0/3735938473 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f409ba70 con 0x7f79f409e170 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79ebfff640 1 -- 172.21.15.144:0/3735938473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 msgr2=0x7f79f4134860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79ebfff640 1 --2- 172.21.15.144:0/3735938473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4134860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f79f40a0f20 tx=0x7f79dc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a8155791820a0ef server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 1 -- 172.21.15.144:0/3735938473 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc032440 con 0x7f79f409e170 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: dump: 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:16.294 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: _finish_auth 0 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 5 monclient: authenticate success, global_id 64829 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 1 -- 172.21.15.144:0/3735938473 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc0325a0 con 0x7f79f409e170 2024-09-18T12:32:16.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 1 -- 172.21.15.144:0/3735938473 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc032940 con 0x7f79f409e170 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 4 set_mon_vals no callback set 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f1d9b640 10 monclient: dump: 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.296 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 10 monclient: shutdown 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 1 -- 172.21.15.144:0/3735938473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 msgr2=0x7f79f4134860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 1 --2- 172.21.15.144:0/3735938473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4134860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f79f40a0f20 tx=0x7f79dc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 1 -- 172.21.15.144:0/3735938473 shutdown_connections 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 1 --2- 172.21.15.144:0/3735938473 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4134860 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 1 -- 172.21.15.144:0/3735938473 >> 172.21.15.144:0/3735938473 conn(0x7f79f4071340 msgr2=0x7f79f40a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:16.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 1 -- 172.21.15.144:0/3735938473 shutdown_connections 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.283+0000 7f79f8d54640 1 -- 172.21.15.144:0/3735938473 wait complete. 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.284+0000 7f79f8d54640 10 monclient: build_initial_monmap 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.284+0000 7f79f8d54640 10 monclient: monmap: 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:16.298 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.284+0000 7f79f8d54640 1 Processor -- start 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.284+0000 7f79f8d54640 1 -- start start 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.284+0000 7f79f8d54640 10 monclient: init 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 0x7f79f4132340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): picked mon.noname-b con 0x7f79f409c250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:16.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): picked mon.noname-c con 0x7f79f409e170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f409e540 0x7f79f4135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): picked mon.noname-a con 0x7f79f409e540 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f411a2a0 con 0x7f79f409e540 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f41372d0 con 0x7f79f409c250 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4137020 con 0x7f79f409e170 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f8d54640 10 monclient(hunting): authenticate will time out at 1171.855154s 2024-09-18T12:32:16.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 0x7f79f4132340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 0x7f79f4132340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43810/0 (socket says 172.21.15.144:43810) 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 -- 172.21.15.144:0/581922981 learned_addr learned my addr 172.21.15.144:0/581922981 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): get_auth_request con 0x7f79f409c250 auth_method 0 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f2d9d640 1 -- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f409e540 msgr2=0x7f79f4135640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f2d9d640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f409e540 0x7f79f4135640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:16.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79ebfff640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79ebfff640 10 monclient(hunting): get_auth_request con 0x7f79f409e170 auth_method 0 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient(hunting): handle_auth_done global_id 6055 payload 824 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 -- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 msgr2=0x7f79f4135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4135100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 -- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f409e540 msgr2=0x7f79f4135640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f409e540 0x7f79f4135640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 monclient: found mon.noname-b 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.285+0000 7f79f259c640 1 -- 172.21.15.144:0/581922981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc009390 con 0x7f79f409c250 2024-09-18T12:32:16.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f259c640 1 -- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 msgr2=0x7f79f4132340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f259c640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 0x7f79f4132340 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f79e400b600 tx=0x7f79e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4015070 con 0x7f79f409c250 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: dump: 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:16.303 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e4003d90 con 0x7f79f409c250 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f8d54640 5 monclient: authenticate success, global_id 6055 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f8d54640 10 monclient: _renew_subs 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f8d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f406e9d0 con 0x7f79f409c250 2024-09-18T12:32:16.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4006380 con 0x7f79f409c250 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79ea7fc640 10 monclient: dump: 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:16.305 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f8d54640 10 monclient: _renew_subs 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.287+0000 7f79f159a640 4 set_mon_vals no callback set 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.288+0000 7f79f8d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.288+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f411af60 con 0x7f79f409c250 2024-09-18T12:32:16.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.288+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f79e4010440 con 0x7f79f409c250 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79f8d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79f8d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79f8d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f40008d0 con 0x7f79f409c250 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79ea7fc640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79d4038440 0x7f79d403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(173..173 src has 13..173) v4 ==== 9787+0+0 (secure 0 0 0) 0x7f79e404c620 con 0x7f79f409c250 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79ebfff640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79d4038440 0x7f79d403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.289+0000 7f79ebfff640 10 monclient: get_auth_request con 0x7f79d4038440 auth_method 0 2024-09-18T12:32:16.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.291+0000 7f79ebfff640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79d4038440 0x7f79d403a8f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f79f40a3fd0 tx=0x7f79dc032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:16.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.293+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f79e4006770 con 0x7f79f409c250 2024-09-18T12:32:16.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.293+0000 7f79ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:16.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.293+0000 7f79ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:16.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.452+0000 7f79f8d54640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:32:16.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.452+0000 7f79f8d54640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:32:16.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.452+0000 7f79f8d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.452+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f79f4064a20 con 0x7f79f409c250 2024-09-18T12:32:16.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.518+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v174) v1 ==== 123+0+0 (secure 0 0 0) 0x7f79e4018690 con 0x7f79f409c250 2024-09-18T12:32:16.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.518+0000 7f79ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:32:16.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.518+0000 7f79ea7fc640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-18T12:32:16.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.618+0000 7f79f8d54640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:32:16.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.618+0000 7f79f8d54640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:32:16.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.618+0000 7f79f8d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:16.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:16.618+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f79f41376e0 con 0x7f79f409c250 2024-09-18T12:32:17.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.602+0000 7f79ea7fc640 1 -- 172.21.15.144:0/581922981 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v176) v1 ==== 149+0+0 (secure 0 0 0) 0x7f79e4018ab0 con 0x7f79f409c250 2024-09-18T12:32:17.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.602+0000 7f79ea7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T12:32:17.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.602+0000 7f79ea7fc640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T12:32:17.605 INFO:tasks.workunit.client.0.smithi144.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T12:32:17.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.604+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79d4038440 msgr2=0x7f79d403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:17.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.604+0000 7f79f8d54640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79d4038440 0x7f79d403a8f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f79f40a3fd0 tx=0x7f79dc032000 comp rx=0 tx=0).stop 2024-09-18T12:32:17.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.604+0000 7f79f8d54640 10 monclient: shutdown 2024-09-18T12:32:17.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.604+0000 7f79f8d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:17.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.604+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 msgr2=0x7f79f4132340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:17.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 0x7f79f4132340 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f79e400b600 tx=0x7f79e40091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 shutdown_connections 2024-09-18T12:32:17.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79d4038440 0x7f79d403a8f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f409e540 0x7f79f4135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79f409e170 0x7f79f4135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 --2- 172.21.15.144:0/581922981 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79f409c250 0x7f79f4132340 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 >> 172.21.15.144:0/581922981 conn(0x7f79f4071340 msgr2=0x7f79f4137cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:17.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 shutdown_connections 2024-09-18T12:32:17.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.605+0000 7f79f8d54640 1 -- 172.21.15.144:0/581922981 wait complete. 2024-09-18T12:32:17.617 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-18T12:32:17.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.694+0000 7fd46066f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:17.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.694+0000 7fd46066f640 10 monclient: build_initial_monmap 2024-09-18T12:32:17.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.694+0000 7fd46066f640 10 monclient: monmap: 2024-09-18T12:32:17.696 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:17.696 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:17.696 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:17.696009+0000 2024-09-18T12:32:17.696 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:17.696009+0000 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:17.697 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:17.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.694+0000 7fd46066f640 1 Processor -- start 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.694+0000 7fd46066f640 1 -- start start 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.694+0000 7fd46066f640 10 monclient: init 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd45809ec50 0x7fd458131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient(hunting): picked mon.noname-a con 0x7fd45809ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd458068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient(hunting): picked mon.noname-b con 0x7fd4581321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd45809f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd458067420 con 0x7fd45809f020 2024-09-18T12:32:17.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45809fdc0 con 0x7fd45809ec50 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580a0510 con 0x7fd4581321a0 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd46066f640 10 monclient(hunting): authenticate will time out at 1173.265458s 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45ec0c640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fd45809f020 legacy=0x7fd458069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45e40b640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fd45809ec50 msgr2=0x7fd458131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45e40b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd45809ec50 0x7fd458131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd458068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd458068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:44038/0 (socket says 172.21.15.144:44038) 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 1 -- 172.21.15.144:0/3865301901 learned_addr learned my addr 172.21.15.144:0/3865301901 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:17.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): get_auth_request con 0x7fd4581321a0 auth_method 0 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.695+0000 7fd45dc0a640 10 monclient(hunting): handle_auth_done global_id 64835 payload 274 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 1 -- 172.21.15.144:0/3865301901 >> v2:172.21.15.116:3300/0 conn(0x7fd45809ec50 msgr2=0x7fd458131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 1 --2- 172.21.15.144:0/3865301901 >> v2:172.21.15.116:3300/0 conn(0x7fd45809ec50 0x7fd458131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 1 -- 172.21.15.144:0/3865301901 >> v1:172.21.15.116:6789/0 conn(0x7fd45809f020 legacy=0x7fd458069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 1 monclient: found mon.noname-b 2024-09-18T12:32:17.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:17.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:17.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.696+0000 7fd45dc0a640 1 -- 172.21.15.144:0/3865301901 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4580a0200 con 0x7fd4581321a0 2024-09-18T12:32:17.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.698+0000 7fd45dc0a640 1 -- 172.21.15.144:0/3865301901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 msgr2=0x7fd458068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.698+0000 7fd45dc0a640 1 --2- 172.21.15.144:0/3865301901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd458068d00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd454009910 tx=0x7fd4540091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=348b50087a3a3527 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 1 -- 172.21.15.144:0/3865301901 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454032440 con 0x7fd4581321a0 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: dump: 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:17.709 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: _finish_auth 0 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 5 monclient: authenticate success, global_id 64835 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 1 -- 172.21.15.144:0/3865301901 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4540325a0 con 0x7fd4581321a0 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 4 set_mon_vals no callback set 2024-09-18T12:32:17.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 1 -- 172.21.15.144:0/3865301901 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454032940 con 0x7fd4581321a0 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd45d409640 10 monclient: dump: 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:17.711 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 10 monclient: shutdown 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 1 -- 172.21.15.144:0/3865301901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 msgr2=0x7fd458068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 1 --2- 172.21.15.144:0/3865301901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd458068d00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd454009910 tx=0x7fd4540091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 1 -- 172.21.15.144:0/3865301901 shutdown_connections 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 1 --2- 172.21.15.144:0/3865301901 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd458068d00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 1 --2- 172.21.15.144:0/3865301901 >> v2:172.21.15.116:3300/0 conn(0x7fd45809ec50 0x7fd458131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.699+0000 7fd46066f640 1 -- 172.21.15.144:0/3865301901 >> 172.21.15.144:0/3865301901 conn(0x7fd4580a5d40 msgr2=0x7fd4580a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:17.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.700+0000 7fd46066f640 1 -- 172.21.15.144:0/3865301901 shutdown_connections 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.700+0000 7fd46066f640 1 -- 172.21.15.144:0/3865301901 wait complete. 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.700+0000 7fd46066f640 10 monclient: build_initial_monmap 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.700+0000 7fd46066f640 10 monclient: monmap: 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:17.713 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.700+0000 7fd46066f640 1 Processor -- start 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.700+0000 7fd46066f640 1 -- start start 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.700+0000 7fd46066f640 10 monclient: init 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 0x7fd45809a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): picked mon.noname-b con 0x7fd45809ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45809f020 0x7fd458130a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): picked mon.noname-a con 0x7fd45809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd45811b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): picked mon.noname-c con 0x7fd4581321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580a4240 con 0x7fd45809f020 2024-09-18T12:32:17.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd458142050 con 0x7fd45809ec50 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45811de10 con 0x7fd4581321a0 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd46066f640 10 monclient(hunting): authenticate will time out at 1173.271316s 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 0x7fd45809a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45dc0a640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45809f020 msgr2=0x7fd458130a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45dc0a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45809f020 0x7fd458130a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 0x7fd45809a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48950/0 (socket says 172.21.15.144:48950) 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 -- 172.21.15.144:0/2189066361 learned_addr learned my addr 172.21.15.144:0/2189066361 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): get_auth_request con 0x7fd45809ec50 auth_method 0 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:17.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45ec0c640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd45811b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45ec0c640 10 monclient(hunting): get_auth_request con 0x7fd4581321a0 auth_method 0 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45ec0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45ec0c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45ec0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient(hunting): handle_auth_done global_id 6058 payload 824 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 -- 172.21.15.144:0/2189066361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 msgr2=0x7fd45811b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:17.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd45811b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 -- 172.21.15.144:0/2189066361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45809f020 msgr2=0x7fd458130a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45809f020 0x7fd458130a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 monclient: found mon.noname-b 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.701+0000 7fd45e40b640 1 -- 172.21.15.144:0/2189066361 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454009450 con 0x7fd45809ec50 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.703+0000 7fd45e40b640 1 -- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 msgr2=0x7fd45809a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.703+0000 7fd45e40b640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 0x7fd45809a9b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd44800b5c0 tx=0x7fd448009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448003cc0 con 0x7fd45809ec50 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:17.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: dump: 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:17.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd46066f640 5 monclient: authenticate success, global_id 6058 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd46066f640 10 monclient: _renew_subs 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd448003e40 con 0x7fd45809ec50 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd46066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45809b310 con 0x7fd45809ec50 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448005340 con 0x7fd45809ec50 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44e7fc640 10 monclient: dump: 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:17.719 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd46066f640 10 monclient: _renew_subs 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd46066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd44ffff640 4 set_mon_vals no callback set 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.704+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd458136740 con 0x7fd45809ec50 2024-09-18T12:32:17.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd44801a460 con 0x7fd45809ec50 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd44e7fc640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd43c038440 0x7fd43c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd46066f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd46066f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd46066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(176..176 src has 13..176) v4 ==== 9098+0+0 (secure 0 0 0) 0x7fd44804c4a0 con 0x7fd45809ec50 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4580008d0 con 0x7fd45809ec50 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd45dc0a640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd43c038440 0x7fd43c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:17.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.705+0000 7fd45dc0a640 10 monclient: get_auth_request con 0x7fd43c038440 auth_method 0 2024-09-18T12:32:17.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.707+0000 7fd45dc0a640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd43c038440 0x7fd43c03a8f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd4580a2b00 tx=0x7fd454032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:17.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.710+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd448017af0 con 0x7fd45809ec50 2024-09-18T12:32:17.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.710+0000 7fd44e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:17.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.710+0000 7fd44e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:17.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.863+0000 7fd46066f640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:17.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.863+0000 7fd46066f640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:17.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.863+0000 7fd46066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:17.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:17.863+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd45811dfd0 con 0x7fd45809ec50 2024-09-18T12:32:18.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.613+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v177) v1 ==== 148+0+0 (secure 0 0 0) 0x7fd448017310 con 0x7fd45809ec50 2024-09-18T12:32:18.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.613+0000 7fd44e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:18.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.613+0000 7fd44e7fc640 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.710+0000 7fd46066f640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.710+0000 7fd46066f640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.710+0000 7fd46066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.710+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd458064900 con 0x7fd45809ec50 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.711+0000 7fd44e7fc640 1 -- 172.21.15.144:0/2189066361 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v177) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd4480174f0 con 0x7fd45809ec50 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.711+0000 7fd44e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.711+0000 7fd44e7fc640 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-18T12:32:18.712 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'cool' does not exist 2024-09-18T12:32:18.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd43c038440 msgr2=0x7fd43c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:18.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd43c038440 0x7fd43c03a8f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd4580a2b00 tx=0x7fd454032000 comp rx=0 tx=0).stop 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 10 monclient: shutdown 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 msgr2=0x7fd45809a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 0x7fd45809a9b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd44800b5c0 tx=0x7fd448009120 comp rx=0 tx=0).stop 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 shutdown_connections 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd43c038440 0x7fd43c03a8f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd4581321a0 0x7fd45811b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd45809f020 0x7fd458130a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 --2- 172.21.15.144:0/2189066361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd45809ec50 0x7fd45809a9b0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 >> 172.21.15.144:0/2189066361 conn(0x7fd4580a5d40 msgr2=0x7fd458134dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 shutdown_connections 2024-09-18T12:32:18.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.713+0000 7fd46066f640 1 -- 172.21.15.144:0/2189066361 wait complete. 2024-09-18T12:32:18.723 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-18T12:32:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.798+0000 7f715c300640 10 monclient: get_monmap_and_config 2024-09-18T12:32:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.798+0000 7f715c300640 10 monclient: build_initial_monmap 2024-09-18T12:32:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.798+0000 7f715c300640 10 monclient: monmap: 2024-09-18T12:32:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:18.799 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:18.799265+0000 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:18.799265+0000 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:18.800 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.798+0000 7f715c300640 1 Processor -- start 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.798+0000 7f715c300640 1 -- start start 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.798+0000 7f715c300640 10 monclient: init 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f715409d140 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7154095a60 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f715413a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient(hunting): picked mon.noname-b con 0x7f715409dc20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:18.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715406f1e0 con 0x7f7154095a60 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715409ecf0 con 0x7f715409d140 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715406f760 con 0x7f715409dc20 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715c300640 10 monclient(hunting): authenticate will time out at 1174.368591s 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715a09c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f715409d140 0x7f715409d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34024/0 (socket says 172.21.15.144:34024) 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715a09c640 1 -- 172.21.15.144:0/2705826068 learned_addr learned my addr 172.21.15.144:0/2705826068 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715989b640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f7154095a60 legacy=0x7f7154136f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715a89d640 1 --2- 172.21.15.144:0/2705826068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f715413a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:18.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715a89d640 10 monclient(hunting): get_auth_request con 0x7f715409dc20 auth_method 0 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715a89d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715a89d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.799+0000 7f715a89d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 10 monclient(hunting): handle_auth_done global_id 64841 payload 274 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 1 -- 172.21.15.144:0/2705826068 >> v1:172.21.15.144:6789/0 conn(0x7f715409d140 legacy=0x7f715409d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 1 -- 172.21.15.144:0/2705826068 >> v1:172.21.15.116:6789/0 conn(0x7f7154095a60 legacy=0x7f7154136f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 10 monclient: _finish_hunting 0 2024-09-18T12:32:18.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 1 monclient: found mon.noname-b 2024-09-18T12:32:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:18.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.800+0000 7f715a89d640 1 -- 172.21.15.144:0/2705826068 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7154098590 con 0x7f715409dc20 2024-09-18T12:32:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715a89d640 1 -- 172.21.15.144:0/2705826068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 msgr2=0x7f715413a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715a89d640 1 --2- 172.21.15.144:0/2705826068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f715413a650 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f714c009e90 tx=0x7f714c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd9b1767abd2d256 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 1 -- 172.21.15.144:0/2705826068 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c007840 con 0x7f715409dc20 2024-09-18T12:32:18.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: dump: 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:18.813 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: _finish_auth 0 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 1 -- 172.21.15.144:0/2705826068 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c02d050 con 0x7f715409dc20 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 1 -- 172.21.15.144:0/2705826068 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c007bd0 con 0x7f715409dc20 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715909a640 10 monclient: dump: 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:18.814 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 5 monclient: authenticate success, global_id 64841 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:18.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 4 set_mon_vals no callback set 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 10 monclient: shutdown 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 1 -- 172.21.15.144:0/2705826068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 msgr2=0x7f715413a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 1 --2- 172.21.15.144:0/2705826068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f715413a650 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f714c009e90 tx=0x7f714c009780 comp rx=0 tx=0).stop 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 1 -- 172.21.15.144:0/2705826068 shutdown_connections 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 1 --2- 172.21.15.144:0/2705826068 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f715413a650 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 1 -- 172.21.15.144:0/2705826068 >> 172.21.15.144:0/2705826068 conn(0x7f71540a2190 msgr2=0x7f715409e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 1 -- 172.21.15.144:0/2705826068 shutdown_connections 2024-09-18T12:32:18.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.803+0000 7f715c300640 1 -- 172.21.15.144:0/2705826068 wait complete. 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.804+0000 7f715c300640 10 monclient: build_initial_monmap 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.804+0000 7f715c300640 10 monclient: monmap: 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:18.817 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.804+0000 7f715c300640 1 Processor -- start 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.804+0000 7f715c300640 1 -- start start 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.804+0000 7f715c300640 10 monclient: init 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154095a60 0x7f71541348f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): picked mon.noname-a con 0x7f7154095a60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 0x7f7154134e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): picked mon.noname-b con 0x7f715409d140 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f7154135370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:18.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): picked mon.noname-c con 0x7f715409dc20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154096200 con 0x7f7154095a60 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154135b10 con 0x7f715409d140 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71540909b0 con 0x7f715409dc20 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715c300640 10 monclient(hunting): authenticate will time out at 1174.374113s 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 0x7f7154134e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715a09c640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154095a60 msgr2=0x7f71541348f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715a09c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154095a60 0x7f71541348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 0x7f7154134e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48964/0 (socket says 172.21.15.144:48964) 2024-09-18T12:32:18.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 -- 172.21.15.144:0/3014313805 learned_addr learned my addr 172.21.15.144:0/3014313805 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): get_auth_request con 0x7f715409d140 auth_method 0 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715a89d640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f7154135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient(hunting): handle_auth_done global_id 6061 payload 824 2024-09-18T12:32:18.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 -- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 msgr2=0x7f7154135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f7154135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 -- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154095a60 msgr2=0x7f71541348f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154095a60 0x7f71541348f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 monclient: found mon.noname-b 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715989b640 1 -- 172.21.15.144:0/3014313805 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c009a20 con 0x7f715409d140 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715a89d640 10 monclient: get_auth_request con 0x7f715409dc20 auth_method 0 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.805+0000 7f715a89d640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f7154135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715989b640 1 -- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 msgr2=0x7f7154134e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:18.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715989b640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 0x7f7154134e30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f714400b630 tx=0x7f71440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144015070 con 0x7f715409d140 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: dump: 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.822 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:18.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715c300640 5 monclient: authenticate success, global_id 6061 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7144003d90 con 0x7f715409d140 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715c300640 10 monclient: _renew_subs 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715c300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154091960 con 0x7f715409d140 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144006380 con 0x7f715409d140 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f714a7fc640 10 monclient: dump: 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:18.824 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715c300640 10 monclient: _renew_subs 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f7158899640 4 set_mon_vals no callback set 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715c300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:18.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.807+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715408eba0 con 0x7f715409d140 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.808+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7144010440 con 0x7f715409d140 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.808+0000 7f715c300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.808+0000 7f715c300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.808+0000 7f715c300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.808+0000 7f714a7fc640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7138038440 0x7f713803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.808+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7154090b70 con 0x7f715409d140 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.809+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(177..177 src has 13..177) v4 ==== 8427+0+0 (secure 0 0 0) 0x7f714404c1c0 con 0x7f715409d140 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.809+0000 7f715a09c640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7138038440 0x7f713803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.809+0000 7f715a09c640 10 monclient: get_auth_request con 0x7f7138038440 auth_method 0 2024-09-18T12:32:18.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.811+0000 7f715a09c640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7138038440 0x7f713803a8f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f71500047d0 tx=0x7f7150007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:18.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:18.813+0000 7f715989b640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 0x7f7154134e30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f714400b630 tx=0x7f71440091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f7144018a70 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:32:19.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:19.641+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7144018a70 con 0x7f715409d140 2024-09-18T12:32:19.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:19.641+0000 7f714a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:19.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:19.641+0000 7f714a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:19.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:19.799+0000 7f715c300640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:19.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:19.799+0000 7f715c300640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:19.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:19.799+0000 7f715c300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:19.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:19.799+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71540008d0 con 0x7f715409d140 2024-09-18T12:32:20.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.628+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v178) v1 ==== 148+0+0 (secure 0 0 0) 0x7f7144049020 con 0x7f715409d140 2024-09-18T12:32:20.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.628+0000 7f714a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:20.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.628+0000 7f714a7fc640 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-18T12:32:20.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.726+0000 7f715c300640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:20.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.726+0000 7f715c300640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:20.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.726+0000 7f715c300640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:20.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.726+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7154064a20 con 0x7f715409d140 2024-09-18T12:32:20.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.726+0000 7f714a7fc640 1 -- 172.21.15.144:0/3014313805 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v178) v1 ==== 155+0+0 (secure 0 0 0) 0x7f7144028070 con 0x7f715409d140 2024-09-18T12:32:20.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.726+0000 7f714a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T12:32:20.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.726+0000 7f714a7fc640 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-18T12:32:20.728 INFO:tasks.workunit.client.0.smithi144.stderr:pool 'cold' does not exist 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7138038440 msgr2=0x7f713803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7138038440 0x7f713803a8f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f71500047d0 tx=0x7f7150007b80 comp rx=0 tx=0).stop 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 10 monclient: shutdown 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 msgr2=0x7f7154134e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 0x7f7154134e30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f714400b630 tx=0x7f71440091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 shutdown_connections 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7138038440 0x7f713803a8f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f715409dc20 0x7f7154135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f715409d140 0x7f7154134e30 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 --2- 172.21.15.144:0/3014313805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154095a60 0x7f71541348f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 >> 172.21.15.144:0/3014313805 conn(0x7f71540a2190 msgr2=0x7f715409ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:20.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 shutdown_connections 2024-09-18T12:32:20.731 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.728+0000 7f715c300640 1 -- 172.21.15.144:0/3014313805 wait complete. 2024-09-18T12:32:20.739 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.812+0000 7fb12c154640 10 monclient: get_monmap_and_config 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.812+0000 7fb12c154640 10 monclient: build_initial_monmap 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.812+0000 7fb12c154640 10 monclient: monmap: 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:20.813878+0000 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:20.813878+0000 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:20.814 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:20.815 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:20.815 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:20.815 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:20.815 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:20.815 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:20.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.813+0000 7fb12c154640 1 Processor -- start 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.813+0000 7fb12c154640 1 -- start start 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.813+0000 7fb12c154640 10 monclient: init 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb124131c00 0x7fb124131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient(hunting): picked mon.noname-a con 0x7fb124131c00 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1240a0b00 0x7fb124132510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient(hunting): picked mon.noname-b con 0x7fb1240a0b00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb124068d00 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12409d230 con 0x7fb124068d00 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1240a08c0 con 0x7fb124131c00 2024-09-18T12:32:20.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1240a0200 con 0x7fb1240a0b00 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12c154640 10 monclient(hunting): authenticate will time out at 1176.383326s 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb129ef0640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fb124131c00 msgr2=0x7fb124131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb129ef0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb124131c00 0x7fb124131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12a6f1640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb124068d00 0x7fb124069100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34026/0 (socket says 172.21.15.144:34026) 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb12a6f1640 1 -- 172.21.15.144:0/4066436730 learned_addr learned my addr 172.21.15.144:0/4066436730 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 1 --2- 172.21.15.144:0/4066436730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1240a0b00 0x7fb124132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): get_auth_request con 0x7fb1240a0b00 auth_method 0 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:20.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient(hunting): handle_auth_done global_id 64844 payload 274 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 1 -- 172.21.15.144:0/4066436730 >> v2:172.21.15.116:3300/0 conn(0x7fb124131c00 msgr2=0x7fb124131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 1 --2- 172.21.15.144:0/4066436730 >> v2:172.21.15.116:3300/0 conn(0x7fb124131c00 0x7fb124131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 1 -- 172.21.15.144:0/4066436730 >> v1:172.21.15.144:6789/0 conn(0x7fb124068d00 legacy=0x7fb124069100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient: _finish_hunting 0 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 1 monclient: found mon.noname-b 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:20.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.814+0000 7fb1296ef640 1 -- 172.21.15.144:0/4066436730 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1240a1b70 con 0x7fb1240a0b00 2024-09-18T12:32:20.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.817+0000 7fb1296ef640 1 -- 172.21.15.144:0/4066436730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1240a0b00 msgr2=0x7fb124132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.817+0000 7fb1296ef640 1 --2- 172.21.15.144:0/4066436730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1240a0b00 0x7fb124132510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb118009850 tx=0x7fb1180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf2df75368dc9da4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 1 -- 172.21.15.144:0/4066436730 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118032440 con 0x7fb1240a0b00 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: dump: 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:20.827 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: _finish_auth 0 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 1 -- 172.21.15.144:0/4066436730 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1180325a0 con 0x7fb1240a0b00 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb12c154640 5 monclient: authenticate success, global_id 64844 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb12c154640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:20.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb12c154640 4 set_mon_vals no callback set 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 1 -- 172.21.15.144:0/4066436730 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118032940 con 0x7fb1240a0b00 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.818+0000 7fb128eee640 10 monclient: dump: 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:20.829 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 10 monclient: shutdown 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 -- 172.21.15.144:0/4066436730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1240a0b00 msgr2=0x7fb124132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 --2- 172.21.15.144:0/4066436730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1240a0b00 0x7fb124132510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb118009850 tx=0x7fb1180090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 -- 172.21.15.144:0/4066436730 shutdown_connections 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 --2- 172.21.15.144:0/4066436730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1240a0b00 0x7fb124132510 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 --2- 172.21.15.144:0/4066436730 >> v2:172.21.15.116:3300/0 conn(0x7fb124131c00 0x7fb124131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 -- 172.21.15.144:0/4066436730 >> 172.21.15.144:0/4066436730 conn(0x7fb124071370 msgr2=0x7fb1240a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 -- 172.21.15.144:0/4066436730 shutdown_connections 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 1 -- 172.21.15.144:0/4066436730 wait complete. 2024-09-18T12:32:20.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 10 monclient: build_initial_monmap 2024-09-18T12:32:20.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.819+0000 7fb12c154640 10 monclient: monmap: 2024-09-18T12:32:20.831 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:20.831 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:20.831 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:20.831 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 Processor -- start 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 -- start start 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient: init 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:20.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb124068d00 0x7fb12409a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): picked mon.noname-c con 0x7fb124068d00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1240a0b00 0x7fb124130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): picked mon.noname-a con 0x7fb1240a0b00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 0x7fb12411b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): picked mon.noname-b con 0x7fb124131c00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12411e010 con 0x7fb1240a0b00 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb124141f80 con 0x7fb124131c00 2024-09-18T12:32:20.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12409b5e0 con 0x7fb124068d00 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12c154640 10 monclient(hunting): authenticate will time out at 1176.389528s 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 0x7fb12411b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb1296ef640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1240a0b00 msgr2=0x7fb124130c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 0x7fb12411b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48974/0 (socket says 172.21.15.144:48974) 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 -- 172.21.15.144:0/623954326 learned_addr learned my addr 172.21.15.144:0/623954326 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): get_auth_request con 0x7fb124131c00 auth_method 0 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:20.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb1296ef640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1240a0b00 0x7fb124130c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb129ef0640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb124068d00 0x7fb12409a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb129ef0640 10 monclient(hunting): get_auth_request con 0x7fb124068d00 auth_method 0 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb129ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb129ef0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb129ef0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient(hunting): handle_auth_done global_id 6064 payload 824 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 -- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb124068d00 msgr2=0x7fb12409a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb124068d00 0x7fb12409a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 -- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1240a0b00 msgr2=0x7fb124130c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1240a0b00 0x7fb124130c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:20.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient: _finish_hunting 0 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 monclient: found mon.noname-b 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.820+0000 7fb12a6f1640 1 -- 172.21.15.144:0/623954326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118009390 con 0x7fb124131c00 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12a6f1640 1 -- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 msgr2=0x7fb12411b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12a6f1640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 0x7fb12411b900 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb11400bb80 tx=0x7fb1140097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11400cc80 con 0x7fb124131c00 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: dump: 2024-09-18T12:32:20.836 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:20.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11400cde0 con 0x7fb124131c00 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12c154640 5 monclient: authenticate success, global_id 6064 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12c154640 10 monclient: _renew_subs 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12c154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12411dd30 con 0x7fb124131c00 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114014690 con 0x7fb124131c00 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb121ffb640 10 monclient: dump: 2024-09-18T12:32:20.838 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12c154640 10 monclient: _renew_subs 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12c154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1241415e0 con 0x7fb124131c00 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.823+0000 7fb123fff640 4 set_mon_vals no callback set 2024-09-18T12:32:20.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb1140133a0 con 0x7fb124131c00 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb121ffb640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb110038440 0x7fb11003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(178..178 src has 13..178) v4 ==== 7620+0+0 (secure 0 0 0) 0x7fb11404bdc0 con 0x7fb124131c00 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb129ef0640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb110038440 0x7fb11003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb129ef0640 10 monclient: get_auth_request con 0x7fb110038440 auth_method 0 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb12c154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb12c154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb12c154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:20.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.824+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1240008d0 con 0x7fb124131c00 2024-09-18T12:32:20.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.827+0000 7fb129ef0640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb110038440 0x7fb11003a8f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb11c0038f0 tx=0x7fb11c006ef0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:20.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.829+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb114019080 con 0x7fb124131c00 2024-09-18T12:32:20.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.829+0000 7fb121ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:20.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.829+0000 7fb121ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:20.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.992+0000 7fb12c154640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:20.993 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.992+0000 7fb12c154640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:20.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.992+0000 7fb12c154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:20.994 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:20.992+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fb12411e490 con 0x7fb124131c00 2024-09-18T12:32:21.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:21.593+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v179) v1 ==== 78+0+0 (secure 0 0 0) 0x7fb114007700 con 0x7fb124131c00 2024-09-18T12:32:21.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:21.593+0000 7fb121ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:21.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:21.593+0000 7fb121ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:21.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:21.702+0000 7fb12c154640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:21.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:21.702+0000 7fb12c154640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:21.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:21.702+0000 7fb12c154640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:21.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:21.702+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fb124072640 con 0x7fb124131c00 2024-09-18T12:32:22.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.593+0000 7fb121ffb640 1 -- 172.21.15.144:0/623954326 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v180) v1 ==== 78+0+0 (secure 0 0 0) 0x7fb1140112a0 con 0x7fb124131c00 2024-09-18T12:32:22.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.593+0000 7fb121ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:22.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.593+0000 7fb121ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:22.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb110038440 msgr2=0x7fb11003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:22.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb110038440 0x7fb11003a8f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb11c0038f0 tx=0x7fb11c006ef0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 10 monclient: shutdown 2024-09-18T12:32:22.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:22.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 msgr2=0x7fb12411b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:22.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 0x7fb12411b900 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb11400bb80 tx=0x7fb1140097e0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 shutdown_connections 2024-09-18T12:32:22.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb110038440 0x7fb11003a8f0 secure :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb11c0038f0 tx=0x7fb11c006ef0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb124131c00 0x7fb12411b900 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1240a0b00 0x7fb124130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 --2- 172.21.15.144:0/623954326 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb124068d00 0x7fb12409a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.600+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 >> 172.21.15.144:0/623954326 conn(0x7fb124071370 msgr2=0x7fb124134d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:22.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.601+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 shutdown_connections 2024-09-18T12:32:22.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.601+0000 7fb12c154640 1 -- 172.21.15.144:0/623954326 wait complete. 2024-09-18T12:32:22.618 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T12:32:22.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.695+0000 7f0a0179d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:22.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.695+0000 7f0a0179d640 10 monclient: build_initial_monmap 2024-09-18T12:32:22.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.695+0000 7f0a0179d640 10 monclient: monmap: 2024-09-18T12:32:22.696 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:22.696 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:22.696373+0000 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:22.696373+0000 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:22.697 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:22.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 1 Processor -- start 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 1 -- start start 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient: init 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09fc09ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09fc09f020 0x7f09fc069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient(hunting): picked mon.noname-a con 0x7f09fc09f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 0x7f09fc139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient(hunting): picked mon.noname-b con 0x7f09fc136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09fc067420 con 0x7f09fc09ec50 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc0a0480 con 0x7f09fc09f020 2024-09-18T12:32:22.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0a0200 con 0x7f09fc136d60 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f0a0179d640 10 monclient(hunting): authenticate will time out at 1178.265612s 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fa7fc640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f09fc09f020 msgr2=0x7f09fc069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09faffd640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f09fc09ec50 legacy=0x7f09fc131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fa7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09fc09f020 0x7f09fc069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fb7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 0x7f09fc139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fb7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 0x7f09fc139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:44092/0 (socket says 172.21.15.144:44092) 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fb7fe640 1 -- 172.21.15.144:0/1085562071 learned_addr learned my addr 172.21.15.144:0/1085562071 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request con 0x7f09fc136d60 auth_method 0 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.696+0000 7f09fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:22.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_done global_id 64850 payload 274 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 1 -- 172.21.15.144:0/1085562071 >> v2:172.21.15.116:3300/0 conn(0x7f09fc09f020 msgr2=0x7f09fc069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 1 --2- 172.21.15.144:0/1085562071 >> v2:172.21.15.116:3300/0 conn(0x7f09fc09f020 0x7f09fc069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 1 -- 172.21.15.144:0/1085562071 >> v1:172.21.15.116:6789/0 conn(0x7f09fc09ec50 legacy=0x7f09fc131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:22.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 1 monclient: found mon.noname-b 2024-09-18T12:32:22.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:22.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:22.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.697+0000 7f09fb7fe640 1 -- 172.21.15.144:0/1085562071 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc09fd80 con 0x7f09fc136d60 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09fb7fe640 1 -- 172.21.15.144:0/1085562071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 msgr2=0x7f09fc139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09fb7fe640 1 --2- 172.21.15.144:0/1085562071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 0x7f09fc139140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009f20 tx=0x7f09f4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e54dd2f31f3a0c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 1 -- 172.21.15.144:0/1085562071 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4007610 con 0x7f09fc136d60 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: dump: 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:22.709 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 1 -- 172.21.15.144:0/1085562071 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f402d050 con 0x7f09fc136d60 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 5 monclient: authenticate success, global_id 64850 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:22.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 1 -- 172.21.15.144:0/1085562071 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40079a0 con 0x7f09fc136d60 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 4 set_mon_vals no callback set 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f09f9ffb640 10 monclient: dump: 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:22.711 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 10 monclient: shutdown 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 -- 172.21.15.144:0/1085562071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 msgr2=0x7f09fc139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 --2- 172.21.15.144:0/1085562071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 0x7f09fc139140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009f20 tx=0x7f09f4009840 comp rx=0 tx=0).stop 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 -- 172.21.15.144:0/1085562071 shutdown_connections 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 --2- 172.21.15.144:0/1085562071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc136d60 0x7f09fc139140 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 --2- 172.21.15.144:0/1085562071 >> v2:172.21.15.116:3300/0 conn(0x7f09fc09f020 0x7f09fc069d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 -- 172.21.15.144:0/1085562071 >> 172.21.15.144:0/1085562071 conn(0x7f09fc0a5cf0 msgr2=0x7f09fc0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 -- 172.21.15.144:0/1085562071 shutdown_connections 2024-09-18T12:32:22.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.700+0000 7f0a0179d640 1 -- 172.21.15.144:0/1085562071 wait complete. 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.701+0000 7f0a0179d640 10 monclient: build_initial_monmap 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.701+0000 7f0a0179d640 10 monclient: monmap: 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:22.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.701+0000 7f0a0179d640 1 Processor -- start 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.701+0000 7f0a0179d640 1 -- start start 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.701+0000 7f0a0179d640 10 monclient: init 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09fc09ec50 0x7f09fc09b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): picked mon.noname-a con 0x7f09fc09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc09f020 0x7f09fc0971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): picked mon.noname-c con 0x7f09fc09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 0x7f09fc097700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): picked mon.noname-b con 0x7f09fc136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc099840 con 0x7f09fc09ec50 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:22.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc09c020 con 0x7f09fc136d60 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0a3c50 con 0x7f09fc09f020 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f0a0179d640 10 monclient(hunting): authenticate will time out at 1178.271351s 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09faffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09fc09ec50 msgr2=0x7f09fc09b460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09fc09ec50 0x7f09fc09b460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 0x7f09fc097700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 0x7f09fc097700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48980/0 (socket says 172.21.15.144:48980) 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 -- 172.21.15.144:0/3300148897 learned_addr learned my addr 172.21.15.144:0/3300148897 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request con 0x7f09fc136d60 auth_method 0 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:22.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc09f020 0x7f09fc0971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 10 monclient(hunting): get_auth_request con 0x7f09fc09f020 auth_method 0 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_done global_id 6067 payload 824 2024-09-18T12:32:22.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 -- 172.21.15.144:0/3300148897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc09f020 msgr2=0x7f09fc0971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc09f020 0x7f09fc0971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 -- 172.21.15.144:0/3300148897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09fc09ec50 msgr2=0x7f09fc09b460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09fc09ec50 0x7f09fc09b460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 monclient: found mon.noname-b 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.702+0000 7f09fb7fe640 1 -- 172.21.15.144:0/3300148897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4009ae0 con 0x7f09fc136d60 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09fb7fe640 1 -- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 msgr2=0x7f09fc097700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09fb7fe640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 0x7f09fc097700 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009f20 tx=0x7f09f4036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4007e10 con 0x7f09fc136d60 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:22.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: dump: 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:22.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f4031cb0 con 0x7f09fc136d60 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f0a0179d640 5 monclient: authenticate success, global_id 6067 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f0a0179d640 10 monclient: _renew_subs 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4044640 con 0x7f09fc136d60 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f0a0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc099180 con 0x7f09fc136d60 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09e2ffd640 10 monclient: dump: 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:22.719 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f09f8ff9640 4 set_mon_vals no callback set 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f0a0179d640 10 monclient: _renew_subs 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f0a0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.705+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc09ae10 con 0x7f09fc136d60 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f09f4038070 con 0x7f09fc136d60 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f0a0179d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:22.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f0a0179d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:22.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f0a0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:22.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc098d30 con 0x7f09fc136d60 2024-09-18T12:32:22.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f09e2ffd640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09e8038440 0x7f09e803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:22.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(180..180 src has 13..180) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f09f406ce60 con 0x7f09fc136d60 2024-09-18T12:32:22.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f09faffd640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09e8038440 0x7f09e803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:22.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.706+0000 7f09faffd640 10 monclient: get_auth_request con 0x7f09e8038440 auth_method 0 2024-09-18T12:32:22.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.709+0000 7f09faffd640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09e8038440 0x7f09e803a8f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006fd0 tx=0x7f09ec006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:22.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:22.710+0000 7f09fb7fe640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 0x7f09fc097700 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009f20 tx=0x7f09f4036000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f09fc098d30 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:32:23.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:23.539+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f09fc098d30 con 0x7f09fc136d60 2024-09-18T12:32:23.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:23.539+0000 7f09e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:23.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:23.539+0000 7f09e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:23.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:23.707+0000 7f0a0179d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:23.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:23.707+0000 7f0a0179d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:23.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:23.707+0000 7f0a0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:23.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:23.707+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f09fc098d30 con 0x7f09fc136d60 2024-09-18T12:32:24.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.640+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v182) v1 ==== 82+0+0 (secure 0 0 0) 0x7f09fc098d30 con 0x7f09fc136d60 2024-09-18T12:32:24.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.640+0000 7f09e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:24.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.640+0000 7f09e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:24.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.744+0000 7f0a0179d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:24.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.744+0000 7f0a0179d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:24.745 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.744+0000 7f0a0179d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:24.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.744+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f09fc098d30 con 0x7f09fc136d60 2024-09-18T12:32:24.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.745+0000 7f09e2ffd640 1 -- 172.21.15.144:0/3300148897 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v182) v1 ==== 115+0+0 (secure 0 0 0) 0x7f09fc098d30 con 0x7f09fc136d60 2024-09-18T12:32:24.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.745+0000 7f09e2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:24.746 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.745+0000 7f09e2ffd640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T12:32:24.746 INFO:tasks.workunit.client.0.smithi144.stderr:compat weight-set already created 2024-09-18T12:32:24.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09e8038440 msgr2=0x7f09e803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:24.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09e8038440 0x7f09e803a8f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006fd0 tx=0x7f09ec006d30 comp rx=0 tx=0).stop 2024-09-18T12:32:24.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 10 monclient: shutdown 2024-09-18T12:32:24.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:24.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 msgr2=0x7f09fc097700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:24.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 0x7f09fc097700 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009f20 tx=0x7f09f4036000 comp rx=0 tx=0).stop 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 shutdown_connections 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09e8038440 0x7f09e803a8f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09fc136d60 0x7f09fc097700 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09fc09f020 0x7f09fc0971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 --2- 172.21.15.144:0/3300148897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09fc09ec50 0x7f09fc09b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 >> 172.21.15.144:0/3300148897 conn(0x7f09fc0a5cf0 msgr2=0x7f09fc09e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 shutdown_connections 2024-09-18T12:32:24.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.747+0000 7f0a0179d640 1 -- 172.21.15.144:0/3300148897 wait complete. 2024-09-18T12:32:24.757 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket fooo host 2024-09-18T12:32:24.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.831+0000 7fa8bb06f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:24.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.831+0000 7fa8bb06f640 10 monclient: build_initial_monmap 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.831+0000 7fa8bb06f640 10 monclient: monmap: 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:24.831954+0000 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:24.831954+0000 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:24.833 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:24.834 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:24.834 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:24.834 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:24.834 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:24.834 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.831+0000 7fa8bb06f640 1 Processor -- start 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.831+0000 7fa8bb06f640 1 -- start start 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.831+0000 7fa8bb06f640 10 monclient: init 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8b409ec50 0x7fa8b4131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b409ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:24.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa8b41321a0 0x7fa8b4068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b41321a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8b409f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b4067420 con 0x7fa8b409f020 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b409fdc0 con 0x7fa8b409ec50 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b40a0510 con 0x7fa8b41321a0 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8bb06f640 10 monclient(hunting): authenticate will time out at 1180.401496s 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b8e0b640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fa8b409ec50 msgr2=0x7fa8b4131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:24.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b8e0b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8b409ec50 0x7fa8b4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa8b41321a0 0x7fa8b4068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa8b41321a0 0x7fa8b4068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48990/0 (socket says 172.21.15.144:48990) 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 -- 172.21.15.144:0/2156241053 learned_addr learned my addr 172.21.15.144:0/2156241053 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b960c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa8b409f020 0x7fa8b4069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34032/0 (socket says 172.21.15.144:34032) 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): get_auth_request con 0x7fa8b41321a0 auth_method 0 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:24.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient(hunting): handle_auth_done global_id 6070 payload 274 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 -- 172.21.15.144:0/2156241053 >> v2:172.21.15.116:3300/0 conn(0x7fa8b409ec50 msgr2=0x7fa8b4131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 --2- 172.21.15.144:0/2156241053 >> v2:172.21.15.116:3300/0 conn(0x7fa8b409ec50 0x7fa8b4131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 -- 172.21.15.144:0/2156241053 >> v1:172.21.15.144:6789/0 conn(0x7fa8b409f020 legacy=0x7fa8b4069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 monclient: found mon.noname-c 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.832+0000 7fa8b3fff640 1 -- 172.21.15.144:0/2156241053 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b40a0200 con 0x7fa8b41321a0 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b3fff640 1 -- 172.21.15.144:0/2156241053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 msgr2=0x7fa8b4068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b3fff640 1 --2- 172.21.15.144:0/2156241053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4068d00 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40a2b00 tx=0x7fa8a80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf4ada053936f990 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:24.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 1 -- 172.21.15.144:0/2156241053 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8032440 con 0x7fa8b41321a0 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: dump: 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:24.848 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 1 -- 172.21.15.144:0/2156241053 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a80325a0 con 0x7fa8b41321a0 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8bb06f640 5 monclient: authenticate success, global_id 6070 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8bb06f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8bb06f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 1 -- 172.21.15.144:0/2156241053 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8032980 con 0x7fa8b41321a0 2024-09-18T12:32:24.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8bb06f640 4 set_mon_vals no callback set 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.835+0000 7fa8b37fe640 10 monclient: dump: 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:24.850 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 10 monclient: shutdown 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 -- 172.21.15.144:0/2156241053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 msgr2=0x7fa8b4068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/2156241053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4068d00 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40a2b00 tx=0x7fa8a80091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 -- 172.21.15.144:0/2156241053 shutdown_connections 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/2156241053 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4068d00 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/2156241053 >> v2:172.21.15.116:3300/0 conn(0x7fa8b409ec50 0x7fa8b4131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 -- 172.21.15.144:0/2156241053 >> 172.21.15.144:0/2156241053 conn(0x7fa8b40a5d40 msgr2=0x7fa8b40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:24.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 -- 172.21.15.144:0/2156241053 shutdown_connections 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 1 -- 172.21.15.144:0/2156241053 wait complete. 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 10 monclient: build_initial_monmap 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.836+0000 7fa8bb06f640 10 monclient: monmap: 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:24.852 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 Processor -- start 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 -- start start 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient: init 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8b409ec50 0x7fa8b409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b409f020 0x7fa8b411bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:24.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b41321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b411e1f0 con 0x7fa8b409f020 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b411dca0 con 0x7fa8b41321a0 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b411df10 con 0x7fa8b409ec50 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.837+0000 7fa8bb06f640 10 monclient(hunting): authenticate will time out at 1180.406997s 2024-09-18T12:32:24.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b3fff640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b409f020 msgr2=0x7fa8b411bc20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b409f020 0x7fa8b411bc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4130b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48998/0 (socket says 172.21.15.144:48998) 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 1 -- 172.21.15.144:0/104347915 learned_addr learned my addr 172.21.15.144:0/104347915 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 10 monclient(hunting): get_auth_request con 0x7fa8b41321a0 auth_method 0 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 1 -- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 msgr2=0x7fa8b4130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 1 -- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 msgr2=0x7fa8b4130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:24.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8b409ec50 0x7fa8b409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b960c640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4130b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): get_auth_request con 0x7fa8b409ec50 auth_method 0 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient(hunting): handle_auth_done global_id 64856 payload 824 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 1 -- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 msgr2=0x7fa8b4130b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:24.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4130b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 1 -- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b409f020 msgr2=0x7fa8b411bc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b409f020 0x7fa8b411bc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 1 monclient: found mon.noname-c 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.838+0000 7fa8b8e0b640 1 -- 172.21.15.144:0/104347915 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a8009450 con 0x7fa8b409ec50 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b8e0b640 1 -- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8b409ec50 msgr2=0x7fa8b409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b8e0b640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8b409ec50 0x7fa8b409a9d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400b5b0 tx=0x7fa8a40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4003d90 con 0x7fa8b409ec50 2024-09-18T12:32:24.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: dump: 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:24.858 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a4006150 con 0x7fa8b409ec50 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4006550 con 0x7fa8b409ec50 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b0ff9640 10 monclient: dump: 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:24.859 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8bb06f640 5 monclient: authenticate success, global_id 64856 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8bb06f640 10 monclient: _renew_subs 2024-09-18T12:32:24.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8bb06f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8b2ffd640 4 set_mon_vals no callback set 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b411c160 con 0x7fa8b409ec50 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8bb06f640 10 monclient: _renew_subs 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8bb06f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.841+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b41311f0 con 0x7fa8b409ec50 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8bb06f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8bb06f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8bb06f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa8a40103e0 con 0x7fa8b409ec50 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b40008d0 con 0x7fa8b409ec50 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8b0ff9640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa8a0038440 0x7fa8a003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:24.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(182..182 src has 13..182) v4 ==== 7525+0+0 (secure 0 0 0) 0x7fa8a404bac0 con 0x7fa8b409ec50 2024-09-18T12:32:24.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8b3fff640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa8a0038440 0x7fa8a003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:24.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.842+0000 7fa8b3fff640 10 monclient: get_auth_request con 0x7fa8a0038440 auth_method 0 2024-09-18T12:32:24.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.844+0000 7fa8b3fff640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa8a0038440 0x7fa8a003a8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8034000 tx=0x7fa8a8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:24.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.847+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa8a400ed00 con 0x7fa8b409ec50 2024-09-18T12:32:24.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.847+0000 7fa8b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:24.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:24.847+0000 7fa8b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:25.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.003+0000 7fa8bb06f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T12:32:25.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.003+0000 7fa8bb06f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T12:32:25.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.003+0000 7fa8bb06f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:25.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.003+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7fa8b40a13e0 con 0x7fa8b409ec50 2024-09-18T12:32:25.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.650+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v183) v1 ==== 140+0+0 (secure 0 0 0) 0x7fa8a40160a0 con 0x7fa8b409ec50 2024-09-18T12:32:25.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.650+0000 7fa8b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T12:32:25.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.650+0000 7fa8b0ff9640 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-18T12:32:25.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.747+0000 7fa8bb06f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T12:32:25.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.747+0000 7fa8bb06f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T12:32:25.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.747+0000 7fa8bb06f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:25.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.747+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7fa8b4072640 con 0x7fa8b409ec50 2024-09-18T12:32:25.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.749+0000 7fa8b0ff9640 1 -- 172.21.15.144:0/104347915 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v183) v1 ==== 128+0+0 (secure 0 0 0) 0x7fa8a400e700 con 0x7fa8b409ec50 2024-09-18T12:32:25.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.749+0000 7fa8b0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T12:32:25.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.749+0000 7fa8b0ff9640 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-18T12:32:25.750 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'fooo' already exists 2024-09-18T12:32:25.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa8a0038440 msgr2=0x7fa8a003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa8a0038440 0x7fa8a003a8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8034000 tx=0x7fa8a8032000 comp rx=0 tx=0).stop 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 10 monclient: shutdown 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8b409ec50 msgr2=0x7fa8b409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8b409ec50 0x7fa8b409a9d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400b5b0 tx=0x7fa8a40090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 shutdown_connections 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa8a0038440 0x7fa8a003a8f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa8b41321a0 0x7fa8b4130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b409f020 0x7fa8b411bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 --2- 172.21.15.144:0/104347915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa8b409ec50 0x7fa8b409a9d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 >> 172.21.15.144:0/104347915 conn(0x7fa8b40a5d40 msgr2=0x7fa8b409e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 shutdown_connections 2024-09-18T12:32:25.753 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.751+0000 7fa8bb06f640 1 -- 172.21.15.144:0/104347915 wait complete. 2024-09-18T12:32:25.762 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush move fooo root=default 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.835+0000 7fed78ec0640 10 monclient: get_monmap_and_config 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.835+0000 7fed78ec0640 10 monclient: build_initial_monmap 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.835+0000 7fed78ec0640 10 monclient: monmap: 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:25.836878+0000 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:25.836878+0000 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:25.837 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:25.838 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:25.838 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:25.838 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:25.838 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:25.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.836+0000 7fed78ec0640 1 Processor -- start 2024-09-18T12:32:25.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.836+0000 7fed78ec0640 1 -- start start 2024-09-18T12:32:25.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.836+0000 7fed78ec0640 10 monclient: init 2024-09-18T12:32:25.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:25.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed740a0ad0 0x7fed740691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fed740a0ad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed7409ee60 0x7fed7409f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fed7409ee60 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed74131ad0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7406a5b0 con 0x7fed74131ad0 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740a0ea0 con 0x7fed7409ee60 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed740a1bb0 con 0x7fed740a0ad0 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:25.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed78ec0640 10 monclient(hunting): authenticate will time out at 1181.406258s 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed7409ee60 0x7fed7409f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fed7409ee60 0x7fed7409f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49014/0 (socket says 172.21.15.144:49014) 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed72d9d640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fed74131ad0 legacy=0x7fed74131f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 -- 172.21.15.144:0/2314890956 learned_addr learned my addr 172.21.15.144:0/2314890956 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): get_auth_request con 0x7fed7409ee60 auth_method 0 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed7159a640 1 -- 172.21.15.144:0/2314890956 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:25.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed7259c640 1 --2- 172.21.15.144:0/2314890956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed740a0ad0 0x7fed740691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient(hunting): handle_auth_done global_id 6073 payload 274 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 -- 172.21.15.144:0/2314890956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed740a0ad0 msgr2=0x7fed740691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 --2- 172.21.15.144:0/2314890956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed740a0ad0 0x7fed740691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 -- 172.21.15.144:0/2314890956 >> v1:172.21.15.116:6789/0 conn(0x7fed74131ad0 legacy=0x7fed74131f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 monclient: found mon.noname-c 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed71d9b640 1 -- 172.21.15.144:0/2314890956 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7409f770 con 0x7fed7409ee60 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed7259c640 10 monclient: get_auth_request con 0x7fed740a0ad0 auth_method 0 2024-09-18T12:32:25.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.837+0000 7fed7259c640 1 --2- 172.21.15.144:0/2314890956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed740a0ad0 0x7fed740691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:32:25.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed71d9b640 1 -- 172.21.15.144:0/2314890956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed7409ee60 msgr2=0x7fed7409f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed71d9b640 1 --2- 172.21.15.144:0/2314890956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed7409ee60 0x7fed7409f230 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fed68009910 tx=0x7fed680091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=526b5a54c2437a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 1 -- 172.21.15.144:0/2314890956 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6802ae00 con 0x7fed7409ee60 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: dump: 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:25.850 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: _finish_auth 0 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 1 -- 172.21.15.144:0/2314890956 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6802f4b0 con 0x7fed7409ee60 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed78ec0640 5 monclient: authenticate success, global_id 6073 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed78ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed78ec0640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 1 -- 172.21.15.144:0/2314890956 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6802f850 con 0x7fed7409ee60 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed78ec0640 4 set_mon_vals no callback set 2024-09-18T12:32:25.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.840+0000 7fed7159a640 10 monclient: dump: 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:25.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 10 monclient: shutdown 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 -- 172.21.15.144:0/2314890956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed7409ee60 msgr2=0x7fed7409f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 --2- 172.21.15.144:0/2314890956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed7409ee60 0x7fed7409f230 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fed68009910 tx=0x7fed680091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 -- 172.21.15.144:0/2314890956 shutdown_connections 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 --2- 172.21.15.144:0/2314890956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed7409ee60 0x7fed7409f230 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 --2- 172.21.15.144:0/2314890956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed740a0ad0 0x7fed740691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 -- 172.21.15.144:0/2314890956 >> 172.21.15.144:0/2314890956 conn(0x7fed74071370 msgr2=0x7fed740a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 -- 172.21.15.144:0/2314890956 shutdown_connections 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 -- 172.21.15.144:0/2314890956 wait complete. 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 10 monclient: build_initial_monmap 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 10 monclient: monmap: 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:25.853 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 Processor -- start 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 1 -- start start 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.841+0000 7fed78ec0640 10 monclient: init 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:25.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed7409ee60 0x7fed74131010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fed7409ee60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 0x7fed74131550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fed740a0ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74131ad0 0x7fed74141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): picked mon.noname-a con 0x7fed74131ad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed740a11e0 con 0x7fed74131ad0 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7411f700 con 0x7fed740a0ad0 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7411d010 con 0x7fed7409ee60 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed78ec0640 10 monclient(hunting): authenticate will time out at 1181.411402s 2024-09-18T12:32:25.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed72d9d640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74131ad0 msgr2=0x7fed74141390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed72d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74131ad0 0x7fed74141390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 0x7fed74131550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 0x7fed74131550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49018/0 (socket says 172.21.15.144:49018) 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 -- 172.21.15.144:0/734293271 learned_addr learned my addr 172.21.15.144:0/734293271 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): get_auth_request con 0x7fed740a0ad0 auth_method 0 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:25.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed7409ee60 0x7fed74131010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 10 monclient(hunting): get_auth_request con 0x7fed7409ee60 auth_method 0 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient(hunting): handle_auth_done global_id 6076 payload 824 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 -- 172.21.15.144:0/734293271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed7409ee60 msgr2=0x7fed74131010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed7409ee60 0x7fed74131010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 -- 172.21.15.144:0/734293271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74131ad0 msgr2=0x7fed74141390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:25.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74131ad0 0x7fed74141390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 monclient: found mon.noname-b 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.842+0000 7fed71d9b640 1 -- 172.21.15.144:0/734293271 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68009450 con 0x7fed740a0ad0 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed71d9b640 1 -- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 msgr2=0x7fed74131550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed71d9b640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 0x7fed74131550 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fed68009970 tx=0x7fed68032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680329b0 con 0x7fed740a0ad0 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: dump: 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.858 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68032b30 con 0x7fed740a0ad0 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed78ec0640 5 monclient: authenticate success, global_id 6076 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6803d440 con 0x7fed740a0ad0 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed78ec0640 10 monclient: _renew_subs 2024-09-18T12:32:25.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed78ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed740a0460 con 0x7fed740a0ad0 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 10 monclient: dump: 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:25.860 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed70d99640 4 set_mon_vals no callback set 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed78ec0640 10 monclient: _renew_subs 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed78ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7411e1e0 con 0x7fed740a0ad0 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.845+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fed68039040 con 0x7fed740a0ad0 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed627fc640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fed54038440 0x7fed5403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(183..183 src has 13..183) v4 ==== 7553+0+0 (secure 0 0 0) 0x7fed6806be30 con 0x7fed740a0ad0 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed78ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed78ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed78ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7411f2e0 con 0x7fed740a0ad0 2024-09-18T12:32:25.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed7259c640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fed54038440 0x7fed5403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:25.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.846+0000 7fed7259c640 10 monclient: get_auth_request con 0x7fed54038440 auth_method 0 2024-09-18T12:32:25.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.848+0000 7fed7259c640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fed54038440 0x7fed5403a8f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fed6400acc0 tx=0x7fed64008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:25.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.848+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:32:25.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.850+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fed68042080 con 0x7fed740a0ad0 2024-09-18T12:32:25.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.850+0000 7fed627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:25.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:25.850+0000 7fed627fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:25.877 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:32:25.877 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:32:25.877 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:32:25.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:32:25.880 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:32:25.880 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:32:25.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.982+0000 7f91a2edd640 10 monclient: get_monmap_and_config 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.982+0000 7f91a2edd640 10 monclient: build_initial_monmap 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.982+0000 7f91a2edd640 10 monclient: monmap: 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:25.984565+0000 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:25.984565+0000 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:25.986 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.983+0000 7f91a2edd640 1 Processor -- start 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.983+0000 7f91a2edd640 1 -- start start 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.983+0000 7f91a2edd640 10 monclient: init 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f919c131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient(hunting): picked mon.noname-b con 0x7f919c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f919c09f220 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:25.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c09faf0 con 0x7f919c131ab0 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c09fd30 con 0x7f919c09f220 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c132590 con 0x7f919c09e910 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a2edd640 10 monclient(hunting): authenticate will time out at 1173.306460s 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a0c79640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f919c131ab0 0x7f919c131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45866/0 (socket says 172.21.15.116:45866) 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f91a0c79640 1 -- 172.21.15.116:0/2521419058 learned_addr learned my addr 172.21.15.116:0/2521419058 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): get_auth_request con 0x7f919c09e910 auth_method 0 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:25.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient(hunting): handle_auth_done global_id 64862 payload 274 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 1 -- 172.21.15.116:0/2521419058 >> v1:172.21.15.144:6789/0 conn(0x7f919c09f220 legacy=0x7f919c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 1 -- 172.21.15.116:0/2521419058 >> v1:172.21.15.116:6789/0 conn(0x7f919c131ab0 legacy=0x7f919c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:25.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.984+0000 7f9193fff640 1 -- 172.21.15.116:0/2521419058 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c0a1110 con 0x7f919c09e910 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.987+0000 7f9193fff640 1 -- 172.21.15.116:0/2521419058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 msgr2=0x7f919c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.987+0000 7f9193fff640 1 --2- 172.21.15.116:0/2521419058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c09ece0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f918c009880 tx=0x7f918c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d237465b0cfe4f6c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 1 -- 172.21.15.116:0/2521419058 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c032440 con 0x7f919c09e910 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: dump: 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 1 -- 172.21.15.116:0/2521419058 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c0325a0 con 0x7f919c09e910 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 1 -- 172.21.15.116:0/2521419058 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c032940 con 0x7f919c09e910 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91937fe640 10 monclient: dump: 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 5 monclient: authenticate success, global_id 64862 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 4 set_mon_vals no callback set 2024-09-18T12:32:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 10 monclient: shutdown 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 1 -- 172.21.15.116:0/2521419058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 msgr2=0x7f919c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 1 --2- 172.21.15.116:0/2521419058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c09ece0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f918c009880 tx=0x7f918c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 1 -- 172.21.15.116:0/2521419058 shutdown_connections 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 1 --2- 172.21.15.116:0/2521419058 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c09ece0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 1 -- 172.21.15.116:0/2521419058 >> 172.21.15.116:0/2521419058 conn(0x7f919c067d30 msgr2=0x7f919c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 1 -- 172.21.15.116:0/2521419058 shutdown_connections 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.988+0000 7f91a2edd640 1 -- 172.21.15.116:0/2521419058 wait complete. 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.989+0000 7f91a2edd640 10 monclient: build_initial_monmap 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.989+0000 7f91a2edd640 10 monclient: monmap: 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.989+0000 7f91a2edd640 1 Processor -- start 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.989+0000 7f91a2edd640 1 -- start start 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.989+0000 7f91a2edd640 10 monclient: init 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c1301c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): picked mon.noname-c con 0x7f919c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 0x7f919c130700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): picked mon.noname-a con 0x7f919c09f220 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f919c131ab0 0x7f919c130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): picked mon.noname-b con 0x7f919c131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c097910 con 0x7f919c09f220 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c098800 con 0x7f919c131ab0 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c09bde0 con 0x7f919c09e910 2024-09-18T12:32:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a2edd640 10 monclient(hunting): authenticate will time out at 1173.312256s 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 0x7f919c130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c1301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 0x7f919c130700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59594/0 (socket says 172.21.15.116:59594) 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 1 -- 172.21.15.116:0/1425810173 learned_addr learned my addr 172.21.15.116:0/1425810173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): get_auth_request con 0x7f919c09e910 auth_method 0 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 10 monclient(hunting): get_auth_request con 0x7f919c09f220 auth_method 0 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f919c131ab0 0x7f919c130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 10 monclient(hunting): get_auth_request con 0x7f919c131ab0 auth_method 0 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 1 -- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 msgr2=0x7f919c130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 1 -- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 msgr2=0x7f919c130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 0x7f919c130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f9193fff640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 0x7f919c130700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a147a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient(hunting): handle_auth_done global_id 64865 payload 824 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 1 -- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f919c131ab0 msgr2=0x7f919c130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f919c131ab0 0x7f919c130c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 1 -- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 msgr2=0x7f919c130700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 0x7f919c130700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient: _finish_hunting 0 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 1 monclient: found mon.noname-c 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.990+0000 7f91a0c79640 1 -- 172.21.15.116:0/1425810173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c009390 con 0x7f919c09e910 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.992+0000 7f91a0c79640 1 -- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 msgr2=0x7f919c1301c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.992+0000 7f91a0c79640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c1301c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f919800c4c0 tx=0x7f9198007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.992+0000 7f9190ff9640 1 -- 172.21.15.116:0/1425810173 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198005cb0 con 0x7f919c09e910 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: dump: 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 5 monclient: authenticate success, global_id 64865 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 1 -- 172.21.15.116:0/1425810173 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198005e30 con 0x7f919c09e910 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 10 monclient: _renew_subs 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c1397e0 con 0x7f919c09e910 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 1 -- 172.21.15.116:0/1425810173 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198007320 con 0x7f919c09e910 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 10 monclient: dump: 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:26.008 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 10 monclient: _renew_subs 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c099120 con 0x7f919c09e910 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9192ffd640 4 set_mon_vals no callback set 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f9190ff9640 1 -- 172.21.15.116:0/1425810173 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f919801b3b0 con 0x7f919c09e910 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.993+0000 7f91a2edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.994+0000 7f91a2edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.994+0000 7f91a2edd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.994+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c04bf60 con 0x7f919c09e910 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.994+0000 7f9190ff9640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91880384c0 0x7f918803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.994+0000 7f9190ff9640 1 -- 172.21.15.116:0/1425810173 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(183..183 src has 13..183) v4 ==== 7553+0+0 (secure 0 0 0) 0x7f919804b7e0 con 0x7f919c09e910 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.994+0000 7f9193fff640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91880384c0 0x7f918803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.994+0000 7f9193fff640 10 monclient: get_auth_request con 0x7f91880384c0 auth_method 0 2024-09-18T12:32:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.996+0000 7f9193fff640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91880384c0 0x7f918803a970 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f919c09bfc0 tx=0x7f918c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.998+0000 7f9190ff9640 1 -- 172.21.15.116:0/1425810173 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9198007710 con 0x7f919c09e910 2024-09-18T12:32:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.998+0000 7f9190ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:25.998+0000 7f9190ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:26.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.011+0000 7fed78ec0640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T12:32:26.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.011+0000 7fed78ec0640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T12:32:26.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.011+0000 7fed78ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:26.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.011+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fed740008d0 con 0x7fed740a0ad0 2024-09-18T12:32:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.216+0000 7f91a2edd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.216+0000 7f91a2edd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.216+0000 7f91a2edd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.216+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f919c066280 con 0x7f919c09e910 2024-09-18T12:32:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.216+0000 7f9190ff9640 1 -- 172.21.15.116:0/1425810173 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9198017610 con 0x7f919c09e910 2024-09-18T12:32:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.216+0000 7f9190ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.216+0000 7f9190ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:26.219 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":76,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91880384c0 msgr2=0x7f918803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91880384c0 0x7f918803a970 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f919c09bfc0 tx=0x7f918c032000 comp rx=0 tx=0).stop 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 10 monclient: shutdown 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 msgr2=0x7f919c1301c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c1301c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f919800c4c0 tx=0x7f9198007f00 comp rx=0 tx=0).stop 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 shutdown_connections 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f91880384c0 0x7f918803a970 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f919c131ab0 0x7f919c130c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f919c09f220 0x7f919c130700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 --2- 172.21.15.116:0/1425810173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f919c09e910 0x7f919c1301c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 >> 172.21.15.116:0/1425810173 conn(0x7f919c067d30 msgr2=0x7f919c0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 shutdown_connections 2024-09-18T12:32:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:26.218+0000 7f91a2edd640 1 -- 172.21.15.116:0/1425810173 wait complete. 2024-09-18T12:32:26.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.649+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v184) v1 ==== 173+0+0 (secure 0 0 0) 0x7fed68037020 con 0x7fed740a0ad0 2024-09-18T12:32:26.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.649+0000 7fed627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T12:32:26.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.649+0000 7fed627fc640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-18T12:32:26.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.747+0000 7fed78ec0640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T12:32:26.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.747+0000 7fed78ec0640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T12:32:26.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.747+0000 7fed78ec0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:26.749 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.747+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fed740008d0 con 0x7fed740a0ad0 2024-09-18T12:32:26.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.748+0000 7fed627fc640 1 -- 172.21.15.144:0/734293271 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v184) v1 ==== 183+0+0 (secure 0 0 0) 0x7fed68037070 con 0x7fed740a0ad0 2024-09-18T12:32:26.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.748+0000 7fed627fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T12:32:26.750 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.748+0000 7fed627fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-18T12:32:26.750 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-18T12:32:26.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fed54038440 msgr2=0x7fed5403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fed54038440 0x7fed5403a8f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fed6400acc0 tx=0x7fed64008040 comp rx=0 tx=0).stop 2024-09-18T12:32:26.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 10 monclient: shutdown 2024-09-18T12:32:26.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:26.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 msgr2=0x7fed74131550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 0x7fed74131550 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fed68009970 tx=0x7fed68032000 comp rx=0 tx=0).stop 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 shutdown_connections 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fed54038440 0x7fed5403a8f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed74131ad0 0x7fed74141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed740a0ad0 0x7fed74131550 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 --2- 172.21.15.144:0/734293271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fed7409ee60 0x7fed74131010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 >> 172.21.15.144:0/734293271 conn(0x7fed74071370 msgr2=0x7fed740a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 shutdown_connections 2024-09-18T12:32:26.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.750+0000 7fed78ec0640 1 -- 172.21.15.144:0/734293271 wait complete. 2024-09-18T12:32:26.759 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket barr rack 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.832+0000 7fbec5b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.832+0000 7fbec5b9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.832+0000 7fbec5b9d640 10 monclient: monmap: 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:26.833679+0000 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:26.833679+0000 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:26.834 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:26.835 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:26.835 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:26.835 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:26.835 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:26.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.833+0000 7fbec5b9d640 1 Processor -- start 2024-09-18T12:32:26.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.833+0000 7fbec5b9d640 1 -- start start 2024-09-18T12:32:26.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.833+0000 7fbec5b9d640 10 monclient: init 2024-09-18T12:32:26.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.833+0000 7fbec5b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.833+0000 7fbec5b9d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.833+0000 7fbec5b9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbec0098c70 0x7fbec0099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fbec0098c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbec0099580 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbec0099c70 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec009f680 con 0x7fbec0099580 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec009e8f0 con 0x7fbec0099c70 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec009f420 con 0x7fbec0098c70 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbec5b9d640 10 monclient(hunting): authenticate will time out at 1182.403072s 2024-09-18T12:32:26.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbec0098c70 0x7fbec0099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebffff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fbec0099c70 0x7fbec009e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34042/0 (socket says 172.21.15.144:34042) 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebffff640 1 -- 172.21.15.144:0/841215510 learned_addr learned my addr 172.21.15.144:0/841215510 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): get_auth_request con 0x7fbec0098c70 auth_method 0 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_done global_id 6082 payload 274 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 1 -- 172.21.15.144:0/841215510 >> v1:172.21.15.144:6789/0 conn(0x7fbec0099c70 legacy=0x7fbec009e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 1 -- 172.21.15.144:0/841215510 >> v1:172.21.15.116:6789/0 conn(0x7fbec0099580 legacy=0x7fbec009dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:26.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 1 monclient: found mon.noname-c 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.834+0000 7fbebf7fe640 1 -- 172.21.15.144:0/841215510 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec0068a20 con 0x7fbec0098c70 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebf7fe640 1 -- 172.21.15.144:0/841215510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 msgr2=0x7fbec0099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebf7fe640 1 --2- 172.21.15.144:0/841215510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0099040 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009880 tx=0x7fbeb00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e6753540983694d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 1 -- 172.21.15.144:0/841215510 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0032440 con 0x7fbec0098c70 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: dump: 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:26.851 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 1 -- 172.21.15.144:0/841215510 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb00325a0 con 0x7fbec0098c70 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 5 monclient: authenticate success, global_id 6082 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:26.852 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 1 -- 172.21.15.144:0/841215510 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0032940 con 0x7fbec0098c70 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 4 set_mon_vals no callback set 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbebe7fc640 10 monclient: dump: 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:26.853 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 10 monclient: shutdown 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 1 -- 172.21.15.144:0/841215510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 msgr2=0x7fbec0099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/841215510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0099040 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009880 tx=0x7fbeb00090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 1 -- 172.21.15.144:0/841215510 shutdown_connections 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/841215510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0099040 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.837+0000 7fbec5b9d640 1 -- 172.21.15.144:0/841215510 >> 172.21.15.144:0/841215510 conn(0x7fbec0070e90 msgr2=0x7fbec00a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.838+0000 7fbec5b9d640 1 -- 172.21.15.144:0/841215510 shutdown_connections 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.838+0000 7fbec5b9d640 1 -- 172.21.15.144:0/841215510 wait complete. 2024-09-18T12:32:26.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.838+0000 7fbec5b9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.838+0000 7fbec5b9d640 10 monclient: monmap: 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.838+0000 7fbec5b9d640 1 Processor -- start 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.838+0000 7fbec5b9d640 1 -- start start 2024-09-18T12:32:26.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.838+0000 7fbec5b9d640 10 monclient: init 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0133dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fbec0098c70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbec0099580 0x7fbec011db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fbec0099580 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 0x7fbec011e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fbec0099c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec0120ef0 con 0x7fbec0099c70 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:26.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec01214b0 con 0x7fbec0098c70 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec0141fd0 con 0x7fbec0099580 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbec5b9d640 10 monclient(hunting): authenticate will time out at 1182.408454s 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0133dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0133dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:49038/0 (socket says 172.21.15.144:49038) 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 -- 172.21.15.144:0/2437242251 learned_addr learned my addr 172.21.15.144:0/2437242251 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): get_auth_request con 0x7fbec0098c70 auth_method 0 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:26.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 0x7fbec011e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebeffd640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbec0099580 0x7fbec011db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 10 monclient(hunting): get_auth_request con 0x7fbec0099c70 auth_method 0 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebeffd640 10 monclient(hunting): get_auth_request con 0x7fbec0099580 auth_method 0 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 1 -- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 msgr2=0x7fbec011e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:32:26.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 1 -- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 msgr2=0x7fbec011e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient(hunting): handle_auth_done global_id 6085 payload 824 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 0x7fbec011e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebffff640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 0x7fbec011e090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 -- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbec0099580 msgr2=0x7fbec011db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbec0099580 0x7fbec011db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 -- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 msgr2=0x7fbec011e090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 0x7fbec011e090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 monclient: found mon.noname-b 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebf7fe640 1 -- 172.21.15.144:0/2437242251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb0009390 con 0x7fbec0098c70 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.839+0000 7fbebeffd640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbec0099580 0x7fbec011db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:26.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbebf7fe640 1 -- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 msgr2=0x7fbec0133dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbebf7fe640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0133dd0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009360 tx=0x7fbeb0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0031990 con 0x7fbec0098c70 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: dump: 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:26.860 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb0031af0 con 0x7fbec0098c70 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbec5b9d640 5 monclient: authenticate success, global_id 6085 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbec5b9d640 10 monclient: _renew_subs 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0031e70 con 0x7fbec0098c70 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbec5b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec0136680 con 0x7fbec0098c70 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:26.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbebdffb640 4 set_mon_vals no callback set 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbea77fe640 10 monclient: dump: 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbec5b9d640 10 monclient: _renew_subs 2024-09-18T12:32:26.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbec5b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.842+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec011ff90 con 0x7fbec0098c70 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbeb0039040 con 0x7fbec0098c70 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbec5b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbec5b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbec5b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec011f380 con 0x7fbec0098c70 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbea77fe640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbeac038490 0x7fbeac03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(184..184 src has 13..184) v4 ==== 7741+0+0 (secure 0 0 0) 0x7fbeb006c6e0 con 0x7fbec0098c70 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbebeffd640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbeac038490 0x7fbeac03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.843+0000 7fbebeffd640 10 monclient: get_auth_request con 0x7fbeac038490 auth_method 0 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.845+0000 7fbebeffd640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbeac038490 0x7fbeac03a940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4002860 tx=0x7fbeb4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.847+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbeb004bd70 con 0x7fbec0098c70 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.847+0000 7fbea77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:26.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:26.847+0000 7fbea77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:27.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.004+0000 7fbec5b9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T12:32:27.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.004+0000 7fbec5b9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T12:32:27.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.004+0000 7fbec5b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:27.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.004+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7fbec011f380 con 0x7fbec0098c70 2024-09-18T12:32:27.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.659+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v185) v1 ==== 140+0+0 (secure 0 0 0) 0x7fbeb00490a0 con 0x7fbec0098c70 2024-09-18T12:32:27.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.659+0000 7fbea77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T12:32:27.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.659+0000 7fbea77fe640 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.755+0000 7fbec5b9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.755+0000 7fbec5b9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.755+0000 7fbec5b9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.755+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7fbec011f380 con 0x7fbec0098c70 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.756+0000 7fbea77fe640 1 -- 172.21.15.144:0/2437242251 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v185) v1 ==== 128+0+0 (secure 0 0 0) 0x7fbeb003f020 con 0x7fbec0098c70 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.756+0000 7fbea77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.756+0000 7fbea77fe640 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-18T12:32:27.757 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'barr' already exists 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.757+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbeac038490 msgr2=0x7fbeac03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.757+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbeac038490 0x7fbeac03a940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4002860 tx=0x7fbeb4009040 comp rx=0 tx=0).stop 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.757+0000 7fbec5b9d640 10 monclient: shutdown 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.757+0000 7fbec5b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.757+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 msgr2=0x7fbec0133dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.757+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0133dd0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009360 tx=0x7fbeb0032000 comp rx=0 tx=0).stop 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.757+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 shutdown_connections 2024-09-18T12:32:27.759 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.758+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbeac038490 0x7fbeac03a940 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.758+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec0099c70 0x7fbec011e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.758+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbec0099580 0x7fbec011db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.758+0000 7fbec5b9d640 1 --2- 172.21.15.144:0/2437242251 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbec0098c70 0x7fbec0133dd0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.758+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 >> 172.21.15.144:0/2437242251 conn(0x7fbec0070e90 msgr2=0x7fbec009d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.758+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 shutdown_connections 2024-09-18T12:32:27.760 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.758+0000 7fbec5b9d640 1 -- 172.21.15.144:0/2437242251 wait complete. 2024-09-18T12:32:27.767 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush move barr root=default 2024-09-18T12:32:27.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.841+0000 7fd05aa99640 10 monclient: get_monmap_and_config 2024-09-18T12:32:27.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.841+0000 7fd05aa99640 10 monclient: build_initial_monmap 2024-09-18T12:32:27.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.841+0000 7fd05aa99640 10 monclient: monmap: 2024-09-18T12:32:27.842 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:27.842436+0000 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:27.842436+0000 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:27.843 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:27.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 1 Processor -- start 2024-09-18T12:32:27.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 1 -- start start 2024-09-18T12:32:27.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient: init 2024-09-18T12:32:27.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054098c70 0x7fd054099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient(hunting): picked mon.noname-b con 0x7fd054098c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd054099580 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd054099c70 0x7fd05409e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient(hunting): picked mon.noname-c con 0x7fd054099c70 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05409f680 con 0x7fd054099580 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05409f420 con 0x7fd054099c70 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd054068a20 con 0x7fd054098c70 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd05aa99640 10 monclient(hunting): authenticate will time out at 1183.411846s 2024-09-18T12:32:27.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.842+0000 7fd059036640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd054099c70 0x7fd05409e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd054099c70 0x7fd05409e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42914/0 (socket says 172.21.15.144:42914) 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 1 -- 172.21.15.144:0/718435213 learned_addr learned my addr 172.21.15.144:0/718435213 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): get_auth_request con 0x7fd054099c70 auth_method 0 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd058835640 1 --2- 172.21.15.144:0/718435213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054098c70 0x7fd054099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd058835640 10 monclient(hunting): get_auth_request con 0x7fd054098c70 auth_method 0 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd058835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:27.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd058835640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd058835640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient(hunting): handle_auth_done global_id 6088 payload 274 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 1 -- 172.21.15.144:0/718435213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054098c70 msgr2=0x7fd054099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 1 --2- 172.21.15.144:0/718435213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054098c70 0x7fd054099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 1 -- 172.21.15.144:0/718435213 >> v1:172.21.15.116:6789/0 conn(0x7fd054099580 legacy=0x7fd05409dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient: _finish_hunting 0 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 1 monclient: found mon.noname-c 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:27.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.843+0000 7fd059036640 1 -- 172.21.15.144:0/718435213 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054096a30 con 0x7fd054099c70 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.845+0000 7fd059036640 1 -- 172.21.15.144:0/718435213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099c70 msgr2=0x7fd05409e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.845+0000 7fd059036640 1 --2- 172.21.15.144:0/718435213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099c70 0x7fd05409e1e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009f20 tx=0x7fd04c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=203be116f57e385d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 1 -- 172.21.15.144:0/718435213 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c007610 con 0x7fd054099c70 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: dump: 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:27.854 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 1 -- 172.21.15.144:0/718435213 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c02d050 con 0x7fd054099c70 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 5 monclient: authenticate success, global_id 6088 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 1 -- 172.21.15.144:0/718435213 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c0079a0 con 0x7fd054099c70 2024-09-18T12:32:27.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 4 set_mon_vals no callback set 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd0537fe640 10 monclient: dump: 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:27.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 10 monclient: shutdown 2024-09-18T12:32:27.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:27.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 -- 172.21.15.144:0/718435213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099c70 msgr2=0x7fd05409e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:27.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 --2- 172.21.15.144:0/718435213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099c70 0x7fd05409e1e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009f20 tx=0x7fd04c009840 comp rx=0 tx=0).stop 2024-09-18T12:32:27.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 -- 172.21.15.144:0/718435213 shutdown_connections 2024-09-18T12:32:27.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 --2- 172.21.15.144:0/718435213 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099c70 0x7fd05409e1e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 --2- 172.21.15.144:0/718435213 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054098c70 0x7fd054099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 -- 172.21.15.144:0/718435213 >> 172.21.15.144:0/718435213 conn(0x7fd054070e90 msgr2=0x7fd0540a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:27.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 -- 172.21.15.144:0/718435213 shutdown_connections 2024-09-18T12:32:27.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.846+0000 7fd05aa99640 1 -- 172.21.15.144:0/718435213 wait complete. 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.847+0000 7fd05aa99640 10 monclient: build_initial_monmap 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.847+0000 7fd05aa99640 10 monclient: monmap: 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.847+0000 7fd05aa99640 1 Processor -- start 2024-09-18T12:32:27.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.847+0000 7fd05aa99640 1 -- start start 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.847+0000 7fd05aa99640 10 monclient: init 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 0x7fd05413d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): picked mon.noname-a con 0x7fd054098c70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 0x7fd05413cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): picked mon.noname-b con 0x7fd054099580 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054099c70 0x7fd054140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): picked mon.noname-c con 0x7fd054099c70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0541210d0 con 0x7fd054098c70 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd054122080 con 0x7fd054099580 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:27.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd054122280 con 0x7fd054099c70 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd05aa99640 10 monclient(hunting): authenticate will time out at 1183.417145s 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 0x7fd05413cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 0x7fd05413cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42930/0 (socket says 172.21.15.144:42930) 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 -- 172.21.15.144:0/1754699684 learned_addr learned my addr 172.21.15.144:0/1754699684 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): get_auth_request con 0x7fd054099580 auth_method 0 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 0x7fd05413d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd059036640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054099c70 0x7fd054140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:27.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 10 monclient(hunting): get_auth_request con 0x7fd054098c70 auth_method 0 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd059036640 10 monclient(hunting): get_auth_request con 0x7fd054099c70 auth_method 0 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd059036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd059036640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd059036640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient(hunting): handle_auth_done global_id 6091 payload 824 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 -- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054099c70 msgr2=0x7fd054140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054099c70 0x7fd054140dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 1 -- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 msgr2=0x7fd05413d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:32:27.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 1 -- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 msgr2=0x7fd05413d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 -- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 msgr2=0x7fd05413d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 0x7fd05413d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd058835640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 0x7fd05413d850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 0x7fd05413d850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.848+0000 7fd053fff640 1 -- 172.21.15.144:0/1754699684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c009ae0 con 0x7fd054099580 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd053fff640 1 -- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 msgr2=0x7fd05413cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd053fff640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 0x7fd05413cf00 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c5e0 tx=0x7fd040007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040012840 con 0x7fd054099580 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:27.865 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: dump: 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0400129a0 con 0x7fd054099580 2024-09-18T12:32:27.866 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd05aa99640 5 monclient: authenticate success, global_id 6091 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd05aa99640 10 monclient: _renew_subs 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd05aa99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd054122840 con 0x7fd054099580 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040012da0 con 0x7fd054099580 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd050ff9640 10 monclient: dump: 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:27.867 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd05aa99640 10 monclient: _renew_subs 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.850+0000 7fd052ffd640 4 set_mon_vals no callback set 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd05aa99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05413e350 con 0x7fd054099580 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd040011590 con 0x7fd054099580 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd05aa99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd05aa99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd05aa99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.851+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0540008d0 con 0x7fd054099580 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.852+0000 7fd050ff9640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd038038490 0x7fd03803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.852+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(185..185 src has 13..185) v4 ==== 7769+0+0 (secure 0 0 0) 0x7fd04004c570 con 0x7fd054099580 2024-09-18T12:32:27.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.852+0000 7fd058835640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd038038490 0x7fd03803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:27.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.852+0000 7fd058835640 10 monclient: get_auth_request con 0x7fd038038490 auth_method 0 2024-09-18T12:32:27.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.854+0000 7fd058835640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd038038490 0x7fd03803a940 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd048006fd0 tx=0x7fd048008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:27.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.856+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd040010ac0 con 0x7fd054099580 2024-09-18T12:32:27.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.856+0000 7fd050ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:27.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:27.856+0000 7fd050ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:28.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.016+0000 7fd05aa99640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T12:32:28.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.016+0000 7fd05aa99640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T12:32:28.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.016+0000 7fd05aa99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:28.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.016+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7fd0541225c0 con 0x7fd054099580 2024-09-18T12:32:28.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.669+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v186) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd040016070 con 0x7fd054099580 2024-09-18T12:32:28.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.669+0000 7fd050ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T12:32:28.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.669+0000 7fd050ff9640 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-18T12:32:28.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.769+0000 7fd05aa99640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T12:32:28.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.769+0000 7fd05aa99640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T12:32:28.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.769+0000 7fd05aa99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:28.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.769+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7fd054072640 con 0x7fd054099580 2024-09-18T12:32:28.770 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.769+0000 7fd050ff9640 1 -- 172.21.15.144:0/1754699684 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v186) v1 ==== 183+0+0 (secure 0 0 0) 0x7fd0400102e0 con 0x7fd054099580 2024-09-18T12:32:28.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.769+0000 7fd050ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T12:32:28.771 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.769+0000 7fd050ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-18T12:32:28.771 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-18T12:32:28.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd038038490 msgr2=0x7fd03803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:28.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd038038490 0x7fd03803a940 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd048006fd0 tx=0x7fd048008040 comp rx=0 tx=0).stop 2024-09-18T12:32:28.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 10 monclient: shutdown 2024-09-18T12:32:28.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:28.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 msgr2=0x7fd05413cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:28.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 0x7fd05413cf00 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c5e0 tx=0x7fd040007ed0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 shutdown_connections 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd038038490 0x7fd03803a940 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd054099c70 0x7fd054140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd054099580 0x7fd05413cf00 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 --2- 172.21.15.144:0/1754699684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd054098c70 0x7fd05413d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 >> 172.21.15.144:0/1754699684 conn(0x7fd054070e90 msgr2=0x7fd05406d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 shutdown_connections 2024-09-18T12:32:28.773 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.771+0000 7fd05aa99640 1 -- 172.21.15.144:0/1754699684 wait complete. 2024-09-18T12:32:28.781 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush move fooo rack=barr 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.854+0000 7f4dae75d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.854+0000 7f4dae75d640 10 monclient: build_initial_monmap 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.854+0000 7f4dae75d640 10 monclient: monmap: 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:28.855862+0000 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:28.855862+0000 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:28.856 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:28.857 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:28.857 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:28.857 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:28.857 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:28.857 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.855+0000 7f4dae75d640 1 Processor -- start 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.855+0000 7f4dae75d640 1 -- start start 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.855+0000 7f4dae75d640 10 monclient: init 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient(hunting): picked mon.noname-b con 0x7f4da809f320 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4da8070920 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4da8071400 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da8096b60 con 0x7f4da8071400 2024-09-18T12:32:28.859 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da809e4d0 con 0x7f4da8070920 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da809e7b0 con 0x7f4da809f320 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4dae75d640 10 monclient(hunting): authenticate will time out at 1184.425307s 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da77fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f4da8070920 0x7f4da8070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:37350/0 (socket says 172.21.15.144:37350) 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da77fe640 1 -- 172.21.15.144:0/3074253501 learned_addr learned my addr 172.21.15.144:0/3074253501 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 1 --2- 172.21.15.144:0/3074253501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): get_auth_request con 0x7f4da809f320 auth_method 0 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:28.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient(hunting): handle_auth_done global_id 64877 payload 274 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 1 -- 172.21.15.144:0/3074253501 >> v1:172.21.15.144:6789/0 conn(0x7f4da8070920 legacy=0x7f4da8070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 1 -- 172.21.15.144:0/3074253501 >> v1:172.21.15.116:6789/0 conn(0x7f4da8071400 legacy=0x7f4da8138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:28.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.856+0000 7f4da7fff640 1 -- 172.21.15.144:0/3074253501 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da809e1f0 con 0x7f4da809f320 2024-09-18T12:32:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.859+0000 7f4da7fff640 1 -- 172.21.15.144:0/3074253501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 msgr2=0x7f4da81321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.859+0000 7f4da7fff640 1 --2- 172.21.15.144:0/3074253501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81321b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009880 tx=0x7f4d980090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4db46b99c9383ed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 1 -- 172.21.15.144:0/3074253501 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9802aea0 con 0x7f4da809f320 2024-09-18T12:32:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:28.867 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: dump: 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:28.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 5 monclient: authenticate success, global_id 64877 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 1 -- 172.21.15.144:0/3074253501 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d98032440 con 0x7f4da809f320 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 4 set_mon_vals no callback set 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 1 -- 172.21.15.144:0/3074253501 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d980327e0 con 0x7f4da809f320 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4da6ffd640 10 monclient: dump: 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:28.869 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 10 monclient: shutdown 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 1 -- 172.21.15.144:0/3074253501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 msgr2=0x7f4da81321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 1 --2- 172.21.15.144:0/3074253501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81321b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009880 tx=0x7f4d980090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 1 -- 172.21.15.144:0/3074253501 shutdown_connections 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 1 --2- 172.21.15.144:0/3074253501 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81321b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.870 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 1 -- 172.21.15.144:0/3074253501 >> 172.21.15.144:0/3074253501 conn(0x7f4da80a2b70 msgr2=0x7f4da809ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 1 -- 172.21.15.144:0/3074253501 shutdown_connections 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.860+0000 7f4dae75d640 1 -- 172.21.15.144:0/3074253501 wait complete. 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.861+0000 7f4dae75d640 10 monclient: build_initial_monmap 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.861+0000 7f4dae75d640 10 monclient: monmap: 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:28.871 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.861+0000 7f4dae75d640 1 Processor -- start 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.861+0000 7f4dae75d640 1 -- start start 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.861+0000 7f4dae75d640 10 monclient: init 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 0x7f4da8136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): picked mon.noname-b con 0x7f4da8070920 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 0x7f4da8136d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): picked mon.noname-a con 0x7f4da8071400 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81372b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:28.872 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): picked mon.noname-c con 0x7f4da809f320 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da809d620 con 0x7f4da8071400 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da8118f10 con 0x7f4da8070920 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da8139000 con 0x7f4da809f320 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4dae75d640 10 monclient(hunting): authenticate will time out at 1184.431279s 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 0x7f4da8136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 0x7f4da8136830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42938/0 (socket says 172.21.15.144:42938) 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 -- 172.21.15.144:0/2335326895 learned_addr learned my addr 172.21.15.144:0/2335326895 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): get_auth_request con 0x7f4da8070920 auth_method 0 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:28.873 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 0x7f4da8136d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 10 monclient(hunting): get_auth_request con 0x7f4da8071400 auth_method 0 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 10 monclient(hunting): get_auth_request con 0x7f4da809f320 auth_method 0 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:28.874 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 1 -- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 msgr2=0x7f4da8136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 1 -- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 msgr2=0x7f4da8136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 0x7f4da8136d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da77fe640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 0x7f4da8136d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4daccfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient(hunting): handle_auth_done global_id 6094 payload 824 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 -- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 msgr2=0x7f4da81372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81372b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 -- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 msgr2=0x7f4da8136d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 0x7f4da8136d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:28.875 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.862+0000 7f4da7fff640 1 -- 172.21.15.144:0/2335326895 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d98009390 con 0x7f4da8070920 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.864+0000 7f4da7fff640 1 -- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 msgr2=0x7f4da8136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.864+0000 7f4da7fff640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 0x7f4da8136830 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009880 tx=0x7f4d98032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9802aea0 con 0x7f4da8070920 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: dump: 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:28.876 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: _finish_auth 0 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4dae75d640 5 monclient: authenticate success, global_id 6094 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4dae75d640 10 monclient: _renew_subs 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4dae75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:28.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da8118c30 con 0x7f4da8070920 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d98031990 con 0x7f4da8070920 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98031d10 con 0x7f4da8070920 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4d8ffff640 10 monclient: dump: 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:28.878 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4dae75d640 10 monclient: _renew_subs 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4da67fc640 4 set_mon_vals no callback set 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4dae75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.865+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da8130ae0 con 0x7f4da8070920 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4d98039070 con 0x7f4da8070920 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4dae75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4dae75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4dae75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da8138250 con 0x7f4da8070920 2024-09-18T12:32:28.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4d8ffff640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d94038490 0x7f4d9403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:28.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(186..186 src has 13..186) v4 ==== 7957+0+0 (secure 0 0 0) 0x7f4d9806c810 con 0x7f4da8070920 2024-09-18T12:32:28.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4da77fe640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d94038490 0x7f4d9403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:28.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.866+0000 7f4da77fe640 10 monclient: get_auth_request con 0x7f4d94038490 auth_method 0 2024-09-18T12:32:28.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.868+0000 7f4da77fe640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d94038490 0x7f4d9403a940 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4da8137fc0 tx=0x7f4d9c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:28.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.870+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4da8138250 con 0x7f4da8070920 2024-09-18T12:32:28.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.870+0000 7f4d8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:28.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:28.870+0000 7f4d8ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:29.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.029+0000 7f4dae75d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T12:32:29.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.029+0000 7f4dae75d640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T12:32:29.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.029+0000 7f4dae75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:29.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.029+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7f4da8130ae0 con 0x7f4da8070920 2024-09-18T12:32:29.231 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.333+0000 7fa79e894640 10 monclient: get_monmap_and_config 2024-09-18T12:32:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.333+0000 7fa79e894640 10 monclient: build_initial_monmap 2024-09-18T12:32:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.333+0000 7fa79e894640 10 monclient: monmap: 2024-09-18T12:32:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:29.335212+0000 2024-09-18T12:32:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:29.335212+0000 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:29.336 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.334+0000 7fa79e894640 1 Processor -- start 2024-09-18T12:32:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.334+0000 7fa79e894640 1 -- start start 2024-09-18T12:32:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.334+0000 7fa79e894640 10 monclient: init 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.334+0000 7fa79e894640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.334+0000 7fa79e894640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.334+0000 7fa79e894640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 0x7fa79809a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.334+0000 7fa79e894640 10 monclient(hunting): picked mon.noname-b con 0x7fa798096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa79809aaa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 0x7fa798135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 10 monclient(hunting): picked mon.noname-a con 0x7fa7980978f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa79809d4a0 con 0x7fa79809aaa0 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa798099620 con 0x7fa7980978f0 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa798099340 con 0x7fa798096e10 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79e894640 10 monclient(hunting): authenticate will time out at 1176.657068s 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 0x7fa798135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 0x7fa79809a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 0x7fa798135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59600/0 (socket says 172.21.15.116:59600) 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 0x7fa79809a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46092/0 (socket says 172.21.15.116:46092) 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 1 -- 172.21.15.116:0/3797573592 learned_addr learned my addr 172.21.15.116:0/3797573592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 10 monclient(hunting): get_auth_request con 0x7fa7980978f0 auth_method 0 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): get_auth_request con 0x7fa798096e10 auth_method 0 2024-09-18T12:32:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 1 -- 172.21.15.116:0/3797573592 >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 msgr2=0x7fa798135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 1 -- 172.21.15.116:0/3797573592 >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 msgr2=0x7fa798135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 1 --2- 172.21.15.116:0/3797573592 >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 0x7fa798135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa79ce31640 1 --2- 172.21.15.116:0/3797573592 >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 0x7fa798135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient(hunting): handle_auth_done global_id 64883 payload 274 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 1 -- 172.21.15.116:0/3797573592 >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 msgr2=0x7fa798135b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 1 --2- 172.21.15.116:0/3797573592 >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 0x7fa798135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 1 -- 172.21.15.116:0/3797573592 >> v1:172.21.15.144:6789/0 conn(0x7fa79809aaa0 legacy=0x7fa7980971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.335+0000 7fa797fff640 1 -- 172.21.15.116:0/3797573592 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798099900 con 0x7fa798096e10 2024-09-18T12:32:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa797fff640 1 -- 172.21.15.116:0/3797573592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 msgr2=0x7fa79809a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa797fff640 1 --2- 172.21.15.116:0/3797573592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 0x7fa79809a560 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa788009850 tx=0x7fa7880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62f528f30ed3dcfe server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 1 -- 172.21.15.116:0/3797573592 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78802add0 con 0x7fa798096e10 2024-09-18T12:32:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: dump: 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa79e894640 5 monclient: authenticate success, global_id 64883 2024-09-18T12:32:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 1 -- 172.21.15.116:0/3797573592 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa788032440 con 0x7fa798096e10 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa79e894640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa79e894640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa79e894640 4 set_mon_vals no callback set 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 1 -- 172.21.15.116:0/3797573592 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880327e0 con 0x7fa798096e10 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.338+0000 7fa7977fe640 10 monclient: dump: 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 10 monclient: shutdown 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 -- 172.21.15.116:0/3797573592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 msgr2=0x7fa79809a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 --2- 172.21.15.116:0/3797573592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 0x7fa79809a560 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa788009850 tx=0x7fa7880090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 -- 172.21.15.116:0/3797573592 shutdown_connections 2024-09-18T12:32:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 --2- 172.21.15.116:0/3797573592 >> v2:172.21.15.116:3300/0 conn(0x7fa7980978f0 0x7fa798135b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 --2- 172.21.15.116:0/3797573592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa798096e10 0x7fa79809a560 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 -- 172.21.15.116:0/3797573592 >> 172.21.15.116:0/3797573592 conn(0x7fa7980672b0 msgr2=0x7fa7980676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 -- 172.21.15.116:0/3797573592 shutdown_connections 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 -- 172.21.15.116:0/3797573592 wait complete. 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 10 monclient: build_initial_monmap 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 10 monclient: monmap: 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 Processor -- start 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 1 -- start start 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.339+0000 7fa79e894640 10 monclient: init 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 0x7fa79813cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): picked mon.noname-a con 0x7fa798096e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7980978f0 0x7fa79812e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): picked mon.noname-c con 0x7fa7980978f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa79809aaa0 0x7fa798118400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): picked mon.noname-b con 0x7fa79809aaa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79811bca0 con 0x7fa798096e10 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79813dc60 con 0x7fa79809aaa0 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa798119f00 con 0x7fa7980978f0 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79e894640 10 monclient(hunting): authenticate will time out at 1176.662398s 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 0x7fa79813cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7980978f0 0x7fa79812e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 0x7fa79813cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59616/0 (socket says 172.21.15.116:59616) 2024-09-18T12:32:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 1 -- 172.21.15.116:0/4217733363 learned_addr learned my addr 172.21.15.116:0/4217733363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): get_auth_request con 0x7fa7980978f0 auth_method 0 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 10 monclient(hunting): get_auth_request con 0x7fa798096e10 auth_method 0 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 1 -- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 msgr2=0x7fa79813cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 1 -- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 msgr2=0x7fa79813cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa79809aaa0 0x7fa798118400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 0x7fa79813cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 10 monclient(hunting): get_auth_request con 0x7fa79809aaa0 auth_method 0 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa797fff640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 0x7fa79813cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:29.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa79ce31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:29.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient(hunting): handle_auth_done global_id 64886 payload 824 2024-09-18T12:32:29.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 1 -- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa79809aaa0 msgr2=0x7fa798118400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:29.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa79809aaa0 0x7fa798118400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 1 -- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 msgr2=0x7fa79813cdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:29.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 0x7fa79813cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 1 monclient: found mon.noname-c 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.340+0000 7fa78ffff640 1 -- 172.21.15.116:0/4217733363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788009390 con 0x7fa7980978f0 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa78ffff640 1 -- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7980978f0 msgr2=0x7fa79812e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa78ffff640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7980978f0 0x7fa79812e8a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa78000c4e0 tx=0x7fa780007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 1 -- 172.21.15.116:0/4217733363 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800117a0 con 0x7fa7980978f0 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: dump: 2024-09-18T12:32:29.360 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 1 -- 172.21.15.116:0/4217733363 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa780011920 con 0x7fa7980978f0 2024-09-18T12:32:29.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa79e894640 5 monclient: authenticate success, global_id 64886 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa79e894640 10 monclient: _renew_subs 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa79e894640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79811b9c0 con 0x7fa7980978f0 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 1 -- 172.21.15.116:0/4217733363 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780011ce0 con 0x7fa7980978f0 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa794ff9640 10 monclient: dump: 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:29.362 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa796ffd640 4 set_mon_vals no callback set 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa79e894640 10 monclient: _renew_subs 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa79e894640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.343+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79811aad0 con 0x7fa7980978f0 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa794ff9640 1 -- 172.21.15.116:0/4217733363 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa780012630 con 0x7fa7980978f0 2024-09-18T12:32:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa79e894640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa79e894640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa79e894640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79811c0a0 con 0x7fa7980978f0 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa794ff9640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa778038490 0x7fa77803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa794ff9640 1 -- 172.21.15.116:0/4217733363 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(186..186 src has 13..186) v4 ==== 7957+0+0 (secure 0 0 0) 0x7fa78004c030 con 0x7fa7980978f0 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa797fff640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa778038490 0x7fa77803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.344+0000 7fa797fff640 10 monclient: get_auth_request con 0x7fa778038490 auth_method 0 2024-09-18T12:32:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.347+0000 7fa797fff640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa778038490 0x7fa77803a940 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa788000e10 tx=0x7fa788032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.348+0000 7fa794ff9640 1 -- 172.21.15.116:0/4217733363 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa78007e030 con 0x7fa7980978f0 2024-09-18T12:32:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.348+0000 7fa794ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.348+0000 7fa794ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:29.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.563+0000 7fa79e894640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.563+0000 7fa79e894640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.563+0000 7fa79e894640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.563+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa79811ad50 con 0x7fa7980978f0 2024-09-18T12:32:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.564+0000 7fa794ff9640 1 -- 172.21.15.116:0/4217733363 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa780010d50 con 0x7fa7980978f0 2024-09-18T12:32:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.564+0000 7fa794ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:29.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.564+0000 7fa794ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:29.566 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":76,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa778038490 msgr2=0x7fa77803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa778038490 0x7fa77803a940 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa788000e10 tx=0x7fa788032000 comp rx=0 tx=0).stop 2024-09-18T12:32:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 10 monclient: shutdown 2024-09-18T12:32:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7980978f0 msgr2=0x7fa79812e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7980978f0 0x7fa79812e8a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa78000c4e0 tx=0x7fa780007e10 comp rx=0 tx=0).stop 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 shutdown_connections 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa778038490 0x7fa77803a940 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa79809aaa0 0x7fa798118400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7980978f0 0x7fa79812e8a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 --2- 172.21.15.116:0/4217733363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798096e10 0x7fa79813cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 >> 172.21.15.116:0/4217733363 conn(0x7fa7980672b0 msgr2=0x7fa79809ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 shutdown_connections 2024-09-18T12:32:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:29.566+0000 7fa79e894640 1 -- 172.21.15.116:0/4217733363 wait complete. 2024-09-18T12:32:29.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.814+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v187) v1 ==== 167+0+0 (secure 0 0 0) 0x7f4da8130ae0 con 0x7f4da8070920 2024-09-18T12:32:29.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.814+0000 7f4d8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T12:32:29.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.814+0000 7f4d8ffff640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.913+0000 7f4dae75d640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.913+0000 7f4dae75d640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.913+0000 7f4dae75d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.913+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7f4da8130ae0 con 0x7f4da8070920 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.914+0000 7f4d8ffff640 1 -- 172.21.15.144:0/2335326895 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v187) v1 ==== 177+0+0 (secure 0 0 0) 0x7f4da8130ae0 con 0x7f4da8070920 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.914+0000 7f4d8ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.914+0000 7f4d8ffff640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-18T12:32:29.915 INFO:tasks.workunit.client.0.smithi144.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-18T12:32:29.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d94038490 msgr2=0x7f4d9403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:29.917 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d94038490 0x7f4d9403a940 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4da8137fc0 tx=0x7f4d9c009040 comp rx=0 tx=0).stop 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 10 monclient: shutdown 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 msgr2=0x7f4da8136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 0x7f4da8136830 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009880 tx=0x7f4d98032000 comp rx=0 tx=0).stop 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 shutdown_connections 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d94038490 0x7f4d9403a940 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4da809f320 0x7f4da81372b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4da8071400 0x7f4da8136d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 --2- 172.21.15.144:0/2335326895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4da8070920 0x7f4da8136830 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 >> 172.21.15.144:0/2335326895 conn(0x7f4da80a2b70 msgr2=0x7f4da8131410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 shutdown_connections 2024-09-18T12:32:29.918 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:29.916+0000 7f4dae75d640 1 -- 172.21.15.144:0/2335326895 wait complete. 2024-09-18T12:32:29.926 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm fooo 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.000+0000 7f012d880640 10 monclient: get_monmap_and_config 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.000+0000 7f012d880640 10 monclient: build_initial_monmap 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.000+0000 7f012d880640 10 monclient: monmap: 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:30.000994+0000 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:30.000994+0000 2024-09-18T12:32:30.001 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:30.002 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:30.002 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:30.002 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:30.002 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:30.002 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:30.002 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:30.002 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:30.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.000+0000 7f012d880640 1 Processor -- start 2024-09-18T12:32:30.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.000+0000 7f012d880640 1 -- start start 2024-09-18T12:32:30.004 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.000+0000 7f012d880640 10 monclient: init 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f0128131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient(hunting): picked mon.noname-b con 0x7f012809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01281321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f012809f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0128067420 con 0x7f01281321a0 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01280a0480 con 0x7f012809f020 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01280a0200 con 0x7f012809ec50 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f012d880640 10 monclient(hunting): authenticate will time out at 1185.570293s 2024-09-18T12:32:30.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f01277fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f012809f020 0x7f0128069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:37356/0 (socket says 172.21.15.144:37356) 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f01277fe640 1 -- 172.21.15.144:0/132171420 learned_addr learned my addr 172.21.15.144:0/132171420 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 1 --2- 172.21.15.144:0/132171420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f0128131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): get_auth_request con 0x7f012809ec50 auth_method 0 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 10 monclient(hunting): handle_auth_done global_id 64889 payload 274 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.001+0000 7f0126ffd640 1 -- 172.21.15.144:0/132171420 >> v1:172.21.15.144:6789/0 conn(0x7f012809f020 legacy=0x7f0128069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.002+0000 7f0126ffd640 1 -- 172.21.15.144:0/132171420 >> v1:172.21.15.116:6789/0 conn(0x7f01281321a0 legacy=0x7f0128068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.002+0000 7f0126ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:30.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.002+0000 7f0126ffd640 1 monclient: found mon.noname-b 2024-09-18T12:32:30.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.002+0000 7f0126ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:30.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.002+0000 7f0126ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:30.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.002+0000 7f0126ffd640 1 -- 172.21.15.144:0/132171420 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012809fd80 con 0x7f012809ec50 2024-09-18T12:32:30.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.004+0000 7f0126ffd640 1 -- 172.21.15.144:0/132171420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 msgr2=0x7f0128131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:30.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.004+0000 7f0126ffd640 1 --2- 172.21.15.144:0/132171420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f0128131c60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0118009880 tx=0x7f01180090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7229a57e4ff129a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:30.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 1 -- 172.21.15.144:0/132171420 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118032440 con 0x7f012809ec50 2024-09-18T12:32:30.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:30.014 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: dump: 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:30.015 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 1 -- 172.21.15.144:0/132171420 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01180325a0 con 0x7f012809ec50 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 1 -- 172.21.15.144:0/132171420 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118032940 con 0x7f012809ec50 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 5 monclient: authenticate success, global_id 64889 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 4 set_mon_vals no callback set 2024-09-18T12:32:30.016 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f01267fc640 10 monclient: dump: 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:30.017 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 10 monclient: shutdown 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 1 -- 172.21.15.144:0/132171420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 msgr2=0x7f0128131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 1 --2- 172.21.15.144:0/132171420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f0128131c60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0118009880 tx=0x7f01180090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 1 -- 172.21.15.144:0/132171420 shutdown_connections 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 1 --2- 172.21.15.144:0/132171420 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f0128131c60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 1 -- 172.21.15.144:0/132171420 >> 172.21.15.144:0/132171420 conn(0x7f01280a5cf0 msgr2=0x7f01280a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 1 -- 172.21.15.144:0/132171420 shutdown_connections 2024-09-18T12:32:30.018 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.005+0000 7f012d880640 1 -- 172.21.15.144:0/132171420 wait complete. 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient: build_initial_monmap 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient: monmap: 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:30.019 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 Processor -- start 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 -- start start 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient: init 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f012809ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): picked mon.noname-c con 0x7f012809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 0x7f0128130d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): picked mon.noname-a con 0x7f012809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01281321a0 0x7f012811b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:30.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): picked mon.noname-b con 0x7f01281321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012811c900 con 0x7f012809f020 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012809dd90 con 0x7f01281321a0 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012809c020 con 0x7f012809ec50 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.006+0000 7f012d880640 10 monclient(hunting): authenticate will time out at 1185.575911s 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01281321a0 0x7f012811b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:30.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f012809ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f012809ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:36406/0 (socket says 172.21.15.144:36406) 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01281321a0 0x7f012811b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42944/0 (socket says 172.21.15.144:42944) 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 -- 172.21.15.144:0/1847787548 learned_addr learned my addr 172.21.15.144:0/1847787548 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): get_auth_request con 0x7f012809ec50 auth_method 0 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 10 monclient(hunting): get_auth_request con 0x7f01281321a0 auth_method 0 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:30.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 0x7f0128130d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 10 monclient(hunting): get_auth_request con 0x7f012809f020 auth_method 0 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:30.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f01277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 1 -- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 msgr2=0x7f0128130d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 1 -- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 msgr2=0x7f0128130d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 0x7f0128130d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f011ffff640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 0x7f0128130d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient(hunting): handle_auth_done global_id 64892 payload 824 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 -- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01281321a0 msgr2=0x7f012811b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01281321a0 0x7f012811b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 -- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 msgr2=0x7f0128130d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:30.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 0x7f0128130d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 monclient: found mon.noname-c 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.007+0000 7f0126ffd640 1 -- 172.21.15.144:0/1847787548 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118009390 con 0x7f012809ec50 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f0126ffd640 1 -- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 msgr2=0x7f012809ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f0126ffd640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f012809ac20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f01180097d0 tx=0x7f0118032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118031990 con 0x7f012809ec50 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:30.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: dump: 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:30.026 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: _finish_auth 0 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0118031af0 con 0x7f012809ec50 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f012d880640 5 monclient: authenticate success, global_id 64892 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f012d880640 10 monclient: _renew_subs 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f012d880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012811cf50 con 0x7f012809ec50 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118031e70 con 0x7f012809ec50 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:30.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f011effd640 10 monclient: dump: 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:30.028 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f0125ffb640 4 set_mon_vals no callback set 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f012d880640 10 monclient: _renew_subs 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f012d880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.009+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012809b7a0 con 0x7f012809ec50 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.010+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0118039040 con 0x7f012809ec50 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.010+0000 7f012d880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.010+0000 7f012d880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.010+0000 7f012d880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.010+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01280008d0 con 0x7f012809ec50 2024-09-18T12:32:30.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.010+0000 7f011effd640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0108038490 0x7f010803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:30.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.010+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(187..187 src has 13..187) v4 ==== 7957+0+0 (secure 0 0 0) 0x7f011806cb00 con 0x7f012809ec50 2024-09-18T12:32:30.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.011+0000 7f011ffff640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0108038490 0x7f010803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:30.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.011+0000 7f011ffff640 10 monclient: get_auth_request con 0x7f0108038490 auth_method 0 2024-09-18T12:32:30.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.013+0000 7f011ffff640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0108038490 0x7f010803a940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f012809b3a0 tx=0x7f0110009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:30.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.015+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f011803b610 con 0x7f012809ec50 2024-09-18T12:32:30.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.015+0000 7f011effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:30.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.015+0000 7f011effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.175+0000 7f012d880640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T12:32:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.175+0000 7f012d880640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T12:32:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.175+0000 7f012d880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:30.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.175+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f01280008d0 con 0x7f012809ec50 2024-09-18T12:32:30.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.821+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v188) v1 ==== 122+0+0 (secure 0 0 0) 0x7f011803f030 con 0x7f012809ec50 2024-09-18T12:32:30.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.821+0000 7f011effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T12:32:30.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.821+0000 7f011effd640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-18T12:32:30.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.919+0000 7f012d880640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T12:32:30.920 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.919+0000 7f012d880640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T12:32:30.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.919+0000 7f012d880640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:30.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.919+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f01280008d0 con 0x7f012809ec50 2024-09-18T12:32:30.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.920+0000 7f011effd640 1 -- 172.21.15.144:0/1847787548 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v188) v1 ==== 122+0+0 (secure 0 0 0) 0x7f011803f030 con 0x7f012809ec50 2024-09-18T12:32:30.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.920+0000 7f011effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T12:32:30.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.920+0000 7f011effd640 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-18T12:32:30.922 INFO:tasks.workunit.client.0.smithi144.stderr:device 'fooo' does not appear in the crush map 2024-09-18T12:32:30.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0108038490 msgr2=0x7f010803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:30.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0108038490 0x7f010803a940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f012809b3a0 tx=0x7f0110009040 comp rx=0 tx=0).stop 2024-09-18T12:32:30.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 10 monclient: shutdown 2024-09-18T12:32:30.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 msgr2=0x7f012809ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f012809ac20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f01180097d0 tx=0x7f0118032000 comp rx=0 tx=0).stop 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 shutdown_connections 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0108038490 0x7f010803a940 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f01281321a0 0x7f012811b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809f020 0x7f0128130d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 --2- 172.21.15.144:0/1847787548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f012809ec50 0x7f012809ac20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 >> 172.21.15.144:0/1847787548 conn(0x7f01280a5cf0 msgr2=0x7f01280a48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 shutdown_connections 2024-09-18T12:32:30.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:30.923+0000 7f012d880640 1 -- 172.21.15.144:0/1847787548 wait complete. 2024-09-18T12:32:30.933 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm barr 2024-09-18T12:32:31.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.005+0000 7fdf4e91d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:31.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.005+0000 7fdf4e91d640 10 monclient: build_initial_monmap 2024-09-18T12:32:31.007 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.005+0000 7fdf4e91d640 10 monclient: monmap: 2024-09-18T12:32:31.007 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:31.007 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.007 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:31.006580+0000 2024-09-18T12:32:31.007 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:31.006580+0000 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:31.008 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 1 Processor -- start 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 1 -- start start 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient: init 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf4806e000 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806a710 0x7fdf4806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf4806a710 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 0x7fdf4809e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf4809e170 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:31.019 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf480699c0 con 0x7fdf4806e000 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4806a360 con 0x7fdf4809e170 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4806be20 con 0x7fdf4806a710 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.006+0000 7fdf4e91d640 10 monclient(hunting): authenticate will time out at 1186.575996s 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf47fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fdf4806e000 0x7fdf4809da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:37370/0 (socket says 172.21.15.144:37370) 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf47fff640 1 -- 172.21.15.144:0/624485615 learned_addr learned my addr 172.21.15.144:0/624485615 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 1 --2- 172.21.15.144:0/624485615 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806a710 0x7fdf4806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): get_auth_request con 0x7fdf4806a710 auth_method 0 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.020 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 1 --2- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 0x7fdf4809e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 10 monclient(hunting): get_auth_request con 0x7fdf4809e170 auth_method 0 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 1 -- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 msgr2=0x7fdf4809e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:31.021 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 1 -- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 msgr2=0x7fdf4809e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 1 --2- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 0x7fdf4809e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf4ceba640 1 --2- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 0x7fdf4809e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient(hunting): handle_auth_done global_id 64895 payload 274 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 1 -- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 msgr2=0x7fdf4809e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 1 --2- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 0x7fdf4809e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 1 -- 172.21.15.144:0/624485615 >> v1:172.21.15.144:6789/0 conn(0x7fdf4806e000 legacy=0x7fdf4809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 1 monclient: found mon.noname-b 2024-09-18T12:32:31.022 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.007+0000 7fdf477fe640 1 -- 172.21.15.144:0/624485615 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf48069f80 con 0x7fdf4806a710 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf477fe640 1 -- 172.21.15.144:0/624485615 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806a710 msgr2=0x7fdf4806e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf477fe640 1 --2- 172.21.15.144:0/624485615 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806a710 0x7fdf4806e3d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009850 tx=0x7fdf3c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73e6877a78307f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 1 -- 172.21.15.144:0/624485615 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c02add0 con 0x7fdf4806a710 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: dump: 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.023 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 1 -- 172.21.15.144:0/624485615 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c032440 con 0x7fdf4806a710 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 1 -- 172.21.15.144:0/624485615 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c032820 con 0x7fdf4806a710 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf4e91d640 5 monclient: authenticate success, global_id 64895 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf4e91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:31.024 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf4e91d640 4 set_mon_vals no callback set 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.010+0000 7fdf46ffd640 10 monclient: dump: 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.025 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 10 monclient: shutdown 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 -- 172.21.15.144:0/624485615 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806a710 msgr2=0x7fdf4806e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/624485615 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806a710 0x7fdf4806e3d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009850 tx=0x7fdf3c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 -- 172.21.15.144:0/624485615 shutdown_connections 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/624485615 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806a710 0x7fdf4806e3d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/624485615 >> v2:172.21.15.116:3300/0 conn(0x7fdf4809e170 0x7fdf4809e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 -- 172.21.15.144:0/624485615 >> 172.21.15.144:0/624485615 conn(0x7fdf48071340 msgr2=0x7fdf480a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 -- 172.21.15.144:0/624485615 shutdown_connections 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 1 -- 172.21.15.144:0/624485615 wait complete. 2024-09-18T12:32:31.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 10 monclient: build_initial_monmap 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.011+0000 7fdf4e91d640 10 monclient: monmap: 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:31.027 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:31.028 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:31.028 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 Processor -- start 2024-09-18T12:32:31.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 -- start start 2024-09-18T12:32:31.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient: init 2024-09-18T12:32:31.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:31.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 0x7fdf481454f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf4806a710 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806e000 0x7fdf481343d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf4806e000 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 0x7fdf48134910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf4809e170 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4811af90 con 0x7fdf4809e170 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4811cac0 con 0x7fdf4806a710 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4811cda0 con 0x7fdf4806e000 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:31.029 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4e91d640 10 monclient(hunting): authenticate will time out at 1186.581675s 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 0x7fdf481454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 0x7fdf481454f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42950/0 (socket says 172.21.15.144:42950) 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 1 -- 172.21.15.144:0/83754491 learned_addr learned my addr 172.21.15.144:0/83754491 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 10 monclient(hunting): get_auth_request con 0x7fdf4806a710 auth_method 0 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806e000 0x7fdf481343d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 10 monclient(hunting): get_auth_request con 0x7fdf4806e000 auth_method 0 2024-09-18T12:32:31.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4ceba640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 0x7fdf48134910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4ceba640 10 monclient(hunting): get_auth_request con 0x7fdf4809e170 auth_method 0 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4ceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4ceba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf4ceba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.012+0000 7fdf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf4ceba640 1 -- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 msgr2=0x7fdf48134910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf4ceba640 1 -- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 msgr2=0x7fdf48134910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:31.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf4ceba640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 0x7fdf48134910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf4ceba640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 0x7fdf48134910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 10 monclient(hunting): handle_auth_done global_id 6103 payload 824 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 1 -- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806e000 msgr2=0x7fdf481343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806e000 0x7fdf481343d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 1 -- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 msgr2=0x7fdf48134910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 0x7fdf48134910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.013+0000 7fdf47fff640 1 -- 172.21.15.144:0/83754491 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c009390 con 0x7fdf4806a710 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf47fff640 1 -- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 msgr2=0x7fdf481454f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:31.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf47fff640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 0x7fdf481454f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800c5e0 tx=0x7fdf38004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf38005cb0 con 0x7fdf4806a710 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: dump: 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.033 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: _finish_auth 0 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf4e91d640 5 monclient: authenticate success, global_id 6103 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf4e91d640 10 monclient: _renew_subs 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf4e91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4811bb60 con 0x7fdf4806a710 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf38005e30 con 0x7fdf4806a710 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf38007340 con 0x7fdf4806a710 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:31.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf2ffff640 10 monclient: dump: 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf4e91d640 10 monclient: _renew_subs 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf4e91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf467fc640 4 set_mon_vals no callback set 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.015+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf48146290 con 0x7fdf4806a710 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdf3801b400 con 0x7fdf4806a710 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf2ffff640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf34038490 0x7fdf3403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(188..188 src has 13..188) v4 ==== 7741+0+0 (secure 0 0 0) 0x7fdf3804bf30 con 0x7fdf4806a710 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf4e91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf4e91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf4e91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4811d920 con 0x7fdf4806a710 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf477fe640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf34038490 0x7fdf3403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.016+0000 7fdf477fe640 10 monclient: get_auth_request con 0x7fdf34038490 auth_method 0 2024-09-18T12:32:31.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.018+0000 7fdf477fe640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf34038490 0x7fdf3403a940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c034000 tx=0x7fdf3c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:31.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.020+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdf3801b6b0 con 0x7fdf4806a710 2024-09-18T12:32:31.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.020+0000 7fdf2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:31.037 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.020+0000 7fdf2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:31.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.179+0000 7fdf4e91d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T12:32:31.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.179+0000 7fdf4e91d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T12:32:31.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.179+0000 7fdf4e91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.179+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7fdf480008d0 con 0x7fdf4806a710 2024-09-18T12:32:31.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.593+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v189) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdf38029090 con 0x7fdf4806a710 2024-09-18T12:32:31.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.593+0000 7fdf2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T12:32:31.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.593+0000 7fdf2ffff640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-18T12:32:31.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.691+0000 7fdf4e91d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T12:32:31.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.691+0000 7fdf4e91d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T12:32:31.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.691+0000 7fdf4e91d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.691+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7fdf48072640 con 0x7fdf4806a710 2024-09-18T12:32:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.691+0000 7fdf2ffff640 1 -- 172.21.15.144:0/83754491 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v189) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdf3801a360 con 0x7fdf4806a710 2024-09-18T12:32:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.691+0000 7fdf2ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T12:32:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.691+0000 7fdf2ffff640 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-18T12:32:31.693 INFO:tasks.workunit.client.0.smithi144.stderr:device 'barr' does not appear in the crush map 2024-09-18T12:32:31.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf34038490 msgr2=0x7fdf3403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:31.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf34038490 0x7fdf3403a940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c034000 tx=0x7fdf3c032000 comp rx=0 tx=0).stop 2024-09-18T12:32:31.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 10 monclient: shutdown 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 msgr2=0x7fdf481454f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 0x7fdf481454f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800c5e0 tx=0x7fdf38004960 comp rx=0 tx=0).stop 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 shutdown_connections 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf34038490 0x7fdf3403a940 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf4809e170 0x7fdf48134910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf4806e000 0x7fdf481343d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 --2- 172.21.15.144:0/83754491 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdf4806a710 0x7fdf481454f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 >> 172.21.15.144:0/83754491 conn(0x7fdf48071340 msgr2=0x7fdf480a1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 shutdown_connections 2024-09-18T12:32:31.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.693+0000 7fdf4e91d640 1 -- 172.21.15.144:0/83754491 wait complete. 2024-09-18T12:32:31.703 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T12:32:31.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.775+0000 7fe35359d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:31.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.775+0000 7fe35359d640 10 monclient: build_initial_monmap 2024-09-18T12:32:31.776 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.775+0000 7fe35359d640 10 monclient: monmap: 2024-09-18T12:32:31.776 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:31.776 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.776 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:31.776167+0000 2024-09-18T12:32:31.776 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:31.776167+0000 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:31.777 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.775+0000 7fe35359d640 1 Processor -- start 2024-09-18T12:32:31.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 1 -- start start 2024-09-18T12:32:31.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient: init 2024-09-18T12:32:31.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c09ec50 0x7fe34c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient(hunting): picked mon.noname-b con 0x7fe34c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe34c1321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe34c09f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c067420 con 0x7fe34c1321a0 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c0a0480 con 0x7fe34c09f020 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.780 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c0a0200 con 0x7fe34c09ec50 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe35359d640 10 monclient(hunting): authenticate will time out at 1187.345581s 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe351b3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe34c09f020 0x7fe34c069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:37374/0 (socket says 172.21.15.144:37374) 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe351b3a640 1 -- 172.21.15.144:0/935044735 learned_addr learned my addr 172.21.15.144:0/935044735 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe351339640 1 --2- 172.21.15.144:0/935044735 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c09ec50 0x7fe34c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe351339640 10 monclient(hunting): get_auth_request con 0x7fe34c09ec50 auth_method 0 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe351339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe351339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.776+0000 7fe351339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 10 monclient(hunting): handle_auth_done global_id 64901 payload 274 2024-09-18T12:32:31.781 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 1 -- 172.21.15.144:0/935044735 >> v1:172.21.15.144:6789/0 conn(0x7fe34c09f020 legacy=0x7fe34c069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:31.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 1 -- 172.21.15.144:0/935044735 >> v1:172.21.15.116:6789/0 conn(0x7fe34c1321a0 legacy=0x7fe34c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:31.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 10 monclient: _finish_hunting 0 2024-09-18T12:32:31.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 1 monclient: found mon.noname-b 2024-09-18T12:32:31.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:31.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:31.782 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.777+0000 7fe351339640 1 -- 172.21.15.144:0/935044735 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c09fd80 con 0x7fe34c09ec50 2024-09-18T12:32:31.788 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe351339640 1 -- 172.21.15.144:0/935044735 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c09ec50 msgr2=0x7fe34c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe351339640 1 --2- 172.21.15.144:0/935044735 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c09ec50 0x7fe34c131c60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe348009880 tx=0x7fe3480090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=881952924c94eb59 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 1 -- 172.21.15.144:0/935044735 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348032440 con 0x7fe34c09ec50 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: dump: 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.789 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: _finish_auth 0 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 1 -- 172.21.15.144:0/935044735 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3480325a0 con 0x7fe34c09ec50 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 5 monclient: authenticate success, global_id 64901 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:31.790 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 1 -- 172.21.15.144:0/935044735 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348032980 con 0x7fe34c09ec50 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 4 set_mon_vals no callback set 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe343fff640 10 monclient: dump: 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.791 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 10 monclient: shutdown 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 1 -- 172.21.15.144:0/935044735 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c09ec50 msgr2=0x7fe34c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 1 --2- 172.21.15.144:0/935044735 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c09ec50 0x7fe34c131c60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe348009880 tx=0x7fe3480090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 1 -- 172.21.15.144:0/935044735 shutdown_connections 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 1 --2- 172.21.15.144:0/935044735 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c09ec50 0x7fe34c131c60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.780+0000 7fe35359d640 1 -- 172.21.15.144:0/935044735 >> 172.21.15.144:0/935044735 conn(0x7fe34c0a5cf0 msgr2=0x7fe34c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.781+0000 7fe35359d640 1 -- 172.21.15.144:0/935044735 shutdown_connections 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.781+0000 7fe35359d640 1 -- 172.21.15.144:0/935044735 wait complete. 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.781+0000 7fe35359d640 10 monclient: build_initial_monmap 2024-09-18T12:32:31.792 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.781+0000 7fe35359d640 10 monclient: monmap: 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:31.793 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.781+0000 7fe35359d640 1 Processor -- start 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.781+0000 7fe35359d640 1 -- start start 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.781+0000 7fe35359d640 10 monclient: init 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 0x7fe34c09ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): picked mon.noname-a con 0x7fe34c09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 0x7fe34c130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): picked mon.noname-b con 0x7fe34c09f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c1321a0 0x7fe34c11b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.794 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): picked mon.noname-c con 0x7fe34c1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c11c850 con 0x7fe34c09ec50 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c11e710 con 0x7fe34c09f020 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c11cf80 con 0x7fe34c1321a0 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe35359d640 10 monclient(hunting): authenticate will time out at 1187.351503s 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 0x7fe34c130d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 0x7fe34c130d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42964/0 (socket says 172.21.15.144:42964) 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 -- 172.21.15.144:0/4272694669 learned_addr learned my addr 172.21.15.144:0/4272694669 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:31.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): get_auth_request con 0x7fe34c09f020 auth_method 0 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 0x7fe34c09ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 10 monclient(hunting): get_auth_request con 0x7fe34c09ec50 auth_method 0 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351b3a640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c1321a0 0x7fe34c11b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351b3a640 10 monclient(hunting): get_auth_request con 0x7fe34c1321a0 auth_method 0 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 1 -- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 msgr2=0x7fe34c09ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 1 -- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 msgr2=0x7fe34c09ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 0x7fe34c09ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient(hunting): handle_auth_done global_id 6106 payload 824 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351339640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 0x7fe34c09ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 -- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c1321a0 msgr2=0x7fe34c11b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c1321a0 0x7fe34c11b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 -- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 msgr2=0x7fe34c09ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 0x7fe34c09ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient: _finish_hunting 0 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 monclient: found mon.noname-b 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe350b38640 1 -- 172.21.15.144:0/4272694669 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe348009390 con 0x7fe34c09f020 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.782+0000 7fe351b3a640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c1321a0 0x7fe34c11b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe350b38640 1 -- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 msgr2=0x7fe34c130d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe350b38640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 0x7fe34c130d50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00c610 tx=0x7fe33c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c012840 con 0x7fe34c09f020 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:31.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: dump: 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: _finish_auth 0 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe35359d640 5 monclient: authenticate success, global_id 6106 2024-09-18T12:32:31.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe35359d640 10 monclient: _renew_subs 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe35359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c1420c0 con 0x7fe34c09f020 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c0129e0 con 0x7fe34c09f020 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c012da0 con 0x7fe34c09f020 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3417fa640 10 monclient: dump: 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.800 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe3437fe640 4 set_mon_vals no callback set 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe35359d640 10 monclient: _renew_subs 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe35359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.785+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c11e200 con 0x7fe34c09f020 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe33c011590 con 0x7fe34c09f020 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe35359d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe35359d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:31.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe35359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c0008d0 con 0x7fe34c09f020 2024-09-18T12:32:31.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe3417fa640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe330038490 0x7fe33003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:31.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(189..189 src has 13..189) v4 ==== 7525+0+0 (secure 0 0 0) 0x7fe33c04be80 con 0x7fe34c09f020 2024-09-18T12:32:31.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe351339640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe330038490 0x7fe33003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:31.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.786+0000 7fe351339640 10 monclient: get_auth_request con 0x7fe330038490 auth_method 0 2024-09-18T12:32:31.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.788+0000 7fe351339640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe330038490 0x7fe33003a940 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe348034000 tx=0x7fe348032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:31.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.790+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe33c019310 con 0x7fe34c09f020 2024-09-18T12:32:31.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.790+0000 7fe3417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:31.803 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.790+0000 7fe3417fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:31.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.954+0000 7fe35359d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:31.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.954+0000 7fe35359d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:31.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.954+0000 7fe35359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:31.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:31.954+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fe34c0a13e0 con 0x7fe34c09f020 2024-09-18T12:32:32.579 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.681+0000 7f377f35d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.681+0000 7f377f35d640 10 monclient: build_initial_monmap 2024-09-18T12:32:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.681+0000 7f377f35d640 10 monclient: monmap: 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:32.683390+0000 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:32.683390+0000 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:32.684 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.682+0000 7f377f35d640 1 Processor -- start 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.682+0000 7f377f35d640 1 -- start start 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.682+0000 7f377f35d640 10 monclient: init 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.682+0000 7f377f35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.682+0000 7f377f35d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.682+0000 7f377f35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f377809a510 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.682+0000 7f377f35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 0x7f37781349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 10 monclient(hunting): picked mon.noname-b con 0x7f37780973b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3778097780 0x7f3778136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 10 monclient(hunting): picked mon.noname-c con 0x7f3778097780 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377806ee10 con 0x7f377809a510 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377809cb00 con 0x7f3778097780 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377809a8e0 con 0x7f37780973b0 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377f35d640 10 monclient(hunting): authenticate will time out at 1180.005049s 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 0x7f37781349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 0x7f37781349a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46130/0 (socket says 172.21.15.116:46130) 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 -- 172.21.15.116:0/3206283408 learned_addr learned my addr 172.21.15.116:0/3206283408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): get_auth_request con 0x7f37780973b0 auth_method 0 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377d8fa640 1 --2- 172.21.15.116:0/3206283408 >> v2:172.21.15.144:3300/0 conn(0x7f3778097780 0x7f3778136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377d8fa640 10 monclient(hunting): get_auth_request con 0x7f3778097780 auth_method 0 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377d8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377d8fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377d8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient(hunting): handle_auth_done global_id 64907 payload 274 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 -- 172.21.15.116:0/3206283408 >> v2:172.21.15.144:3300/0 conn(0x7f3778097780 msgr2=0x7f3778136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 --2- 172.21.15.116:0/3206283408 >> v2:172.21.15.144:3300/0 conn(0x7f3778097780 0x7f3778136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 -- 172.21.15.116:0/3206283408 >> v1:172.21.15.144:6789/0 conn(0x7f377809a510 legacy=0x7f3778096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:32.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient: _finish_hunting 0 2024-09-18T12:32:32.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 monclient: found mon.noname-b 2024-09-18T12:32:32.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:32.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:32.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.683+0000 7f377c8f8640 1 -- 172.21.15.116:0/3206283408 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377809d590 con 0x7f37780973b0 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377c8f8640 1 -- 172.21.15.116:0/3206283408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 msgr2=0x7f37781349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377c8f8640 1 --2- 172.21.15.116:0/3206283408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 0x7f37781349a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3768009910 tx=0x7f37680091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee770e155aa2d975 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 1 -- 172.21.15.116:0/3206283408 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768032440 con 0x7f37780973b0 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: dump: 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: _finish_auth 0 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377f35d640 5 monclient: authenticate success, global_id 64907 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 1 -- 172.21.15.116:0/3206283408 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37680325a0 con 0x7f37780973b0 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:32.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377f35d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377f35d640 4 set_mon_vals no callback set 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 1 -- 172.21.15.116:0/3206283408 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768032940 con 0x7f37780973b0 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f376ffff640 10 monclient: dump: 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:32.699 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377f35d640 10 monclient: shutdown 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377f35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377f35d640 1 -- 172.21.15.116:0/3206283408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 msgr2=0x7f37781349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.686+0000 7f377f35d640 1 --2- 172.21.15.116:0/3206283408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 0x7f37781349a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3768009910 tx=0x7f37680091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 -- 172.21.15.116:0/3206283408 shutdown_connections 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 --2- 172.21.15.116:0/3206283408 >> v2:172.21.15.144:3300/0 conn(0x7f3778097780 0x7f3778136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 --2- 172.21.15.116:0/3206283408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37780973b0 0x7f37781349a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 -- 172.21.15.116:0/3206283408 >> 172.21.15.116:0/3206283408 conn(0x7f37780672b0 msgr2=0x7f37780676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:32.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 -- 172.21.15.116:0/3206283408 shutdown_connections 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 -- 172.21.15.116:0/3206283408 wait complete. 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 10 monclient: build_initial_monmap 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 10 monclient: monmap: 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:32.701 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 Processor -- start 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 1 -- start start 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.687+0000 7f377f35d640 10 monclient: init 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37780973b0 0x7f3778140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): picked mon.noname-b con 0x7f37780973b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 0x7f377812fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): picked mon.noname-c con 0x7f3778097780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 0x7f3778098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): picked mon.noname-a con 0x7f377809a510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377811ee90 con 0x7f377809a510 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377811dc50 con 0x7f37780973b0 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377811ebb0 con 0x7f3778097780 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377f35d640 10 monclient(hunting): authenticate will time out at 1180.010694s 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 0x7f377812fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 0x7f377812fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46144/0 (socket says 172.21.15.116:46144) 2024-09-18T12:32:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 1 -- 172.21.15.116:0/512756035 learned_addr learned my addr 172.21.15.116:0/512756035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 0x7f3778098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 10 monclient(hunting): get_auth_request con 0x7f3778097780 auth_method 0 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 10 monclient(hunting): get_auth_request con 0x7f377809a510 auth_method 0 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:32.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 1 -- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 msgr2=0x7f3778098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 1 -- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 msgr2=0x7f3778098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377c8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d0f9640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37780973b0 0x7f3778140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 0x7f3778098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d0f9640 10 monclient(hunting): get_auth_request con 0x7f37780973b0 auth_method 0 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.688+0000 7f377d8fa640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 0x7f3778098300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377d0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377d0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:32.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377d0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 10 monclient(hunting): handle_auth_done global_id 64910 payload 824 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 1 -- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37780973b0 msgr2=0x7f3778140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37780973b0 0x7f3778140fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 1 -- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 msgr2=0x7f3778098300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 0x7f3778098300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 10 monclient: _finish_hunting 0 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 1 monclient: found mon.noname-c 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.689+0000 7f377c8f8640 1 -- 172.21.15.116:0/512756035 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3768009450 con 0x7f3778097780 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377c8f8640 1 -- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 msgr2=0x7f377812fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377c8f8640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 0x7f377812fa70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3768035040 tx=0x7f3768032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 1 -- 172.21.15.116:0/512756035 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768006b30 con 0x7f3778097780 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: dump: 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 1 -- 172.21.15.116:0/512756035 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3768000d10 con 0x7f3778097780 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377f35d640 5 monclient: authenticate success, global_id 64910 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377f35d640 10 monclient: _renew_subs 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377f35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377811f170 con 0x7f3778097780 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 1 -- 172.21.15.116:0/512756035 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376803d440 con 0x7f3778097780 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376d7fa640 10 monclient: dump: 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f376f7fe640 4 set_mon_vals no callback set 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377f35d640 10 monclient: _renew_subs 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377f35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.691+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3778141d80 con 0x7f3778097780 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.692+0000 7f376d7fa640 1 -- 172.21.15.116:0/512756035 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3768039040 con 0x7f3778097780 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.692+0000 7f377f35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.692+0000 7f377f35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.692+0000 7f377f35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.692+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377804bf60 con 0x7f3778097780 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.692+0000 7f376d7fa640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f375c038490 0x7f375c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.692+0000 7f376d7fa640 1 -- 172.21.15.116:0/512756035 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(189..189 src has 13..189) v4 ==== 7525+0+0 (secure 0 0 0) 0x7f376806c0a0 con 0x7f3778097780 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.693+0000 7f377d0f9640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f375c038490 0x7f375c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.693+0000 7f377d0f9640 10 monclient: get_auth_request con 0x7f375c038490 auth_method 0 2024-09-18T12:32:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.695+0000 7f377d0f9640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f375c038490 0x7f375c03a940 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3774007fd0 tx=0x7f3774002630 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.697+0000 7f376d7fa640 1 -- 172.21.15.116:0/512756035 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f376802f2a0 con 0x7f3778097780 2024-09-18T12:32:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.697+0000 7f376d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.697+0000 7f376d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:32.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:32.824+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v190) v1 ==== 78+0+0 (secure 0 0 0) 0x7fe33c019310 con 0x7fe34c09f020 2024-09-18T12:32:32.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:32.824+0000 7fe3417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:32.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:32.824+0000 7fe3417fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.910+0000 7f377f35d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.910+0000 7f377f35d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.910+0000 7f377f35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:32.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.910+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f377811f4e0 con 0x7f3778097780 2024-09-18T12:32:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.911+0000 7f376d7fa640 1 -- 172.21.15.116:0/512756035 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f3768042080 con 0x7f3778097780 2024-09-18T12:32:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.911+0000 7f376d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:32.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.911+0000 7f376d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:32.913 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":76,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.912+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f375c038490 msgr2=0x7f375c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.912+0000 7f377f35d640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f375c038490 0x7f375c03a940 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3774007fd0 tx=0x7f3774002630 comp rx=0 tx=0).stop 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.912+0000 7f377f35d640 10 monclient: shutdown 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.912+0000 7f377f35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.912+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 msgr2=0x7f377812fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.912+0000 7f377f35d640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 0x7f377812fa70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3768035040 tx=0x7f3768032000 comp rx=0 tx=0).stop 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 shutdown_connections 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f375c038490 0x7f375c03a940 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f377809a510 0x7f3778098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3778097780 0x7f377812fa70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 --2- 172.21.15.116:0/512756035 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37780973b0 0x7f3778140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:32.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 >> 172.21.15.116:0/512756035 conn(0x7f37780672b0 msgr2=0x7f377809faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:32.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 shutdown_connections 2024-09-18T12:32:32.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:32.913+0000 7f377f35d640 1 -- 172.21.15.116:0/512756035 wait complete. 2024-09-18T12:32:32.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:32.927+0000 7fe35359d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:32.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:32.927+0000 7fe35359d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:32.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:32.927+0000 7fe35359d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:32.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:32.927+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fe34c09b890 con 0x7fe34c09f020 2024-09-18T12:32:33.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.831+0000 7fe3417fa640 1 -- 172.21.15.144:0/4272694669 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v191) v1 ==== 78+0+0 (secure 0 0 0) 0x7fe33c019310 con 0x7fe34c09f020 2024-09-18T12:32:33.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.831+0000 7fe3417fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:32:33.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.831+0000 7fe3417fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe330038490 msgr2=0x7fe33003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe330038490 0x7fe33003a940 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe348034000 tx=0x7fe348032000 comp rx=0 tx=0).stop 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 10 monclient: shutdown 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 msgr2=0x7fe34c130d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 0x7fe34c130d50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00c610 tx=0x7fe33c007ed0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 shutdown_connections 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe330038490 0x7fe33003a940 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe34c1321a0 0x7fe34c11b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c09f020 0x7fe34c130d50 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 --2- 172.21.15.144:0/4272694669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe34c09ec50 0x7fe34c09ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 >> 172.21.15.144:0/4272694669 conn(0x7fe34c0a5cf0 msgr2=0x7fe34c0a48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:33.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 shutdown_connections 2024-09-18T12:32:33.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.834+0000 7fe35359d640 1 -- 172.21.15.144:0/4272694669 wait complete. 2024-09-18T12:32:33.846 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.922+0000 7f54e6b1d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.923+0000 7f54e6b1d640 10 monclient: build_initial_monmap 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.923+0000 7f54e6b1d640 10 monclient: monmap: 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:33.923923+0000 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:33.923923+0000 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:33.924 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:33.925 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:33.925 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:33.925 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:33.925 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:33.925 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:33.925 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:33.927 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.923+0000 7f54e6b1d640 1 Processor -- start 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.923+0000 7f54e6b1d640 1 -- start start 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.923+0000 7f54e6b1d640 10 monclient: init 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0131c90 0x7f54e0132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f54e0131c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f54e009d210 0x7f54e01325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f54e009d210 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:33.928 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54e00a0ab0 0x7f54e0068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f54e00a0ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e00a1800 con 0x7f54e00a0ab0 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e009fea0 con 0x7f54e009d210 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00a1520 con 0x7f54e0131c90 2024-09-18T12:32:33.929 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e6b1d640 10 monclient(hunting): authenticate will time out at 1189.493303s 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f54e009d210 0x7f54e01325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f54e009d210 0x7f54e01325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42980/0 (socket says 172.21.15.144:42980) 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 -- 172.21.15.144:0/2366645637 learned_addr learned my addr 172.21.15.144:0/2366645637 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): get_auth_request con 0x7f54e009d210 auth_method 0 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:33.930 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e48b9640 1 --2- 172.21.15.144:0/2366645637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0131c90 0x7f54e0132060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e48b9640 10 monclient(hunting): get_auth_request con 0x7f54e0131c90 auth_method 0 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e48b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e48b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e48b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54e50ba640 1 --2- 172.21.15.144:0/2366645637 >> v2:172.21.15.116:3300/0 conn(0x7f54e00a0ab0 0x7f54e0068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient(hunting): handle_auth_done global_id 6115 payload 274 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 -- 172.21.15.144:0/2366645637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0131c90 msgr2=0x7f54e0132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 --2- 172.21.15.144:0/2366645637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0131c90 0x7f54e0132060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 -- 172.21.15.144:0/2366645637 >> v2:172.21.15.116:3300/0 conn(0x7f54e00a0ab0 msgr2=0x7f54e0068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 --2- 172.21.15.144:0/2366645637 >> v2:172.21.15.116:3300/0 conn(0x7f54e00a0ab0 0x7f54e0068e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.931 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:33.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 monclient: found mon.noname-c 2024-09-18T12:32:33.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:33.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:33.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.924+0000 7f54dffff640 1 -- 172.21.15.144:0/2366645637 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e0069340 con 0x7f54e009d210 2024-09-18T12:32:33.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54dffff640 1 -- 172.21.15.144:0/2366645637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e009d210 msgr2=0x7f54e01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:33.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54dffff640 1 --2- 172.21.15.144:0/2366645637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e009d210 0x7f54e01325a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f54e00a51b0 tx=0x7f54d0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d507bce409e8b0e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:33.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 1 -- 172.21.15.144:0/2366645637 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0007610 con 0x7f54e009d210 2024-09-18T12:32:33.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:33.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: dump: 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:33.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 1 -- 172.21.15.144:0/2366645637 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d002d050 con 0x7f54e009d210 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54e6b1d640 5 monclient: authenticate success, global_id 6115 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54e6b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54e6b1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 1 -- 172.21.15.144:0/2366645637 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d00079e0 con 0x7f54e009d210 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54e6b1d640 4 set_mon_vals no callback set 2024-09-18T12:32:33.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.927+0000 7f54df7fe640 10 monclient: dump: 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:33.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 10 monclient: shutdown 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2366645637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e009d210 msgr2=0x7f54e01325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2366645637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e009d210 0x7f54e01325a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f54e00a51b0 tx=0x7f54d0009840 comp rx=0 tx=0).stop 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2366645637 shutdown_connections 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2366645637 >> v2:172.21.15.116:3300/0 conn(0x7f54e00a0ab0 0x7f54e0068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2366645637 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e009d210 0x7f54e01325a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2366645637 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0131c90 0x7f54e0132060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2366645637 >> 172.21.15.144:0/2366645637 conn(0x7f54e0071370 msgr2=0x7f54e00a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2366645637 shutdown_connections 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2366645637 wait complete. 2024-09-18T12:32:33.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 10 monclient: build_initial_monmap 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.928+0000 7f54e6b1d640 10 monclient: monmap: 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:33.944 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 Processor -- start 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 -- start start 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient: init 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 0x7f54e0141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f54e00a0ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 0x7f54e0141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f54e0131c90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0141bd0 0x7f54e011e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f54e0141bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:33.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00a0fb0 con 0x7f54e00a0ab0 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00a4bc0 con 0x7f54e0131c90 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e0122000 con 0x7f54e0141bd0 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54e6b1d640 10 monclient(hunting): authenticate will time out at 1189.498888s 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54dffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 0x7f54e0141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54dffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 0x7f54e0141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42982/0 (socket says 172.21.15.144:42982) 2024-09-18T12:32:33.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54dffff640 1 -- 172.21.15.144:0/2075639368 learned_addr learned my addr 172.21.15.144:0/2075639368 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54dffff640 10 monclient(hunting): get_auth_request con 0x7f54e0131c90 auth_method 0 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.929+0000 7f54dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 0x7f54e0141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 10 monclient(hunting): get_auth_request con 0x7f54e00a0ab0 auth_method 0 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:33.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0141bd0 0x7f54e011e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 10 monclient(hunting): get_auth_request con 0x7f54e0141bd0 auth_method 0 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 1 -- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 msgr2=0x7f54e0141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 1 -- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 msgr2=0x7f54e0141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 0x7f54e0141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:33.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e48b9640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 0x7f54e0141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54e50ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 10 monclient(hunting): handle_auth_done global_id 6118 payload 824 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 1 -- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0141bd0 msgr2=0x7f54e011e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0141bd0 0x7f54e011e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 1 -- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 msgr2=0x7f54e0141150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 0x7f54e0141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:33.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 1 monclient: found mon.noname-b 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.930+0000 7f54dffff640 1 -- 172.21.15.144:0/2075639368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0009ae0 con 0x7f54e0131c90 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dffff640 1 -- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 msgr2=0x7f54e0141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dffff640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 0x7f54e0141690 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009f50 tx=0x7f54d0036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0032930 con 0x7f54e0131c90 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:33.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: dump: 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:33.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d0032ab0 con 0x7f54e0131c90 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54e6b1d640 5 monclient: authenticate success, global_id 6118 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54e6b1d640 10 monclient: _renew_subs 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54e6b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e0122d70 con 0x7f54e0131c90 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0032e70 con 0x7f54e0131c90 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54dcff9640 10 monclient: dump: 2024-09-18T12:32:33.952 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54deffd640 4 set_mon_vals no callback set 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54e6b1d640 10 monclient: _renew_subs 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54e6b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:33.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.932+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e01219d0 con 0x7f54e0131c90 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.933+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f54d0038050 con 0x7f54e0131c90 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.933+0000 7f54e6b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.933+0000 7f54e6b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.933+0000 7f54e6b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.933+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e00008d0 con 0x7f54e0131c90 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.933+0000 7f54dcff9640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f54c00384e0 0x7f54c003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.933+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(191..191 src has 13..191) v4 ==== 7197+0+0 (secure 0 0 0) 0x7f54d006caf0 con 0x7f54e0131c90 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.934+0000 7f54e48b9640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f54c00384e0 0x7f54c003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.934+0000 7f54e48b9640 10 monclient: get_auth_request con 0x7f54c00384e0 auth_method 0 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.936+0000 7f54e48b9640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f54c00384e0 0x7f54c003a990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f54d8006fd0 tx=0x7f54d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.938+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f54d003b620 con 0x7f54e0131c90 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.938+0000 7f54dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:33.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:33.938+0000 7f54dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:34.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.098+0000 7f54e6b1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:34.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.098+0000 7f54e6b1d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:34.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.098+0000 7f54e6b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:34.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.098+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f54e00695e0 con 0x7f54e0131c90 2024-09-18T12:32:34.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.838+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v192) v1 ==== 82+0+0 (secure 0 0 0) 0x7f54d003f070 con 0x7f54e0131c90 2024-09-18T12:32:34.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.838+0000 7f54dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:34.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.838+0000 7f54dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:34.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.938+0000 7f54e6b1d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:34.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.938+0000 7f54e6b1d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:34.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.938+0000 7f54e6b1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:34.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.938+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f54e00695e0 con 0x7f54e0131c90 2024-09-18T12:32:34.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.938+0000 7f54dcff9640 1 -- 172.21.15.144:0/2075639368 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v192) v1 ==== 115+0+0 (secure 0 0 0) 0x7f54d004cb40 con 0x7f54e0131c90 2024-09-18T12:32:34.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.938+0000 7f54dcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T12:32:34.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.938+0000 7f54dcff9640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T12:32:34.940 INFO:tasks.workunit.client.0.smithi144.stderr:compat weight-set already created 2024-09-18T12:32:34.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f54c00384e0 msgr2=0x7f54c003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:34.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f54c00384e0 0x7f54c003a990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f54d8006fd0 tx=0x7f54d8008040 comp rx=0 tx=0).stop 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 10 monclient: shutdown 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 msgr2=0x7f54e0141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 0x7f54e0141690 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009f50 tx=0x7f54d0036000 comp rx=0 tx=0).stop 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 shutdown_connections 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f54c00384e0 0x7f54c003a990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f54e0141bd0 0x7f54e011e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f54e0131c90 0x7f54e0141690 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 --2- 172.21.15.144:0/2075639368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e00a0ab0 0x7f54e0141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 >> 172.21.15.144:0/2075639368 conn(0x7f54e0071370 msgr2=0x7f54e00a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 shutdown_connections 2024-09-18T12:32:34.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:34.940+0000 7f54e6b1d640 1 -- 172.21.15.144:0/2075639368 wait complete. 2024-09-18T12:32:34.950 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-18T12:32:35.025 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.024+0000 7f8084494640 10 monclient: get_monmap_and_config 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.024+0000 7f8084494640 10 monclient: build_initial_monmap 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.024+0000 7f8084494640 10 monclient: monmap: 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:35.025195+0000 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:35.025195+0000 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:35.026 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:35.027 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:35.027 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:35.027 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:35.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.024+0000 7f8084494640 1 Processor -- start 2024-09-18T12:32:35.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.024+0000 7f8084494640 1 -- start start 2024-09-18T12:32:35.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.024+0000 7f8084494640 10 monclient: init 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09ec50 0x7f807c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient(hunting): picked mon.noname-b con 0x7f807c09ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f807c1321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 0x7f807c069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient(hunting): picked mon.noname-a con 0x7f807c09f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:35.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c067420 con 0x7f807c1321a0 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c09fdc0 con 0x7f807c09f020 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0a0510 con 0x7f807c09ec50 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8084494640 10 monclient(hunting): authenticate will time out at 1190.594597s 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082a31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 0x7f807c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082230640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09ec50 0x7f807c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082a31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 0x7f807c069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:59924/0 (socket says 172.21.15.144:59924) 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082a31640 1 -- 172.21.15.144:0/288379245 learned_addr learned my addr 172.21.15.144:0/288379245 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082a31640 10 monclient(hunting): get_auth_request con 0x7f807c09f020 auth_method 0 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:35.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082a31640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8081a2f640 1 --1- 172.21.15.144:0/288379245 >> v1:172.21.15.116:6789/0 conn(0x7f807c1321a0 0x7f807c068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:40920/0 (socket says 172.21.15.144:40920) 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082230640 10 monclient(hunting): get_auth_request con 0x7f807c09ec50 auth_method 0 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082230640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082230640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.025+0000 7f8082230640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082a31640 1 -- 172.21.15.144:0/288379245 >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 msgr2=0x7f807c069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082a31640 1 -- 172.21.15.144:0/288379245 >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 msgr2=0x7f807c069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082a31640 1 --2- 172.21.15.144:0/288379245 >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 0x7f807c069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:35.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082a31640 1 --2- 172.21.15.144:0/288379245 >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 0x7f807c069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 10 monclient(hunting): handle_auth_done global_id 64919 payload 274 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 1 -- 172.21.15.144:0/288379245 >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 msgr2=0x7f807c069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 1 --2- 172.21.15.144:0/288379245 >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 0x7f807c069410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 1 -- 172.21.15.144:0/288379245 >> v1:172.21.15.116:6789/0 conn(0x7f807c1321a0 legacy=0x7f807c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 10 monclient: _finish_hunting 0 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 1 monclient: found mon.noname-b 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.026+0000 7f8082230640 1 -- 172.21.15.144:0/288379245 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0a0200 con 0x7f807c09ec50 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8082230640 1 -- 172.21.15.144:0/288379245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09ec50 msgr2=0x7f807c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8082230640 1 --2- 172.21.15.144:0/288379245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09ec50 0x7f807c131c60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8078009850 tx=0x7f80780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=394cbbb9543eee32 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:35.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 1 -- 172.21.15.144:0/288379245 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807802add0 con 0x7f807c09ec50 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: dump: 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:35.034 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: _finish_auth 0 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 1 -- 172.21.15.144:0/288379245 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8078032440 con 0x7f807c09ec50 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 1 -- 172.21.15.144:0/288379245 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780327e0 con 0x7f807c09ec50 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f808122e640 10 monclient: dump: 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.035 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 5 monclient: authenticate success, global_id 64919 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:35.036 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 4 set_mon_vals no callback set 2024-09-18T12:32:35.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 10 monclient: shutdown 2024-09-18T12:32:35.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:35.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 1 -- 172.21.15.144:0/288379245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09ec50 msgr2=0x7f807c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:35.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 1 --2- 172.21.15.144:0/288379245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09ec50 0x7f807c131c60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8078009850 tx=0x7f80780090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 1 -- 172.21.15.144:0/288379245 shutdown_connections 2024-09-18T12:32:35.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 1 --2- 172.21.15.144:0/288379245 >> v2:172.21.15.116:3300/0 conn(0x7f807c09f020 0x7f807c069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.029+0000 7f8084494640 1 --2- 172.21.15.144:0/288379245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09ec50 0x7f807c131c60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 1 -- 172.21.15.144:0/288379245 >> 172.21.15.144:0/288379245 conn(0x7f807c0a5d40 msgr2=0x7f807c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 1 -- 172.21.15.144:0/288379245 shutdown_connections 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 1 -- 172.21.15.144:0/288379245 wait complete. 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 10 monclient: build_initial_monmap 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 10 monclient: monmap: 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:35.042 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 1 Processor -- start 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 1 -- start start 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.030+0000 7f8084494640 10 monclient: init 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 0x7f807c09ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): picked mon.noname-b con 0x7f807c09ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09f020 0x7f807c130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): picked mon.noname-c con 0x7f807c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c1321a0 0x7f807c1412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:35.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): picked mon.noname-a con 0x7f807c1321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c11e000 con 0x7f807c1321a0 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c09c000 con 0x7f807c09ec50 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c11c750 con 0x7f807c09f020 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8084494640 10 monclient(hunting): authenticate will time out at 1190.600240s 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 0x7f807c09ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 0x7f807c09ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42990/0 (socket says 172.21.15.144:42990) 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 -- 172.21.15.144:0/3815073866 learned_addr learned my addr 172.21.15.144:0/3815073866 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): get_auth_request con 0x7f807c09ec50 auth_method 0 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:35.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082a31640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c1321a0 0x7f807c1412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8081a2f640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09f020 0x7f807c130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082a31640 10 monclient(hunting): get_auth_request con 0x7f807c1321a0 auth_method 0 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082a31640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8081a2f640 10 monclient(hunting): get_auth_request con 0x7f807c09f020 auth_method 0 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8081a2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:35.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8081a2f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8081a2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient(hunting): handle_auth_done global_id 6121 payload 824 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 -- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09f020 msgr2=0x7f807c130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09f020 0x7f807c130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 -- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c1321a0 msgr2=0x7f807c1412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c1321a0 0x7f807c1412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient: _finish_hunting 0 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 monclient: found mon.noname-b 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.031+0000 7f8082230640 1 -- 172.21.15.144:0/3815073866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8078009390 con 0x7f807c09ec50 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f8082230640 1 -- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 msgr2=0x7f807c09ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:35.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f8082230640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 0x7f807c09ab00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8078009850 tx=0x7f8078032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078031d40 con 0x7f807c09ec50 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f80727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f80727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f80727fc640 10 monclient: dump: 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:35.047 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f80727fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.033+0000 7f80727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807803d440 con 0x7f807c09ec50 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8084494640 5 monclient: authenticate success, global_id 6121 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8084494640 10 monclient: _renew_subs 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8084494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c0a4300 con 0x7f807c09ec50 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f80727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078031ea0 con 0x7f807c09ec50 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f80727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f80727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f80727fc640 10 monclient: dump: 2024-09-18T12:32:35.048 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8080a2d640 4 set_mon_vals no callback set 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8084494640 10 monclient: _renew_subs 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8084494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.034+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c11ce30 con 0x7f807c09ec50 2024-09-18T12:32:35.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f8084494640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f8084494640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f8084494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c0008d0 con 0x7f807c09ec50 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8078039040 con 0x7f807c09ec50 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f80727fc640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8060038490 0x7f806003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(192..192 src has 13..192) v4 ==== 7525+0+0 (secure 0 0 0) 0x7f807806c430 con 0x7f807c09ec50 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f8081a2f640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8060038490 0x7f806003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.035+0000 7f8081a2f640 10 monclient: get_auth_request con 0x7f8060038490 auth_method 0 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.037+0000 7f8081a2f640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8060038490 0x7f806003a940 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f807c11d340 tx=0x7f806c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.039+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f807802f2a0 con 0x7f807c09ec50 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.039+0000 7f80727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:35.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.039+0000 7f80727fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:35.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.197+0000 7f8084494640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T12:32:35.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.197+0000 7f8084494640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T12:32:35.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.197+0000 7f8084494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:35.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.197+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f807c0a13e0 con 0x7f807c09ec50 2024-09-18T12:32:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.842+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v193) v1 ==== 176+0+0 (secure 0 0 0) 0x7f8078042070 con 0x7f807c09ec50 2024-09-18T12:32:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.842+0000 7f80727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T12:32:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.842+0000 7f80727fc640 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-18T12:32:35.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:35.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.939+0000 7f8084494640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T12:32:35.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.939+0000 7f8084494640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T12:32:35.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.939+0000 7f8084494640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:35.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.939+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f807c0a13e0 con 0x7f807c09ec50 2024-09-18T12:32:35.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.939+0000 7f80727fc640 1 -- 172.21.15.144:0/3815073866 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v193) v1 ==== 150+0+0 (secure 0 0 0) 0x7f8078037030 con 0x7f807c09ec50 2024-09-18T12:32:35.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.939+0000 7f80727fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T12:32:35.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.939+0000 7f80727fc640 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-18T12:32:35.941 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'r1' already exists 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8060038490 msgr2=0x7f806003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8060038490 0x7f806003a940 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f807c11d340 tx=0x7f806c009040 comp rx=0 tx=0).stop 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 10 monclient: shutdown 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 msgr2=0x7f807c09ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 0x7f807c09ab00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8078009850 tx=0x7f8078032000 comp rx=0 tx=0).stop 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 shutdown_connections 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.941+0000 7f8084494640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8060038490 0x7f806003a940 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.942+0000 7f8084494640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c1321a0 0x7f807c1412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.942+0000 7f8084494640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f807c09f020 0x7f807c130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.942+0000 7f8084494640 1 --2- 172.21.15.144:0/3815073866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f807c09ec50 0x7f807c09ab00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:35.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.942+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 >> 172.21.15.144:0/3815073866 conn(0x7f807c0a5d40 msgr2=0x7f807c0a4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:35.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.942+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 shutdown_connections 2024-09-18T12:32:35.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:35.942+0000 7f8084494640 1 -- 172.21.15.144:0/3815073866 wait complete. 2024-09-18T12:32:35.951 INFO:tasks.workunit.client.0.smithi144.stderr:++ seq 1 32 2024-09-18T12:32:35.955 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:35.955 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-18T12:32:36.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.027+0000 7f685dd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.027+0000 7f685dd9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.027+0000 7f685dd9d640 10 monclient: monmap: 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:36.028756+0000 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:36.028756+0000 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:36.029 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:36.030 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:36.030 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:36.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.027+0000 7f685dd9d640 1 Processor -- start 2024-09-18T12:32:36.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.027+0000 7f685dd9d640 1 -- start start 2024-09-18T12:32:36.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.027+0000 7f685dd9d640 10 monclient: init 2024-09-18T12:32:36.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:36.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T12:32:36.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 0x7f6858131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f6858131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f685809e910 0x7f685809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f685809e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f685809f250 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685809faf0 con 0x7f685809f250 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685809fd30 con 0x7f6858131ab0 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68580a1110 con 0x7f685809e910 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f685dd9d640 10 monclient(hunting): authenticate will time out at 1183.350502s 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 0x7f6858131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 0x7f6858131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43866/0 (socket says 172.21.15.116:43866) 2024-09-18T12:32:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 1 -- 172.21.15.116:0/2118951695 learned_addr learned my addr 172.21.15.116:0/2118951695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 10 monclient(hunting): get_auth_request con 0x7f6858131ab0 auth_method 0 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 1 --2- 172.21.15.116:0/2118951695 >> v2:172.21.15.144:3300/0 conn(0x7f685809e910 0x7f685809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): get_auth_request con 0x7f685809e910 auth_method 0 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 1 -- 172.21.15.116:0/2118951695 >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 msgr2=0x7f6858131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 1 -- 172.21.15.116:0/2118951695 >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 msgr2=0x7f6858131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 1 --2- 172.21.15.116:0/2118951695 >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 0x7f6858131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f68577fe640 1 --2- 172.21.15.116:0/2118951695 >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 0x7f6858131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.028+0000 7f6856ffd640 10 monclient(hunting): handle_auth_done global_id 6124 payload 274 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 1 -- 172.21.15.116:0/2118951695 >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 msgr2=0x7f6858131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 1 --2- 172.21.15.116:0/2118951695 >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 0x7f6858131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 1 -- 172.21.15.116:0/2118951695 >> v1:172.21.15.144:6789/0 conn(0x7f685809f250 legacy=0x7f6858068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 1 monclient: found mon.noname-c 2024-09-18T12:32:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.029+0000 7f6856ffd640 1 -- 172.21.15.116:0/2118951695 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68580a13f0 con 0x7f685809e910 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.031+0000 7f6856ffd640 1 -- 172.21.15.116:0/2118951695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 msgr2=0x7f685809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.031+0000 7f6856ffd640 1 --2- 172.21.15.116:0/2118951695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685809ed10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f685809bdb0 tx=0x7f684c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6038cafce0730b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 1 -- 172.21.15.116:0/2118951695 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c02adf0 con 0x7f685809e910 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: dump: 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.050 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 1 -- 172.21.15.116:0/2118951695 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684c032440 con 0x7f685809e910 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 5 monclient: authenticate success, global_id 6124 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:36.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 1 -- 172.21.15.116:0/2118951695 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c032820 con 0x7f685809e910 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 4 set_mon_vals no callback set 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f68567fc640 10 monclient: dump: 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:36.052 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 10 monclient: shutdown 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 1 -- 172.21.15.116:0/2118951695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 msgr2=0x7f685809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 1 --2- 172.21.15.116:0/2118951695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685809ed10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f685809bdb0 tx=0x7f684c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 1 -- 172.21.15.116:0/2118951695 shutdown_connections 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 1 --2- 172.21.15.116:0/2118951695 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685809ed10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 1 --2- 172.21.15.116:0/2118951695 >> v2:172.21.15.116:3300/0 conn(0x7f6858131ab0 0x7f6858131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 1 -- 172.21.15.116:0/2118951695 >> 172.21.15.116:0/2118951695 conn(0x7f6858067d30 msgr2=0x7f68580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.034+0000 7f685dd9d640 1 -- 172.21.15.116:0/2118951695 shutdown_connections 2024-09-18T12:32:36.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.035+0000 7f685dd9d640 1 -- 172.21.15.116:0/2118951695 wait complete. 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.035+0000 7f685dd9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.035+0000 7f685dd9d640 10 monclient: monmap: 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 Processor -- start 2024-09-18T12:32:36.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 -- start start 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient: init 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685811ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f685809e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 0x7f685811d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f685809f250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 0x7f685811d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f6858131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685811ea20 con 0x7f685809f250 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685809bde0 con 0x7f685809e910 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68580a43a0 con 0x7f6858131ab0 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f685dd9d640 10 monclient(hunting): authenticate will time out at 1183.358534s 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 0x7f685811d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685811ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 0x7f685811d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42864/0 (socket says 172.21.15.116:42864) 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 0x7f685811d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 -- 172.21.15.116:0/1417626029 learned_addr learned my addr 172.21.15.116:0/1417626029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): get_auth_request con 0x7f6858131ab0 auth_method 0 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 10 monclient(hunting): get_auth_request con 0x7f685809e910 auth_method 0 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 10 monclient(hunting): get_auth_request con 0x7f685809f250 auth_method 0 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 1 -- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 msgr2=0x7f685811d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 1 -- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 msgr2=0x7f685811d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 0x7f685811d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6856ffd640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 0x7f685811d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient(hunting): handle_auth_done global_id 64925 payload 824 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 -- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 msgr2=0x7f685811ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685811ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 -- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 msgr2=0x7f685811d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 0x7f685811d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:36.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 monclient: found mon.noname-c 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f6857fff640 1 -- 172.21.15.116:0/1417626029 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c009390 con 0x7f6858131ab0 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.036+0000 7f68577fe640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685811ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f6857fff640 1 -- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 msgr2=0x7f685811d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f6857fff640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 0x7f685811d8d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f685000c8f0 tx=0x7f6850002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 1 -- 172.21.15.116:0/1417626029 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685000aa00 con 0x7f6858131ab0 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: dump: 2024-09-18T12:32:36.059 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:36.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f685dd9d640 5 monclient: authenticate success, global_id 64925 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 1 -- 172.21.15.116:0/1417626029 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685000ab80 con 0x7f6858131ab0 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f685dd9d640 10 monclient: _renew_subs 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 1 -- 172.21.15.116:0/1417626029 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850012420 con 0x7f6858131ab0 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f685dd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685811f670 con 0x7f6858131ab0 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f683f7fe640 10 monclient: dump: 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.061 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f6855ffb640 4 set_mon_vals no callback set 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f685dd9d640 10 monclient: _renew_subs 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f685dd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.039+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68581304e0 con 0x7f6858131ab0 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f683f7fe640 1 -- 172.21.15.116:0/1417626029 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6850012810 con 0x7f6858131ab0 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f683f7fe640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6844038490 0x7f684403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f683f7fe640 1 -- 172.21.15.116:0/1417626029 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(193..193 src has 13..193) v4 ==== 7735+0+0 (secure 0 0 0) 0x7f685004c8c0 con 0x7f6858131ab0 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f685dd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f685dd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f685dd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f68577fe640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6844038490 0x7f684403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685804bf60 con 0x7f6858131ab0 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.040+0000 7f68577fe640 10 monclient: get_auth_request con 0x7f6844038490 auth_method 0 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.043+0000 7f68577fe640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6844038490 0x7f684403a940 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6848006a70 tx=0x7f6848008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.045+0000 7f683f7fe640 1 -- 172.21.15.116:0/1417626029 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6850016030 con 0x7f6858131ab0 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.045+0000 7f683f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.045+0000 7f683f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:36.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.030+0000 7f62f3a9f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:36.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.030+0000 7f62f3a9f640 10 monclient: build_initial_monmap 2024-09-18T12:32:36.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.030+0000 7f62f3a9f640 10 monclient: monmap: 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:36.031142+0000 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:36.031142+0000 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.030+0000 7f62f3a9f640 1 Processor -- start 2024-09-18T12:32:36.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 1 -- start start 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient: init 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62ec131bd0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 0x7f62ec131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient(hunting): picked mon.noname-a con 0x7f62ec069e10 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62ec136d60 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec0a06d0 con 0x7f62ec136d60 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec071780 con 0x7f62ec131bd0 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ec06a210 con 0x7f62ec069e10 2024-09-18T12:32:36.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f3a9f640 10 monclient(hunting): authenticate will time out at 1191.600574s 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f203c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f62ec136d60 0x7f62ec139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:40934/0 (socket says 172.21.15.144:40934) 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 0x7f62ec131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f203c640 1 -- 172.21.15.144:0/2635742206 learned_addr learned my addr 172.21.15.144:0/2635742206 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 10 monclient(hunting): get_auth_request con 0x7f62ec069e10 auth_method 0 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 1 -- 172.21.15.144:0/2635742206 >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 msgr2=0x7f62ec131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 1 -- 172.21.15.144:0/2635742206 >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 msgr2=0x7f62ec131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 1 --2- 172.21.15.144:0/2635742206 >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 0x7f62ec131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.031+0000 7f62f103a640 1 --2- 172.21.15.144:0/2635742206 >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 0x7f62ec131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:36.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.033+0000 7f62f203c640 1 -- 172.21.15.144:0/2635742206 >> v1:172.21.15.116:6789/0 conn(0x7f62ec136d60 legacy=0x7f62ec139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.034+0000 7f62f183b640 1 -- 172.21.15.144:0/2635742206 >> v1:172.21.15.144:6789/0 conn(0x7f62ec131bd0 legacy=0x7f62ec09ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749934037 0 0) 0x7f62ec071780 con 0x7f62ec131bd0 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): my global_id is 6127 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d40034e0 con 0x7f62ec131bd0 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567437703 0 0) 0x7f62d40034e0 con 0x7f62ec131bd0 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 msgr2=0x7f62ec131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 --2- 172.21.15.144:0/2635742206 >> v2:172.21.15.116:3300/0 conn(0x7f62ec069e10 0x7f62ec131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 >> v1:172.21.15.116:6789/0 conn(0x7f62ec136d60 legacy=0x7f62ec139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: _finish_hunting 0 2024-09-18T12:32:36.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec0a0010 con 0x7f62ec131bd0 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: _finish_auth 0 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f62e4001b70 con 0x7f62ec131bd0 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f3a9f640 5 monclient: authenticate success, global_id 6127 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f3a9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: dump: 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.074 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 >> v1:172.21.15.144:6789/0 conn(0x7f62ec131bd0 legacy=0x7f62ec09ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62d4003bc0 0x7f62d4003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): picked mon.c con 0x7f62d4003bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d4004780 0x7f62d4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): picked mon.a con 0x7f62d4004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62d400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): picked mon.b con 0x7f62d40099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec06a210 con 0x7f62d4004780 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d400c420 con 0x7f62d40099c0 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d400c5e0 con 0x7f62d4003bc0 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f0839640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:32:36.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f3a9f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62d400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): get_auth_request con 0x7f62d40099c0 auth_method 0 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62d4003bc0 0x7f62d4003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 10 monclient(hunting): get_auth_request con 0x7f62d4003bc0 auth_method 0 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d4004780 0x7f62d4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 10 monclient(hunting): get_auth_request con 0x7f62d4004780 auth_method 0 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f183b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f103a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient(hunting): handle_auth_done global_id 6127 payload 293 2024-09-18T12:32:36.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 1 -- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62d4003bc0 msgr2=0x7f62d4003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62d4003bc0 0x7f62d4003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 1 -- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d4004780 msgr2=0x7f62d4004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d4004780 0x7f62d4004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 1 monclient: found mon.b 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 1 -- 172.21.15.144:0/2635742206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d40034e0 con 0x7f62d40099c0 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 10 monclient: _finish_auth 0 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.035+0000 7f62f203c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.036+0000 7f62f203c640 1 -- 172.21.15.144:0/2635742206 reap_dead start 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f203c640 1 -- 172.21.15.144:0/2635742206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 msgr2=0x7f62d400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f203c640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62d400bda0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f62dc001bf0 tx=0x7f62dc007d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaf30480458f898a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:36.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc02ac60 con 0x7f62d40099c0 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 10 monclient: dump: 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:36.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc02adc0 con 0x7f62d40099c0 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f3a9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 1 -- 172.21.15.144:0/2635742206 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc007340 con 0x7f62d40099c0 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f0839640 10 monclient: dump: 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:36.081 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f3a9f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.038+0000 7f62f3a9f640 4 set_mon_vals no callback set 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 10 monclient: shutdown 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 1 -- 172.21.15.144:0/2635742206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 msgr2=0x7f62d400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62d400bda0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f62dc001bf0 tx=0x7f62dc007d90 comp rx=0 tx=0).stop 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 1 -- 172.21.15.144:0/2635742206 shutdown_connections 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/2635742206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62d400bda0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 1 -- 172.21.15.144:0/2635742206 >> 172.21.15.144:0/2635742206 conn(0x7f62ec0713b0 msgr2=0x7f62ec0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 1 -- 172.21.15.144:0/2635742206 shutdown_connections 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.039+0000 7f62f3a9f640 1 -- 172.21.15.144:0/2635742206 wait complete. 2024-09-18T12:32:36.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient: build_initial_monmap 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient: monmap: 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 Processor -- start 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 -- start start 2024-09-18T12:32:36.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient: init 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62ec130910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): picked mon.noname-b con 0x7f62d40099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 0x7f62ec0997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): picked mon.noname-a con 0x7f62ec130e50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62ec099d10 0x7f62ec09a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): picked mon.noname-c con 0x7f62ec099d10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec11ce00 con 0x7f62ec130e50 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec131240 con 0x7f62d40099c0 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:36.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec11c7a0 con 0x7f62ec099d10 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f3a9f640 10 monclient(hunting): authenticate will time out at 1191.609867s 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f183b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62ec130910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f183b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62ec130910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43000/0 (socket says 172.21.15.144:43000) 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f183b640 1 -- 172.21.15.144:0/497409653 learned_addr learned my addr 172.21.15.144:0/497409653 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f183b640 10 monclient(hunting): get_auth_request con 0x7f62d40099c0 auth_method 0 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f183b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f183b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.040+0000 7f62f183b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62ec099d10 0x7f62ec09a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 0x7f62ec0997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 10 monclient(hunting): get_auth_request con 0x7f62ec099d10 auth_method 0 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 10 monclient(hunting): get_auth_request con 0x7f62ec130e50 auth_method 0 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 1 -- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 msgr2=0x7f62ec0997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 1 -- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 msgr2=0x7f62ec0997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:36.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f203c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 0x7f62ec0997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 10 monclient(hunting): handle_auth_done global_id 6133 payload 824 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f103a640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 0x7f62ec0997d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 1 -- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62ec099d10 msgr2=0x7f62ec09a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62ec099d10 0x7f62ec09a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 1 -- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 msgr2=0x7f62ec0997d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 0x7f62ec0997d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 1 monclient: found mon.noname-b 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.041+0000 7f62f183b640 1 -- 172.21.15.144:0/497409653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec09bec0 con 0x7f62d40099c0 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f183b640 1 -- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 msgr2=0x7f62ec130910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:36.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f183b640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62ec130910 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f62e4007730 tx=0x7f62ec06a350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400fc80 con 0x7f62d40099c0 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: dump: 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.088 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f3a9f640 5 monclient: authenticate success, global_id 6133 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e400fde0 con 0x7f62d40099c0 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e40146a0 con 0x7f62d40099c0 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f3a9f640 10 monclient: _renew_subs 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f3a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec0a4990 con 0x7f62d40099c0 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62e9ffb640 10 monclient: dump: 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:36.089 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f3a9f640 10 monclient: _renew_subs 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f3a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec11bac0 con 0x7f62d40099c0 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.043+0000 7f62ebfff640 4 set_mon_vals no callback set 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.044+0000 7f62f3a9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62f3a9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:36.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62f3a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec09b370 con 0x7f62d40099c0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f62e4014a10 con 0x7f62d40099c0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62e9ffb640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62d8038490 0x7f62d803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(193..193 src has 13..193) v4 ==== 7735+0+0 (secure 0 0 0) 0x7f62e404cc00 con 0x7f62d40099c0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62f103a640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62d8038490 0x7f62d803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.045+0000 7f62f103a640 10 monclient: get_auth_request con 0x7f62d8038490 auth_method 0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.047+0000 7f62f103a640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62d8038490 0x7f62d803a940 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f62e0006e70 tx=0x7f62e0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.049+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f62e4024ae0 con 0x7f62d40099c0 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.049+0000 7f62e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:36.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.049+0000 7f62e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:36.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.208+0000 7f62f3a9f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:36.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.208+0000 7f62f3a9f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:36.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.208+0000 7f62f3a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.208+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f62ec04bf60 con 0x7f62d40099c0 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.257+0000 7f685dd9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.258+0000 7f685dd9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.258+0000 7f685dd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.258+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6858120c50 con 0x7f6858131ab0 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.258+0000 7f683f7fe640 1 -- 172.21.15.116:0/1417626029 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f6850016020 con 0x7f6858131ab0 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.258+0000 7f683f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.258+0000 7f683f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:36.260 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":76,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6844038490 msgr2=0x7f684403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6844038490 0x7f684403a940 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6848006a70 tx=0x7f6848008040 comp rx=0 tx=0).stop 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 10 monclient: shutdown 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 msgr2=0x7f685811d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 0x7f685811d8d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f685000c8f0 tx=0x7f6850002640 comp rx=0 tx=0).stop 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 shutdown_connections 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6844038490 0x7f684403a940 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6858131ab0 0x7f685811d8d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f685809f250 0x7f685811d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 --2- 172.21.15.116:0/1417626029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f685809e910 0x7f685811ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 >> 172.21.15.116:0/1417626029 conn(0x7f6858067d30 msgr2=0x7f68581310a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 shutdown_connections 2024-09-18T12:32:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:36.260+0000 7f685dd9d640 1 -- 172.21.15.116:0/1417626029 wait complete. 2024-09-18T12:32:36.854 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.853+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v194) v1 ==== 166+0+0 (secure 0 0 0) 0x7f62e400b0d0 con 0x7f62d40099c0 2024-09-18T12:32:36.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.853+0000 7f62e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:36.855 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.853+0000 7f62e9ffb640 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-18T12:32:36.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.950+0000 7f62f3a9f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:36.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.950+0000 7f62f3a9f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:36.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.950+0000 7f62f3a9f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:36.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.950+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f62ec136210 con 0x7f62d40099c0 2024-09-18T12:32:36.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.950+0000 7f62e9ffb640 1 -- 172.21.15.144:0/497409653 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v194) v1 ==== 145+0+0 (secure 0 0 0) 0x7f62e4024300 con 0x7f62d40099c0 2024-09-18T12:32:36.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.950+0000 7f62e9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:36.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.950+0000 7f62e9ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-18T12:32:36.952 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h1' already exists 2024-09-18T12:32:36.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.952+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62d8038490 msgr2=0x7f62d803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.952+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62d8038490 0x7f62d803a940 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f62e0006e70 tx=0x7f62e0008040 comp rx=0 tx=0).stop 2024-09-18T12:32:36.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.952+0000 7f62f3a9f640 10 monclient: shutdown 2024-09-18T12:32:36.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.952+0000 7f62f3a9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:36.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.952+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 msgr2=0x7f62ec130910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:36.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.952+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62ec130910 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f62e4007730 tx=0x7f62ec06a350 comp rx=0 tx=0).stop 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 shutdown_connections 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62d8038490 0x7f62d803a940 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f62ec099d10 0x7f62ec09a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec130e50 0x7f62ec0997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 --2- 172.21.15.144:0/497409653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f62d40099c0 0x7f62ec130910 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 >> 172.21.15.144:0/497409653 conn(0x7f62ec0713b0 msgr2=0x7f62ec0a3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 shutdown_connections 2024-09-18T12:32:36.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:36.953+0000 7f62f3a9f640 1 -- 172.21.15.144:0/497409653 wait complete. 2024-09-18T12:32:36.963 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:36.963 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-18T12:32:37.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.037+0000 7f648e39d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:37.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.037+0000 7f648e39d640 10 monclient: build_initial_monmap 2024-09-18T12:32:37.038 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.037+0000 7f648e39d640 10 monclient: monmap: 2024-09-18T12:32:37.038 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:37.038 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:37.038 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:37.038219+0000 2024-09-18T12:32:37.038 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:37.038219+0000 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:37.039 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 1 Processor -- start 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 1 -- start start 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient: init 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6488096a30 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f648809d530 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 0x7f6488097510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.041 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient(hunting): picked mon.noname-a con 0x7f6488099d70 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648809a1d0 con 0x7f6488096a30 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6488097e80 con 0x7f648809d530 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6488097a70 con 0x7f6488099d70 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648e39d640 10 monclient(hunting): authenticate will time out at 1192.607633s 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f64877fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f648809d530 0x7f6488133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:37392/0 (socket says 172.21.15.144:37392) 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f64877fe640 1 -- 172.21.15.144:0/4283809023 learned_addr learned my addr 172.21.15.144:0/4283809023 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.038+0000 7f648c93a640 1 --2- 172.21.15.144:0/4283809023 >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 0x7f6488097510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 10 monclient(hunting): get_auth_request con 0x7f6488099d70 auth_method 0 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:37.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 1 -- 172.21.15.144:0/4283809023 >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 msgr2=0x7f6488097510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:37.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 1 -- 172.21.15.144:0/4283809023 >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 msgr2=0x7f6488097510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:37.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 1 --2- 172.21.15.144:0/4283809023 >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 0x7f6488097510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:37.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.039+0000 7f648c93a640 1 --2- 172.21.15.144:0/4283809023 >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 0x7f6488097510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:37.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.040+0000 7f64877fe640 1 -- 172.21.15.144:0/4283809023 >> v1:172.21.15.144:6789/0 conn(0x7f648809d530 legacy=0x7f6488133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:37.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740850399 0 0) 0x7f6488097e80 con 0x7f648809d530 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): my global_id is 6136 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64700034e0 con 0x7f648809d530 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 -- 172.21.15.144:0/4283809023 >> v1:172.21.15.116:6789/0 conn(0x7f6488096a30 legacy=0x7f6488096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1277308979 0 0) 0x7f64700034e0 con 0x7f648809d530 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 msgr2=0x7f6488097510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 --2- 172.21.15.144:0/4283809023 >> v2:172.21.15.116:3300/0 conn(0x7f6488099d70 0x7f6488097510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 >> v1:172.21.15.116:6789/0 conn(0x7f6488096a30 legacy=0x7f6488096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:32:37.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6488098be0 con 0x7f648809d530 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f647c0017a0 con 0x7f648809d530 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648e39d640 5 monclient: authenticate success, global_id 6136 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648e39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: dump: 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:37.054 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 >> v1:172.21.15.144:6789/0 conn(0x7f648809d530 legacy=0x7f6488133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f6470003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): picked mon.b con 0x7f6470003bc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6470004780 0x7f6470004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): picked mon.c con 0x7f6470004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64700099c0 0x7f647000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): picked mon.a con 0x7f64700099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6488097a70 con 0x7f64700099c0 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647000c420 con 0x7f6470003bc0 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647000c5e0 con 0x7f6470004780 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:37.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6486ffd640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648e39d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f6470003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): get_auth_request con 0x7f6470003bc0 auth_method 0 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6470004780 0x7f6470004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 10 monclient(hunting): get_auth_request con 0x7f6470004780 auth_method 0 2024-09-18T12:32:37.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648c93a640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64700099c0 0x7f647000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648c93a640 10 monclient(hunting): get_auth_request con 0x7f64700099c0 auth_method 0 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648c93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648c93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648c93a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f64877fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient(hunting): handle_auth_done global_id 6136 payload 293 2024-09-18T12:32:37.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 -- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6470004780 msgr2=0x7f6470004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6470004780 0x7f6470004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 -- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64700099c0 msgr2=0x7f647000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64700099c0 0x7f647000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 monclient: found mon.b 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 1 -- 172.21.15.144:0/4283809023 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700034e0 con 0x7f6470003bc0 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 10 monclient: _finish_auth 0 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f6487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648c93a640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64700099c0 0x7f647000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:37.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.041+0000 7f648c93a640 1 -- 172.21.15.144:0/4283809023 reap_dead start 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6487fff640 1 -- 172.21.15.144:0/4283809023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 msgr2=0x7f6470003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6487fff640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f6470003fd0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6478001b10 tx=0x7f6478007c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b633e73d3a61956 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647802ab50 con 0x7f6470003bc0 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 10 monclient: dump: 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:37.060 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f648e39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647802acb0 con 0x7f6470003bc0 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f648e39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f648e39d640 4 set_mon_vals no callback set 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 1 -- 172.21.15.144:0/4283809023 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478007340 con 0x7f6470003bc0 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.044+0000 7f6486ffd640 10 monclient: dump: 2024-09-18T12:32:37.061 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 10 monclient: shutdown 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 1 -- 172.21.15.144:0/4283809023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 msgr2=0x7f6470003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f6470003fd0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6478001b10 tx=0x7f6478007c80 comp rx=0 tx=0).stop 2024-09-18T12:32:37.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 1 -- 172.21.15.144:0/4283809023 shutdown_connections 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 1 --2- 172.21.15.144:0/4283809023 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f6470003fd0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 1 -- 172.21.15.144:0/4283809023 >> 172.21.15.144:0/4283809023 conn(0x7f64880a0fe0 msgr2=0x7f64880a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 1 -- 172.21.15.144:0/4283809023 shutdown_connections 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 1 -- 172.21.15.144:0/4283809023 wait complete. 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 10 monclient: build_initial_monmap 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.045+0000 7f648e39d640 10 monclient: monmap: 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:37.063 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 Processor -- start 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 -- start start 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient: init 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f648813f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): picked mon.noname-b con 0x7f6470003bc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 0x7f648812ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): picked mon.noname-a con 0x7f648812eb50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f648811a6a0 0x7f648811ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): picked mon.noname-c con 0x7f648811a6a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648811e250 con 0x7f648812eb50 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648811c2a0 con 0x7f6470003bc0 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648812f460 con 0x7f648811a6a0 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f648e39d640 10 monclient(hunting): authenticate will time out at 1192.615863s 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f6487fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f648813f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f6487fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f648813f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:43016/0 (socket says 172.21.15.144:43016) 2024-09-18T12:32:37.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f6487fff640 1 -- 172.21.15.144:0/2101616924 learned_addr learned my addr 172.21.15.144:0/2101616924 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f6487fff640 10 monclient(hunting): get_auth_request con 0x7f6470003bc0 auth_method 0 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f6487fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f6487fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.046+0000 7f6487fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 0x7f648812ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f648811a6a0 0x7f648811ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 10 monclient(hunting): get_auth_request con 0x7f648812eb50 auth_method 0 2024-09-18T12:32:37.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 10 monclient(hunting): get_auth_request con 0x7f648811a6a0 auth_method 0 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 1 -- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 msgr2=0x7f648812ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 1 -- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 msgr2=0x7f648812ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f648c93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 0x7f648812ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:37.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f64877fe640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 0x7f648812ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 10 monclient(hunting): handle_auth_done global_id 6139 payload 824 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 1 -- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f648811a6a0 msgr2=0x7f648811ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f648811a6a0 0x7f648811ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 1 -- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 msgr2=0x7f648812ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 0x7f648812ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.047+0000 7f6487fff640 1 -- 172.21.15.144:0/2101616924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648811df70 con 0x7f6470003bc0 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f6487fff640 1 -- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 msgr2=0x7f648813f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:37.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f6487fff640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f648813f0a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6478007790 tx=0x7f6478030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647803b070 con 0x7f6470003bc0 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: dump: 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:37.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: _finish_auth 0 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6478033470 con 0x7f6470003bc0 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f648e39d640 5 monclient: authenticate success, global_id 6139 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f648e39d640 10 monclient: _renew_subs 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f648e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648813ff00 con 0x7f6470003bc0 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64780337f0 con 0x7f6470003bc0 2024-09-18T12:32:37.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f646ffff640 10 monclient: dump: 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:37.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f648e39d640 10 monclient: _renew_subs 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f648e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f64867fc640 4 set_mon_vals no callback set 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.049+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648811cd30 con 0x7f6470003bc0 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f647803e070 con 0x7f6470003bc0 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f648e39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f648e39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f648e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648804bf60 con 0x7f6470003bc0 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f646ffff640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6474038490 0x7f647403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(194..194 src has 13..194) v4 ==== 7945+0+0 (secure 0 0 0) 0x7f647806b620 con 0x7f6470003bc0 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f64877fe640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6474038490 0x7f647403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:37.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.050+0000 7f64877fe640 10 monclient: get_auth_request con 0x7f6474038490 auth_method 0 2024-09-18T12:32:37.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.053+0000 7f64877fe640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6474038490 0x7f647403a940 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f647c001950 tx=0x7f647c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:37.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.055+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f647803fde0 con 0x7f6470003bc0 2024-09-18T12:32:37.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.055+0000 7f646ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:37.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.055+0000 7f646ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:37.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.212+0000 7f648e39d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:37.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.212+0000 7f648e39d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:37.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.212+0000 7f648e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:37.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.212+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f648809a1d0 con 0x7f6470003bc0 2024-09-18T12:32:37.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.856+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v195) v1 ==== 166+0+0 (secure 0 0 0) 0x7f648809a1d0 con 0x7f6470003bc0 2024-09-18T12:32:37.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.856+0000 7f646ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:37.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.856+0000 7f646ffff640 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-18T12:32:37.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.952+0000 7f648e39d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:37.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.952+0000 7f648e39d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:37.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.952+0000 7f648e39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:37.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.952+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f648809a1d0 con 0x7f6470003bc0 2024-09-18T12:32:37.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.952+0000 7f646ffff640 1 -- 172.21.15.144:0/2101616924 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v195) v1 ==== 145+0+0 (secure 0 0 0) 0x7f648809a1d0 con 0x7f6470003bc0 2024-09-18T12:32:37.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.952+0000 7f646ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:37.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.952+0000 7f646ffff640 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-18T12:32:37.954 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h2' already exists 2024-09-18T12:32:37.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6474038490 msgr2=0x7f647403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:37.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6474038490 0x7f647403a940 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f647c001950 tx=0x7f647c009040 comp rx=0 tx=0).stop 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 10 monclient: shutdown 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 msgr2=0x7f648813f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f648813f0a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6478007790 tx=0x7f6478030000 comp rx=0 tx=0).stop 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 shutdown_connections 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6474038490 0x7f647403a940 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f648811a6a0 0x7f648811ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f648812eb50 0x7f648812ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 --2- 172.21.15.144:0/2101616924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6470003bc0 0x7f648813f0a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:37.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.954+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 >> 172.21.15.144:0/2101616924 conn(0x7f64880a0fe0 msgr2=0x7f648809d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:37.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.955+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 shutdown_connections 2024-09-18T12:32:37.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:37.955+0000 7f648e39d640 1 -- 172.21.15.144:0/2101616924 wait complete. 2024-09-18T12:32:37.964 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:37.964 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-18T12:32:38.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.038+0000 7f391275d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:38.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.038+0000 7f391275d640 10 monclient: build_initial_monmap 2024-09-18T12:32:38.039 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.038+0000 7f391275d640 10 monclient: monmap: 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:38.039411+0000 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:38.039411+0000 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:38.040 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:38.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 1 Processor -- start 2024-09-18T12:32:38.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 1 -- start start 2024-09-18T12:32:38.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient: init 2024-09-18T12:32:38.043 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c098050 0x7f390c132000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient(hunting): picked mon.noname-b con 0x7f390c098050 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f390c09fe90 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f390c136cf0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390c0a1230 con 0x7f390c136cf0 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390c0a1510 con 0x7f390c09fe90 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c06a230 con 0x7f390c098050 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.039+0000 7f391275d640 10 monclient(hunting): authenticate will time out at 1193.608970s 2024-09-18T12:32:38.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390b7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f390c09fe90 0x7f390c098420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58850/0 (socket says 172.21.15.144:58850) 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390b7fe640 1 -- 172.21.15.144:0/4013351224 learned_addr learned my addr 172.21.15.144:0/4013351224 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 1 --2- 172.21.15.144:0/4013351224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c098050 0x7f390c132000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): get_auth_request con 0x7f390c098050 auth_method 0 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:38.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient(hunting): handle_auth_done global_id 64934 payload 274 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 1 -- 172.21.15.144:0/4013351224 >> v1:172.21.15.144:6789/0 conn(0x7f390c09fe90 legacy=0x7f390c098420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 1 -- 172.21.15.144:0/4013351224 >> v1:172.21.15.116:6789/0 conn(0x7f390c136cf0 legacy=0x7f390c1390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 1 monclient: found mon.noname-b 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:38.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:38.047 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.040+0000 7f390bfff640 1 -- 172.21.15.144:0/4013351224 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c098b30 con 0x7f390c098050 2024-09-18T12:32:38.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390bfff640 1 -- 172.21.15.144:0/4013351224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c098050 msgr2=0x7f390c132000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:38.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390bfff640 1 --2- 172.21.15.144:0/4013351224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c098050 0x7f390c132000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009880 tx=0x7f38fc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb5756e9b8cd4c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:38.055 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390affd640 1 -- 172.21.15.144:0/4013351224 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc02aea0 con 0x7f390c098050 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390affd640 10 monclient: dump: 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:38.056 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390affd640 10 monclient: _finish_auth 0 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f390affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f391275d640 5 monclient: authenticate success, global_id 64934 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.043+0000 7f391275d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f390affd640 1 -- 172.21.15.144:0/4013351224 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc032440 con 0x7f390c098050 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f390affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f390affd640 1 -- 172.21.15.144:0/4013351224 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0327e0 con 0x7f390c098050 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 4 set_mon_vals no callback set 2024-09-18T12:32:38.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f390affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f390affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f390affd640 10 monclient: dump: 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:38.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 10 monclient: shutdown 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 1 -- 172.21.15.144:0/4013351224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c098050 msgr2=0x7f390c132000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 1 --2- 172.21.15.144:0/4013351224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c098050 0x7f390c132000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009880 tx=0x7f38fc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 1 -- 172.21.15.144:0/4013351224 shutdown_connections 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 1 --2- 172.21.15.144:0/4013351224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c098050 0x7f390c132000 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 1 -- 172.21.15.144:0/4013351224 >> 172.21.15.144:0/4013351224 conn(0x7f390c0a5190 msgr2=0x7f390c0a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 1 -- 172.21.15.144:0/4013351224 shutdown_connections 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.044+0000 7f391275d640 1 -- 172.21.15.144:0/4013351224 wait complete. 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.045+0000 7f391275d640 10 monclient: build_initial_monmap 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.045+0000 7f391275d640 10 monclient: monmap: 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:38.059 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.045+0000 7f391275d640 1 Processor -- start 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.045+0000 7f391275d640 1 -- start start 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.045+0000 7f391275d640 10 monclient: init 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.045+0000 7f391275d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:38.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 0x7f390c130ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): picked mon.noname-b con 0x7f390c098050 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 0x7f390c134840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): picked mon.noname-a con 0x7f390c09fe90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c136cf0 0x7f390c134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): picked mon.noname-c con 0x7f390c136cf0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c098d70 con 0x7f390c09fe90 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c141fc0 con 0x7f390c098050 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:38.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0a3600 con 0x7f390c136cf0 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f391275d640 10 monclient(hunting): authenticate will time out at 1193.615087s 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 0x7f390c130ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 0x7f390c130ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54268/0 (socket says 172.21.15.144:54268) 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 -- 172.21.15.144:0/1105499514 learned_addr learned my addr 172.21.15.144:0/1105499514 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): get_auth_request con 0x7f390c098050 auth_method 0 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:38.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 0x7f390c134840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f3910cfa640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c136cf0 0x7f390c134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 10 monclient(hunting): get_auth_request con 0x7f390c09fe90 auth_method 0 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f3910cfa640 10 monclient(hunting): get_auth_request con 0x7f390c136cf0 auth_method 0 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f3910cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f3910cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f3910cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 1 -- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 msgr2=0x7f390c134840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient(hunting): handle_auth_done global_id 6142 payload 824 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 1 -- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 msgr2=0x7f390c134840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:38.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 0x7f390c134840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390b7fe640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 0x7f390c134840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 -- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c136cf0 msgr2=0x7f390c134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c136cf0 0x7f390c134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 -- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 msgr2=0x7f390c134840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 0x7f390c134840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 monclient: found mon.noname-b 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f390bfff640 1 -- 172.21.15.144:0/1105499514 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc009390 con 0x7f390c098050 2024-09-18T12:32:38.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.046+0000 7f3910cfa640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c136cf0 0x7f390c134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f390bfff640 1 -- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 msgr2=0x7f390c130ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f390bfff640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 0x7f390c130ac0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0097d0 tx=0x7f38fc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc02aea0 con 0x7f390c098050 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: dump: 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:38.065 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: _finish_auth 0 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc031990 con 0x7f390c098050 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f391275d640 5 monclient: authenticate success, global_id 6142 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f391275d640 10 monclient: _renew_subs 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f391275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c1363e0 con 0x7f390c098050 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc031d50 con 0x7f390c098050 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:38.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f38f3fff640 10 monclient: dump: 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:38.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f390a7fc640 4 set_mon_vals no callback set 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f391275d640 10 monclient: _renew_subs 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f391275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.048+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c137590 con 0x7f390c098050 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.049+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f38fc039070 con 0x7f390c098050 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.049+0000 7f391275d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.049+0000 7f391275d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.049+0000 7f391275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.049+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c04bf60 con 0x7f390c098050 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.049+0000 7f38f3fff640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f38f8038490 0x7f38f803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.050+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(195..195 src has 13..195) v4 ==== 8155+0+0 (secure 0 0 0) 0x7f38fc06c8f0 con 0x7f390c098050 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.050+0000 7f390b7fe640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f38f8038490 0x7f38f803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.050+0000 7f390b7fe640 10 monclient: get_auth_request con 0x7f38f8038490 auth_method 0 2024-09-18T12:32:38.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.052+0000 7f390b7fe640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f38f8038490 0x7f38f803a940 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f390c135f00 tx=0x7f3900009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:38.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.054+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f38fc030b20 con 0x7f390c098050 2024-09-18T12:32:38.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.054+0000 7f38f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:38.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.054+0000 7f38f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:38.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.211+0000 7f391275d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:38.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.211+0000 7f391275d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:38.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.211+0000 7f391275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:38.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:38.211+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f390c1363e0 con 0x7f390c098050 2024-09-18T12:32:39.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:39.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.374+0000 7f244671d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:39.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.374+0000 7f244671d640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.374+0000 7f244671d640 10 monclient: monmap: 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:39.376453+0000 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:39.376453+0000 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:39.377 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:39.378 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.375+0000 7f244671d640 1 Processor -- start 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.375+0000 7f244671d640 1 -- start start 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.375+0000 7f244671d640 10 monclient: init 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 0x7f244012ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): picked mon.noname-a con 0x7f24400974f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244009a8a0 0x7f244009acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): picked mon.noname-b con 0x7f244009a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f244012dd40 0x7f244012e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): picked mon.noname-c con 0x7f244012dd40 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24400978c0 con 0x7f24400974f0 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244009ca20 con 0x7f244012dd40 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244009b1f0 con 0x7f244009a8a0 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f244671d640 10 monclient(hunting): authenticate will time out at 1186.698462s 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 0x7f244012ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 0x7f244012ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43890/0 (socket says 172.21.15.116:43890) 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 1 -- 172.21.15.116:0/846141152 learned_addr learned my addr 172.21.15.116:0/846141152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 10 monclient(hunting): get_auth_request con 0x7f24400974f0 auth_method 0 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.144:3300/0 conn(0x7f244012dd40 0x7f244012e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 10 monclient(hunting): get_auth_request con 0x7f244012dd40 auth_method 0 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 1 --2- 172.21.15.116:0/846141152 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244009a8a0 0x7f244009acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 10 monclient(hunting): get_auth_request con 0x7f244009a8a0 auth_method 0 2024-09-18T12:32:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f2444cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 1 -- 172.21.15.116:0/846141152 >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 msgr2=0x7f244012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 1 -- 172.21.15.116:0/846141152 >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 msgr2=0x7f244012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 0x7f244012ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243ffff640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 0x7f244012ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.376+0000 7f243f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 10 monclient(hunting): handle_auth_done global_id 64940 payload 274 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 1 -- 172.21.15.116:0/846141152 >> v2:172.21.15.144:3300/0 conn(0x7f244012dd40 msgr2=0x7f244012e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.144:3300/0 conn(0x7f244012dd40 0x7f244012e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 1 -- 172.21.15.116:0/846141152 >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 msgr2=0x7f244012ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 0x7f244012ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:39.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f243f7fe640 1 -- 172.21.15.116:0/846141152 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440098ad0 con 0x7f244009a8a0 2024-09-18T12:32:39.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.377+0000 7f2444cba640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.144:3300/0 conn(0x7f244012dd40 0x7f244012e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:39.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.380+0000 7f243f7fe640 1 -- 172.21.15.116:0/846141152 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244009a8a0 msgr2=0x7f244009acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.380+0000 7f243f7fe640 1 --2- 172.21.15.116:0/846141152 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244009a8a0 0x7f244009acb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2440065310 tx=0x7f243400b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5054e6eeaadede6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 1 -- 172.21.15.116:0/846141152 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2434009760 con 0x7f244009a8a0 2024-09-18T12:32:39.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: dump: 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 1 -- 172.21.15.116:0/846141152 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243402f050 con 0x7f244009a8a0 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f244671d640 5 monclient: authenticate success, global_id 64940 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f244671d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f244671d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f244671d640 4 set_mon_vals no callback set 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 1 -- 172.21.15.116:0/846141152 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2434009af0 con 0x7f244009a8a0 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.382+0000 7f243effd640 10 monclient: dump: 2024-09-18T12:32:39.392 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 10 monclient: shutdown 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 -- 172.21.15.116:0/846141152 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244009a8a0 msgr2=0x7f244009acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 --2- 172.21.15.116:0/846141152 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244009a8a0 0x7f244009acb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2440065310 tx=0x7f243400b780 comp rx=0 tx=0).stop 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 -- 172.21.15.116:0/846141152 shutdown_connections 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.144:3300/0 conn(0x7f244012dd40 0x7f244012e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 --2- 172.21.15.116:0/846141152 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244009a8a0 0x7f244009acb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 --2- 172.21.15.116:0/846141152 >> v2:172.21.15.116:3300/0 conn(0x7f24400974f0 0x7f244012ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 -- 172.21.15.116:0/846141152 >> 172.21.15.116:0/846141152 conn(0x7f24400672b0 msgr2=0x7f24400676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 -- 172.21.15.116:0/846141152 shutdown_connections 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.383+0000 7f244671d640 1 -- 172.21.15.116:0/846141152 wait complete. 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.384+0000 7f244671d640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.384+0000 7f244671d640 10 monclient: monmap: 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:39.394 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.384+0000 7f244671d640 1 Processor -- start 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.384+0000 7f244671d640 1 -- start start 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.384+0000 7f244671d640 10 monclient: init 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24400974f0 0x7f244006cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): picked mon.noname-b con 0x7f24400974f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:39.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244009a8a0 0x7f244006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): picked mon.noname-a con 0x7f244009a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 0x7f244006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): picked mon.noname-c con 0x7f244012dd40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24401189f0 con 0x7f244009a8a0 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440118610 con 0x7f24400974f0 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2440128340 con 0x7f244012dd40 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f244671d640 10 monclient(hunting): authenticate will time out at 1186.707323s 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244009a8a0 0x7f244006d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 0x7f244006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24400974f0 0x7f244006cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244009a8a0 0x7f244006d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43898/0 (socket says 172.21.15.116:43898) 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 1 -- 172.21.15.116:0/2102362199 learned_addr learned my addr 172.21.15.116:0/2102362199 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 10 monclient(hunting): get_auth_request con 0x7f244009a8a0 auth_method 0 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 0x7f244006d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42894/0 (socket says 172.21.15.116:42894) 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): get_auth_request con 0x7f244012dd40 auth_method 0 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 10 monclient(hunting): get_auth_request con 0x7f24400974f0 auth_method 0 2024-09-18T12:32:39.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient(hunting): handle_auth_done global_id 64943 payload 824 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 -- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24400974f0 msgr2=0x7f244006cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24400974f0 0x7f244006cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 -- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244009a8a0 msgr2=0x7f244006d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244009a8a0 0x7f244006d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 monclient: found mon.noname-c 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f2444cba640 1 -- 172.21.15.116:0/2102362199 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2438009a20 con 0x7f244012dd40 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.385+0000 7f243f7fe640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244009a8a0 0x7f244006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.388+0000 7f2444cba640 1 -- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 msgr2=0x7f244006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.388+0000 7f2444cba640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 0x7f244006d9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2438009f10 tx=0x7f243800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 1 -- 172.21.15.116:0/2102362199 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438014480 con 0x7f244012dd40 2024-09-18T12:32:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: dump: 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 1 -- 172.21.15.116:0/2102362199 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243800f040 con 0x7f244012dd40 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f244671d640 5 monclient: authenticate success, global_id 64943 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f244671d640 10 monclient: _renew_subs 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 1 -- 172.21.15.116:0/2102362199 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438014850 con 0x7f244012dd40 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f244671d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243400ba20 con 0x7f244012dd40 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f2427fff640 10 monclient: dump: 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f243e7fc640 4 set_mon_vals no callback set 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f244671d640 10 monclient: _renew_subs 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f244671d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.389+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244009df50 con 0x7f244012dd40 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f2427fff640 1 -- 172.21.15.116:0/2102362199 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2438003020 con 0x7f244012dd40 2024-09-18T12:32:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f2427fff640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f242c0384e0 0x7f242c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f2427fff640 1 -- 172.21.15.116:0/2102362199 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(195..195 src has 13..195) v4 ==== 8155+0+0 (secure 0 0 0) 0x7f243804cf70 con 0x7f244012dd40 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f243ffff640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f242c0384e0 0x7f242c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f243ffff640 10 monclient: get_auth_request con 0x7f242c0384e0 auth_method 0 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f244671d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f244671d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f244671d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.390+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244004bf60 con 0x7f244012dd40 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.393+0000 7f243ffff640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f242c0384e0 0x7f242c03a990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2430006a70 tx=0x7f2430008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.395+0000 7f2427fff640 1 -- 172.21.15.116:0/2102362199 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f24380112c0 con 0x7f244012dd40 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.395+0000 7f2427fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.395+0000 7f2427fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:39.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.439+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v196) v1 ==== 166+0+0 (secure 0 0 0) 0x7f390c1363e0 con 0x7f390c098050 2024-09-18T12:32:39.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.439+0000 7f38f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:39.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.439+0000 7f38f3fff640 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-18T12:32:39.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.540+0000 7f391275d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:39.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.540+0000 7f391275d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:39.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.540+0000 7f391275d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:39.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.540+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f390c136380 con 0x7f390c098050 2024-09-18T12:32:39.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.541+0000 7f38f3fff640 1 -- 172.21.15.144:0/1105499514 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v196) v1 ==== 145+0+0 (secure 0 0 0) 0x7f38fc037030 con 0x7f390c098050 2024-09-18T12:32:39.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.541+0000 7f38f3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:39.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.541+0000 7f38f3fff640 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-18T12:32:39.543 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h3' already exists 2024-09-18T12:32:39.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.543+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f38f8038490 msgr2=0x7f38f803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.543+0000 7f391275d640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f38f8038490 0x7f38f803a940 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f390c135f00 tx=0x7f3900009040 comp rx=0 tx=0).stop 2024-09-18T12:32:39.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.543+0000 7f391275d640 10 monclient: shutdown 2024-09-18T12:32:39.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.543+0000 7f391275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.545 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.543+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 msgr2=0x7f390c130ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.543+0000 7f391275d640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 0x7f390c130ac0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0097d0 tx=0x7f38fc032000 comp rx=0 tx=0).stop 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 shutdown_connections 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f38f8038490 0x7f38f803a940 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f390c136cf0 0x7f390c134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f390c09fe90 0x7f390c134840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 --2- 172.21.15.144:0/1105499514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f390c098050 0x7f390c130ac0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 >> 172.21.15.144:0/1105499514 conn(0x7f390c0a5190 msgr2=0x7f390c09e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 shutdown_connections 2024-09-18T12:32:39.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.544+0000 7f391275d640 1 -- 172.21.15.144:0/1105499514 wait complete. 2024-09-18T12:32:39.554 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:39.554 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-18T12:32:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.613+0000 7f244671d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.613+0000 7f244671d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.613+0000 7f244671d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.613+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f24400638d0 con 0x7f244012dd40 2024-09-18T12:32:39.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.613+0000 7f2427fff640 1 -- 172.21.15.116:0/2102362199 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2438012340 con 0x7f244012dd40 2024-09-18T12:32:39.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.613+0000 7f2427fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:39.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.613+0000 7f2427fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:39.616 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:39.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f242c0384e0 msgr2=0x7f242c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f242c0384e0 0x7f242c03a990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2430006a70 tx=0x7f2430008040 comp rx=0 tx=0).stop 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 10 monclient: shutdown 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 msgr2=0x7f244006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 0x7f244006d9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2438009f10 tx=0x7f243800b040 comp rx=0 tx=0).stop 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 shutdown_connections 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f242c0384e0 0x7f242c03a990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f244012dd40 0x7f244006d9f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f244009a8a0 0x7f244006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 --2- 172.21.15.116:0/2102362199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24400974f0 0x7f244006cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 >> 172.21.15.116:0/2102362199 conn(0x7f24400672b0 msgr2=0x7f244012d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 shutdown_connections 2024-09-18T12:32:39.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.615+0000 7f244671d640 1 -- 172.21.15.116:0/2102362199 wait complete. 2024-09-18T12:32:39.627 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:32:39.627 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:32:39.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.629+0000 7fac6986f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.629+0000 7fac6986f640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.629+0000 7fac6986f640 10 monclient: monmap: 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:39.630382+0000 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:39.630382+0000 2024-09-18T12:32:39.634 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.635 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:39.635 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:39.635 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:39.635 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:39.635 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:39.635 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 Processor -- start 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 -- start start 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient: init 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fac64131bd0 0x7fac64131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): picked mon.noname-a con 0x7fac64131bd0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:39.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fac6409ebe0 0x7fac64068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): picked mon.noname-c con 0x7fac6409ebe0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:39.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac6409efe0 0x7fac64069340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): picked mon.noname-b con 0x7fac6409efe0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:39.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac640a06d0 con 0x7fac64131bd0 2024-09-18T12:32:39.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac64071780 con 0x7fac6409ebe0 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac640a0010 con 0x7fac6409efe0 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac6986f640 10 monclient(hunting): authenticate will time out at 1195.199556s 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fac6409ebe0 0x7fac64068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fac6409ebe0 0x7fac64068e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54284/0 (socket says 172.21.15.144:54284) 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 1 -- 172.21.15.144:0/1122854578 learned_addr learned my addr 172.21.15.144:0/1122854578 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): get_auth_request con 0x7fac6409ebe0 auth_method 0 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 1 --2- 172.21.15.144:0/1122854578 >> v2:172.21.15.116:3300/0 conn(0x7fac64131bd0 0x7fac64131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 1 --2- 172.21.15.144:0/1122854578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac6409efe0 0x7fac64069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 10 monclient(hunting): get_auth_request con 0x7fac6409efe0 auth_method 0 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 10 monclient(hunting): get_auth_request con 0x7fac64131bd0 auth_method 0 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 10 monclient(hunting): handle_auth_done global_id 6151 payload 274 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 1 -- 172.21.15.144:0/1122854578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac6409efe0 msgr2=0x7fac64069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.630+0000 7fac627fc640 1 --2- 172.21.15.144:0/1122854578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac6409efe0 0x7fac64069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.631+0000 7fac627fc640 1 -- 172.21.15.144:0/1122854578 >> v2:172.21.15.116:3300/0 conn(0x7fac64131bd0 msgr2=0x7fac64131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:39.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.631+0000 7fac627fc640 1 --2- 172.21.15.144:0/1122854578 >> v2:172.21.15.116:3300/0 conn(0x7fac64131bd0 0x7fac64131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.631+0000 7fac627fc640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.631+0000 7fac627fc640 1 monclient: found mon.noname-c 2024-09-18T12:32:39.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.631+0000 7fac627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.631+0000 7fac627fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:39.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.631+0000 7fac627fc640 1 -- 172.21.15.144:0/1122854578 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac640a1910 con 0x7fac6409ebe0 2024-09-18T12:32:39.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac627fc640 1 -- 172.21.15.144:0/1122854578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 msgr2=0x7fac64068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac627fc640 1 --2- 172.21.15.144:0/1122854578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac64068e00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fac64068d90 tx=0x7fac580091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c5653beddda163 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 1 -- 172.21.15.144:0/1122854578 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58032440 con 0x7fac6409ebe0 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 10 monclient: dump: 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.651 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:39.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 1 -- 172.21.15.144:0/1122854578 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac580325a0 con 0x7fac6409ebe0 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac6986f640 5 monclient: authenticate success, global_id 6151 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac6986f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac6986f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 1 -- 172.21.15.144:0/1122854578 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58032940 con 0x7fac6409ebe0 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac6986f640 4 set_mon_vals no callback set 2024-09-18T12:32:39.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.633+0000 7fac61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac61ffb640 10 monclient: dump: 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.654 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 10 monclient: shutdown 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 -- 172.21.15.144:0/1122854578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 msgr2=0x7fac64068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 --2- 172.21.15.144:0/1122854578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac64068e00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fac64068d90 tx=0x7fac580091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 -- 172.21.15.144:0/1122854578 shutdown_connections 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 --2- 172.21.15.144:0/1122854578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac6409efe0 0x7fac64069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 --2- 172.21.15.144:0/1122854578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac64068e00 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 --2- 172.21.15.144:0/1122854578 >> v2:172.21.15.116:3300/0 conn(0x7fac64131bd0 0x7fac64131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 -- 172.21.15.144:0/1122854578 >> 172.21.15.144:0/1122854578 conn(0x7fac640713b0 msgr2=0x7fac640a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 -- 172.21.15.144:0/1122854578 shutdown_connections 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.634+0000 7fac6986f640 1 -- 172.21.15.144:0/1122854578 wait complete. 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient: monmap: 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:39.657 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 Processor -- start 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 -- start start 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient: init 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac6409a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): picked mon.noname-b con 0x7fac6409ebe0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac6409efe0 0x7fac641309a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): picked mon.noname-a con 0x7fac6409efe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac64131bd0 0x7fac64130ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): picked mon.noname-c con 0x7fac64131bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6411dda0 con 0x7fac6409efe0 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac641361c0 con 0x7fac6409ebe0 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6411c890 con 0x7fac64131bd0 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.635+0000 7fac6986f640 10 monclient(hunting): authenticate will time out at 1195.204936s 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac6409a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac6409a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54288/0 (socket says 172.21.15.144:54288) 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 -- 172.21.15.144:0/1356790577 learned_addr learned my addr 172.21.15.144:0/1356790577 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): get_auth_request con 0x7fac6409ebe0 auth_method 0 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac64131bd0 0x7fac64130ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac6409efe0 0x7fac641309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 10 monclient(hunting): get_auth_request con 0x7fac64131bd0 auth_method 0 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 10 monclient(hunting): get_auth_request con 0x7fac6409efe0 auth_method 0 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient(hunting): handle_auth_done global_id 6154 payload 824 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 -- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac64131bd0 msgr2=0x7fac64130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac64131bd0 0x7fac64130ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 -- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac6409efe0 msgr2=0x7fac641309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac6409efe0 0x7fac641309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 monclient: found mon.noname-b 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.636+0000 7fac62ffd640 1 -- 172.21.15.144:0/1356790577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58009450 con 0x7fac6409ebe0 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac62ffd640 1 -- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 msgr2=0x7fac6409a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac62ffd640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac6409a870 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fac5400ea20 tx=0x7fac540090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5400cc20 con 0x7fac6409ebe0 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: dump: 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.663 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac6986f640 5 monclient: authenticate success, global_id 6154 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5400cd80 con 0x7fac6409ebe0 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac6986f640 10 monclient: _renew_subs 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac6986f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:39.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6411db20 con 0x7fac6409ebe0 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54013680 con 0x7fac6409ebe0 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac4affd640 10 monclient: dump: 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:39.665 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac6986f640 10 monclient: _renew_subs 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac6986f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac64141b20 con 0x7fac6409ebe0 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.638+0000 7fac617fa640 4 set_mon_vals no callback set 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.639+0000 7fac6986f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.639+0000 7fac6986f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.639+0000 7fac6986f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.639+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6404bf60 con 0x7fac6409ebe0 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.639+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fac54013a70 con 0x7fac6409ebe0 2024-09-18T12:32:39.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.639+0000 7fac4affd640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fac500384c0 0x7fac5003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.639+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(196..196 src has 13..196) v4 ==== 8365+0+0 (secure 0 0 0) 0x7fac5404ca50 con 0x7fac6409ebe0 2024-09-18T12:32:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.640+0000 7fac627fc640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fac500384c0 0x7fac5003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.640+0000 7fac627fc640 10 monclient: get_auth_request con 0x7fac500384c0 auth_method 0 2024-09-18T12:32:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.642+0000 7fac627fc640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fac500384c0 0x7fac5003a970 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac64068370 tx=0x7fac58032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.644+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fac540123d0 con 0x7fac6409ebe0 2024-09-18T12:32:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.644+0000 7fac4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.644+0000 7fac4affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.719+0000 7f1f4e8f8640 10 monclient: get_monmap_and_config 2024-09-18T12:32:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.719+0000 7f1f4e8f8640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.719+0000 7f1f4e8f8640 10 monclient: monmap: 2024-09-18T12:32:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:39.721019+0000 2024-09-18T12:32:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:39.721019+0000 2024-09-18T12:32:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:39.723 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:39.723 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 1 Processor -- start 2024-09-18T12:32:39.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 1 -- start start 2024-09-18T12:32:39.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient: init 2024-09-18T12:32:39.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T12:32:39.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f480974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:39.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f4809a8a0 0x7f1f4809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4809a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f48133070 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4812b210 con 0x7f1f48133070 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f480978c0 con 0x7f1f480974f0 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4809b220 con 0x7f1f4809a8a0 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4e8f8640 10 monclient(hunting): authenticate will time out at 1187.042629s 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f4809a8a0 0x7f1f4809ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4ce95640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1f48133070 0x7f1f48135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42530/0 (socket says 172.21.15.116:42530) 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f4809a8a0 0x7f1f4809ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43902/0 (socket says 172.21.15.116:43902) 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f4ce95640 1 -- 172.21.15.116:0/2285874325 learned_addr learned my addr 172.21.15.116:0/2285874325 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:32:39.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 10 monclient(hunting): get_auth_request con 0x7f1f4809a8a0 auth_method 0 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.720+0000 7f1f477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 10 monclient(hunting): handle_auth_done global_id 104106 payload 274 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 1 -- 172.21.15.116:0/2285874325 >> v1:172.21.15.144:6789/0 conn(0x7f1f480974f0 legacy=0x7f1f4812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 1 -- 172.21.15.116:0/2285874325 >> v1:172.21.15.116:6789/0 conn(0x7f1f48133070 legacy=0x7f1f48135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 1 monclient: found mon.noname-a 2024-09-18T12:32:39.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.721+0000 7f1f477fe640 1 -- 172.21.15.116:0/2285874325 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4809ca20 con 0x7f1f4809a8a0 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.723+0000 7f1f477fe640 1 -- 172.21.15.116:0/2285874325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4809a8a0 msgr2=0x7f1f4809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.723+0000 7f1f477fe640 1 --2- 172.21.15.116:0/2285874325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4809a8a0 0x7f1f4809ace0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009850 tx=0x7f1f300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea232f7faf95d6a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 1 -- 172.21.15.116:0/2285874325 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f30032440 con 0x7f1f4809a8a0 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:39.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: dump: 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 5 monclient: authenticate success, global_id 104106 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 1 -- 172.21.15.116:0/2285874325 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f300325a0 con 0x7f1f4809a8a0 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 4 set_mon_vals no callback set 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 1 -- 172.21.15.116:0/2285874325 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f30032980 con 0x7f1f4809a8a0 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f46ffd640 10 monclient: dump: 2024-09-18T12:32:39.738 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 10 monclient: shutdown 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/2285874325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4809a8a0 msgr2=0x7f1f4809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 1 --2- 172.21.15.116:0/2285874325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4809a8a0 0x7f1f4809ace0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009850 tx=0x7f1f300090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/2285874325 shutdown_connections 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 1 --2- 172.21.15.116:0/2285874325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4809a8a0 0x7f1f4809ace0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/2285874325 >> 172.21.15.116:0/2285874325 conn(0x7f1f480672b0 msgr2=0x7f1f480676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/2285874325 shutdown_connections 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.724+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/2285874325 wait complete. 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.725+0000 7f1f4e8f8640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.725+0000 7f1f4e8f8640 10 monclient: monmap: 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.740 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.725+0000 7f1f4e8f8640 1 Processor -- start 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.725+0000 7f1f4e8f8640 1 -- start start 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.725+0000 7f1f4e8f8640 10 monclient: init 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f480974f0 0x7f1f4806ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): picked mon.noname-c con 0x7f1f480974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f4809a8a0 0x7f1f481321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): picked mon.noname-b con 0x7f1f4809a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48133070 0x7f1f4813ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): picked mon.noname-a con 0x7f1f48133070 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4812d0e0 con 0x7f1f48133070 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48098300 con 0x7f1f4809a8a0 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48099fe0 con 0x7f1f480974f0 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4e8f8640 10 monclient(hunting): authenticate will time out at 1187.048170s 2024-09-18T12:32:39.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48133070 0x7f1f4813ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f480974f0 0x7f1f4806ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48133070 0x7f1f4813ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43912/0 (socket says 172.21.15.116:43912) 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 -- 172.21.15.116:0/559429344 learned_addr learned my addr 172.21.15.116:0/559429344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): get_auth_request con 0x7f1f48133070 auth_method 0 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 10 monclient(hunting): get_auth_request con 0x7f1f480974f0 auth_method 0 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f4809a8a0 0x7f1f481321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 10 monclient(hunting): get_auth_request con 0x7f1f4809a8a0 auth_method 0 2024-09-18T12:32:39.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient(hunting): handle_auth_done global_id 104109 payload 824 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f480974f0 msgr2=0x7f1f4806ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f480974f0 0x7f1f4806ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f4809a8a0 msgr2=0x7f1f481321d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f4809a8a0 0x7f1f481321d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 monclient: found mon.noname-a 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f4ce95640 1 -- 172.21.15.116:0/559429344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f30009390 con 0x7f1f48133070 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.726+0000 7f1f47fff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f480974f0 0x7f1f4806ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f4ce95640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48133070 msgr2=0x7f1f4813ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f4ce95640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48133070 0x7f1f4813ee60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000cc20 tx=0x7f1f40005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000ce40 con 0x7f1f48133070 2024-09-18T12:32:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f3ffff640 10 monclient: dump: 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f3ffff640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f4e8f8640 5 monclient: authenticate success, global_id 104109 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f4e8f8640 10 monclient: _renew_subs 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4000aa60 con 0x7f1f48133070 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f4e8f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.728+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4812e3e0 con 0x7f1f48133070 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f3ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000ae20 con 0x7f1f48133070 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f3ffff640 10 monclient: dump: 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f467fc640 4 set_mon_vals no callback set 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f4e8f8640 10 monclient: _renew_subs 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f4e8f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4812d570 con 0x7f1f48133070 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f4e8f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1f40007300 con 0x7f1f48133070 2024-09-18T12:32:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f4e8f8640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 0x7f1f4812bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f4e8f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48065f20 con 0x7f1f48132710 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f47fff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 0x7f1f4812bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f47fff640 10 monclient: get_auth_request con 0x7f1f48132710 auth_method 0 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f47fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.729+0000 7f1f47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f3ffff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f28038490 0x7f1f2803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(196..196 src has 13..196) v4 ==== 8365+0+0 (secure 0 0 0) 0x7f1f4004c790 con 0x7f1f48133070 2024-09-18T12:32:39.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f47fff640 10 monclient(hunting): handle_auth_done global_id 104112 payload 824 2024-09-18T12:32:39.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f47fff640 1 -- 172.21.15.116:0/559429344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f4812e900 con 0x7f1f48132710 2024-09-18T12:32:39.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f477fe640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f28038490 0x7f1f2803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.730+0000 7f1f477fe640 10 monclient: get_auth_request con 0x7f1f28038490 auth_method 0 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f47fff640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 msgr2=0x7f1f4812bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f47fff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 0x7f1f4812bde0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800fa50 tx=0x7f1f38007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3800dda0 con 0x7f1f48132710 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 10 monclient: dump: 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.755 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1f38014470 con 0x7f1f48132710 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 msgr2=0x7f1f4812bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 0x7f1f4812bde0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800fa50 tx=0x7f1f38007e10 comp rx=0 tx=0).stop 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f477fe640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f28038490 0x7f1f2803a940 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1f30034000 tx=0x7f1f30032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 msgr2=0x7f1f4812bde0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.732+0000 7f1f3ffff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48132710 0x7f1f4812bde0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f4e8f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f4e8f8640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 0x7f1f4813f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f4e8f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4813fce0 con 0x7f1f4813f3a0 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 0x7f1f4813f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient: get_auth_request con 0x7f1f4813f3a0 auth_method 0 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 10 monclient(hunting): handle_auth_done global_id 104115 payload 824 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.738+0000 7f1f47fff640 1 -- 172.21.15.116:0/559429344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1f4804bf60 con 0x7f1f4813f3a0 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f47fff640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 msgr2=0x7f1f4813f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f47fff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 0x7f1f4813f7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1f38019000 tx=0x7f1f3801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3801edc0 con 0x7f1f4813f3a0 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 10 monclient: dump: 2024-09-18T12:32:39.758 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f1f3800db90 con 0x7f1f4813f3a0 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 msgr2=0x7f1f4813f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 0x7f1f4813f7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1f38019000 tx=0x7f1f3801a000 comp rx=0 tx=0).stop 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 msgr2=0x7f1f4813f7a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f3ffff640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f4813f3a0 0x7f1f4813f7a0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1f38019000 tx=0x7f1f3801a000 comp rx=0 tx=0).stop 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 82, 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:39.760 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:39.761 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:39.762 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:39.763 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:39.764 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.766 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:39.768 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:39.769 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:39.770 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.771 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f28038490 msgr2=0x7f1f2803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f28038490 0x7f1f2803a940 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1f30034000 tx=0x7f1f30032000 comp rx=0 tx=0).stop 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 10 monclient: shutdown 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48133070 msgr2=0x7f1f4813ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 --2- 172.21.15.116:0/559429344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f48133070 0x7f1f4813ee60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000cc20 tx=0x7f1f40005340 comp rx=0 tx=0).stop 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4ce95640 1 -- 172.21.15.116:0/559429344 reap_dead start 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4ce95640 1 -- 172.21.15.116:0/559429344 reap_dead start 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 shutdown_connections 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 >> 172.21.15.116:0/559429344 conn(0x7f1f480672b0 msgr2=0x7f1f4809d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 shutdown_connections 2024-09-18T12:32:39.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.741+0000 7f1f4e8f8640 1 -- 172.21.15.116:0/559429344 wait complete. 2024-09-18T12:32:39.775 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:32:39.775 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:32:39.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:32:39.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.803+0000 7fac6986f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:39.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.803+0000 7fac6986f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:39.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.803+0000 7fac6986f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:39.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:39.803+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fac6411e0e0 con 0x7fac6409ebe0 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.867+0000 7f46e1f1d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.867+0000 7f46e1f1d640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.867+0000 7f46e1f1d640 10 monclient: monmap: 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:39.868842+0000 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:39.868842+0000 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:39.870 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:39.870 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:39.870 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:39.870 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:39.870 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 1 Processor -- start 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 1 -- start start 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient: init 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46dc131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 0x7f46dc09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f46dc09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46dc09f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc09faf0 con 0x7f46dc09f220 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc09fd30 con 0x7f46dc131ab0 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc132590 con 0x7f46dc09e910 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46e1f1d640 10 monclient(hunting): authenticate will time out at 1187.190666s 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 0x7f46dc09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 0x7f46dc09ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42902/0 (socket says 172.21.15.116:42902) 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46dbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f46dc09f220 0x7f46dc138bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42546/0 (socket says 172.21.15.116:42546) 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 1 -- 172.21.15.116:0/3180016443 learned_addr learned my addr 172.21.15.116:0/3180016443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 10 monclient(hunting): get_auth_request con 0x7f46dc09e910 auth_method 0 2024-09-18T12:32:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.868+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 10 monclient(hunting): handle_auth_done global_id 64955 payload 274 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 1 -- 172.21.15.116:0/3180016443 >> v1:172.21.15.144:6789/0 conn(0x7f46dc131ab0 legacy=0x7f46dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 1 -- 172.21.15.116:0/3180016443 >> v1:172.21.15.116:6789/0 conn(0x7f46dc09f220 legacy=0x7f46dc138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 1 monclient: found mon.noname-b 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:39.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.869+0000 7f46d3fff640 1 -- 172.21.15.116:0/3180016443 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0a1110 con 0x7f46dc09e910 2024-09-18T12:32:39.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46d3fff640 1 -- 172.21.15.116:0/3180016443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 msgr2=0x7f46dc09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46d3fff640 1 --2- 172.21.15.116:0/3180016443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 0x7f46dc09ece0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009940 tx=0x7f46c40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7dafe8695902657 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 1 -- 172.21.15.116:0/3180016443 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4032440 con 0x7f46dc09e910 2024-09-18T12:32:39.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 10 monclient: dump: 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.887 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 1 -- 172.21.15.116:0/3180016443 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c40325a0 con 0x7f46dc09e910 2024-09-18T12:32:39.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46e1f1d640 5 monclient: authenticate success, global_id 64955 2024-09-18T12:32:39.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46e1f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46e1f1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.871+0000 7f46daffd640 1 -- 172.21.15.116:0/3180016443 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4032940 con 0x7f46dc09e910 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 4 set_mon_vals no callback set 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46daffd640 10 monclient: dump: 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 10 monclient: shutdown 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 1 -- 172.21.15.116:0/3180016443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 msgr2=0x7f46dc09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 1 --2- 172.21.15.116:0/3180016443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 0x7f46dc09ece0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009940 tx=0x7f46c40091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 1 -- 172.21.15.116:0/3180016443 shutdown_connections 2024-09-18T12:32:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 1 --2- 172.21.15.116:0/3180016443 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09e910 0x7f46dc09ece0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 1 -- 172.21.15.116:0/3180016443 >> 172.21.15.116:0/3180016443 conn(0x7f46dc067d30 msgr2=0x7f46dc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 1 -- 172.21.15.116:0/3180016443 shutdown_connections 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.872+0000 7f46e1f1d640 1 -- 172.21.15.116:0/3180016443 wait complete. 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient: build_initial_monmap 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient: monmap: 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 Processor -- start 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 -- start start 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient: init 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 0x7f46dc134100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f46dc09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09f220 0x7f46dc134640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f46dc09f220 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46dc131ab0 0x7f46dc134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f46dc131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc09bde0 con 0x7f46dc09e910 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc1397e0 con 0x7f46dc131ab0 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc1376c0 con 0x7f46dc09f220 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46e1f1d640 10 monclient(hunting): authenticate will time out at 1187.195830s 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46d3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09f220 0x7f46dc134640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 0x7f46dc134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 0x7f46dc134100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43932/0 (socket says 172.21.15.116:43932) 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46db7fe640 1 -- 172.21.15.116:0/581698200 learned_addr learned my addr 172.21.15.116:0/581698200 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46d3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09f220 0x7f46dc134640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42912/0 (socket says 172.21.15.116:42912) 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46db7fe640 10 monclient(hunting): get_auth_request con 0x7f46dc09e910 auth_method 0 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46d3fff640 10 monclient(hunting): get_auth_request con 0x7f46dc09f220 auth_method 0 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.873+0000 7f46d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46dc131ab0 0x7f46dc134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 10 monclient(hunting): get_auth_request con 0x7f46dc131ab0 auth_method 0 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 10 monclient(hunting): handle_auth_done global_id 104118 payload 824 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09f220 msgr2=0x7f46dc134640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09f220 0x7f46dc134640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46dc131ab0 msgr2=0x7f46dc134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46dc131ab0 0x7f46dc134b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 1 monclient: found mon.noname-a 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46db7fe640 1 -- 172.21.15.116:0/581698200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c4009450 con 0x7f46dc09e910 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.874+0000 7f46d3fff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc09f220 0x7f46dc134640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46db7fe640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 msgr2=0x7f46dc134100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46db7fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 0x7f46dc134100 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00c5e0 tx=0x7f46cc007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc012840 con 0x7f46dc09e910 2024-09-18T12:32:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: dump: 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc0129a0 con 0x7f46dc09e910 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46e1f1d640 5 monclient: authenticate success, global_id 104118 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46e1f1d640 10 monclient: _renew_subs 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc012d60 con 0x7f46dc09e910 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46e1f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc1379a0 con 0x7f46dc09e910 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46da7fc640 4 set_mon_vals no callback set 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46d37fe640 10 monclient: dump: 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46e1f1d640 10 monclient: _renew_subs 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46e1f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.876+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc135c00 con 0x7f46dc09e910 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46e1f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46e1f1d640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 0x7f46dc02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46e1f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc063740 con 0x7f46dc138620 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f46cc011590 con 0x7f46dc09e910 2024-09-18T12:32:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 0x7f46dc02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient: get_auth_request con 0x7f46dc138620 auth_method 0 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d37fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46bc038490 0x7f46bc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 10 monclient(hunting): handle_auth_done global_id 64961 payload 824 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d3fff640 1 -- 172.21.15.116:0/581698200 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46dc04bf60 con 0x7f46dc138620 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.877+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(196..196 src has 13..196) v4 ==== 8365+0+0 (secure 0 0 0) 0x7f46cc04c6d0 con 0x7f46dc09e910 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.878+0000 7f46dbfff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46bc038490 0x7f46bc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.878+0000 7f46dbfff640 10 monclient: get_auth_request con 0x7f46bc038490 auth_method 0 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.879+0000 7f46d3fff640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 msgr2=0x7f46dc02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d3fff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 0x7f46dc02aa40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46c4034000 tx=0x7f46c4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4000bc0 con 0x7f46dc138620 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 10 monclient: dump: 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.901 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f46c4046420 con 0x7f46dc138620 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 msgr2=0x7f46dc02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 0x7f46dc02aa40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46c4034000 tx=0x7f46c4032000 comp rx=0 tx=0).stop 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 msgr2=0x7f46dc02aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46d37fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc138620 0x7f46dc02aa40 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46c4034000 tx=0x7f46c4032000 comp rx=0 tx=0).stop 2024-09-18T12:32:39.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.880+0000 7f46dbfff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46bc038490 0x7f46bc03a940 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f46d40079b0 tx=0x7f46d4004e20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.885+0000 7f46e1f1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46e1f1d640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 0x7f46dc05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46e1f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc137be0 con 0x7f46dc05e310 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 0x7f46dc05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient: get_auth_request con 0x7f46dc05e310 auth_method 0 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:39.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 10 monclient(hunting): handle_auth_done global_id 64964 payload 824 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.886+0000 7f46d3fff640 1 -- 172.21.15.116:0/581698200 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46dc136830 con 0x7f46dc05e310 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d3fff640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 msgr2=0x7f46dc05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d3fff640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 0x7f46dc05e710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009210 tx=0x7f46c403e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c403bba0 con 0x7f46dc05e310 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 10 monclient: dump: 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.905 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f46c403bd00 con 0x7f46dc05e310 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 msgr2=0x7f46dc05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 0x7f46dc05e710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009210 tx=0x7f46c403e040 comp rx=0 tx=0).stop 2024-09-18T12:32:39.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 msgr2=0x7f46dc05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:39.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.888+0000 7f46d37fe640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46dc05e310 0x7f46dc05e710 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009210 tx=0x7f46c403e040 comp rx=0 tx=0).stop 2024-09-18T12:32:39.907 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:39.907 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 82, 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:39.908 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:39.909 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:39.909 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:39.909 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:39.909 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:39.909 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:39.909 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:39.910 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:39.910 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:39.910 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:39.910 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.910 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:39.911 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:39.911 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:39.911 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:39.911 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:39.911 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:39.911 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:39.912 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:39.913 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:39.914 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.915 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:39.916 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:39.918 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:39.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:39.920 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.922 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:39.923 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:39.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46bc038490 msgr2=0x7f46bc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46bc038490 0x7f46bc03a940 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f46d40079b0 tx=0x7f46d4004e20 comp rx=0 tx=0).stop 2024-09-18T12:32:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 10 monclient: shutdown 2024-09-18T12:32:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 msgr2=0x7f46dc134100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 0x7f46dc134100 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00c5e0 tx=0x7f46cc007ed0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46dbfff640 1 -- 172.21.15.116:0/581698200 reap_dead start 2024-09-18T12:32:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 shutdown_connections 2024-09-18T12:32:39.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 --2- 172.21.15.116:0/581698200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46dc09e910 0x7f46dc134100 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:39.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 >> 172.21.15.116:0/581698200 conn(0x7f46dc067d30 msgr2=0x7f46dc09d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:39.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 shutdown_connections 2024-09-18T12:32:39.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:39.889+0000 7f46e1f1d640 1 -- 172.21.15.116:0/581698200 wait complete. 2024-09-18T12:32:39.926 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:32:39.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:32:39.927 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:32:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.019+0000 7fe1dd39d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.019+0000 7fe1dd39d640 10 monclient: build_initial_monmap 2024-09-18T12:32:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.019+0000 7fe1dd39d640 10 monclient: monmap: 2024-09-18T12:32:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:40.021626+0000 2024-09-18T12:32:40.022 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:40.021626+0000 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.020+0000 7fe1dd39d640 1 Processor -- start 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.020+0000 7fe1dd39d640 1 -- start start 2024-09-18T12:32:40.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.020+0000 7fe1dd39d640 10 monclient: init 2024-09-18T12:32:40.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.020+0000 7fe1dd39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:40.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.020+0000 7fe1dd39d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T12:32:40.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.020+0000 7fe1dd39d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1d8131980 0x7fe1d8131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d8131980 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1d809fda0 0x7fe1d8132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d809fda0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1d809e9c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d809f510 con 0x7fe1d809e9c0 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d80a1180 con 0x7fe1d809fda0 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d80a1460 con 0x7fe1d8131980 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1dd39d640 10 monclient(hunting): authenticate will time out at 1187.343086s 2024-09-18T12:32:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1d809fda0 0x7fe1d8132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1d809fda0 0x7fe1d8132290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43946/0 (socket says 172.21.15.116:43946) 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 -- 172.21.15.116:0/2137208185 learned_addr learned my addr 172.21.15.116:0/2137208185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): get_auth_request con 0x7fe1d809fda0 auth_method 0 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 1 --2- 172.21.15.116:0/2137208185 >> v2:172.21.15.144:3300/0 conn(0x7fe1d8131980 0x7fe1d8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 10 monclient(hunting): get_auth_request con 0x7fe1d8131980 auth_method 0 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_done global_id 104121 payload 274 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 -- 172.21.15.116:0/2137208185 >> v2:172.21.15.144:3300/0 conn(0x7fe1d8131980 msgr2=0x7fe1d8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 --2- 172.21.15.116:0/2137208185 >> v2:172.21.15.144:3300/0 conn(0x7fe1d8131980 0x7fe1d8131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 -- 172.21.15.116:0/2137208185 >> v1:172.21.15.144:6789/0 conn(0x7fe1d809e9c0 legacy=0x7fe1d809ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient: _finish_hunting 0 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 monclient: found mon.noname-a 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.021+0000 7fe1d67fc640 1 -- 172.21.15.116:0/2137208185 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8139cd0 con 0x7fe1d809fda0 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d67fc640 1 -- 172.21.15.116:0/2137208185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d809fda0 msgr2=0x7fe1d8132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d67fc640 1 --2- 172.21.15.116:0/2137208185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d809fda0 0x7fe1d8132290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80a4fb0 tx=0x7fe1cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4365b705080fc22d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 1 -- 172.21.15.116:0/2137208185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc032440 con 0x7fe1d809fda0 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: dump: 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 5 monclient: authenticate success, global_id 104121 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 1 -- 172.21.15.116:0/2137208185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc0325a0 con 0x7fe1d809fda0 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:40.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 4 set_mon_vals no callback set 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 1 -- 172.21.15.116:0/2137208185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc032980 con 0x7fe1d809fda0 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1d5ffb640 10 monclient: dump: 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.041 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 10 monclient: shutdown 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 1 -- 172.21.15.116:0/2137208185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d809fda0 msgr2=0x7fe1d8132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 1 --2- 172.21.15.116:0/2137208185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d809fda0 0x7fe1d8132290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80a4fb0 tx=0x7fe1cc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 1 -- 172.21.15.116:0/2137208185 shutdown_connections 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 1 --2- 172.21.15.116:0/2137208185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d809fda0 0x7fe1d8132290 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 1 --2- 172.21.15.116:0/2137208185 >> v2:172.21.15.144:3300/0 conn(0x7fe1d8131980 0x7fe1d8131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.024+0000 7fe1dd39d640 1 -- 172.21.15.116:0/2137208185 >> 172.21.15.116:0/2137208185 conn(0x7fe1d8067bd0 msgr2=0x7fe1d80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.025+0000 7fe1dd39d640 1 -- 172.21.15.116:0/2137208185 shutdown_connections 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.025+0000 7fe1dd39d640 1 -- 172.21.15.116:0/2137208185 wait complete. 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.025+0000 7fe1dd39d640 10 monclient: build_initial_monmap 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.025+0000 7fe1dd39d640 10 monclient: monmap: 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:40.043 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.025+0000 7fe1dd39d640 1 Processor -- start 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.025+0000 7fe1dd39d640 1 -- start start 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.025+0000 7fe1dd39d640 10 monclient: init 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d809e9c0 0x7fe1d81374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d809e9c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d809fda0 0x7fe1d8130c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d809fda0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d8131980 0x7fe1d81311a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d8131980 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d809e490 con 0x7fe1d8131980 2024-09-18T12:32:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d8136bc0 con 0x7fe1d809fda0 2024-09-18T12:32:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d8136570 con 0x7fe1d809e9c0 2024-09-18T12:32:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1dd39d640 10 monclient(hunting): authenticate will time out at 1187.348417s 2024-09-18T12:32:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d809e9c0 0x7fe1d81374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d8131980 0x7fe1d81311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d8131980 0x7fe1d81311a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43960/0 (socket says 172.21.15.116:43960) 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 -- 172.21.15.116:0/3583578888 learned_addr learned my addr 172.21.15.116:0/3583578888 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d809e9c0 0x7fe1d81374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42944/0 (socket says 172.21.15.116:42944) 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): get_auth_request con 0x7fe1d8131980 auth_method 0 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 10 monclient(hunting): get_auth_request con 0x7fe1d809e9c0 auth_method 0 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d809fda0 0x7fe1d8130c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 10 monclient(hunting): get_auth_request con 0x7fe1d809fda0 auth_method 0 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient(hunting): handle_auth_done global_id 104124 payload 824 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d809e9c0 msgr2=0x7fe1d81374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d809e9c0 0x7fe1d81374d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d809fda0 msgr2=0x7fe1d8130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d809fda0 0x7fe1d8130c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 monclient: found mon.noname-a 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d77fe640 1 -- 172.21.15.116:0/3583578888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc009450 con 0x7fe1d8131980 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.026+0000 7fe1d6ffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1d809e9c0 0x7fe1d81374d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1d77fe640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d8131980 msgr2=0x7fe1d81311a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1d77fe640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d8131980 0x7fe1d81311a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000c890 tx=0x7fe1d0002640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d000ce80 con 0x7fe1d8131980 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: dump: 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.050 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d000aa00 con 0x7fe1d8131980 2024-09-18T12:32:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1dd39d640 5 monclient: authenticate success, global_id 104124 2024-09-18T12:32:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1dd39d640 10 monclient: _renew_subs 2024-09-18T12:32:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d000ae00 con 0x7fe1d8131980 2024-09-18T12:32:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1dd39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d8136850 con 0x7fe1d8131980 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 10 monclient: dump: 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1d57fa640 4 set_mon_vals no callback set 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1dd39d640 10 monclient: _renew_subs 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1dd39d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d8135610 con 0x7fe1d8131980 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.029+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe1d0012630 con 0x7fe1d8131980 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1beffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1c4038490 0x7fe1c403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(196..196 src has 13..196) v4 ==== 8365+0+0 (secure 0 0 0) 0x7fe1d004c2e0 con 0x7fe1d8131980 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1dd39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1dd39d640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 0x7fe1d8141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1dd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d8065e00 con 0x7fe1d81411c0 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d6ffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1c4038490 0x7fe1c403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d6ffd640 10 monclient: get_auth_request con 0x7fe1c4038490 auth_method 0 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 0x7fe1d8141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient: get_auth_request con 0x7fe1d81411c0 auth_method 0 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_done global_id 6169 payload 824 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.030+0000 7fe1d67fc640 1 -- 172.21.15.116:0/3583578888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1d804bf60 con 0x7fe1d81411c0 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1d6ffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1c4038490 0x7fe1c403a940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80113e0 tx=0x7fe1c80090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1d67fc640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 msgr2=0x7fe1d8141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1d67fc640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 0x7fe1d8141590 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8068aa0 tx=0x7fe1cc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc031b00 con 0x7fe1d81411c0 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 10 monclient: dump: 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.056 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.057 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe1cc031c60 con 0x7fe1d81411c0 2024-09-18T12:32:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 msgr2=0x7fe1d8141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 0x7fe1d8141590 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8068aa0 tx=0x7fe1cc032000 comp rx=0 tx=0).stop 2024-09-18T12:32:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 msgr2=0x7fe1d8141590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.033+0000 7fe1beffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d81411c0 0x7fe1d8141590 secure :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8068aa0 tx=0x7fe1cc032000 comp rx=0 tx=0).stop 2024-09-18T12:32:40.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1dd39d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:40.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1dd39d640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 0x7fe1d8098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1dd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d8134960 con 0x7fe1d8141ad0 2024-09-18T12:32:40.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 0x7fe1d8098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient: get_auth_request con 0x7fe1d8141ad0 auth_method 0 2024-09-18T12:32:40.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 10 monclient(hunting): handle_auth_done global_id 6172 payload 824 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.039+0000 7fe1d67fc640 1 -- 172.21.15.116:0/3583578888 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe1d80638d0 con 0x7fe1d8141ad0 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.041+0000 7fe1d67fc640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 msgr2=0x7fe1d8098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.041+0000 7fe1d67fc640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 0x7fe1d8098a70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe1d81358c0 tx=0x7fe1cc03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc03c650 con 0x7fe1d8141ad0 2024-09-18T12:32:40.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 10 monclient: dump: 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fe1cc03c7b0 con 0x7fe1d8141ad0 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 msgr2=0x7fe1d8098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 0x7fe1d8098a70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe1d81358c0 tx=0x7fe1cc03d000 comp rx=0 tx=0).stop 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 msgr2=0x7fe1d8098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.042+0000 7fe1beffd640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1d8141ad0 0x7fe1d8098a70 secure :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe1d81358c0 tx=0x7fe1cc03d000 comp rx=0 tx=0).stop 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 82, 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:40.064 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:40.065 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:40.066 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:40.067 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:40.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.069 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.070 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:40.073 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:40.074 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.075 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:40.076 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:40.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:40.077 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:40.077 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:40.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1c4038490 msgr2=0x7fe1c403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1c4038490 0x7fe1c403a940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80113e0 tx=0x7fe1c80090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 10 monclient: shutdown 2024-09-18T12:32:40.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d8131980 msgr2=0x7fe1d81311a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 --2- 172.21.15.116:0/3583578888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1d8131980 0x7fe1d81311a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000c890 tx=0x7fe1d0002640 comp rx=0 tx=0).stop 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1d77fe640 1 -- 172.21.15.116:0/3583578888 reap_dead start 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1d77fe640 1 -- 172.21.15.116:0/3583578888 reap_dead start 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 shutdown_connections 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 >> 172.21.15.116:0/3583578888 conn(0x7fe1d8067bd0 msgr2=0x7fe1d809d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 shutdown_connections 2024-09-18T12:32:40.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.043+0000 7fe1dd39d640 1 -- 172.21.15.116:0/3583578888 wait complete. 2024-09-18T12:32:40.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:32:40.079 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:32:40.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:32:40.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.169+0000 7f06c7d5d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.169+0000 7f06c7d5d640 10 monclient: build_initial_monmap 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.170+0000 7f06c7d5d640 10 monclient: monmap: 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:40.171645+0000 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:40.171645+0000 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:40.173 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:40.173 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.170+0000 7f06c7d5d640 1 Processor -- start 2024-09-18T12:32:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.170+0000 7f06c7d5d640 1 -- start start 2024-09-18T12:32:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.170+0000 7f06c7d5d640 10 monclient: init 2024-09-18T12:32:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:40.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T12:32:40.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f06c00974f0 0x7f06c012ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f06c00974f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:40.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06c009a8a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:40.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06c0133070 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:40.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c00978c0 con 0x7f06c009a8a0 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c009ca20 con 0x7f06c0133070 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c0098ad0 con 0x7f06c00974f0 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c7d5d640 10 monclient(hunting): authenticate will time out at 1187.493328s 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f06c00974f0 0x7f06c012ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c52f8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f06c009a8a0 0x7f06c009ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42562/0 (socket says 172.21.15.116:42562) 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c52f8640 1 -- 172.21.15.116:0/778633689 learned_addr learned my addr 172.21.15.116:0/778633689 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:32:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): get_auth_request con 0x7f06c00974f0 auth_method 0 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient(hunting): handle_auth_done global_id 104127 payload 274 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 1 -- 172.21.15.116:0/778633689 >> v1:172.21.15.144:6789/0 conn(0x7f06c0133070 legacy=0x7f06c0135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 1 -- 172.21.15.116:0/778633689 >> v1:172.21.15.116:6789/0 conn(0x7f06c009a8a0 legacy=0x7f06c009ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient: _finish_hunting 0 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 1 monclient: found mon.noname-a 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.171+0000 7f06c5af9640 1 -- 172.21.15.116:0/778633689 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c0098db0 con 0x7f06c00974f0 2024-09-18T12:32:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c5af9640 1 -- 172.21.15.116:0/778633689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 msgr2=0x7f06c012ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c5af9640 1 --2- 172.21.15.116:0/778633689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c012ab00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009850 tx=0x7f06bc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef035b45cec4ba8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 1 -- 172.21.15.116:0/778633689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc032440 con 0x7f06c00974f0 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: dump: 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: _finish_auth 0 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c7d5d640 5 monclient: authenticate success, global_id 104127 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c7d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 1 -- 172.21.15.116:0/778633689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06bc0325a0 con 0x7f06c00974f0 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c7d5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 1 -- 172.21.15.116:0/778633689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc032940 con 0x7f06c00974f0 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c7d5d640 4 set_mon_vals no callback set 2024-09-18T12:32:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.174+0000 7f06c4af7640 10 monclient: dump: 2024-09-18T12:32:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 10 monclient: shutdown 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 1 -- 172.21.15.116:0/778633689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 msgr2=0x7f06c012ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/778633689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c012ab00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009850 tx=0x7f06bc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 1 -- 172.21.15.116:0/778633689 shutdown_connections 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/778633689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c012ab00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 1 -- 172.21.15.116:0/778633689 >> 172.21.15.116:0/778633689 conn(0x7f06c00672b0 msgr2=0x7f06c00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 1 -- 172.21.15.116:0/778633689 shutdown_connections 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 1 -- 172.21.15.116:0/778633689 wait complete. 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 10 monclient: build_initial_monmap 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.175+0000 7f06c7d5d640 10 monclient: monmap: 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 Processor -- start 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 -- start start 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient: init 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c006e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f06c00974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06c009a8a0 0x7f06c006a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f06c009a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c0133070 0x7f06c006a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f06c0133070 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c006dc70 con 0x7f06c00974f0 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c006b550 con 0x7f06c0133070 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c006d990 con 0x7f06c009a8a0 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c7d5d640 10 monclient(hunting): authenticate will time out at 1187.498539s 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c006e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06c009a8a0 0x7f06c006a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c006e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43976/0 (socket says 172.21.15.116:43976) 2024-09-18T12:32:40.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 -- 172.21.15.116:0/2490039597 learned_addr learned my addr 172.21.15.116:0/2490039597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): get_auth_request con 0x7f06c00974f0 auth_method 0 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 10 monclient(hunting): get_auth_request con 0x7f06c009a8a0 auth_method 0 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c52f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c62fa640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c0133070 0x7f06c006a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c62fa640 10 monclient(hunting): get_auth_request con 0x7f06c0133070 auth_method 0 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c62fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c62fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c62fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient(hunting): handle_auth_done global_id 104130 payload 824 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 -- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06c009a8a0 msgr2=0x7f06c006a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06c009a8a0 0x7f06c006a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 -- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c0133070 msgr2=0x7f06c006a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c0133070 0x7f06c006a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient: _finish_hunting 0 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 monclient: found mon.noname-a 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.176+0000 7f06c5af9640 1 -- 172.21.15.116:0/2490039597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc009390 con 0x7f06c00974f0 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c5af9640 1 -- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 msgr2=0x7f06c006e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c5af9640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c006e370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009360 tx=0x7f06bc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 1 -- 172.21.15.116:0/2490039597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc031990 con 0x7f06c00974f0 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: dump: 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.195 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.196 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c7d5d640 5 monclient: authenticate success, global_id 104130 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 1 -- 172.21.15.116:0/2490039597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06bc031af0 con 0x7f06c00974f0 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c7d5d640 10 monclient: _renew_subs 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c7d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c006c7b0 con 0x7f06c00974f0 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 1 -- 172.21.15.116:0/2490039597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc03b400 con 0x7f06c00974f0 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:40.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b5ffb640 10 monclient: dump: 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.198 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.199 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c7d5d640 10 monclient: _renew_subs 2024-09-18T12:32:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06b7fff640 4 set_mon_vals no callback set 2024-09-18T12:32:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c7d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.179+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c006e8b0 con 0x7f06c00974f0 2024-09-18T12:32:40.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06b5ffb640 1 -- 172.21.15.116:0/2490039597 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f06bc043030 con 0x7f06c00974f0 2024-09-18T12:32:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06c7d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06c7d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06c7d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c004bf60 con 0x7f06c00974f0 2024-09-18T12:32:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06b5ffb640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f06a4038490 0x7f06a403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06b5ffb640 1 -- 172.21.15.116:0/2490039597 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(196..196 src has 13..196) v4 ==== 8365+0+0 (secure 0 0 0) 0x7f06bc041070 con 0x7f06c00974f0 2024-09-18T12:32:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06c52f8640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f06a4038490 0x7f06a403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.180+0000 7f06c52f8640 10 monclient: get_auth_request con 0x7f06a4038490 auth_method 0 2024-09-18T12:32:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.183+0000 7f06c52f8640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f06a4038490 0x7f06a403a940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f06b0007fd0 tx=0x7f06b0007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.184+0000 7f06b5ffb640 1 -- 172.21.15.116:0/2490039597 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f06bc044380 con 0x7f06c00974f0 2024-09-18T12:32:40.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.184+0000 7f06b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.184+0000 7f06b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.401+0000 7f06c7d5d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:32:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.401+0000 7f06c7d5d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:32:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.401+0000 7f06c7d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.401+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f06c004bf60 con 0x7f06c00974f0 2024-09-18T12:32:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f06b5ffb640 1 -- 172.21.15.116:0/2490039597 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f06bc037070 con 0x7f06c00974f0 2024-09-18T12:32:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f06b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:32:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f06b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:40.404 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.404 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=9,logm=62} crc {auth=2212447033,config=3140514009,health=1406400352,kv=2187923472,logm=3440043132}) 2024-09-18T12:32:40.404 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=3371248845,config=3140514009,health=4240550191,kv=2187923472,logm=2510745403}) 2024-09-18T12:32:40.404 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.404 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=9,logm=62} crc {auth=2212447033,config=3140514009,health=1406400352,kv=2187923472,logm=3440043132}) 2024-09-18T12:32:40.405 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.402+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=9,logm=62} crc {auth=3960957967,config=3140514009,health=1555756363,kv=2187923472,logm=3130288808}) 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780542098}) 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=785918852}) 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780542098}) 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3260357769}) 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.403+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339621459}) 2024-09-18T12:32:40.406 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3592933492}) 2024-09-18T12:32:40.407 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.407 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339621459}) 2024-09-18T12:32:40.407 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=950444208}) 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f06a4038490 msgr2=0x7f06a403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f06a4038490 0x7f06a403a940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f06b0007fd0 tx=0x7f06b0007f00 comp rx=0 tx=0).stop 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 10 monclient: shutdown 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 msgr2=0x7f06c006e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c006e370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009360 tx=0x7f06bc032000 comp rx=0 tx=0).stop 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 shutdown_connections 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f06a4038490 0x7f06a403a940 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06c0133070 0x7f06c006a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06c009a8a0 0x7f06c006a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 --2- 172.21.15.116:0/2490039597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c00974f0 0x7f06c006e370 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 >> 172.21.15.116:0/2490039597 conn(0x7f06c00672b0 msgr2=0x7f06c0125a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:40.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 shutdown_connections 2024-09-18T12:32:40.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f06c7d5d640 1 -- 172.21.15.116:0/2490039597 wait complete. 2024-09-18T12:32:40.414 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.414 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602266921}) 2024-09-18T12:32:40.414 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449903287}) 2024-09-18T12:32:40.414 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.414 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602266921}) 2024-09-18T12:32:40.414 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.404+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511734216}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.405+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.405+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265838434}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.405+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3155154527}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.405+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.405+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265838434}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.405+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728941313}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603183113}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260371445}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603183113}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147525074}) 2024-09-18T12:32:40.415 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272679584}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3926232090}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272679584}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1757622591}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424131896}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2161329835}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424131896}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2882232864}) 2024-09-18T12:32:40.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.408+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.408+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273835241}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.408+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681103726}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.408+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.408+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273835241}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.408+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929489578}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556909565}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4214569441}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556909565}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567509712}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994534062}) 2024-09-18T12:32:40.417 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865054452}) 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994534062}) 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1312589475}) 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472680151}) 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1737045676}) 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472680151}) 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484155039}) 2024-09-18T12:32:40.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946617969}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1053569284}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946617969}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3940391117}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163848837}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4148786148}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163848837}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688245144}) 2024-09-18T12:32:40.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324051878}) 2024-09-18T12:32:40.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262940645}) 2024-09-18T12:32:40.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324051878}) 2024-09-18T12:32:40.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4191355289}) 2024-09-18T12:32:40.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407807092}) 2024-09-18T12:32:40.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=752778344}) 2024-09-18T12:32:40.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407807092}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202811518}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511886638}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505383726}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511886638}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590107521}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025612862}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365796787}) 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025612862}) 2024-09-18T12:32:40.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261337079}) 2024-09-18T12:32:40.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766671669}) 2024-09-18T12:32:40.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349580980}) 2024-09-18T12:32:40.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766671669}) 2024-09-18T12:32:40.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188851437}) 2024-09-18T12:32:40.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:32:40.432 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.432 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377675946}) 2024-09-18T12:32:40.432 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493230834}) 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377675946}) 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2918641882}) 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573975900}) 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316494732}) 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573975900}) 2024-09-18T12:32:40.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526311711}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313835241}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333348825}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313835241}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=227514594}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3078142466}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983189803}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3078142466}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2113133534}) 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.434 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213062064}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3209000948}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213062064}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638293170}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2040046711}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315710141}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2040046711}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152846957}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523710223}) 2024-09-18T12:32:40.435 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590911944}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523710223}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=856450798}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901465284}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2776734405}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901465284}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=571563447}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402682263}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1447360340}) 2024-09-18T12:32:40.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402682263}) 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558737155}) 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279310802}) 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1141164615}) 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279310802}) 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1951369552}) 2024-09-18T12:32:40.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152612614}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190293218}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152612614}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.421+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084247655}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.422+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.422+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84671157}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.422+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1169275887}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.422+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.422+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84671157}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.422+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226940032}) 2024-09-18T12:32:40.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.423+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.423+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266676065}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.423+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3214979045}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.423+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.423+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266676065}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.423+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=3339149977,mdsmap=3585761405}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {logm=3044300602,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3020089879,mon_config_key=3655569570,monmap=148614106,osd_metadata=687973059,osd_pg_creating=267676565,osd_snap=2033388227,osdmap=1623666235}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=4,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=2,osdmap=60} crc {logm=4166876469,mdsmap=3585761405,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1405089338,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_snap=2033388227,osdmap=1587537077}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {logm=3044300602,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3020089879,mon_config_key=3655569570,monmap=148614106,osd_metadata=687973059,osd_pg_creating=267676565,osd_snap=2033388227,osdmap=1623666235}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=58} crc {logm=2938483645,mdsmap=1531151339,mgr=175081845,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3848122197,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=267676565,osd_snap=2033388227,osdmap=3596505148}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.425+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.425+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4134019048}) 2024-09-18T12:32:40.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.425+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=639743470}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.425+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.425+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4134019048}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.425+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1652897348}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.426+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.426+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3737199253}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.426+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3315278146}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.426+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.426+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3737199253}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.426+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1244496743}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.429+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.429+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=1171551246}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.429+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=21} crc {osdmap=4065000273}) 2024-09-18T12:32:40.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.429+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:32:40.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.429+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=1171551246}) 2024-09-18T12:32:40.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:32:40.430+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=3505620720}) 2024-09-18T12:32:40.452 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.451+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v197) v1 ==== 166+0+0 (secure 0 0 0) 0x7fac54023760 con 0x7fac6409ebe0 2024-09-18T12:32:40.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.452+0000 7fac4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:40.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.452+0000 7fac4affd640 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-18T12:32:40.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.547+0000 7fac6986f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:40.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.547+0000 7fac6986f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:40.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.547+0000 7fac6986f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:40.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.547+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fac64064990 con 0x7fac6409ebe0 2024-09-18T12:32:40.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.548+0000 7fac4affd640 1 -- 172.21.15.144:0/1356790577 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v197) v1 ==== 145+0+0 (secure 0 0 0) 0x7fac54023d40 con 0x7fac6409ebe0 2024-09-18T12:32:40.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.548+0000 7fac4affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:40.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.548+0000 7fac4affd640 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-18T12:32:40.550 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h4' already exists 2024-09-18T12:32:40.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fac500384c0 msgr2=0x7fac5003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fac500384c0 0x7fac5003a970 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac64068370 tx=0x7fac58032000 comp rx=0 tx=0).stop 2024-09-18T12:32:40.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 10 monclient: shutdown 2024-09-18T12:32:40.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:40.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 msgr2=0x7fac6409a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac6409a870 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fac5400ea20 tx=0x7fac540090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 shutdown_connections 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fac500384c0 0x7fac5003a970 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fac64131bd0 0x7fac64130ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac6409efe0 0x7fac641309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 --2- 172.21.15.144:0/1356790577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fac6409ebe0 0x7fac6409a870 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 >> 172.21.15.144:0/1356790577 conn(0x7fac640713b0 msgr2=0x7fac640a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 shutdown_connections 2024-09-18T12:32:40.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.550+0000 7fac6986f640 1 -- 172.21.15.144:0/1356790577 wait complete. 2024-09-18T12:32:40.560 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:40.560 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-18T12:32:40.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.634+0000 7f2375d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:40.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.634+0000 7f2375d1d640 10 monclient: build_initial_monmap 2024-09-18T12:32:40.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.634+0000 7f2375d1d640 10 monclient: monmap: 2024-09-18T12:32:40.635 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:40.635 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:40.634963+0000 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:40.634963+0000 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:40.636 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:40.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.634+0000 7f2375d1d640 1 Processor -- start 2024-09-18T12:32:40.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.634+0000 7f2375d1d640 1 -- start start 2024-09-18T12:32:40.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.634+0000 7f2375d1d640 10 monclient: init 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2370131bd0 0x7f2370131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2370131bd0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f237009ebe0 0x7f2370068e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f237009ebe0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f237009efe0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23700a06d0 con 0x7f237009efe0 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2370071780 con 0x7f2370131bd0 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23700a0010 con 0x7f237009ebe0 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f2375d1d640 10 monclient(hunting): authenticate will time out at 1196.204352s 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f237009ebe0 0x7f2370068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236ffff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f237009efe0 0x7f2370069340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58860/0 (socket says 172.21.15.144:58860) 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236ffff640 1 -- 172.21.15.144:0/2716785987 learned_addr learned my addr 172.21.15.144:0/2716785987 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): get_auth_request con 0x7f237009ebe0 auth_method 0 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236f7fe640 1 --2- 172.21.15.144:0/2716785987 >> v2:172.21.15.116:3300/0 conn(0x7f2370131bd0 0x7f2370131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236f7fe640 10 monclient(hunting): get_auth_request con 0x7f2370131bd0 auth_method 0 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient(hunting): handle_auth_done global_id 6178 payload 274 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 1 -- 172.21.15.144:0/2716785987 >> v2:172.21.15.116:3300/0 conn(0x7f2370131bd0 msgr2=0x7f2370131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 1 --2- 172.21.15.144:0/2716785987 >> v2:172.21.15.116:3300/0 conn(0x7f2370131bd0 0x7f2370131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 1 -- 172.21.15.144:0/2716785987 >> v1:172.21.15.144:6789/0 conn(0x7f237009efe0 legacy=0x7f2370069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 1 monclient: found mon.noname-c 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236effd640 1 -- 172.21.15.144:0/2716785987 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23700a1910 con 0x7f237009ebe0 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.635+0000 7f236f7fe640 1 --2- 172.21.15.144:0/2716785987 >> v2:172.21.15.116:3300/0 conn(0x7f2370131bd0 0x7f2370131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236effd640 1 -- 172.21.15.144:0/2716785987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009ebe0 msgr2=0x7f2370068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236effd640 1 --2- 172.21.15.144:0/2716785987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009ebe0 0x7f2370068e00 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2370068d90 tx=0x7f23640090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98a5b9fcee16a241 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 1 -- 172.21.15.144:0/2716785987 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236402adf0 con 0x7f237009ebe0 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: dump: 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.647 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 1 -- 172.21.15.144:0/2716785987 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364032440 con 0x7f237009ebe0 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 1 -- 172.21.15.144:0/2716785987 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23640327e0 con 0x7f237009ebe0 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f2375d1d640 5 monclient: authenticate success, global_id 6178 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f2375d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f2375d1d640 4 set_mon_vals no callback set 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.638+0000 7f236e7fc640 10 monclient: dump: 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 10 monclient: shutdown 2024-09-18T12:32:40.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 -- 172.21.15.144:0/2716785987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009ebe0 msgr2=0x7f2370068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 --2- 172.21.15.144:0/2716785987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009ebe0 0x7f2370068e00 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2370068d90 tx=0x7f23640090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 -- 172.21.15.144:0/2716785987 shutdown_connections 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 --2- 172.21.15.144:0/2716785987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009ebe0 0x7f2370068e00 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 --2- 172.21.15.144:0/2716785987 >> v2:172.21.15.116:3300/0 conn(0x7f2370131bd0 0x7f2370131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 -- 172.21.15.144:0/2716785987 >> 172.21.15.144:0/2716785987 conn(0x7f23700713b0 msgr2=0x7f23700a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 -- 172.21.15.144:0/2716785987 shutdown_connections 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 1 -- 172.21.15.144:0/2716785987 wait complete. 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 10 monclient: build_initial_monmap 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.639+0000 7f2375d1d640 10 monclient: monmap: 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:40.650 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 Processor -- start 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 -- start start 2024-09-18T12:32:40.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient: init 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237009ebe0 0x7f237009a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f237009ebe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 0x7f2370130960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f237009efe0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2370131bd0 0x7f237011b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2370131bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237009b3d0 con 0x7f237009ebe0 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237009bec0 con 0x7f237009efe0 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237009e410 con 0x7f2370131bd0 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f2375d1d640 10 monclient(hunting): authenticate will time out at 1196.209767s 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 0x7f2370130960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 0x7f2370130960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54298/0 (socket says 172.21.15.144:54298) 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 1 -- 172.21.15.144:0/3812364126 learned_addr learned my addr 172.21.15.144:0/3812364126 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:40.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 10 monclient(hunting): get_auth_request con 0x7f237009efe0 auth_method 0 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236f7fe640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237009ebe0 0x7f237009a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236ffff640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2370131bd0 0x7f237011b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236f7fe640 10 monclient(hunting): get_auth_request con 0x7f237009ebe0 auth_method 0 2024-09-18T12:32:40.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236ffff640 10 monclient(hunting): get_auth_request con 0x7f2370131bd0 auth_method 0 2024-09-18T12:32:40.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:40.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:40.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.640+0000 7f236ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:40.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:40.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:40.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:40.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 10 monclient(hunting): handle_auth_done global_id 6181 payload 824 2024-09-18T12:32:40.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 1 -- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2370131bd0 msgr2=0x7f237011b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2370131bd0 0x7f237011b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 1 -- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237009ebe0 msgr2=0x7f237009a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:40.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237009ebe0 0x7f237009a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:40.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:40.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 1 monclient: found mon.noname-b 2024-09-18T12:32:40.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:40.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:40.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.641+0000 7f236effd640 1 -- 172.21.15.144:0/3812364126 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009390 con 0x7f237009efe0 2024-09-18T12:32:40.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f236effd640 1 -- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 msgr2=0x7f2370130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f236effd640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 0x7f2370130960 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2364009880 tx=0x7f2364032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364031a10 con 0x7f237009efe0 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: dump: 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.661 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364031b70 con 0x7f237009efe0 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f2375d1d640 5 monclient: authenticate success, global_id 6181 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f2375d1d640 10 monclient: _renew_subs 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f2375d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:40.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237011dd60 con 0x7f237009efe0 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236403d440 con 0x7f237009efe0 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f23577fe640 10 monclient: dump: 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:40.663 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f236dffb640 4 set_mon_vals no callback set 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f2375d1d640 10 monclient: _renew_subs 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f2375d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.643+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237011cb20 con 0x7f237009efe0 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2364039040 con 0x7f237009efe0 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f2375d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f2375d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f2375d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237004bf60 con 0x7f237009efe0 2024-09-18T12:32:40.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f23577fe640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f235c0384c0 0x7f235c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:40.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(197..197 src has 13..197) v4 ==== 8575+0+0 (secure 0 0 0) 0x7f236406c270 con 0x7f237009efe0 2024-09-18T12:32:40.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f236f7fe640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f235c0384c0 0x7f235c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:40.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.644+0000 7f236f7fe640 10 monclient: get_auth_request con 0x7f235c0384c0 auth_method 0 2024-09-18T12:32:40.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.647+0000 7f236f7fe640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f235c0384c0 0x7f235c03a970 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2360006e70 tx=0x7f2360008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:40.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.649+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2364044b00 con 0x7f237009efe0 2024-09-18T12:32:40.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.649+0000 7f23577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:40.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.649+0000 7f23577fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:40.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.805+0000 7f2375d1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:40.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.805+0000 7f2375d1d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:40.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.805+0000 7f2375d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:40.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:40.805+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f237011e0d0 con 0x7f237009efe0 2024-09-18T12:32:41.426 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:32:41.426 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:41.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.456+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v198) v1 ==== 166+0+0 (secure 0 0 0) 0x7f2364037070 con 0x7f237009efe0 2024-09-18T12:32:41.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.457+0000 7f23577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:41.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.457+0000 7f23577fe640 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-18T12:32:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.523+0000 7f5c10b14640 10 monclient: get_monmap_and_config 2024-09-18T12:32:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.523+0000 7f5c10b14640 10 monclient: build_initial_monmap 2024-09-18T12:32:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.523+0000 7f5c10b14640 10 monclient: monmap: 2024-09-18T12:32:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:41.524778+0000 2024-09-18T12:32:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:41.524778+0000 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.523+0000 7f5c10b14640 1 Processor -- start 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.523+0000 7f5c10b14640 1 -- start start 2024-09-18T12:32:41.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.523+0000 7f5c10b14640 10 monclient: init 2024-09-18T12:32:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T12:32:41.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c0c1318f0 0x7f5c0c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0c1318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c09ea50 0x7f5c0c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): picked mon.noname-b con 0x7f5c0c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5c0c068d90 0x7f5c0c0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0c068d90 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0c131cc0 con 0x7f5c0c1318f0 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0c09fbf0 con 0x7f5c0c068d90 2024-09-18T12:32:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c09fe30 con 0x7f5c0c09ea50 2024-09-18T12:32:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c10b14640 10 monclient(hunting): authenticate will time out at 1188.846356s 2024-09-18T12:32:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c0c1318f0 0x7f5c0c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c09ea50 0x7f5c0c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c0c1318f0 0x7f5c0c131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43986/0 (socket says 172.21.15.116:43986) 2024-09-18T12:32:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 1 -- 172.21.15.116:0/3241743145 learned_addr learned my addr 172.21.15.116:0/3241743145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 10 monclient(hunting): get_auth_request con 0x7f5c0c1318f0 auth_method 0 2024-09-18T12:32:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): get_auth_request con 0x7f5c0c09ea50 auth_method 0 2024-09-18T12:32:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 1 --2- 172.21.15.116:0/3241743145 >> v2:172.21.15.144:3300/0 conn(0x7f5c0c068d90 0x7f5c0c0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 10 monclient(hunting): get_auth_request con 0x7f5c0c068d90 auth_method 0 2024-09-18T12:32:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient(hunting): handle_auth_done global_id 64976 payload 274 2024-09-18T12:32:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 1 -- 172.21.15.116:0/3241743145 >> v2:172.21.15.144:3300/0 conn(0x7f5c0c068d90 msgr2=0x7f5c0c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 1 --2- 172.21.15.116:0/3241743145 >> v2:172.21.15.144:3300/0 conn(0x7f5c0c068d90 0x7f5c0c0691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 1 -- 172.21.15.116:0/3241743145 >> v2:172.21.15.116:3300/0 conn(0x7f5c0c1318f0 msgr2=0x7f5c0c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 1 --2- 172.21.15.116:0/3241743145 >> v2:172.21.15.116:3300/0 conn(0x7f5c0c1318f0 0x7f5c0c131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 1 monclient: found mon.noname-b 2024-09-18T12:32:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:41.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c09d9b640 1 -- 172.21.15.116:0/3241743145 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c0a1210 con 0x7f5c0c09ea50 2024-09-18T12:32:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.524+0000 7f5c0a59c640 1 --2- 172.21.15.116:0/3241743145 >> v2:172.21.15.116:3300/0 conn(0x7f5c0c1318f0 0x7f5c0c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c09d9b640 1 -- 172.21.15.116:0/3241743145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c09ea50 msgr2=0x7f5c0c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c09d9b640 1 --2- 172.21.15.116:0/3241743145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c09ea50 0x7f5c0c09ee20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c064050 tx=0x7f5c00009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d76e50c1b93a0b8c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 1 -- 172.21.15.116:0/3241743145 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007760 con 0x7f5c0c09ea50 2024-09-18T12:32:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: dump: 2024-09-18T12:32:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: _finish_auth 0 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 1 -- 172.21.15.116:0/3241743145 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0002d050 con 0x7f5c0c09ea50 2024-09-18T12:32:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 1 -- 172.21.15.116:0/3241743145 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007b30 con 0x7f5c0c09ea50 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c10b14640 5 monclient: authenticate success, global_id 64976 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c10b14640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c10b14640 4 set_mon_vals no callback set 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.527+0000 7f5c0959a640 10 monclient: dump: 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 10 monclient: shutdown 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 -- 172.21.15.116:0/3241743145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c09ea50 msgr2=0x7f5c0c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 --2- 172.21.15.116:0/3241743145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c09ea50 0x7f5c0c09ee20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c064050 tx=0x7f5c00009780 comp rx=0 tx=0).stop 2024-09-18T12:32:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 -- 172.21.15.116:0/3241743145 shutdown_connections 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 --2- 172.21.15.116:0/3241743145 >> v2:172.21.15.144:3300/0 conn(0x7f5c0c068d90 0x7f5c0c0691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 --2- 172.21.15.116:0/3241743145 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c09ea50 0x7f5c0c09ee20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 --2- 172.21.15.116:0/3241743145 >> v2:172.21.15.116:3300/0 conn(0x7f5c0c1318f0 0x7f5c0c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 -- 172.21.15.116:0/3241743145 >> 172.21.15.116:0/3241743145 conn(0x7f5c0c067110 msgr2=0x7f5c0c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 -- 172.21.15.116:0/3241743145 shutdown_connections 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 1 -- 172.21.15.116:0/3241743145 wait complete. 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 10 monclient: build_initial_monmap 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.528+0000 7f5c10b14640 10 monclient: monmap: 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 Processor -- start 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 -- start start 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient: init 2024-09-18T12:32:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 0x7f5c0c09ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0c068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0c09ea50 0x7f5c0c130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0c09ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c0c1318f0 0x7f5c0c131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): picked mon.noname-b con 0x7f5c0c1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c09bf10 con 0x7f5c0c09ea50 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c09e750 con 0x7f5c0c1318f0 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c141e70 con 0x7f5c0c068d90 2024-09-18T12:32:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c10b14640 10 monclient(hunting): authenticate will time out at 1188.851749s 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 0x7f5c0c09ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0c09ea50 0x7f5c0c130d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 0x7f5c0c09ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42974/0 (socket says 172.21.15.116:42974) 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 1 -- 172.21.15.116:0/2662035280 learned_addr learned my addr 172.21.15.116:0/2662035280 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 10 monclient(hunting): get_auth_request con 0x7f5c0c068d90 auth_method 0 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 10 monclient(hunting): get_auth_request con 0x7f5c0c09ea50 auth_method 0 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0ad9d640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c0c1318f0 0x7f5c0c131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0ad9d640 10 monclient(hunting): get_auth_request con 0x7f5c0c1318f0 auth_method 0 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.529+0000 7f5c0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 10 monclient(hunting): handle_auth_done global_id 64979 payload 824 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 1 -- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c0c1318f0 msgr2=0x7f5c0c131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c0c1318f0 0x7f5c0c131290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 1 -- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0c09ea50 msgr2=0x7f5c0c130d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0c09ea50 0x7f5c0c130d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 1 monclient: found mon.noname-c 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0a59c640 1 -- 172.21.15.116:0/2662035280 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c04009450 con 0x7f5c0c068d90 2024-09-18T12:32:41.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c0ad9d640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c0c1318f0 0x7f5c0c131290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.530+0000 7f5c09d9b640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0c09ea50 0x7f5c0c130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c0a59c640 1 -- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 msgr2=0x7f5c0c09ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c0a59c640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 0x7f5c0c09ab60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ec80 tx=0x7f5c040091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 1 -- 172.21.15.116:0/2662035280 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04009730 con 0x7f5c0c068d90 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: dump: 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.558 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 1 -- 172.21.15.116:0/2662035280 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0400eec0 con 0x7f5c0c068d90 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c10b14640 5 monclient: authenticate success, global_id 64979 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c10b14640 10 monclient: _renew_subs 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c10b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c00009a20 con 0x7f5c0c068d90 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 1 -- 172.21.15.116:0/2662035280 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04013470 con 0x7f5c0c068d90 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5bfa7fc640 10 monclient: dump: 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c10b14640 10 monclient: _renew_subs 2024-09-18T12:32:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c10b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c08d99640 4 set_mon_vals no callback set 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.532+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0c11cf80 con 0x7f5c0c068d90 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5bfa7fc640 1 -- 172.21.15.116:0/2662035280 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5c04013820 con 0x7f5c0c068d90 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5c10b14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5c10b14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5c10b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0c11df80 con 0x7f5c0c068d90 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5bfa7fc640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5bec038490 0x7f5bec03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5bfa7fc640 1 -- 172.21.15.116:0/2662035280 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f5c0404cc80 con 0x7f5c0c068d90 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5c09d9b640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5bec038490 0x7f5bec03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.533+0000 7f5c09d9b640 10 monclient: get_auth_request con 0x7f5bec038490 auth_method 0 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.536+0000 7f5c09d9b640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5bec038490 0x7f5bec03a940 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5c00034000 tx=0x7f5c00037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.538+0000 7f5bfa7fc640 1 -- 172.21.15.116:0/2662035280 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5c04023ad0 con 0x7f5c0c068d90 2024-09-18T12:32:41.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.538+0000 7f5bfa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.538+0000 7f5bfa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.555+0000 7f2375d1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.555+0000 7f2375d1d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.555+0000 7f2375d1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.555+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f237011e0d0 con 0x7f237009efe0 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.556+0000 7f23577fe640 1 -- 172.21.15.144:0/3812364126 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v198) v1 ==== 145+0+0 (secure 0 0 0) 0x7f2364034030 con 0x7f237009efe0 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.556+0000 7f23577fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.556+0000 7f23577fe640 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h5' already exists 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f235c0384c0 msgr2=0x7f235c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f235c0384c0 0x7f235c03a970 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2360006e70 tx=0x7f2360008040 comp rx=0 tx=0).stop 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 10 monclient: shutdown 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 msgr2=0x7f2370130960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 0x7f2370130960 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2364009880 tx=0x7f2364032000 comp rx=0 tx=0).stop 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 shutdown_connections 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f235c0384c0 0x7f235c03a970 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2370131bd0 0x7f237011b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f237009efe0 0x7f2370130960 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 --2- 172.21.15.144:0/3812364126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237009ebe0 0x7f237009a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 >> 172.21.15.144:0/3812364126 conn(0x7f23700713b0 msgr2=0x7f237009cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 shutdown_connections 2024-09-18T12:32:41.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.558+0000 7f2375d1d640 1 -- 172.21.15.144:0/3812364126 wait complete. 2024-09-18T12:32:41.569 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:41.569 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-18T12:32:41.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.644+0000 7f4f3aa0f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.644+0000 7f4f3aa0f640 10 monclient: build_initial_monmap 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.644+0000 7f4f3aa0f640 10 monclient: monmap: 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:41.645744+0000 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:41.645744+0000 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:41.646 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:41.647 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:41.647 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:41.647 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:41.647 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:41.647 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:41.647 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.645+0000 7f4f3aa0f640 1 Processor -- start 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.645+0000 7f4f3aa0f640 1 -- start start 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.645+0000 7f4f3aa0f640 10 monclient: init 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f34131ac0 0x7f4f34131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f34131ac0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f3409ff50 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f341323d0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f341327a0 con 0x7f4f341323d0 2024-09-18T12:32:41.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f340a1330 con 0x7f4f3409ff50 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f34098860 con 0x7f4f34131ac0 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f3aa0f640 10 monclient(hunting): authenticate will time out at 1197.215273s 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f34131ac0 0x7f4f34131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f34131ac0 0x7f4f34131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54308/0 (socket says 172.21.15.144:54308) 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 1 -- 172.21.15.144:0/3297132185 learned_addr learned my addr 172.21.15.144:0/3297132185 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): get_auth_request con 0x7f4f34131ac0 auth_method 0 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f337fe640 1 --1- 172.21.15.144:0/3297132185 >> v1:172.21.15.144:6789/0 conn(0x7f4f3409ff50 0x7f4f34098150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:58872/0 (socket says 172.21.15.144:58872) 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient(hunting): handle_auth_done global_id 6190 payload 274 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 1 -- 172.21.15.144:0/3297132185 >> v1:172.21.15.144:6789/0 conn(0x7f4f3409ff50 legacy=0x7f4f34098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 1 -- 172.21.15.144:0/3297132185 >> v1:172.21.15.116:6789/0 conn(0x7f4f341323d0 legacy=0x7f4f34138e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 1 monclient: found mon.noname-c 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:32:41.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.646+0000 7f4f33fff640 1 -- 172.21.15.144:0/3297132185 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340989a0 con 0x7f4f34131ac0 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f33fff640 1 -- 172.21.15.144:0/3297132185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f34131ac0 msgr2=0x7f4f34131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f33fff640 1 --2- 172.21.15.144:0/3297132185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f34131ac0 0x7f4f34131e90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009910 tx=0x7f4f240091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac3a84643aac2ec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 1 -- 172.21.15.144:0/3297132185 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24032440 con 0x7f4f34131ac0 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: dump: 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.660 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 1 -- 172.21.15.144:0/3297132185 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f240325a0 con 0x7f4f34131ac0 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f3aa0f640 5 monclient: authenticate success, global_id 6190 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f3aa0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:41.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f3aa0f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 1 -- 172.21.15.144:0/3297132185 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24032940 con 0x7f4f34131ac0 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f3aa0f640 4 set_mon_vals no callback set 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.649+0000 7f4f32ffd640 10 monclient: dump: 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.662 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 10 monclient: shutdown 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/3297132185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f34131ac0 msgr2=0x7f4f34131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/3297132185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f34131ac0 0x7f4f34131e90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009910 tx=0x7f4f240091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/3297132185 shutdown_connections 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/3297132185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f34131ac0 0x7f4f34131e90 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/3297132185 >> 172.21.15.144:0/3297132185 conn(0x7f4f340a5190 msgr2=0x7f4f340a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/3297132185 shutdown_connections 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/3297132185 wait complete. 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 10 monclient: build_initial_monmap 2024-09-18T12:32:41.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 10 monclient: monmap: 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.650+0000 7f4f3aa0f640 1 Processor -- start 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 1 -- start start 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient: init 2024-09-18T12:32:41.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f341385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3409ff50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 0x7f4f34134280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f34131ac0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 0x7f4f341347c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f341323d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f34098e70 con 0x7f4f34131ac0 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f340a3090 con 0x7f4f341323d0 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f341378f0 con 0x7f4f3409ff50 2024-09-18T12:32:41.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f3aa0f640 10 monclient(hunting): authenticate will time out at 1197.220589s 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 0x7f4f341347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 0x7f4f341347c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54310/0 (socket says 172.21.15.144:54310) 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 learned_addr learned my addr 172.21.15.144:0/4174960782 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): get_auth_request con 0x7f4f341323d0 auth_method 0 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f341385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 0x7f4f34134280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 10 monclient(hunting): get_auth_request con 0x7f4f34131ac0 auth_method 0 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 10 monclient(hunting): get_auth_request con 0x7f4f3409ff50 auth_method 0 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.667 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 10 monclient(hunting): handle_auth_done global_id 6193 payload 824 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.651+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 msgr2=0x7f4f341385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f341385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 msgr2=0x7f4f34134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 0x7f4f34134280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 10 monclient: _finish_hunting 0 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 1 monclient: found mon.noname-b 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.652+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f24009450 con 0x7f4f341323d0 2024-09-18T12:32:41.668 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 msgr2=0x7f4f341347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 0x7f4f341347c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c890 tx=0x7f4f2c002640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00ce80 con 0x7f4f341323d0 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: dump: 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.669 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.670 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:41.670 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.670 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.670 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.670 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:41.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: _finish_auth 0 2024-09-18T12:32:41.670 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f3aa0f640 5 monclient: authenticate success, global_id 6193 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f3aa0f640 10 monclient: _renew_subs 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c00aa00 con 0x7f4f341323d0 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f3aa0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f34135dc0 con 0x7f4f341323d0 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f327fc640 4 set_mon_vals no callback set 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f3aa0f640 10 monclient: _renew_subs 2024-09-18T12:32:41.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f3aa0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00adc0 con 0x7f4f341323d0 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f34138af0 con 0x7f4f341323d0 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.654+0000 7f4f1bfff640 10 monclient: dump: 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.672 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4f2c012630 con 0x7f4f341323d0 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f1bfff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f20038490 0x7f4f2003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f4f2c04cbd0 con 0x7f4f341323d0 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f3aa0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f3aa0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f3aa0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f33fff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f20038490 0x7f4f2003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f34138d60 con 0x7f4f341323d0 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.655+0000 7f4f33fff640 10 monclient: get_auth_request con 0x7f4f20038490 auth_method 0 2024-09-18T12:32:41.673 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.657+0000 7f4f33fff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f20038490 0x7f4f2003a940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4f3409feb0 tx=0x7f4f24032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.660+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4f34138d60 con 0x7f4f341323d0 2024-09-18T12:32:41.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.660+0000 7f4f1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.674 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.660+0000 7f4f1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.756+0000 7f5c10b14640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.756+0000 7f5c10b14640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.756+0000 7f5c10b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.756+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c0c04bf60 con 0x7f5c0c068d90 2024-09-18T12:32:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.757+0000 7f5bfa7fc640 1 -- 172.21.15.116:0/2662035280 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f5c040232f0 con 0x7f5c0c068d90 2024-09-18T12:32:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.757+0000 7f5bfa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.757+0000 7f5bfa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:41.759 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5bec038490 msgr2=0x7f5bec03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5bec038490 0x7f5bec03a940 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5c00034000 tx=0x7f5c00037000 comp rx=0 tx=0).stop 2024-09-18T12:32:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 10 monclient: shutdown 2024-09-18T12:32:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 msgr2=0x7f5c0c09ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 0x7f5c0c09ab60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ec80 tx=0x7f5c040091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 shutdown_connections 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5bec038490 0x7f5bec03a940 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c0c1318f0 0x7f5c0c131290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c0c09ea50 0x7f5c0c130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 --2- 172.21.15.116:0/2662035280 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c0c068d90 0x7f5c0c09ab60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 >> 172.21.15.116:0/2662035280 conn(0x7f5c0c067110 msgr2=0x7f5c0c0a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 shutdown_connections 2024-09-18T12:32:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.760+0000 7f5c10b14640 1 -- 172.21.15.116:0/2662035280 wait complete. 2024-09-18T12:32:41.771 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:32:41.771 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:32:41.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:32:41.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:32:41.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.817+0000 7f4f3aa0f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:41.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.817+0000 7f4f3aa0f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:41.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.817+0000 7f4f3aa0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:32:41.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:41.817+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4f34138d60 con 0x7f4f341323d0 2024-09-18T12:32:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient: get_monmap_and_config 2024-09-18T12:32:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient: build_initial_monmap 2024-09-18T12:32:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient: monmap: 2024-09-18T12:32:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:41.865186+0000 2024-09-18T12:32:41.865 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:41.865186+0000 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:41.866 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 1 Processor -- start 2024-09-18T12:32:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 1 -- start start 2024-09-18T12:32:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient: init 2024-09-18T12:32:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T12:32:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 0x7f96ac09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac09c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96ac09aba0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f96ac09af70 0x7f96ac09df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac09af70 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac06a320 con 0x7f96ac09aba0 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac09a9d0 con 0x7f96ac09af70 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac068a20 con 0x7f96ac09c860 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b32dd640 10 monclient(hunting): authenticate will time out at 1189.186823s 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b1079640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 0x7f96ac09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b0878640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f96ac09aba0 0x7f96ac09d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42564/0 (socket says 172.21.15.116:42564) 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b1079640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 0x7f96ac09cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42982/0 (socket says 172.21.15.116:42982) 2024-09-18T12:32:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b0878640 1 -- 172.21.15.116:0/2097660342 learned_addr learned my addr 172.21.15.116:0/2097660342 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b1079640 10 monclient(hunting): get_auth_request con 0x7f96ac09c860 auth_method 0 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b1079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b1079640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.864+0000 7f96b1079640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b187a640 1 --2- 172.21.15.116:0/2097660342 >> v2:172.21.15.144:3300/0 conn(0x7f96ac09af70 0x7f96ac09df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b187a640 10 monclient(hunting): get_auth_request con 0x7f96ac09af70 auth_method 0 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b187a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b187a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b187a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 10 monclient(hunting): handle_auth_done global_id 64985 payload 274 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 1 -- 172.21.15.116:0/2097660342 >> v2:172.21.15.144:3300/0 conn(0x7f96ac09af70 msgr2=0x7f96ac09df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 1 --2- 172.21.15.116:0/2097660342 >> v2:172.21.15.144:3300/0 conn(0x7f96ac09af70 0x7f96ac09df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 1 -- 172.21.15.116:0/2097660342 >> v1:172.21.15.116:6789/0 conn(0x7f96ac09aba0 legacy=0x7f96ac09d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 10 monclient: _finish_hunting 0 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 1 monclient: found mon.noname-b 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.865+0000 7f96b1079640 1 -- 172.21.15.116:0/2097660342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac068e60 con 0x7f96ac09c860 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.867+0000 7f96b1079640 1 -- 172.21.15.116:0/2097660342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 msgr2=0x7f96ac09cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.867+0000 7f96b1079640 1 --2- 172.21.15.116:0/2097660342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 0x7f96ac09cc30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009850 tx=0x7f96a80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a79a7a9eff698812 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 1 -- 172.21.15.116:0/2097660342 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a802add0 con 0x7f96ac09c860 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: dump: 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.880 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: _finish_auth 0 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 5 monclient: authenticate success, global_id 64985 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 1 -- 172.21.15.116:0/2097660342 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a8032440 con 0x7f96ac09c860 2024-09-18T12:32:41.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 4 set_mon_vals no callback set 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 1 -- 172.21.15.116:0/2097660342 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8032820 con 0x7f96ac09c860 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96a3fff640 10 monclient: dump: 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 10 monclient: shutdown 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 -- 172.21.15.116:0/2097660342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 msgr2=0x7f96ac09cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 --2- 172.21.15.116:0/2097660342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 0x7f96ac09cc30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009850 tx=0x7f96a80090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 -- 172.21.15.116:0/2097660342 shutdown_connections 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 --2- 172.21.15.116:0/2097660342 >> v2:172.21.15.144:3300/0 conn(0x7f96ac09af70 0x7f96ac09df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 --2- 172.21.15.116:0/2097660342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09c860 0x7f96ac09cc30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 -- 172.21.15.116:0/2097660342 >> 172.21.15.116:0/2097660342 conn(0x7f96ac0a2e40 msgr2=0x7f96ac0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:41.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 -- 172.21.15.116:0/2097660342 shutdown_connections 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.868+0000 7f96b32dd640 1 -- 172.21.15.116:0/2097660342 wait complete. 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.869+0000 7f96b32dd640 10 monclient: build_initial_monmap 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.869+0000 7f96b32dd640 10 monclient: monmap: 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:41.884 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:41.885 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.869+0000 7f96b32dd640 1 Processor -- start 2024-09-18T12:32:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.869+0000 7f96b32dd640 1 -- start start 2024-09-18T12:32:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.869+0000 7f96b32dd640 10 monclient: init 2024-09-18T12:32:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09aba0 0x7f96ac133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac09aba0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96ac09af70 0x7f96ac11d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac09af70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96ac09c860 0x7f96ac11ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac09c860 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac11f620 con 0x7f96ac09af70 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac134b20 con 0x7f96ac09c860 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac099cc0 con 0x7f96ac09aba0 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b32dd640 10 monclient(hunting): authenticate will time out at 1189.192149s 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09aba0 0x7f96ac133ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09aba0 0x7f96ac133ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42998/0 (socket says 172.21.15.116:42998) 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 -- 172.21.15.116:0/796440722 learned_addr learned my addr 172.21.15.116:0/796440722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): get_auth_request con 0x7f96ac09aba0 auth_method 0 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96ac09af70 0x7f96ac11d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 10 monclient(hunting): get_auth_request con 0x7f96ac09af70 auth_method 0 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96ac09c860 0x7f96ac11ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 10 monclient(hunting): get_auth_request con 0x7f96ac09c860 auth_method 0 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b187a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient(hunting): handle_auth_done global_id 64988 payload 824 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96ac09c860 msgr2=0x7f96ac11ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96ac09c860 0x7f96ac11ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96ac09af70 msgr2=0x7f96ac11d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96ac09af70 0x7f96ac11d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient: _finish_hunting 0 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 monclient: found mon.noname-c 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b1079640 1 -- 172.21.15.116:0/796440722 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8009390 con 0x7f96ac09aba0 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.870+0000 7f96b0878640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96ac09af70 0x7f96ac11d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b1079640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09aba0 msgr2=0x7f96ac133ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b1079640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09aba0 0x7f96ac133ab0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009850 tx=0x7f96a8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8031d40 con 0x7f96ac09aba0 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96a17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96a17fa640 10 monclient: dump: 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.891 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.891 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96a17fa640 10 monclient: _finish_auth 0 2024-09-18T12:32:41.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:41.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b32dd640 5 monclient: authenticate success, global_id 64988 2024-09-18T12:32:41.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b32dd640 10 monclient: _renew_subs 2024-09-18T12:32:41.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b32dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac11fc70 con 0x7f96ac09aba0 2024-09-18T12:32:41.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b32dd640 10 monclient: _renew_subs 2024-09-18T12:32:41.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b32dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.872+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac120c80 con 0x7f96ac09aba0 2024-09-18T12:32:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a803c400 con 0x7f96ac09aba0 2024-09-18T12:32:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8031ea0 con 0x7f96ac09aba0 2024-09-18T12:32:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a17fa640 10 monclient: dump: 2024-09-18T12:32:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a37fe640 4 set_mon_vals no callback set 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b32dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b32dd640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 0x7f968c004ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b32dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c007c30 con 0x7f968c002850 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b0878640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 0x7f968c004ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f96a803b3a0 con 0x7f96ac09aba0 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b0878640 10 monclient: get_auth_request con 0x7f968c002850 auth_method 0 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b0878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b0878640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.873+0000 7f96b0878640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96a17fa640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9690038490 0x7f969003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f96a806d320 con 0x7f96ac09aba0 2024-09-18T12:32:41.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96b0878640 10 monclient(hunting): handle_auth_done global_id 104151 payload 824 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96b0878640 1 -- 172.21.15.116:0/796440722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f968c007df0 con 0x7f968c002850 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96b187a640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9690038490 0x7f969003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.874+0000 7f96b187a640 10 monclient: get_auth_request con 0x7f9690038490 auth_method 0 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96b0878640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 msgr2=0x7f968c004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96b0878640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 0x7f968c004ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f969c005140 tx=0x7f969c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c00fd50 con 0x7f968c002850 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 10 monclient: dump: 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.897 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f969c00feb0 con 0x7f968c002850 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 msgr2=0x7f968c004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 0x7f968c004ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f969c005140 tx=0x7f969c007f00 comp rx=0 tx=0).stop 2024-09-18T12:32:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 msgr2=0x7f968c004ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:41.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96a17fa640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c002850 0x7f968c004ce0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f969c005140 tx=0x7f969c007f00 comp rx=0 tx=0).stop 2024-09-18T12:32:41.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.876+0000 7f96b187a640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9690038490 0x7f969003a940 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f96ac120f60 tx=0x7f96a4009850 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b32dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b32dd640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 0x7f968c0070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b32dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c0076e0 con 0x7f968c006c20 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 0x7f968c0070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient: get_auth_request con 0x7f968c006c20 auth_method 0 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 10 monclient(hunting): handle_auth_done global_id 104154 payload 824 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.883+0000 7f96b0878640 1 -- 172.21.15.116:0/796440722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f968c0078a0 con 0x7f968c006c20 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b0878640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 msgr2=0x7f968c0070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b0878640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 0x7f968c0070a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f969c005140 tx=0x7f969c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c021870 con 0x7f968c006c20 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 10 monclient: dump: 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f969c0219d0 con 0x7f968c006c20 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 msgr2=0x7f968c0070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 0x7f968c0070a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f969c005140 tx=0x7f969c018000 comp rx=0 tx=0).stop 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 msgr2=0x7f968c0070a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96a17fa640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c006c20 0x7f968c0070a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 82, 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:32:41.905 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:41.906 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:41.907 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:41.908 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:41.909 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:41.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:41.911 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.913 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:41.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:41.915 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:41.916 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:41.917 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:41.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:41.917 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:41.917 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:41.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:41.917 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:41.917 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:41.918 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:41.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:41.918 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:41.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:41.918 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:41.918 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:41.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9690038490 msgr2=0x7f969003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9690038490 0x7f969003a940 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f96ac120f60 tx=0x7f96a4009850 comp rx=0 tx=0).stop 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 10 monclient: shutdown 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09aba0 msgr2=0x7f96ac133ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 --2- 172.21.15.116:0/796440722 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96ac09aba0 0x7f96ac133ab0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009850 tx=0x7f96a8032000 comp rx=0 tx=0).stop 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b187a640 1 -- 172.21.15.116:0/796440722 reap_dead start 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b187a640 1 -- 172.21.15.116:0/796440722 reap_dead start 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 shutdown_connections 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 >> 172.21.15.116:0/796440722 conn(0x7f96ac0a2e40 msgr2=0x7f96ac098cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 shutdown_connections 2024-09-18T12:32:41.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:41.885+0000 7f96b32dd640 1 -- 172.21.15.116:0/796440722 wait complete. 2024-09-18T12:32:41.920 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:32:41.921 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:32:41.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:32:42.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.012+0000 7f79f06ef640 10 monclient: get_monmap_and_config 2024-09-18T12:32:42.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.012+0000 7f79f06ef640 10 monclient: build_initial_monmap 2024-09-18T12:32:42.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.012+0000 7f79f06ef640 10 monclient: monmap: 2024-09-18T12:32:42.014 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:42.014 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:42.014172+0000 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:42.014172+0000 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:42.015 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:42.016 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 1 Processor -- start 2024-09-18T12:32:42.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 1 -- start start 2024-09-18T12:32:42.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient: init 2024-09-18T12:32:42.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79e8096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79e80971e0 0x7f79e80975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient(hunting): picked mon.noname-a con 0x7f79e80971e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79e8133b30 0x7f79e8135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient(hunting): picked mon.noname-c con 0x7f79e8133b30 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e8099620 con 0x7f79e8096e10 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e8099340 con 0x7f79e80971e0 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e8099900 con 0x7f79e8133b30 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:42.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79f06ef640 10 monclient(hunting): authenticate will time out at 1189.335777s 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79edc8a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79e80971e0 0x7f79e80975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79ee48b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f79e8096e10 0x7f79e809a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42580/0 (socket says 172.21.15.116:42580) 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79ee48b640 1 -- 172.21.15.116:0/967426600 learned_addr learned my addr 172.21.15.116:0/967426600 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79edc8a640 1 --2- 172.21.15.116:0/967426600 >> v2:172.21.15.116:3300/0 conn(0x7f79e80971e0 0x7f79e80975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44030/0 (socket says 172.21.15.116:44030) 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79edc8a640 10 monclient(hunting): get_auth_request con 0x7f79e80971e0 auth_method 0 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79edc8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79edc8a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79edc8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.013+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79eec8c640 1 --2- 172.21.15.116:0/967426600 >> v2:172.21.15.144:3300/0 conn(0x7f79e8133b30 0x7f79e8135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 10 monclient(hunting): handle_auth_done global_id 104157 payload 274 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 1 -- 172.21.15.116:0/967426600 >> v2:172.21.15.144:3300/0 conn(0x7f79e8133b30 msgr2=0x7f79e8135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 1 --2- 172.21.15.116:0/967426600 >> v2:172.21.15.144:3300/0 conn(0x7f79e8133b30 0x7f79e8135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 1 -- 172.21.15.116:0/967426600 >> v1:172.21.15.116:6789/0 conn(0x7f79e8096e10 legacy=0x7f79e809a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 1 monclient: found mon.noname-a 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.014+0000 7f79edc8a640 1 -- 172.21.15.116:0/967426600 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e809ac70 con 0x7f79e80971e0 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.016+0000 7f79edc8a640 1 -- 172.21.15.116:0/967426600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80971e0 msgr2=0x7f79e80975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.016+0000 7f79edc8a640 1 --2- 172.21.15.116:0/967426600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80971e0 0x7f79e80975e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f79e809b020 tx=0x7f79d80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f11b3d26a2510a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 1 -- 172.21.15.116:0/967426600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d802ae90 con 0x7f79e80971e0 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: dump: 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.029 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: _finish_auth 0 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 1 -- 172.21.15.116:0/967426600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d8032440 con 0x7f79e80971e0 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 1 -- 172.21.15.116:0/967426600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d80327e0 con 0x7f79e80971e0 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79ed489640 10 monclient: dump: 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 5 monclient: authenticate success, global_id 104157 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 4 set_mon_vals no callback set 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 10 monclient: shutdown 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 -- 172.21.15.116:0/967426600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80971e0 msgr2=0x7f79e80975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 --2- 172.21.15.116:0/967426600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80971e0 0x7f79e80975e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f79e809b020 tx=0x7f79d80090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 -- 172.21.15.116:0/967426600 shutdown_connections 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 --2- 172.21.15.116:0/967426600 >> v2:172.21.15.144:3300/0 conn(0x7f79e8133b30 0x7f79e8135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 --2- 172.21.15.116:0/967426600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e80971e0 0x7f79e80975e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 -- 172.21.15.116:0/967426600 >> 172.21.15.116:0/967426600 conn(0x7f79e80672b0 msgr2=0x7f79e80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 -- 172.21.15.116:0/967426600 shutdown_connections 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 -- 172.21.15.116:0/967426600 wait complete. 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 10 monclient: build_initial_monmap 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 10 monclient: monmap: 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 Processor -- start 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 1 -- start start 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.018+0000 7f79f06ef640 10 monclient: init 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e8096e10 0x7f79e809b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): picked mon.noname-a con 0x7f79e8096e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80971e0 0x7f79e812e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): picked mon.noname-b con 0x7f79e80971e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8133b30 0x7f79e8119690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): picked mon.noname-c con 0x7f79e8133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e809b050 con 0x7f79e8096e10 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e812f790 con 0x7f79e80971e0 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e811b900 con 0x7f79e8133b30 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79f06ef640 10 monclient(hunting): authenticate will time out at 1189.341299s 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e8096e10 0x7f79e809b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8133b30 0x7f79e8119690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e8096e10 0x7f79e809b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44046/0 (socket says 172.21.15.116:44046) 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 -- 172.21.15.116:0/557086467 learned_addr learned my addr 172.21.15.116:0/557086467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): get_auth_request con 0x7f79e8096e10 auth_method 0 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 10 monclient(hunting): get_auth_request con 0x7f79e8133b30 auth_method 0 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79edc8a640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80971e0 0x7f79e812e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79edc8a640 10 monclient(hunting): get_auth_request con 0x7f79e80971e0 auth_method 0 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79edc8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79edc8a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79edc8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient(hunting): handle_auth_done global_id 104160 payload 824 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8133b30 msgr2=0x7f79e8119690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8133b30 0x7f79e8119690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80971e0 msgr2=0x7f79e812e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80971e0 0x7f79e812e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 monclient: found mon.noname-a 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79ee48b640 1 -- 172.21.15.116:0/557086467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8009390 con 0x7f79e8096e10 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79eec8c640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e8133b30 0x7f79e8119690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.019+0000 7f79edc8a640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79e80971e0 0x7f79e812e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.021+0000 7f79ee48b640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e8096e10 msgr2=0x7f79e809b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.021+0000 7f79ee48b640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e8096e10 0x7f79e809b810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f79e400c5e0 tx=0x7f79e4004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4005cb0 con 0x7f79e8096e10 2024-09-18T12:32:42.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: dump: 2024-09-18T12:32:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.040 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.041 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e4005e30 con 0x7f79e8096e10 2024-09-18T12:32:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 5 monclient: authenticate success, global_id 104160 2024-09-18T12:32:42.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 10 monclient: _renew_subs 2024-09-18T12:32:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4007340 con 0x7f79e8096e10 2024-09-18T12:32:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e809be50 con 0x7f79e8096e10 2024-09-18T12:32:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 10 monclient: dump: 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.043 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.044 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79ecc88640 4 set_mon_vals no callback set 2024-09-18T12:32:42.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 10 monclient: _renew_subs 2024-09-18T12:32:42.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:42.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e81343d0 con 0x7f79e8096e10 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f79e401b400 con 0x7f79e8096e10 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 0x7f79e8119bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8065f20 con 0x7f79e812ea40 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.022+0000 7f79de7fc640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79cc0384e0 0x7f79cc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f79e404c3c0 con 0x7f79e8096e10 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 0x7f79e8119bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient: get_auth_request con 0x7f79e812ea40 auth_method 0 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 10 monclient(hunting): handle_auth_done global_id 64994 payload 824 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79edc8a640 1 -- 172.21.15.116:0/557086467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79e804bf60 con 0x7f79e812ea40 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79eec8c640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79cc0384e0 0x7f79cc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.023+0000 7f79eec8c640 10 monclient: get_auth_request con 0x7f79cc0384e0 auth_method 0 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79edc8a640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 msgr2=0x7f79e8119bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79edc8a640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 0x7f79e8119bd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f79d8034000 tx=0x7f79d8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8031e00 con 0x7f79e812ea40 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 10 monclient: dump: 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.049 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79eec8c640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79cc0384e0 0x7f79cc03a990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d990 tx=0x7f79e00077b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f79d802ae90 con 0x7f79e812ea40 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 msgr2=0x7f79e8119bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 0x7f79e8119bd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f79d8034000 tx=0x7f79d8032000 comp rx=0 tx=0).stop 2024-09-18T12:32:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 msgr2=0x7f79e8119bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.025+0000 7f79de7fc640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e812ea40 0x7f79e8119bd0 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f79d8034000 tx=0x7f79d8032000 comp rx=0 tx=0).stop 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79f06ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79f06ef640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 0x7f79e806b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79f06ef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e812ee10 con 0x7f79e811a110 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 0x7f79e806b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient: get_auth_request con 0x7f79e811a110 auth_method 0 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 10 monclient(hunting): handle_auth_done global_id 64997 payload 824 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.032+0000 7f79edc8a640 1 -- 172.21.15.116:0/557086467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f79e811a510 con 0x7f79e811a110 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79edc8a640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 msgr2=0x7f79e806b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79edc8a640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 0x7f79e806b6b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79d803e000 tx=0x7f79d803c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d803cc90 con 0x7f79e811a110 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 10 monclient: dump: 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f79d803cdf0 con 0x7f79e811a110 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 msgr2=0x7f79e806b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 0x7f79e806b6b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79d803e000 tx=0x7f79d803c000 comp rx=0 tx=0).stop 2024-09-18T12:32:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 msgr2=0x7f79e806b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79de7fc640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f79e811a110 0x7f79e806b6b0 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79d803e000 tx=0x7f79d803c000 comp rx=0 tx=0).stop 2024-09-18T12:32:42.054 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:42.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:42.054 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:42.054 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 82, 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:42.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:42.056 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:42.057 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:42.058 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:42.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:42.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:42.062 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:42.063 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.064 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.064 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:42.064 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:42.064 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.066 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:42.067 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:42.068 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79cc0384e0 msgr2=0x7f79cc03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79cc0384e0 0x7f79cc03a990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d990 tx=0x7f79e00077b0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 10 monclient: shutdown 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e8096e10 msgr2=0x7f79e809b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 --2- 172.21.15.116:0/557086467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e8096e10 0x7f79e809b810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f79e400c5e0 tx=0x7f79e4004960 comp rx=0 tx=0).stop 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79eec8c640 1 -- 172.21.15.116:0/557086467 reap_dead start 2024-09-18T12:32:42.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79eec8c640 1 -- 172.21.15.116:0/557086467 reap_dead start 2024-09-18T12:32:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 shutdown_connections 2024-09-18T12:32:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 >> 172.21.15.116:0/557086467 conn(0x7f79e80672b0 msgr2=0x7f79e809fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 shutdown_connections 2024-09-18T12:32:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.034+0000 7f79f06ef640 1 -- 172.21.15.116:0/557086467 wait complete. 2024-09-18T12:32:42.071 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:32:42.072 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:32:42.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:32:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.164+0000 7ff56c44f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.164+0000 7ff56c44f640 10 monclient: build_initial_monmap 2024-09-18T12:32:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.164+0000 7ff56c44f640 10 monclient: monmap: 2024-09-18T12:32:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:42.165436+0000 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:42.165436+0000 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:42.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.165+0000 7ff56c44f640 1 Processor -- start 2024-09-18T12:32:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.165+0000 7ff56c44f640 1 -- start start 2024-09-18T12:32:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.165+0000 7ff56c44f640 10 monclient: init 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff564096e10 0x7ff56409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient(hunting): picked mon.noname-c con 0x7ff564096e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 0x7ff5640971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient(hunting): picked mon.noname-b con 0x7ff56409aaa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff564097720 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:42.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564099620 con 0x7ff564097720 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff564099340 con 0x7ff564096e10 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff564099900 con 0x7ff56409aaa0 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56c44f640 10 monclient(hunting): authenticate will time out at 1189.487127s 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 0x7ff5640971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 0x7ff5640971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43046/0 (socket says 172.21.15.116:43046) 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 -- 172.21.15.116:0/3892729034 learned_addr learned my addr 172.21.15.116:0/3892729034 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): get_auth_request con 0x7ff56409aaa0 auth_method 0 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56a1eb640 1 --2- 172.21.15.116:0/3892729034 >> v2:172.21.15.144:3300/0 conn(0x7ff564096e10 0x7ff56409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56a1eb640 10 monclient(hunting): get_auth_request con 0x7ff564096e10 auth_method 0 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56a1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56a1eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff56a1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient(hunting): handle_auth_done global_id 65000 payload 274 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 -- 172.21.15.116:0/3892729034 >> v2:172.21.15.144:3300/0 conn(0x7ff564096e10 msgr2=0x7ff56409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 --2- 172.21.15.116:0/3892729034 >> v2:172.21.15.144:3300/0 conn(0x7ff564096e10 0x7ff56409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 -- 172.21.15.116:0/3892729034 >> v1:172.21.15.144:6789/0 conn(0x7ff564097720 legacy=0x7ff56412e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient: _finish_hunting 0 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 monclient: found mon.noname-b 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.166+0000 7ff5699ea640 1 -- 172.21.15.116:0/3892729034 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff564097b10 con 0x7ff56409aaa0 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5699ea640 1 -- 172.21.15.116:0/3892729034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 msgr2=0x7ff5640971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5699ea640 1 --2- 172.21.15.116:0/3892729034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 0x7ff5640971e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff554009910 tx=0x7ff5540091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae8247831c43750f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 1 -- 172.21.15.116:0/3892729034 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55402ae90 con 0x7ff56409aaa0 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: dump: 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: _finish_auth 0 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff56c44f640 5 monclient: authenticate success, global_id 65000 2024-09-18T12:32:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff56c44f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 1 -- 172.21.15.116:0/3892729034 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff554032440 con 0x7ff56409aaa0 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff56c44f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 1 -- 172.21.15.116:0/3892729034 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5540327e0 con 0x7ff56409aaa0 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff56c44f640 4 set_mon_vals no callback set 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.169+0000 7ff5691e9640 10 monclient: dump: 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 10 monclient: shutdown 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 -- 172.21.15.116:0/3892729034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 msgr2=0x7ff5640971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 --2- 172.21.15.116:0/3892729034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 0x7ff5640971e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff554009910 tx=0x7ff5540091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 -- 172.21.15.116:0/3892729034 shutdown_connections 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 --2- 172.21.15.116:0/3892729034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff56409aaa0 0x7ff5640971e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 --2- 172.21.15.116:0/3892729034 >> v2:172.21.15.144:3300/0 conn(0x7ff564096e10 0x7ff56409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 -- 172.21.15.116:0/3892729034 >> 172.21.15.116:0/3892729034 conn(0x7ff5640672b0 msgr2=0x7ff5640676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 -- 172.21.15.116:0/3892729034 shutdown_connections 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 1 -- 172.21.15.116:0/3892729034 wait complete. 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 10 monclient: build_initial_monmap 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.170+0000 7ff56c44f640 10 monclient: monmap: 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 Processor -- start 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 -- start start 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient: init 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff564096e10 0x7ff56413de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): picked mon.noname-c con 0x7ff564096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff564097720 0x7ff56412db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): picked mon.noname-a con 0x7ff564097720 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409aaa0 0x7ff564117840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): picked mon.noname-b con 0x7ff56409aaa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56412e890 con 0x7ff564097720 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56411b1f0 con 0x7ff56409aaa0 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56411af10 con 0x7ff564096e10 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56c44f640 10 monclient(hunting): authenticate will time out at 1189.492533s 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff564096e10 0x7ff56413de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff564096e10 0x7ff56413de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43048/0 (socket says 172.21.15.116:43048) 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 1 -- 172.21.15.116:0/4098876010 learned_addr learned my addr 172.21.15.116:0/4098876010 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 10 monclient(hunting): get_auth_request con 0x7ff564096e10 auth_method 0 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff564097720 0x7ff56412db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): get_auth_request con 0x7ff564097720 auth_method 0 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a9ec640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409aaa0 0x7ff564117840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient(hunting): handle_auth_done global_id 104163 payload 824 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff564096e10 msgr2=0x7ff56413de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff564096e10 0x7ff56413de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409aaa0 msgr2=0x7ff564117840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409aaa0 0x7ff564117840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient: _finish_hunting 0 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 1 monclient: found mon.noname-a 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff5699ea640 1 -- 172.21.15.116:0/4098876010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff554009450 con 0x7ff564097720 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a9ec640 10 monclient: get_auth_request con 0x7ff56409aaa0 auth_method 0 2024-09-18T12:32:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a1eb640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff564096e10 0x7ff56413de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.171+0000 7ff56a9ec640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409aaa0 0x7ff564117840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff5699ea640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff564097720 msgr2=0x7ff56412db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff5699ea640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff564097720 0x7ff56412db10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff554031000 tx=0x7ff554032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554031d40 con 0x7ff564097720 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: dump: 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff56c44f640 5 monclient: authenticate success, global_id 104163 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55402ae90 con 0x7ff564097720 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff56c44f640 10 monclient: _renew_subs 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55403d690 con 0x7ff564097720 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff56c44f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56413ece0 con 0x7ff564097720 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff55a7fc640 10 monclient: dump: 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.195 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff5689e8640 4 set_mon_vals no callback set 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff56c44f640 10 monclient: _renew_subs 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff56c44f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.174+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff564119180 con 0x7ff564097720 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56c44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56c44f640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 0x7ff56409b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56c44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff564065f20 con 0x7ff56412e050 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff554039040 con 0x7ff564097720 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 0x7ff56409b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient: get_auth_request con 0x7ff56412e050 auth_method 0 2024-09-18T12:32:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff55a7fc640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff548038490 0x7ff54803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7ff55406bff0 con 0x7ff564097720 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a9ec640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff548038490 0x7ff54803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a9ec640 10 monclient: get_auth_request con 0x7ff548038490 auth_method 0 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_done global_id 6208 payload 824 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.175+0000 7ff56a1eb640 1 -- 172.21.15.116:0/4098876010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff56411b360 con 0x7ff56412e050 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.177+0000 7ff56a1eb640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 msgr2=0x7ff56409b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.177+0000 7ff56a1eb640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 0x7ff56409b7d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff560010ce0 tx=0x7ff560008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff56a9ec640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff548038490 0x7ff54803a940 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff55c007fd0 tx=0x7ff55c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560014640 con 0x7ff56412e050 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 10 monclient: dump: 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff5600147a0 con 0x7ff56412e050 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 msgr2=0x7ff56409b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 0x7ff56409b7d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff560010ce0 tx=0x7ff560008040 comp rx=0 tx=0).stop 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 msgr2=0x7ff56409b7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.178+0000 7ff55a7fc640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56412e050 0x7ff56409b7d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.183+0000 7ff56c44f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56c44f640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 0x7ff56409c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56c44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56412e420 con 0x7ff56409bd10 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 0x7ff56409c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient: get_auth_request con 0x7ff56409bd10 auth_method 0 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 10 monclient(hunting): handle_auth_done global_id 6211 payload 824 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.184+0000 7ff56a1eb640 1 -- 172.21.15.116:0/4098876010 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff56404bf60 con 0x7ff56409bd10 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.186+0000 7ff56a1eb640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 msgr2=0x7ff56409c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.186+0000 7ff56a1eb640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 0x7ff56409c110 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff560007b50 tx=0x7ff56001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5600137f0 con 0x7ff56409bd10 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 10 monclient: dump: 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7ff560014430 con 0x7ff56409bd10 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 msgr2=0x7ff56409c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 0x7ff56409c110 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff560007b50 tx=0x7ff56001c000 comp rx=0 tx=0).stop 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 msgr2=0x7ff56409c110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff55a7fc640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff56409bd10 0x7ff56409c110 secure :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff560007b50 tx=0x7ff56001c000 comp rx=0 tx=0).stop 2024-09-18T12:32:42.203 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 82, 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:42.204 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:42.205 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:42.206 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:42.207 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.208 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.209 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:42.210 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:42.212 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:42.213 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:42.214 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:42.215 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:42.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:42.216 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:42.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:42.216 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:42.216 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff548038490 msgr2=0x7ff54803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff548038490 0x7ff54803a940 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff55c007fd0 tx=0x7ff55c009040 comp rx=0 tx=0).stop 2024-09-18T12:32:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 10 monclient: shutdown 2024-09-18T12:32:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff564097720 msgr2=0x7ff56412db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff564097720 0x7ff56412db10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff554031000 tx=0x7ff554032000 comp rx=0 tx=0).stop 2024-09-18T12:32:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56a9ec640 1 -- 172.21.15.116:0/4098876010 reap_dead start 2024-09-18T12:32:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 shutdown_connections 2024-09-18T12:32:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 --2- 172.21.15.116:0/4098876010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff564097720 0x7ff56412db10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 >> 172.21.15.116:0/4098876010 conn(0x7ff5640672b0 msgr2=0x7ff56409f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 shutdown_connections 2024-09-18T12:32:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.187+0000 7ff56c44f640 1 -- 172.21.15.116:0/4098876010 wait complete. 2024-09-18T12:32:42.219 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:32:42.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T12:32:42.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:32:42.219 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T12:32:42.219 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:32:42.219 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:32:42.219 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 msgr2=0x7f4f341347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 msgr2=0x7f4f341347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 0x7f4f341347c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c890 tx=0x7f4f2c002640 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 0x7f4f341347c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c890 tx=0x7f4f2c002640 comp rx=0 tx=0).stop 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f1bfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f1bfff640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 msgr2=0x7f4f341347c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f1bfff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f341323d0 0x7f4f341347c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.224+0000 7f4f1bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f20043a10 0x7f4f20045e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 10 monclient(hunting): picked mon.b con 0x7f4f20043a10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f337fe640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f20043a10 msgr2=0x7f4f20045e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f20048710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f337fe640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f20043a10 0x7f4f20045e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 10 monclient(hunting): picked mon.c con 0x7f4f3409ff50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 0x7f4f2004b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 10 monclient(hunting): picked mon.a con 0x7f4f34131ac0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2004bad0 con 0x7f4f34131ac0 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2004bc10 con 0x7f4f20043a10 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2004bda0 con 0x7f4f3409ff50 2024-09-18T12:32:42.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f1bfff640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:42.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f20048710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f38fac640 10 monclient(hunting): get_auth_request con 0x7f4f3409ff50 auth_method 0 2024-09-18T12:32:42.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f38fac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f38fac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f38fac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f33fff640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 0x7f4f2004b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f33fff640 10 monclient(hunting): get_auth_request con 0x7f4f34131ac0 auth_method 0 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.225+0000 7f4f33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient(hunting): handle_auth_done global_id 6193 payload 293 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f20043a10 msgr2=0x7f4f20045e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f20043a10 0x7f4f20045e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 msgr2=0x7f4f2004b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 0x7f4f2004b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient: _finish_hunting 0 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 1 monclient: found mon.c 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f4f2004bf30 con 0x7f4f3409ff50 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4f34138af0 con 0x7f4f3409ff50 2024-09-18T12:32:42.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 10 monclient: _finish_auth 0 2024-09-18T12:32:42.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.226+0000 7f4f38fac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:42.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.230+0000 7f4f38fac640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 msgr2=0x7f4f20048710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.230+0000 7f4f38fac640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f20048710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c04a480 tx=0x7f4f2c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c022770 con 0x7f4f3409ff50 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f1bfff640 10 monclient: dump: 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.233 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.234 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:42.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c011cf0 con 0x7f4f3409ff50 2024-09-18T12:32:42.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f1bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:42.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f327fc640 4 set_mon_vals no callback set 2024-09-18T12:32:42.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:42.231+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4f2c022300 con 0x7f4f3409ff50 2024-09-18T12:32:42.235 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:32:42.235 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:32:42.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:42.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.326+0000 7f9e6b51d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.326+0000 7f9e6b51d640 10 monclient: build_initial_monmap 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.326+0000 7f9e6b51d640 10 monclient: monmap: 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:42.326741+0000 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:42.326741+0000 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:42.327 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:42.328 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.326+0000 7f9e6b51d640 1 Processor -- start 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.326+0000 7f9e6b51d640 1 -- start start 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.326+0000 7f9e6b51d640 10 monclient: init 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e640981c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 0x7f9e64136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e64131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e64132170 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6409ca60 con 0x7f9e640981c0 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6409cca0 con 0x7f9e64132170 2024-09-18T12:32:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64131bd0 con 0x7f9e64131da0 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e6b51d640 10 monclient(hunting): authenticate will time out at 1189.648393s 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 0x7f9e64136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 0x7f9e64136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43070/0 (socket says 172.21.15.116:43070) 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 1 -- 172.21.15.116:0/1061987054 learned_addr learned my addr 172.21.15.116:0/1061987054 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): get_auth_request con 0x7f9e64131da0 auth_method 0 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e69aba640 1 -- 172.21.15.116:0/1061987054 >> v1:172.21.15.144:6789/0 conn(0x7f9e64132170 legacy=0x7f9e641390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient(hunting): handle_auth_done global_id 65006 payload 274 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 1 -- 172.21.15.116:0/1061987054 >> v1:172.21.15.144:6789/0 conn(0x7f9e64132170 legacy=0x7f9e641390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 1 -- 172.21.15.116:0/1061987054 >> v1:172.21.15.116:6789/0 conn(0x7f9e640981c0 legacy=0x7f9e64098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient: _finish_hunting 0 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 1 monclient: found mon.noname-b 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.327+0000 7f9e68ab8640 1 -- 172.21.15.116:0/1061987054 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6409e010 con 0x7f9e64131da0 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e68ab8640 1 -- 172.21.15.116:0/1061987054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 msgr2=0x7f9e64136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e68ab8640 1 --2- 172.21.15.116:0/1061987054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 0x7f9e64136b70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009940 tx=0x7f9e540091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcc769b89cc2934 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 1 -- 172.21.15.116:0/1061987054 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54032440 con 0x7f9e64131da0 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: dump: 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: _finish_auth 0 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e6b51d640 5 monclient: authenticate success, global_id 65006 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e6b51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 1 -- 172.21.15.116:0/1061987054 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e540325a0 con 0x7f9e64131da0 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e6b51d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e6b51d640 4 set_mon_vals no callback set 2024-09-18T12:32:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 1 -- 172.21.15.116:0/1061987054 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e54032980 con 0x7f9e64131da0 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.330+0000 7f9e5bfff640 10 monclient: dump: 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 10 monclient: shutdown 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1061987054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 msgr2=0x7f9e64136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1061987054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 0x7f9e64136b70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009940 tx=0x7f9e540091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1061987054 shutdown_connections 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1061987054 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64131da0 0x7f9e64136b70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1061987054 >> 172.21.15.116:0/1061987054 conn(0x7f9e640a1e30 msgr2=0x7f9e640a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1061987054 shutdown_connections 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1061987054 wait complete. 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 10 monclient: build_initial_monmap 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.331+0000 7f9e6b51d640 10 monclient: monmap: 2024-09-18T12:32:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:42.340 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 Processor -- start 2024-09-18T12:32:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 -- start start 2024-09-18T12:32:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient: init 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e640981c0 0x7f9e64138260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e640981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e64131da0 0x7f9e641387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e64131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64132170 0x7f9e64138ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e64132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:42.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64119580 con 0x7f9e640981c0 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64119bd0 con 0x7f9e64131da0 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64118690 con 0x7f9e64132170 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e6b51d640 10 monclient(hunting): authenticate will time out at 1189.653817s 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e692b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e640981c0 0x7f9e64138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e69aba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64132170 0x7f9e64138ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e692b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e640981c0 0x7f9e64138260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44066/0 (socket says 172.21.15.116:44066) 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e692b9640 1 -- 172.21.15.116:0/1129897749 learned_addr learned my addr 172.21.15.116:0/1129897749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e692b9640 10 monclient(hunting): get_auth_request con 0x7f9e640981c0 auth_method 0 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e692b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e692b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e692b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e69aba640 10 monclient(hunting): get_auth_request con 0x7f9e64132170 auth_method 0 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e69aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e69aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e69aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e68ab8640 1 -- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e64131da0 msgr2=0x7f9e641387a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.332+0000 7f9e68ab8640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e64131da0 0x7f9e641387a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e692b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e692b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e692b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 10 monclient(hunting): handle_auth_done global_id 65009 payload 824 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 1 -- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e64131da0 msgr2=0x7f9e641387a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e64131da0 0x7f9e641387a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 1 -- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e640981c0 msgr2=0x7f9e64138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e640981c0 0x7f9e64138260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 10 monclient: _finish_hunting 0 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 1 monclient: found mon.noname-c 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e69aba640 1 -- 172.21.15.116:0/1129897749 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54009450 con 0x7f9e64132170 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.333+0000 7f9e692b9640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e640981c0 0x7f9e64138260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e69aba640 1 -- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64132170 msgr2=0x7f9e64138ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e69aba640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64132170 0x7f9e64138ce0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009fd0 tx=0x7f9e5c009740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 1 -- 172.21.15.116:0/1129897749 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c006150 con 0x7f9e64132170 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: dump: 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: _finish_auth 0 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 1 -- 172.21.15.116:0/1129897749 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c0062b0 con 0x7f9e64132170 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e6b51d640 5 monclient: authenticate success, global_id 65009 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e6b51d640 10 monclient: _renew_subs 2024-09-18T12:32:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e6b51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6413a360 con 0x7f9e64132170 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 1 -- 172.21.15.116:0/1129897749 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c006440 con 0x7f9e64132170 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e597fa640 10 monclient: dump: 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e5b7fe640 4 set_mon_vals no callback set 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e6b51d640 10 monclient: _renew_subs 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e6b51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.335+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e64139de0 con 0x7f9e64132170 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e597fa640 1 -- 172.21.15.116:0/1129897749 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9e5c012020 con 0x7f9e64132170 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e6b51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e6b51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e6b51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6404bf60 con 0x7f9e64132170 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e597fa640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e48038440 0x7f9e4803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e597fa640 1 -- 172.21.15.116:0/1129897749 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f9e5c04ca10 con 0x7f9e64132170 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.336+0000 7f9e692b9640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e48038440 0x7f9e4803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.337+0000 7f9e692b9640 10 monclient: get_auth_request con 0x7f9e48038440 auth_method 0 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.339+0000 7f9e692b9640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e48038440 0x7f9e4803a8f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9e60002ed0 tx=0x7f9e60007ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.341+0000 7f9e597fa640 1 -- 172.21.15.116:0/1129897749 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9e5c0068b0 con 0x7f9e64132170 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.341+0000 7f9e597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.341+0000 7f9e597fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.557+0000 7f9e6b51d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.557+0000 7f9e6b51d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.557+0000 7f9e6b51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.557+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e64098ee0 con 0x7f9e64132170 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.557+0000 7f9e597fa640 1 -- 172.21.15.116:0/1129897749 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f9e5c007300 con 0x7f9e64132170 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.557+0000 7f9e597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.557+0000 7f9e597fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:42.558 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e48038440 msgr2=0x7f9e4803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e48038440 0x7f9e4803a8f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9e60002ed0 tx=0x7f9e60007ed0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 10 monclient: shutdown 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64132170 msgr2=0x7f9e64138ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64132170 0x7f9e64138ce0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009fd0 tx=0x7f9e5c009740 comp rx=0 tx=0).stop 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 shutdown_connections 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e48038440 0x7f9e4803a8f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e64132170 0x7f9e64138ce0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e64131da0 0x7f9e641387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 --2- 172.21.15.116:0/1129897749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e640981c0 0x7f9e64138260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 >> 172.21.15.116:0/1129897749 conn(0x7f9e640a1e30 msgr2=0x7f9e64096f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 shutdown_connections 2024-09-18T12:32:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:42.559+0000 7f9e6b51d640 1 -- 172.21.15.116:0/1129897749 wait complete. 2024-09-18T12:32:44.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:44.651+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:44.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:44.651+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:44.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:44.651+0000 7f4f1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:45.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:45.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:45.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:45.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:45.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:45.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.674+0000 7f6a2951d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.674+0000 7f6a2951d640 10 monclient: build_initial_monmap 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.674+0000 7f6a2951d640 10 monclient: monmap: 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:45.675621+0000 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:45.675621+0000 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:45.676 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:45.677 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:45.677 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:45.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.675+0000 7f6a2951d640 1 Processor -- start 2024-09-18T12:32:45.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.675+0000 7f6a2951d640 1 -- start start 2024-09-18T12:32:45.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.675+0000 7f6a2951d640 10 monclient: init 2024-09-18T12:32:45.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:45.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T12:32:45.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a2409c860 0x7f6a2409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:45.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2409c860 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a2409aba0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a2409afa0 0x7f6a2409df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2409afa0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2406a320 con 0x7f6a2409aba0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a24068a20 con 0x7f6a2409afa0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a24068e60 con 0x7f6a2409c860 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a2951d640 10 monclient(hunting): authenticate will time out at 1192.997197s 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a227fc640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f6a2409aba0 legacy=0x7f6a2409d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a2409afa0 0x7f6a2409df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a22ffd640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f6a2409c860 msgr2=0x7f6a2409cc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:45.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a22ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a2409c860 0x7f6a2409cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a2409afa0 0x7f6a2409df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48900/0 (socket says 172.21.15.116:48900) 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 learned_addr learned my addr 172.21.15.116:0/3190077997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): get_auth_request con 0x7f6a2409afa0 auth_method 0 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient(hunting): handle_auth_done global_id 104169 payload 274 2024-09-18T12:32:45.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 >> v2:172.21.15.144:3300/0 conn(0x7f6a2409c860 msgr2=0x7f6a2409cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:45.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 --2- 172.21.15.116:0/3190077997 >> v2:172.21.15.144:3300/0 conn(0x7f6a2409c860 0x7f6a2409cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 >> v1:172.21.15.144:6789/0 conn(0x7f6a2409aba0 legacy=0x7f6a2409d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:45.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:45.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 monclient: found mon.noname-a 2024-09-18T12:32:45.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:45.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:45.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.676+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2406a040 con 0x7f6a2409afa0 2024-09-18T12:32:45.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 msgr2=0x7f6a2409df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a237fe640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a2409df40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a24071740 tx=0x7f6a1c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8520704614d69a55 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 msgr2=0x7f6a2409df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 msgr2=0x7f6a2409df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a237fe640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a2409df40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a24071740 tx=0x7f6a1c009840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a237fe640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a2409df40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a24071740 tx=0x7f6a1c009840 comp rx=0 tx=0).stop 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 1 -- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 msgr2=0x7f6a2409df40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a2409df40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient: _add_conns ranks=[0,2] 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a0c0034c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:45.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a0c009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0c0073e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 1 -- 172.21.15.116:0/3190077997 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c00a130 con 0x7f6a0c0034c0 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 1 -- 172.21.15.116:0/3190077997 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c00b2f0 con 0x7f6a0c0073e0 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a21ffb640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a0c009890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): get_auth_request con 0x7f6a0c0073e0 auth_method 0 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:45.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient(hunting): handle_auth_done global_id 104169 payload 293 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 1 -- 172.21.15.116:0/3190077997 >> v1:172.21.15.116:6789/0 conn(0x7f6a0c0034c0 legacy=0x7f6a0c005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient: _finish_hunting 0 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 1 monclient: found mon.noname-b 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.679+0000 7f6a227fc640 1 -- 172.21.15.116:0/3190077997 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c00b570 con 0x7f6a0c0073e0 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a227fc640 1 -- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 msgr2=0x7f6a0c009890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a227fc640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a0c009890 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009940 tx=0x7f6a18009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c180edf7e90dc704 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 1 -- 172.21.15.116:0/3190077997 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a180092b0 con 0x7f6a0c0073e0 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:45.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: dump: 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:45.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 1 -- 172.21.15.116:0/3190077997 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a18003b80 con 0x7f6a0c0073e0 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 5 monclient: authenticate success, global_id 104169 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 4 set_mon_vals no callback set 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 1 -- 172.21.15.116:0/3190077997 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a180050f0 con 0x7f6a0c0073e0 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a21ffb640 10 monclient: dump: 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:45.694 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 10 monclient: shutdown 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 1 -- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 msgr2=0x7f6a0c009890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a0c009890 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009940 tx=0x7f6a18009060 comp rx=0 tx=0).stop 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a237fe640 1 -- 172.21.15.116:0/3190077997 reap_dead start 2024-09-18T12:32:45.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 1 -- 172.21.15.116:0/3190077997 shutdown_connections 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 1 --2- 172.21.15.116:0/3190077997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a2409df40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 1 -- 172.21.15.116:0/3190077997 >> 172.21.15.116:0/3190077997 conn(0x7f6a240a2e40 msgr2=0x7f6a240a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 1 -- 172.21.15.116:0/3190077997 shutdown_connections 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.682+0000 7f6a2951d640 1 -- 172.21.15.116:0/3190077997 wait complete. 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.683+0000 7f6a2951d640 10 monclient: build_initial_monmap 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.683+0000 7f6a2951d640 10 monclient: monmap: 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:45.696 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.683+0000 7f6a2951d640 1 Processor -- start 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.683+0000 7f6a2951d640 1 -- start start 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient: init 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a24140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2409afa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a24134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0c0073e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:45.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a24120e40 0x7f6a2411e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a24120e40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a241222f0 con 0x7f6a2409afa0 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240a1280 con 0x7f6a24120e40 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24121d30 con 0x7f6a0c0073e0 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a2951d640 10 monclient(hunting): authenticate will time out at 1193.005477s 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a24140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a24140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48912/0 (socket says 172.21.15.116:48912) 2024-09-18T12:32:45.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 -- 172.21.15.116:0/2564993166 learned_addr learned my addr 172.21.15.116:0/2564993166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): get_auth_request con 0x7f6a2409afa0 auth_method 0 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a24134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 10 monclient(hunting): get_auth_request con 0x7f6a0c0073e0 auth_method 0 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a237fe640 1 -- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a24120e40 msgr2=0x7f6a2411e6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a237fe640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a24120e40 0x7f6a2411e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:45.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient(hunting): handle_auth_done global_id 104172 payload 824 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 -- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 msgr2=0x7f6a24134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a24134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 -- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a24120e40 msgr2=0x7f6a2411e6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a24120e40 0x7f6a2411e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 monclient: found mon.noname-a 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.684+0000 7f6a22ffd640 1 -- 172.21.15.116:0/2564993166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a24122010 con 0x7f6a2409afa0 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a22ffd640 1 -- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 msgr2=0x7f6a24140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a22ffd640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a24140dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400b5c0 tx=0x7f6a140090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 1 -- 172.21.15.116:0/2564993166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14003d90 con 0x7f6a2409afa0 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: dump: 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: _finish_auth 0 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a2951d640 5 monclient: authenticate success, global_id 104172 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 1 -- 172.21.15.116:0/2564993166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a14006150 con 0x7f6a2409afa0 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 1 -- 172.21.15.116:0/2564993166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14006510 con 0x7f6a2409afa0 2024-09-18T12:32:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a2951d640 10 monclient: _renew_subs 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a2951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a24141310 con 0x7f6a2409afa0 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a0affd640 10 monclient: dump: 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a217fa640 4 set_mon_vals no callback set 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a2951d640 10 monclient: _renew_subs 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a2951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.687+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a241352f0 con 0x7f6a2409afa0 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a0affd640 1 -- 172.21.15.116:0/2564993166 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6a140103c0 con 0x7f6a2409afa0 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a0affd640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a10038470 0x7f6a1003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a2951d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a2951d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a2951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a0affd640 1 -- 172.21.15.116:0/2564993166 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f6a1404bfd0 con 0x7f6a2409afa0 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a24122460 con 0x7f6a2409afa0 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a227fc640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a10038470 0x7f6a1003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.688+0000 7f6a227fc640 10 monclient: get_auth_request con 0x7f6a10038470 auth_method 0 2024-09-18T12:32:45.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.691+0000 7f6a227fc640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a10038470 0x7f6a1003a920 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6a180038f0 tx=0x7f6a18015000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:45.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.693+0000 7f6a0affd640 1 -- 172.21.15.116:0/2564993166 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6a14017030 con 0x7f6a2409afa0 2024-09-18T12:32:45.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.693+0000 7f6a0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:45.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.693+0000 7f6a0affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.905+0000 7f6a2951d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.905+0000 7f6a2951d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.905+0000 7f6a2951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.905+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6a2404bf60 con 0x7f6a2409afa0 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.906+0000 7f6a0affd640 1 -- 172.21.15.116:0/2564993166 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f6a14017030 con 0x7f6a2409afa0 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.906+0000 7f6a0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.906+0000 7f6a0affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:45.907 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a10038470 msgr2=0x7f6a1003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a10038470 0x7f6a1003a920 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6a180038f0 tx=0x7f6a18015000 comp rx=0 tx=0).stop 2024-09-18T12:32:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 10 monclient: shutdown 2024-09-18T12:32:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 msgr2=0x7f6a24140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a24140dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400b5c0 tx=0x7f6a140090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 shutdown_connections 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a10038470 0x7f6a1003a920 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a24120e40 0x7f6a2411e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a0c0073e0 0x7f6a24134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 --2- 172.21.15.116:0/2564993166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2409afa0 0x7f6a24140dd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 >> 172.21.15.116:0/2564993166 conn(0x7f6a240a2e40 msgr2=0x7f6a24098db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 shutdown_connections 2024-09-18T12:32:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:45.908+0000 7f6a2951d640 1 -- 172.21.15.116:0/2564993166 wait complete. 2024-09-18T12:32:46.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:46.651+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:46.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:46.651+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:47.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:47.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:47.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:47.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:48.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:48.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:48.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:48.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:48.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.023+0000 7febcd034640 10 monclient: get_monmap_and_config 2024-09-18T12:32:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.023+0000 7febcd034640 10 monclient: build_initial_monmap 2024-09-18T12:32:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.023+0000 7febcd034640 10 monclient: monmap: 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:49.024550+0000 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:49.024550+0000 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:49.027 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.024+0000 7febcd034640 1 Processor -- start 2024-09-18T12:32:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.024+0000 7febcd034640 1 -- start start 2024-09-18T12:32:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.024+0000 7febcd034640 10 monclient: init 2024-09-18T12:32:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T12:32:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febc8096c60 0x7febc809a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): picked mon.noname-c con 0x7febc8096c60 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 0x7febc8097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): picked mon.noname-b con 0x7febc8097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7febc812e5f0 0x7febc812ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): picked mon.noname-a con 0x7febc812e5f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc809d2e0 con 0x7febc812e5f0 2024-09-18T12:32:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc809ac00 con 0x7febc8096c60 2024-09-18T12:32:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc809d6b0 con 0x7febc8097030 2024-09-18T12:32:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febcd034640 10 monclient(hunting): authenticate will time out at 1196.346156s 2024-09-18T12:32:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc6d9d640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7febc8096c60 msgr2=0x7febc809a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 0x7febc8097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7febc812e5f0 0x7febc812ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc6d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7febc8096c60 0x7febc809a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7febc812e5f0 0x7febc812ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48928/0 (socket says 172.21.15.116:48928) 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 0x7febc8097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54516/0 (socket says 172.21.15.116:54516) 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 1 -- 172.21.15.116:0/2287462661 learned_addr learned my addr 172.21.15.116:0/2287462661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 10 monclient(hunting): get_auth_request con 0x7febc812e5f0 auth_method 0 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): get_auth_request con 0x7febc8097030 auth_method 0 2024-09-18T12:32:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:49.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient(hunting): handle_auth_done global_id 65015 payload 274 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 -- 172.21.15.116:0/2287462661 >> v2:172.21.15.144:3300/0 conn(0x7febc8096c60 msgr2=0x7febc809a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 --2- 172.21.15.116:0/2287462661 >> v2:172.21.15.144:3300/0 conn(0x7febc8096c60 0x7febc809a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 -- 172.21.15.116:0/2287462661 >> v2:172.21.15.116:3300/0 conn(0x7febc812e5f0 msgr2=0x7febc812ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 --2- 172.21.15.116:0/2287462661 >> v2:172.21.15.116:3300/0 conn(0x7febc812e5f0 0x7febc812ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 monclient: found mon.noname-b 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febbe59c640 1 -- 172.21.15.116:0/2287462661 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc80992f0 con 0x7febc8097030 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.025+0000 7febc759e640 1 --2- 172.21.15.116:0/2287462661 >> v2:172.21.15.116:3300/0 conn(0x7febc812e5f0 0x7febc812ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febbe59c640 1 -- 172.21.15.116:0/2287462661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 msgr2=0x7febc8097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febbe59c640 1 --2- 172.21.15.116:0/2287462661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 0x7febc8097430 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7febc80637c0 tx=0x7febb00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15257df8cc4d2c52 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 1 -- 172.21.15.116:0/2287462661 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb002ae10 con 0x7febc8097030 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: dump: 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: _finish_auth 0 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febcd034640 5 monclient: authenticate success, global_id 65015 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 1 -- 172.21.15.116:0/2287462661 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb0032440 con 0x7febc8097030 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febcd034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febcd034640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 1 -- 172.21.15.116:0/2287462661 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb00327e0 con 0x7febc8097030 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febcd034640 4 set_mon_vals no callback set 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.028+0000 7febc659c640 10 monclient: dump: 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 10 monclient: shutdown 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 -- 172.21.15.116:0/2287462661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 msgr2=0x7febc8097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 --2- 172.21.15.116:0/2287462661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 0x7febc8097430 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7febc80637c0 tx=0x7febb00090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 -- 172.21.15.116:0/2287462661 shutdown_connections 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 --2- 172.21.15.116:0/2287462661 >> v2:172.21.15.116:3300/0 conn(0x7febc812e5f0 0x7febc812ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 --2- 172.21.15.116:0/2287462661 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8097030 0x7febc8097430 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 --2- 172.21.15.116:0/2287462661 >> v2:172.21.15.144:3300/0 conn(0x7febc8096c60 0x7febc809a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 -- 172.21.15.116:0/2287462661 >> 172.21.15.116:0/2287462661 conn(0x7febc80672b0 msgr2=0x7febc80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 -- 172.21.15.116:0/2287462661 shutdown_connections 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 1 -- 172.21.15.116:0/2287462661 wait complete. 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 10 monclient: build_initial_monmap 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.029+0000 7febcd034640 10 monclient: monmap: 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:49.052 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 Processor -- start 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 -- start start 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient: init 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8096c60 0x7febc812cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): picked mon.noname-c con 0x7febc8096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febc8097030 0x7febc8069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): picked mon.noname-b con 0x7febc8097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febc812e5f0 0x7febc8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): picked mon.noname-a con 0x7febc812e5f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc809ae00 con 0x7febc812e5f0 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc806eaa0 con 0x7febc8097030 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc812d560 con 0x7febc8096c60 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febcd034640 10 monclient(hunting): authenticate will time out at 1196.351730s 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8096c60 0x7febc812cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febc812e5f0 0x7febc8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8096c60 0x7febc812cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54522/0 (socket says 172.21.15.116:54522) 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 10 monclient(hunting): get_auth_request con 0x7febc812e5f0 auth_method 0 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 1 -- 172.21.15.116:0/3261866361 learned_addr learned my addr 172.21.15.116:0/3261866361 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:49.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febbe59c640 1 -- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febc8097030 msgr2=0x7febc8069860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 10 monclient(hunting): get_auth_request con 0x7febc8096c60 auth_method 0 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febbe59c640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febc8097030 0x7febc8069860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:49.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.030+0000 7febc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 10 monclient(hunting): handle_auth_done global_id 104178 payload 824 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 1 -- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8096c60 msgr2=0x7febc812cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8096c60 0x7febc812cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 1 -- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febc8097030 msgr2=0x7febc8069860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febc8097030 0x7febc8069860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 10 monclient: _finish_hunting 0 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 1 monclient: found mon.noname-a 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc759e640 1 -- 172.21.15.116:0/3261866361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc0009ae0 con 0x7febc812e5f0 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.031+0000 7febc6d9d640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8096c60 0x7febc812cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:49.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febc759e640 1 -- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febc812e5f0 msgr2=0x7febc8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febc759e640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febc812e5f0 0x7febc8069da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7febc0009f10 tx=0x7febc000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 1 -- 172.21.15.116:0/3261866361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0018070 con 0x7febc812e5f0 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: dump: 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:49.057 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 1 -- 172.21.15.116:0/3261866361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc000f040 con 0x7febc812e5f0 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febcd034640 5 monclient: authenticate success, global_id 104178 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febcd034640 10 monclient: _renew_subs 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febcd034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb0009390 con 0x7febc812e5f0 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 1 -- 172.21.15.116:0/3261866361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc00092e0 con 0x7febc812e5f0 2024-09-18T12:32:49.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febbf7fe640 10 monclient: dump: 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:49.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febc5d9b640 4 set_mon_vals no callback set 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febcd034640 10 monclient: _renew_subs 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febcd034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.033+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc811bb70 con 0x7febc812e5f0 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.034+0000 7febbf7fe640 1 -- 172.21.15.116:0/3261866361 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7febc0003020 con 0x7febc812e5f0 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.034+0000 7febcd034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.034+0000 7febcd034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.034+0000 7febcd034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.034+0000 7febbf7fe640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feba8038440 0x7feba803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.034+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc804bf60 con 0x7febc812e5f0 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.034+0000 7febbf7fe640 1 -- 172.21.15.116:0/3261866361 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7febc004c890 con 0x7febc812e5f0 2024-09-18T12:32:49.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.035+0000 7febc6d9d640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feba8038440 0x7feba803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.035+0000 7febc6d9d640 10 monclient: get_auth_request con 0x7feba8038440 auth_method 0 2024-09-18T12:32:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.037+0000 7febc6d9d640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feba8038440 0x7feba803a8f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7febb80098d0 tx=0x7febb80091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.039+0000 7febbf7fe640 1 -- 172.21.15.116:0/3261866361 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7febc00112c0 con 0x7febc812e5f0 2024-09-18T12:32:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.039+0000 7febbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:49.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.039+0000 7febbf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:49.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.251+0000 7febcd034640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:49.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.251+0000 7febcd034640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:49.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.251+0000 7febcd034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:49.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.251+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7febc809fa40 con 0x7febc812e5f0 2024-09-18T12:32:49.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.251+0000 7febbf7fe640 1 -- 172.21.15.116:0/3261866361 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7febc0012340 con 0x7febc812e5f0 2024-09-18T12:32:49.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.251+0000 7febbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:49.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.251+0000 7febbf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:49.253 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:49.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feba8038440 msgr2=0x7feba803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feba8038440 0x7feba803a8f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7febb80098d0 tx=0x7febb80091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 10 monclient: shutdown 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febc812e5f0 msgr2=0x7febc8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febc812e5f0 0x7febc8069da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7febc0009f10 tx=0x7febc000b040 comp rx=0 tx=0).stop 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 shutdown_connections 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feba8038440 0x7feba803a8f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febc812e5f0 0x7febc8069da0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7febc8097030 0x7febc8069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 --2- 172.21.15.116:0/3261866361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7febc8096c60 0x7febc812cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 >> 172.21.15.116:0/3261866361 conn(0x7febc80672b0 msgr2=0x7febc806ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 shutdown_connections 2024-09-18T12:32:49.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:49.253+0000 7febcd034640 1 -- 172.21.15.116:0/3261866361 wait complete. 2024-09-18T12:32:49.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:49.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:49.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:49.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:50.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:50.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:50.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:50.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:51.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:51.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:51.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:51.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:52.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:32:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.370+0000 7f7ca6cc0640 10 monclient: get_monmap_and_config 2024-09-18T12:32:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.370+0000 7f7ca6cc0640 10 monclient: build_initial_monmap 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.370+0000 7f7ca6cc0640 10 monclient: monmap: 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:52.371485+0000 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:52.371485+0000 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:52.372 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.371+0000 7f7ca6cc0640 1 Processor -- start 2024-09-18T12:32:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.371+0000 7f7ca6cc0640 1 -- start start 2024-09-18T12:32:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.371+0000 7f7ca6cc0640 10 monclient: init 2024-09-18T12:32:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:32:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f7ca0096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f7ca00971e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 0x7f7ca012ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f7ca012e5f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca0098510 con 0x7f7ca00971e0 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca0099340 con 0x7f7ca012e5f0 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca0099900 con 0x7f7ca0096e10 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca6cc0640 10 monclient(hunting): authenticate will time out at 1199.693246s 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54530/0 (socket says 172.21.15.116:54530) 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca525d640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 msgr2=0x7f7ca012ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 learned_addr learned my addr 172.21.15.116:0/2234204438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request con 0x7f7ca0096e10 auth_method 0 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request con 0x7f7ca00971e0 auth_method 0 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca525d640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 0x7f7ca012ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:32:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.372+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:52.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:52.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request con 0x7f7ca00971e0 auth_method 0 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request con 0x7f7ca0096e10 auth_method 0 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca525d640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 msgr2=0x7f7ca012ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca525d640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 0x7f7ca012ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:52.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:32:52.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:52.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:52.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:52.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:32:52.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.573+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:32:52.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:52.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:52.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:52.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca525d640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 msgr2=0x7f7ca012ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request con 0x7f7ca0096e10 auth_method 0 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request con 0x7f7ca00971e0 auth_method 0 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca525d640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 0x7f7ca012ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:32:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:52.974+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:32:53.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:53.653+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:53.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:53.653+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca525d640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 msgr2=0x7f7ca012ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca525d640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 0x7f7ca012ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request con 0x7f7ca0096e10 auth_method 0 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request con 0x7f7ca00971e0 auth_method 0 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:32:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:53.775+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:32:54.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:54.652+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:54.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:54.652+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:54.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:54.652+0000 7f4f1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:55.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 10 monclient(hunting): tick 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 monclient(hunting): continuing hunt 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 msgr2=0x7f7ca009a7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ca0096e10 0x7f7ca009a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 msgr2=0x7f7ca012ea90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.144:3300/0 conn(0x7f7ca012e5f0 0x7f7ca012ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 -- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 msgr2=0x7f7ca00975b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 --2- 172.21.15.116:0/2234204438 >> v2:172.21.15.116:3300/0 conn(0x7f7ca00971e0 0x7f7ca00975b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 10 monclient: _add_conns ranks=[1,0] 2024-09-18T12:32:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c80003650 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:55.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c80007370 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:55.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 -- 172.21.15.116:0/2234204438 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca009ad00 con 0x7f7c80007370 2024-09-18T12:32:55.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 1 -- 172.21.15.116:0/2234204438 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8000c0a0 con 0x7f7c80003650 2024-09-18T12:32:55.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7c9e7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:55.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.372+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> v1:172.21.15.144:6789/0 conn(0x7f7c80003650 legacy=0x7f7c80005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:55.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.375+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> v1:172.21.15.116:6789/0 conn(0x7f7c80007370 legacy=0x7f7c80009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:55.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.573+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> v1:172.21.15.144:6789/0 conn(0x7f7c80003650 legacy=0x7f7c80005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:55.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:55.653+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:55.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:55.653+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:55.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:55.974+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> v1:172.21.15.144:6789/0 conn(0x7f7c80003650 legacy=0x7f7c80005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.475+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507792098 0 0) 0x7f7ca009ad00 con 0x7f7c80007370 2024-09-18T12:32:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.475+0000 7f7c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.475+0000 7f7c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.475+0000 7f7c9f7fe640 10 monclient(hunting): my global_id is 104181 2024-09-18T12:32:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.475+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c8c0034e0 con 0x7f7c80007370 2024-09-18T12:32:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 305418450 0 0) 0x7f7c8c0034e0 con 0x7f7c80007370 2024-09-18T12:32:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 >> v1:172.21.15.144:6789/0 conn(0x7f7c80003650 legacy=0x7f7c80005aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8000d260 con 0x7f7c80007370 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f7c8800f8a0 con 0x7f7c80007370 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: dump: 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 >> v1:172.21.15.116:6789/0 conn(0x7f7c80007370 legacy=0x7f7c80009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7c9f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.476+0000 7f7ca525d640 1 -- 172.21.15.116:0/2234204438 reap_dead start 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c003a00 0x7f7c8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): picked mon.a con 0x7f7c8c003a00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7c8c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): picked mon.b con 0x7f7c8c004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c005330 0x7f7c8c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): picked mon.c con 0x7f7c8c005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c00c210 con 0x7f7c8c003a00 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c00c380 con 0x7f7c8c004780 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c00c540 con 0x7f7c8c005330 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c003a00 0x7f7c8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c005330 0x7f7c8c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9ffff640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 msgr2=0x7f7c8c004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request con 0x7f7c8c003a00 auth_method 0 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7c8c004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 10 monclient(hunting): get_auth_request con 0x7f7c8c005330 auth_method 0 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient(hunting): handle_auth_done global_id 104181 payload 293 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c005330 msgr2=0x7f7c8c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c005330 0x7f7c8c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 msgr2=0x7f7c8c004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7c8c004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 1 monclient: found mon.a 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c0034e0 con 0x7f7c8c003a00 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 10 monclient: _finish_auth 0 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca4a5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca6cc0640 5 monclient: authenticate success, global_id 104181 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca6cc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.477+0000 7f7ca525d640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c005330 0x7f7c8c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:56.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.480+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c003a00 msgr2=0x7f7c8c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.480+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c003a00 0x7f7c8c003dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006fd0 tx=0x7f7c90008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de6a7c80c8a34e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.481+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9002b940 con 0x7f7c8c003a00 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.481+0000 7f7c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.481+0000 7f7c9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.481+0000 7f7c9f7fe640 10 monclient: dump: 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.491 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.481+0000 7f7ca6cc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9000c040 con 0x7f7c8c003a00 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7c9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7c9f7fe640 1 -- 172.21.15.116:0/2234204438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9002bcf0 con 0x7f7c8c003a00 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7c9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7c9f7fe640 10 monclient: dump: 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7ca6cc0640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.482+0000 7f7ca6cc0640 4 set_mon_vals no callback set 2024-09-18T12:32:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 10 monclient: shutdown 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c003a00 msgr2=0x7f7c8c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c003a00 0x7f7c8c003dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006fd0 tx=0x7f7c90008040 comp rx=0 tx=0).stop 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/2234204438 shutdown_connections 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c005330 0x7f7c8c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7c8c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/2234204438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c003a00 0x7f7c8c003dd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/2234204438 >> 172.21.15.116:0/2234204438 conn(0x7f7ca00672b0 msgr2=0x7f7ca00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/2234204438 shutdown_connections 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/2234204438 wait complete. 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 10 monclient: build_initial_monmap 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.483+0000 7f7ca6cc0640 10 monclient: monmap: 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.484+0000 7f7ca6cc0640 1 Processor -- start 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.484+0000 7f7ca6cc0640 1 -- start start 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.484+0000 7f7ca6cc0640 10 monclient: init 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c003a00 0x7f7ca0118c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f7c8c003a00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7ca0119170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f7c8c004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 0x7f7ca01196b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f7c8c005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca011c580 con 0x7f7c8c005330 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca009b050 con 0x7f7c8c004780 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00a03b0 con 0x7f7c8c003a00 2024-09-18T12:32:56.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:56.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca6cc0640 10 monclient(hunting): authenticate will time out at 1203.806327s 2024-09-18T12:32:56.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c003a00 0x7f7ca0118c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 0x7f7ca01196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c003a00 0x7f7ca0118c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46394/0 (socket says 172.21.15.116:46394) 2024-09-18T12:32:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 0x7f7ca01196b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44860/0 (socket says 172.21.15.116:44860) 2024-09-18T12:32:56.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 1 -- 172.21.15.116:0/1112176310 learned_addr learned my addr 172.21.15.116:0/1112176310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request con 0x7f7c8c003a00 auth_method 0 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7c9ffff640 1 -- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 msgr2=0x7f7ca0119170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): get_auth_request con 0x7f7c8c005330 auth_method 0 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7c9ffff640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7ca0119170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca4a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient(hunting): handle_auth_done global_id 104184 payload 824 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 -- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c003a00 msgr2=0x7f7ca0118c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c003a00 0x7f7ca0118c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 -- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 msgr2=0x7f7ca0119170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7ca0119170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient: _finish_hunting 0 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 monclient: found mon.noname-a 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.485+0000 7f7ca525d640 1 -- 172.21.15.116:0/1112176310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca011b3e0 con 0x7f7c8c005330 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca525d640 1 -- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 msgr2=0x7f7ca01196b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca525d640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 0x7f7ca01196b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ca012e9c0 tx=0x7f7c9800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 1 -- 172.21.15.116:0/1112176310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98013050 con 0x7f7c8c005330 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: dump: 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 1 -- 172.21.15.116:0/1112176310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c98014040 con 0x7f7c8c005330 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca6cc0640 5 monclient: authenticate success, global_id 104184 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca6cc0640 10 monclient: _renew_subs 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca6cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca011c2a0 con 0x7f7c8c005330 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 1 -- 172.21.15.116:0/1112176310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98016a20 con 0x7f7c8c005330 2024-09-18T12:32:56.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9cff9640 10 monclient: dump: 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca6cc0640 10 monclient: _renew_subs 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca6cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7c9effd640 4 set_mon_vals no callback set 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.488+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca0119bf0 con 0x7f7c8c005330 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7c9cff9640 1 -- 172.21.15.116:0/1112176310 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7c980183c0 con 0x7f7c8c005330 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7ca6cc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7ca6cc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7ca6cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca004bf60 con 0x7f7c8c005330 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7c9cff9640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c80038df0 0x7f7c8003b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7c9cff9640 1 -- 172.21.15.116:0/1112176310 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f7c9804fc00 con 0x7f7c8c005330 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c80038df0 0x7f7c8003b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.489+0000 7f7ca4a5c640 10 monclient: get_auth_request con 0x7f7c80038df0 auth_method 0 2024-09-18T12:32:56.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.492+0000 7f7ca4a5c640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c80038df0 0x7f7c8003b2a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006c70 tx=0x7f7c90036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.494+0000 7f7c9cff9640 1 -- 172.21.15.116:0/1112176310 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7c98018670 con 0x7f7c8c005330 2024-09-18T12:32:56.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.494+0000 7f7c9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:56.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.494+0000 7f7c9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:56.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:56.653+0000 7f4f1affd640 10 monclient: tick 2024-09-18T12:32:56.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:56.653+0000 7f4f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:56.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.711+0000 7f7ca6cc0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:32:56.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.711+0000 7f7ca6cc0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:56.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.711+0000 7f7ca6cc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.711+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7ca006cd80 con 0x7f7c8c005330 2024-09-18T12:32:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.711+0000 7f7c9cff9640 1 -- 172.21.15.116:0/1112176310 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f7c98019e60 con 0x7f7c8c005330 2024-09-18T12:32:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.711+0000 7f7c9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:32:56.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.711+0000 7f7c9cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:56.713 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":86,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c80038df0 msgr2=0x7f7c8003b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c80038df0 0x7f7c8003b2a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006c70 tx=0x7f7c90036000 comp rx=0 tx=0).stop 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 10 monclient: shutdown 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 msgr2=0x7f7ca01196b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 0x7f7ca01196b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ca012e9c0 tx=0x7f7c9800e830 comp rx=0 tx=0).stop 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 shutdown_connections 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c80038df0 0x7f7c8003b2a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8c005330 0x7f7ca01196b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c8c004780 0x7f7ca0119170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 --2- 172.21.15.116:0/1112176310 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c8c003a00 0x7f7ca0118c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 >> 172.21.15.116:0/1112176310 conn(0x7f7ca00672b0 msgr2=0x7f7ca00a06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:56.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 shutdown_connections 2024-09-18T12:32:56.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.713+0000 7f7ca6cc0640 1 -- 172.21.15.116:0/1112176310 wait complete. 2024-09-18T12:32:56.725 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:32:56.725 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:32:56.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:32:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.820+0000 7f047d554640 10 monclient: get_monmap_and_config 2024-09-18T12:32:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.820+0000 7f047d554640 10 monclient: build_initial_monmap 2024-09-18T12:32:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.820+0000 7f047d554640 10 monclient: monmap: 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:56.821022+0000 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:56.821022+0000 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:56.822 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.820+0000 7f047d554640 1 Processor -- start 2024-09-18T12:32:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.820+0000 7f047d554640 1 -- start start 2024-09-18T12:32:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.820+0000 7f047d554640 10 monclient: init 2024-09-18T12:32:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f047809a510 0x7f0478096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient(hunting): picked mon.noname-b con 0x7f047809a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04780971e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04780975b0 0x7f047812fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient(hunting): picked mon.noname-a con 0x7f04780975b0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047806ee10 con 0x7f04780971e0 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f047809cb20 con 0x7f04780975b0 2024-09-18T12:32:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047809a8e0 con 0x7f047809a510 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f047d554640 10 monclient(hunting): authenticate will time out at 1204.142482s 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04780975b0 0x7f047812fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f047809a510 0x7f0478096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04767fc640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f04780971e0 legacy=0x7f047812f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04780975b0 0x7f047812fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44872/0 (socket says 172.21.15.116:44872) 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f047809a510 0x7f0478096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46398/0 (socket says 172.21.15.116:46398) 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 -- 172.21.15.116:0/535970916 learned_addr learned my addr 172.21.15.116:0/535970916 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): get_auth_request con 0x7f04780975b0 auth_method 0 2024-09-18T12:32:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 10 monclient(hunting): get_auth_request con 0x7f047809a510 auth_method 0 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient(hunting): handle_auth_done global_id 104187 payload 274 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 -- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f047809a510 msgr2=0x7f0478096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 --2- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f047809a510 0x7f0478096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 -- 172.21.15.116:0/535970916 >> v1:172.21.15.144:6789/0 conn(0x7f04780971e0 legacy=0x7f047812f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 monclient: found mon.noname-a 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f04777fe640 1 -- 172.21.15.116:0/535970916 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047809d590 con 0x7f04780975b0 2024-09-18T12:32:56.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.821+0000 7f0476ffd640 1 --2- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f047809a510 0x7f0478096ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f04777fe640 1 -- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04780975b0 msgr2=0x7f047812fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f04777fe640 1 --2- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04780975b0 0x7f047812fd80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0470009f20 tx=0x7f0470009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc82979751587c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 1 -- 172.21.15.116:0/535970916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470007610 con 0x7f04780975b0 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: dump: 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.834 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: _finish_auth 0 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 1 -- 172.21.15.116:0/535970916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047002d050 con 0x7f04780975b0 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f047d554640 5 monclient: authenticate success, global_id 104187 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f047d554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:56.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f047d554640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 1 -- 172.21.15.116:0/535970916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04700079a0 con 0x7f04780975b0 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f047d554640 4 set_mon_vals no callback set 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.824+0000 7f0475ffb640 10 monclient: dump: 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.836 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 10 monclient: shutdown 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 -- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04780975b0 msgr2=0x7f047812fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 --2- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04780975b0 0x7f047812fd80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0470009f20 tx=0x7f0470009840 comp rx=0 tx=0).stop 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 -- 172.21.15.116:0/535970916 shutdown_connections 2024-09-18T12:32:56.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 --2- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04780975b0 0x7f047812fd80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 --2- 172.21.15.116:0/535970916 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f047809a510 0x7f0478096ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 -- 172.21.15.116:0/535970916 >> 172.21.15.116:0/535970916 conn(0x7f04780672b0 msgr2=0x7f04780676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 -- 172.21.15.116:0/535970916 shutdown_connections 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.825+0000 7f047d554640 1 -- 172.21.15.116:0/535970916 wait complete. 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.826+0000 7f047d554640 10 monclient: build_initial_monmap 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.826+0000 7f047d554640 10 monclient: monmap: 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:56.838 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.826+0000 7f047d554640 1 Processor -- start 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.826+0000 7f047d554640 1 -- start start 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.826+0000 7f047d554640 10 monclient: init 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:56.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f04780971e0 0x7f04781316c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): picked mon.noname-c con 0x7f04780971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04780975b0 0x7f047811b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): picked mon.noname-b con 0x7f04780975b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047809a510 0x7f047811b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): picked mon.noname-a con 0x7f047809a510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047811d2a0 con 0x7f047809a510 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047809c370 con 0x7f04780975b0 2024-09-18T12:32:56.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0478132690 con 0x7f04780971e0 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f047d554640 10 monclient(hunting): authenticate will time out at 1204.148146s 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f04780971e0 0x7f04781316c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f04780971e0 0x7f04781316c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46412/0 (socket says 172.21.15.116:46412) 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047809a510 0x7f047811b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 1 -- 172.21.15.116:0/1700481682 learned_addr learned my addr 172.21.15.116:0/1700481682 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 10 monclient(hunting): get_auth_request con 0x7f04780971e0 auth_method 0 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): get_auth_request con 0x7f047809a510 auth_method 0 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04767fc640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04780975b0 msgr2=0x7f047811b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04767fc640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04780975b0 0x7f047811b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient(hunting): handle_auth_done global_id 104190 payload 824 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f04780971e0 msgr2=0x7f04781316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f04780971e0 0x7f04781316c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04780975b0 msgr2=0x7f047811b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f04780975b0 0x7f047811b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient: _finish_hunting 0 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 1 monclient: found mon.noname-a 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.827+0000 7f04777fe640 1 -- 172.21.15.116:0/1700481682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468009390 con 0x7f047809a510 2024-09-18T12:32:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f04777fe640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047809a510 msgr2=0x7f047811b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:56.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f04777fe640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047809a510 0x7f047811b710 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0470030000 tx=0x7f0470036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470007170 con 0x7f047809a510 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: dump: 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.850 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: _finish_auth 0 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f047d554640 5 monclient: authenticate success, global_id 104190 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047003b410 con 0x7f047809a510 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:56.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047003b810 con 0x7f047809a510 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f047d554640 10 monclient: _renew_subs 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f047d554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0470009ae0 con 0x7f047809a510 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f045effd640 10 monclient: dump: 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.852 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.829+0000 7f04757fa640 4 set_mon_vals no callback set 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f047d554640 10 monclient: _renew_subs 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f047d554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047811cd20 con 0x7f047809a510 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f047d554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0470003020 con 0x7f047809a510 2024-09-18T12:32:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f047d554640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 0x7f047811bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f047d554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047811e980 con 0x7f0478131c00 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 0x7f047811bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 10 monclient: get_auth_request con 0x7f0478131c00 auth_method 0 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.830+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.831+0000 7f045effd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0464038440 0x7f046403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.831+0000 7f0476ffd640 10 monclient(hunting): handle_auth_done global_id 104193 payload 824 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.831+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f047006cfe0 con 0x7f047809a510 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.831+0000 7f0476ffd640 1 -- 172.21.15.116:0/1700481682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f047804bf60 con 0x7f0478131c00 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.831+0000 7f04767fc640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0464038440 0x7f046403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.831+0000 7f04767fc640 10 monclient: get_auth_request con 0x7f0464038440 auth_method 0 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f0476ffd640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 msgr2=0x7f047811bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f0476ffd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 0x7f047811bc50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0468011cb0 tx=0x7f04680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468014650 con 0x7f0478131c00 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 10 monclient: dump: 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f04680147b0 con 0x7f0478131c00 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 msgr2=0x7f047811bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 0x7f047811bc50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0468011cb0 tx=0x7f04680090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 msgr2=0x7f047811bc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f045effd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0478131c00 0x7f047811bc50 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0468011cb0 tx=0x7f04680090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.833+0000 7f04767fc640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0464038440 0x7f046403a8f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f046c0038f0 tx=0x7f046c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f047d554640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f047d554640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 0x7f0478117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f047d554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0478065f20 con 0x7f047811c190 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 0x7f0478117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient: get_auth_request con 0x7f047811c190 auth_method 0 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 10 monclient(hunting): handle_auth_done global_id 104196 payload 824 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.839+0000 7f0476ffd640 1 -- 172.21.15.116:0/1700481682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04780638d0 con 0x7f047811c190 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f0476ffd640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 msgr2=0x7f0478117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f0476ffd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 0x7f0478117480 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f046801b000 tx=0x7f046801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468024040 con 0x7f047811c190 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 10 monclient: dump: 2024-09-18T12:32:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f04680135b0 con 0x7f047811c190 2024-09-18T12:32:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 msgr2=0x7f0478117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 0x7f0478117480 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f046801b000 tx=0x7f046801c000 comp rx=0 tx=0).stop 2024-09-18T12:32:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 msgr2=0x7f0478117480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f045effd640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047811c190 0x7f0478117480 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f046801b000 tx=0x7f046801c000 comp rx=0 tx=0).stop 2024-09-18T12:32:56.860 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:56.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 86, 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:56.861 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:56.862 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:56.862 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:56.868 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.869 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:56.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:56.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:56.873 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:56.874 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:56.877 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0464038440 msgr2=0x7f046403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.842+0000 7f047d554640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0464038440 0x7f046403a8f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f046c0038f0 tx=0x7f046c006d80 comp rx=0 tx=0).stop 2024-09-18T12:32:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 10 monclient: shutdown 2024-09-18T12:32:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047809a510 msgr2=0x7f047811b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f04777fe640 1 -- 172.21.15.116:0/1700481682 reap_dead start 2024-09-18T12:32:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047809a510 0x7f047811b710 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0470030000 tx=0x7f0470036000 comp rx=0 tx=0).stop 2024-09-18T12:32:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 shutdown_connections 2024-09-18T12:32:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 1 --2- 172.21.15.116:0/1700481682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047809a510 0x7f047811b710 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 >> 172.21.15.116:0/1700481682 conn(0x7f04780672b0 msgr2=0x7f04781328c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 shutdown_connections 2024-09-18T12:32:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.843+0000 7f047d554640 1 -- 172.21.15.116:0/1700481682 wait complete. 2024-09-18T12:32:56.880 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:32:56.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:32:56.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:32:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.973+0000 7f539099d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.973+0000 7f539099d640 10 monclient: build_initial_monmap 2024-09-18T12:32:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.973+0000 7f539099d640 10 monclient: monmap: 2024-09-18T12:32:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:32:56.974526+0000 2024-09-18T12:32:56.975 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:32:56.974526+0000 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.974+0000 7f539099d640 1 Processor -- start 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.974+0000 7f539099d640 1 -- start start 2024-09-18T12:32:56.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.974+0000 7f539099d640 10 monclient: init 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 0x7f538c09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient(hunting): picked mon.noname-b con 0x7f538c09c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f538c09aba0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f538c09af70 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c06a320 con 0x7f538c09af70 2024-09-18T12:32:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c068a20 con 0x7f538c09aba0 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c09a9d0 con 0x7f538c09c860 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f539099d640 10 monclient(hunting): authenticate will time out at 1204.296105s 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 0x7f538c09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538ad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f538c09af70 0x7f538c09df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:50030/0 (socket says 172.21.15.116:50030) 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538ad9d640 1 -- 172.21.15.116:0/1696130654 learned_addr learned my addr 172.21.15.116:0/1696130654 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 0x7f538c09cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46418/0 (socket says 172.21.15.116:46418) 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): get_auth_request con 0x7f538c09c860 auth_method 0 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f5389d9b640 1 -- 172.21.15.116:0/1696130654 >> v1:172.21.15.144:6789/0 conn(0x7f538c09aba0 legacy=0x7f538c09d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient(hunting): handle_auth_done global_id 65030 payload 274 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 1 -- 172.21.15.116:0/1696130654 >> v1:172.21.15.144:6789/0 conn(0x7f538c09aba0 legacy=0x7f538c09d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 1 -- 172.21.15.116:0/1696130654 >> v1:172.21.15.116:6789/0 conn(0x7f538c09af70 legacy=0x7f538c09df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 1 monclient: found mon.noname-b 2024-09-18T12:32:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.975+0000 7f538a59c640 1 -- 172.21.15.116:0/1696130654 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c068e60 con 0x7f538c09c860 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538a59c640 1 -- 172.21.15.116:0/1696130654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 msgr2=0x7f538c09cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538a59c640 1 --2- 172.21.15.116:0/1696130654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 0x7f538c09cc30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5384009e90 tx=0x7f5384009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3c22a4eb1c91b5b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 1 -- 172.21.15.116:0/1696130654 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384007840 con 0x7f538c09c860 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: dump: 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: _finish_auth 0 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 1 -- 172.21.15.116:0/1696130654 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538402d050 con 0x7f538c09c860 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 5 monclient: authenticate success, global_id 65030 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 1 -- 172.21.15.116:0/1696130654 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384007c10 con 0x7f538c09c860 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 4 set_mon_vals no callback set 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f538959a640 10 monclient: dump: 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 10 monclient: shutdown 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:56.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 1 -- 172.21.15.116:0/1696130654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 msgr2=0x7f538c09cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 1 --2- 172.21.15.116:0/1696130654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 0x7f538c09cc30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5384009e90 tx=0x7f5384009780 comp rx=0 tx=0).stop 2024-09-18T12:32:56.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 1 -- 172.21.15.116:0/1696130654 shutdown_connections 2024-09-18T12:32:56.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.978+0000 7f539099d640 1 --2- 172.21.15.116:0/1696130654 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09c860 0x7f538c09cc30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 1 -- 172.21.15.116:0/1696130654 >> 172.21.15.116:0/1696130654 conn(0x7f538c0a2e40 msgr2=0x7f538c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:56.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 1 -- 172.21.15.116:0/1696130654 shutdown_connections 2024-09-18T12:32:56.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 1 -- 172.21.15.116:0/1696130654 wait complete. 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 10 monclient: build_initial_monmap 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 10 monclient: monmap: 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:56.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 1 Processor -- start 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 1 -- start start 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.979+0000 7f539099d640 10 monclient: init 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09aba0 0x7f538c06e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): picked mon.noname-c con 0x7f538c09aba0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c09af70 0x7f538c132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): picked mon.noname-a con 0x7f538c09af70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c09c860 0x7f538c11d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): picked mon.noname-b con 0x7f538c09c860 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c11eb30 con 0x7f538c09af70 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c141f20 con 0x7f538c09c860 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c134c00 con 0x7f538c09aba0 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f539099d640 10 monclient(hunting): authenticate will time out at 1204.301250s 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09aba0 0x7f538c06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c09af70 0x7f538c132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c09af70 0x7f538c132b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44900/0 (socket says 172.21.15.116:44900) 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 -- 172.21.15.116:0/3746262038 learned_addr learned my addr 172.21.15.116:0/3746262038 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09aba0 0x7f538c06e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46422/0 (socket says 172.21.15.116:46422) 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): get_auth_request con 0x7f538c09af70 auth_method 0 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 10 monclient(hunting): get_auth_request con 0x7f538c09aba0 auth_method 0 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538ad9d640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c09c860 msgr2=0x7f538c11d8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538ad9d640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c09c860 0x7f538c11d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient(hunting): handle_auth_done global_id 104199 payload 824 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09aba0 msgr2=0x7f538c06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c09aba0 0x7f538c06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c09c860 msgr2=0x7f538c11d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f538c09c860 0x7f538c11d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 monclient: found mon.noname-a 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.980+0000 7f5389d9b640 1 -- 172.21.15.116:0/3746262038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384009a20 con 0x7f538c09af70 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f5389d9b640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c09af70 msgr2=0x7f538c132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f5389d9b640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c09af70 0x7f538c132b40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f537400b630 tx=0x7f53740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374003d90 con 0x7f538c09af70 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 10 monclient: dump: 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374006150 con 0x7f538c09af70 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f539099d640 5 monclient: authenticate success, global_id 104199 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f539099d640 10 monclient: _renew_subs 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f539099d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c06f180 con 0x7f538c09af70 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.982+0000 7f53827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374006510 con 0x7f538c09af70 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f53827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f53827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f53827fc640 10 monclient: dump: 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.003 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f5388d99640 4 set_mon_vals no callback set 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f539099d640 10 monclient: _renew_subs 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f539099d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538c11fce0 con 0x7f538c09af70 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f539099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f539099d640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 0x7f538c11ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f539099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c065e00 con 0x7f538c133080 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f53740103e0 con 0x7f538c09af70 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f538a59c640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 0x7f538c11ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f538a59c640 10 monclient: get_auth_request con 0x7f538c133080 auth_method 0 2024-09-18T12:32:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f538a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f538a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.983+0000 7f538a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f538a59c640 10 monclient(hunting): handle_auth_done global_id 65036 payload 824 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f538a59c640 1 -- 172.21.15.116:0/3746262038 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f538c136660 con 0x7f538c133080 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f53827fc640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f536c038440 0x7f536c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(198..198 src has 13..198) v4 ==== 8913+0+0 (secure 0 0 0) 0x7f537404c080 con 0x7f538c09af70 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f538ad9d640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f536c038440 0x7f536c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.984+0000 7f538ad9d640 10 monclient: get_auth_request con 0x7f536c038440 auth_method 0 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f538a59c640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 msgr2=0x7f538c11ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f538a59c640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 0x7f538c11ddf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f538c068db0 tx=0x7f5384037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384041070 con 0x7f538c133080 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 10 monclient: dump: 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5384031cf0 con 0x7f538c133080 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 msgr2=0x7f538c11ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 0x7f538c11ddf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f538c068db0 tx=0x7f5384037000 comp rx=0 tx=0).stop 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 msgr2=0x7f538c11ddf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.986+0000 7f53827fc640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c133080 0x7f538c11ddf0 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f538c068db0 tx=0x7f5384037000 comp rx=0 tx=0).stop 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.987+0000 7f538ad9d640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f536c038440 0x7f536c03a8f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f538c06f490 tx=0x7f537c007b60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f539099d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f539099d640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 0x7f538c06b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f539099d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c133450 con 0x7f538c11e330 2024-09-18T12:32:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 0x7f538c06b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient: get_auth_request con 0x7f538c11e330 auth_method 0 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 10 monclient(hunting): handle_auth_done global_id 65039 payload 824 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.992+0000 7f538a59c640 1 -- 172.21.15.116:0/3746262038 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f538c11e730 con 0x7f538c11e330 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f538a59c640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 msgr2=0x7f538c06b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:57.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f538a59c640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 0x7f538c06b8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5384009de0 tx=0x7f5384041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538404b070 con 0x7f538c11e330 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 10 monclient: dump: 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:32:57.012 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f538403b410 con 0x7f538c11e330 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 msgr2=0x7f538c06b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 0x7f538c06b8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5384009de0 tx=0x7f5384041000 comp rx=0 tx=0).stop 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 msgr2=0x7f538c06b8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:32:57.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f53827fc640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f538c11e330 0x7f538c06b8b0 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5384009de0 tx=0x7f5384041000 comp rx=0 tx=0).stop 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 86, 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:32:57.014 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:57.015 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:57.016 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:32:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:32:57.018 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.019 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:57.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:57.023 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:32:57.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.026 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:32:57.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:32:57.028 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f536c038440 msgr2=0x7f536c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f536c038440 0x7f536c03a8f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f538c06f490 tx=0x7f537c007b60 comp rx=0 tx=0).stop 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 10 monclient: shutdown 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c09af70 msgr2=0x7f538c132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 --2- 172.21.15.116:0/3746262038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f538c09af70 0x7f538c132b40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f537400b630 tx=0x7f53740091a0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f538ad9d640 1 -- 172.21.15.116:0/3746262038 reap_dead start 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f538ad9d640 1 -- 172.21.15.116:0/3746262038 reap_dead start 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 shutdown_connections 2024-09-18T12:32:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 >> 172.21.15.116:0/3746262038 conn(0x7f538c0a2e40 msgr2=0x7f538c09b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 shutdown_connections 2024-09-18T12:32:57.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:32:56.995+0000 7f539099d640 1 -- 172.21.15.116:0/3746262038 wait complete. 2024-09-18T12:32:57.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:32:57.031 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:32:57.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.479+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v199) v1 ==== 166+0+0 (secure 0 0 0) 0x7f4f2c016050 con 0x7f4f3409ff50 2024-09-18T12:32:57.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.479+0000 7f4f1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:57.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.479+0000 7f4f1bfff640 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-18T12:32:57.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.581+0000 7f4f3aa0f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:57.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.581+0000 7f4f3aa0f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:57.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.581+0000 7f4f3aa0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:57.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.581+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f4f34138d60 con 0x7f4f3409ff50 2024-09-18T12:32:57.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.582+0000 7f4f1bfff640 1 -- 172.21.15.144:0/4174960782 <== mon.2 v2:172.21.15.116:3301/0 5 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v199) v1 ==== 145+0+0 (secure 0 0 0) 0x7f4f34138d60 con 0x7f4f3409ff50 2024-09-18T12:32:57.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.582+0000 7f4f1bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:57.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.582+0000 7f4f1bfff640 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-18T12:32:57.583 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h6' already exists 2024-09-18T12:32:57.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f20038490 msgr2=0x7f4f2003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f20038490 0x7f4f2003a940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4f3409feb0 tx=0x7f4f24032000 comp rx=0 tx=0).stop 2024-09-18T12:32:57.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 10 monclient: shutdown 2024-09-18T12:32:57.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 msgr2=0x7f4f20048710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f20048710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c04a480 tx=0x7f4f2c01d000 comp rx=0 tx=0).stop 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 shutdown_connections 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34131ac0 0x7f4f2004b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f3409ff50 0x7f4f20048710 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f20043a10 0x7f4f20045e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 --2- 172.21.15.144:0/4174960782 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f20038490 0x7f4f2003a940 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 >> 172.21.15.144:0/4174960782 conn(0x7f4f340a5190 msgr2=0x7f4f34096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 shutdown_connections 2024-09-18T12:32:57.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.584+0000 7f4f3aa0f640 1 -- 172.21.15.144:0/4174960782 wait complete. 2024-09-18T12:32:57.595 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:57.595 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-18T12:32:57.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.669+0000 7f15f446f640 10 monclient: get_monmap_and_config 2024-09-18T12:32:57.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.669+0000 7f15f446f640 10 monclient: build_initial_monmap 2024-09-18T12:32:57.671 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.669+0000 7f15f446f640 10 monclient: monmap: 2024-09-18T12:32:57.671 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:57.671 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.671 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:57.670809+0000 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:57.670809+0000 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.670+0000 7f15f446f640 1 Processor -- start 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.670+0000 7f15f446f640 1 -- start start 2024-09-18T12:32:57.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.670+0000 7f15f446f640 10 monclient: init 2024-09-18T12:32:57.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:57.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:32:57.679 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15ec096a30 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 0x7f15ec099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient(hunting): picked mon.noname-b con 0x7f15ec09d530 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f15ec097510 0x7f15ec135c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient(hunting): picked mon.noname-c con 0x7f15ec097510 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec097e80 con 0x7f15ec096a30 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ec098be0 con 0x7f15ec097510 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec098ec0 con 0x7f15ec09d530 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f446f640 10 monclient(hunting): authenticate will time out at 1213.240141s 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f2a0c640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f15ec097510 msgr2=0x7f15ec135c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:57.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f220b640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f15ec096a30 legacy=0x7f15ec096e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f2a0c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f15ec097510 0x7f15ec135c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 0x7f15ec099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 0x7f15ec099d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55430/0 (socket says 172.21.15.144:55430) 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 -- 172.21.15.144:0/1681297342 learned_addr learned my addr 172.21.15.144:0/1681297342 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): get_auth_request con 0x7f15ec09d530 auth_method 0 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient(hunting): handle_auth_done global_id 65042 payload 274 2024-09-18T12:32:57.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 -- 172.21.15.144:0/1681297342 >> v2:172.21.15.144:3300/0 conn(0x7f15ec097510 msgr2=0x7f15ec135c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 --2- 172.21.15.144:0/1681297342 >> v2:172.21.15.144:3300/0 conn(0x7f15ec097510 0x7f15ec135c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 -- 172.21.15.144:0/1681297342 >> v1:172.21.15.144:6789/0 conn(0x7f15ec096a30 legacy=0x7f15ec096e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 monclient: found mon.noname-b 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.671+0000 7f15f1a0a640 1 -- 172.21.15.144:0/1681297342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec097990 con 0x7f15ec09d530 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1a0a640 1 -- 172.21.15.144:0/1681297342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 msgr2=0x7f15ec099d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1a0a640 1 --2- 172.21.15.144:0/1681297342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 0x7f15ec099d70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009910 tx=0x7f15dc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55e1319e0f7d905c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1209640 1 -- 172.21.15.144:0/1681297342 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc032440 con 0x7f15ec09d530 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1209640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1209640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1209640 10 monclient: dump: 2024-09-18T12:32:57.682 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1209640 10 monclient: _finish_auth 0 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1209640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.674+0000 7f15f1209640 1 -- 172.21.15.144:0/1681297342 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc0325a0 con 0x7f15ec09d530 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f1209640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 5 monclient: authenticate success, global_id 65042 2024-09-18T12:32:57.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f1209640 1 -- 172.21.15.144:0/1681297342 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc032940 con 0x7f15ec09d530 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 4 set_mon_vals no callback set 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f1209640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f1209640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f1209640 10 monclient: dump: 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.684 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 10 monclient: shutdown 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 -- 172.21.15.144:0/1681297342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 msgr2=0x7f15ec099d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 --2- 172.21.15.144:0/1681297342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 0x7f15ec099d70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009910 tx=0x7f15dc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 -- 172.21.15.144:0/1681297342 shutdown_connections 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 --2- 172.21.15.144:0/1681297342 >> v2:172.21.15.144:3300/0 conn(0x7f15ec097510 0x7f15ec135c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 --2- 172.21.15.144:0/1681297342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec09d530 0x7f15ec099d70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 -- 172.21.15.144:0/1681297342 >> 172.21.15.144:0/1681297342 conn(0x7f15ec0a0fe0 msgr2=0x7f15ec0a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 -- 172.21.15.144:0/1681297342 shutdown_connections 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.675+0000 7f15f446f640 1 -- 172.21.15.144:0/1681297342 wait complete. 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.676+0000 7f15f446f640 10 monclient: build_initial_monmap 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.676+0000 7f15f446f640 10 monclient: monmap: 2024-09-18T12:32:57.685 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.676+0000 7f15f446f640 1 Processor -- start 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.676+0000 7f15f446f640 1 -- start start 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.676+0000 7f15f446f640 10 monclient: init 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:32:57.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 0x7f15ec141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): picked mon.noname-c con 0x7f15ec096a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ec097510 0x7f15ec130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): picked mon.noname-b con 0x7f15ec097510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ec09d530 0x7f15ec1317e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): picked mon.noname-a con 0x7f15ec09d530 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec0984c0 con 0x7f15ec09d530 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec11e230 con 0x7f15ec097510 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec11df50 con 0x7f15ec096a30 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f446f640 10 monclient(hunting): authenticate will time out at 1213.246138s 2024-09-18T12:32:57.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f1a0a640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ec097510 msgr2=0x7f15ec130b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f1a0a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ec097510 0x7f15ec130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 0x7f15ec141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 0x7f15ec141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55438/0 (socket says 172.21.15.144:55438) 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 -- 172.21.15.144:0/2162763155 learned_addr learned my addr 172.21.15.144:0/2162763155 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): get_auth_request con 0x7f15ec096a30 auth_method 0 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ec09d530 0x7f15ec1317e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 10 monclient(hunting): get_auth_request con 0x7f15ec09d530 auth_method 0 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:57.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f2a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient(hunting): handle_auth_done global_id 65045 payload 824 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 -- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ec097510 msgr2=0x7f15ec130b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ec097510 0x7f15ec130b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 -- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ec09d530 msgr2=0x7f15ec1317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ec09d530 0x7f15ec1317e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient: _finish_hunting 0 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 monclient: found mon.noname-c 2024-09-18T12:32:57.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:57.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:57.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.677+0000 7f15f220b640 1 -- 172.21.15.144:0/2162763155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc009450 con 0x7f15ec096a30 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.679+0000 7f15f220b640 1 -- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 msgr2=0x7f15ec141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.679+0000 7f15f220b640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 0x7f15ec141600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f15e800b5f0 tx=0x7f15e8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.679+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8015070 con 0x7f15ec096a30 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.679+0000 7f15f220b640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 0x7f15ec141600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f15e800b5f0 tx=0x7f15e8009120 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f15e8003d90 config(0 keys) v1 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.679+0000 7f15e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.679+0000 7f15e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.679+0000 7f15e27fc640 10 monclient: dump: 2024-09-18T12:32:57.692 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 10 monclient: _finish_auth 0 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e8003d90 con 0x7f15ec096a30 2024-09-18T12:32:57.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 5 monclient: authenticate success, global_id 65045 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 10 monclient: _renew_subs 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec141eb0 con 0x7f15ec096a30 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f0a08640 4 set_mon_vals no callback set 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 10 monclient: _renew_subs 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec132cc0 con 0x7f15ec096a30 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e80063a0 con 0x7f15ec096a30 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 10 monclient: dump: 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:57.694 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f15e8010440 con 0x7f15ec096a30 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:57.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.680+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ec04bf60 con 0x7f15ec096a30 2024-09-18T12:32:57.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.681+0000 7f15e27fc640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15d0038440 0x7f15d003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:57.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.681+0000 7f15f1a0a640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15d0038440 0x7f15d003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:57.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.681+0000 7f15f1a0a640 10 monclient: get_auth_request con 0x7f15d0038440 auth_method 0 2024-09-18T12:32:57.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.683+0000 7f15f1a0a640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15d0038440 0x7f15d003a8f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f15dc031000 tx=0x7f15dc032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:57.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:57.685+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(199..199 src has 13..199) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f15e804c4e0 con 0x7f15ec096a30 2024-09-18T12:32:58.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.302+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f15e8018d60 con 0x7f15ec096a30 2024-09-18T12:32:58.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.302+0000 7f15e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:58.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.302+0000 7f15e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:58.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.465+0000 7f15f446f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.465+0000 7f15f446f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.465+0000 7f15f446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:58.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.465+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f15ec09cd10 con 0x7f15ec096a30 2024-09-18T12:32:58.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.487+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v200) v1 ==== 166+0+0 (secure 0 0 0) 0x7f15e8018780 con 0x7f15ec096a30 2024-09-18T12:32:58.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.487+0000 7f15e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.487+0000 7f15e27fc640 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-18T12:32:58.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.583+0000 7f15f446f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.583+0000 7f15f446f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.583+0000 7f15f446f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:58.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.583+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f15ec09e840 con 0x7f15ec096a30 2024-09-18T12:32:58.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.584+0000 7f15e27fc640 1 -- 172.21.15.144:0/2162763155 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v200) v1 ==== 145+0+0 (secure 0 0 0) 0x7f15e8018d60 con 0x7f15ec096a30 2024-09-18T12:32:58.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.584+0000 7f15e27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.584+0000 7f15e27fc640 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-18T12:32:58.586 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h7' already exists 2024-09-18T12:32:58.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15d0038440 msgr2=0x7f15d003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:58.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15d0038440 0x7f15d003a8f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f15dc031000 tx=0x7f15dc032000 comp rx=0 tx=0).stop 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 10 monclient: shutdown 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 msgr2=0x7f15ec141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 0x7f15ec141600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f15e800b5f0 tx=0x7f15e8009120 comp rx=0 tx=0).stop 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 shutdown_connections 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f15d0038440 0x7f15d003a8f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15ec09d530 0x7f15ec1317e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15ec097510 0x7f15ec130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 --2- 172.21.15.144:0/2162763155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f15ec096a30 0x7f15ec141600 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 >> 172.21.15.144:0/2162763155 conn(0x7f15ec0a0fe0 msgr2=0x7f15ec0a0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 shutdown_connections 2024-09-18T12:32:58.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.586+0000 7f15f446f640 1 -- 172.21.15.144:0/2162763155 wait complete. 2024-09-18T12:32:58.596 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:58.597 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.670+0000 7faafea60640 10 monclient: get_monmap_and_config 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.670+0000 7faafea60640 10 monclient: build_initial_monmap 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.670+0000 7faafea60640 10 monclient: monmap: 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:58.671856+0000 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:58.671856+0000 2024-09-18T12:32:58.672 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:58.673 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:58.673 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:58.673 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:58.673 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:58.673 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:58.673 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:58.673 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:58.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.671+0000 7faafea60640 1 Processor -- start 2024-09-18T12:32:58.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.671+0000 7faafea60640 1 -- start start 2024-09-18T12:32:58.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.671+0000 7faafea60640 10 monclient: init 2024-09-18T12:32:58.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:58.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:32:58.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faaf8131ac0 0x7faaf8131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:58.675 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient(hunting): picked mon.noname-c con 0x7faaf8131ac0 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaf809ff50 addr v1:172.21.15.116:6789/0 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faaf81323d0 0x7faaf8098860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient(hunting): picked mon.noname-a con 0x7faaf81323d0 addr v2:172.21.15.116:3300/0 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf80a1610 con 0x7faaf809ff50 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf81327a0 con 0x7faaf81323d0 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf80a1330 con 0x7faaf8131ac0 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafea60640 10 monclient(hunting): authenticate will time out at 1214.241250s 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faaf7fff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7faaf8131ac0 msgr2=0x7faaf8131e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faaf7fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faaf8131ac0 0x7faaf8131e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:58.676 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faaf81323d0 0x7faaf8098860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faaf77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faaf809ff50 0x7faaf8098150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:50424/0 (socket says 172.21.15.144:50424) 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faaf77fe640 1 -- 172.21.15.144:0/3936255524 learned_addr learned my addr 172.21.15.144:0/3936255524 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): get_auth_request con 0x7faaf81323d0 auth_method 0 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient(hunting): handle_auth_done global_id 104205 payload 274 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 1 -- 172.21.15.144:0/3936255524 >> v2:172.21.15.144:3300/0 conn(0x7faaf8131ac0 msgr2=0x7faaf8131e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 1 --2- 172.21.15.144:0/3936255524 >> v2:172.21.15.144:3300/0 conn(0x7faaf8131ac0 0x7faaf8131e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 1 -- 172.21.15.144:0/3936255524 >> v1:172.21.15.116:6789/0 conn(0x7faaf809ff50 legacy=0x7faaf8098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:32:58.677 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient: _finish_hunting 0 2024-09-18T12:32:58.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 1 monclient: found mon.noname-a 2024-09-18T12:32:58.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:58.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:58.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.672+0000 7faafcffd640 1 -- 172.21.15.144:0/3936255524 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf8139e10 con 0x7faaf81323d0 2024-09-18T12:32:58.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.675+0000 7faafcffd640 1 -- 172.21.15.144:0/3936255524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf81323d0 msgr2=0x7faaf8098860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:58.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.675+0000 7faafcffd640 1 --2- 172.21.15.144:0/3936255524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf81323d0 0x7faaf8098860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faaf80607f0 tx=0x7faaf0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e760faaf838843 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:58.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 1 -- 172.21.15.144:0/3936255524 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0007840 con 0x7faaf81323d0 2024-09-18T12:32:58.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: dump: 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:32:58.683 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 1 -- 172.21.15.144:0/3936255524 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf002d050 con 0x7faaf81323d0 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 1 -- 172.21.15.144:0/3936255524 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0007c10 con 0x7faaf81323d0 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faaf6ffd640 10 monclient: dump: 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:58.684 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 5 monclient: authenticate success, global_id 104205 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 4 set_mon_vals no callback set 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 10 monclient: shutdown 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 -- 172.21.15.144:0/3936255524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf81323d0 msgr2=0x7faaf8098860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 --2- 172.21.15.144:0/3936255524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf81323d0 0x7faaf8098860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faaf80607f0 tx=0x7faaf0009780 comp rx=0 tx=0).stop 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 -- 172.21.15.144:0/3936255524 shutdown_connections 2024-09-18T12:32:58.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 --2- 172.21.15.144:0/3936255524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf81323d0 0x7faaf8098860 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 --2- 172.21.15.144:0/3936255524 >> v2:172.21.15.144:3300/0 conn(0x7faaf8131ac0 0x7faaf8131e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 -- 172.21.15.144:0/3936255524 >> 172.21.15.144:0/3936255524 conn(0x7faaf80a5190 msgr2=0x7faaf80a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 -- 172.21.15.144:0/3936255524 shutdown_connections 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.676+0000 7faafea60640 1 -- 172.21.15.144:0/3936255524 wait complete. 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.677+0000 7faafea60640 10 monclient: build_initial_monmap 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.677+0000 7faafea60640 10 monclient: monmap: 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:58.686 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.677+0000 7faafea60640 1 Processor -- start 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.677+0000 7faafea60640 1 -- start start 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.677+0000 7faafea60640 10 monclient: init 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 0x7faaf8138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): picked mon.noname-a con 0x7faaf809ff50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf8131ac0 0x7faaf8133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): picked mon.noname-b con 0x7faaf8131ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:58.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf81323d0 0x7faaf8134110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): picked mon.noname-c con 0x7faaf81323d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8098e70 con 0x7faaf809ff50 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8097f20 con 0x7faaf8131ac0 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8137480 con 0x7faaf81323d0 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafea60640 10 monclient(hunting): authenticate will time out at 1214.247350s 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf77fe640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf8131ac0 msgr2=0x7faaf8133bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:58.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf77fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf8131ac0 0x7faaf8133bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf81323d0 0x7faaf8134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 0x7faaf8138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf81323d0 0x7faaf8134110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55450/0 (socket says 172.21.15.144:55450) 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 0x7faaf8138190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57498/0 (socket says 172.21.15.144:57498) 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 1 -- 172.21.15.144:0/926806835 learned_addr learned my addr 172.21.15.144:0/926806835 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 10 monclient(hunting): get_auth_request con 0x7faaf81323d0 auth_method 0 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): get_auth_request con 0x7faaf809ff50 auth_method 0 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:58.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faafcffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient(hunting): handle_auth_done global_id 104208 payload 824 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 -- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf81323d0 msgr2=0x7faaf8134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf81323d0 0x7faaf8134110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 -- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf8131ac0 msgr2=0x7faaf8133bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:58.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf8131ac0 0x7faaf8133bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 monclient: found mon.noname-a 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.678+0000 7faaf7fff640 1 -- 172.21.15.144:0/926806835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf0009a20 con 0x7faaf809ff50 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faaf7fff640 1 -- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 msgr2=0x7faaf8138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faaf7fff640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 0x7faaf8138190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faae800b530 tx=0x7faae80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8003d90 con 0x7faaf809ff50 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: dump: 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:58.691 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: _finish_auth 0 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae8006150 con 0x7faaf809ff50 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faafea60640 5 monclient: authenticate success, global_id 104208 2024-09-18T12:32:58.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faafea60640 10 monclient: _renew_subs 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faafea60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf81356d0 con 0x7faaf809ff50 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8006510 con 0x7faaf809ff50 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faadffff640 10 monclient: dump: 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:58.693 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:58.694 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:58.694 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:58.694 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:58.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faafea60640 10 monclient: _renew_subs 2024-09-18T12:32:58.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faafea60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:58.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf81386d0 con 0x7faaf809ff50 2024-09-18T12:32:58.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.681+0000 7faaf67fc640 4 set_mon_vals no callback set 2024-09-18T12:32:58.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7faae801b3b0 con 0x7faaf809ff50 2024-09-18T12:32:58.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faadffff640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faae4038440 0x7faae403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:58.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faafea60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:58.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faafea60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:58.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faafea60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:58.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(200..200 src has 13..200) v4 ==== 9333+0+0 (secure 0 0 0) 0x7faae804be90 con 0x7faaf809ff50 2024-09-18T12:32:58.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf804bf60 con 0x7faaf809ff50 2024-09-18T12:32:58.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faaf77fe640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faae4038440 0x7faae403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:58.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.682+0000 7faaf77fe640 10 monclient: get_auth_request con 0x7faae4038440 auth_method 0 2024-09-18T12:32:58.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.684+0000 7faaf77fe640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faae4038440 0x7faae403a8f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faaec0047d0 tx=0x7faaec007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:58.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.686+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7faae8017070 con 0x7faaf809ff50 2024-09-18T12:32:58.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.686+0000 7faadffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:58.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.686+0000 7faadffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:58.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.844+0000 7faafea60640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.844+0000 7faafea60640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:58.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.844+0000 7faafea60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:58.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:58.844+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7faaf809c100 con 0x7faaf809ff50 2024-09-18T12:32:59.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.498+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v201) v1 ==== 166+0+0 (secure 0 0 0) 0x7faae801a360 con 0x7faaf809ff50 2024-09-18T12:32:59.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.498+0000 7faadffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:59.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.498+0000 7faadffff640 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-18T12:32:59.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.598+0000 7faafea60640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:59.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.598+0000 7faafea60640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:59.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.598+0000 7faafea60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:32:59.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.598+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7faaf8138ec0 con 0x7faaf809ff50 2024-09-18T12:32:59.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.599+0000 7faadffff640 1 -- 172.21.15.144:0/926806835 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v201) v1 ==== 145+0+0 (secure 0 0 0) 0x7faae800d290 con 0x7faaf809ff50 2024-09-18T12:32:59.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.599+0000 7faadffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:59.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.599+0000 7faadffff640 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-18T12:32:59.600 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h8' already exists 2024-09-18T12:32:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faae4038440 msgr2=0x7faae403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faae4038440 0x7faae403a8f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faaec0047d0 tx=0x7faaec007b80 comp rx=0 tx=0).stop 2024-09-18T12:32:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 10 monclient: shutdown 2024-09-18T12:32:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:59.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 msgr2=0x7faaf8138190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:59.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 0x7faaf8138190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faae800b530 tx=0x7faae80090f0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 shutdown_connections 2024-09-18T12:32:59.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faae4038440 0x7faae403a8f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf81323d0 0x7faaf8134110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf8131ac0 0x7faaf8133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 --2- 172.21.15.144:0/926806835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf809ff50 0x7faaf8138190 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 >> 172.21.15.144:0/926806835 conn(0x7faaf80a5190 msgr2=0x7faaf80a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 shutdown_connections 2024-09-18T12:32:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.601+0000 7faafea60640 1 -- 172.21.15.144:0/926806835 wait complete. 2024-09-18T12:32:59.612 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:32:59.612 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-18T12:32:59.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.687+0000 7f80d2f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:32:59.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.687+0000 7f80d2f9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.687+0000 7f80d2f9d640 10 monclient: monmap: 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:32:59.688100+0000 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:32:59.688100+0000 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:32:59.689 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:59.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.687+0000 7f80d2f9d640 1 Processor -- start 2024-09-18T12:32:59.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.687+0000 7f80d2f9d640 1 -- start start 2024-09-18T12:32:59.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.687+0000 7f80d2f9d640 10 monclient: init 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80cc131bd0 addr v1:172.21.15.144:6789/0 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f80cc069e10 0x7f80cc131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f80cc069e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f80cc136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80cc0a06d0 con 0x7f80cc131bd0 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80cc071780 con 0x7f80cc069e10 2024-09-18T12:32:59.692 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc06a210 con 0x7f80cc136d60 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d2f9d640 10 monclient(hunting): authenticate will time out at 1215.257458s 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d0d39640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f80cc131bd0 legacy=0x7f80cc09ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80cbfff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f80cc069e10 msgr2=0x7f80cc131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80cbfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f80cc069e10 0x7f80cc131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55464/0 (socket says 172.21.15.144:55464) 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 1 -- 172.21.15.144:0/3463036138 learned_addr learned my addr 172.21.15.144:0/3463036138 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 10 monclient(hunting): get_auth_request con 0x7f80cc136d60 auth_method 0 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:59.693 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.688+0000 7f80d153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 10 monclient(hunting): handle_auth_done global_id 65051 payload 274 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 1 -- 172.21.15.144:0/3463036138 >> v2:172.21.15.144:3300/0 conn(0x7f80cc069e10 msgr2=0x7f80cc131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 1 --2- 172.21.15.144:0/3463036138 >> v2:172.21.15.144:3300/0 conn(0x7f80cc069e10 0x7f80cc131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 1 -- 172.21.15.144:0/3463036138 >> v1:172.21.15.144:6789/0 conn(0x7f80cc131bd0 legacy=0x7f80cc09ebe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 1 monclient: found mon.noname-b 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:32:59.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.689+0000 7f80d153a640 1 -- 172.21.15.144:0/3463036138 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc0a0010 con 0x7f80cc136d60 2024-09-18T12:32:59.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d153a640 1 -- 172.21.15.144:0/3463036138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 msgr2=0x7f80cc139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:59.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d153a640 1 --2- 172.21.15.144:0/3463036138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc139140 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009f20 tx=0x7f80c4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a17a910e65aacadf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:59.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 1 -- 172.21.15.144:0/3463036138 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4007610 con 0x7f80cc136d60 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: dump: 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:59.701 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: _finish_auth 0 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 1 -- 172.21.15.144:0/3463036138 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c402d050 con 0x7f80cc136d60 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 1 -- 172.21.15.144:0/3463036138 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c40079a0 con 0x7f80cc136d60 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80cb7fe640 10 monclient: dump: 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:59.702 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:59.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 5 monclient: authenticate success, global_id 65051 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 4 set_mon_vals no callback set 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 10 monclient: shutdown 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 1 -- 172.21.15.144:0/3463036138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 msgr2=0x7f80cc139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/3463036138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc139140 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009f20 tx=0x7f80c4009840 comp rx=0 tx=0).stop 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 1 -- 172.21.15.144:0/3463036138 shutdown_connections 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/3463036138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc139140 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/3463036138 >> v2:172.21.15.144:3300/0 conn(0x7f80cc069e10 0x7f80cc131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.692+0000 7f80d2f9d640 1 -- 172.21.15.144:0/3463036138 >> 172.21.15.144:0/3463036138 conn(0x7f80cc0713b0 msgr2=0x7f80cc0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.693+0000 7f80d2f9d640 1 -- 172.21.15.144:0/3463036138 shutdown_connections 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.693+0000 7f80d2f9d640 1 -- 172.21.15.144:0/3463036138 wait complete. 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.693+0000 7f80d2f9d640 10 monclient: build_initial_monmap 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.693+0000 7f80d2f9d640 10 monclient: monmap: 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:32:59.705 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.693+0000 7f80d2f9d640 1 Processor -- start 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.693+0000 7f80d2f9d640 1 -- start start 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.693+0000 7f80d2f9d640 10 monclient: init 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80cc069e10 0x7f80cc130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f80cc069e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80cc131bd0 0x7f80cc131050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f80cc131bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:32:59.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f80cc136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc1420e0 con 0x7f80cc131bd0 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc09ea00 con 0x7f80cc069e10 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc11d1e0 con 0x7f80cc136d60 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d2f9d640 10 monclient(hunting): authenticate will time out at 1215.263357s 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d0d39640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80cc069e10 msgr2=0x7f80cc130b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:32:59.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d0d39640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80cc069e10 0x7f80cc130b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:32:59.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:59.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80cc131bd0 0x7f80cc131050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:59.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc099890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55472/0 (socket says 172.21.15.144:55472) 2024-09-18T12:32:59.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80cc131bd0 0x7f80cc131050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57504/0 (socket says 172.21.15.144:57504) 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 1 -- 172.21.15.144:0/2680186392 learned_addr learned my addr 172.21.15.144:0/2680186392 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): get_auth_request con 0x7f80cc136d60 auth_method 0 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 10 monclient(hunting): get_auth_request con 0x7f80cc131bd0 auth_method 0 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:32:59.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 10 monclient(hunting): handle_auth_done global_id 65054 payload 824 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 1 -- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80cc069e10 msgr2=0x7f80cc130b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.694+0000 7f80d153a640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80cc069e10 0x7f80cc130b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80d153a640 1 -- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80cc131bd0 msgr2=0x7f80cc131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80d153a640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80cc131bd0 0x7f80cc131050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:32:59.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80d153a640 10 monclient: _finish_hunting 0 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80d153a640 1 monclient: found mon.noname-c 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80d153a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80d153a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80d153a640 1 -- 172.21.15.144:0/2680186392 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c4009ae0 con 0x7f80cc136d60 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.695+0000 7f80cbfff640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80cc131bd0 0x7f80cc131050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d153a640 1 -- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 msgr2=0x7f80cc099890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d153a640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc099890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009f20 tx=0x7f80c4036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4007e10 con 0x7f80cc136d60 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: dump: 2024-09-18T12:32:59.711 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: _finish_auth 0 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d2f9d640 5 monclient: authenticate success, global_id 65054 2024-09-18T12:32:59.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d2f9d640 10 monclient: _renew_subs 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d2f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc11cf00 con 0x7f80cc136d60 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d2f9d640 10 monclient: _renew_subs 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d2f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc137600 con 0x7f80cc136d60 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c4031cd0 con 0x7f80cc136d60 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4031e40 con 0x7f80cc136d60 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80c8ff9640 10 monclient: dump: 2024-09-18T12:32:59.713 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.697+0000 7f80caffd640 4 set_mon_vals no callback set 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80d2f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80d2f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:59.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80d2f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:59.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f80c4038070 con 0x7f80cc136d60 2024-09-18T12:32:59.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc136920 con 0x7f80cc136d60 2024-09-18T12:32:59.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80c8ff9640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80b8038440 0x7f80b803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:32:59.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(201..201 src has 13..201) v4 ==== 9543+0+0 (secure 0 0 0) 0x7f80c406dcf0 con 0x7f80cc136d60 2024-09-18T12:32:59.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80d0d39640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80b8038440 0x7f80b803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:32:59.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.698+0000 7f80d0d39640 10 monclient: get_auth_request con 0x7f80b8038440 auth_method 0 2024-09-18T12:32:59.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.701+0000 7f80d0d39640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80b8038440 0x7f80b803a8f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f80bc003940 tx=0x7f80bc006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:32:59.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.703+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f80cc136920 con 0x7f80cc136d60 2024-09-18T12:32:59.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.703+0000 7f80c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:32:59.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.703+0000 7f80c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:32:59.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.861+0000 7f80d2f9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:59.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.861+0000 7f80d2f9d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:32:59.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.861+0000 7f80d2f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:32:59.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:32:59.861+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f80cc136920 con 0x7f80cc136d60 2024-09-18T12:33:01.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.043+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v202) v1 ==== 166+0+0 (secure 0 0 0) 0x7f80cc136920 con 0x7f80cc136d60 2024-09-18T12:33:01.044 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.043+0000 7f80c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:01.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.043+0000 7f80c8ff9640 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-18T12:33:01.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.141+0000 7f80d2f9d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:01.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.141+0000 7f80d2f9d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:01.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.141+0000 7f80d2f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:01.142 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.141+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f80cc136920 con 0x7f80cc136d60 2024-09-18T12:33:01.143 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.143+0000 7f80c8ff9640 1 -- 172.21.15.144:0/2680186392 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v202) v1 ==== 145+0+0 (secure 0 0 0) 0x7f80cc136920 con 0x7f80cc136d60 2024-09-18T12:33:01.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.143+0000 7f80c8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:01.144 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.143+0000 7f80c8ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-18T12:33:01.144 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h9' already exists 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.144+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80b8038440 msgr2=0x7f80b803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.144+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80b8038440 0x7f80b803a8f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f80bc003940 tx=0x7f80bc006d80 comp rx=0 tx=0).stop 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.144+0000 7f80d2f9d640 10 monclient: shutdown 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.144+0000 7f80d2f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.144+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 msgr2=0x7f80cc099890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.144+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc099890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009f20 tx=0x7f80c4036000 comp rx=0 tx=0).stop 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 shutdown_connections 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80b8038440 0x7f80b803a8f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80cc136d60 0x7f80cc099890 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80cc131bd0 0x7f80cc131050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 --2- 172.21.15.144:0/2680186392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80cc069e10 0x7f80cc130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 >> 172.21.15.144:0/2680186392 conn(0x7f80cc0713b0 msgr2=0x7f80cc09d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:01.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 shutdown_connections 2024-09-18T12:33:01.147 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.145+0000 7f80d2f9d640 1 -- 172.21.15.144:0/2680186392 wait complete. 2024-09-18T12:33:01.155 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:01.155 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-18T12:33:01.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.231+0000 7f5643e94640 10 monclient: get_monmap_and_config 2024-09-18T12:33:01.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.231+0000 7f5643e94640 10 monclient: build_initial_monmap 2024-09-18T12:33:01.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.231+0000 7f5643e94640 10 monclient: monmap: 2024-09-18T12:33:01.232 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:01.232 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:01.232 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:01.231990+0000 2024-09-18T12:33:01.232 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:01.231990+0000 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:01.233 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.231+0000 7f5643e94640 1 Processor -- start 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.231+0000 7f5643e94640 1 -- start start 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.231+0000 7f5643e94640 10 monclient: init 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f563c131bd0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f563c069e10 0x7f563c131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient(hunting): picked mon.noname-c con 0x7f563c069e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:01.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f563c136d60 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0a06d0 con 0x7f563c136d60 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5641c30640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f563c131bd0 legacy=0x7f563c09ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c071780 con 0x7f563c131bd0 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c06a210 con 0x7f563c069e10 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5643e94640 10 monclient(hunting): authenticate will time out at 1216.801439s 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f564142f640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f563c069e10 msgr2=0x7f563c131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f564142f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f563c069e10 0x7f563c131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5642431640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f563c136d60 0x7f563c139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:50432/0 (socket says 172.21.15.144:50432) 2024-09-18T12:33:01.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.232+0000 7f5642431640 1 -- 172.21.15.144:0/3554714178 learned_addr learned my addr 172.21.15.144:0/3554714178 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:01.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 1 -- 172.21.15.144:0/3554714178 >> v1:172.21.15.116:6789/0 conn(0x7f563c136d60 legacy=0x7f563c139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:01.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496514211 0 0) 0x7f563c0a06d0 con 0x7f563c136d60 2024-09-18T12:33:01.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:01.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:01.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): my global_id is 104214 2024-09-18T12:33:01.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56240034e0 con 0x7f563c136d60 2024-09-18T12:33:01.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949774141 0 0) 0x7f56240034e0 con 0x7f563c136d60 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 >> v2:172.21.15.144:3300/0 conn(0x7f563c069e10 msgr2=0x7f563c131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 --2- 172.21.15.144:0/3554714178 >> v2:172.21.15.144:3300/0 conn(0x7f563c069e10 0x7f563c131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 >> v1:172.21.15.144:6789/0 conn(0x7f563c131bd0 legacy=0x7f563c09ebe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: _finish_hunting 0 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0a0010 con 0x7f563c136d60 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: _finish_auth 0 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f562c001860 con 0x7f563c136d60 2024-09-18T12:33:01.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5643e94640 5 monclient: authenticate success, global_id 104214 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5643e94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: dump: 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:01.246 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 >> v1:172.21.15.116:6789/0 conn(0x7f563c136d60 legacy=0x7f563c139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f5624003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): picked mon.c con 0x7f56240039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5624004780 0x7f5624004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): picked mon.b con 0x7f5624004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5624005330 0x7f562400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): picked mon.a con 0x7f5624005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c06a210 con 0x7f5624005330 2024-09-18T12:33:01.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562400c290 con 0x7f5624004780 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562400c450 con 0x7f56240039d0 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5640c2e640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5643e94640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f564142f640 1 -- 172.21.15.144:0/3554714178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5624004780 msgr2=0x7f5624004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f564142f640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5624004780 0x7f5624004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f5624003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5624005330 0x7f562400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 10 monclient(hunting): get_auth_request con 0x7f56240039d0 auth_method 0 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:01.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 10 monclient(hunting): get_auth_request con 0x7f5624005330 auth_method 0 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5641c30640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.235+0000 7f5642431640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 10 monclient(hunting): handle_auth_done global_id 104214 payload 293 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 1 -- 172.21.15.144:0/3554714178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5624004780 msgr2=0x7f5624004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:01.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5624004780 0x7f5624004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 1 -- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5624005330 msgr2=0x7f562400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5624005330 0x7f562400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 10 monclient: _finish_hunting 0 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 1 monclient: found mon.c 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5642431640 1 -- 172.21.15.144:0/3554714178 reap_dead start 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 1 -- 172.21.15.144:0/3554714178 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56240034e0 con 0x7f56240039d0 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 10 monclient: _finish_auth 0 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.236+0000 7f5641c30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.238+0000 7f5641c30640 1 -- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 msgr2=0x7f5624003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.238+0000 7f5641c30640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f5624003dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5634009f90 tx=0x7f56340095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e568ec4b01426c22 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:01.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634032430 con 0x7f56240039d0 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 10 monclient: dump: 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:01.251 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563402d070 con 0x7f56240039d0 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 1 -- 172.21.15.144:0/3554714178 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634032820 con 0x7f56240039d0 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 4 set_mon_vals no callback set 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5640c2e640 10 monclient: dump: 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.252 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 10 monclient: shutdown 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 1 -- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 msgr2=0x7f5624003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f5624003dc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5634009f90 tx=0x7f56340095d0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 1 -- 172.21.15.144:0/3554714178 shutdown_connections 2024-09-18T12:33:01.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 1 --2- 172.21.15.144:0/3554714178 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f5624003dc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.239+0000 7f5643e94640 1 -- 172.21.15.144:0/3554714178 >> 172.21.15.144:0/3554714178 conn(0x7f563c0713b0 msgr2=0x7f563c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.240+0000 7f5643e94640 1 -- 172.21.15.144:0/3554714178 shutdown_connections 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.240+0000 7f5643e94640 1 -- 172.21.15.144:0/3554714178 wait complete. 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.240+0000 7f5643e94640 10 monclient: build_initial_monmap 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.240+0000 7f5643e94640 10 monclient: monmap: 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:01.254 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.240+0000 7f5643e94640 1 Processor -- start 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.240+0000 7f5643e94640 1 -- start start 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.240+0000 7f5643e94640 10 monclient: init 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f563c099410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): picked mon.noname-c con 0x7f56240039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f563c099950 0x7f563c099d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): picked mon.noname-a con 0x7f563c099950 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c09a260 0x7f563c147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): picked mon.noname-b con 0x7f563c09a260 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:01.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c09bec0 con 0x7f563c099950 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0a4630 con 0x7f563c09a260 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c11b800 con 0x7f56240039d0 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5643e94640 10 monclient(hunting): authenticate will time out at 1216.810347s 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5642431640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c09a260 msgr2=0x7f563c147200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5639ffb640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5642431640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c09a260 0x7f563c147200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f563c099950 0x7f563c099d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f563c099950 0x7f563c099d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57530/0 (socket says 172.21.15.144:57530) 2024-09-18T12:33:01.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f563c099410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 1 -- 172.21.15.144:0/3627652951 learned_addr learned my addr 172.21.15.144:0/3627652951 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 10 monclient(hunting): get_auth_request con 0x7f563c099950 auth_method 0 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): get_auth_request con 0x7f56240039d0 auth_method 0 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:01.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient(hunting): handle_auth_done global_id 65057 payload 824 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 1 -- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c09a260 msgr2=0x7f563c147200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c09a260 0x7f563c147200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 1 -- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f563c099950 msgr2=0x7f563c099d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f563c099950 0x7f563c099d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient: _finish_hunting 0 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 1 monclient: found mon.noname-c 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f5641c30640 1 -- 172.21.15.144:0/3627652951 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c09b040 con 0x7f56240039d0 2024-09-18T12:33:01.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.241+0000 7f564142f640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f563c099950 0x7f563c099d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:01.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5641c30640 1 -- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 msgr2=0x7f563c099410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5641c30640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f563c099410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f562400c5e0 tx=0x7f5634035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56340315e0 con 0x7f56240039d0 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: dump: 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:01.261 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5643e94640 5 monclient: authenticate success, global_id 65057 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5634031740 con 0x7f56240039d0 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634031b00 con 0x7f56240039d0 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5643e94640 10 monclient: _renew_subs 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5643e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c11c610 con 0x7f56240039d0 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:01.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5639ffb640 10 monclient: dump: 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f563bfff640 4 set_mon_vals no callback set 2024-09-18T12:33:01.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5643e94640 10 monclient: _renew_subs 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5643e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.244+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c09a6d0 con 0x7f56240039d0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.245+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5634039070 con 0x7f56240039d0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.245+0000 7f5643e94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.245+0000 7f5643e94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.245+0000 7f5643e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.245+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c04bf60 con 0x7f56240039d0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.246+0000 7f5639ffb640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5628038490 0x7f562803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.246+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(202..202 src has 13..202) v4 ==== 9753+0+0 (secure 0 0 0) 0x7f5634006e40 con 0x7f56240039d0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.246+0000 7f564142f640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5628038490 0x7f562803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.246+0000 7f564142f640 10 monclient: get_auth_request con 0x7f5628038490 auth_method 0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.248+0000 7f564142f640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5628038490 0x7f562803a940 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f563c09f6b0 tx=0x7f5630009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.250+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f563403e400 con 0x7f56240039d0 2024-09-18T12:33:01.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.250+0000 7f5639ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:01.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.250+0000 7f5639ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:01.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.406+0000 7f5643e94640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:01.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.406+0000 7f5643e94640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:01.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.406+0000 7f5643e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:01.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:01.406+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f563c136210 con 0x7f56240039d0 2024-09-18T12:33:02.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.051+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v203) v1 ==== 168+0+0 (secure 0 0 0) 0x7f5634041120 con 0x7f56240039d0 2024-09-18T12:33:02.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.051+0000 7f5639ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:02.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.051+0000 7f5639ffb640 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-18T12:33:02.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.148+0000 7f5643e94640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:02.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.148+0000 7f5643e94640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:02.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.148+0000 7f5643e94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:02.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.148+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f563c136210 con 0x7f56240039d0 2024-09-18T12:33:02.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.149+0000 7f5639ffb640 1 -- 172.21.15.144:0/3627652951 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v203) v1 ==== 147+0+0 (secure 0 0 0) 0x7f5634030380 con 0x7f56240039d0 2024-09-18T12:33:02.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.149+0000 7f5639ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:02.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.149+0000 7f5639ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-18T12:33:02.150 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h10' already exists 2024-09-18T12:33:02.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5628038490 msgr2=0x7f562803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:02.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5628038490 0x7f562803a940 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f563c09f6b0 tx=0x7f5630009250 comp rx=0 tx=0).stop 2024-09-18T12:33:02.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 10 monclient: shutdown 2024-09-18T12:33:02.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:02.152 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 msgr2=0x7f563c099410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f563c099410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f562400c5e0 tx=0x7f5634035000 comp rx=0 tx=0).stop 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 shutdown_connections 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5628038490 0x7f562803a940 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f563c09a260 0x7f563c147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f563c099950 0x7f563c099d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 --2- 172.21.15.144:0/3627652951 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f56240039d0 0x7f563c099410 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 >> 172.21.15.144:0/3627652951 conn(0x7f563c0713b0 msgr2=0x7f563c0a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 shutdown_connections 2024-09-18T12:33:02.153 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.151+0000 7f5643e94640 1 -- 172.21.15.144:0/3627652951 wait complete. 2024-09-18T12:33:02.161 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:02.161 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-18T12:33:02.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.235+0000 7faa3fad4640 10 monclient: get_monmap_and_config 2024-09-18T12:33:02.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.235+0000 7faa3fad4640 10 monclient: build_initial_monmap 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.235+0000 7faa3fad4640 10 monclient: monmap: 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:02.236363+0000 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:02.236363+0000 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:02.237 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:02.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.235+0000 7faa3fad4640 1 Processor -- start 2024-09-18T12:33:02.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 1 -- start start 2024-09-18T12:33:02.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient: init 2024-09-18T12:33:02.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:02.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T12:33:02.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa380972c0 0x7faa38099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient(hunting): picked mon.noname-b con 0x7faa380972c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faa38097cb0 0x7faa3809b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient(hunting): picked mon.noname-c con 0x7faa38097cb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa3809bf10 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa38097690 con 0x7faa3809bf10 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3809d8d0 con 0x7faa38097cb0 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa38097a70 con 0x7faa380972c0 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3fad4640 10 monclient(hunting): authenticate will time out at 1217.805654s 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3d06f640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7faa38097cb0 msgr2=0x7faa3809b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:02.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.236+0000 7faa3d06f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faa38097cb0 0x7faa3809b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa380972c0 0x7faa38099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3e071640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faa3809bf10 0x7faa381347e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:50444/0 (socket says 172.21.15.144:50444) 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3e071640 1 -- 172.21.15.144:0/2024035632 learned_addr learned my addr 172.21.15.144:0/2024035632 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): get_auth_request con 0x7faa380972c0 auth_method 0 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient(hunting): handle_auth_done global_id 65060 payload 274 2024-09-18T12:33:02.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 1 -- 172.21.15.144:0/2024035632 >> v2:172.21.15.144:3300/0 conn(0x7faa38097cb0 msgr2=0x7faa3809b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:02.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 1 --2- 172.21.15.144:0/2024035632 >> v2:172.21.15.144:3300/0 conn(0x7faa38097cb0 0x7faa3809b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 1 -- 172.21.15.144:0/2024035632 >> v1:172.21.15.116:6789/0 conn(0x7faa3809bf10 legacy=0x7faa381347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:02.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient: _finish_hunting 0 2024-09-18T12:33:02.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 1 monclient: found mon.noname-b 2024-09-18T12:33:02.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:02.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:02.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.237+0000 7faa3d870640 1 -- 172.21.15.144:0/2024035632 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3806ebf0 con 0x7faa380972c0 2024-09-18T12:33:02.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3d870640 1 -- 172.21.15.144:0/2024035632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa380972c0 msgr2=0x7faa38099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:02.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3d870640 1 --2- 172.21.15.144:0/2024035632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa380972c0 0x7faa38099040 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7faa34009850 tx=0x7faa340090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e23437af9b9aa165 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:02.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 1 -- 172.21.15.144:0/2024035632 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34032440 con 0x7faa380972c0 2024-09-18T12:33:02.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:02.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: dump: 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: _finish_auth 0 2024-09-18T12:33:02.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 1 -- 172.21.15.144:0/2024035632 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa340325a0 con 0x7faa380972c0 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 1 -- 172.21.15.144:0/2024035632 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34032980 con 0x7faa380972c0 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3fad4640 5 monclient: authenticate success, global_id 65060 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3fad4640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3fad4640 4 set_mon_vals no callback set 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.240+0000 7faa3c86e640 10 monclient: dump: 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.251 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 10 monclient: shutdown 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 -- 172.21.15.144:0/2024035632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa380972c0 msgr2=0x7faa38099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 --2- 172.21.15.144:0/2024035632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa380972c0 0x7faa38099040 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7faa34009850 tx=0x7faa340090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 -- 172.21.15.144:0/2024035632 shutdown_connections 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 --2- 172.21.15.144:0/2024035632 >> v2:172.21.15.144:3300/0 conn(0x7faa38097cb0 0x7faa3809b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 --2- 172.21.15.144:0/2024035632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa380972c0 0x7faa38099040 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 -- 172.21.15.144:0/2024035632 >> 172.21.15.144:0/2024035632 conn(0x7faa380a0fe0 msgr2=0x7faa380a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 -- 172.21.15.144:0/2024035632 shutdown_connections 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 -- 172.21.15.144:0/2024035632 wait complete. 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 10 monclient: build_initial_monmap 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 10 monclient: monmap: 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:02.253 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.241+0000 7faa3fad4640 1 Processor -- start 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 1 -- start start 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient: init 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa380972c0 0x7faa3812f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): picked mon.noname-b con 0x7faa380972c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa38097cb0 0x7faa38133770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): picked mon.noname-c con 0x7faa38097cb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 0x7faa38133cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): picked mon.noname-a con 0x7faa3809bf10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:02.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3809ccc0 con 0x7faa3809bf10 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa381373b0 con 0x7faa380972c0 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3813a780 con 0x7faa38097cb0 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3fad4640 10 monclient(hunting): authenticate will time out at 1217.811461s 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d870640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa380972c0 msgr2=0x7faa3812f8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d870640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa380972c0 0x7faa3812f8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 0x7faa38133cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa38097cb0 0x7faa38133770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 0x7faa38133cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57534/0 (socket says 172.21.15.144:57534) 2024-09-18T12:33:02.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 1 -- 172.21.15.144:0/682804835 learned_addr learned my addr 172.21.15.144:0/682804835 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 10 monclient(hunting): get_auth_request con 0x7faa3809bf10 auth_method 0 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa38097cb0 0x7faa38133770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55508/0 (socket says 172.21.15.144:55508) 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 10 monclient(hunting): get_auth_request con 0x7faa38097cb0 auth_method 0 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3e071640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:02.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.242+0000 7faa3d06f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 10 monclient(hunting): handle_auth_done global_id 104220 payload 824 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 1 -- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa38097cb0 msgr2=0x7faa38133770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa38097cb0 0x7faa38133770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 1 -- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa380972c0 msgr2=0x7faa3812f8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa380972c0 0x7faa3812f8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 10 monclient: _finish_hunting 0 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 1 monclient: found mon.noname-a 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3e071640 1 -- 172.21.15.144:0/682804835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa34009390 con 0x7faa3809bf10 2024-09-18T12:33:02.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.243+0000 7faa3d06f640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa38097cb0 0x7faa38133770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3e071640 1 -- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 msgr2=0x7faa38133cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3e071640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 0x7faa38133cb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faa3000ccc0 tx=0x7faa30005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3000ce80 con 0x7faa3809bf10 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: dump: 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:02.258 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3fad4640 5 monclient: authenticate success, global_id 104220 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3fad4640 10 monclient: _renew_subs 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3fad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa38137010 con 0x7faa3809bf10 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3000aa60 con 0x7faa3809bf10 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3000ade0 con 0x7faa3809bf10 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:02.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2dffb640 10 monclient: dump: 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3fad4640 10 monclient: _renew_subs 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa2ffff640 4 set_mon_vals no callback set 2024-09-18T12:33:02.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3fad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.245+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3813ff20 con 0x7faa3809bf10 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.246+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7faa30007300 con 0x7faa3809bf10 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa3fad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa3fad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa3fad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3804bf60 con 0x7faa3809bf10 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa2dffb640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa1c038440 0x7faa1c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(203..203 src has 13..203) v4 ==== 9966+0+0 (secure 0 0 0) 0x7faa3004cd90 con 0x7faa3809bf10 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa3d870640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa1c038440 0x7faa1c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:02.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.247+0000 7faa3d870640 10 monclient: get_auth_request con 0x7faa1c038440 auth_method 0 2024-09-18T12:33:02.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.249+0000 7faa3d870640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa1c038440 0x7faa1c03a8f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faa34034000 tx=0x7faa34032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:02.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.252+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7faa30051070 con 0x7faa3809bf10 2024-09-18T12:33:02.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.252+0000 7faa2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:02.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.252+0000 7faa2dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:02.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.408+0000 7faa3fad4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:02.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.408+0000 7faa3fad4640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:02.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.408+0000 7faa3fad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:02.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:02.408+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7faa38136d90 con 0x7faa3809bf10 2024-09-18T12:33:03.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.063+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v204) v1 ==== 168+0+0 (secure 0 0 0) 0x7faa3001b400 con 0x7faa3809bf10 2024-09-18T12:33:03.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.063+0000 7faa2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:03.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.063+0000 7faa2dffb640 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-18T12:33:03.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.163+0000 7faa3fad4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:03.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.163+0000 7faa3fad4640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:03.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.163+0000 7faa3fad4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:03.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.163+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7faa38064b40 con 0x7faa3809bf10 2024-09-18T12:33:03.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.163+0000 7faa2dffb640 1 -- 172.21.15.144:0/682804835 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v204) v1 ==== 147+0+0 (secure 0 0 0) 0x7faa30010af0 con 0x7faa3809bf10 2024-09-18T12:33:03.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.164+0000 7faa2dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:03.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.164+0000 7faa2dffb640 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-18T12:33:03.165 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h11' already exists 2024-09-18T12:33:03.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa1c038440 msgr2=0x7faa1c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:03.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa1c038440 0x7faa1c03a8f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7faa34034000 tx=0x7faa34032000 comp rx=0 tx=0).stop 2024-09-18T12:33:03.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 10 monclient: shutdown 2024-09-18T12:33:03.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:03.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 msgr2=0x7faa38133cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 0x7faa38133cb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faa3000ccc0 tx=0x7faa30005370 comp rx=0 tx=0).stop 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 shutdown_connections 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa1c038440 0x7faa1c03a8f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa3809bf10 0x7faa38133cb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa38097cb0 0x7faa38133770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 --2- 172.21.15.144:0/682804835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa380972c0 0x7faa3812f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 >> 172.21.15.144:0/682804835 conn(0x7faa380a0fe0 msgr2=0x7faa3809dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 shutdown_connections 2024-09-18T12:33:03.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.166+0000 7faa3fad4640 1 -- 172.21.15.144:0/682804835 wait complete. 2024-09-18T12:33:03.177 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:03.177 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-18T12:33:03.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.251+0000 7fbc0e91d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:03.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.251+0000 7fbc0e91d640 10 monclient: build_initial_monmap 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.251+0000 7fbc0e91d640 10 monclient: monmap: 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:03.252589+0000 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:03.252589+0000 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:03.253 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:03.254 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:03.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 1 Processor -- start 2024-09-18T12:33:03.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 1 -- start start 2024-09-18T12:33:03.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 10 monclient: init 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbc0806df30 0x7fbc0806a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0806df30 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc0806e300 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc0809e1e0 0x7fbc0809e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.252+0000 7fbc0e91d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0809e1e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0e91d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc080696e0 con 0x7fbc0806e300 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0e91d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc0806a290 con 0x7fbc0806df30 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0e91d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc0809f840 con 0x7fbc0809e1e0 2024-09-18T12:33:03.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0e91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0e91d640 10 monclient(hunting): authenticate will time out at 1218.822040s 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0ceba640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fbc0809e1e0 msgr2=0x7fbc0809e600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc077fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fbc0806e300 legacy=0x7fbc0809dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc0ceba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbc0809e1e0 0x7fbc0809e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbc0806df30 0x7fbc0806a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbc0806df30 0x7fbc0806a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57546/0 (socket says 172.21.15.144:57546) 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 -- 172.21.15.144:0/2721034688 learned_addr learned my addr 172.21.15.144:0/2721034688 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): get_auth_request con 0x7fbc0806df30 auth_method 0 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:03.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient(hunting): handle_auth_done global_id 104223 payload 274 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 -- 172.21.15.144:0/2721034688 >> v2:172.21.15.144:3300/0 conn(0x7fbc0809e1e0 msgr2=0x7fbc0809e600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 --2- 172.21.15.144:0/2721034688 >> v2:172.21.15.144:3300/0 conn(0x7fbc0809e1e0 0x7fbc0809e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 -- 172.21.15.144:0/2721034688 >> v1:172.21.15.144:6789/0 conn(0x7fbc0806e300 legacy=0x7fbc0809dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 monclient: found mon.noname-a 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:03.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.253+0000 7fbc07fff640 1 -- 172.21.15.144:0/2721034688 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0809f600 con 0x7fbc0806df30 2024-09-18T12:33:03.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc07fff640 1 -- 172.21.15.144:0/2721034688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806df30 msgr2=0x7fbc0806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:03.265 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc07fff640 1 --2- 172.21.15.144:0/2721034688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806df30 0x7fbc0806a6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbc0806a3d0 tx=0x7fbbf80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8902bf68068c91 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 1 -- 172.21.15.144:0/2721034688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8032440 con 0x7fbc0806df30 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: dump: 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:03.266 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 1 -- 172.21.15.144:0/2721034688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf80325a0 con 0x7fbc0806df30 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 1 -- 172.21.15.144:0/2721034688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8032940 con 0x7fbc0806df30 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc0e91d640 5 monclient: authenticate success, global_id 104223 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc0e91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc0e91d640 4 set_mon_vals no callback set 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.256+0000 7fbc06ffd640 10 monclient: dump: 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:03.267 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 10 monclient: shutdown 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2721034688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806df30 msgr2=0x7fbc0806a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2721034688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806df30 0x7fbc0806a6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbc0806a3d0 tx=0x7fbbf80091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2721034688 shutdown_connections 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2721034688 >> v2:172.21.15.144:3300/0 conn(0x7fbc0809e1e0 0x7fbc0809e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2721034688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806df30 0x7fbc0806a6b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2721034688 >> 172.21.15.144:0/2721034688 conn(0x7fbc080a3b30 msgr2=0x7fbc0809fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2721034688 shutdown_connections 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2721034688 wait complete. 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 10 monclient: build_initial_monmap 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.257+0000 7fbc0e91d640 10 monclient: monmap: 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:03.269 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 Processor -- start 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 -- start start 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient: init 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc0806df30 0x7fbc081412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0806df30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806e300 0x7fbc08131ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0806e300 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:03.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 0x7fbc080979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc07fff640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc0806df30 msgr2=0x7fbc081412f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0809e1e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc08099ea0 con 0x7fbc0806e300 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0806d9b0 con 0x7fbc0806df30 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc0806df30 0x7fbc081412f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0811f2c0 con 0x7fbc0809e1e0 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.258+0000 7fbc0e91d640 10 monclient(hunting): authenticate will time out at 1218.827930s 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806e300 0x7fbc08131ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:03.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 0x7fbc080979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806e300 0x7fbc08131ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57556/0 (socket says 172.21.15.144:57556) 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 1 -- 172.21.15.144:0/2330576252 learned_addr learned my addr 172.21.15.144:0/2330576252 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 10 monclient(hunting): get_auth_request con 0x7fbc0806e300 auth_method 0 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): get_auth_request con 0x7fbc0809e1e0 auth_method 0 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:03.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient(hunting): handle_auth_done global_id 65066 payload 824 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 1 -- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc0806df30 msgr2=0x7fbc081412f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc0806df30 0x7fbc081412f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 1 -- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806e300 msgr2=0x7fbc08131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806e300 0x7fbc08131ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient: _finish_hunting 0 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 1 monclient: found mon.noname-c 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc0ceba640 1 -- 172.21.15.144:0/2330576252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8009450 con 0x7fbc0809e1e0 2024-09-18T12:33:03.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.259+0000 7fbc077fe640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806e300 0x7fbc08131ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.261+0000 7fbc0ceba640 1 -- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 msgr2=0x7fbc080979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.261+0000 7fbc0ceba640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 0x7fbc080979d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000bdf0 tx=0x7fbc000096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0ceba640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 0x7fbc080979d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000bdf0 tx=0x7fbc000096a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fbc00006570 mon_map magic: 0 v1 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00006570 con 0x7fbc0809e1e0 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 10 monclient: dump: 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:03.274 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 10 monclient: _finish_auth 0 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0000f070 con 0x7fbc0809e1e0 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbbeffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0e91d640 5 monclient: authenticate success, global_id 65066 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0e91d640 10 monclient: _renew_subs 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0811efe0 con 0x7fbc0809e1e0 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc067fc640 4 set_mon_vals no callback set 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0e91d640 10 monclient: _renew_subs 2024-09-18T12:33:03.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.262+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc081420d0 con 0x7fbc0809e1e0 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00006750 con 0x7fbc0809e1e0 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbbeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbbeffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbbeffff640 10 monclient: dump: 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.276 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbc0001e3a0 con 0x7fbc0809e1e0 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbc0e91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbc0e91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbc0e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0804bf60 con 0x7fbc0809e1e0 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbbeffff640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbbf4038440 0x7fbbf403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:03.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbc07fff640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbbf4038440 0x7fbbf403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:03.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.263+0000 7fbc07fff640 10 monclient: get_auth_request con 0x7fbbf4038440 auth_method 0 2024-09-18T12:33:03.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.265+0000 7fbc07fff640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbbf4038440 0x7fbbf403a8f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbc08071980 tx=0x7fbbf8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:03.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.268+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(204..204 src has 13..204) v4 ==== 10179+0+0 (secure 0 0 0) 0x7fbc0004d6d0 con 0x7fbc0809e1e0 2024-09-18T12:33:03.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.885+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbc00006b80 con 0x7fbc0809e1e0 2024-09-18T12:33:03.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.885+0000 7fbbeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:03.886 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:03.885+0000 7fbbeffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:04.045 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.044+0000 7fbc0e91d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.044+0000 7fbc0e91d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.044+0000 7fbc0e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:04.046 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.044+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbc08068a20 con 0x7fbc0809e1e0 2024-09-18T12:33:04.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.064+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v205) v1 ==== 168+0+0 (secure 0 0 0) 0x7fbc0001a030 con 0x7fbc0809e1e0 2024-09-18T12:33:04.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.064+0000 7fbbeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.064+0000 7fbbeffff640 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-18T12:33:04.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.162+0000 7fbc0e91d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.162+0000 7fbc0e91d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.162+0000 7fbc0e91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:04.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.162+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fbc08064a90 con 0x7fbc0809e1e0 2024-09-18T12:33:04.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.164+0000 7fbbeffff640 1 -- 172.21.15.144:0/2330576252 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v205) v1 ==== 147+0+0 (secure 0 0 0) 0x7fbc0001a030 con 0x7fbc0809e1e0 2024-09-18T12:33:04.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.164+0000 7fbbeffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.164+0000 7fbbeffff640 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-18T12:33:04.165 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h12' already exists 2024-09-18T12:33:04.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.165+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbbf4038440 msgr2=0x7fbbf403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:04.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.165+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbbf4038440 0x7fbbf403a8f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbc08071980 tx=0x7fbbf8032000 comp rx=0 tx=0).stop 2024-09-18T12:33:04.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.165+0000 7fbc0e91d640 10 monclient: shutdown 2024-09-18T12:33:04.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.165+0000 7fbc0e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:04.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.165+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 msgr2=0x7fbc080979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.165+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 0x7fbc080979d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000bdf0 tx=0x7fbc000096a0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 shutdown_connections 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbbf4038440 0x7fbbf403a8f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbc0809e1e0 0x7fbc080979d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc0806e300 0x7fbc08131ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 --2- 172.21.15.144:0/2330576252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbc0806df30 0x7fbc081412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 >> 172.21.15.144:0/2330576252 conn(0x7fbc080a3b30 msgr2=0x7fbc0806c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 shutdown_connections 2024-09-18T12:33:04.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.166+0000 7fbc0e91d640 1 -- 172.21.15.144:0/2330576252 wait complete. 2024-09-18T12:33:04.176 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:04.176 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.249+0000 7fb963f1d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.249+0000 7fb963f1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.249+0000 7fb963f1d640 10 monclient: monmap: 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:04.250788+0000 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:04.250788+0000 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:04.251 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:04.252 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:04.252 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.250+0000 7fb963f1d640 1 Processor -- start 2024-09-18T12:33:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.250+0000 7fb963f1d640 1 -- start start 2024-09-18T12:33:04.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.250+0000 7fb963f1d640 10 monclient: init 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb95c098050 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb95c09fe90 0x7fb95c098420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c09fe90 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb95c098960 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c132710 con 0x7fb95c098050 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c0a1230 con 0x7fb95c098960 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95c0a1510 con 0x7fb95c09fe90 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb963f1d640 10 monclient(hunting): authenticate will time out at 1219.820207s 2024-09-18T12:33:04.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb9624ba640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fb95c098960 legacy=0x7fb95c138d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb9614b8640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fb95c09fe90 msgr2=0x7fb95c098420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb9614b8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb95c09fe90 0x7fb95c098420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb961cb9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb95c098050 0x7fb95c132000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:50450/0 (socket says 172.21.15.144:50450) 2024-09-18T12:33:04.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.251+0000 7fb961cb9640 1 -- 172.21.15.144:0/130452426 learned_addr learned my addr 172.21.15.144:0/130452426 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.253+0000 7fb961cb9640 1 -- 172.21.15.144:0/130452426 >> v1:172.21.15.116:6789/0 conn(0x7fb95c098050 legacy=0x7fb95c132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894184535 0 0) 0x7fb95c132710 con 0x7fb95c098050 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): my global_id is 104229 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9480034e0 con 0x7fb95c098050 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781563226 0 0) 0x7fb9480034e0 con 0x7fb95c098050 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 >> v2:172.21.15.144:3300/0 conn(0x7fb95c09fe90 msgr2=0x7fb95c098420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 --2- 172.21.15.144:0/130452426 >> v2:172.21.15.144:3300/0 conn(0x7fb95c09fe90 0x7fb95c098420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 >> v1:172.21.15.144:6789/0 conn(0x7fb95c098960 legacy=0x7fb95c138d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: _finish_hunting 0 2024-09-18T12:33:04.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c131bf0 con 0x7fb95c098050 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: _finish_auth 0 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fb958001860 con 0x7fb95c098050 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: dump: 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:04.259 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 >> v1:172.21.15.116:6789/0 conn(0x7fb95c098050 legacy=0x7fb95c132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9480039d0 0x7fb948003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): picked mon.c con 0x7fb9480039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb948004780 0x7fb948004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): picked mon.a con 0x7fb948004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb948005230 0x7fb94800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): picked mon.b con 0x7fb948005230 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c0a1510 con 0x7fb948004780 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94800c1f0 con 0x7fb948005230 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94800c3b0 con 0x7fb9480039d0 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb960cb7640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:04.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9624ba640 1 -- 172.21.15.144:0/130452426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb948005230 msgr2=0x7fb94800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9624ba640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb948005230 0x7fb94800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9480039d0 0x7fb948003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb948004780 0x7fb948004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): get_auth_request con 0x7fb9480039d0 auth_method 0 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 10 monclient(hunting): get_auth_request con 0x7fb948004780 auth_method 0 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:04.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient(hunting): handle_auth_done global_id 104229 payload 293 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 1 -- 172.21.15.144:0/130452426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb948005230 msgr2=0x7fb94800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb948005230 0x7fb94800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 1 -- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb948004780 msgr2=0x7fb948004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb948004780 0x7fb948004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient: _finish_hunting 0 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 1 monclient: found mon.c 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9624ba640 1 -- 172.21.15.144:0/130452426 reap_dead start 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:04.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 1 -- 172.21.15.144:0/130452426 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9480034e0 con 0x7fb9480039d0 2024-09-18T12:33:04.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 10 monclient: _finish_auth 0 2024-09-18T12:33:04.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb961cb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:04.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb963f1d640 5 monclient: authenticate success, global_id 104229 2024-09-18T12:33:04.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb963f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:04.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.254+0000 7fb9614b8640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb948004780 0x7fb948004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:04.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.257+0000 7fb961cb9640 1 -- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9480039d0 msgr2=0x7fb948003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.257+0000 7fb961cb9640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9480039d0 0x7fb948003dc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb9580014d0 tx=0x7fb958007af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55455697215baab6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.257+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb958005890 con 0x7fb9480039d0 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.257+0000 7fb960cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.257+0000 7fb960cb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.257+0000 7fb960cb7640 10 monclient: dump: 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:04.270 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb958005a30 con 0x7fb9480039d0 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb960cb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb960cb7640 1 -- 172.21.15.144:0/130452426 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb958005df0 con 0x7fb9480039d0 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 4 set_mon_vals no callback set 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb960cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb960cb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb960cb7640 10 monclient: dump: 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:04.271 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 10 monclient: shutdown 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 1 -- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9480039d0 msgr2=0x7fb948003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9480039d0 0x7fb948003dc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb9580014d0 tx=0x7fb958007af0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 1 -- 172.21.15.144:0/130452426 shutdown_connections 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 1 --2- 172.21.15.144:0/130452426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb9480039d0 0x7fb948003dc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.272 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 1 -- 172.21.15.144:0/130452426 >> 172.21.15.144:0/130452426 conn(0x7fb95c0a5190 msgr2=0x7fb95c0a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 1 -- 172.21.15.144:0/130452426 shutdown_connections 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.258+0000 7fb963f1d640 1 -- 172.21.15.144:0/130452426 wait complete. 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient: monmap: 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:04.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 Processor -- start 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 -- start start 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient: init 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9480039d0 0x7fb95c09b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9480039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 0x7fb95c130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c11bcb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c11cda0 0x7fb95c140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c11cda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c0a3a20 con 0x7fb95c11bcb0 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:04.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c11e2c0 con 0x7fb9480039d0 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c11dfe0 con 0x7fb95c11cda0 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb963f1d640 10 monclient(hunting): authenticate will time out at 1219.828813s 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb961cb9640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9480039d0 msgr2=0x7fb95c09b240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb961cb9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9480039d0 0x7fb95c09b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb9614b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 0x7fb95c130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb9624ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c11cda0 0x7fb95c140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.259+0000 7fb9614b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 0x7fb95c130be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57576/0 (socket says 172.21.15.144:57576) 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 1 -- 172.21.15.144:0/614315762 learned_addr learned my addr 172.21.15.144:0/614315762 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): get_auth_request con 0x7fb95c11bcb0 auth_method 0 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:04.275 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 10 monclient(hunting): get_auth_request con 0x7fb95c11cda0 auth_method 0 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient(hunting): handle_auth_done global_id 104232 payload 824 2024-09-18T12:33:04.276 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 1 -- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c11cda0 msgr2=0x7fb95c140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c11cda0 0x7fb95c140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 1 -- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9480039d0 msgr2=0x7fb95c09b240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9480039d0 0x7fb95c09b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient: _finish_hunting 0 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 1 monclient: found mon.noname-a 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9614b8640 1 -- 172.21.15.144:0/614315762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c11dd00 con 0x7fb95c11bcb0 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.260+0000 7fb9624ba640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c11cda0 0x7fb95c140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb9614b8640 1 -- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 msgr2=0x7fb95c130be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb9614b8640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 0x7fb95c130be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009740 tx=0x7fb94c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c0128e0 con 0x7fb95c11bcb0 2024-09-18T12:33:04.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: dump: 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:04.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c012a40 con 0x7fb95c11bcb0 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb963f1d640 5 monclient: authenticate success, global_id 104232 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb963f1d640 10 monclient: _renew_subs 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb963f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c11c8d0 con 0x7fb95c11bcb0 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c012dc0 con 0x7fb95c11bcb0 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:04.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb951ffb640 10 monclient: dump: 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:04.280 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.262+0000 7fb953fff640 4 set_mon_vals no callback set 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.263+0000 7fb963f1d640 10 monclient: _renew_subs 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.263+0000 7fb963f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.263+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c131370 con 0x7fb95c11bcb0 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.263+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb94c011560 con 0x7fb95c11bcb0 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.263+0000 7fb951ffb640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb940038470 0x7fb94003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.263+0000 7fb963f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.264+0000 7fb963f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.264+0000 7fb963f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.264+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c04bf60 con 0x7fb95c11bcb0 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.264+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(205..205 src has 13..205) v4 ==== 10392+0+0 (secure 0 0 0) 0x7fb94c04c810 con 0x7fb95c11bcb0 2024-09-18T12:33:04.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.264+0000 7fb961cb9640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb940038470 0x7fb94003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:04.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.264+0000 7fb961cb9640 10 monclient: get_auth_request con 0x7fb940038470 auth_method 0 2024-09-18T12:33:04.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.266+0000 7fb961cb9640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb940038470 0x7fb94003a920 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb958001ed0 tx=0x7fb958034000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:04.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.269+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb94c0198c0 con 0x7fb95c11bcb0 2024-09-18T12:33:04.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.269+0000 7fb951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:04.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.269+0000 7fb951ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:04.427 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.426+0000 7fb963f1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.426+0000 7fb963f1d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:04.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.426+0000 7fb963f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:04.428 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:04.426+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb95c0649e0 con 0x7fb95c11bcb0 2024-09-18T12:33:05.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.078+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v206) v1 ==== 168+0+0 (secure 0 0 0) 0x7fb94c019310 con 0x7fb95c11bcb0 2024-09-18T12:33:05.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.078+0000 7fb951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:05.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.078+0000 7fb951ffb640 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.176+0000 7fb963f1d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.176+0000 7fb963f1d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.176+0000 7fb963f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.176+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb95c141b90 con 0x7fb95c11bcb0 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.177+0000 7fb951ffb640 1 -- 172.21.15.144:0/614315762 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v206) v1 ==== 147+0+0 (secure 0 0 0) 0x7fb94c019310 con 0x7fb95c11bcb0 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.177+0000 7fb951ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.177+0000 7fb951ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-18T12:33:05.178 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h13' already exists 2024-09-18T12:33:05.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb940038470 msgr2=0x7fb94003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:05.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb940038470 0x7fb94003a920 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb958001ed0 tx=0x7fb958034000 comp rx=0 tx=0).stop 2024-09-18T12:33:05.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 10 monclient: shutdown 2024-09-18T12:33:05.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:05.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 msgr2=0x7fb95c130be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:05.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 0x7fb95c130be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009740 tx=0x7fb94c009060 comp rx=0 tx=0).stop 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 shutdown_connections 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb940038470 0x7fb94003a920 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c11cda0 0x7fb95c140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c11bcb0 0x7fb95c130be0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 --2- 172.21.15.144:0/614315762 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb9480039d0 0x7fb95c09b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 >> 172.21.15.144:0/614315762 conn(0x7fb95c0a5190 msgr2=0x7fb95c069650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 shutdown_connections 2024-09-18T12:33:05.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.179+0000 7fb963f1d640 1 -- 172.21.15.144:0/614315762 wait complete. 2024-09-18T12:33:05.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:05.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.262+0000 7f7f86817640 10 monclient: get_monmap_and_config 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.262+0000 7f7f86817640 10 monclient: build_initial_monmap 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.262+0000 7f7f86817640 10 monclient: monmap: 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:05.263776+0000 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:05.263776+0000 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:05.264 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:05.265 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:05.265 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:05.265 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:05.265 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:05.265 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:05.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.263+0000 7f7f86817640 1 Processor -- start 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.263+0000 7f7f86817640 1 -- start start 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.263+0000 7f7f86817640 10 monclient: init 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.263+0000 7f7f86817640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f80094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient(hunting): picked mon.noname-b con 0x7f7f80067640 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f800945c0 0x7f7f80094990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient(hunting): picked mon.noname-a con 0x7f7f800945c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f8009c020 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8009d930 con 0x7f7f8009c020 2024-09-18T12:33:05.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f80099370 con 0x7f7f800945c0 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8006f7d0 con 0x7f7f80067640 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f86817640 10 monclient(hunting): authenticate will time out at 1220.833079s 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f84db4640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f7f8009c020 legacy=0x7f7f8009c4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f800945c0 0x7f7f80094990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f80094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f800945c0 0x7f7f80094990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57590/0 (socket says 172.21.15.144:57590) 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/41864487 learned_addr learned my addr 172.21.15.144:0/41864487 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): get_auth_request con 0x7f7f800945c0 auth_method 0 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:05.269 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request con 0x7f7f80067640 auth_method 0 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:05.270 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_done global_id 104235 payload 274 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 msgr2=0x7f7f80094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 --2- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f80094080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/41864487 >> v1:172.21.15.144:6789/0 conn(0x7f7f8009c020 legacy=0x7f7f8009c4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:05.271 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.264+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/41864487 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f80067a10 con 0x7f7f800945c0 2024-09-18T12:33:05.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.267+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 msgr2=0x7f7f80094990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:05.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.267+0000 7f7f7f7fe640 1 --2- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80094990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009910 tx=0x7f7f740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9faaa74a7b242d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:05.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.267+0000 7f7f7effd640 1 -- 172.21.15.144:0/41864487 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7402ae90 con 0x7f7f800945c0 2024-09-18T12:33:05.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:05.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:05.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: dump: 2024-09-18T12:33:05.277 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: _finish_auth 0 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 1 -- 172.21.15.144:0/41864487 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f74032440 con 0x7f7f800945c0 2024-09-18T12:33:05.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 5 monclient: authenticate success, global_id 104235 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 1 -- 172.21.15.144:0/41864487 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f740327e0 con 0x7f7f800945c0 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 4 set_mon_vals no callback set 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f7effd640 10 monclient: dump: 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.279 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 10 monclient: shutdown 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 -- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 msgr2=0x7f7f80094990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:05.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 --2- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80094990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009910 tx=0x7f7f740091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 -- 172.21.15.144:0/41864487 shutdown_connections 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 --2- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80094990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 --2- 172.21.15.144:0/41864487 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f80094080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 -- 172.21.15.144:0/41864487 >> 172.21.15.144:0/41864487 conn(0x7f7f800a1630 msgr2=0x7f7f800a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 -- 172.21.15.144:0/41864487 shutdown_connections 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.268+0000 7f7f86817640 1 -- 172.21.15.144:0/41864487 wait complete. 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 10 monclient: build_initial_monmap 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 10 monclient: monmap: 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:05.281 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 1 Processor -- start 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 1 -- start start 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 10 monclient: init 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f8008d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.269+0000 7f7f86817640 10 monclient(hunting): picked mon.noname-c con 0x7f7f80067640 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:05.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80093510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 10 monclient(hunting): picked mon.noname-a con 0x7f7f800945c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f8009c020 0x7f7f80093a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 10 monclient(hunting): picked mon.noname-b con 0x7f7f8009c020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8008cd40 con 0x7f7f800945c0 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8008c780 con 0x7f7f8009c020 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8008ca60 con 0x7f7f80067640 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f86817640 10 monclient(hunting): authenticate will time out at 1220.839035s 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f84db4640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f8009c020 msgr2=0x7f7f80093a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f84db4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f8009c020 0x7f7f80093a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:05.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f8008d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80093510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80093510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:57604/0 (socket says 172.21.15.144:57604) 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f8008d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55566/0 (socket says 172.21.15.144:55566) 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/928914371 learned_addr learned my addr 172.21.15.144:0/928914371 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): get_auth_request con 0x7f7f800945c0 auth_method 0 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request con 0x7f7f80067640 auth_method 0 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:05.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient(hunting): handle_auth_done global_id 104238 payload 824 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 msgr2=0x7f7f8008d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f8008d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f8009c020 msgr2=0x7f7f80093a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f8009c020 0x7f7f80093a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:05.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/928914371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f74009450 con 0x7f7f800945c0 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.270+0000 7f7f7ffff640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f8008d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f7f7fe640 1 -- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 msgr2=0x7f7f80093510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f7f7fe640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80093510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009910 tx=0x7f7f74032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f74000c40 con 0x7f7f800945c0 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: dump: 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.286 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: _finish_auth 0 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7402ae90 con 0x7f7f800945c0 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f86817640 5 monclient: authenticate success, global_id 104238 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f86817640 10 monclient: _renew_subs 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7403c400 con 0x7f7f800945c0 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f86817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:05.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8008b820 con 0x7f7f800945c0 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f67fff640 10 monclient: dump: 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:05.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f7e7fc640 4 set_mon_vals no callback set 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f86817640 10 monclient: _renew_subs 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f86817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.273+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f800894f0 con 0x7f7f800945c0 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f86817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f86817640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f86817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8004bf60 con 0x7f7f800945c0 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7f7403c770 con 0x7f7f800945c0 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f67fff640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f6c038440 0x7f7f6c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(206..206 src has 13..206) v4 ==== 10605+0+0 (secure 0 0 0) 0x7f7f7406d780 con 0x7f7f800945c0 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f7ffff640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f6c038440 0x7f7f6c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:05.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.274+0000 7f7f7ffff640 10 monclient: get_auth_request con 0x7f7f6c038440 auth_method 0 2024-09-18T12:33:05.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.277+0000 7f7f7ffff640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f6c038440 0x7f7f6c03a8f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7f700113e0 tx=0x7f7f700090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:05.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.279+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7f7404ad10 con 0x7f7f800945c0 2024-09-18T12:33:05.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.279+0000 7f7f67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:05.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.279+0000 7f7f67fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:05.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.435+0000 7f7f86817640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:05.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.435+0000 7f7f86817640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:05.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.435+0000 7f7f86817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:05.436 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.435+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7f80064a70 con 0x7f7f800945c0 2024-09-18T12:33:05.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.975+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v207) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7f74037070 con 0x7f7f800945c0 2024-09-18T12:33:05.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.975+0000 7f7f67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:05.977 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:05.975+0000 7f7f67fff640 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-18T12:33:06.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.072+0000 7f7f86817640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:06.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.072+0000 7f7f86817640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:06.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.072+0000 7f7f86817640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:06.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.072+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7f80064a70 con 0x7f7f800945c0 2024-09-18T12:33:06.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.073+0000 7f7f67fff640 1 -- 172.21.15.144:0/928914371 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v207) v1 ==== 147+0+0 (secure 0 0 0) 0x7f7f74030e40 con 0x7f7f800945c0 2024-09-18T12:33:06.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.073+0000 7f7f67fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:06.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.073+0000 7f7f67fff640 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-18T12:33:06.075 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h14' already exists 2024-09-18T12:33:06.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f6c038440 msgr2=0x7f7f6c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:06.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f6c038440 0x7f7f6c03a8f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7f700113e0 tx=0x7f7f700090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 10 monclient: shutdown 2024-09-18T12:33:06.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:06.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 msgr2=0x7f7f80093510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:06.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80093510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009910 tx=0x7f7f74032000 comp rx=0 tx=0).stop 2024-09-18T12:33:06.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 shutdown_connections 2024-09-18T12:33:06.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7f6c038440 0x7f7f6c03a8f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f8009c020 0x7f7f80093a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f800945c0 0x7f7f80093510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 --2- 172.21.15.144:0/928914371 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f80067640 0x7f7f8008d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.074+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 >> 172.21.15.144:0/928914371 conn(0x7f7f800a1630 msgr2=0x7f7f80128710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:06.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.075+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 shutdown_connections 2024-09-18T12:33:06.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.075+0000 7f7f86817640 1 -- 172.21.15.144:0/928914371 wait complete. 2024-09-18T12:33:06.084 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:06.085 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-18T12:33:06.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.157+0000 7f6a3219d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.157+0000 7f6a3219d640 10 monclient: build_initial_monmap 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.157+0000 7f6a3219d640 10 monclient: monmap: 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:06.158683+0000 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:06.158683+0000 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:06.159 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:06.160 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:06.160 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:06.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.158+0000 7f6a3219d640 1 Processor -- start 2024-09-18T12:33:06.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.158+0000 7f6a3219d640 1 -- start start 2024-09-18T12:33:06.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.158+0000 7f6a3219d640 10 monclient: init 2024-09-18T12:33:06.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.158+0000 7f6a3219d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:06.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.158+0000 7f6a3219d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T12:33:06.161 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.158+0000 7f6a3219d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a2c06df30 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a2c06a6b0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a2c06e300 0x7f6a2c09e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2c06e300 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0696e0 con 0x7f6a2c06a6b0 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c09f600 con 0x7f6a2c06df30 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2c06aab0 con 0x7f6a2c06e300 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a3219d640 10 monclient(hunting): authenticate will time out at 1221.728048s 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a2b7fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f6a2c06df30 legacy=0x7f6a2c09dad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a2bfff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f6a2c06e300 msgr2=0x7f6a2c09e8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a2bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a2c06e300 0x7f6a2c09e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:06.162 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a2affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6a2c06a6b0 0x7f6a2c09e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:50458/0 (socket says 172.21.15.144:50458) 2024-09-18T12:33:06.163 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.159+0000 7f6a2affd640 1 -- 172.21.15.144:0/3252383036 learned_addr learned my addr 172.21.15.144:0/3252383036 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:06.165 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.161+0000 7f6a2affd640 1 -- 172.21.15.144:0/3252383036 >> v1:172.21.15.116:6789/0 conn(0x7f6a2c06a6b0 legacy=0x7f6a2c09e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:06.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.161+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355316154 0 0) 0x7f6a2c0696e0 con 0x7f6a2c06a6b0 2024-09-18T12:33:06.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.161+0000 7f6a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:06.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.161+0000 7f6a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:06.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.161+0000 7f6a2a7fc640 10 monclient(hunting): my global_id is 104241 2024-09-18T12:33:06.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.161+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a140034e0 con 0x7f6a2c06a6b0 2024-09-18T12:33:06.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229627188 0 0) 0x7f6a140034e0 con 0x7f6a2c06a6b0 2024-09-18T12:33:06.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 >> v2:172.21.15.144:3300/0 conn(0x7f6a2c06e300 msgr2=0x7f6a2c09e8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:06.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 --2- 172.21.15.144:0/3252383036 >> v2:172.21.15.144:3300/0 conn(0x7f6a2c06e300 0x7f6a2c09e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 >> v1:172.21.15.144:6789/0 conn(0x7f6a2c06df30 legacy=0x7f6a2c09dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:06.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:33:06.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:06.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:06.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:06.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c069f40 con 0x7f6a2c06a6b0 2024-09-18T12:33:06.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:06.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:06.168 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f6a20001860 con 0x7f6a2c06a6b0 2024-09-18T12:33:06.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a3219d640 5 monclient: authenticate success, global_id 104241 2024-09-18T12:33:06.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a3219d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:06.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:06.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:06.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: dump: 2024-09-18T12:33:06.169 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:06.170 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:06.170 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.170 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.170 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:06.170 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:06.171 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:06.171 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:06.171 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:06.171 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:06.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:06.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:06.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 >> v1:172.21.15.116:6789/0 conn(0x7f6a2c06a6b0 legacy=0x7f6a2c09e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:06.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:06.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a140039d0 0x7f6a14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): picked mon.b con 0x7f6a140039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:06.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a14004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): picked mon.c con 0x7f6a14004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:06.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a14005330 0x7f6a1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): picked mon.a con 0x7f6a14005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:06.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:06.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c06aab0 con 0x7f6a14005330 2024-09-18T12:33:06.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:06.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1400c290 con 0x7f6a140039d0 2024-09-18T12:33:06.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:06.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1400c450 con 0x7f6a14004780 2024-09-18T12:33:06.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:06.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:06.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a3219d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:06.174 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2b7fe640 1 -- 172.21.15.144:0/3252383036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a140039d0 msgr2=0x7f6a14003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:06.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2b7fe640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a140039d0 0x7f6a14003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:06.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a14004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:06.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a14005330 0x7f6a1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:06.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): get_auth_request con 0x7f6a14004780 auth_method 0 2024-09-18T12:33:06.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:06.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:06.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:06.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 10 monclient(hunting): get_auth_request con 0x7f6a14005330 auth_method 0 2024-09-18T12:33:06.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:06.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:06.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:06.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:06.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:06.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:06.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:06.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:06.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:06.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient(hunting): handle_auth_done global_id 104241 payload 293 2024-09-18T12:33:06.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 1 -- 172.21.15.144:0/3252383036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a140039d0 msgr2=0x7f6a14003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:06.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a140039d0 0x7f6a14003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 1 -- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a14005330 msgr2=0x7f6a1400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:06.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a14005330 0x7f6a1400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:06.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 1 monclient: found mon.c 2024-09-18T12:33:06.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:06.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:06.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 1 -- 172.21.15.144:0/3252383036 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140034e0 con 0x7f6a14004780 2024-09-18T12:33:06.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 10 monclient: _finish_auth 0 2024-09-18T12:33:06.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:06.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a14005330 0x7f6a1400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:06.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.162+0000 7f6a2bfff640 1 -- 172.21.15.144:0/3252383036 reap_dead start 2024-09-18T12:33:06.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.165+0000 7f6a2affd640 1 -- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 msgr2=0x7f6a14004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:06.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.165+0000 7f6a2affd640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a14004bf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6a200014d0 tx=0x7f6a2c064d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c764775b0c216e52 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:06.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.165+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20005890 con 0x7f6a14004780 2024-09-18T12:33:06.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:06.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:06.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2a7fc640 10 monclient: dump: 2024-09-18T12:33:06.191 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:06.192 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:06.192 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.192 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.192 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:06.192 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:06.192 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:06.193 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:06.193 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:06.193 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:06.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:06.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2affd640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a14004bf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6a200014d0 tx=0x7f6a2c064d20 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f6a20005df0 mon_map magic: 0 v1 2024-09-18T12:33:06.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a20005a30 con 0x7f6a14004780 2024-09-18T12:33:06.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:06.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:06.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 4 set_mon_vals no callback set 2024-09-18T12:33:06.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 10 monclient: shutdown 2024-09-18T12:33:06.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:06.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 1 -- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 msgr2=0x7f6a14004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:06.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a14004bf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6a200014d0 tx=0x7f6a2c064d20 comp rx=0 tx=0).stop 2024-09-18T12:33:06.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2a7fc640 1 -- 172.21.15.144:0/3252383036 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f6a20005df0 con 0x7f6a14004780 2024-09-18T12:33:06.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a2a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T12:33:06.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 1 -- 172.21.15.144:0/3252383036 shutdown_connections 2024-09-18T12:33:06.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 1 --2- 172.21.15.144:0/3252383036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a14004bf0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 1 -- 172.21.15.144:0/3252383036 >> 172.21.15.144:0/3252383036 conn(0x7f6a2c0a3b30 msgr2=0x7f6a2c09fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:06.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 1 -- 172.21.15.144:0/3252383036 shutdown_connections 2024-09-18T12:33:06.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.166+0000 7f6a3219d640 1 -- 172.21.15.144:0/3252383036 wait complete. 2024-09-18T12:33:06.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.167+0000 7f6a3219d640 10 monclient: build_initial_monmap 2024-09-18T12:33:06.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.167+0000 7f6a3219d640 10 monclient: monmap: 2024-09-18T12:33:06.197 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:06.197 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:06.197 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:06.197 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:06.197 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:06.198 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:06.198 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:06.198 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:06.198 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:06.198 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:06.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.167+0000 7f6a3219d640 1 Processor -- start 2024-09-18T12:33:06.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.167+0000 7f6a3219d640 1 -- start start 2024-09-18T12:33:06.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.167+0000 7f6a3219d640 10 monclient: init 2024-09-18T12:33:06.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:06.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:06.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a2c134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a14004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:06.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2c1353c0 0x7f6a2c11ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2c1353c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:06.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a2c11efa0 0x7f6a2c11f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a2c11efa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:06.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:06.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c099ea0 con 0x7f6a2c1353c0 2024-09-18T12:33:06.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:06.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c06d7b0 con 0x7f6a2c11efa0 2024-09-18T12:33:06.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:06.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c141e60 con 0x7f6a14004780 2024-09-18T12:33:06.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:06.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a3219d640 10 monclient(hunting): authenticate will time out at 1221.737115s 2024-09-18T12:33:06.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2bfff640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a2c11efa0 msgr2=0x7f6a2c11f3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:06.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a2c11efa0 0x7f6a2c11f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:06.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a2c134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:06.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2c1353c0 0x7f6a2c11ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:06.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a2c134e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:55578/0 (socket says 172.21.15.144:55578) 2024-09-18T12:33:06.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 -- 172.21.15.144:0/2759105070 learned_addr learned my addr 172.21.15.144:0/2759105070 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:06.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): get_auth_request con 0x7f6a14004780 auth_method 0 2024-09-18T12:33:06.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:06.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:06.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:06.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 10 monclient(hunting): get_auth_request con 0x7f6a2c1353c0 auth_method 0 2024-09-18T12:33:06.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:06.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:06.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:06.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:06.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:06.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:06.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:06.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:06.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:06.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient(hunting): handle_auth_done global_id 65078 payload 824 2024-09-18T12:33:06.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 -- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a2c11efa0 msgr2=0x7f6a2c11f3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:06.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a2c11efa0 0x7f6a2c11f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 -- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2c1353c0 msgr2=0x7f6a2c11ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:06.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2c1353c0 0x7f6a2c11ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:06.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:06.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:33:06.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:06.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:06.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.168+0000 7f6a2b7fe640 1 -- 172.21.15.144:0/2759105070 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c122280 con 0x7f6a14004780 2024-09-18T12:33:06.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.170+0000 7f6a2b7fe640 1 -- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 msgr2=0x7f6a2c134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:06.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.170+0000 7f6a2b7fe640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a2c134e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00b630 tx=0x7f6a1c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:06.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a2b7fe640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a2c134e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00b630 tx=0x7f6a1c0091a0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f6a1c015070 mon_map magic: 0 v1 2024-09-18T12:33:06.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c015070 con 0x7f6a14004780 2024-09-18T12:33:06.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:06.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:06.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: dump: 2024-09-18T12:33:06.208 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:06.208 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:06.208 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.208 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.208 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:06.209 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:06.209 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:06.209 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:06.209 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:06.209 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:06.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:06.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:06.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c003d90 con 0x7f6a14004780 2024-09-18T12:33:06.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a3219d640 5 monclient: authenticate success, global_id 65078 2024-09-18T12:33:06.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:06.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a3219d640 10 monclient: _renew_subs 2024-09-18T12:33:06.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a3219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:06.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c120470 con 0x7f6a14004780 2024-09-18T12:33:06.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a29ffb640 4 set_mon_vals no callback set 2024-09-18T12:33:06.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a3219d640 10 monclient: _renew_subs 2024-09-18T12:33:06.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a3219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:06.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c1363b0 con 0x7f6a14004780 2024-09-18T12:33:06.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c006380 con 0x7f6a14004780 2024-09-18T12:33:06.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:06.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:06.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 10 monclient: dump: 2024-09-18T12:33:06.212 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:06.213 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:06.213 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.213 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:06.213 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:06.213 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:06.213 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:06.214 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:06.214 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:06.214 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:06.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.171+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6a1c010440 con 0x7f6a14004780 2024-09-18T12:33:06.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.172+0000 7f6a137fe640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a18038490 0x7f6a1803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.172+0000 7f6a3219d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.172+0000 7f6a3219d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.172+0000 7f6a3219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.172+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2c1224c0 con 0x7f6a14004780 2024-09-18T12:33:06.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.172+0000 7f6a2affd640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a18038490 0x7f6a1803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:06.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.172+0000 7f6a2affd640 10 monclient: get_auth_request con 0x7f6a18038490 auth_method 0 2024-09-18T12:33:06.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.174+0000 7f6a2affd640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a18038490 0x7f6a1803a940 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6a20001ed0 tx=0x7f6a20034000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:06.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.177+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(207..207 src has 13..207) v4 ==== 10818+0+0 (secure 0 0 0) 0x7f6a1c04cc40 con 0x7f6a14004780 2024-09-18T12:33:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.999+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6a2c1224c0 con 0x7f6a14004780 2024-09-18T12:33:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.999+0000 7f6a137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:07.000 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:06.999+0000 7f6a137fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:07.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:07.157+0000 7f6a3219d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:07.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:07.157+0000 7f6a3219d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:07.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:07.157+0000 7f6a3219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:07.158 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:07.157+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6a2c1224c0 con 0x7f6a14004780 2024-09-18T12:33:08.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.084+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v208) v1 ==== 168+0+0 (secure 0 0 0) 0x7f6a2c1224c0 con 0x7f6a14004780 2024-09-18T12:33:08.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.084+0000 7f6a137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:08.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.084+0000 7f6a137fe640 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-18T12:33:08.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.182+0000 7f6a3219d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:08.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.182+0000 7f6a3219d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:08.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.182+0000 7f6a3219d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:08.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.182+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6a2c1224c0 con 0x7f6a14004780 2024-09-18T12:33:08.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.184+0000 7f6a137fe640 1 -- 172.21.15.144:0/2759105070 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v208) v1 ==== 147+0+0 (secure 0 0 0) 0x7f6a2c1224c0 con 0x7f6a14004780 2024-09-18T12:33:08.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.184+0000 7f6a137fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:08.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.184+0000 7f6a137fe640 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-18T12:33:08.185 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h15' already exists 2024-09-18T12:33:08.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a18038490 msgr2=0x7f6a1803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:08.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a18038490 0x7f6a1803a940 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6a20001ed0 tx=0x7f6a20034000 comp rx=0 tx=0).stop 2024-09-18T12:33:08.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 10 monclient: shutdown 2024-09-18T12:33:08.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:08.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 msgr2=0x7f6a2c134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:08.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a2c134e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00b630 tx=0x7f6a1c0091a0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 shutdown_connections 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a18038490 0x7f6a1803a940 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a2c11efa0 0x7f6a2c11f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a2c1353c0 0x7f6a2c11ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 --2- 172.21.15.144:0/2759105070 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a14004780 0x7f6a2c134e80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 >> 172.21.15.144:0/2759105070 conn(0x7f6a2c0a3b30 msgr2=0x7f6a2c09e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 shutdown_connections 2024-09-18T12:33:08.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.186+0000 7f6a3219d640 1 -- 172.21.15.144:0/2759105070 wait complete. 2024-09-18T12:33:08.197 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:08.197 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-18T12:33:08.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.272+0000 7f17c559d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:08.273 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.272+0000 7f17c559d640 10 monclient: build_initial_monmap 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.272+0000 7f17c559d640 10 monclient: monmap: 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:08.273635+0000 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:08.273635+0000 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:08.274 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 1 Processor -- start 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 1 -- start start 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient: init 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17c0131bd0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f17c0069e10 0x7f17c0131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient(hunting): picked mon.noname-c con 0x7f17c0069e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:08.277 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17c0136d60 0x7f17c0139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient(hunting): picked mon.noname-a con 0x7f17c0136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c00a06d0 con 0x7f17c0131bd0 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c0071780 con 0x7f17c0136d60 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c006a210 con 0x7f17c0069e10 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17c559d640 10 monclient(hunting): authenticate will time out at 1223.842812s 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17be7fc640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f17c0069e10 msgr2=0x7f17c0131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.273+0000 7f17be7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f17c0069e10 0x7f17c0131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17c0136d60 0x7f17c0139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17beffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17c0131bd0 0x7f17c009ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:51418/0 (socket says 172.21.15.144:51418) 2024-09-18T12:33:08.278 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17c0136d60 0x7f17c0139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58834/0 (socket says 172.21.15.144:58834) 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17beffd640 1 -- 172.21.15.144:0/2130106966 learned_addr learned my addr 172.21.15.144:0/2130106966 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): get_auth_request con 0x7f17c0136d60 auth_method 0 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_done global_id 104247 payload 274 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2130106966 >> v2:172.21.15.144:3300/0 conn(0x7f17c0069e10 msgr2=0x7f17c0131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 1 --2- 172.21.15.144:0/2130106966 >> v2:172.21.15.144:3300/0 conn(0x7f17c0069e10 0x7f17c0131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2130106966 >> v1:172.21.15.116:6789/0 conn(0x7f17c0131bd0 legacy=0x7f17c009ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:08.279 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:08.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:08.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:08.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:08.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.274+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2130106966 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c00a0010 con 0x7f17c0136d60 2024-09-18T12:33:08.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2130106966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 msgr2=0x7f17c0139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:08.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bf7fe640 1 --2- 172.21.15.144:0/2130106966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0139140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f17c0068d90 tx=0x7f17b8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0d47f307abe982 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:08.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 1 -- 172.21.15.144:0/2130106966 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8007840 con 0x7f17c0136d60 2024-09-18T12:33:08.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:08.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: dump: 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:08.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 1 -- 172.21.15.144:0/2130106966 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b802d050 con 0x7f17c0136d60 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 1 -- 172.21.15.144:0/2130106966 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8007c10 con 0x7f17c0136d60 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17bdffb640 10 monclient: dump: 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:08.288 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17c559d640 5 monclient: authenticate success, global_id 104247 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17c559d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.277+0000 7f17c559d640 4 set_mon_vals no callback set 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 10 monclient: shutdown 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 -- 172.21.15.144:0/2130106966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 msgr2=0x7f17c0139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 --2- 172.21.15.144:0/2130106966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0139140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f17c0068d90 tx=0x7f17b8009780 comp rx=0 tx=0).stop 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 -- 172.21.15.144:0/2130106966 shutdown_connections 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 --2- 172.21.15.144:0/2130106966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0139140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 --2- 172.21.15.144:0/2130106966 >> v2:172.21.15.144:3300/0 conn(0x7f17c0069e10 0x7f17c0131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 -- 172.21.15.144:0/2130106966 >> 172.21.15.144:0/2130106966 conn(0x7f17c00713b0 msgr2=0x7f17c00a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 -- 172.21.15.144:0/2130106966 shutdown_connections 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 1 -- 172.21.15.144:0/2130106966 wait complete. 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 10 monclient: build_initial_monmap 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.278+0000 7f17c559d640 10 monclient: monmap: 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:08.290 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 Processor -- start 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 -- start start 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient: init 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c0069e10 0x7f17c0141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): picked mon.noname-b con 0x7f17c0069e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17c0131bd0 0x7f17c0141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): picked mon.noname-c con 0x7f17c0131bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0099620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): picked mon.noname-a con 0x7f17c0136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:08.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c009bec0 con 0x7f17c0136d60 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c009e970 con 0x7f17c0069e10 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c011cc20 con 0x7f17c0131bd0 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17c559d640 10 monclient(hunting): authenticate will time out at 1223.848691s 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17beffd640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c0069e10 msgr2=0x7f17c0141400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17beffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c0069e10 0x7f17c0141400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0099620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17be7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17c0131bd0 0x7f17c0141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0099620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58846/0 (socket says 172.21.15.144:58846) 2024-09-18T12:33:08.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2287236380 learned_addr learned my addr 172.21.15.144:0/2287236380 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17bf7fe640 10 monclient(hunting): get_auth_request con 0x7f17c0136d60 auth_method 0 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17be7fc640 10 monclient(hunting): get_auth_request con 0x7f17c0131bd0 auth_method 0 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.279+0000 7f17be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:08.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 10 monclient(hunting): handle_auth_done global_id 104250 payload 824 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17c0131bd0 msgr2=0x7f17c0141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17c0131bd0 0x7f17c0141940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c0069e10 msgr2=0x7f17c0141400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c0069e10 0x7f17c0141400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2287236380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8009a20 con 0x7f17c0136d60 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.280+0000 7f17be7fc640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17c0131bd0 0x7f17c0141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:08.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17bf7fe640 1 -- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 msgr2=0x7f17c0099620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17bf7fe640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0099620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f17b80099f0 tx=0x7f17b8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8031da0 con 0x7f17c0136d60 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: dump: 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:08.295 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17c559d640 5 monclient: authenticate success, global_id 104250 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b8040070 con 0x7f17c0136d60 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17c559d640 10 monclient: _renew_subs 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17c559d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c009b9f0 con 0x7f17c0136d60 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8045640 con 0x7f17c0136d60 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:08.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17a6ffd640 10 monclient: dump: 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17c559d640 10 monclient: _renew_subs 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17c559d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c011c940 con 0x7f17c0136d60 2024-09-18T12:33:08.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.282+0000 7f17bd7fa640 4 set_mon_vals no callback set 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.283+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f17b8003020 con 0x7f17c0136d60 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.283+0000 7f17c559d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.283+0000 7f17c559d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.283+0000 7f17c559d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.283+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c0063750 con 0x7f17c0136d60 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.283+0000 7f17a6ffd640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f17ac038440 0x7f17ac03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.283+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(208..208 src has 13..208) v4 ==== 11031+0+0 (secure 0 0 0) 0x7f17b806ee20 con 0x7f17c0136d60 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.284+0000 7f17beffd640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f17ac038440 0x7f17ac03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:08.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.284+0000 7f17beffd640 10 monclient: get_auth_request con 0x7f17ac038440 auth_method 0 2024-09-18T12:33:08.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.286+0000 7f17beffd640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f17ac038440 0x7f17ac03a8f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f17b00047d0 tx=0x7f17b0007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:08.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.288+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f17c0063750 con 0x7f17c0136d60 2024-09-18T12:33:08.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.288+0000 7f17a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:08.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.288+0000 7f17a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:08.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.447+0000 7f17c559d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:08.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.447+0000 7f17c559d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:08.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.447+0000 7f17c559d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:08.449 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:08.447+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f17c0063750 con 0x7f17c0136d60 2024-09-18T12:33:09.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.093+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v209) v1 ==== 168+0+0 (secure 0 0 0) 0x7f17c0063750 con 0x7f17c0136d60 2024-09-18T12:33:09.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.093+0000 7f17a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:09.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.093+0000 7f17a6ffd640 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-18T12:33:09.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.193+0000 7f17c559d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:09.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.193+0000 7f17c559d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:09.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.193+0000 7f17c559d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:09.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.193+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f17c0063750 con 0x7f17c0136d60 2024-09-18T12:33:09.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.194+0000 7f17a6ffd640 1 -- 172.21.15.144:0/2287236380 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v209) v1 ==== 147+0+0 (secure 0 0 0) 0x7f17c0063750 con 0x7f17c0136d60 2024-09-18T12:33:09.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.194+0000 7f17a6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:09.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.194+0000 7f17a6ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-18T12:33:09.195 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h16' already exists 2024-09-18T12:33:09.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f17ac038440 msgr2=0x7f17ac03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:09.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f17ac038440 0x7f17ac03a8f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f17b00047d0 tx=0x7f17b0007b80 comp rx=0 tx=0).stop 2024-09-18T12:33:09.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 10 monclient: shutdown 2024-09-18T12:33:09.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:09.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 msgr2=0x7f17c0099620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0099620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f17b80099f0 tx=0x7f17b8037000 comp rx=0 tx=0).stop 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 shutdown_connections 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f17ac038440 0x7f17ac03a8f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0136d60 0x7f17c0099620 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f17c0131bd0 0x7f17c0141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 --2- 172.21.15.144:0/2287236380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17c0069e10 0x7f17c0141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 >> 172.21.15.144:0/2287236380 conn(0x7f17c00713b0 msgr2=0x7f17c009eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 shutdown_connections 2024-09-18T12:33:09.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.196+0000 7f17c559d640 1 -- 172.21.15.144:0/2287236380 wait complete. 2024-09-18T12:33:09.207 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:09.207 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.281+0000 7f15344e8640 10 monclient: get_monmap_and_config 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.281+0000 7f15344e8640 10 monclient: build_initial_monmap 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.281+0000 7f15344e8640 10 monclient: monmap: 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:09.282709+0000 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:09.282709+0000 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:09.283 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:09.284 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:09.284 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:09.284 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:09.284 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:09.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.282+0000 7f15344e8640 1 Processor -- start 2024-09-18T12:33:09.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.282+0000 7f15344e8640 1 -- start start 2024-09-18T12:33:09.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.282+0000 7f15344e8640 10 monclient: init 2024-09-18T12:33:09.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f152c09c2b0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f152c09c680 0x7f152c09aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient(hunting): picked mon.noname-a con 0x7f152c09c680 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c134780 0x7f152c136b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient(hunting): picked mon.noname-b con 0x7f152c134780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c09bb40 con 0x7f152c09c2b0 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152c099f60 con 0x7f152c09c680 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152c09f700 con 0x7f152c134780 2024-09-18T12:33:09.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f15344e8640 10 monclient(hunting): authenticate will time out at 1224.852135s 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532284640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f152c09c2b0 legacy=0x7f152c09a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f152c09c680 0x7f152c09aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c134780 0x7f152c136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f152c09c680 0x7f152c09aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58850/0 (socket says 172.21.15.144:58850) 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 -- 172.21.15.144:0/3822420030 learned_addr learned my addr 172.21.15.144:0/3822420030 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c134780 0x7f152c136b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:36480/0 (socket says 172.21.15.144:36480) 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): get_auth_request con 0x7f152c09c680 auth_method 0 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 10 monclient(hunting): get_auth_request con 0x7f152c134780 auth_method 0 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:09.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient(hunting): handle_auth_done global_id 104253 payload 274 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 -- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c134780 msgr2=0x7f152c136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 --2- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c134780 0x7f152c136b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 -- 172.21.15.144:0/3822420030 >> v1:172.21.15.144:6789/0 conn(0x7f152c09c2b0 legacy=0x7f152c09a7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient: _finish_hunting 0 2024-09-18T12:33:09.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 monclient: found mon.noname-a 2024-09-18T12:33:09.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:09.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:09.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1531a83640 1 -- 172.21.15.144:0/3822420030 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c09b4f0 con 0x7f152c09c680 2024-09-18T12:33:09.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.283+0000 7f1532a85640 1 --2- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c134780 0x7f152c136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:09.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531a83640 1 -- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c680 msgr2=0x7f152c09aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:09.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531a83640 1 --2- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c680 0x7f152c09aeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1520009910 tx=0x7f15200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c308f21df491a94 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:09.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 1 -- 172.21.15.144:0/3822420030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152002ae90 con 0x7f152c09c680 2024-09-18T12:33:09.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:09.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:09.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: dump: 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: _finish_auth 0 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:09.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 1 -- 172.21.15.144:0/3822420030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1520032440 con 0x7f152c09c680 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f15344e8640 5 monclient: authenticate success, global_id 104253 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f15344e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f15344e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 1 -- 172.21.15.144:0/3822420030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520032820 con 0x7f152c09c680 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f15344e8640 4 set_mon_vals no callback set 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.286+0000 7f1531282640 10 monclient: dump: 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.297 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 10 monclient: shutdown 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 -- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c680 msgr2=0x7f152c09aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 --2- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c680 0x7f152c09aeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1520009910 tx=0x7f15200091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 -- 172.21.15.144:0/3822420030 shutdown_connections 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 --2- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c134780 0x7f152c136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 --2- 172.21.15.144:0/3822420030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c680 0x7f152c09aeb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 -- 172.21.15.144:0/3822420030 >> 172.21.15.144:0/3822420030 conn(0x7f152c0713b0 msgr2=0x7f152c0a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 -- 172.21.15.144:0/3822420030 shutdown_connections 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 1 -- 172.21.15.144:0/3822420030 wait complete. 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 10 monclient: build_initial_monmap 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.287+0000 7f15344e8640 10 monclient: monmap: 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:09.299 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 Processor -- start 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 -- start start 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient: init 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c2b0 0x7f152c098190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): picked mon.noname-a con 0x7f152c09c2b0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 0x7f152c0986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): picked mon.noname-c con 0x7f152c09c680 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f152c134780 0x7f152c098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:09.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): picked mon.noname-b con 0x7f152c134780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c11f7b0 con 0x7f152c09c2b0 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152c11fa00 con 0x7f152c134780 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152c06e9e0 con 0x7f152c09c680 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f15344e8640 10 monclient(hunting): authenticate will time out at 1224.857678s 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1532a85640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f152c134780 msgr2=0x7f152c098c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1532a85640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f152c134780 0x7f152c098c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 0x7f152c0986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1532284640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c2b0 0x7f152c098190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:09.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 0x7f152c0986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:36488/0 (socket says 172.21.15.144:36488) 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 1 -- 172.21.15.144:0/3912045047 learned_addr learned my addr 172.21.15.144:0/3912045047 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 10 monclient(hunting): get_auth_request con 0x7f152c09c680 auth_method 0 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1532284640 10 monclient(hunting): get_auth_request con 0x7f152c09c2b0 auth_method 0 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1532284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1532284640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1532284640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.288+0000 7f1531a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:09.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1532284640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1532284640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1532284640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 10 monclient(hunting): handle_auth_done global_id 65087 payload 824 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 1 -- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f152c134780 msgr2=0x7f152c098c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f152c134780 0x7f152c098c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 1 -- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c2b0 msgr2=0x7f152c098190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c2b0 0x7f152c098190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 10 monclient: _finish_hunting 0 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 1 monclient: found mon.noname-c 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1531a83640 1 -- 172.21.15.144:0/3912045047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1524009a20 con 0x7f152c09c680 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.289+0000 7f1532284640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c2b0 0x7f152c098190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f1531a83640 1 -- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 msgr2=0x7f152c0986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:09.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f1531a83640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 0x7f152c0986d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1520009970 tx=0x7f1520032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520000d10 con 0x7f152c09c680 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: dump: 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:09.304 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f15344e8640 5 monclient: authenticate success, global_id 65087 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f15344e8640 10 monclient: _renew_subs 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f15344e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1520009450 con 0x7f152c09c680 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f15344e8640 10 monclient: _renew_subs 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f15344e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152c11de90 con 0x7f152c09c680 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152002ae90 con 0x7f152c09c680 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:09.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520031e80 con 0x7f152c09c680 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f152a7fc640 10 monclient: dump: 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:09.306 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.291+0000 7f1530a81640 4 set_mon_vals no callback set 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1520039040 con 0x7f152c09c680 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f15344e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f15344e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f15344e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c005010 con 0x7f152c09c680 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f152a7fc640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1518038440 0x7f151803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(209..209 src has 13..209) v4 ==== 11500+0+0 (secure 0 0 0) 0x7f152006cf20 con 0x7f152c09c680 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.292+0000 7f1532284640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1518038440 0x7f151803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:09.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.293+0000 7f1532284640 10 monclient: get_auth_request con 0x7f1518038440 auth_method 0 2024-09-18T12:33:09.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.295+0000 7f1532284640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1518038440 0x7f151803a8f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f151c003940 tx=0x7f151c0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:09.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.297+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f152002f4d0 con 0x7f152c09c680 2024-09-18T12:33:09.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.297+0000 7f152a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:09.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.297+0000 7f152a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:09.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.454+0000 7f15344e8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:09.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.454+0000 7f15344e8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:09.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.454+0000 7f15344e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:09.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:09.454+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f150c005840 con 0x7f152c09c680 2024-09-18T12:33:10.101 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.099+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v210) v1 ==== 168+0+0 (secure 0 0 0) 0x7f15200370b0 con 0x7f152c09c680 2024-09-18T12:33:10.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.099+0000 7f152a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:10.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.099+0000 7f152a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-18T12:33:10.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.194+0000 7f15344e8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:10.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.194+0000 7f15344e8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:10.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.194+0000 7f15344e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:10.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.194+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f150c0052c0 con 0x7f152c09c680 2024-09-18T12:33:10.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.196+0000 7f152a7fc640 1 -- 172.21.15.144:0/3912045047 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v210) v1 ==== 147+0+0 (secure 0 0 0) 0x7f1520042020 con 0x7f152c09c680 2024-09-18T12:33:10.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.196+0000 7f152a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:10.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.196+0000 7f152a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-18T12:33:10.198 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h17' already exists 2024-09-18T12:33:10.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1518038440 msgr2=0x7f151803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1518038440 0x7f151803a8f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f151c003940 tx=0x7f151c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 10 monclient: shutdown 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 msgr2=0x7f152c0986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 0x7f152c0986d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1520009970 tx=0x7f1520032000 comp rx=0 tx=0).stop 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 shutdown_connections 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1518038440 0x7f151803a8f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f152c134780 0x7f152c098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f152c09c680 0x7f152c0986d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 --2- 172.21.15.144:0/3912045047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f152c09c2b0 0x7f152c098190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 >> 172.21.15.144:0/3912045047 conn(0x7f152c0713b0 msgr2=0x7f152c135ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:10.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 shutdown_connections 2024-09-18T12:33:10.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.198+0000 7f15344e8640 1 -- 172.21.15.144:0/3912045047 wait complete. 2024-09-18T12:33:10.209 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:10.209 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-18T12:33:10.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.282+0000 7f72a6a89640 10 monclient: get_monmap_and_config 2024-09-18T12:33:10.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.282+0000 7f72a6a89640 10 monclient: build_initial_monmap 2024-09-18T12:33:10.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.282+0000 7f72a6a89640 10 monclient: monmap: 2024-09-18T12:33:10.284 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:10.284255+0000 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:10.284255+0000 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:10.285 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:10.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 1 Processor -- start 2024-09-18T12:33:10.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 1 -- start start 2024-09-18T12:33:10.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient: init 2024-09-18T12:33:10.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:10.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T12:33:10.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72a0131ac0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a009ff50 0x7f72a0098150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient(hunting): picked mon.noname-b con 0x7f72a009ff50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72a0098690 0x7f72a0138e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient(hunting): picked mon.noname-a con 0x7f72a0098690 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a00a1330 con 0x7f72a0131ac0 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a01325a0 con 0x7f72a0098690 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a01326e0 con 0x7f72a009ff50 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a6a89640 10 monclient(hunting): authenticate will time out at 1225.853585s 2024-09-18T12:33:10.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f729ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a009ff50 0x7f72a0098150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a5026640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72a0098690 0x7f72a0138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f729ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a009ff50 0x7f72a0098150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:36502/0 (socket says 172.21.15.144:36502) 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a5026640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72a0098690 0x7f72a0138e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58870/0 (socket says 172.21.15.144:58870) 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f729ffff640 1 -- 172.21.15.144:0/3692612413 learned_addr learned my addr 172.21.15.144:0/3692612413 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a4825640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f72a0131ac0 0x7f72a0131e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:51424/0 (socket says 172.21.15.144:51424) 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f729ffff640 10 monclient(hunting): get_auth_request con 0x7f72a009ff50 auth_method 0 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f729ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f729ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f729ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a5026640 10 monclient(hunting): get_auth_request con 0x7f72a0098690 auth_method 0 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a5026640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a5026640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:10.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.283+0000 7f72a5026640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f729ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f729ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f729ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 10 monclient(hunting): handle_auth_done global_id 104259 payload 274 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 1 -- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a009ff50 msgr2=0x7f72a0098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 1 --2- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a009ff50 0x7f72a0098150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 1 -- 172.21.15.144:0/3692612413 >> v1:172.21.15.116:6789/0 conn(0x7f72a0131ac0 legacy=0x7f72a0131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 10 monclient: _finish_hunting 0 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 1 monclient: found mon.noname-a 2024-09-18T12:33:10.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:10.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:10.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f72a5026640 1 -- 172.21.15.144:0/3692612413 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0098ab0 con 0x7f72a0098690 2024-09-18T12:33:10.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.284+0000 7f729ffff640 1 --2- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a009ff50 0x7f72a0098150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:10.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a5026640 1 -- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 msgr2=0x7f72a0138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a5026640 1 --2- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138e00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7298009e60 tx=0x7f7298009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3143427d2877e4f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 1 -- 172.21.15.144:0/3692612413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298007760 con 0x7f72a0098690 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: dump: 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:10.298 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 1 -- 172.21.15.144:0/3692612413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729802d050 con 0x7f72a0098690 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 5 monclient: authenticate success, global_id 104259 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 4 set_mon_vals no callback set 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 1 -- 172.21.15.144:0/3692612413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298007af0 con 0x7f72a0098690 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:10.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f729f7fe640 10 monclient: dump: 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 10 monclient: shutdown 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:10.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 -- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 msgr2=0x7f72a0138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138e00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7298009e60 tx=0x7f7298009780 comp rx=0 tx=0).stop 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 -- 172.21.15.144:0/3692612413 shutdown_connections 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138e00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3692612413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a009ff50 0x7f72a0098150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 -- 172.21.15.144:0/3692612413 >> 172.21.15.144:0/3692612413 conn(0x7f72a00a5190 msgr2=0x7f72a00a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 -- 172.21.15.144:0/3692612413 shutdown_connections 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.287+0000 7f72a6a89640 1 -- 172.21.15.144:0/3692612413 wait complete. 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.288+0000 7f72a6a89640 10 monclient: build_initial_monmap 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.288+0000 7f72a6a89640 10 monclient: monmap: 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:10.301 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.288+0000 7f72a6a89640 1 Processor -- start 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.288+0000 7f72a6a89640 1 -- start start 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.288+0000 7f72a6a89640 10 monclient: init 2024-09-18T12:33:10.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): picked mon.noname-a con 0x7f72a0098690 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72a009ff50 0x7f72a01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): picked mon.noname-b con 0x7f72a009ff50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a0131ac0 0x7f72a0134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): picked mon.noname-c con 0x7f72a0131ac0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0135610 con 0x7f72a0098690 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0098e70 con 0x7f72a009ff50 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:10.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a01361e0 con 0x7f72a0131ac0 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a6a89640 10 monclient(hunting): authenticate will time out at 1225.859367s 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f729ffff640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72a009ff50 msgr2=0x7f72a01340c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f729ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72a009ff50 0x7f72a01340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58884/0 (socket says 172.21.15.144:58884) 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 1 -- 172.21.15.144:0/3339408935 learned_addr learned my addr 172.21.15.144:0/3339408935 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): get_auth_request con 0x7f72a0098690 auth_method 0 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:10.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a0131ac0 0x7f72a0134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 10 monclient(hunting): get_auth_request con 0x7f72a0131ac0 auth_method 0 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a5026640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 10 monclient(hunting): handle_auth_done global_id 104262 payload 824 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 1 -- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a0131ac0 msgr2=0x7f72a0134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:10.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.289+0000 7f72a4825640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a0131ac0 0x7f72a0134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.290+0000 7f72a4825640 1 -- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72a009ff50 msgr2=0x7f72a01340c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.290+0000 7f72a4825640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72a009ff50 0x7f72a01340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.290+0000 7f72a4825640 10 monclient: _finish_hunting 0 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.290+0000 7f72a4825640 1 monclient: found mon.noname-a 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.290+0000 7f72a4825640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.290+0000 7f72a4825640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.290+0000 7f72a4825640 1 -- 172.21.15.144:0/3339408935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c009450 con 0x7f72a0098690 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a4825640 1 -- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 msgr2=0x7f72a0138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a4825640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f729400c5b0 tx=0x7f7294007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940074f0 con 0x7f72a0098690 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: dump: 2024-09-18T12:33:10.306 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7294007670 con 0x7f72a0098690 2024-09-18T12:33:10.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a6a89640 5 monclient: authenticate success, global_id 104262 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a6a89640 10 monclient: _renew_subs 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a6a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7298009a20 con 0x7f72a0098690 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294005a00 con 0x7f72a0098690 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729cff9640 10 monclient: dump: 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.308 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a6a89640 10 monclient: _renew_subs 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a6a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f729e7fc640 4 set_mon_vals no callback set 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.292+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a0135cd0 con 0x7f72a0098690 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f729401b3b0 con 0x7f72a0098690 2024-09-18T12:33:10.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f72a6a89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f72a6a89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f72a6a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a004bf60 con 0x7f72a0098690 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f729cff9640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7284038440 0x7f728403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(210..210 src has 13..210) v4 ==== 11713+0+0 (secure 0 0 0) 0x7f729404c800 con 0x7f72a0098690 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f729ffff640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7284038440 0x7f728403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.293+0000 7f729ffff640 10 monclient: get_auth_request con 0x7f7284038440 auth_method 0 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.296+0000 7f729ffff640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7284038440 0x7f728403a8f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f72a0138b70 tx=0x7f728c0091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:10.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.298+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7294018660 con 0x7f72a0098690 2024-09-18T12:33:10.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.298+0000 7f729cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:10.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.298+0000 7f729cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:10.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.457+0000 7f72a6a89640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:10.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.457+0000 7f72a6a89640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:10.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.457+0000 7f72a6a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:10.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.457+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f72a0064a70 con 0x7f72a0098690 2024-09-18T12:33:10.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.981+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v211) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7294017020 con 0x7f72a0098690 2024-09-18T12:33:10.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.981+0000 7f729cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:10.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:10.981+0000 7f729cff9640 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-18T12:33:11.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.078+0000 7f72a6a89640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:11.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.078+0000 7f72a6a89640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:11.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.078+0000 7f72a6a89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:11.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.078+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f72a01368a0 con 0x7f72a0098690 2024-09-18T12:33:11.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.079+0000 7f729cff9640 1 -- 172.21.15.144:0/3339408935 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v211) v1 ==== 147+0+0 (secure 0 0 0) 0x7f7294017020 con 0x7f72a0098690 2024-09-18T12:33:11.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.079+0000 7f729cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:11.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.079+0000 7f729cff9640 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-18T12:33:11.082 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h18' already exists 2024-09-18T12:33:11.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7284038440 msgr2=0x7f728403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:11.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7284038440 0x7f728403a8f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f72a0138b70 tx=0x7f728c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 10 monclient: shutdown 2024-09-18T12:33:11.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 msgr2=0x7f72a0138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f729400c5b0 tx=0x7f7294007f00 comp rx=0 tx=0).stop 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 shutdown_connections 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7284038440 0x7f728403a8f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72a0131ac0 0x7f72a0134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72a009ff50 0x7f72a01340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 --2- 172.21.15.144:0/3339408935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0098690 0x7f72a0138500 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 >> 172.21.15.144:0/3339408935 conn(0x7f72a00a5190 msgr2=0x7f72a009b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 shutdown_connections 2024-09-18T12:33:11.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.081+0000 7f72a6a89640 1 -- 172.21.15.144:0/3339408935 wait complete. 2024-09-18T12:33:11.092 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:11.092 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-18T12:33:11.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.164+0000 7f9616eef640 10 monclient: get_monmap_and_config 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.164+0000 7f9616eef640 10 monclient: build_initial_monmap 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.164+0000 7f9616eef640 10 monclient: monmap: 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:11.166744+0000 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:11.166744+0000 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:11.167 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:11.168 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:11.168 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.165+0000 7f9616eef640 1 Processor -- start 2024-09-18T12:33:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.165+0000 7f9616eef640 1 -- start start 2024-09-18T12:33:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.165+0000 7f9616eef640 10 monclient: init 2024-09-18T12:33:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:11.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9610069bd0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f961009cf60 0x7f96100981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient(hunting): picked mon.noname-c con 0x7f961009cf60 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96100986f0 0x7f9610138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient(hunting): picked mon.noname-a con 0x7f96100986f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961009cd00 con 0x7f9610069bd0 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9610132640 con 0x7f96100986f0 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961009d6d0 con 0x7f961009cf60 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9616eef640 10 monclient(hunting): authenticate will time out at 1226.736106s 2024-09-18T12:33:11.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9614c8b640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f9610069bd0 legacy=0x7f9610131f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9607fff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f961009cf60 msgr2=0x7f96100981b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f9607fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f961009cf60 0x7f96100981b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96100986f0 0x7f9610138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96100986f0 0x7f9610138ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58908/0 (socket says 172.21.15.144:58908) 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 -- 172.21.15.144:0/3465995137 learned_addr learned my addr 172.21.15.144:0/3465995137 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): get_auth_request con 0x7f96100986f0 auth_method 0 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:11.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient(hunting): handle_auth_done global_id 104265 payload 274 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 -- 172.21.15.144:0/3465995137 >> v2:172.21.15.144:3300/0 conn(0x7f961009cf60 msgr2=0x7f96100981b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 --2- 172.21.15.144:0/3465995137 >> v2:172.21.15.144:3300/0 conn(0x7f961009cf60 0x7f96100981b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 -- 172.21.15.144:0/3465995137 >> v1:172.21.15.144:6789/0 conn(0x7f9610069bd0 legacy=0x7f9610131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 monclient: found mon.noname-a 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:11.173 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.166+0000 7f961548c640 1 -- 172.21.15.144:0/3465995137 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961009e320 con 0x7f96100986f0 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.169+0000 7f961548c640 1 -- 172.21.15.144:0/3465995137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96100986f0 msgr2=0x7f9610138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.169+0000 7f961548c640 1 --2- 172.21.15.144:0/3465995137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96100986f0 0x7f9610138ea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f961009e950 tx=0x7f9608009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3cdd84c6f32c34d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.169+0000 7f96077fe640 1 -- 172.21.15.144:0/3465995137 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608007610 con 0x7f96100986f0 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.169+0000 7f96077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.169+0000 7f96077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 10 monclient: dump: 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:11.179 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 1 -- 172.21.15.144:0/3465995137 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960802d050 con 0x7f96100986f0 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 5 monclient: authenticate success, global_id 104265 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:11.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 1 -- 172.21.15.144:0/3465995137 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96080079a0 con 0x7f96100986f0 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 4 set_mon_vals no callback set 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f96077fe640 10 monclient: dump: 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:11.181 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 10 monclient: shutdown 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 -- 172.21.15.144:0/3465995137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96100986f0 msgr2=0x7f9610138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 --2- 172.21.15.144:0/3465995137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96100986f0 0x7f9610138ea0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f961009e950 tx=0x7f9608009840 comp rx=0 tx=0).stop 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 -- 172.21.15.144:0/3465995137 shutdown_connections 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 --2- 172.21.15.144:0/3465995137 >> v2:172.21.15.144:3300/0 conn(0x7f961009cf60 0x7f96100981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 --2- 172.21.15.144:0/3465995137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96100986f0 0x7f9610138ea0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 -- 172.21.15.144:0/3465995137 >> 172.21.15.144:0/3465995137 conn(0x7f961009e9f0 msgr2=0x7f961009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 -- 172.21.15.144:0/3465995137 shutdown_connections 2024-09-18T12:33:11.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.170+0000 7f9616eef640 1 -- 172.21.15.144:0/3465995137 wait complete. 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 10 monclient: build_initial_monmap 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 10 monclient: monmap: 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:11.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 1 Processor -- start 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 1 -- start start 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 10 monclient: init 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 0x7f96101353d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.171+0000 7f9616eef640 10 monclient(hunting): picked mon.noname-a con 0x7f9610069bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96100986f0 0x7f9610135910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 10 monclient(hunting): picked mon.noname-b con 0x7f96100986f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f961009cf60 0x7f9610135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 10 monclient(hunting): picked mon.noname-c con 0x7f961009cf60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96100a01d0 con 0x7f9610069bd0 2024-09-18T12:33:11.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961009d8a0 con 0x7f96100986f0 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9610138a20 con 0x7f961009cf60 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9616eef640 10 monclient(hunting): authenticate will time out at 1226.742051s 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9607fff640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96100986f0 msgr2=0x7f9610135910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9607fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96100986f0 0x7f9610135910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 0x7f96101353d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f961009cf60 0x7f9610135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 0x7f96101353d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58920/0 (socket says 172.21.15.144:58920) 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f961009cf60 0x7f9610135e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:36538/0 (socket says 172.21.15.144:36538) 2024-09-18T12:33:11.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 -- 172.21.15.144:0/1376169426 learned_addr learned my addr 172.21.15.144:0/1376169426 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): get_auth_request con 0x7f9610069bd0 auth_method 0 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 10 monclient(hunting): get_auth_request con 0x7f961009cf60 auth_method 0 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:11.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f961548c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient(hunting): handle_auth_done global_id 104268 payload 824 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 -- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f961009cf60 msgr2=0x7f9610135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f961009cf60 0x7f9610135e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 -- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96100986f0 msgr2=0x7f9610135910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96100986f0 0x7f9610135910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient: _finish_hunting 0 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 monclient: found mon.noname-a 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.172+0000 7f9614c8b640 1 -- 172.21.15.144:0/1376169426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9608009ae0 con 0x7f9610069bd0 2024-09-18T12:33:11.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.174+0000 7f9614c8b640 1 -- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 msgr2=0x7f96101353d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.174+0000 7f9614c8b640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 0x7f96101353d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f960c00b600 tx=0x7f960c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c015070 con 0x7f9610069bd0 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: dump: 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:11.188 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c003d90 con 0x7f9610069bd0 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9616eef640 5 monclient: authenticate success, global_id 104268 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9616eef640 10 monclient: _renew_subs 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9616eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9610137bb0 con 0x7f9610069bd0 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c006380 con 0x7f9610069bd0 2024-09-18T12:33:11.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 10 monclient: dump: 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:11.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9616eef640 10 monclient: _renew_subs 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9616eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961012f3b0 con 0x7f9610069bd0 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9606ffd640 4 set_mon_vals no callback set 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.175+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f960c010440 con 0x7f9610069bd0 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9616eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9616eef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9616eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96101187d0 con 0x7f9610069bd0 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9604ff9640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95fc038440 0x7f95fc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(211..211 src has 13..211) v4 ==== 11926+0+0 (secure 0 0 0) 0x7f960c04ce90 con 0x7f9610069bd0 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9607fff640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95fc038440 0x7f95fc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:11.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.176+0000 7f9607fff640 10 monclient: get_auth_request con 0x7f95fc038440 auth_method 0 2024-09-18T12:33:11.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.178+0000 7f9607fff640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95fc038440 0x7f95fc03a8f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f961012ea70 tx=0x7f9600006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:11.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.181+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f96101187d0 con 0x7f9610069bd0 2024-09-18T12:33:11.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.181+0000 7f9604ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:11.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.181+0000 7f9604ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:11.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.336+0000 7f9616eef640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:11.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.336+0000 7f9616eef640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:11.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.336+0000 7f9616eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:11.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:11.336+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f96101187d0 con 0x7f9610069bd0 2024-09-18T12:33:12.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.107+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v212) v1 ==== 168+0+0 (secure 0 0 0) 0x7f96101187d0 con 0x7f9610069bd0 2024-09-18T12:33:12.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.107+0000 7f9604ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:12.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.107+0000 7f9604ff9640 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-18T12:33:12.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.205+0000 7f9616eef640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:12.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.205+0000 7f9616eef640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:12.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.205+0000 7f9616eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:12.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.205+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f96101187d0 con 0x7f9610069bd0 2024-09-18T12:33:12.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.205+0000 7f9604ff9640 1 -- 172.21.15.144:0/1376169426 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v212) v1 ==== 147+0+0 (secure 0 0 0) 0x7f96101187d0 con 0x7f9610069bd0 2024-09-18T12:33:12.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.205+0000 7f9604ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:12.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.206+0000 7f9604ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-18T12:33:12.208 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h19' already exists 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95fc038440 msgr2=0x7f95fc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95fc038440 0x7f95fc03a8f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f961012ea70 tx=0x7f9600006d80 comp rx=0 tx=0).stop 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 10 monclient: shutdown 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 msgr2=0x7f96101353d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 0x7f96101353d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f960c00b600 tx=0x7f960c0091a0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 shutdown_connections 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f95fc038440 0x7f95fc03a8f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f961009cf60 0x7f9610135e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.207+0000 7f9616eef640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f96100986f0 0x7f9610135910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.208+0000 7f9616eef640 1 --2- 172.21.15.144:0/1376169426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9610069bd0 0x7f96101353d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.208+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 >> 172.21.15.144:0/1376169426 conn(0x7f961009e9f0 msgr2=0x7f96100a1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:12.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.208+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 shutdown_connections 2024-09-18T12:33:12.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.208+0000 7f9616eef640 1 -- 172.21.15.144:0/1376169426 wait complete. 2024-09-18T12:33:12.219 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:12.219 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-18T12:33:12.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.292+0000 7f3ae075d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:12.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.292+0000 7f3ae075d640 10 monclient: build_initial_monmap 2024-09-18T12:33:12.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.292+0000 7f3ae075d640 10 monclient: monmap: 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:12.294167+0000 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:12.294167+0000 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:12.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.292+0000 7f3ae075d640 1 Processor -- start 2024-09-18T12:33:12.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.292+0000 7f3ae075d640 1 -- start start 2024-09-18T12:33:12.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.292+0000 7f3ae075d640 10 monclient: init 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ad8131bd0 0x7f3ad8131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad8131bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ad809ebe0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ad809efe0 0x7f3ad8069510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad809efe0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad80a06d0 con 0x7f3ad809ebe0 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad8071780 con 0x7f3ad809efe0 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:12.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad80a0010 con 0x7f3ad8131bd0 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ae075d640 10 monclient(hunting): authenticate will time out at 1227.863573s 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ade4f9640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f3ad8131bd0 msgr2=0x7f3ad8131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3addcf8640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f3ad809ebe0 legacy=0x7f3ad8068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3ade4f9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3ad8131bd0 0x7f3ad8131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3adecfa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ad809efe0 0x7f3ad8069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3adecfa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ad809efe0 0x7f3ad8069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58932/0 (socket says 172.21.15.144:58932) 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3adecfa640 1 -- 172.21.15.144:0/591469311 learned_addr learned my addr 172.21.15.144:0/591469311 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3adecfa640 10 monclient(hunting): get_auth_request con 0x7f3ad809efe0 auth_method 0 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3adecfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3adecfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.293+0000 7f3adecfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:12.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 10 monclient(hunting): handle_auth_done global_id 104271 payload 274 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 1 -- 172.21.15.144:0/591469311 >> v2:172.21.15.144:3300/0 conn(0x7f3ad8131bd0 msgr2=0x7f3ad8131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 1 --2- 172.21.15.144:0/591469311 >> v2:172.21.15.144:3300/0 conn(0x7f3ad8131bd0 0x7f3ad8131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 1 -- 172.21.15.144:0/591469311 >> v1:172.21.15.144:6789/0 conn(0x7f3ad809ebe0 legacy=0x7f3ad8068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 10 monclient: _finish_hunting 0 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 1 monclient: found mon.noname-a 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:12.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.294+0000 7f3adecfa640 1 -- 172.21.15.144:0/591469311 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad80a1910 con 0x7f3ad809efe0 2024-09-18T12:33:12.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3adecfa640 1 -- 172.21.15.144:0/591469311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 msgr2=0x7f3ad8069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3adecfa640 1 --2- 172.21.15.144:0/591469311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8069510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8068370 tx=0x7f3ad0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62667966e1c3cf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 1 -- 172.21.15.144:0/591469311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0007610 con 0x7f3ad809efe0 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: dump: 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:12.307 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: _finish_auth 0 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 5 monclient: authenticate success, global_id 104271 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 1 -- 172.21.15.144:0/591469311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad002d050 con 0x7f3ad809efe0 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 4 set_mon_vals no callback set 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 1 -- 172.21.15.144:0/591469311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad00079e0 con 0x7f3ad809efe0 2024-09-18T12:33:12.308 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3add4f7640 10 monclient: dump: 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:12.309 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 10 monclient: shutdown 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 1 -- 172.21.15.144:0/591469311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 msgr2=0x7f3ad8069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.297+0000 7f3ae075d640 1 --2- 172.21.15.144:0/591469311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8069510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8068370 tx=0x7f3ad0009840 comp rx=0 tx=0).stop 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 -- 172.21.15.144:0/591469311 shutdown_connections 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 --2- 172.21.15.144:0/591469311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8069510 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 --2- 172.21.15.144:0/591469311 >> v2:172.21.15.144:3300/0 conn(0x7f3ad8131bd0 0x7f3ad8131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 -- 172.21.15.144:0/591469311 >> 172.21.15.144:0/591469311 conn(0x7f3ad80713b0 msgr2=0x7f3ad80a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 -- 172.21.15.144:0/591469311 shutdown_connections 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 -- 172.21.15.144:0/591469311 wait complete. 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 10 monclient: build_initial_monmap 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 10 monclient: monmap: 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:12.310 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 Processor -- start 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 1 -- start start 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.298+0000 7f3ae075d640 10 monclient: init 2024-09-18T12:33:12.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ad809ebe0 0x7f3ad8140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad809ebe0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad809efe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad8131bd0 0x7f3ad811d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ad8131bd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad809e790 con 0x7f3ad809efe0 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8120eb0 con 0x7f3ad8131bd0 2024-09-18T12:33:12.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8141b60 con 0x7f3ad809ebe0 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ae075d640 10 monclient(hunting): authenticate will time out at 1227.869314s 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3adecfa640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad8131bd0 msgr2=0x7f3ad811d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3adecfa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad8131bd0 0x7f3ad811d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8130a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58946/0 (socket says 172.21.15.144:58946) 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 -- 172.21.15.144:0/37349160 learned_addr learned my addr 172.21.15.144:0/37349160 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): get_auth_request con 0x7f3ad809efe0 auth_method 0 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ad809ebe0 0x7f3ad8140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:12.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 10 monclient(hunting): get_auth_request con 0x7f3ad809ebe0 auth_method 0 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3ade4f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient(hunting): handle_auth_done global_id 104274 payload 824 2024-09-18T12:33:12.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 -- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ad809ebe0 msgr2=0x7f3ad8140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ad809ebe0 0x7f3ad8140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 -- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad8131bd0 msgr2=0x7f3ad811d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad8131bd0 0x7f3ad811d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient: _finish_hunting 0 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 monclient: found mon.noname-a 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.299+0000 7f3addcf8640 1 -- 172.21.15.144:0/37349160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad0009ae0 con 0x7f3ad809efe0 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3addcf8640 1 -- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 msgr2=0x7f3ad8130a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3addcf8640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8130a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b5c0 tx=0x7f3ac8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8003cc0 con 0x7f3ad809efe0 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:12.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: dump: 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:12.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac8003e40 con 0x7f3ad809efe0 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ae075d640 5 monclient: authenticate success, global_id 104274 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ae075d640 10 monclient: _renew_subs 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ae075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad8120bd0 con 0x7f3ad809efe0 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8005340 con 0x7f3ad809efe0 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ace7fc640 10 monclient: dump: 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.317 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ae075d640 10 monclient: _renew_subs 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3adccf6640 4 set_mon_vals no callback set 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ae075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.302+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad81413d0 con 0x7f3ad809efe0 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ae075d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ae075d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ae075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:12.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad804bf60 con 0x7f3ad809efe0 2024-09-18T12:33:12.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3ac801a460 con 0x7f3ad809efe0 2024-09-18T12:33:12.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ace7fc640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3abc038440 0x7f3abc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:12.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(212..212 src has 13..212) v4 ==== 12139+0+0 (secure 0 0 0) 0x7f3ac804d130 con 0x7f3ad809efe0 2024-09-18T12:33:12.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ade4f9640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3abc038440 0x7f3abc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:12.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.303+0000 7f3ade4f9640 10 monclient: get_auth_request con 0x7f3abc038440 auth_method 0 2024-09-18T12:33:12.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.306+0000 7f3ade4f9640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3abc038440 0x7f3abc03a8f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4003850 tx=0x7f3ad40090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:12.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.307+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3ac8017960 con 0x7f3ad809efe0 2024-09-18T12:33:12.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.308+0000 7f3ace7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:12.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.308+0000 7f3ace7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:12.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.466+0000 7f3ae075d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:12.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.466+0000 7f3ae075d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:12.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.466+0000 7f3ae075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:12.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:12.466+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3ad8064990 con 0x7f3ad809efe0 2024-09-18T12:33:13.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.112+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v213) v1 ==== 168+0+0 (secure 0 0 0) 0x7f3ac8017780 con 0x7f3ad809efe0 2024-09-18T12:33:13.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.112+0000 7f3ace7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:13.114 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.112+0000 7f3ace7fc640 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-18T12:33:13.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.212+0000 7f3ae075d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:13.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.212+0000 7f3ae075d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:13.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.212+0000 7f3ae075d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:13.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.212+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3ad811fcd0 con 0x7f3ad809efe0 2024-09-18T12:33:13.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.213+0000 7f3ace7fc640 1 -- 172.21.15.144:0/37349160 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v213) v1 ==== 147+0+0 (secure 0 0 0) 0x7f3ac8017960 con 0x7f3ad809efe0 2024-09-18T12:33:13.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.213+0000 7f3ace7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:13.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.213+0000 7f3ace7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-18T12:33:13.215 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h20' already exists 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3abc038440 msgr2=0x7f3abc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3abc038440 0x7f3abc03a8f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4003850 tx=0x7f3ad40090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 10 monclient: shutdown 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 msgr2=0x7f3ad8130a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8130a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800b5c0 tx=0x7f3ac8009120 comp rx=0 tx=0).stop 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 shutdown_connections 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3abc038440 0x7f3abc03a8f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ad8131bd0 0x7f3ad811d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ad809efe0 0x7f3ad8130a80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 --2- 172.21.15.144:0/37349160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ad809ebe0 0x7f3ad8140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 >> 172.21.15.144:0/37349160 conn(0x7f3ad80713b0 msgr2=0x7f3ad80a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:13.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 shutdown_connections 2024-09-18T12:33:13.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.215+0000 7f3ae075d640 1 -- 172.21.15.144:0/37349160 wait complete. 2024-09-18T12:33:13.226 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:13.226 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-18T12:33:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.299+0000 7ffb7709f640 10 monclient: get_monmap_and_config 2024-09-18T12:33:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.299+0000 7ffb7709f640 10 monclient: build_initial_monmap 2024-09-18T12:33:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.299+0000 7ffb7709f640 10 monclient: monmap: 2024-09-18T12:33:13.301 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:13.301238+0000 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:13.301238+0000 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:13.302 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 1 Processor -- start 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 1 -- start start 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient: init 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb70099e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffb7009dfa0 0x7ffb7009d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient(hunting): picked mon.noname-c con 0x7ffb7009dfa0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb701348d0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7009f620 con 0x7ffb701348d0 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7009f860 con 0x7ffb70099e10 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb7009b7c0 con 0x7ffb7009dfa0 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb7709f640 10 monclient(hunting): authenticate will time out at 1228.870727s 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb74e3b640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7ffb70099e10 legacy=0x7ffb7009e650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb6ffff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7ffb7009dfa0 msgr2=0x7ffb7009d1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.300+0000 7ffb6ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffb7009dfa0 0x7ffb7009d1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.301+0000 7ffb7563c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffb701348d0 0x7ffb70136cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:51432/0 (socket says 172.21.15.144:51432) 2024-09-18T12:33:13.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.301+0000 7ffb7563c640 1 -- 172.21.15.144:0/3857287619 learned_addr learned my addr 172.21.15.144:0/3857287619 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb7563c640 1 -- 172.21.15.144:0/3857287619 >> v1:172.21.15.116:6789/0 conn(0x7ffb701348d0 legacy=0x7ffb70136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556530144 0 0) 0x7ffb7009f620 con 0x7ffb701348d0 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): my global_id is 104277 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb600034e0 con 0x7ffb701348d0 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2958215067 0 0) 0x7ffb600034e0 con 0x7ffb701348d0 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 >> v2:172.21.15.144:3300/0 conn(0x7ffb7009dfa0 msgr2=0x7ffb7009d1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 --2- 172.21.15.144:0/3857287619 >> v2:172.21.15.144:3300/0 conn(0x7ffb7009dfa0 0x7ffb7009d1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 >> v1:172.21.15.144:6789/0 conn(0x7ffb70099e10 legacy=0x7ffb7009e650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:13.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7009b4e0 con 0x7ffb701348d0 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7ffb68001be0 con 0x7ffb701348d0 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: dump: 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.317 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 >> v1:172.21.15.116:6789/0 conn(0x7ffb701348d0 legacy=0x7ffb70136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): picked mon.a con 0x7ffb600039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:13.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb60004780 0x7ffb60004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): picked mon.c con 0x7ffb60004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb60005280 0x7ffb6000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): picked mon.b con 0x7ffb60005280 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7009b7c0 con 0x7ffb600039d0 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6000c240 con 0x7ffb60005280 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6000c400 con 0x7ffb60004780 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb6f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb7563c640 1 -- 172.21.15.144:0/3857287619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb60005280 msgr2=0x7ffb6000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:13.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.303+0000 7ffb7563c640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb60005280 0x7ffb6000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb60004780 0x7ffb60004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): get_auth_request con 0x7ffb600039d0 auth_method 0 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 10 monclient(hunting): get_auth_request con 0x7ffb60004780 auth_method 0 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:13.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_done global_id 104277 payload 293 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 1 -- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb60004780 msgr2=0x7ffb60004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb60004780 0x7ffb60004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 1 -- 172.21.15.144:0/3857287619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb60005280 msgr2=0x7ffb6000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb60005280 0x7ffb6000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient: _finish_hunting 0 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 1 monclient: found mon.a 2024-09-18T12:33:13.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb7563c640 1 -- 172.21.15.144:0/3857287619 reap_dead start 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 1 -- 172.21.15.144:0/3857287619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb600034e0 con 0x7ffb600039d0 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 10 monclient: _finish_auth 0 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb74e3b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb7709f640 5 monclient: authenticate success, global_id 104277 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.304+0000 7ffb7709f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb74e3b640 1 -- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 msgr2=0x7ffb60003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb74e3b640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb60003dc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffb640099c0 tx=0x7ffb64009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f81323078091e8d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64003b80 con 0x7ffb600039d0 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 10 monclient: dump: 2024-09-18T12:33:13.322 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64003d20 con 0x7ffb600039d0 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:13.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 1 -- 172.21.15.144:0/3857287619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6402ab80 con 0x7ffb600039d0 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 4 set_mon_vals no callback set 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb6f7fe640 10 monclient: dump: 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:13.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 10 monclient: shutdown 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 1 -- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 msgr2=0x7ffb60003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb60003dc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffb640099c0 tx=0x7ffb64009060 comp rx=0 tx=0).stop 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 1 -- 172.21.15.144:0/3857287619 shutdown_connections 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 1 --2- 172.21.15.144:0/3857287619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb60003dc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 1 -- 172.21.15.144:0/3857287619 >> 172.21.15.144:0/3857287619 conn(0x7ffb700a3b30 msgr2=0x7ffb7009fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 1 -- 172.21.15.144:0/3857287619 shutdown_connections 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.307+0000 7ffb7709f640 1 -- 172.21.15.144:0/3857287619 wait complete. 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.308+0000 7ffb7709f640 10 monclient: build_initial_monmap 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.308+0000 7ffb7709f640 10 monclient: monmap: 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:13.325 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.308+0000 7ffb7709f640 1 Processor -- start 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.308+0000 7ffb7709f640 1 -- start start 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.308+0000 7ffb7709f640 10 monclient: init 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb70140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): picked mon.noname-a con 0x7ffb600039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:13.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb70133c50 0x7ffb70134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): picked mon.noname-c con 0x7ffb70133c50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb70068a40 0x7ffb70068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): picked mon.noname-b con 0x7ffb70068a40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7006eb80 con 0x7ffb600039d0 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700a1e40 con 0x7ffb70068a40 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7006a610 con 0x7ffb70133c50 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7709f640 10 monclient(hunting): authenticate will time out at 1228.879349s 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7563c640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb70068a40 msgr2=0x7ffb70068e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb7563c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb70068a40 0x7ffb70068e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:13.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb70140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb70140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58976/0 (socket says 172.21.15.144:58976) 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 -- 172.21.15.144:0/1855791634 learned_addr learned my addr 172.21.15.144:0/1855791634 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): get_auth_request con 0x7ffb600039d0 auth_method 0 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb70133c50 0x7ffb70134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 10 monclient(hunting): get_auth_request con 0x7ffb70133c50 auth_method 0 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:13.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient(hunting): handle_auth_done global_id 104280 payload 824 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 -- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb70133c50 msgr2=0x7ffb70134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb70133c50 0x7ffb70134020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 -- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb70068a40 msgr2=0x7ffb70068e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb70068a40 0x7ffb70068e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient: _finish_hunting 0 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 monclient: found mon.noname-a 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.309+0000 7ffb74e3b640 1 -- 172.21.15.144:0/1855791634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb70120a50 con 0x7ffb600039d0 2024-09-18T12:33:13.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb74e3b640 1 -- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 msgr2=0x7ffb70140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb74e3b640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb70140f40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffb640099c0 tx=0x7ffb64030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64003b80 con 0x7ffb600039d0 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: dump: 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:13.330 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64038070 con 0x7ffb600039d0 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb7709f640 5 monclient: authenticate success, global_id 104280 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb7709f640 10 monclient: _renew_subs 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb7709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb701418a0 con 0x7ffb600039d0 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64005e30 con 0x7ffb600039d0 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:13.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6cff9640 10 monclient: dump: 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb7709f640 10 monclient: _renew_subs 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb7709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:13.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb70141480 con 0x7ffb600039d0 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.312+0000 7ffb6effd640 4 set_mon_vals no callback set 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ffb64030070 con 0x7ffb600039d0 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb7709f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb7709f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb7709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb701221f0 con 0x7ffb600039d0 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb6cff9640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb54038440 0x7ffb5403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(213..213 src has 13..213) v4 ==== 12352+0+0 (secure 0 0 0) 0x7ffb6406d340 con 0x7ffb600039d0 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb6ffff640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb54038440 0x7ffb5403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:13.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.313+0000 7ffb6ffff640 10 monclient: get_auth_request con 0x7ffb54038440 auth_method 0 2024-09-18T12:33:13.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.316+0000 7ffb6ffff640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb54038440 0x7ffb5403a8f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c003fd0 tx=0x7ffb5c006ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:13.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.318+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ffb64032df0 con 0x7ffb600039d0 2024-09-18T12:33:13.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.318+0000 7ffb6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:13.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.318+0000 7ffb6cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:13.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.476+0000 7ffb7709f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:13.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.476+0000 7ffb7709f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:13.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.476+0000 7ffb7709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:13.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:13.476+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ffb7004bf60 con 0x7ffb600039d0 2024-09-18T12:33:14.115 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.113+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v214) v1 ==== 168+0+0 (secure 0 0 0) 0x7ffb64032840 con 0x7ffb600039d0 2024-09-18T12:33:14.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.113+0000 7ffb6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:14.116 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.113+0000 7ffb6cff9640 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.209+0000 7ffb7709f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.209+0000 7ffb7709f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.209+0000 7ffb7709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.209+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ffb701219f0 con 0x7ffb600039d0 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.210+0000 7ffb6cff9640 1 -- 172.21.15.144:0/1855791634 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v214) v1 ==== 147+0+0 (secure 0 0 0) 0x7ffb64069fc0 con 0x7ffb600039d0 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.210+0000 7ffb6cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.210+0000 7ffb6cff9640 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-18T12:33:14.212 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h21' already exists 2024-09-18T12:33:14.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb54038440 msgr2=0x7ffb5403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:14.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb54038440 0x7ffb5403a8f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c003fd0 tx=0x7ffb5c006ed0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 10 monclient: shutdown 2024-09-18T12:33:14.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:14.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 msgr2=0x7ffb70140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:14.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb70140f40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffb640099c0 tx=0x7ffb64030000 comp rx=0 tx=0).stop 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 shutdown_connections 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb54038440 0x7ffb5403a8f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb70068a40 0x7ffb70068e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb70133c50 0x7ffb70134020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 --2- 172.21.15.144:0/1855791634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb600039d0 0x7ffb70140f40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 >> 172.21.15.144:0/1855791634 conn(0x7ffb700a3b30 msgr2=0x7ffb7009c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 shutdown_connections 2024-09-18T12:33:14.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.212+0000 7ffb7709f640 1 -- 172.21.15.144:0/1855791634 wait complete. 2024-09-18T12:33:14.223 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:14.223 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-18T12:33:14.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.296+0000 7fb10979d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.296+0000 7fb10979d640 10 monclient: build_initial_monmap 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.296+0000 7fb10979d640 10 monclient: monmap: 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:14.298591+0000 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:14.298591+0000 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:14.299 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:14.300 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:14.300 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:14.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 1 Processor -- start 2024-09-18T12:33:14.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 1 -- start start 2024-09-18T12:33:14.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient: init 2024-09-18T12:33:14.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:14.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T12:33:14.302 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb104131bd0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:14.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb104069e10 0x7fb104131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:14.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient(hunting): picked mon.noname-c con 0x7fb104069e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:14.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb104136d60 0x7fb104139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:14.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient(hunting): picked mon.noname-a con 0x7fb104136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:14.303 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1040a06d0 con 0x7fb104131bd0 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb104071780 con 0x7fb104136d60 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10406a210 con 0x7fb104069e10 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb10979d640 10 monclient(hunting): authenticate will time out at 1229.867902s 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb102ffd640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fb104131bd0 legacy=0x7fb10409ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb0fbfff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fb104069e10 msgr2=0x7fb104131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.297+0000 7fb0fbfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb104069e10 0x7fb104131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb104136d60 0x7fb104139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb104136d60 0x7fb104139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58988/0 (socket says 172.21.15.144:58988) 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 -- 172.21.15.144:0/3393616314 learned_addr learned my addr 172.21.15.144:0/3393616314 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:14.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): get_auth_request con 0x7fb104136d60 auth_method 0 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient(hunting): handle_auth_done global_id 104283 payload 274 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 -- 172.21.15.144:0/3393616314 >> v2:172.21.15.144:3300/0 conn(0x7fb104069e10 msgr2=0x7fb104131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 --2- 172.21.15.144:0/3393616314 >> v2:172.21.15.144:3300/0 conn(0x7fb104069e10 0x7fb104131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 -- 172.21.15.144:0/3393616314 >> v1:172.21.15.144:6789/0 conn(0x7fb104131bd0 legacy=0x7fb10409ebe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:14.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:14.306 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.298+0000 7fb1037fe640 1 -- 172.21.15.144:0/3393616314 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1040a0010 con 0x7fb104136d60 2024-09-18T12:33:14.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1037fe640 1 -- 172.21.15.144:0/3393616314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104136d60 msgr2=0x7fb104139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:14.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1037fe640 1 --2- 172.21.15.144:0/3393616314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104136d60 0x7fb104139140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb104068d90 tx=0x7fb0fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a1c96eb8db7dfe server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:14.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 1 -- 172.21.15.144:0/3393616314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc007610 con 0x7fb104136d60 2024-09-18T12:33:14.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:14.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:14.311 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: dump: 2024-09-18T12:33:14.311 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 1 -- 172.21.15.144:0/3393616314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc02d050 con 0x7fb104136d60 2024-09-18T12:33:14.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 1 -- 172.21.15.144:0/3393616314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0079a0 con 0x7fb104136d60 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb1027fc640 10 monclient: dump: 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:14.313 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb10979d640 5 monclient: authenticate success, global_id 104283 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb10979d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.301+0000 7fb10979d640 4 set_mon_vals no callback set 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 10 monclient: shutdown 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 -- 172.21.15.144:0/3393616314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104136d60 msgr2=0x7fb104139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 --2- 172.21.15.144:0/3393616314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104136d60 0x7fb104139140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb104068d90 tx=0x7fb0fc009840 comp rx=0 tx=0).stop 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 -- 172.21.15.144:0/3393616314 shutdown_connections 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 --2- 172.21.15.144:0/3393616314 >> v2:172.21.15.144:3300/0 conn(0x7fb104069e10 0x7fb104131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 --2- 172.21.15.144:0/3393616314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104136d60 0x7fb104139140 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 -- 172.21.15.144:0/3393616314 >> 172.21.15.144:0/3393616314 conn(0x7fb1040713b0 msgr2=0x7fb1040a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:14.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 -- 172.21.15.144:0/3393616314 shutdown_connections 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 1 -- 172.21.15.144:0/3393616314 wait complete. 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 10 monclient: build_initial_monmap 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.302+0000 7fb10979d640 10 monclient: monmap: 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 Processor -- start 2024-09-18T12:33:14.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 -- start start 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient: init 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb104069e10 0x7fb104099690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): picked mon.noname-b con 0x7fb104069e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 0x7fb104099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): picked mon.noname-a con 0x7fb104131bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb104136d60 0x7fb10409a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): picked mon.noname-c con 0x7fb104136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10411ba40 con 0x7fb104131bd0 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10409b070 con 0x7fb104069e10 2024-09-18T12:33:14.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10409e910 con 0x7fb104136d60 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb10979d640 10 monclient(hunting): authenticate will time out at 1229.873494s 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb102ffd640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb104069e10 msgr2=0x7fb104099690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb102ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb104069e10 0x7fb104099690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 0x7fb104099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 0x7fb104099bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58990/0 (socket says 172.21.15.144:58990) 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 1 -- 172.21.15.144:0/2698540625 learned_addr learned my addr 172.21.15.144:0/2698540625 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb104136d60 0x7fb10409a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 10 monclient(hunting): get_auth_request con 0x7fb104131bd0 auth_method 0 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:14.317 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 10 monclient(hunting): get_auth_request con 0x7fb104136d60 auth_method 0 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb0fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.303+0000 7fb1037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 10 monclient(hunting): handle_auth_done global_id 104286 payload 824 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 1 -- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb104136d60 msgr2=0x7fb10409a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:14.318 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb104136d60 0x7fb10409a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 1 -- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb104069e10 msgr2=0x7fb104099690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb104069e10 0x7fb104099690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 1 monclient: found mon.noname-a 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb0fbfff640 1 -- 172.21.15.144:0/2698540625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc009ae0 con 0x7fb104131bd0 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.304+0000 7fb1037fe640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb104136d60 0x7fb10409a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0fbfff640 1 -- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 msgr2=0x7fb104099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0fbfff640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 0x7fb104099bd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00b620 tx=0x7fb0ec009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec003cc0 con 0x7fb104131bd0 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:14.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: dump: 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:14.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ec003e20 con 0x7fb104131bd0 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec005340 con 0x7fb104131bd0 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb10979d640 5 monclient: authenticate success, global_id 104286 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb0faffd640 10 monclient: dump: 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:14.321 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb10979d640 10 monclient: _renew_subs 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb101ffb640 4 set_mon_vals no callback set 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb10979d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10409bf30 con 0x7fb104131bd0 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb10979d640 10 monclient: _renew_subs 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb10979d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.306+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10411cd50 con 0x7fb104131bd0 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb0ec010480 con 0x7fb104131bd0 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb10979d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb10979d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:14.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb10979d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb104131870 con 0x7fb104131bd0 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb0faffd640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb0e4038440 0x7fb0e403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(214..214 src has 13..214) v4 ==== 12565+0+0 (secure 0 0 0) 0x7fb0ec04d210 con 0x7fb104131bd0 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb102ffd640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb0e4038440 0x7fb0e403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.307+0000 7fb102ffd640 10 monclient: get_auth_request con 0x7fb0e4038440 auth_method 0 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.310+0000 7fb102ffd640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb0e4038440 0x7fb0e403a8f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4003940 tx=0x7fb0f4006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.312+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb0ec019e30 con 0x7fb104131bd0 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.312+0000 7fb0faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:14.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.312+0000 7fb0faffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:14.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.471+0000 7fb10979d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:14.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.471+0000 7fb10979d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:14.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.471+0000 7fb10979d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:14.473 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:14.471+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb10409bf30 con 0x7fb104131bd0 2024-09-18T12:33:15.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.123+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v215) v1 ==== 168+0+0 (secure 0 0 0) 0x7fb10409bf30 con 0x7fb104131bd0 2024-09-18T12:33:15.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.123+0000 7fb0faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:15.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.123+0000 7fb0faffd640 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-18T12:33:15.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.219+0000 7fb10979d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:15.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.219+0000 7fb10979d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:15.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.219+0000 7fb10979d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:15.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.219+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fb10409bf30 con 0x7fb104131bd0 2024-09-18T12:33:15.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.219+0000 7fb0faffd640 1 -- 172.21.15.144:0/2698540625 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v215) v1 ==== 147+0+0 (secure 0 0 0) 0x7fb10409bf30 con 0x7fb104131bd0 2024-09-18T12:33:15.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.219+0000 7fb0faffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:15.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.219+0000 7fb0faffd640 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-18T12:33:15.222 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h22' already exists 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb0e4038440 msgr2=0x7fb0e403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb0e4038440 0x7fb0e403a8f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4003940 tx=0x7fb0f4006d80 comp rx=0 tx=0).stop 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 10 monclient: shutdown 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 msgr2=0x7fb104099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 0x7fb104099bd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00b620 tx=0x7fb0ec009120 comp rx=0 tx=0).stop 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 shutdown_connections 2024-09-18T12:33:15.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb0e4038440 0x7fb0e403a8f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb104136d60 0x7fb10409a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb104131bd0 0x7fb104099bd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 --2- 172.21.15.144:0/2698540625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb104069e10 0x7fb104099690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 >> 172.21.15.144:0/2698540625 conn(0x7fb1040713b0 msgr2=0x7fb10409d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:15.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.221+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 shutdown_connections 2024-09-18T12:33:15.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.222+0000 7fb10979d640 1 -- 172.21.15.144:0/2698540625 wait complete. 2024-09-18T12:33:15.233 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:15.233 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.306+0000 7f1d13e46640 10 monclient: get_monmap_and_config 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.306+0000 7f1d13e46640 10 monclient: build_initial_monmap 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.306+0000 7f1d13e46640 10 monclient: monmap: 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:15.308621+0000 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:15.308621+0000 2024-09-18T12:33:15.309 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 Processor -- start 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 -- start start 2024-09-18T12:33:15.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient: init 2024-09-18T12:33:15.312 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d0c09c2b0 0x7f1d0c09a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): picked mon.noname-a con 0x7f1d0c09c2b0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1d0c09c680 0x7f1d0c09ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): picked mon.noname-c con 0x7f1d0c09c680 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c09dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): picked mon.noname-b con 0x7f1d0c09d980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c099f60 con 0x7f1d0c09c2b0 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c09bb40 con 0x7f1d0c09c680 2024-09-18T12:33:15.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c09f700 con 0x7f1d0c09d980 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.307+0000 7f1d13e46640 10 monclient(hunting): authenticate will time out at 1230.877965s 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d113e1640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f1d0c09c680 msgr2=0x7f1d0c09ace0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d113e1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1d0c09c680 0x7f1d0c09ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d0c09c2b0 0x7f1d0c09a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c09dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d0c09c2b0 0x7f1d0c09a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:59000/0 (socket says 172.21.15.144:59000) 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 -- 172.21.15.144:0/3017879046 learned_addr learned my addr 172.21.15.144:0/3017879046 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): get_auth_request con 0x7f1d0c09c2b0 auth_method 0 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:15.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 10 monclient(hunting): get_auth_request con 0x7f1d0c09d980 auth_method 0 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d123e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient(hunting): handle_auth_done global_id 104289 payload 274 2024-09-18T12:33:15.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 -- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 msgr2=0x7f1d0c09dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 --2- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c09dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 -- 172.21.15.144:0/3017879046 >> v2:172.21.15.144:3300/0 conn(0x7f1d0c09c680 msgr2=0x7f1d0c09ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 --2- 172.21.15.144:0/3017879046 >> v2:172.21.15.144:3300/0 conn(0x7f1d0c09c680 0x7f1d0c09ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient: _finish_hunting 0 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 monclient: found mon.noname-a 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:15.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.308+0000 7f1d11be2640 1 -- 172.21.15.144:0/3017879046 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c09b220 con 0x7f1d0c09c2b0 2024-09-18T12:33:15.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.311+0000 7f1d11be2640 1 -- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c2b0 msgr2=0x7f1d0c09a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.311+0000 7f1d11be2640 1 --2- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c2b0 0x7f1d0c09a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c09f3d0 tx=0x7f1d080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41602a631ba6b7c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 1 -- 172.21.15.144:0/3017879046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08032440 con 0x7f1d0c09c2b0 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: dump: 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:15.323 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: _finish_auth 0 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 5 monclient: authenticate success, global_id 104289 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 1 -- 172.21.15.144:0/3017879046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d080325a0 con 0x7f1d0c09c2b0 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 4 set_mon_vals no callback set 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 1 -- 172.21.15.144:0/3017879046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08032980 con 0x7f1d0c09c2b0 2024-09-18T12:33:15.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d10be0640 10 monclient: dump: 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 10 monclient: shutdown 2024-09-18T12:33:15.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 -- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c2b0 msgr2=0x7f1d0c09a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c2b0 0x7f1d0c09a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c09f3d0 tx=0x7f1d080091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 -- 172.21.15.144:0/3017879046 shutdown_connections 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c09dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3017879046 >> v2:172.21.15.144:3300/0 conn(0x7f1d0c09c680 0x7f1d0c09ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3017879046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c2b0 0x7f1d0c09a7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 -- 172.21.15.144:0/3017879046 >> 172.21.15.144:0/3017879046 conn(0x7f1d0c0713b0 msgr2=0x7f1d0c0a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 -- 172.21.15.144:0/3017879046 shutdown_connections 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.312+0000 7f1d13e46640 1 -- 172.21.15.144:0/3017879046 wait complete. 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.313+0000 7f1d13e46640 10 monclient: build_initial_monmap 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.313+0000 7f1d13e46640 10 monclient: monmap: 2024-09-18T12:33:15.326 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.313+0000 7f1d13e46640 1 Processor -- start 2024-09-18T12:33:15.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.313+0000 7f1d13e46640 1 -- start start 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.313+0000 7f1d13e46640 10 monclient: init 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d0c09c2b0 0x7f1d0c13cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): picked mon.noname-b con 0x7f1d0c09c2b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 0x7f1d0c130930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): picked mon.noname-a con 0x7f1d0c09c680 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c11a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): picked mon.noname-c con 0x7f1d0c09d980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c06e9e0 con 0x7f1d0c09c680 2024-09-18T12:33:15.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0a2650 con 0x7f1d0c09c2b0 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c11ddc0 con 0x7f1d0c09d980 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d11be2640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d0c09c2b0 msgr2=0x7f1d0c13cb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d13e46640 10 monclient(hunting): authenticate will time out at 1230.884122s 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d11be2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d0c09c2b0 0x7f1d0c13cb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 0x7f1d0c130930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c11a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 0x7f1d0c130930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:59002/0 (socket says 172.21.15.144:59002) 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 -- 172.21.15.144:0/3932484159 learned_addr learned my addr 172.21.15.144:0/3932484159 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): get_auth_request con 0x7f1d0c09c680 auth_method 0 2024-09-18T12:33:15.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 10 monclient(hunting): get_auth_request con 0x7f1d0c09d980 auth_method 0 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:15.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d123e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient(hunting): handle_auth_done global_id 104292 payload 824 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 -- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 msgr2=0x7f1d0c11a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c11a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 -- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d0c09c2b0 msgr2=0x7f1d0c13cb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d0c09c2b0 0x7f1d0c13cb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient: _finish_hunting 0 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 monclient: found mon.noname-a 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.314+0000 7f1d113e1640 1 -- 172.21.15.144:0/3932484159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08009450 con 0x7f1d0c09c680 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d113e1640 1 -- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 msgr2=0x7f1d0c130930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d113e1640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 0x7f1d0c130930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00b5f0 tx=0x7f1cfc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc015070 con 0x7f1d0c09c680 2024-09-18T12:33:15.331 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: dump: 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:15.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d13e46640 5 monclient: authenticate success, global_id 104292 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d13e46640 10 monclient: _renew_subs 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d13e46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c11bf20 con 0x7f1d0c09c680 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cfc006150 con 0x7f1d0c09c680 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc0062c0 con 0x7f1d0c09c680 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d01ffb640 10 monclient: dump: 2024-09-18T12:33:15.333 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d13e46640 10 monclient: _renew_subs 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d13e46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d03fff640 4 set_mon_vals no callback set 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.317+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c13d060 con 0x7f1d0c09c680 2024-09-18T12:33:15.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1cfc0066f0 con 0x7f1d0c09c680 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d13e46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d13e46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d13e46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d01ffb640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1cf0038440 0x7f1cf003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0c04bf60 con 0x7f1d0c09c680 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(215..215 src has 13..215) v4 ==== 12778+0+0 (secure 0 0 0) 0x7f1cfc04cfb0 con 0x7f1d0c09c680 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d11be2640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1cf0038440 0x7f1cf003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.318+0000 7f1d11be2640 10 monclient: get_auth_request con 0x7f1cf0038440 auth_method 0 2024-09-18T12:33:15.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.321+0000 7f1d11be2640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1cf0038440 0x7f1cf003a8f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1d08034000 tx=0x7f1d08032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:15.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.323+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1cfc01a5d0 con 0x7f1d0c09c680 2024-09-18T12:33:15.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.323+0000 7f1d01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:15.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.323+0000 7f1d01ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:15.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.479+0000 7f1d13e46640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:15.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.480+0000 7f1d13e46640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:15.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.480+0000 7f1d13e46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:15.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:15.480+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1d0c064a70 con 0x7f1d0c09c680 2024-09-18T12:33:16.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.127+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v216) v1 ==== 168+0+0 (secure 0 0 0) 0x7f1cfc018310 con 0x7f1d0c09c680 2024-09-18T12:33:16.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.127+0000 7f1d01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:16.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.127+0000 7f1d01ffb640 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-18T12:33:16.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.221+0000 7f1d13e46640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:16.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.221+0000 7f1d13e46640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:16.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.221+0000 7f1d13e46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:16.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.221+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1d0c11caf0 con 0x7f1d0c09c680 2024-09-18T12:33:16.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.223+0000 7f1d01ffb640 1 -- 172.21.15.144:0/3932484159 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v216) v1 ==== 147+0+0 (secure 0 0 0) 0x7f1cfc0184f0 con 0x7f1d0c09c680 2024-09-18T12:33:16.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.223+0000 7f1d01ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:16.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.223+0000 7f1d01ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-18T12:33:16.225 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h23' already exists 2024-09-18T12:33:16.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1cf0038440 msgr2=0x7f1cf003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:16.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1cf0038440 0x7f1cf003a8f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1d08034000 tx=0x7f1d08032000 comp rx=0 tx=0).stop 2024-09-18T12:33:16.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 10 monclient: shutdown 2024-09-18T12:33:16.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 msgr2=0x7f1d0c130930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 0x7f1d0c130930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00b5f0 tx=0x7f1cfc009120 comp rx=0 tx=0).stop 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 shutdown_connections 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1cf0038440 0x7f1cf003a8f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d0c09d980 0x7f1d0c11a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0c09c680 0x7f1d0c130930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 --2- 172.21.15.144:0/3932484159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d0c09c2b0 0x7f1d0c13cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 >> 172.21.15.144:0/3932484159 conn(0x7f1d0c0713b0 msgr2=0x7f1d0c06d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 shutdown_connections 2024-09-18T12:33:16.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.224+0000 7f1d13e46640 1 -- 172.21.15.144:0/3932484159 wait complete. 2024-09-18T12:33:16.235 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:16.235 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient: get_monmap_and_config 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient: build_initial_monmap 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient: monmap: 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:16.310067+0000 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:16.310067+0000 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:16.310 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:16.311 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:16.311 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:16.311 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:16.311 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:16.311 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:16.311 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:16.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 1 Processor -- start 2024-09-18T12:33:16.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 1 -- start start 2024-09-18T12:33:16.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient: init 2024-09-18T12:33:16.313 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8f08131bd0 0x7f8f08131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient(hunting): picked mon.noname-c con 0x7f8f08131bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f0809ebe0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f0809efe0 0x7f8f08069510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient(hunting): picked mon.noname-a con 0x7f8f0809efe0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080a06d0 con 0x7f8f0809ebe0 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f08071780 con 0x7f8f0809efe0 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f080a0010 con 0x7f8f08131bd0 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0cfeb640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f8f08131bd0 msgr2=0x7f8f08131fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:16.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0f24f640 10 monclient(hunting): authenticate will time out at 1231.879561s 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0cfeb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8f08131bd0 0x7f8f08131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f07fff640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f8f0809ebe0 legacy=0x7f8f08068e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.309+0000 7f8f0d7ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f0809efe0 0x7f8f08069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f0809efe0 0x7f8f08069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:59006/0 (socket says 172.21.15.144:59006) 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 1 -- 172.21.15.144:0/1250774844 learned_addr learned my addr 172.21.15.144:0/1250774844 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): get_auth_request con 0x7f8f0809efe0 auth_method 0 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient(hunting): handle_auth_done global_id 104295 payload 274 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 1 -- 172.21.15.144:0/1250774844 >> v2:172.21.15.144:3300/0 conn(0x7f8f08131bd0 msgr2=0x7f8f08131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:16.315 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 1 --2- 172.21.15.144:0/1250774844 >> v2:172.21.15.144:3300/0 conn(0x7f8f08131bd0 0x7f8f08131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 1 -- 172.21.15.144:0/1250774844 >> v1:172.21.15.144:6789/0 conn(0x7f8f0809ebe0 legacy=0x7f8f08068e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:16.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient: _finish_hunting 0 2024-09-18T12:33:16.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 1 monclient: found mon.noname-a 2024-09-18T12:33:16.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:16.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:16.316 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.310+0000 7f8f0d7ec640 1 -- 172.21.15.144:0/1250774844 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080a1910 con 0x7f8f0809efe0 2024-09-18T12:33:16.319 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0d7ec640 1 -- 172.21.15.144:0/1250774844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 msgr2=0x7f8f08069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0d7ec640 1 --2- 172.21.15.144:0/1250774844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08069510 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f08068370 tx=0x7f8ef00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3c443d0a58b956 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 1 -- 172.21.15.144:0/1250774844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0032440 con 0x7f8f0809efe0 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: dump: 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:16.320 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 1 -- 172.21.15.144:0/1250774844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef00325a0 con 0x7f8f0809efe0 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 5 monclient: authenticate success, global_id 104295 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 1 -- 172.21.15.144:0/1250774844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0032940 con 0x7f8f0809efe0 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:16.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 4 set_mon_vals no callback set 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f077fe640 10 monclient: dump: 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:16.322 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 10 monclient: shutdown 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 -- 172.21.15.144:0/1250774844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 msgr2=0x7f8f08069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/1250774844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08069510 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f08068370 tx=0x7f8ef00091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 -- 172.21.15.144:0/1250774844 shutdown_connections 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/1250774844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08069510 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/1250774844 >> v2:172.21.15.144:3300/0 conn(0x7f8f08131bd0 0x7f8f08131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 -- 172.21.15.144:0/1250774844 >> 172.21.15.144:0/1250774844 conn(0x7f8f080713b0 msgr2=0x7f8f080a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 -- 172.21.15.144:0/1250774844 shutdown_connections 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 -- 172.21.15.144:0/1250774844 wait complete. 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 10 monclient: build_initial_monmap 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 10 monclient: monmap: 2024-09-18T12:33:16.323 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 Processor -- start 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 1 -- start start 2024-09-18T12:33:16.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.314+0000 7f8f0f24f640 10 monclient: init 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f0809ebe0 0x7f8f08140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): picked mon.noname-c con 0x7f8f0809ebe0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08130af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): picked mon.noname-a con 0x7f8f0809efe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f08131bd0 0x7f8f0811d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): picked mon.noname-b con 0x7f8f08131bd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f08120c40 con 0x7f8f0809efe0 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0809bec0 con 0x7f8f08131bd0 2024-09-18T12:33:16.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0809e8a0 con 0x7f8f0809ebe0 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0f24f640 10 monclient(hunting): authenticate will time out at 1231.885413s 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0d7ec640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f08131bd0 msgr2=0x7f8f0811d750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0d7ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f08131bd0 0x7f8f0811d750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08130af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08130af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:59010/0 (socket says 172.21.15.144:59010) 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f0809ebe0 0x7f8f08140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 -- 172.21.15.144:0/3039951895 learned_addr learned my addr 172.21.15.144:0/3039951895 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): get_auth_request con 0x7f8f0809efe0 auth_method 0 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:16.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 10 monclient(hunting): get_auth_request con 0x7f8f0809ebe0 auth_method 0 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f0cfeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient(hunting): handle_auth_done global_id 104298 payload 824 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 -- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f0809ebe0 msgr2=0x7f8f08140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f0809ebe0 0x7f8f08140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 -- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f08131bd0 msgr2=0x7f8f0811d750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:16.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f08131bd0 0x7f8f0811d750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:16.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:16.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.315+0000 7f8f07fff640 1 monclient: found mon.noname-a 2024-09-18T12:33:16.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.316+0000 7f8f07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:16.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.316+0000 7f8f07fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:16.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.316+0000 7f8f07fff640 1 -- 172.21.15.144:0/3039951895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef0009450 con 0x7f8f0809efe0 2024-09-18T12:33:16.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f07fff640 1 -- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 msgr2=0x7f8f08130af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:16.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f07fff640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08130af0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800bbb0 tx=0x7f8ef80097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:16.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef800cc80 con 0x7f8f0809efe0 2024-09-18T12:33:16.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:16.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:16.330 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: dump: 2024-09-18T12:33:16.330 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:16.331 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef800cde0 con 0x7f8f0809efe0 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f0f24f640 5 monclient: authenticate success, global_id 104298 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8016430 con 0x7f8f0809efe0 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f0f24f640 10 monclient: _renew_subs 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f0f24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0811fce0 con 0x7f8f0809efe0 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:16.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f04ff9640 10 monclient: dump: 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f06ffd640 4 set_mon_vals no callback set 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f0f24f640 10 monclient: _renew_subs 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f0f24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:16.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.318+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0811f110 con 0x7f8f0809efe0 2024-09-18T12:33:16.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.319+0000 7f8f0f24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:16.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.319+0000 7f8f0f24f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:16.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.319+0000 7f8f0f24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:16.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.319+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8ef8012020 con 0x7f8f0809efe0 2024-09-18T12:33:16.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.319+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0804bf60 con 0x7f8f0809efe0 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.320+0000 7f8f04ff9640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ee8038440 0x7f8ee803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.320+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(216..216 src has 13..216) v4 ==== 12991+0+0 (secure 0 0 0) 0x7f8ef804d790 con 0x7f8f0809efe0 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.320+0000 7f8f0cfeb640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ee8038440 0x7f8ee803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.320+0000 7f8f0cfeb640 10 monclient: get_auth_request con 0x7f8ee8038440 auth_method 0 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.322+0000 7f8f0cfeb640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ee8038440 0x7f8ee803a8f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8f00003940 tx=0x7f8f000090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.325+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8ef801d900 con 0x7f8f0809efe0 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.325+0000 7f8f04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:16.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.325+0000 7f8f04ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:16.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.479+0000 7f8f0f24f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:16.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.479+0000 7f8f0f24f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:16.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.479+0000 7f8f0f24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:16.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:16.479+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8f08120fe0 con 0x7f8f0809efe0 2024-09-18T12:33:17.032 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:33:17.032 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T12:33:17.032 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T12:33:17.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T12:33:17.036 INFO:tasks.ceph.mon.b:Started 2024-09-18T12:33:17.036 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:33:17.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:17.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.131+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v217) v1 ==== 168+0+0 (secure 0 0 0) 0x7f8ef8014340 con 0x7f8f0809efe0 2024-09-18T12:33:17.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.132+0000 7f8f04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:17.133 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.132+0000 7f8f04ff9640 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-18T12:33:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.142+0000 7f4f2cf9d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.142+0000 7f4f2cf9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.142+0000 7f4f2cf9d640 10 monclient: monmap: 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:17.142878+0000 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:17.142878+0000 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:17.144 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.142+0000 7f4f2cf9d640 1 Processor -- start 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.142+0000 7f4f2cf9d640 1 -- start start 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.142+0000 7f4f2cf9d640 10 monclient: init 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f280974f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 0x7f4f2809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f2809a8a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f28133070 0x7f4f28135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f28133070 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2812b210 con 0x7f4f280974f0 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f280978c0 con 0x7f4f28133070 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2809b220 con 0x7f4f2809a8a0 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2cf9d640 10 monclient(hunting): authenticate will time out at 1224.464511s 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f28133070 0x7f4f28135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4f280974f0 0x7f4f2812ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43998/0 (socket says 172.21.15.116:43998) 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f2659c640 1 -- 172.21.15.116:0/456280985 learned_addr learned my addr 172.21.15.116:0/456280985 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f28133070 0x7f4f28135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55452/0 (socket says 172.21.15.116:55452) 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request con 0x7f4f28133070 auth_method 0 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 1 --2- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 0x7f4f2809ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 10 monclient(hunting): get_auth_request con 0x7f4f2809a8a0 auth_method 0 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 1 -- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 msgr2=0x7f4f2809ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 1 -- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 msgr2=0x7f4f2809ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 1 --2- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 0x7f4f2809ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f25d9b640 1 --2- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 0x7f4f2809ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:17.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_done global_id 104301 payload 274 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 1 -- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 msgr2=0x7f4f2809ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 1 --2- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 0x7f4f2809ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 1 -- 172.21.15.116:0/456280985 >> v1:172.21.15.116:6789/0 conn(0x7f4f280974f0 legacy=0x7f4f2812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 1 monclient: found mon.noname-a 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.143+0000 7f4f26d9d640 1 -- 172.21.15.116:0/456280985 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2809ca20 con 0x7f4f28133070 2024-09-18T12:33:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.146+0000 7f4f26d9d640 1 -- 172.21.15.116:0/456280985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f28133070 msgr2=0x7f4f28135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.146+0000 7f4f26d9d640 1 --2- 172.21.15.116:0/456280985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f28133070 0x7f4f28135450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4f2809d040 tx=0x7f4f1800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bf8c2f581d9ce0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 1 -- 172.21.15.116:0/456280985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18031720 con 0x7f4f28133070 2024-09-18T12:33:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: dump: 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: _finish_auth 0 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 1 -- 172.21.15.116:0/456280985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1802f050 con 0x7f4f28133070 2024-09-18T12:33:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 5 monclient: authenticate success, global_id 104301 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 1 -- 172.21.15.116:0/456280985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18031ab0 con 0x7f4f28133070 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 4 set_mon_vals no callback set 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2559a640 10 monclient: dump: 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 10 monclient: shutdown 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/456280985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f28133070 msgr2=0x7f4f28135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/456280985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f28133070 0x7f4f28135450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4f2809d040 tx=0x7f4f1800b3b0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/456280985 shutdown_connections 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/456280985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f28133070 0x7f4f28135450 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/456280985 >> v2:172.21.15.144:3300/0 conn(0x7f4f2809a8a0 0x7f4f2809ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/456280985 >> 172.21.15.116:0/456280985 conn(0x7f4f280672b0 msgr2=0x7f4f280676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/456280985 shutdown_connections 2024-09-18T12:33:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.147+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/456280985 wait complete. 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.148+0000 7f4f2cf9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.148+0000 7f4f2cf9d640 10 monclient: monmap: 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.148+0000 7f4f2cf9d640 1 Processor -- start 2024-09-18T12:33:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 1 -- start start 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient: init 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 0x7f4f2812fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4f280974f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 0x7f4f28130260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4f2809a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f28133070 0x7f4f281307a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4f28133070 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28098300 con 0x7f4f2809a8a0 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2809e430 con 0x7f4f280974f0 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28131ed0 con 0x7f4f28133070 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2cf9d640 10 monclient(hunting): authenticate will time out at 1224.470589s 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 0x7f4f28130260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 0x7f4f28130260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55456/0 (socket says 172.21.15.116:55456) 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 1 -- 172.21.15.116:0/2515093088 learned_addr learned my addr 172.21.15.116:0/2515093088 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): get_auth_request con 0x7f4f2809a8a0 auth_method 0 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f28133070 0x7f4f281307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request con 0x7f4f28133070 auth_method 0 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 0x7f4f2812fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 10 monclient(hunting): get_auth_request con 0x7f4f280974f0 auth_method 0 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:17.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 1 -- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 msgr2=0x7f4f2812fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 1 -- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 msgr2=0x7f4f2812fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 0x7f4f2812fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f2659c640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 0x7f4f2812fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 10 monclient(hunting): handle_auth_done global_id 104304 payload 824 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 1 -- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f28133070 msgr2=0x7f4f281307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.149+0000 7f4f25d9b640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f28133070 0x7f4f281307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.150+0000 7f4f25d9b640 1 -- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 msgr2=0x7f4f2812fd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.150+0000 7f4f25d9b640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 0x7f4f2812fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.150+0000 7f4f25d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.150+0000 7f4f25d9b640 1 monclient: found mon.noname-a 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.150+0000 7f4f25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.150+0000 7f4f25d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.150+0000 7f4f25d9b640 1 -- 172.21.15.116:0/2515093088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1800b630 con 0x7f4f2809a8a0 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f25d9b640 1 -- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 msgr2=0x7f4f28130260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f25d9b640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 0x7f4f28130260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000d890 tx=0x7f4f10008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 1 -- 172.21.15.116:0/2515093088 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1000de20 con 0x7f4f2809a8a0 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: dump: 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 1 -- 172.21.15.116:0/2515093088 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1000bae0 con 0x7f4f2809a8a0 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f2cf9d640 5 monclient: authenticate success, global_id 104304 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f2cf9d640 10 monclient: _renew_subs 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f2cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2806c520 con 0x7f4f2809a8a0 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 1 -- 172.21.15.116:0/2515093088 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1000bea0 con 0x7f4f2809a8a0 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f1e7fc640 10 monclient: dump: 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f24d99640 4 set_mon_vals no callback set 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f2cf9d640 10 monclient: _renew_subs 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f2cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.152+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f28132320 con 0x7f4f2809a8a0 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f1e7fc640 1 -- 172.21.15.116:0/2515093088 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4f10014640 con 0x7f4f2809a8a0 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f2cf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f2cf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f2cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f1e7fc640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f08038490 0x7f4f0803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2804bf60 con 0x7f4f2809a8a0 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f1e7fc640 1 -- 172.21.15.116:0/2515093088 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(217..217 src has 13..217) v4 ==== 13204+0+0 (secure 0 0 0) 0x7f4f1004df10 con 0x7f4f2809a8a0 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f2659c640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f08038490 0x7f4f0803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.153+0000 7f4f2659c640 10 monclient: get_auth_request con 0x7f4f08038490 auth_method 0 2024-09-18T12:33:17.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.156+0000 7f4f2659c640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f08038490 0x7f4f0803a940 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4f20005140 tx=0x7f4f20009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:17.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.158+0000 7f4f1e7fc640 1 -- 172.21.15.116:0/2515093088 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4f1001e460 con 0x7f4f2809a8a0 2024-09-18T12:33:17.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.158+0000 7f4f1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:17.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.158+0000 7f4f1e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:17.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.226+0000 7f8f0f24f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:17.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.226+0000 7f8f0f24f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:17.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.226+0000 7f8f0f24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.226+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8f08064990 con 0x7f8f0809efe0 2024-09-18T12:33:17.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.227+0000 7f8f04ff9640 1 -- 172.21.15.144:0/3039951895 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v217) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8ef801d300 con 0x7f8f0809efe0 2024-09-18T12:33:17.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.227+0000 7f8f04ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:17.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.227+0000 7f8f04ff9640 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-18T12:33:17.228 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h24' already exists 2024-09-18T12:33:17.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ee8038440 msgr2=0x7f8ee803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ee8038440 0x7f8ee803a8f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8f00003940 tx=0x7f8f000090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 10 monclient: shutdown 2024-09-18T12:33:17.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:17.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 msgr2=0x7f8f08130af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08130af0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800bbb0 tx=0x7f8ef80097f0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 shutdown_connections 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8ee8038440 0x7f8ee803a8f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8f08131bd0 0x7f8f0811d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f0809efe0 0x7f8f08130af0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 --2- 172.21.15.144:0/3039951895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8f0809ebe0 0x7f8f08140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 >> 172.21.15.144:0/3039951895 conn(0x7f8f080713b0 msgr2=0x7f8f080a4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 shutdown_connections 2024-09-18T12:33:17.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.229+0000 7f8f0f24f640 1 -- 172.21.15.144:0/3039951895 wait complete. 2024-09-18T12:33:17.239 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:17.239 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-18T12:33:17.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.312+0000 7effc864f640 10 monclient: get_monmap_and_config 2024-09-18T12:33:17.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.312+0000 7effc864f640 10 monclient: build_initial_monmap 2024-09-18T12:33:17.314 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.312+0000 7effc864f640 10 monclient: monmap: 2024-09-18T12:33:17.314 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:17.314 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.314 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:17.313759+0000 2024-09-18T12:33:17.315 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:17.313759+0000 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:17.316 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.313+0000 7effc864f640 1 Processor -- start 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.313+0000 7effc864f640 1 -- start start 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.313+0000 7effc864f640 10 monclient: init 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7effc0096a30 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7effc009d530 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:17.320 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7effc0099d70 0x7effc0097510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient(hunting): picked mon.noname-a con 0x7effc0099d70 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc009a1d0 con 0x7effc009d530 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc0097e80 con 0x7effc0096a30 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc0097a70 con 0x7effc0099d70 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc864f640 10 monclient(hunting): authenticate will time out at 1232.883241s 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc63eb640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7effc0096a30 0x7effc0096e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34126/0 (socket says 172.21.15.144:34126) 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc63eb640 1 -- 172.21.15.144:0/3616771821 learned_addr learned my addr 172.21.15.144:0/3616771821 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 1 --2- 172.21.15.144:0/3616771821 >> v2:172.21.15.116:3300/0 conn(0x7effc0099d70 0x7effc0097510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): get_auth_request con 0x7effc0099d70 auth_method 0 2024-09-18T12:33:17.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient(hunting): handle_auth_done global_id 104307 payload 274 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 1 -- 172.21.15.144:0/3616771821 >> v1:172.21.15.144:6789/0 conn(0x7effc0096a30 legacy=0x7effc0096e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 1 -- 172.21.15.144:0/3616771821 >> v1:172.21.15.116:6789/0 conn(0x7effc009d530 legacy=0x7effc0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient: _finish_hunting 0 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 1 monclient: found mon.noname-a 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.322 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.314+0000 7effc6bec640 1 -- 172.21.15.144:0/3616771821 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc0098be0 con 0x7effc0099d70 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc6bec640 1 -- 172.21.15.144:0/3616771821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc0099d70 msgr2=0x7effc0097510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc6bec640 1 --2- 172.21.15.144:0/3616771821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc0099d70 0x7effc0097510 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7effb8009e60 tx=0x7effb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f040a7360844c02f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 1 -- 172.21.15.144:0/3616771821 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8007840 con 0x7effc0099d70 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: dump: 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.323 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: _finish_auth 0 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 1 -- 172.21.15.144:0/3616771821 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb802d050 con 0x7effc0099d70 2024-09-18T12:33:17.324 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc864f640 5 monclient: authenticate success, global_id 104307 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc864f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc864f640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc864f640 4 set_mon_vals no callback set 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 1 -- 172.21.15.144:0/3616771821 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8007bd0 con 0x7effc0099d70 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.317+0000 7effc53e9640 10 monclient: dump: 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.325 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 10 monclient: shutdown 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 1 -- 172.21.15.144:0/3616771821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc0099d70 msgr2=0x7effc0097510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 1 --2- 172.21.15.144:0/3616771821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc0099d70 0x7effc0097510 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7effb8009e60 tx=0x7effb8009780 comp rx=0 tx=0).stop 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 1 -- 172.21.15.144:0/3616771821 shutdown_connections 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 1 --2- 172.21.15.144:0/3616771821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc0099d70 0x7effc0097510 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 1 -- 172.21.15.144:0/3616771821 >> 172.21.15.144:0/3616771821 conn(0x7effc00a0fe0 msgr2=0x7effc00a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:17.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 1 -- 172.21.15.144:0/3616771821 shutdown_connections 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.318+0000 7effc864f640 1 -- 172.21.15.144:0/3616771821 wait complete. 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient: build_initial_monmap 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient: monmap: 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:17.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 Processor -- start 2024-09-18T12:33:17.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 -- start start 2024-09-18T12:33:17.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient: init 2024-09-18T12:33:17.332 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 0x7effc01190b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): picked mon.noname-b con 0x7effc0096a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effc0099d70 0x7effc01195f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): picked mon.noname-c con 0x7effc0099d70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc009d530 0x7effc0119b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): picked mon.noname-a con 0x7effc009d530 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0132aa0 con 0x7effc009d530 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc011aec0 con 0x7effc0096a30 2024-09-18T12:33:17.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc00984c0 con 0x7effc0099d70 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.319+0000 7effc864f640 10 monclient(hunting): authenticate will time out at 1232.888921s 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 0x7effc01190b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 0x7effc01190b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55200/0 (socket says 172.21.15.144:55200) 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 1 -- 172.21.15.144:0/1492458057 learned_addr learned my addr 172.21.15.144:0/1492458057 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 10 monclient(hunting): get_auth_request con 0x7effc0096a30 auth_method 0 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc009d530 0x7effc0119b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): get_auth_request con 0x7effc009d530 auth_method 0 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effc0099d70 0x7effc01195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 10 monclient(hunting): get_auth_request con 0x7effc0099d70 auth_method 0 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 1 -- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 msgr2=0x7effc01190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 1 -- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 msgr2=0x7effc01190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 0x7effc01190b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc63eb640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 0x7effc01190b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:17.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc5bea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient(hunting): handle_auth_done global_id 104310 payload 824 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 1 -- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effc0099d70 msgr2=0x7effc01195f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effc0099d70 0x7effc01195f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 1 -- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 msgr2=0x7effc01190b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 0x7effc01190b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient: _finish_hunting 0 2024-09-18T12:33:17.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 1 monclient: found mon.noname-a 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.320+0000 7effc6bec640 1 -- 172.21.15.144:0/1492458057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb8009a20 con 0x7effc009d530 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc6bec640 1 -- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc009d530 msgr2=0x7effc0119b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc6bec640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc009d530 0x7effc0119b30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7effb8031000 tx=0x7effb8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb8031da0 con 0x7effc009d530 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: dump: 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.337 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb8040070 con 0x7effc009d530 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc864f640 5 monclient: authenticate success, global_id 104310 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc864f640 10 monclient: _renew_subs 2024-09-18T12:33:17.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc864f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc00a00e0 con 0x7effc009d530 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb80456d0 con 0x7effc009d530 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effb67fc640 10 monclient: dump: 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:17.339 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc864f640 10 monclient: _renew_subs 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc4be8640 4 set_mon_vals no callback set 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc864f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.322+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc011a9c0 con 0x7effc009d530 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.323+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7effb8042030 con 0x7effc009d530 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.323+0000 7effc864f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.323+0000 7effc864f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.323+0000 7effc864f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.323+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc004bf60 con 0x7effc009d530 2024-09-18T12:33:17.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.323+0000 7effb67fc640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effa4038490 0x7effa403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:17.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.323+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(217..217 src has 13..217) v4 ==== 13204+0+0 (secure 0 0 0) 0x7effb806ef50 con 0x7effc009d530 2024-09-18T12:33:17.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.324+0000 7effc63eb640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effa4038490 0x7effa403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:17.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.324+0000 7effc63eb640 10 monclient: get_auth_request con 0x7effa4038490 auth_method 0 2024-09-18T12:33:17.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.326+0000 7effc63eb640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effa4038490 0x7effa403a940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7effbc007dd0 tx=0x7effbc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:17.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.328+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7effb803abd0 con 0x7effc009d530 2024-09-18T12:33:17.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.328+0000 7effb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:17.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.328+0000 7effb67fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.374+0000 7f4f2cf9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.374+0000 7f4f2cf9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.374+0000 7f4f2cf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.374+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4f28131810 con 0x7f4f2809a8a0 2024-09-18T12:33:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.374+0000 7f4f1e7fc640 1 -- 172.21.15.116:0/2515093088 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f4f1001dbb0 con 0x7f4f2809a8a0 2024-09-18T12:33:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.374+0000 7f4f1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:17.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.374+0000 7f4f1e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:17.376 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":86,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:17.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f08038490 msgr2=0x7f4f0803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f08038490 0x7f4f0803a940 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4f20005140 tx=0x7f4f20009040 comp rx=0 tx=0).stop 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 10 monclient: shutdown 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 msgr2=0x7f4f28130260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 0x7f4f28130260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000d890 tx=0x7f4f10008040 comp rx=0 tx=0).stop 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 shutdown_connections 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4f08038490 0x7f4f0803a940 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4f28133070 0x7f4f281307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2809a8a0 0x7f4f28130260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 --2- 172.21.15.116:0/2515093088 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4f280974f0 0x7f4f2812fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 >> 172.21.15.116:0/2515093088 conn(0x7f4f280672b0 msgr2=0x7f4f281255d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 shutdown_connections 2024-09-18T12:33:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:17.376+0000 7f4f2cf9d640 1 -- 172.21.15.116:0/2515093088 wait complete. 2024-09-18T12:33:17.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.487+0000 7effc864f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:17.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.487+0000 7effc864f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:17.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.487+0000 7effc864f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:17.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:17.487+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7effc009a1d0 con 0x7effc009d530 2024-09-18T12:33:18.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.137+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v218) v1 ==== 168+0+0 (secure 0 0 0) 0x7effc009a1d0 con 0x7effc009d530 2024-09-18T12:33:18.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.137+0000 7effb67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:18.139 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.137+0000 7effb67fc640 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-18T12:33:18.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.233+0000 7effc864f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:18.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.233+0000 7effc864f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:18.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.233+0000 7effc864f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:18.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.233+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7effc009a1d0 con 0x7effc009d530 2024-09-18T12:33:18.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.234+0000 7effb67fc640 1 -- 172.21.15.144:0/1492458057 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v218) v1 ==== 147+0+0 (secure 0 0 0) 0x7effc009a1d0 con 0x7effc009d530 2024-09-18T12:33:18.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.234+0000 7effb67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:18.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.234+0000 7effb67fc640 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-18T12:33:18.236 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h25' already exists 2024-09-18T12:33:18.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effa4038490 msgr2=0x7effa403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:18.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effa4038490 0x7effa403a940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7effbc007dd0 tx=0x7effbc009040 comp rx=0 tx=0).stop 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 10 monclient: shutdown 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc009d530 msgr2=0x7effc0119b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc009d530 0x7effc0119b30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7effb8031000 tx=0x7effb8037000 comp rx=0 tx=0).stop 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 shutdown_connections 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effa4038490 0x7effa403a940 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc009d530 0x7effc0119b30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effc0099d70 0x7effc01195f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 --2- 172.21.15.144:0/1492458057 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effc0096a30 0x7effc01190b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 >> 172.21.15.144:0/1492458057 conn(0x7effc00a0fe0 msgr2=0x7effc009c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 shutdown_connections 2024-09-18T12:33:18.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.236+0000 7effc864f640 1 -- 172.21.15.144:0/1492458057 wait complete. 2024-09-18T12:33:18.246 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:18.246 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-18T12:33:18.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.320+0000 7f543199d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:18.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.320+0000 7f543199d640 10 monclient: build_initial_monmap 2024-09-18T12:33:18.321 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.320+0000 7f543199d640 10 monclient: monmap: 2024-09-18T12:33:18.321 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:18.321 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:18.321 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:18.321172+0000 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:18.321172+0000 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:18.322 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.320+0000 7f543199d640 1 Processor -- start 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.320+0000 7f543199d640 1 -- start start 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.320+0000 7f543199d640 10 monclient: init 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f542c131bd0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 0x7f542c131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient(hunting): picked mon.noname-c con 0x7f542c069e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f542c136d60 0x7f542c139140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient(hunting): picked mon.noname-a con 0x7f542c136d60 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:18.325 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c0a06d0 con 0x7f542c131bd0 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542c071780 con 0x7f542c136d60 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542c06a210 con 0x7f542c069e10 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f543199d640 10 monclient(hunting): authenticate will time out at 1233.890366s 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 0x7f542c131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 0x7f542c131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55216/0 (socket says 172.21.15.144:55216) 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 1 -- 172.21.15.144:0/2168995234 learned_addr learned my addr 172.21.15.144:0/2168995234 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 10 monclient(hunting): get_auth_request con 0x7f542c069e10 auth_method 0 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:18.326 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 1 -- 172.21.15.144:0/2168995234 >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 msgr2=0x7f542c131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 1 -- 172.21.15.144:0/2168995234 >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 msgr2=0x7f542c131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 1 --2- 172.21.15.144:0/2168995234 >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 0x7f542c131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f54227fc640 1 --2- 172.21.15.144:0/2168995234 >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 0x7f542c131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 1 --2- 172.21.15.144:0/2168995234 >> v2:172.21.15.116:3300/0 conn(0x7f542c136d60 0x7f542c139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 10 monclient(hunting): get_auth_request con 0x7f542c136d60 auth_method 0 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.321+0000 7f542b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:18.327 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 10 monclient(hunting): handle_auth_done global_id 104313 payload 274 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 1 -- 172.21.15.144:0/2168995234 >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 msgr2=0x7f542c131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 1 --2- 172.21.15.144:0/2168995234 >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 0x7f542c131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 1 -- 172.21.15.144:0/2168995234 >> v1:172.21.15.116:6789/0 conn(0x7f542c131bd0 legacy=0x7f542c09ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:18.328 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.322+0000 7f542b7fe640 1 -- 172.21.15.144:0/2168995234 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0a0010 con 0x7f542c136d60 2024-09-18T12:33:18.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.324+0000 7f542b7fe640 1 -- 172.21.15.144:0/2168995234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c136d60 msgr2=0x7f542c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:18.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.324+0000 7f542b7fe640 1 --2- 172.21.15.144:0/2168995234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c136d60 0x7f542c139140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f542c068d90 tx=0x7f5424009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853ed16829531448 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:18.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 1 -- 172.21.15.144:0/2168995234 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424007760 con 0x7f542c136d60 2024-09-18T12:33:18.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:18.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: dump: 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:18.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 5 monclient: authenticate success, global_id 104313 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 1 -- 172.21.15.144:0/2168995234 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542402d050 con 0x7f542c136d60 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 4 set_mon_vals no callback set 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 1 -- 172.21.15.144:0/2168995234 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5424007b30 con 0x7f542c136d60 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f542a7fc640 10 monclient: dump: 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:18.337 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 10 monclient: shutdown 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 1 -- 172.21.15.144:0/2168995234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c136d60 msgr2=0x7f542c139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 1 --2- 172.21.15.144:0/2168995234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c136d60 0x7f542c139140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f542c068d90 tx=0x7f5424009780 comp rx=0 tx=0).stop 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 1 -- 172.21.15.144:0/2168995234 shutdown_connections 2024-09-18T12:33:18.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 1 --2- 172.21.15.144:0/2168995234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c136d60 0x7f542c139140 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 1 --2- 172.21.15.144:0/2168995234 >> v2:172.21.15.144:3300/0 conn(0x7f542c069e10 0x7f542c131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 1 -- 172.21.15.144:0/2168995234 >> 172.21.15.144:0/2168995234 conn(0x7f542c0713b0 msgr2=0x7f542c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.325+0000 7f543199d640 1 -- 172.21.15.144:0/2168995234 shutdown_connections 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.326+0000 7f543199d640 1 -- 172.21.15.144:0/2168995234 wait complete. 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.326+0000 7f543199d640 10 monclient: build_initial_monmap 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.326+0000 7f543199d640 10 monclient: monmap: 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:18.339 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:18.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.326+0000 7f543199d640 1 Processor -- start 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.326+0000 7f543199d640 1 -- start start 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.326+0000 7f543199d640 10 monclient: init 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c069e10 0x7f542c141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): picked mon.noname-a con 0x7f542c069e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f542c131bd0 0x7f542c141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): picked mon.noname-c con 0x7f542c131bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 0x7f542c099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:18.341 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): picked mon.noname-b con 0x7f542c136d60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c09b040 con 0x7f542c069e10 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c09bec0 con 0x7f542c136d60 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c09ea60 con 0x7f542c131bd0 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f543199d640 10 monclient(hunting): authenticate will time out at 1233.896385s 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 0x7f542c099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 0x7f542c099580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55232/0 (socket says 172.21.15.144:55232) 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 1 -- 172.21.15.144:0/723324535 learned_addr learned my addr 172.21.15.144:0/723324535 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:18.342 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 10 monclient(hunting): get_auth_request con 0x7f542c136d60 auth_method 0 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 1 -- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 msgr2=0x7f542c099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 1 -- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 msgr2=0x7f542c099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 0x7f542c099580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542b7fe640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 0x7f542c099580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c069e10 0x7f542c141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f542c131bd0 0x7f542c141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 10 monclient(hunting): get_auth_request con 0x7f542c069e10 auth_method 0 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:18.343 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 10 monclient(hunting): get_auth_request con 0x7f542c131bd0 auth_method 0 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f542affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.327+0000 7f54227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 10 monclient(hunting): handle_auth_done global_id 104316 payload 824 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 1 -- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f542c131bd0 msgr2=0x7f542c141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:18.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f542c131bd0 0x7f542c141940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 1 -- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 msgr2=0x7f542c099580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 0x7f542c099580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 1 monclient: found mon.noname-a 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.328+0000 7f542affd640 1 -- 172.21.15.144:0/723324535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424009a20 con 0x7f542c069e10 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f542affd640 1 -- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c069e10 msgr2=0x7f542c141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f542affd640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c069e10 0x7f542c141400 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f541c00c4e0 tx=0x7f541c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c0117a0 con 0x7f542c069e10 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:18.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: dump: 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:18.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c011920 con 0x7f542c069e10 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f543199d640 5 monclient: authenticate success, global_id 104316 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f543199d640 10 monclient: _renew_subs 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c011ce0 con 0x7f542c069e10 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f543199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542c11c7b0 con 0x7f542c069e10 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54297fa640 4 set_mon_vals no callback set 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f54237fe640 10 monclient: dump: 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:18.347 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f543199d640 10 monclient: _renew_subs 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f543199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:18.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.330+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c141e80 con 0x7f542c069e10 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f543199d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f543199d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f543199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f541c012630 con 0x7f542c069e10 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542c04bf60 con 0x7f542c069e10 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f54237fe640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f540c038490 0x7f540c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(218..218 src has 13..218) v4 ==== 13417+0+0 (secure 0 0 0) 0x7f541c04d650 con 0x7f542c069e10 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f54227fc640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f540c038490 0x7f540c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.331+0000 7f54227fc640 10 monclient: get_auth_request con 0x7f540c038490 auth_method 0 2024-09-18T12:33:18.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.334+0000 7f54227fc640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f540c038490 0x7f540c03a940 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f542c09b880 tx=0x7f5414008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:18.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.336+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f541c010900 con 0x7f542c069e10 2024-09-18T12:33:18.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.336+0000 7f54237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:18.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.336+0000 7f54237fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:18.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.493+0000 7f543199d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:18.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.493+0000 7f543199d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:18.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.493+0000 7f543199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:18.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:18.493+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f542c11b810 con 0x7f542c069e10 2024-09-18T12:33:19.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.144+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v219) v1 ==== 168+0+0 (secure 0 0 0) 0x7f541c010320 con 0x7f542c069e10 2024-09-18T12:33:19.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.144+0000 7f54237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:19.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.144+0000 7f54237fe640 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-18T12:33:19.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.239+0000 7f543199d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:19.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.239+0000 7f543199d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:19.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.239+0000 7f543199d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:19.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.239+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f542c064990 con 0x7f542c069e10 2024-09-18T12:33:19.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.240+0000 7f54237fe640 1 -- 172.21.15.144:0/723324535 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v219) v1 ==== 147+0+0 (secure 0 0 0) 0x7f541c04b020 con 0x7f542c069e10 2024-09-18T12:33:19.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.240+0000 7f54237fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:19.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.240+0000 7f54237fe640 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-18T12:33:19.241 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h26' already exists 2024-09-18T12:33:19.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f540c038490 msgr2=0x7f540c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:19.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f540c038490 0x7f540c03a940 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f542c09b880 tx=0x7f5414008040 comp rx=0 tx=0).stop 2024-09-18T12:33:19.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 10 monclient: shutdown 2024-09-18T12:33:19.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:19.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c069e10 msgr2=0x7f542c141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:19.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c069e10 0x7f542c141400 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f541c00c4e0 tx=0x7f541c007e10 comp rx=0 tx=0).stop 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 shutdown_connections 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f540c038490 0x7f540c03a940 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f542c136d60 0x7f542c099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f542c131bd0 0x7f542c141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 --2- 172.21.15.144:0/723324535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f542c069e10 0x7f542c141400 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 >> 172.21.15.144:0/723324535 conn(0x7f542c0713b0 msgr2=0x7f542c0a3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 shutdown_connections 2024-09-18T12:33:19.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.242+0000 7f543199d640 1 -- 172.21.15.144:0/723324535 wait complete. 2024-09-18T12:33:19.254 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:19.254 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-18T12:33:19.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.328+0000 7f1689034640 10 monclient: get_monmap_and_config 2024-09-18T12:33:19.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.328+0000 7f1689034640 10 monclient: build_initial_monmap 2024-09-18T12:33:19.329 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.328+0000 7f1689034640 10 monclient: monmap: 2024-09-18T12:33:19.329 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:19.329 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:19.329 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:19.329149+0000 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:19.329149+0000 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:19.330 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.328+0000 7f1689034640 1 Processor -- start 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.328+0000 7f1689034640 1 -- start start 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.328+0000 7f1689034640 10 monclient: init 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16840972c0 0x7f1684099040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient(hunting): picked mon.noname-b con 0x7f16840972c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1684097cb0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f168409c0e0 0x7f16841357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:19.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient(hunting): picked mon.noname-a con 0x7f168409c0e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:19.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1684097690 con 0x7f1684097cb0 2024-09-18T12:33:19.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:19.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168409d8d0 con 0x7f168409c0e0 2024-09-18T12:33:19.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:19.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1684097a70 con 0x7f16840972c0 2024-09-18T12:33:19.334 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:19.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1689034640 10 monclient(hunting): authenticate will time out at 1234.898503s 2024-09-18T12:33:19.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168259c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1684097cb0 0x7f168409b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34138/0 (socket says 172.21.15.144:34138) 2024-09-18T12:33:19.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168259c640 1 -- 172.21.15.144:0/739405931 learned_addr learned my addr 172.21.15.144:0/739405931 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:19.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 1 --2- 172.21.15.144:0/739405931 >> v2:172.21.15.116:3300/0 conn(0x7f168409c0e0 0x7f16841357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:19.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 1 --2- 172.21.15.144:0/739405931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16840972c0 0x7f1684099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:19.335 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 10 monclient(hunting): get_auth_request con 0x7f168409c0e0 auth_method 0 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 10 monclient(hunting): get_auth_request con 0x7f16840972c0 auth_method 0 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f1682d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:19.336 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.329+0000 7f168359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 10 monclient(hunting): handle_auth_done global_id 65126 payload 274 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 1 -- 172.21.15.144:0/739405931 >> v2:172.21.15.116:3300/0 conn(0x7f168409c0e0 msgr2=0x7f16841357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 1 --2- 172.21.15.144:0/739405931 >> v2:172.21.15.116:3300/0 conn(0x7f168409c0e0 0x7f16841357f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 1 -- 172.21.15.144:0/739405931 >> v1:172.21.15.144:6789/0 conn(0x7f1684097cb0 legacy=0x7f168409b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 1 monclient: found mon.noname-b 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:19.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.330+0000 7f1682d9d640 1 -- 172.21.15.144:0/739405931 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168406ebf0 con 0x7f16840972c0 2024-09-18T12:33:19.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1682d9d640 1 -- 172.21.15.144:0/739405931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16840972c0 msgr2=0x7f1684099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:19.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1682d9d640 1 --2- 172.21.15.144:0/739405931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16840972c0 0x7f1684099040 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1674009850 tx=0x7f16740090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=459b188a7c67112 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:19.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 1 -- 172.21.15.144:0/739405931 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167402add0 con 0x7f16840972c0 2024-09-18T12:33:19.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:19.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:19.344 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: dump: 2024-09-18T12:33:19.344 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: _finish_auth 0 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:19.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 5 monclient: authenticate success, global_id 65126 2024-09-18T12:33:19.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:19.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 1 -- 172.21.15.144:0/739405931 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1674032440 con 0x7f16840972c0 2024-09-18T12:33:19.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:19.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 1 -- 172.21.15.144:0/739405931 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16740327e0 con 0x7f16840972c0 2024-09-18T12:33:19.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:19.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:19.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 4 set_mon_vals no callback set 2024-09-18T12:33:19.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:19.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1681d9b640 10 monclient: dump: 2024-09-18T12:33:19.347 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:19.347 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 10 monclient: shutdown 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:19.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 1 -- 172.21.15.144:0/739405931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16840972c0 msgr2=0x7f1684099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.333+0000 7f1689034640 1 --2- 172.21.15.144:0/739405931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16840972c0 0x7f1684099040 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1674009850 tx=0x7f16740090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 1 -- 172.21.15.144:0/739405931 shutdown_connections 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 1 --2- 172.21.15.144:0/739405931 >> v2:172.21.15.116:3300/0 conn(0x7f168409c0e0 0x7f16841357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 1 --2- 172.21.15.144:0/739405931 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f16840972c0 0x7f1684099040 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 1 -- 172.21.15.144:0/739405931 >> 172.21.15.144:0/739405931 conn(0x7f16840a0fe0 msgr2=0x7f16840a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 1 -- 172.21.15.144:0/739405931 shutdown_connections 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 1 -- 172.21.15.144:0/739405931 wait complete. 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 10 monclient: build_initial_monmap 2024-09-18T12:33:19.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 10 monclient: monmap: 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.334+0000 7f1689034640 1 Processor -- start 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 1 -- start start 2024-09-18T12:33:19.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient: init 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 0x7f168412ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): picked mon.noname-b con 0x7f16840972c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684097cb0 0x7f168413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): picked mon.noname-a con 0x7f1684097cb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168409c0e0 0x7f168413fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): picked mon.noname-c con 0x7f168409c0e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16841307d0 con 0x7f1684097cb0 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:19.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168411a030 con 0x7f16840972c0 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1684136610 con 0x7f168409c0e0 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1689034640 10 monclient(hunting): authenticate will time out at 1234.904596s 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 0x7f168412ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 0x7f168412ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55246/0 (socket says 172.21.15.144:55246) 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 1 -- 172.21.15.144:0/461629239 learned_addr learned my addr 172.21.15.144:0/461629239 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 10 monclient(hunting): get_auth_request con 0x7f16840972c0 auth_method 0 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:19.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 1 -- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 msgr2=0x7f168412ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 1 -- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 msgr2=0x7f168412ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 0x7f168412ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f1682d9d640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 0x7f168412ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168259c640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684097cb0 0x7f168413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168409c0e0 0x7f168413fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168259c640 10 monclient(hunting): get_auth_request con 0x7f1684097cb0 auth_method 0 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 10 monclient(hunting): get_auth_request con 0x7f168409c0e0 auth_method 0 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:19.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.335+0000 7f168259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 10 monclient(hunting): handle_auth_done global_id 65129 payload 824 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 1 -- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 msgr2=0x7f168412ff30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 0x7f168412ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 1 -- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684097cb0 msgr2=0x7f168413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684097cb0 0x7f168413f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 10 monclient: _finish_hunting 0 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 1 monclient: found mon.noname-c 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.336+0000 7f168359e640 1 -- 172.21.15.144:0/461629239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1674009390 con 0x7f168409c0e0 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f168359e640 1 -- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168409c0e0 msgr2=0x7f168413fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f168359e640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168409c0e0 0x7f168413fb00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f167c009e80 tx=0x7f167c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c014480 con 0x7f168409c0e0 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: dump: 2024-09-18T12:33:19.355 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:19.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: _finish_auth 0 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f1689034640 5 monclient: authenticate success, global_id 65129 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f1689034640 10 monclient: _renew_subs 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c00f040 con 0x7f168409c0e0 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f1689034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1684119d50 con 0x7f168409c0e0 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c014830 con 0x7f168409c0e0 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f166affd640 10 monclient: dump: 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:19.357 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:19.359 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:19.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f168159a640 4 set_mon_vals no callback set 2024-09-18T12:33:19.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f1689034640 10 monclient: _renew_subs 2024-09-18T12:33:19.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f1689034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:19.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.338+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1684135eb0 con 0x7f168409c0e0 2024-09-18T12:33:19.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.339+0000 7f1689034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:19.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.339+0000 7f1689034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:19.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.339+0000 7f1689034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:19.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.339+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168404bf60 con 0x7f168409c0e0 2024-09-18T12:33:19.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.339+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f167c014ba0 con 0x7f168409c0e0 2024-09-18T12:33:19.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.339+0000 7f166affd640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1670038490 0x7f167003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:19.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.339+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(219..219 src has 13..219) v4 ==== 13630+0+0 (secure 0 0 0) 0x7f167c04e8c0 con 0x7f168409c0e0 2024-09-18T12:33:19.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.340+0000 7f1682d9d640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1670038490 0x7f167003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:19.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.340+0000 7f1682d9d640 10 monclient: get_auth_request con 0x7f1670038490 auth_method 0 2024-09-18T12:33:19.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.342+0000 7f1682d9d640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1670038490 0x7f167003a940 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1674035040 tx=0x7f1674032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:19.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.344+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f167c024530 con 0x7f168409c0e0 2024-09-18T12:33:19.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.344+0000 7f166affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:19.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.344+0000 7f166affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:19.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.501+0000 7f1689034640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:19.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.501+0000 7f1689034640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:19.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.501+0000 7f1689034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:19.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:19.501+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1684064b40 con 0x7f168409c0e0 2024-09-18T12:33:20.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.144+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v220) v1 ==== 168+0+0 (secure 0 0 0) 0x7f167c018070 con 0x7f168409c0e0 2024-09-18T12:33:20.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.144+0000 7f166affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:20.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.144+0000 7f166affd640 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-18T12:33:20.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.243+0000 7f1689034640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:20.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.243+0000 7f1689034640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:20.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.243+0000 7f1689034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:20.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.243+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f1684068ee0 con 0x7f168409c0e0 2024-09-18T12:33:20.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.244+0000 7f166affd640 1 -- 172.21.15.144:0/461629239 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v220) v1 ==== 147+0+0 (secure 0 0 0) 0x7f167c007e60 con 0x7f168409c0e0 2024-09-18T12:33:20.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.244+0000 7f166affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:20.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.244+0000 7f166affd640 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-18T12:33:20.245 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h27' already exists 2024-09-18T12:33:20.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1670038490 msgr2=0x7f167003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1670038490 0x7f167003a940 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1674035040 tx=0x7f1674032000 comp rx=0 tx=0).stop 2024-09-18T12:33:20.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 10 monclient: shutdown 2024-09-18T12:33:20.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:20.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168409c0e0 msgr2=0x7f168413fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168409c0e0 0x7f168413fb00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f167c009e80 tx=0x7f167c00b040 comp rx=0 tx=0).stop 2024-09-18T12:33:20.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 shutdown_connections 2024-09-18T12:33:20.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1670038490 0x7f167003a940 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f168409c0e0 0x7f168413fb00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1684097cb0 0x7f168413f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 --2- 172.21.15.144:0/461629239 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16840972c0 0x7f168412ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 >> 172.21.15.144:0/461629239 conn(0x7f16840a0fe0 msgr2=0x7f168409dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:20.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 shutdown_connections 2024-09-18T12:33:20.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.246+0000 7f1689034640 1 -- 172.21.15.144:0/461629239 wait complete. 2024-09-18T12:33:20.257 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:20.257 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-18T12:33:20.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.332+0000 7f7bc819d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:20.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.332+0000 7f7bc819d640 10 monclient: build_initial_monmap 2024-09-18T12:33:20.333 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.332+0000 7f7bc819d640 10 monclient: monmap: 2024-09-18T12:33:20.333 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:20.333509+0000 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:20.333509+0000 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:20.334 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:20.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 1 Processor -- start 2024-09-18T12:33:20.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 1 -- start start 2024-09-18T12:33:20.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient: init 2024-09-18T12:33:20.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:20.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T12:33:20.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc009c2b0 0x7f7bc009a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.337 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient(hunting): picked mon.noname-b con 0x7f7bc009c2b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7bc009c680 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 0x7f7bc0136b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc0134780 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc009bb40 con 0x7f7bc009c680 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc0099f60 con 0x7f7bc0134780 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc009f700 con 0x7f7bc009c2b0 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc819d640 10 monclient(hunting): authenticate will time out at 1235.902866s 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.333+0000 7f7bc673a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 0x7f7bc0136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 0x7f7bc0136b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55250/0 (socket says 172.21.15.144:55250) 2024-09-18T12:33:20.338 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 1 -- 172.21.15.144:0/2206269355 learned_addr learned my addr 172.21.15.144:0/2206269355 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 10 monclient(hunting): get_auth_request con 0x7f7bc0134780 auth_method 0 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 1 -- 172.21.15.144:0/2206269355 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 msgr2=0x7f7bc0136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 1 -- 172.21.15.144:0/2206269355 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 msgr2=0x7f7bc0136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 1 --2- 172.21.15.144:0/2206269355 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 0x7f7bc0136b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc673a640 1 --2- 172.21.15.144:0/2206269355 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 0x7f7bc0136b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 1 --2- 172.21.15.144:0/2206269355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc009c2b0 0x7f7bc009a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): get_auth_request con 0x7f7bc009c2b0 auth_method 0 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.339 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_done global_id 65132 payload 274 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 1 -- 172.21.15.144:0/2206269355 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 msgr2=0x7f7bc0136b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 1 --2- 172.21.15.144:0/2206269355 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 0x7f7bc0136b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 1 -- 172.21.15.144:0/2206269355 >> v1:172.21.15.116:6789/0 conn(0x7f7bc009c680 legacy=0x7f7bc009ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient: _finish_hunting 0 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 1 monclient: found mon.noname-b 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:20.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.334+0000 7f7bc5f39640 1 -- 172.21.15.144:0/2206269355 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc009b4f0 con 0x7f7bc009c2b0 2024-09-18T12:33:20.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc5f39640 1 -- 172.21.15.144:0/2206269355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc009c2b0 msgr2=0x7f7bc009a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc5f39640 1 --2- 172.21.15.144:0/2206269355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc009c2b0 0x7f7bc009a7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009850 tx=0x7f7bbc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc54a19cff37c2d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 1 -- 172.21.15.144:0/2206269355 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc02add0 con 0x7f7bc009c2b0 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: dump: 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.347 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: _finish_auth 0 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 1 -- 172.21.15.144:0/2206269355 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc032440 con 0x7f7bc009c2b0 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 1 -- 172.21.15.144:0/2206269355 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc0327e0 con 0x7f7bc009c2b0 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc819d640 5 monclient: authenticate success, global_id 65132 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc819d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc819d640 4 set_mon_vals no callback set 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:20.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.337+0000 7f7bc4f37640 10 monclient: dump: 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 10 monclient: shutdown 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:20.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 -- 172.21.15.144:0/2206269355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc009c2b0 msgr2=0x7f7bc009a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 --2- 172.21.15.144:0/2206269355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc009c2b0 0x7f7bc009a7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009850 tx=0x7f7bbc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 -- 172.21.15.144:0/2206269355 shutdown_connections 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 --2- 172.21.15.144:0/2206269355 >> v2:172.21.15.144:3300/0 conn(0x7f7bc0134780 0x7f7bc0136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 --2- 172.21.15.144:0/2206269355 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc009c2b0 0x7f7bc009a7a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 -- 172.21.15.144:0/2206269355 >> 172.21.15.144:0/2206269355 conn(0x7f7bc00713b0 msgr2=0x7f7bc00a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 -- 172.21.15.144:0/2206269355 shutdown_connections 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 1 -- 172.21.15.144:0/2206269355 wait complete. 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 10 monclient: build_initial_monmap 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.338+0000 7f7bc819d640 10 monclient: monmap: 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:20.350 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 Processor -- start 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 -- start start 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient: init 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc009c2b0 0x7f7bc0119d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): picked mon.noname-a con 0x7f7bc009c2b0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 0x7f7bc011a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): picked mon.noname-b con 0x7f7bc009c680 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc0134780 0x7f7bc011a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc0134780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc006e9e0 con 0x7f7bc009c2b0 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc01364d0 con 0x7f7bc009c680 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc011d5e0 con 0x7f7bc0134780 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc819d640 10 monclient(hunting): authenticate will time out at 1235.908550s 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 0x7f7bc011a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 0x7f7bc011a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55256/0 (socket says 172.21.15.144:55256) 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 1 -- 172.21.15.144:0/644612764 learned_addr learned my addr 172.21.15.144:0/644612764 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 10 monclient(hunting): get_auth_request con 0x7f7bc009c680 auth_method 0 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 1 -- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 msgr2=0x7f7bc011a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 1 -- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 msgr2=0x7f7bc011a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 0x7f7bc011a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5738640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 0x7f7bc011a2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc009c2b0 0x7f7bc0119d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc0134780 0x7f7bc011a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 10 monclient(hunting): get_auth_request con 0x7f7bc009c2b0 auth_method 0 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 10 monclient(hunting): get_auth_request con 0x7f7bc0134780 auth_method 0 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:20.354 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.339+0000 7f7bc673a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 10 monclient(hunting): handle_auth_done global_id 104325 payload 824 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 1 -- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc0134780 msgr2=0x7f7bc011a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc0134780 0x7f7bc011a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 1 -- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 msgr2=0x7f7bc011a2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 0x7f7bc011a2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 10 monclient: _finish_hunting 0 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 1 monclient: found mon.noname-a 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc5f39640 1 -- 172.21.15.144:0/644612764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc009390 con 0x7f7bc009c2b0 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.340+0000 7f7bc673a640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc0134780 0x7f7bc011a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:20.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bc5f39640 1 -- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc009c2b0 msgr2=0x7f7bc0119d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bc5f39640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc009c2b0 0x7f7bc0119d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc031000 tx=0x7f7bbc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc031dc0 con 0x7f7bc009c2b0 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: dump: 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.356 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bc819d640 5 monclient: authenticate success, global_id 104325 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bc819d640 10 monclient: _renew_subs 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bc819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc011c3a0 con 0x7f7bc009c2b0 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc03d440 con 0x7f7bc009c2b0 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc03d800 con 0x7f7bc009c2b0 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb5ffb640 10 monclient: dump: 2024-09-18T12:33:20.357 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.342+0000 7f7bb7fff640 4 set_mon_vals no callback set 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.343+0000 7f7bc819d640 10 monclient: _renew_subs 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.343+0000 7f7bc819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.343+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc0134390 con 0x7f7bc009c2b0 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.343+0000 7f7bc819d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.343+0000 7f7bc819d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.343+0000 7f7bc819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.343+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc011d800 con 0x7f7bc009c2b0 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.344+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7bbc039040 con 0x7f7bc009c2b0 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.344+0000 7f7bb5ffb640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ba40384c0 0x7f7ba403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.344+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(220..220 src has 13..220) v4 ==== 13843+0+0 (secure 0 0 0) 0x7f7bbc06dc60 con 0x7f7bc009c2b0 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.344+0000 7f7bc5738640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ba40384c0 0x7f7ba403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.344+0000 7f7bc5738640 10 monclient: get_auth_request con 0x7f7ba40384c0 auth_method 0 2024-09-18T12:33:20.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.346+0000 7f7bc5738640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ba40384c0 0x7f7ba403a970 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7bc00638c0 tx=0x7f7bb0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:20.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.349+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7bbc044580 con 0x7f7bc009c2b0 2024-09-18T12:33:20.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.349+0000 7f7bb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:20.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.349+0000 7f7bb5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:20.388 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.492+0000 7f917135d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.492+0000 7f917135d640 10 monclient: build_initial_monmap 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.492+0000 7f917135d640 10 monclient: monmap: 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:20.493731+0000 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:20.493731+0000 2024-09-18T12:33:20.494 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.492+0000 7f917135d640 1 Processor -- start 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.492+0000 7f917135d640 1 -- start start 2024-09-18T12:33:20.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.492+0000 7f917135d640 10 monclient: init 2024-09-18T12:33:20.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f916c1318f0 0x7f916c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient(hunting): picked mon.noname-a con 0x7f916c1318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c09ea50 0x7f916c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient(hunting): picked mon.noname-b con 0x7f916c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f916c068d90 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c131cc0 con 0x7f916c068d90 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c09fbf0 con 0x7f916c1318f0 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c09fe30 con 0x7f916c09ea50 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f917135d640 10 monclient(hunting): authenticate will time out at 1227.815266s 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c09ea50 0x7f916c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c09ea50 0x7f916c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45928/0 (socket says 172.21.15.116:45928) 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f916c068d90 0x7f916c0691f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44002/0 (socket says 172.21.15.116:44002) 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f916c1318f0 0x7f916c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 1 -- 172.21.15.116:0/3289791265 learned_addr learned my addr 172.21.15.116:0/3289791265 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 10 monclient(hunting): get_auth_request con 0x7f916c09ea50 auth_method 0 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): get_auth_request con 0x7f916c1318f0 auth_method 0 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient(hunting): handle_auth_done global_id 104328 payload 274 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 1 -- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c09ea50 msgr2=0x7f916c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 1 --2- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c09ea50 0x7f916c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 1 -- 172.21.15.116:0/3289791265 >> v1:172.21.15.116:6789/0 conn(0x7f916c068d90 legacy=0x7f916c0691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 1 monclient: found mon.noname-a 2024-09-18T12:33:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:20.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916affd640 1 -- 172.21.15.116:0/3289791265 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0a1210 con 0x7f916c1318f0 2024-09-18T12:33:20.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.493+0000 7f916a7fc640 1 --2- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c09ea50 0x7f916c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:20.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f916affd640 1 -- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c1318f0 msgr2=0x7f916c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f916affd640 1 --2- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c1318f0 0x7f916c131f30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9164009e60 tx=0x7f9164009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d30504c094cfb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 1 -- 172.21.15.116:0/3289791265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164007760 con 0x7f916c1318f0 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: dump: 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:20.507 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f917135d640 5 monclient: authenticate success, global_id 104328 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f917135d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 1 -- 172.21.15.116:0/3289791265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916402d050 con 0x7f916c1318f0 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 1 -- 172.21.15.116:0/3289791265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164007af0 con 0x7f916c1318f0 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f917135d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f917135d640 4 set_mon_vals no callback set 2024-09-18T12:33:20.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.496+0000 7f9169ffb640 10 monclient: dump: 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 10 monclient: shutdown 2024-09-18T12:33:20.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 -- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c1318f0 msgr2=0x7f916c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 --2- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c1318f0 0x7f916c131f30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9164009e60 tx=0x7f9164009780 comp rx=0 tx=0).stop 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 -- 172.21.15.116:0/3289791265 shutdown_connections 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 --2- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c09ea50 0x7f916c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 --2- 172.21.15.116:0/3289791265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c1318f0 0x7f916c131f30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 -- 172.21.15.116:0/3289791265 >> 172.21.15.116:0/3289791265 conn(0x7f916c067110 msgr2=0x7f916c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 -- 172.21.15.116:0/3289791265 shutdown_connections 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.497+0000 7f917135d640 1 -- 172.21.15.116:0/3289791265 wait complete. 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.498+0000 7f917135d640 10 monclient: build_initial_monmap 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.498+0000 7f917135d640 10 monclient: monmap: 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:20.510 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.498+0000 7f917135d640 1 Processor -- start 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.498+0000 7f917135d640 1 -- start start 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.498+0000 7f917135d640 10 monclient: init 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c068d90 0x7f916c09a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): picked mon.noname-c con 0x7f916c068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 0x7f916c130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): picked mon.noname-a con 0x7f916c09ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 0x7f916c11b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): picked mon.noname-b con 0x7f916c1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c11df10 con 0x7f916c09ea50 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c11e1f0 con 0x7f916c1318f0 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c09b420 con 0x7f916c068d90 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f917135d640 10 monclient(hunting): authenticate will time out at 1227.821110s 2024-09-18T12:33:20.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 0x7f916c130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c068d90 0x7f916c09a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 0x7f916c130b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55468/0 (socket says 172.21.15.116:55468) 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 -- 172.21.15.116:0/1026314593 learned_addr learned my addr 172.21.15.116:0/1026314593 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): get_auth_request con 0x7f916c09ea50 auth_method 0 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 10 monclient(hunting): get_auth_request con 0x7f916c068d90 auth_method 0 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:20.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 0x7f916c11b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 10 monclient(hunting): get_auth_request con 0x7f916c1318f0 auth_method 0 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient(hunting): handle_auth_done global_id 104331 payload 824 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 1 -- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 msgr2=0x7f916c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 1 -- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 msgr2=0x7f916c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 -- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c068d90 msgr2=0x7f916c09a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c068d90 0x7f916c09a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 0x7f916c11b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 -- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 msgr2=0x7f916c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916b7fe640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 0x7f916c11b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 0x7f916c11b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 monclient: found mon.noname-a 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916a7fc640 1 -- 172.21.15.116:0/1026314593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9154009450 con 0x7f916c09ea50 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.499+0000 7f916affd640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c068d90 0x7f916c09a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f916a7fc640 1 -- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 msgr2=0x7f916c130b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f916a7fc640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 0x7f916c130b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f915400ec80 tx=0x7f91540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 1 -- 172.21.15.116:0/1026314593 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154009730 con 0x7f916c09ea50 2024-09-18T12:33:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: dump: 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:20.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f917135d640 5 monclient: authenticate success, global_id 104331 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f917135d640 10 monclient: _renew_subs 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f917135d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9164009a20 con 0x7f916c09ea50 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 1 -- 172.21.15.116:0/1026314593 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915400eec0 con 0x7f916c09ea50 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 1 -- 172.21.15.116:0/1026314593 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9154013470 con 0x7f916c09ea50 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f9162ffd640 10 monclient: dump: 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:20.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f917135d640 10 monclient: _renew_subs 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.501+0000 7f91697fa640 4 set_mon_vals no callback set 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f917135d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c11ccd0 con 0x7f916c09ea50 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f917135d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:20.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f917135d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:20.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f917135d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f9162ffd640 1 -- 172.21.15.116:0/1026314593 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9154013820 con 0x7f916c09ea50 2024-09-18T12:33:20.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.502+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c09b930 con 0x7f916c09ea50 2024-09-18T12:33:20.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.503+0000 7f9162ffd640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f914c038490 0x7f914c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:20.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.503+0000 7f9162ffd640 1 -- 172.21.15.116:0/1026314593 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(220..220 src has 13..220) v4 ==== 13843+0+0 (secure 0 0 0) 0x7f915404e010 con 0x7f916c09ea50 2024-09-18T12:33:20.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.503+0000 7f916affd640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f914c038490 0x7f914c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:20.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.503+0000 7f916affd640 10 monclient: get_auth_request con 0x7f914c038490 auth_method 0 2024-09-18T12:33:20.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.506+0000 7f7bc819d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:20.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.506+0000 7f7bc819d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:20.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.506+0000 7f7bc819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.506+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7bc004bf60 con 0x7f7bc009c2b0 2024-09-18T12:33:20.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.505+0000 7f916affd640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f914c038490 0x7f914c03a940 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f916c064050 tx=0x7f9164037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:20.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.507+0000 7f9162ffd640 1 -- 172.21.15.116:0/1026314593 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9154080030 con 0x7f916c09ea50 2024-09-18T12:33:20.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.507+0000 7f9162ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:20.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.507+0000 7f9162ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.725+0000 7f917135d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.725+0000 7f917135d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.725+0000 7f917135d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.725+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f916c04bf60 con 0x7f916c09ea50 2024-09-18T12:33:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.726+0000 7f9162ffd640 1 -- 172.21.15.116:0/1026314593 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9154021070 con 0x7f916c09ea50 2024-09-18T12:33:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.726+0000 7f9162ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.726+0000 7f9162ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:20.728 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":86,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f914c038490 msgr2=0x7f914c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f914c038490 0x7f914c03a940 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f916c064050 tx=0x7f9164037000 comp rx=0 tx=0).stop 2024-09-18T12:33:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 10 monclient: shutdown 2024-09-18T12:33:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 msgr2=0x7f916c130b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 0x7f916c130b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f915400ec80 tx=0x7f91540091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 shutdown_connections 2024-09-18T12:33:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f914c038490 0x7f914c03a940 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f916c1318f0 0x7f916c11b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c09ea50 0x7f916c130b10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 --2- 172.21.15.116:0/1026314593 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f916c068d90 0x7f916c09a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 >> 172.21.15.116:0/1026314593 conn(0x7f916c067110 msgr2=0x7f916c09d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 shutdown_connections 2024-09-18T12:33:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:20.728+0000 7f917135d640 1 -- 172.21.15.116:0/1026314593 wait complete. 2024-09-18T12:33:20.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.980+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v221) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7bbc041020 con 0x7f7bc009c2b0 2024-09-18T12:33:20.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.980+0000 7f7bb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:20.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:20.980+0000 7f7bb5ffb640 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-18T12:33:21.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.076+0000 7f7bc819d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:21.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.076+0000 7f7bc819d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:21.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.076+0000 7f7bc819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:21.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.076+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7bc004bf60 con 0x7f7bc009c2b0 2024-09-18T12:33:21.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.077+0000 7f7bb5ffb640 1 -- 172.21.15.144:0/644612764 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v221) v1 ==== 147+0+0 (secure 0 0 0) 0x7f7bbc037070 con 0x7f7bc009c2b0 2024-09-18T12:33:21.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.077+0000 7f7bb5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:21.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.077+0000 7f7bb5ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-18T12:33:21.078 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h28' already exists 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ba40384c0 msgr2=0x7f7ba403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ba40384c0 0x7f7ba403a970 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7bc00638c0 tx=0x7f7bb0009040 comp rx=0 tx=0).stop 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 10 monclient: shutdown 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc009c2b0 msgr2=0x7f7bc0119d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc009c2b0 0x7f7bc0119d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc031000 tx=0x7f7bbc032000 comp rx=0 tx=0).stop 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 shutdown_connections 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ba40384c0 0x7f7ba403a970 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7bc0134780 0x7f7bc011a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7bc009c680 0x7f7bc011a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 --2- 172.21.15.144:0/644612764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc009c2b0 0x7f7bc0119d80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 >> 172.21.15.144:0/644612764 conn(0x7f7bc00713b0 msgr2=0x7f7bc009ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:21.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.079+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 shutdown_connections 2024-09-18T12:33:21.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.080+0000 7f7bc819d640 1 -- 172.21.15.144:0/644612764 wait complete. 2024-09-18T12:33:21.089 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:21.090 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-18T12:33:21.166 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.165+0000 7fcc8659d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.165+0000 7fcc8659d640 10 monclient: build_initial_monmap 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.165+0000 7fcc8659d640 10 monclient: monmap: 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:21.166740+0000 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:21.166740+0000 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:21.167 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:21.169 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.166+0000 7fcc8659d640 1 Processor -- start 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.166+0000 7fcc8659d640 1 -- start start 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.166+0000 7fcc8659d640 10 monclient: init 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 0x7fcc80131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc80131ac0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc8009ff50 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc801323d0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc801327a0 con 0x7fcc8009ff50 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc800a1330 con 0x7fcc801323d0 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc80098860 con 0x7fcc80131ac0 2024-09-18T12:33:21.170 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc8659d640 10 monclient(hunting): authenticate will time out at 1236.736257s 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 0x7fcc80131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 0x7fcc80131e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55272/0 (socket says 172.21.15.144:55272) 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc84b3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fcc801323d0 0x7fcc80138e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34154/0 (socket says 172.21.15.144:34154) 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 1 -- 172.21.15.144:0/2366872150 learned_addr learned my addr 172.21.15.144:0/2366872150 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 10 monclient(hunting): get_auth_request con 0x7fcc80131ac0 auth_method 0 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 1 -- 172.21.15.144:0/2366872150 >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 msgr2=0x7fcc80131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 1 -- 172.21.15.144:0/2366872150 >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 msgr2=0x7fcc80131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 1 --2- 172.21.15.144:0/2366872150 >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 0x7fcc80131e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:21.171 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.167+0000 7fcc7ffff640 1 --2- 172.21.15.144:0/2366872150 >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 0x7fcc80131e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.169+0000 7fcc84b3a640 1 -- 172.21.15.144:0/2366872150 >> v1:172.21.15.144:6789/0 conn(0x7fcc801323d0 legacy=0x7fcc80138e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.169+0000 7fcc77fff640 1 -- 172.21.15.144:0/2366872150 >> v1:172.21.15.116:6789/0 conn(0x7fcc8009ff50 legacy=0x7fcc80098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.169+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397928236 0 0) 0x7fcc801327a0 con 0x7fcc8009ff50 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.169+0000 7fcc7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.169+0000 7fcc7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.169+0000 7fcc7f7fe640 10 monclient(hunting): my global_id is 104334 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.169+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6c0034e0 con 0x7fcc8009ff50 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124149208 0 0) 0x7fcc6c0034e0 con 0x7fcc8009ff50 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 msgr2=0x7fcc80131e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 --2- 172.21.15.144:0/2366872150 >> v2:172.21.15.144:3300/0 conn(0x7fcc80131ac0 0x7fcc80131e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 >> v1:172.21.15.144:6789/0 conn(0x7fcc801323d0 legacy=0x7fcc80138e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:21.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800989a0 con 0x7fcc8009ff50 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fcc680018b0 con 0x7fcc8009ff50 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc8659d640 5 monclient: authenticate success, global_id 104334 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc8659d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: dump: 2024-09-18T12:33:21.183 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:21.184 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 >> v1:172.21.15.116:6789/0 conn(0x7fcc8009ff50 legacy=0x7fcc80098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc6c003af0 0x7fcc6c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): picked mon.c con 0x7fcc6c003af0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6c004780 0x7fcc6c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): picked mon.b con 0x7fcc6c004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c005380 0x7fcc6c00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): picked mon.a con 0x7fcc6c005380 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80098860 con 0x7fcc6c005380 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c00c350 con 0x7fcc6c004780 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:21.185 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c00c510 con 0x7fcc6c003af0 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc8659d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6c004780 0x7fcc6c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): get_auth_request con 0x7fcc6c004780 auth_method 0 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc6c003af0 0x7fcc6c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c005380 0x7fcc6c00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:21.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 10 monclient(hunting): get_auth_request con 0x7fcc6c003af0 auth_method 0 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 10 monclient(hunting): get_auth_request con 0x7fcc6c005380 auth_method 0 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:21.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient(hunting): handle_auth_done global_id 104334 payload 293 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 1 -- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc6c003af0 msgr2=0x7fcc6c003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc6c003af0 0x7fcc6c003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 1 -- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c005380 msgr2=0x7fcc6c00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c005380 0x7fcc6c00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 1 monclient: found mon.b 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc84b3a640 1 -- 172.21.15.144:0/2366872150 reap_dead start 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 1 -- 172.21.15.144:0/2366872150 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0034e0 con 0x7fcc6c004780 2024-09-18T12:33:21.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 10 monclient: _finish_auth 0 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.170+0000 7fcc77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc77fff640 1 -- 172.21.15.144:0/2366872150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6c004780 msgr2=0x7fcc6c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc77fff640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6c004780 0x7fcc6c004bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c5c0 tx=0x7fcc68007a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a43152b5b3977e93 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc680070f0 con 0x7fcc6c004780 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 10 monclient: dump: 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:21.189 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc68007270 con 0x7fcc6c004780 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc8659d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc8659d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 1 -- 172.21.15.144:0/2366872150 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68007420 con 0x7fcc6c004780 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc8659d640 4 set_mon_vals no callback set 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.173+0000 7fcc7f7fe640 10 monclient: dump: 2024-09-18T12:33:21.190 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 10 monclient: shutdown 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 -- 172.21.15.144:0/2366872150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6c004780 msgr2=0x7fcc6c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6c004780 0x7fcc6c004bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c5c0 tx=0x7fcc68007a60 comp rx=0 tx=0).stop 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 -- 172.21.15.144:0/2366872150 shutdown_connections 2024-09-18T12:33:21.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 --2- 172.21.15.144:0/2366872150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc6c004780 0x7fcc6c004bf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 -- 172.21.15.144:0/2366872150 >> 172.21.15.144:0/2366872150 conn(0x7fcc800a5190 msgr2=0x7fcc800a5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 -- 172.21.15.144:0/2366872150 shutdown_connections 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 -- 172.21.15.144:0/2366872150 wait complete. 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 10 monclient: build_initial_monmap 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 10 monclient: monmap: 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:21.192 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 Processor -- start 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 1 -- start start 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.174+0000 7fcc8659d640 10 monclient: init 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c004780 0x7fcc80140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc6c004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 0x7fcc80130d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): picked mon.noname-b con 0x7fcc80141530 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:21.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc801312a0 0x7fcc8011ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc801312a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80141900 con 0x7fcc6c004780 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80098e70 con 0x7fcc80141530 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8011cec0 con 0x7fcc801312a0 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc8659d640 10 monclient(hunting): authenticate will time out at 1236.744410s 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 0x7fcc80130d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 0x7fcc80130d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55296/0 (socket says 172.21.15.144:55296) 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 1 -- 172.21.15.144:0/4164288183 learned_addr learned my addr 172.21.15.144:0/4164288183 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 10 monclient(hunting): get_auth_request con 0x7fcc80141530 auth_method 0 2024-09-18T12:33:21.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 1 -- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 msgr2=0x7fcc80130d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 1 -- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 msgr2=0x7fcc80130d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 0x7fcc80130d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc77fff640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 0x7fcc80130d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc801312a0 0x7fcc8011ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c004780 0x7fcc80140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): get_auth_request con 0x7fcc801312a0 auth_method 0 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:21.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 10 monclient(hunting): get_auth_request con 0x7fcc6c004780 auth_method 0 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient(hunting): handle_auth_done global_id 65144 payload 824 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 1 -- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 msgr2=0x7fcc80130d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:21.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 0x7fcc80130d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 1 -- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c004780 msgr2=0x7fcc80140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c004780 0x7fcc80140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 1 monclient: found mon.noname-c 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc84b3a640 1 -- 172.21.15.144:0/4164288183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8011f170 con 0x7fcc801312a0 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.175+0000 7fcc7ffff640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c004780 0x7fcc80140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc84b3a640 1 -- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc801312a0 msgr2=0x7fcc8011ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc84b3a640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc801312a0 0x7fcc8011ba50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcc800a07c0 tx=0x7fcc780018d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78014450 con 0x7fcc801312a0 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:21.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: dump: 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc8659d640 5 monclient: authenticate success, global_id 65144 2024-09-18T12:33:21.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc780145b0 con 0x7fcc801312a0 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc8659d640 10 monclient: _renew_subs 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc8659d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8011df30 con 0x7fcc801312a0 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78014970 con 0x7fcc801312a0 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7cff9640 10 monclient: dump: 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:21.199 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc8659d640 10 monclient: _renew_subs 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc8659d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8011bf90 con 0x7fcc801312a0 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.178+0000 7fcc7effd640 4 set_mon_vals no callback set 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fcc7801d3b0 con 0x7fcc801312a0 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc8659d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc8659d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc8659d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:21.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8004bf60 con 0x7fcc801312a0 2024-09-18T12:33:21.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc7cff9640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc60038490 0x7fcc6003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:21.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(221..221 src has 13..221) v4 ==== 14056+0+0 (secure 0 0 0) 0x7fcc7804d0c0 con 0x7fcc801312a0 2024-09-18T12:33:21.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc7ffff640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc60038490 0x7fcc6003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:21.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.179+0000 7fcc7ffff640 10 monclient: get_auth_request con 0x7fcc60038490 auth_method 0 2024-09-18T12:33:21.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.182+0000 7fcc7ffff640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc60038490 0x7fcc6003a940 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcc70006a70 tx=0x7fcc70008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:21.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.184+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fcc7801b8a0 con 0x7fcc801312a0 2024-09-18T12:33:21.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.184+0000 7fcc7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:21.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.184+0000 7fcc7cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:21.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.339+0000 7fcc8659d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:21.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.339+0000 7fcc8659d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:21.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.339+0000 7fcc8659d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:21.340 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:21.339+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcc80098b90 con 0x7fcc801312a0 2024-09-18T12:33:22.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.154+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v222) v1 ==== 168+0+0 (secure 0 0 0) 0x7fcc78019020 con 0x7fcc801312a0 2024-09-18T12:33:22.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.154+0000 7fcc7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:22.155 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.154+0000 7fcc7cff9640 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-18T12:33:22.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.255+0000 7fcc8659d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:22.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.256+0000 7fcc8659d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:22.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.256+0000 7fcc8659d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:22.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.256+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fcc80072640 con 0x7fcc801312a0 2024-09-18T12:33:22.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.257+0000 7fcc7cff9640 1 -- 172.21.15.144:0/4164288183 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v222) v1 ==== 147+0+0 (secure 0 0 0) 0x7fcc7801b2c0 con 0x7fcc801312a0 2024-09-18T12:33:22.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.257+0000 7fcc7cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:22.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.257+0000 7fcc7cff9640 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-18T12:33:22.258 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h29' already exists 2024-09-18T12:33:22.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc60038490 msgr2=0x7fcc6003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:22.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc60038490 0x7fcc6003a940 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcc70006a70 tx=0x7fcc70008040 comp rx=0 tx=0).stop 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 10 monclient: shutdown 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc801312a0 msgr2=0x7fcc8011ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc801312a0 0x7fcc8011ba50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcc800a07c0 tx=0x7fcc780018d0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 shutdown_connections 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcc60038490 0x7fcc6003a940 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcc801312a0 0x7fcc8011ba50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcc80141530 0x7fcc80130d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 --2- 172.21.15.144:0/4164288183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc6c004780 0x7fcc80140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 >> 172.21.15.144:0/4164288183 conn(0x7fcc800a5190 msgr2=0x7fcc8009b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 shutdown_connections 2024-09-18T12:33:22.261 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.259+0000 7fcc8659d640 1 -- 172.21.15.144:0/4164288183 wait complete. 2024-09-18T12:33:22.272 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:22.272 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.345+0000 7fd79475d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.345+0000 7fd79475d640 10 monclient: build_initial_monmap 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.345+0000 7fd79475d640 10 monclient: monmap: 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:22.346137+0000 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:22.346137+0000 2024-09-18T12:33:22.346 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:22.347 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:22.347 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:22.347 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:22.347 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:22.347 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:22.347 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:22.347 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:22.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.345+0000 7fd79475d640 1 Processor -- start 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.345+0000 7fd79475d640 1 -- start start 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.345+0000 7fd79475d640 10 monclient: init 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 0x7fd78c131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient(hunting): picked mon.noname-c con 0x7fd78c131bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd78c09ebe0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd78c09efe0 0x7fd78c069510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient(hunting): picked mon.noname-a con 0x7fd78c09efe0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c0a06d0 con 0x7fd78c09ebe0 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:22.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78c071780 con 0x7fd78c09efe0 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd78c0a0010 con 0x7fd78c131bd0 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd79475d640 10 monclient(hunting): authenticate will time out at 1237.915551s 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 0x7fd78c131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd791cf8640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd78c09ebe0 0x7fd78c068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34158/0 (socket says 172.21.15.144:34158) 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd791cf8640 1 -- 172.21.15.144:0/1556234019 learned_addr learned my addr 172.21.15.144:0/1556234019 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 10 monclient(hunting): get_auth_request con 0x7fd78c131bd0 auth_method 0 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:22.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 1 -- 172.21.15.144:0/1556234019 >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 msgr2=0x7fd78c131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 1 -- 172.21.15.144:0/1556234019 >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 msgr2=0x7fd78c131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 1 --2- 172.21.15.144:0/1556234019 >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 0x7fd78c131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd7924f9640 1 --2- 172.21.15.144:0/1556234019 >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 0x7fd78c131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd792cfa640 1 --2- 172.21.15.144:0/1556234019 >> v2:172.21.15.116:3300/0 conn(0x7fd78c09efe0 0x7fd78c069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd792cfa640 10 monclient(hunting): get_auth_request con 0x7fd78c09efe0 auth_method 0 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd792cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd792cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd792cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd792cfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.346+0000 7fd792cfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 10 monclient(hunting): handle_auth_done global_id 104340 payload 274 2024-09-18T12:33:22.352 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 1 -- 172.21.15.144:0/1556234019 >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 msgr2=0x7fd78c131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:22.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 1 --2- 172.21.15.144:0/1556234019 >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 0x7fd78c131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 1 -- 172.21.15.144:0/1556234019 >> v1:172.21.15.144:6789/0 conn(0x7fd78c09ebe0 legacy=0x7fd78c068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:22.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 10 monclient: _finish_hunting 0 2024-09-18T12:33:22.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 1 monclient: found mon.noname-a 2024-09-18T12:33:22.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:22.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:22.353 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.347+0000 7fd792cfa640 1 -- 172.21.15.144:0/1556234019 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c0a1910 con 0x7fd78c09efe0 2024-09-18T12:33:22.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd792cfa640 1 -- 172.21.15.144:0/1556234019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09efe0 msgr2=0x7fd78c069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:22.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd792cfa640 1 --2- 172.21.15.144:0/1556234019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09efe0 0x7fd78c069510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd78c068370 tx=0x7fd784009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704f31e17d6c8b21 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:22.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 1 -- 172.21.15.144:0/1556234019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784007760 con 0x7fd78c09efe0 2024-09-18T12:33:22.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:22.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:22.361 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: dump: 2024-09-18T12:33:22.361 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: _finish_auth 0 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd79475d640 5 monclient: authenticate success, global_id 104340 2024-09-18T12:33:22.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd79475d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 1 -- 172.21.15.144:0/1556234019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78402d050 con 0x7fd78c09efe0 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 1 -- 172.21.15.144:0/1556234019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784007af0 con 0x7fd78c09efe0 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd79475d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd79475d640 4 set_mon_vals no callback set 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.350+0000 7fd7914f7640 10 monclient: dump: 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.363 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 10 monclient: shutdown 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 -- 172.21.15.144:0/1556234019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09efe0 msgr2=0x7fd78c069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:22.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 --2- 172.21.15.144:0/1556234019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09efe0 0x7fd78c069510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd78c068370 tx=0x7fd784009780 comp rx=0 tx=0).stop 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 -- 172.21.15.144:0/1556234019 shutdown_connections 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 --2- 172.21.15.144:0/1556234019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09efe0 0x7fd78c069510 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 --2- 172.21.15.144:0/1556234019 >> v2:172.21.15.144:3300/0 conn(0x7fd78c131bd0 0x7fd78c131fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 -- 172.21.15.144:0/1556234019 >> 172.21.15.144:0/1556234019 conn(0x7fd78c0713b0 msgr2=0x7fd78c0a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 -- 172.21.15.144:0/1556234019 shutdown_connections 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 1 -- 172.21.15.144:0/1556234019 wait complete. 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 10 monclient: build_initial_monmap 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.351+0000 7fd79475d640 10 monclient: monmap: 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:22.365 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 Processor -- start 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 -- start start 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient: init 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:22.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09ebe0 0x7fd78c140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): picked mon.noname-a con 0x7fd78c09ebe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd78c09efe0 0x7fd78c130d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): picked mon.noname-c con 0x7fd78c09efe0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 0x7fd78c11d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): picked mon.noname-b con 0x7fd78c131bd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c11fb50 con 0x7fd78c09ebe0 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c09e8a0 con 0x7fd78c131bd0 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c09bec0 con 0x7fd78c09efe0 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:22.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd79475d640 10 monclient(hunting): authenticate will time out at 1237.921566s 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 0x7fd78c11d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 0x7fd78c11d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55310/0 (socket says 172.21.15.144:55310) 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 1 -- 172.21.15.144:0/1104874490 learned_addr learned my addr 172.21.15.144:0/1104874490 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 10 monclient(hunting): get_auth_request con 0x7fd78c131bd0 auth_method 0 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 1 -- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 msgr2=0x7fd78c11d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 1 -- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 msgr2=0x7fd78c11d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 0x7fd78c11d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd792cfa640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 0x7fd78c11d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:22.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd7924f9640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09ebe0 0x7fd78c140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd791cf8640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd78c09efe0 0x7fd78c130d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd7924f9640 10 monclient(hunting): get_auth_request con 0x7fd78c09ebe0 auth_method 0 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd7924f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd7924f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd7924f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd791cf8640 10 monclient(hunting): get_auth_request con 0x7fd78c09efe0 auth_method 0 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd791cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd791cf8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.352+0000 7fd791cf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:22.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd791cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd791cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd791cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 10 monclient(hunting): handle_auth_done global_id 104343 payload 824 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 1 -- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd78c09efe0 msgr2=0x7fd78c130d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd78c09efe0 0x7fd78c130d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 1 -- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 msgr2=0x7fd78c11d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 0x7fd78c11d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 10 monclient: _finish_hunting 0 2024-09-18T12:33:22.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 1 monclient: found mon.noname-a 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.353+0000 7fd7924f9640 1 -- 172.21.15.144:0/1104874490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784009a20 con 0x7fd78c09ebe0 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7924f9640 1 -- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09ebe0 msgr2=0x7fd78c140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7924f9640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09ebe0 0x7fd78c140f00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd788006ea0 tx=0x7fd788008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78800b870 con 0x7fd78c09ebe0 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: dump: 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.371 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd79475d640 5 monclient: authenticate success, global_id 104343 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78800b9d0 con 0x7fd78c09ebe0 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd79475d640 10 monclient: _renew_subs 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd79475d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:22.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78c141db0 con 0x7fd78c09ebe0 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78800bd90 con 0x7fd78c09ebe0 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd7827fc640 10 monclient: dump: 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:22.373 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd79475d640 10 monclient: _renew_subs 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd79475d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c11f350 con 0x7fd78c09ebe0 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.355+0000 7fd790cf6640 4 set_mon_vals no callback set 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd79475d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd79475d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:22.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd79475d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:22.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c121360 con 0x7fd78c09ebe0 2024-09-18T12:33:22.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd788014640 con 0x7fd78c09ebe0 2024-09-18T12:33:22.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd7827fc640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd770038490 0x7fd77003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:22.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(222..222 src has 13..222) v4 ==== 14269+0+0 (secure 0 0 0) 0x7fd78804e1a0 con 0x7fd78c09ebe0 2024-09-18T12:33:22.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.356+0000 7fd791cf8640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd770038490 0x7fd77003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:22.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.357+0000 7fd791cf8640 10 monclient: get_auth_request con 0x7fd770038490 auth_method 0 2024-09-18T12:33:22.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.359+0000 7fd791cf8640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd770038490 0x7fd77003a940 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009a40 tx=0x7fd77c006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:22.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.361+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd78801d400 con 0x7fd78c09ebe0 2024-09-18T12:33:22.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.361+0000 7fd7827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:22.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.361+0000 7fd7827fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:22.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.519+0000 7fd79475d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:22.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.519+0000 7fd79475d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:22.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.519+0000 7fd79475d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:22.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:22.519+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd78c04bf60 con 0x7fd78c09ebe0 2024-09-18T12:33:23.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.162+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v223) v1 ==== 168+0+0 (secure 0 0 0) 0x7fd788011e60 con 0x7fd78c09ebe0 2024-09-18T12:33:23.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.162+0000 7fd7827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:23.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.162+0000 7fd7827fc640 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-18T12:33:23.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.258+0000 7fd79475d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:23.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.258+0000 7fd79475d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:23.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.258+0000 7fd79475d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:23.259 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.258+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd78c069490 con 0x7fd78c09ebe0 2024-09-18T12:33:23.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.258+0000 7fd7827fc640 1 -- 172.21.15.144:0/1104874490 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v223) v1 ==== 147+0+0 (secure 0 0 0) 0x7fd78801be90 con 0x7fd78c09ebe0 2024-09-18T12:33:23.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.258+0000 7fd7827fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:23.260 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.258+0000 7fd7827fc640 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-18T12:33:23.260 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h30' already exists 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd770038490 msgr2=0x7fd77003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd770038490 0x7fd77003a940 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009a40 tx=0x7fd77c006f50 comp rx=0 tx=0).stop 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 10 monclient: shutdown 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09ebe0 msgr2=0x7fd78c140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09ebe0 0x7fd78c140f00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd788006ea0 tx=0x7fd788008040 comp rx=0 tx=0).stop 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 shutdown_connections 2024-09-18T12:33:23.262 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd770038490 0x7fd77003a940 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd78c131bd0 0x7fd78c11d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd78c09efe0 0x7fd78c130d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 --2- 172.21.15.144:0/1104874490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd78c09ebe0 0x7fd78c140f00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 >> 172.21.15.144:0/1104874490 conn(0x7fd78c0713b0 msgr2=0x7fd78c0a31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:23.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 shutdown_connections 2024-09-18T12:33:23.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.260+0000 7fd79475d640 1 -- 172.21.15.144:0/1104874490 wait complete. 2024-09-18T12:33:23.270 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:23.270 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-18T12:33:23.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.344+0000 7fa3dd44f640 10 monclient: get_monmap_and_config 2024-09-18T12:33:23.345 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.344+0000 7fa3dd44f640 10 monclient: build_initial_monmap 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.344+0000 7fa3dd44f640 10 monclient: monmap: 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:23.345192+0000 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:23.345192+0000 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:23.346 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:23.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.344+0000 7fa3dd44f640 1 Processor -- start 2024-09-18T12:33:23.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.344+0000 7fa3dd44f640 1 -- start start 2024-09-18T12:33:23.347 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.344+0000 7fa3dd44f640 10 monclient: init 2024-09-18T12:33:23.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 0x7fa3d8131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient(hunting): picked mon.noname-c con 0x7fa3d8131bd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3d809ebe0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3d809efe0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d80a06d0 con 0x7fa3d809ebe0 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d8071780 con 0x7fa3d809efe0 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d80a0010 con 0x7fa3d8131bd0 2024-09-18T12:33:23.349 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3dd44f640 10 monclient(hunting): authenticate will time out at 1238.914494s 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 0x7fa3d8131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 0x7fa3d8131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55318/0 (socket says 172.21.15.144:55318) 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/3927943405 learned_addr learned my addr 172.21.15.144:0/3927943405 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 10 monclient(hunting): get_auth_request con 0x7fa3d8131bd0 auth_method 0 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/3927943405 >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 msgr2=0x7fa3d8131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/3927943405 >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 msgr2=0x7fa3d8131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:23.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/3927943405 >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 0x7fa3d8131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:23.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.345+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/3927943405 >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 0x7fa3d8131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:23.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.347+0000 7fa3d77fe640 1 -- 172.21.15.144:0/3927943405 >> v1:172.21.15.144:6789/0 conn(0x7fa3d809efe0 legacy=0x7fa3d8069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:23.351 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.347+0000 7fa3d67fc640 1 -- 172.21.15.144:0/3927943405 >> v1:172.21.15.116:6789/0 conn(0x7fa3d809ebe0 legacy=0x7fa3d8068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:23.362 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.348+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215754841 0 0) 0x7fa3d80a06d0 con 0x7fa3d809ebe0 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.348+0000 7fa3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.348+0000 7fa3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.348+0000 7fa3d5ffb640 10 monclient(hunting): my global_id is 104346 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.348+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c00034e0 con 0x7fa3d809ebe0 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 536249348 0 0) 0x7fa3c00034e0 con 0x7fa3d809ebe0 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 msgr2=0x7fa3d8131fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 --2- 172.21.15.144:0/3927943405 >> v2:172.21.15.144:3300/0 conn(0x7fa3d8131bd0 0x7fa3d8131fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 >> v1:172.21.15.144:6789/0 conn(0x7fa3d809efe0 legacy=0x7fa3d8069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80a1910 con 0x7fa3d809ebe0 2024-09-18T12:33:23.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fa3cc0017f0 con 0x7fa3d809ebe0 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3dd44f640 5 monclient: authenticate success, global_id 104346 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: dump: 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.364 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 >> v1:172.21.15.116:6789/0 conn(0x7fa3d809ebe0 legacy=0x7fa3d8068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3c0003bc0 0x7fa3c0003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): picked mon.c con 0x7fa3c0003bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c0004780 0x7fa3c0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): picked mon.a con 0x7fa3c0004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 0x7fa3c000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): picked mon.b con 0x7fa3c00099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80a0010 con 0x7fa3c0004780 2024-09-18T12:33:23.365 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c000c420 con 0x7fa3c00099c0 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c000c5e0 con 0x7fa3c0003bc0 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3dd44f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d77fe640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 0x7fa3c000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d77fe640 1 -- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 msgr2=0x7fa3c000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d77fe640 1 -- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 msgr2=0x7fa3c000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3c0003bc0 0x7fa3c0003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d77fe640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 0x7fa3c000bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): get_auth_request con 0x7fa3c0003bc0 auth_method 0 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c0004780 0x7fa3c0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d77fe640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 0x7fa3c000bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 10 monclient(hunting): get_auth_request con 0x7fa3c0004780 auth_method 0 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_done global_id 104346 payload 293 2024-09-18T12:33:23.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 msgr2=0x7fa3c000bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c00099c0 0x7fa3c000bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c0004780 msgr2=0x7fa3c0004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c0004780 0x7fa3c0004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 1 monclient: found mon.c 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d77fe640 1 -- 172.21.15.144:0/3927943405 reap_dead start 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/3927943405 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00034e0 con 0x7fa3c0003bc0 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.349+0000 7fa3d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3c0003bc0 msgr2=0x7fa3c0003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3c0003bc0 0x7fa3c0003fd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009650 tx=0x7fa3d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62715edc7851ee76 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:23.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0030480 con 0x7fa3c0003bc0 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 10 monclient: dump: 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3dd44f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:23.369 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d002d030 con 0x7fa3c0003bc0 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 1 -- 172.21.15.144:0/3927943405 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0030850 con 0x7fa3c0003bc0 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3dd44f640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3dd44f640 4 set_mon_vals no callback set 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.352+0000 7fa3d5ffb640 10 monclient: dump: 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:23.370 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 10 monclient: shutdown 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 1 -- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3c0003bc0 msgr2=0x7fa3c0003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3c0003bc0 0x7fa3c0003fd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009650 tx=0x7fa3d00091a0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 1 -- 172.21.15.144:0/3927943405 shutdown_connections 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/3927943405 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3c0003bc0 0x7fa3c0003fd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 1 -- 172.21.15.144:0/3927943405 >> 172.21.15.144:0/3927943405 conn(0x7fa3d80713b0 msgr2=0x7fa3d80a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 1 -- 172.21.15.144:0/3927943405 shutdown_connections 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.353+0000 7fa3dd44f640 1 -- 172.21.15.144:0/3927943405 wait complete. 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.354+0000 7fa3dd44f640 10 monclient: build_initial_monmap 2024-09-18T12:33:23.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.354+0000 7fa3dd44f640 10 monclient: monmap: 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.354+0000 7fa3dd44f640 1 Processor -- start 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.354+0000 7fa3dd44f640 1 -- start start 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.354+0000 7fa3dd44f640 10 monclient: init 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:23.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 0x7fa3d809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): picked mon.noname-b con 0x7fa3c0003bc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d811bbc0 0x7fa3d8130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): picked mon.noname-a con 0x7fa3d811bbc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3d8131140 0x7fa3d81410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): picked mon.noname-c con 0x7fa3d8131140 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d809b600 con 0x7fa3d811bbc0 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d809bec0 con 0x7fa3c0003bc0 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d809e520 con 0x7fa3d8131140 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3dd44f640 10 monclient(hunting): authenticate will time out at 1238.924076s 2024-09-18T12:33:23.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 0x7fa3d809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 0x7fa3d809aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55324/0 (socket says 172.21.15.144:55324) 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/2780513731 learned_addr learned my addr 172.21.15.144:0/2780513731 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): get_auth_request con 0x7fa3c0003bc0 auth_method 0 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d811bbc0 0x7fa3d8130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 10 monclient(hunting): get_auth_request con 0x7fa3d811bbc0 auth_method 0 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3d8131140 0x7fa3d81410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 10 monclient(hunting): get_auth_request con 0x7fa3d8131140 auth_method 0 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient(hunting): handle_auth_done global_id 114100 payload 824 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3d8131140 msgr2=0x7fa3d81410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3d8131140 0x7fa3d81410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d811bbc0 msgr2=0x7fa3d8130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d811bbc0 0x7fa3d8130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 monclient: found mon.noname-b 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.355+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/2780513731 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d811dd20 con 0x7fa3c0003bc0 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3d6ffd640 1 -- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 msgr2=0x7fa3d809aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3d6ffd640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 0x7fa3d809aa90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009de0 tx=0x7fa3d0038000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d003d890 con 0x7fa3c0003bc0 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: dump: 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.376 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3dd44f640 5 monclient: authenticate success, global_id 114100 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d003d9f0 con 0x7fa3c0003bc0 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3dd44f640 10 monclient: _renew_subs 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3dd44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:23.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d811e000 con 0x7fa3c0003bc0 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d003ddb0 con 0x7fa3c0003bc0 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3beffd640 10 monclient: dump: 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:23.378 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3d57fa640 4 set_mon_vals no callback set 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3dd44f640 10 monclient: _renew_subs 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3dd44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.357+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d811c910 con 0x7fa3c0003bc0 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.358+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa3d003a040 con 0x7fa3c0003bc0 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.358+0000 7fa3beffd640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3c40384c0 0x7fa3c403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.358+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(223..223 src has 13..223) v4 ==== 14482+0+0 (secure 0 0 0) 0x7fa3d0043070 con 0x7fa3c0003bc0 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.358+0000 7fa3d67fc640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3c40384c0 0x7fa3c403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.358+0000 7fa3d67fc640 10 monclient: get_auth_request con 0x7fa3c40384c0 auth_method 0 2024-09-18T12:33:23.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.359+0000 7fa3dd44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:23.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.359+0000 7fa3dd44f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:23.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.359+0000 7fa3dd44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:23.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.359+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d809bf10 con 0x7fa3c0003bc0 2024-09-18T12:33:23.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.361+0000 7fa3d67fc640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3c40384c0 0x7fa3c403a970 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0019a0 tx=0x7fa3cc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:23.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.363+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa3d00314f0 con 0x7fa3c0003bc0 2024-09-18T12:33:23.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.363+0000 7fa3beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:23.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.363+0000 7fa3beffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:23.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.520+0000 7fa3dd44f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:23.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.520+0000 7fa3dd44f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:23.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.520+0000 7fa3dd44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:23.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:23.520+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa3d804bf60 con 0x7fa3c0003bc0 2024-09-18T12:33:23.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.842+0000 7f4c6bb9d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.842+0000 7f4c6bb9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.842+0000 7f4c6bb9d640 10 monclient: monmap: 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:23.844504+0000 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:23.844504+0000 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:23.845 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.843+0000 7f4c6bb9d640 1 Processor -- start 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.843+0000 7f4c6bb9d640 1 -- start start 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.843+0000 7f4c6bb9d640 10 monclient: init 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c640974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c6409a8a0 0x7f4c6409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c6409a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4c64133070 0x7f4c64135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4c64133070 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6412b210 con 0x7f4c640974f0 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c640978c0 con 0x7f4c64133070 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6409ca20 con 0x7f4c6409a8a0 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6bb9d640 10 monclient(hunting): authenticate will time out at 1231.166254s 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c6409a8a0 0x7f4c6409acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4c64133070 0x7f4c64135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c6409a8a0 0x7f4c6409acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45946/0 (socket says 172.21.15.116:45946) 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 1 -- 172.21.15.116:0/186642567 learned_addr learned my addr 172.21.15.116:0/186642567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 10 monclient(hunting): get_auth_request con 0x7f4c6409a8a0 auth_method 0 2024-09-18T12:33:23.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): get_auth_request con 0x7f4c64133070 auth_method 0 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient(hunting): handle_auth_done global_id 104352 payload 274 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 1 -- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c6409a8a0 msgr2=0x7f4c6409acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 1 --2- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c6409a8a0 0x7f4c6409acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 1 -- 172.21.15.116:0/186642567 >> v1:172.21.15.144:6789/0 conn(0x7f4c640974f0 legacy=0x7f4c6412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient: _finish_hunting 0 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 1 monclient: found mon.noname-a 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c6a13a640 1 -- 172.21.15.116:0/186642567 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6409b1f0 con 0x7f4c64133070 2024-09-18T12:33:23.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.844+0000 7f4c69138640 1 --2- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c6409a8a0 0x7f4c6409acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:23.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c6a13a640 1 -- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c64133070 msgr2=0x7f4c64135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:23.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c6a13a640 1 --2- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c64133070 0x7f4c64135450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009e90 tx=0x7f4c5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef4515678402e80 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:23.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 1 -- 172.21.15.116:0/186642567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c007760 con 0x7f4c64133070 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: dump: 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.857 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: _finish_auth 0 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 1 -- 172.21.15.116:0/186642567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c02d050 con 0x7f4c64133070 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c6bb9d640 5 monclient: authenticate success, global_id 104352 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c6bb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 1 -- 172.21.15.116:0/186642567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c007af0 con 0x7f4c64133070 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c68937640 10 monclient: dump: 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c6bb9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.847+0000 7f4c6bb9d640 4 set_mon_vals no callback set 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 10 monclient: shutdown 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c64133070 msgr2=0x7f4c64135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c64133070 0x7f4c64135450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009e90 tx=0x7f4c5c009780 comp rx=0 tx=0).stop 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/186642567 shutdown_connections 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c6409a8a0 0x7f4c6409acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/186642567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c64133070 0x7f4c64135450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/186642567 >> 172.21.15.116:0/186642567 conn(0x7f4c640672b0 msgr2=0x7f4c640676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/186642567 shutdown_connections 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/186642567 wait complete. 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.848+0000 7f4c6bb9d640 10 monclient: monmap: 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 Processor -- start 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 -- start start 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient: init 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 0x7f4c6406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4c640974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c6409a8a0 0x7f4c6412bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c6409a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c64133070 0x7f4c64132190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4c64133070 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6409de20 con 0x7f4c640974f0 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6412e9f0 con 0x7f4c6409a8a0 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6412e3a0 con 0x7f4c64133070 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6bb9d640 10 monclient(hunting): authenticate will time out at 1231.171765s 2024-09-18T12:33:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 0x7f4c6406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6a13a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c64133070 0x7f4c64132190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 0x7f4c6406eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55474/0 (socket says 172.21.15.116:55474) 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 1 -- 172.21.15.116:0/2470623874 learned_addr learned my addr 172.21.15.116:0/2470623874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 10 monclient(hunting): get_auth_request con 0x7f4c640974f0 auth_method 0 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6a13a640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c64133070 0x7f4c64132190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45952/0 (socket says 172.21.15.116:45952) 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6a13a640 10 monclient(hunting): get_auth_request con 0x7f4c64133070 auth_method 0 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6a13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6a13a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6a13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c69138640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c6409a8a0 0x7f4c6412bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.849+0000 7f4c6a13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c6a13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c6a13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69138640 10 monclient(hunting): get_auth_request con 0x7f4c6409a8a0 auth_method 0 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69138640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69138640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:23.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 10 monclient(hunting): handle_auth_done global_id 104355 payload 824 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 1 -- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c64133070 msgr2=0x7f4c64132190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c64133070 0x7f4c64132190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 1 -- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c6409a8a0 msgr2=0x7f4c6412bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c6409a8a0 0x7f4c6412bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 10 monclient: _finish_hunting 0 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 1 monclient: found mon.noname-a 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.850+0000 7f4c69939640 1 -- 172.21.15.116:0/2470623874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c54009450 con 0x7f4c640974f0 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c69939640 1 -- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 msgr2=0x7f4c6406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c69939640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 0x7f4c6406eca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000c610 tx=0x7f4c60007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 1 -- 172.21.15.116:0/2470623874 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60012840 con 0x7f4c640974f0 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: dump: 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.866 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 1 -- 172.21.15.116:0/2470623874 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c600129e0 con 0x7f4c640974f0 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c6bb9d640 5 monclient: authenticate success, global_id 104355 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c6bb9d640 10 monclient: _renew_subs 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c6bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c009a20 con 0x7f4c640974f0 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:23.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 1 -- 172.21.15.116:0/2470623874 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60012da0 con 0x7f4c640974f0 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c59ffb640 10 monclient: dump: 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:23.868 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c5bfff640 4 set_mon_vals no callback set 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c6bb9d640 10 monclient: _renew_subs 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c6bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.852+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6412e730 con 0x7f4c640974f0 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.853+0000 7f4c59ffb640 1 -- 172.21.15.116:0/2470623874 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4c60011590 con 0x7f4c640974f0 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.853+0000 7f4c6bb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.853+0000 7f4c6bb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.853+0000 7f4c6bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.853+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6404bf60 con 0x7f4c640974f0 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.853+0000 7f4c59ffb640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c48038490 0x7f4c4803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.853+0000 7f4c59ffb640 1 -- 172.21.15.116:0/2470623874 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(223..223 src has 13..223) v4 ==== 14482+0+0 (secure 0 0 0) 0x7f4c6004d8f0 con 0x7f4c640974f0 2024-09-18T12:33:23.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.854+0000 7f4c69138640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c48038490 0x7f4c4803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:23.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.854+0000 7f4c69138640 10 monclient: get_auth_request con 0x7f4c48038490 auth_method 0 2024-09-18T12:33:23.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.856+0000 7f4c69138640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c48038490 0x7f4c4803a940 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4c64098930 tx=0x7f4c540091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:23.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.858+0000 7f4c59ffb640 1 -- 172.21.15.116:0/2470623874 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4c600198e0 con 0x7f4c640974f0 2024-09-18T12:33:23.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.858+0000 7f4c59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:23.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:23.858+0000 7f4c59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:24.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.072+0000 7f4c6bb9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:24.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.072+0000 7f4c6bb9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:24.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.072+0000 7f4c6bb9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:24.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.072+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4c6412e010 con 0x7f4c640974f0 2024-09-18T12:33:24.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.073+0000 7f4c59ffb640 1 -- 172.21.15.116:0/2470623874 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f4c60019310 con 0x7f4c640974f0 2024-09-18T12:33:24.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.073+0000 7f4c59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:24.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.073+0000 7f4c59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:24.075 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:24.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c48038490 msgr2=0x7f4c4803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c48038490 0x7f4c4803a940 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4c64098930 tx=0x7f4c540091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 10 monclient: shutdown 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 msgr2=0x7f4c6406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 0x7f4c6406eca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000c610 tx=0x7f4c60007ed0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 shutdown_connections 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c48038490 0x7f4c4803a940 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c64133070 0x7f4c64132190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c6409a8a0 0x7f4c6412bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 --2- 172.21.15.116:0/2470623874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c640974f0 0x7f4c6406eca0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 >> 172.21.15.116:0/2470623874 conn(0x7f4c640672b0 msgr2=0x7f4c64129a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 shutdown_connections 2024-09-18T12:33:24.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.075+0000 7f4c6bb9d640 1 -- 172.21.15.116:0/2470623874 wait complete. 2024-09-18T12:33:24.087 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:33:24.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:33:24.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:33:24.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.180+0000 7f6aa079d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:24.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.180+0000 7f6aa079d640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.180+0000 7f6aa079d640 10 monclient: monmap: 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:24.182201+0000 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:24.182201+0000 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:24.183 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 1 Processor -- start 2024-09-18T12:33:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 1 -- start start 2024-09-18T12:33:24.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient: init 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6a980974f0 0x7f6a9812ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a980974f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a9809a8a0 0x7f6a9809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a9809a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a9812dd40 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a980978c0 con 0x7f6a9812dd40 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9809b220 con 0x7f6a9809a8a0 2024-09-18T12:33:24.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9809ca20 con 0x7f6a980974f0 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6aa079d640 10 monclient(hunting): authenticate will time out at 1231.503760s 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a9809a8a0 0x7f6a9809ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a9809a8a0 0x7f6a9809ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55478/0 (socket says 172.21.15.116:55478) 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 1 -- 172.21.15.116:0/3392318045 learned_addr learned my addr 172.21.15.116:0/3392318045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 10 monclient(hunting): get_auth_request con 0x7f6a9809a8a0 auth_method 0 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.181+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9e539640 1 --2- 172.21.15.116:0/3392318045 >> v2:172.21.15.144:3300/0 conn(0x7f6a980974f0 0x7f6a9812ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9e539640 10 monclient(hunting): get_auth_request con 0x7f6a980974f0 auth_method 0 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9e539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9e539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_done global_id 104358 payload 274 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 1 -- 172.21.15.116:0/3392318045 >> v2:172.21.15.144:3300/0 conn(0x7f6a980974f0 msgr2=0x7f6a9812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 1 --2- 172.21.15.116:0/3392318045 >> v2:172.21.15.144:3300/0 conn(0x7f6a980974f0 0x7f6a9812ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 1 -- 172.21.15.116:0/3392318045 >> v1:172.21.15.144:6789/0 conn(0x7f6a9812dd40 legacy=0x7f6a9812e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 1 monclient: found mon.noname-a 2024-09-18T12:33:24.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:24.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.182+0000 7f6a9dd38640 1 -- 172.21.15.116:0/3392318045 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a98098ad0 con 0x7f6a9809a8a0 2024-09-18T12:33:24.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.184+0000 7f6a9dd38640 1 -- 172.21.15.116:0/3392318045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9809a8a0 msgr2=0x7f6a9809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.184+0000 7f6a9dd38640 1 --2- 172.21.15.116:0/3392318045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9809a8a0 0x7f6a9809ace0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a980768c0 tx=0x7f6a8c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c8947719f72bca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 1 -- 172.21.15.116:0/3392318045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c032440 con 0x7f6a9809a8a0 2024-09-18T12:33:24.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:24.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: dump: 2024-09-18T12:33:24.196 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 5 monclient: authenticate success, global_id 104358 2024-09-18T12:33:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 1 -- 172.21.15.116:0/3392318045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c0325a0 con 0x7f6a9809a8a0 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 1 -- 172.21.15.116:0/3392318045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c032940 con 0x7f6a9809a8a0 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 4 set_mon_vals no callback set 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6a9d537640 10 monclient: dump: 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 10 monclient: shutdown 2024-09-18T12:33:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 -- 172.21.15.116:0/3392318045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9809a8a0 msgr2=0x7f6a9809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 --2- 172.21.15.116:0/3392318045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9809a8a0 0x7f6a9809ace0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a980768c0 tx=0x7f6a8c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 -- 172.21.15.116:0/3392318045 shutdown_connections 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 --2- 172.21.15.116:0/3392318045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9809a8a0 0x7f6a9809ace0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 --2- 172.21.15.116:0/3392318045 >> v2:172.21.15.144:3300/0 conn(0x7f6a980974f0 0x7f6a9812ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 -- 172.21.15.116:0/3392318045 >> 172.21.15.116:0/3392318045 conn(0x7f6a980672b0 msgr2=0x7f6a980676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 -- 172.21.15.116:0/3392318045 shutdown_connections 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.185+0000 7f6aa079d640 1 -- 172.21.15.116:0/3392318045 wait complete. 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient: monmap: 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 1 Processor -- start 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 1 -- start start 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient: init 2024-09-18T12:33:24.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a980974f0 0x7f6a9811daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a980974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a9809a8a0 0x7f6a98127820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a9809a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a9812dd40 0x7f6a9806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.186+0000 7f6aa079d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a9812dd40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9809e0b0 con 0x7f6a980974f0 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9811caf0 con 0x7f6a9812dd40 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9811c810 con 0x7f6a9809a8a0 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6aa079d640 10 monclient(hunting): authenticate will time out at 1231.509053s 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a980974f0 0x7f6a9811daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a9809a8a0 0x7f6a98127820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a980974f0 0x7f6a9811daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55480/0 (socket says 172.21.15.116:55480) 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 -- 172.21.15.116:0/147299517 learned_addr learned my addr 172.21.15.116:0/147299517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): get_auth_request con 0x7f6a980974f0 auth_method 0 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 10 monclient(hunting): get_auth_request con 0x7f6a9809a8a0 auth_method 0 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient(hunting): handle_auth_done global_id 104361 payload 824 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a9809a8a0 msgr2=0x7f6a98127820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6a9809a8a0 0x7f6a98127820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a9812dd40 msgr2=0x7f6a9806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6a9812dd40 0x7f6a9806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 monclient: found mon.noname-a 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.187+0000 7f6a9e539640 1 -- 172.21.15.116:0/147299517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c009450 con 0x7f6a980974f0 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a9e539640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a980974f0 msgr2=0x7f6a9811daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a9e539640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a980974f0 0x7f6a9811daa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000e190 tx=0x7f6a90009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90013590 con 0x7f6a980974f0 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: dump: 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a900136f0 con 0x7f6a980974f0 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6aa079d640 5 monclient: authenticate success, global_id 104361 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90013ab0 con 0x7f6a980974f0 2024-09-18T12:33:24.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6aa079d640 10 monclient: _renew_subs 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6aa079d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9811ce30 con 0x7f6a980974f0 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a967fc640 10 monclient: dump: 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.208 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6a9cd36640 4 set_mon_vals no callback set 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6aa079d640 10 monclient: _renew_subs 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6aa079d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.189+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a98127f00 con 0x7f6a980974f0 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6aa079d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6aa079d640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 0x7f6a9806da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6aa079d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a98065f20 con 0x7f6a9806d630 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6a9001e460 con 0x7f6a980974f0 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 0x7f6a9806da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient: get_auth_request con 0x7f6a9806d630 auth_method 0 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a967fc640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a84038490 0x7f6a8403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_done global_id 104364 payload 824 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.190+0000 7f6a9dd38640 1 -- 172.21.15.116:0/147299517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a9811ccb0 con 0x7f6a9806d630 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.191+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(223..223 src has 13..223) v4 ==== 14482+0+0 (secure 0 0 0) 0x7f6a9004e650 con 0x7f6a980974f0 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.191+0000 7f6a9ed3a640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a84038490 0x7f6a8403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.191+0000 7f6a9ed3a640 10 monclient: get_auth_request con 0x7f6a84038490 auth_method 0 2024-09-18T12:33:24.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a9dd38640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 msgr2=0x7f6a9806da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a9dd38640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 0x7f6a9806da00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c034000 tx=0x7f6a8c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c031c50 con 0x7f6a9806d630 2024-09-18T12:33:24.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:24.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 10 monclient: dump: 2024-09-18T12:33:24.213 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.213 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.213 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.213 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6a8c031db0 con 0x7f6a9806d630 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 msgr2=0x7f6a9806da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 0x7f6a9806da00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c034000 tx=0x7f6a8c032000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 msgr2=0x7f6a9806da00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:24.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a967fc640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806d630 0x7f6a9806da00 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c034000 tx=0x7f6a8c032000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.193+0000 7f6a9ed3a640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a84038490 0x7f6a8403a940 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6a88007fd0 tx=0x7f6a88007a90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6aa079d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:24.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6aa079d640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 0x7f6a980696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6aa079d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9806ec30 con 0x7f6a9806df40 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 0x7f6a980696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient: get_auth_request con 0x7f6a9806df40 auth_method 0 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 10 monclient(hunting): handle_auth_done global_id 104367 payload 824 2024-09-18T12:33:24.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.199+0000 7f6a9dd38640 1 -- 172.21.15.116:0/147299517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6a9804bf60 con 0x7f6a9806df40 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.201+0000 7f6a9dd38640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 msgr2=0x7f6a980696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.201+0000 7f6a9dd38640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 0x7f6a980696a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a9811b960 tx=0x7f6a8c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.201+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c055070 con 0x7f6a9806df40 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.201+0000 7f6a967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.201+0000 7f6a967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.201+0000 7f6a967fc640 10 monclient: dump: 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.218 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f6a8c051450 con 0x7f6a9806df40 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a967fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 msgr2=0x7f6a980696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a967fc640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 0x7f6a980696a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a9811b960 tx=0x7f6a8c03d000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a967fc640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 msgr2=0x7f6a980696a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a967fc640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a9806df40 0x7f6a980696a0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a9811b960 tx=0x7f6a8c03d000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.219 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 88, 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:33:24.220 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.221 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.222 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.222 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.222 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.222 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:24.222 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:24.222 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:24.223 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:24.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.228 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:24.228 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:24.228 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.228 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.228 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.229 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:24.229 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.229 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.229 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:24.229 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:24.230 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.236 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.236 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:33:24.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.236 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.236 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.236 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.237 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.238 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:24.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.238 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.238 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:24.238 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a84038490 msgr2=0x7f6a8403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6a84038490 0x7f6a8403a940 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6a88007fd0 tx=0x7f6a88007a90 comp rx=0 tx=0).stop 2024-09-18T12:33:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 10 monclient: shutdown 2024-09-18T12:33:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a980974f0 msgr2=0x7f6a9811daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 --2- 172.21.15.116:0/147299517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a980974f0 0x7f6a9811daa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000e190 tx=0x7f6a90009920 comp rx=0 tx=0).stop 2024-09-18T12:33:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a9ed3a640 1 -- 172.21.15.116:0/147299517 reap_dead start 2024-09-18T12:33:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6a9ed3a640 1 -- 172.21.15.116:0/147299517 reap_dead start 2024-09-18T12:33:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 shutdown_connections 2024-09-18T12:33:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 >> 172.21.15.116:0/147299517 conn(0x7f6a980672b0 msgr2=0x7f6a9812d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 shutdown_connections 2024-09-18T12:33:24.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.202+0000 7f6aa079d640 1 -- 172.21.15.116:0/147299517 wait complete. 2024-09-18T12:33:24.242 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:33:24.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:33:24.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:33:24.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.333+0000 7f82e4f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:24.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.333+0000 7f82e4f9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.333+0000 7f82e4f9d640 10 monclient: monmap: 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:24.335178+0000 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:24.335178+0000 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:24.337 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:24.337 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:24.337 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 1 Processor -- start 2024-09-18T12:33:24.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 1 -- start start 2024-09-18T12:33:24.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient: init 2024-09-18T12:33:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T12:33:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f82e009ad90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82e009b160 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82e009b530 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e00692c0 con 0x7f82e009b530 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e009b990 con 0x7f82e009b160 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e009db70 con 0x7f82e009ad90 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82e4f9d640 10 monclient(hunting): authenticate will time out at 1231.656580s 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82ded9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f82e009b530 0x7f82e0132070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44016/0 (socket says 172.21.15.116:44016) 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82ded9d640 1 -- 172.21.15.116:0/3194313410 learned_addr learned my addr 172.21.15.116:0/3194313410 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): get_auth_request con 0x7f82e009ad90 auth_method 0 2024-09-18T12:33:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient(hunting): handle_auth_done global_id 65162 payload 274 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 1 -- 172.21.15.116:0/3194313410 >> v1:172.21.15.144:6789/0 conn(0x7f82e009b160 legacy=0x7f82e0131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 1 -- 172.21.15.116:0/3194313410 >> v1:172.21.15.116:6789/0 conn(0x7f82e009b530 legacy=0x7f82e0132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 1 monclient: found mon.noname-b 2024-09-18T12:33:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.334+0000 7f82de59c640 1 -- 172.21.15.116:0/3194313410 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e009d5b0 con 0x7f82e009ad90 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82de59c640 1 -- 172.21.15.116:0/3194313410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 msgr2=0x7f82e009c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82de59c640 1 --2- 172.21.15.116:0/3194313410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e009c5d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009880 tx=0x7f82d00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f3419d4ae49e56b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 1 -- 172.21.15.116:0/3194313410 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d002aea0 con 0x7f82e009ad90 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: dump: 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 1 -- 172.21.15.116:0/3194313410 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d0032440 con 0x7f82e009ad90 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82e4f9d640 5 monclient: authenticate success, global_id 65162 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82e4f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82e4f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:24.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82e4f9d640 4 set_mon_vals no callback set 2024-09-18T12:33:24.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 1 -- 172.21.15.116:0/3194313410 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0032820 con 0x7f82e009ad90 2024-09-18T12:33:24.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:24.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.337+0000 7f82ddd9b640 10 monclient: dump: 2024-09-18T12:33:24.351 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.351 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.352 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 10 monclient: shutdown 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 1 -- 172.21.15.116:0/3194313410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 msgr2=0x7f82e009c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 1 --2- 172.21.15.116:0/3194313410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e009c5d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009880 tx=0x7f82d00090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 1 -- 172.21.15.116:0/3194313410 shutdown_connections 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 1 --2- 172.21.15.116:0/3194313410 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e009c5d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 1 -- 172.21.15.116:0/3194313410 >> 172.21.15.116:0/3194313410 conn(0x7f82e00678a0 msgr2=0x7f82e00a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 1 -- 172.21.15.116:0/3194313410 shutdown_connections 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.338+0000 7f82e4f9d640 1 -- 172.21.15.116:0/3194313410 wait complete. 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient: monmap: 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 Processor -- start 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 -- start start 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient: init 2024-09-18T12:33:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e0141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f82e009ad90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82e009b160 0x7f82e0130d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f82e009b160 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82e009b530 0x7f82e0127a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f82e009b530 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e012b230 con 0x7f82e009b530 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e012af50 con 0x7f82e009b160 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.339+0000 7f82e4f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e012b510 con 0x7f82e009ad90 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82e4f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82e4f9d640 10 monclient(hunting): authenticate will time out at 1231.662020s 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e0141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e0141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45990/0 (socket says 172.21.15.116:45990) 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 -- 172.21.15.116:0/2717168377 learned_addr learned my addr 172.21.15.116:0/2717168377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): get_auth_request con 0x7f82e009ad90 auth_method 0 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82e009b530 0x7f82e0127a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 10 monclient(hunting): get_auth_request con 0x7f82e009b530 auth_method 0 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82ded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient(hunting): handle_auth_done global_id 65165 payload 824 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82d5d9b640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82e009b160 0x7f82e0130d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82e009b160 msgr2=0x7f82e0130d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82e009b160 0x7f82e0130d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82e009b530 msgr2=0x7f82e0127a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82e009b530 0x7f82e0127a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 monclient: found mon.noname-c 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.340+0000 7f82de59c640 1 -- 172.21.15.116:0/2717168377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0009390 con 0x7f82e009ad90 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82de59c640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 msgr2=0x7f82e0141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82de59c640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e0141150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009880 tx=0x7f82d0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d002aea0 con 0x7f82e009ad90 2024-09-18T12:33:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: dump: 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82e4f9d640 5 monclient: authenticate success, global_id 65165 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d0031990 con 0x7f82e009ad90 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82e4f9d640 10 monclient: _renew_subs 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82e4f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e0129920 con 0x7f82e009ad90 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0031d50 con 0x7f82e009ad90 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82d6ffd640 10 monclient: dump: 2024-09-18T12:33:24.360 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.361 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82dd59a640 4 set_mon_vals no callback set 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82e4f9d640 10 monclient: _renew_subs 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82e4f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.342+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e0131440 con 0x7f82e009ad90 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82e4f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82e4f9d640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 0x7f82e0127f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82e4f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0065e00 con 0x7f82e0141690 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f82d0039040 con 0x7f82e009ad90 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 0x7f82e0127f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 10 monclient: get_auth_request con 0x7f82e0141690 auth_method 0 2024-09-18T12:33:24.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.343+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.344+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82c0038490 0x7f82c003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.344+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(223..223 src has 13..223) v4 ==== 14482+0+0 (secure 0 0 0) 0x7f82d006e090 con 0x7f82e009ad90 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.344+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_done global_id 65168 payload 824 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.344+0000 7f82d5d9b640 1 -- 172.21.15.116:0/2717168377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e012b0c0 con 0x7f82e0141690 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.344+0000 7f82ded9d640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82c0038490 0x7f82c003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.344+0000 7f82ded9d640 10 monclient: get_auth_request con 0x7f82c0038490 auth_method 0 2024-09-18T12:33:24.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.359+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v224) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa3d0031310 con 0x7fa3c0003bc0 2024-09-18T12:33:24.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.359+0000 7fa3beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:24.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.359+0000 7fa3beffd640 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d5d9b640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 msgr2=0x7f82e0127f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d5d9b640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 0x7f82e0127f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d400 tx=0x7f82c8009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8012810 con 0x7f82e0141690 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 10 monclient: dump: 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.367 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f82c8012970 con 0x7f82e0141690 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 msgr2=0x7f82e0127f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 0x7f82e0127f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d400 tx=0x7f82c8009040 comp rx=0 tx=0).stop 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 msgr2=0x7f82e0127f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:24.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e0141690 0x7f82e0127f80 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d400 tx=0x7f82c8009040 comp rx=0 tx=0).stop 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.346+0000 7f82ded9d640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82c0038490 0x7f82c003a940 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f82d8006d70 tx=0x7f82d80052b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82e4f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82e4f9d640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 0x7f82e0095430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82e4f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0141a60 con 0x7f82e01284c0 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 0x7f82e0095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient: get_auth_request con 0x7f82e01284c0 auth_method 0 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 10 monclient(hunting): handle_auth_done global_id 65171 payload 824 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.352+0000 7f82d5d9b640 1 -- 172.21.15.116:0/2717168377 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f82e012b3a0 con 0x7f82e01284c0 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.354+0000 7f82d5d9b640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 msgr2=0x7f82e0095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.354+0000 7f82d5d9b640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 0x7f82e0095430 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f82c8018000 tx=0x7f82c801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.355+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8020070 con 0x7f82e01284c0 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.355+0000 7f82d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.355+0000 7f82d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.355+0000 7f82d6ffd640 10 monclient: dump: 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f82c801a7d0 con 0x7f82e01284c0 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82d6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 msgr2=0x7f82e0095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 0x7f82e0095430 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f82c8018000 tx=0x7f82c801b000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82d6ffd640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 msgr2=0x7f82e0095430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:24.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e01284c0 0x7f82e0095430 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f82c8018000 tx=0x7f82c801b000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.372 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:24.372 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:24.372 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:24.372 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:33:24.372 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 88, 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.373 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.374 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:24.375 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:24.376 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:24.377 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:24.377 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.377 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.377 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.377 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.377 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.378 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.379 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.380 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.381 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.382 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.384 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.385 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:24.386 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82c0038490 msgr2=0x7f82c003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82c0038490 0x7f82c003a940 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f82d8006d70 tx=0x7f82d80052b0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 10 monclient: shutdown 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 msgr2=0x7f82e0141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 --2- 172.21.15.116:0/2717168377 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82e009ad90 0x7f82e0141150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009880 tx=0x7f82d0032000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82ded9d640 1 -- 172.21.15.116:0/2717168377 reap_dead start 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82ded9d640 1 -- 172.21.15.116:0/2717168377 reap_dead start 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 shutdown_connections 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 >> 172.21.15.116:0/2717168377 conn(0x7f82e00678a0 msgr2=0x7f82e009f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 shutdown_connections 2024-09-18T12:33:24.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.356+0000 7f82e4f9d640 1 -- 172.21.15.116:0/2717168377 wait complete. 2024-09-18T12:33:24.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:33:24.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:33:24.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:33:24.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.455+0000 7fa3dd44f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:24.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.455+0000 7fa3dd44f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:24.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.455+0000 7fa3dd44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:24.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.455+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa3d8072640 con 0x7fa3c0003bc0 2024-09-18T12:33:24.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.456+0000 7fa3beffd640 1 -- 172.21.15.144:0/2780513731 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v224) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa3d0031ab0 con 0x7fa3c0003bc0 2024-09-18T12:33:24.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.456+0000 7fa3beffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:24.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.456+0000 7fa3beffd640 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-18T12:33:24.457 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h31' already exists 2024-09-18T12:33:24.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3c40384c0 msgr2=0x7fa3c403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3c40384c0 0x7fa3c403a970 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0019a0 tx=0x7fa3cc009040 comp rx=0 tx=0).stop 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 10 monclient: shutdown 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 msgr2=0x7fa3d809aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 0x7fa3d809aa90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009de0 tx=0x7fa3d0038000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 shutdown_connections 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3c40384c0 0x7fa3c403a970 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3d8131140 0x7fa3d81410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d811bbc0 0x7fa3d8130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 --2- 172.21.15.144:0/2780513731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3c0003bc0 0x7fa3d809aa90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 >> 172.21.15.144:0/2780513731 conn(0x7fa3d80713b0 msgr2=0x7fa3d8069770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 shutdown_connections 2024-09-18T12:33:24.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.458+0000 7fa3dd44f640 1 -- 172.21.15.144:0/2780513731 wait complete. 2024-09-18T12:33:24.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:24.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-18T12:33:24.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.481+0000 7f724566f640 10 monclient: get_monmap_and_config 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.481+0000 7f724566f640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.481+0000 7f724566f640 10 monclient: monmap: 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:24.482900+0000 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:24.482900+0000 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:24.484 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:24.484 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:24.484 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.481+0000 7f724566f640 1 Processor -- start 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.481+0000 7f724566f640 1 -- start start 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.481+0000 7f724566f640 10 monclient: init 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f72401318f0 0x7f7240131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): picked mon.noname-c con 0x7f72401318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f724009ea50 0x7f724009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): picked mon.noname-a con 0x7f724009ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f72400691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): picked mon.noname-b con 0x7f7240068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7240131cc0 con 0x7f724009ea50 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f724009fbf0 con 0x7f72401318f0 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724009fe30 con 0x7f7240068d90 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f724566f640 10 monclient(hunting): authenticate will time out at 1231.804366s 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f72400691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f724009ea50 0x7f724009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f72400691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46012/0 (socket says 172.21.15.116:46012) 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f724009ea50 0x7f724009ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55512/0 (socket says 172.21.15.116:55512) 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 1 -- 172.21.15.116:0/1087569885 learned_addr learned my addr 172.21.15.116:0/1087569885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 10 monclient(hunting): get_auth_request con 0x7f7240068d90 auth_method 0 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): get_auth_request con 0x7f724009ea50 auth_method 0 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 1 --2- 172.21.15.116:0/1087569885 >> v2:172.21.15.144:3300/0 conn(0x7f72401318f0 0x7f7240131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 10 monclient(hunting): get_auth_request con 0x7f72401318f0 auth_method 0 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient(hunting): handle_auth_done global_id 104373 payload 274 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 -- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 msgr2=0x7f72400691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 --2- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f72400691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 -- 172.21.15.116:0/1087569885 >> v2:172.21.15.144:3300/0 conn(0x7f72401318f0 msgr2=0x7f7240131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 --2- 172.21.15.116:0/1087569885 >> v2:172.21.15.144:3300/0 conn(0x7f72401318f0 0x7f7240131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 monclient: found mon.noname-a 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723e7fc640 1 -- 172.21.15.116:0/1087569885 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72400a1210 con 0x7f724009ea50 2024-09-18T12:33:24.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.482+0000 7f723f7fe640 1 --2- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f72400691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723e7fc640 1 -- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 msgr2=0x7f724009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723e7fc640 1 --2- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 0x7f724009ee20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7240076600 tx=0x7f7234009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd0eb8c46edbed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 1 -- 172.21.15.116:0/1087569885 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234007760 con 0x7f724009ea50 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: dump: 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.499 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.500 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f724566f640 5 monclient: authenticate success, global_id 104373 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f724566f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 1 -- 172.21.15.116:0/1087569885 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723402d050 con 0x7f724009ea50 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 1 -- 172.21.15.116:0/1087569885 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234007af0 con 0x7f724009ea50 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f724566f640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f724566f640 4 set_mon_vals no callback set 2024-09-18T12:33:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.485+0000 7f723dffb640 10 monclient: dump: 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 10 monclient: shutdown 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 -- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 msgr2=0x7f724009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 --2- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 0x7f724009ee20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7240076600 tx=0x7f7234009780 comp rx=0 tx=0).stop 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 -- 172.21.15.116:0/1087569885 shutdown_connections 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 --2- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f72400691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 --2- 172.21.15.116:0/1087569885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 0x7f724009ee20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 --2- 172.21.15.116:0/1087569885 >> v2:172.21.15.144:3300/0 conn(0x7f72401318f0 0x7f7240131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 -- 172.21.15.116:0/1087569885 >> 172.21.15.116:0/1087569885 conn(0x7f7240067110 msgr2=0x7f7240067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 -- 172.21.15.116:0/1087569885 shutdown_connections 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 1 -- 172.21.15.116:0/1087569885 wait complete. 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.486+0000 7f724566f640 10 monclient: monmap: 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 Processor -- start 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 -- start start 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient: init 2024-09-18T12:33:24.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f724009a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): picked mon.noname-c con 0x7f7240068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 0x7f7240130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): picked mon.noname-a con 0x7f724009ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401318f0 0x7f7240140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): picked mon.noname-b con 0x7f72401318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724009bf10 con 0x7f724009ea50 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724009e6b0 con 0x7f72401318f0 2024-09-18T12:33:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724011df10 con 0x7f7240068d90 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f724566f640 10 monclient(hunting): authenticate will time out at 1231.809739s 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f724009a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 0x7f7240130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f724009a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46026/0 (socket says 172.21.15.116:46026) 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 1 -- 172.21.15.116:0/125812701 learned_addr learned my addr 172.21.15.116:0/125812701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 10 monclient(hunting): get_auth_request con 0x7f7240068d90 auth_method 0 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 10 monclient(hunting): get_auth_request con 0x7f724009ea50 auth_method 0 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723f7fe640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401318f0 0x7f7240140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723f7fe640 10 monclient(hunting): get_auth_request con 0x7f72401318f0 auth_method 0 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.487+0000 7f723f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 10 monclient(hunting): handle_auth_done global_id 65177 payload 824 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401318f0 msgr2=0x7f7240140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401318f0 0x7f7240140d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 msgr2=0x7f7240130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 0x7f7240130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 1 monclient: found mon.noname-c 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723effd640 1 -- 172.21.15.116:0/125812701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7230009450 con 0x7f7240068d90 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.488+0000 7f723e7fc640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724009ea50 0x7f7240130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f723effd640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 msgr2=0x7f724009a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f723effd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f724009a900 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f723800ea50 tx=0x7f72380090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723800cc20 con 0x7f7240068d90 2024-09-18T12:33:24.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: dump: 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f724566f640 5 monclient: authenticate success, global_id 65177 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f724566f640 10 monclient: _renew_subs 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f724566f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7234009a20 con 0x7f7240068d90 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723800cd80 con 0x7f7240068d90 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238013680 con 0x7f7240068d90 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f7226ffd640 10 monclient: dump: 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.511 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f724566f640 10 monclient: _renew_subs 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f724566f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724011bbf0 con 0x7f7240068d90 2024-09-18T12:33:24.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.490+0000 7f723d7fa640 4 set_mon_vals no callback set 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f724566f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7238013a70 con 0x7f7240068d90 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f724566f640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 0x7f7240141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724011ddd0 con 0x7f7240131050 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f7226ffd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f722c038490 0x7f722c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 0x7f7240141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient: get_auth_request con 0x7f7240131050 auth_method 0 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(224..224 src has 13..224) v4 ==== 14695+0+0 (secure 0 0 0) 0x7f723804e2a0 con 0x7f7240068d90 2024-09-18T12:33:24.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723f7fe640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f722c038490 0x7f722c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723f7fe640 10 monclient: get_auth_request con 0x7f722c038490 auth_method 0 2024-09-18T12:33:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 10 monclient(hunting): handle_auth_done global_id 114115 payload 824 2024-09-18T12:33:24.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.491+0000 7f723e7fc640 1 -- 172.21.15.116:0/125812701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f724009b4e0 con 0x7f7240131050 2024-09-18T12:33:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f723f7fe640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f722c038490 0x7f722c03a940 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7240069960 tx=0x7f72300091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f723e7fc640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 msgr2=0x7f7240141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f723e7fc640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 0x7f7240141280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7234034000 tx=0x7f7234037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234041070 con 0x7f7240131050 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 10 monclient: dump: 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7234045410 con 0x7f7240131050 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 msgr2=0x7f7240141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.494+0000 7f7226ffd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 0x7f7240141280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7234034000 tx=0x7f7234037000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.495+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 msgr2=0x7f7240141280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.495+0000 7f7226ffd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240131050 0x7f7240141280 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.500+0000 7f724566f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.500+0000 7f724566f640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 0x7f7240096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.500+0000 7f724566f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.500+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7240065e00 con 0x7f72401417c0 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 0x7f7240096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient: get_auth_request con 0x7f72401417c0 auth_method 0 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 10 monclient(hunting): handle_auth_done global_id 114118 payload 824 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.501+0000 7f723e7fc640 1 -- 172.21.15.116:0/125812701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f724011c8f0 con 0x7f72401417c0 2024-09-18T12:33:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.503+0000 7f723e7fc640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 msgr2=0x7f7240096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.503+0000 7f723e7fc640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 0x7f7240096a60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f72340097c0 tx=0x7f7234041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234057070 con 0x7f72401417c0 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 10 monclient: dump: 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f7234000f40 con 0x7f72401417c0 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 msgr2=0x7f7240096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 0x7f7240096a60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f72340097c0 tx=0x7f7234041000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 msgr2=0x7f7240096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f7226ffd640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72401417c0 0x7f7240096a60 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f72340097c0 tx=0x7f7234041000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 88, 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:33:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:24.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:24.526 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:24.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:24.529 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.530 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:24.531 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:24.532 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:24.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:24.534 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:24.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f722c038490 msgr2=0x7f722c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f722c038490 0x7f722c03a940 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7240069960 tx=0x7f72300091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 10 monclient: shutdown 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 msgr2=0x7f724009a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 --2- 172.21.15.116:0/125812701 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240068d90 0x7f724009a900 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f723800ea50 tx=0x7f72380090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f723f7fe640 1 -- 172.21.15.116:0/125812701 reap_dead start 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f723f7fe640 1 -- 172.21.15.116:0/125812701 reap_dead start 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 shutdown_connections 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 >> 172.21.15.116:0/125812701 conn(0x7f7240067110 msgr2=0x7f724009d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 shutdown_connections 2024-09-18T12:33:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.504+0000 7f724566f640 1 -- 172.21.15.116:0/125812701 wait complete. 2024-09-18T12:33:24.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:33:24.538 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:33:24.538 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:33:24.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.543+0000 7f508cc80640 10 monclient: get_monmap_and_config 2024-09-18T12:33:24.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.543+0000 7f508cc80640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.544 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.543+0000 7f508cc80640 10 monclient: monmap: 2024-09-18T12:33:24.544 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:24.544647+0000 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:24.544647+0000 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:24.545 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 1 Processor -- start 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 1 -- start start 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient: init 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5088067640 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f508809d030 0x7f5088094790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient(hunting): picked mon.noname-b con 0x7f508809d030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5088094cd0 0x7f5088139240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient(hunting): picked mon.noname-c con 0x7f5088094cd0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:24.548 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508809d930 con 0x7f5088067640 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5088099370 con 0x7f5088094cd0 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508806f7d0 con 0x7f508809d030 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.544+0000 7f508cc80640 10 monclient(hunting): authenticate will time out at 1240.113923s 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5088094cd0 0x7f5088139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f508809d030 0x7f5088094790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5088094cd0 0x7f5088139240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55336/0 (socket says 172.21.15.144:55336) 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f508659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5088067640 0x7f5088094080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:39540/0 (socket says 172.21.15.144:39540) 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 -- 172.21.15.144:0/2844952380 learned_addr learned my addr 172.21.15.144:0/2844952380 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:24.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): get_auth_request con 0x7f5088094cd0 auth_method 0 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 10 monclient(hunting): get_auth_request con 0x7f508809d030 auth_method 0 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f507ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient(hunting): handle_auth_done global_id 114121 payload 274 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 -- 172.21.15.144:0/2844952380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f508809d030 msgr2=0x7f5088094790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 --2- 172.21.15.144:0/2844952380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f508809d030 0x7f5088094790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 -- 172.21.15.144:0/2844952380 >> v1:172.21.15.116:6789/0 conn(0x7f5088067640 legacy=0x7f5088094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 monclient: found mon.noname-c 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:33:24.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.545+0000 7f5086d9d640 1 -- 172.21.15.144:0/2844952380 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088067a10 con 0x7f5088094cd0 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5086d9d640 1 -- 172.21.15.144:0/2844952380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088094cd0 msgr2=0x7f5088139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5086d9d640 1 --2- 172.21.15.144:0/2844952380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088094cd0 0x7f5088139240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5080009f20 tx=0x7f5080009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=771d52cfefdc228d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 1 -- 172.21.15.144:0/2844952380 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080007610 con 0x7f5088094cd0 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: dump: 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.557 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 5 monclient: authenticate success, global_id 114121 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 1 -- 172.21.15.144:0/2844952380 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508002d050 con 0x7f5088094cd0 2024-09-18T12:33:24.558 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 1 -- 172.21.15.144:0/2844952380 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800079a0 con 0x7f5088094cd0 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 4 set_mon_vals no callback set 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f5085d9b640 10 monclient: dump: 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:24.559 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 10 monclient: shutdown 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 1 -- 172.21.15.144:0/2844952380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088094cd0 msgr2=0x7f5088139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.548+0000 7f508cc80640 1 --2- 172.21.15.144:0/2844952380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088094cd0 0x7f5088139240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5080009f20 tx=0x7f5080009840 comp rx=0 tx=0).stop 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 -- 172.21.15.144:0/2844952380 shutdown_connections 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 --2- 172.21.15.144:0/2844952380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088094cd0 0x7f5088139240 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 --2- 172.21.15.144:0/2844952380 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f508809d030 0x7f5088094790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 -- 172.21.15.144:0/2844952380 >> 172.21.15.144:0/2844952380 conn(0x7f50880a1630 msgr2=0x7f50880a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.560 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 -- 172.21.15.144:0/2844952380 shutdown_connections 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 -- 172.21.15.144:0/2844952380 wait complete. 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 10 monclient: monmap: 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:24.561 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 Processor -- start 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 1 -- start start 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.549+0000 7f508cc80640 10 monclient: init 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 0x7f5088140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): picked mon.noname-b con 0x7f5088067640 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5088094cd0 0x7f50880941e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): picked mon.noname-c con 0x7f5088094cd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508809d030 0x7f5088094720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): picked mon.noname-a con 0x7f508809d030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508808b920 con 0x7f508809d030 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5088141d20 con 0x7f5088067640 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508809b220 con 0x7f5088094cd0 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508cc80640 10 monclient(hunting): authenticate will time out at 1240.119369s 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 0x7f5088140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 0x7f5088140e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55352/0 (socket says 172.21.15.144:55352) 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 -- 172.21.15.144:0/971973955 learned_addr learned my addr 172.21.15.144:0/971973955 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): get_auth_request con 0x7f5088067640 auth_method 0 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508809d030 0x7f5088094720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5088094cd0 0x7f50880941e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 10 monclient(hunting): get_auth_request con 0x7f5088094cd0 auth_method 0 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 10 monclient(hunting): get_auth_request con 0x7f508809d030 auth_method 0 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f5086d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f507ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient(hunting): handle_auth_done global_id 114124 payload 824 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 -- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5088094cd0 msgr2=0x7f50880941e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5088094cd0 0x7f50880941e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 -- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508809d030 msgr2=0x7f5088094720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508809d030 0x7f5088094720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 monclient: found mon.noname-b 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.550+0000 7f508659c640 1 -- 172.21.15.144:0/971973955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5080009ae0 con 0x7f5088067640 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.552+0000 7f508659c640 1 -- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 msgr2=0x7f5088140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.552+0000 7f508659c640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 0x7f5088140e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f507800c5e0 tx=0x7f5078007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078007510 con 0x7f5088067640 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: dump: 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.566 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50780057d0 con 0x7f5088067640 2024-09-18T12:33:24.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508cc80640 5 monclient: authenticate success, global_id 114124 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508cc80640 10 monclient: _renew_subs 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508cc80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508808d6a0 con 0x7f5088067640 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078007690 con 0x7f5088067640 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 10 monclient: dump: 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.568 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508559a640 4 set_mon_vals no callback set 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508cc80640 10 monclient: _renew_subs 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508cc80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508808d470 con 0x7f5088067640 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.553+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f507801b400 con 0x7f5088067640 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f507e7fc640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5068038490 0x7f506803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f508cc80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f508cc80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f508cc80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:24.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508808dd80 con 0x7f5088067640 2024-09-18T12:33:24.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(224..224 src has 13..224) v4 ==== 14695+0+0 (secure 0 0 0) 0x7f507804d970 con 0x7f5088067640 2024-09-18T12:33:24.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f507ffff640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5068038490 0x7f506803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.554+0000 7f507ffff640 10 monclient: get_auth_request con 0x7f5068038490 auth_method 0 2024-09-18T12:33:24.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.556+0000 7f507ffff640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5068038490 0x7f506803a940 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f508809b050 tx=0x7f50700090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.558+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5078005e20 con 0x7f5088067640 2024-09-18T12:33:24.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.558+0000 7f507e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.558+0000 7f507e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.628+0000 7efe3c00f640 10 monclient: get_monmap_and_config 2024-09-18T12:33:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.628+0000 7efe3c00f640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.628+0000 7efe3c00f640 10 monclient: monmap: 2024-09-18T12:33:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:24.630187+0000 2024-09-18T12:33:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:24.630187+0000 2024-09-18T12:33:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:24.632 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 1 Processor -- start 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 1 -- start start 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient: init 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 0x7efe3409c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient(hunting): picked mon.noname-b con 0x7efe3409ad90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe3409b160 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe3409b530 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe340692c0 con 0x7efe3409b530 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3409b990 con 0x7efe3409b160 2024-09-18T12:33:24.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3409db70 con 0x7efe3409ad90 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3c00f640 10 monclient(hunting): authenticate will time out at 1231.951752s 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 0x7efe3409c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3a5ac640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efe3409b530 0x7efe34132070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44026/0 (socket says 172.21.15.116:44026) 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe3a5ac640 1 -- 172.21.15.116:0/828804984 learned_addr learned my addr 172.21.15.116:0/828804984 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 1 --2- 172.21.15.116:0/828804984 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 0x7efe3409c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46038/0 (socket says 172.21.15.116:46038) 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 10 monclient(hunting): get_auth_request con 0x7efe3409ad90 auth_method 0 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.629+0000 7efe39dab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 10 monclient(hunting): handle_auth_done global_id 65186 payload 274 2024-09-18T12:33:24.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 1 -- 172.21.15.116:0/828804984 >> v1:172.21.15.144:6789/0 conn(0x7efe3409b160 legacy=0x7efe34131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 1 -- 172.21.15.116:0/828804984 >> v1:172.21.15.116:6789/0 conn(0x7efe3409b530 legacy=0x7efe34132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 1 monclient: found mon.noname-b 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.630+0000 7efe39dab640 1 -- 172.21.15.116:0/828804984 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3409d5b0 con 0x7efe3409ad90 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.632+0000 7efe39dab640 1 -- 172.21.15.116:0/828804984 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 msgr2=0x7efe3409c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.632+0000 7efe39dab640 1 --2- 172.21.15.116:0/828804984 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 0x7efe3409c5d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efe30009880 tx=0x7efe300090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c87d2ad001334fd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 1 -- 172.21.15.116:0/828804984 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3002aea0 con 0x7efe3409ad90 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:24.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: dump: 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 1 -- 172.21.15.116:0/828804984 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe30032440 con 0x7efe3409ad90 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 5 monclient: authenticate success, global_id 65186 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 1 -- 172.21.15.116:0/828804984 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe30032820 con 0x7efe3409ad90 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 4 set_mon_vals no callback set 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe38da9640 10 monclient: dump: 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.650 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 10 monclient: shutdown 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 1 -- 172.21.15.116:0/828804984 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 msgr2=0x7efe3409c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 1 --2- 172.21.15.116:0/828804984 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 0x7efe3409c5d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efe30009880 tx=0x7efe300090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 1 -- 172.21.15.116:0/828804984 shutdown_connections 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 1 --2- 172.21.15.116:0/828804984 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409ad90 0x7efe3409c5d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 1 -- 172.21.15.116:0/828804984 >> 172.21.15.116:0/828804984 conn(0x7efe340678a0 msgr2=0x7efe340a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 1 -- 172.21.15.116:0/828804984 shutdown_connections 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.633+0000 7efe3c00f640 1 -- 172.21.15.116:0/828804984 wait complete. 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient: build_initial_monmap 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient: monmap: 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 1 Processor -- start 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 1 -- start start 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient: init 2024-09-18T12:33:24.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe3409ad90 0x7efe34141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient(hunting): picked mon.noname-b con 0x7efe3409ad90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe3409b160 0x7efe34130d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient(hunting): picked mon.noname-a con 0x7efe3409b160 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 0x7efe34127a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.634+0000 7efe3c00f640 10 monclient(hunting): picked mon.noname-c con 0x7efe3409b530 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3412b230 con 0x7efe3409b160 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3412af50 con 0x7efe3409ad90 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3412b510 con 0x7efe3409b530 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3c00f640 10 monclient(hunting): authenticate will time out at 1231.957025s 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe3409b160 0x7efe34130d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 0x7efe34127a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 0x7efe34127a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46054/0 (socket says 172.21.15.116:46054) 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 -- 172.21.15.116:0/3089235600 learned_addr learned my addr 172.21.15.116:0/3089235600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe3409b160 0x7efe34130d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55536/0 (socket says 172.21.15.116:55536) 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): get_auth_request con 0x7efe3409b530 auth_method 0 2024-09-18T12:33:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 10 monclient(hunting): get_auth_request con 0x7efe3409b160 auth_method 0 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe3409ad90 0x7efe34141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 10 monclient(hunting): get_auth_request con 0x7efe3409ad90 auth_method 0 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe395aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe39dab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient(hunting): handle_auth_done global_id 65189 payload 824 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 -- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe3409ad90 msgr2=0x7efe34141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe3409ad90 0x7efe34141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 -- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe3409b160 msgr2=0x7efe34130d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe3409b160 0x7efe34130d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient: _finish_hunting 0 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 monclient: found mon.noname-c 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.635+0000 7efe3a5ac640 1 -- 172.21.15.116:0/3089235600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30009390 con 0x7efe3409b530 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe3a5ac640 1 -- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 msgr2=0x7efe34127a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe3a5ac640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 0x7efe34127a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efe2c00ccc0 tx=0x7efe2c005370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 1 -- 172.21.15.116:0/3089235600 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c00ce80 con 0x7efe3409b530 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: dump: 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe3c00f640 5 monclient: authenticate success, global_id 65189 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe3c00f640 10 monclient: _renew_subs 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 1 -- 172.21.15.116:0/3089235600 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2c00aa60 con 0x7efe3409b530 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe3c00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe34129920 con 0x7efe3409b530 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 1 -- 172.21.15.116:0/3089235600 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c00ae20 con 0x7efe3409b530 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe29ffb640 10 monclient: dump: 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.637+0000 7efe2bfff640 4 set_mon_vals no callback set 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe3c00f640 10 monclient: _renew_subs 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe3c00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe34131440 con 0x7efe3409b530 2024-09-18T12:33:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe29ffb640 1 -- 172.21.15.116:0/3089235600 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7efe2c007300 con 0x7efe3409b530 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe29ffb640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe18038490 0x7efe1803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe3c00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe3c00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe3c00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3404bf60 con 0x7efe3409b530 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.638+0000 7efe29ffb640 1 -- 172.21.15.116:0/3089235600 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(224..224 src has 13..224) v4 ==== 14695+0+0 (secure 0 0 0) 0x7efe2c04e050 con 0x7efe3409b530 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.639+0000 7efe39dab640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe18038490 0x7efe1803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.639+0000 7efe39dab640 10 monclient: get_auth_request con 0x7efe18038490 auth_method 0 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.641+0000 7efe39dab640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe18038490 0x7efe1803a940 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efe30034000 tx=0x7efe30032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.643+0000 7efe29ffb640 1 -- 172.21.15.116:0/3089235600 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7efe2c0075b0 con 0x7efe3409b530 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.643+0000 7efe29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:24.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.643+0000 7efe29ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:24.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.715+0000 7f508cc80640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:24.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.715+0000 7f508cc80640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:24.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.715+0000 7f508cc80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:24.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:24.715+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f508804bf60 con 0x7f5088067640 2024-09-18T12:33:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.856+0000 7efe3c00f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:33:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.856+0000 7efe3c00f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:33:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.856+0000 7efe3c00f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.856+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7efe3412abc0 con 0x7efe3409b530 2024-09-18T12:33:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.858+0000 7efe29ffb640 1 -- 172.21.15.116:0/3089235600 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7efe2c010af0 con 0x7efe3409b530 2024-09-18T12:33:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.858+0000 7efe29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:33:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.858+0000 7efe29ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.858+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.858+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=3092340946,config=3140514009,health=1301148488,kv=2187923472,logm=885594689}) 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.858+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=3340797349,config=3140514009,health=1301148488,kv=2187923472,logm=1382438021}) 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.858+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.858+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=3092340946,config=3140514009,health=1301148488,kv=2187923472,logm=885594689}) 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.858+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=9,logm=59} crc {auth=3092340946,config=3140514009,health=797350388,kv=2187923472,logm=560024583}) 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.859+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.859+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989091419}) 2024-09-18T12:33:24.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.859+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955338200}) 2024-09-18T12:33:24.862 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.859+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.862 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.859+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989091419}) 2024-09-18T12:33:24.862 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.859+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126351473}) 2024-09-18T12:33:24.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe18038490 msgr2=0x7efe1803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe18038490 0x7efe1803a940 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efe30034000 tx=0x7efe30032000 comp rx=0 tx=0).stop 2024-09-18T12:33:24.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 10 monclient: shutdown 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 msgr2=0x7efe34127a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 0x7efe34127a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efe2c00ccc0 tx=0x7efe2c005370 comp rx=0 tx=0).stop 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 shutdown_connections 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe18038490 0x7efe1803a940 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe3409b530 0x7efe34127a40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe3409b160 0x7efe34130d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 --2- 172.21.15.116:0/3089235600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe3409ad90 0x7efe34141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 >> 172.21.15.116:0/3089235600 conn(0x7efe340678a0 msgr2=0x7efe3409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 shutdown_connections 2024-09-18T12:33:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:24.860+0000 7efe3c00f640 1 -- 172.21.15.116:0/3089235600 wait complete. 2024-09-18T12:33:24.869 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.860+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.869 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.860+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013437392}) 2024-09-18T12:33:24.869 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.860+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260585450}) 2024-09-18T12:33:24.869 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.860+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.869 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.860+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013437392}) 2024-09-18T12:33:24.869 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.860+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1074078883}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194875341}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855368826}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194875341}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304948690}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955918626}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=583072134}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955918626}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.861+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662741019}) 2024-09-18T12:33:24.870 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.862+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.862+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=268464442}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.862+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39908043}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.862+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.862+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=268464442}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.862+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871688051}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749633598}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2574277499}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749633598}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320466238}) 2024-09-18T12:33:24.871 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.863+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2064597811}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=768987138}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2064597811}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4035122823}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782492041}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2455765276}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782492041}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.864+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088343002}) 2024-09-18T12:33:24.872 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.865+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.865+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426424508}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.865+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128007996}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.865+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.865+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426424508}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.865+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212747225}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568120766}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=356015175}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568120766}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763649398}) 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.873 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953093581}) 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352770477}) 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953093581}) 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.866+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2859526048}) 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.867+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.867+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1221617707}) 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.867+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1662722302}) 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.867+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.867+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1221617707}) 2024-09-18T12:33:24.874 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.867+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096780412}) 2024-09-18T12:33:24.876 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.876 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063434885}) 2024-09-18T12:33:24.876 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539347443}) 2024-09-18T12:33:24.876 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.876 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063434885}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883212893}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85965702}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243768525}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85965702}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.868+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3055891040}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.869+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.869+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149178788}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.869+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239688333}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.869+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.869+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149178788}) 2024-09-18T12:33:24.877 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.869+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320099040}) 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761030876}) 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3039992864}) 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761030876}) 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783568871}) 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949748784}) 2024-09-18T12:33:24.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2235092840}) 2024-09-18T12:33:24.879 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:33:24.879 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:24.879 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949748784}) 2024-09-18T12:33:24.879 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:24.870+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42721028}) 2024-09-18T12:33:25.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.169+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.169+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291979252}) 2024-09-18T12:33:25.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.169+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3737768313}) 2024-09-18T12:33:25.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.169+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.169+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291979252}) 2024-09-18T12:33:25.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.169+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684335714}) 2024-09-18T12:33:25.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.174+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.174+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364712869}) 2024-09-18T12:33:25.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.174+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892228531}) 2024-09-18T12:33:25.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.174+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.174+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364712869}) 2024-09-18T12:33:25.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.174+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434194954}) 2024-09-18T12:33:25.356 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.354+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.356 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.354+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527547351}) 2024-09-18T12:33:25.356 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.354+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3580915463}) 2024-09-18T12:33:25.356 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.354+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.356 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.354+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527547351}) 2024-09-18T12:33:25.356 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.354+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=640487462}) 2024-09-18T12:33:25.358 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.356+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.359 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.356+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1516359282}) 2024-09-18T12:33:25.359 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.357+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787656387}) 2024-09-18T12:33:25.359 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.357+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.359 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.357+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1516359282}) 2024-09-18T12:33:25.359 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.357+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722304206}) 2024-09-18T12:33:25.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.360+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.360+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160502500}) 2024-09-18T12:33:25.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.360+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801471508}) 2024-09-18T12:33:25.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.360+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.360+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160502500}) 2024-09-18T12:33:25.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.360+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2946928865}) 2024-09-18T12:33:25.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.363+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v225) v1 ==== 168+0+0 (secure 0 0 0) 0x7f507801a360 con 0x7f5088067640 2024-09-18T12:33:25.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.363+0000 7f507e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:25.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.363+0000 7f507e7fc640 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-18T12:33:25.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.459+0000 7f508cc80640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:25.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.459+0000 7f508cc80640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:25.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.459+0000 7f508cc80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:25.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.459+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5088064a70 con 0x7f5088067640 2024-09-18T12:33:25.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.460+0000 7f507e7fc640 1 -- 172.21.15.144:0/971973955 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v225) v1 ==== 147+0+0 (secure 0 0 0) 0x7f507801a540 con 0x7f5088067640 2024-09-18T12:33:25.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.460+0000 7f507e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T12:33:25.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.460+0000 7f507e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-18T12:33:25.462 INFO:tasks.workunit.client.0.smithi144.stderr:bucket 'h32' already exists 2024-09-18T12:33:25.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5068038490 msgr2=0x7f506803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:25.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5068038490 0x7f506803a940 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f508809b050 tx=0x7f50700090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 10 monclient: shutdown 2024-09-18T12:33:25.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:25.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 msgr2=0x7f5088140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:25.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 0x7f5088140e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f507800c5e0 tx=0x7f5078007f00 comp rx=0 tx=0).stop 2024-09-18T12:33:25.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 shutdown_connections 2024-09-18T12:33:25.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5068038490 0x7f506803a940 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508809d030 0x7f5088094720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5088094cd0 0x7f50880941e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 --2- 172.21.15.144:0/971973955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5088067640 0x7f5088140e30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 >> 172.21.15.144:0/971973955 conn(0x7f50880a1630 msgr2=0x7f5088128710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:25.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 shutdown_connections 2024-09-18T12:33:25.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.462+0000 7f508cc80640 1 -- 172.21.15.144:0/971973955 wait complete. 2024-09-18T12:33:25.472 INFO:tasks.workunit.client.0.smithi144.stderr:++ seq 1 32 2024-09-18T12:33:25.473 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:25.473 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h1 2024-09-18T12:33:25.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.545+0000 7f297acc0640 10 monclient: get_monmap_and_config 2024-09-18T12:33:25.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.545+0000 7f297acc0640 10 monclient: build_initial_monmap 2024-09-18T12:33:25.546 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.545+0000 7f297acc0640 10 monclient: monmap: 2024-09-18T12:33:25.546 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:25.546 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.546 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:25.546025+0000 2024-09-18T12:33:25.546 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:25.546025+0000 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:25.547 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:25.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.545+0000 7f297acc0640 1 Processor -- start 2024-09-18T12:33:25.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.545+0000 7f297acc0640 1 -- start start 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.545+0000 7f297acc0640 10 monclient: init 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f297409ec50 0x7f2974131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient(hunting): picked mon.noname-c con 0x7f297409ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29741321a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f297409f020 0x7f2974069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient(hunting): picked mon.noname-a con 0x7f297409f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:25.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2974067420 con 0x7f29741321a0 2024-09-18T12:33:25.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29740a0480 con 0x7f297409f020 2024-09-18T12:33:25.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29740a0200 con 0x7f297409ec50 2024-09-18T12:33:25.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:25.551 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297acc0640 10 monclient(hunting): authenticate will time out at 1241.115499s 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f297409ec50 0x7f2974131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f297409ec50 0x7f2974131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55366/0 (socket says 172.21.15.144:55366) 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 -- 172.21.15.144:0/1758877447 learned_addr learned my addr 172.21.15.144:0/1758877447 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): get_auth_request con 0x7f297409ec50 auth_method 0 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 1 --2- 172.21.15.144:0/1758877447 >> v2:172.21.15.116:3300/0 conn(0x7f297409f020 0x7f2974069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 10 monclient(hunting): get_auth_request con 0x7f297409f020 auth_method 0 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f297925d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient(hunting): handle_auth_done global_id 114130 payload 274 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 -- 172.21.15.144:0/1758877447 >> v2:172.21.15.116:3300/0 conn(0x7f297409f020 msgr2=0x7f2974069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 --2- 172.21.15.144:0/1758877447 >> v2:172.21.15.116:3300/0 conn(0x7f297409f020 0x7f2974069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 -- 172.21.15.144:0/1758877447 >> v1:172.21.15.116:6789/0 conn(0x7f29741321a0 legacy=0x7f2974068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:25.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:25.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 monclient: found mon.noname-c 2024-09-18T12:33:25.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:25.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:33:25.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.546+0000 7f2978a5c640 1 -- 172.21.15.144:0/1758877447 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297409fd80 con 0x7f297409ec50 2024-09-18T12:33:25.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f2978a5c640 1 -- 172.21.15.144:0/1758877447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f297409ec50 msgr2=0x7f2974131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:25.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f2978a5c640 1 --2- 172.21.15.144:0/1758877447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f297409ec50 0x7f2974131c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f29740719d0 tx=0x7f296c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cf5a87d943c9460 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:25.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 1 -- 172.21.15.144:0/1758877447 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c007610 con 0x7f297409ec50 2024-09-18T12:33:25.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:25.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:25.561 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: dump: 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:25.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 1 -- 172.21.15.144:0/1758877447 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296c02d050 con 0x7f297409ec50 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f297acc0640 5 monclient: authenticate success, global_id 114130 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f297acc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 1 -- 172.21.15.144:0/1758877447 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c0079a0 con 0x7f297409ec50 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f29737fe640 10 monclient: dump: 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.563 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f297acc0640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.549+0000 7f297acc0640 4 set_mon_vals no callback set 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 10 monclient: shutdown 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:25.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 -- 172.21.15.144:0/1758877447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f297409ec50 msgr2=0x7f2974131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 --2- 172.21.15.144:0/1758877447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f297409ec50 0x7f2974131c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f29740719d0 tx=0x7f296c009840 comp rx=0 tx=0).stop 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 -- 172.21.15.144:0/1758877447 shutdown_connections 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 --2- 172.21.15.144:0/1758877447 >> v2:172.21.15.116:3300/0 conn(0x7f297409f020 0x7f2974069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 --2- 172.21.15.144:0/1758877447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f297409ec50 0x7f2974131c60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 -- 172.21.15.144:0/1758877447 >> 172.21.15.144:0/1758877447 conn(0x7f29740a5cf0 msgr2=0x7f29740a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 -- 172.21.15.144:0/1758877447 shutdown_connections 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 1 -- 172.21.15.144:0/1758877447 wait complete. 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 10 monclient: build_initial_monmap 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.550+0000 7f297acc0640 10 monclient: monmap: 2024-09-18T12:33:25.565 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 Processor -- start 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 -- start start 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient: init 2024-09-18T12:33:25.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f297409ec50 0x7f297409acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): picked mon.noname-c con 0x7f297409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f297409f020 0x7f2974130d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): picked mon.noname-a con 0x7f297409f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 0x7f297411b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): picked mon.noname-b con 0x7f29741321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297409c020 con 0x7f297409f020 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297409e120 con 0x7f29741321a0 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297409b850 con 0x7f297409ec50 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297acc0640 10 monclient(hunting): authenticate will time out at 1241.120852s 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297925d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 0x7f297411b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297925d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 0x7f297411b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55368/0 (socket says 172.21.15.144:55368) 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297925d640 1 -- 172.21.15.144:0/1125967505 learned_addr learned my addr 172.21.15.144:0/1125967505 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297925d640 10 monclient(hunting): get_auth_request con 0x7f29741321a0 auth_method 0 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297925d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297925d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f297925d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f2978a5c640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f297409ec50 0x7f297409acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2978a5c640 10 monclient(hunting): get_auth_request con 0x7f297409ec50 auth_method 0 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2978a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2978a5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2978a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.551+0000 7f2973fff640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f297409f020 0x7f2974130d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2973fff640 10 monclient(hunting): get_auth_request con 0x7f297409f020 auth_method 0 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2973fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2978a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2978a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2978a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f2973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 10 monclient(hunting): handle_auth_done global_id 114133 payload 824 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 1 -- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f297409ec50 msgr2=0x7f297409acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f297409ec50 0x7f297409acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 1 -- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f297409f020 msgr2=0x7f2974130d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f297409f020 0x7f2974130d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 1 monclient: found mon.noname-b 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.552+0000 7f297925d640 1 -- 172.21.15.144:0/1125967505 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c009ae0 con 0x7f29741321a0 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297925d640 1 -- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 msgr2=0x7f297411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:25.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297925d640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 0x7f297411b760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f295c00ea50 tx=0x7f295c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c00cb50 con 0x7f29741321a0 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: dump: 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:25.572 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:25.573 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:25.573 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:25.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:25.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:25.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297acc0640 5 monclient: authenticate success, global_id 114133 2024-09-18T12:33:25.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297acc0640 10 monclient: _renew_subs 2024-09-18T12:33:25.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c00ccb0 con 0x7f29741321a0 2024-09-18T12:33:25.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297acc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:25.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297411e100 con 0x7f29741321a0 2024-09-18T12:33:25.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:25.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c013470 con 0x7f29741321a0 2024-09-18T12:33:25.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:25.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:25.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2970ff9640 10 monclient: dump: 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297acc0640 10 monclient: _renew_subs 2024-09-18T12:33:25.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f2972ffd640 4 set_mon_vals no callback set 2024-09-18T12:33:25.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297acc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:25.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.554+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2974141670 con 0x7f29741321a0 2024-09-18T12:33:25.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f295c013880 con 0x7f29741321a0 2024-09-18T12:33:25.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f297acc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:25.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f297acc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f297acc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f2970ff9640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f29540384c0 0x7f295403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29741418f0 con 0x7f29741321a0 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(225..225 src has 13..225) v4 ==== 14908+0+0 (secure 0 0 0) 0x7f295c04e2c0 con 0x7f29741321a0 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f2978a5c640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f29540384c0 0x7f295403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.555+0000 7f2978a5c640 10 monclient: get_auth_request con 0x7f29540384c0 auth_method 0 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.558+0000 7f2978a5c640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f29540384c0 0x7f295403a970 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f29741325a0 tx=0x7f296c036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:25.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.560+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f295c023ab0 con 0x7f29741321a0 2024-09-18T12:33:25.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.560+0000 7f2970ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:25.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.560+0000 7f2970ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:25.611 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.610+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.611 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.610+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592246983}) 2024-09-18T12:33:25.612 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.610+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4259242723}) 2024-09-18T12:33:25.612 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.610+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.612 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.610+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592246983}) 2024-09-18T12:33:25.612 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.610+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347503601}) 2024-09-18T12:33:25.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.718+0000 7f297acc0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T12:33:25.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.718+0000 7f297acc0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T12:33:25.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.718+0000 7f297acc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:25.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:25.718+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f29740008d0 con 0x7f29741321a0 2024-09-18T12:33:25.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.826+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.826+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211420110}) 2024-09-18T12:33:25.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.826+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760395574}) 2024-09-18T12:33:25.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.826+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:25.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.826+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211420110}) 2024-09-18T12:33:25.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:25.826+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237476156}) 2024-09-18T12:33:25.880 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:33:25.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.976+0000 7fbe59c17640 10 monclient: get_monmap_and_config 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.976+0000 7fbe59c17640 10 monclient: build_initial_monmap 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.976+0000 7fbe59c17640 10 monclient: monmap: 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:25.977877+0000 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:25.977877+0000 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:25.979 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:25.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.976+0000 7fbe59c17640 1 Processor -- start 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.976+0000 7fbe59c17640 1 -- start start 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.976+0000 7fbe59c17640 10 monclient: init 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe54131ab0 0x7fbe54131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient(hunting): picked mon.noname-a con 0x7fbe54131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe5409e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient(hunting): picked mon.noname-b con 0x7fbe54136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5409faf0 con 0x7fbe5409e910 2024-09-18T12:33:25.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5409fd30 con 0x7fbe54131ab0 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe540a1110 con 0x7fbe54136bb0 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe59c17640 10 monclient(hunting): authenticate will time out at 1233.299524s 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe54131ab0 0x7fbe54131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe54131ab0 0x7fbe54131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36644/0 (socket says 172.21.15.116:36644) 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 -- 172.21.15.116:0/1620114638 learned_addr learned my addr 172.21.15.116:0/1620114638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): get_auth_request con 0x7fbe54131ab0 auth_method 0 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 1 --2- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53044/0 (socket says 172.21.15.116:53044) 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 10 monclient(hunting): get_auth_request con 0x7fbe54136bb0 auth_method 0 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient(hunting): handle_auth_done global_id 104391 payload 274 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 -- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 msgr2=0x7fbe54138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 --2- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 -- 172.21.15.116:0/1620114638 >> v1:172.21.15.144:6789/0 conn(0x7fbe5409e910 legacy=0x7fbe5409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe537fe640 1 -- 172.21.15.116:0/1620114638 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe540a13f0 con 0x7fbe54131ab0 2024-09-18T12:33:25.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.977+0000 7fbe53fff640 1 --2- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:25.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.980+0000 7fbe537fe640 1 -- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe54131ab0 msgr2=0x7fbe54131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:25.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.980+0000 7fbe537fe640 1 --2- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe54131ab0 0x7fbe54131e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009910 tx=0x7fbe440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40a8d5731640bc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:25.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 1 -- 172.21.15.116:0/1620114638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44032440 con 0x7fbe54131ab0 2024-09-18T12:33:25.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:25.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:25.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: dump: 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:25.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 5 monclient: authenticate success, global_id 104391 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 1 -- 172.21.15.116:0/1620114638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe440325a0 con 0x7fbe54131ab0 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 1 -- 172.21.15.116:0/1620114638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44032940 con 0x7fbe54131ab0 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 4 set_mon_vals no callback set 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe527fc640 10 monclient: dump: 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:25.993 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 10 monclient: shutdown 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 -- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe54131ab0 msgr2=0x7fbe54131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 --2- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe54131ab0 0x7fbe54131e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009910 tx=0x7fbe440091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 -- 172.21.15.116:0/1620114638 shutdown_connections 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 --2- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 --2- 172.21.15.116:0/1620114638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe54131ab0 0x7fbe54131e80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 -- 172.21.15.116:0/1620114638 >> 172.21.15.116:0/1620114638 conn(0x7fbe54067d30 msgr2=0x7fbe540a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 -- 172.21.15.116:0/1620114638 shutdown_connections 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.981+0000 7fbe59c17640 1 -- 172.21.15.116:0/1620114638 wait complete. 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.982+0000 7fbe59c17640 10 monclient: build_initial_monmap 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.982+0000 7fbe59c17640 10 monclient: monmap: 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:25.995 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.982+0000 7fbe59c17640 1 Processor -- start 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.982+0000 7fbe59c17640 1 -- start start 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.982+0000 7fbe59c17640 10 monclient: init 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 0x7fbe540973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe54131ab0 0x7fbe54097930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): picked mon.noname-b con 0x7fbe54131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54097e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): picked mon.noname-c con 0x7fbe54136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe54099920 con 0x7fbe5409e910 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe540992d0 con 0x7fbe54131ab0 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5409bde0 con 0x7fbe54136bb0 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe59c17640 10 monclient(hunting): authenticate will time out at 1233.305111s 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 0x7fbe540973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 0x7fbe540973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36646/0 (socket says 172.21.15.116:36646) 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 -- 172.21.15.116:0/2682005075 learned_addr learned my addr 172.21.15.116:0/2682005075 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54097e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): get_auth_request con 0x7fbe5409e910 auth_method 0 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 10 monclient(hunting): get_auth_request con 0x7fbe54136bb0 auth_method 0 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe54131ab0 0x7fbe54097930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 10 monclient(hunting): get_auth_request con 0x7fbe54131ab0 auth_method 0 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:25.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient(hunting): handle_auth_done global_id 104394 payload 824 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 -- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 msgr2=0x7fbe54097e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54097e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 -- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe54131ab0 msgr2=0x7fbe54097930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe54131ab0 0x7fbe54097930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe537fe640 1 -- 172.21.15.116:0/2682005075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c009a20 con 0x7fbe5409e910 2024-09-18T12:33:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.983+0000 7fbe53fff640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54097e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe537fe640 1 -- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 msgr2=0x7fbe540973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe537fe640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 0x7fbe540973f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009970 tx=0x7fbe44032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/2682005075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe440319c0 con 0x7fbe5409e910 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: dump: 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:26.001 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe59c17640 5 monclient: authenticate success, global_id 104394 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe59c17640 10 monclient: _renew_subs 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe59c17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe44009450 con 0x7fbe5409e910 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/2682005075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe44031b40 con 0x7fbe5409e910 2024-09-18T12:33:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/2682005075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4403d440 con 0x7fbe5409e910 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe3b7fe640 10 monclient: dump: 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.986+0000 7fbe59c17640 10 monclient: _renew_subs 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.986+0000 7fbe59c17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.986+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5409a810 con 0x7fbe5409e910 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.985+0000 7fbe51ffb640 4 set_mon_vals no callback set 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.986+0000 7fbe537fe640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 0x7fbe540973f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009970 tx=0x7fbe44032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fbe4406d890 osd_map(225..225 src has 13..225) v4 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.986+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/2682005075 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbe44039040 con 0x7fbe5409e910 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.987+0000 7fbe59c17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.987+0000 7fbe59c17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.987+0000 7fbe59c17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.987+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5404bf60 con 0x7fbe5409e910 2024-09-18T12:33:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.987+0000 7fbe3b7fe640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe40038490 0x7fbe4003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.987+0000 7fbe52ffd640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe40038490 0x7fbe4003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.987+0000 7fbe52ffd640 10 monclient: get_auth_request con 0x7fbe40038490 auth_method 0 2024-09-18T12:33:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.990+0000 7fbe52ffd640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe40038490 0x7fbe4003a940 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbe54098950 tx=0x7fbe48006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:25.991+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/2682005075 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(225..225 src has 13..225) v4 ==== 14908+0+0 (secure 0 0 0) 0x7fbe4406d890 con 0x7fbe5409e910 2024-09-18T12:33:26.144 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.145 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239270650}) 2024-09-18T12:33:26.145 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1896571943}) 2024-09-18T12:33:26.145 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.145 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239270650}) 2024-09-18T12:33:26.145 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712997889}) 2024-09-18T12:33:26.146 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517650853}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799628851}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517650853}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014640498}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226159317}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434084679}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1226159317}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647060051}) 2024-09-18T12:33:26.147 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.148 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1015076638}) 2024-09-18T12:33:26.148 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3723551664}) 2024-09-18T12:33:26.148 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.148 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1015076638}) 2024-09-18T12:33:26.148 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693283760}) 2024-09-18T12:33:26.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3378138415}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3300741460}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3378138415}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3131519265}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677618563}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460318366}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677618563}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719912339}) 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826010534}) 2024-09-18T12:33:26.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3507259562}) 2024-09-18T12:33:26.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826010534}) 2024-09-18T12:33:26.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499244868}) 2024-09-18T12:33:26.164 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470521048}) 2024-09-18T12:33:26.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=555748266}) 2024-09-18T12:33:26.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470521048}) 2024-09-18T12:33:26.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=394423578}) 2024-09-18T12:33:26.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.172+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.172+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107633354}) 2024-09-18T12:33:26.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.172+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=437757451}) 2024-09-18T12:33:26.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.172+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.172+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107633354}) 2024-09-18T12:33:26.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.172+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151860654}) 2024-09-18T12:33:26.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.177+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.177+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744936213}) 2024-09-18T12:33:26.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.177+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3177397948}) 2024-09-18T12:33:26.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.177+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.177+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744936213}) 2024-09-18T12:33:26.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.177+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38319242}) 2024-09-18T12:33:26.361 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.359+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.361 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.359+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104358140}) 2024-09-18T12:33:26.361 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.359+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447111828}) 2024-09-18T12:33:26.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.359+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.359+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104358140}) 2024-09-18T12:33:26.362 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.359+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1089671532}) 2024-09-18T12:33:26.365 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.363+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.365 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.363+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449298174}) 2024-09-18T12:33:26.365 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.363+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841524552}) 2024-09-18T12:33:26.365 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.363+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.365 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.363+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449298174}) 2024-09-18T12:33:26.365 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.363+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1525786184}) 2024-09-18T12:33:26.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.367+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v226) v1 ==== 118+0+0 (secure 0 0 0) 0x7f295c0232f0 con 0x7f29741321a0 2024-09-18T12:33:26.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.367+0000 7f2970ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T12:33:26.368 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.367+0000 7f2970ff9640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-18T12:33:26.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016807951}) 2024-09-18T12:33:26.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=934427111}) 2024-09-18T12:33:26.410 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.410 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1016807951}) 2024-09-18T12:33:26.410 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.407+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501960509}) 2024-09-18T12:33:26.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021987721}) 2024-09-18T12:33:26.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3064741424}) 2024-09-18T12:33:26.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021987721}) 2024-09-18T12:33:26.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466502627}) 2024-09-18T12:33:26.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=59042532,mdsmap=1531151339,mgr=3608348820}) 2024-09-18T12:33:26.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=1224651511,mdsmap=1531151339,mgr=3280793501}) 2024-09-18T12:33:26.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=59042532,mdsmap=1531151339,mgr=3608348820}) 2024-09-18T12:33:26.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=87,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1} crc {logm=1424792811,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319}) 2024-09-18T12:33:26.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.421 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=73} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4015377062,mon_config_key=3655569570,monmap=3750608922,osd_metadata=4229638783,osd_pg_creating=3165743630,osd_snap=3823785925,osdmap=4244705497}) 2024-09-18T12:33:26.421 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=69} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=198565790,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=3165743630,osd_snap=3823785925,osdmap=2502691266}) 2024-09-18T12:33:26.421 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.421 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=73} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4015377062,mon_config_key=3655569570,monmap=3750608922,osd_metadata=4229638783,osd_pg_creating=3165743630,osd_snap=3823785925,osdmap=4244705497}) 2024-09-18T12:33:26.421 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.418+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=3,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=71} crc {mgr=543305786,mgr_metadata=3239289319,mgrstat=198565790,mon_config_key=3655569570,monmap=3140514009,osd_metadata=3127564625,osd_pg_creating=3165743630,osd_snap=2033388227,osdmap=1742327717}) 2024-09-18T12:33:26.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1642555256}) 2024-09-18T12:33:26.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2260122497}) 2024-09-18T12:33:26.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1642555256}) 2024-09-18T12:33:26.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.419+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2378383510}) 2024-09-18T12:33:26.422 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1390162943}) 2024-09-18T12:33:26.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151279046}) 2024-09-18T12:33:26.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1390162943}) 2024-09-18T12:33:26.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.420+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=948966158}) 2024-09-18T12:33:26.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=57} crc {osdmap=3337196473}) 2024-09-18T12:33:26.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=57} crc {osdmap=2359069713}) 2024-09-18T12:33:26.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:33:26.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.424+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=57} crc {osdmap=3337196473}) 2024-09-18T12:33:26.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:33:26.425+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=1010649709}) 2024-09-18T12:33:26.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.466+0000 7f297acc0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T12:33:26.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.466+0000 7f297acc0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T12:33:26.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.466+0000 7f297acc0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:26.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.466+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f29740a13e0 con 0x7f29741321a0 2024-09-18T12:33:26.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.467+0000 7f2970ff9640 1 -- 172.21.15.144:0/1125967505 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v226) v1 ==== 118+0+0 (secure 0 0 0) 0x7f295c0234d0 con 0x7f29741321a0 2024-09-18T12:33:26.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.467+0000 7f2970ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T12:33:26.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.467+0000 7f2970ff9640 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-18T12:33:26.468 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h1' does not appear in the crush map 2024-09-18T12:33:26.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f29540384c0 msgr2=0x7f295403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:26.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f29540384c0 0x7f295403a970 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f29741325a0 tx=0x7f296c036000 comp rx=0 tx=0).stop 2024-09-18T12:33:26.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 10 monclient: shutdown 2024-09-18T12:33:26.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:26.470 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 msgr2=0x7f297411b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 0x7f297411b760 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f295c00ea50 tx=0x7f295c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 shutdown_connections 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f29540384c0 0x7f295403a970 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29741321a0 0x7f297411b760 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f297409f020 0x7f2974130d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 --2- 172.21.15.144:0/1125967505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f297409ec50 0x7f297409acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 >> 172.21.15.144:0/1125967505 conn(0x7f29740a5cf0 msgr2=0x7f29740a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 shutdown_connections 2024-09-18T12:33:26.471 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.469+0000 7f297acc0640 1 -- 172.21.15.144:0/1125967505 wait complete. 2024-09-18T12:33:26.478 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:26.479 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h2 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.551+0000 7f83ace68640 10 monclient: get_monmap_and_config 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.551+0000 7f83ace68640 10 monclient: build_initial_monmap 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.551+0000 7f83ace68640 10 monclient: monmap: 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:26.552001+0000 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:26.552001+0000 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:26.552 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:26.553 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:26.553 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:26.553 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:26.553 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:26.553 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:26.553 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.551+0000 7f83ace68640 1 Processor -- start 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.551+0000 7f83ace68640 1 -- start start 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.551+0000 7f83ace68640 10 monclient: init 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83a809ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a809f020 0x7f83a8069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient(hunting): picked mon.noname-b con 0x7f83a809f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83a8136d60 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:26.555 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a8067420 con 0x7f83a809ec50 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a80a0480 con 0x7f83a8136d60 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a809f3f0 con 0x7f83a809f020 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83ace68640 10 monclient(hunting): authenticate will time out at 1242.121439s 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a6d9d640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f83a8136d60 0x7f83a8139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:34170/0 (socket says 172.21.15.144:34170) 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a6d9d640 1 -- 172.21.15.144:0/323508550 learned_addr learned my addr 172.21.15.144:0/323508550 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 1 --2- 172.21.15.144:0/323508550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a809f020 0x7f83a8069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): get_auth_request con 0x7f83a809f020 auth_method 0 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:26.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient(hunting): handle_auth_done global_id 65201 payload 274 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 1 -- 172.21.15.144:0/323508550 >> v1:172.21.15.144:6789/0 conn(0x7f83a8136d60 legacy=0x7f83a8139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 1 -- 172.21.15.144:0/323508550 >> v1:172.21.15.116:6789/0 conn(0x7f83a809ec50 legacy=0x7f83a8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 1 monclient: found mon.noname-b 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:26.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.552+0000 7f83a5d9b640 1 -- 172.21.15.144:0/323508550 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80a0200 con 0x7f83a809f020 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.555+0000 7f83a5d9b640 1 -- 172.21.15.144:0/323508550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a809f020 msgr2=0x7f83a8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.555+0000 7f83a5d9b640 1 --2- 172.21.15.144:0/323508550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a809f020 0x7f83a8069d10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f839c009880 tx=0x7f839c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce9527d80a8470d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 1 -- 172.21.15.144:0/323508550 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c02aea0 con 0x7f83a809f020 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: dump: 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:26.562 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: _finish_auth 0 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 5 monclient: authenticate success, global_id 65201 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 1 -- 172.21.15.144:0/323508550 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c032440 con 0x7f83a809f020 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:26.563 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 4 set_mon_vals no callback set 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 1 -- 172.21.15.144:0/323508550 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c032820 con 0x7f83a809f020 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83a559a640 10 monclient: dump: 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:26.564 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 10 monclient: shutdown 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 1 -- 172.21.15.144:0/323508550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a809f020 msgr2=0x7f83a8069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 1 --2- 172.21.15.144:0/323508550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a809f020 0x7f83a8069d10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f839c009880 tx=0x7f839c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 1 -- 172.21.15.144:0/323508550 shutdown_connections 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 1 --2- 172.21.15.144:0/323508550 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a809f020 0x7f83a8069d10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 1 -- 172.21.15.144:0/323508550 >> 172.21.15.144:0/323508550 conn(0x7f83a80a5cf0 msgr2=0x7f83a80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 1 -- 172.21.15.144:0/323508550 shutdown_connections 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.556+0000 7f83ace68640 1 -- 172.21.15.144:0/323508550 wait complete. 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient: build_initial_monmap 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient: monmap: 2024-09-18T12:33:26.565 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:26.566 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 Processor -- start 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 -- start start 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient: init 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 0x7f83a8141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): picked mon.noname-b con 0x7f83a809ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f83a8130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): picked mon.noname-a con 0x7f83a809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f83a8099c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): picked mon.noname-c con 0x7f83a8136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80a45e0 con 0x7f83a809f020 2024-09-18T12:33:26.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a811d3b0 con 0x7f83a809ec50 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a811d0d0 con 0x7f83a8136d60 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83ace68640 10 monclient(hunting): authenticate will time out at 1242.126927s 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.557+0000 7f83a659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 0x7f83a8141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 0x7f83a8141340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55380/0 (socket says 172.21.15.144:55380) 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 learned_addr learned my addr 172.21.15.144:0/1335661995 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): get_auth_request con 0x7f83a809ec50 auth_method 0 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:26.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f83a8130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f83a8099c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 10 monclient(hunting): get_auth_request con 0x7f83a809f020 auth_method 0 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 10 monclient(hunting): get_auth_request con 0x7f83a8136d60 auth_method 0 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:26.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient(hunting): handle_auth_done global_id 114139 payload 824 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 msgr2=0x7f83a8099c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f83a8099c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 msgr2=0x7f83a8130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f83a8130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:26.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 monclient: found mon.noname-b 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.558+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c009390 con 0x7f83a809ec50 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 msgr2=0x7f83a8141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 0x7f83a8141340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f839800c8e0 tx=0x7f83980026c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839800aa60 con 0x7f83a809ec50 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: dump: 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:26.571 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839800abe0 con 0x7f83a809ec50 2024-09-18T12:33:26.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83ace68640 5 monclient: authenticate success, global_id 114139 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83ace68640 10 monclient: _renew_subs 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83ace68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a81421f0 con 0x7f83a809ec50 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398012420 con 0x7f83a809ec50 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83967fc640 10 monclient: dump: 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:26.573 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83a4d99640 4 set_mon_vals no callback set 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83ace68640 10 monclient: _renew_subs 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83ace68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:26.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.560+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a809b700 con 0x7f83a809ec50 2024-09-18T12:33:26.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.561+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8398012890 con 0x7f83a809ec50 2024-09-18T12:33:26.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.561+0000 7f83ace68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.561+0000 7f83ace68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.561+0000 7f83ace68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.561+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a81363b0 con 0x7f83a809ec50 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.561+0000 7f83967fc640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8388038490 0x7f838803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.562+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(226..226 src has 13..226) v4 ==== 14698+0+0 (secure 0 0 0) 0x7f839804e360 con 0x7f83a809ec50 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.562+0000 7f83a5d9b640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8388038490 0x7f838803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.562+0000 7f83a5d9b640 10 monclient: get_auth_request con 0x7f8388038490 auth_method 0 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.564+0000 7f83a5d9b640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8388038490 0x7f838803a940 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f839c034000 tx=0x7f839c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.566+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f83a81363b0 con 0x7f83a809ec50 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.566+0000 7f83967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:26.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.566+0000 7f83967fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:26.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.728+0000 7f83ace68640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T12:33:26.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.728+0000 7f83ace68640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T12:33:26.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.728+0000 7f83ace68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:26.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:26.728+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f83a81363b0 con 0x7f83a809ec50 2024-09-18T12:33:26.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:26.815+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/2682005075 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbe44044ac0 con 0x7fbe5409e910 2024-09-18T12:33:26.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:26.815+0000 7fbe3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:26.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:26.815+0000 7fbe3b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.028+0000 7fbe59c17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.028+0000 7fbe59c17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.028+0000 7fbe59c17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.028+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe5409abc0 con 0x7fbe5409e910 2024-09-18T12:33:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.029+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/2682005075 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fbe44037020 con 0x7fbe5409e910 2024-09-18T12:33:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.029+0000 7fbe3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.029+0000 7fbe3b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:27.031 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe40038490 msgr2=0x7fbe4003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe40038490 0x7fbe4003a940 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbe54098950 tx=0x7fbe48006f50 comp rx=0 tx=0).stop 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 10 monclient: shutdown 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 msgr2=0x7fbe540973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 0x7fbe540973f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009970 tx=0x7fbe44032000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 shutdown_connections 2024-09-18T12:33:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe40038490 0x7fbe4003a940 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe54136bb0 0x7fbe54097e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe54131ab0 0x7fbe54097930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 --2- 172.21.15.116:0/2682005075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5409e910 0x7fbe540973f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 >> 172.21.15.116:0/2682005075 conn(0x7fbe54067d30 msgr2=0x7fbe540a47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 shutdown_connections 2024-09-18T12:33:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.031+0000 7fbe59c17640 1 -- 172.21.15.116:0/2682005075 wait complete. 2024-09-18T12:33:27.043 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:33:27.043 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:33:27.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:33:27.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.136+0000 7f7f1cf1d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.136+0000 7f7f1cf1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.136+0000 7f7f1cf1d640 10 monclient: monmap: 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:27.137869+0000 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:27.137869+0000 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:27.139 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.136+0000 7f7f1cf1d640 1 Processor -- start 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.136+0000 7f7f1cf1d640 1 -- start start 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.136+0000 7f7f1cf1d640 10 monclient: init 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7f1806eb80 0x7f7f1806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f1806eb80 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f18068a20 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f18136ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1806a0e0 con 0x7f7f18136ab0 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f18069e00 con 0x7f7f18068a20 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1806a7a0 con 0x7f7f1806eb80 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1cf1d640 10 monclient(hunting): authenticate will time out at 1234.459471s 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f16d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7f18136ab0 0x7f7f18138e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59640/0 (socket says 172.21.15.116:59640) 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f16d9d640 1 -- 172.21.15.116:0/1790688779 learned_addr learned my addr 172.21.15.116:0/1790688779 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 1 --2- 172.21.15.116:0/1790688779 >> v2:172.21.15.144:3300/0 conn(0x7f7f1806eb80 0x7f7f1806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 10 monclient(hunting): get_auth_request con 0x7f7f1806eb80 auth_method 0 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.137+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 10 monclient(hunting): handle_auth_done global_id 114142 payload 274 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 1 -- 172.21.15.116:0/1790688779 >> v1:172.21.15.144:6789/0 conn(0x7f7f18068a20 legacy=0x7f7f1809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 1 -- 172.21.15.116:0/1790688779 >> v1:172.21.15.116:6789/0 conn(0x7f7f18136ab0 legacy=0x7f7f18138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 1 monclient: found mon.noname-c 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.138+0000 7f7f1659c640 1 -- 172.21.15.116:0/1790688779 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1806f490 con 0x7f7f1806eb80 2024-09-18T12:33:27.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.141+0000 7f7f1659c640 1 -- 172.21.15.116:0/1790688779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1806eb80 msgr2=0x7f7f1806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.141+0000 7f7f1659c640 1 --2- 172.21.15.116:0/1790688779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1806eb80 0x7f7f1806ef50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009880 tx=0x7f7f080090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2c2800213bdc150 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 1 -- 172.21.15.116:0/1790688779 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08032440 con 0x7f7f1806eb80 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: dump: 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f1cf1d640 5 monclient: authenticate success, global_id 114142 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 1 -- 172.21.15.116:0/1790688779 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f080325a0 con 0x7f7f1806eb80 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f1cf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f1cf1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f1cf1d640 4 set_mon_vals no callback set 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 1 -- 172.21.15.116:0/1790688779 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08032940 con 0x7f7f1806eb80 2024-09-18T12:33:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.142+0000 7f7f15d9b640 10 monclient: dump: 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 10 monclient: shutdown 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/1790688779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1806eb80 msgr2=0x7f7f1806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 --2- 172.21.15.116:0/1790688779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1806eb80 0x7f7f1806ef50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009880 tx=0x7f7f080090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/1790688779 shutdown_connections 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 --2- 172.21.15.116:0/1790688779 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f1806eb80 0x7f7f1806ef50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/1790688779 >> 172.21.15.116:0/1790688779 conn(0x7f7f18067cd0 msgr2=0x7f7f180a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/1790688779 shutdown_connections 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/1790688779 wait complete. 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 10 monclient: monmap: 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.143+0000 7f7f1cf1d640 1 Processor -- start 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 1 -- start start 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient: init 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f18068a20 0x7f7f18097200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f18068a20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1806eb80 0x7f7f18097740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f1806eb80 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f18136ab0 0x7f7f18097c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f18136ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1809d050 con 0x7f7f1806eb80 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1813a380 con 0x7f7f18068a20 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1811e9f0 con 0x7f7f18136ab0 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1cf1d640 10 monclient(hunting): authenticate will time out at 1234.466511s 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1806eb80 0x7f7f18097740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f18136ab0 0x7f7f18097c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1806eb80 0x7f7f18097740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36658/0 (socket says 172.21.15.116:36658) 2024-09-18T12:33:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): get_auth_request con 0x7f7f18136ab0 auth_method 0 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 1 -- 172.21.15.116:0/539603430 learned_addr learned my addr 172.21.15.116:0/539603430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 10 monclient(hunting): get_auth_request con 0x7f7f1806eb80 auth_method 0 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1659c640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f18068a20 0x7f7f18097200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1659c640 10 monclient(hunting): get_auth_request con 0x7f7f18068a20 auth_method 0 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient(hunting): handle_auth_done global_id 65207 payload 824 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f18068a20 msgr2=0x7f7f18097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7f18068a20 0x7f7f18097200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1806eb80 msgr2=0x7f7f18097740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1806eb80 0x7f7f18097740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 1 monclient: found mon.noname-c 2024-09-18T12:33:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f16d9d640 1 -- 172.21.15.116:0/539603430 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08009390 con 0x7f7f18136ab0 2024-09-18T12:33:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.144+0000 7f7f0ffff640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1806eb80 0x7f7f18097740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f16d9d640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f18136ab0 msgr2=0x7f7f18097c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f16d9d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f18136ab0 0x7f7f18097c80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000cb90 tx=0x7f7f100052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000a850 con 0x7f7f18136ab0 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: dump: 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1cf1d640 5 monclient: authenticate success, global_id 65207 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1cf1d640 10 monclient: _renew_subs 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1cf1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1000a9d0 con 0x7f7f18136ab0 2024-09-18T12:33:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1811efb0 con 0x7f7f18136ab0 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000ad90 con 0x7f7f18136ab0 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f0e7fc640 10 monclient: dump: 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1559a640 4 set_mon_vals no callback set 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1cf1d640 10 monclient: _renew_subs 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1cf1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.147+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f18136820 con 0x7f7f18136ab0 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7f10007300 con 0x7f7f18136ab0 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1cf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1cf1d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 0x7f7f1809a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1cf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1811e8b0 con 0x7f7f180981c0 2024-09-18T12:33:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 0x7f7f1809a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient: get_auth_request con 0x7f7f180981c0 auth_method 0 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f0e7fc640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ef8038490 0x7f7ef803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(226..226 src has 13..226) v4 ==== 14698+0+0 (secure 0 0 0) 0x7f7f1004dfb0 con 0x7f7f18136ab0 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 10 monclient(hunting): handle_auth_done global_id 104403 payload 824 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f0ffff640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ef8038490 0x7f7ef803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f1659c640 1 -- 172.21.15.116:0/539603430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f1811eba0 con 0x7f7f180981c0 2024-09-18T12:33:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.148+0000 7f7f0ffff640 10 monclient: get_auth_request con 0x7f7ef8038490 auth_method 0 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.150+0000 7f7f1659c640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 msgr2=0x7f7f1809a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.150+0000 7f7f1659c640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 0x7f7f1809a590 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009850 tx=0x7f7f08032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.150+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0803d950 con 0x7f7f180981c0 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.150+0000 7f7f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.150+0000 7f7f0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.150+0000 7f7f0e7fc640 10 monclient: dump: 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0ffff640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ef8038490 0x7f7ef803a940 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7f1811e070 tx=0x7f7f000090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7f0803dab0 con 0x7f7f180981c0 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 msgr2=0x7f7f1809a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0e7fc640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 0x7f7f1809a590 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009850 tx=0x7f7f08032000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 msgr2=0x7f7f1809a590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.151+0000 7f7f0e7fc640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f180981c0 0x7f7f1809a590 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1cf1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1cf1d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 0x7f7f1809aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1cf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18065e00 con 0x7f7f1809aad0 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1659c640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 0x7f7f1809aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1659c640 10 monclient: get_auth_request con 0x7f7f1809aad0 auth_method 0 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.156+0000 7f7f1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.157+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.157+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.157+0000 7f7f1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.157+0000 7f7f1659c640 10 monclient(hunting): handle_auth_done global_id 104406 payload 824 2024-09-18T12:33:27.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.157+0000 7f7f1659c640 1 -- 172.21.15.116:0/539603430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7f1804bf60 con 0x7f7f1809aad0 2024-09-18T12:33:27.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f1659c640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 msgr2=0x7f7f1809aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f1659c640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 0x7f7f1809aed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f18136540 tx=0x7f7f0803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0803c880 con 0x7f7f1809aad0 2024-09-18T12:33:27.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 10 monclient: dump: 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f7f0803d740 con 0x7f7f1809aad0 2024-09-18T12:33:27.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 msgr2=0x7f7f1809aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 0x7f7f1809aed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f18136540 tx=0x7f7f0803e000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 msgr2=0x7f7f1809aed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.159+0000 7f7f0e7fc640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f1809aad0 0x7f7f1809aed0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f18136540 tx=0x7f7f0803e000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:27.170 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 88, 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.174 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:27.175 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.176 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:27.178 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.181 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.182 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:27.183 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ef8038490 msgr2=0x7f7ef803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7ef8038490 0x7f7ef803a940 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7f1811e070 tx=0x7f7f000090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 10 monclient: shutdown 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f18136ab0 msgr2=0x7f7f18097c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f18136ab0 0x7f7f18097c80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000cb90 tx=0x7f7f100052b0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f16d9d640 1 -- 172.21.15.116:0/539603430 reap_dead start 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 shutdown_connections 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 --2- 172.21.15.116:0/539603430 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7f18136ab0 0x7f7f18097c80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 >> 172.21.15.116:0/539603430 conn(0x7f7f18067cd0 msgr2=0x7f7f180a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 shutdown_connections 2024-09-18T12:33:27.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.160+0000 7f7f1cf1d640 1 -- 172.21.15.116:0/539603430 wait complete. 2024-09-18T12:33:27.185 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:33:27.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:33:27.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:33:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.277+0000 7f127619d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.277+0000 7f127619d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.277+0000 7f127619d640 10 monclient: monmap: 2024-09-18T12:33:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:27.279457+0000 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:27.279457+0000 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:27.280 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.278+0000 7f127619d640 1 Processor -- start 2024-09-18T12:33:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.278+0000 7f127619d640 1 -- start start 2024-09-18T12:33:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.278+0000 7f127619d640 10 monclient: init 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.278+0000 7f127619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.278+0000 7f127619d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.278+0000 7f127619d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 10 monclient(hunting): picked mon.noname-b con 0x7f1270096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f127009aaa0 0x7f12700971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 10 monclient(hunting): picked mon.noname-a con 0x7f127009aaa0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1270097720 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1270099620 con 0x7f1270097720 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1270099340 con 0x7f127009aaa0 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1270099900 con 0x7f1270096e10 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f127619d640 10 monclient(hunting): authenticate will time out at 1234.601062s 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f127009aaa0 0x7f12700971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f127009aaa0 0x7f12700971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36692/0 (socket says 172.21.15.116:36692) 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 -- 172.21.15.116:0/592432001 learned_addr learned my addr 172.21.15.116:0/592432001 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): get_auth_request con 0x7f127009aaa0 auth_method 0 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 10 monclient(hunting): get_auth_request con 0x7f1270096e10 auth_method 0 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient(hunting): handle_auth_done global_id 104409 payload 274 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 -- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 msgr2=0x7f127009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 --2- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 -- 172.21.15.116:0/592432001 >> v1:172.21.15.144:6789/0 conn(0x7f1270097720 legacy=0x7f127012e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 monclient: found mon.noname-a 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126effd640 1 -- 172.21.15.116:0/592432001 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1270097b10 con 0x7f127009aaa0 2024-09-18T12:33:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.279+0000 7f126f7fe640 1 --2- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126effd640 1 -- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f127009aaa0 msgr2=0x7f12700971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126effd640 1 --2- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f127009aaa0 0x7f12700971e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1264009910 tx=0x7f12640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77683f12dca8848 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 1 -- 172.21.15.116:0/592432001 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126402aea0 con 0x7f127009aaa0 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: dump: 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 5 monclient: authenticate success, global_id 104409 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 1 -- 172.21.15.116:0/592432001 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1264032440 con 0x7f127009aaa0 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 1 -- 172.21.15.116:0/592432001 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12640327e0 con 0x7f127009aaa0 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 4 set_mon_vals no callback set 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f126e7fc640 10 monclient: dump: 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 10 monclient: shutdown 2024-09-18T12:33:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 1 -- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f127009aaa0 msgr2=0x7f12700971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 1 --2- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f127009aaa0 0x7f12700971e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1264009910 tx=0x7f12640091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 1 -- 172.21.15.116:0/592432001 shutdown_connections 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 1 --2- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f127009aaa0 0x7f12700971e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 1 --2- 172.21.15.116:0/592432001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 1 -- 172.21.15.116:0/592432001 >> 172.21.15.116:0/592432001 conn(0x7f12700672b0 msgr2=0x7f12700676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.282+0000 7f127619d640 1 -- 172.21.15.116:0/592432001 shutdown_connections 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.283+0000 7f127619d640 1 -- 172.21.15.116:0/592432001 wait complete. 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.283+0000 7f127619d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.283+0000 7f127619d640 10 monclient: monmap: 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.283+0000 7f127619d640 1 Processor -- start 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.283+0000 7f127619d640 1 -- start start 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.283+0000 7f127619d640 10 monclient: init 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): picked mon.noname-c con 0x7f1270096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1270097720 0x7f1270119b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): picked mon.noname-a con 0x7f1270097720 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127009aaa0 0x7f127012da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): picked mon.noname-b con 0x7f127009aaa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127011be80 con 0x7f1270097720 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127011a870 con 0x7f127009aaa0 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127011acb0 con 0x7f1270096e10 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f127619d640 10 monclient(hunting): authenticate will time out at 1234.606378s 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1270097720 0x7f1270119b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53082/0 (socket says 172.21.15.116:53082) 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): get_auth_request con 0x7f1270097720 auth_method 0 2024-09-18T12:33:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 1 -- 172.21.15.116:0/2515528113 learned_addr learned my addr 172.21.15.116:0/2515528113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 10 monclient(hunting): get_auth_request con 0x7f1270096e10 auth_method 0 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127009aaa0 0x7f127012da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 10 monclient(hunting): get_auth_request con 0x7f127009aaa0 auth_method 0 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient(hunting): handle_auth_done global_id 104412 payload 824 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 msgr2=0x7f127009b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127009aaa0 msgr2=0x7f127012da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f127009aaa0 0x7f127012da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 1 monclient: found mon.noname-a 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126effd640 1 -- 172.21.15.116:0/2515528113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1260009a20 con 0x7f1270097720 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.284+0000 7f126f7fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1270096e10 0x7f127009b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f126effd640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1270097720 msgr2=0x7f1270119b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f126effd640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1270097720 0x7f1270119b50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1264031000 tx=0x7f1264032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126402aea0 con 0x7f1270097720 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: dump: 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f127619d640 5 monclient: authenticate success, global_id 104412 2024-09-18T12:33:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1264031c70 con 0x7f1270097720 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f127619d640 10 monclient: _renew_subs 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f127619d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1264009450 con 0x7f1270097720 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126403d420 con 0x7f1270097720 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f12577fe640 10 monclient: dump: 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f126dffb640 4 set_mon_vals no callback set 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f127619d640 10 monclient: _renew_subs 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f127619d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.287+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127011c160 con 0x7f1270097720 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f127619d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f127619d640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 0x7f127013ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f127619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1270065f20 con 0x7f127012dfb0 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1264039040 con 0x7f1270097720 2024-09-18T12:33:27.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 0x7f127013ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient: get_auth_request con 0x7f127012dfb0 auth_method 0 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f12577fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f125c038490 0x7f125c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 10 monclient(hunting): handle_auth_done global_id 65216 payload 824 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(226..226 src has 13..226) v4 ==== 14698+0+0 (secure 0 0 0) 0x7f126406da20 con 0x7f1270097720 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126f7fe640 1 -- 172.21.15.116:0/2515528113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f127011c570 con 0x7f127012dfb0 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126ffff640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f125c038490 0x7f125c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.288+0000 7f126ffff640 10 monclient: get_auth_request con 0x7f125c038490 auth_method 0 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.290+0000 7f126f7fe640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 msgr2=0x7f127013ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.290+0000 7f126f7fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 0x7f127013ed60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f12600027e0 tx=0x7f126000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.290+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260011db0 con 0x7f127012dfb0 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.290+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.290+0000 7f12577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.290+0000 7f12577fe640 10 monclient: dump: 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f126000f040 con 0x7f127012dfb0 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f12577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f12577fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 msgr2=0x7f127013ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f12577fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 0x7f127013ed60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f12600027e0 tx=0x7f126000b040 comp rx=0 tx=0).stop 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 msgr2=0x7f127013ed60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f12577fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127012dfb0 0x7f127013ed60 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f12600027e0 tx=0x7f126000b040 comp rx=0 tx=0).stop 2024-09-18T12:33:27.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.291+0000 7f126ffff640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f125c038490 0x7f125c03a940 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1268007fd0 tx=0x7f1268007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f127619d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f127619d640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 0x7f127013f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f127619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127013fbe0 con 0x7f127013f2a0 2024-09-18T12:33:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 0x7f127013f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 10 monclient: get_auth_request con 0x7f127013f2a0 auth_method 0 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.296+0000 7f126f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.297+0000 7f126f7fe640 10 monclient(hunting): handle_auth_done global_id 65219 payload 824 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.297+0000 7f126f7fe640 1 -- 172.21.15.116:0/2515528113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f127009c2d0 con 0x7f127013f2a0 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f126f7fe640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 msgr2=0x7f127013f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f126f7fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 0x7f127013f6a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f12600027e0 tx=0x7f126001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260015e30 con 0x7f127013f2a0 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 10 monclient: dump: 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f1260026070 con 0x7f127013f2a0 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 msgr2=0x7f127013f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 0x7f127013f6a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f12600027e0 tx=0x7f126001d000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 msgr2=0x7f127013f6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f12577fe640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f127013f2a0 0x7f127013f6a0 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f12600027e0 tx=0x7f126001d000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 88, 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:27.313 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.314 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:27.316 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.317 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:27.318 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:27.321 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.324 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:27.325 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f125c038490 msgr2=0x7f125c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f125c038490 0x7f125c03a940 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1268007fd0 tx=0x7f1268007e10 comp rx=0 tx=0).stop 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 10 monclient: shutdown 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1270097720 msgr2=0x7f1270119b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 1 --2- 172.21.15.116:0/2515528113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1270097720 0x7f1270119b50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1264031000 tx=0x7f1264032000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f126ffff640 1 -- 172.21.15.116:0/2515528113 reap_dead start 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f126ffff640 1 -- 172.21.15.116:0/2515528113 reap_dead start 2024-09-18T12:33:27.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 shutdown_connections 2024-09-18T12:33:27.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.299+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 >> 172.21.15.116:0/2515528113 conn(0x7f12700672b0 msgr2=0x7f1270131700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.300+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 shutdown_connections 2024-09-18T12:33:27.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.300+0000 7f127619d640 1 -- 172.21.15.116:0/2515528113 wait complete. 2024-09-18T12:33:27.328 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:33:27.328 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:33:27.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:33:27.370 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.369+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v227) v1 ==== 118+0+0 (secure 0 0 0) 0x7f83a81363b0 con 0x7f83a809ec50 2024-09-18T12:33:27.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.369+0000 7f83967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T12:33:27.371 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.369+0000 7f83967fc640 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-18T12:33:27.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.422+0000 7f5332f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.422+0000 7f5332f9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.422+0000 7f5332f9d640 10 monclient: monmap: 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:27.424575+0000 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:27.424575+0000 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:27.425 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:27.426 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:27.426 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.423+0000 7f5332f9d640 1 Processor -- start 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.423+0000 7f5332f9d640 1 -- start start 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.423+0000 7f5332f9d640 10 monclient: init 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f532c096c60 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c097030 0x7f532c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f532c097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f532c133b40 0x7f532c135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f532c133b40 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c09d2e0 con 0x7f532c096c60 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c09d6b0 con 0x7f532c133b40 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c0992f0 con 0x7f532c097030 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5332f9d640 10 monclient(hunting): authenticate will time out at 1234.746183s 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5330d39640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f532c096c60 0x7f532c09a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59652/0 (socket says 172.21.15.116:59652) 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c097030 0x7f532c097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f5330d39640 1 -- 172.21.15.116:0/2699756151 learned_addr learned my addr 172.21.15.116:0/2699756151 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): get_auth_request con 0x7f532c097030 auth_method 0 2024-09-18T12:33:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f533153a640 1 --2- 172.21.15.116:0/2699756151 >> v2:172.21.15.144:3300/0 conn(0x7f532c133b40 0x7f532c135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f533153a640 10 monclient(hunting): get_auth_request con 0x7f532c133b40 auth_method 0 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f533153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f533153a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f533153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient(hunting): handle_auth_done global_id 65222 payload 274 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 1 -- 172.21.15.116:0/2699756151 >> v2:172.21.15.144:3300/0 conn(0x7f532c133b40 msgr2=0x7f532c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 1 --2- 172.21.15.116:0/2699756151 >> v2:172.21.15.144:3300/0 conn(0x7f532c133b40 0x7f532c135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 1 -- 172.21.15.116:0/2699756151 >> v1:172.21.15.116:6789/0 conn(0x7f532c096c60 legacy=0x7f532c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 1 monclient: found mon.noname-b 2024-09-18T12:33:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f532bfff640 1 -- 172.21.15.116:0/2699756151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c0995d0 con 0x7f532c097030 2024-09-18T12:33:27.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.424+0000 7f533153a640 1 --2- 172.21.15.116:0/2699756151 >> v2:172.21.15.144:3300/0 conn(0x7f532c133b40 0x7f532c135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:33:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532bfff640 1 -- 172.21.15.116:0/2699756151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c097030 msgr2=0x7f532c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532bfff640 1 --2- 172.21.15.116:0/2699756151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c097030 0x7f532c097430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f531c009e60 tx=0x7f531c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5735bcea3535fb57 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 1 -- 172.21.15.116:0/2699756151 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c007760 con 0x7f532c097030 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: dump: 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f5332f9d640 5 monclient: authenticate success, global_id 65222 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 1 -- 172.21.15.116:0/2699756151 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531c02d050 con 0x7f532c097030 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f5332f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f5332f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 1 -- 172.21.15.116:0/2699756151 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c007af0 con 0x7f532c097030 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f5332f9d640 4 set_mon_vals no callback set 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.427+0000 7f532b7fe640 10 monclient: dump: 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 10 monclient: shutdown 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 -- 172.21.15.116:0/2699756151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c097030 msgr2=0x7f532c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 --2- 172.21.15.116:0/2699756151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c097030 0x7f532c097430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f531c009e60 tx=0x7f531c009780 comp rx=0 tx=0).stop 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 -- 172.21.15.116:0/2699756151 shutdown_connections 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 --2- 172.21.15.116:0/2699756151 >> v2:172.21.15.144:3300/0 conn(0x7f532c133b40 0x7f532c135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 --2- 172.21.15.116:0/2699756151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c097030 0x7f532c097430 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 -- 172.21.15.116:0/2699756151 >> 172.21.15.116:0/2699756151 conn(0x7f532c0672b0 msgr2=0x7f532c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 -- 172.21.15.116:0/2699756151 shutdown_connections 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 1 -- 172.21.15.116:0/2699756151 wait complete. 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.428+0000 7f5332f9d640 10 monclient: monmap: 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 Processor -- start 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 -- start start 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient: init 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f532c096c60 0x7f532c11c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f532c096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c097030 0x7f532c12e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f532c097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c133b40 0x7f532c12e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f532c133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c06e0e0 con 0x7f532c096c60 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c06db60 con 0x7f532c097030 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c09add0 con 0x7f532c133b40 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5332f9d640 10 monclient(hunting): authenticate will time out at 1234.751865s 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5330d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f532c096c60 0x7f532c11c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5330d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f532c096c60 0x7f532c11c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36712/0 (socket says 172.21.15.116:36712) 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5330d39640 1 -- 172.21.15.116:0/4274132294 learned_addr learned my addr 172.21.15.116:0/4274132294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5330d39640 10 monclient(hunting): get_auth_request con 0x7f532c096c60 auth_method 0 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5330d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5330d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f5330d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f533153a640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c133b40 0x7f532c12e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f533153a640 10 monclient(hunting): get_auth_request con 0x7f532c133b40 auth_method 0 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f533153a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f533153a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.429+0000 7f533153a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c097030 0x7f532c12e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 10 monclient(hunting): get_auth_request con 0x7f532c097030 auth_method 0 2024-09-18T12:33:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f533153a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f533153a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f533153a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 10 monclient(hunting): handle_auth_done global_id 104415 payload 824 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c133b40 msgr2=0x7f532c12e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f532c133b40 0x7f532c12e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c097030 msgr2=0x7f532c12e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c097030 0x7f532c12e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 1 monclient: found mon.noname-a 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.430+0000 7f5330d39640 1 -- 172.21.15.116:0/4274132294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c009a20 con 0x7f532c096c60 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5330d39640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f532c096c60 msgr2=0x7f532c11c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5330d39640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f532c096c60 0x7f532c11c880 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f532400c5e0 tx=0x7f5324004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324005cb0 con 0x7f532c096c60 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: dump: 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.447 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5332f9d640 5 monclient: authenticate success, global_id 104415 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5332f9d640 10 monclient: _renew_subs 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5324005e30 con 0x7f532c096c60 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5332f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c09f730 con 0x7f532c096c60 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324007340 con 0x7f532c096c60 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5328ff9640 10 monclient: dump: 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f532affd640 4 set_mon_vals no callback set 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5332f9d640 10 monclient: _renew_subs 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5332f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.432+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c06bfb0 con 0x7f532c096c60 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f5332f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f5332f9d640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 0x7f532c131c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f5332f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c065f20 con 0x7f532c1318a0 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f532401b400 con 0x7f532c096c60 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 0x7f532c131c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient: get_auth_request con 0x7f532c1318a0 auth_method 0 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f5328ff9640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384c0 0x7f530c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7f532404d930 con 0x7f532c096c60 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 10 monclient(hunting): handle_auth_done global_id 114157 payload 824 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f532bfff640 1 -- 172.21.15.116:0/4274132294 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f532c06d3b0 con 0x7f532c1318a0 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f533153a640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384c0 0x7f530c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.433+0000 7f533153a640 10 monclient: get_auth_request con 0x7f530c0384c0 auth_method 0 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f532bfff640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 msgr2=0x7f532c131c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f532bfff640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 0x7f532c131c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f531c034000 tx=0x7f531c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c041070 con 0x7f532c1318a0 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 10 monclient: dump: 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f533153a640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384c0 0x7f530c03a970 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5314007d60 tx=0x7f5314008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f531c03cc00 con 0x7f532c1318a0 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 msgr2=0x7f532c131c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 0x7f532c131c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f531c034000 tx=0x7f531c037000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 msgr2=0x7f532c131c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.436+0000 7f5328ff9640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1318a0 0x7f532c131c70 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f531c034000 tx=0x7f531c037000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f5332f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f5332f9d640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 0x7f532c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f5332f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c04bf60 con 0x7f532c1321b0 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 0x7f532c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient: get_auth_request con 0x7f532c1321b0 auth_method 0 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 10 monclient(hunting): handle_auth_done global_id 114160 payload 824 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.442+0000 7f532bfff640 1 -- 172.21.15.116:0/4274132294 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f532c0983b0 con 0x7f532c1321b0 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f532bfff640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 msgr2=0x7f532c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f532bfff640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 0x7f532c06c230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f531c0097c0 tx=0x7f531c041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c056070 con 0x7f532c1321b0 2024-09-18T12:33:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 10 monclient: dump: 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f531c03b9e0 con 0x7f532c1321b0 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 msgr2=0x7f532c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 0x7f532c06c230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f531c0097c0 tx=0x7f531c041000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 msgr2=0x7f532c06c230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5328ff9640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f532c1321b0 0x7f532c06c230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 88, 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:27.457 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.458 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.459 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:27.469 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384c0 msgr2=0x7f530c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5332f9d640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f530c0384c0 0x7f530c03a970 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5314007d60 tx=0x7f5314008040 comp rx=0 tx=0).stop 2024-09-18T12:33:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5332f9d640 10 monclient: shutdown 2024-09-18T12:33:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5332f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f532c096c60 msgr2=0x7f532c11c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f5332f9d640 1 --2- 172.21.15.116:0/4274132294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f532c096c60 0x7f532c11c880 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f532400c5e0 tx=0x7f5324004960 comp rx=0 tx=0).stop 2024-09-18T12:33:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f533153a640 1 -- 172.21.15.116:0/4274132294 reap_dead start 2024-09-18T12:33:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.445+0000 7f533153a640 1 -- 172.21.15.116:0/4274132294 reap_dead start 2024-09-18T12:33:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.446+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 shutdown_connections 2024-09-18T12:33:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.446+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 >> 172.21.15.116:0/4274132294 conn(0x7f532c0672b0 msgr2=0x7f532c1351b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.446+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 shutdown_connections 2024-09-18T12:33:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.446+0000 7f5332f9d640 1 -- 172.21.15.116:0/4274132294 wait complete. 2024-09-18T12:33:27.471 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:33:27.471 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T12:33:27.471 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:33:27.471 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T12:33:27.472 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:33:27.472 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:33:27.472 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:33:27.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.470+0000 7f83ace68640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T12:33:27.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.470+0000 7f83ace68640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T12:33:27.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.470+0000 7f83ace68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:33:27.472 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.470+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f83a81363b0 con 0x7f83a809ec50 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 msgr2=0x7f83a8141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 msgr2=0x7f83a8141340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 0x7f83a8141340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f839800c8e0 tx=0x7f83980026c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 0x7f83a8141340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f839800c8e0 tx=0x7f83980026c0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 msgr2=0x7f83a8141340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f83a809ec50 0x7f83a8141340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.478 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f838804ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient(hunting): picked mon.c con 0x7f83a8136d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f8388051640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient(hunting): picked mon.a con 0x7f83a809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8388051fc0 0x7f8388054440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient(hunting): picked mon.b con 0x7f8388051fc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388054e60 con 0x7f83a809f020 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388054fa0 con 0x7f8388051fc0 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8388055130 con 0x7f83a8136d60 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.474+0000 7f83967fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8388051fc0 msgr2=0x7f8388054440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:27.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8388051fc0 0x7f8388054440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f8388051640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): get_auth_request con 0x7f83a809f020 auth_method 0 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f838804ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 10 monclient(hunting): get_auth_request con 0x7f83a8136d60 auth_method 0 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient(hunting): handle_auth_done global_id 114139 payload 293 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 msgr2=0x7f838804ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f838804ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8388051fc0 msgr2=0x7f8388054440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8388051fc0 0x7f8388054440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 monclient: found mon.a 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f83880552f0 con 0x7f83a809f020 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 1 -- 172.21.15.144:0/1335661995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f83a000f040 con 0x7f83a809f020 2024-09-18T12:33:27.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.475+0000 7f83a659c640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f838804ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.479+0000 7f83a6d9d640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 msgr2=0x7f8388051640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.479+0000 7f83a6d9d640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f8388051640 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f83a0007590 tx=0x7f83a00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0004c60 con 0x7f83a809f020 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f83967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f83967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f83967fc640 10 monclient: dump: 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.483 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0004dc0 con 0x7f83a809f020 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f83967fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f8397fff640 4 set_mon_vals no callback set 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.480+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f83a000ea20 con 0x7f83a809f020 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.481+0000 7f83967fc640 1 -- 172.21.15.144:0/1335661995 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v227) v1 ==== 118+0+0 (secure 0 0 0) 0x7f83a0047860 con 0x7f83a809f020 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.481+0000 7f83967fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.481+0000 7f83967fc640 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-18T12:33:27.484 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h2' does not appear in the crush map 2024-09-18T12:33:27.485 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:33:27.485 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:33:27.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:27.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8388038490 msgr2=0x7f838803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8388038490 0x7f838803a940 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f839c034000 tx=0x7f839c032000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 10 monclient: shutdown 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 msgr2=0x7f8388051640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f8388051640 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f83a0007590 tx=0x7f83a00077d0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 shutdown_connections 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8388051fc0 0x7f8388054440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a809f020 0x7f8388051640 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f83a8136d60 0x7f838804ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 --2- 172.21.15.144:0/1335661995 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8388038490 0x7f838803a940 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 >> 172.21.15.144:0/1335661995 conn(0x7f83a80a5cf0 msgr2=0x7f83a80a47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 shutdown_connections 2024-09-18T12:33:27.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.483+0000 7f83ace68640 1 -- 172.21.15.144:0/1335661995 wait complete. 2024-09-18T12:33:27.493 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:27.493 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h3 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.565+0000 7f368b857640 10 monclient: get_monmap_and_config 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.565+0000 7f368b857640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.565+0000 7f368b857640 10 monclient: monmap: 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:27.566805+0000 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:27.566805+0000 2024-09-18T12:33:27.567 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.566+0000 7f368b857640 1 Processor -- start 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.566+0000 7f368b857640 1 -- start start 2024-09-18T12:33:27.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.566+0000 7f368b857640 10 monclient: init 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f368409c250 0x7f368409da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient(hunting): picked mon.noname-c con 0x7f368409c250 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f368409dfa0 0x7f368409e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient(hunting): picked mon.noname-a con 0x7f368409dfa0 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f368409c620 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368409e8e0 con 0x7f368409c620 2024-09-18T12:33:27.573 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368409b230 con 0x7f368409dfa0 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368409b790 con 0x7f368409c250 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f368b857640 10 monclient(hunting): authenticate will time out at 1243.136174s 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f36895f3640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f368409c250 msgr2=0x7f368409da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f36895f3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f368409c250 0x7f368409da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f368409dfa0 0x7f368409e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3689df4640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f368409c620 0x7f368409ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:42440/0 (socket says 172.21.15.144:42440) 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 1 --2- 172.21.15.144:0/2796983042 >> v2:172.21.15.116:3300/0 conn(0x7f368409dfa0 0x7f368409e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:40590/0 (socket says 172.21.15.144:40590) 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3689df4640 1 -- 172.21.15.144:0/2796983042 learned_addr learned my addr 172.21.15.144:0/2796983042 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): get_auth_request con 0x7f368409dfa0 auth_method 0 2024-09-18T12:33:27.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient(hunting): handle_auth_done global_id 104418 payload 274 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 1 -- 172.21.15.144:0/2796983042 >> v2:172.21.15.144:3300/0 conn(0x7f368409c250 msgr2=0x7f368409da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 1 --2- 172.21.15.144:0/2796983042 >> v2:172.21.15.144:3300/0 conn(0x7f368409c250 0x7f368409da60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 1 -- 172.21.15.144:0/2796983042 >> v1:172.21.15.116:6789/0 conn(0x7f368409c620 legacy=0x7f368409ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 1 monclient: found mon.noname-a 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.567+0000 7f3688df2640 1 -- 172.21.15.144:0/2796983042 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368409b4b0 con 0x7f368409dfa0 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3688df2640 1 -- 172.21.15.144:0/2796983042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409dfa0 msgr2=0x7f368409e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3688df2640 1 --2- 172.21.15.144:0/2796983042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409dfa0 0x7f368409e3a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f36840a3fd0 tx=0x7f36780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a8f26024de78ac server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 1 -- 172.21.15.144:0/2796983042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3678032440 con 0x7f368409dfa0 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: dump: 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.576 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 1 -- 172.21.15.144:0/2796983042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36780325a0 con 0x7f368409dfa0 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 1 -- 172.21.15.144:0/2796983042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3678032940 con 0x7f368409dfa0 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f3683fff640 10 monclient: dump: 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:27.577 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f368b857640 5 monclient: authenticate success, global_id 104418 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f368b857640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.570+0000 7f368b857640 4 set_mon_vals no callback set 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 10 monclient: shutdown 2024-09-18T12:33:27.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 -- 172.21.15.144:0/2796983042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409dfa0 msgr2=0x7f368409e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 --2- 172.21.15.144:0/2796983042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409dfa0 0x7f368409e3a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f36840a3fd0 tx=0x7f36780090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 -- 172.21.15.144:0/2796983042 shutdown_connections 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 --2- 172.21.15.144:0/2796983042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409dfa0 0x7f368409e3a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 --2- 172.21.15.144:0/2796983042 >> v2:172.21.15.144:3300/0 conn(0x7f368409c250 0x7f368409da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 -- 172.21.15.144:0/2796983042 >> 172.21.15.144:0/2796983042 conn(0x7f3684071340 msgr2=0x7f36840a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 -- 172.21.15.144:0/2796983042 shutdown_connections 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 1 -- 172.21.15.144:0/2796983042 wait complete. 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.571+0000 7f368b857640 10 monclient: monmap: 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:27.579 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 Processor -- start 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 -- start start 2024-09-18T12:33:27.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient: init 2024-09-18T12:33:27.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:27.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 0x7f368413ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): picked mon.noname-a con 0x7f368409c250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f368409c620 0x7f3684131ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): picked mon.noname-b con 0x7f368409c620 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f368409dfa0 0x7f368411b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): picked mon.noname-c con 0x7f368409dfa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368411d140 con 0x7f368409c250 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368411eec0 con 0x7f368409c620 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368411ebe0 con 0x7f368409dfa0 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f3688df2640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f368409c620 msgr2=0x7f3684131ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f368b857640 10 monclient(hunting): authenticate will time out at 1243.141979s 2024-09-18T12:33:27.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.572+0000 7f3688df2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f368409c620 0x7f3684131ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 0x7f368413ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 0x7f368413ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:40602/0 (socket says 172.21.15.144:40602) 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 -- 172.21.15.144:0/2771057763 learned_addr learned my addr 172.21.15.144:0/2771057763 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): get_auth_request con 0x7f368409c250 auth_method 0 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f368409dfa0 0x7f368411b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 10 monclient(hunting): get_auth_request con 0x7f368409dfa0 auth_method 0 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f3689df4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient(hunting): handle_auth_done global_id 104421 payload 824 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 -- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f368409dfa0 msgr2=0x7f368411b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f368409dfa0 0x7f368411b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 -- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f368409c620 msgr2=0x7f3684131ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f368409c620 0x7f3684131ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 monclient: found mon.noname-a 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.573+0000 7f36895f3640 1 -- 172.21.15.144:0/2771057763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3678009390 con 0x7f368409c250 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36895f3640 1 -- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 msgr2=0x7f368413ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36895f3640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 0x7f368413ee30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f367c00bb80 tx=0x7f367c0097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c00cc80 con 0x7f368409c250 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: dump: 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:27.592 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367c00cde0 con 0x7f368409c250 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f368b857640 5 monclient: authenticate success, global_id 104421 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f368b857640 10 monclient: _renew_subs 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f368b857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368411f1a0 con 0x7f368409c250 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c016430 con 0x7f368409c250 2024-09-18T12:33:27.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36817fa640 10 monclient: dump: 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f368b857640 10 monclient: _renew_subs 2024-09-18T12:33:27.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f368b857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368413f510 con 0x7f368409c250 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.575+0000 7f36837fe640 4 set_mon_vals no callback set 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f367c012020 con 0x7f368409c250 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f368b857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f368b857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f368b857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36840008d0 con 0x7f368409c250 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f36817fa640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3670038440 0x7f367003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7f367c04df70 con 0x7f368409c250 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.576+0000 7f3688df2640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3670038440 0x7f367003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.577+0000 7f3688df2640 10 monclient: get_auth_request con 0x7f3670038440 auth_method 0 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.579+0000 7f3688df2640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3670038440 0x7f367003a8f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f368409c200 tx=0x7f3678032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.581+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f367c016860 con 0x7f368409c250 2024-09-18T12:33:27.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.581+0000 7f36817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.581+0000 7f36817fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.577+0000 7fad96f5d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.577+0000 7fad96f5d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.577+0000 7fad96f5d640 10 monclient: monmap: 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:27.578734+0000 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:27.578734+0000 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.577+0000 7fad96f5d640 1 Processor -- start 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.577+0000 7fad96f5d640 1 -- start start 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.577+0000 7fad96f5d640 10 monclient: init 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad901318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad9009ea50 0x7fad9009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fad9009ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad90136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad90131cc0 con 0x7fad901318f0 2024-09-18T12:33:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9009fbf0 con 0x7fad90136c20 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9009fe30 con 0x7fad9009ea50 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad96f5d640 10 monclient(hunting): authenticate will time out at 1234.900358s 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad8ffff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fad9009ea50 msgr2=0x7fad9009ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad8ffff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad9009ea50 0x7fad9009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad954fa640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fad90136c20 legacy=0x7fad90139000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad94cf9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fad901318f0 0x7fad90131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59660/0 (socket says 172.21.15.116:59660) 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.578+0000 7fad94cf9640 1 -- 172.21.15.116:0/2590307825 learned_addr learned my addr 172.21.15.116:0/2590307825 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad94cf9640 1 -- 172.21.15.116:0/2590307825 >> v1:172.21.15.116:6789/0 conn(0x7fad901318f0 legacy=0x7fad90131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145443257 0 0) 0x7fad90131cc0 con 0x7fad901318f0 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient(hunting): my global_id is 104424 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad800034e0 con 0x7fad901318f0 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944626450 0 0) 0x7fad800034e0 con 0x7fad901318f0 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 >> v2:172.21.15.144:3300/0 conn(0x7fad9009ea50 msgr2=0x7fad9009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 --2- 172.21.15.116:0/2590307825 >> v2:172.21.15.144:3300/0 conn(0x7fad9009ea50 0x7fad9009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 >> v1:172.21.15.144:6789/0 conn(0x7fad90136c20 legacy=0x7fad90139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad900a1210 con 0x7fad901318f0 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad96f5d640 5 monclient: authenticate success, global_id 104424 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fad840017a0 con 0x7fad901318f0 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad96f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: dump: 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.607 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 >> v1:172.21.15.116:6789/0 conn(0x7fad901318f0 legacy=0x7fad90131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.580+0000 7fad8f7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad800039d0 0x7fad80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): picked mon.b con 0x7fad800039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad80004780 0x7fad80004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): picked mon.a con 0x7fad80004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80005330 0x7fad8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): picked mon.c con 0x7fad80005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9009fe30 con 0x7fad80004780 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8000c2f0 con 0x7fad800039d0 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8000c4b0 con 0x7fad80005330 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad94cf9640 1 -- 172.21.15.116:0/2590307825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad800039d0 msgr2=0x7fad80003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad94cf9640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad800039d0 0x7fad80003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad96f5d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad80004780 0x7fad80004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): get_auth_request con 0x7fad80004780 auth_method 0 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80005330 0x7fad8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 10 monclient(hunting): get_auth_request con 0x7fad80005330 auth_method 0 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient(hunting): handle_auth_done global_id 104424 payload 293 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 1 -- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80005330 msgr2=0x7fad8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80005330 0x7fad8000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 1 -- 172.21.15.116:0/2590307825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad800039d0 msgr2=0x7fad80003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad800039d0 0x7fad80003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 1 monclient: found mon.a 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 1 -- 172.21.15.116:0/2590307825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad800034e0 con 0x7fad80004780 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80005330 0x7fad8000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.581+0000 7fad954fa640 1 -- 172.21.15.116:0/2590307825 reap_dead start 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8ffff640 1 -- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad80004780 msgr2=0x7fad80004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8ffff640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad80004780 0x7fad80004bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0099c0 tx=0x7fad7c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cd209312f96609 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c003b80 con 0x7fad80004780 2024-09-18T12:33:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 10 monclient: dump: 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c003d20 con 0x7fad80004780 2024-09-18T12:33:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 1 -- 172.21.15.116:0/2590307825 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c02ab80 con 0x7fad80004780 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 4 set_mon_vals no callback set 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad8f7fe640 10 monclient: dump: 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 10 monclient: shutdown 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 1 -- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad80004780 msgr2=0x7fad80004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad80004780 0x7fad80004bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0099c0 tx=0x7fad7c009060 comp rx=0 tx=0).stop 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 1 -- 172.21.15.116:0/2590307825 shutdown_connections 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 1 --2- 172.21.15.116:0/2590307825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad80004780 0x7fad80004bf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.584+0000 7fad96f5d640 1 -- 172.21.15.116:0/2590307825 >> 172.21.15.116:0/2590307825 conn(0x7fad90067110 msgr2=0x7fad90067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.585+0000 7fad96f5d640 1 -- 172.21.15.116:0/2590307825 shutdown_connections 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.585+0000 7fad96f5d640 1 -- 172.21.15.116:0/2590307825 wait complete. 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.585+0000 7fad96f5d640 10 monclient: build_initial_monmap 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.585+0000 7fad96f5d640 10 monclient: monmap: 2024-09-18T12:33:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.585+0000 7fad96f5d640 1 Processor -- start 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.585+0000 7fad96f5d640 1 -- start start 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.585+0000 7fad96f5d640 10 monclient: init 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:27.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 0x7fad90099670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fad80004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad90099bb0 0x7fad90099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fad90099bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad9009a4e0 0x7fad901470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fad9009a4e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9009b620 con 0x7fad90099bb0 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9009db70 con 0x7fad9009a4e0 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9011c760 con 0x7fad80004780 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad96f5d640 10 monclient(hunting): authenticate will time out at 1234.908377s 2024-09-18T12:33:27.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad954fa640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad9009a4e0 msgr2=0x7fad901470c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 0x7fad90099670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad954fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad9009a4e0 0x7fad901470c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad90099bb0 0x7fad90099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 0x7fad90099670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53132/0 (socket says 172.21.15.116:53132) 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 -- 172.21.15.116:0/3391237113 learned_addr learned my addr 172.21.15.116:0/3391237113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 10 monclient(hunting): get_auth_request con 0x7fad90099bb0 auth_method 0 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): get_auth_request con 0x7fad80004780 auth_method 0 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient(hunting): handle_auth_done global_id 65231 payload 824 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 -- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad9009a4e0 msgr2=0x7fad901470c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad9009a4e0 0x7fad901470c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 -- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad90099bb0 msgr2=0x7fad90099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad90099bb0 0x7fad90099fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient: _finish_hunting 0 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 monclient: found mon.noname-c 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad94cf9640 1 -- 172.21.15.116:0/3391237113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9011b870 con 0x7fad80004780 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.586+0000 7fad8ffff640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad90099bb0 0x7fad90099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.588+0000 7fad94cf9640 1 -- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 msgr2=0x7fad90099670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.588+0000 7fad94cf9640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 0x7fad90099670 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fad8400c7c0 tx=0x7fad9009fd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 1 -- 172.21.15.116:0/3391237113 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84007360 con 0x7fad80004780 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: dump: 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad96f5d640 5 monclient: authenticate success, global_id 65231 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad96f5d640 10 monclient: _renew_subs 2024-09-18T12:33:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 1 -- 172.21.15.116:0/3391237113 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad840074e0 con 0x7fad80004780 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad96f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9009bf10 con 0x7fad80004780 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 1 -- 172.21.15.116:0/3391237113 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad840049f0 con 0x7fad80004780 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8cff9640 10 monclient: dump: 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad8effd640 4 set_mon_vals no callback set 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad96f5d640 10 monclient: _renew_subs 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad96f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.589+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9009b0a0 con 0x7fad80004780 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad8cff9640 1 -- 172.21.15.116:0/3391237113 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fad8400a640 con 0x7fad80004780 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad96f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad96f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad96f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9004bf60 con 0x7fad80004780 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad8cff9640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad74038440 0x7fad7403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad8cff9640 1 -- 172.21.15.116:0/3391237113 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7fad8404d9b0 con 0x7fad80004780 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad8ffff640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad74038440 0x7fad7403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.590+0000 7fad8ffff640 10 monclient: get_auth_request con 0x7fad74038440 auth_method 0 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.593+0000 7fad8ffff640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad74038440 0x7fad7403a8f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0038f0 tx=0x7fad7c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.595+0000 7fad8cff9640 1 -- 172.21.15.116:0/3391237113 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fad84014420 con 0x7fad80004780 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.595+0000 7fad8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:27.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.595+0000 7fad8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:27.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.738+0000 7f368b857640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T12:33:27.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.738+0000 7f368b857640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T12:33:27.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.738+0000 7f368b857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:27.740 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:27.738+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f3684064a20 con 0x7f368409c250 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.812+0000 7fad96f5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.812+0000 7fad96f5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.812+0000 7fad96f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.812+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad9011b870 con 0x7fad80004780 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.812+0000 7fad8cff9640 1 -- 172.21.15.116:0/3391237113 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fad9011b870 con 0x7fad80004780 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.812+0000 7fad8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.812+0000 7fad8cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:27.814 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:27.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad74038440 msgr2=0x7fad7403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad74038440 0x7fad7403a8f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0038f0 tx=0x7fad7c030000 comp rx=0 tx=0).stop 2024-09-18T12:33:27.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 10 monclient: shutdown 2024-09-18T12:33:27.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:27.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 msgr2=0x7fad90099670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 0x7fad90099670 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fad8400c7c0 tx=0x7fad9009fd80 comp rx=0 tx=0).stop 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 shutdown_connections 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad74038440 0x7fad7403a8f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad9009a4e0 0x7fad901470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad90099bb0 0x7fad90099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 --2- 172.21.15.116:0/3391237113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad80004780 0x7fad90099670 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 >> 172.21.15.116:0/3391237113 conn(0x7fad90067110 msgr2=0x7fad900a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 shutdown_connections 2024-09-18T12:33:27.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:27.814+0000 7fad96f5d640 1 -- 172.21.15.116:0/3391237113 wait complete. 2024-09-18T12:33:30.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:30.573+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:30.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:30.573+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:30.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:30.573+0000 7f366bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:30.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.928+0000 7fce2d55d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.928+0000 7fce2d55d640 10 monclient: build_initial_monmap 2024-09-18T12:33:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.928+0000 7fce2d55d640 10 monclient: monmap: 2024-09-18T12:33:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:30.930347+0000 2024-09-18T12:33:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:30.930347+0000 2024-09-18T12:33:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:30.931 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:30.932 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:30.932 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:30.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 1 Processor -- start 2024-09-18T12:33:30.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 1 -- start start 2024-09-18T12:33:30.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient: init 2024-09-18T12:33:30.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:30.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T12:33:30.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce28131980 0x7fce28131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:30.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient(hunting): picked mon.noname-a con 0x7fce28131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:30.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce2809fe10 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:30.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce2809f140 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce280a14d0 con 0x7fce2809f140 2024-09-18T12:33:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce281325b0 con 0x7fce2809fe10 2024-09-18T12:33:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce280a11f0 con 0x7fce28131980 2024-09-18T12:33:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.929+0000 7fce2d55d640 10 monclient(hunting): authenticate will time out at 1238.251925s 2024-09-18T12:33:30.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce267fc640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fce2809fe10 legacy=0x7fce2809ea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:30.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce28131980 0x7fce28131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:30.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce277fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fce2809f140 0x7fce28138cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59670/0 (socket says 172.21.15.116:59670) 2024-09-18T12:33:30.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce277fe640 1 -- 172.21.15.116:0/1232754960 learned_addr learned my addr 172.21.15.116:0/1232754960 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:30.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): get_auth_request con 0x7fce28131980 auth_method 0 2024-09-18T12:33:30.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient(hunting): handle_auth_done global_id 104430 payload 274 2024-09-18T12:33:30.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 1 -- 172.21.15.116:0/1232754960 >> v1:172.21.15.144:6789/0 conn(0x7fce2809fe10 legacy=0x7fce2809ea30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:30.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 1 -- 172.21.15.116:0/1232754960 >> v1:172.21.15.116:6789/0 conn(0x7fce2809f140 legacy=0x7fce28138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:30.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:30.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 1 monclient: found mon.noname-a 2024-09-18T12:33:30.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:30.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.930+0000 7fce26ffd640 1 -- 172.21.15.116:0/1232754960 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28132290 con 0x7fce28131980 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce26ffd640 1 -- 172.21.15.116:0/1232754960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 msgr2=0x7fce28131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce26ffd640 1 --2- 172.21.15.116:0/1232754960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28131d50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fce18009850 tx=0x7fce180090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919128dfdbe2d19c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 1 -- 172.21.15.116:0/1232754960 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18032440 con 0x7fce28131980 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: dump: 2024-09-18T12:33:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 1 -- 172.21.15.116:0/1232754960 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce180325a0 con 0x7fce28131980 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce2d55d640 5 monclient: authenticate success, global_id 104430 2024-09-18T12:33:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce2d55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce2d55d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 1 -- 172.21.15.116:0/1232754960 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18032980 con 0x7fce28131980 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce2d55d640 4 set_mon_vals no callback set 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce25ffb640 10 monclient: dump: 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce2d55d640 10 monclient: shutdown 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce2d55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.933+0000 7fce2d55d640 1 -- 172.21.15.116:0/1232754960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 msgr2=0x7fce28131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 1 --2- 172.21.15.116:0/1232754960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28131d50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fce18009850 tx=0x7fce180090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 1 -- 172.21.15.116:0/1232754960 shutdown_connections 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 1 --2- 172.21.15.116:0/1232754960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28131d50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 1 -- 172.21.15.116:0/1232754960 >> 172.21.15.116:0/1232754960 conn(0x7fce280a5050 msgr2=0x7fce280a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 1 -- 172.21.15.116:0/1232754960 shutdown_connections 2024-09-18T12:33:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 1 -- 172.21.15.116:0/1232754960 wait complete. 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 10 monclient: build_initial_monmap 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.934+0000 7fce2d55d640 10 monclient: monmap: 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:30.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 Processor -- start 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 -- start start 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient: init 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce2809f140 0x7fce28134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): picked mon.noname-b con 0x7fce2809f140 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce2809fe10 0x7fce28134a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): picked mon.noname-c con 0x7fce2809fe10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): picked mon.noname-a con 0x7fce28131980 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce28137dc0 con 0x7fce28131980 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce28136bf0 con 0x7fce2809f140 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce28137ae0 con 0x7fce2809fe10 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce2d55d640 10 monclient(hunting): authenticate will time out at 1238.257561s 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce26ffd640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce2809f140 msgr2=0x7fce28134550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce2809fe10 0x7fce28134a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce26ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce2809f140 0x7fce28134550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28134fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36756/0 (socket says 172.21.15.116:36756) 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 1 -- 172.21.15.116:0/2538909956 learned_addr learned my addr 172.21.15.116:0/2538909956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:30.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): get_auth_request con 0x7fce28131980 auth_method 0 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 10 monclient(hunting): get_auth_request con 0x7fce2809fe10 auth_method 0 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:30.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 10 monclient(hunting): handle_auth_done global_id 104433 payload 824 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 1 -- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce2809fe10 msgr2=0x7fce28134a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.935+0000 7fce277fe640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce2809fe10 0x7fce28134a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce277fe640 1 -- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce2809f140 msgr2=0x7fce28134550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce277fe640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce2809f140 0x7fce28134550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce277fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce277fe640 1 monclient: found mon.noname-a 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce277fe640 1 -- 172.21.15.116:0/2538909956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18009390 con 0x7fce28131980 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.936+0000 7fce267fc640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce2809fe10 0x7fce28134a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:30.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce277fe640 1 -- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 msgr2=0x7fce28134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce277fe640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28134fd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce2000bb80 tx=0x7fce200097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 1 -- 172.21.15.116:0/2538909956 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2000cc80 con 0x7fce28131980 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: dump: 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: _finish_auth 0 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 1 -- 172.21.15.116:0/2538909956 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2000cde0 con 0x7fce28131980 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce2d55d640 5 monclient: authenticate success, global_id 104433 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce2d55d640 10 monclient: _renew_subs 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce2d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce281380a0 con 0x7fce28131980 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 1 -- 172.21.15.116:0/2538909956 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20014690 con 0x7fce28131980 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce0effd640 10 monclient: dump: 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce257fa640 4 set_mon_vals no callback set 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce2d55d640 10 monclient: _renew_subs 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce2d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.938+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce28139a30 con 0x7fce28131980 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce0effd640 1 -- 172.21.15.116:0/2538909956 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fce200133a0 con 0x7fce28131980 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce0effd640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce14038440 0x7fce1403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce0effd640 1 -- 172.21.15.116:0/2538909956 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7fce2004d890 con 0x7fce28131980 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce2d55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce2d55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce2d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2804bf60 con 0x7fce28131980 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.939+0000 7fce26ffd640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce14038440 0x7fce1403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.940+0000 7fce26ffd640 10 monclient: get_auth_request con 0x7fce14038440 auth_method 0 2024-09-18T12:33:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.942+0000 7fce26ffd640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce14038440 0x7fce1403a8f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fce18034000 tx=0x7fce18032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.944+0000 7fce0effd640 1 -- 172.21.15.116:0/2538909956 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fce20019080 con 0x7fce28131980 2024-09-18T12:33:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.944+0000 7fce0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:30.944+0000 7fce0effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.161+0000 7fce2d55d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.161+0000 7fce2d55d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.161+0000 7fce2d55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.161+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fce28067f00 con 0x7fce28131980 2024-09-18T12:33:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.161+0000 7fce0effd640 1 -- 172.21.15.116:0/2538909956 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fce20007700 con 0x7fce28131980 2024-09-18T12:33:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.161+0000 7fce0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.161+0000 7fce0effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:31.164 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce14038440 msgr2=0x7fce1403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce14038440 0x7fce1403a8f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fce18034000 tx=0x7fce18032000 comp rx=0 tx=0).stop 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 10 monclient: shutdown 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 msgr2=0x7fce28134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28134fd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce2000bb80 tx=0x7fce200097e0 comp rx=0 tx=0).stop 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 shutdown_connections 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fce14038440 0x7fce1403a8f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce28131980 0x7fce28134fd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fce2809fe10 0x7fce28134a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 --2- 172.21.15.116:0/2538909956 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fce2809f140 0x7fce28134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.163+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 >> 172.21.15.116:0/2538909956 conn(0x7fce280a5050 msgr2=0x7fce280a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.164+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 shutdown_connections 2024-09-18T12:33:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:31.164+0000 7fce2d55d640 1 -- 172.21.15.116:0/2538909956 wait complete. 2024-09-18T12:33:31.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:31.573+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:31.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:31.573+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:32.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:32.573+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:32.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:32.573+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:33.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:33.573+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:33.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:33.573+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:34.176 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.278+0000 7fdd97034640 10 monclient: get_monmap_and_config 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.278+0000 7fdd97034640 10 monclient: build_initial_monmap 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.278+0000 7fdd97034640 10 monclient: monmap: 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:34.279887+0000 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:34.279887+0000 2024-09-18T12:33:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:34.281 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.278+0000 7fdd97034640 1 Processor -- start 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 1 -- start start 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient: init 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 0x7fdd9009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient(hunting): picked mon.noname-b con 0x7fdd9009ad90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd9009b160 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd9009b530 0x7fdd90132070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9009b530 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd900692c0 con 0x7fdd9009b160 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9009d890 con 0x7fdd9009b530 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9009b990 con 0x7fdd9009ad90 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd97034640 10 monclient(hunting): authenticate will time out at 1241.601538s 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd8ffff640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fdd9009b160 legacy=0x7fdd90131960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd955d1640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fdd9009b530 msgr2=0x7fdd90132070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 0x7fdd9009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd955d1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd9009b530 0x7fdd90132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 0x7fdd9009c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53152/0 (socket says 172.21.15.116:53152) 2024-09-18T12:33:34.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 -- 172.21.15.116:0/1823300653 learned_addr learned my addr 172.21.15.116:0/1823300653 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): get_auth_request con 0x7fdd9009ad90 auth_method 0 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient(hunting): handle_auth_done global_id 65237 payload 274 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 -- 172.21.15.116:0/1823300653 >> v2:172.21.15.144:3300/0 conn(0x7fdd9009b530 msgr2=0x7fdd90132070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 --2- 172.21.15.116:0/1823300653 >> v2:172.21.15.144:3300/0 conn(0x7fdd9009b530 0x7fdd90132070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 -- 172.21.15.116:0/1823300653 >> v1:172.21.15.144:6789/0 conn(0x7fdd9009b160 legacy=0x7fdd90131960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient: _finish_hunting 0 2024-09-18T12:33:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 monclient: found mon.noname-b 2024-09-18T12:33:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.279+0000 7fdd94dd0640 1 -- 172.21.15.116:0/1823300653 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9009db70 con 0x7fdd9009ad90 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.282+0000 7fdd94dd0640 1 -- 172.21.15.116:0/1823300653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 msgr2=0x7fdd9009c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.282+0000 7fdd94dd0640 1 --2- 172.21.15.116:0/1823300653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 0x7fdd9009c5d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009910 tx=0x7fdd840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b7551c858e7f50 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 1 -- 172.21.15.116:0/1823300653 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd84032440 con 0x7fdd9009ad90 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: dump: 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.293 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 1 -- 172.21.15.116:0/1823300653 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd840325a0 con 0x7fdd9009ad90 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 5 monclient: authenticate success, global_id 65237 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:34.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 1 -- 172.21.15.116:0/1823300653 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd84032980 con 0x7fdd9009ad90 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 4 set_mon_vals no callback set 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd8f7fe640 10 monclient: dump: 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:34.295 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 10 monclient: shutdown 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 -- 172.21.15.116:0/1823300653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 msgr2=0x7fdd9009c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 --2- 172.21.15.116:0/1823300653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 0x7fdd9009c5d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009910 tx=0x7fdd840091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 -- 172.21.15.116:0/1823300653 shutdown_connections 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 --2- 172.21.15.116:0/1823300653 >> v2:172.21.15.144:3300/0 conn(0x7fdd9009b530 0x7fdd90132070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 --2- 172.21.15.116:0/1823300653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009ad90 0x7fdd9009c5d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 -- 172.21.15.116:0/1823300653 >> 172.21.15.116:0/1823300653 conn(0x7fdd900678a0 msgr2=0x7fdd900a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 -- 172.21.15.116:0/1823300653 shutdown_connections 2024-09-18T12:33:34.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.283+0000 7fdd97034640 1 -- 172.21.15.116:0/1823300653 wait complete. 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.284+0000 7fdd97034640 10 monclient: build_initial_monmap 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.284+0000 7fdd97034640 10 monclient: monmap: 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:34.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.284+0000 7fdd97034640 1 Processor -- start 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.284+0000 7fdd97034640 1 -- start start 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.284+0000 7fdd97034640 10 monclient: init 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9009ad90 0x7fdd901413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): picked mon.noname-b con 0x7fdd9009ad90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 0x7fdd90130b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9009b160 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009b530 0x7fdd90127820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9009b530 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9012ad80 con 0x7fdd9009b160 2024-09-18T12:33:34.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd901299a0 con 0x7fdd9009ad90 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90129290 con 0x7fdd9009b530 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd97034640 10 monclient(hunting): authenticate will time out at 1241.607303s 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd94dd0640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9009ad90 msgr2=0x7fdd901413a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd94dd0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9009ad90 0x7fdd901413a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009b530 0x7fdd90127820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 0x7fdd90130b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 0x7fdd90130b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36770/0 (socket says 172.21.15.116:36770) 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009b530 0x7fdd90127820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53160/0 (socket says 172.21.15.116:53160) 2024-09-18T12:33:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 -- 172.21.15.116:0/218320727 learned_addr learned my addr 172.21.15.116:0/218320727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): get_auth_request con 0x7fdd9009b160 auth_method 0 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 10 monclient(hunting): get_auth_request con 0x7fdd9009b530 auth_method 0 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd955d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient(hunting): handle_auth_done global_id 104436 payload 824 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 -- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009b530 msgr2=0x7fdd90127820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009b530 0x7fdd90127820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 -- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9009ad90 msgr2=0x7fdd901413a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9009ad90 0x7fdd901413a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 monclient: found mon.noname-a 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.285+0000 7fdd8ffff640 1 -- 172.21.15.116:0/218320727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84009450 con 0x7fdd9009b160 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.287+0000 7fdd8ffff640 1 -- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 msgr2=0x7fdd90130b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.287+0000 7fdd8ffff640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 0x7fdd90130b90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00b600 tx=0x7fdd7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 1 -- 172.21.15.116:0/218320727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c015070 con 0x7fdd9009b160 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: dump: 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:34.302 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd97034640 5 monclient: authenticate success, global_id 104436 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd97034640 10 monclient: _renew_subs 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd97034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9006f020 con 0x7fdd9009b160 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 1 -- 172.21.15.116:0/218320727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c003d90 con 0x7fdd9009b160 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 1 -- 172.21.15.116:0/218320727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c006380 con 0x7fdd9009b160 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8cff9640 10 monclient: dump: 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd8effd640 4 set_mon_vals no callback set 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd97034640 10 monclient: _renew_subs 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd97034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.288+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd90142160 con 0x7fdd9009b160 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd8cff9640 1 -- 172.21.15.116:0/218320727 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdd7c010440 con 0x7fdd9009b160 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd97034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd97034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd97034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9004bf60 con 0x7fdd9009b160 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd8cff9640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd74038440 0x7fdd7403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd8cff9640 1 -- 172.21.15.116:0/218320727 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7fdd7c04d900 con 0x7fdd9009b160 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd94dd0640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd74038440 0x7fdd7403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.289+0000 7fdd94dd0640 10 monclient: get_auth_request con 0x7fdd74038440 auth_method 0 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.292+0000 7fdd94dd0640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd74038440 0x7fdd7403a8f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdd84034000 tx=0x7fdd84032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.293+0000 7fdd8cff9640 1 -- 172.21.15.116:0/218320727 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdd7c018910 con 0x7fdd9009b160 2024-09-18T12:33:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.293+0000 7fdd8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.293+0000 7fdd8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:34.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.507+0000 7fdd97034640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:34.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.507+0000 7fdd97034640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:34.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.507+0000 7fdd97034640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:34.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.507+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdd900979e0 con 0x7fdd9009b160 2024-09-18T12:33:34.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.507+0000 7fdd8cff9640 1 -- 172.21.15.116:0/218320727 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fdd7c018310 con 0x7fdd9009b160 2024-09-18T12:33:34.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.508+0000 7fdd8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:34.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.508+0000 7fdd8cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:34.510 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd74038440 msgr2=0x7fdd7403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:34.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd74038440 0x7fdd7403a8f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdd84034000 tx=0x7fdd84032000 comp rx=0 tx=0).stop 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 10 monclient: shutdown 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 msgr2=0x7fdd90130b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 0x7fdd90130b90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00b600 tx=0x7fdd7c0091a0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 shutdown_connections 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdd74038440 0x7fdd7403a8f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd9009b530 0x7fdd90127820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9009b160 0x7fdd90130b90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 --2- 172.21.15.116:0/218320727 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd9009ad90 0x7fdd901413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 >> 172.21.15.116:0/218320727 conn(0x7fdd900678a0 msgr2=0x7fdd90099ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 shutdown_connections 2024-09-18T12:33:34.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:34.509+0000 7fdd97034640 1 -- 172.21.15.116:0/218320727 wait complete. 2024-09-18T12:33:34.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:34.573+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:34.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:34.573+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:35.574 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:35.574+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:35.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:35.574+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:36.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:36.573+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:36.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:36.573+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:37.523 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:33:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:37.574+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:37.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:37.574+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:37.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.626+0000 7f52d7e68640 10 monclient: get_monmap_and_config 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.626+0000 7f52d7e68640 10 monclient: build_initial_monmap 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.626+0000 7f52d7e68640 10 monclient: monmap: 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:37.627913+0000 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:37.627913+0000 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:37.628 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:37.629 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.626+0000 7f52d7e68640 1 Processor -- start 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 1 -- start start 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient: init 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient(hunting): picked mon.noname-b con 0x7f52d0131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient(hunting): picked mon.noname-a con 0x7f52d009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52d009f220 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d009faf0 con 0x7f52d009f220 2024-09-18T12:33:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d009fd30 con 0x7f52d009e910 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00a1110 con 0x7f52d0131ab0 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d7e68640 10 monclient(hunting): authenticate will time out at 1244.949587s 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d6405640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f52d009f220 legacy=0x7f52d0068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5c04640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5403640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5403640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37586/0 (socket says 172.21.15.116:37586) 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5c04640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33182/0 (socket says 172.21.15.116:33182) 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 learned_addr learned my addr 172.21.15.116:0/2241463036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5403640 10 monclient(hunting): get_auth_request con 0x7f52d009e910 auth_method 0 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5403640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5403640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5403640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5c04640 10 monclient(hunting): get_auth_request con 0x7f52d0131ab0 auth_method 0 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5c04640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.627+0000 7f52d5c04640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.628+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:37.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d6405640 1 -- 172.21.15.116:0/2241463036 >> v1:172.21.15.144:6789/0 conn(0x7f52d009f220 legacy=0x7f52d0068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 10 monclient(hunting): get_auth_request con 0x7f52d009e910 auth_method 0 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 10 monclient(hunting): get_auth_request con 0x7f52d0131ab0 auth_method 0 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:37.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:33:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:33:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:33:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:37.829+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:33:38.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d6405640 1 -- 172.21.15.116:0/2241463036 >> v1:172.21.15.144:6789/0 conn(0x7f52d009f220 legacy=0x7f52d0068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 10 monclient(hunting): get_auth_request con 0x7f52d009e910 auth_method 0 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 10 monclient(hunting): get_auth_request con 0x7f52d0131ab0 auth_method 0 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:38.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:33:38.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:38.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:38.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:33:38.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:33:38.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:38.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:38.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:38.230+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:33:38.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:38.574+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:38.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:38.574+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d6405640 1 -- 172.21.15.116:0/2241463036 >> v1:172.21.15.144:6789/0 conn(0x7f52d009f220 legacy=0x7f52d0068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 10 monclient(hunting): get_auth_request con 0x7f52d0131ab0 auth_method 0 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 10 monclient(hunting): get_auth_request con 0x7f52d009e910 auth_method 0 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:39.031+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:33:39.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:39.574+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:39.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:39.574+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:40.574+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:40.574+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:40.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:40.574+0000 7f366bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 10 monclient(hunting): tick 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 msgr2=0x7f52d0131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52d0131ab0 0x7f52d0131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 msgr2=0x7f52d009ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.116:3300/0 conn(0x7f52d009e910 0x7f52d009ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 -- 172.21.15.116:0/2241463036 >> v1:172.21.15.144:6789/0 conn(0x7f52d009f220 legacy=0x7f52d0068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 10 monclient: _add_conns ranks=[0,4] 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52bc003680 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 0x7f52bc009790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f52bc0072e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 -- 172.21.15.116:0/2241463036 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00a13f0 con 0x7f52bc003680 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 1 -- 172.21.15.116:0/2241463036 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d009fd30 con 0x7f52bc0072e0 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52cf7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 msgr2=0x7f52bc009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.628+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 0x7f52bc009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.631+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> v1:172.21.15.116:6789/0 conn(0x7f52bc003680 legacy=0x7f52bc005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:40.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.829+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 msgr2=0x7f52bc009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:40.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:40.829+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 0x7f52bc009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:33:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:41.230+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 msgr2=0x7f52bc009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:41.230+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 0x7f52bc009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:33:41.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:41.574+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:41.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:41.574+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.032+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 msgr2=0x7f52bc009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.032+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 0x7f52bc009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.382+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756287012 0 0) 0x7f52d00a13f0 con 0x7f52bc003680 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.382+0000 7f52d4c02640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.382+0000 7f52d4c02640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.382+0000 7f52d4c02640 10 monclient(hunting): my global_id is 104439 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.382+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b80034e0 con 0x7f52bc003680 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294159911 0 0) 0x7f52b80034e0 con 0x7f52bc003680 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 msgr2=0x7f52bc009790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 --2- 172.21.15.116:0/2241463036 >> v2:172.21.15.144:3300/0 conn(0x7f52bc0072e0 0x7f52bc009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: _finish_hunting 0 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d009faf0 con 0x7f52bc003680 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: _finish_auth 0 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f52c800d480 con 0x7f52bc003680 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d7e68640 5 monclient: authenticate success, global_id 104439 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d7e68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: dump: 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 >> v1:172.21.15.116:6789/0 conn(0x7f52bc003680 legacy=0x7f52bc005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d6405640 1 -- 172.21.15.116:0/2241463036 reap_dead start 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): picked mon.c con 0x7f52b80039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8004780 0x7f52b8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): picked mon.b con 0x7f52b8004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8005330 0x7f52b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): picked mon.a con 0x7f52b8005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d009fd30 con 0x7f52b8005330 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b800c290 con 0x7f52b8004780 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b800c450 con 0x7f52b80039d0 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d4c02640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d7e68640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 10 monclient(hunting): get_auth_request con 0x7f52b80039d0 auth_method 0 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d6405640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8005330 0x7f52b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5403640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8004780 msgr2=0x7f52b8004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d6405640 10 monclient(hunting): get_auth_request con 0x7f52b8005330 auth_method 0 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5403640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8004780 0x7f52b8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d6405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d6405640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d6405640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d5c04640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.383+0000 7f52d6405640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d6405640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d6405640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 10 monclient(hunting): handle_auth_done global_id 104439 payload 293 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8004780 msgr2=0x7f52b8004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8004780 0x7f52b8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8005330 msgr2=0x7f52b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8005330 0x7f52b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 10 monclient: _finish_hunting 0 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 1 monclient: found mon.c 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80034e0 con 0x7f52b80039d0 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 10 monclient: _finish_auth 0 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d5c04640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.384+0000 7f52d6405640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8005330 0x7f52b800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:42.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d5c04640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 msgr2=0x7f52b8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d5c04640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52b8003dc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f52c8006fd0 tx=0x7f52c8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada6178e3013385b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c802bc20 con 0x7f52b80039d0 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 10 monclient: dump: 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c802f070 con 0x7f52b80039d0 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d7e68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d7e68640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d7e68640 4 set_mon_vals no callback set 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 1 -- 172.21.15.116:0/2241463036 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8035400 con 0x7f52b80039d0 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.387+0000 7f52d4c02640 10 monclient: dump: 2024-09-18T12:33:42.400 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 10 monclient: shutdown 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 -- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 msgr2=0x7f52b8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52b8003dc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f52c8006fd0 tx=0x7f52c8008040 comp rx=0 tx=0).stop 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 -- 172.21.15.116:0/2241463036 shutdown_connections 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8005330 0x7f52b800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8004780 0x7f52b8004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 --2- 172.21.15.116:0/2241463036 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52b8003dc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 -- 172.21.15.116:0/2241463036 >> 172.21.15.116:0/2241463036 conn(0x7f52d0067d30 msgr2=0x7f52d00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 -- 172.21.15.116:0/2241463036 shutdown_connections 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 1 -- 172.21.15.116:0/2241463036 wait complete. 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 10 monclient: build_initial_monmap 2024-09-18T12:33:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.388+0000 7f52d7e68640 10 monclient: monmap: 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 Processor -- start 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 -- start start 2024-09-18T12:33:42.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient: init 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52d011cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): picked mon.noname-c con 0x7f52b80039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8004780 0x7f52d011d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): picked mon.noname-a con 0x7f52b8004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8005330 0x7f52d011da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): picked mon.noname-b con 0x7f52b8005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d011f630 con 0x7f52b8004780 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d009bde0 con 0x7f52b8005330 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00a44d0 con 0x7f52b80039d0 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.389+0000 7f52d7e68640 10 monclient(hunting): authenticate will time out at 1249.710933s 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52d011cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8004780 0x7f52d011d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52d011cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33222/0 (socket says 172.21.15.116:33222) 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 -- 172.21.15.116:0/4137262232 learned_addr learned my addr 172.21.15.116:0/4137262232 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): get_auth_request con 0x7f52b80039d0 auth_method 0 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d6405640 1 -- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8005330 msgr2=0x7f52d011da50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 10 monclient(hunting): get_auth_request con 0x7f52b8004780 auth_method 0 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d6405640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8005330 0x7f52d011da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient(hunting): handle_auth_done global_id 65243 payload 824 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 -- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8005330 msgr2=0x7f52d011da50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8005330 0x7f52d011da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 -- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8004780 msgr2=0x7f52d011d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8004780 0x7f52d011d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient: _finish_hunting 0 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 monclient: found mon.noname-c 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5c04640 1 -- 172.21.15.116:0/4137262232 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d011ea00 con 0x7f52b80039d0 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.390+0000 7f52d5403640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8004780 0x7f52d011d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52d5c04640 1 -- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 msgr2=0x7f52d011cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52d5c04640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52d011cfd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f52c80134d0 tx=0x7f52c8037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 1 -- 172.21.15.116:0/4137262232 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8006bc0 con 0x7f52b80039d0 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 10 monclient: dump: 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.408 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 1 -- 172.21.15.116:0/4137262232 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c800d480 con 0x7f52b80039d0 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52d7e68640 5 monclient: authenticate success, global_id 65243 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52d7e68640 10 monclient: _renew_subs 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cdffb640 1 -- 172.21.15.116:0/4137262232 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8005db0 con 0x7f52b80039d0 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52d7e68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d01204f0 con 0x7f52b80039d0 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.393+0000 7f52cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.393+0000 7f52cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.393+0000 7f52cdffb640 10 monclient: dump: 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.392+0000 7f52cffff640 4 set_mon_vals no callback set 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.393+0000 7f52d7e68640 10 monclient: _renew_subs 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.393+0000 7f52d7e68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.393+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d01306b0 con 0x7f52b80039d0 2024-09-18T12:33:42.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.394+0000 7f52cdffb640 1 -- 172.21.15.116:0/4137262232 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f52c8006240 con 0x7f52b80039d0 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.394+0000 7f52d7e68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.394+0000 7f52d7e68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.394+0000 7f52d7e68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.394+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d004bf60 con 0x7f52b80039d0 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.394+0000 7f52cdffb640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52bc038a10 0x7f52bc03aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.394+0000 7f52cdffb640 1 -- 172.21.15.116:0/4137262232 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7f52c8000e40 con 0x7f52b80039d0 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.395+0000 7f52d5403640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52bc038a10 0x7f52bc03aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.395+0000 7f52d5403640 10 monclient: get_auth_request con 0x7f52bc038a10 auth_method 0 2024-09-18T12:33:42.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.397+0000 7f52d5403640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52bc038a10 0x7f52bc03aec0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f52c4008010 tx=0x7f52c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.399+0000 7f52cdffb640 1 -- 172.21.15.116:0/4137262232 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f52c8046dd0 con 0x7f52b80039d0 2024-09-18T12:33:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.399+0000 7f52cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:42.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.399+0000 7f52cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:42.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:42.574+0000 7f366bfff640 10 monclient: tick 2024-09-18T12:33:42.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:42.574+0000 7f366bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.619+0000 7f52d7e68640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:33:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.619+0000 7f52d7e68640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.619+0000 7f52d7e68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.619+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f52d004bf60 con 0x7f52b80039d0 2024-09-18T12:33:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.619+0000 7f52cdffb640 1 -- 172.21.15.116:0/4137262232 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f52c80467d0 con 0x7f52b80039d0 2024-09-18T12:33:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.619+0000 7f52cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:33:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.619+0000 7f52cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:42.621 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:33:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52bc038a10 msgr2=0x7f52bc03aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52bc038a10 0x7f52bc03aec0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f52c4008010 tx=0x7f52c4008040 comp rx=0 tx=0).stop 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 10 monclient: shutdown 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 msgr2=0x7f52d011cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52d011cfd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f52c80134d0 tx=0x7f52c8037000 comp rx=0 tx=0).stop 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 shutdown_connections 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52bc038a10 0x7f52bc03aec0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f52b8005330 0x7f52d011da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b8004780 0x7f52d011d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 --2- 172.21.15.116:0/4137262232 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f52b80039d0 0x7f52d011cfd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 >> 172.21.15.116:0/4137262232 conn(0x7f52d0067d30 msgr2=0x7f52d01310a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 shutdown_connections 2024-09-18T12:33:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.621+0000 7f52d7e68640 1 -- 172.21.15.116:0/4137262232 wait complete. 2024-09-18T12:33:42.633 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:33:42.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:33:42.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:33:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.726+0000 7f4d8746f640 10 monclient: get_monmap_and_config 2024-09-18T12:33:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.726+0000 7f4d8746f640 10 monclient: build_initial_monmap 2024-09-18T12:33:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.726+0000 7f4d8746f640 10 monclient: monmap: 2024-09-18T12:33:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:42.727195+0000 2024-09-18T12:33:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:42.727195+0000 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:42.728 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 1 Processor -- start 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 1 -- start start 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient: init 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d80096e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d800971e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4d80134b40 0x7f4d80136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient(hunting): picked mon.noname-a con 0x7f4d80134b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8009d4a0 con 0x7f4d800971e0 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d80099620 con 0x7f4d80096e10 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d80099340 con 0x7f4d80134b40 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8746f640 10 monclient(hunting): authenticate will time out at 1250.048781s 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d85a0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4d80134b40 0x7f4d80136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d84a0a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4d800971e0 0x7f4d800975e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54582/0 (socket says 172.21.15.116:54582) 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d84a0a640 1 -- 172.21.15.116:0/486601277 learned_addr learned my addr 172.21.15.116:0/486601277 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d8520b640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f4d80096e10 legacy=0x7f4d8009a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d85a0c640 10 monclient(hunting): get_auth_request con 0x7f4d80134b40 auth_method 0 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d85a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d85a0c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.727+0000 7f4d85a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 10 monclient(hunting): handle_auth_done global_id 104445 payload 274 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 1 -- 172.21.15.116:0/486601277 >> v1:172.21.15.144:6789/0 conn(0x7f4d80096e10 legacy=0x7f4d8009a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 1 -- 172.21.15.116:0/486601277 >> v1:172.21.15.116:6789/0 conn(0x7f4d800971e0 legacy=0x7f4d800975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 1 monclient: found mon.noname-a 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.728+0000 7f4d85a0c640 1 -- 172.21.15.116:0/486601277 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d80099900 con 0x7f4d80134b40 2024-09-18T12:33:42.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d85a0c640 1 -- 172.21.15.116:0/486601277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80134b40 msgr2=0x7f4d80136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d85a0c640 1 --2- 172.21.15.116:0/486601277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80134b40 0x7f4d80136f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4d80076be0 tx=0x7f4d78009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4389def91348c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 1 -- 172.21.15.116:0/486601277 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78007840 con 0x7f4d80134b40 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: dump: 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.739 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: _finish_auth 0 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 5 monclient: authenticate success, global_id 104445 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 1 -- 172.21.15.116:0/486601277 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7802d050 con 0x7f4d80134b40 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 4 set_mon_vals no callback set 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 1 -- 172.21.15.116:0/486601277 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78007bd0 con 0x7f4d80134b40 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:42.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d77fff640 10 monclient: dump: 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 10 monclient: shutdown 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 1 -- 172.21.15.116:0/486601277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80134b40 msgr2=0x7f4d80136f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 1 --2- 172.21.15.116:0/486601277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80134b40 0x7f4d80136f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4d80076be0 tx=0x7f4d78009780 comp rx=0 tx=0).stop 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 1 -- 172.21.15.116:0/486601277 shutdown_connections 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 1 --2- 172.21.15.116:0/486601277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80134b40 0x7f4d80136f20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.731+0000 7f4d8746f640 1 -- 172.21.15.116:0/486601277 >> 172.21.15.116:0/486601277 conn(0x7f4d800672b0 msgr2=0x7f4d800676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.732+0000 7f4d8746f640 1 -- 172.21.15.116:0/486601277 shutdown_connections 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.732+0000 7f4d8746f640 1 -- 172.21.15.116:0/486601277 wait complete. 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.732+0000 7f4d8746f640 10 monclient: build_initial_monmap 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.732+0000 7f4d8746f640 10 monclient: monmap: 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:42.742 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.732+0000 7f4d8746f640 1 Processor -- start 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.732+0000 7f4d8746f640 1 -- start start 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.732+0000 7f4d8746f640 10 monclient: init 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d80096e10 0x7f4d8009b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): picked mon.noname-c con 0x7f4d80096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 0x7f4d80132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): picked mon.noname-a con 0x7f4d800971e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d80134b40 0x7f4d80133510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): picked mon.noname-b con 0x7f4d80134b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8006a930 con 0x7f4d800971e0 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8009b050 con 0x7f4d80134b40 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8009e540 con 0x7f4d80096e10 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8746f640 10 monclient(hunting): authenticate will time out at 1250.054451s 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d80096e10 0x7f4d8009b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 0x7f4d80132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 0x7f4d80132fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37630/0 (socket says 172.21.15.116:37630) 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d85a0c640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d80134b40 msgr2=0x7f4d80133510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d80096e10 0x7f4d8009b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33230/0 (socket says 172.21.15.116:33230) 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 -- 172.21.15.116:0/1906217540 learned_addr learned my addr 172.21.15.116:0/1906217540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): get_auth_request con 0x7f4d800971e0 auth_method 0 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 10 monclient(hunting): get_auth_request con 0x7f4d80096e10 auth_method 0 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d85a0c640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d80134b40 0x7f4d80133510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient(hunting): handle_auth_done global_id 104448 payload 824 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d80096e10 msgr2=0x7f4d8009b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d80096e10 0x7f4d8009b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d80134b40 msgr2=0x7f4d80133510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4d80134b40 0x7f4d80133510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient: _finish_hunting 0 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 monclient: found mon.noname-a 2024-09-18T12:33:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d84a0a640 1 -- 172.21.15.116:0/1906217540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d78009a20 con 0x7f4d800971e0 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.733+0000 7f4d8520b640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4d80096e10 0x7f4d8009b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d84a0a640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 msgr2=0x7f4d80132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d84a0a640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 0x7f4d80132fd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000c5e0 tx=0x7f4d70004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70005cb0 con 0x7f4d800971e0 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: dump: 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: _finish_auth 0 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d8746f640 5 monclient: authenticate success, global_id 104448 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d8746f640 10 monclient: _renew_subs 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d8746f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:42.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8009c640 con 0x7f4d800971e0 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70005e30 con 0x7f4d800971e0 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70007340 con 0x7f4d800971e0 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 10 monclient: dump: 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d777fe640 4 set_mon_vals no callback set 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d8746f640 10 monclient: _renew_subs 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d8746f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8006f250 con 0x7f4d800971e0 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.736+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4d7001b400 con 0x7f4d800971e0 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8746f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8746f640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 0x7f4d8012c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8746f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d80065f20 con 0x7f4d80133a50 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d757fa640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d64038440 0x7f4d6403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 0x7f4d8012c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient: get_auth_request con 0x7f4d80133a50 auth_method 0 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7f4d7004d930 con 0x7f4d800971e0 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 10 monclient(hunting): handle_auth_done global_id 104451 payload 824 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d85a0c640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d64038440 0x7f4d6403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d8520b640 1 -- 172.21.15.116:0/1906217540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d8006a3f0 con 0x7f4d80133a50 2024-09-18T12:33:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.737+0000 7f4d85a0c640 10 monclient: get_auth_request con 0x7f4d64038440 auth_method 0 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.739+0000 7f4d8520b640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 msgr2=0x7f4d8012c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.739+0000 7f4d8520b640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 0x7f4d8012c950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c006330 tx=0x7f4d7c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.739+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c00ee30 con 0x7f4d80133a50 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.739+0000 7f4d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.739+0000 7f4d757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.739+0000 7f4d757fa640 10 monclient: dump: 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.754 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d85a0c640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d64038440 0x7f4d6403a8f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4d8009b020 tx=0x7f4d78037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4d7c0123f0 con 0x7f4d80133a50 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d757fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 msgr2=0x7f4d8012c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d757fa640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 0x7f4d8012c950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c006330 tx=0x7f4d7c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 msgr2=0x7f4d8012c950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.740+0000 7f4d757fa640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d80133a50 0x7f4d8012c950 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c006330 tx=0x7f4d7c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.745+0000 7f4d8746f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.745+0000 7f4d8746f640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 0x7f4d8012d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.745+0000 7f4d8746f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.745+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8004bf60 con 0x7f4d8012ce90 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.745+0000 7f4d8520b640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 0x7f4d8012d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient: get_auth_request con 0x7f4d8012ce90 auth_method 0 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 10 monclient(hunting): handle_auth_done global_id 104454 payload 824 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.746+0000 7f4d8520b640 1 -- 172.21.15.116:0/1906217540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d80133e20 con 0x7f4d8012ce90 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d8520b640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 msgr2=0x7f4d8012d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d8520b640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 0x7f4d8012d290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c006330 tx=0x7f4d7c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c025070 con 0x7f4d8012ce90 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 10 monclient: dump: 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.757 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f4d7c020a40 con 0x7f4d8012ce90 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 msgr2=0x7f4d8012d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 0x7f4d8012d290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c006330 tx=0x7f4d7c015000 comp rx=0 tx=0).stop 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 msgr2=0x7f4d8012d290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.748+0000 7f4d757fa640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d8012ce90 0x7f4d8012d290 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 92, 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.759 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:42.760 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:42.761 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:42.762 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:42.763 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:42.764 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:42.765 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.768 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:42.769 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:42.771 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d64038440 msgr2=0x7f4d6403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4d64038440 0x7f4d6403a8f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4d8009b020 tx=0x7f4d78037000 comp rx=0 tx=0).stop 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 10 monclient: shutdown 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 msgr2=0x7f4d80132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d85a0c640 1 -- 172.21.15.116:0/1906217540 reap_dead start 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 0x7f4d80132fd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000c5e0 tx=0x7f4d70004960 comp rx=0 tx=0).stop 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 shutdown_connections 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 --2- 172.21.15.116:0/1906217540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d800971e0 0x7f4d80132fd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 >> 172.21.15.116:0/1906217540 conn(0x7f4d800672b0 msgr2=0x7f4d800a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 shutdown_connections 2024-09-18T12:33:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.749+0000 7f4d8746f640 1 -- 172.21.15.116:0/1906217540 wait complete. 2024-09-18T12:33:42.774 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:33:42.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:33:42.774 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.867+0000 7f4284b1d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.867+0000 7f4284b1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.867+0000 7f4284b1d640 10 monclient: monmap: 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:33:42.867870+0000 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:33:42.867870+0000 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.868 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:42.869 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:42.869 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:42.869 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:42.869 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:42.869 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.867+0000 7f4284b1d640 1 Processor -- start 2024-09-18T12:33:42.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.867+0000 7f4284b1d640 1 -- start start 2024-09-18T12:33:42.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.867+0000 7f4284b1d640 10 monclient: init 2024-09-18T12:33:42.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:42.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T12:33:42.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4280131ab0 0x7f4280131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f4280131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f428009e910 0x7f428009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f428009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009f250 0x7f4280068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f428009f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428009faf0 con 0x7f428009e910 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428009fd30 con 0x7f4280131ab0 2024-09-18T12:33:42.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42800a1110 con 0x7f428009f250 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4284b1d640 10 monclient(hunting): authenticate will time out at 1250.189318s 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427e59c640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f4280131ab0 msgr2=0x7f4280131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f428009e910 0x7f428009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009f250 0x7f4280068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427e59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4280131ab0 0x7f4280131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f428009e910 0x7f428009ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37662/0 (socket says 172.21.15.116:37662) 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 1 -- 172.21.15.116:0/2423558886 learned_addr learned my addr 172.21.15.116:0/2423558886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 10 monclient(hunting): get_auth_request con 0x7f428009e910 auth_method 0 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): get_auth_request con 0x7f428009f250 auth_method 0 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient(hunting): handle_auth_done global_id 65249 payload 274 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 1 -- 172.21.15.116:0/2423558886 >> v2:172.21.15.144:3300/0 conn(0x7f4280131ab0 msgr2=0x7f4280131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 1 --2- 172.21.15.116:0/2423558886 >> v2:172.21.15.144:3300/0 conn(0x7f4280131ab0 0x7f4280131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 1 -- 172.21.15.116:0/2423558886 >> v2:172.21.15.116:3300/0 conn(0x7f428009e910 msgr2=0x7f428009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:42.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 1 --2- 172.21.15.116:0/2423558886 >> v2:172.21.15.116:3300/0 conn(0x7f428009e910 0x7f428009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:42.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 1 monclient: found mon.noname-b 2024-09-18T12:33:42.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:42.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f427ed9d640 1 -- 172.21.15.116:0/2423558886 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800a13f0 con 0x7f428009f250 2024-09-18T12:33:42.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.868+0000 7f4275d9b640 1 --2- 172.21.15.116:0/2423558886 >> v2:172.21.15.116:3300/0 conn(0x7f428009e910 0x7f428009ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:42.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427ed9d640 1 -- 172.21.15.116:0/2423558886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009f250 msgr2=0x7f4280068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427ed9d640 1 --2- 172.21.15.116:0/2423558886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009f250 0x7f4280068ce0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4280076760 tx=0x7f4278009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b72c6d812ee3b504 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 1 -- 172.21.15.116:0/2423558886 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278007760 con 0x7f428009f250 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: dump: 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f4284b1d640 5 monclient: authenticate success, global_id 65249 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f4284b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 1 -- 172.21.15.116:0/2423558886 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427802d050 con 0x7f428009f250 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f4284b1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f4284b1d640 4 set_mon_vals no callback set 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 1 -- 172.21.15.116:0/2423558886 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278007af0 con 0x7f428009f250 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.871+0000 7f427dd9b640 10 monclient: dump: 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 10 monclient: shutdown 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:42.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 -- 172.21.15.116:0/2423558886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009f250 msgr2=0x7f4280068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 --2- 172.21.15.116:0/2423558886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009f250 0x7f4280068ce0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4280076760 tx=0x7f4278009780 comp rx=0 tx=0).stop 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 -- 172.21.15.116:0/2423558886 shutdown_connections 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 --2- 172.21.15.116:0/2423558886 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009f250 0x7f4280068ce0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 --2- 172.21.15.116:0/2423558886 >> v2:172.21.15.116:3300/0 conn(0x7f428009e910 0x7f428009ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 --2- 172.21.15.116:0/2423558886 >> v2:172.21.15.144:3300/0 conn(0x7f4280131ab0 0x7f4280131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 -- 172.21.15.116:0/2423558886 >> 172.21.15.116:0/2423558886 conn(0x7f4280067d30 msgr2=0x7f42800a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 -- 172.21.15.116:0/2423558886 shutdown_connections 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 1 -- 172.21.15.116:0/2423558886 wait complete. 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.872+0000 7f4284b1d640 10 monclient: monmap: 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 Processor -- start 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 -- start start 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient: init 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009e910 0x7f428009a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f428009e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428009f250 0x7f42801308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f428009f250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4280131ab0 0x7f428011b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f4280131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4280141ab0 con 0x7f428009f250 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428009bde0 con 0x7f4280131ab0 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4280134720 con 0x7f428009e910 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4284b1d640 10 monclient(hunting): authenticate will time out at 1250.194598s 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009e910 0x7f428009a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428009f250 0x7f42801308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427ed9d640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4280131ab0 msgr2=0x7f428011b6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009e910 0x7f428009a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33256/0 (socket says 172.21.15.116:33256) 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 -- 172.21.15.116:0/770114666 learned_addr learned my addr 172.21.15.116:0/770114666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): get_auth_request con 0x7f428009e910 auth_method 0 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 10 monclient(hunting): get_auth_request con 0x7f428009f250 auth_method 0 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427ed9d640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4280131ab0 0x7f428011b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient(hunting): handle_auth_done global_id 65252 payload 824 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4280131ab0 msgr2=0x7f428011b6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4280131ab0 0x7f428011b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428009f250 msgr2=0x7f42801308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428009f250 0x7f42801308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 monclient: found mon.noname-c 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f427e59c640 1 -- 172.21.15.116:0/770114666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4268009450 con 0x7f428009e910 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.873+0000 7f4275d9b640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f428009f250 0x7f42801308d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f427e59c640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009e910 msgr2=0x7f428009a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f427e59c640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009e910 0x7f428009a7f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f427000b660 tx=0x7f42700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270014030 con 0x7f428009e910 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: dump: 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4284b1d640 5 monclient: authenticate success, global_id 65252 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4284b1d640 10 monclient: _renew_subs 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4284b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4278009a20 con 0x7f428009e910 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4284b1d640 10 monclient: _renew_subs 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4284b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428011c7e0 con 0x7f428009e910 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4270003d90 con 0x7f428009e910 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270006380 con 0x7f428009e910 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f4276ffd640 10 monclient: dump: 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.876+0000 7f427d59a640 4 set_mon_vals no callback set 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4284b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4284b1d640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 0x7f428011bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428011db90 con 0x7f4280130e10 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4270017070 con 0x7f428009e910 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 0x7f428011bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient: get_auth_request con 0x7f4280130e10 auth_method 0 2024-09-18T12:33:42.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4276ffd640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4260038440 0x7f426003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 10 monclient(hunting): handle_auth_done global_id 65255 payload 824 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(227..227 src has 13..227) v4 ==== 14488+0+0 (secure 0 0 0) 0x7f4270015070 con 0x7f428009e910 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f4275d9b640 1 -- 172.21.15.116:0/770114666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428004bf60 con 0x7f4280130e10 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f427ed9d640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4260038440 0x7f426003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.877+0000 7f427ed9d640 10 monclient: get_auth_request con 0x7f4260038440 auth_method 0 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.879+0000 7f4275d9b640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 msgr2=0x7f428011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.879+0000 7f4275d9b640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 0x7f428011bbf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f426800f470 tx=0x7f42680091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.879+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268014940 con 0x7f4280130e10 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.879+0000 7f4276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.879+0000 7f4276ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.879+0000 7f4276ffd640 10 monclient: dump: 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4268014aa0 con 0x7f4280130e10 2024-09-18T12:33:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f4276ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f4276ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 msgr2=0x7f428011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f4276ffd640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 0x7f428011bbf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f426800f470 tx=0x7f42680091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 msgr2=0x7f428011bbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f4276ffd640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4280130e10 0x7f428011bbf0 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f426800f470 tx=0x7f42680091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.880+0000 7f427ed9d640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4260038440 0x7f426003a8f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f428009bdb0 tx=0x7f4278037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4284b1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4284b1d640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 0x7f4280096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4284b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4280063740 con 0x7f428011c130 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4275d9b640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 0x7f4280096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4275d9b640 10 monclient: get_auth_request con 0x7f428011c130 auth_method 0 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4275d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4275d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.885+0000 7f4275d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.886+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.886+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.886+0000 7f4275d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.886+0000 7f4275d9b640 10 monclient(hunting): handle_auth_done global_id 65258 payload 824 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.886+0000 7f4275d9b640 1 -- 172.21.15.116:0/770114666 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f428011c530 con 0x7f428011c130 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4275d9b640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 msgr2=0x7f4280096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4275d9b640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 0x7f4280096a60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f426800f470 tx=0x7f426801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426801c800 con 0x7f428011c130 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 10 monclient: dump: 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:33:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f426801c960 con 0x7f428011c130 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 msgr2=0x7f4280096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 0x7f4280096a60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f426800f470 tx=0x7f426801d000 comp rx=0 tx=0).stop 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 msgr2=0x7f4280096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.888+0000 7f4276ffd640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428011c130 0x7f4280096a60 secure :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f426800f470 tx=0x7f426801d000 comp rx=0 tx=0).stop 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 92, 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:33:42.902 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:42.903 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:42.904 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:33:42.905 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:33:42.906 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.907 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.908 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:42.909 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.910 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:33:42.911 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:33:42.912 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:33:42.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:33:42.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:33:42.915 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:33:42.915 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4260038440 msgr2=0x7f426003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4260038440 0x7f426003a8f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f428009bdb0 tx=0x7f4278037000 comp rx=0 tx=0).stop 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 10 monclient: shutdown 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009e910 msgr2=0x7f428009a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 --2- 172.21.15.116:0/770114666 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f428009e910 0x7f428009a7f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f427000b660 tx=0x7f42700091a0 comp rx=0 tx=0).stop 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f427ed9d640 1 -- 172.21.15.116:0/770114666 reap_dead start 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f427ed9d640 1 -- 172.21.15.116:0/770114666 reap_dead start 2024-09-18T12:33:42.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 shutdown_connections 2024-09-18T12:33:42.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 >> 172.21.15.116:0/770114666 conn(0x7f4280067d30 msgr2=0x7f4280134930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:42.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 shutdown_connections 2024-09-18T12:33:42.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:33:42.889+0000 7f4284b1d640 1 -- 172.21.15.116:0/770114666 wait complete. 2024-09-18T12:33:42.917 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:33:42.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:33:43.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.381+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v228) v1 ==== 118+0+0 (secure 0 0 0) 0x7f367c014340 con 0x7f368409c250 2024-09-18T12:33:43.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.381+0000 7f36817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T12:33:43.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.381+0000 7f36817fa640 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-18T12:33:43.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.479+0000 7f368b857640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T12:33:43.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.479+0000 7f368b857640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T12:33:43.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.479+0000 7f368b857640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:43.481 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.479+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f368411dc80 con 0x7f368409c250 2024-09-18T12:33:43.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.480+0000 7f36817fa640 1 -- 172.21.15.144:0/2771057763 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v228) v1 ==== 118+0+0 (secure 0 0 0) 0x7f367c014990 con 0x7f368409c250 2024-09-18T12:33:43.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.480+0000 7f36817fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T12:33:43.482 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.480+0000 7f36817fa640 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-18T12:33:43.482 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h3' does not appear in the crush map 2024-09-18T12:33:43.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3670038440 msgr2=0x7f367003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:43.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3670038440 0x7f367003a8f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f368409c200 tx=0x7f3678032000 comp rx=0 tx=0).stop 2024-09-18T12:33:43.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 10 monclient: shutdown 2024-09-18T12:33:43.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 msgr2=0x7f368413ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 0x7f368413ee30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f367c00bb80 tx=0x7f367c0097f0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 shutdown_connections 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3670038440 0x7f367003a8f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f368409dfa0 0x7f368411b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f368409c620 0x7f3684131ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 --2- 172.21.15.144:0/2771057763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f368409c250 0x7f368413ee30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 >> 172.21.15.144:0/2771057763 conn(0x7f3684071340 msgr2=0x7f3684132af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:43.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 shutdown_connections 2024-09-18T12:33:43.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.482+0000 7f368b857640 1 -- 172.21.15.144:0/2771057763 wait complete. 2024-09-18T12:33:43.492 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:43.492 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h4 2024-09-18T12:33:43.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.566+0000 7fccd479d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:43.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.566+0000 7fccd479d640 10 monclient: build_initial_monmap 2024-09-18T12:33:43.568 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.566+0000 7fccd479d640 10 monclient: monmap: 2024-09-18T12:33:43.568 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:43.568 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:43.567847+0000 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:43.567847+0000 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:43.569 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.567+0000 7fccd479d640 1 Processor -- start 2024-09-18T12:33:43.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.567+0000 7fccd479d640 1 -- start start 2024-09-18T12:33:43.570 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.567+0000 7fccd479d640 10 monclient: init 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcccc131c90 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcccc09d210 0x7fcccc069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient(hunting): picked mon.noname-c con 0x7fcccc09d210 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcccc0a0ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0a1800 con 0x7fcccc131c90 2024-09-18T12:33:43.575 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc09fea0 con 0x7fcccc0a0ab0 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0a1520 con 0x7fcccc09d210 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd2d3a640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fcccc0a0ab0 legacy=0x7fcccc138fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd1d38640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fcccc09d210 msgr2=0x7fcccc069e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd479d640 10 monclient(hunting): authenticate will time out at 1259.137268s 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd1d38640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcccc09d210 0x7fcccc069e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd2539640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcccc131c90 0x7fcccc132060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:58544/0 (socket says 172.21.15.144:58544) 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.568+0000 7fccd2539640 1 -- 172.21.15.144:0/2365753750 learned_addr learned my addr 172.21.15.144:0/2365753750 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:43.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.570+0000 7fccd2539640 1 -- 172.21.15.144:0/2365753750 >> v1:172.21.15.116:6789/0 conn(0x7fcccc131c90 legacy=0x7fcccc132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.570+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527635942 0 0) 0x7fcccc0a1800 con 0x7fcccc131c90 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): my global_id is 104463 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb40034e0 con 0x7fcccc131c90 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 12037315 0 0) 0x7fccb40034e0 con 0x7fcccc131c90 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 >> v2:172.21.15.144:3300/0 conn(0x7fcccc09d210 msgr2=0x7fcccc069e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 --2- 172.21.15.144:0/2365753750 >> v2:172.21.15.144:3300/0 conn(0x7fcccc09d210 0x7fcccc069e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 >> v1:172.21.15.144:6789/0 conn(0x7fcccc0a0ab0 legacy=0x7fcccc138fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:43.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: _finish_hunting 0 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc132770 con 0x7fcccc131c90 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: _finish_auth 0 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd479d640 5 monclient: authenticate success, global_id 104463 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fccc4001bd0 con 0x7fcccc131c90 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd479d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:43.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: dump: 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:43.579 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 >> v1:172.21.15.116:6789/0 conn(0x7fcccc131c90 legacy=0x7fcccc132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb40039d0 0x7fccb4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): picked mon.b con 0x7fccb40039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:43.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4004780 0x7fccb4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): picked mon.c con 0x7fccb4004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb4005330 0x7fccb400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): picked mon.a con 0x7fccb4005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0a1520 con 0x7fccb4005330 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb400c2f0 con 0x7fccb40039d0 2024-09-18T12:33:43.581 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2539640 1 -- 172.21.15.144:0/2365753750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb40039d0 msgr2=0x7fccb4003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb400c4b0 con 0x7fccb4004780 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1537640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd479d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2539640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb40039d0 0x7fccb4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb4005330 0x7fccb400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4004780 0x7fccb4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): get_auth_request con 0x7fccb4005330 auth_method 0 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:43.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 10 monclient(hunting): get_auth_request con 0x7fccb4004780 auth_method 0 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:43.583 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd1d38640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient(hunting): handle_auth_done global_id 104463 payload 293 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 1 -- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4004780 msgr2=0x7fccb4004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4004780 0x7fccb4004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 1 -- 172.21.15.144:0/2365753750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb40039d0 msgr2=0x7fccb4003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb40039d0 0x7fccb4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient: _finish_hunting 0 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 1 monclient: found mon.a 2024-09-18T12:33:43.584 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:43.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:43.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 1 -- 172.21.15.144:0/2365753750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40034e0 con 0x7fccb4005330 2024-09-18T12:33:43.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 10 monclient: _finish_auth 0 2024-09-18T12:33:43.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.571+0000 7fccd2d3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:43.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.572+0000 7fccd2d3a640 1 -- 172.21.15.144:0/2365753750 reap_dead start 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.574+0000 7fccd2d3a640 1 -- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb4005330 msgr2=0x7fccb400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.574+0000 7fccd2d3a640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb4005330 0x7fccb400ba10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0099c0 tx=0x7fccbc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e3a8a45d1ba8a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.574+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc003b80 con 0x7fccb4005330 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.574+0000 7fccd1537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 10 monclient: dump: 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:43.592 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc003d20 con 0x7fccb4005330 2024-09-18T12:33:43.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 4 set_mon_vals no callback set 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 1 -- 172.21.15.144:0/2365753750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc02ab80 con 0x7fccb4005330 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd1537640 10 monclient: dump: 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:43.594 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 10 monclient: shutdown 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 1 -- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb4005330 msgr2=0x7fccb400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb4005330 0x7fccb400ba10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0099c0 tx=0x7fccbc009060 comp rx=0 tx=0).stop 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 1 -- 172.21.15.144:0/2365753750 shutdown_connections 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 1 --2- 172.21.15.144:0/2365753750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccb4005330 0x7fccb400ba10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 1 -- 172.21.15.144:0/2365753750 >> 172.21.15.144:0/2365753750 conn(0x7fcccc071370 msgr2=0x7fcccc0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 1 -- 172.21.15.144:0/2365753750 shutdown_connections 2024-09-18T12:33:43.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.575+0000 7fccd479d640 1 -- 172.21.15.144:0/2365753750 wait complete. 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient: build_initial_monmap 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient: monmap: 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:43.596 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 Processor -- start 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 -- start start 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient: init 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 0x7fcccc09aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): picked mon.noname-c con 0x7fccb4005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcccc141a10 0x7fcccc130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): picked mon.noname-b con 0x7fcccc141a10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcccc131130 0x7fcccc11b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): picked mon.noname-a con 0x7fcccc131130 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc11e010 con 0x7fcccc131130 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0a0fb0 con 0x7fcccc141a10 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd1d38640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcccc141a10 msgr2=0x7fcccc130bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0a44e0 con 0x7fccb4005330 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccd479d640 10 monclient(hunting): authenticate will time out at 1259.145993s 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.576+0000 7fccca7fc640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:33:43.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd1d38640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcccc141a10 0x7fcccc130bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 0x7fcccc09aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcccc131130 0x7fcccc11b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 0x7fcccc09aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59044/0 (socket says 172.21.15.144:59044) 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 -- 172.21.15.144:0/4132918223 learned_addr learned my addr 172.21.15.144:0/4132918223 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): get_auth_request con 0x7fccb4005330 auth_method 0 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 10 monclient(hunting): get_auth_request con 0x7fcccc131130 auth_method 0 2024-09-18T12:33:43.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient(hunting): handle_auth_done global_id 65261 payload 824 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 -- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcccc141a10 msgr2=0x7fcccc130bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:43.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcccc141a10 0x7fcccc130bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 -- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcccc131130 msgr2=0x7fcccc11b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcccc131130 0x7fcccc11b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient: _finish_hunting 0 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 monclient: found mon.noname-c 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2539640 1 -- 172.21.15.144:0/4132918223 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc11dd30 con 0x7fccb4005330 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.577+0000 7fccd2d3a640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcccc131130 0x7fcccc11b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.579+0000 7fccd2539640 1 -- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 msgr2=0x7fcccc09aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.579+0000 7fccd2539640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 0x7fcccc09aa20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fccc400cb70 tx=0x7fccc4001520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:43.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400a7d0 con 0x7fccb4005330 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: dump: 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:43.602 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc400a930 con 0x7fccb4005330 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 5 monclient: authenticate success, global_id 65261 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 10 monclient: _renew_subs 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc09b590 con 0x7fccb4005330 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:43.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400acf0 con 0x7fccb4005330 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 10 monclient: dump: 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:43.604 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd0d36640 4 set_mon_vals no callback set 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 10 monclient: _renew_subs 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc11ce40 con 0x7fccb4005330 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fccc40073a0 con 0x7fccb4005330 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.580+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc0008d0 con 0x7fccb4005330 2024-09-18T12:33:43.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.581+0000 7fccca7fc640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccb8038440 0x7fccb803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.581+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(228..228 src has 13..228) v4 ==== 14278+0+0 (secure 0 0 0) 0x7fccc404de70 con 0x7fccb4005330 2024-09-18T12:33:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.581+0000 7fccd1d38640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccb8038440 0x7fccb803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.581+0000 7fccd1d38640 10 monclient: get_auth_request con 0x7fccb8038440 auth_method 0 2024-09-18T12:33:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.583+0000 7fccd1d38640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccb8038440 0x7fccb803a8f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fccc0003870 tx=0x7fccc0006ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.585+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fccc40104f0 con 0x7fccb4005330 2024-09-18T12:33:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.585+0000 7fccca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:43.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.585+0000 7fccca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:43.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.747+0000 7fccd479d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T12:33:43.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.747+0000 7fccd479d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T12:33:43.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.747+0000 7fccd479d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:43.748 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:43.747+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fcccc064a20 con 0x7fccb4005330 2024-09-18T12:33:44.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.386+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v229) v1 ==== 118+0+0 (secure 0 0 0) 0x7fccc401baf0 con 0x7fccb4005330 2024-09-18T12:33:44.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.386+0000 7fccca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T12:33:44.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.386+0000 7fccca7fc640 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-18T12:33:44.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.486+0000 7fccd479d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T12:33:44.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.486+0000 7fccd479d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T12:33:44.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.486+0000 7fccd479d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:44.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.486+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fcccc11be40 con 0x7fccb4005330 2024-09-18T12:33:44.488 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.488+0000 7fccca7fc640 1 -- 172.21.15.144:0/4132918223 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v229) v1 ==== 118+0+0 (secure 0 0 0) 0x7fccc401bcd0 con 0x7fccb4005330 2024-09-18T12:33:44.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.488+0000 7fccca7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T12:33:44.489 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.488+0000 7fccca7fc640 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-18T12:33:44.489 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h4' does not appear in the crush map 2024-09-18T12:33:44.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccb8038440 msgr2=0x7fccb803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:44.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccb8038440 0x7fccb803a8f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fccc0003870 tx=0x7fccc0006ed0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.491 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 10 monclient: shutdown 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 msgr2=0x7fcccc09aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 0x7fcccc09aa20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fccc400cb70 tx=0x7fccc4001520 comp rx=0 tx=0).stop 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 shutdown_connections 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccb8038440 0x7fccb803a8f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcccc131130 0x7fcccc11b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcccc141a10 0x7fcccc130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 --2- 172.21.15.144:0/4132918223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccb4005330 0x7fcccc09aa20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 >> 172.21.15.144:0/4132918223 conn(0x7fcccc071370 msgr2=0x7fcccc0a46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 shutdown_connections 2024-09-18T12:33:44.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.490+0000 7fccd479d640 1 -- 172.21.15.144:0/4132918223 wait complete. 2024-09-18T12:33:44.500 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:44.500 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h5 2024-09-18T12:33:44.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.571+0000 7ffb0aa80640 10 monclient: get_monmap_and_config 2024-09-18T12:33:44.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.571+0000 7ffb0aa80640 10 monclient: build_initial_monmap 2024-09-18T12:33:44.572 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.571+0000 7ffb0aa80640 10 monclient: monmap: 2024-09-18T12:33:44.572 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:44.572444+0000 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:44.572444+0000 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:44.573 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:44.576 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 1 Processor -- start 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 1 -- start start 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient: init 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffb0406e130 0x7ffb0409da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient(hunting): picked mon.noname-a con 0x7ffb0406e130 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb0409dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient(hunting): picked mon.noname-b con 0x7ffb0406a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb0406e500 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0406a0b0 con 0x7ffb0406e500 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:44.577 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb04068c30 con 0x7ffb0406e130 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04069750 con 0x7ffb0406a790 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0aa80640 10 monclient(hunting): authenticate will time out at 1260.141869s 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.572+0000 7ffb0901d640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7ffb0406e500 legacy=0x7ffb0409e4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb0409dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffb0406e130 0x7ffb0409da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb0409dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59050/0 (socket says 172.21.15.144:59050) 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): get_auth_request con 0x7ffb0406e130 auth_method 0 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 1 -- 172.21.15.144:0/2297032217 learned_addr learned my addr 172.21.15.144:0/2297032217 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:44.578 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 10 monclient(hunting): get_auth_request con 0x7ffb0406a790 auth_method 0 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient(hunting): handle_auth_done global_id 104469 payload 274 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 1 -- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 msgr2=0x7ffb0409dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 1 --2- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb0409dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.579 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 1 -- 172.21.15.144:0/2297032217 >> v1:172.21.15.144:6789/0 conn(0x7ffb0406e500 legacy=0x7ffb0409e4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:44.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:44.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 1 monclient: found mon.noname-a 2024-09-18T12:33:44.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:44.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:44.580 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.573+0000 7ffb0881c640 1 -- 172.21.15.144:0/2297032217 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04069a60 con 0x7ffb0406e130 2024-09-18T12:33:44.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb0881c640 1 -- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e130 msgr2=0x7ffb0409da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:44.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb0881c640 1 --2- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e130 0x7ffb0409da60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009850 tx=0x7ffaf80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab64cc901fd38e9f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:44.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 1 -- 172.21.15.144:0/2297032217 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf802ae60 con 0x7ffb0406e130 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: dump: 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:44.588 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 1 -- 172.21.15.144:0/2297032217 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf8032440 con 0x7ffb0406e130 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 1 -- 172.21.15.144:0/2297032217 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80327e0 con 0x7ffb0406e130 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb037fe640 10 monclient: dump: 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:44.589 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb0aa80640 5 monclient: authenticate success, global_id 104469 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb0aa80640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.576+0000 7ffb0aa80640 4 set_mon_vals no callback set 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 10 monclient: shutdown 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:44.590 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e130 msgr2=0x7ffb0409da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e130 0x7ffb0409da60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009850 tx=0x7ffaf80090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2297032217 shutdown_connections 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb0409dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2297032217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e130 0x7ffb0409da60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2297032217 >> 172.21.15.144:0/2297032217 conn(0x7ffb04071340 msgr2=0x7ffb040a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2297032217 shutdown_connections 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2297032217 wait complete. 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 10 monclient: build_initial_monmap 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.577+0000 7ffb0aa80640 10 monclient: monmap: 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:44.591 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 Processor -- start 2024-09-18T12:33:44.592 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 -- start start 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient: init 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb041365e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): picked mon.noname-c con 0x7ffb0406a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb0406e130 0x7ffb04133e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): picked mon.noname-b con 0x7ffb0406e130 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 0x7ffb0411dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): picked mon.noname-a con 0x7ffb0406e500 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04121740 con 0x7ffb0406e500 2024-09-18T12:33:44.593 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0406e970 con 0x7ffb0406e130 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04134c50 con 0x7ffb0406a790 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb0aa80640 10 monclient(hunting): authenticate will time out at 1260.147835s 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb03fff640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb0406e130 msgr2=0x7ffb04133e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.578+0000 7ffb03fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb0406e130 0x7ffb04133e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb041365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 0x7ffb0411dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb041365e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59060/0 (socket says 172.21.15.144:59060) 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 0x7ffb0411dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:54578/0 (socket says 172.21.15.144:54578) 2024-09-18T12:33:44.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 1 -- 172.21.15.144:0/2377259971 learned_addr learned my addr 172.21.15.144:0/2377259971 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 10 monclient(hunting): get_auth_request con 0x7ffb0406a790 auth_method 0 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): get_auth_request con 0x7ffb0406e500 auth_method 0 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:44.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient(hunting): handle_auth_done global_id 104472 payload 824 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 -- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 msgr2=0x7ffb041365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb041365e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 -- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb0406e130 msgr2=0x7ffb04133e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb0406e130 0x7ffb04133e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 monclient: found mon.noname-a 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:44.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0901d640 1 -- 172.21.15.144:0/2377259971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf8009390 con 0x7ffb0406e500 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.579+0000 7ffb0881c640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb041365e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.581+0000 7ffb0901d640 1 -- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 msgr2=0x7ffb0411dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.581+0000 7ffb0901d640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 0x7ffb0411dae0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00be20 tx=0x7ffafc0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc006570 con 0x7ffb0406e500 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: dump: 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.597 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 5 monclient: authenticate success, global_id 104472 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 10 monclient: _renew_subs 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0411f2e0 con 0x7ffb0406e500 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc00f070 con 0x7ffb0406e500 2024-09-18T12:33:44.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb02ffd640 4 set_mon_vals no callback set 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc006730 con 0x7ffb0406e500 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 10 monclient: dump: 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:44.599 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 10 monclient: _renew_subs 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb04134360 con 0x7ffb0406e500 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb04120fb0 con 0x7ffb0406e500 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.582+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ffafc01f3f0 con 0x7ffb0406e500 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.583+0000 7ffb00ff9640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffae8038440 0x7ffae803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:44.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.583+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(229..229 src has 13..229) v4 ==== 14068+0+0 (secure 0 0 0) 0x7ffafc04e650 con 0x7ffb0406e500 2024-09-18T12:33:44.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.583+0000 7ffb0881c640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffae8038440 0x7ffae803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:44.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.583+0000 7ffb0881c640 10 monclient: get_auth_request con 0x7ffae8038440 auth_method 0 2024-09-18T12:33:44.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.585+0000 7ffb0881c640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffae8038440 0x7ffae803a8f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009130 tx=0x7ffaf8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:44.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.587+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ffb04120fb0 con 0x7ffb0406e500 2024-09-18T12:33:44.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.587+0000 7ffb00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:44.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.587+0000 7ffb00ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:44.751 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.750+0000 7ffb0aa80640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T12:33:44.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.750+0000 7ffb0aa80640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T12:33:44.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.750+0000 7ffb0aa80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:44.752 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:44.750+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7ffb04120fb0 con 0x7ffb0406e500 2024-09-18T12:33:45.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.397+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v230) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffb04120fb0 con 0x7ffb0406e500 2024-09-18T12:33:45.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.397+0000 7ffb00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T12:33:45.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.397+0000 7ffb00ff9640 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-18T12:33:45.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.496+0000 7ffb0aa80640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T12:33:45.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.496+0000 7ffb0aa80640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T12:33:45.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.496+0000 7ffb0aa80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:45.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.496+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7ffb04120fb0 con 0x7ffb0406e500 2024-09-18T12:33:45.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.497+0000 7ffb00ff9640 1 -- 172.21.15.144:0/2377259971 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v230) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffb04120fb0 con 0x7ffb0406e500 2024-09-18T12:33:45.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.497+0000 7ffb00ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T12:33:45.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.497+0000 7ffb00ff9640 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-18T12:33:45.498 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h5' does not appear in the crush map 2024-09-18T12:33:45.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffae8038440 msgr2=0x7ffae803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffae8038440 0x7ffae803a8f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009130 tx=0x7ffaf8032000 comp rx=0 tx=0).stop 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 10 monclient: shutdown 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 msgr2=0x7ffb0411dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 0x7ffb0411dae0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00be20 tx=0x7ffafc0096a0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 shutdown_connections 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffae8038440 0x7ffae803a8f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb0406e500 0x7ffb0411dae0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffb0406e130 0x7ffb04133e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 --2- 172.21.15.144:0/2377259971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffb0406a790 0x7ffb041365e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 >> 172.21.15.144:0/2377259971 conn(0x7ffb04071340 msgr2=0x7ffb04134e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 shutdown_connections 2024-09-18T12:33:45.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.499+0000 7ffb0aa80640 1 -- 172.21.15.144:0/2377259971 wait complete. 2024-09-18T12:33:45.509 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:45.509 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h6 2024-09-18T12:33:45.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.581+0000 7fa7d5094640 10 monclient: get_monmap_and_config 2024-09-18T12:33:45.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.581+0000 7fa7d5094640 10 monclient: build_initial_monmap 2024-09-18T12:33:45.582 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.581+0000 7fa7d5094640 10 monclient: monmap: 2024-09-18T12:33:45.582 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:45.582424+0000 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:45.582424+0000 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:45.583 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:45.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 1 Processor -- start 2024-09-18T12:33:45.585 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 1 -- start start 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient: init 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7d006e130 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d009e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d006a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7d006e500 0x7fa7d009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d006e500 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d006a0b0 con 0x7fa7d006e130 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d0068c30 con 0x7fa7d006e500 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d0069750 con 0x7fa7d006a790 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:45.586 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7d5094640 10 monclient(hunting): authenticate will time out at 1261.151752s 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7ced9d640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fa7d006e130 legacy=0x7fa7d009da60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7cf59e640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fa7d006e500 msgr2=0x7fa7d009e6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.582+0000 7fa7cf59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7d006e500 0x7fa7d009e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d009e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d009e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59062/0 (socket says 172.21.15.144:59062) 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 -- 172.21.15.144:0/1823940043 learned_addr learned my addr 172.21.15.144:0/1823940043 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): get_auth_request con 0x7fa7d006a790 auth_method 0 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:45.587 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient(hunting): handle_auth_done global_id 65270 payload 274 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 -- 172.21.15.144:0/1823940043 >> v2:172.21.15.144:3300/0 conn(0x7fa7d006e500 msgr2=0x7fa7d009e6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 --2- 172.21.15.144:0/1823940043 >> v2:172.21.15.144:3300/0 conn(0x7fa7d006e500 0x7fa7d009e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 -- 172.21.15.144:0/1823940043 >> v1:172.21.15.144:6789/0 conn(0x7fa7d006e130 legacy=0x7fa7d009da60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 monclient: found mon.noname-b 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:45.588 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.583+0000 7fa7c7fff640 1 -- 172.21.15.144:0/1823940043 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d0069a60 con 0x7fa7d006a790 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7c7fff640 1 -- 172.21.15.144:0/1823940043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 msgr2=0x7fa7d009e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7c7fff640 1 --2- 172.21.15.144:0/1823940043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d009e170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8009910 tx=0x7fa7b80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e3b8e3bd71dc00 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 1 -- 172.21.15.144:0/1823940043 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b802ae20 con 0x7fa7d006a790 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: dump: 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:45.594 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: _finish_auth 0 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 1 -- 172.21.15.144:0/1823940043 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b802f4b0 con 0x7fa7d006a790 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7d5094640 5 monclient: authenticate success, global_id 65270 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7d5094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:45.595 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 1 -- 172.21.15.144:0/1823940043 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b802f850 con 0x7fa7d006a790 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7d5094640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7d5094640 4 set_mon_vals no callback set 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.586+0000 7fa7ce59c640 10 monclient: dump: 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:45.596 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 10 monclient: shutdown 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 -- 172.21.15.144:0/1823940043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 msgr2=0x7fa7d009e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 --2- 172.21.15.144:0/1823940043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d009e170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8009910 tx=0x7fa7b80091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 -- 172.21.15.144:0/1823940043 shutdown_connections 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 --2- 172.21.15.144:0/1823940043 >> v2:172.21.15.144:3300/0 conn(0x7fa7d006e500 0x7fa7d009e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 --2- 172.21.15.144:0/1823940043 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d009e170 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 -- 172.21.15.144:0/1823940043 >> 172.21.15.144:0/1823940043 conn(0x7fa7d0071340 msgr2=0x7fa7d00a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 -- 172.21.15.144:0/1823940043 shutdown_connections 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 1 -- 172.21.15.144:0/1823940043 wait complete. 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 10 monclient: build_initial_monmap 2024-09-18T12:33:45.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.587+0000 7fa7d5094640 10 monclient: monmap: 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 Processor -- start 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 -- start start 2024-09-18T12:33:45.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient: init 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d0141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d006a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d006e130 0x7fa7d0133ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d006e130 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d006e500 0x7fa7d011dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d006e500 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d0142200 con 0x7fa7d006e500 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d006e970 con 0x7fa7d006e130 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:45.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d0134c30 con 0x7fa7d006a790 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7c7fff640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d006e130 msgr2=0x7fa7d0133ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7d5094640 10 monclient(hunting): authenticate will time out at 1261.157590s 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7c7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d006e130 0x7fa7d0133ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d0141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d006e500 0x7fa7d011dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d006e500 0x7fa7d011dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:54586/0 (socket says 172.21.15.144:54586) 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d0141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59070/0 (socket says 172.21.15.144:59070) 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 1 -- 172.21.15.144:0/2790407055 learned_addr learned my addr 172.21.15.144:0/2790407055 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 10 monclient(hunting): get_auth_request con 0x7fa7d006e500 auth_method 0 2024-09-18T12:33:45.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 10 monclient(hunting): get_auth_request con 0x7fa7d006a790 auth_method 0 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.588+0000 7fa7ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:45.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 10 monclient(hunting): handle_auth_done global_id 65273 payload 824 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 1 -- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d006e130 msgr2=0x7fa7d0133ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d006e130 0x7fa7d0133ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 1 -- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d006e500 msgr2=0x7fa7d011dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d006e500 0x7fa7d011dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 1 monclient: found mon.noname-c 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7ced9d640 1 -- 172.21.15.144:0/2790407055 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8009450 con 0x7fa7d006a790 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.589+0000 7fa7cf59e640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d006e500 0x7fa7d011dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7ced9d640 1 -- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 msgr2=0x7fa7d0141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:45.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7ced9d640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d0141320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000b5b0 tx=0x7fa7c00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7ced9d640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d0141320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000b5b0 tx=0x7fa7c00090f0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fa7c0003d90 mon_map magic: 0 v1 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0003d90 con 0x7fa7d006a790 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 10 monclient: dump: 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:45.603 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c0006150 con 0x7fa7d006a790 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7d5094640 5 monclient: authenticate success, global_id 65273 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7d5094640 10 monclient: _renew_subs 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7d5094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d011f510 con 0x7fa7d006a790 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7cdd9b640 4 set_mon_vals no callback set 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.591+0000 7fa7d5094640 10 monclient: _renew_subs 2024-09-18T12:33:45.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.592+0000 7fa7d5094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.592+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d01200a0 con 0x7fa7d006a790 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.592+0000 7fa7d5094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.592+0000 7fa7d5094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.592+0000 7fa7d5094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.592+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d00008d0 con 0x7fa7d006a790 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.592+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0006510 con 0x7fa7d006a790 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.593+0000 7fa7c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.593+0000 7fa7c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.593+0000 7fa7c6ffd640 10 monclient: dump: 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:45.605 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.593+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa7c001b440 con 0x7fa7d006a790 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.593+0000 7fa7c6ffd640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7b0038490 0x7fa7b003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.593+0000 7fa7c7fff640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7b0038490 0x7fa7b003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:45.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.593+0000 7fa7c7fff640 10 monclient: get_auth_request con 0x7fa7b0038490 auth_method 0 2024-09-18T12:33:45.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.595+0000 7fa7c7fff640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7b0038490 0x7fa7b003a940 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa7d009fac0 tx=0x7fa7b8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:45.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:45.598+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(230..230 src has 13..230) v4 ==== 13858+0+0 (secure 0 0 0) 0x7fa7c004d3d0 con 0x7fa7d006a790 2024-09-18T12:33:46.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.214+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa7c000dd80 con 0x7fa7d006a790 2024-09-18T12:33:46.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.214+0000 7fa7c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:46.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.214+0000 7fa7c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:46.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.371+0000 7fa7d5094640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T12:33:46.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.371+0000 7fa7d5094640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T12:33:46.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.371+0000 7fa7d5094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:46.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.371+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7fa7d006d060 con 0x7fa7d006a790 2024-09-18T12:33:46.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.990+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v232) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa7c0016030 con 0x7fa7d006a790 2024-09-18T12:33:46.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.990+0000 7fa7c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T12:33:46.991 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:46.990+0000 7fa7c6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-18T12:33:47.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.088+0000 7fa7d5094640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T12:33:47.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.088+0000 7fa7d5094640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T12:33:47.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.088+0000 7fa7d5094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:47.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.088+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7fa7d0121060 con 0x7fa7d006a790 2024-09-18T12:33:47.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.089+0000 7fa7c6ffd640 1 -- 172.21.15.144:0/2790407055 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v232) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa7c0016030 con 0x7fa7d006a790 2024-09-18T12:33:47.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.089+0000 7fa7c6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T12:33:47.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.089+0000 7fa7c6ffd640 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-18T12:33:47.091 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h6' does not appear in the crush map 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.091+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7b0038490 msgr2=0x7fa7b003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.091+0000 7fa7d5094640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7b0038490 0x7fa7b003a940 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa7d009fac0 tx=0x7fa7b8032000 comp rx=0 tx=0).stop 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.091+0000 7fa7d5094640 10 monclient: shutdown 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.091+0000 7fa7d5094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.091+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 msgr2=0x7fa7d0141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.091+0000 7fa7d5094640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d0141320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000b5b0 tx=0x7fa7c00090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 shutdown_connections 2024-09-18T12:33:47.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa7b0038490 0x7fa7b003a940 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7d006e500 0x7fa7d011dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7d006e130 0x7fa7d0133ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 --2- 172.21.15.144:0/2790407055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa7d006a790 0x7fa7d0141320 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 >> 172.21.15.144:0/2790407055 conn(0x7fa7d0071340 msgr2=0x7fa7d00a21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:47.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 shutdown_connections 2024-09-18T12:33:47.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.092+0000 7fa7d5094640 1 -- 172.21.15.144:0/2790407055 wait complete. 2024-09-18T12:33:47.101 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:47.102 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h7 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.174+0000 7fbfdc11d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.174+0000 7fbfdc11d640 10 monclient: build_initial_monmap 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.174+0000 7fbfdc11d640 10 monclient: monmap: 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:47.175117+0000 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:47.175117+0000 2024-09-18T12:33:47.175 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:47.176 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:47.176 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:47.176 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:47.176 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:47.176 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:47.176 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:47.176 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:47.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.174+0000 7fbfdc11d640 1 Processor -- start 2024-09-18T12:33:47.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 1 -- start start 2024-09-18T12:33:47.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient: init 2024-09-18T12:33:47.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:47.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:33:47.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfd409ec50 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfd409f020 0x7fbfd4069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd409f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfd4136d60 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd4067420 con 0x7fbfd4136d60 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd40a0480 con 0x7fbfd409ec50 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd40a0200 con 0x7fbfd409f020 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfdc11d640 10 monclient(hunting): authenticate will time out at 1262.744531s 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfd9eb9640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fbfd409ec50 legacy=0x7fbfd4131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfd96b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfd409f020 0x7fbfd4069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfda6ba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbfd4136d60 0x7fbfd4139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:58548/0 (socket says 172.21.15.144:58548) 2024-09-18T12:33:47.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfda6ba640 1 -- 172.21.15.144:0/3339039260 learned_addr learned my addr 172.21.15.144:0/3339039260 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfd96b8640 10 monclient(hunting): get_auth_request con 0x7fbfd409f020 auth_method 0 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfd96b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfd96b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.175+0000 7fbfd96b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 10 monclient(hunting): handle_auth_done global_id 104478 payload 274 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 1 -- 172.21.15.144:0/3339039260 >> v1:172.21.15.144:6789/0 conn(0x7fbfd409ec50 legacy=0x7fbfd4131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 1 -- 172.21.15.144:0/3339039260 >> v1:172.21.15.116:6789/0 conn(0x7fbfd4136d60 legacy=0x7fbfd4139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 10 monclient: _finish_hunting 0 2024-09-18T12:33:47.180 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 1 monclient: found mon.noname-a 2024-09-18T12:33:47.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:47.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:47.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.176+0000 7fbfd96b8640 1 -- 172.21.15.144:0/3339039260 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd409fd80 con 0x7fbfd409f020 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd96b8640 1 -- 172.21.15.144:0/3339039260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd409f020 msgr2=0x7fbfd4069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd96b8640 1 --2- 172.21.15.144:0/3339039260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd409f020 0x7fbfd4069d10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009850 tx=0x7fbfc40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fb62bbe03175f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 1 -- 172.21.15.144:0/3339039260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4032440 con 0x7fbfd409f020 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: dump: 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:47.187 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: _finish_auth 0 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 1 -- 172.21.15.144:0/3339039260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc40325a0 con 0x7fbfd409f020 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfdc11d640 5 monclient: authenticate success, global_id 104478 2024-09-18T12:33:47.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfdc11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 1 -- 172.21.15.144:0/3339039260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4032940 con 0x7fbfd409f020 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfdc11d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfdc11d640 4 set_mon_vals no callback set 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.179+0000 7fbfd8eb7640 10 monclient: dump: 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:47.189 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 10 monclient: shutdown 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 1 -- 172.21.15.144:0/3339039260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd409f020 msgr2=0x7fbfd4069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/3339039260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd409f020 0x7fbfd4069d10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009850 tx=0x7fbfc40090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 1 -- 172.21.15.144:0/3339039260 shutdown_connections 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/3339039260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd409f020 0x7fbfd4069d10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 1 -- 172.21.15.144:0/3339039260 >> 172.21.15.144:0/3339039260 conn(0x7fbfd40a5cf0 msgr2=0x7fbfd40a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 1 -- 172.21.15.144:0/3339039260 shutdown_connections 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.180+0000 7fbfdc11d640 1 -- 172.21.15.144:0/3339039260 wait complete. 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient: build_initial_monmap 2024-09-18T12:33:47.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient: monmap: 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 Processor -- start 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 -- start start 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient: init 2024-09-18T12:33:47.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 0x7fbfd4135da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd409ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd409f020 0x7fbfd4096a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd409f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd4136d60 0x7fbfd4096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd4136d60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4097d80 con 0x7fbfd4136d60 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd409c020 con 0x7fbfd409f020 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40a3f30 con 0x7fbfd409ec50 2024-09-18T12:33:47.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfdc11d640 10 monclient(hunting): authenticate will time out at 1262.750928s 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfd96b8640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd409f020 msgr2=0x7fbfd4096a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.181+0000 7fbfd96b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd409f020 0x7fbfd4096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 0x7fbfd4135da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 0x7fbfd4135da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59084/0 (socket says 172.21.15.144:59084) 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 -- 172.21.15.144:0/2878773534 learned_addr learned my addr 172.21.15.144:0/2878773534 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): get_auth_request con 0x7fbfd409ec50 auth_method 0 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd4136d60 0x7fbfd4096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 10 monclient(hunting): get_auth_request con 0x7fbfd4136d60 auth_method 0 2024-09-18T12:33:47.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient(hunting): handle_auth_done global_id 65276 payload 824 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 -- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd409f020 msgr2=0x7fbfd4096a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd409f020 0x7fbfd4096a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 -- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd4136d60 msgr2=0x7fbfd4096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd4136d60 0x7fbfd4096f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:47.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient: _finish_hunting 0 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 monclient: found mon.noname-c 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfd9eb9640 1 -- 172.21.15.144:0/2878773534 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc4009390 con 0x7fbfd409ec50 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.182+0000 7fbfda6ba640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd4136d60 0x7fbfd4096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.184+0000 7fbfd9eb9640 1 -- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 msgr2=0x7fbfd4135da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.184+0000 7fbfd9eb9640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 0x7fbfd4135da0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000b620 tx=0x7fbfd0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0015070 con 0x7fbfd409ec50 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: dump: 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:47.195 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfdc11d640 5 monclient: authenticate success, global_id 65276 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfdc11d640 10 monclient: _renew_subs 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0003d90 con 0x7fbfd409ec50 2024-09-18T12:33:47.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd4099820 con 0x7fbfd409ec50 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0006380 con 0x7fbfd409ec50 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfc9ffb640 10 monclient: dump: 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:47.197 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfdc11d640 10 monclient: _renew_subs 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfcbfff640 4 set_mon_vals no callback set 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.185+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd4098a60 con 0x7fbfd409ec50 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbfd0010440 con 0x7fbfd409ec50 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfdc11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfdc11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd4099f00 con 0x7fbfd409ec50 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfc9ffb640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbfb8038440 0x7fbfb803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:47.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(232..232 src has 13..232) v4 ==== 13648+0+0 (secure 0 0 0) 0x7fbfd004d5a0 con 0x7fbfd409ec50 2024-09-18T12:33:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfd96b8640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbfb8038440 0x7fbfb803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.186+0000 7fbfd96b8640 10 monclient: get_auth_request con 0x7fbfb8038440 auth_method 0 2024-09-18T12:33:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.188+0000 7fbfd96b8640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbfb8038440 0x7fbfb803a8f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4031000 tx=0x7fbfc4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.190+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbfd0006770 con 0x7fbfd409ec50 2024-09-18T12:33:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.190+0000 7fbfc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:47.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.190+0000 7fbfc9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:47.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.347+0000 7fbfdc11d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T12:33:47.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.347+0000 7fbfdc11d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T12:33:47.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.347+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:47.348 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.347+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7fbfd40008d0 con 0x7fbfd409ec50 2024-09-18T12:33:47.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.995+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v233) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbfd0018a90 con 0x7fbfd409ec50 2024-09-18T12:33:47.995 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.995+0000 7fbfc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T12:33:47.996 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:47.995+0000 7fbfc9ffb640 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-18T12:33:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.091+0000 7fbfdc11d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T12:33:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.091+0000 7fbfdc11d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T12:33:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.091+0000 7fbfdc11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:48.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.091+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7fbfd40a13e0 con 0x7fbfd409ec50 2024-09-18T12:33:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.092+0000 7fbfc9ffb640 1 -- 172.21.15.144:0/2878773534 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v233) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbfd0006770 con 0x7fbfd409ec50 2024-09-18T12:33:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.092+0000 7fbfc9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T12:33:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.092+0000 7fbfc9ffb640 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-18T12:33:48.093 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h7' does not appear in the crush map 2024-09-18T12:33:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbfb8038440 msgr2=0x7fbfb803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:48.095 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbfb8038440 0x7fbfb803a8f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4031000 tx=0x7fbfc4032000 comp rx=0 tx=0).stop 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 10 monclient: shutdown 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 msgr2=0x7fbfd4135da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 0x7fbfd4135da0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000b620 tx=0x7fbfd0009120 comp rx=0 tx=0).stop 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 shutdown_connections 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbfb8038440 0x7fbfb803a8f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfd4136d60 0x7fbfd4096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbfd409f020 0x7fbfd4096a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 --2- 172.21.15.144:0/2878773534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbfd409ec50 0x7fbfd4135da0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 >> 172.21.15.144:0/2878773534 conn(0x7fbfd40a5cf0 msgr2=0x7fbfd409e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:48.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 shutdown_connections 2024-09-18T12:33:48.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.094+0000 7fbfdc11d640 1 -- 172.21.15.144:0/2878773534 wait complete. 2024-09-18T12:33:48.104 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:48.104 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h8 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.177+0000 7f2fed79d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.177+0000 7f2fed79d640 10 monclient: build_initial_monmap 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.177+0000 7f2fed79d640 10 monclient: monmap: 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:48.178855+0000 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:48.178855+0000 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:48.179 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:48.180 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:48.180 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:48.180 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:48.180 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:48.181 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 1 Processor -- start 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 1 -- start start 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 10 monclient: init 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2fe8131c90 0x7f2fe8132060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 10 monclient(hunting): picked mon.noname-c con 0x7f2fe8131c90 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.178+0000 7f2fed79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fe809d210 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fed79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fe80a0ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fed79d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe80a1800 con 0x7f2fe809d210 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fed79d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe809fea0 con 0x7f2fe80a0ab0 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fed79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:48.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fed79d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80a1520 con 0x7f2fe8131c90 2024-09-18T12:33:48.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fed79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:48.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fed79d640 10 monclient(hunting): authenticate will time out at 1263.748042s 2024-09-18T12:33:48.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fe6ffd640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f2fe8131c90 msgr2=0x7f2fe8132060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:48.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fe77fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f2fe80a0ab0 legacy=0x7f2fe8069510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:48.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fe6ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2fe8131c90 0x7f2fe8132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:48.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fe67fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2fe809d210 0x7f2fe8068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:56368/0 (socket says 172.21.15.144:56368) 2024-09-18T12:33:48.183 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.179+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1901981133 learned_addr learned my addr 172.21.15.144:0/1901981133 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:48.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.181+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1901981133 >> v1:172.21.15.116:6789/0 conn(0x7f2fe809d210 legacy=0x7f2fe8068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:48.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.181+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408564668 0 0) 0x7f2fe80a1800 con 0x7f2fe809d210 2024-09-18T12:33:48.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.181+0000 7f2fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:48.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.181+0000 7f2fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:48.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.181+0000 7f2fe5ffb640 10 monclient(hunting): my global_id is 104484 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.181+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd40034e0 con 0x7f2fe809d210 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934798704 0 0) 0x7f2fd40034e0 con 0x7f2fe809d210 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 >> v2:172.21.15.144:3300/0 conn(0x7f2fe8131c90 msgr2=0x7f2fe8132060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 --2- 172.21.15.144:0/1901981133 >> v2:172.21.15.144:3300/0 conn(0x7f2fe8131c90 0x7f2fe8132060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 >> v1:172.21.15.144:6789/0 conn(0x7f2fe80a0ab0 legacy=0x7f2fe8069510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe81325a0 con 0x7f2fe809d210 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:48.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f2fd0001850 con 0x7f2fe809d210 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: dump: 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:48.195 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 >> v1:172.21.15.116:6789/0 conn(0x7f2fe809d210 legacy=0x7f2fe8068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd40039d0 0x7f2fd4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): picked mon.c con 0x7f2fd40039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fd4004780 0x7f2fd4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): picked mon.a con 0x7f2fd4004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd4005330 0x7f2fd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): picked mon.b con 0x7f2fd4005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80a1520 con 0x7f2fd4004780 2024-09-18T12:33:48.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd400c290 con 0x7f2fd4005330 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd400c450 con 0x7f2fd40039d0 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe77fe640 1 -- 172.21.15.144:0/1901981133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd4005330 msgr2=0x7f2fd400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe77fe640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd4005330 0x7f2fd400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fd4004780 0x7f2fd4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): get_auth_request con 0x7f2fd4004780 auth_method 0 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:48.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd40039d0 0x7f2fd4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 10 monclient(hunting): get_auth_request con 0x7f2fd40039d0 auth_method 0 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_done global_id 104484 payload 293 2024-09-18T12:33:48.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd40039d0 msgr2=0x7f2fd4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd40039d0 0x7f2fd4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1901981133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd4005330 msgr2=0x7f2fd400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fd4005330 0x7f2fd400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient: _finish_hunting 0 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 1 monclient: found mon.a 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe77fe640 1 -- 172.21.15.144:0/1901981133 reap_dead start 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1901981133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40034e0 con 0x7f2fd4004780 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fed79d640 5 monclient: authenticate success, global_id 104484 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.182+0000 7f2fed79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:48.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fd4004780 msgr2=0x7f2fd4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe67fc640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fd4004780 0x7f2fd4004bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c500 tx=0x7f2fd0007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c215ccbb45b06c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0005890 con 0x7f2fd4004780 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 10 monclient: dump: 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:48.200 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fed79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0005a10 con 0x7f2fd4004780 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 1 -- 172.21.15.144:0/1901981133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0005dd0 con 0x7f2fd4004780 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fe5ffb640 10 monclient: dump: 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.201 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fed79d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.185+0000 7f2fed79d640 4 set_mon_vals no callback set 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 10 monclient: shutdown 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 1 -- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fd4004780 msgr2=0x7f2fd4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fd4004780 0x7f2fd4004bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c500 tx=0x7f2fd0007ad0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 1 -- 172.21.15.144:0/1901981133 shutdown_connections 2024-09-18T12:33:48.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1901981133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fd4004780 0x7f2fd4004bf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 1 -- 172.21.15.144:0/1901981133 >> 172.21.15.144:0/1901981133 conn(0x7f2fe8071370 msgr2=0x7f2fe80a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 1 -- 172.21.15.144:0/1901981133 shutdown_connections 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 1 -- 172.21.15.144:0/1901981133 wait complete. 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 10 monclient: build_initial_monmap 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.186+0000 7f2fed79d640 10 monclient: monmap: 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:48.203 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 Processor -- start 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 -- start start 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient: init 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd4004780 0x7f2fe8130cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd4004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 0x7f2fe8140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): picked mon.noname-a con 0x7f2fe8131200 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe81413f0 0x7f2fe811b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): picked mon.noname-b con 0x7f2fe81413f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:48.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe811d330 con 0x7f2fe8131200 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81315d0 con 0x7f2fe81413f0 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe811f0d0 con 0x7f2fd4004780 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fed79d640 10 monclient(hunting): authenticate will time out at 1263.756835s 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fe77fe640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe81413f0 msgr2=0x7f2fe811b900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fe77fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe81413f0 0x7f2fe811b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.187+0000 7f2fdeffd640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd4004780 0x7f2fe8130cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 0x7f2fe8140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:48.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 0x7f2fe8140eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:34424/0 (socket says 172.21.15.144:34424) 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd4004780 0x7f2fe8130cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59748/0 (socket says 172.21.15.144:59748) 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1818852176 learned_addr learned my addr 172.21.15.144:0/1818852176 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): get_auth_request con 0x7f2fe8131200 auth_method 0 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 10 monclient(hunting): get_auth_request con 0x7f2fd4004780 auth_method 0 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:48.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient(hunting): handle_auth_done global_id 104487 payload 824 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd4004780 msgr2=0x7f2fe8130cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd4004780 0x7f2fe8130cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe81413f0 msgr2=0x7f2fe811b900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe81413f0 0x7f2fe811b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient: _finish_hunting 0 2024-09-18T12:33:48.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 monclient: found mon.noname-a 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1818852176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe811edf0 con 0x7f2fe8131200 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.188+0000 7f2fe6ffd640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd4004780 0x7f2fe8130cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.190+0000 7f2fe67fc640 1 -- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 msgr2=0x7f2fe8140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.190+0000 7f2fe67fc640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 0x7f2fe8140eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c560 tx=0x7f2fd0034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0007510 con 0x7f2fe8131200 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: dump: 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:48.208 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 5 monclient: authenticate success, global_id 104487 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0007670 con 0x7f2fe8131200 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd002f650 con 0x7f2fe8131200 2024-09-18T12:33:48.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 10 monclient: _renew_subs 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe811f3b0 con 0x7f2fe8131200 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fe57fa640 4 set_mon_vals no callback set 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fdeffd640 10 monclient: dump: 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:48.210 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 10 monclient: _renew_subs 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe81418a0 con 0x7f2fe8131200 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.191+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe80008d0 con 0x7f2fe8131200 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.192+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2fd002fa00 con 0x7f2fe8131200 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.192+0000 7f2fdeffd640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2fc8038440 0x7f2fc803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.192+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(233..233 src has 13..233) v4 ==== 13438+0+0 (secure 0 0 0) 0x7f2fd006d970 con 0x7f2fe8131200 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.192+0000 7f2fe6ffd640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2fc8038440 0x7f2fc803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:48.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.192+0000 7f2fe6ffd640 10 monclient: get_auth_request con 0x7f2fc8038440 auth_method 0 2024-09-18T12:33:48.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.194+0000 7f2fe6ffd640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2fc8038440 0x7f2fc803a8f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8066f10 tx=0x7f2fe000e040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:48.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.197+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2fd003a410 con 0x7f2fe8131200 2024-09-18T12:33:48.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.197+0000 7f2fdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:48.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.197+0000 7f2fdeffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:48.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.354+0000 7f2fed79d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T12:33:48.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.354+0000 7f2fed79d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T12:33:48.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.354+0000 7f2fed79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:48.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.354+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f2fe8064a20 con 0x7f2fe8131200 2024-09-18T12:33:48.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.998+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v234) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2fd0043390 con 0x7f2fe8131200 2024-09-18T12:33:48.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.998+0000 7f2fdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T12:33:48.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:48.998+0000 7f2fdeffd640 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-18T12:33:49.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.098+0000 7f2fed79d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T12:33:49.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.098+0000 7f2fed79d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T12:33:49.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.098+0000 7f2fed79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:49.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.098+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f2fe811be40 con 0x7f2fe8131200 2024-09-18T12:33:49.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.099+0000 7f2fdeffd640 1 -- 172.21.15.144:0/1818852176 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v234) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2fd0043570 con 0x7f2fe8131200 2024-09-18T12:33:49.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.099+0000 7f2fdeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T12:33:49.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.099+0000 7f2fdeffd640 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-18T12:33:49.100 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h8' does not appear in the crush map 2024-09-18T12:33:49.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2fc8038440 msgr2=0x7f2fc803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:49.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2fc8038440 0x7f2fc803a8f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8066f10 tx=0x7f2fe000e040 comp rx=0 tx=0).stop 2024-09-18T12:33:49.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 10 monclient: shutdown 2024-09-18T12:33:49.102 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 msgr2=0x7f2fe8140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 0x7f2fe8140eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c560 tx=0x7f2fd0034000 comp rx=0 tx=0).stop 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 shutdown_connections 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2fc8038440 0x7f2fc803a8f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2fe81413f0 0x7f2fe811b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2fe8131200 0x7f2fe8140eb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 --2- 172.21.15.144:0/1818852176 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2fd4004780 0x7f2fe8130cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 >> 172.21.15.144:0/1818852176 conn(0x7f2fe8071370 msgr2=0x7f2fe809dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 shutdown_connections 2024-09-18T12:33:49.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.101+0000 7f2fed79d640 1 -- 172.21.15.144:0/1818852176 wait complete. 2024-09-18T12:33:49.111 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:49.111 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h9 2024-09-18T12:33:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.185+0000 7f5f0ed1d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:49.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.185+0000 7f5f0ed1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.185+0000 7f5f0ed1d640 10 monclient: monmap: 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:49.186412+0000 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:49.186412+0000 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:49.187 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:49.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 1 Processor -- start 2024-09-18T12:33:49.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 1 -- start start 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient: init 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f08099130 0x7f5f08099500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f08099130 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f0806a790 0x7f5f0809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0806a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f0809dfa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f08069770 con 0x7f5f0809dfa0 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f0809a080 con 0x7f5f08099130 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:49.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0806a040 con 0x7f5f0806a790 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0ed1d640 10 monclient(hunting): authenticate will time out at 1264.755779s 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.186+0000 7f5f0d2ba640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f5f0809dfa0 legacy=0x7f5f0809e400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f08099130 0x7f5f08099500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f0806a790 0x7f5f0809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f08099130 0x7f5f08099500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:34438/0 (socket says 172.21.15.144:34438) 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 -- 172.21.15.144:0/3527250362 learned_addr learned my addr 172.21.15.144:0/3527250362 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): get_auth_request con 0x7f5f08099130 auth_method 0 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 10 monclient(hunting): get_auth_request con 0x7f5f0806a790 auth_method 0 2024-09-18T12:33:49.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient(hunting): handle_auth_done global_id 104490 payload 274 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 -- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f0806a790 msgr2=0x7f5f0809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 --2- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f0806a790 0x7f5f0809da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 -- 172.21.15.144:0/3527250362 >> v1:172.21.15.144:6789/0 conn(0x7f5f0809dfa0 legacy=0x7f5f0809e400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient: _finish_hunting 0 2024-09-18T12:33:49.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 monclient: found mon.noname-a 2024-09-18T12:33:49.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:49.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:49.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f0cab9640 1 -- 172.21.15.144:0/3527250362 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0809f5d0 con 0x7f5f08099130 2024-09-18T12:33:49.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.187+0000 7f5f07fff640 1 --2- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f0806a790 0x7f5f0809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:49.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0cab9640 1 -- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f08099130 msgr2=0x7f5f08099500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:49.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0cab9640 1 --2- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f08099130 0x7f5f08099500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009940 tx=0x7f5efc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd23e8758c5969e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:49.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 1 -- 172.21.15.144:0/3527250362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc02ae80 con 0x7f5f08099130 2024-09-18T12:33:49.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:49.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: dump: 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:49.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 1 -- 172.21.15.144:0/3527250362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc032440 con 0x7f5f08099130 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 5 monclient: authenticate success, global_id 104490 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 4 set_mon_vals no callback set 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 1 -- 172.21.15.144:0/3527250362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc0327c0 con 0x7f5f08099130 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:49.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f077fe640 10 monclient: dump: 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 10 monclient: shutdown 2024-09-18T12:33:49.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f08099130 msgr2=0x7f5f08099500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.190+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f08099130 0x7f5f08099500 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009940 tx=0x7f5efc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3527250362 shutdown_connections 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f0806a790 0x7f5f0809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3527250362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f08099130 0x7f5f08099500 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3527250362 >> 172.21.15.144:0/3527250362 conn(0x7f5f08071340 msgr2=0x7f5f080a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3527250362 shutdown_connections 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3527250362 wait complete. 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 10 monclient: build_initial_monmap 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 10 monclient: monmap: 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:49.204 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.191+0000 7f5f0ed1d640 1 Processor -- start 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 1 -- start start 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient: init 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:49.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f0806a790 0x7f5f08141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0806a790 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f08099130 0x7f5f08133ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f08099130 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 0x7f5f0811dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0809dfa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0806eba0 con 0x7f5f0809dfa0 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0811f9f0 con 0x7f5f0806a790 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08097ff0 con 0x7f5f08099130 2024-09-18T12:33:49.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0cab9640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f0806a790 msgr2=0x7f5f08141210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0ed1d640 10 monclient(hunting): authenticate will time out at 1264.761452s 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0cab9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f0806a790 0x7f5f08141210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f08099130 0x7f5f08133ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 0x7f5f0811dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 0x7f5f0811dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:34450/0 (socket says 172.21.15.144:34450) 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f08099130 0x7f5f08133ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59774/0 (socket says 172.21.15.144:59774) 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 1 -- 172.21.15.144:0/3360967612 learned_addr learned my addr 172.21.15.144:0/3360967612 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 10 monclient(hunting): get_auth_request con 0x7f5f0809dfa0 auth_method 0 2024-09-18T12:33:49.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 10 monclient(hunting): get_auth_request con 0x7f5f08099130 auth_method 0 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f0d2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:49.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.192+0000 7f5f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 10 monclient(hunting): handle_auth_done global_id 104493 payload 824 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 1 -- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f08099130 msgr2=0x7f5f08133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f08099130 0x7f5f08133ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 1 -- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f0806a790 msgr2=0x7f5f08141210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f0806a790 0x7f5f08141210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 10 monclient: _finish_hunting 0 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 1 monclient: found mon.noname-a 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f0d2ba640 1 -- 172.21.15.144:0/3360967612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4009390 con 0x7f5f0809dfa0 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.193+0000 7f5f07fff640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f08099130 0x7f5f08133ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0d2ba640 1 -- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 msgr2=0x7f5f0811dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0d2ba640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 0x7f5f0811dae0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000bf80 tx=0x7f5f00009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:49.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f00016430 con 0x7f5f0809dfa0 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: dump: 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:49.210 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: _finish_auth 0 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0ed1d640 5 monclient: authenticate success, global_id 104493 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0ed1d640 10 monclient: _renew_subs 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0ed1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc009450 con 0x7f5f0809dfa0 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0ed1d640 10 monclient: _renew_subs 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0ed1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f081210c0 con 0x7f5f0809dfa0 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f00016590 con 0x7f5f0809dfa0 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:49.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f00016700 con 0x7f5f0809dfa0 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f04ff9640 10 monclient: dump: 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:49.212 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.195+0000 7f5f067fc640 4 set_mon_vals no callback set 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.196+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5f00012020 con 0x7f5f0809dfa0 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.196+0000 7f5f0ed1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.196+0000 7f5f0ed1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.196+0000 7f5f0ed1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.196+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee8005010 con 0x7f5f0809dfa0 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.196+0000 7f5f04ff9640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5eec038440 0x7f5eec03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.197+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(234..234 src has 13..234) v4 ==== 13228+0+0 (secure 0 0 0) 0x7f5f0004da40 con 0x7f5f0809dfa0 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.197+0000 7f5f0cab9640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5eec038440 0x7f5eec03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.197+0000 7f5f0cab9640 10 monclient: get_auth_request con 0x7f5eec038440 auth_method 0 2024-09-18T12:33:49.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.199+0000 7f5f0cab9640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5eec038440 0x7f5eec03a8f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5f0809f460 tx=0x7f5efc032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:49.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.201+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5f00006150 con 0x7f5f0809dfa0 2024-09-18T12:33:49.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.201+0000 7f5f04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:49.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.201+0000 7f5f04ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:49.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.358+0000 7f5f0ed1d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T12:33:49.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.359+0000 7f5f0ed1d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T12:33:49.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.359+0000 7f5f0ed1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:49.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.359+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f5ee80052c0 con 0x7f5f0809dfa0 2024-09-18T12:33:49.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.998+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v235) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5f0001d300 con 0x7f5f0809dfa0 2024-09-18T12:33:49.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.998+0000 7f5f04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T12:33:49.999 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:49.998+0000 7f5f04ff9640 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-18T12:33:50.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.095+0000 7f5f0ed1d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T12:33:50.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.095+0000 7f5f0ed1d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T12:33:50.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.095+0000 7f5f0ed1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:50.096 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.095+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f5ee8005810 con 0x7f5f0809dfa0 2024-09-18T12:33:50.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.096+0000 7f5f04ff9640 1 -- 172.21.15.144:0/3360967612 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v235) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5f0001d950 con 0x7f5f0809dfa0 2024-09-18T12:33:50.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.096+0000 7f5f04ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T12:33:50.097 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.096+0000 7f5f04ff9640 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-18T12:33:50.097 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h9' does not appear in the crush map 2024-09-18T12:33:50.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5eec038440 msgr2=0x7f5eec03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:50.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5eec038440 0x7f5eec03a8f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5f0809f460 tx=0x7f5efc032000 comp rx=0 tx=0).stop 2024-09-18T12:33:50.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 10 monclient: shutdown 2024-09-18T12:33:50.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:50.099 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 msgr2=0x7f5f0811dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 0x7f5f0811dae0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000bf80 tx=0x7f5f00009730 comp rx=0 tx=0).stop 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 shutdown_connections 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5eec038440 0x7f5eec03a8f0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0809dfa0 0x7f5f0811dae0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f08099130 0x7f5f08133ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 --2- 172.21.15.144:0/3360967612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f0806a790 0x7f5f08141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 >> 172.21.15.144:0/3360967612 conn(0x7f5f08071340 msgr2=0x7f5f08098660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 shutdown_connections 2024-09-18T12:33:50.100 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.098+0000 7f5f0ed1d640 1 -- 172.21.15.144:0/3360967612 wait complete. 2024-09-18T12:33:50.108 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:50.108 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h10 2024-09-18T12:33:50.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.181+0000 7f90bc8d4640 10 monclient: get_monmap_and_config 2024-09-18T12:33:50.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.181+0000 7f90bc8d4640 10 monclient: build_initial_monmap 2024-09-18T12:33:50.182 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.181+0000 7f90bc8d4640 10 monclient: monmap: 2024-09-18T12:33:50.182 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:50.182 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:50.182 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:50.182319+0000 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:50.182319+0000 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:50.183 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:50.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 Processor -- start 2024-09-18T12:33:50.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 -- start start 2024-09-18T12:33:50.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient: init 2024-09-18T12:33:50.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:50.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T12:33:50.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90b806e130 0x7f90b809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:50.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): picked mon.noname-a con 0x7f90b806e130 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806a790 0x7f90b809dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): picked mon.noname-b con 0x7f90b806a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f90b806e500 0x7f90b809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): picked mon.noname-c con 0x7f90b806e500 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b806a0b0 con 0x7f90b806e130 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b8068c30 con 0x7f90b806e500 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b8069750 con 0x7f90b806a790 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:50.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90bc8d4640 10 monclient(hunting): authenticate will time out at 1265.751735s 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b6d9d640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f90b806e500 msgr2=0x7f90b809e4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b6d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f90b806e500 0x7f90b809e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90b806e130 0x7f90b809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90affff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806a790 0x7f90b809dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90b806e130 0x7f90b809da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:34456/0 (socket says 172.21.15.144:34456) 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90affff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806a790 0x7f90b809dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59786/0 (socket says 172.21.15.144:59786) 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b659c640 1 -- 172.21.15.144:0/2897395956 learned_addr learned my addr 172.21.15.144:0/2897395956 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b659c640 10 monclient(hunting): get_auth_request con 0x7f90b806e130 auth_method 0 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90affff640 10 monclient(hunting): get_auth_request con 0x7f90b806a790 auth_method 0 2024-09-18T12:33:50.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90affff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.182+0000 7f90affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 10 monclient(hunting): handle_auth_done global_id 104496 payload 274 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 1 -- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806a790 msgr2=0x7f90b809dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 1 --2- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806a790 0x7f90b809dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 1 -- 172.21.15.144:0/2897395956 >> v2:172.21.15.144:3300/0 conn(0x7f90b806e500 msgr2=0x7f90b809e4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:50.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 1 --2- 172.21.15.144:0/2897395956 >> v2:172.21.15.144:3300/0 conn(0x7f90b806e500 0x7f90b809e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:50.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 1 monclient: found mon.noname-a 2024-09-18T12:33:50.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:50.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:50.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.183+0000 7f90b659c640 1 -- 172.21.15.144:0/2897395956 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b8069a60 con 0x7f90b806e130 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b659c640 1 -- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806e130 msgr2=0x7f90b809da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b659c640 1 --2- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806e130 0x7f90b809da60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f90b8064d60 tx=0x7f90a80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbf9cee46b25c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 1 -- 172.21.15.144:0/2897395956 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8032440 con 0x7f90b806e130 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: dump: 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.196 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: _finish_auth 0 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 5 monclient: authenticate success, global_id 104496 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 1 -- 172.21.15.144:0/2897395956 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a80325a0 con 0x7f90b806e130 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 1 -- 172.21.15.144:0/2897395956 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a8032940 con 0x7f90b806e130 2024-09-18T12:33:50.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 4 set_mon_vals no callback set 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90b5d9b640 10 monclient: dump: 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:50.198 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 10 monclient: shutdown 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 -- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806e130 msgr2=0x7f90b809da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806e130 0x7f90b809da60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f90b8064d60 tx=0x7f90a80091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 -- 172.21.15.144:0/2897395956 shutdown_connections 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/2897395956 >> v2:172.21.15.144:3300/0 conn(0x7f90b806e500 0x7f90b809e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806a790 0x7f90b809dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/2897395956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806e130 0x7f90b809da60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 -- 172.21.15.144:0/2897395956 >> 172.21.15.144:0/2897395956 conn(0x7f90b8071340 msgr2=0x7f90b80a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.186+0000 7f90bc8d4640 1 -- 172.21.15.144:0/2897395956 shutdown_connections 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.187+0000 7f90bc8d4640 1 -- 172.21.15.144:0/2897395956 wait complete. 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.187+0000 7f90bc8d4640 10 monclient: build_initial_monmap 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.187+0000 7f90bc8d4640 10 monclient: monmap: 2024-09-18T12:33:50.199 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.187+0000 7f90bc8d4640 1 Processor -- start 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.187+0000 7f90bc8d4640 1 -- start start 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.187+0000 7f90bc8d4640 10 monclient: init 2024-09-18T12:33:50.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806a790 0x7f90b8141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): picked mon.noname-a con 0x7f90b806a790 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90b806e130 0x7f90b8133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): picked mon.noname-b con 0x7f90b806e130 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806e500 0x7f90b811e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): picked mon.noname-c con 0x7f90b806e500 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b806e970 con 0x7f90b806a790 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b8122290 con 0x7f90b806e130 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:50.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b8121fb0 con 0x7f90b806e500 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90bc8d4640 10 monclient(hunting): authenticate will time out at 1265.757245s 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90affff640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90b806e130 msgr2=0x7f90b8133e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90affff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90b806e130 0x7f90b8133e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806e500 0x7f90b811e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806a790 0x7f90b8141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806e500 0x7f90b811e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59790/0 (socket says 172.21.15.144:59790) 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): get_auth_request con 0x7f90b806a790 auth_method 0 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 1 -- 172.21.15.144:0/3324965630 learned_addr learned my addr 172.21.15.144:0/3324965630 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:50.202 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 10 monclient(hunting): get_auth_request con 0x7f90b806e500 auth_method 0 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient(hunting): handle_auth_done global_id 104499 payload 824 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 1 -- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806e500 msgr2=0x7f90b811e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806e500 0x7f90b811e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 1 -- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90b806e130 msgr2=0x7f90b8133e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:50.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90b806e130 0x7f90b8133e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 1 monclient: found mon.noname-a 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.188+0000 7f90b659c640 1 -- 172.21.15.144:0/3324965630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8009450 con 0x7f90b806a790 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90b659c640 1 -- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806a790 msgr2=0x7f90b8141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90b659c640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806a790 0x7f90b8141090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f90a8031000 tx=0x7f90a8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:50.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a80319c0 con 0x7f90b806a790 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: dump: 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:50.205 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a8031b60 con 0x7f90b806a790 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90bc8d4640 5 monclient: authenticate success, global_id 104499 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:50.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90bc8d4640 10 monclient: _renew_subs 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90bc8d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a803d440 con 0x7f90b806a790 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b8122570 con 0x7f90b806a790 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90ae7fc640 10 monclient: dump: 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:50.207 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90b559a640 4 set_mon_vals no callback set 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90bc8d4640 10 monclient: _renew_subs 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90bc8d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.191+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b81415d0 con 0x7f90b806a790 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90bc8d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90bc8d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90bc8d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b80008d0 con 0x7f90b806a790 2024-09-18T12:33:50.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f90a8039040 con 0x7f90b806a790 2024-09-18T12:33:50.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90ae7fc640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9098038440 0x7f909803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:50.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(235..235 src has 13..235) v4 ==== 13018+0+0 (secure 0 0 0) 0x7f90a806d0d0 con 0x7f90b806a790 2024-09-18T12:33:50.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90affff640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9098038440 0x7f909803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:50.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.192+0000 7f90affff640 10 monclient: get_auth_request con 0x7f9098038440 auth_method 0 2024-09-18T12:33:50.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.195+0000 7f90affff640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9098038440 0x7f909803a8f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f90a0009480 tx=0x7f90a00090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:50.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.196+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f90a809f030 con 0x7f90b806a790 2024-09-18T12:33:50.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.196+0000 7f90ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:50.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.196+0000 7f90ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:50.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.354+0000 7f90bc8d4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T12:33:50.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.354+0000 7f90bc8d4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T12:33:50.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.354+0000 7f90bc8d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:50.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:50.354+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f90b81417c0 con 0x7f90b806a790 2024-09-18T12:33:51.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.010+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f90a8037030 con 0x7f90b806a790 2024-09-18T12:33:51.011 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.010+0000 7f90ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T12:33:51.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.010+0000 7f90ae7fc640 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-18T12:33:51.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.107+0000 7f90bc8d4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T12:33:51.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.107+0000 7f90bc8d4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T12:33:51.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.107+0000 7f90bc8d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:51.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.107+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f90b81417c0 con 0x7f90b806a790 2024-09-18T12:33:51.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.108+0000 7f90ae7fc640 1 -- 172.21.15.144:0/3324965630 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f90a8037020 con 0x7f90b806a790 2024-09-18T12:33:51.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.108+0000 7f90ae7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T12:33:51.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.108+0000 7f90ae7fc640 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-18T12:33:51.109 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h10' does not appear in the crush map 2024-09-18T12:33:51.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.109+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9098038440 msgr2=0x7f909803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:51.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.109+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9098038440 0x7f909803a8f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f90a0009480 tx=0x7f90a00090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.109+0000 7f90bc8d4640 10 monclient: shutdown 2024-09-18T12:33:51.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.109+0000 7f90bc8d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:51.111 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.109+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806a790 msgr2=0x7f90b8141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.109+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806a790 0x7f90b8141090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f90a8031000 tx=0x7f90a8032000 comp rx=0 tx=0).stop 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 shutdown_connections 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9098038440 0x7f909803a8f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90b806e500 0x7f90b811e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90b806e130 0x7f90b8133e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 --2- 172.21.15.144:0/3324965630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b806a790 0x7f90b8141090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 >> 172.21.15.144:0/3324965630 conn(0x7f90b8071340 msgr2=0x7f90b809c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 shutdown_connections 2024-09-18T12:33:51.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.110+0000 7f90bc8d4640 1 -- 172.21.15.144:0/3324965630 wait complete. 2024-09-18T12:33:51.120 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:51.120 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h11 2024-09-18T12:33:51.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.193+0000 7fe1148af640 10 monclient: get_monmap_and_config 2024-09-18T12:33:51.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.193+0000 7fe1148af640 10 monclient: build_initial_monmap 2024-09-18T12:33:51.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.193+0000 7fe1148af640 10 monclient: monmap: 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:51.194472+0000 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:51.194472+0000 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:51.195 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:51.196 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:51.196 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:51.196 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:51.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 1 Processor -- start 2024-09-18T12:33:51.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 1 -- start start 2024-09-18T12:33:51.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient: init 2024-09-18T12:33:51.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:51.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe110131c00 0x7fe110131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient(hunting): picked mon.noname-c con 0x7fe110131c00 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1100a0b00 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 0x7fe110069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient(hunting): picked mon.noname-b con 0x7fe110132510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11009d230 con 0x7fe1100a0b00 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1100a08c0 con 0x7fe110131c00 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe110132910 con 0x7fe110132510 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:51.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe1148af640 10 monclient(hunting): authenticate will time out at 1266.763782s 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe10e59c640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fe110131c00 msgr2=0x7fe110131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.194+0000 7fe10e59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe110131c00 0x7fe110131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 0x7fe110069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 0x7fe110069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59804/0 (socket says 172.21.15.144:59804) 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10dd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe1100a0b00 0x7fe110068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:56374/0 (socket says 172.21.15.144:56374) 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 -- 172.21.15.144:0/285526048 learned_addr learned my addr 172.21.15.144:0/285526048 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): get_auth_request con 0x7fe110132510 auth_method 0 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:51.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient(hunting): handle_auth_done global_id 65294 payload 274 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 -- 172.21.15.144:0/285526048 >> v2:172.21.15.144:3300/0 conn(0x7fe110131c00 msgr2=0x7fe110131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 --2- 172.21.15.144:0/285526048 >> v2:172.21.15.144:3300/0 conn(0x7fe110131c00 0x7fe110131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 -- 172.21.15.144:0/285526048 >> v1:172.21.15.116:6789/0 conn(0x7fe1100a0b00 legacy=0x7fe110068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 monclient: found mon.noname-b 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:51.201 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.195+0000 7fe10ed9d640 1 -- 172.21.15.144:0/285526048 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1100a1b70 con 0x7fe110132510 2024-09-18T12:33:51.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10ed9d640 1 -- 172.21.15.144:0/285526048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 msgr2=0x7fe110069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:51.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10ed9d640 1 --2- 172.21.15.144:0/285526048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 0x7fe110069410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe108009f20 tx=0x7fe108009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc33e6f716348a59 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:51.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 1 -- 172.21.15.144:0/285526048 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe108007610 con 0x7fe110132510 2024-09-18T12:33:51.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:51.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: dump: 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: _finish_auth 0 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:51.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe1148af640 5 monclient: authenticate success, global_id 65294 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe1148af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 1 -- 172.21.15.144:0/285526048 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10802d050 con 0x7fe110132510 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 1 -- 172.21.15.144:0/285526048 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1080079a0 con 0x7fe110132510 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe10d59a640 10 monclient: dump: 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:51.209 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe1148af640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.198+0000 7fe1148af640 4 set_mon_vals no callback set 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 10 monclient: shutdown 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 -- 172.21.15.144:0/285526048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 msgr2=0x7fe110069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 --2- 172.21.15.144:0/285526048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 0x7fe110069410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe108009f20 tx=0x7fe108009840 comp rx=0 tx=0).stop 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 -- 172.21.15.144:0/285526048 shutdown_connections 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 --2- 172.21.15.144:0/285526048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe110132510 0x7fe110069410 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 --2- 172.21.15.144:0/285526048 >> v2:172.21.15.144:3300/0 conn(0x7fe110131c00 0x7fe110131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 -- 172.21.15.144:0/285526048 >> 172.21.15.144:0/285526048 conn(0x7fe110071370 msgr2=0x7fe1100a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 -- 172.21.15.144:0/285526048 shutdown_connections 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 -- 172.21.15.144:0/285526048 wait complete. 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 10 monclient: build_initial_monmap 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 10 monclient: monmap: 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:51.211 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.199+0000 7fe1148af640 1 Processor -- start 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 1 -- start start 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient: init 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 0x7fe11009aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): picked mon.noname-c con 0x7fe1100a0b00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe110131c00 0x7fe110131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): picked mon.noname-a con 0x7fe110131c00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110132510 0x7fe11011b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:51.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): picked mon.noname-b con 0x7fe110132510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1100a0fd0 con 0x7fe110131c00 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11009f3a0 con 0x7fe110132510 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11011e7a0 con 0x7fe1100a0b00 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe1148af640 10 monclient(hunting): authenticate will time out at 1266.769523s 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10ed9d640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110132510 msgr2=0x7fe11011b900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110132510 0x7fe11011b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 0x7fe11009aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe110131c00 0x7fe110131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:51.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 0x7fe11009aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59820/0 (socket says 172.21.15.144:59820) 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 -- 172.21.15.144:0/4258253656 learned_addr learned my addr 172.21.15.144:0/4258253656 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): get_auth_request con 0x7fe1100a0b00 auth_method 0 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 10 monclient(hunting): get_auth_request con 0x7fe110131c00 auth_method 0 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:51.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient(hunting): handle_auth_done global_id 65297 payload 824 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 -- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110132510 msgr2=0x7fe11011b900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110132510 0x7fe11011b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 -- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe110131c00 msgr2=0x7fe110131090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe110131c00 0x7fe110131090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 1 monclient: found mon.noname-c 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.200+0000 7fe10e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.201+0000 7fe10e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:51.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.201+0000 7fe10e59c640 1 -- 172.21.15.144:0/4258253656 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe108009ae0 con 0x7fe1100a0b00 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe10e59c640 1 -- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 msgr2=0x7fe11009aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe10e59c640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 0x7fe11009aed0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe10000b600 tx=0x7fe1000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe100015070 con 0x7fe1100a0b00 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: dump: 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:51.216 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe100003d90 con 0x7fe1100a0b00 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe100006380 con 0x7fe1100a0b00 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe1148af640 5 monclient: authenticate success, global_id 65297 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe1148af640 10 monclient: _renew_subs 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe1148af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11011e1b0 con 0x7fe1100a0b00 2024-09-18T12:33:51.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe10cd99640 4 set_mon_vals no callback set 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe0fe7fc640 10 monclient: dump: 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:51.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe1148af640 10 monclient: _renew_subs 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe1148af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.203+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11011d2c0 con 0x7fe1100a0b00 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe1148af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe1148af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe1148af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1100008d0 con 0x7fe1100a0b00 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe100010440 con 0x7fe1100a0b00 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe0fe7fc640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0f0038470 0x7fe0f003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(236..236 src has 13..236) v4 ==== 12805+0+0 (secure 0 0 0) 0x7fe10004d3e0 con 0x7fe1100a0b00 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe10dd9b640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0f0038470 0x7fe0f003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:51.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.204+0000 7fe10dd9b640 10 monclient: get_auth_request con 0x7fe0f0038470 auth_method 0 2024-09-18T12:33:51.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.207+0000 7fe10dd9b640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0f0038470 0x7fe0f003a920 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe104007310 tx=0x7fe104007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:51.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.209+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe100018d60 con 0x7fe1100a0b00 2024-09-18T12:33:51.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.209+0000 7fe0fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:51.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.209+0000 7fe0fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:51.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.365+0000 7fe1148af640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T12:33:51.366 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.365+0000 7fe1148af640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T12:33:51.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.365+0000 7fe1148af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:51.367 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:51.365+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7fe1100a1890 con 0x7fe1100a0b00 2024-09-18T12:33:52.012 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.011+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe100018780 con 0x7fe1100a0b00 2024-09-18T12:33:52.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.011+0000 7fe0fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T12:33:52.013 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.011+0000 7fe0fe7fc640 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-18T12:33:52.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.106+0000 7fe1148af640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T12:33:52.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.107+0000 7fe1148af640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T12:33:52.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.107+0000 7fe1148af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:52.108 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.107+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7fe11009dad0 con 0x7fe1100a0b00 2024-09-18T12:33:52.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.109+0000 7fe0fe7fc640 1 -- 172.21.15.144:0/4258253656 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe1000067b0 con 0x7fe1100a0b00 2024-09-18T12:33:52.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.109+0000 7fe0fe7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T12:33:52.109 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.109+0000 7fe0fe7fc640 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-18T12:33:52.109 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h11' does not appear in the crush map 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0f0038470 msgr2=0x7fe0f003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0f0038470 0x7fe0f003a920 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe104007310 tx=0x7fe104007e40 comp rx=0 tx=0).stop 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 10 monclient: shutdown 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 msgr2=0x7fe11009aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 0x7fe11009aed0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe10000b600 tx=0x7fe1000091a0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 shutdown_connections 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe0f0038470 0x7fe0f003a920 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110132510 0x7fe11011b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.112 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe110131c00 0x7fe110131090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 --2- 172.21.15.144:0/4258253656 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe1100a0b00 0x7fe11009aed0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 >> 172.21.15.144:0/4258253656 conn(0x7fe110071370 msgr2=0x7fe11009d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:52.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 shutdown_connections 2024-09-18T12:33:52.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.110+0000 7fe1148af640 1 -- 172.21.15.144:0/4258253656 wait complete. 2024-09-18T12:33:52.119 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:52.119 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h12 2024-09-18T12:33:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.192+0000 7f624459d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.192+0000 7f624459d640 10 monclient: build_initial_monmap 2024-09-18T12:33:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.192+0000 7f624459d640 10 monclient: monmap: 2024-09-18T12:33:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:52.193158+0000 2024-09-18T12:33:52.193 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:52.193158+0000 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:52.194 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.192+0000 7f624459d640 1 Processor -- start 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.192+0000 7f624459d640 1 -- start start 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.192+0000 7f624459d640 10 monclient: init 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f623c09ec50 0x7f623c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient(hunting): picked mon.noname-c con 0x7f623c09ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 0x7f623c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient(hunting): picked mon.noname-b con 0x7f623c1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:52.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f623c09f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623c067420 con 0x7f623c09f020 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623c0a0480 con 0x7f623c09ec50 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623c0a0200 con 0x7f623c1321a0 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f624459d640 10 monclient(hunting): authenticate will time out at 1267.762557s 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f6242339640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f623c09ec50 msgr2=0x7f623c131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f6242339640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f623c09ec50 0x7f623c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.193+0000 7f6241b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 0x7f623c068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 0x7f623c068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59822/0 (socket says 172.21.15.144:59822) 2024-09-18T12:33:52.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 1 -- 172.21.15.144:0/986572480 learned_addr learned my addr 172.21.15.144:0/986572480 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:52.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6242b3a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f623c09f020 0x7f623c069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:56378/0 (socket says 172.21.15.144:56378) 2024-09-18T12:33:52.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): get_auth_request con 0x7f623c1321a0 auth_method 0 2024-09-18T12:33:52.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:52.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:52.199 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient(hunting): handle_auth_done global_id 65300 payload 274 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 1 -- 172.21.15.144:0/986572480 >> v2:172.21.15.144:3300/0 conn(0x7f623c09ec50 msgr2=0x7f623c131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 1 --2- 172.21.15.144:0/986572480 >> v2:172.21.15.144:3300/0 conn(0x7f623c09ec50 0x7f623c131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 1 -- 172.21.15.144:0/986572480 >> v1:172.21.15.116:6789/0 conn(0x7f623c09f020 legacy=0x7f623c069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient: _finish_hunting 0 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 1 monclient: found mon.noname-b 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:52.200 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.194+0000 7f6241b38640 1 -- 172.21.15.144:0/986572480 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c09fd80 con 0x7f623c1321a0 2024-09-18T12:33:52.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241b38640 1 -- 172.21.15.144:0/986572480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 msgr2=0x7f623c068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:52.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241b38640 1 --2- 172.21.15.144:0/986572480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 0x7f623c068d00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f622c009910 tx=0x7f622c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daefa8d4ecaf3da2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:52.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 1 -- 172.21.15.144:0/986572480 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c02ae20 con 0x7f623c1321a0 2024-09-18T12:33:52.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:52.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:52.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: dump: 2024-09-18T12:33:52.207 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:52.208 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:52.208 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.208 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.208 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:52.208 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:52.208 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:52.209 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:52.209 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:52.209 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:52.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: _finish_auth 0 2024-09-18T12:33:52.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:52.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 1 -- 172.21.15.144:0/986572480 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c02f4b0 con 0x7f623c1321a0 2024-09-18T12:33:52.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 5 monclient: authenticate success, global_id 65300 2024-09-18T12:33:52.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:52.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:52.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 1 -- 172.21.15.144:0/986572480 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c02f850 con 0x7f623c1321a0 2024-09-18T12:33:52.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:52.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 4 set_mon_vals no callback set 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f6241337640 10 monclient: dump: 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:52.211 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:52.212 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:52.212 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:52.212 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:52.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 10 monclient: shutdown 2024-09-18T12:33:52.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:52.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 1 -- 172.21.15.144:0/986572480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 msgr2=0x7f623c068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:52.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.197+0000 7f624459d640 1 --2- 172.21.15.144:0/986572480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 0x7f623c068d00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f622c009910 tx=0x7f622c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 -- 172.21.15.144:0/986572480 shutdown_connections 2024-09-18T12:33:52.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 --2- 172.21.15.144:0/986572480 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c1321a0 0x7f623c068d00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 --2- 172.21.15.144:0/986572480 >> v2:172.21.15.144:3300/0 conn(0x7f623c09ec50 0x7f623c131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 -- 172.21.15.144:0/986572480 >> 172.21.15.144:0/986572480 conn(0x7f623c0a5cf0 msgr2=0x7f623c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:52.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 -- 172.21.15.144:0/986572480 shutdown_connections 2024-09-18T12:33:52.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 -- 172.21.15.144:0/986572480 wait complete. 2024-09-18T12:33:52.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 10 monclient: build_initial_monmap 2024-09-18T12:33:52.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 10 monclient: monmap: 2024-09-18T12:33:52.214 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 Processor -- start 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 1 -- start start 2024-09-18T12:33:52.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.198+0000 7f624459d640 10 monclient: init 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c09ec50 0x7f623c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): picked mon.noname-a con 0x7f623c09ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 0x7f623c130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): picked mon.noname-c con 0x7f623c09f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f623c1321a0 0x7f623c131020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): picked mon.noname-b con 0x7f623c1321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c09c070 con 0x7f623c09ec50 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c134b80 con 0x7f623c1321a0 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:52.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623c11dc00 con 0x7f623c09f020 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f624459d640 10 monclient(hunting): authenticate will time out at 1267.768323s 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242b3a640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f623c1321a0 msgr2=0x7f623c131020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242b3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f623c1321a0 0x7f623c131020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 0x7f623c130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c09ec50 0x7f623c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 0x7f623c130ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59830/0 (socket says 172.21.15.144:59830) 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 -- 172.21.15.144:0/2904121621 learned_addr learned my addr 172.21.15.144:0/2904121621 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): get_auth_request con 0x7f623c09f020 auth_method 0 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:52.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 10 monclient(hunting): get_auth_request con 0x7f623c09ec50 auth_method 0 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient(hunting): handle_auth_done global_id 65303 payload 824 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 -- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f623c1321a0 msgr2=0x7f623c131020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f623c1321a0 0x7f623c131020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 -- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c09ec50 msgr2=0x7f623c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c09ec50 0x7f623c09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient: _finish_hunting 0 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 monclient: found mon.noname-c 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6241b38640 1 -- 172.21.15.144:0/2904121621 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c009450 con 0x7f623c09f020 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.199+0000 7f6242339640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c09ec50 0x7f623c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.202+0000 7f6241b38640 1 -- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 msgr2=0x7f623c130ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.202+0000 7f6241b38640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 0x7f623c130ae0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f622c009910 tx=0x7f622c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.202+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c032a30 con 0x7f623c09f020 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: dump: 2024-09-18T12:33:52.219 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c032bb0 con 0x7f623c09f020 2024-09-18T12:33:52.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c03b400 con 0x7f623c09f020 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f62327fc640 10 monclient: dump: 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:52.221 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f624459d640 5 monclient: authenticate success, global_id 65303 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f624459d640 10 monclient: _renew_subs 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f624459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c11dee0 con 0x7f623c09f020 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f6240b36640 4 set_mon_vals no callback set 2024-09-18T12:33:52.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f624459d640 10 monclient: _renew_subs 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f624459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.204+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c141570 con 0x7f623c09f020 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f624459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f624459d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f624459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c0008d0 con 0x7f623c09f020 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f622c039040 con 0x7f623c09f020 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f62327fc640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6220038490 0x7f622003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(237..237 src has 13..237) v4 ==== 12592+0+0 (secure 0 0 0) 0x7f622c06d7a0 con 0x7f623c09f020 2024-09-18T12:33:52.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.205+0000 7f6242339640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6220038490 0x7f622003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:52.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.206+0000 7f6242339640 10 monclient: get_auth_request con 0x7f6220038490 auth_method 0 2024-09-18T12:33:52.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.208+0000 7f6242339640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6220038490 0x7f622003a940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6238003940 tx=0x7f62380090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:52.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.210+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f622c03ab40 con 0x7f623c09f020 2024-09-18T12:33:52.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.210+0000 7f62327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:52.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.210+0000 7f62327fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:52.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.372+0000 7f624459d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T12:33:52.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.372+0000 7f624459d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T12:33:52.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.372+0000 7f624459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:52.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:52.372+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f623c0a13e0 con 0x7f623c09f020 2024-09-18T12:33:53.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.025+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f622c03f070 con 0x7f623c09f020 2024-09-18T12:33:53.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.025+0000 7f62327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T12:33:53.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.025+0000 7f62327fc640 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-18T12:33:53.127 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.126+0000 7f624459d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T12:33:53.127 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.126+0000 7f624459d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T12:33:53.127 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.126+0000 7f624459d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:53.127 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.126+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f623c0a13e0 con 0x7f623c09f020 2024-09-18T12:33:53.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.127+0000 7f62327fc640 1 -- 172.21.15.144:0/2904121621 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f622c037030 con 0x7f623c09f020 2024-09-18T12:33:53.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.127+0000 7f62327fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T12:33:53.129 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.127+0000 7f62327fc640 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-18T12:33:53.129 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h12' does not appear in the crush map 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6220038490 msgr2=0x7f622003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6220038490 0x7f622003a940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6238003940 tx=0x7f62380090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 10 monclient: shutdown 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 msgr2=0x7f623c130ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 0x7f623c130ae0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f622c009910 tx=0x7f622c032000 comp rx=0 tx=0).stop 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 shutdown_connections 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6220038490 0x7f622003a940 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f623c1321a0 0x7f623c131020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f623c09f020 0x7f623c130ae0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.131 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 --2- 172.21.15.144:0/2904121621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f623c09ec50 0x7f623c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 >> 172.21.15.144:0/2904121621 conn(0x7f623c0a5cf0 msgr2=0x7f623c134d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:53.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 shutdown_connections 2024-09-18T12:33:53.132 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.129+0000 7f624459d640 1 -- 172.21.15.144:0/2904121621 wait complete. 2024-09-18T12:33:53.139 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:53.139 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h13 2024-09-18T12:33:53.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.212+0000 7f8d9d71d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.212+0000 7f8d9d71d640 10 monclient: build_initial_monmap 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.212+0000 7f8d9d71d640 10 monclient: monmap: 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:53.213788+0000 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:53.213788+0000 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:53.214 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:53.215 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:53.215 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:53.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 1 Processor -- start 2024-09-18T12:33:53.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 1 -- start start 2024-09-18T12:33:53.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 10 monclient: init 2024-09-18T12:33:53.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:53.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:33:53.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8d98131c00 0x7f8d98131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 10 monclient(hunting): picked mon.noname-c con 0x7f8d98131c00 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d980a0b00 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.213+0000 7f8d9d71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d98132510 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d9d71d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980a1b70 con 0x7f8d980a0b00 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d9d71d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980a08c0 con 0x7f8d98132510 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d9d71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d9d71d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d98132910 con 0x7f8d98131c00 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d9d71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d96ffd640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f8d98131c00 msgr2=0x7f8d98131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d9d71d640 10 monclient(hunting): authenticate will time out at 1268.783045s 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d96ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8d98131c00 0x7f8d98131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:53.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d977fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f8d98132510 legacy=0x7f8d98069410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:53.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d967fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8d980a0b00 0x7f8d98068d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:56392/0 (socket says 172.21.15.144:56392) 2024-09-18T12:33:53.218 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.214+0000 7f8d967fc640 1 -- 172.21.15.144:0/3088799768 learned_addr learned my addr 172.21.15.144:0/3088799768 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:53.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.216+0000 7f8d967fc640 1 -- 172.21.15.144:0/3088799768 >> v1:172.21.15.116:6789/0 conn(0x7f8d980a0b00 legacy=0x7f8d98068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:53.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.216+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686487642 0 0) 0x7f8d980a1b70 con 0x7f8d980a0b00 2024-09-18T12:33:53.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.216+0000 7f8d95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:53.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.216+0000 7f8d95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:53.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.216+0000 7f8d95ffb640 10 monclient(hunting): my global_id is 104508 2024-09-18T12:33:53.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.216+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d800034e0 con 0x7f8d980a0b00 2024-09-18T12:33:53.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845779878 0 0) 0x7f8d800034e0 con 0x7f8d980a0b00 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 >> v2:172.21.15.144:3300/0 conn(0x7f8d98131c00 msgr2=0x7f8d98131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 --2- 172.21.15.144:0/3088799768 >> v2:172.21.15.144:3300/0 conn(0x7f8d98131c00 0x7f8d98131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 >> v1:172.21.15.144:6789/0 conn(0x7f8d98132510 legacy=0x7f8d98069410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980a0200 con 0x7f8d980a0b00 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: _finish_auth 0 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f8d8c001860 con 0x7f8d980a0b00 2024-09-18T12:33:53.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d9d71d640 5 monclient: authenticate success, global_id 104508 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d9d71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: dump: 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:53.225 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 >> v1:172.21.15.116:6789/0 conn(0x7f8d980a0b00 legacy=0x7f8d98068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d800039d0 0x7f8d80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): picked mon.b con 0x7f8d800039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80004780 0x7f8d80004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): picked mon.a con 0x7f8d80004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d80005330 0x7f8d8000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): picked mon.c con 0x7f8d80005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d96ffd640 1 -- 172.21.15.144:0/3088799768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d800039d0 msgr2=0x7f8d80003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:53.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98132910 con 0x7f8d80004780 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8000c290 con 0x7f8d800039d0 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d96ffd640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d800039d0 0x7f8d80003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8000c450 con 0x7f8d80005330 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d95ffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d9d71d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d80005330 0x7f8d8000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80004780 0x7f8d80004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): get_auth_request con 0x7f8d80005330 auth_method 0 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:53.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 10 monclient(hunting): get_auth_request con 0x7f8d80004780 auth_method 0 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:53.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient(hunting): handle_auth_done global_id 104508 payload 293 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 1 -- 172.21.15.144:0/3088799768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d800039d0 msgr2=0x7f8d80003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d800039d0 0x7f8d80003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 1 -- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80004780 msgr2=0x7f8d80004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80004780 0x7f8d80004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 1 monclient: found mon.c 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 1 -- 172.21.15.144:0/3088799768 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d800034e0 con 0x7f8d80005330 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 10 monclient: _finish_auth 0 2024-09-18T12:33:53.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.217+0000 7f8d967fc640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80004780 0x7f8d80004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.218+0000 7f8d977fe640 1 -- 172.21.15.144:0/3088799768 reap_dead start 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.220+0000 7f8d977fe640 1 -- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d80005330 msgr2=0x7f8d8000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.220+0000 7f8d977fe640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d80005330 0x7f8d8000b9d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009f90 tx=0x7f8d900095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a4295aa29d8b19 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.220+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90032430 con 0x7f8d80005330 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.220+0000 7f8d95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.220+0000 7f8d95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.220+0000 7f8d95ffb640 10 monclient: dump: 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.230 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9002d070 con 0x7f8d80005330 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d95ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 4 set_mon_vals no callback set 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d95ffb640 1 -- 172.21.15.144:0/3088799768 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90032820 con 0x7f8d80005330 2024-09-18T12:33:53.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d95ffb640 10 monclient: dump: 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:53.232 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 10 monclient: shutdown 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 1 -- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d80005330 msgr2=0x7f8d8000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d80005330 0x7f8d8000b9d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009f90 tx=0x7f8d900095d0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 1 -- 172.21.15.144:0/3088799768 shutdown_connections 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/3088799768 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d80005330 0x7f8d8000b9d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 1 -- 172.21.15.144:0/3088799768 >> 172.21.15.144:0/3088799768 conn(0x7f8d98071370 msgr2=0x7f8d980a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 1 -- 172.21.15.144:0/3088799768 shutdown_connections 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.221+0000 7f8d9d71d640 1 -- 172.21.15.144:0/3088799768 wait complete. 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 10 monclient: build_initial_monmap 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 10 monclient: monmap: 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:53.233 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 1 Processor -- start 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 1 -- start start 2024-09-18T12:33:53.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 10 monclient: init 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 0x7f8d9809ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 10 monclient(hunting): picked mon.noname-a con 0x7f8d80005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.222+0000 7f8d9d71d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9811ce40 0x7f8d98130c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 10 monclient(hunting): picked mon.noname-b con 0x7f8d9811ce40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d981311c0 0x7f8d9811b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 10 monclient(hunting): picked mon.noname-c con 0x7f8d981311c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:53.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9811e360 con 0x7f8d80005330 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9809f3e0 con 0x7f8d9811ce40 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9811dda0 con 0x7f8d981311c0 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d9d71d640 10 monclient(hunting): authenticate will time out at 1268.792043s 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d967fc640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9811ce40 msgr2=0x7f8d98130c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:53.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d967fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9811ce40 0x7f8d98130c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 0x7f8d9809ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 0x7f8d9809ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:34498/0 (socket says 172.21.15.144:34498) 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 -- 172.21.15.144:0/777285895 learned_addr learned my addr 172.21.15.144:0/777285895 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): get_auth_request con 0x7f8d80005330 auth_method 0 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d981311c0 0x7f8d9811b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 10 monclient(hunting): get_auth_request con 0x7f8d981311c0 auth_method 0 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:53.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient(hunting): handle_auth_done global_id 104511 payload 824 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 -- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d981311c0 msgr2=0x7f8d9811b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d981311c0 0x7f8d9811b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 -- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9811ce40 msgr2=0x7f8d98130c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:53.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9811ce40 0x7f8d98130c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 monclient: found mon.noname-a 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d96ffd640 1 -- 172.21.15.144:0/777285895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9811e080 con 0x7f8d80005330 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.223+0000 7f8d977fe640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d981311c0 0x7f8d9811b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.225+0000 7f8d96ffd640 1 -- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 msgr2=0x7f8d9809ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.225+0000 7f8d96ffd640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 0x7f8d9809ab40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800b810 tx=0x7f8d880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88003de0 con 0x7f8d80005330 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: dump: 2024-09-18T12:33:53.243 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: _finish_auth 0 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d9d71d640 5 monclient: authenticate success, global_id 104511 2024-09-18T12:33:53.244 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d88006150 con 0x7f8d80005330 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d9d71d640 10 monclient: _renew_subs 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d9d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d98141c70 con 0x7f8d80005330 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88006300 con 0x7f8d80005330 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d7effd640 10 monclient: dump: 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:53.245 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d9d71d640 10 monclient: _renew_subs 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d9d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d98141530 con 0x7f8d80005330 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.226+0000 7f8d957fa640 4 set_mon_vals no callback set 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.227+0000 7f8d9d71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.227+0000 7f8d9d71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.227+0000 7f8d9d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.227+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d980008d0 con 0x7f8d80005330 2024-09-18T12:33:53.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.227+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8d88012020 con 0x7f8d80005330 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.228+0000 7f8d7effd640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8d84038490 0x7f8d8403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.228+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(238..238 src has 13..238) v4 ==== 12379+0+0 (secure 0 0 0) 0x7f8d8804c830 con 0x7f8d80005330 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.228+0000 7f8d967fc640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8d84038490 0x7f8d8403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.228+0000 7f8d967fc640 10 monclient: get_auth_request con 0x7f8d84038490 auth_method 0 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.230+0000 7f8d967fc640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8d84038490 0x7f8d8403a940 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c007fd0 tx=0x7f8d980a00e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.232+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f8d8801c2b0 con 0x7f8d80005330 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.232+0000 7f8d7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:53.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.232+0000 7f8d7effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:53.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.392+0000 7f8d9d71d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T12:33:53.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.392+0000 7f8d9d71d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T12:33:53.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.392+0000 7f8d9d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:53.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:53.392+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f8d980a1890 con 0x7f8d80005330 2024-09-18T12:33:54.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.026+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8d88014c90 con 0x7f8d80005330 2024-09-18T12:33:54.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.026+0000 7f8d7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T12:33:54.027 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.026+0000 7f8d7effd640 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-18T12:33:54.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.123+0000 7f8d9d71d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T12:33:54.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.123+0000 7f8d9d71d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T12:33:54.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.123+0000 7f8d9d71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:33:54.124 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.123+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f8d98072640 con 0x7f8d80005330 2024-09-18T12:33:54.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.124+0000 7f8d7effd640 1 -- 172.21.15.144:0/777285895 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8d88014e50 con 0x7f8d80005330 2024-09-18T12:33:54.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.124+0000 7f8d7effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T12:33:54.125 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.124+0000 7f8d7effd640 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-18T12:33:54.125 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h13' does not appear in the crush map 2024-09-18T12:33:54.127 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8d84038490 msgr2=0x7f8d8403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:54.127 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8d84038490 0x7f8d8403a940 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c007fd0 tx=0x7f8d980a00e0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 10 monclient: shutdown 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 msgr2=0x7f8d9809ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 0x7f8d9809ab40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800b810 tx=0x7f8d880091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 shutdown_connections 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8d84038490 0x7f8d8403a940 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8d981311c0 0x7f8d9811b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d9811ce40 0x7f8d98130c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 --2- 172.21.15.144:0/777285895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d80005330 0x7f8d9809ab40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 >> 172.21.15.144:0/777285895 conn(0x7f8d98071370 msgr2=0x7f8d98134ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 shutdown_connections 2024-09-18T12:33:54.128 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.126+0000 7f8d9d71d640 1 -- 172.21.15.144:0/777285895 wait complete. 2024-09-18T12:33:54.136 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:54.136 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h14 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.208+0000 7f725cb9d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.208+0000 7f725cb9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.208+0000 7f725cb9d640 10 monclient: monmap: 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:54.209193+0000 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:54.209193+0000 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:54.209 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:54.210 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:54.210 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:54.210 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:54.212 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:54.212 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:54.212 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.208+0000 7f725cb9d640 1 Processor -- start 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.208+0000 7f725cb9d640 1 -- start start 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.208+0000 7f725cb9d640 10 monclient: init 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f725809ec50 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f725809f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7258069d10 0x7f725813a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7258069d10 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7258067420 con 0x7f725809ec50 2024-09-18T12:33:54.219 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580a0480 con 0x7f725809f020 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725806a170 con 0x7f7258069d10 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f725cb9d640 10 monclient(hunting): authenticate will time out at 1269.778698s 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f7256d9d640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f7258069d10 msgr2=0x7f725813a490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f7255d9b640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f725809f020 legacy=0x7f7258136d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.209+0000 7f7256d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7258069d10 0x7f725813a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.210+0000 7f725659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f725809ec50 0x7f7258131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:56398/0 (socket says 172.21.15.144:56398) 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.210+0000 7f725659c640 1 -- 172.21.15.144:0/2117551142 learned_addr learned my addr 172.21.15.144:0/2117551142 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725659c640 1 -- 172.21.15.144:0/2117551142 >> v1:172.21.15.116:6789/0 conn(0x7f725809ec50 legacy=0x7f7258131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894687390 0 0) 0x7f7258067420 con 0x7f725809ec50 2024-09-18T12:33:54.220 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient(hunting): my global_id is 104514 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72400034e0 con 0x7f725809ec50 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257829486 0 0) 0x7f72400034e0 con 0x7f725809ec50 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 >> v2:172.21.15.144:3300/0 conn(0x7f7258069d10 msgr2=0x7f725813a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 --2- 172.21.15.144:0/2117551142 >> v2:172.21.15.144:3300/0 conn(0x7f7258069d10 0x7f725813a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 >> v1:172.21.15.144:6789/0 conn(0x7f725809f020 legacy=0x7f7258136d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: _finish_hunting 0 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580a0200 con 0x7f725809ec50 2024-09-18T12:33:54.221 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: _finish_auth 0 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f72480017a0 con 0x7f725809ec50 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725cb9d640 5 monclient: authenticate success, global_id 104514 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725cb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: dump: 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:54.222 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 >> v1:172.21.15.116:6789/0 conn(0x7f725809ec50 legacy=0x7f7258131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240003aa0 0x7f7240003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient(hunting): picked mon.c con 0x7f7240003aa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240004780 0x7f7240004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 10 monclient(hunting): picked mon.b con 0x7f7240004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.212+0000 7f725559a640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240005380 0x7f724000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.223 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 10 monclient(hunting): picked mon.a con 0x7f7240005380 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724000c2e0 con 0x7f7240005380 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724000c450 con 0x7f7240004780 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7255d9b640 1 -- 172.21.15.144:0/2117551142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240004780 msgr2=0x7f7240004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724000c610 con 0x7f7240003aa0 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725559a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7255d9b640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240004780 0x7f7240004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725cb9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:54.224 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240003aa0 0x7f7240003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240005380 0x7f724000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): get_auth_request con 0x7f7240003aa0 auth_method 0 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 10 monclient(hunting): get_auth_request con 0x7f7240005380 auth_method 0 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:54.225 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient(hunting): handle_auth_done global_id 104514 payload 293 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 1 -- 172.21.15.144:0/2117551142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240004780 msgr2=0x7f7240004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240004780 0x7f7240004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 1 -- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240005380 msgr2=0x7f724000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240005380 0x7f724000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 1 monclient: found mon.c 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 1 -- 172.21.15.144:0/2117551142 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72400034e0 con 0x7f7240003aa0 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 10 monclient: _finish_auth 0 2024-09-18T12:33:54.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f725659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240005380 0x7f724000ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.213+0000 7f7256d9d640 1 -- 172.21.15.144:0/2117551142 reap_dead start 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725659c640 1 -- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240003aa0 msgr2=0x7f7240003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725659c640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240003aa0 0x7f7240003e90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72480014d0 tx=0x7f7248007a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3fa3da413e38335 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7248005890 con 0x7f7240003aa0 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 10 monclient: dump: 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.227 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7248005a10 con 0x7f7240003aa0 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 1 -- 172.21.15.144:0/2117551142 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7248005dd0 con 0x7f7240003aa0 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725559a640 10 monclient: dump: 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:54.228 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725cb9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.216+0000 7f725cb9d640 4 set_mon_vals no callback set 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 10 monclient: shutdown 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 1 -- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240003aa0 msgr2=0x7f7240003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:54.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240003aa0 0x7f7240003e90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72480014d0 tx=0x7f7248007a90 comp rx=0 tx=0).stop 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 1 -- 172.21.15.144:0/2117551142 shutdown_connections 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 1 --2- 172.21.15.144:0/2117551142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7240003aa0 0x7f7240003e90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 1 -- 172.21.15.144:0/2117551142 >> 172.21.15.144:0/2117551142 conn(0x7f72580a5cf0 msgr2=0x7f72580a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 1 -- 172.21.15.144:0/2117551142 shutdown_connections 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 1 -- 172.21.15.144:0/2117551142 wait complete. 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 10 monclient: build_initial_monmap 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.217+0000 7f725cb9d640 10 monclient: monmap: 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:54.230 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:54.231 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:54.231 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:54.231 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:54.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 Processor -- start 2024-09-18T12:33:54.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 -- start start 2024-09-18T12:33:54.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient: init 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240003aa0 0x7f725809a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7240003aa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72581309c0 0x7f7258130d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f72581309c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 0x7f725811bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.235 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f725811b760 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725809c020 con 0x7f72581309c0 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725809dfa0 con 0x7f7240003aa0 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725811dae0 con 0x7f725811b760 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725659c640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240003aa0 msgr2=0x7f725809a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725cb9d640 10 monclient(hunting): authenticate will time out at 1269.787529s 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f725659c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240003aa0 0x7f725809a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7256d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 0x7f725811bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7255d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72581309c0 0x7f7258130d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7255d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72581309c0 0x7f7258130d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:34512/0 (socket says 172.21.15.144:34512) 2024-09-18T12:33:54.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7256d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 0x7f725811bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59862/0 (socket says 172.21.15.144:59862) 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7255d9b640 1 -- 172.21.15.144:0/1248312329 learned_addr learned my addr 172.21.15.144:0/1248312329 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7255d9b640 10 monclient(hunting): get_auth_request con 0x7f72581309c0 auth_method 0 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7255d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7255d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7255d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7256d9d640 10 monclient(hunting): get_auth_request con 0x7f725811b760 auth_method 0 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7256d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7256d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.218+0000 7f7256d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7255d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:54.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7255d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7255d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 10 monclient(hunting): handle_auth_done global_id 65309 payload 824 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 1 -- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240003aa0 msgr2=0x7f725809a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240003aa0 0x7f725809a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 1 -- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72581309c0 msgr2=0x7f7258130d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72581309c0 0x7f7258130d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 1 monclient: found mon.noname-c 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.219+0000 7f7256d9d640 1 -- 172.21.15.144:0/1248312329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725811ddc0 con 0x7f725811b760 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f7256d9d640 1 -- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 msgr2=0x7f725811bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f7256d9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 0x7f725811bb90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f72580a2f00 tx=0x7f72500095d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f7256d9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 0x7f725811bb90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f72580a2f00 tx=0x7f72500095d0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f7250011070 mon_map magic: 0 v1 2024-09-18T12:33:54.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250011070 con 0x7f725811b760 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f72467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f72467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f72467fc640 10 monclient: dump: 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f72467fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:54.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f72467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725000b890 con 0x7f725811b760 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f725cb9d640 5 monclient: authenticate success, global_id 65309 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.221+0000 7f725cb9d640 10 monclient: _renew_subs 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725809b4a0 con 0x7f725811b760 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f72467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 10 monclient: _renew_subs 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f7254d99640 4 set_mon_vals no callback set 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725811cbf0 con 0x7f725811b760 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:54.240 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72580008d0 con 0x7f725811b760 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725000bc70 con 0x7f725811b760 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f72467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f72467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f72467fc640 10 monclient: dump: 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:54.241 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:54.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.222+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7250016420 con 0x7f725811b760 2024-09-18T12:33:54.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.223+0000 7f72467fc640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7238038440 0x7f723803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:54.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.223+0000 7f725659c640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7238038440 0x7f723803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:54.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.223+0000 7f725659c640 10 monclient: get_auth_request con 0x7f7238038440 auth_method 0 2024-09-18T12:33:54.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.225+0000 7f725659c640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7238038440 0x7f723803a8f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7248001980 tx=0x7f7248034000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:54.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:54.227+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(239..239 src has 13..239) v4 ==== 12166+0+0 (secure 0 0 0) 0x7f7250006e40 con 0x7f725811b760 2024-09-18T12:33:55.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:55.050+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f725001f8a0 con 0x7f725811b760 2024-09-18T12:33:55.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:55.050+0000 7f72467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:55.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:55.050+0000 7f72467fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:55.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:55.211+0000 7f725cb9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T12:33:55.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:55.211+0000 7f725cb9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T12:33:55.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:55.211+0000 7f725cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:55.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:55.211+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f725811dc20 con 0x7f725811b760 2024-09-18T12:33:56.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.040+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v240) v1 ==== 120+0+0 (secure 0 0 0) 0x7f725811dc20 con 0x7f725811b760 2024-09-18T12:33:56.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.041+0000 7f72467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T12:33:56.042 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.041+0000 7f72467fc640 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-18T12:33:56.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.143+0000 7f725cb9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T12:33:56.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.143+0000 7f725cb9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T12:33:56.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.143+0000 7f725cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:56.145 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.143+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f725811dc20 con 0x7f725811b760 2024-09-18T12:33:56.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.145+0000 7f72467fc640 1 -- 172.21.15.144:0/1248312329 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v240) v1 ==== 120+0+0 (secure 0 0 0) 0x7f725811dc20 con 0x7f725811b760 2024-09-18T12:33:56.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.145+0000 7f72467fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T12:33:56.146 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.145+0000 7f72467fc640 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-18T12:33:56.146 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h14' does not appear in the crush map 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7238038440 msgr2=0x7f723803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7238038440 0x7f723803a8f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7248001980 tx=0x7f7248034000 comp rx=0 tx=0).stop 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 10 monclient: shutdown 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 msgr2=0x7f725811bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 0x7f725811bb90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f72580a2f00 tx=0x7f72500095d0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 shutdown_connections 2024-09-18T12:33:56.149 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7238038440 0x7f723803a8f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f725811b760 0x7f725811bb90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72581309c0 0x7f7258130d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 --2- 172.21.15.144:0/1248312329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7240003aa0 0x7f725809a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.147+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 >> 172.21.15.144:0/1248312329 conn(0x7f72580a5cf0 msgr2=0x7f7258131c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:56.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.148+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 shutdown_connections 2024-09-18T12:33:56.150 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.148+0000 7f725cb9d640 1 -- 172.21.15.144:0/1248312329 wait complete. 2024-09-18T12:33:56.158 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:56.158 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h15 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.231+0000 7fe63c39d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.231+0000 7fe63c39d640 10 monclient: build_initial_monmap 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.231+0000 7fe63c39d640 10 monclient: monmap: 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:56.232425+0000 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:56.232425+0000 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:56.233 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:56.234 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:56.234 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:56.234 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:56.234 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:56.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 1 Processor -- start 2024-09-18T12:33:56.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 1 -- start start 2024-09-18T12:33:56.234 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient: init 2024-09-18T12:33:56.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:56.236 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe634131c90 addr v1:172.21.15.144:6789/0 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 0x7fe634069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient(hunting): picked mon.noname-b con 0x7fe63409d210 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe6340a0ab0 0x7fe634138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe6340a0ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6340a1800 con 0x7fe634131c90 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63a139640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fe634131c90 legacy=0x7fe634132060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63409fea0 con 0x7fe6340a0ab0 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6340a1520 con 0x7fe63409d210 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:56.237 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63c39d640 10 monclient(hunting): authenticate will time out at 1271.801819s 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63a93a640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fe6340a0ab0 msgr2=0x7fe634138fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.232+0000 7fe63a93a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe6340a0ab0 0x7fe634138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 0x7fe634069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 0x7fe634069e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59864/0 (socket says 172.21.15.144:59864) 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 -- 172.21.15.144:0/3829399522 learned_addr learned my addr 172.21.15.144:0/3829399522 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): get_auth_request con 0x7fe63409d210 auth_method 0 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient(hunting): handle_auth_done global_id 65312 payload 274 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 -- 172.21.15.144:0/3829399522 >> v2:172.21.15.144:3300/0 conn(0x7fe6340a0ab0 msgr2=0x7fe634138fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:56.238 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 --2- 172.21.15.144:0/3829399522 >> v2:172.21.15.144:3300/0 conn(0x7fe6340a0ab0 0x7fe634138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 -- 172.21.15.144:0/3829399522 >> v1:172.21.15.144:6789/0 conn(0x7fe634131c90 legacy=0x7fe634132060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:56.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient: _finish_hunting 0 2024-09-18T12:33:56.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 monclient: found mon.noname-b 2024-09-18T12:33:56.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:56.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:56.239 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.233+0000 7fe639938640 1 -- 172.21.15.144:0/3829399522 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634132770 con 0x7fe63409d210 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639938640 1 -- 172.21.15.144:0/3829399522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 msgr2=0x7fe634069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639938640 1 --2- 172.21.15.144:0/3829399522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 0x7fe634069e10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe624009910 tx=0x7fe6240091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbbf69ee788d1c56 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 1 -- 172.21.15.144:0/3829399522 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624032440 con 0x7fe63409d210 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: dump: 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.245 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: _finish_auth 0 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 1 -- 172.21.15.144:0/3829399522 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6240325a0 con 0x7fe63409d210 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 1 -- 172.21.15.144:0/3829399522 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624032980 con 0x7fe63409d210 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe63c39d640 5 monclient: authenticate success, global_id 65312 2024-09-18T12:33:56.246 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe63c39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe63c39d640 4 set_mon_vals no callback set 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.236+0000 7fe639137640 10 monclient: dump: 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:56.247 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 10 monclient: shutdown 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 -- 172.21.15.144:0/3829399522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 msgr2=0x7fe634069e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3829399522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 0x7fe634069e10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe624009910 tx=0x7fe6240091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 -- 172.21.15.144:0/3829399522 shutdown_connections 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3829399522 >> v2:172.21.15.144:3300/0 conn(0x7fe6340a0ab0 0x7fe634138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3829399522 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe63409d210 0x7fe634069e10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 -- 172.21.15.144:0/3829399522 >> 172.21.15.144:0/3829399522 conn(0x7fe634071370 msgr2=0x7fe6340a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 -- 172.21.15.144:0/3829399522 shutdown_connections 2024-09-18T12:33:56.248 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 1 -- 172.21.15.144:0/3829399522 wait complete. 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 10 monclient: build_initial_monmap 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.237+0000 7fe63c39d640 10 monclient: monmap: 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:56.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 Processor -- start 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 -- start start 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient: init 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6340a0ab0 0x7fe634134210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6340a0ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 0x7fe634134750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe634131c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe634134c90 0x7fe634147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): picked mon.noname-a con 0x7fe634134c90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6340a0fb0 con 0x7fe634134c90 2024-09-18T12:33:56.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63a139640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6340a0ab0 msgr2=0x7fe634134210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe634137bd0 con 0x7fe6340a0ab0 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6341362b0 con 0x7fe634131c90 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63c39d640 10 monclient(hunting): authenticate will time out at 1271.807726s 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.238+0000 7fe63a139640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6340a0ab0 0x7fe634134210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 0x7fe634134750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe634134c90 0x7fe634147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 0x7fe634134750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:59868/0 (socket says 172.21.15.144:59868) 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 -- 172.21.15.144:0/3461518040 learned_addr learned my addr 172.21.15.144:0/3461518040 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): get_auth_request con 0x7fe634131c90 auth_method 0 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:56.251 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 10 monclient(hunting): get_auth_request con 0x7fe634134c90 auth_method 0 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:56.252 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient(hunting): handle_auth_done global_id 65315 payload 824 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 -- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6340a0ab0 msgr2=0x7fe634134210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6340a0ab0 0x7fe634134210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 -- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe634134c90 msgr2=0x7fe634147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe634134c90 0x7fe634147840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient: _finish_hunting 0 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 monclient: found mon.noname-c 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe639938640 1 -- 172.21.15.144:0/3461518040 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624009450 con 0x7fe634131c90 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.239+0000 7fe63a93a640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe634134c90 0x7fe634147840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.241+0000 7fe639938640 1 -- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 msgr2=0x7fe634134750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.241+0000 7fe639938640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 0x7fe634134750 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe624009910 tx=0x7fe624032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:56.253 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62403c960 con 0x7fe634131c90 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: dump: 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:56.254 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe63c39d640 5 monclient: authenticate success, global_id 65315 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62403cac0 con 0x7fe634131c90 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62403ce60 con 0x7fe634131c90 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe62a7fc640 10 monclient: dump: 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.255 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe638936640 4 set_mon_vals no callback set 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe63c39d640 10 monclient: _renew_subs 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe63c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe634137860 con 0x7fe634131c90 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe63c39d640 10 monclient: _renew_subs 2024-09-18T12:33:56.256 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe63c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.242+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe634131720 con 0x7fe634131c90 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe63c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe63c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe63c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe634135060 con 0x7fe634131c90 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe624039040 con 0x7fe634131c90 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe62a7fc640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe618038490 0x7fe61803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(240..240 src has 13..240) v4 ==== 11953+0+0 (secure 0 0 0) 0x7fe62406d6c0 con 0x7fe634131c90 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe63a139640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe618038490 0x7fe61803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:56.257 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.243+0000 7fe63a139640 10 monclient: get_auth_request con 0x7fe618038490 auth_method 0 2024-09-18T12:33:56.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.245+0000 7fe63a139640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe618038490 0x7fe61803a940 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe630003940 tx=0x7fe630006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:56.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.247+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe6240315b0 con 0x7fe634131c90 2024-09-18T12:33:56.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.248+0000 7fe62a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:56.258 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.248+0000 7fe62a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:56.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.408+0000 7fe63c39d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T12:33:56.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.408+0000 7fe63c39d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T12:33:56.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.408+0000 7fe63c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:56.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:56.408+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7fe6340008d0 con 0x7fe634131c90 2024-09-18T12:33:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.406+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe624040030 con 0x7fe634131c90 2024-09-18T12:33:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.406+0000 7fe62a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T12:33:57.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.406+0000 7fe62a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-18T12:33:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.509+0000 7fe63c39d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T12:33:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.509+0000 7fe63c39d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T12:33:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.509+0000 7fe63c39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.509+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7fe6340008d0 con 0x7fe634131c90 2024-09-18T12:33:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.510+0000 7fe62a7fc640 1 -- 172.21.15.144:0/3461518040 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7fe624040020 con 0x7fe634131c90 2024-09-18T12:33:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.510+0000 7fe62a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T12:33:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.510+0000 7fe62a7fc640 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-18T12:33:57.511 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h15' does not appear in the crush map 2024-09-18T12:33:57.513 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe618038490 msgr2=0x7fe61803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe618038490 0x7fe61803a940 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe630003940 tx=0x7fe630006d80 comp rx=0 tx=0).stop 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 10 monclient: shutdown 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 msgr2=0x7fe634134750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 0x7fe634134750 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe624009910 tx=0x7fe624032000 comp rx=0 tx=0).stop 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 shutdown_connections 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe618038490 0x7fe61803a940 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe634134c90 0x7fe634147840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe634131c90 0x7fe634134750 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 --2- 172.21.15.144:0/3461518040 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6340a0ab0 0x7fe634134210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 >> 172.21.15.144:0/3461518040 conn(0x7fe634071370 msgr2=0x7fe63409db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 shutdown_connections 2024-09-18T12:33:57.514 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.512+0000 7fe63c39d640 1 -- 172.21.15.144:0/3461518040 wait complete. 2024-09-18T12:33:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h16 2024-09-18T12:33:57.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.596+0000 7ff555060640 10 monclient: get_monmap_and_config 2024-09-18T12:33:57.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.596+0000 7ff555060640 10 monclient: build_initial_monmap 2024-09-18T12:33:57.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.596+0000 7ff555060640 10 monclient: monmap: 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:57.597076+0000 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:57.597076+0000 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:57.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.596+0000 7ff555060640 1 Processor -- start 2024-09-18T12:33:57.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.596+0000 7ff555060640 1 -- start start 2024-09-18T12:33:57.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.596+0000 7ff555060640 10 monclient: init 2024-09-18T12:33:57.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:57.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T12:33:57.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff55009ec50 0x7ff550131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:57.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): picked mon.noname-c con 0x7ff55009ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 0x7ff550068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): picked mon.noname-b con 0x7ff5501321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff55009f020 0x7ff550069240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): picked mon.noname-a con 0x7ff55009f020 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff550067420 con 0x7ff55009f020 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5500a0480 con 0x7ff55009ec50 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5500a0200 con 0x7ff5501321a0 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:57.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff555060640 10 monclient(hunting): authenticate will time out at 1273.166342s 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54ed9d640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7ff55009ec50 msgr2=0x7ff550131c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54ed9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff55009ec50 0x7ff550131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 0x7ff550068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 0x7ff550068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56622/0 (socket says 172.21.15.144:56622) 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff55009f020 0x7ff550069240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 1 -- 172.21.15.144:0/1116743151 learned_addr learned my addr 172.21.15.144:0/1116743151 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 10 monclient(hunting): get_auth_request con 0x7ff5501321a0 auth_method 0 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 10 monclient(hunting): get_auth_request con 0x7ff55009f020 auth_method 0 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:57.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.597+0000 7ff54f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 10 monclient(hunting): handle_auth_done global_id 65318 payload 274 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 1 -- 172.21.15.144:0/1116743151 >> v2:172.21.15.144:3300/0 conn(0x7ff55009ec50 msgr2=0x7ff550131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 1 --2- 172.21.15.144:0/1116743151 >> v2:172.21.15.144:3300/0 conn(0x7ff55009ec50 0x7ff550131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 1 -- 172.21.15.144:0/1116743151 >> v2:172.21.15.116:3300/0 conn(0x7ff55009f020 msgr2=0x7ff550069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 1 --2- 172.21.15.144:0/1116743151 >> v2:172.21.15.116:3300/0 conn(0x7ff55009f020 0x7ff550069240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 1 monclient: found mon.noname-b 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54e59c640 1 -- 172.21.15.144:0/1116743151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55009fd80 con 0x7ff5501321a0 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.598+0000 7ff54f59e640 1 --2- 172.21.15.144:0/1116743151 >> v2:172.21.15.116:3300/0 conn(0x7ff55009f020 0x7ff550069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.600+0000 7ff54e59c640 1 -- 172.21.15.144:0/1116743151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 msgr2=0x7ff550068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.600+0000 7ff54e59c640 1 --2- 172.21.15.144:0/1116743151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 0x7ff550068d00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff5500719d0 tx=0x7ff5380091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d70e2c016aa0ea04 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 1 -- 172.21.15.144:0/1116743151 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538032440 con 0x7ff5501321a0 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:57.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: dump: 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:33:57.608 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 5 monclient: authenticate success, global_id 65318 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 1 -- 172.21.15.144:0/1116743151 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5380325a0 con 0x7ff5501321a0 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 1 -- 172.21.15.144:0/1116743151 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538032940 con 0x7ff5501321a0 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 4 set_mon_vals no callback set 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff54dd9b640 10 monclient: dump: 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:57.609 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 10 monclient: shutdown 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 -- 172.21.15.144:0/1116743151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 msgr2=0x7ff550068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 --2- 172.21.15.144:0/1116743151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 0x7ff550068d00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff5500719d0 tx=0x7ff5380091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 -- 172.21.15.144:0/1116743151 shutdown_connections 2024-09-18T12:33:57.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 --2- 172.21.15.144:0/1116743151 >> v2:172.21.15.116:3300/0 conn(0x7ff55009f020 0x7ff550069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 --2- 172.21.15.144:0/1116743151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff5501321a0 0x7ff550068d00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 --2- 172.21.15.144:0/1116743151 >> v2:172.21.15.144:3300/0 conn(0x7ff55009ec50 0x7ff550131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 -- 172.21.15.144:0/1116743151 >> 172.21.15.144:0/1116743151 conn(0x7ff5500a5cf0 msgr2=0x7ff5500a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 -- 172.21.15.144:0/1116743151 shutdown_connections 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.601+0000 7ff555060640 1 -- 172.21.15.144:0/1116743151 wait complete. 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.602+0000 7ff555060640 10 monclient: build_initial_monmap 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.602+0000 7ff555060640 10 monclient: monmap: 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:57.611 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:57.612 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:57.612 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:57.612 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:57.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.602+0000 7ff555060640 1 Processor -- start 2024-09-18T12:33:57.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.602+0000 7ff555060640 1 -- start start 2024-09-18T12:33:57.612 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.602+0000 7ff555060640 10 monclient: init 2024-09-18T12:33:57.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:57.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:57.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 0x7ff55009a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): picked mon.noname-c con 0x7ff55009ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff55009f020 0x7ff550130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): picked mon.noname-a con 0x7ff55009f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5501321a0 0x7ff5501410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): picked mon.noname-b con 0x7ff5501321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55009b540 con 0x7ff55009f020 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55009c050 con 0x7ff5501321a0 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55011dc00 con 0x7ff55009ec50 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:57.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff555060640 10 monclient(hunting): authenticate will time out at 1273.172168s 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54f59e640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5501321a0 msgr2=0x7ff5501410a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54f59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5501321a0 0x7ff5501410a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 0x7ff55009a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 0x7ff55009a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56638/0 (socket says 172.21.15.144:56638) 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 -- 172.21.15.144:0/1734773978 learned_addr learned my addr 172.21.15.144:0/1734773978 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): get_auth_request con 0x7ff55009ec50 auth_method 0 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:57.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff55009f020 0x7ff550130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 10 monclient(hunting): get_auth_request con 0x7ff55009f020 auth_method 0 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:57.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient(hunting): handle_auth_done global_id 65321 payload 824 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 -- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5501321a0 msgr2=0x7ff5501410a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5501321a0 0x7ff5501410a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 -- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff55009f020 msgr2=0x7ff550130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff55009f020 0x7ff550130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 monclient: found mon.noname-c 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.603+0000 7ff54ed9d640 1 -- 172.21.15.144:0/1734773978 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff548009390 con 0x7ff55009ec50 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.605+0000 7ff54ed9d640 1 -- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 msgr2=0x7ff55009a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.605+0000 7ff54ed9d640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 0x7ff55009a990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff54000bb80 tx=0x7ff5400097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000cc80 con 0x7ff55009ec50 2024-09-18T12:33:57.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: dump: 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:57.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 5 monclient: authenticate success, global_id 65321 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54000cde0 con 0x7ff55009ec50 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540016430 con 0x7ff55009ec50 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 10 monclient: _renew_subs 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff538009450 con 0x7ff55009ec50 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff54d59a640 4 set_mon_vals no callback set 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff546ffd640 10 monclient: dump: 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:57.624 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 10 monclient: _renew_subs 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55011bc30 con 0x7ff55009ec50 2024-09-18T12:33:57.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.606+0000 7ff555060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff555060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff555060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5500008d0 con 0x7ff55009ec50 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff540012020 con 0x7ff55009ec50 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff546ffd640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff530038440 0x7ff53003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(241..241 src has 13..241) v4 ==== 11740+0+0 (secure 0 0 0) 0x7ff54004d230 con 0x7ff55009ec50 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff54e59c640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff530038440 0x7ff53003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.607+0000 7ff54e59c640 10 monclient: get_auth_request con 0x7ff530038440 auth_method 0 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.609+0000 7ff54e59c640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff530038440 0x7ff53003a8f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff55009bf50 tx=0x7ff538032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.612+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff54001d900 con 0x7ff55009ec50 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.612+0000 7ff546ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:57.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.612+0000 7ff546ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:57.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.771+0000 7ff555060640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T12:33:57.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.771+0000 7ff555060640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T12:33:57.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.771+0000 7ff555060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:57.772 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:57.771+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7ff5500a13e0 con 0x7ff55009ec50 2024-09-18T12:33:58.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.408+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff540014340 con 0x7ff55009ec50 2024-09-18T12:33:58.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.408+0000 7ff546ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T12:33:58.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.408+0000 7ff546ffd640 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-18T12:33:58.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.505+0000 7ff555060640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T12:33:58.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.506+0000 7ff555060640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T12:33:58.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.506+0000 7ff555060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:58.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.506+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7ff550072640 con 0x7ff55009ec50 2024-09-18T12:33:58.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.718+0000 7ff546ffd640 1 -- 172.21.15.144:0/1734773978 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff54001d300 con 0x7ff55009ec50 2024-09-18T12:33:58.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.718+0000 7ff546ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T12:33:58.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.718+0000 7ff546ffd640 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-18T12:33:58.719 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h16' does not appear in the crush map 2024-09-18T12:33:58.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff530038440 msgr2=0x7ff53003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:58.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff530038440 0x7ff53003a8f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff55009bf50 tx=0x7ff538032000 comp rx=0 tx=0).stop 2024-09-18T12:33:58.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 10 monclient: shutdown 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 msgr2=0x7ff55009a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 0x7ff55009a990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff54000bb80 tx=0x7ff5400097f0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 shutdown_connections 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff530038440 0x7ff53003a8f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff5501321a0 0x7ff5501410a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff55009f020 0x7ff550130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 --2- 172.21.15.144:0/1734773978 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff55009ec50 0x7ff55009a990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 >> 172.21.15.144:0/1734773978 conn(0x7ff5500a5cf0 msgr2=0x7ff550134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 shutdown_connections 2024-09-18T12:33:58.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.720+0000 7ff555060640 1 -- 172.21.15.144:0/1734773978 wait complete. 2024-09-18T12:33:58.730 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:58.730 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h17 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.803+0000 7f2eae6f8640 10 monclient: get_monmap_and_config 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.803+0000 7f2eae6f8640 10 monclient: build_initial_monmap 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.803+0000 7f2eae6f8640 10 monclient: monmap: 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:58.804068+0000 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:58.804068+0000 2024-09-18T12:33:58.804 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:58.805 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:58.805 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:58.805 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:58.805 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:58.805 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:58.805 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:58.805 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:58.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.803+0000 7f2eae6f8640 1 Processor -- start 2024-09-18T12:33:58.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.803+0000 7f2eae6f8640 1 -- start start 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.803+0000 7f2eae6f8640 10 monclient: init 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ea809ec50 0x7f2ea8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ea809ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 0x7f2ea8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea81321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ea809f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea8067420 con 0x7f2ea809f020 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea80a0480 con 0x7f2ea809ec50 2024-09-18T12:33:58.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea80a0200 con 0x7f2ea81321a0 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eae6f8640 10 monclient(hunting): authenticate will time out at 1274.373472s 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 0x7f2ea8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ea809ec50 0x7f2ea8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ea809ec50 0x7f2ea8131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:46376/0 (socket says 172.21.15.144:46376) 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 0x7f2ea8068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56646/0 (socket says 172.21.15.144:56646) 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 1 -- 172.21.15.144:0/4287340393 learned_addr learned my addr 172.21.15.144:0/4287340393 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 10 monclient(hunting): get_auth_request con 0x7f2ea809ec50 auth_method 0 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:58.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2eacc95640 1 --1- 172.21.15.144:0/4287340393 >> v1:172.21.15.116:6789/0 conn(0x7f2ea809f020 0x7f2ea8069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:45702/0 (socket says 172.21.15.144:45702) 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 10 monclient(hunting): get_auth_request con 0x7f2ea81321a0 auth_method 0 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.804+0000 7f2ea7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 10 monclient(hunting): handle_auth_done global_id 65324 payload 274 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 1 -- 172.21.15.144:0/4287340393 >> v2:172.21.15.116:3300/0 conn(0x7f2ea809ec50 msgr2=0x7f2ea8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:58.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 1 --2- 172.21.15.144:0/4287340393 >> v2:172.21.15.116:3300/0 conn(0x7f2ea809ec50 0x7f2ea8131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 1 -- 172.21.15.144:0/4287340393 >> v1:172.21.15.116:6789/0 conn(0x7f2ea809f020 legacy=0x7f2ea8069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:58.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:33:58.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 1 monclient: found mon.noname-b 2024-09-18T12:33:58.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:58.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:58.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea77fe640 1 -- 172.21.15.144:0/4287340393 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea809fd80 con 0x7f2ea81321a0 2024-09-18T12:33:58.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.805+0000 7f2ea7fff640 1 --2- 172.21.15.144:0/4287340393 >> v2:172.21.15.116:3300/0 conn(0x7f2ea809ec50 0x7f2ea8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:33:58.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.807+0000 7f2ea77fe640 1 -- 172.21.15.144:0/4287340393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 msgr2=0x7f2ea8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.807+0000 7f2ea77fe640 1 --2- 172.21.15.144:0/4287340393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 0x7f2ea8068d00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009850 tx=0x7f2e9c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433db79a04cc2c16 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 1 -- 172.21.15.144:0/4287340393 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c02ae60 con 0x7f2ea81321a0 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: dump: 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:58.818 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 5 monclient: authenticate success, global_id 65324 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 1 -- 172.21.15.144:0/4287340393 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9c032440 con 0x7f2ea81321a0 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 1 -- 172.21.15.144:0/4287340393 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0327e0 con 0x7f2ea81321a0 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 4 set_mon_vals no callback set 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2ea6ffd640 10 monclient: dump: 2024-09-18T12:33:58.819 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 10 monclient: shutdown 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 -- 172.21.15.144:0/4287340393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 msgr2=0x7f2ea8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:58.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/4287340393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 0x7f2ea8068d00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009850 tx=0x7f2e9c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 -- 172.21.15.144:0/4287340393 shutdown_connections 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/4287340393 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea81321a0 0x7f2ea8068d00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/4287340393 >> v2:172.21.15.116:3300/0 conn(0x7f2ea809ec50 0x7f2ea8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 -- 172.21.15.144:0/4287340393 >> 172.21.15.144:0/4287340393 conn(0x7f2ea80a5cf0 msgr2=0x7f2ea80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 -- 172.21.15.144:0/4287340393 shutdown_connections 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.808+0000 7f2eae6f8640 1 -- 172.21.15.144:0/4287340393 wait complete. 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.809+0000 7f2eae6f8640 10 monclient: build_initial_monmap 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.809+0000 7f2eae6f8640 10 monclient: monmap: 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:58.821 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.809+0000 7f2eae6f8640 1 Processor -- start 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.809+0000 7f2eae6f8640 1 -- start start 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.809+0000 7f2eae6f8640 10 monclient: init 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 0x7f2ea8130b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ea809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea809f020 0x7f2ea8140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:58.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ea809f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea81321a0 0x7f2ea81414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea81321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea811cbd0 con 0x7f2ea809f020 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea809c020 con 0x7f2ea81321a0 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea811f230 con 0x7f2ea809ec50 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eae6f8640 10 monclient(hunting): authenticate will time out at 1274.379103s 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eacc95640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea81321a0 msgr2=0x7f2ea81414d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2eacc95640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea81321a0 0x7f2ea81414d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:58.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 0x7f2ea8130b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 0x7f2ea8130b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56652/0 (socket says 172.21.15.144:56652) 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea809f020 0x7f2ea8140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 -- 172.21.15.144:0/3784029389 learned_addr learned my addr 172.21.15.144:0/3784029389 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): get_auth_request con 0x7f2ea809ec50 auth_method 0 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 10 monclient(hunting): get_auth_request con 0x7f2ea809f020 auth_method 0 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:58.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient(hunting): handle_auth_done global_id 65327 payload 824 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 -- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea81321a0 msgr2=0x7f2ea81414d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea81321a0 0x7f2ea81414d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 -- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea809f020 msgr2=0x7f2ea8140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea809f020 0x7f2ea8140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 monclient: found mon.noname-c 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:58.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.810+0000 7f2ea7fff640 1 -- 172.21.15.144:0/3784029389 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e98009450 con 0x7f2ea809ec50 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.812+0000 7f2ea7fff640 1 -- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 msgr2=0x7f2ea8130b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.812+0000 7f2ea7fff640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 0x7f2ea8130b40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800ecb0 tx=0x7f2e980091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98009730 con 0x7f2ea809ec50 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: dump: 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:58.826 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: _finish_auth 0 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9800cdd0 con 0x7f2ea809ec50 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2eae6f8640 5 monclient: authenticate success, global_id 65327 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e980136a0 con 0x7f2ea809ec50 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2eae6f8640 10 monclient: _renew_subs 2024-09-18T12:33:58.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2eae6f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9c009390 con 0x7f2ea809ec50 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2e8ffff640 10 monclient: dump: 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:58.828 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2ea67fc640 4 set_mon_vals no callback set 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2eae6f8640 10 monclient: _renew_subs 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2eae6f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.813+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea811dd80 con 0x7f2ea809ec50 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2e98013a10 con 0x7f2ea809ec50 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2eae6f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2eae6f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2eae6f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea80008d0 con 0x7f2ea809ec50 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2e8ffff640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e94038440 0x7f2e9403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(242..242 src has 13..242) v4 ==== 11527+0+0 (secure 0 0 0) 0x7f2e9804d780 con 0x7f2ea809ec50 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2ea77fe640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e94038440 0x7f2e9403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.814+0000 7f2ea77fe640 10 monclient: get_auth_request con 0x7f2e94038440 auth_method 0 2024-09-18T12:33:58.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.816+0000 7f2ea77fe640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e94038440 0x7f2e9403a8f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2ea809bf50 tx=0x7f2e9c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:58.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.818+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2e98023ad0 con 0x7f2ea809ec50 2024-09-18T12:33:58.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.818+0000 7f2e8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:58.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.818+0000 7f2e8ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:58.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.977+0000 7f2eae6f8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T12:33:58.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.977+0000 7f2eae6f8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T12:33:58.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.977+0000 7f2eae6f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:58.978 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:58.977+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f2ea80a13e0 con 0x7f2ea809ec50 2024-09-18T12:33:59.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.416+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v243) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2e980232f0 con 0x7f2ea809ec50 2024-09-18T12:33:59.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.416+0000 7f2e8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T12:33:59.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.416+0000 7f2e8ffff640 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-18T12:33:59.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.514+0000 7f2eae6f8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T12:33:59.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.514+0000 7f2eae6f8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T12:33:59.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.514+0000 7f2eae6f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:59.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.514+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f2ea8064900 con 0x7f2ea809ec50 2024-09-18T12:33:59.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.516+0000 7f2e8ffff640 1 -- 172.21.15.144:0/3784029389 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v243) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2e980234d0 con 0x7f2ea809ec50 2024-09-18T12:33:59.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.516+0000 7f2e8ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T12:33:59.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.516+0000 7f2e8ffff640 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-18T12:33:59.517 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h17' does not appear in the crush map 2024-09-18T12:33:59.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e94038440 msgr2=0x7f2e9403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:59.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e94038440 0x7f2e9403a8f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2ea809bf50 tx=0x7f2e9c032000 comp rx=0 tx=0).stop 2024-09-18T12:33:59.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 10 monclient: shutdown 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 msgr2=0x7f2ea8130b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 0x7f2ea8130b40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800ecb0 tx=0x7f2e980091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 shutdown_connections 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e94038440 0x7f2e9403a8f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ea81321a0 0x7f2ea81414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea809f020 0x7f2ea8140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 --2- 172.21.15.144:0/3784029389 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2ea809ec50 0x7f2ea8130b40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 >> 172.21.15.144:0/3784029389 conn(0x7f2ea80a5cf0 msgr2=0x7f2ea809d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 shutdown_connections 2024-09-18T12:33:59.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.518+0000 7f2eae6f8640 1 -- 172.21.15.144:0/3784029389 wait complete. 2024-09-18T12:33:59.529 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:33:59.529 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h18 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.600+0000 7f9dd755d640 10 monclient: get_monmap_and_config 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.600+0000 7f9dd755d640 10 monclient: build_initial_monmap 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.600+0000 7f9dd755d640 10 monclient: monmap: 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:33:59.601175+0000 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:33:59.601175+0000 2024-09-18T12:33:59.601 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:33:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:33:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:33:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:33:59.602 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:33:59.602 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.600+0000 7f9dd755d640 1 Processor -- start 2024-09-18T12:33:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 1 -- start start 2024-09-18T12:33:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient: init 2024-09-18T12:33:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T12:33:59.604 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd0131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient(hunting): picked mon.noname-b con 0x7f9dd009ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9dd01321a0 0x7f9dd0068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient(hunting): picked mon.noname-c con 0x7f9dd01321a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9dd009f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd0067420 con 0x7f9dd009f020 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd00a0480 con 0x7f9dd01321a0 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd00a0200 con 0x7f9dd009ec50 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd755d640 10 monclient(hunting): authenticate will time out at 1275.170652s 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd4af8640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f9dd01321a0 msgr2=0x7f9dd0068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:59.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd4af8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9dd01321a0 0x7f9dd0068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd0131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd0131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56666/0 (socket says 172.21.15.144:56666) 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 1 -- 172.21.15.144:0/2154778995 learned_addr learned my addr 172.21.15.144:0/2154778995 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 10 monclient(hunting): get_auth_request con 0x7f9dd009ec50 auth_method 0 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd5afa640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9dd009f020 0x7f9dd0069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:45718/0 (socket says 172.21.15.144:45718) 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:59.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.601+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_done global_id 65330 payload 274 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 1 -- 172.21.15.144:0/2154778995 >> v2:172.21.15.144:3300/0 conn(0x7f9dd01321a0 msgr2=0x7f9dd0068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 1 --2- 172.21.15.144:0/2154778995 >> v2:172.21.15.144:3300/0 conn(0x7f9dd01321a0 0x7f9dd0068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 1 -- 172.21.15.144:0/2154778995 >> v1:172.21.15.116:6789/0 conn(0x7f9dd009f020 legacy=0x7f9dd0069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 10 monclient: _finish_hunting 0 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 1 monclient: found mon.noname-b 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:33:59.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.602+0000 7f9dd52f9640 1 -- 172.21.15.144:0/2154778995 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd009fd80 con 0x7f9dd009ec50 2024-09-18T12:33:59.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.604+0000 7f9dd52f9640 1 -- 172.21.15.144:0/2154778995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 msgr2=0x7f9dd0131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:59.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.604+0000 7f9dd52f9640 1 --2- 172.21.15.144:0/2154778995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd0131c60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009910 tx=0x7f9dc00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=947c09d3e4e4b722 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:59.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 1 -- 172.21.15.144:0/2154778995 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc002ae20 con 0x7f9dd009ec50 2024-09-18T12:33:59.613 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: dump: 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: _finish_auth 0 2024-09-18T12:33:59.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 1 -- 172.21.15.144:0/2154778995 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc002f4b0 con 0x7f9dd009ec50 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 1 -- 172.21.15.144:0/2154778995 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc002f850 con 0x7f9dd009ec50 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dc7fff640 10 monclient: dump: 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:59.615 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dd755d640 5 monclient: authenticate success, global_id 65330 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dd755d640 10 monclient: get_monmap_and_config success 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dd755d640 4 set_mon_vals no callback set 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dd755d640 10 monclient: shutdown 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dd755d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dd755d640 1 -- 172.21.15.144:0/2154778995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 msgr2=0x7f9dd0131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.605+0000 7f9dd755d640 1 --2- 172.21.15.144:0/2154778995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd0131c60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009910 tx=0x7f9dc00091d0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 -- 172.21.15.144:0/2154778995 shutdown_connections 2024-09-18T12:33:59.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 --2- 172.21.15.144:0/2154778995 >> v2:172.21.15.144:3300/0 conn(0x7f9dd01321a0 0x7f9dd0068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 --2- 172.21.15.144:0/2154778995 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd0131c60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 -- 172.21.15.144:0/2154778995 >> 172.21.15.144:0/2154778995 conn(0x7f9dd00a5cf0 msgr2=0x7f9dd00a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 -- 172.21.15.144:0/2154778995 shutdown_connections 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 -- 172.21.15.144:0/2154778995 wait complete. 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 10 monclient: build_initial_monmap 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 10 monclient: monmap: 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:59.617 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 Processor -- start 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 1 -- start start 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.606+0000 7f9dd755d640 10 monclient: init 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient: _reopen_session rank -1 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd009a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): picked mon.noname-c con 0x7f9dd009ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd009f020 0x7f9dd0130a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): picked mon.noname-a con 0x7f9dd009f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01321a0 0x7f9dd011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:59.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): picked mon.noname-b con 0x7f9dd01321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0134b50 con 0x7f9dd009f020 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0141f70 con 0x7f9dd01321a0 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd011de70 con 0x7f9dd009ec50 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): _renew_subs 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd755d640 10 monclient(hunting): authenticate will time out at 1275.176527s 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd5afa640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01321a0 msgr2=0x7f9dd011b760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd5afa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01321a0 0x7f9dd011b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd52f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd009a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd4af8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd009f020 0x7f9dd0130a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd52f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd009a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56674/0 (socket says 172.21.15.144:56674) 2024-09-18T12:33:59.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd52f9640 1 -- 172.21.15.144:0/1093603753 learned_addr learned my addr 172.21.15.144:0/1093603753 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd52f9640 10 monclient(hunting): get_auth_request con 0x7f9dd009ec50 auth_method 0 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd52f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd52f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd52f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd4af8640 10 monclient(hunting): get_auth_request con 0x7f9dd009f020 auth_method 0 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd4af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd4af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.607+0000 7f9dd4af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd4af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:33:59.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd4af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd4af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 10 monclient(hunting): handle_auth_done global_id 65333 payload 824 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 1 -- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01321a0 msgr2=0x7f9dd011b760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01321a0 0x7f9dd011b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 1 -- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd009f020 msgr2=0x7f9dd0130a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd009f020 0x7f9dd0130a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 10 monclient: _finish_hunting 0 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 1 monclient: found mon.noname-c 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.608+0000 7f9dd52f9640 1 -- 172.21.15.144:0/1093603753 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0009450 con 0x7f9dd009ec50 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dd52f9640 1 -- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 msgr2=0x7f9dd009a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dd52f9640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd009a960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009210 tx=0x7f9dc0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:59.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0032930 con 0x7f9dd009ec50 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: dump: 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:59.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: _finish_auth 0 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0032ab0 con 0x7f9dd009ec50 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dd755d640 5 monclient: authenticate success, global_id 65333 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dd755d640 10 monclient: _renew_subs 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dd755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd009b4e0 con 0x7f9dd009ec50 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0032e70 con 0x7f9dd009ec50 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc57fa640 10 monclient: dump: 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:33:59.623 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.610+0000 7f9dc77fe640 4 set_mon_vals no callback set 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dd755d640 10 monclient: _renew_subs 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dd755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd0141570 con 0x7f9dd009ec50 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9dc0039070 con 0x7f9dd009ec50 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dd755d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:59.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dd755d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:59.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dd755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:59.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd011e030 con 0x7f9dd009ec50 2024-09-18T12:33:59.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.611+0000 7f9dc57fa640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9db4038440 0x7f9db403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:33:59.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.612+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(243..243 src has 13..243) v4 ==== 11314+0+0 (secure 0 0 0) 0x7f9dc006d7d0 con 0x7f9dd009ec50 2024-09-18T12:33:59.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.612+0000 7f9dd4af8640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9db4038440 0x7f9db403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:33:59.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.612+0000 7f9dd4af8640 10 monclient: get_auth_request con 0x7f9db4038440 auth_method 0 2024-09-18T12:33:59.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.614+0000 7f9dd4af8640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9db4038440 0x7f9db403a8f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc003850 tx=0x7f9dcc0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:33:59.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.616+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9dc003aae0 con 0x7f9dd009ec50 2024-09-18T12:33:59.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.616+0000 7f9dc57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:33:59.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.616+0000 7f9dc57fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:33:59.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.777+0000 7f9dd755d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T12:33:59.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.777+0000 7f9dd755d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T12:33:59.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.777+0000 7f9dd755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:33:59.779 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:33:59.777+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f9dd00008d0 con 0x7f9dd009ec50 2024-09-18T12:34:00.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.420+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v244) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9dc0037090 con 0x7f9dd009ec50 2024-09-18T12:34:00.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.420+0000 7f9dc57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T12:34:00.421 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.420+0000 7f9dc57fa640 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-18T12:34:00.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.519+0000 7f9dd755d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T12:34:00.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.519+0000 7f9dd755d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T12:34:00.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.519+0000 7f9dd755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:00.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.519+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f9dd00008d0 con 0x7f9dd009ec50 2024-09-18T12:34:00.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.521+0000 7f9dc57fa640 1 -- 172.21.15.144:0/1093603753 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v244) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9dc003a300 con 0x7f9dd009ec50 2024-09-18T12:34:00.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.521+0000 7f9dc57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T12:34:00.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.521+0000 7f9dc57fa640 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-18T12:34:00.522 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h18' does not appear in the crush map 2024-09-18T12:34:00.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9db4038440 msgr2=0x7f9db403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:00.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9db4038440 0x7f9db403a8f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc003850 tx=0x7f9dcc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 10 monclient: shutdown 2024-09-18T12:34:00.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:00.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 msgr2=0x7f9dd009a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd009a960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009210 tx=0x7f9dc0032000 comp rx=0 tx=0).stop 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 shutdown_connections 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9db4038440 0x7f9db403a8f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9dd01321a0 0x7f9dd011b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd009f020 0x7f9dd0130a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 --2- 172.21.15.144:0/1093603753 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9dd009ec50 0x7f9dd009a960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 >> 172.21.15.144:0/1093603753 conn(0x7f9dd00a5cf0 msgr2=0x7f9dd00a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 shutdown_connections 2024-09-18T12:34:00.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.523+0000 7f9dd755d640 1 -- 172.21.15.144:0/1093603753 wait complete. 2024-09-18T12:34:00.533 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:00.533 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h19 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.605+0000 7f218411d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.605+0000 7f218411d640 10 monclient: build_initial_monmap 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.605+0000 7f218411d640 10 monclient: monmap: 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:00.606842+0000 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:00.606842+0000 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:00.607 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:00.608 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:00.608 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:00.608 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:00.608 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.606+0000 7f218411d640 1 Processor -- start 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.606+0000 7f218411d640 1 -- start start 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.606+0000 7f218411d640 10 monclient: init 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f217c131c00 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f217c1326e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f217c0a0b00 0x7f217c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient(hunting): picked mon.noname-c con 0x7f217c0a0b00 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:00.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217c0a1b70 con 0x7f217c1326e0 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f2181eb9640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f217c131c00 legacy=0x7f217c131fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217c0a08c0 con 0x7f217c131c00 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217c0a0200 con 0x7f217c0a0b00 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f21826ba640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f217c0a0b00 msgr2=0x7f217c069d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f218411d640 10 monclient(hunting): authenticate will time out at 1276.176352s 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f21826ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f217c0a0b00 0x7f217c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f21816b8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f217c1326e0 0x7f217c136fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:45720/0 (socket says 172.21.15.144:45720) 2024-09-18T12:34:00.611 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.607+0000 7f21816b8640 1 -- 172.21.15.144:0/2622936871 learned_addr learned my addr 172.21.15.144:0/2622936871 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:00.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.609+0000 7f21816b8640 1 -- 172.21.15.144:0/2622936871 >> v1:172.21.15.116:6789/0 conn(0x7f217c1326e0 legacy=0x7f217c136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:00.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175832353 0 0) 0x7f217c0a1b70 con 0x7f217c1326e0 2024-09-18T12:34:00.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:00.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:00.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): my global_id is 104538 2024-09-18T12:34:00.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680034e0 con 0x7f217c1326e0 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616395098 0 0) 0x7f21680034e0 con 0x7f217c1326e0 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 >> v2:172.21.15.144:3300/0 conn(0x7f217c0a0b00 msgr2=0x7f217c069d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 --2- 172.21.15.144:0/2622936871 >> v2:172.21.15.144:3300/0 conn(0x7f217c0a0b00 0x7f217c069d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 >> v1:172.21.15.144:6789/0 conn(0x7f217c131c00 legacy=0x7f217c131fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: _finish_hunting 0 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c0a1890 con 0x7f217c1326e0 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: _finish_auth 0 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:00.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f216c0017a0 con 0x7f217c1326e0 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: dump: 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:00.621 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 >> v1:172.21.15.116:6789/0 conn(0x7f217c1326e0 legacy=0x7f217c136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21680039d0 0x7f2168003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): picked mon.c con 0x7f21680039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f2168004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): picked mon.a con 0x7f2168004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2168005330 0x7f216800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): picked mon.b con 0x7f2168005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0a0200 con 0x7f2168004780 2024-09-18T12:34:00.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216800c2f0 con 0x7f2168005330 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216800c4b0 con 0x7f21680039d0 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2180eb7640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f21826ba640 1 -- 172.21.15.144:0/2622936871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2168005330 msgr2=0x7f216800ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f21826ba640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2168005330 0x7f216800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2181eb9640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21680039d0 0x7f2168003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f21816b8640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f2168004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2181eb9640 10 monclient(hunting): get_auth_request con 0x7f21680039d0 auth_method 0 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2181eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2181eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:00.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f2181eb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f21816b8640 10 monclient(hunting): get_auth_request con 0x7f2168004780 auth_method 0 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f21816b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f21816b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.610+0000 7f21816b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f2181eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f2181eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f2181eb9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 10 monclient(hunting): handle_auth_done global_id 104538 payload 293 2024-09-18T12:34:00.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 1 -- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21680039d0 msgr2=0x7f2168003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21680039d0 0x7f2168003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 1 -- 172.21.15.144:0/2622936871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2168005330 msgr2=0x7f216800ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2168005330 0x7f216800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 10 monclient: _finish_hunting 0 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 1 monclient: found mon.a 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21826ba640 1 -- 172.21.15.144:0/2622936871 reap_dead start 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 1 -- 172.21.15.144:0/2622936871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680034e0 con 0x7f2168004780 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 10 monclient: _finish_auth 0 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f21816b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f218411d640 5 monclient: authenticate success, global_id 104538 2024-09-18T12:34:00.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f218411d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.611+0000 7f2181eb9640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f21680039d0 0x7f2168003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f21816b8640 1 -- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 msgr2=0x7f2168004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f21816b8640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f2168004bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f216c0014d0 tx=0x7f216c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744c41b623ace025 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c005890 con 0x7f2168004780 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 10 monclient: dump: 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:00.626 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f218411d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c005a10 con 0x7f2168004780 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 1 -- 172.21.15.144:0/2622936871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c005dd0 con 0x7f2168004780 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f218411d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f218411d640 4 set_mon_vals no callback set 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f2180eb7640 10 monclient: dump: 2024-09-18T12:34:00.627 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f218411d640 10 monclient: shutdown 2024-09-18T12:34:00.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f218411d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f218411d640 1 -- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 msgr2=0x7f2168004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.614+0000 7f218411d640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f2168004bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f216c0014d0 tx=0x7f216c007a90 comp rx=0 tx=0).stop 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 1 -- 172.21.15.144:0/2622936871 shutdown_connections 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 1 --2- 172.21.15.144:0/2622936871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f2168004bf0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 1 -- 172.21.15.144:0/2622936871 >> 172.21.15.144:0/2622936871 conn(0x7f217c071370 msgr2=0x7f217c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 1 -- 172.21.15.144:0/2622936871 shutdown_connections 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 1 -- 172.21.15.144:0/2622936871 wait complete. 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 10 monclient: build_initial_monmap 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 10 monclient: monmap: 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:00.629 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 1 Processor -- start 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 1 -- start start 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.615+0000 7f218411d640 10 monclient: init 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f217c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): picked mon.noname-a con 0x7f2168004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:00.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f217c141c90 0x7f217c130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): picked mon.noname-c con 0x7f217c141c90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c131130 0x7f217c11b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): picked mon.noname-b con 0x7f217c131130 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c11e010 con 0x7f2168004780 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0a0fd0 con 0x7f217c131130 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217c134b00 con 0x7f217c141c90 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:00.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f218411d640 10 monclient(hunting): authenticate will time out at 1276.185424s 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21826ba640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c131130 msgr2=0x7f217c11b900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2171ffb640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21826ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c131130 0x7f217c11b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f217c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f217c141c90 0x7f217c130bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f217c09a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:46402/0 (socket says 172.21.15.144:46402) 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 -- 172.21.15.144:0/3884260323 learned_addr learned my addr 172.21.15.144:0/3884260323 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): get_auth_request con 0x7f2168004780 auth_method 0 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:00.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 10 monclient(hunting): get_auth_request con 0x7f217c141c90 auth_method 0 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f21816b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient(hunting): handle_auth_done global_id 104541 payload 824 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 -- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f217c141c90 msgr2=0x7f217c130bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f217c141c90 0x7f217c130bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.633 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 -- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c131130 msgr2=0x7f217c11b900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:00.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c131130 0x7f217c11b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:00.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient: _finish_hunting 0 2024-09-18T12:34:00.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 1 monclient: found mon.noname-a 2024-09-18T12:34:00.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:00.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.616+0000 7f2181eb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:00.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.617+0000 7f2181eb9640 1 -- 172.21.15.144:0/3884260323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c11dd30 con 0x7f2168004780 2024-09-18T12:34:00.634 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.617+0000 7f21816b8640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f217c141c90 0x7f217c130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:00.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2181eb9640 1 -- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 msgr2=0x7f217c09a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:00.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2181eb9640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f217c09a9d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2178009740 tx=0x7f2178009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:00.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21780128e0 con 0x7f2168004780 2024-09-18T12:34:00.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:00.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: dump: 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:00.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2178012a40 con 0x7f2168004780 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178012dc0 con 0x7f2168004780 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f218411d640 5 monclient: authenticate success, global_id 104541 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2171ffb640 10 monclient: dump: 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:00.638 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f2173fff640 4 set_mon_vals no callback set 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f218411d640 10 monclient: _renew_subs 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f218411d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.619+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217c09b590 con 0x7f2168004780 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f218411d640 10 monclient: _renew_subs 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f218411d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217c11ce40 con 0x7f2168004780 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f218411d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:00.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f218411d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f218411d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217c0008d0 con 0x7f2168004780 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.620+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2178011560 con 0x7f2168004780 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.621+0000 7f2171ffb640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2160038490 0x7f216003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.621+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(244..244 src has 13..244) v4 ==== 11101+0+0 (secure 0 0 0) 0x7f217804cac0 con 0x7f2168004780 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.621+0000 7f21816b8640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2160038490 0x7f216003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.621+0000 7f21816b8640 10 monclient: get_auth_request con 0x7f2160038490 auth_method 0 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.623+0000 7f21816b8640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2160038490 0x7f216003a940 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f216c001980 tx=0x7f216c034000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.626+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f21780198a0 con 0x7f2168004780 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.626+0000 7f2171ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:00.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.626+0000 7f2171ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:00.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.783+0000 7f218411d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T12:34:00.784 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.783+0000 7f218411d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T12:34:00.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.783+0000 7f218411d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:00.785 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.783+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f217c0a0440 con 0x7f2168004780 2024-09-18T12:34:00.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.988+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v245) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2178019670 con 0x7f2168004780 2024-09-18T12:34:00.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.988+0000 7f2171ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T12:34:00.989 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:00.988+0000 7f2171ffb640 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-18T12:34:01.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.086+0000 7f218411d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T12:34:01.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.086+0000 7f218411d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T12:34:01.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.086+0000 7f218411d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:01.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.086+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f217c072640 con 0x7f2168004780 2024-09-18T12:34:01.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.087+0000 7f2171ffb640 1 -- 172.21.15.144:0/3884260323 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v245) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2178019670 con 0x7f2168004780 2024-09-18T12:34:01.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.087+0000 7f2171ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T12:34:01.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.087+0000 7f2171ffb640 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-18T12:34:01.088 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h19' does not appear in the crush map 2024-09-18T12:34:01.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2160038490 msgr2=0x7f216003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:01.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2160038490 0x7f216003a940 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f216c001980 tx=0x7f216c034000 comp rx=0 tx=0).stop 2024-09-18T12:34:01.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 10 monclient: shutdown 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 msgr2=0x7f217c09a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f217c09a9d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2178009740 tx=0x7f2178009060 comp rx=0 tx=0).stop 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 shutdown_connections 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2160038490 0x7f216003a940 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f217c131130 0x7f217c11b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f217c141c90 0x7f217c130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 --2- 172.21.15.144:0/3884260323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2168004780 0x7f217c09a9d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 >> 172.21.15.144:0/3884260323 conn(0x7f217c071370 msgr2=0x7f217c134d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 shutdown_connections 2024-09-18T12:34:01.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.089+0000 7f218411d640 1 -- 172.21.15.144:0/3884260323 wait complete. 2024-09-18T12:34:01.099 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:01.099 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h20 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.171+0000 7fdffcd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.171+0000 7fdffcd9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.171+0000 7fdffcd9d640 10 monclient: monmap: 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:01.171944+0000 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:01.171944+0000 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:01.172 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:01.173 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:01.173 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:01.173 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:01.173 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:01.175 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.171+0000 7fdffcd9d640 1 Processor -- start 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.171+0000 7fdffcd9d640 1 -- start start 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.171+0000 7fdffcd9d640 10 monclient: init 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff8131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdff809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdff81321a0 0x7fdff8068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdff81321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdff809f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:01.176 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8067420 con 0x7fdff809f020 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff80a0480 con 0x7fdff81321a0 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80a0200 con 0x7fdff809ec50 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdffcd9d640 10 monclient(hunting): authenticate will time out at 1276.741250s 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff6d9d640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fdff809f020 legacy=0x7fdff8069240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdff81321a0 0x7fdff8068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:01.177 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff8131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdff81321a0 0x7fdff8068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:46406/0 (socket says 172.21.15.144:46406) 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff8131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56690/0 (socket says 172.21.15.144:56690) 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 -- 172.21.15.144:0/1052507368 learned_addr learned my addr 172.21.15.144:0/1052507368 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): get_auth_request con 0x7fdff81321a0 auth_method 0 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 10 monclient(hunting): get_auth_request con 0x7fdff809ec50 auth_method 0 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:01.178 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient(hunting): handle_auth_done global_id 104544 payload 274 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 -- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 msgr2=0x7fdff8131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 --2- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff8131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 -- 172.21.15.144:0/1052507368 >> v1:172.21.15.144:6789/0 conn(0x7fdff809f020 legacy=0x7fdff8069240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient: _finish_hunting 0 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 monclient: found mon.noname-a 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdfedd9b640 1 -- 172.21.15.144:0/1052507368 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff809fd80 con 0x7fdff81321a0 2024-09-18T12:34:01.179 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.172+0000 7fdff659c640 1 --2- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.175+0000 7fdfedd9b640 1 -- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 msgr2=0x7fdff8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.175+0000 7fdfedd9b640 1 --2- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 0x7fdff8068d00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009910 tx=0x7fdfe00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89c31d94aac8685 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 1 -- 172.21.15.144:0/1052507368 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0032440 con 0x7fdff81321a0 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: dump: 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:01.186 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: _finish_auth 0 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 1 -- 172.21.15.144:0/1052507368 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe00325a0 con 0x7fdff81321a0 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 1 -- 172.21.15.144:0/1052507368 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0032980 con 0x7fdff81321a0 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 5 monclient: authenticate success, global_id 104544 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 4 set_mon_vals no callback set 2024-09-18T12:34:01.187 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdff5d9b640 10 monclient: dump: 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:01.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 10 monclient: shutdown 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 -- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 msgr2=0x7fdff8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 0x7fdff8068d00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009910 tx=0x7fdfe00091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 -- 172.21.15.144:0/1052507368 shutdown_connections 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 0x7fdff8068d00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/1052507368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff8131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 -- 172.21.15.144:0/1052507368 >> 172.21.15.144:0/1052507368 conn(0x7fdff80a5cf0 msgr2=0x7fdff80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 -- 172.21.15.144:0/1052507368 shutdown_connections 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.176+0000 7fdffcd9d640 1 -- 172.21.15.144:0/1052507368 wait complete. 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.177+0000 7fdffcd9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.177+0000 7fdffcd9d640 10 monclient: monmap: 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:01.189 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.177+0000 7fdffcd9d640 1 Processor -- start 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.177+0000 7fdffcd9d640 1 -- start start 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.177+0000 7fdffcd9d640 10 monclient: init 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:01.190 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff812fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdff809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdff809f020 0x7fdff80999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdff809f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 0x7fdff8099ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdff81321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff811ce90 con 0x7fdff81321a0 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff8141f00 con 0x7fdff809f020 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff811cbb0 con 0x7fdff809ec50 2024-09-18T12:34:01.191 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdffcd9d640 10 monclient(hunting): authenticate will time out at 1276.747244s 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdfedd9b640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdff809f020 msgr2=0x7fdff80999a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdfedd9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdff809f020 0x7fdff80999a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff812fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff812fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56698/0 (socket says 172.21.15.144:56698) 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 -- 172.21.15.144:0/284025757 learned_addr learned my addr 172.21.15.144:0/284025757 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): get_auth_request con 0x7fdff809ec50 auth_method 0 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 0x7fdff8099ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 10 monclient(hunting): get_auth_request con 0x7fdff81321a0 auth_method 0 2024-09-18T12:34:01.192 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient(hunting): handle_auth_done global_id 65342 payload 824 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 -- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdff809f020 msgr2=0x7fdff80999a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdff809f020 0x7fdff80999a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 -- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 msgr2=0x7fdff8099ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:01.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 0x7fdff8099ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 monclient: found mon.noname-c 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.178+0000 7fdff659c640 1 -- 172.21.15.144:0/284025757 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe8009390 con 0x7fdff809ec50 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdff659c640 1 -- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 msgr2=0x7fdff812fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdff659c640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff812fa90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800ec80 tx=0x7fdfe80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe800eec0 con 0x7fdff809ec50 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: dump: 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:01.194 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe800cdd0 con 0x7fdff809ec50 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdffcd9d640 5 monclient: authenticate success, global_id 65342 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdffcd9d640 10 monclient: _renew_subs 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdffcd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:01.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe0009450 con 0x7fdff809ec50 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8013680 con 0x7fdff809ec50 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdfeeffd640 10 monclient: dump: 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:01.196 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdffcd9d640 10 monclient: _renew_subs 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdff559a640 4 set_mon_vals no callback set 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdffcd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.181+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff811d220 con 0x7fdff809ec50 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdffcd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdffcd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdffcd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdfe80139f0 con 0x7fdff809ec50 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff811cfd0 con 0x7fdff809ec50 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdfeeffd640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdfd8038440 0x7fdfd803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(245..245 src has 13..245) v4 ==== 10888+0+0 (secure 0 0 0) 0x7fdfe804d3f0 con 0x7fdff809ec50 2024-09-18T12:34:01.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdfedd9b640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdfd8038440 0x7fdfd803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:01.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.182+0000 7fdfedd9b640 10 monclient: get_auth_request con 0x7fdfd8038440 auth_method 0 2024-09-18T12:34:01.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.184+0000 7fdfedd9b640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdfd8038440 0x7fdfd803a8f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0034000 tx=0x7fdfe0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:01.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.187+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdfe80236b0 con 0x7fdff809ec50 2024-09-18T12:34:01.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.187+0000 7fdfeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:01.198 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.187+0000 7fdfeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:01.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.344+0000 7fdffcd9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T12:34:01.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.344+0000 7fdffcd9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T12:34:01.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.344+0000 7fdffcd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:01.346 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:01.344+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7fdff80008d0 con 0x7fdff809ec50 2024-09-18T12:34:02.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.290+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v246) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdfe80232f0 con 0x7fdff809ec50 2024-09-18T12:34:02.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.290+0000 7fdfeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T12:34:02.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.290+0000 7fdfeeffd640 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-18T12:34:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.392+0000 7fdffcd9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T12:34:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.392+0000 7fdffcd9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T12:34:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.392+0000 7fdffcd9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.392+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7fdff80a13e0 con 0x7fdff809ec50 2024-09-18T12:34:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.393+0000 7fdfeeffd640 1 -- 172.21.15.144:0/284025757 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v246) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdfe80234d0 con 0x7fdff809ec50 2024-09-18T12:34:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.393+0000 7fdfeeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T12:34:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.393+0000 7fdfeeffd640 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-18T12:34:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h20' does not appear in the crush map 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdfd8038440 msgr2=0x7fdfd803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdfd8038440 0x7fdfd803a8f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0034000 tx=0x7fdfe0032000 comp rx=0 tx=0).stop 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 10 monclient: shutdown 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 msgr2=0x7fdff812fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff812fa90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800ec80 tx=0x7fdfe80090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 shutdown_connections 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdfd8038440 0x7fdfd803a8f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff81321a0 0x7fdff8099ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdff809f020 0x7fdff80999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 --2- 172.21.15.144:0/284025757 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdff809ec50 0x7fdff812fa90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 >> 172.21.15.144:0/284025757 conn(0x7fdff80a5cf0 msgr2=0x7fdff80a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 shutdown_connections 2024-09-18T12:34:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.395+0000 7fdffcd9d640 1 -- 172.21.15.144:0/284025757 wait complete. 2024-09-18T12:34:02.406 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:02.406 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h21 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.478+0000 7fb777d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.478+0000 7fb777d9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.478+0000 7fb777d9d640 10 monclient: monmap: 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:02.479856+0000 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:02.479856+0000 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:02.480 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:02.481 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:02.481 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:02.481 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:02.483 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.479+0000 7fb777d9d640 1 Processor -- start 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.479+0000 7fb777d9d640 1 -- start start 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.479+0000 7fb777d9d640 10 monclient: init 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77012f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb770069a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb77009baa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb77009bea0 0x7fb770071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb77009bea0 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb770069e00 con 0x7fb77009baa0 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77009d100 con 0x7fb77009bea0 2024-09-18T12:34:02.484 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77009e270 con 0x7fb770069a30 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb777d9d640 10 monclient(hunting): authenticate will time out at 1278.049359s 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775338640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fb77009baa0 legacy=0x7fb770070920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77012f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb77009bea0 0x7fb770071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77012f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56714/0 (socket says 172.21.15.144:56714) 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb77009bea0 0x7fb770071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:46430/0 (socket says 172.21.15.144:46430) 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 1 -- 172.21.15.144:0/1865045354 learned_addr learned my addr 172.21.15.144:0/1865045354 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 10 monclient(hunting): get_auth_request con 0x7fb770069a30 auth_method 0 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:02.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 10 monclient(hunting): get_auth_request con 0x7fb77009bea0 auth_method 0 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb775b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.480+0000 7fb77633a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 10 monclient(hunting): handle_auth_done global_id 65345 payload 274 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 1 -- 172.21.15.144:0/1865045354 >> v2:172.21.15.116:3300/0 conn(0x7fb77009bea0 msgr2=0x7fb770071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:02.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 1 --2- 172.21.15.144:0/1865045354 >> v2:172.21.15.116:3300/0 conn(0x7fb77009bea0 0x7fb770071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 1 -- 172.21.15.144:0/1865045354 >> v1:172.21.15.144:6789/0 conn(0x7fb77009baa0 legacy=0x7fb770070920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:02.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 10 monclient: _finish_hunting 0 2024-09-18T12:34:02.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 1 monclient: found mon.noname-b 2024-09-18T12:34:02.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:02.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:02.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb775b39640 1 -- 172.21.15.144:0/1865045354 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77009dc40 con 0x7fb770069a30 2024-09-18T12:34:02.487 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.481+0000 7fb77633a640 1 --2- 172.21.15.144:0/1865045354 >> v2:172.21.15.116:3300/0 conn(0x7fb77009bea0 0x7fb770071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:02.492 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.483+0000 7fb775b39640 1 -- 172.21.15.144:0/1865045354 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 msgr2=0x7fb77012f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.483+0000 7fb775b39640 1 --2- 172.21.15.144:0/1865045354 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77012f840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009910 tx=0x7fb76c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b77f56d1343f1679 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.483+0000 7fb774b37640 1 -- 172.21.15.144:0/1865045354 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c02ae90 con 0x7fb770069a30 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: dump: 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:02.493 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: _finish_auth 0 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 5 monclient: authenticate success, global_id 65345 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 1 -- 172.21.15.144:0/1865045354 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c032440 con 0x7fb770069a30 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 4 set_mon_vals no callback set 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 1 -- 172.21.15.144:0/1865045354 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0327e0 con 0x7fb770069a30 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:02.494 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb774b37640 10 monclient: dump: 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 10 monclient: shutdown 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 -- 172.21.15.144:0/1865045354 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 msgr2=0x7fb77012f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:02.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 --2- 172.21.15.144:0/1865045354 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77012f840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009910 tx=0x7fb76c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 -- 172.21.15.144:0/1865045354 shutdown_connections 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 --2- 172.21.15.144:0/1865045354 >> v2:172.21.15.116:3300/0 conn(0x7fb77009bea0 0x7fb770071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 --2- 172.21.15.144:0/1865045354 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77012f840 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 -- 172.21.15.144:0/1865045354 >> 172.21.15.144:0/1865045354 conn(0x7fb77009e9f0 msgr2=0x7fb77009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 -- 172.21.15.144:0/1865045354 shutdown_connections 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.484+0000 7fb777d9d640 1 -- 172.21.15.144:0/1865045354 wait complete. 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient: monmap: 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:02.496 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 1 Processor -- start 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 1 -- start start 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient: init 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77013ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb770069a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:02.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb77009baa0 0x7fb77012ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb77009baa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77009bea0 0x7fb770118800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.485+0000 7fb777d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb77009bea0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77009dea0 con 0x7fb77009bea0 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb770132920 con 0x7fb77009baa0 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77011bd00 con 0x7fb770069a30 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775338640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb77009baa0 msgr2=0x7fb77012ebe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb777d9d640 10 monclient(hunting): authenticate will time out at 1278.055025s 2024-09-18T12:34:02.498 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775338640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb77009baa0 0x7fb77012ebe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77013ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77009bea0 0x7fb770118800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77013ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56724/0 (socket says 172.21.15.144:56724) 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 1 -- 172.21.15.144:0/2852465059 learned_addr learned my addr 172.21.15.144:0/2852465059 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 10 monclient(hunting): get_auth_request con 0x7fb770069a30 auth_method 0 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): get_auth_request con 0x7fb77009bea0 auth_method 0 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:02.499 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient(hunting): handle_auth_done global_id 104553 payload 824 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 1 -- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 msgr2=0x7fb77013ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77013ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 1 -- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb77009baa0 msgr2=0x7fb77012ebe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb77009baa0 0x7fb77012ebe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient: _finish_hunting 0 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 1 monclient: found mon.noname-a 2024-09-18T12:34:02.500 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb77633a640 1 -- 172.21.15.144:0/2852465059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb768009a20 con 0x7fb77009bea0 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.486+0000 7fb775b39640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77013ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.488+0000 7fb77633a640 1 -- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77009bea0 msgr2=0x7fb770118800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.488+0000 7fb77633a640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77009bea0 0x7fb770118800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb768009f10 tx=0x7fb76800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768014480 con 0x7fb77009bea0 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: dump: 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:02.501 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76800f040 con 0x7fb77009bea0 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb777d9d640 5 monclient: authenticate success, global_id 104553 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768014850 con 0x7fb77009bea0 2024-09-18T12:34:02.502 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb765ffb640 10 monclient: dump: 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb777d9d640 10 monclient: _renew_subs 2024-09-18T12:34:02.503 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb777d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb767fff640 4 set_mon_vals no callback set 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c009450 con 0x7fb77009bea0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb777d9d640 10 monclient: _renew_subs 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb777d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.489+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77013fc30 con 0x7fb77009bea0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb768003020 con 0x7fb77009bea0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb777d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb777d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb777d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77011c180 con 0x7fb77009bea0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb765ffb640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb754038440 0x7fb75403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(246..246 src has 13..246) v4 ==== 10675+0+0 (secure 0 0 0) 0x7fb76804d8d0 con 0x7fb77009bea0 2024-09-18T12:34:02.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb775b39640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb754038440 0x7fb75403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:02.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.490+0000 7fb775b39640 10 monclient: get_auth_request con 0x7fb754038440 auth_method 0 2024-09-18T12:34:02.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.492+0000 7fb775b39640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb754038440 0x7fb75403a8f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009210 tx=0x7fb76c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:02.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.495+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb7680112c0 con 0x7fb77009bea0 2024-09-18T12:34:02.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.495+0000 7fb765ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:02.505 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.495+0000 7fb765ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:02.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.654+0000 7fb777d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T12:34:02.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.654+0000 7fb777d9d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T12:34:02.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.654+0000 7fb777d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:02.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:02.654+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7fb7700008d0 con 0x7fb77009bea0 2024-09-18T12:34:02.918 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:34:02.919 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T12:34:02.919 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T12:34:02.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T12:34:02.922 INFO:tasks.ceph.mon.b:Started 2024-09-18T12:34:02.922 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:34:02.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:34:03.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.004+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v247) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb768012950 con 0x7fb77009bea0 2024-09-18T12:34:03.005 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.004+0000 7fb765ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T12:34:03.006 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.004+0000 7fb765ffb640 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.028+0000 7fe58b309640 10 monclient: get_monmap_and_config 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.028+0000 7fe58b309640 10 monclient: build_initial_monmap 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.028+0000 7fe58b309640 10 monclient: monmap: 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:03.028861+0000 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:03.028861+0000 2024-09-18T12:34:03.029 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:03.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.028+0000 7fe58b309640 1 Processor -- start 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 1 -- start start 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient: init 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5841318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe58409ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 0x7fe58413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient(hunting): picked mon.noname-c con 0x7fe584137c30 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe584131cc0 con 0x7fe58409ea50 2024-09-18T12:34:03.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58409f530 con 0x7fe5841318f0 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58409fbf0 con 0x7fe584137c30 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe58b309640 10 monclient(hunting): authenticate will time out at 1270.350491s 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5888a4640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe58409ea50 0x7fe58409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59612/0 (socket says 172.21.15.116:59612) 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5888a4640 1 -- 172.21.15.116:0/1542712519 learned_addr learned my addr 172.21.15.116:0/1542712519 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 1 --2- 172.21.15.116:0/1542712519 >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 0x7fe58413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 10 monclient(hunting): get_auth_request con 0x7fe584137c30 auth_method 0 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 1 -- 172.21.15.116:0/1542712519 >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 msgr2=0x7fe58413a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 1 -- 172.21.15.116:0/1542712519 >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 msgr2=0x7fe58413a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 1 --2- 172.21.15.116:0/1542712519 >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 0x7fe58413a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.029+0000 7fe5898a6640 1 --2- 172.21.15.116:0/1542712519 >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 0x7fe58413a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.031+0000 7fe5888a4640 1 -- 172.21.15.116:0/1542712519 >> v1:172.21.15.116:6789/0 conn(0x7fe58409ea50 legacy=0x7fe58409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.031+0000 7fe5890a5640 1 -- 172.21.15.116:0/1542712519 >> v1:172.21.15.144:6789/0 conn(0x7fe5841318f0 legacy=0x7fe584131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.031+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368118440 0 0) 0x7fe584131cc0 con 0x7fe58409ea50 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): my global_id is 104556 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5700034e0 con 0x7fe58409ea50 2024-09-18T12:34:03.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116851866 0 0) 0x7fe5700034e0 con 0x7fe58409ea50 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 msgr2=0x7fe58413a010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 --2- 172.21.15.116:0/1542712519 >> v2:172.21.15.144:3300/0 conn(0x7fe584137c30 0x7fe58413a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 >> v1:172.21.15.144:6789/0 conn(0x7fe5841318f0 legacy=0x7fe584131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58409fe30 con 0x7fe58409ea50 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: _finish_auth 0 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fe574001860 con 0x7fe58409ea50 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe58b309640 5 monclient: authenticate success, global_id 104556 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe58b309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:03.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: dump: 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:03.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 >> v1:172.21.15.116:6789/0 conn(0x7fe58409ea50 legacy=0x7fe58409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe570003af0 0x7fe570003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): picked mon.a con 0x7fe570003af0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570004780 0x7fe570004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): picked mon.b con 0x7fe570004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe570005380 0x7fe57000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): picked mon.c con 0x7fe570005380 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58409fbf0 con 0x7fe570003af0 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57000c350 con 0x7fe570004780 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57000c510 con 0x7fe570005380 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe57bfff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe58b309640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe570003af0 0x7fe570003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): get_auth_request con 0x7fe570003af0 auth_method 0 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe570005380 0x7fe57000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570004780 0x7fe570004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 10 monclient(hunting): get_auth_request con 0x7fe570004780 auth_method 0 2024-09-18T12:34:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 10 monclient(hunting): get_auth_request con 0x7fe570005380 auth_method 0 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5888a4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient(hunting): handle_auth_done global_id 104556 payload 293 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 1 -- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe570005380 msgr2=0x7fe57000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe570005380 0x7fe57000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 1 -- 172.21.15.116:0/1542712519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570004780 msgr2=0x7fe570004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570004780 0x7fe570004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient: _finish_hunting 0 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 1 monclient: found mon.a 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5898a6640 1 -- 172.21.15.116:0/1542712519 reap_dead start 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 1 -- 172.21.15.116:0/1542712519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700034e0 con 0x7fe570003af0 2024-09-18T12:34:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 10 monclient: _finish_auth 0 2024-09-18T12:34:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.032+0000 7fe5890a5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe5890a5640 1 -- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe570003af0 msgr2=0x7fe570003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe5890a5640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe570003af0 0x7fe570003f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe580001b10 tx=0x7fe580007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7a3b1a7d96102f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58002ab50 con 0x7fe570003af0 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 10 monclient: dump: 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.051 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe58b309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58002acb0 con 0x7fe570003af0 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe58b309640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe58b309640 4 set_mon_vals no callback set 2024-09-18T12:34:03.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 1 -- 172.21.15.116:0/1542712519 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580007340 con 0x7fe570003af0 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.035+0000 7fe57bfff640 10 monclient: dump: 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 10 monclient: shutdown 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 1 -- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe570003af0 msgr2=0x7fe570003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe570003af0 0x7fe570003f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe580001b10 tx=0x7fe580007c80 comp rx=0 tx=0).stop 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 1 -- 172.21.15.116:0/1542712519 shutdown_connections 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 1 --2- 172.21.15.116:0/1542712519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe570003af0 0x7fe570003f00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 1 -- 172.21.15.116:0/1542712519 >> 172.21.15.116:0/1542712519 conn(0x7fe584067110 msgr2=0x7fe584067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 1 -- 172.21.15.116:0/1542712519 shutdown_connections 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 1 -- 172.21.15.116:0/1542712519 wait complete. 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 10 monclient: build_initial_monmap 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.036+0000 7fe58b309640 10 monclient: monmap: 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 Processor -- start 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 -- start start 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient: init 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 0x7fe584141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): picked mon.noname-b con 0x7fe570003af0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe584141bc0 0x7fe584098820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): picked mon.noname-a con 0x7fe584141bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 0x7fe5840991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): picked mon.noname-c con 0x7fe584098d60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58409de30 con 0x7fe584141bc0 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58411b9e0 con 0x7fe570003af0 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58409a820 con 0x7fe584098d60 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe58b309640 10 monclient(hunting): authenticate will time out at 1270.358849s 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5898a6640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 0x7fe5840991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5888a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe584141bc0 0x7fe584098820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5898a6640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 0x7fe5840991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45720/0 (socket says 172.21.15.116:45720) 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5888a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe584141bc0 0x7fe584098820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36328/0 (socket says 172.21.15.116:36328) 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5898a6640 1 -- 172.21.15.116:0/3091677212 learned_addr learned my addr 172.21.15.116:0/3091677212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5898a6640 10 monclient(hunting): get_auth_request con 0x7fe584098d60 auth_method 0 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5898a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5898a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5898a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5888a4640 10 monclient(hunting): get_auth_request con 0x7fe584141bc0 auth_method 0 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5888a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5888a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.037+0000 7fe5888a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 0x7fe584141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 10 monclient(hunting): get_auth_request con 0x7fe570003af0 auth_method 0 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5888a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5888a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5888a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 1 -- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 msgr2=0x7fe584141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 1 -- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 msgr2=0x7fe584141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 0x7fe584141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5890a5640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 0x7fe584141680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 10 monclient(hunting): handle_auth_done global_id 65351 payload 824 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 1 -- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 msgr2=0x7fe584141680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 0x7fe584141680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 1 -- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe584141bc0 msgr2=0x7fe584098820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe584141bc0 0x7fe584098820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 10 monclient: _finish_hunting 0 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 1 monclient: found mon.noname-c 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5898a6640 1 -- 172.21.15.116:0/3091677212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58409bf10 con 0x7fe584098d60 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.038+0000 7fe5888a4640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe584141bc0 0x7fe584098820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5898a6640 1 -- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 msgr2=0x7fe5840991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5898a6640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 0x7fe5840991b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009590 tx=0x7fe57c009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 1 -- 172.21.15.116:0/3091677212 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c007040 con 0x7fe584098d60 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: dump: 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: _finish_auth 0 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe58b309640 5 monclient: authenticate success, global_id 65351 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 1 -- 172.21.15.116:0/3091677212 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57c00f910 con 0x7fe584098d60 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe58b309640 10 monclient: _renew_subs 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe58b309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58411bfa0 con 0x7fe584098d60 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 1 -- 172.21.15.116:0/3091677212 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c00fcb0 con 0x7fe584098d60 2024-09-18T12:34:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe5797fa640 10 monclient: dump: 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:03.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe57b7fe640 4 set_mon_vals no callback set 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe58b309640 10 monclient: _renew_subs 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe58b309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.040+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58409aee0 con 0x7fe584098d60 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.041+0000 7fe5797fa640 1 -- 172.21.15.116:0/3091677212 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe57c014600 con 0x7fe584098d60 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.041+0000 7fe58b309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.041+0000 7fe58b309640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.041+0000 7fe58b309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.041+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58411bb50 con 0x7fe584098d60 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.041+0000 7fe5797fa640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038490 0x7fe56803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.041+0000 7fe5797fa640 1 -- 172.21.15.116:0/3091677212 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(247..247 src has 13..247) v4 ==== 10462+0+0 (secure 0 0 0) 0x7fe57c04cf80 con 0x7fe584098d60 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.042+0000 7fe5890a5640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038490 0x7fe56803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.042+0000 7fe5890a5640 10 monclient: get_auth_request con 0x7fe568038490 auth_method 0 2024-09-18T12:34:03.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.044+0000 7fe5890a5640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038490 0x7fe56803a940 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe580032000 tx=0x7fe580030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:03.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.046+0000 7fe5797fa640 1 -- 172.21.15.116:0/3091677212 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe57c01a880 con 0x7fe584098d60 2024-09-18T12:34:03.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.046+0000 7fe5797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:03.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.046+0000 7fe5797fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:03.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.101+0000 7fb777d9d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T12:34:03.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.101+0000 7fb777d9d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T12:34:03.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.101+0000 7fb777d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:03.103 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.101+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7fb77009c2c0 con 0x7fb77009bea0 2024-09-18T12:34:03.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.102+0000 7fb765ffb640 1 -- 172.21.15.144:0/2852465059 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v247) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb7680112c0 con 0x7fb77009bea0 2024-09-18T12:34:03.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.102+0000 7fb765ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T12:34:03.104 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.102+0000 7fb765ffb640 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-18T12:34:03.104 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h21' does not appear in the crush map 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb754038440 msgr2=0x7fb75403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb754038440 0x7fb75403a8f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009210 tx=0x7fb76c032000 comp rx=0 tx=0).stop 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 10 monclient: shutdown 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77009bea0 msgr2=0x7fb770118800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77009bea0 0x7fb770118800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb768009f10 tx=0x7fb76800b040 comp rx=0 tx=0).stop 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 shutdown_connections 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb754038440 0x7fb75403a8f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77009bea0 0x7fb770118800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb77009baa0 0x7fb77012ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.106 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 --2- 172.21.15.144:0/2852465059 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb770069a30 0x7fb77013ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 >> 172.21.15.144:0/2852465059 conn(0x7fb77009e9f0 msgr2=0x7fb770132b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:03.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 shutdown_connections 2024-09-18T12:34:03.107 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.104+0000 7fb777d9d640 1 -- 172.21.15.144:0/2852465059 wait complete. 2024-09-18T12:34:03.114 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:03.115 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h22 2024-09-18T12:34:03.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.187+0000 7f1affc34640 10 monclient: get_monmap_and_config 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.187+0000 7f1affc34640 10 monclient: build_initial_monmap 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.187+0000 7f1affc34640 10 monclient: monmap: 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:03.188654+0000 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:03.188654+0000 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:03.189 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:03.190 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:03.190 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:03.190 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:03.190 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:03.190 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 1 Processor -- start 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 1 -- start start 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient: init 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1af809c250 0x7f1af809da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient(hunting): picked mon.noname-a con 0x7f1af809c250 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af809e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient(hunting): picked mon.noname-b con 0x7f1af809dfa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1af809c620 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:03.193 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af809b790 con 0x7f1af809c620 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af809b230 con 0x7f1af809c250 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af809e8b0 con 0x7f1af809dfa0 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.188+0000 7f1affc34640 10 monclient(hunting): authenticate will time out at 1278.757987s 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af809e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1af809c250 0x7f1af809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af809e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56746/0 (socket says 172.21.15.144:56746) 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afe1d1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1af809c620 0x7f1af809ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:45730/0 (socket says 172.21.15.144:45730) 2024-09-18T12:34:03.194 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 -- 172.21.15.144:0/2895201716 learned_addr learned my addr 172.21.15.144:0/2895201716 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): get_auth_request con 0x7f1af809dfa0 auth_method 0 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 10 monclient(hunting): get_auth_request con 0x7f1af809c250 auth_method 0 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.195 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient(hunting): handle_auth_done global_id 65354 payload 274 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 -- 172.21.15.144:0/2895201716 >> v2:172.21.15.116:3300/0 conn(0x7f1af809c250 msgr2=0x7f1af809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 --2- 172.21.15.144:0/2895201716 >> v2:172.21.15.116:3300/0 conn(0x7f1af809c250 0x7f1af809da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 -- 172.21.15.144:0/2895201716 >> v1:172.21.15.116:6789/0 conn(0x7f1af809c620 legacy=0x7f1af809ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient: _finish_hunting 0 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 monclient: found mon.noname-b 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.196 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd1cf640 1 -- 172.21.15.144:0/2895201716 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af809b4b0 con 0x7f1af809dfa0 2024-09-18T12:34:03.197 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.189+0000 7f1afd9d0640 1 --2- 172.21.15.144:0/2895201716 >> v2:172.21.15.116:3300/0 conn(0x7f1af809c250 0x7f1af809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afd1cf640 1 -- 172.21.15.144:0/2895201716 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 msgr2=0x7f1af809e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afd1cf640 1 --2- 172.21.15.144:0/2895201716 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af809e370 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1af80a3fd0 tx=0x7f1ae80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1f8380f167338ca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 1 -- 172.21.15.144:0/2895201716 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae802ae60 con 0x7f1af809dfa0 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: dump: 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.203 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: _finish_auth 0 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 1 -- 172.21.15.144:0/2895201716 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae8032440 con 0x7f1af809dfa0 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1affc34640 5 monclient: authenticate success, global_id 65354 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1affc34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:03.204 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 1 -- 172.21.15.144:0/2895201716 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae80327e0 con 0x7f1af809dfa0 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1affc34640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1affc34640 4 set_mon_vals no callback set 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.192+0000 7f1afc9ce640 10 monclient: dump: 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.205 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 10 monclient: shutdown 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 -- 172.21.15.144:0/2895201716 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 msgr2=0x7f1af809e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 --2- 172.21.15.144:0/2895201716 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af809e370 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1af80a3fd0 tx=0x7f1ae80091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 -- 172.21.15.144:0/2895201716 shutdown_connections 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 --2- 172.21.15.144:0/2895201716 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af809e370 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 --2- 172.21.15.144:0/2895201716 >> v2:172.21.15.116:3300/0 conn(0x7f1af809c250 0x7f1af809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 -- 172.21.15.144:0/2895201716 >> 172.21.15.144:0/2895201716 conn(0x7f1af8071340 msgr2=0x7f1af80a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 -- 172.21.15.144:0/2895201716 shutdown_connections 2024-09-18T12:34:03.206 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.193+0000 7f1affc34640 1 -- 172.21.15.144:0/2895201716 wait complete. 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient: build_initial_monmap 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient: monmap: 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:03.207 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:03.208 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:03.208 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:03.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 Processor -- start 2024-09-18T12:34:03.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 -- start start 2024-09-18T12:34:03.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient: init 2024-09-18T12:34:03.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:03.208 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 0x7f1af813ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): picked mon.noname-b con 0x7f1af809c250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af809c620 0x7f1af8131a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): picked mon.noname-a con 0x7f1af809c620 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af811b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): picked mon.noname-c con 0x7f1af809dfa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af806e970 con 0x7f1af809c620 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af811f1f0 con 0x7f1af809c250 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af811eba0 con 0x7f1af809dfa0 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:03.209 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1affc34640 10 monclient(hunting): authenticate will time out at 1278.763848s 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd9d0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 0x7f1af813ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd9d0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 0x7f1af813ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48336/0 (socket says 172.21.15.144:48336) 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd9d0640 1 -- 172.21.15.144:0/265970398 learned_addr learned my addr 172.21.15.144:0/265970398 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd9d0640 10 monclient(hunting): get_auth_request con 0x7f1af809c250 auth_method 0 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd9d0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afd1cf640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af809c620 0x7f1af8131a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.194+0000 7f1afe1d1640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af811b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.210 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd1cf640 10 monclient(hunting): get_auth_request con 0x7f1af809c620 auth_method 0 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd1cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd1cf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd1cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): get_auth_request con 0x7f1af809dfa0 auth_method 0 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd9d0640 1 -- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 msgr2=0x7f1af813ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd9d0640 1 -- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 msgr2=0x7f1af813ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd9d0640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 0x7f1af813ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:03.211 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd1cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd1cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd1cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afd9d0640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 0x7f1af813ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient(hunting): handle_auth_done global_id 65357 payload 824 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 1 -- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 msgr2=0x7f1af813ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 0x7f1af813ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 1 -- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af809c620 msgr2=0x7f1af8131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af809c620 0x7f1af8131a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient: _finish_hunting 0 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 1 monclient: found mon.noname-c 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.195+0000 7f1afe1d1640 1 -- 172.21.15.144:0/265970398 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af4009a20 con 0x7f1af809dfa0 2024-09-18T12:34:03.212 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1afe1d1640 1 -- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 msgr2=0x7f1af811b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1afe1d1640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af811b660 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c410 tx=0x7f1af0007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af00074f0 con 0x7f1af809dfa0 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: dump: 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.213 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af0007650 con 0x7f1af809dfa0 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0005a20 con 0x7f1af809dfa0 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1affc34640 5 monclient: authenticate success, global_id 65357 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1affc34640 10 monclient: _renew_subs 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1affc34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae8009450 con 0x7f1af809dfa0 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aedffb640 10 monclient: dump: 2024-09-18T12:34:03.214 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.197+0000 7f1aeffff640 4 set_mon_vals no callback set 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1affc34640 10 monclient: _renew_subs 2024-09-18T12:34:03.215 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1affc34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af8132820 con 0x7f1af809dfa0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1af001b3b0 con 0x7f1af809dfa0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1aedffb640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1adc038490 0x7f1adc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1affc34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1affc34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(247..247 src has 13..247) v4 ==== 10462+0+0 (secure 0 0 0) 0x7f1af004c270 con 0x7f1af809dfa0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1affc34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af811edc0 con 0x7f1af809dfa0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1afd9d0640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1adc038490 0x7f1adc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.198+0000 7f1afd9d0640 10 monclient: get_auth_request con 0x7f1adc038490 auth_method 0 2024-09-18T12:34:03.216 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.201+0000 7f1afd9d0640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1adc038490 0x7f1adc03a940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1af4009cd0 tx=0x7f1af400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:03.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.203+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1af0017070 con 0x7f1af809dfa0 2024-09-18T12:34:03.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.203+0000 7f1aedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:03.217 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.203+0000 7f1aedffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.262+0000 7fe58b309640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:34:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.262+0000 7fe58b309640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.262+0000 7fe58b309640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.262+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe58404bf60 con 0x7fe584098d60 2024-09-18T12:34:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.262+0000 7fe5797fa640 1 -- 172.21.15.116:0/3091677212 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fe57c01a2b0 con 0x7fe584098d60 2024-09-18T12:34:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.263+0000 7fe5797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.263+0000 7fe5797fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:03.264 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.264+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038490 msgr2=0x7fe56803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.264+0000 7fe58b309640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038490 0x7fe56803a940 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe580032000 tx=0x7fe580030000 comp rx=0 tx=0).stop 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.264+0000 7fe58b309640 10 monclient: shutdown 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.264+0000 7fe58b309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.264+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 msgr2=0x7fe5840991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.264+0000 7fe58b309640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 0x7fe5840991b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009590 tx=0x7fe57c009720 comp rx=0 tx=0).stop 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 shutdown_connections 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe568038490 0x7fe56803a940 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe584098d60 0x7fe5840991b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe584141bc0 0x7fe584098820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 --2- 172.21.15.116:0/3091677212 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe570003af0 0x7fe584141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 >> 172.21.15.116:0/3091677212 conn(0x7fe584067110 msgr2=0x7fe5840a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 shutdown_connections 2024-09-18T12:34:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:03.265+0000 7fe58b309640 1 -- 172.21.15.116:0/3091677212 wait complete. 2024-09-18T12:34:03.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.362+0000 7f1affc34640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T12:34:03.363 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.362+0000 7f1affc34640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T12:34:03.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.362+0000 7f1affc34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:03.364 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:03.362+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f1af80008d0 con 0x7f1af809dfa0 2024-09-18T12:34:04.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.431+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 removed item id -82 name 'h22' from crush map v248) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1af001a360 con 0x7f1af809dfa0 2024-09-18T12:34:04.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.431+0000 7f1aedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T12:34:04.433 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.431+0000 7f1aedffb640 10 monclient: _finish_command 2 = system:0 removed item id -82 name 'h22' from crush map 2024-09-18T12:34:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.532+0000 7f1affc34640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T12:34:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.532+0000 7f1affc34640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T12:34:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.532+0000 7f1affc34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:04.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.532+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7f1af8072640 con 0x7f1af809dfa0 2024-09-18T12:34:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.534+0000 7f1aedffb640 1 -- 172.21.15.144:0/265970398 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v248) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1af000f260 con 0x7f1af809dfa0 2024-09-18T12:34:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.534+0000 7f1aedffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T12:34:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.534+0000 7f1aedffb640 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-18T12:34:04.535 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h22' does not appear in the crush map 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1adc038490 msgr2=0x7f1adc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1adc038490 0x7f1adc03a940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1af4009cd0 tx=0x7f1af400b040 comp rx=0 tx=0).stop 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 10 monclient: shutdown 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 msgr2=0x7f1af811b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af811b660 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c410 tx=0x7f1af0007e40 comp rx=0 tx=0).stop 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 shutdown_connections 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1adc038490 0x7f1adc03a940 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1af809dfa0 0x7f1af811b660 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1af809c620 0x7f1af8131a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 --2- 172.21.15.144:0/265970398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1af809c250 0x7f1af813ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 >> 172.21.15.144:0/265970398 conn(0x7f1af8071340 msgr2=0x7f1af8132af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 shutdown_connections 2024-09-18T12:34:04.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.536+0000 7f1affc34640 1 -- 172.21.15.144:0/265970398 wait complete. 2024-09-18T12:34:04.547 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:04.547 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h23 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.619+0000 7f965464f640 10 monclient: get_monmap_and_config 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.619+0000 7f965464f640 10 monclient: build_initial_monmap 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.619+0000 7f965464f640 10 monclient: monmap: 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:04.620796+0000 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:04.620796+0000 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:04.621 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:04.622 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:04.622 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:04.622 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:04.622 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.620+0000 7f965464f640 1 Processor -- start 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.620+0000 7f965464f640 1 -- start start 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.620+0000 7f965464f640 10 monclient: init 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f964c099130 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f964c06a790 0x7f964c099500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:04.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient(hunting): picked mon.noname-a con 0x7f964c06a790 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 0x7f964c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient(hunting): picked mon.noname-b con 0x7f964c09e170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c069770 con 0x7f964c099130 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964c09a080 con 0x7f964c06a790 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06a040 con 0x7f964c09e170 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f965464f640 10 monclient(hunting): authenticate will time out at 1280.190310s 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 0x7f964c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f964c06a790 0x7f964c099500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 0x7f964c09e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56804/0 (socket says 172.21.15.144:56804) 2024-09-18T12:34:04.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 -- 172.21.15.144:0/2118634910 learned_addr learned my addr 172.21.15.144:0/2118634910 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): get_auth_request con 0x7f964c09e170 auth_method 0 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 10 monclient(hunting): get_auth_request con 0x7f964c06a790 auth_method 0 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:04.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient(hunting): handle_auth_done global_id 65360 payload 274 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 -- 172.21.15.144:0/2118634910 >> v2:172.21.15.116:3300/0 conn(0x7f964c06a790 msgr2=0x7f964c099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 --2- 172.21.15.144:0/2118634910 >> v2:172.21.15.116:3300/0 conn(0x7f964c06a790 0x7f964c099500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 -- 172.21.15.144:0/2118634910 >> v1:172.21.15.116:6789/0 conn(0x7f964c099130 legacy=0x7f964c09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient: _finish_hunting 0 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 monclient: found mon.noname-b 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:04.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:04.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9652bec640 1 -- 172.21.15.144:0/2118634910 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c09f5d0 con 0x7f964c09e170 2024-09-18T12:34:04.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.621+0000 7f9651bea640 1 --2- 172.21.15.144:0/2118634910 >> v2:172.21.15.116:3300/0 conn(0x7f964c06a790 0x7f964c099500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:04.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.624+0000 7f9652bec640 1 -- 172.21.15.144:0/2118634910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 msgr2=0x7f964c09e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:04.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.624+0000 7f9652bec640 1 --2- 172.21.15.144:0/2118634910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 0x7f964c09e5d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9644009f20 tx=0x7f9644009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4fa6939b43f071 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:04.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 1 -- 172.21.15.144:0/2118634910 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644007610 con 0x7f964c09e170 2024-09-18T12:34:04.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:04.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:04.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: dump: 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: _finish_auth 0 2024-09-18T12:34:04.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 5 monclient: authenticate success, global_id 65360 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 1 -- 172.21.15.144:0/2118634910 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964402d050 con 0x7f964c09e170 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 4 set_mon_vals no callback set 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 1 -- 172.21.15.144:0/2118634910 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96440079e0 con 0x7f964c09e170 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f96513e9640 10 monclient: dump: 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:04.637 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 10 monclient: shutdown 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 -- 172.21.15.144:0/2118634910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 msgr2=0x7f964c09e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 --2- 172.21.15.144:0/2118634910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 0x7f964c09e5d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9644009f20 tx=0x7f9644009840 comp rx=0 tx=0).stop 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 -- 172.21.15.144:0/2118634910 shutdown_connections 2024-09-18T12:34:04.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 --2- 172.21.15.144:0/2118634910 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c09e170 0x7f964c09e5d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 --2- 172.21.15.144:0/2118634910 >> v2:172.21.15.116:3300/0 conn(0x7f964c06a790 0x7f964c099500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 -- 172.21.15.144:0/2118634910 >> 172.21.15.144:0/2118634910 conn(0x7f964c071340 msgr2=0x7f964c0a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 -- 172.21.15.144:0/2118634910 shutdown_connections 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.625+0000 7f965464f640 1 -- 172.21.15.144:0/2118634910 wait complete. 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient: build_initial_monmap 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient: monmap: 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:04.639 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 Processor -- start 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 -- start start 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient: init 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c06a790 0x7f964c134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): picked mon.noname-c con 0x7f964c06a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f964c099130 0x7f964c11e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): picked mon.noname-a con 0x7f964c099130 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 0x7f964c11ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:04.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): picked mon.noname-b con 0x7f964c09e170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0a2580 con 0x7f964c099130 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06eba0 con 0x7f964c09e170 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c120740 con 0x7f964c06a790 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.626+0000 7f965464f640 10 monclient(hunting): authenticate will time out at 1280.195980s 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 0x7f964c11ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 0x7f964c11ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48344/0 (socket says 172.21.15.144:48344) 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 1 -- 172.21.15.144:0/2733713573 learned_addr learned my addr 172.21.15.144:0/2733713573 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:04.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 10 monclient(hunting): get_auth_request con 0x7f964c09e170 auth_method 0 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c06a790 0x7f964c134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 1 -- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 msgr2=0x7f964c11ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 1 -- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 msgr2=0x7f964c11ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 0x7f964c11ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f964c099130 0x7f964c11e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 10 monclient(hunting): get_auth_request con 0x7f964c06a790 auth_method 0 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:04.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9652bec640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 0x7f964c11ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): get_auth_request con 0x7f964c099130 auth_method 0 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient(hunting): handle_auth_done global_id 104571 payload 824 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 1 -- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c06a790 msgr2=0x7f964c134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c06a790 0x7f964c134bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 1 -- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 msgr2=0x7f964c11ec70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 0x7f964c11ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient: _finish_hunting 0 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 1 monclient: found mon.noname-a 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f9651bea640 1 -- 172.21.15.144:0/2733713573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c009390 con 0x7f964c099130 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.627+0000 7f96523eb640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c06a790 0x7f964c134bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.629+0000 7f9651bea640 1 -- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f964c099130 msgr2=0x7f964c11e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.629+0000 7f9651bea640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f964c099130 0x7f964c11e730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f963c00ea50 tx=0x7f963c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c009670 con 0x7f964c099130 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:04.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: dump: 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:04.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 5 monclient: authenticate success, global_id 104571 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963c00cc20 con 0x7f964c099130 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 10 monclient: _renew_subs 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9644009ae0 con 0x7f964c099130 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963c013470 con 0x7f964c099130 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f96427fc640 10 monclient: dump: 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.646 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 10 monclient: _renew_subs 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f9650be8640 4 set_mon_vals no callback set 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c120230 con 0x7f964c099130 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:04.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.630+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c0008d0 con 0x7f964c099130 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.631+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f963c013820 con 0x7f964c099130 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.631+0000 7f96427fc640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9630038490 0x7f963003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.631+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(248..248 src has 13..248) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f963c04d140 con 0x7f964c099130 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.631+0000 7f96523eb640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9630038490 0x7f963003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.631+0000 7f96523eb640 10 monclient: get_auth_request con 0x7f9630038490 auth_method 0 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.633+0000 7f96523eb640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9630038490 0x7f963003a940 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9648007fd0 tx=0x7f9648007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.636+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f963c0236b0 con 0x7f964c099130 2024-09-18T12:34:04.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.636+0000 7f96427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:04.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.636+0000 7f96427fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:04.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.795+0000 7f965464f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T12:34:04.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.795+0000 7f965464f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T12:34:04.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.795+0000 7f965464f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:04.868 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:04.795+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7f964c064a20 con 0x7f964c099130 2024-09-18T12:34:05.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.438+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v249) v1 ==== 120+0+0 (secure 0 0 0) 0x7f963c0232f0 con 0x7f964c099130 2024-09-18T12:34:05.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.438+0000 7f96427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T12:34:05.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.438+0000 7f96427fc640 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-18T12:34:05.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.537+0000 7f965464f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T12:34:05.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.537+0000 7f965464f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T12:34:05.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.537+0000 7f965464f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:05.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.537+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7f964c120e00 con 0x7f964c099130 2024-09-18T12:34:05.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.538+0000 7f96427fc640 1 -- 172.21.15.144:0/2733713573 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v249) v1 ==== 120+0+0 (secure 0 0 0) 0x7f963c0234d0 con 0x7f964c099130 2024-09-18T12:34:05.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.538+0000 7f96427fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T12:34:05.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.538+0000 7f96427fc640 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-18T12:34:05.539 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h23' does not appear in the crush map 2024-09-18T12:34:05.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9630038490 msgr2=0x7f963003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:05.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9630038490 0x7f963003a940 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9648007fd0 tx=0x7f9648007e10 comp rx=0 tx=0).stop 2024-09-18T12:34:05.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 10 monclient: shutdown 2024-09-18T12:34:05.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:05.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f964c099130 msgr2=0x7f964c11e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f964c099130 0x7f964c11e730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f963c00ea50 tx=0x7f963c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 shutdown_connections 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9630038490 0x7f963003a940 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f964c09e170 0x7f964c11ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f964c099130 0x7f964c11e730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 --2- 172.21.15.144:0/2733713573 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f964c06a790 0x7f964c134bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 >> 172.21.15.144:0/2733713573 conn(0x7f964c071340 msgr2=0x7f964c06d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 shutdown_connections 2024-09-18T12:34:05.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.540+0000 7f965464f640 1 -- 172.21.15.144:0/2733713573 wait complete. 2024-09-18T12:34:05.550 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:05.550 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h24 2024-09-18T12:34:05.623 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.622+0000 7f67ca55d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.622+0000 7f67ca55d640 10 monclient: build_initial_monmap 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.622+0000 7f67ca55d640 10 monclient: monmap: 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:05.623808+0000 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:05.623808+0000 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:05.624 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:05.625 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:05.625 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:05.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.623+0000 7f67ca55d640 1 Processor -- start 2024-09-18T12:34:05.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.623+0000 7f67ca55d640 1 -- start start 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.623+0000 7f67ca55d640 10 monclient: init 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 0x7f67c4099500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient(hunting): picked mon.noname-c con 0x7f67c4099130 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67c406a790 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67c409e170 0x7f67c409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient(hunting): picked mon.noname-a con 0x7f67c409e170 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c4069770 con 0x7f67c406a790 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c409a080 con 0x7f67c409e170 2024-09-18T12:34:05.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c406a040 con 0x7f67c4099130 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67ca55d640 10 monclient(hunting): authenticate will time out at 1281.193212s 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 0x7f67c4099500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c37fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f67c406a790 0x7f67c409da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:56830/0 (socket says 172.21.15.144:56830) 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c37fe640 1 -- 172.21.15.144:0/3924894304 learned_addr learned my addr 172.21.15.144:0/3924894304 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 0x7f67c4099500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48350/0 (socket says 172.21.15.144:48350) 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 10 monclient(hunting): get_auth_request con 0x7f67c4099130 auth_method 0 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 1 -- 172.21.15.144:0/3924894304 >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 msgr2=0x7f67c4099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:34:05.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 1 -- 172.21.15.144:0/3924894304 >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 msgr2=0x7f67c4099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 1 --2- 172.21.15.144:0/3924894304 >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 0x7f67c4099500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c3fff640 1 --2- 172.21.15.144:0/3924894304 >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 0x7f67c4099500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 1 --2- 172.21.15.144:0/3924894304 >> v2:172.21.15.116:3300/0 conn(0x7f67c409e170 0x7f67c409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): get_auth_request con 0x7f67c409e170 auth_method 0 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient(hunting): handle_auth_done global_id 104574 payload 274 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 1 -- 172.21.15.144:0/3924894304 >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 msgr2=0x7f67c4099500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:05.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 1 --2- 172.21.15.144:0/3924894304 >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 0x7f67c4099500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 1 -- 172.21.15.144:0/3924894304 >> v1:172.21.15.144:6789/0 conn(0x7f67c406a790 legacy=0x7f67c409da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:05.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient: _finish_hunting 0 2024-09-18T12:34:05.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 1 monclient: found mon.noname-a 2024-09-18T12:34:05.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:05.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:05.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.624+0000 7f67c8afa640 1 -- 172.21.15.144:0/3924894304 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c409f5d0 con 0x7f67c409e170 2024-09-18T12:34:05.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.627+0000 7f67c8afa640 1 -- 172.21.15.144:0/3924894304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c409e170 msgr2=0x7f67c409e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.627+0000 7f67c8afa640 1 --2- 172.21.15.144:0/3924894304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c409e170 0x7f67c409e5d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67c409fac0 tx=0x7f67bc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36246fc84bbca44b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 1 -- 172.21.15.144:0/3924894304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc007760 con 0x7f67c409e170 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: dump: 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:05.638 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 1 -- 172.21.15.144:0/3924894304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc02d050 con 0x7f67c409e170 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 5 monclient: authenticate success, global_id 104574 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 1 -- 172.21.15.144:0/3924894304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc007af0 con 0x7f67c409e170 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:05.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 4 set_mon_vals no callback set 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67c2ffd640 10 monclient: dump: 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:05.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 10 monclient: shutdown 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 -- 172.21.15.144:0/3924894304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c409e170 msgr2=0x7f67c409e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 --2- 172.21.15.144:0/3924894304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c409e170 0x7f67c409e5d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67c409fac0 tx=0x7f67bc009780 comp rx=0 tx=0).stop 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 -- 172.21.15.144:0/3924894304 shutdown_connections 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 --2- 172.21.15.144:0/3924894304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c409e170 0x7f67c409e5d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 --2- 172.21.15.144:0/3924894304 >> v2:172.21.15.144:3300/0 conn(0x7f67c4099130 0x7f67c4099500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 -- 172.21.15.144:0/3924894304 >> 172.21.15.144:0/3924894304 conn(0x7f67c4071340 msgr2=0x7f67c40a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 -- 172.21.15.144:0/3924894304 shutdown_connections 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.628+0000 7f67ca55d640 1 -- 172.21.15.144:0/3924894304 wait complete. 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.629+0000 7f67ca55d640 10 monclient: build_initial_monmap 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.629+0000 7f67ca55d640 10 monclient: monmap: 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:05.641 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.629+0000 7f67ca55d640 1 Processor -- start 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.629+0000 7f67ca55d640 1 -- start start 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.629+0000 7f67ca55d640 10 monclient: init 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:05.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67c406a790 0x7f67c413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): picked mon.noname-c con 0x7f67c406a790 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c4099130 0x7f67c413ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): picked mon.noname-a con 0x7f67c4099130 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 0x7f67c4140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): picked mon.noname-b con 0x7f67c409e170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c41204f0 con 0x7f67c4099130 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c413e2e0 con 0x7f67c409e170 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c41221e0 con 0x7f67c406a790 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:05.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67ca55d640 10 monclient(hunting): authenticate will time out at 1281.199196s 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 0x7f67c4140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 0x7f67c4140d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48362/0 (socket says 172.21.15.144:48362) 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 1 -- 172.21.15.144:0/2288542263 learned_addr learned my addr 172.21.15.144:0/2288542263 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 10 monclient(hunting): get_auth_request con 0x7f67c409e170 auth_method 0 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67c406a790 0x7f67c413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 10 monclient(hunting): get_auth_request con 0x7f67c406a790 auth_method 0 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:05.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c4099130 0x7f67c413ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): get_auth_request con 0x7f67c4099130 auth_method 0 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 1 -- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 msgr2=0x7f67c4140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 1 -- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 msgr2=0x7f67c4140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 0x7f67c4140d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c8afa640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 0x7f67c4140d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:05.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient(hunting): handle_auth_done global_id 104577 payload 824 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 1 -- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67c406a790 msgr2=0x7f67c413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67c406a790 0x7f67c413d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 1 -- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 msgr2=0x7f67c4140d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 0x7f67c4140d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 1 monclient: found mon.noname-a 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c37fe640 1 -- 172.21.15.144:0/2288542263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc009a20 con 0x7f67c4099130 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.630+0000 7f67c3fff640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67c406a790 0x7f67c413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:05.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.632+0000 7f67c37fe640 1 -- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c4099130 msgr2=0x7f67c413ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.632+0000 7f67c37fe640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c4099130 0x7f67c413ce60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f67b800b490 tx=0x7f67b8006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8011840 con 0x7f67c4099130 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: dump: 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:05.647 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: _finish_auth 0 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b80119c0 con 0x7f67c4099130 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67ca55d640 5 monclient: authenticate success, global_id 104577 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67ca55d640 10 monclient: _renew_subs 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8011d80 con 0x7f67c4099130 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67ca55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c4121f00 con 0x7f67c4099130 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:05.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67abfff640 10 monclient: dump: 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67c27fc640 4 set_mon_vals no callback set 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67ca55d640 10 monclient: _renew_subs 2024-09-18T12:34:05.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67ca55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.633+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c413d540 con 0x7f67c4099130 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f67b8010590 con 0x7f67c4099130 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67ca55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67ca55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67ca55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c41223a0 con 0x7f67c4099130 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67abfff640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67b0038490 0x7f67b003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(249..249 src has 13..249) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f67b804c970 con 0x7f67c4099130 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67c3fff640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67b0038490 0x7f67b003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.634+0000 7f67c3fff640 10 monclient: get_auth_request con 0x7f67b0038490 auth_method 0 2024-09-18T12:34:05.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.636+0000 7f67c3fff640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67b0038490 0x7f67b003a940 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f67b4006fd0 tx=0x7f67b4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:05.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.639+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f67b80193a0 con 0x7f67c4099130 2024-09-18T12:34:05.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.639+0000 7f67abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:05.651 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.639+0000 7f67abfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:05.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.797+0000 7f67ca55d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T12:34:05.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.797+0000 7f67ca55d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T12:34:05.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.797+0000 7f67ca55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:05.798 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:05.797+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7f67c40008d0 con 0x7f67c4099130 2024-09-18T12:34:06.276 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:34:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.379+0000 7f1850f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.379+0000 7f1850f9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.379+0000 7f1850f9d640 10 monclient: monmap: 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:06.379686+0000 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:06.379686+0000 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:06.381 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.379+0000 7f1850f9d640 1 Processor -- start 2024-09-18T12:34:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.379+0000 7f1850f9d640 1 -- start start 2024-09-18T12:34:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.379+0000 7f1850f9d640 10 monclient: init 2024-09-18T12:34:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T12:34:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f184c096e10 0x7f184c09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f184c096e10 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f184c09aaa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f184c0978f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c09d4a0 con 0x7f184c0978f0 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c099620 con 0x7f184c09aaa0 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c099340 con 0x7f184c096e10 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f1850f9d640 10 monclient(hunting): authenticate will time out at 1273.701426s 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f184c096e10 0x7f184c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184ad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f184c0978f0 0x7f184c135b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46480/0 (socket says 172.21.15.116:46480) 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184ad9d640 1 -- 172.21.15.116:0/2157614062 learned_addr learned my addr 172.21.15.116:0/2157614062 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:34:06.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): get_auth_request con 0x7f184c096e10 auth_method 0 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient(hunting): handle_auth_done global_id 104580 payload 274 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 1 -- 172.21.15.116:0/2157614062 >> v1:172.21.15.144:6789/0 conn(0x7f184c09aaa0 legacy=0x7f184c0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 1 -- 172.21.15.116:0/2157614062 >> v1:172.21.15.116:6789/0 conn(0x7f184c0978f0 legacy=0x7f184c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 1 monclient: found mon.noname-a 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:06.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.380+0000 7f184a59c640 1 -- 172.21.15.116:0/2157614062 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c099900 con 0x7f184c096e10 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184a59c640 1 -- 172.21.15.116:0/2157614062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c096e10 msgr2=0x7f184c09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184a59c640 1 --2- 172.21.15.116:0/2157614062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c096e10 0x7f184c09a560 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f184c076be0 tx=0x7f183c00b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf7abf31c97bb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 1 -- 172.21.15.116:0/2157614062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c031800 con 0x7f184c096e10 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: dump: 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:06.391 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: _finish_auth 0 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f1850f9d640 5 monclient: authenticate success, global_id 104580 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 1 -- 172.21.15.116:0/2157614062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c02f050 con 0x7f184c096e10 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f1850f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:06.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f1850f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f1850f9d640 4 set_mon_vals no callback set 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 1 -- 172.21.15.116:0/2157614062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c031b90 con 0x7f184c096e10 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.383+0000 7f184959a640 10 monclient: dump: 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.393 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 10 monclient: shutdown 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 1 -- 172.21.15.116:0/2157614062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c096e10 msgr2=0x7f184c09a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 1 --2- 172.21.15.116:0/2157614062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c096e10 0x7f184c09a560 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f184c076be0 tx=0x7f183c00b3b0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 1 -- 172.21.15.116:0/2157614062 shutdown_connections 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 1 --2- 172.21.15.116:0/2157614062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c096e10 0x7f184c09a560 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 1 -- 172.21.15.116:0/2157614062 >> 172.21.15.116:0/2157614062 conn(0x7f184c0672b0 msgr2=0x7f184c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 1 -- 172.21.15.116:0/2157614062 shutdown_connections 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.384+0000 7f1850f9d640 1 -- 172.21.15.116:0/2157614062 wait complete. 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:06.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient: monmap: 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 Processor -- start 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 -- start start 2024-09-18T12:34:06.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient: init 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184c096e10 0x7f184c11cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f184c096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184c0978f0 0x7f184c11d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f184c0978f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 0x7f184c11d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f184c09aaa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c11ec10 con 0x7f184c09aaa0 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c120e10 con 0x7f184c096e10 2024-09-18T12:34:06.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c09b050 con 0x7f184c0978f0 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1850f9d640 10 monclient(hunting): authenticate will time out at 1273.706845s 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f184ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 0x7f184c11d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1849d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184c0978f0 0x7f184c11d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f184ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 0x7f184c11d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45832/0 (socket says 172.21.15.116:45832) 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f184ad9d640 1 -- 172.21.15.116:0/383080704 learned_addr learned my addr 172.21.15.116:0/383080704 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f184ad9d640 10 monclient(hunting): get_auth_request con 0x7f184c09aaa0 auth_method 0 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f184ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f184ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f184ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1849d9b640 10 monclient(hunting): get_auth_request con 0x7f184c0978f0 auth_method 0 2024-09-18T12:34:06.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1849d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1849d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.385+0000 7f1849d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f1849d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f1849d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f1849d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184a59c640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184c096e10 0x7f184c11cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 10 monclient(hunting): handle_auth_done global_id 104583 payload 824 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 1 -- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184c0978f0 msgr2=0x7f184c11d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184c0978f0 0x7f184c11d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 1 -- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184c096e10 msgr2=0x7f184c11cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184c096e10 0x7f184c11cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 1 monclient: found mon.noname-a 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184ad9d640 1 -- 172.21.15.116:0/383080704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c00b630 con 0x7f184c09aaa0 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184a59c640 10 monclient: get_auth_request con 0x7f184c096e10 auth_method 0 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.386+0000 7f184a59c640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184c096e10 0x7f184c11cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f184ad9d640 1 -- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 msgr2=0x7f184c11d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f184ad9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 0x7f184c11d560 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f184400ccc0 tx=0x7f1844005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 1 -- 172.21.15.116:0/383080704 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184400ce80 con 0x7f184c09aaa0 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: dump: 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:06.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 1 -- 172.21.15.116:0/383080704 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184400aa60 con 0x7f184c09aaa0 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1850f9d640 5 monclient: authenticate success, global_id 104583 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1850f9d640 10 monclient: _renew_subs 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1850f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c130720 con 0x7f184c09aaa0 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 1 -- 172.21.15.116:0/383080704 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184400ae20 con 0x7f184c09aaa0 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f183a7fc640 10 monclient: dump: 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.401 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1848d99640 4 set_mon_vals no callback set 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1850f9d640 10 monclient: _renew_subs 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1850f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.388+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c120200 con 0x7f184c09aaa0 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.389+0000 7f183a7fc640 1 -- 172.21.15.116:0/383080704 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1844007300 con 0x7f184c09aaa0 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.389+0000 7f1850f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.389+0000 7f1850f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.389+0000 7f1850f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.389+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c04bf60 con 0x7f184c09aaa0 2024-09-18T12:34:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.389+0000 7f183a7fc640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f182c038490 0x7f182c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.389+0000 7f183a7fc640 1 -- 172.21.15.116:0/383080704 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(249..249 src has 13..249) v4 ==== 10036+0+0 (secure 0 0 0) 0x7f184404cde0 con 0x7f184c09aaa0 2024-09-18T12:34:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.390+0000 7f184a59c640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f182c038490 0x7f182c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.390+0000 7f184a59c640 10 monclient: get_auth_request con 0x7f182c038490 auth_method 0 2024-09-18T12:34:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.392+0000 7f184a59c640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f182c038490 0x7f182c03a940 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f184c09d0e0 tx=0x7f183c039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.394+0000 7f183a7fc640 1 -- 172.21.15.116:0/383080704 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f184401a590 con 0x7f184c09aaa0 2024-09-18T12:34:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.394+0000 7f183a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:06.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.394+0000 7f183a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:06.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.437+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v250) v1 ==== 120+0+0 (secure 0 0 0) 0x7f67b800d700 con 0x7f67c4099130 2024-09-18T12:34:06.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.437+0000 7f67abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T12:34:06.438 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.437+0000 7f67abfff640 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-18T12:34:06.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.535+0000 7f67ca55d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T12:34:06.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.535+0000 7f67ca55d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T12:34:06.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.535+0000 7f67ca55d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.535+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7f67c4064a20 con 0x7f67c4099130 2024-09-18T12:34:06.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.536+0000 7f67abfff640 1 -- 172.21.15.144:0/2288542263 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v250) v1 ==== 120+0+0 (secure 0 0 0) 0x7f67b800d8e0 con 0x7f67c4099130 2024-09-18T12:34:06.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.536+0000 7f67abfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T12:34:06.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.536+0000 7f67abfff640 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-18T12:34:06.538 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h24' does not appear in the crush map 2024-09-18T12:34:06.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67b0038490 msgr2=0x7f67b003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67b0038490 0x7f67b003a940 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f67b4006fd0 tx=0x7f67b4008040 comp rx=0 tx=0).stop 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 10 monclient: shutdown 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c4099130 msgr2=0x7f67c413ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c4099130 0x7f67c413ce60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f67b800b490 tx=0x7f67b8006f50 comp rx=0 tx=0).stop 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 shutdown_connections 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67b0038490 0x7f67b003a940 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67c409e170 0x7f67c4140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67c4099130 0x7f67c413ce60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 --2- 172.21.15.144:0/2288542263 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f67c406a790 0x7f67c413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 >> 172.21.15.144:0/2288542263 conn(0x7f67c4071340 msgr2=0x7f67c406d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:06.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 shutdown_connections 2024-09-18T12:34:06.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.538+0000 7f67ca55d640 1 -- 172.21.15.144:0/2288542263 wait complete. 2024-09-18T12:34:06.548 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:06.549 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h25 2024-09-18T12:34:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.612+0000 7f1850f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:34:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.612+0000 7f1850f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.612+0000 7f1850f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.612+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f184c0976b0 con 0x7f184c09aaa0 2024-09-18T12:34:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.612+0000 7f183a7fc640 1 -- 172.21.15.116:0/383080704 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f1844010af0 con 0x7f184c09aaa0 2024-09-18T12:34:06.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.612+0000 7f183a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:06.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.612+0000 7f183a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:06.614 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:34:06.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f182c038490 msgr2=0x7f182c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f182c038490 0x7f182c03a940 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f184c09d0e0 tx=0x7f183c039000 comp rx=0 tx=0).stop 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 10 monclient: shutdown 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 msgr2=0x7f184c11d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 0x7f184c11d560 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f184400ccc0 tx=0x7f1844005370 comp rx=0 tx=0).stop 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 shutdown_connections 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f182c038490 0x7f182c03a940 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c09aaa0 0x7f184c11d560 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f184c0978f0 0x7f184c11d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 --2- 172.21.15.116:0/383080704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f184c096e10 0x7f184c11cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 >> 172.21.15.116:0/383080704 conn(0x7f184c0672b0 msgr2=0x7f184c130930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 shutdown_connections 2024-09-18T12:34:06.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:06.614+0000 7f1850f9d640 1 -- 172.21.15.116:0/383080704 wait complete. 2024-09-18T12:34:06.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.620+0000 7fa231e68640 10 monclient: get_monmap_and_config 2024-09-18T12:34:06.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.620+0000 7fa231e68640 10 monclient: build_initial_monmap 2024-09-18T12:34:06.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.620+0000 7fa231e68640 10 monclient: monmap: 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:06.621793+0000 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:06.621793+0000 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:06.622 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:06.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.621+0000 7fa231e68640 1 Processor -- start 2024-09-18T12:34:06.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.621+0000 7fa231e68640 1 -- start start 2024-09-18T12:34:06.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.621+0000 7fa231e68640 10 monclient: init 2024-09-18T12:34:06.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c0a0b00 0x7fa22c131a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient(hunting): picked mon.noname-b con 0x7fa22c0a0b00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa22c131fa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa22c06ec50 0x7fa22c069d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient(hunting): picked mon.noname-a con 0x7fa22c06ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22c0a1b70 con 0x7fa22c131fa0 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22c0a0200 con 0x7fa22c06ec50 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c0a1890 con 0x7fa22c0a0b00 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:06.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa231e68640 10 monclient(hunting): authenticate will time out at 1282.191272s 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa22c06ec50 0x7fa22c069d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c0a0b00 0x7fa22c131a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c0a0b00 0x7fa22c131a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:56830/0 (socket says 172.21.15.144:56830) 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa22c06ec50 0x7fa22c069d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:46548/0 (socket says 172.21.15.144:46548) 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 1 -- 172.21.15.144:0/3295106894 learned_addr learned my addr 172.21.15.144:0/3295106894 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa22c131fa0 0x7fa22c132370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:45746/0 (socket says 172.21.15.144:45746) 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 10 monclient(hunting): get_auth_request con 0x7fa22c0a0b00 auth_method 0 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): get_auth_request con 0x7fa22c06ec50 auth_method 0 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient(hunting): handle_auth_done global_id 104586 payload 274 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 1 -- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c0a0b00 msgr2=0x7fa22c131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 1 --2- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c0a0b00 0x7fa22c131a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 1 -- 172.21.15.144:0/3295106894 >> v1:172.21.15.116:6789/0 conn(0x7fa22c131fa0 legacy=0x7fa22c132370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:06.628 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 1 monclient: found mon.noname-a 2024-09-18T12:34:06.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:06.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22bfff640 1 -- 172.21.15.144:0/3295106894 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c0a0440 con 0x7fa22c06ec50 2024-09-18T12:34:06.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.622+0000 7fa22b7fe640 1 --2- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c0a0b00 0x7fa22c131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:06.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.625+0000 7fa22bfff640 1 -- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c06ec50 msgr2=0x7fa22c069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:06.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.625+0000 7fa22bfff640 1 --2- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c06ec50 0x7fa22c069d10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa224009e60 tx=0x7fa224009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251b02298bea85db server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:06.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.625+0000 7fa22a7fc640 1 -- 172.21.15.144:0/3295106894 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224007760 con 0x7fa22c06ec50 2024-09-18T12:34:06.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:06.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: dump: 2024-09-18T12:34:06.636 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 1 -- 172.21.15.144:0/3295106894 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22402d050 con 0x7fa22c06ec50 2024-09-18T12:34:06.637 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 5 monclient: authenticate success, global_id 104586 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 4 set_mon_vals no callback set 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 1 -- 172.21.15.144:0/3295106894 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224007af0 con 0x7fa22c06ec50 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa22a7fc640 10 monclient: dump: 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.638 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 10 monclient: shutdown 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 -- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c06ec50 msgr2=0x7fa22c069d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 --2- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c06ec50 0x7fa22c069d10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa224009e60 tx=0x7fa224009780 comp rx=0 tx=0).stop 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 -- 172.21.15.144:0/3295106894 shutdown_connections 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 --2- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c06ec50 0x7fa22c069d10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 --2- 172.21.15.144:0/3295106894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c0a0b00 0x7fa22c131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 -- 172.21.15.144:0/3295106894 >> 172.21.15.144:0/3295106894 conn(0x7fa22c071370 msgr2=0x7fa22c0a52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 -- 172.21.15.144:0/3295106894 shutdown_connections 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.626+0000 7fa231e68640 1 -- 172.21.15.144:0/3295106894 wait complete. 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient: build_initial_monmap 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient: monmap: 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:06.640 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 Processor -- start 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 -- start start 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient: init 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c06ec50 0x7fa22c0973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): picked mon.noname-c con 0x7fa22c06ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 0x7fa22c097930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): picked mon.noname-b con 0x7fa22c0a0b00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c131fa0 0x7fa22c097e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): picked mon.noname-a con 0x7fa22c131fa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.641 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c0988d0 con 0x7fa22c131fa0 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c09b800 con 0x7fa22c0a0b00 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c0a0fd0 con 0x7fa22c06ec50 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.627+0000 7fa231e68640 10 monclient(hunting): authenticate will time out at 1282.196992s 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 0x7fa22c097930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c06ec50 0x7fa22c0973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c131fa0 0x7fa22c097e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 0x7fa22c097930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:48366/0 (socket says 172.21.15.144:48366) 2024-09-18T12:34:06.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 1 -- 172.21.15.144:0/2091359952 learned_addr learned my addr 172.21.15.144:0/2091359952 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 10 monclient(hunting): get_auth_request con 0x7fa22c06ec50 auth_method 0 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): get_auth_request con 0x7fa22c131fa0 auth_method 0 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 10 monclient(hunting): get_auth_request con 0x7fa22c0a0b00 auth_method 0 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:06.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 1 -- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 msgr2=0x7fa22c097930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 1 -- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 msgr2=0x7fa22c097930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 0x7fa22c097930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22affd640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 0x7fa22c097930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient(hunting): handle_auth_done global_id 104589 payload 824 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 1 -- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c06ec50 msgr2=0x7fa22c0973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c06ec50 0x7fa22c0973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 1 -- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 msgr2=0x7fa22c097930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 0x7fa22c097930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 1 monclient: found mon.noname-a 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22bfff640 1 -- 172.21.15.144:0/2091359952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c009450 con 0x7fa22c131fa0 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.628+0000 7fa22b7fe640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c06ec50 0x7fa22c0973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa22bfff640 1 -- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c131fa0 msgr2=0x7fa22c097e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa22bfff640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c131fa0 0x7fa22c097e70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa224031000 tx=0x7fa224037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224031aa0 con 0x7fa22c131fa0 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: dump: 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: _finish_auth 0 2024-09-18T12:34:06.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa224040070 con 0x7fa22c131fa0 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa231e68640 5 monclient: authenticate success, global_id 104589 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224031e90 con 0x7fa22c131fa0 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa231e68640 10 monclient: _renew_subs 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa231e68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa224009a20 con 0x7fa22c131fa0 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa2137fe640 10 monclient: dump: 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:06.647 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa229ffb640 4 set_mon_vals no callback set 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa231e68640 10 monclient: _renew_subs 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa231e68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.630+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22c09aa60 con 0x7fa22c131fa0 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.631+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa224042030 con 0x7fa22c131fa0 2024-09-18T12:34:06.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.631+0000 7fa231e68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.631+0000 7fa231e68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.631+0000 7fa231e68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.631+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22c0008d0 con 0x7fa22c131fa0 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.631+0000 7fa2137fe640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa218038490 0x7fa21803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.632+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(250..250 src has 13..250) v4 ==== 9823+0+0 (secure 0 0 0) 0x7fa22406e130 con 0x7fa22c131fa0 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.632+0000 7fa22b7fe640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa218038490 0x7fa21803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.632+0000 7fa22b7fe640 10 monclient: get_auth_request con 0x7fa218038490 auth_method 0 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.634+0000 7fa22b7fe640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa218038490 0x7fa21803a940 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa21c006c90 tx=0x7fa21c0091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:06.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.636+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa22403ab80 con 0x7fa22c131fa0 2024-09-18T12:34:06.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.636+0000 7fa2137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:06.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.636+0000 7fa2137fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:06.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.796+0000 7fa231e68640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T12:34:06.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.796+0000 7fa231e68640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T12:34:06.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.796+0000 7fa231e68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:06.797 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:06.796+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7fa22c13d7a0 con 0x7fa22c131fa0 2024-09-18T12:34:07.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.446+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 removed item id -91 name 'h25' from crush map v251) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa22404a030 con 0x7fa22c131fa0 2024-09-18T12:34:07.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.447+0000 7fa2137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T12:34:07.448 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.447+0000 7fa2137fe640 10 monclient: _finish_command 2 = system:0 removed item id -91 name 'h25' from crush map 2024-09-18T12:34:07.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.548+0000 7fa231e68640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T12:34:07.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.548+0000 7fa231e68640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T12:34:07.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.548+0000 7fa231e68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:07.549 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.548+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7fa22c098f30 con 0x7fa22c131fa0 2024-09-18T12:34:07.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.549+0000 7fa2137fe640 1 -- 172.21.15.144:0/2091359952 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v251) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa22404a020 con 0x7fa22c131fa0 2024-09-18T12:34:07.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.549+0000 7fa2137fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T12:34:07.550 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.549+0000 7fa2137fe640 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-18T12:34:07.550 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h25' does not appear in the crush map 2024-09-18T12:34:07.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa218038490 msgr2=0x7fa21803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:07.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa218038490 0x7fa21803a940 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa21c006c90 tx=0x7fa21c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.552 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 10 monclient: shutdown 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c131fa0 msgr2=0x7fa22c097e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c131fa0 0x7fa22c097e70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa224031000 tx=0x7fa224037000 comp rx=0 tx=0).stop 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 shutdown_connections 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa218038490 0x7fa21803a940 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c131fa0 0x7fa22c097e70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa22c0a0b00 0x7fa22c097930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 --2- 172.21.15.144:0/2091359952 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa22c06ec50 0x7fa22c0973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 >> 172.21.15.144:0/2091359952 conn(0x7fa22c071370 msgr2=0x7fa22c0a4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 shutdown_connections 2024-09-18T12:34:07.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.551+0000 7fa231e68640 1 -- 172.21.15.144:0/2091359952 wait complete. 2024-09-18T12:34:07.561 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:07.561 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h26 2024-09-18T12:34:07.635 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.634+0000 7ff654954640 10 monclient: get_monmap_and_config 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.634+0000 7ff654954640 10 monclient: build_initial_monmap 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.634+0000 7ff654954640 10 monclient: monmap: 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:07.635656+0000 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:07.635656+0000 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:07.636 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:07.637 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:07.637 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.635+0000 7ff654954640 1 Processor -- start 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.635+0000 7ff654954640 1 -- start start 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.635+0000 7ff654954640 10 monclient: init 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.635+0000 7ff654954640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.635+0000 7ff654954640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.635+0000 7ff654954640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650069bd0 0x7ff6501320a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.635+0000 7ff654954640 10 monclient(hunting): picked mon.noname-b con 0x7ff650069bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff650098330 0x7ff650098740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 10 monclient(hunting): picked mon.noname-a con 0x7ff650098330 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 0x7ff650070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:07.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 10 monclient(hunting): picked mon.noname-c con 0x7ff650098c80 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff650131db0 con 0x7ff650098330 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff650096bd0 con 0x7ff650098c80 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65009d7f0 con 0x7ff650069bd0 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff654954640 10 monclient(hunting): authenticate will time out at 1283.205069s 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 0x7ff650070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 0x7ff650070920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44284/0 (socket says 172.21.15.144:44284) 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 1 -- 172.21.15.144:0/2969860732 learned_addr learned my addr 172.21.15.144:0/2969860732 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 10 monclient(hunting): get_auth_request con 0x7ff650098c80 auth_method 0 2024-09-18T12:34:07.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 1 -- 172.21.15.144:0/2969860732 >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 msgr2=0x7ff650070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 1 -- 172.21.15.144:0/2969860732 >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 msgr2=0x7ff650070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 1 --2- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650069bd0 0x7ff6501320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 1 --2- 172.21.15.144:0/2969860732 >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 0x7ff650070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/2969860732 >> v2:172.21.15.116:3300/0 conn(0x7ff650098330 0x7ff650098740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 10 monclient(hunting): get_auth_request con 0x7ff650069bd0 auth_method 0 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64ed9d640 1 --2- 172.21.15.144:0/2969860732 >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 0x7ff650070920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:07.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): get_auth_request con 0x7ff650098330 auth_method 0 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient(hunting): handle_auth_done global_id 104592 payload 274 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 1 -- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650069bd0 msgr2=0x7ff6501320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650069bd0 0x7ff6501320a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 1 -- 172.21.15.144:0/2969860732 >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 msgr2=0x7ff650070920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/2969860732 >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 0x7ff650070920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient: _finish_hunting 0 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 1 monclient: found mon.noname-a 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64dd9b640 1 -- 172.21.15.144:0/2969860732 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65009e500 con 0x7ff650098330 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.636+0000 7ff64e59c640 1 --2- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650069bd0 0x7ff6501320a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.639+0000 7ff64dd9b640 1 -- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650098330 msgr2=0x7ff650098740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.639+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650098330 0x7ff650098740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff65009e910 tx=0x7ff6440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354f3e54f5ce3d8d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 1 -- 172.21.15.144:0/2969860732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64402adf0 con 0x7ff650098330 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:07.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: dump: 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: _finish_auth 0 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:07.647 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 1 -- 172.21.15.144:0/2969860732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff644032440 con 0x7ff650098330 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 5 monclient: authenticate success, global_id 104592 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 1 -- 172.21.15.144:0/2969860732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6440327e0 con 0x7ff650098330 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 4 set_mon_vals no callback set 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff64d59a640 10 monclient: dump: 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.648 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 10 monclient: shutdown 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 -- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650098330 msgr2=0x7ff650098740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 --2- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650098330 0x7ff650098740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff65009e910 tx=0x7ff6440090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 -- 172.21.15.144:0/2969860732 shutdown_connections 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 --2- 172.21.15.144:0/2969860732 >> v2:172.21.15.144:3300/0 conn(0x7ff650098c80 0x7ff650070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.649 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 --2- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650098330 0x7ff650098740 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 --2- 172.21.15.144:0/2969860732 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650069bd0 0x7ff6501320a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 -- 172.21.15.144:0/2969860732 >> 172.21.15.144:0/2969860732 conn(0x7ff65009e9f0 msgr2=0x7ff65009edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 -- 172.21.15.144:0/2969860732 shutdown_connections 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.640+0000 7ff654954640 1 -- 172.21.15.144:0/2969860732 wait complete. 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.641+0000 7ff654954640 10 monclient: build_initial_monmap 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.641+0000 7ff654954640 10 monclient: monmap: 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:07.650 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:07.651 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:07.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.641+0000 7ff654954640 1 Processor -- start 2024-09-18T12:34:07.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.641+0000 7ff654954640 1 -- start start 2024-09-18T12:34:07.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.641+0000 7ff654954640 10 monclient: init 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650069bd0 0x7ff65012fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): picked mon.noname-a con 0x7ff650069bd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 0x7ff650140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): picked mon.noname-b con 0x7ff650098330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650098c80 0x7ff650141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): picked mon.noname-c con 0x7ff650098c80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65009d5e0 con 0x7ff650069bd0 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff650130de0 con 0x7ff650098330 2024-09-18T12:34:07.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65011b340 con 0x7ff650098c80 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff654954640 10 monclient(hunting): authenticate will time out at 1283.211196s 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 0x7ff650140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 0x7ff650140fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44288/0 (socket says 172.21.15.144:44288) 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 1 -- 172.21.15.144:0/3648065155 learned_addr learned my addr 172.21.15.144:0/3648065155 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 10 monclient(hunting): get_auth_request con 0x7ff650098330 auth_method 0 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650069bd0 0x7ff65012fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:07.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 1 -- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 msgr2=0x7ff650140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 1 -- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 msgr2=0x7ff650140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 0x7ff650140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): get_auth_request con 0x7ff650069bd0 auth_method 0 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650098c80 0x7ff650141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 0x7ff650140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 10 monclient(hunting): get_auth_request con 0x7ff650098c80 auth_method 0 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:07.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient(hunting): handle_auth_done global_id 104595 payload 824 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 1 -- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650098c80 msgr2=0x7ff650141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650098c80 0x7ff650141520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 1 -- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 msgr2=0x7ff650140fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 0x7ff650140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:07.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 1 monclient: found mon.noname-a 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.642+0000 7ff64e59c640 1 -- 172.21.15.144:0/3648065155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640009390 con 0x7ff650069bd0 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.644+0000 7ff64e59c640 1 -- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650069bd0 msgr2=0x7ff65012fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff64e59c640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650069bd0 0x7ff65012fce0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff64000e970 tx=0x7ff6400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6400095d0 con 0x7ff650069bd0 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: dump: 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.660 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64000cb50 con 0x7ff650069bd0 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff640013470 con 0x7ff650069bd0 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff654954640 5 monclient: authenticate success, global_id 104595 2024-09-18T12:34:07.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff654954640 10 monclient: _renew_subs 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff654954640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff644009390 con 0x7ff650069bd0 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff64cd99640 4 set_mon_vals no callback set 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff63e7fc640 10 monclient: dump: 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:07.662 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff654954640 10 monclient: _renew_subs 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff654954640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.645+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff650130220 con 0x7ff650069bd0 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff654954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff654954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff654954640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65011b850 con 0x7ff650069bd0 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff640013820 con 0x7ff650069bd0 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff63e7fc640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6300384e0 0x7ff63003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:07.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(251..251 src has 13..251) v4 ==== 9610+0+0 (secure 0 0 0) 0x7ff64004cd40 con 0x7ff650069bd0 2024-09-18T12:34:07.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.646+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6300384e0 0x7ff63003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:07.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.647+0000 7ff64dd9b640 10 monclient: get_auth_request con 0x7ff6300384e0 auth_method 0 2024-09-18T12:34:07.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.649+0000 7ff64dd9b640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6300384e0 0x7ff63003a990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff650142030 tx=0x7ff644032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:07.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.651+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff64001b770 con 0x7ff650069bd0 2024-09-18T12:34:07.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.651+0000 7ff63e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:07.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.651+0000 7ff63e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:07.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.808+0000 7ff654954640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T12:34:07.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.808+0000 7ff654954640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T12:34:07.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.808+0000 7ff654954640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:07.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:07.808+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7ff6500008d0 con 0x7ff650069bd0 2024-09-18T12:34:08.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.451+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v252) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff640001f00 con 0x7ff650069bd0 2024-09-18T12:34:08.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.451+0000 7ff63e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T12:34:08.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.451+0000 7ff63e7fc640 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-18T12:34:08.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.551+0000 7ff654954640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T12:34:08.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.551+0000 7ff654954640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T12:34:08.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.551+0000 7ff654954640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:08.553 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.551+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7ff65011ca70 con 0x7ff650069bd0 2024-09-18T12:34:08.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.551+0000 7ff63e7fc640 1 -- 172.21.15.144:0/3648065155 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v252) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff64001b3b0 con 0x7ff650069bd0 2024-09-18T12:34:08.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.552+0000 7ff63e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T12:34:08.554 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.552+0000 7ff63e7fc640 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-18T12:34:08.554 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h26' does not appear in the crush map 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6300384e0 msgr2=0x7ff63003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6300384e0 0x7ff63003a990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff650142030 tx=0x7ff644032000 comp rx=0 tx=0).stop 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 10 monclient: shutdown 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650069bd0 msgr2=0x7ff65012fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650069bd0 0x7ff65012fce0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff64000e970 tx=0x7ff6400090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 shutdown_connections 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6300384e0 0x7ff63003a990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff650098c80 0x7ff650141520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.556 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff650098330 0x7ff650140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 --2- 172.21.15.144:0/3648065155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff650069bd0 0x7ff65012fce0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.553+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 >> 172.21.15.144:0/3648065155 conn(0x7ff65009e9f0 msgr2=0x7ff650097080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:08.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.554+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 shutdown_connections 2024-09-18T12:34:08.557 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.554+0000 7ff654954640 1 -- 172.21.15.144:0/3648065155 wait complete. 2024-09-18T12:34:08.565 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:08.565 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h27 2024-09-18T12:34:08.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.637+0000 7fad2cc17640 10 monclient: get_monmap_and_config 2024-09-18T12:34:08.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.637+0000 7fad2cc17640 10 monclient: build_initial_monmap 2024-09-18T12:34:08.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.637+0000 7fad2cc17640 10 monclient: monmap: 2024-09-18T12:34:08.639 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:08.639 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:08.639 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:08.639222+0000 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:08.639222+0000 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:08.640 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.637+0000 7fad2cc17640 1 Processor -- start 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.637+0000 7fad2cc17640 1 -- start start 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.637+0000 7fad2cc17640 10 monclient: init 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 0x7fad28131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient(hunting): picked mon.noname-c con 0x7fad2809ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad281321a0 0x7fad28068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient(hunting): picked mon.noname-a con 0x7fad281321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad2809f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:08.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad28067420 con 0x7fad2809f020 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad280a0480 con 0x7fad281321a0 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad280a0200 con 0x7fad2809ec50 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2cc17640 10 monclient(hunting): authenticate will time out at 1284.208605s 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 0x7fad28131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 0x7fad28131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44302/0 (socket says 172.21.15.144:44302) 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 1 -- 172.21.15.144:0/2471522107 learned_addr learned my addr 172.21.15.144:0/2471522107 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 10 monclient(hunting): get_auth_request con 0x7fad2809ec50 auth_method 0 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:08.644 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 1 -- 172.21.15.144:0/2471522107 >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 msgr2=0x7fad28131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 1 -- 172.21.15.144:0/2471522107 >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 msgr2=0x7fad28131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 1 --2- 172.21.15.144:0/2471522107 >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 0x7fad28131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad2659c640 1 --2- 172.21.15.144:0/2471522107 >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 0x7fad28131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.638+0000 7fad25d9b640 1 --2- 172.21.15.144:0/2471522107 >> v2:172.21.15.116:3300/0 conn(0x7fad281321a0 0x7fad28068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): get_auth_request con 0x7fad281321a0 auth_method 0 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:08.645 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient(hunting): handle_auth_done global_id 104598 payload 274 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 1 -- 172.21.15.144:0/2471522107 >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 msgr2=0x7fad28131c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 1 --2- 172.21.15.144:0/2471522107 >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 0x7fad28131c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 1 -- 172.21.15.144:0/2471522107 >> v1:172.21.15.116:6789/0 conn(0x7fad2809f020 legacy=0x7fad28069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 1 monclient: found mon.noname-a 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:08.646 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.639+0000 7fad25d9b640 1 -- 172.21.15.144:0/2471522107 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2809fd80 con 0x7fad281321a0 2024-09-18T12:34:08.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad25d9b640 1 -- 172.21.15.144:0/2471522107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 msgr2=0x7fad28068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:08.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad25d9b640 1 --2- 172.21.15.144:0/2471522107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 0x7fad28068d00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad280719d0 tx=0x7fad100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b432c8cc137134ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:08.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 1 -- 172.21.15.144:0/2471522107 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1002adf0 con 0x7fad281321a0 2024-09-18T12:34:08.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:08.652 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: dump: 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: _finish_auth 0 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:08.653 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 1 -- 172.21.15.144:0/2471522107 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad10032440 con 0x7fad281321a0 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 1 -- 172.21.15.144:0/2471522107 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad100327e0 con 0x7fad281321a0 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2cc17640 5 monclient: authenticate success, global_id 104598 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2cc17640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2cc17640 4 set_mon_vals no callback set 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.642+0000 7fad2559a640 10 monclient: dump: 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.654 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 10 monclient: shutdown 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 -- 172.21.15.144:0/2471522107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 msgr2=0x7fad28068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 --2- 172.21.15.144:0/2471522107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 0x7fad28068d00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad280719d0 tx=0x7fad100090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 -- 172.21.15.144:0/2471522107 shutdown_connections 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 --2- 172.21.15.144:0/2471522107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 0x7fad28068d00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.655 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 --2- 172.21.15.144:0/2471522107 >> v2:172.21.15.144:3300/0 conn(0x7fad2809ec50 0x7fad28131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 -- 172.21.15.144:0/2471522107 >> 172.21.15.144:0/2471522107 conn(0x7fad280a5cf0 msgr2=0x7fad280a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 -- 172.21.15.144:0/2471522107 shutdown_connections 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.643+0000 7fad2cc17640 1 -- 172.21.15.144:0/2471522107 wait complete. 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient: build_initial_monmap 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient: monmap: 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:08.656 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 Processor -- start 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 -- start start 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient: init 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad2809ec50 0x7fad2809ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): picked mon.noname-c con 0x7fad2809ec50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 0x7fad2811bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): picked mon.noname-b con 0x7fad2809f020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 0x7fad28130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:08.657 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): picked mon.noname-a con 0x7fad281321a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2809b5f0 con 0x7fad281321a0 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2811dc30 con 0x7fad2809f020 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2811e1f0 con 0x7fad2809ec50 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad2cc17640 10 monclient(hunting): authenticate will time out at 1284.214868s 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad25d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 0x7fad2811bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad25d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 0x7fad2811bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44314/0 (socket says 172.21.15.144:44314) 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad25d9b640 1 -- 172.21.15.144:0/339270515 learned_addr learned my addr 172.21.15.144:0/339270515 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad25d9b640 10 monclient(hunting): get_auth_request con 0x7fad2809f020 auth_method 0 2024-09-18T12:34:08.658 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.644+0000 7fad25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad25d9b640 1 -- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 msgr2=0x7fad2811bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad25d9b640 1 -- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 msgr2=0x7fad2811bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad25d9b640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 0x7fad2811bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad2809ec50 0x7fad2809ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad25d9b640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 0x7fad2811bc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): get_auth_request con 0x7fad2809ec50 auth_method 0 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 0x7fad28130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 10 monclient(hunting): get_auth_request con 0x7fad281321a0 auth_method 0 2024-09-18T12:34:08.659 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:08.660 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient(hunting): handle_auth_done global_id 65384 payload 824 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 1 -- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 msgr2=0x7fad2811bc20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 0x7fad2811bc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 1 -- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 msgr2=0x7fad28130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 0x7fad28130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 1 monclient: found mon.noname-c 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.645+0000 7fad2659c640 1 -- 172.21.15.144:0/339270515 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad10009390 con 0x7fad2809ec50 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.647+0000 7fad2659c640 1 -- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad2809ec50 msgr2=0x7fad2809ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.647+0000 7fad2659c640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad2809ec50 0x7fad2809ab00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fad18009650 tx=0x7fad18009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:08.661 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1800dd50 con 0x7fad2809ec50 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: dump: 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:08.662 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad2cc17640 5 monclient: authenticate success, global_id 65384 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad2cc17640 10 monclient: _renew_subs 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad2cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2811df70 con 0x7fad2809ec50 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1800f040 con 0x7fad2809ec50 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18015690 con 0x7fad2809ec50 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad1e7fc640 10 monclient: dump: 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:08.663 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad24d99640 4 set_mon_vals no callback set 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad2cc17640 10 monclient: _renew_subs 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad2cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.648+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad28131050 con 0x7fad2809ec50 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad2cc17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:08.664 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad2cc17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad2cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad280008d0 con 0x7fad2809ec50 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fad1801e3b0 con 0x7fad2809ec50 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad1e7fc640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad08038490 0x7fad0803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(252..252 src has 13..252) v4 ==== 9397+0+0 (secure 0 0 0) 0x7fad1804cbb0 con 0x7fad2809ec50 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad25d9b640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad08038490 0x7fad0803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.649+0000 7fad25d9b640 10 monclient: get_auth_request con 0x7fad08038490 auth_method 0 2024-09-18T12:34:08.665 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.651+0000 7fad25d9b640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad08038490 0x7fad0803a940 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fad10035040 tx=0x7fad10032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:08.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.654+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fad180118b0 con 0x7fad2809ec50 2024-09-18T12:34:08.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.654+0000 7fad1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:08.666 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.654+0000 7fad1e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:08.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.817+0000 7fad2cc17640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T12:34:08.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.817+0000 7fad2cc17640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T12:34:08.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.817+0000 7fad2cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:08.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:08.817+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7fad280a13e0 con 0x7fad2809ec50 2024-09-18T12:34:09.632 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.735+0000 7f40bb40f640 10 monclient: get_monmap_and_config 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.735+0000 7f40bb40f640 10 monclient: build_initial_monmap 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.735+0000 7f40bb40f640 10 monclient: monmap: 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:09.735996+0000 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:09.735996+0000 2024-09-18T12:34:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:09.737 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:09.737 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:09.737 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:09.737 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:09.737 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:09.737 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:09.737 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.735+0000 7f40bb40f640 1 Processor -- start 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 1 -- start start 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient: init 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40b4096c60 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40b4097030 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 0x7f40b4136f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient(hunting): picked mon.noname-c con 0x7f40b4134b50 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40696a0 con 0x7f40b4096c60 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b409d2e0 con 0x7f40b4097030 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b409d710 con 0x7f40b4134b50 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40bb40f640 10 monclient(hunting): authenticate will time out at 1277.057725s 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40b91ab640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f40b4096c60 0x7f40b409a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46484/0 (socket says 172.21.15.116:46484) 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40b91ab640 1 -- 172.21.15.116:0/947791447 learned_addr learned my addr 172.21.15.116:0/947791447 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.736+0000 7f40b99ac640 1 --2- 172.21.15.116:0/947791447 >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 0x7f40b4136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 10 monclient(hunting): get_auth_request con 0x7f40b4134b50 auth_method 0 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 1 -- 172.21.15.116:0/947791447 >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 msgr2=0x7f40b4136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:34:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 1 -- 172.21.15.116:0/947791447 >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 msgr2=0x7f40b4136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 1 --2- 172.21.15.116:0/947791447 >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 0x7f40b4136f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.737+0000 7f40b99ac640 1 --2- 172.21.15.116:0/947791447 >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 0x7f40b4136f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.739+0000 7f40b91ab640 1 -- 172.21.15.116:0/947791447 >> v1:172.21.15.116:6789/0 conn(0x7f40b4096c60 legacy=0x7f40b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.739+0000 7f40b89aa640 1 -- 172.21.15.116:0/947791447 >> v1:172.21.15.144:6789/0 conn(0x7f40b4097030 legacy=0x7f40b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.743+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817436410 0 0) 0x7f40b40696a0 con 0x7f40b4096c60 2024-09-18T12:34:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.743+0000 7f40abfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.743+0000 7f40abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.743+0000 7f40abfff640 10 monclient(hunting): my global_id is 104604 2024-09-18T12:34:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.743+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a00034e0 con 0x7f40b4096c60 2024-09-18T12:34:09.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.742+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v253) v1 ==== 120+0+0 (secure 0 0 0) 0x7fad1801a020 con 0x7fad2809ec50 2024-09-18T12:34:09.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.742+0000 7fad1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T12:34:09.744 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.742+0000 7fad1e7fc640 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-18T12:34:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241159072 0 0) 0x7f40a00034e0 con 0x7f40b4096c60 2024-09-18T12:34:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 msgr2=0x7f40b4136f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 --2- 172.21.15.116:0/947791447 >> v2:172.21.15.144:3300/0 conn(0x7f40b4134b50 0x7f40b4136f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 >> v1:172.21.15.144:6789/0 conn(0x7f40b4097030 legacy=0x7f40b4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40992f0 con 0x7f40b4096c60 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: _finish_auth 0 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40bb40f640 5 monclient: authenticate success, global_id 104604 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f40a40017a0 con 0x7f40b4096c60 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:34:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: dump: 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.749 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 >> v1:172.21.15.116:6789/0 conn(0x7f40b4096c60 legacy=0x7f40b409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40a0003bc0 0x7f40a0003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): picked mon.b con 0x7f40a0003bc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40a0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): picked mon.c con 0x7f40a0004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40a00099c0 0x7f40a000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): picked mon.a con 0x7f40a00099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b409d710 con 0x7f40a00099c0 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a000c420 con 0x7f40a0003bc0 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a000c5e0 con 0x7f40a0004780 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40abfff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40bb40f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40a0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40a00099c0 0x7f40a000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): get_auth_request con 0x7f40a0004780 auth_method 0 2024-09-18T12:34:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40a0003bc0 0x7f40a0003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 10 monclient(hunting): get_auth_request con 0x7f40a00099c0 auth_method 0 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 10 monclient(hunting): get_auth_request con 0x7f40a0003bc0 auth_method 0 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b91ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient(hunting): handle_auth_done global_id 104604 payload 293 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 1 -- 172.21.15.116:0/947791447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40a0003bc0 msgr2=0x7f40a0003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40a0003bc0 0x7f40a0003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 1 -- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40a00099c0 msgr2=0x7f40a000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40a00099c0 0x7f40a000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient: _finish_hunting 0 2024-09-18T12:34:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 1 monclient: found mon.c 2024-09-18T12:34:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 1 -- 172.21.15.116:0/947791447 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a00034e0 con 0x7f40a0004780 2024-09-18T12:34:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 10 monclient: _finish_auth 0 2024-09-18T12:34:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b89aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40a00099c0 0x7f40a000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.744+0000 7f40b99ac640 1 -- 172.21.15.116:0/947791447 reap_dead start 2024-09-18T12:34:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.747+0000 7f40b89aa640 1 -- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 msgr2=0x7f40a0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.747+0000 7f40b89aa640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40a0004bf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001b10 tx=0x7f40b0007c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2efe822fd230e561 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b002ab50 con 0x7f40a0004780 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 10 monclient: dump: 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.761 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b002acb0 con 0x7f40a0004780 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40bb40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40bb40f640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 1 -- 172.21.15.116:0/947791447 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0007340 con 0x7f40a0004780 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40bb40f640 4 set_mon_vals no callback set 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.748+0000 7f40abfff640 10 monclient: dump: 2024-09-18T12:34:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 10 monclient: shutdown 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 1 -- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 msgr2=0x7f40a0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40a0004bf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001b10 tx=0x7f40b0007c80 comp rx=0 tx=0).stop 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 1 -- 172.21.15.116:0/947791447 shutdown_connections 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 1 --2- 172.21.15.116:0/947791447 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40a0004bf0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 1 -- 172.21.15.116:0/947791447 >> 172.21.15.116:0/947791447 conn(0x7f40b40672b0 msgr2=0x7f40b40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 1 -- 172.21.15.116:0/947791447 shutdown_connections 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 1 -- 172.21.15.116:0/947791447 wait complete. 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 10 monclient: build_initial_monmap 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.749+0000 7f40bb40f640 10 monclient: monmap: 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 Processor -- start 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 -- start start 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient: init 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40b412ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): picked mon.noname-c con 0x7f40a0004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40b412ef40 0x7f40b41411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): picked mon.noname-a con 0x7f40b412ef40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b4141720 0x7f40b406d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): picked mon.noname-b con 0x7f40b4141720 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b412f330 con 0x7f40b412ef40 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4141bc0 con 0x7f40b4141720 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4134510 con 0x7f40a0004780 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40bb40f640 10 monclient(hunting): authenticate will time out at 1277.071837s 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b91ab640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40b412ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b89aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40b412ef40 0x7f40b41411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b91ab640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40b412ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43376/0 (socket says 172.21.15.116:43376) 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b91ab640 1 -- 172.21.15.116:0/2947363794 learned_addr learned my addr 172.21.15.116:0/2947363794 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b89aa640 10 monclient(hunting): get_auth_request con 0x7f40b412ef40 auth_method 0 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b89aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b89aa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b89aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b91ab640 10 monclient(hunting): get_auth_request con 0x7f40a0004780 auth_method 0 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b91ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b91ab640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.750+0000 7f40b91ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b4141720 0x7f40b406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 10 monclient(hunting): get_auth_request con 0x7f40b4141720 auth_method 0 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b89aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b89aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b89aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b99ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 10 monclient(hunting): handle_auth_done global_id 65387 payload 824 2024-09-18T12:34:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 1 -- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b4141720 msgr2=0x7f40b406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b4141720 0x7f40b406d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 1 -- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40b412ef40 msgr2=0x7f40b41411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40b412ef40 0x7f40b41411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 10 monclient: _finish_hunting 0 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 1 monclient: found mon.noname-c 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.751+0000 7f40b91ab640 1 -- 172.21.15.116:0/2947363794 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b406f630 con 0x7f40a0004780 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40b91ab640 1 -- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 msgr2=0x7f40b412ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40b91ab640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40b412ea00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f40a4007140 tx=0x7f40a4007a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 1 -- 172.21.15.116:0/2947363794 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4018030 con 0x7f40a0004780 2024-09-18T12:34:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: dump: 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: _finish_auth 0 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40bb40f640 5 monclient: authenticate success, global_id 65387 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 1 -- 172.21.15.116:0/2947363794 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a400f8e0 con 0x7f40a0004780 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40bb40f640 10 monclient: _renew_subs 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 1 -- 172.21.15.116:0/2947363794 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a400fca0 con 0x7f40a0004780 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40bb40f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b406eee0 con 0x7f40a0004780 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40ab7fe640 4 set_mon_vals no callback set 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40a97fa640 10 monclient: dump: 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:09.771 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40bb40f640 10 monclient: _renew_subs 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40bb40f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.753+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b406d860 con 0x7f40a0004780 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.754+0000 7f40a97fa640 1 -- 172.21.15.116:0/2947363794 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f40a401b070 con 0x7f40a0004780 2024-09-18T12:34:09.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.754+0000 7f40a97fa640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4098038490 0x7f409803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.754+0000 7f40a97fa640 1 -- 172.21.15.116:0/2947363794 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(253..253 src has 13..253) v4 ==== 9184+0+0 (secure 0 0 0) 0x7f40a4019070 con 0x7f40a0004780 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.754+0000 7f40bb40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.754+0000 7f40bb40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.754+0000 7f40bb40f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.754+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b404bf60 con 0x7f40a0004780 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.755+0000 7f40b89aa640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4098038490 0x7f409803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.755+0000 7f40b89aa640 10 monclient: get_auth_request con 0x7f4098038490 auth_method 0 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.757+0000 7f40b89aa640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4098038490 0x7f409803a940 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f40b002f000 tx=0x7f40b0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.759+0000 7f40a97fa640 1 -- 172.21.15.116:0/2947363794 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f40a4013e00 con 0x7f40a0004780 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.759+0000 7f40a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.759+0000 7f40a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.843+0000 7fad2cc17640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T12:34:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.843+0000 7fad2cc17640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T12:34:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.843+0000 7fad2cc17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:09.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.843+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7fad28072640 con 0x7fad2809ec50 2024-09-18T12:34:09.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.845+0000 7fad1e7fc640 1 -- 172.21.15.144:0/339270515 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v253) v1 ==== 120+0+0 (secure 0 0 0) 0x7fad1801a030 con 0x7fad2809ec50 2024-09-18T12:34:09.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.845+0000 7fad1e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T12:34:09.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.845+0000 7fad1e7fc640 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-18T12:34:09.847 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h27' does not appear in the crush map 2024-09-18T12:34:09.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad08038490 msgr2=0x7fad0803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad08038490 0x7fad0803a940 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fad10035040 tx=0x7fad10032000 comp rx=0 tx=0).stop 2024-09-18T12:34:09.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 10 monclient: shutdown 2024-09-18T12:34:09.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad2809ec50 msgr2=0x7fad2809ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad2809ec50 0x7fad2809ab00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fad18009650 tx=0x7fad18009250 comp rx=0 tx=0).stop 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 shutdown_connections 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad08038490 0x7fad0803a940 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad281321a0 0x7fad28130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad2809f020 0x7fad2811bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 --2- 172.21.15.144:0/339270515 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad2809ec50 0x7fad2809ab00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 >> 172.21.15.144:0/339270515 conn(0x7fad280a5cf0 msgr2=0x7fad28134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 shutdown_connections 2024-09-18T12:34:09.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.847+0000 7fad2cc17640 1 -- 172.21.15.144:0/339270515 wait complete. 2024-09-18T12:34:09.858 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:09.859 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h28 2024-09-18T12:34:09.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.930+0000 7f7c258d4640 10 monclient: get_monmap_and_config 2024-09-18T12:34:09.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.930+0000 7f7c258d4640 10 monclient: build_initial_monmap 2024-09-18T12:34:09.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.930+0000 7f7c258d4640 10 monclient: monmap: 2024-09-18T12:34:09.932 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:09.932139+0000 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:09.932139+0000 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.930+0000 7f7c258d4640 1 Processor -- start 2024-09-18T12:34:09.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.930+0000 7f7c258d4640 1 -- start start 2024-09-18T12:34:09.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.930+0000 7f7c258d4640 10 monclient: init 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c2009ec50 0x7f7c20131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient(hunting): picked mon.noname-c con 0x7f7c2009ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7c201321a0 0x7f7c20068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient(hunting): picked mon.noname-a con 0x7f7c201321a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c2009f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c20067420 con 0x7f7c2009f020 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200a0480 con 0x7f7c201321a0 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200a0200 con 0x7f7c2009ec50 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c258d4640 10 monclient(hunting): authenticate will time out at 1285.501321s 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c2009ec50 0x7f7c20131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c2009ec50 0x7f7c20131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44318/0 (socket says 172.21.15.144:44318) 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1effd640 1 -- 172.21.15.144:0/2147841520 learned_addr learned my addr 172.21.15.144:0/2147841520 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1effd640 10 monclient(hunting): get_auth_request con 0x7f7c2009ec50 auth_method 0 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1f7fe640 1 --1- 172.21.15.144:0/2147841520 >> v1:172.21.15.144:6789/0 conn(0x7f7c2009f020 0x7f7c20069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:55926/0 (socket says 172.21.15.144:55926) 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 1 --2- 172.21.15.144:0/2147841520 >> v2:172.21.15.116:3300/0 conn(0x7f7c201321a0 0x7f7c20068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7c201321a0 auth_method 0 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient(hunting): handle_auth_done global_id 104610 payload 274 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 1 -- 172.21.15.144:0/2147841520 >> v2:172.21.15.144:3300/0 conn(0x7f7c2009ec50 msgr2=0x7f7c20131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 1 --2- 172.21.15.144:0/2147841520 >> v2:172.21.15.144:3300/0 conn(0x7f7c2009ec50 0x7f7c20131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 1 -- 172.21.15.144:0/2147841520 >> v1:172.21.15.144:6789/0 conn(0x7f7c2009f020 legacy=0x7f7c20069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:34:09.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 1 monclient: found mon.noname-a 2024-09-18T12:34:09.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:09.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:09.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.931+0000 7f7c1e7fc640 1 -- 172.21.15.144:0/2147841520 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2009fd80 con 0x7f7c201321a0 2024-09-18T12:34:09.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.934+0000 7f7c1e7fc640 1 -- 172.21.15.144:0/2147841520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c201321a0 msgr2=0x7f7c20068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:09.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.934+0000 7f7c1e7fc640 1 --2- 172.21.15.144:0/2147841520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c201321a0 0x7f7c20068d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c200719d0 tx=0x7f7c140090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0045b0e7bf7d8d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 1 -- 172.21.15.144:0/2147841520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1402adf0 con 0x7f7c201321a0 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: dump: 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.945 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 1 -- 172.21.15.144:0/2147841520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c14032440 con 0x7f7c201321a0 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 1 -- 172.21.15.144:0/2147841520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c140327e0 con 0x7f7c201321a0 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 5 monclient: authenticate success, global_id 104610 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 4 set_mon_vals no callback set 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c1dffb640 10 monclient: dump: 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:09.946 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 10 monclient: shutdown 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 -- 172.21.15.144:0/2147841520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c201321a0 msgr2=0x7f7c20068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 --2- 172.21.15.144:0/2147841520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c201321a0 0x7f7c20068d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c200719d0 tx=0x7f7c140090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 -- 172.21.15.144:0/2147841520 shutdown_connections 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 --2- 172.21.15.144:0/2147841520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c201321a0 0x7f7c20068d00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 --2- 172.21.15.144:0/2147841520 >> v2:172.21.15.144:3300/0 conn(0x7f7c2009ec50 0x7f7c20131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 -- 172.21.15.144:0/2147841520 >> 172.21.15.144:0/2147841520 conn(0x7f7c200a5cf0 msgr2=0x7f7c200a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 -- 172.21.15.144:0/2147841520 shutdown_connections 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.935+0000 7f7c258d4640 1 -- 172.21.15.144:0/2147841520 wait complete. 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.936+0000 7f7c258d4640 10 monclient: build_initial_monmap 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.936+0000 7f7c258d4640 10 monclient: monmap: 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:09.948 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.936+0000 7f7c258d4640 1 Processor -- start 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.936+0000 7f7c258d4640 1 -- start start 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.936+0000 7f7c258d4640 10 monclient: init 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 0x7f7c2009a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): picked mon.noname-b con 0x7f7c2009ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2009f020 0x7f7c20130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): picked mon.noname-a con 0x7f7c2009f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:09.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c201321a0 0x7f7c2011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): picked mon.noname-c con 0x7f7c201321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20134ad0 con 0x7f7c2009f020 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2011de70 con 0x7f7c2009ec50 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2009b4f0 con 0x7f7c201321a0 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c258d4640 10 monclient(hunting): authenticate will time out at 1285.507084s 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 0x7f7c2009a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 0x7f7c2009a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44326/0 (socket says 172.21.15.144:44326) 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 -- 172.21.15.144:0/1464330520 learned_addr learned my addr 172.21.15.144:0/1464330520 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:09.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): get_auth_request con 0x7f7c2009ec50 auth_method 0 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2009f020 0x7f7c20130a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c201321a0 0x7f7c2011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 10 monclient(hunting): get_auth_request con 0x7f7c201321a0 auth_method 0 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 10 monclient(hunting): get_auth_request con 0x7f7c2009f020 auth_method 0 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient(hunting): handle_auth_done global_id 124105 payload 824 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 -- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c201321a0 msgr2=0x7f7c2011b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c201321a0 0x7f7c2011b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 -- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2009f020 msgr2=0x7f7c20130a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2009f020 0x7f7c20130a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient: _finish_hunting 0 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 monclient: found mon.noname-b 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.937+0000 7f7c1effd640 1 -- 172.21.15.144:0/1464330520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c14009390 con 0x7f7c2009ec50 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c1effd640 1 -- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 msgr2=0x7f7c2009a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c1effd640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 0x7f7c2009a980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7c10007eb0 tx=0x7f7c10009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10015400 con 0x7f7c2009ec50 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: dump: 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:09.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c258d4640 5 monclient: authenticate success, global_id 124105 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c258d4640 10 monclient: _renew_subs 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c258d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c20141df0 con 0x7f7c2009ec50 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c258d4640 10 monclient: _renew_subs 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c258d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2011cca0 con 0x7f7c2009ec50 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1000f040 con 0x7f7c2009ec50 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c100157b0 con 0x7f7c2009ec50 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c06ffd640 10 monclient: dump: 2024-09-18T12:34:09.955 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.939+0000 7f7c1d7fa640 4 set_mon_vals no callback set 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.940+0000 7f7c258d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.940+0000 7f7c258d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:09.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.940+0000 7f7c258d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.940+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7c10015b20 con 0x7f7c2009ec50 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.940+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2011e030 con 0x7f7c2009ec50 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.941+0000 7f7c06ffd640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c0c038490 0x7f7c0c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.941+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(253..253 src has 13..253) v4 ==== 9184+0+0 (secure 0 0 0) 0x7f7c1004ce00 con 0x7f7c2009ec50 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.941+0000 7f7c1e7fc640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c0c038490 0x7f7c0c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.941+0000 7f7c1e7fc640 10 monclient: get_auth_request con 0x7f7c0c038490 auth_method 0 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.943+0000 7f7c1e7fc640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c0c038490 0x7f7c0c03a940 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7c14031000 tx=0x7f7c14032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:09.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.945+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7c1001d960 con 0x7f7c2009ec50 2024-09-18T12:34:09.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.945+0000 7f7c06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:09.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:09.945+0000 7f7c06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.976+0000 7f40bb40f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:34:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.976+0000 7f40bb40f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.976+0000 7f40bb40f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.976+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f40b409add0 con 0x7f40a0004780 2024-09-18T12:34:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.976+0000 7f40a97fa640 1 -- 172.21.15.116:0/2947363794 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f40a402c620 con 0x7f40a0004780 2024-09-18T12:34:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.976+0000 7f40a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.976+0000 7f40a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:09.978 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:34:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4098038490 msgr2=0x7f409803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4098038490 0x7f409803a940 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f40b002f000 tx=0x7f40b0030000 comp rx=0 tx=0).stop 2024-09-18T12:34:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 10 monclient: shutdown 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 msgr2=0x7f40b412ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40b412ea00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f40a4007140 tx=0x7f40a4007a90 comp rx=0 tx=0).stop 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 shutdown_connections 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4098038490 0x7f409803a940 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40b4141720 0x7f40b406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40b412ef40 0x7f40b41411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 --2- 172.21.15.116:0/2947363794 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40a0004780 0x7f40b412ea00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 >> 172.21.15.116:0/2947363794 conn(0x7f40b40672b0 msgr2=0x7f40b40a01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 shutdown_connections 2024-09-18T12:34:09.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:09.978+0000 7f40bb40f640 1 -- 172.21.15.116:0/2947363794 wait complete. 2024-09-18T12:34:09.990 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:34:09.990 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:34:09.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.082+0000 7f343859d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.082+0000 7f343859d640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.082+0000 7f343859d640 10 monclient: monmap: 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:10.082880+0000 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:10.082880+0000 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:10.084 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.082+0000 7f343859d640 1 Processor -- start 2024-09-18T12:34:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.082+0000 7f343859d640 1 -- start start 2024-09-18T12:34:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.082+0000 7f343859d640 10 monclient: init 2024-09-18T12:34:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T12:34:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3430131ab0 0x7f3430131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient(hunting): picked mon.noname-a con 0x7f3430131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f343009e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3430136bb0 0x7f3430138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient(hunting): picked mon.noname-c con 0x7f3430136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343009faf0 con 0x7f343009e910 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343009fd30 con 0x7f3430131ab0 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34300a1110 con 0x7f3430136bb0 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f343859d640 10 monclient(hunting): authenticate will time out at 1277.404455s 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3430131ab0 0x7f3430131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3435b38640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f343009e910 0x7f343009ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46486/0 (socket says 172.21.15.116:46486) 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3430131ab0 0x7f3430131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45850/0 (socket says 172.21.15.116:45850) 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3435b38640 1 -- 172.21.15.116:0/3606652577 learned_addr learned my addr 172.21.15.116:0/3606652577 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): get_auth_request con 0x7f3430131ab0 auth_method 0 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436b3a640 1 --2- 172.21.15.116:0/3606652577 >> v2:172.21.15.144:3300/0 conn(0x7f3430136bb0 0x7f3430138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436b3a640 10 monclient(hunting): get_auth_request con 0x7f3430136bb0 auth_method 0 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient(hunting): handle_auth_done global_id 104616 payload 274 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 1 -- 172.21.15.116:0/3606652577 >> v2:172.21.15.144:3300/0 conn(0x7f3430136bb0 msgr2=0x7f3430138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 1 --2- 172.21.15.116:0/3606652577 >> v2:172.21.15.144:3300/0 conn(0x7f3430136bb0 0x7f3430138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 1 -- 172.21.15.116:0/3606652577 >> v1:172.21.15.116:6789/0 conn(0x7f343009e910 legacy=0x7f343009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 1 monclient: found mon.noname-a 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.083+0000 7f3436339640 1 -- 172.21.15.116:0/3606652577 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300a13f0 con 0x7f3430131ab0 2024-09-18T12:34:10.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3436339640 1 -- 172.21.15.116:0/3606652577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430131ab0 msgr2=0x7f3430131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3436339640 1 --2- 172.21.15.116:0/3606652577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430131ab0 0x7f3430131e80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3428009850 tx=0x7f34280090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ff09f995d6e61a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 1 -- 172.21.15.116:0/3606652577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342802adf0 con 0x7f3430131ab0 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: dump: 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.097 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 1 -- 172.21.15.116:0/3606652577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3428032440 con 0x7f3430131ab0 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f343859d640 5 monclient: authenticate success, global_id 104616 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f343859d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f343859d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 1 -- 172.21.15.116:0/3606652577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34280327e0 con 0x7f3430131ab0 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f343859d640 4 set_mon_vals no callback set 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.086+0000 7f3435337640 10 monclient: dump: 2024-09-18T12:34:10.098 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 10 monclient: shutdown 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 -- 172.21.15.116:0/3606652577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430131ab0 msgr2=0x7f3430131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 --2- 172.21.15.116:0/3606652577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430131ab0 0x7f3430131e80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3428009850 tx=0x7f34280090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 -- 172.21.15.116:0/3606652577 shutdown_connections 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 --2- 172.21.15.116:0/3606652577 >> v2:172.21.15.144:3300/0 conn(0x7f3430136bb0 0x7f3430138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 --2- 172.21.15.116:0/3606652577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430131ab0 0x7f3430131e80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 -- 172.21.15.116:0/3606652577 >> 172.21.15.116:0/3606652577 conn(0x7f3430067d30 msgr2=0x7f34300a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 -- 172.21.15.116:0/3606652577 shutdown_connections 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 1 -- 172.21.15.116:0/3606652577 wait complete. 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.087+0000 7f343859d640 10 monclient: monmap: 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 Processor -- start 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 -- start start 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient: init 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343009e910 0x7f34300975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): picked mon.noname-b con 0x7f343009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3430131ab0 0x7f3430097b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): picked mon.noname-c con 0x7f3430131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:10.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430136bb0 0x7f3430098060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): picked mon.noname-a con 0x7f3430136bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343009ac20 con 0x7f3430136bb0 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343009bde0 con 0x7f343009e910 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343009d900 con 0x7f3430131ab0 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f343859d640 10 monclient(hunting): authenticate will time out at 1277.409685s 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430136bb0 0x7f3430098060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3430131ab0 0x7f3430097b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430136bb0 0x7f3430098060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45854/0 (socket says 172.21.15.116:45854) 2024-09-18T12:34:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 1 -- 172.21.15.116:0/4104179221 learned_addr learned my addr 172.21.15.116:0/4104179221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 10 monclient(hunting): get_auth_request con 0x7f3430136bb0 auth_method 0 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 10 monclient(hunting): get_auth_request con 0x7f3430131ab0 auth_method 0 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343009e910 0x7f34300975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 10 monclient(hunting): get_auth_request con 0x7f343009e910 auth_method 0 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3435b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.088+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 10 monclient(hunting): handle_auth_done global_id 104619 payload 824 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3430131ab0 msgr2=0x7f3430097b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3430131ab0 0x7f3430097b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343009e910 msgr2=0x7f34300975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343009e910 0x7f34300975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 1 monclient: found mon.noname-a 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3436b3a640 1 -- 172.21.15.116:0/4104179221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428009390 con 0x7f3430136bb0 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.089+0000 7f3435b38640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3430131ab0 0x7f3430097b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f3436b3a640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430136bb0 msgr2=0x7f3430098060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f3436b3a640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430136bb0 0x7f3430098060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3420009fa0 tx=0x7f3420009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342000ee60 con 0x7f3430136bb0 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: dump: 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3420013590 con 0x7f3430136bb0 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f343859d640 5 monclient: authenticate success, global_id 104619 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420013950 con 0x7f3430136bb0 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f343859d640 10 monclient: _renew_subs 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f343859d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3430098e10 con 0x7f3430136bb0 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f342e7fc640 10 monclient: dump: 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f3434b36640 4 set_mon_vals no callback set 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f343859d640 10 monclient: _renew_subs 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f343859d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.091+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3430131560 con 0x7f3430136bb0 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f343859d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f343859d640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 0x7f3430133690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f343859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343009aae0 con 0x7f34300985a0 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3420013cc0 con 0x7f3430136bb0 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 0x7f3430133690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient: get_auth_request con 0x7f34300985a0 auth_method 0 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f342e7fc640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 10 monclient(hunting): handle_auth_done global_id 104622 payload 824 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(253..253 src has 13..253) v4 ==== 9184+0+0 (secure 0 0 0) 0x7f342004d0a0 con 0x7f3430136bb0 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3436339640 1 -- 172.21.15.116:0/4104179221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f343004bf60 con 0x7f34300985a0 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3435b38640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.092+0000 7f3435b38640 10 monclient: get_auth_request con 0x7f341c038490 auth_method 0 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.094+0000 7f3436339640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 msgr2=0x7f3430133690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.094+0000 7f3436339640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 0x7f3430133690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3428034000 tx=0x7f3428032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.094+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428031b80 con 0x7f34300985a0 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.094+0000 7f342e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.094+0000 7f342e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.094+0000 7f342e7fc640 10 monclient: dump: 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.110 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3428031ce0 con 0x7f34300985a0 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f342e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f342e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 msgr2=0x7f3430133690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f342e7fc640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 0x7f3430133690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3428034000 tx=0x7f3428032000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 msgr2=0x7f3430133690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f3435b38640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3424007fd0 tx=0x7f3424007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.095+0000 7f342e7fc640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34300985a0 0x7f3430133690 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.104+0000 7f7c258d4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T12:34:10.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.104+0000 7f7c258d4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T12:34:10.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.104+0000 7f7c258d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:10.113 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.104+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f7c200008d0 con 0x7f7c2009ec50 2024-09-18T12:34:10.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.100+0000 7f343859d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.100+0000 7f343859d640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 0x7f3430133fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.100+0000 7f343859d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.100+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3430063740 con 0x7f3430133bd0 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 0x7f3430133fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient: get_auth_request con 0x7f3430133bd0 auth_method 0 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 10 monclient(hunting): handle_auth_done global_id 104625 payload 824 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.101+0000 7f3436339640 1 -- 172.21.15.116:0/4104179221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3430066280 con 0x7f3430133bd0 2024-09-18T12:34:10.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f3436339640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 msgr2=0x7f3430133fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f3436339640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 0x7f3430133fd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3428009130 tx=0x7f342803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34280468f0 con 0x7f3430133bd0 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 10 monclient: dump: 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.116 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f3428031970 con 0x7f3430133bd0 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 msgr2=0x7f3430133fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 0x7f3430133fd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3428009130 tx=0x7f342803d000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 msgr2=0x7f3430133fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.103+0000 7f342e7fc640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430133bd0 0x7f3430133fd0 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3428009130 tx=0x7f342803d000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.117 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 94, 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.122 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:10.125 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.128 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.130 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:10.130 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.130 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.130 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:10.130 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:10.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 msgr2=0x7f341c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3424007fd0 tx=0x7f3424007f00 comp rx=0 tx=0).stop 2024-09-18T12:34:10.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 10 monclient: shutdown 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430136bb0 msgr2=0x7f3430098060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 --2- 172.21.15.116:0/4104179221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3430136bb0 0x7f3430098060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3420009fa0 tx=0x7f3420009900 comp rx=0 tx=0).stop 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f3436b3a640 1 -- 172.21.15.116:0/4104179221 reap_dead start 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f3436b3a640 1 -- 172.21.15.116:0/4104179221 reap_dead start 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 shutdown_connections 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 >> 172.21.15.116:0/4104179221 conn(0x7f3430067d30 msgr2=0x7f343009c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 shutdown_connections 2024-09-18T12:34:10.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.104+0000 7f343859d640 1 -- 172.21.15.116:0/4104179221 wait complete. 2024-09-18T12:34:10.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:34:10.133 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:34:10.133 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:34:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.226+0000 7fb1efc46640 10 monclient: get_monmap_and_config 2024-09-18T12:34:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.226+0000 7fb1efc46640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.226+0000 7fb1efc46640 10 monclient: monmap: 2024-09-18T12:34:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:10.226949+0000 2024-09-18T12:34:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:10.226949+0000 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.226+0000 7fb1efc46640 1 Processor -- start 2024-09-18T12:34:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 1 -- start start 2024-09-18T12:34:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient: init 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e81318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1e809ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1e8136c20 0x7fb1e8139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e8136c20 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e8131cc0 con 0x7fb1e809ea50 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e809f530 con 0x7fb1e8136c20 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e809fbf0 con 0x7fb1e81318f0 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1efc46640 10 monclient(hunting): authenticate will time out at 1277.548546s 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1e8136c20 0x7fb1e8139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1e8136c20 0x7fb1e8139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45880/0 (socket says 172.21.15.116:45880) 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e8131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43392/0 (socket says 172.21.15.116:43392) 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 1 -- 172.21.15.116:0/3383456727 learned_addr learned my addr 172.21.15.116:0/3383456727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed1e1640 1 --1- 172.21.15.116:0/3383456727 >> v1:172.21.15.144:6789/0 conn(0x7fb1e809ea50 0x7fb1e809ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:45304/0 (socket says 172.21.15.116:45304) 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 10 monclient(hunting): get_auth_request con 0x7fb1e8136c20 auth_method 0 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): get_auth_request con 0x7fb1e81318f0 auth_method 0 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ee1e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_done global_id 65396 payload 274 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/3383456727 >> v2:172.21.15.116:3300/0 conn(0x7fb1e8136c20 msgr2=0x7fb1e8139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 1 --2- 172.21.15.116:0/3383456727 >> v2:172.21.15.116:3300/0 conn(0x7fb1e8136c20 0x7fb1e8139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/3383456727 >> v1:172.21.15.144:6789/0 conn(0x7fb1e809ea50 legacy=0x7fb1e809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 1 monclient: found mon.noname-b 2024-09-18T12:34:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.227+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/3383456727 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e809fe30 con 0x7fb1e81318f0 2024-09-18T12:34:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.228+0000 7fb1ee1e3640 1 --2- 172.21.15.116:0/3383456727 >> v2:172.21.15.116:3300/0 conn(0x7fb1e8136c20 0x7fb1e8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/3383456727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 msgr2=0x7fb1e8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ed9e2640 1 --2- 172.21.15.116:0/3383456727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e8131f30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009850 tx=0x7fb1e00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c772a693d264fe35 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 1 -- 172.21.15.116:0/3383456727 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e002add0 con 0x7fb1e81318f0 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: dump: 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1efc46640 5 monclient: authenticate success, global_id 65396 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1efc46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 1 -- 172.21.15.116:0/3383456727 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e0032440 con 0x7fb1e81318f0 2024-09-18T12:34:10.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 1 -- 172.21.15.116:0/3383456727 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00327e0 con 0x7fb1e81318f0 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1efc46640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1efc46640 4 set_mon_vals no callback set 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.230+0000 7fb1ec9e0640 10 monclient: dump: 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.243 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 10 monclient: shutdown 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 -- 172.21.15.116:0/3383456727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 msgr2=0x7fb1e8131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 --2- 172.21.15.116:0/3383456727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e8131f30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009850 tx=0x7fb1e00090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 -- 172.21.15.116:0/3383456727 shutdown_connections 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 --2- 172.21.15.116:0/3383456727 >> v2:172.21.15.116:3300/0 conn(0x7fb1e8136c20 0x7fb1e8139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 --2- 172.21.15.116:0/3383456727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e8131f30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 -- 172.21.15.116:0/3383456727 >> 172.21.15.116:0/3383456727 conn(0x7fb1e8067110 msgr2=0x7fb1e8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 -- 172.21.15.116:0/3383456727 shutdown_connections 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.231+0000 7fb1efc46640 1 -- 172.21.15.116:0/3383456727 wait complete. 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient: monmap: 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:10.245 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 Processor -- start 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 -- start start 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient: init 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1e809ea50 0x7fb1e81308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e809ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e80998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e81318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 0x7fb1e8099e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e8136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e809bf10 con 0x7fb1e8136c20 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e809b8d0 con 0x7fb1e809ea50 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e809e060 con 0x7fb1e81318f0 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1efc46640 10 monclient(hunting): authenticate will time out at 1277.553898s 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1ee1e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 0x7fb1e8099e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.232+0000 7fb1ed1e1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e80998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 0x7fb1e8099e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45886/0 (socket says 172.21.15.116:45886) 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 -- 172.21.15.116:0/152410379 learned_addr learned my addr 172.21.15.116:0/152410379 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): get_auth_request con 0x7fb1e8136c20 auth_method 0 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 10 monclient(hunting): get_auth_request con 0x7fb1e81318f0 auth_method 0 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed9e2640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1e809ea50 0x7fb1e81308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed9e2640 10 monclient(hunting): get_auth_request con 0x7fb1e809ea50 auth_method 0 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed9e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient(hunting): handle_auth_done global_id 104631 payload 824 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 msgr2=0x7fb1e80998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e80998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1e809ea50 msgr2=0x7fb1e81308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1e809ea50 0x7fb1e81308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 monclient: found mon.noname-a 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ee1e3640 1 -- 172.21.15.116:0/152410379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d8009ae0 con 0x7fb1e8136c20 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.233+0000 7fb1ed1e1640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e81318f0 0x7fb1e80998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1ee1e3640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 msgr2=0x7fb1e8099e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1ee1e3640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 0x7fb1e8099e00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009fb0 tx=0x7fb1d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d800ce70 con 0x7fb1e8136c20 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: dump: 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d800f040 con 0x7fb1e8136c20 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1efc46640 5 monclient: authenticate success, global_id 104631 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80146f0 con 0x7fb1e8136c20 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1efc46640 10 monclient: _renew_subs 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1efc46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e0009390 con 0x7fb1e8136c20 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e5ffb640 10 monclient: dump: 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.252 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1e7fff640 4 set_mon_vals no callback set 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1efc46640 10 monclient: _renew_subs 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1efc46640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.235+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e809b3c0 con 0x7fb1e8136c20 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb1d8014a60 con 0x7fb1e8136c20 2024-09-18T12:34:10.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1efc46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1efc46640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 0x7fb1e809a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1efc46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e8065e00 con 0x7fb1e8130e10 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1e5ffb640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb1d4038490 0x7fb1d403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(253..253 src has 13..253) v4 ==== 9184+0+0 (secure 0 0 0) 0x7fb1d804d100 con 0x7fb1e8136c20 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 0x7fb1e809a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 10 monclient: get_auth_request con 0x7fb1e8130e10 auth_method 0 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.236+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.237+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_done global_id 65402 payload 824 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.237+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/152410379 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1e811cd50 con 0x7fb1e8130e10 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.237+0000 7fb1ed1e1640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb1d4038490 0x7fb1d403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.237+0000 7fb1ed1e1640 10 monclient: get_auth_request con 0x7fb1d4038490 auth_method 0 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 msgr2=0x7fb1e809a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1ed9e2640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 0x7fb1e809a340 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0034000 tx=0x7fb1e0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0031ca0 con 0x7fb1e8130e10 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 10 monclient: dump: 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb1e0031e00 con 0x7fb1e8130e10 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1ed1e1640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb1d4038490 0x7fb1d403a940 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb1e809b050 tx=0x7fb1dc007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 msgr2=0x7fb1e809a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 0x7fb1e809a340 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0034000 tx=0x7fb1e0032000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 msgr2=0x7fb1e809a340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.239+0000 7fb1e5ffb640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e8130e10 0x7fb1e809a340 secure :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0034000 tx=0x7fb1e0032000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1efc46640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1efc46640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 0x7fb1e81347e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1efc46640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e809ac80 con 0x7fb1e809a880 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 0x7fb1e81347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient: get_auth_request con 0x7fb1e809a880 auth_method 0 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 10 monclient(hunting): handle_auth_done global_id 65405 payload 824 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.245+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/152410379 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb1e804bf60 con 0x7fb1e809a880 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.247+0000 7fb1ed9e2640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 msgr2=0x7fb1e81347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.247+0000 7fb1ed9e2640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 0x7fb1e81347e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8142010 tx=0x7fb1e003c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e003bbf0 con 0x7fb1e809a880 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 10 monclient: dump: 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.260 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fb1e004b070 con 0x7fb1e809a880 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 msgr2=0x7fb1e81347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 0x7fb1e81347e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8142010 tx=0x7fb1e003c000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 msgr2=0x7fb1e81347e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1e5ffb640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb1e809a880 0x7fb1e81347e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 94, 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.262 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.263 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:10.264 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.265 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:10.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.267 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.268 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:10.269 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.272 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:10.273 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb1d4038490 msgr2=0x7fb1d403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb1d4038490 0x7fb1d403a940 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb1e809b050 tx=0x7fb1dc007e40 comp rx=0 tx=0).stop 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 10 monclient: shutdown 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 msgr2=0x7fb1e8099e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 0x7fb1e8099e00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009fb0 tx=0x7fb1d800b040 comp rx=0 tx=0).stop 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1ee1e3640 1 -- 172.21.15.116:0/152410379 reap_dead start 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 shutdown_connections 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 --2- 172.21.15.116:0/152410379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e8136c20 0x7fb1e8099e00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 >> 172.21.15.116:0/152410379 conn(0x7fb1e8067110 msgr2=0x7fb1e809e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 shutdown_connections 2024-09-18T12:34:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.248+0000 7fb1efc46640 1 -- 172.21.15.116:0/152410379 wait complete. 2024-09-18T12:34:10.276 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:34:10.276 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:34:10.276 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:34:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.368+0000 7fd9b5109640 10 monclient: get_monmap_and_config 2024-09-18T12:34:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.368+0000 7fd9b5109640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.368+0000 7fd9b5109640 10 monclient: monmap: 2024-09-18T12:34:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.369 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:10.369244+0000 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:10.369244+0000 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:10.370 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 1 Processor -- start 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 1 -- start start 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient: init 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9b0096c60 0x7fd9b009a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b0096c60 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9b0097030 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9b0133b40 0x7fd9b0135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient(hunting): picked mon.noname-c con 0x7fd9b0133b40 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b00696a0 con 0x7fd9b0097030 2024-09-18T12:34:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b009ac40 con 0x7fd9b0096c60 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b009d6b0 con 0x7fd9b0133b40 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9b5109640 10 monclient(hunting): authenticate will time out at 1277.690679s 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9b0096c60 0x7fd9b009a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9b0096c60 0x7fd9b009a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45898/0 (socket says 172.21.15.116:45898) 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 1 -- 172.21.15.116:0/4193857233 learned_addr learned my addr 172.21.15.116:0/4193857233 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 10 monclient(hunting): get_auth_request con 0x7fd9b0096c60 auth_method 0 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9af59e640 1 --2- 172.21.15.116:0/4193857233 >> v2:172.21.15.144:3300/0 conn(0x7fd9b0133b40 0x7fd9b0135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9af59e640 10 monclient(hunting): get_auth_request con 0x7fd9b0133b40 auth_method 0 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9af59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9af59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.369+0000 7fd9af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_done global_id 104634 payload 274 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 1 -- 172.21.15.116:0/4193857233 >> v2:172.21.15.144:3300/0 conn(0x7fd9b0133b40 msgr2=0x7fd9b0135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 1 --2- 172.21.15.116:0/4193857233 >> v2:172.21.15.144:3300/0 conn(0x7fd9b0133b40 0x7fd9b0135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 1 -- 172.21.15.116:0/4193857233 >> v1:172.21.15.144:6789/0 conn(0x7fd9b0097030 legacy=0x7fd9b0097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 1 monclient: found mon.noname-a 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.370+0000 7fd9aed9d640 1 -- 172.21.15.116:0/4193857233 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b009d2e0 con 0x7fd9b0096c60 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.372+0000 7fd9aed9d640 1 -- 172.21.15.116:0/4193857233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0096c60 msgr2=0x7fd9b009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.372+0000 7fd9aed9d640 1 --2- 172.21.15.116:0/4193857233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0096c60 0x7fd9b009a6c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0097b70 tx=0x7fd9a00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0aa9e710068f5c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 1 -- 172.21.15.116:0/4193857233 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0032440 con 0x7fd9b0096c60 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: dump: 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 1 -- 172.21.15.116:0/4193857233 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a00325a0 con 0x7fd9b0096c60 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 5 monclient: authenticate success, global_id 104634 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 1 -- 172.21.15.116:0/4193857233 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0032940 con 0x7fd9b0096c60 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 4 set_mon_vals no callback set 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9add9b640 10 monclient: dump: 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.380 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.380 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.380 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.380 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.380 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.380 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 10 monclient: shutdown 2024-09-18T12:34:10.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 -- 172.21.15.116:0/4193857233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0096c60 msgr2=0x7fd9b009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 --2- 172.21.15.116:0/4193857233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0096c60 0x7fd9b009a6c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0097b70 tx=0x7fd9a00091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 -- 172.21.15.116:0/4193857233 shutdown_connections 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 --2- 172.21.15.116:0/4193857233 >> v2:172.21.15.144:3300/0 conn(0x7fd9b0133b40 0x7fd9b0135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 --2- 172.21.15.116:0/4193857233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0096c60 0x7fd9b009a6c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 -- 172.21.15.116:0/4193857233 >> 172.21.15.116:0/4193857233 conn(0x7fd9b00672b0 msgr2=0x7fd9b00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 -- 172.21.15.116:0/4193857233 shutdown_connections 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.373+0000 7fd9b5109640 1 -- 172.21.15.116:0/4193857233 wait complete. 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.374+0000 7fd9b5109640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.374+0000 7fd9b5109640 10 monclient: monmap: 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:10.387 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.374+0000 7fd9b5109640 1 Processor -- start 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.374+0000 7fd9b5109640 1 -- start start 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.374+0000 7fd9b5109640 10 monclient: init 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:10.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9b0096c60 0x7fd9b006e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): picked mon.noname-c con 0x7fd9b0096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0097030 0x7fd9b012e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): picked mon.noname-b con 0x7fd9b0097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0133b40 0x7fd9b0117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b0133b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b0119b10 con 0x7fd9b0133b40 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b009add0 con 0x7fd9b0097030 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b013de00 con 0x7fd9b0096c60 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9b5109640 10 monclient(hunting): authenticate will time out at 1277.696272s 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9b0096c60 0x7fd9b006e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0133b40 0x7fd9b0117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9b0096c60 0x7fd9b006e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43430/0 (socket says 172.21.15.116:43430) 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 -- 172.21.15.116:0/3107098971 learned_addr learned my addr 172.21.15.116:0/3107098971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): get_auth_request con 0x7fd9b0096c60 auth_method 0 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 10 monclient(hunting): get_auth_request con 0x7fd9b0133b40 auth_method 0 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0097030 0x7fd9b012e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 10 monclient(hunting): get_auth_request con 0x7fd9b0097030 auth_method 0 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient(hunting): handle_auth_done global_id 65408 payload 824 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0097030 msgr2=0x7fd9b012e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0097030 0x7fd9b012e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0133b40 msgr2=0x7fd9b0117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0133b40 0x7fd9b0117480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 monclient: found mon.noname-c 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9aed9d640 1 -- 172.21.15.116:0/3107098971 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a0009450 con 0x7fd9b0096c60 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.375+0000 7fd9af59e640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0133b40 0x7fd9b0117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.377+0000 7fd9aed9d640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9b0096c60 msgr2=0x7fd9b006e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.377+0000 7fd9aed9d640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9b0096c60 0x7fd9b006e6a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0035040 tx=0x7fd9a0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00319c0 con 0x7fd9b0096c60 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: dump: 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.393 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a0031b40 con 0x7fd9b0096c60 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 5 monclient: authenticate success, global_id 65408 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 10 monclient: _renew_subs 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b0133790 con 0x7fd9b0096c60 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a003d440 con 0x7fd9b0096c60 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd996ffd640 10 monclient: dump: 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.394 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 10 monclient: _renew_subs 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b0119830 con 0x7fd9b0096c60 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9ad59a640 4 set_mon_vals no callback set 2024-09-18T12:34:10.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 0x7fd9b00298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.378+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b0065f20 con 0x7fd9b012ec70 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd9a0039040 con 0x7fd9b0096c60 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 0x7fd9b00298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient: get_auth_request con 0x7fd9b012ec70 auth_method 0 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd996ffd640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd99c038490 0x7fd99c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(253..253 src has 13..253) v4 ==== 9184+0+0 (secure 0 0 0) 0x7fd9a006c380 con 0x7fd9b0096c60 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9af59e640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd99c038490 0x7fd99c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_done global_id 124123 payload 824 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9ae59c640 1 -- 172.21.15.116:0/3107098971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9b004bf60 con 0x7fd9b012ec70 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.379+0000 7fd9af59e640 10 monclient: get_auth_request con 0x7fd99c038490 auth_method 0 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.381+0000 7fd9ae59c640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 msgr2=0x7fd9b00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.381+0000 7fd9ae59c640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 0x7fd9b00298c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd9b009ff70 tx=0x7fd9a4007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.381+0000 7fd9af59e640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd99c038490 0x7fd99c03a940 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b5a0 tx=0x7fd9a800d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.381+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400d7e0 con 0x7fd9b012ec70 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.381+0000 7fd996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.381+0000 7fd996ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.381+0000 7fd996ffd640 10 monclient: dump: 2024-09-18T12:34:10.397 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.382+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd9a400d940 con 0x7fd9b012ec70 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.382+0000 7fd996ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.382+0000 7fd996ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:10.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.382+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 msgr2=0x7fd9b00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.382+0000 7fd996ffd640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 0x7fd9b00298c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd9b009ff70 tx=0x7fd9a4007e10 comp rx=0 tx=0).stop 2024-09-18T12:34:10.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.382+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 msgr2=0x7fd9b00298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:10.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.382+0000 7fd996ffd640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b012ec70 0x7fd9b00298c0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd9b009ff70 tx=0x7fd9a4007e10 comp rx=0 tx=0).stop 2024-09-18T12:34:10.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9b5109640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9b5109640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 0x7fd9b002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9b5109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b0119d30 con 0x7fd9b0029e00 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 0x7fd9b002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient: get_auth_request con 0x7fd9b0029e00 auth_method 0 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 10 monclient(hunting): handle_auth_done global_id 124126 payload 824 2024-09-18T12:34:10.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.388+0000 7fd9ae59c640 1 -- 172.21.15.116:0/3107098971 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd9b00983b0 con 0x7fd9b0029e00 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.390+0000 7fd9ae59c640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 msgr2=0x7fd9b002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.390+0000 7fd9ae59c640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 0x7fd9b002a200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4018000 tx=0x7fd9a4019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4014e20 con 0x7fd9b0029e00 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 10 monclient: dump: 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.402 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fd9a401d610 con 0x7fd9b0029e00 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 msgr2=0x7fd9b002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 0x7fd9b002a200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4018000 tx=0x7fd9a4019000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 msgr2=0x7fd9b002a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd996ffd640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9b0029e00 0x7fd9b002a200 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4018000 tx=0x7fd9a4019000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.403 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 94, 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:10.404 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.405 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.406 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:10.407 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:10.408 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.409 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.410 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:10.411 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.412 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.414 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:10.415 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:10.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:10.416 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:10.416 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:10.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd99c038490 msgr2=0x7fd99c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd99c038490 0x7fd99c03a940 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b5a0 tx=0x7fd9a800d040 comp rx=0 tx=0).stop 2024-09-18T12:34:10.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 10 monclient: shutdown 2024-09-18T12:34:10.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9b0096c60 msgr2=0x7fd9b006e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 1 --2- 172.21.15.116:0/3107098971 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9b0096c60 0x7fd9b006e6a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0035040 tx=0x7fd9a0032000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9af59e640 1 -- 172.21.15.116:0/3107098971 reap_dead start 2024-09-18T12:34:10.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9af59e640 1 -- 172.21.15.116:0/3107098971 reap_dead start 2024-09-18T12:34:10.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 shutdown_connections 2024-09-18T12:34:10.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 >> 172.21.15.116:0/3107098971 conn(0x7fd9b00672b0 msgr2=0x7fd9b00a0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.391+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 shutdown_connections 2024-09-18T12:34:10.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.392+0000 7fd9b5109640 1 -- 172.21.15.116:0/3107098971 wait complete. 2024-09-18T12:34:10.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:34:10.418 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:34:10.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:34:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.510+0000 7f07596e8640 10 monclient: get_monmap_and_config 2024-09-18T12:34:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.510+0000 7f07596e8640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.510+0000 7f07596e8640 10 monclient: monmap: 2024-09-18T12:34:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:10.510848+0000 2024-09-18T12:34:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:10.510848+0000 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.510+0000 7f07596e8640 1 Processor -- start 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.510+0000 7f07596e8640 1 -- start start 2024-09-18T12:34:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.510+0000 7f07596e8640 10 monclient: init 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07541318f0 0x7f0754131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): picked mon.noname-a con 0x7f07541318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f075409ea50 0x7f075409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): picked mon.noname-c con 0x7f075409ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f07540691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): picked mon.noname-b con 0x7f0754068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0754131cc0 con 0x7f07541318f0 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075409fbf0 con 0x7f075409ea50 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075409fe30 con 0x7f0754068d90 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07596e8640 10 monclient(hunting): authenticate will time out at 1277.832297s 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f07540691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07541318f0 0x7f0754131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f07540691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43438/0 (socket says 172.21.15.116:43438) 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07541318f0 0x7f0754131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45922/0 (socket says 172.21.15.116:45922) 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 -- 172.21.15.116:0/2864898729 learned_addr learned my addr 172.21.15.116:0/2864898729 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): get_auth_request con 0x7f0754068d90 auth_method 0 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 10 monclient(hunting): get_auth_request con 0x7f07541318f0 auth_method 0 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 1 --2- 172.21.15.116:0/2864898729 >> v2:172.21.15.144:3300/0 conn(0x7f075409ea50 0x7f075409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 10 monclient(hunting): get_auth_request con 0x7f075409ea50 auth_method 0 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient(hunting): handle_auth_done global_id 65411 payload 274 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 -- 172.21.15.116:0/2864898729 >> v2:172.21.15.144:3300/0 conn(0x7f075409ea50 msgr2=0x7f075409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 --2- 172.21.15.116:0/2864898729 >> v2:172.21.15.144:3300/0 conn(0x7f075409ea50 0x7f075409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 -- 172.21.15.116:0/2864898729 >> v2:172.21.15.116:3300/0 conn(0x7f07541318f0 msgr2=0x7f0754131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 --2- 172.21.15.116:0/2864898729 >> v2:172.21.15.116:3300/0 conn(0x7f07541318f0 0x7f0754131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 monclient: found mon.noname-b 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07537fe640 1 -- 172.21.15.116:0/2864898729 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07540a1210 con 0x7f0754068d90 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f0752ffd640 1 --2- 172.21.15.116:0/2864898729 >> v2:172.21.15.116:3300/0 conn(0x7f07541318f0 0x7f0754131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.511+0000 7f07527fc640 1 --2- 172.21.15.116:0/2864898729 >> v2:172.21.15.144:3300/0 conn(0x7f075409ea50 0x7f075409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f07537fe640 1 -- 172.21.15.116:0/2864898729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 msgr2=0x7f07540691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f07537fe640 1 --2- 172.21.15.116:0/2864898729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f07540691d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f075406f600 tx=0x7f074c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b0f94580b756e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 1 -- 172.21.15.116:0/2864898729 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c032440 con 0x7f0754068d90 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: dump: 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 1 -- 172.21.15.116:0/2864898729 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074c0325a0 con 0x7f0754068d90 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 1 -- 172.21.15.116:0/2864898729 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c032940 con 0x7f0754068d90 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f07596e8640 5 monclient: authenticate success, global_id 65411 2024-09-18T12:34:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f07596e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f07596e8640 4 set_mon_vals no callback set 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.514+0000 7f0751ffb640 10 monclient: dump: 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.527 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 10 monclient: shutdown 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 -- 172.21.15.116:0/2864898729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 msgr2=0x7f07540691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 --2- 172.21.15.116:0/2864898729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f07540691d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f075406f600 tx=0x7f074c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 -- 172.21.15.116:0/2864898729 shutdown_connections 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 --2- 172.21.15.116:0/2864898729 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f07540691d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 --2- 172.21.15.116:0/2864898729 >> v2:172.21.15.144:3300/0 conn(0x7f075409ea50 0x7f075409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 --2- 172.21.15.116:0/2864898729 >> v2:172.21.15.116:3300/0 conn(0x7f07541318f0 0x7f0754131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 -- 172.21.15.116:0/2864898729 >> 172.21.15.116:0/2864898729 conn(0x7f0754067110 msgr2=0x7f0754067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 -- 172.21.15.116:0/2864898729 shutdown_connections 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 1 -- 172.21.15.116:0/2864898729 wait complete. 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.515+0000 7f07596e8640 10 monclient: monmap: 2024-09-18T12:34:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 Processor -- start 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 -- start start 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient: init 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f075409a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): picked mon.noname-c con 0x7f0754068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f075409ea50 0x7f0754130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): picked mon.noname-a con 0x7f075409ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07541318f0 0x7f0754140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): picked mon.noname-b con 0x7f07541318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075409b940 con 0x7f075409ea50 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075411dc30 con 0x7f07541318f0 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075411e1f0 con 0x7f0754068d90 2024-09-18T12:34:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07596e8640 10 monclient(hunting): authenticate will time out at 1277.837687s 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f075409a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f075409ea50 0x7f0754130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f075409a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43442/0 (socket says 172.21.15.116:43442) 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f075409ea50 0x7f0754130b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45932/0 (socket says 172.21.15.116:45932) 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 1 -- 172.21.15.116:0/1440909010 learned_addr learned my addr 172.21.15.116:0/1440909010 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): get_auth_request con 0x7f0754068d90 auth_method 0 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 10 monclient(hunting): get_auth_request con 0x7f075409ea50 auth_method 0 2024-09-18T12:34:10.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f07527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.516+0000 7f0752ffd640 10 monclient(hunting): handle_auth_done global_id 65414 payload 824 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f07537fe640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07541318f0 0x7f0754140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 1 -- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07541318f0 msgr2=0x7f0754140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07541318f0 0x7f0754140b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 1 -- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f075409ea50 msgr2=0x7f0754130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f075409ea50 0x7f0754130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 1 monclient: found mon.noname-c 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f0752ffd640 1 -- 172.21.15.116:0/1440909010 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0748009390 con 0x7f0754068d90 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.517+0000 7f07527fc640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f075409ea50 0x7f0754130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f0752ffd640 1 -- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 msgr2=0x7f075409a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f0752ffd640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f075409a900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f074400c170 tx=0x7f074400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 1 -- 172.21.15.116:0/1440909010 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744014690 con 0x7f0754068d90 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: dump: 2024-09-18T12:34:10.533 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07596e8640 5 monclient: authenticate success, global_id 65414 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 1 -- 172.21.15.116:0/1440909010 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074400f040 con 0x7f0754068d90 2024-09-18T12:34:10.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 1 -- 172.21.15.116:0/1440909010 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744014a20 con 0x7f0754068d90 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f073affd640 10 monclient: dump: 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.535 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07596e8640 10 monclient: _renew_subs 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07596e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07517fa640 4 set_mon_vals no callback set 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0744009a20 con 0x7f0754068d90 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07596e8640 10 monclient: _renew_subs 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07596e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.519+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c009450 con 0x7f0754068d90 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f073affd640 1 -- 172.21.15.116:0/1440909010 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f074401d400 con 0x7f0754068d90 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f07596e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f07596e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f07596e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075404bf60 con 0x7f0754068d90 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f073affd640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0740038490 0x7f074003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f073affd640 1 -- 172.21.15.116:0/1440909010 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(253..253 src has 13..253) v4 ==== 9184+0+0 (secure 0 0 0) 0x7f074404d870 con 0x7f0754068d90 2024-09-18T12:34:10.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f07527fc640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0740038490 0x7f074003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.520+0000 7f07527fc640 10 monclient: get_auth_request con 0x7f0740038490 auth_method 0 2024-09-18T12:34:10.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.523+0000 7f07527fc640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0740038490 0x7f074003a940 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f075411d640 tx=0x7f07480090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.524+0000 7f073affd640 1 -- 172.21.15.116:0/1440909010 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f07440133e0 con 0x7f0754068d90 2024-09-18T12:34:10.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.525+0000 7f073affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.525+0000 7f073affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:10.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.740+0000 7f07596e8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:34:10.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.740+0000 7f07596e8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:34:10.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.740+0000 7f07596e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:10.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.740+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f075411df10 con 0x7f0754068d90 2024-09-18T12:34:10.743 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.743 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=1957933107,config=3140514009,health=2127886669,kv=2187923472,logm=1282580313}) 2024-09-18T12:34:10.743 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=11,kv=9,logm=59} crc {auth=1815234670,config=3140514009,health=1862850906,kv=2187923472,logm=2235874925}) 2024-09-18T12:34:10.743 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.743 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=1957933107,config=3140514009,health=2127886669,kv=2187923472,logm=1282580313}) 2024-09-18T12:34:10.743 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=1957933107,config=3140514009,health=2127886669,kv=2187923472,logm=779203346}) 2024-09-18T12:34:10.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f073affd640 1 -- 172.21.15.116:0/1440909010 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0744012340 con 0x7f0754068d90 2024-09-18T12:34:10.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f073affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:34:10.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.742+0000 7f073affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:10.744 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.743+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.744 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.743+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743984620}) 2024-09-18T12:34:10.744 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.743+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025274795}) 2024-09-18T12:34:10.744 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.743+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.743+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743984620}) 2024-09-18T12:34:10.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.743+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243470223}) 2024-09-18T12:34:10.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230096743}) 2024-09-18T12:34:10.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045147560}) 2024-09-18T12:34:10.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0740038490 msgr2=0x7f074003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.746 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230096743}) 2024-09-18T12:34:10.746 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2924404984}) 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f07596e8640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0740038490 0x7f074003a940 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f075411d640 tx=0x7f07480090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f07596e8640 10 monclient: shutdown 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f07596e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.744+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 msgr2=0x7f075409a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f075409a900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f074400c170 tx=0x7f074400b040 comp rx=0 tx=0).stop 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 shutdown_connections 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0740038490 0x7f074003a940 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07541318f0 0x7f0754140b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f075409ea50 0x7f0754130b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 --2- 172.21.15.116:0/1440909010 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0754068d90 0x7f075409a900 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 >> 172.21.15.116:0/1440909010 conn(0x7f0754067110 msgr2=0x7f07540a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 shutdown_connections 2024-09-18T12:34:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f07596e8640 1 -- 172.21.15.116:0/1440909010 wait complete. 2024-09-18T12:34:10.749 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.749 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989073316}) 2024-09-18T12:34:10.749 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624112914}) 2024-09-18T12:34:10.749 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.749 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989073316}) 2024-09-18T12:34:10.749 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.745+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3028425786}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.746+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.746+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749058521}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.746+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803904499}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.746+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.746+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749058521}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.746+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1201649000}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.747+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.747+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866279895}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.747+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893473716}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.747+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.747+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866279895}) 2024-09-18T12:34:10.750 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.747+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316090177}) 2024-09-18T12:34:10.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.754+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v254) v1 ==== 121+0+0 (secure 0 0 0) 0x7f7c10013300 con 0x7f7c2009ec50 2024-09-18T12:34:10.756 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.754+0000 7f7c06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T12:34:10.757 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.754+0000 7f7c06ffd640 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-18T12:34:10.757 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.755+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.757 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.755+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540909594}) 2024-09-18T12:34:10.757 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.755+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363814143}) 2024-09-18T12:34:10.757 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.757 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540909594}) 2024-09-18T12:34:10.758 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156276431}) 2024-09-18T12:34:10.758 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:34:10.762 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.762 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937671113}) 2024-09-18T12:34:10.762 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133109447}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937671113}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.756+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469114255}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.757+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.757+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425276146}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.757+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721861153}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.757+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.757+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425276146}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.757+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3658659552}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141751930}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1726735427}) 2024-09-18T12:34:10.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141751930}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704130437}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.758+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525623671}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2473289578}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525623671}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2025691203}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040450796}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=99981515}) 2024-09-18T12:34:10.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040450796}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.759+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229392612}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.760+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.760+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425925817}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.760+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=630034580}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.760+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.760+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425925817}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.760+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270603108}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498613060}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294278054}) 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498613060}) 2024-09-18T12:34:10.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808527051}) 2024-09-18T12:34:10.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709514124}) 2024-09-18T12:34:10.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2904749381}) 2024-09-18T12:34:10.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709514124}) 2024-09-18T12:34:10.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.761+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564723653}) 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.762+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.762+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812732098}) 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.762+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=248672000}) 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.762+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.762+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812732098}) 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.762+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363655096}) 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48859025}) 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284672440}) 2024-09-18T12:34:10.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48859025}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260489649}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960871180}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=343180327}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960871180}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.763+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2985597631}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252371504}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304533499}) 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252371504}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105010518}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943619963}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.764+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3517691428}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943619963}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122725993}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=847120331}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3300590465}) 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=847120331}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.765+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327245575}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754301251}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=581260879}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754301251}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137215607}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=113169179}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=584089593}) 2024-09-18T12:34:10.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=113169179}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.766+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824385459}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.767+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.767+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=183547064}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.767+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747897344}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.767+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.767+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=183547064}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.767+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45737415}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246758965}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96860001}) 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246758965}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2792399536}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090822678}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3337732031}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090822678}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.768+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3987093169}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.769+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.769+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030269523}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.769+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775032248}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.769+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.769+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030269523}) 2024-09-18T12:34:10.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.769+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605404807}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142473597}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367745745}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142473597}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927806930}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456448089}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2341146979}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456448089}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.770+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225573354}) 2024-09-18T12:34:10.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.771+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.771+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=452714276}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.771+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37432614}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.771+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.771+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=452714276}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.771+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3371071184}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.771+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917820946}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3097174795}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917820946}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081865271}) 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029089699}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882050180}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029089699}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.772+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358669083}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734553706}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405601715}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734553706}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700689121}) 2024-09-18T12:34:10.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658403767}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1267701987}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658403767}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.773+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979388990}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.774+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.774+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109070160}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.774+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2281510513}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.774+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.774+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109070160}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.774+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2561330489}) 2024-09-18T12:34:10.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679761105}) 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851054124}) 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679761105}) 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4148425610}) 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331804393}) 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4261923010}) 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331804393}) 2024-09-18T12:34:10.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.775+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229189843}) 2024-09-18T12:34:10.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.776+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.776+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790965357}) 2024-09-18T12:34:10.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.776+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4111636887}) 2024-09-18T12:34:10.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.776+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.776+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790965357}) 2024-09-18T12:34:10.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.776+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=314449657}) 2024-09-18T12:34:10.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482598645}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274266399}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482598645}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2115964726}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121936293}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1143798022}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121936293}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.777+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685502628}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241617024}) 2024-09-18T12:34:10.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509423245}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241617024}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3784581665}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216779480}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851471518}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216779480}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.778+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260541286}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.779+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.779+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303713344}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.779+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1834692219}) 2024-09-18T12:34:10.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.779+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.779+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303713344}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.779+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=486125719}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118007721}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4111715817}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118007721}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1852938170}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605140919}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193688027}) 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.796 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605140919}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.780+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2674336466}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.781+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.781+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {logm=3086857815,mdsmap=1531151339,mgr=3328594750,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4049335394,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=3927986319,osd_snap=2033388227,osdmap=765294349}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.781+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {logm=685310753,mdsmap=3140514009,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=505004825,mon_config_key=1611758519,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=3927986319,osd_snap=2033388227,osdmap=1964664266}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.781+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.781+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {logm=3086857815,mdsmap=1531151339,mgr=3328594750,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4049335394,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=3927986319,osd_snap=2033388227,osdmap=765294349}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.781+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {logm=3086857815,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4049335394,mon_config_key=3655569570,monmap=148614106,osd_metadata=4229638783,osd_pg_creating=3927986319,osd_snap=2033388227,osdmap=2658524980}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.782+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.782+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3608146116}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.782+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=769233845}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.782+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.782+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3608146116}) 2024-09-18T12:34:10.797 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.782+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2680761246}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.783+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.783+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2556280157}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.783+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1485394560}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.783+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.783+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2556280157}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.783+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2453936991}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.785+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.785+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3319979507}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.785+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=684876245}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.785+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.785+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3319979507}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.785+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=400113512}) 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.790+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:34:10.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.790+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=12} crc {osdmap=2439037504}) 2024-09-18T12:34:10.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:10.790+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=9} crc {osdmap=970364817}) 2024-09-18T12:34:10.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.855+0000 7f7c258d4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T12:34:10.857 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.855+0000 7f7c258d4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T12:34:10.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.855+0000 7f7c258d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:10.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.855+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7f7c200a13e0 con 0x7f7c2009ec50 2024-09-18T12:34:10.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.856+0000 7f7c06ffd640 1 -- 172.21.15.144:0/1464330520 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v254) v1 ==== 120+0+0 (secure 0 0 0) 0x7f7c1001d360 con 0x7f7c2009ec50 2024-09-18T12:34:10.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.856+0000 7f7c06ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T12:34:10.858 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.856+0000 7f7c06ffd640 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-18T12:34:10.858 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h28' does not appear in the crush map 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c0c038490 msgr2=0x7f7c0c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c0c038490 0x7f7c0c03a940 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7c14031000 tx=0x7f7c14032000 comp rx=0 tx=0).stop 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 10 monclient: shutdown 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 msgr2=0x7f7c2009a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 0x7f7c2009a980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7c10007eb0 tx=0x7f7c10009330 comp rx=0 tx=0).stop 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 shutdown_connections 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c0c038490 0x7f7c0c03a940 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c201321a0 0x7f7c2011b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2009f020 0x7f7c20130a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 --2- 172.21.15.144:0/1464330520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c2009ec50 0x7f7c2009a980 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 >> 172.21.15.144:0/1464330520 conn(0x7f7c200a5cf0 msgr2=0x7f7c20134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 shutdown_connections 2024-09-18T12:34:10.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.858+0000 7f7c258d4640 1 -- 172.21.15.144:0/1464330520 wait complete. 2024-09-18T12:34:10.869 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:10.869 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h29 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.940+0000 7f696159d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.940+0000 7f696159d640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.940+0000 7f696159d640 10 monclient: monmap: 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:10.942039+0000 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:10.942039+0000 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.942 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:10.943 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:10.943 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:10.943 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:10.943 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:10.943 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:10.943 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:10.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.940+0000 7f696159d640 1 Processor -- start 2024-09-18T12:34:10.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.940+0000 7f696159d640 1 -- start start 2024-09-18T12:34:10.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.940+0000 7f696159d640 10 monclient: init 2024-09-18T12:34:10.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f695c09ec50 0x7f695c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient(hunting): picked mon.noname-a con 0x7f695c09ec50 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f695c1321a0 0x7f695c068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient(hunting): picked mon.noname-c con 0x7f695c1321a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f695c09f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c067420 con 0x7f695c09f020 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c0a0480 con 0x7f695c09ec50 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c0a0200 con 0x7f695c1321a0 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f696159d640 10 monclient(hunting): authenticate will time out at 1286.511236s 2024-09-18T12:34:10.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f695c1321a0 0x7f695c068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695a7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f695c1321a0 0x7f695c068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44338/0 (socket says 172.21.15.144:44338) 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695a7fc640 1 -- 172.21.15.144:0/2108702488 learned_addr learned my addr 172.21.15.144:0/2108702488 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695a7fc640 10 monclient(hunting): get_auth_request con 0x7f695c1321a0 auth_method 0 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 1 --2- 172.21.15.144:0/2108702488 >> v2:172.21.15.116:3300/0 conn(0x7f695c09ec50 0x7f695c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): get_auth_request con 0x7f695c09ec50 auth_method 0 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient(hunting): handle_auth_done global_id 104646 payload 274 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 1 -- 172.21.15.144:0/2108702488 >> v2:172.21.15.144:3300/0 conn(0x7f695c1321a0 msgr2=0x7f695c068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 1 --2- 172.21.15.144:0/2108702488 >> v2:172.21.15.144:3300/0 conn(0x7f695c1321a0 0x7f695c068d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 1 -- 172.21.15.144:0/2108702488 >> v1:172.21.15.144:6789/0 conn(0x7f695c09f020 legacy=0x7f695c069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 1 monclient: found mon.noname-a 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:10.948 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.941+0000 7f695affd640 1 -- 172.21.15.144:0/2108702488 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c09fd80 con 0x7f695c09ec50 2024-09-18T12:34:10.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f695affd640 1 -- 172.21.15.144:0/2108702488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09ec50 msgr2=0x7f695c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f695affd640 1 --2- 172.21.15.144:0/2108702488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09ec50 0x7f695c131c60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f695c1325a0 tx=0x7f694c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32daebe2ba3887f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 1 -- 172.21.15.144:0/2108702488 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c007760 con 0x7f695c09ec50 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: dump: 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:10.954 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 1 -- 172.21.15.144:0/2108702488 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c02d050 con 0x7f695c09ec50 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 1 -- 172.21.15.144:0/2108702488 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c007af0 con 0x7f695c09ec50 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f6959ffb640 10 monclient: dump: 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:10.955 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f696159d640 5 monclient: authenticate success, global_id 104646 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f696159d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.944+0000 7f696159d640 4 set_mon_vals no callback set 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 10 monclient: shutdown 2024-09-18T12:34:10.956 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 -- 172.21.15.144:0/2108702488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09ec50 msgr2=0x7f695c131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 --2- 172.21.15.144:0/2108702488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09ec50 0x7f695c131c60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f695c1325a0 tx=0x7f694c009780 comp rx=0 tx=0).stop 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 -- 172.21.15.144:0/2108702488 shutdown_connections 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 --2- 172.21.15.144:0/2108702488 >> v2:172.21.15.144:3300/0 conn(0x7f695c1321a0 0x7f695c068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 --2- 172.21.15.144:0/2108702488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09ec50 0x7f695c131c60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 -- 172.21.15.144:0/2108702488 >> 172.21.15.144:0/2108702488 conn(0x7f695c0a5cf0 msgr2=0x7f695c0a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 -- 172.21.15.144:0/2108702488 shutdown_connections 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.945+0000 7f696159d640 1 -- 172.21.15.144:0/2108702488 wait complete. 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient: build_initial_monmap 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient: monmap: 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:10.957 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 Processor -- start 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 -- start start 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient: init 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 0x7f695c11cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.958 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): picked mon.noname-b con 0x7f695c09ec50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09f020 0x7f695c11d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): picked mon.noname-a con 0x7f695c09f020 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f695c1321a0 0x7f695c11d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): picked mon.noname-c con 0x7f695c1321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c1207b0 con 0x7f695c09f020 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c11f570 con 0x7f695c09ec50 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c09c020 con 0x7f695c1321a0 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f696159d640 10 monclient(hunting): authenticate will time out at 1286.516883s 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f695affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 0x7f695c11cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f695affd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 0x7f695c11cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44354/0 (socket says 172.21.15.144:44354) 2024-09-18T12:34:10.959 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f695affd640 1 -- 172.21.15.144:0/3429932102 learned_addr learned my addr 172.21.15.144:0/3429932102 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f695affd640 10 monclient(hunting): get_auth_request con 0x7f695c09ec50 auth_method 0 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.946+0000 7f695affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09f020 0x7f695c11d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f695c1321a0 0x7f695c11d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 10 monclient(hunting): get_auth_request con 0x7f695c09f020 auth_method 0 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.960 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 10 monclient(hunting): get_auth_request con 0x7f695c1321a0 auth_method 0 2024-09-18T12:34:10.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:10.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:10.961 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:10.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.963 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient(hunting): handle_auth_done global_id 124135 payload 824 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 1 -- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f695c1321a0 msgr2=0x7f695c11d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f695c1321a0 0x7f695c11d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 1 -- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09f020 msgr2=0x7f695c11d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09f020 0x7f695c11d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient: _finish_hunting 0 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 1 monclient: found mon.noname-b 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.947+0000 7f695affd640 1 -- 172.21.15.144:0/3429932102 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c009a20 con 0x7f695c09ec50 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f695affd640 1 -- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 msgr2=0x7f695c11cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:10.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f695affd640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 0x7f695c11cf40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f694c0099f0 tx=0x7f694c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c031ac0 con 0x7f695c09ec50 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: dump: 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.965 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c040070 con 0x7f695c09ec50 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f696159d640 5 monclient: authenticate success, global_id 124135 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f696159d640 10 monclient: _renew_subs 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f696159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c09e050 con 0x7f695c09ec50 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c031e30 con 0x7f695c09ec50 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f6942ffd640 10 monclient: dump: 2024-09-18T12:34:10.966 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f696159d640 10 monclient: _renew_subs 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f696159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f69597fa640 4 set_mon_vals no callback set 2024-09-18T12:34:10.967 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.949+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c120580 con 0x7f695c09ec50 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f694c042030 con 0x7f695c09ec50 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f696159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f696159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f696159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c0008d0 con 0x7f695c09ec50 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f6942ffd640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6948038490 0x7f694803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(254..254 src has 13..254) v4 ==== 8971+0+0 (secure 0 0 0) 0x7f694c06dfc0 con 0x7f695c09ec50 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f695a7fc640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6948038490 0x7f694803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:10.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.950+0000 7f695a7fc640 10 monclient: get_auth_request con 0x7f6948038490 auth_method 0 2024-09-18T12:34:10.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.953+0000 7f695a7fc640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6948038490 0x7f694803a940 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f695c0a0990 tx=0x7f6950008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:10.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.955+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f694c03ab80 con 0x7f695c09ec50 2024-09-18T12:34:10.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.955+0000 7f6942ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:10.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:10.955+0000 7f6942ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:11.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:11.117+0000 7f696159d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T12:34:11.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:11.117+0000 7f696159d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T12:34:11.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:11.117+0000 7f696159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:11.119 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:11.117+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f695c0a13e0 con 0x7f695c09ec50 2024-09-18T12:34:11.760 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:34:11.760 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.859+0000 7f107e6c0640 10 monclient: get_monmap_and_config 2024-09-18T12:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.859+0000 7f107e6c0640 10 monclient: build_initial_monmap 2024-09-18T12:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.859+0000 7f107e6c0640 10 monclient: monmap: 2024-09-18T12:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:11.860333+0000 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:11.860333+0000 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.860+0000 7f107e6c0640 1 Processor -- start 2024-09-18T12:34:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.860+0000 7f107e6c0640 1 -- start start 2024-09-18T12:34:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.860+0000 7f107e6c0640 10 monclient: init 2024-09-18T12:34:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1078067110 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f107809b010 0x7f107809b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient(hunting): picked mon.noname-a con 0x7f107809b010 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f10781349a0 0x7f1078136d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient(hunting): picked mon.noname-c con 0x7f10781349a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078130540 con 0x7f1078067110 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1078067ed0 con 0x7f107809b010 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1078099800 con 0x7f10781349a0 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107e6c0640 10 monclient(hunting): authenticate will time out at 1279.182263s 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f107809b010 0x7f107809b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f1077fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1078067110 0x7f107812f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46488/0 (socket says 172.21.15.116:46488) 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f107809b010 0x7f107809b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45942/0 (socket says 172.21.15.116:45942) 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f1077fff640 1 -- 172.21.15.116:0/4014730435 learned_addr learned my addr 172.21.15.116:0/4014730435 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): get_auth_request con 0x7f107809b010 auth_method 0 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107cc5d640 1 --2- 172.21.15.116:0/4014730435 >> v2:172.21.15.144:3300/0 conn(0x7f10781349a0 0x7f1078136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107cc5d640 10 monclient(hunting): get_auth_request con 0x7f10781349a0 auth_method 0 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107cc5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107cc5d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107cc5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient(hunting): handle_auth_done global_id 104652 payload 274 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 1 -- 172.21.15.116:0/4014730435 >> v2:172.21.15.144:3300/0 conn(0x7f10781349a0 msgr2=0x7f1078136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 1 --2- 172.21.15.116:0/4014730435 >> v2:172.21.15.144:3300/0 conn(0x7f10781349a0 0x7f1078136d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 1 -- 172.21.15.116:0/4014730435 >> v1:172.21.15.116:6789/0 conn(0x7f1078067110 legacy=0x7f107812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 1 monclient: found mon.noname-a 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f106ffff640 1 -- 172.21.15.116:0/4014730435 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10780674e0 con 0x7f107809b010 2024-09-18T12:34:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.861+0000 7f107cc5d640 1 --2- 172.21.15.116:0/4014730435 >> v2:172.21.15.144:3300/0 conn(0x7f10781349a0 0x7f1078136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f106ffff640 1 -- 172.21.15.116:0/4014730435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f107809b010 msgr2=0x7f107809b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f106ffff640 1 --2- 172.21.15.116:0/4014730435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f107809b010 0x7f107809b430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1078067de0 tx=0x7f10600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40ca6c10501884 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 1 -- 172.21.15.116:0/4014730435 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106002ae90 con 0x7f107809b010 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: dump: 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: _finish_auth 0 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f107e6c0640 5 monclient: authenticate success, global_id 104652 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f107e6c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 1 -- 172.21.15.116:0/4014730435 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1060032440 con 0x7f107809b010 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 1 -- 172.21.15.116:0/4014730435 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600327e0 con 0x7f107809b010 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f107e6c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f107e6c0640 4 set_mon_vals no callback set 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.864+0000 7f10777fe640 10 monclient: dump: 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 10 monclient: shutdown 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 -- 172.21.15.116:0/4014730435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f107809b010 msgr2=0x7f107809b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 --2- 172.21.15.116:0/4014730435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f107809b010 0x7f107809b430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1078067de0 tx=0x7f10600090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 -- 172.21.15.116:0/4014730435 shutdown_connections 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 --2- 172.21.15.116:0/4014730435 >> v2:172.21.15.144:3300/0 conn(0x7f10781349a0 0x7f1078136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 --2- 172.21.15.116:0/4014730435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f107809b010 0x7f107809b430 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 -- 172.21.15.116:0/4014730435 >> 172.21.15.116:0/4014730435 conn(0x7f10780a1e30 msgr2=0x7f10780a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 -- 172.21.15.116:0/4014730435 shutdown_connections 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 1 -- 172.21.15.116:0/4014730435 wait complete. 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 10 monclient: build_initial_monmap 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.865+0000 7f107e6c0640 10 monclient: monmap: 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 Processor -- start 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 -- start start 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient: init 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1078067110 0x7f107806ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1078067110 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f107809b010 0x7f107812b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): picked mon.noname-c con 0x7f107809b010 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 0x7f1078133210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): picked mon.noname-a con 0x7f10781349a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107809e330 con 0x7f10781349a0 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10780a0860 con 0x7f1078067110 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1078137830 con 0x7f107809b010 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107e6c0640 10 monclient(hunting): authenticate will time out at 1279.187637s 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 0x7f1078133210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f107809b010 0x7f107812b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 0x7f1078133210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45948/0 (socket says 172.21.15.116:45948) 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 1 -- 172.21.15.116:0/1283260943 learned_addr learned my addr 172.21.15.116:0/1283260943 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): get_auth_request con 0x7f10781349a0 auth_method 0 2024-09-18T12:34:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 10 monclient(hunting): get_auth_request con 0x7f107809b010 auth_method 0 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f106ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f1077fff640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1078067110 0x7f107806ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:11.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f1077fff640 10 monclient(hunting): get_auth_request con 0x7f1078067110 auth_method 0 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f1077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f1077fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f1077fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.866+0000 7f107cc5d640 10 monclient(hunting): handle_auth_done global_id 104655 payload 824 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 1 -- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f107809b010 msgr2=0x7f107812b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f107809b010 0x7f107812b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 1 -- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1078067110 msgr2=0x7f107806ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1078067110 0x7f107806ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 1 monclient: found mon.noname-a 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f107cc5d640 1 -- 172.21.15.116:0/1283260943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060009390 con 0x7f10781349a0 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.867+0000 7f1077fff640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1078067110 0x7f107806ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107cc5d640 1 -- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 msgr2=0x7f1078133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107cc5d640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 0x7f1078133210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f107000d180 tx=0x7f1070009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 1 -- 172.21.15.116:0/1283260943 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070012560 con 0x7f10781349a0 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: dump: 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: _finish_auth 0 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107e6c0640 5 monclient: authenticate success, global_id 104655 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 1 -- 172.21.15.116:0/1283260943 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107000f040 con 0x7f10781349a0 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107e6c0640 10 monclient: _renew_subs 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107e6c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1078098520 con 0x7f10781349a0 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 1 -- 172.21.15.116:0/1283260943 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070012930 con 0x7f10781349a0 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1074ff9640 10 monclient: dump: 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f1076ffd640 4 set_mon_vals no callback set 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107e6c0640 10 monclient: _renew_subs 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107e6c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.869+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1078096bf0 con 0x7f10781349a0 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.870+0000 7f1074ff9640 1 -- 172.21.15.116:0/1283260943 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1070012ce0 con 0x7f10781349a0 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.870+0000 7f107e6c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.870+0000 7f107e6c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.870+0000 7f107e6c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.870+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107809d8c0 con 0x7f10781349a0 2024-09-18T12:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.870+0000 7f1074ff9640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1058038490 0x7f105803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.870+0000 7f1074ff9640 1 -- 172.21.15.116:0/1283260943 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(255..255 src has 13..255) v4 ==== 8971+0+0 (secure 0 0 0) 0x7f107004d1f0 con 0x7f10781349a0 2024-09-18T12:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.871+0000 7f1077fff640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1058038490 0x7f105803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.871+0000 7f1077fff640 10 monclient: get_auth_request con 0x7f1058038490 auth_method 0 2024-09-18T12:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.873+0000 7f1077fff640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1058038490 0x7f105803a940 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1068007fd0 tx=0x7f1068009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.875+0000 7f1074ff9640 1 -- 172.21.15.116:0/1283260943 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f107809d8c0 con 0x7f10781349a0 2024-09-18T12:34:11.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.875+0000 7f1074ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:11.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:11.875+0000 7f1074ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:12.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.087+0000 7f107e6c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:34:12.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.087+0000 7f107e6c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:12.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.087+0000 7f107e6c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:12.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.087+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f107809d8c0 con 0x7f10781349a0 2024-09-18T12:34:12.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.087+0000 7f1074ff9640 1 -- 172.21.15.116:0/1283260943 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f107809d8c0 con 0x7f10781349a0 2024-09-18T12:34:12.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.087+0000 7f1074ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:12.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.087+0000 7f1074ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:12.089 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:34:12.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1058038490 msgr2=0x7f105803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1058038490 0x7f105803a940 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1068007fd0 tx=0x7f1068009040 comp rx=0 tx=0).stop 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 10 monclient: shutdown 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 msgr2=0x7f1078133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 0x7f1078133210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f107000d180 tx=0x7f1070009a50 comp rx=0 tx=0).stop 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 shutdown_connections 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1058038490 0x7f105803a940 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10781349a0 0x7f1078133210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f107809b010 0x7f107812b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 --2- 172.21.15.116:0/1283260943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1078067110 0x7f107806ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 >> 172.21.15.116:0/1283260943 conn(0x7f10780a1e30 msgr2=0x7f107809d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 shutdown_connections 2024-09-18T12:34:12.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.089+0000 7f107e6c0640 1 -- 172.21.15.116:0/1283260943 wait complete. 2024-09-18T12:34:12.100 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:34:12.101 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:34:12.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:34:12.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:34:12.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.162+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v256) v1 ==== 121+0+0 (secure 0 0 0) 0x7f694c03a3a0 con 0x7f695c09ec50 2024-09-18T12:34:12.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.162+0000 7f6942ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T12:34:12.164 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.162+0000 7f6942ffd640 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-18T12:34:12.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.195+0000 7f871c19d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:12.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.195+0000 7f871c19d640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.195+0000 7f871c19d640 10 monclient: monmap: 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:12.196493+0000 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:12.196493+0000 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:12.197 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.196+0000 7f871c19d640 1 Processor -- start 2024-09-18T12:34:12.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.196+0000 7f871c19d640 1 -- start start 2024-09-18T12:34:12.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.196+0000 7f871c19d640 10 monclient: init 2024-09-18T12:34:12.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.196+0000 7f871c19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.196+0000 7f871c19d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 10 monclient(hunting): picked mon.noname-b con 0x7f8714096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8714097030 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8714133b40 0x7f8714135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 10 monclient(hunting): picked mon.noname-c con 0x7f8714133b40 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87140696a0 con 0x7f8714097030 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871409ac00 con 0x7f8714133b40 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871409d6b0 con 0x7f8714096c60 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871c19d640 10 monclient(hunting): authenticate will time out at 1279.518070s 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43454/0 (socket says 172.21.15.116:43454) 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 -- 172.21.15.116:0/536011474 learned_addr learned my addr 172.21.15.116:0/536011474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): get_auth_request con 0x7f8714096c60 auth_method 0 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719738640 1 --1- 172.21.15.116:0/536011474 >> v1:172.21.15.116:6789/0 conn(0x7f8714097030 0x7f8714097400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46500/0 (socket says 172.21.15.116:46500) 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 1 --2- 172.21.15.116:0/536011474 >> v2:172.21.15.144:3300/0 conn(0x7f8714133b40 0x7f8714135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 10 monclient(hunting): get_auth_request con 0x7f8714133b40 auth_method 0 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f871a73a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient(hunting): handle_auth_done global_id 65423 payload 274 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 -- 172.21.15.116:0/536011474 >> v2:172.21.15.144:3300/0 conn(0x7f8714133b40 msgr2=0x7f8714135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 --2- 172.21.15.116:0/536011474 >> v2:172.21.15.144:3300/0 conn(0x7f8714133b40 0x7f8714135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 -- 172.21.15.116:0/536011474 >> v1:172.21.15.116:6789/0 conn(0x7f8714097030 legacy=0x7f8714097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 monclient: found mon.noname-b 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.197+0000 7f8719f39640 1 -- 172.21.15.116:0/536011474 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871409d2e0 con 0x7f8714096c60 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8719f39640 1 -- 172.21.15.116:0/536011474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 msgr2=0x7f871409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8719f39640 1 --2- 172.21.15.116:0/536011474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871409a6c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f870c00bbc0 tx=0x7f870c00b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f883d16b4a5842e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 1 -- 172.21.15.116:0/536011474 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c032700 con 0x7f8714096c60 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:12.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: dump: 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.214 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.214 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 5 monclient: authenticate success, global_id 65423 2024-09-18T12:34:12.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 1 -- 172.21.15.116:0/536011474 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c032860 con 0x7f8714096c60 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 4 set_mon_vals no callback set 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 1 -- 172.21.15.116:0/536011474 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c032c00 con 0x7f8714096c60 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f8718f37640 10 monclient: dump: 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 10 monclient: shutdown 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 1 -- 172.21.15.116:0/536011474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 msgr2=0x7f871409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 1 --2- 172.21.15.116:0/536011474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871409a6c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f870c00bbc0 tx=0x7f870c00b460 comp rx=0 tx=0).stop 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 1 -- 172.21.15.116:0/536011474 shutdown_connections 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 1 --2- 172.21.15.116:0/536011474 >> v2:172.21.15.144:3300/0 conn(0x7f8714133b40 0x7f8714135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 1 --2- 172.21.15.116:0/536011474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871409a6c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.200+0000 7f871c19d640 1 -- 172.21.15.116:0/536011474 >> 172.21.15.116:0/536011474 conn(0x7f87140672b0 msgr2=0x7f87140676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.201+0000 7f871c19d640 1 -- 172.21.15.116:0/536011474 shutdown_connections 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.201+0000 7f871c19d640 1 -- 172.21.15.116:0/536011474 wait complete. 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.201+0000 7f871c19d640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.201+0000 7f871c19d640 10 monclient: monmap: 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:12.217 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.201+0000 7f871c19d640 1 Processor -- start 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.201+0000 7f871c19d640 1 -- start start 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.201+0000 7f871c19d640 10 monclient: init 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): picked mon.noname-c con 0x7f8714096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8714097030 0x7f8714117920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): picked mon.noname-a con 0x7f8714097030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8714133b40 0x7f871412e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): picked mon.noname-b con 0x7f8714133b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87141199c0 con 0x7f8714097030 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8714119f80 con 0x7f8714133b40 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8714117e60 con 0x7f8714096c60 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871c19d640 10 monclient(hunting): authenticate will time out at 1279.523249s 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871406e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43456/0 (socket says 172.21.15.116:43456) 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 -- 172.21.15.116:0/2625873045 learned_addr learned my addr 172.21.15.116:0/2625873045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:12.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): get_auth_request con 0x7f8714096c60 auth_method 0 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8714097030 0x7f8714117920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 10 monclient(hunting): get_auth_request con 0x7f8714097030 auth_method 0 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8714133b40 0x7f871412e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 10 monclient(hunting): get_auth_request con 0x7f8714133b40 auth_method 0 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f871a73a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient(hunting): handle_auth_done global_id 65426 payload 824 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8714133b40 msgr2=0x7f871412e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8714133b40 0x7f871412e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8714097030 msgr2=0x7f8714117920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8714097030 0x7f8714117920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 monclient: found mon.noname-c 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.202+0000 7f8719f39640 1 -- 172.21.15.116:0/2625873045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c00b700 con 0x7f8714096c60 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.204+0000 7f8719f39640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 msgr2=0x7f871406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.204+0000 7f8719f39640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871406e840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b4a0 tx=0x7f870c03a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.204+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c035b30 con 0x7f8714096c60 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.204+0000 7f8711ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.204+0000 7f8711ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.204+0000 7f8711ffb640 10 monclient: dump: 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.222 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c035cb0 con 0x7f8714096c60 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f871c19d640 5 monclient: authenticate success, global_id 65426 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f871c19d640 10 monclient: _renew_subs 2024-09-18T12:34:12.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f871c19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8714119ca0 con 0x7f8714096c60 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c03f660 con 0x7f8714096c60 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 10 monclient: dump: 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.224 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f871c19d640 10 monclient: _renew_subs 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f871c19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8714118540 con 0x7f8714096c60 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8713fff640 4 set_mon_vals no callback set 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.205+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f870c03c050 con 0x7f8714096c60 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f871c19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f871c19d640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 0x7f871413d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f871c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8714065f20 con 0x7f871412ee00 2024-09-18T12:34:12.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8711ffb640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8700038490 0x7f870003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(256..256 src has 13..256) v4 ==== 8758+0+0 (secure 0 0 0) 0x7f870c06f690 con 0x7f8714096c60 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 0x7f871413d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient: get_auth_request con 0x7f871412ee00 auth_method 0 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f871a73a640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8700038490 0x7f870003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f871a73a640 10 monclient: get_auth_request con 0x7f8700038490 auth_method 0 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 10 monclient(hunting): handle_auth_done global_id 104661 payload 824 2024-09-18T12:34:12.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.206+0000 7f8719738640 1 -- 172.21.15.116:0/2625873045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8714118a60 con 0x7f871412ee00 2024-09-18T12:34:12.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.208+0000 7f8719738640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 msgr2=0x7f871413d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.208+0000 7f8719738640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 0x7f871413d7d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f870800e740 tx=0x7f8708007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.208+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708005050 con 0x7f871412ee00 2024-09-18T12:34:12.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.208+0000 7f8711ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.208+0000 7f8711ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.208+0000 7f8711ffb640 10 monclient: dump: 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.208+0000 7f871a73a640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8700038490 0x7f870003a940 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8704009fd0 tx=0x7f870400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.209+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f87080051b0 con 0x7f871412ee00 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.209+0000 7f8711ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.209+0000 7f8711ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.209+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 msgr2=0x7f871413d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.209+0000 7f8711ffb640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 0x7f871413d7d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f870800e740 tx=0x7f8708007e40 comp rx=0 tx=0).stop 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.209+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 msgr2=0x7f871413d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.209+0000 7f8711ffb640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f871412ee00 0x7f871413d7d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f871c19d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f871c19d640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 0x7f8714029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f871c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871404bf60 con 0x7f87140298c0 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 0x7f8714029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 10 monclient: get_auth_request con 0x7f87140298c0 auth_method 0 2024-09-18T12:34:12.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.214+0000 7f8719738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.215+0000 7f8719738640 10 monclient(hunting): handle_auth_done global_id 104664 payload 824 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.215+0000 7f8719738640 1 -- 172.21.15.116:0/2625873045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f87140983b0 con 0x7f87140298c0 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8719738640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 msgr2=0x7f8714029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8719738640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 0x7f8714029cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f870800e740 tx=0x7f8708019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870801d6f0 con 0x7f87140298c0 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:12.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 10 monclient: dump: 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f8708025070 con 0x7f87140298c0 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:12.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 msgr2=0x7f8714029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 0x7f8714029cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f870800e740 tx=0x7f8708019000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 msgr2=0x7f8714029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f8711ffb640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87140298c0 0x7f8714029cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 94, 2024-09-18T12:34:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:12.239 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.240 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.241 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.243 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.244 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:34:12.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:34:12.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:12.247 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:12.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8700038490 msgr2=0x7f870003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8700038490 0x7f870003a940 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8704009fd0 tx=0x7f870400b040 comp rx=0 tx=0).stop 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 10 monclient: shutdown 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 msgr2=0x7f871406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 1 --2- 172.21.15.116:0/2625873045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8714096c60 0x7f871406e840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b4a0 tx=0x7f870c03a000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871a73a640 1 -- 172.21.15.116:0/2625873045 reap_dead start 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871a73a640 1 -- 172.21.15.116:0/2625873045 reap_dead start 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 shutdown_connections 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 >> 172.21.15.116:0/2625873045 conn(0x7f87140672b0 msgr2=0x7f871409f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.217+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 shutdown_connections 2024-09-18T12:34:12.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.218+0000 7f871c19d640 1 -- 172.21.15.116:0/2625873045 wait complete. 2024-09-18T12:34:12.250 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:34:12.250 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:34:12.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:34:12.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.261+0000 7f696159d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T12:34:12.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.261+0000 7f696159d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T12:34:12.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.261+0000 7f696159d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:12.263 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.261+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f695c0a13e0 con 0x7f695c09ec50 2024-09-18T12:34:12.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.262+0000 7f6942ffd640 1 -- 172.21.15.144:0/3429932102 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v256) v1 ==== 120+0+0 (secure 0 0 0) 0x7f694c03a580 con 0x7f695c09ec50 2024-09-18T12:34:12.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.262+0000 7f6942ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T12:34:12.264 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.262+0000 7f6942ffd640 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-18T12:34:12.265 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h29' does not appear in the crush map 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6948038490 msgr2=0x7f694803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6948038490 0x7f694803a940 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f695c0a0990 tx=0x7f6950008040 comp rx=0 tx=0).stop 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 10 monclient: shutdown 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 msgr2=0x7f695c11cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 0x7f695c11cf40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f694c0099f0 tx=0x7f694c037000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 shutdown_connections 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6948038490 0x7f694803a940 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f695c1321a0 0x7f695c11d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.267 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c09f020 0x7f695c11d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 --2- 172.21.15.144:0/3429932102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695c09ec50 0x7f695c11cf40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.264+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 >> 172.21.15.144:0/3429932102 conn(0x7f695c0a5cf0 msgr2=0x7f695c131110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.265+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 shutdown_connections 2024-09-18T12:34:12.268 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.265+0000 7f696159d640 1 -- 172.21.15.144:0/3429932102 wait complete. 2024-09-18T12:34:12.275 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:12.275 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h30 2024-09-18T12:34:12.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.342+0000 7f2e83e80640 10 monclient: get_monmap_and_config 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.342+0000 7f2e83e80640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.342+0000 7f2e83e80640 10 monclient: monmap: 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:12.343625+0000 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:12.343625+0000 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:12.344 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:12.345 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.343+0000 7f2e83e80640 1 Processor -- start 2024-09-18T12:34:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.343+0000 7f2e83e80640 1 -- start start 2024-09-18T12:34:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.343+0000 7f2e83e80640 10 monclient: init 2024-09-18T12:34:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:34:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c12f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c067110 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e7c09b010 0x7f2e7c09b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c09b010 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e7c09c820 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c130540 con 0x7f2e7c09c820 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c09b940 con 0x7f2e7c09b010 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c067ed0 con 0x7f2e7c067110 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e83e80640 10 monclient(hunting): authenticate will time out at 1279.665289s 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c12f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e7c09b010 0x7f2e7c09b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c12f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43470/0 (socket says 172.21.15.116:43470) 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 -- 172.21.15.116:0/1799890520 learned_addr learned my addr 172.21.15.116:0/1799890520 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): get_auth_request con 0x7f2e7c067110 auth_method 0 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 10 monclient(hunting): get_auth_request con 0x7f2e7c09b010 auth_method 0 2024-09-18T12:34:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_done global_id 65429 payload 274 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 -- 172.21.15.116:0/1799890520 >> v2:172.21.15.116:3300/0 conn(0x7f2e7c09b010 msgr2=0x7f2e7c09b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 --2- 172.21.15.116:0/1799890520 >> v2:172.21.15.116:3300/0 conn(0x7f2e7c09b010 0x7f2e7c09b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 -- 172.21.15.116:0/1799890520 >> v1:172.21.15.144:6789/0 conn(0x7f2e7c09c820 legacy=0x7f2e7c09cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 monclient: found mon.noname-b 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e81c1c640 1 -- 172.21.15.116:0/1799890520 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c099800 con 0x7f2e7c067110 2024-09-18T12:34:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.344+0000 7f2e8141b640 1 --2- 172.21.15.116:0/1799890520 >> v2:172.21.15.116:3300/0 conn(0x7f2e7c09b010 0x7f2e7c09b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.346+0000 7fbf4719d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.346+0000 7fbf4719d640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.346+0000 7fbf4719d640 10 monclient: monmap: 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:12.348825+0000 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:12.348825+0000 2024-09-18T12:34:12.372 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.347+0000 7fbf4719d640 1 Processor -- start 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.347+0000 7fbf4719d640 1 -- start start 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.347+0000 7fbf4719d640 10 monclient: init 2024-09-18T12:34:12.373 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbf4009ec50 0x7fbf40131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient(hunting): picked mon.noname-c con 0x7fbf4009ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf401321a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf4009f020 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf40067420 con 0x7fbf4009f020 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf400a0480 con 0x7fbf401321a0 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.374 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf400a0200 con 0x7fbf4009ec50 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4719d640 10 monclient(hunting): authenticate will time out at 1287.918165s 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf44f39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbf4009ec50 0x7fbf40131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf44f39640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbf4009ec50 0x7fbf40131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44358/0 (socket says 172.21.15.144:44358) 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf44f39640 1 -- 172.21.15.144:0/1645812790 learned_addr learned my addr 172.21.15.144:0/1645812790 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf44f39640 10 monclient(hunting): get_auth_request con 0x7fbf4009ec50 auth_method 0 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf44f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf44f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf44f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.348+0000 7fbf4573a640 1 --1- 172.21.15.144:0/1645812790 >> v1:172.21.15.116:6789/0 conn(0x7fbf4009f020 0x7fbf40069410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:35772/0 (socket says 172.21.15.144:35772) 2024-09-18T12:34:12.375 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf3ffff640 1 -- 172.21.15.144:0/1645812790 >> v1:172.21.15.144:6789/0 conn(0x7fbf401321a0 legacy=0x7fbf40068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf4573a640 1 -- 172.21.15.144:0/1645812790 >> v1:172.21.15.116:6789/0 conn(0x7fbf4009f020 legacy=0x7fbf40069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 10 monclient(hunting): handle_auth_done global_id 124150 payload 274 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 1 -- 172.21.15.144:0/1645812790 >> v1:172.21.15.144:6789/0 conn(0x7fbf401321a0 legacy=0x7fbf40068d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 1 -- 172.21.15.144:0/1645812790 >> v1:172.21.15.116:6789/0 conn(0x7fbf4009f020 legacy=0x7fbf40069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.376 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 1 monclient: found mon.noname-c 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf3f7fe640 1 -- 172.21.15.144:0/1645812790 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012029492 0 0) 0x7fbf40067420 con 0x7fbf4009f020 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf44f39640 1 -- 172.21.15.144:0/1645812790 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4009fd80 con 0x7fbf4009ec50 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.350+0000 7fbf3f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf44f39640 1 -- 172.21.15.144:0/1645812790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf4009ec50 msgr2=0x7fbf40131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf44f39640 1 --2- 172.21.15.144:0/1645812790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf4009ec50 0x7fbf40131c60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009a10 tx=0x7fbf380091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33a4c7263f8afce7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 1 -- 172.21.15.144:0/1645812790 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3802f4b0 con 0x7fbf4009ec50 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.377 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: dump: 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.378 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 1 -- 172.21.15.144:0/1645812790 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3802f610 con 0x7fbf4009ec50 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf4719d640 5 monclient: authenticate success, global_id 124150 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf4719d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf4719d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 1 -- 172.21.15.144:0/1645812790 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3802f9b0 con 0x7fbf4009ec50 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf4719d640 4 set_mon_vals no callback set 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:12.379 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.353+0000 7fbf3f7fe640 10 monclient: dump: 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:12.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 10 monclient: shutdown 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 -- 172.21.15.144:0/1645812790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf4009ec50 msgr2=0x7fbf40131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1645812790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf4009ec50 0x7fbf40131c60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009a10 tx=0x7fbf380091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 -- 172.21.15.144:0/1645812790 shutdown_connections 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1645812790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf4009ec50 0x7fbf40131c60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 -- 172.21.15.144:0/1645812790 >> 172.21.15.144:0/1645812790 conn(0x7fbf400a5cf0 msgr2=0x7fbf400a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 -- 172.21.15.144:0/1645812790 shutdown_connections 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 -- 172.21.15.144:0/1645812790 wait complete. 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 10 monclient: monmap: 2024-09-18T12:34:12.381 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:12.382 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.354+0000 7fbf4719d640 1 Processor -- start 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 1 -- start start 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient: init 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4009ec50 0x7fbf4009a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): picked mon.noname-a con 0x7fbf4009ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf4009f020 0x7fbf40130a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): picked mon.noname-c con 0x7fbf4009f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 0x7fbf4011b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): picked mon.noname-b con 0x7fbf401321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4009b510 con 0x7fbf4009ec50 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4009c020 con 0x7fbf401321a0 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40134ad0 con 0x7fbf4009f020 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4719d640 10 monclient(hunting): authenticate will time out at 1287.925523s 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 0x7fbf4011b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 0x7fbf4011b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:44374/0 (socket says 172.21.15.144:44374) 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 -- 172.21.15.144:0/1894471539 learned_addr learned my addr 172.21.15.144:0/1894471539 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): get_auth_request con 0x7fbf401321a0 auth_method 0 2024-09-18T12:34:12.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4009ec50 0x7fbf4009a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf4009f020 0x7fbf40130a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 10 monclient(hunting): get_auth_request con 0x7fbf4009ec50 auth_method 0 2024-09-18T12:34:12.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 10 monclient(hunting): get_auth_request con 0x7fbf4009f020 auth_method 0 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf44f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient(hunting): handle_auth_done global_id 124153 payload 824 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 -- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf4009f020 msgr2=0x7fbf40130a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf4009f020 0x7fbf40130a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 -- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4009ec50 msgr2=0x7fbf4009a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4009ec50 0x7fbf4009a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 monclient: found mon.noname-b 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.355+0000 7fbf4573a640 1 -- 172.21.15.144:0/1894471539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf38009450 con 0x7fbf401321a0 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4573a640 1 -- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 msgr2=0x7fbf4011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4573a640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 0x7fbf4011b760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000c910 tx=0x7fbf30007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000cea0 con 0x7fbf401321a0 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4573a640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 0x7fbf4011b760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000c910 tx=0x7fbf30007ec0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fbf3000aa20 mon_map magic: 0 v1 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: dump: 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:12.388 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.389 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.389 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.389 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.389 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:12.389 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.389 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.391 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4719d640 5 monclient: authenticate success, global_id 124153 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3000a640 con 0x7fbf401321a0 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4719d640 10 monclient: _renew_subs 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4011db30 con 0x7fbf401321a0 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4719d640 10 monclient: _renew_subs 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3effd640 4 set_mon_vals no callback set 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf40136570 con 0x7fbf401321a0 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000aa20 con 0x7fbf401321a0 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 10 monclient: dump: 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.393 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.358+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbf300070f0 con 0x7fbf401321a0 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.359+0000 7fbf4719d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.359+0000 7fbf3cff9640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf20038490 0x7fbf2003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.359+0000 7fbf4719d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.359+0000 7fbf4719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:12.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.359+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf400008d0 con 0x7fbf401321a0 2024-09-18T12:34:12.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.359+0000 7fbf44f39640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf20038490 0x7fbf2003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.359+0000 7fbf44f39640 10 monclient: get_auth_request con 0x7fbf20038490 auth_method 0 2024-09-18T12:34:12.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.361+0000 7fbf44f39640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf20038490 0x7fbf2003a940 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbf4009bf50 tx=0x7fbf38032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:12.363+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(256..256 src has 13..256) v4 ==== 8758+0+0 (secure 0 0 0) 0x7fbf3004c520 con 0x7fbf401321a0 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e81c1c640 1 -- 172.21.15.116:0/1799890520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 msgr2=0x7f2e7c12f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e81c1c640 1 --2- 172.21.15.116:0/1799890520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c12f670 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009910 tx=0x7f2e780091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285d3a1edfb156aa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 1 -- 172.21.15.116:0/1799890520 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7802aea0 con 0x7f2e7c067110 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: dump: 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.396 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 1 -- 172.21.15.116:0/1799890520 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78032440 con 0x7f2e7c067110 2024-09-18T12:34:12.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e83e80640 5 monclient: authenticate success, global_id 65429 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e83e80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e83e80640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 1 -- 172.21.15.116:0/1799890520 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e780327e0 con 0x7f2e7c067110 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e83e80640 4 set_mon_vals no callback set 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.347+0000 7f2e80c1a640 10 monclient: dump: 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.398 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 10 monclient: shutdown 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 -- 172.21.15.116:0/1799890520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 msgr2=0x7f2e7c12f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 --2- 172.21.15.116:0/1799890520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c12f670 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009910 tx=0x7f2e780091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 -- 172.21.15.116:0/1799890520 shutdown_connections 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 --2- 172.21.15.116:0/1799890520 >> v2:172.21.15.116:3300/0 conn(0x7f2e7c09b010 0x7f2e7c09b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 --2- 172.21.15.116:0/1799890520 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c12f670 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 -- 172.21.15.116:0/1799890520 >> 172.21.15.116:0/1799890520 conn(0x7f2e7c0a1e30 msgr2=0x7f2e7c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 -- 172.21.15.116:0/1799890520 shutdown_connections 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 -- 172.21.15.116:0/1799890520 wait complete. 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 10 monclient: monmap: 2024-09-18T12:34:12.400 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:12.401 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.348+0000 7f2e83e80640 1 Processor -- start 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 1 -- start start 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient: init 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c13ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): picked mon.noname-c con 0x7f2e7c067110 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7c09b010 0x7f2e7c12ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c09b010 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e7c09c820 0x7f2e7c11b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c09c820 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c11dae0 con 0x7f2e7c09c820 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c09e330 con 0x7f2e7c09b010 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c06eb40 con 0x7f2e7c067110 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e83e80640 10 monclient(hunting): authenticate will time out at 1279.670550s 2024-09-18T12:34:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c13ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c13ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43482/0 (socket says 172.21.15.116:43482) 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 -- 172.21.15.116:0/3840021367 learned_addr learned my addr 172.21.15.116:0/3840021367 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): get_auth_request con 0x7f2e7c067110 auth_method 0 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e7c09c820 0x7f2e7c11b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 10 monclient(hunting): get_auth_request con 0x7f2e7c09c820 auth_method 0 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e8241d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient(hunting): handle_auth_done global_id 65432 payload 824 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7c09b010 msgr2=0x7f2e7c12ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e7c09b010 0x7f2e7c12ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e7c09c820 msgr2=0x7f2e7c11b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e7c09c820 0x7f2e7c11b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 monclient: found mon.noname-c 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.349+0000 7f2e81c1c640 1 -- 172.21.15.116:0/3840021367 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c009390 con 0x7f2e7c067110 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e81c1c640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 msgr2=0x7f2e7c13ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e81c1c640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c13ec80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2e78035040 tx=0x7f2e78032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7802aea0 con 0x7f2e7c067110 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: dump: 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78000d10 con 0x7f2e7c067110 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e83e80640 5 monclient: authenticate success, global_id 65432 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e83e80640 10 monclient: _renew_subs 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e83e80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e78009450 con 0x7f2e7c067110 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7803d420 con 0x7f2e7c067110 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e71ffb640 10 monclient: dump: 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e73fff640 4 set_mon_vals no callback set 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e83e80640 10 monclient: _renew_subs 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e83e80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.352+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c13fac0 con 0x7f2e7c067110 2024-09-18T12:34:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e83e80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e83e80640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 0x7f2e7c11bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e83e80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2e78039040 con 0x7f2e7c067110 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c065f90 con 0x7f2e7c12f200 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 0x7f2e7c11bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient: get_auth_request con 0x7f2e7c12f200 auth_method 0 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e71ffb640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e60038490 0x7f2e6003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(256..256 src has 13..256) v4 ==== 8758+0+0 (secure 0 0 0) 0x7f2e7806c6a0 con 0x7f2e7c067110 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 10 monclient(hunting): handle_auth_done global_id 65435 payload 824 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8141b640 1 -- 172.21.15.116:0/3840021367 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e7c11ef50 con 0x7f2e7c12f200 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8241d640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e60038490 0x7f2e6003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.353+0000 7f2e8241d640 10 monclient: get_auth_request con 0x7f2e60038490 auth_method 0 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.355+0000 7f2e8141b640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 msgr2=0x7f2e7c11bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.355+0000 7f2e8141b640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 0x7f2e7c11bdc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00e8d0 tx=0x7f2e6c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.355+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c00ccd0 con 0x7f2e7c12f200 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.355+0000 7f2e71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.355+0000 7f2e71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.355+0000 7f2e71ffb640 10 monclient: dump: 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e8241d640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e60038490 0x7f2e6003a940 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c11f830 tx=0x7f2e740025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2e6c00ce30 con 0x7f2e7c12f200 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e71ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 msgr2=0x7f2e7c11bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e71ffb640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 0x7f2e7c11bdc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00e8d0 tx=0x7f2e6c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 msgr2=0x7f2e7c11bdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.356+0000 7f2e71ffb640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c12f200 0x7f2e7c11bdc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.361+0000 7f2e83e80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e83e80640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 0x7f2e7c118490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e83e80640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c11c700 con 0x7f2e7c11c300 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 0x7f2e7c118490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient: get_auth_request con 0x7f2e7c11c300 auth_method 0 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 10 monclient(hunting): handle_auth_done global_id 65441 payload 824 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.362+0000 7f2e8141b640 1 -- 172.21.15.116:0/3840021367 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e7c11f2f0 con 0x7f2e7c11c300 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e8141b640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 msgr2=0x7f2e7c118490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e8141b640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 0x7f2e7c118490 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c018000 tx=0x7f2e6c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c005970 con 0x7f2e7c11c300 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 10 monclient: dump: 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f2e6c00cac0 con 0x7f2e7c11c300 2024-09-18T12:34:12.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:12.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:12.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 msgr2=0x7f2e7c118490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 0x7f2e7c118490 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c018000 tx=0x7f2e6c019000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 msgr2=0x7f2e7c118490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:12.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.364+0000 7f2e71ffb640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c11c300 0x7f2e7c118490 secure :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c018000 tx=0x7f2e6c019000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.418 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 94, 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:34:12.419 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.420 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.421 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:12.422 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.423 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:12.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.425 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.426 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.428 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.430 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:12.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.432 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.433 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:34:12.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.433 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.433 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:12.433 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e60038490 msgr2=0x7f2e6003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e60038490 0x7f2e6003a940 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c11f830 tx=0x7f2e740025b0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 10 monclient: shutdown 2024-09-18T12:34:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 msgr2=0x7f2e7c13ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c13ec80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2e78035040 tx=0x7f2e78032000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e8241d640 1 -- 172.21.15.116:0/3840021367 reap_dead start 2024-09-18T12:34:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 shutdown_connections 2024-09-18T12:34:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 --2- 172.21.15.116:0/3840021367 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e7c067110 0x7f2e7c13ec80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 >> 172.21.15.116:0/3840021367 conn(0x7f2e7c0a1e30 msgr2=0x7f2e7c06df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 shutdown_connections 2024-09-18T12:34:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.365+0000 7f2e83e80640 1 -- 172.21.15.116:0/3840021367 wait complete. 2024-09-18T12:34:12.436 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:34:12.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:34:12.437 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:34:12.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.529+0000 7f2f362d4640 10 monclient: get_monmap_and_config 2024-09-18T12:34:12.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.529+0000 7f2f362d4640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.529+0000 7f2f362d4640 10 monclient: monmap: 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:12.530430+0000 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:12.530430+0000 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:12.532 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:12.532 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 Processor -- start 2024-09-18T12:34:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 -- start start 2024-09-18T12:34:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient: init 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f3009a510 0x7f2f30096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3009a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2f300971a0 0x7f2f300975a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f300971a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 0x7f2f3012fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3012f670 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f30097ae0 con 0x7f2f3009a510 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3009cb00 con 0x7f2f300971a0 2024-09-18T12:34:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3009a8e0 con 0x7f2f3012f670 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.530+0000 7f2f362d4640 10 monclient(hunting): authenticate will time out at 1279.851979s 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f3009a510 0x7f2f30096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 0x7f2f3012fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f3009a510 0x7f2f30096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46000/0 (socket says 172.21.15.116:46000) 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 0x7f2f3012fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43520/0 (socket says 172.21.15.116:43520) 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 1 -- 172.21.15.116:0/1259897375 learned_addr learned my addr 172.21.15.116:0/1259897375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f3009a510 auth_method 0 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): get_auth_request con 0x7f2f3012f670 auth_method 0 2024-09-18T12:34:12.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2f7fe640 1 --2- 172.21.15.116:0/1259897375 >> v2:172.21.15.144:3300/0 conn(0x7f2f300971a0 0x7f2f300975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2f7fe640 10 monclient(hunting): get_auth_request con 0x7f2f300971a0 auth_method 0 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient(hunting): handle_auth_done global_id 65444 payload 274 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 -- 172.21.15.116:0/1259897375 >> v2:172.21.15.144:3300/0 conn(0x7f2f300971a0 msgr2=0x7f2f300975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 --2- 172.21.15.116:0/1259897375 >> v2:172.21.15.144:3300/0 conn(0x7f2f300971a0 0x7f2f300975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 -- 172.21.15.116:0/1259897375 >> v2:172.21.15.116:3300/0 conn(0x7f2f3009a510 msgr2=0x7f2f30096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 --2- 172.21.15.116:0/1259897375 >> v2:172.21.15.116:3300/0 conn(0x7f2f3009a510 0x7f2f30096c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 monclient: found mon.noname-b 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f34871640 1 -- 172.21.15.116:0/1259897375 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3006ed80 con 0x7f2f3012f670 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.531+0000 7f2f2ffff640 1 --2- 172.21.15.116:0/1259897375 >> v2:172.21.15.116:3300/0 conn(0x7f2f3009a510 0x7f2f30096c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f34871640 1 -- 172.21.15.116:0/1259897375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 msgr2=0x7f2f3012fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f34871640 1 --2- 172.21.15.116:0/1259897375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 0x7f2f3012fa90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009e60 tx=0x7f2f28009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba966184e388835 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 1 -- 172.21.15.116:0/1259897375 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28007760 con 0x7f2f3012f670 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: dump: 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 5 monclient: authenticate success, global_id 65444 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 1 -- 172.21.15.116:0/1259897375 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2802d050 con 0x7f2f3012f670 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 1 -- 172.21.15.116:0/1259897375 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28007af0 con 0x7f2f3012f670 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:12.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 4 set_mon_vals no callback set 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f2effd640 10 monclient: dump: 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.543 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 10 monclient: shutdown 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 -- 172.21.15.116:0/1259897375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 msgr2=0x7f2f3012fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 --2- 172.21.15.116:0/1259897375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 0x7f2f3012fa90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009e60 tx=0x7f2f28009780 comp rx=0 tx=0).stop 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 -- 172.21.15.116:0/1259897375 shutdown_connections 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 --2- 172.21.15.116:0/1259897375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3012f670 0x7f2f3012fa90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 --2- 172.21.15.116:0/1259897375 >> v2:172.21.15.144:3300/0 conn(0x7f2f300971a0 0x7f2f300975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 --2- 172.21.15.116:0/1259897375 >> v2:172.21.15.116:3300/0 conn(0x7f2f3009a510 0x7f2f30096c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 -- 172.21.15.116:0/1259897375 >> 172.21.15.116:0/1259897375 conn(0x7f2f300672b0 msgr2=0x7f2f300676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 -- 172.21.15.116:0/1259897375 shutdown_connections 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.534+0000 7f2f362d4640 1 -- 172.21.15.116:0/1259897375 wait complete. 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.535+0000 7f2f362d4640 10 monclient: build_initial_monmap 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.535+0000 7f2f362d4640 10 monclient: monmap: 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:12.546 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.535+0000 7f2f362d4640 1 Processor -- start 2024-09-18T12:34:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.535+0000 7f2f362d4640 1 -- start start 2024-09-18T12:34:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.535+0000 7f2f362d4640 10 monclient: init 2024-09-18T12:34:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300971a0 0x7f2f3013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f300971a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3009a510 0x7f2f3013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3009a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f3012f670 0x7f2f3011bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3012f670 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3006f040 con 0x7f2f3012f670 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3009e3e0 con 0x7f2f300971a0 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3011f8d0 con 0x7f2f3009a510 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f362d4640 10 monclient(hunting): authenticate will time out at 1279.857388s 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3009a510 0x7f2f3013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f3012f670 0x7f2f3011bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3009a510 0x7f2f3013f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:43524/0 (socket says 172.21.15.116:43524) 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 -- 172.21.15.116:0/3131763363 learned_addr learned my addr 172.21.15.116:0/3131763363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): get_auth_request con 0x7f2f3009a510 auth_method 0 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300971a0 0x7f2f3013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 10 monclient(hunting): get_auth_request con 0x7f2f3012f670 auth_method 0 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request con 0x7f2f300971a0 auth_method 0 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient(hunting): handle_auth_done global_id 65447 payload 824 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300971a0 msgr2=0x7f2f3013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300971a0 0x7f2f3013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f3012f670 msgr2=0x7f2f3011bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f3012f670 0x7f2f3011bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:34:12.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f2f7fe640 1 -- 172.21.15.116:0/3131763363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20009450 con 0x7f2f3009a510 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.536+0000 7f2f34871640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f3012f670 0x7f2f3011bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.538+0000 7f2f2f7fe640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3009a510 msgr2=0x7f2f3013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.538+0000 7f2f2f7fe640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3009a510 0x7f2f3013f2d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800d890 tx=0x7f2f18008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1800de20 con 0x7f2f3009a510 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: dump: 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.558 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: _finish_auth 0 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1800bae0 con 0x7f2f3009a510 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 5 monclient: authenticate success, global_id 65447 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 10 monclient: _renew_subs 2024-09-18T12:34:12.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f28009a20 con 0x7f2f3009a510 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1800bea0 con 0x7f2f3009a510 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 10 monclient: dump: 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.560 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f2e7fc640 4 set_mon_vals no callback set 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 10 monclient: _renew_subs 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3011f290 con 0x7f2f3009a510 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2f18014640 con 0x7f2f3009a510 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 0x7f2f3011d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.539+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f30065f20 con 0x7f2f3011ccd0 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f27fff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f10038490 0x7f2f1003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(256..256 src has 13..256) v4 ==== 8758+0+0 (secure 0 0 0) 0x7f2f1804ce30 con 0x7f2f3009a510 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 0x7f2f3011d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient: get_auth_request con 0x7f2f3011ccd0 auth_method 0 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f34871640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f10038490 0x7f2f1003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f34871640 10 monclient: get_auth_request con 0x7f2f10038490 auth_method 0 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_done global_id 124162 payload 824 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.540+0000 7f2f2ffff640 1 -- 172.21.15.116:0/3131763363 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f3004bf60 con 0x7f2f3011ccd0 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.542+0000 7f2f34871640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f10038490 0x7f2f1003a940 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2f3009b450 tx=0x7f2f28037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.542+0000 7f2f2ffff640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 msgr2=0x7f2f3011d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.542+0000 7f2f2ffff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 0x7f2f3011d0a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f20011bf0 tx=0x7f2f200091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000fe90 con 0x7f2f3011ccd0 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 10 monclient: dump: 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.563 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2f20014650 con 0x7f2f3011ccd0 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 msgr2=0x7f2f3011d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 0x7f2f3011d0a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f20011bf0 tx=0x7f2f200091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 msgr2=0x7f2f3011d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:12.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.543+0000 7f2f27fff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f3011ccd0 0x7f2f3011d0a0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f20011bf0 tx=0x7f2f200091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.548+0000 7f2f362d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.548+0000 7f2f362d4640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 0x7f2f30029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.548+0000 7f2f362d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.548+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f30118b50 con 0x7f2f300298c0 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 0x7f2f30029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient: get_auth_request con 0x7f2f300298c0 auth_method 0 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 10 monclient(hunting): handle_auth_done global_id 124165 payload 824 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.549+0000 7f2f2ffff640 1 -- 172.21.15.116:0/3131763363 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2f300638d0 con 0x7f2f300298c0 2024-09-18T12:34:12.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.551+0000 7f2f2ffff640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 msgr2=0x7f2f30029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.551+0000 7f2f2ffff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 0x7f2f30029cc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f20011bf0 tx=0x7f2f2001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.551+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20025040 con 0x7f2f300298c0 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.551+0000 7f2f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.551+0000 7f2f27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.551+0000 7f2f27fff640 10 monclient: dump: 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f2f2001c3e0 con 0x7f2f300298c0 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f27fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f27fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 msgr2=0x7f2f30029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f27fff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 0x7f2f30029cc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f20011bf0 tx=0x7f2f2001d000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f27fff640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 msgr2=0x7f2f30029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f27fff640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f300298c0 0x7f2f30029cc0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f20011bf0 tx=0x7f2f2001d000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 94, 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.574 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:12.575 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:34:12.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:12.579 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:12.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:12.579 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:12.579 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:12.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f10038490 msgr2=0x7f2f1003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f10038490 0x7f2f1003a940 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2f3009b450 tx=0x7f2f28037000 comp rx=0 tx=0).stop 2024-09-18T12:34:12.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 10 monclient: shutdown 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3009a510 msgr2=0x7f2f3013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 --2- 172.21.15.116:0/3131763363 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f3009a510 0x7f2f3013f2d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800d890 tx=0x7f2f18008040 comp rx=0 tx=0).stop 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f34871640 1 -- 172.21.15.116:0/3131763363 reap_dead start 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f34871640 1 -- 172.21.15.116:0/3131763363 reap_dead start 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 shutdown_connections 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 >> 172.21.15.116:0/3131763363 conn(0x7f2f300672b0 msgr2=0x7f2f3006cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 shutdown_connections 2024-09-18T12:34:12.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:12.552+0000 7f2f362d4640 1 -- 172.21.15.116:0/3131763363 wait complete. 2024-09-18T12:34:12.581 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:34:12.582 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:34:12.582 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-18T12:34:12.582 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:34:12.582 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:34:12.582 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:34:12.582 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:34:12.597 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:34:12.597 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-18T12:34:12.597 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-18T12:34:12.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:34:13.188 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:13.187+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbf3000ad90 con 0x7fbf401321a0 2024-09-18T12:34:13.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:13.187+0000 7fbf3cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:13.189 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:13.187+0000 7fbf3cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:13.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:13.347+0000 7fbf4719d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T12:34:13.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:13.348+0000 7fbf4719d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T12:34:13.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:13.348+0000 7fbf4719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:13.350 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:13.348+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fbf400a13e0 con 0x7fbf401321a0 2024-09-18T12:34:15.357 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:15.356+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:15.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:15.356+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:15.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:15.356+0000 7fbf377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:16.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:16.356+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:16.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:16.356+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:17.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:17.356+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:17.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:17.356+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:18.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:18.357+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:18.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:18.357+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:19.357+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:19.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:19.357+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:20.357+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:20.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:20.357+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:21.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:21.357+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:21.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:21.357+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:22.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:22.357+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:22.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:22.357+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:23.358 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:23.357+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:23.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:23.357+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:24.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:24.358+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:24.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:24.358+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:25.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:25.358+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:25.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:25.358+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:25.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:25.358+0000 7fbf377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:26.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:26.358+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:26.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:26.358+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:27.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:27.358+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:27.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:27.358+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:27.598 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-18T12:34:27.598 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-18T12:34:27.598 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:34:27.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:34:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.699+0000 7f471f089640 10 monclient: get_monmap_and_config 2024-09-18T12:34:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.699+0000 7f471f089640 10 monclient: build_initial_monmap 2024-09-18T12:34:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.699+0000 7f471f089640 10 monclient: monmap: 2024-09-18T12:34:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:27.701345+0000 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:27.701345+0000 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:27.702 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 Processor -- start 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 -- start start 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient: init 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): picked mon.noname-a con 0x7f471809a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 0x7f47180975a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): picked mon.noname-b con 0x7f47180971a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): picked mon.noname-c con 0x7f471812f670 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4718097ae0 con 0x7f471809a510 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471809cb00 con 0x7f471812f670 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471809a8e0 con 0x7f47180971a0 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.700+0000 7f471f089640 10 monclient(hunting): authenticate will time out at 1295.022973s 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f4717fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 msgr2=0x7f47180975a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f4717fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 0x7f47180975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58740/0 (socket says 172.21.15.116:58740) 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 learned_addr learned my addr 172.21.15.116:0/530658357 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 10 monclient(hunting): get_auth_request con 0x7f471809a510 auth_method 0 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 10 monclient(hunting): get_auth_request con 0x7f471812f670 auth_method 0 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:34:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.701+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:27.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f4717fff640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 msgr2=0x7f47180975a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:27.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f4717fff640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 0x7f47180975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:34:27.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:27.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 10 monclient(hunting): get_auth_request con 0x7f471809a510 auth_method 0 2024-09-18T12:34:27.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:27.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 10 monclient(hunting): get_auth_request con 0x7f471812f670 auth_method 0 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:27.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:27.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:27.902+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f4717fff640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 msgr2=0x7f47180975a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f4717fff640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 0x7f47180975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 10 monclient(hunting): get_auth_request con 0x7f471809a510 auth_method 0 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:34:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 10 monclient(hunting): get_auth_request con 0x7f471812f670 auth_method 0 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:28.303+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:34:28.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:28.358+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:28.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:28.358+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f4717fff640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 msgr2=0x7f47180975a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f4717fff640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 0x7f47180975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 10 monclient(hunting): get_auth_request con 0x7f471809a510 auth_method 0 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 10 monclient(hunting): get_auth_request con 0x7f471812f670 auth_method 0 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:34:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:29.104+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:34:29.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:29.358+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:29.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:29.358+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:30.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:30.358+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:30.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:30.358+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:30.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 10 monclient(hunting): tick 2024-09-18T12:34:30.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 monclient(hunting): continuing hunt 2024-09-18T12:34:30.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:34:30.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 msgr2=0x7f47180975a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:30.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47180971a0 0x7f47180975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 msgr2=0x7f471812fa90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.144:3300/0 conn(0x7f471812f670 0x7f471812fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 -- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 msgr2=0x7f4718096c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 --2- 172.21.15.116:0/530658357 >> v2:172.21.15.116:3300/0 conn(0x7f471809a510 0x7f4718096c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 10 monclient: _add_conns ranks=[0,1] 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46fc003670 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46fc007370 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 -- 172.21.15.116:0/530658357 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471806ed80 con 0x7f46fc003670 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 1 -- 172.21.15.116:0/530658357 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc00c0a0 con 0x7f46fc007370 2024-09-18T12:34:30.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.701+0000 7f47167fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:30.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.703+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> v1:172.21.15.116:6789/0 conn(0x7f46fc003670 legacy=0x7f46fc005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:30.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:30.703+0000 7f4717fff640 1 -- 172.21.15.116:0/530658357 >> v1:172.21.15.144:6789/0 conn(0x7f46fc007370 legacy=0x7f46fc009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:31.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:31.359+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:31.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:31.359+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:32.359+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:32.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:32.359+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.609+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234838677 0 0) 0x7f46fc00c0a0 con 0x7f46fc007370 2024-09-18T12:34:32.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.609+0000 7f47177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.609+0000 7f47177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.609+0000 7f47177fe640 10 monclient(hunting): my global_id is 124168 2024-09-18T12:34:32.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.609+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47080034e0 con 0x7f46fc007370 2024-09-18T12:34:32.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.615+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329886788 0 0) 0x7f471806ed80 con 0x7f46fc003670 2024-09-18T12:34:32.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.615+0000 7f47177fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.615+0000 7f47177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.615+0000 7f47177fe640 10 monclient(hunting): my global_id is 104685 2024-09-18T12:34:32.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.615+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46fc00c0a0 con 0x7f46fc003670 2024-09-18T12:34:32.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.618+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685435608 0 0) 0x7f46fc00c0a0 con 0x7f46fc003670 2024-09-18T12:34:32.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.618+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 >> v1:172.21.15.144:6789/0 conn(0x7f46fc007370 legacy=0x7f46fc009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.618+0000 7f47177fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:32.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.618+0000 7f47177fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.618+0000 7f47177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc00d2c0 con 0x7f46fc003670 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient: _finish_auth 0 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f470c00c680 con 0x7f46fc003670 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471f089640 5 monclient: authenticate success, global_id 104685 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471f089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient: dump: 2024-09-18T12:34:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.626 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 >> v1:172.21.15.116:6789/0 conn(0x7f46fc003670 legacy=0x7f46fc005aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 reap_dead start 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708003be0 0x7f4708003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): picked mon.a con 0x7f4708003be0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708004780 0x7f4708004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): picked mon.b con 0x7f4708004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708005360 0x7f470800ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): picked mon.c con 0x7f4708005360 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470800c300 con 0x7f4708003be0 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470800c470 con 0x7f4708004780 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470800c630 con 0x7f4708005360 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f47177fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471f089640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471d626640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708005360 msgr2=0x7f470800ba50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471d626640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708005360 0x7f470800ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708003be0 0x7f4708003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): get_auth_request con 0x7f4708003be0 auth_method 0 2024-09-18T12:34:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708004780 0x7f4708004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 10 monclient(hunting): get_auth_request con 0x7f4708004780 auth_method 0 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f4717fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient(hunting): handle_auth_done global_id 104685 payload 293 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708005360 msgr2=0x7f470800ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708005360 0x7f470800ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708004780 msgr2=0x7f4708004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708004780 0x7f4708004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient: _finish_hunting 0 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 1 monclient: found mon.a 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc00c0a0 con 0x7f4708003be0 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 10 monclient: _finish_auth 0 2024-09-18T12:34:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.619+0000 7f471ce25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f471ce25640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708003be0 msgr2=0x7f4708003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f471ce25640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708003be0 0x7f4708003fb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f470c006f60 tx=0x7f470c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16faec6f0b857e4f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c034410 con 0x7f4708003be0 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 10 monclient: dump: 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c02e030 con 0x7f4708003be0 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f471f089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 1 -- 172.21.15.116:0/530658357 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c0345f0 con 0x7f4708003be0 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f471f089640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f471f089640 4 set_mon_vals no callback set 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.622+0000 7f47177fe640 10 monclient: dump: 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.633 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 10 monclient: shutdown 2024-09-18T12:34:32.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 -- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708003be0 msgr2=0x7f4708003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708003be0 0x7f4708003fb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f470c006f60 tx=0x7f470c008040 comp rx=0 tx=0).stop 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 -- 172.21.15.116:0/530658357 shutdown_connections 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708005360 0x7f470800ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708004780 0x7f4708004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 --2- 172.21.15.116:0/530658357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708003be0 0x7f4708003fb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 -- 172.21.15.116:0/530658357 >> 172.21.15.116:0/530658357 conn(0x7f47180672b0 msgr2=0x7f47180676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 -- 172.21.15.116:0/530658357 shutdown_connections 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.623+0000 7f471f089640 1 -- 172.21.15.116:0/530658357 wait complete. 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.624+0000 7f471f089640 10 monclient: build_initial_monmap 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.624+0000 7f471f089640 10 monclient: monmap: 2024-09-18T12:34:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.624+0000 7f471f089640 1 Processor -- start 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 1 -- start start 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient: init 2024-09-18T12:34:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 0x7f4718140db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): picked mon.noname-b con 0x7f4708003be0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708004780 0x7f47181412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): picked mon.noname-c con 0x7f4708004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708005360 0x7f4718098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): picked mon.noname-a con 0x7f4708005360 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471809c370 con 0x7f4708005360 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471806dc90 con 0x7f4708003be0 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4718121950 con 0x7f4708004780 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471f089640 10 monclient(hunting): authenticate will time out at 1299.947610s 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f4717fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708004780 msgr2=0x7f47181412f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f4717fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708004780 0x7f47181412f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708005360 0x7f4718098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 0x7f4718140db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 0x7f4718140db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:39738/0 (socket says 172.21.15.116:39738) 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 1 -- 172.21.15.116:0/1443038640 learned_addr learned my addr 172.21.15.116:0/1443038640 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708005360 0x7f4718098300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58782/0 (socket says 172.21.15.116:58782) 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 10 monclient(hunting): get_auth_request con 0x7f4708003be0 auth_method 0 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 10 monclient(hunting): get_auth_request con 0x7f4708005360 auth_method 0 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471ce25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.625+0000 7f471d626640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 10 monclient(hunting): handle_auth_done global_id 124171 payload 824 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 1 -- 172.21.15.116:0/1443038640 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708004780 msgr2=0x7f47181412f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708004780 0x7f47181412f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 1 -- 172.21.15.116:0/1443038640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708005360 msgr2=0x7f4718098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708005360 0x7f4718098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 10 monclient: _finish_hunting 0 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 1 monclient: found mon.noname-b 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471ce25640 1 -- 172.21.15.116:0/1443038640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4718121e80 con 0x7f4708003be0 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.626+0000 7f471d626640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708005360 0x7f4718098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471ce25640 1 -- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 msgr2=0x7f4718140db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471ce25640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 0x7f4718140db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f470c006fd0 tx=0x7f470c039000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 1 -- 172.21.15.116:0/1443038640 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c00c890 con 0x7f4708003be0 2024-09-18T12:34:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: dump: 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.646 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: _finish_auth 0 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471f089640 5 monclient: authenticate success, global_id 124171 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471f089640 10 monclient: _renew_subs 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471f089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47181206e0 con 0x7f4708003be0 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 1 -- 172.21.15.116:0/1443038640 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c00c040 con 0x7f4708003be0 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 1 -- 172.21.15.116:0/1443038640 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c040680 con 0x7f4708003be0 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4714ff9640 10 monclient: dump: 2024-09-18T12:34:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f4716ffd640 4 set_mon_vals no callback set 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471f089640 10 monclient: _renew_subs 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471f089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:32.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.628+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4718122770 con 0x7f4708003be0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.629+0000 7f4714ff9640 1 -- 172.21.15.116:0/1443038640 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f470c03e400 con 0x7f4708003be0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.629+0000 7f4714ff9640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46fc038ec0 0x7f46fc03b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.629+0000 7f471f089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.629+0000 7f471f089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.629+0000 7f471f089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.629+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471804bf60 con 0x7f4708003be0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.629+0000 7f4714ff9640 1 -- 172.21.15.116:0/1443038640 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(256..256 src has 13..256) v4 ==== 8758+0+0 (secure 0 0 0) 0x7f470c06d1d0 con 0x7f4708003be0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.630+0000 7f4717fff640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46fc038ec0 0x7f46fc03b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.630+0000 7f4717fff640 10 monclient: get_auth_request con 0x7f46fc038ec0 auth_method 0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.632+0000 7f4717fff640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46fc038ec0 0x7f46fc03b370 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f471809a4e0 tx=0x7f4704007800 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.634+0000 7f4714ff9640 1 -- 172.21.15.116:0/1443038640 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f470c045020 con 0x7f4708003be0 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.635+0000 7f4714ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:32.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.635+0000 7f4714ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:32.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.853+0000 7f471f089640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:34:32.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.853+0000 7f471f089640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:32.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.853+0000 7f471f089640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:32.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.853+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4718062f40 con 0x7f4708003be0 2024-09-18T12:34:32.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.854+0000 7f4714ff9640 1 -- 172.21.15.116:0/1443038640 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f470c034590 con 0x7f4708003be0 2024-09-18T12:34:32.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.854+0000 7f4714ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:34:32.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.854+0000 7f4714ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:32.856 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:34:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46fc038ec0 msgr2=0x7f46fc03b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46fc038ec0 0x7f46fc03b370 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f471809a4e0 tx=0x7f4704007800 comp rx=0 tx=0).stop 2024-09-18T12:34:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 10 monclient: shutdown 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 msgr2=0x7f4718140db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 0x7f4718140db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f470c006fd0 tx=0x7f470c039000 comp rx=0 tx=0).stop 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 shutdown_connections 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46fc038ec0 0x7f46fc03b370 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708005360 0x7f4718098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4708004780 0x7f47181412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 --2- 172.21.15.116:0/1443038640 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4708003be0 0x7f4718140db0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 >> 172.21.15.116:0/1443038640 conn(0x7f47180672b0 msgr2=0x7f471806d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 shutdown_connections 2024-09-18T12:34:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.856+0000 7f471f089640 1 -- 172.21.15.116:0/1443038640 wait complete. 2024-09-18T12:34:32.869 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:34:32.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:34:32.869 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:34:32.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.960+0000 7f7027d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.960+0000 7f7027d9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.960+0000 7f7027d9d640 10 monclient: monmap: 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:32.962272+0000 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:32.962272+0000 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:32.963 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 1 Processor -- start 2024-09-18T12:34:32.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 1 -- start start 2024-09-18T12:34:32.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient: init 2024-09-18T12:34:32.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f702009a650 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 0x7f702009cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7020099eb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7020133840 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7020098300 con 0x7f702009a650 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702009a280 con 0x7f7020133840 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7020098fe0 con 0x7f7020099eb0 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7027d9d640 10 monclient(hunting): authenticate will time out at 1300.283850s 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7025338640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 msgr2=0x7f702009cae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7025338640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 0x7f702009cae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7025b39640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f702009a650 0x7f702009aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34124/0 (socket says 172.21.15.116:34124) 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.961+0000 7f7025b39640 1 -- 172.21.15.116:0/3152869160 learned_addr learned my addr 172.21.15.116:0/3152869160 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 1 -- 172.21.15.116:0/3152869160 >> v1:172.21.15.116:6789/0 conn(0x7f702009a650 legacy=0x7f702009aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973217745 0 0) 0x7f7020098300 con 0x7f702009a650 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): my global_id is 104691 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700c0034e0 con 0x7f702009a650 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f702633a640 1 -- 172.21.15.116:0/3152869160 >> v1:172.21.15.144:6789/0 conn(0x7f7020133840 legacy=0x7f7020135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62224666 0 0) 0x7f700c0034e0 con 0x7f702009a650 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 msgr2=0x7f702009cae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 0x7f702009cae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 >> v1:172.21.15.144:6789/0 conn(0x7f7020133840 legacy=0x7f7020135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: _finish_hunting 0 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020099250 con 0x7f702009a650 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: _finish_auth 0 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7027d9d640 5 monclient: authenticate success, global_id 104691 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f701c0017a0 con 0x7f702009a650 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7027d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: dump: 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:34:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 >> v1:172.21.15.116:6789/0 conn(0x7f702009a650 legacy=0x7f702009aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f700c003ad0 0x7f700c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): picked mon.b con 0x7f700c003ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f700c004780 0x7f700c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): picked mon.a con 0x7f700c004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 0x7f700c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): picked mon.c con 0x7f7020099eb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020098fe0 con 0x7f700c004780 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c005630 con 0x7f700c003ad0 2024-09-18T12:34:32.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c00bf90 con 0x7f7020099eb0 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7024b37640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7027d9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f702633a640 1 -- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 msgr2=0x7f700c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f702633a640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 0x7f700c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f700c004780 0x7f700c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): get_auth_request con 0x7f700c004780 auth_method 0 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f700c003ad0 0x7f700c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 10 monclient(hunting): get_auth_request con 0x7f700c003ad0 auth_method 0 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient(hunting): handle_auth_done global_id 104691 payload 293 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 1 -- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 msgr2=0x7f700c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7020099eb0 0x7f700c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 1 -- 172.21.15.116:0/3152869160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f700c003ad0 msgr2=0x7f700c003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f700c003ad0 0x7f700c003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient: _finish_hunting 0 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 1 monclient: found mon.a 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 1 -- 172.21.15.116:0/3152869160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c0034e0 con 0x7f700c004780 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 10 monclient: _finish_auth 0 2024-09-18T12:34:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.964+0000 7f7025338640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7025338640 1 -- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f700c004780 msgr2=0x7f700c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7025338640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f700c004780 0x7f700c004bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f70100099c0 tx=0x7f7010009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7f086c7c094d01 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010003b80 con 0x7f700c004780 2024-09-18T12:34:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 10 monclient: dump: 2024-09-18T12:34:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7027d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7010003d20 con 0x7f700c004780 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7027d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7027d9d640 4 set_mon_vals no callback set 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 1 -- 172.21.15.116:0/3152869160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701002ab80 con 0x7f700c004780 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.967+0000 7f7024b37640 10 monclient: dump: 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 10 monclient: shutdown 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 1 -- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f700c004780 msgr2=0x7f700c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 1 --2- 172.21.15.116:0/3152869160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f700c004780 0x7f700c004bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f70100099c0 tx=0x7f7010009060 comp rx=0 tx=0).stop 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f702633a640 1 -- 172.21.15.116:0/3152869160 reap_dead start 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 1 -- 172.21.15.116:0/3152869160 shutdown_connections 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 1 -- 172.21.15.116:0/3152869160 >> 172.21.15.116:0/3152869160 conn(0x7f7020067d00 msgr2=0x7f70200a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 1 -- 172.21.15.116:0/3152869160 shutdown_connections 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 1 -- 172.21.15.116:0/3152869160 wait complete. 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.968+0000 7f7027d9d640 10 monclient: monmap: 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 Processor -- start 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 -- start start 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient: init 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f700c004780 0x7f702009b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f700c004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702011b8f0 0x7f7020130880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f702011b8f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020130dc0 0x7f702013eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7020130dc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702011bd00 con 0x7f702011b8f0 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702009c360 con 0x7f7020130dc0 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702011d9a0 con 0x7f700c004780 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7027d9d640 10 monclient(hunting): authenticate will time out at 1300.291725s 2024-09-18T12:34:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025b39640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f700c004780 msgr2=0x7f702009b7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f700c004780 0x7f702009b7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702011b8f0 0x7f7020130880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702011b8f0 0x7f7020130880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58790/0 (socket says 172.21.15.116:58790) 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 1 -- 172.21.15.116:0/3035081387 learned_addr learned my addr 172.21.15.116:0/3035081387 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 10 monclient(hunting): get_auth_request con 0x7f702011b8f0 auth_method 0 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f7025338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f702633a640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020130dc0 0x7f702013eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f702633a640 10 monclient(hunting): get_auth_request con 0x7f7020130dc0 auth_method 0 2024-09-18T12:34:32.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f702633a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f702633a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.969+0000 7f702633a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f702633a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f702633a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f702633a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 10 monclient(hunting): handle_auth_done global_id 104694 payload 824 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f700c004780 msgr2=0x7f702009b7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f700c004780 0x7f702009b7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020130dc0 msgr2=0x7f702013eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7020130dc0 0x7f702013eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 10 monclient: _finish_hunting 0 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 1 monclient: found mon.noname-a 2024-09-18T12:34:32.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.970+0000 7f7025338640 1 -- 172.21.15.116:0/3035081387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702011dc80 con 0x7f702011b8f0 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7025338640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702011b8f0 msgr2=0x7f7020130880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7025338640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702011b8f0 0x7f7020130880 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7010009420 tx=0x7f7010030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701003b070 con 0x7f702011b8f0 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: dump: 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.992 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7010005510 con 0x7f702011b8f0 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7027d9d640 5 monclient: authenticate success, global_id 104694 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7027d9d640 10 monclient: _renew_subs 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7027d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:32.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702011df60 con 0x7f702011b8f0 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010033470 con 0x7f702011b8f0 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7015ffb640 10 monclient: dump: 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.994 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7027d9d640 10 monclient: _renew_subs 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7027d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702011cab0 con 0x7f702011b8f0 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.972+0000 7f7017fff640 4 set_mon_vals no callback set 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7010030070 con 0x7f702011b8f0 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7027d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7027d9d640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 0x7f702013f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7027d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020065f90 con 0x7f702013f3e0 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 0x7f702013f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient: get_auth_request con 0x7f702013f3e0 auth_method 0 2024-09-18T12:34:32.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7015ffb640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(256..256 src has 13..256) v4 ==== 8758+0+0 (secure 0 0 0) 0x7f701006bb00 con 0x7f702011b8f0 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 10 monclient(hunting): handle_auth_done global_id 104697 payload 824 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f7025b39640 1 -- 172.21.15.116:0/3035081387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f702011db10 con 0x7f702013f3e0 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.973+0000 7f702633a640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.974+0000 7f702633a640 10 monclient: get_auth_request con 0x7f7004038440 auth_method 0 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.975+0000 7f7025b39640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 msgr2=0x7f702013f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:32.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.975+0000 7f7025b39640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 0x7f702013f7b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f701c001980 tx=0x7f701c007a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c00f990 con 0x7f702013f3e0 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 10 monclient: dump: 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:32.997 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f701c00faf0 con 0x7f702013f3e0 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 msgr2=0x7f702013f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 0x7f702013f7b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f701c001980 tx=0x7f701c007a70 comp rx=0 tx=0).stop 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 msgr2=0x7f702013f7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f7015ffb640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702013f3e0 0x7f702013f7b0 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f701c001980 tx=0x7f701c007a70 comp rx=0 tx=0).stop 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.976+0000 7f702633a640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7020097d30 tx=0x7f7018009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.981+0000 7f7027d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.981+0000 7f7027d9d640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 0x7f702006be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7027d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:32.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020131190 con 0x7f702006ba50 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 0x7f702006be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient: get_auth_request con 0x7f702006ba50 auth_method 0 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 10 monclient(hunting): handle_auth_done global_id 104700 payload 824 2024-09-18T12:34:32.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.982+0000 7f7025b39640 1 -- 172.21.15.116:0/3035081387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f702013fcf0 con 0x7f702006ba50 2024-09-18T12:34:33.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.984+0000 7f7025b39640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 msgr2=0x7f702006be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.984+0000 7f7025b39640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 0x7f702006be50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f701c001980 tx=0x7f701c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.984+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c0219e0 con 0x7f702006ba50 2024-09-18T12:34:33.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.984+0000 7f7015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.984+0000 7f7015ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.984+0000 7f7015ffb640 10 monclient: dump: 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.001 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f701c021b40 con 0x7f702006ba50 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7015ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7015ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 msgr2=0x7f702006be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7015ffb640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 0x7f702006be50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f701c001980 tx=0x7f701c013000 comp rx=0 tx=0).stop 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7015ffb640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 msgr2=0x7f702006be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7015ffb640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702006ba50 0x7f702006be50 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f701c001980 tx=0x7f701c013000 comp rx=0 tx=0).stop 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 96, 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:33.009 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:33.010 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.012 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.013 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:33.014 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:33.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.015 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.015 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:33.015 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 msgr2=0x7f700403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7004038440 0x7f700403a8f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7020097d30 tx=0x7f7018009040 comp rx=0 tx=0).stop 2024-09-18T12:34:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 10 monclient: shutdown 2024-09-18T12:34:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702011b8f0 msgr2=0x7f7020130880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 --2- 172.21.15.116:0/3035081387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f702011b8f0 0x7f7020130880 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7010009420 tx=0x7f7010030000 comp rx=0 tx=0).stop 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f702633a640 1 -- 172.21.15.116:0/3035081387 reap_dead start 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f702633a640 1 -- 172.21.15.116:0/3035081387 reap_dead start 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 shutdown_connections 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 >> 172.21.15.116:0/3035081387 conn(0x7f7020067d00 msgr2=0x7f702009ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 shutdown_connections 2024-09-18T12:34:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:32.985+0000 7f7027d9d640 1 -- 172.21.15.116:0/3035081387 wait complete. 2024-09-18T12:34:33.017 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:34:33.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:34:33.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:34:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.109+0000 7f5178b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.109+0000 7f5178b9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.109+0000 7f5178b9d640 10 monclient: monmap: 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:34:33.111082+0000 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:34:33.111082+0000 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:33.113 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.109+0000 7f5178b9d640 1 Processor -- start 2024-09-18T12:34:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 1 -- start start 2024-09-18T12:34:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient: init 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 0x7f5174098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f51740981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5174131da0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5174132170 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5174098ad0 con 0x7f5174131da0 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5174067f30 con 0x7f5174132170 2024-09-18T12:34:33.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5174131bd0 con 0x7f51740981c0 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5178b9d640 10 monclient(hunting): authenticate will time out at 1300.432654s 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f517259c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 msgr2=0x7f5174098590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f517259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 0x7f5174098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5171d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5174131da0 0x7f5174067110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34136/0 (socket says 172.21.15.116:34136) 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.110+0000 7f5171d9b640 1 -- 172.21.15.116:0/3772349925 learned_addr learned my addr 172.21.15.116:0/3772349925 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.112+0000 7f5171d9b640 1 -- 172.21.15.116:0/3772349925 >> v1:172.21.15.116:6789/0 conn(0x7f5174131da0 legacy=0x7f5174067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754794415 0 0) 0x7f5174098ad0 con 0x7f5174131da0 2024-09-18T12:34:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): my global_id is 104703 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51600034e0 con 0x7f5174131da0 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 -- 172.21.15.116:0/3772349925 >> v1:172.21.15.144:6789/0 conn(0x7f5174132170 legacy=0x7f5174067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4252479751 0 0) 0x7f51600034e0 con 0x7f5174131da0 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 msgr2=0x7f5174098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 0x7f5174098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 >> v1:172.21.15.144:6789/0 conn(0x7f5174132170 legacy=0x7f5174067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: _finish_hunting 0 2024-09-18T12:34:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517409ca60 con 0x7f5174131da0 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: _finish_auth 0 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f515c0017a0 con 0x7f5174131da0 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5178b9d640 5 monclient: authenticate success, global_id 104703 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5178b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: dump: 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 >> v1:172.21.15.116:6789/0 conn(0x7f5174131da0 legacy=0x7f5174067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 0x7f5160003af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): picked mon.c con 0x7f51740981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004250 0x7f5160004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): picked mon.b con 0x7f5160004250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:33.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5160004f10 0x7f516000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): picked mon.a con 0x7f5160004f10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5174131bd0 con 0x7f5160004f10 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516000bf10 con 0x7f5160004250 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517259c640 1 -- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 msgr2=0x7f5160003af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516000c080 con 0x7f51740981c0 2024-09-18T12:34:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517259c640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 0x7f5160003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f517159a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5178b9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5160004f10 0x7f516000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): get_auth_request con 0x7f5160004f10 auth_method 0 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:33.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004250 0x7f5160004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 10 monclient(hunting): get_auth_request con 0x7f5160004250 auth_method 0 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:34:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient(hunting): handle_auth_done global_id 104703 payload 293 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 -- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 msgr2=0x7f5160003af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51740981c0 0x7f5160003af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 -- 172.21.15.116:0/3772349925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004250 msgr2=0x7f5160004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004250 0x7f5160004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 monclient: found mon.a 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 1 -- 172.21.15.116:0/3772349925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51600034e0 con 0x7f5160004f10 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 10 monclient: _finish_auth 0 2024-09-18T12:34:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.113+0000 7f5172d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f5172d9d640 1 -- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5160004f10 msgr2=0x7f516000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f5172d9d640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5160004f10 0x7f516000b9d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f516c0021d0 tx=0x7f516c001830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82332a925195554c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c005b00 con 0x7f5160004f10 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 10 monclient: dump: 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f5178b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c02d070 con 0x7f5160004f10 2024-09-18T12:34:33.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 1 -- 172.21.15.116:0/3772349925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c005cc0 con 0x7f5160004f10 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f5178b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f5178b9d640 4 set_mon_vals no callback set 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.116+0000 7f517159a640 10 monclient: dump: 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 10 monclient: shutdown 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 1 -- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5160004f10 msgr2=0x7f516000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 1 --2- 172.21.15.116:0/3772349925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5160004f10 0x7f516000b9d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f516c0021d0 tx=0x7f516c001830 comp rx=0 tx=0).stop 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5172d9d640 1 -- 172.21.15.116:0/3772349925 reap_dead start 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 1 -- 172.21.15.116:0/3772349925 shutdown_connections 2024-09-18T12:34:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 1 -- 172.21.15.116:0/3772349925 >> 172.21.15.116:0/3772349925 conn(0x7f51740a1e30 msgr2=0x7f51740a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 1 -- 172.21.15.116:0/3772349925 shutdown_connections 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 1 -- 172.21.15.116:0/3772349925 wait complete. 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.117+0000 7f5178b9d640 10 monclient: monmap: 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 Processor -- start 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 -- start start 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient: init 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:33.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004f10 0x7f5174141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5160004f10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51741415b0 0x7f5174130a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f51741415b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 0x7f517411a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5174130fd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5174098cb0 con 0x7f5174130fd0 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51740a0590 con 0x7f5160004f10 2024-09-18T12:34:33.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517411de90 con 0x7f51741415b0 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5178b9d640 10 monclient(hunting): authenticate will time out at 1300.440569s 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5171d9b640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51741415b0 msgr2=0x7f5174130a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5171d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51741415b0 0x7f5174130a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 0x7f517411a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 0x7f517411a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58830/0 (socket says 172.21.15.116:58830) 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 -- 172.21.15.116:0/1065048912 learned_addr learned my addr 172.21.15.116:0/1065048912 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): get_auth_request con 0x7f5174130fd0 auth_method 0 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004f10 0x7f5174141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 10 monclient(hunting): get_auth_request con 0x7f5160004f10 auth_method 0 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f517259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient(hunting): handle_auth_done global_id 104706 payload 824 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51741415b0 msgr2=0x7f5174130a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f51741415b0 0x7f5174130a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004f10 msgr2=0x7f5174141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5160004f10 0x7f5174141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 monclient: found mon.noname-a 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.118+0000 7f5172d9d640 1 -- 172.21.15.116:0/1065048912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517411c080 con 0x7f5174130fd0 2024-09-18T12:34:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5172d9d640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 msgr2=0x7f517411a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5172d9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 0x7f517411a670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f516c0018b0 tx=0x7f516c033000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c0079f0 con 0x7f5174130fd0 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: dump: 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c032430 con 0x7f5174130fd0 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5178b9d640 5 monclient: authenticate success, global_id 104706 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5178b9d640 10 monclient: _renew_subs 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5178b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517411dbb0 con 0x7f5174130fd0 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c0327f0 con 0x7f5174130fd0 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f516a7fc640 10 monclient: dump: 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5170d99640 4 set_mon_vals no callback set 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5178b9d640 10 monclient: _renew_subs 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5178b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.121+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5174141d70 con 0x7f5174130fd0 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f516c02f030 con 0x7f5174130fd0 2024-09-18T12:34:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5178b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f516a7fc640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5154038440 0x7f515403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5178b9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 0x7f517411af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5178b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5174065f90 con 0x7f517411abb0 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(256..256 src has 13..256) v4 ==== 8758+0+0 (secure 0 0 0) 0x7f516c06d090 con 0x7f5174130fd0 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f517259c640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5154038440 0x7f515403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 0x7f517411af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f517259c640 10 monclient: get_auth_request con 0x7f5154038440 auth_method 0 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient: get_auth_request con 0x7f517411abb0 auth_method 0 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 10 monclient(hunting): handle_auth_done global_id 124186 payload 824 2024-09-18T12:34:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.122+0000 7f5171d9b640 1 -- 172.21.15.116:0/1065048912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f517411e020 con 0x7f517411abb0 2024-09-18T12:34:33.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.124+0000 7f517259c640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5154038440 0x7f515403a8f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5164003850 tx=0x7f51640090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.124+0000 7f5171d9b640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 msgr2=0x7f517411af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.124+0000 7f5171d9b640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 0x7f517411af80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f515c0079c0 tx=0x7f5174096f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00f910 con 0x7f517411abb0 2024-09-18T12:34:33.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 10 monclient: dump: 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.155 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f515c00fa70 con 0x7f517411abb0 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 msgr2=0x7f517411af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 0x7f517411af80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f515c0079c0 tx=0x7f5174096f10 comp rx=0 tx=0).stop 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 msgr2=0x7f517411af80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.125+0000 7f516a7fc640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f517411abb0 0x7f517411af80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5178b9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:34:33.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5178b9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 0x7f5174099710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5178b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517411dd40 con 0x7f5174099310 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5171d9b640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 0x7f5174099710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5171d9b640 10 monclient: get_auth_request con 0x7f5174099310 auth_method 0 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5171d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5171d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.131+0000 7f5171d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.132+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.132+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.132+0000 7f5171d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.132+0000 7f5171d9b640 10 monclient(hunting): handle_auth_done global_id 124189 payload 824 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.132+0000 7f5171d9b640 1 -- 172.21.15.116:0/1065048912 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f517411b4c0 con 0x7f5174099310 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.134+0000 7f5171d9b640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 msgr2=0x7f5174099710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.134+0000 7f5171d9b640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 0x7f5174099710 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f515c014000 tx=0x7f515c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.134+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c01cc90 con 0x7f5174099310 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.134+0000 7f516a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.134+0000 7f516a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.134+0000 7f516a7fc640 10 monclient: dump: 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.158 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f515c01cdf0 con 0x7f5174099310 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f516a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f516a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 msgr2=0x7f5174099710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f516a7fc640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 0x7f5174099710 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f515c014000 tx=0x7f515c013000 comp rx=0 tx=0).stop 2024-09-18T12:34:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f516a7fc640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 msgr2=0x7f5174099710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:34:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f516a7fc640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5174099310 0x7f5174099710 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f515c014000 tx=0x7f515c013000 comp rx=0 tx=0).stop 2024-09-18T12:34:33.160 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:34:33.160 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:33.160 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:33.160 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:34:33.160 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 96, 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:33.161 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:33.162 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:34:33.163 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:34:33.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:34:33.165 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.166 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:33.167 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:33.169 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:34:33.171 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.172 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:33.173 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:34:33.174 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:34:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5154038440 msgr2=0x7f515403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5154038440 0x7f515403a8f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5164003850 tx=0x7f51640090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 10 monclient: shutdown 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5172d9d640 1 -- 172.21.15.116:0/1065048912 reap_dead start 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 msgr2=0x7f517411a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 0x7f517411a670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f516c0018b0 tx=0x7f516c033000 comp rx=0 tx=0).stop 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 shutdown_connections 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 --2- 172.21.15.116:0/1065048912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5174130fd0 0x7f517411a670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 >> 172.21.15.116:0/1065048912 conn(0x7f51740a1e30 msgr2=0x7f5174096f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 shutdown_connections 2024-09-18T12:34:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:34:33.135+0000 7f5178b9d640 1 -- 172.21.15.116:0/1065048912 wait complete. 2024-09-18T12:34:33.178 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:34:33.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:34:33.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.359+0000 7fbf377fe640 10 monclient: tick 2024-09-18T12:34:33.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.359+0000 7fbf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:33.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.609+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v257) v1 ==== 121+0+0 (secure 0 0 0) 0x7fbf30010770 con 0x7fbf401321a0 2024-09-18T12:34:33.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.609+0000 7fbf3cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T12:34:33.610 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.609+0000 7fbf3cff9640 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-18T12:34:33.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.709+0000 7fbf4719d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T12:34:33.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.709+0000 7fbf4719d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T12:34:33.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.709+0000 7fbf4719d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:33.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.709+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fbf40072640 con 0x7fbf401321a0 2024-09-18T12:34:33.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.710+0000 7fbf3cff9640 1 -- 172.21.15.144:0/1894471539 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v257) v1 ==== 120+0+0 (secure 0 0 0) 0x7fbf30010950 con 0x7fbf401321a0 2024-09-18T12:34:33.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.710+0000 7fbf3cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T12:34:33.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.710+0000 7fbf3cff9640 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-18T12:34:33.711 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h30' does not appear in the crush map 2024-09-18T12:34:33.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf20038490 msgr2=0x7fbf2003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf20038490 0x7fbf2003a940 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbf4009bf50 tx=0x7fbf38032000 comp rx=0 tx=0).stop 2024-09-18T12:34:33.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 10 monclient: shutdown 2024-09-18T12:34:33.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:33.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 msgr2=0x7fbf4011b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 0x7fbf4011b760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000c910 tx=0x7fbf30007ec0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 shutdown_connections 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbf20038490 0x7fbf2003a940 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf401321a0 0x7fbf4011b760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbf4009f020 0x7fbf40130a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 --2- 172.21.15.144:0/1894471539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4009ec50 0x7fbf4009a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 >> 172.21.15.144:0/1894471539 conn(0x7fbf400a5cf0 msgr2=0x7fbf40134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 shutdown_connections 2024-09-18T12:34:33.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.712+0000 7fbf4719d640 1 -- 172.21.15.144:0/1894471539 wait complete. 2024-09-18T12:34:33.722 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:33.722 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h31 2024-09-18T12:34:33.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.794+0000 7ff70279d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:33.795 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.794+0000 7ff70279d640 10 monclient: build_initial_monmap 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.794+0000 7ff70279d640 10 monclient: monmap: 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:33.795484+0000 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:33.795484+0000 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:33.796 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:33.797 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 1 Processor -- start 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 1 -- start start 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient: init 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6fc0677e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff6fc094080 0x7ff6fc094480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6fc094080 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6fc0949c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:33.799 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc09d570 con 0x7ff6fc0677e0 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc095f10 con 0x7ff6fc0949c0 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6fc09b560 con 0x7ff6fc094080 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.795+0000 7ff70279d640 10 monclient(hunting): authenticate will time out at 1309.364988s 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff6fc094080 0x7ff6fc094480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff700d3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7ff6fc0949c0 0x7ff6fc136d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:46920/0 (socket says 172.21.15.144:46920) 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff700d3a640 1 -- 172.21.15.144:0/1150796026 learned_addr learned my addr 172.21.15.144:0/1150796026 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): get_auth_request con 0x7ff6fc094080 auth_method 0 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.800 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient(hunting): handle_auth_done global_id 124192 payload 274 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 1 -- 172.21.15.144:0/1150796026 >> v1:172.21.15.144:6789/0 conn(0x7ff6fc0949c0 legacy=0x7ff6fc136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 1 -- 172.21.15.144:0/1150796026 >> v1:172.21.15.116:6789/0 conn(0x7ff6fc0677e0 legacy=0x7ff6fc095800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 1 monclient: found mon.noname-c 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:33.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.796+0000 7ff6fb7fe640 1 -- 172.21.15.144:0/1150796026 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc09d800 con 0x7ff6fc094080 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6fb7fe640 1 -- 172.21.15.144:0/1150796026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc094080 msgr2=0x7ff6fc094480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6fb7fe640 1 --2- 172.21.15.144:0/1150796026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc094080 0x7ff6fc094480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009850 tx=0x7ff6f00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaefa6460b5fa55b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 1 -- 172.21.15.144:0/1150796026 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0032440 con 0x7ff6fc094080 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: dump: 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.807 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 1 -- 172.21.15.144:0/1150796026 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f00325a0 con 0x7ff6fc094080 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 5 monclient: authenticate success, global_id 124192 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 1 -- 172.21.15.144:0/1150796026 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0032980 con 0x7ff6fc094080 2024-09-18T12:34:33.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff6faffd640 10 monclient: dump: 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:33.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 4 set_mon_vals no callback set 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 10 monclient: shutdown 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 1 -- 172.21.15.144:0/1150796026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc094080 msgr2=0x7ff6fc094480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 1 --2- 172.21.15.144:0/1150796026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc094080 0x7ff6fc094480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009850 tx=0x7ff6f00090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 1 -- 172.21.15.144:0/1150796026 shutdown_connections 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 1 --2- 172.21.15.144:0/1150796026 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc094080 0x7ff6fc094480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 1 -- 172.21.15.144:0/1150796026 >> 172.21.15.144:0/1150796026 conn(0x7ff6fc0a16a0 msgr2=0x7ff6fc0a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 1 -- 172.21.15.144:0/1150796026 shutdown_connections 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.799+0000 7ff70279d640 1 -- 172.21.15.144:0/1150796026 wait complete. 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.800+0000 7ff70279d640 10 monclient: build_initial_monmap 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.800+0000 7ff70279d640 10 monclient: monmap: 2024-09-18T12:34:33.810 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.800+0000 7ff70279d640 1 Processor -- start 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.800+0000 7ff70279d640 1 -- start start 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.800+0000 7ff70279d640 10 monclient: init 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:33.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 0x7ff6fc098db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): picked mon.noname-b con 0x7ff6fc0677e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6fc094080 0x7ff6fc0992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6fc094080 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6fc0949c0 0x7ff6fc099830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6fc0949c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc095090 con 0x7ff6fc0949c0 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0a0260 con 0x7ff6fc0677e0 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc08f520 con 0x7ff6fc094080 2024-09-18T12:34:33.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff70279d640 10 monclient(hunting): authenticate will time out at 1309.370186s 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 0x7ff6fc098db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 0x7ff6fc098db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39556/0 (socket says 172.21.15.144:39556) 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 -- 172.21.15.144:0/3066648158 learned_addr learned my addr 172.21.15.144:0/3066648158 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): get_auth_request con 0x7ff6fc0677e0 auth_method 0 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fb7fe640 1 -- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6fc094080 msgr2=0x7ff6fc0992f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fb7fe640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6fc094080 0x7ff6fc0992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:33.813 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient(hunting): handle_auth_done global_id 124195 payload 824 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 -- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6fc094080 msgr2=0x7ff6fc0992f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff700d3a640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6fc0949c0 0x7ff6fc099830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6fc094080 0x7ff6fc0992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 -- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6fc0949c0 msgr2=0x7ff6fc099830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6fc0949c0 0x7ff6fc099830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 monclient: found mon.noname-b 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.801+0000 7ff6fbfff640 1 -- 172.21.15.144:0/3066648158 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0009390 con 0x7ff6fc0677e0 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6fbfff640 1 -- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 msgr2=0x7ff6fc098db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6fbfff640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 0x7ff6fc098db0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00c5e0 tx=0x7ff6ec007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec012920 con 0x7ff6fc0677e0 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: dump: 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:33.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: _finish_auth 0 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec012a80 con 0x7ff6fc0677e0 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff70279d640 5 monclient: authenticate success, global_id 124195 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff70279d640 10 monclient: _renew_subs 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff70279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6fc09a720 con 0x7ff6fc0677e0 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec012e80 con 0x7ff6fc0677e0 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6e3fff640 10 monclient: dump: 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:33.816 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff6fa7fc640 4 set_mon_vals no callback set 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff70279d640 10 monclient: _renew_subs 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff70279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.803+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6fc08dd80 con 0x7ff6fc0677e0 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.804+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff6ec011590 con 0x7ff6fc0677e0 2024-09-18T12:34:33.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.804+0000 7ff70279d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.804+0000 7ff70279d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.804+0000 7ff70279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.804+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6fc0008d0 con 0x7ff6fc0677e0 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.805+0000 7ff6e3fff640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6e8038440 0x7ff6e803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.805+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(257..257 src has 13..257) v4 ==== 8545+0+0 (secure 0 0 0) 0x7ff6ec04c920 con 0x7ff6fc0677e0 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.805+0000 7ff6fb7fe640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6e8038440 0x7ff6e803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.805+0000 7ff6fb7fe640 10 monclient: get_auth_request con 0x7ff6e8038440 auth_method 0 2024-09-18T12:34:33.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.807+0000 7ff6fb7fe640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6e8038440 0x7ff6e803a8f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0034000 tx=0x7ff6f0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:33.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.809+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff6ec010ac0 con 0x7ff6fc0677e0 2024-09-18T12:34:33.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.809+0000 7ff6e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:33.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.809+0000 7ff6e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:33.972 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.971+0000 7ff70279d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T12:34:33.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.971+0000 7ff70279d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T12:34:33.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.971+0000 7ff70279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:33.973 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:33.971+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7ff6fc094dc0 con 0x7ff6fc0677e0 2024-09-18T12:34:34.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.617+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v258) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff6ec016030 con 0x7ff6fc0677e0 2024-09-18T12:34:34.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.617+0000 7ff6e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T12:34:34.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.617+0000 7ff6e3fff640 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-18T12:34:34.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.716+0000 7ff70279d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T12:34:34.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.716+0000 7ff70279d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T12:34:34.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.716+0000 7ff70279d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:34.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.716+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7ff6fc072640 con 0x7ff6fc0677e0 2024-09-18T12:34:34.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.717+0000 7ff6e3fff640 1 -- 172.21.15.144:0/3066648158 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v258) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff6ec016030 con 0x7ff6fc0677e0 2024-09-18T12:34:34.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.717+0000 7ff6e3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T12:34:34.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.717+0000 7ff6e3fff640 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-18T12:34:34.718 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h31' does not appear in the crush map 2024-09-18T12:34:34.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6e8038440 msgr2=0x7ff6e803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:34.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6e8038440 0x7ff6e803a8f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0034000 tx=0x7ff6f0032000 comp rx=0 tx=0).stop 2024-09-18T12:34:34.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 10 monclient: shutdown 2024-09-18T12:34:34.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:34.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 msgr2=0x7ff6fc098db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 0x7ff6fc098db0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00c5e0 tx=0x7ff6ec007ed0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 shutdown_connections 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff6e8038440 0x7ff6e803a8f0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6fc0949c0 0x7ff6fc099830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff6fc094080 0x7ff6fc0992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 --2- 172.21.15.144:0/3066648158 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6fc0677e0 0x7ff6fc098db0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 >> 172.21.15.144:0/3066648158 conn(0x7ff6fc0a16a0 msgr2=0x7ff6fc09bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 shutdown_connections 2024-09-18T12:34:34.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.719+0000 7ff70279d640 1 -- 172.21.15.144:0/3066648158 wait complete. 2024-09-18T12:34:34.729 INFO:tasks.workunit.client.0.smithi144.stderr:+ for f in `seq 1 32` 2024-09-18T12:34:34.729 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm h32 2024-09-18T12:34:34.801 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.800+0000 7fef5555d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.800+0000 7fef5555d640 10 monclient: build_initial_monmap 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.800+0000 7fef5555d640 10 monclient: monmap: 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:34.801834+0000 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:34.801834+0000 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:34.802 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:34.803 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 1 Processor -- start 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 1 -- start start 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 10 monclient: init 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef500a1800 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef500983d0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fef5012ec70 0x7fef50137200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 10 monclient(hunting): picked mon.noname-c con 0x7fef5012ec70 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef500a1bd0 con 0x7fef500983d0 2024-09-18T12:34:34.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.801+0000 7fef5555d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5009cbf0 con 0x7fef500a1800 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef5555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef5555d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef5012f600 con 0x7fef5012ec70 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef5555d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef5555d640 10 monclient(hunting): authenticate will time out at 1310.371013s 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4effd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fef500a1800 0x7fef5009aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:46936/0 (socket says 172.21.15.144:46936) 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4effd640 1 -- 172.21.15.144:0/1097438857 learned_addr learned my addr 172.21.15.144:0/1097438857 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 1 --2- 172.21.15.144:0/1097438857 >> v2:172.21.15.144:3300/0 conn(0x7fef5012ec70 0x7fef50137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): get_auth_request con 0x7fef5012ec70 auth_method 0 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:34.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_done global_id 124198 payload 274 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 1 -- 172.21.15.144:0/1097438857 >> v1:172.21.15.144:6789/0 conn(0x7fef500a1800 legacy=0x7fef5009aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 1 -- 172.21.15.144:0/1097438857 >> v1:172.21.15.116:6789/0 conn(0x7fef500983d0 legacy=0x7fef50133ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:34.807 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.802+0000 7fef4f7fe640 1 -- 172.21.15.144:0/1097438857 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50098cc0 con 0x7fef5012ec70 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4f7fe640 1 -- 172.21.15.144:0/1097438857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5012ec70 msgr2=0x7fef50137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4f7fe640 1 --2- 172.21.15.144:0/1097438857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5012ec70 0x7fef50137200 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef48009e60 tx=0x7fef48009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67ace883f2f874bf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 1 -- 172.21.15.144:0/1097438857 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48007840 con 0x7fef5012ec70 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: dump: 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:34.814 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef5555d640 5 monclient: authenticate success, global_id 124198 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef5555d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:34.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 1 -- 172.21.15.144:0/1097438857 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4802d050 con 0x7fef5012ec70 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef5555d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef5555d640 4 set_mon_vals no callback set 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 1 -- 172.21.15.144:0/1097438857 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48007c10 con 0x7fef5012ec70 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.805+0000 7fef4dffb640 10 monclient: dump: 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.816 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 10 monclient: shutdown 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 1 -- 172.21.15.144:0/1097438857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5012ec70 msgr2=0x7fef50137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 1 --2- 172.21.15.144:0/1097438857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5012ec70 0x7fef50137200 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef48009e60 tx=0x7fef48009780 comp rx=0 tx=0).stop 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 1 -- 172.21.15.144:0/1097438857 shutdown_connections 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 1 --2- 172.21.15.144:0/1097438857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5012ec70 0x7fef50137200 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 1 -- 172.21.15.144:0/1097438857 >> 172.21.15.144:0/1097438857 conn(0x7fef50071100 msgr2=0x7fef500a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:34.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 1 -- 172.21.15.144:0/1097438857 shutdown_connections 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 1 -- 172.21.15.144:0/1097438857 wait complete. 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 10 monclient: build_initial_monmap 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.806+0000 7fef5555d640 10 monclient: monmap: 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:34.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 Processor -- start 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 -- start start 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient: init 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef500a1800 0x7fef5013ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): picked mon.noname-c con 0x7fef500a1800 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5012ec70 0x7fef5012d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): picked mon.noname-a con 0x7fef5012ec70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 0x7fef501184e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): picked mon.noname-b con 0x7fef5013f1d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5011a0c0 con 0x7fef5012ec70 2024-09-18T12:34:34.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef50098830 con 0x7fef5013f1d0 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5011a540 con 0x7fef500a1800 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef5555d640 10 monclient(hunting): authenticate will time out at 1310.376606s 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 0x7fef501184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4effd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef500a1800 msgr2=0x7fef5013ec90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef500a1800 0x7fef5013ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 0x7fef501184e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39582/0 (socket says 172.21.15.144:39582) 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 1 -- 172.21.15.144:0/3482409633 learned_addr learned my addr 172.21.15.144:0/3482409633 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): get_auth_request con 0x7fef5013f1d0 auth_method 0 2024-09-18T12:34:34.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4e7fc640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5012ec70 0x7fef5012d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4e7fc640 10 monclient(hunting): get_auth_request con 0x7fef5012ec70 auth_method 0 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 10 monclient(hunting): handle_auth_done global_id 124201 payload 824 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 1 -- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef500a1800 msgr2=0x7fef5013ec90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.807+0000 7fef4f7fe640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef500a1800 0x7fef5013ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4f7fe640 1 -- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5012ec70 msgr2=0x7fef5012d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4f7fe640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5012ec70 0x7fef5012d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4f7fe640 1 -- 172.21.15.144:0/3482409633 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48009a20 con 0x7fef5013f1d0 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.808+0000 7fef4e7fc640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5012ec70 0x7fef5012d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef4f7fe640 1 -- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 msgr2=0x7fef501184e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef4f7fe640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 0x7fef501184e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef480099f0 tx=0x7fef48037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48031cd0 con 0x7fef5013f1d0 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:34.822 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: dump: 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef48040070 con 0x7fef5013f1d0 2024-09-18T12:34:34.823 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef5555d640 5 monclient: authenticate success, global_id 124201 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef5555d640 10 monclient: _renew_subs 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef5555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5013faf0 con 0x7fef5013f1d0 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef48045680 con 0x7fef5013f1d0 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef36ffd640 10 monclient: dump: 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:34.824 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef4d7fa640 4 set_mon_vals no callback set 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef5555d640 10 monclient: _renew_subs 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef5555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.810+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5011ac00 con 0x7fef5013f1d0 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fef48003020 con 0x7fef5013f1d0 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef36ffd640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef3c038490 0x7fef3c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:34.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(258..258 src has 13..258) v4 ==== 8332+0+0 (secure 0 0 0) 0x7fef4806e140 con 0x7fef5013f1d0 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef4effd640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef3c038490 0x7fef3c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef4effd640 10 monclient: get_auth_request con 0x7fef3c038490 auth_method 0 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef5555d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef5555d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef5555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.811+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5011bef0 con 0x7fef5013f1d0 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.813+0000 7fef4effd640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef3c038490 0x7fef3c03a940 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fef400047d0 tx=0x7fef40007b80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:34.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.815+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fef480443d0 con 0x7fef5013f1d0 2024-09-18T12:34:34.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.815+0000 7fef36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:34.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.815+0000 7fef36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:34.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.975+0000 7fef5555d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T12:34:34.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.975+0000 7fef5555d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T12:34:34.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.975+0000 7fef5555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:34.976 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:34.975+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7fef5011bef0 con 0x7fef5013f1d0 2024-09-18T12:34:35.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.624+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v259) v1 ==== 121+0+0 (secure 0 0 0) 0x7fef48043380 con 0x7fef5013f1d0 2024-09-18T12:34:35.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.624+0000 7fef36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T12:34:35.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.624+0000 7fef36ffd640 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-18T12:34:35.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.723+0000 7fef5555d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T12:34:35.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.723+0000 7fef5555d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T12:34:35.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.723+0000 7fef5555d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:35.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.723+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7fef5011bef0 con 0x7fef5013f1d0 2024-09-18T12:34:35.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.724+0000 7fef36ffd640 1 -- 172.21.15.144:0/3482409633 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v259) v1 ==== 120+0+0 (secure 0 0 0) 0x7fef48043560 con 0x7fef5013f1d0 2024-09-18T12:34:35.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.724+0000 7fef36ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T12:34:35.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.724+0000 7fef36ffd640 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-18T12:34:35.726 INFO:tasks.workunit.client.0.smithi144.stderr:device 'h32' does not appear in the crush map 2024-09-18T12:34:35.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef3c038490 msgr2=0x7fef3c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:35.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef3c038490 0x7fef3c03a940 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fef400047d0 tx=0x7fef40007b80 comp rx=0 tx=0).stop 2024-09-18T12:34:35.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 10 monclient: shutdown 2024-09-18T12:34:35.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:35.728 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 msgr2=0x7fef501184e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:35.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 0x7fef501184e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef480099f0 tx=0x7fef48037000 comp rx=0 tx=0).stop 2024-09-18T12:34:35.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 shutdown_connections 2024-09-18T12:34:35.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fef3c038490 0x7fef3c03a940 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fef5013f1d0 0x7fef501184e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5012ec70 0x7fef5012d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 --2- 172.21.15.144:0/3482409633 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fef500a1800 0x7fef5013ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.729 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 >> 172.21.15.144:0/3482409633 conn(0x7fef50071100 msgr2=0x7fef5009b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:35.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 shutdown_connections 2024-09-18T12:34:35.730 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.726+0000 7fef5555d640 1 -- 172.21.15.144:0/3482409633 wait complete. 2024-09-18T12:34:35.736 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush rm r1 2024-09-18T12:34:35.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.807+0000 7f1c4b39d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:35.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.807+0000 7f1c4b39d640 10 monclient: build_initial_monmap 2024-09-18T12:34:35.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.807+0000 7f1c4b39d640 10 monclient: monmap: 2024-09-18T12:34:35.810 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:35.810 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:35.810 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:35.808693+0000 2024-09-18T12:34:35.810 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:35.808693+0000 2024-09-18T12:34:35.810 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:35.810 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:35.811 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:35.811 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:35.811 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:35.811 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:35.811 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:35.811 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:35.811 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 1 Processor -- start 2024-09-18T12:34:35.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 1 -- start start 2024-09-18T12:34:35.812 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 10 monclient: init 2024-09-18T12:34:35.824 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:35.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:34:35.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c44069a30 0x7f1c4412f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:35.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c44069a30 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:35.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c4409baa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:35.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c4409bea0 0x7f1c44071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:35.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4409bea0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:35.825 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.808+0000 7f1c4b39d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c44069e00 con 0x7f1c4409baa0 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4b39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4b39d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c4409d100 con 0x7f1c44069a30 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4b39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4b39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4409e270 con 0x7f1c4409bea0 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4b39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4b39d640 10 monclient(hunting): authenticate will time out at 1311.378021s 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4993a640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c4409bea0 msgr2=0x7f1c44071030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:35.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c4993a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c4409bea0 0x7f1c44071030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c44069a30 0x7f1c4412f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c48938640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1c4409baa0 0x7f1c44070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:42144/0 (socket says 172.21.15.144:42144) 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c48938640 1 -- 172.21.15.144:0/2206337413 learned_addr learned my addr 172.21.15.144:0/2206337413 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): get_auth_request con 0x7f1c44069a30 auth_method 0 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:35.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:35.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:35.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient(hunting): handle_auth_done global_id 104712 payload 274 2024-09-18T12:34:35.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 1 -- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c4409bea0 msgr2=0x7f1c44071030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:35.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 1 --2- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c4409bea0 0x7f1c44071030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 1 -- 172.21.15.144:0/2206337413 >> v1:172.21.15.116:6789/0 conn(0x7f1c4409baa0 legacy=0x7f1c44070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:35.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient: _finish_hunting 0 2024-09-18T12:34:35.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 1 monclient: found mon.noname-a 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.809+0000 7f1c49139640 1 -- 172.21.15.144:0/2206337413 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4409dc40 con 0x7f1c44069a30 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c49139640 1 -- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c44069a30 msgr2=0x7f1c4412f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c49139640 1 --2- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c44069a30 0x7f1c4412f840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009850 tx=0x7f1c400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e37fbf2c22f35e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 1 -- 172.21.15.144:0/2206337413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40032440 con 0x7f1c44069a30 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:35.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: dump: 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:35.830 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: _finish_auth 0 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c4b39d640 5 monclient: authenticate success, global_id 104712 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c4b39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:35.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 1 -- 172.21.15.144:0/2206337413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c400325a0 con 0x7f1c44069a30 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 1 -- 172.21.15.144:0/2206337413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40032940 con 0x7f1c44069a30 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c4b39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c4b39d640 4 set_mon_vals no callback set 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.812+0000 7f1c3bfff640 10 monclient: dump: 2024-09-18T12:34:35.832 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:35.833 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:35.833 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.833 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.833 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:35.833 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:35.833 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:35.833 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:35.834 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:35.834 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:35.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 10 monclient: shutdown 2024-09-18T12:34:35.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:35.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 -- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c44069a30 msgr2=0x7f1c4412f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:35.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c44069a30 0x7f1c4412f840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009850 tx=0x7f1c400090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 -- 172.21.15.144:0/2206337413 shutdown_connections 2024-09-18T12:34:35.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c4409bea0 0x7f1c44071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/2206337413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c44069a30 0x7f1c4412f840 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 -- 172.21.15.144:0/2206337413 >> 172.21.15.144:0/2206337413 conn(0x7f1c4409e9f0 msgr2=0x7f1c4409edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:35.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 -- 172.21.15.144:0/2206337413 shutdown_connections 2024-09-18T12:34:35.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.813+0000 7f1c4b39d640 1 -- 172.21.15.144:0/2206337413 wait complete. 2024-09-18T12:34:35.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient: build_initial_monmap 2024-09-18T12:34:35.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient: monmap: 2024-09-18T12:34:35.836 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:35.836 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:35.836 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:35.836 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:35.836 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:35.836 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:35.836 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 Processor -- start 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 -- start start 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient: init 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:35.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c44069a30 0x7f1c4413eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): picked mon.noname-c con 0x7f1c44069a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 0x7f1c4412ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4409baa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c4409bea0 0x7f1c44118560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c4409bea0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:35.838 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4411bb60 con 0x7f1c4409bea0 2024-09-18T12:34:35.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:35.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4409dea0 con 0x7f1c4409baa0 2024-09-18T12:34:35.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:35.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4411a540 con 0x7f1c44069a30 2024-09-18T12:34:35.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:35.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.814+0000 7f1c4b39d640 10 monclient(hunting): authenticate will time out at 1311.383968s 2024-09-18T12:34:35.839 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c49139640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c44069a30 msgr2=0x7f1c4413eee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c49139640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c44069a30 0x7f1c4413eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 0x7f1c4412ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 0x7f1c4412ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39590/0 (socket says 172.21.15.144:39590) 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 -- 172.21.15.144:0/1856243924 learned_addr learned my addr 172.21.15.144:0/1856243924 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): get_auth_request con 0x7f1c4409baa0 auth_method 0 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:35.840 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:35.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c4409bea0 0x7f1c44118560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:35.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 10 monclient(hunting): get_auth_request con 0x7f1c4409bea0 auth_method 0 2024-09-18T12:34:35.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:35.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:35.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:35.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:35.841 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c4993a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient(hunting): handle_auth_done global_id 124204 payload 824 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 -- 172.21.15.144:0/1856243924 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c44069a30 msgr2=0x7f1c4413eee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c44069a30 0x7f1c4413eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 -- 172.21.15.144:0/1856243924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c4409bea0 msgr2=0x7f1c44118560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c4409bea0 0x7f1c44118560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient: _finish_hunting 0 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 monclient: found mon.noname-b 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.815+0000 7f1c48938640 1 -- 172.21.15.144:0/1856243924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c40009390 con 0x7f1c4409baa0 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c48938640 1 -- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 msgr2=0x7f1c4412ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:35.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c48938640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 0x7f1c4412ea40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400c5e0 tx=0x7f1c34004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c34005cb0 con 0x7f1c4409baa0 2024-09-18T12:34:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: dump: 2024-09-18T12:34:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:35.844 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.845 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.845 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:35.845 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:35.845 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:35.845 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:35.845 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:35.845 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: _finish_auth 0 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c34005e30 con 0x7f1c4409baa0 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c4b39d640 5 monclient: authenticate success, global_id 124204 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c4b39d640 10 monclient: _renew_subs 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c4b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4412fa00 con 0x7f1c4409baa0 2024-09-18T12:34:35.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c34007340 con 0x7f1c4409baa0 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c397fa640 10 monclient: dump: 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.847 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:35.848 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:35.848 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:35.848 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:35.848 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:35.848 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:35.848 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:35.848 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c3b7fe640 4 set_mon_vals no callback set 2024-09-18T12:34:35.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c4b39d640 10 monclient: _renew_subs 2024-09-18T12:34:35.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c4b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:35.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.817+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4411a030 con 0x7f1c4409baa0 2024-09-18T12:34:35.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.818+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1c3401b400 con 0x7f1c4409baa0 2024-09-18T12:34:35.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.818+0000 7f1c4b39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:35.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.818+0000 7f1c4b39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:35.849 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.818+0000 7f1c4b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.818+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c440008d0 con 0x7f1c4409baa0 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.818+0000 7f1c397fa640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c28038470 0x7f1c2803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.819+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(259..259 src has 13..259) v4 ==== 8119+0+0 (secure 0 0 0) 0x7f1c3404c000 con 0x7f1c4409baa0 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.819+0000 7f1c49139640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c28038470 0x7f1c2803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.819+0000 7f1c49139640 10 monclient: get_auth_request con 0x7f1c28038470 auth_method 0 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.821+0000 7f1c49139640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c28038470 0x7f1c2803a920 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1c4409d070 tx=0x7f1c40032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.823+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1c3401a540 con 0x7f1c4409baa0 2024-09-18T12:34:35.850 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.823+0000 7f1c397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:35.851 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.823+0000 7f1c397fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.981+0000 7f1c4b39d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T12:34:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.981+0000 7f1c4b39d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T12:34:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.981+0000 7f1c4b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:35.982 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:35.981+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f1c4409c2e0 con 0x7f1c4409baa0 2024-09-18T12:34:36.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.623+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v260) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1c3401a360 con 0x7f1c4409baa0 2024-09-18T12:34:36.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.623+0000 7f1c397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T12:34:36.624 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.623+0000 7f1c397fa640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-18T12:34:36.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.722+0000 7f1c4b39d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T12:34:36.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.722+0000 7f1c4b39d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T12:34:36.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.722+0000 7f1c4b39d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:36.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.722+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f1c4411ac60 con 0x7f1c4409baa0 2024-09-18T12:34:36.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.723+0000 7f1c397fa640 1 -- 172.21.15.144:0/1856243924 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v260) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1c34029090 con 0x7f1c4409baa0 2024-09-18T12:34:36.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.723+0000 7f1c397fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T12:34:36.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.723+0000 7f1c397fa640 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-18T12:34:36.724 INFO:tasks.workunit.client.0.smithi144.stderr:device 'r1' does not appear in the crush map 2024-09-18T12:34:36.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c28038470 msgr2=0x7f1c2803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:36.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c28038470 0x7f1c2803a920 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1c4409d070 tx=0x7f1c40032000 comp rx=0 tx=0).stop 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 10 monclient: shutdown 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 msgr2=0x7f1c4412ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 0x7f1c4412ea40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400c5e0 tx=0x7f1c34004960 comp rx=0 tx=0).stop 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 shutdown_connections 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c28038470 0x7f1c2803a920 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c4409bea0 0x7f1c44118560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c4409baa0 0x7f1c4412ea40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 --2- 172.21.15.144:0/1856243924 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c44069a30 0x7f1c4413eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 >> 172.21.15.144:0/1856243924 conn(0x7f1c4409e9f0 msgr2=0x7f1c4409aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 shutdown_connections 2024-09-18T12:34:36.727 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.725+0000 7f1c4b39d640 1 -- 172.21.15.144:0/1856243924 wait complete. 2024-09-18T12:34:36.734 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T12:34:36.808 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.807+0000 7f248c79d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.807+0000 7f248c79d640 10 monclient: build_initial_monmap 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.807+0000 7f248c79d640 10 monclient: monmap: 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:36.807997+0000 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:36.807997+0000 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:36.809 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:36.810 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:36.810 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:36.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.807+0000 7f248c79d640 1 Processor -- start 2024-09-18T12:34:36.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.807+0000 7f248c79d640 1 -- start start 2024-09-18T12:34:36.810 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.807+0000 7f248c79d640 10 monclient: init 2024-09-18T12:34:36.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:36.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T12:34:36.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f248409ec50 0x7f2484131c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:36.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient(hunting): picked mon.noname-c con 0x7f248409ec50 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:36.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24841321a0 0x7f2484068d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:36.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient(hunting): picked mon.noname-b con 0x7f24841321a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:36.814 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f248409f020 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2484067420 con 0x7f248409f020 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24840a0480 con 0x7f248409ec50 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24840a0200 con 0x7f24841321a0 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248c79d640 10 monclient(hunting): authenticate will time out at 1312.377510s 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f248409ec50 0x7f2484131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248ad3a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f248409f020 0x7f2484069240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:46940/0 (socket says 172.21.15.144:46940) 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): get_auth_request con 0x7f248409ec50 auth_method 0 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248ad3a640 1 -- 172.21.15.144:0/2020924672 learned_addr learned my addr 172.21.15.144:0/2020924672 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:36.815 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f2489d38640 1 -- 172.21.15.144:0/2020924672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24841321a0 msgr2=0x7f2484068d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f2489d38640 1 --2- 172.21.15.144:0/2020924672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24841321a0 0x7f2484068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient(hunting): handle_auth_done global_id 124207 payload 274 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 1 -- 172.21.15.144:0/2020924672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24841321a0 msgr2=0x7f2484068d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 1 --2- 172.21.15.144:0/2020924672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24841321a0 0x7f2484068d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 1 -- 172.21.15.144:0/2020924672 >> v1:172.21.15.144:6789/0 conn(0x7f248409f020 legacy=0x7f2484069240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient: _finish_hunting 0 2024-09-18T12:34:36.816 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 1 monclient: found mon.noname-c 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.808+0000 7f248a539640 1 -- 172.21.15.144:0/2020924672 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248409fd80 con 0x7f248409ec50 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.811+0000 7f248a539640 1 -- 172.21.15.144:0/2020924672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f248409ec50 msgr2=0x7f2484131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.811+0000 7f248a539640 1 --2- 172.21.15.144:0/2020924672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f248409ec50 0x7f2484131c60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2474009e60 tx=0x7f2474009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f28f3781e6d15c85 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 1 -- 172.21.15.144:0/2020924672 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2474007840 con 0x7f248409ec50 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: dump: 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.817 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: _finish_auth 0 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 5 monclient: authenticate success, global_id 124207 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 1 -- 172.21.15.144:0/2020924672 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247402d050 con 0x7f248409ec50 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:36.818 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 4 set_mon_vals no callback set 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 1 -- 172.21.15.144:0/2020924672 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2474007bd0 con 0x7f248409ec50 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f2489537640 10 monclient: dump: 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:36.819 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 10 monclient: shutdown 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 -- 172.21.15.144:0/2020924672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f248409ec50 msgr2=0x7f2484131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 --2- 172.21.15.144:0/2020924672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f248409ec50 0x7f2484131c60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2474009e60 tx=0x7f2474009780 comp rx=0 tx=0).stop 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 -- 172.21.15.144:0/2020924672 shutdown_connections 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 --2- 172.21.15.144:0/2020924672 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f24841321a0 0x7f2484068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 --2- 172.21.15.144:0/2020924672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f248409ec50 0x7f2484131c60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 -- 172.21.15.144:0/2020924672 >> 172.21.15.144:0/2020924672 conn(0x7f24840a5cf0 msgr2=0x7f24840a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 -- 172.21.15.144:0/2020924672 shutdown_connections 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.812+0000 7f248c79d640 1 -- 172.21.15.144:0/2020924672 wait complete. 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.813+0000 7f248c79d640 10 monclient: build_initial_monmap 2024-09-18T12:34:36.820 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.813+0000 7f248c79d640 10 monclient: monmap: 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.813+0000 7f248c79d640 1 Processor -- start 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.813+0000 7f248c79d640 1 -- start start 2024-09-18T12:34:36.821 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.813+0000 7f248c79d640 10 monclient: init 2024-09-18T12:34:36.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:36.826 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:34:36.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f248409ec50 0x7f2484130f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:36.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): picked mon.noname-a con 0x7f248409ec50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:36.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248409f020 0x7f2484141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:36.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): picked mon.noname-c con 0x7f248409f020 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:36.827 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 0x7f248411ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:36.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): picked mon.noname-b con 0x7f24841321a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:36.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:36.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248411f630 con 0x7f248409ec50 2024-09-18T12:34:36.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:36.828 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248411ce70 con 0x7f24841321a0 2024-09-18T12:34:36.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:36.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248409c020 con 0x7f248409f020 2024-09-18T12:34:36.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:36.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248c79d640 10 monclient(hunting): authenticate will time out at 1312.383091s 2024-09-18T12:34:36.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 0x7f248411ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:36.829 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 0x7f248411ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39606/0 (socket says 172.21.15.144:39606) 2024-09-18T12:34:36.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 -- 172.21.15.144:0/253929486 learned_addr learned my addr 172.21.15.144:0/253929486 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:36.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): get_auth_request con 0x7f24841321a0 auth_method 0 2024-09-18T12:34:36.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:36.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:36.830 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:36.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f2489d38640 1 -- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248409f020 msgr2=0x7f2484141290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:36.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f2489d38640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248409f020 0x7f2484141290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:36.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:36.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:36.831 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:36.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient(hunting): handle_auth_done global_id 124210 payload 824 2024-09-18T12:34:36.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 -- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248409f020 msgr2=0x7f2484141290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:36.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248409f020 0x7f2484141290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 -- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f248409ec50 msgr2=0x7f2484130f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:36.832 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f248409ec50 0x7f2484130f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient: _finish_hunting 0 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 monclient: found mon.noname-b 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.814+0000 7f248ad3a640 1 -- 172.21.15.144:0/253929486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2474009a20 con 0x7f24841321a0 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248ad3a640 1 -- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 msgr2=0x7f248411ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248ad3a640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 0x7f248411ba00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c510 tx=0x7f247c004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c005cb0 con 0x7f24841321a0 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248ad3a640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 0x7f248411ba00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c510 tx=0x7f247c004900 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f247c007320 mon_map magic: 0 v1 2024-09-18T12:34:36.833 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 10 monclient: dump: 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248c79d640 5 monclient: authenticate success, global_id 124210 2024-09-18T12:34:36.834 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c005e30 con 0x7f24841321a0 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f24827fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248c79d640 10 monclient: _renew_subs 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248c79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248411f070 con 0x7f24841321a0 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f2488d36640 4 set_mon_vals no callback set 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248c79d640 10 monclient: _renew_subs 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248c79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.816+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248411e110 con 0x7f24841321a0 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c007320 con 0x7f24841321a0 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f24827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f24827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f24827fc640 10 monclient: dump: 2024-09-18T12:34:36.835 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f247c01b3b0 con 0x7f24841321a0 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f248c79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f248c79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:36.836 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f248c79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:36.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248411f7e0 con 0x7f24841321a0 2024-09-18T12:34:36.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f24827fc640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2470038440 0x7f247003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:36.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f248a539640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2470038440 0x7f247003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:36.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.817+0000 7f248a539640 10 monclient: get_auth_request con 0x7f2470038440 auth_method 0 2024-09-18T12:34:36.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.820+0000 7f248a539640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2470038440 0x7f247003a8f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f248409bf50 tx=0x7f2474037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:36.837 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:36.822+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(260..260 src has 192..260) v4 ==== 7909+0+0 (secure 0 0 0) 0x7f247c04ba60 con 0x7f24841321a0 2024-09-18T12:34:37.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.439+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f247c018650 con 0x7f24841321a0 2024-09-18T12:34:37.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.439+0000 7f24827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:37.440 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.439+0000 7f24827fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:37.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.604+0000 7f248c79d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:34:37.605 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.604+0000 7f248c79d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:34:37.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.604+0000 7f248c79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:37.606 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.604+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f24840008d0 con 0x7f24841321a0 2024-09-18T12:34:37.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.625+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v261) v1 ==== 78+0+0 (secure 0 0 0) 0x7f247c017030 con 0x7f24841321a0 2024-09-18T12:34:37.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.625+0000 7f24827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:34:37.627 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.625+0000 7f24827fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:37.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.725+0000 7f248c79d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:34:37.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.725+0000 7f248c79d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:34:37.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.725+0000 7f248c79d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:37.726 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:37.725+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f24840a4df0 con 0x7f24841321a0 2024-09-18T12:34:38.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.676+0000 7f24827fc640 1 -- 172.21.15.144:0/253929486 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v262) v1 ==== 78+0+0 (secure 0 0 0) 0x7f247c017030 con 0x7f24841321a0 2024-09-18T12:34:38.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.676+0000 7f24827fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T12:34:38.678 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.676+0000 7f24827fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:34:38.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.679+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2470038440 msgr2=0x7f247003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:38.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.679+0000 7f248c79d640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2470038440 0x7f247003a8f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f248409bf50 tx=0x7f2474037000 comp rx=0 tx=0).stop 2024-09-18T12:34:38.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.679+0000 7f248c79d640 10 monclient: shutdown 2024-09-18T12:34:38.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.679+0000 7f248c79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:38.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.679+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 msgr2=0x7f248411ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:38.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.679+0000 7f248c79d640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 0x7f248411ba00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f247c00c510 tx=0x7f247c004900 comp rx=0 tx=0).stop 2024-09-18T12:34:38.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.679+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 shutdown_connections 2024-09-18T12:34:38.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.680+0000 7f248c79d640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2470038440 0x7f247003a8f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.680+0000 7f248c79d640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f24841321a0 0x7f248411ba00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.680+0000 7f248c79d640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f248409f020 0x7f2484141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.680+0000 7f248c79d640 1 --2- 172.21.15.144:0/253929486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f248409ec50 0x7f2484130f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.680+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 >> 172.21.15.144:0/253929486 conn(0x7f24840a5cf0 msgr2=0x7f248409d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:38.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.680+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 shutdown_connections 2024-09-18T12:34:38.682 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.680+0000 7f248c79d640 1 -- 172.21.15.144:0/253929486 wait complete. 2024-09-18T12:34:38.692 INFO:tasks.workunit.client.0.smithi144.stdout:OK 2024-09-18T12:34:38.693 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo OK 2024-09-18T12:34:38.693 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:34:38.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:34:38.721 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-18T12:34:38.722 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-18T12:34:38.722 DEBUG:teuthology.orchestra.run.smithi144:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-18T12:34:38.782 INFO:tasks.workunit.client.0.smithi144.stderr:+ set -e 2024-09-18T12:34:38.782 INFO:tasks.workunit.client.0.smithi144.stderr:++ uuidgen 2024-09-18T12:34:38.808 INFO:tasks.workunit.client.0.smithi144.stderr:+ ua=4b751cce-d39e-4f85-9cff-56f6325f3ab2 2024-09-18T12:34:38.809 INFO:tasks.workunit.client.0.smithi144.stderr:++ uuidgen 2024-09-18T12:34:38.809 INFO:tasks.workunit.client.0.smithi144.stderr:+ ub=2826fa46-0d84-40c9-bf29-d833af7ba458 2024-09-18T12:34:38.809 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 4b751cce-d39e-4f85-9cff-56f6325f3ab2 2024-09-18T12:34:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.882+0000 7f194639d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.882+0000 7f194639d640 10 monclient: build_initial_monmap 2024-09-18T12:34:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.882+0000 7f194639d640 10 monclient: monmap: 2024-09-18T12:34:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:38.883 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:38.882978+0000 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:38.882978+0000 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:38.884 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:38.888 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.882+0000 7f194639d640 1 Processor -- start 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.882+0000 7f194639d640 1 -- start start 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.882+0000 7f194639d640 10 monclient: init 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1940131b40 0x7f194009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient(hunting): picked mon.noname-b con 0x7f1940131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f194009f240 0x7f1940131f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient(hunting): picked mon.noname-c con 0x7f194009f240 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1940132450 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194006a230 con 0x7f1940132450 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:38.889 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1940132890 con 0x7f194009f240 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19400a0300 con 0x7f1940131b40 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194639d640 10 monclient(hunting): authenticate will time out at 1314.452452s 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193ffff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1940131b40 msgr2=0x7f194009ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1940131b40 0x7f194009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f194009f240 0x7f1940131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194493a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1940132450 0x7f1940068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57462/0 (socket says 172.21.15.144:57462) 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f194493a640 1 -- 172.21.15.144:0/3915470047 learned_addr learned my addr 172.21.15.144:0/3915470047 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f194009f240 0x7f1940131f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42098/0 (socket says 172.21.15.144:42098) 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): get_auth_request con 0x7f194009f240 auth_method 0 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:38.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient(hunting): handle_auth_done global_id 124213 payload 274 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 1 -- 172.21.15.144:0/3915470047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1940131b40 msgr2=0x7f194009ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 1 --2- 172.21.15.144:0/3915470047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1940131b40 0x7f194009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 1 -- 172.21.15.144:0/3915470047 >> v1:172.21.15.144:6789/0 conn(0x7f1940132450 legacy=0x7f1940068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:38.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.883+0000 7f193f7fe640 1 -- 172.21.15.144:0/3915470047 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400a00c0 con 0x7f194009f240 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193f7fe640 1 -- 172.21.15.144:0/3915470047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f194009f240 msgr2=0x7f1940131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193f7fe640 1 --2- 172.21.15.144:0/3915470047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f194009f240 0x7f1940131f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1934009850 tx=0x7f19340090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75e014c7bd93e7b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 1 -- 172.21.15.144:0/3915470047 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934032440 con 0x7f194009f240 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: dump: 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.892 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: _finish_auth 0 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 1 -- 172.21.15.144:0/3915470047 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19340325a0 con 0x7f194009f240 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f194639d640 5 monclient: authenticate success, global_id 124213 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f194639d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f194639d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 1 -- 172.21.15.144:0/3915470047 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934032980 con 0x7f194009f240 2024-09-18T12:34:38.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f194639d640 4 set_mon_vals no callback set 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.886+0000 7f193effd640 10 monclient: dump: 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:38.894 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 10 monclient: shutdown 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 -- 172.21.15.144:0/3915470047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f194009f240 msgr2=0x7f1940131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 --2- 172.21.15.144:0/3915470047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f194009f240 0x7f1940131f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1934009850 tx=0x7f19340090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 -- 172.21.15.144:0/3915470047 shutdown_connections 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 --2- 172.21.15.144:0/3915470047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f194009f240 0x7f1940131f10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 --2- 172.21.15.144:0/3915470047 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1940131b40 0x7f194009ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 -- 172.21.15.144:0/3915470047 >> 172.21.15.144:0/3915470047 conn(0x7f19400a5100 msgr2=0x7f19400a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 -- 172.21.15.144:0/3915470047 shutdown_connections 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 1 -- 172.21.15.144:0/3915470047 wait complete. 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 10 monclient: build_initial_monmap 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.887+0000 7f194639d640 10 monclient: monmap: 2024-09-18T12:34:38.895 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:38.896 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 Processor -- start 2024-09-18T12:34:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 -- start start 2024-09-18T12:34:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient: init 2024-09-18T12:34:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f194009f240 0x7f1940141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:38.900 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): picked mon.noname-c con 0x7f194009f240 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1940131b40 0x7f194012ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): picked mon.noname-a con 0x7f1940131b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 0x7f1940098d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): picked mon.noname-b con 0x7f1940132450 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194009a950 con 0x7f1940131b40 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19400a0a30 con 0x7f1940132450 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1940131620 con 0x7f194009f240 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194639d640 10 monclient(hunting): authenticate will time out at 1314.457842s 2024-09-18T12:34:38.901 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f193ffff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f194009f240 msgr2=0x7f1940141180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f193ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f194009f240 0x7f1940141180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 0x7f1940098d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 0x7f1940098d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42106/0 (socket says 172.21.15.144:42106) 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 1 -- 172.21.15.144:0/3246962349 learned_addr learned my addr 172.21.15.144:0/3246962349 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 10 monclient(hunting): get_auth_request con 0x7f1940132450 auth_method 0 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.888+0000 7f194493a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 10 monclient(hunting): handle_auth_done global_id 124216 payload 824 2024-09-18T12:34:38.902 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f193f7fe640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1940131b40 0x7f194012ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 1 -- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f194009f240 msgr2=0x7f1940141180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f194009f240 0x7f1940141180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 1 -- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1940131b40 msgr2=0x7f194012ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1940131b40 0x7f194012ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 10 monclient: _finish_hunting 0 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 1 monclient: found mon.noname-b 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f194493a640 1 -- 172.21.15.144:0/3246962349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1934009390 con 0x7f1940132450 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f193f7fe640 10 monclient: get_auth_request con 0x7f1940131b40 auth_method 0 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.889+0000 7f193f7fe640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1940131b40 0x7f194012ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194493a640 1 -- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 msgr2=0x7f1940098d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:38.903 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194493a640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 0x7f1940098d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f193800ccf0 tx=0x7f1938005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800aa60 con 0x7f1940132450 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: dump: 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:38.904 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: _finish_auth 0 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193800abe0 con 0x7f1940132450 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194639d640 5 monclient: authenticate success, global_id 124216 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380070f0 con 0x7f1940132450 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f1927fff640 10 monclient: dump: 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:38.905 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194639d640 10 monclient: _renew_subs 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194639d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194011be30 con 0x7f1940132450 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f193e7fc640 4 set_mon_vals no callback set 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194639d640 10 monclient: _renew_subs 2024-09-18T12:34:38.906 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194639d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.891+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19401416c0 con 0x7f1940132450 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f19380074a0 con 0x7f1940132450 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f194639d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f1927fff640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f192c038440 0x7f192c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f194639d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f194639d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19400636e0 con 0x7f1940132450 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(262..262 src has 192..262) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f193804c640 con 0x7f1940132450 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f193ffff640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f192c038440 0x7f192c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.892+0000 7f193ffff640 10 monclient: get_auth_request con 0x7f192c038440 auth_method 0 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.895+0000 7f193ffff640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f192c038440 0x7f192c03a8f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f19300038f0 tx=0x7f1930006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.897+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1938007750 con 0x7f1940132450 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.897+0000 7f1927fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:38.907 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:38.897+0000 7f1927fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:39.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.062+0000 7f194639d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:39.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.062+0000 7f194639d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:39.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.062+0000 7f194639d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:39.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.062+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"} v 0) v1 -- 0x7f194011c230 con 0x7f1940132450 2024-09-18T12:34:39.880 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.879+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}]=0 v263) v1 ==== 106+0+3 (secure 0 0 0) 0x7f19380290b0 con 0x7f1940132450 2024-09-18T12:34:39.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.879+0000 7f1927fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:39.881 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.879+0000 7f1927fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:39.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.982+0000 7f194639d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:39.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.982+0000 7f194639d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:39.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.982+0000 7f194639d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:39.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.982+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"} v 0) v1 -- 0x7f1940072640 con 0x7f1940132450 2024-09-18T12:34:39.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.984+0000 7f1927fff640 1 -- 172.21.15.144:0/3246962349 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}]=0 v263) v1 ==== 106+0+3 (secure 0 0 0) 0x7f1938010af0 con 0x7f1940132450 2024-09-18T12:34:39.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.984+0000 7f1927fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:39.985 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.984+0000 7f1927fff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:34:39.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f192c038440 msgr2=0x7f192c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:39.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f192c038440 0x7f192c03a8f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f19300038f0 tx=0x7f1930006d80 comp rx=0 tx=0).stop 2024-09-18T12:34:39.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 10 monclient: shutdown 2024-09-18T12:34:39.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:39.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 msgr2=0x7f1940098d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:39.987 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 0x7f1940098d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f193800ccf0 tx=0x7f1938005370 comp rx=0 tx=0).stop 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 shutdown_connections 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f192c038440 0x7f192c03a8f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1940132450 0x7f1940098d60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1940131b40 0x7f194012ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 --2- 172.21.15.144:0/3246962349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f194009f240 0x7f1940141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 >> 172.21.15.144:0/3246962349 conn(0x7f19400a5100 msgr2=0x7f19400a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 shutdown_connections 2024-09-18T12:34:39.988 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:39.986+0000 7f194639d640 1 -- 172.21.15.144:0/3246962349 wait complete. 2024-09-18T12:34:39.996 INFO:tasks.workunit.client.0.smithi144.stderr:+ na=12 2024-09-18T12:34:39.996 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 4b751cce-d39e-4f85-9cff-56f6325f3ab2 2024-09-18T12:34:40.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.067+0000 7f49b039d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:40.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.067+0000 7f49b039d640 10 monclient: build_initial_monmap 2024-09-18T12:34:40.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.067+0000 7f49b039d640 10 monclient: monmap: 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:40.068472+0000 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:40.068472+0000 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:40.069 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 1 Processor -- start 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 1 -- start start 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient: init 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49a809c230 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49a809c600 0x7f49a809ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient(hunting): picked mon.noname-a con 0x7f49a809c600 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49a8134860 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a809b1d0 con 0x7f49a8134860 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a809f5b0 con 0x7f49a809c230 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a809b470 con 0x7f49a809c600 2024-09-18T12:34:40.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49b039d640 10 monclient(hunting): authenticate will time out at 1315.637842s 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49ae139640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f49a809c230 0x7f49a809cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57468/0 (socket says 172.21.15.144:57468) 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.068+0000 7f49ae139640 1 -- 172.21.15.144:0/3849974433 learned_addr learned my addr 172.21.15.144:0/3849974433 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 1 --2- 172.21.15.144:0/3849974433 >> v2:172.21.15.116:3300/0 conn(0x7f49a809c600 0x7f49a809ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): get_auth_request con 0x7f49a809c600 auth_method 0 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient(hunting): handle_auth_done global_id 104718 payload 274 2024-09-18T12:34:40.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 1 -- 172.21.15.144:0/3849974433 >> v1:172.21.15.144:6789/0 conn(0x7f49a809c230 legacy=0x7f49a809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:40.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 1 -- 172.21.15.144:0/3849974433 >> v1:172.21.15.116:6789/0 conn(0x7f49a8134860 legacy=0x7f49a8136c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:40.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient: _finish_hunting 0 2024-09-18T12:34:40.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 1 monclient: found mon.noname-a 2024-09-18T12:34:40.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:40.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:40.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.069+0000 7f49ad938640 1 -- 172.21.15.144:0/3849974433 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a809d340 con 0x7f49a809c600 2024-09-18T12:34:40.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.072+0000 7f49ad938640 1 -- 172.21.15.144:0/3849974433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a809c600 msgr2=0x7f49a809ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:40.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.072+0000 7f49ad938640 1 --2- 172.21.15.144:0/3849974433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a809c600 0x7f49a809ea70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4998009850 tx=0x7f49980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92fb1670f945ba1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:40.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 1 -- 172.21.15.144:0/3849974433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998032440 con 0x7f49a809c600 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: dump: 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.081 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: _finish_auth 0 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 1 -- 172.21.15.144:0/3849974433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49980325a0 con 0x7f49a809c600 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 1 -- 172.21.15.144:0/3849974433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998032940 con 0x7f49a809c600 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 5 monclient: authenticate success, global_id 104718 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 4 set_mon_vals no callback set 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49ad137640 10 monclient: dump: 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.082 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 10 monclient: shutdown 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 1 -- 172.21.15.144:0/3849974433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a809c600 msgr2=0x7f49a809ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:40.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 1 --2- 172.21.15.144:0/3849974433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a809c600 0x7f49a809ea70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4998009850 tx=0x7f49980090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 1 -- 172.21.15.144:0/3849974433 shutdown_connections 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 1 --2- 172.21.15.144:0/3849974433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a809c600 0x7f49a809ea70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 1 -- 172.21.15.144:0/3849974433 >> 172.21.15.144:0/3849974433 conn(0x7f49a8071350 msgr2=0x7f49a80a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 1 -- 172.21.15.144:0/3849974433 shutdown_connections 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.073+0000 7f49b039d640 1 -- 172.21.15.144:0/3849974433 wait complete. 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient: build_initial_monmap 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient: monmap: 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:40.084 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 Processor -- start 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 -- start start 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient: init 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 0x7f49a81416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): picked mon.noname-b con 0x7f49a809c230 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a809c600 0x7f49a8141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): picked mon.noname-c con 0x7f49a809c600 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8134860 0x7f49a8142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): picked mon.noname-a con 0x7f49a8134860 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a806ec10 con 0x7f49a8134860 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a8137fb0 con 0x7f49a809c230 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a8134fb0 con 0x7f49a809c600 2024-09-18T12:34:40.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49b039d640 10 monclient(hunting): authenticate will time out at 1315.643910s 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49ae139640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 0x7f49a81416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.074+0000 7f49ad938640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a809c600 msgr2=0x7f49a8141be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ad938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a809c600 0x7f49a8141be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 0x7f49a81416a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42108/0 (socket says 172.21.15.144:42108) 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 -- 172.21.15.144:0/3436764351 learned_addr learned my addr 172.21.15.144:0/3436764351 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): get_auth_request con 0x7f49a809c230 auth_method 0 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:40.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae93a640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8134860 0x7f49a8142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae93a640 10 monclient(hunting): get_auth_request con 0x7f49a8134860 auth_method 0 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae93a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae93a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae93a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient(hunting): handle_auth_done global_id 124219 payload 824 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 -- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a809c600 msgr2=0x7f49a8141be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a809c600 0x7f49a8141be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 -- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8134860 msgr2=0x7f49a8142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8134860 0x7f49a8142120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient: _finish_hunting 0 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 monclient: found mon.noname-b 2024-09-18T12:34:40.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae139640 1 -- 172.21.15.144:0/3436764351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4998009390 con 0x7f49a809c230 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.075+0000 7f49ae93a640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8134860 0x7f49a8142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49ae139640 1 -- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 msgr2=0x7f49a81416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49ae139640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 0x7f49a81416a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49a400cca0 tx=0x7f49a40053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a400aa60 con 0x7f49a809c230 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: dump: 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.089 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49b039d640 5 monclient: authenticate success, global_id 124219 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49b039d640 10 monclient: _renew_subs 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49b039d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a8134d30 con 0x7f49a809c230 2024-09-18T12:34:40.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a400abe0 con 0x7f49a809c230 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40070f0 con 0x7f49a809c230 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f499e7fc640 10 monclient: dump: 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.091 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49b039d640 10 monclient: _renew_subs 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49b039d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49ac936640 4 set_mon_vals no callback set 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.077+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a8098ee0 con 0x7f49a809c230 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f49a4007560 con 0x7f49a809c230 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f49b039d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f49b039d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f49b039d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a8099130 con 0x7f49a809c230 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f499e7fc640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f498c038470 0x7f498c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(263..263 src has 192..263) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f49a404c550 con 0x7f49a809c230 2024-09-18T12:34:40.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f49ad938640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f498c038470 0x7f498c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.078+0000 7f49ad938640 10 monclient: get_auth_request con 0x7f498c038470 auth_method 0 2024-09-18T12:34:40.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.081+0000 7f49ad938640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f498c038470 0x7f498c03a920 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4998031000 tx=0x7f4998032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:40.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.083+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f49a401a360 con 0x7f49a809c230 2024-09-18T12:34:40.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.083+0000 7f499e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:40.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.083+0000 7f499e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:40.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.248+0000 7f49b039d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:40.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.248+0000 7f49b039d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:40.249 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.248+0000 7f49b039d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.248+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"} v 0) v1 -- 0x7f49a80636e0 con 0x7f49a809c230 2024-09-18T12:34:40.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.249+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}]=0 v263) v1 ==== 106+0+3 (secure 0 0 0) 0x7f49a4010af0 con 0x7f49a809c230 2024-09-18T12:34:40.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.249+0000 7f499e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:40.250 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.249+0000 7f499e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:40.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.354+0000 7f49b039d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:40.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.354+0000 7f49b039d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:40.355 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.354+0000 7f49b039d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.354+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"} v 0) v1 -- 0x7f49a809f120 con 0x7f49a809c230 2024-09-18T12:34:40.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.355+0000 7f499e7fc640 1 -- 172.21.15.144:0/3436764351 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}]=0 v263) v1 ==== 106+0+3 (secure 0 0 0) 0x7f49a401a360 con 0x7f49a809c230 2024-09-18T12:34:40.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.355+0000 7f499e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:40.356 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.355+0000 7f499e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f498c038470 msgr2=0x7f498c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f498c038470 0x7f498c03a920 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4998031000 tx=0x7f4998032000 comp rx=0 tx=0).stop 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 10 monclient: shutdown 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 msgr2=0x7f49a81416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 0x7f49a81416a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49a400cca0 tx=0x7f49a40053f0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 shutdown_connections 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f498c038470 0x7f498c03a920 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8134860 0x7f49a8142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f49a809c600 0x7f49a8141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.359 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 --2- 172.21.15.144:0/3436764351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f49a809c230 0x7f49a81416a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 >> 172.21.15.144:0/3436764351 conn(0x7f49a8071350 msgr2=0x7f49a80a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:40.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 shutdown_connections 2024-09-18T12:34:40.360 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.357+0000 7f49b039d640 1 -- 172.21.15.144:0/3436764351 wait complete. 2024-09-18T12:34:40.367 INFO:tasks.workunit.client.0.smithi144.stderr:+ test 12 -eq 12 2024-09-18T12:34:40.367 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 2826fa46-0d84-40c9-bf29-d833af7ba458 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.440+0000 7f7c64f5d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.440+0000 7f7c64f5d640 10 monclient: build_initial_monmap 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.440+0000 7f7c64f5d640 10 monclient: monmap: 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:40.441099+0000 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:40.441099+0000 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.441 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:40.442 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:40.442 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:40.442 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:40.442 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:40.442 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.440+0000 7f7c64f5d640 1 Processor -- start 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.440+0000 7f7c64f5d640 1 -- start start 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.440+0000 7f7c64f5d640 10 monclient: init 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c60131b40 0x7f7c6009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c60131b40 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c6009f240 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:40.444 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7c60136e70 0x7f7c60139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c60136e70 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c600a0300 con 0x7f7c6009f240 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c600a00c0 con 0x7f7c60136e70 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c6009bee0 con 0x7f7c60131b40 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c64f5d640 10 monclient(hunting): authenticate will time out at 1316.010430s 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c60131b40 0x7f7c6009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c60131b40 0x7f7c6009ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42118/0 (socket says 172.21.15.144:42118) 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 -- 172.21.15.144:0/949302585 learned_addr learned my addr 172.21.15.144:0/949302585 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): get_auth_request con 0x7f7c60131b40 auth_method 0 2024-09-18T12:34:40.445 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5ed9d640 1 --2- 172.21.15.144:0/949302585 >> v2:172.21.15.116:3300/0 conn(0x7f7c60136e70 0x7f7c60139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient(hunting): handle_auth_done global_id 124222 payload 274 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 -- 172.21.15.144:0/949302585 >> v2:172.21.15.116:3300/0 conn(0x7f7c60136e70 msgr2=0x7f7c60139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 --2- 172.21.15.144:0/949302585 >> v2:172.21.15.116:3300/0 conn(0x7f7c60136e70 0x7f7c60139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 -- 172.21.15.144:0/949302585 >> v1:172.21.15.116:6789/0 conn(0x7f7c6009f240 legacy=0x7f7c60131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 monclient: found mon.noname-c 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:40.446 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.447 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.441+0000 7f7c5e59c640 1 -- 172.21.15.144:0/949302585 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c600a14b0 con 0x7f7c60131b40 2024-09-18T12:34:40.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5e59c640 1 -- 172.21.15.144:0/949302585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60131b40 msgr2=0x7f7c6009ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:40.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5e59c640 1 --2- 172.21.15.144:0/949302585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60131b40 0x7f7c6009ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c6006a6b0 tx=0x7f7c500091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9192daa7ba598d6e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 1 -- 172.21.15.144:0/949302585 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50032440 con 0x7f7c60131b40 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: dump: 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.455 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 1 -- 172.21.15.144:0/949302585 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c500325a0 con 0x7f7c60131b40 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c64f5d640 5 monclient: authenticate success, global_id 124222 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c64f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c64f5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 1 -- 172.21.15.144:0/949302585 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50032980 con 0x7f7c60131b40 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c64f5d640 4 set_mon_vals no callback set 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.444+0000 7f7c5dd9b640 10 monclient: dump: 2024-09-18T12:34:40.456 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 10 monclient: shutdown 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 -- 172.21.15.144:0/949302585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60131b40 msgr2=0x7f7c6009ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:40.457 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/949302585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60131b40 0x7f7c6009ed00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c6006a6b0 tx=0x7f7c500091d0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 -- 172.21.15.144:0/949302585 shutdown_connections 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/949302585 >> v2:172.21.15.116:3300/0 conn(0x7f7c60136e70 0x7f7c60139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/949302585 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60131b40 0x7f7c6009ed00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 -- 172.21.15.144:0/949302585 >> 172.21.15.144:0/949302585 conn(0x7f7c600a5100 msgr2=0x7f7c600a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 -- 172.21.15.144:0/949302585 shutdown_connections 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 1 -- 172.21.15.144:0/949302585 wait complete. 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 10 monclient: build_initial_monmap 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.445+0000 7f7c64f5d640 10 monclient: monmap: 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:40.458 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 Processor -- start 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 -- start start 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient: init 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c6009f240 0x7f7c601413c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c6009f240 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:40.459 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c60131b40 0x7f7c6012ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c60131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 0x7f7c60097c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c60136e70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6009b930 con 0x7f7c6009f240 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6009deb0 con 0x7f7c60136e70 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6009b370 con 0x7f7c60131b40 2024-09-18T12:34:40.460 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c64f5d640 10 monclient(hunting): authenticate will time out at 1316.015748s 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 0x7f7c60097c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c57fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c60131b40 msgr2=0x7f7c6012ed20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 0x7f7c60097c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42134/0 (socket says 172.21.15.144:42134) 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 1 -- 172.21.15.144:0/2765624741 learned_addr learned my addr 172.21.15.144:0/2765624741 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 10 monclient(hunting): get_auth_request con 0x7f7c60136e70 auth_method 0 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:40.461 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c57fff640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c60131b40 0x7f7c6012ed20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5e59c640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c6009f240 0x7f7c601413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5e59c640 10 monclient(hunting): get_auth_request con 0x7f7c6009f240 auth_method 0 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.446+0000 7f7c5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 10 monclient(hunting): handle_auth_done global_id 124225 payload 824 2024-09-18T12:34:40.462 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 1 -- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c60131b40 msgr2=0x7f7c6012ed20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c60131b40 0x7f7c6012ed20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 1 -- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c6009f240 msgr2=0x7f7c601413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c6009f240 0x7f7c601413c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 1 monclient: found mon.noname-b 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.447+0000 7f7c5ed9d640 1 -- 172.21.15.144:0/2765624741 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c50009450 con 0x7f7c60136e70 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c5ed9d640 1 -- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 msgr2=0x7f7c60097c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c5ed9d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 0x7f7c60097c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800ceb0 tx=0x7f7c58007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5800e610 con 0x7f7c60136e70 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:40.463 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: dump: 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:40.464 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c64f5d640 5 monclient: authenticate success, global_id 124225 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5800e770 con 0x7f7c60136e70 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c64f5d640 10 monclient: _renew_subs 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5800eb10 con 0x7f7c60136e70 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c64f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6009b650 con 0x7f7c60136e70 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c567fc640 10 monclient: dump: 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:40.465 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c5d59a640 4 set_mon_vals no callback set 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c64f5d640 10 monclient: _renew_subs 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c64f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.449+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6009a620 con 0x7f7c60136e70 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7c58003020 con 0x7f7c60136e70 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c64f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c64f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:40.466 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c64f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c600636e0 con 0x7f7c60136e70 2024-09-18T12:34:40.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c567fc640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c40038440 0x7f7c4003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:40.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(263..263 src has 192..263) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7c58012070 con 0x7f7c60136e70 2024-09-18T12:34:40.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c5e59c640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c40038440 0x7f7c4003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:40.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.450+0000 7f7c5e59c640 10 monclient: get_auth_request con 0x7f7c40038440 auth_method 0 2024-09-18T12:34:40.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.453+0000 7f7c5e59c640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c40038440 0x7f7c4003a8f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7c50034000 tx=0x7f7c50032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:40.467 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.454+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7c58016430 con 0x7f7c60136e70 2024-09-18T12:34:40.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.454+0000 7f7c567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:40.468 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.454+0000 7f7c567fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:40.621 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.620+0000 7f7c64f5d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:40.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.620+0000 7f7c64f5d640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:40.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.620+0000 7f7c64f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:40.622 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:40.620+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"} v 0) v1 -- 0x7f7c6009f9f0 con 0x7f7c60136e70 2024-09-18T12:34:41.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.494+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}]=0 v264) v1 ==== 106+0+3 (secure 0 0 0) 0x7f7c58014330 con 0x7f7c60136e70 2024-09-18T12:34:41.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.494+0000 7f7c567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.495 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.494+0000 7f7c567fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:41.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.598+0000 7f7c64f5d640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.598+0000 7f7c64f5d640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.598+0000 7f7c64f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:41.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.598+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"} v 0) v1 -- 0x7f7c60072640 con 0x7f7c60136e70 2024-09-18T12:34:41.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.599+0000 7f7c567fc640 1 -- 172.21.15.144:0/2765624741 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}]=0 v264) v1 ==== 106+0+3 (secure 0 0 0) 0x7f7c58014510 con 0x7f7c60136e70 2024-09-18T12:34:41.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.599+0000 7f7c567fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.600 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.599+0000 7f7c567fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:34:41.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c40038440 msgr2=0x7f7c4003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:41.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c40038440 0x7f7c4003a8f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7c50034000 tx=0x7f7c50032000 comp rx=0 tx=0).stop 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 10 monclient: shutdown 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 msgr2=0x7f7c60097c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 0x7f7c60097c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800ceb0 tx=0x7f7c58007870 comp rx=0 tx=0).stop 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 shutdown_connections 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c40038440 0x7f7c4003a8f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c60136e70 0x7f7c60097c50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c60131b40 0x7f7c6012ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 --2- 172.21.15.144:0/2765624741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c6009f240 0x7f7c601413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 >> 172.21.15.144:0/2765624741 conn(0x7f7c600a5100 msgr2=0x7f7c60130a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 shutdown_connections 2024-09-18T12:34:41.603 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.601+0000 7f7c64f5d640 1 -- 172.21.15.144:0/2765624741 wait complete. 2024-09-18T12:34:41.611 INFO:tasks.workunit.client.0.smithi144.stderr:+ nb=14 2024-09-18T12:34:41.611 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 2826fa46-0d84-40c9-bf29-d833af7ba458 2024-09-18T12:34:41.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.683+0000 7f2f90c2b640 10 monclient: get_monmap_and_config 2024-09-18T12:34:41.684 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.683+0000 7f2f90c2b640 10 monclient: build_initial_monmap 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.683+0000 7f2f90c2b640 10 monclient: monmap: 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:41.684175+0000 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:41.684175+0000 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:41.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.683+0000 7f2f90c2b640 1 Processor -- start 2024-09-18T12:34:41.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.683+0000 7f2f90c2b640 1 -- start start 2024-09-18T12:34:41.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.683+0000 7f2f90c2b640 10 monclient: init 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c131b40 0x7f2f8c09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f8c09f240 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f8c136e70 0x7f2f8c139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c136e70 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c06a230 con 0x7f2f8c09f240 2024-09-18T12:34:41.688 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8c0a0300 con 0x7f2f8c136e70 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c0a00c0 con 0x7f2f8c131b40 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f90c2b640 10 monclient(hunting): authenticate will time out at 1317.253513s 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f8a59c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c131b40 msgr2=0x7f2f8c09ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f8a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c131b40 0x7f2f8c09ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f8ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f8c136e70 0x7f2f8c139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f89d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2f8c09f240 0x7f2f8c131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:56050/0 (socket says 172.21.15.144:56050) 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f89d9b640 1 -- 172.21.15.144:0/2710389528 learned_addr learned my addr 172.21.15.144:0/2710389528 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f8ad9d640 10 monclient(hunting): get_auth_request con 0x7f2f8c136e70 auth_method 0 2024-09-18T12:34:41.689 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.684+0000 7f2f8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_done global_id 104727 payload 274 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c131b40 msgr2=0x7f2f8c09ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 1 --2- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c131b40 0x7f2f8c09ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/2710389528 >> v1:172.21.15.116:6789/0 conn(0x7f2f8c09f240 legacy=0x7f2f8c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 1 monclient: found mon.noname-a 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:41.690 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:41.691 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.685+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/2710389528 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c09bee0 con 0x7f2f8c136e70 2024-09-18T12:34:41.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c136e70 msgr2=0x7f2f8c139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:41.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8ad9d640 1 --2- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c136e70 0x7f2f8c139250 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009e60 tx=0x7f2f84009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2749effe32f6f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:41.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 1 -- 172.21.15.144:0/2710389528 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84007840 con 0x7f2f8c136e70 2024-09-18T12:34:41.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:41.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: dump: 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: _finish_auth 0 2024-09-18T12:34:41.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 1 -- 172.21.15.144:0/2710389528 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8402d050 con 0x7f2f8c136e70 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 5 monclient: authenticate success, global_id 104727 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 4 set_mon_vals no callback set 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 1 -- 172.21.15.144:0/2710389528 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84007c10 con 0x7f2f8c136e70 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f8959a640 10 monclient: dump: 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:41.699 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 10 monclient: shutdown 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 1 -- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c136e70 msgr2=0x7f2f8c139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c136e70 0x7f2f8c139250 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009e60 tx=0x7f2f84009780 comp rx=0 tx=0).stop 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 1 -- 172.21.15.144:0/2710389528 shutdown_connections 2024-09-18T12:34:41.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c136e70 0x7f2f8c139250 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/2710389528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c131b40 0x7f2f8c09ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 1 -- 172.21.15.144:0/2710389528 >> 172.21.15.144:0/2710389528 conn(0x7f2f8c0a5100 msgr2=0x7f2f8c0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.688+0000 7f2f90c2b640 1 -- 172.21.15.144:0/2710389528 shutdown_connections 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.689+0000 7f2f90c2b640 1 -- 172.21.15.144:0/2710389528 wait complete. 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.689+0000 7f2f90c2b640 10 monclient: build_initial_monmap 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.689+0000 7f2f90c2b640 10 monclient: monmap: 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:41.701 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.689+0000 7f2f90c2b640 1 Processor -- start 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.689+0000 7f2f90c2b640 1 -- start start 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.689+0000 7f2f90c2b640 10 monclient: init 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c09f240 0x7f2f8c1412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c09f240 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c131b40 0x7f2f8c141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c131b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:41.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 0x7f2f8c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c136e70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c0a0a30 con 0x7f2f8c131b40 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c130a60 con 0x7f2f8c136e70 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c11be10 con 0x7f2f8c09f240 2024-09-18T12:34:41.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f90c2b640 10 monclient(hunting): authenticate will time out at 1317.259199s 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8a59c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c09f240 msgr2=0x7f2f8c1412e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c09f240 0x7f2f8c1412e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 0x7f2f8c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 0x7f2f8c141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42150/0 (socket says 172.21.15.144:42150) 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/3477114206 learned_addr learned my addr 172.21.15.144:0/3477114206 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): get_auth_request con 0x7f2f8c136e70 auth_method 0 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:41.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c131b40 0x7f2f8c141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 10 monclient(hunting): get_auth_request con 0x7f2f8c131b40 auth_method 0 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:41.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient(hunting): handle_auth_done global_id 124228 payload 824 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/3477114206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c09f240 msgr2=0x7f2f8c1412e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c09f240 0x7f2f8c1412e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/3477114206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c131b40 msgr2=0x7f2f8c141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c131b40 0x7f2f8c141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 monclient: found mon.noname-b 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.690+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/3477114206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f84009a20 con 0x7f2f8c136e70 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.692+0000 7f2f8ad9d640 1 -- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 msgr2=0x7f2f8c141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:41.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.692+0000 7f2f8ad9d640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 0x7f2f8c141d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f840099f0 tx=0x7f2f84037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84031cd0 con 0x7f2f8c136e70 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: dump: 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:41.707 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f84040070 con 0x7f2f8c136e70 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 5 monclient: authenticate success, global_id 124228 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 10 monclient: _renew_subs 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:41.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c11cb20 con 0x7f2f8c136e70 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84045640 con 0x7f2f8c136e70 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 10 monclient: dump: 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:41.709 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 10 monclient: _renew_subs 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f88d99640 4 set_mon_vals no callback set 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c09b980 con 0x7f2f8c136e70 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2f84003020 con 0x7f2f8c136e70 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.693+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c11bc40 con 0x7f2f8c136e70 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.694+0000 7f2f7a7fc640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f6c038470 0x7f2f6c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:41.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.694+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(264..264 src has 192..264) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2f8406def0 con 0x7f2f8c136e70 2024-09-18T12:34:41.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.694+0000 7f2f8a59c640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f6c038470 0x7f2f6c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:41.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.694+0000 7f2f8a59c640 10 monclient: get_auth_request con 0x7f2f6c038470 auth_method 0 2024-09-18T12:34:41.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.696+0000 7f2f8a59c640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f6c038470 0x7f2f6c03a920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0038f0 tx=0x7f2f7c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:41.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.698+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2f84031e30 con 0x7f2f8c136e70 2024-09-18T12:34:41.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.698+0000 7f2f7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:41.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.698+0000 7f2f7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:41.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.862+0000 7f2f90c2b640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.862+0000 7f2f90c2b640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.862+0000 7f2f90c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:41.863 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.862+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"} v 0) v1 -- 0x7f2f8c11ba20 con 0x7f2f8c136e70 2024-09-18T12:34:41.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.863+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}]=0 v264) v1 ==== 106+0+3 (secure 0 0 0) 0x7f2f840393c0 con 0x7f2f8c136e70 2024-09-18T12:34:41.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.863+0000 7f2f7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.864 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.863+0000 7f2f7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:41.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.964+0000 7f2f90c2b640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.964+0000 7f2f90c2b640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.964+0000 7f2f90c2b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:41.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.964+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"} v 0) v1 -- 0x7f2f8c11ba20 con 0x7f2f8c136e70 2024-09-18T12:34:41.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.965+0000 7f2f7a7fc640 1 -- 172.21.15.144:0/3477114206 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}]=0 v264) v1 ==== 106+0+3 (secure 0 0 0) 0x7f2f8403a3a0 con 0x7f2f8c136e70 2024-09-18T12:34:41.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.965+0000 7f2f7a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "2826fa46-0d84-40c9-bf29-d833af7ba458"}] 2024-09-18T12:34:41.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.965+0000 7f2f7a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:34:41.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f6c038470 msgr2=0x7f2f6c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:41.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f6c038470 0x7f2f6c03a920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0038f0 tx=0x7f2f7c006d80 comp rx=0 tx=0).stop 2024-09-18T12:34:41.968 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 10 monclient: shutdown 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 msgr2=0x7f2f8c141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 0x7f2f8c141d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f840099f0 tx=0x7f2f84037000 comp rx=0 tx=0).stop 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 shutdown_connections 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2f6c038470 0x7f2f6c03a920 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2f8c136e70 0x7f2f8c141d60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f8c131b40 0x7f2f8c141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 --2- 172.21.15.144:0/3477114206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2f8c09f240 0x7f2f8c1412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 >> 172.21.15.144:0/3477114206 conn(0x7f2f8c0a5100 msgr2=0x7f2f8c0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 shutdown_connections 2024-09-18T12:34:41.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:41.967+0000 7f2f90c2b640 1 -- 172.21.15.144:0/3477114206 wait complete. 2024-09-18T12:34:41.977 INFO:tasks.workunit.client.0.smithi144.stderr:+ test 14 -eq 14 2024-09-18T12:34:41.977 INFO:tasks.workunit.client.0.smithi144.stderr:+ test 14 -ne 12 2024-09-18T12:34:41.977 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd rm 12 2024-09-18T12:34:42.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.048+0000 7f00dab9d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.048+0000 7f00dab9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.048+0000 7f00dab9d640 10 monclient: monmap: 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:42.049244+0000 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:42.049244+0000 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:42.050 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:42.051 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:42.051 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:42.051 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:42.051 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 1 Processor -- start 2024-09-18T12:34:42.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 1 -- start start 2024-09-18T12:34:42.051 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient: init 2024-09-18T12:34:42.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00d4131b40 0x7f00d409ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient(hunting): picked mon.noname-a con 0x7f00d4131b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00d409f240 0x7f00d4131f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient(hunting): picked mon.noname-c con 0x7f00d409f240 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00d4132450 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d406a230 con 0x7f00d4132450 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d40a0300 con 0x7f00d4131b40 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d40a00c0 con 0x7f00d409f240 2024-09-18T12:34:42.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00dab9d640 10 monclient(hunting): authenticate will time out at 1317.618585s 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f00d409f240 0x7f00d4131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d913a640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f00d4132450 0x7f00d4068e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57478/0 (socket says 172.21.15.144:57478) 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d913a640 1 -- 172.21.15.144:0/3036623853 learned_addr learned my addr 172.21.15.144:0/3036623853 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): get_auth_request con 0x7f00d409f240 auth_method 0 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:42.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d8939640 1 --2- 172.21.15.144:0/3036623853 >> v2:172.21.15.116:3300/0 conn(0x7f00d4131b40 0x7f00d409ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d8939640 10 monclient(hunting): get_auth_request con 0x7f00d4131b40 auth_method 0 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d8939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d8939640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d8939640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient(hunting): handle_auth_done global_id 124231 payload 274 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 1 -- 172.21.15.144:0/3036623853 >> v2:172.21.15.116:3300/0 conn(0x7f00d4131b40 msgr2=0x7f00d409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 1 --2- 172.21.15.144:0/3036623853 >> v2:172.21.15.116:3300/0 conn(0x7f00d4131b40 0x7f00d409ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 1 -- 172.21.15.144:0/3036623853 >> v1:172.21.15.144:6789/0 conn(0x7f00d4132450 legacy=0x7f00d4068e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 1 monclient: found mon.noname-c 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.049+0000 7f00d3fff640 1 -- 172.21.15.144:0/3036623853 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d409bee0 con 0x7f00d409f240 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.052+0000 7f00d3fff640 1 -- 172.21.15.144:0/3036623853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d409f240 msgr2=0x7f00d4131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.052+0000 7f00d3fff640 1 --2- 172.21.15.144:0/3036623853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d409f240 0x7f00d4131f10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00d409f660 tx=0x7f00c00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15d868078dca40ba server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 1 -- 172.21.15.144:0/3036623853 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c002adf0 con 0x7f00d409f240 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: dump: 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.061 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: _finish_auth 0 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 1 -- 172.21.15.144:0/3036623853 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c0032440 con 0x7f00d409f240 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 5 monclient: authenticate success, global_id 124231 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 1 -- 172.21.15.144:0/3036623853 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c00327e0 con 0x7f00d409f240 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 4 set_mon_vals no callback set 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00d37fe640 10 monclient: dump: 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 10 monclient: shutdown 2024-09-18T12:34:42.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 -- 172.21.15.144:0/3036623853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d409f240 msgr2=0x7f00d4131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3036623853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d409f240 0x7f00d4131f10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f00d409f660 tx=0x7f00c00090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 -- 172.21.15.144:0/3036623853 shutdown_connections 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3036623853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d409f240 0x7f00d4131f10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3036623853 >> v2:172.21.15.116:3300/0 conn(0x7f00d4131b40 0x7f00d409ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 -- 172.21.15.144:0/3036623853 >> 172.21.15.144:0/3036623853 conn(0x7f00d40a5100 msgr2=0x7f00d40a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 -- 172.21.15.144:0/3036623853 shutdown_connections 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.053+0000 7f00dab9d640 1 -- 172.21.15.144:0/3036623853 wait complete. 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.054+0000 7f00dab9d640 10 monclient: build_initial_monmap 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.054+0000 7f00dab9d640 10 monclient: monmap: 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:42.064 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.054+0000 7f00dab9d640 1 Processor -- start 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.054+0000 7f00dab9d640 1 -- start start 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.054+0000 7f00dab9d640 10 monclient: init 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00d409f240 0x7f00d4141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): picked mon.noname-a con 0x7f00d409f240 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00d4131b40 0x7f00d412ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): picked mon.noname-c con 0x7f00d4131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 0x7f00d4098c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): picked mon.noname-b con 0x7f00d4132450 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d4142040 con 0x7f00d409f240 2024-09-18T12:34:42.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40a0a30 con 0x7f00d4132450 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d4131650 con 0x7f00d4131b40 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00dab9d640 10 monclient(hunting): authenticate will time out at 1317.624105s 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 0x7f00d4098c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d3fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00d4131b40 msgr2=0x7f00d412ee10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00d4131b40 0x7f00d412ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 0x7f00d4098c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42162/0 (socket says 172.21.15.144:42162) 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 -- 172.21.15.144:0/3359463837 learned_addr learned my addr 172.21.15.144:0/3359463837 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): get_auth_request con 0x7f00d4132450 auth_method 0 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:42.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient(hunting): handle_auth_done global_id 124234 payload 824 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d8939640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00d409f240 0x7f00d4141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 -- 172.21.15.144:0/3359463837 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00d4131b40 msgr2=0x7f00d412ee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00d4131b40 0x7f00d412ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 -- 172.21.15.144:0/3359463837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00d409f240 msgr2=0x7f00d4141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00d409f240 0x7f00d4141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient: _finish_hunting 0 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 monclient: found mon.noname-b 2024-09-18T12:34:42.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:42.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.055+0000 7f00d913a640 1 -- 172.21.15.144:0/3359463837 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c0009390 con 0x7f00d4132450 2024-09-18T12:34:42.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d913a640 1 -- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 msgr2=0x7f00d4098c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:42.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d913a640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 0x7f00d4098c60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00ccc0 tx=0x7f00cc005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:42.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc00ce80 con 0x7f00d4132450 2024-09-18T12:34:42.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:42.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: dump: 2024-09-18T12:34:42.071 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:42.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc00aa60 con 0x7f00d4132450 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00dab9d640 5 monclient: authenticate success, global_id 124234 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00dab9d640 10 monclient: _renew_subs 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00dab9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d411c210 con 0x7f00d4132450 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc00ae20 con 0x7f00d4132450 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d0ff9640 10 monclient: dump: 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.073 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00dab9d640 10 monclient: _renew_subs 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00d2ffd640 4 set_mon_vals no callback set 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00dab9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.057+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d409b290 con 0x7f00d4132450 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.058+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f00cc007300 con 0x7f00d4132450 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.058+0000 7f00dab9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:42.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.058+0000 7f00dab9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.058+0000 7f00dab9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.058+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d411c3a0 con 0x7f00d4132450 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.059+0000 7f00d0ff9640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00b8038440 0x7f00b803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.059+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(264..264 src has 192..264) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f00cc04c580 con 0x7f00d4132450 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.059+0000 7f00d8939640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00b8038440 0x7f00b803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.059+0000 7f00d8939640 10 monclient: get_auth_request con 0x7f00b8038440 auth_method 0 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.061+0000 7f00d8939640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00b8038440 0x7f00b803a8f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f00c8006fd0 tx=0x7f00c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.063+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f00cc01a590 con 0x7f00d4132450 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.063+0000 7f00d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:42.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.063+0000 7f00d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.224+0000 7f00dab9d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.224+0000 7f00dab9d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.224+0000 7f00dab9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.226 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.224+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f00d409a5e0 con 0x7f00d4132450 2024-09-18T12:34:42.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:42.225+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e264 definitely_dead 0 2024-09-18T12:34:42.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.496+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 removed osd.12 v265) v1 ==== 83+0+0 (secure 0 0 0) 0x7f00cc010af0 con 0x7f00d4132450 2024-09-18T12:34:42.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.496+0000 7f00d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.497 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.496+0000 7f00d0ff9640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-18T12:34:42.597 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.596+0000 7f00dab9d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.596+0000 7f00dab9d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.596+0000 7f00dab9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.598 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.596+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f00d40636e0 con 0x7f00d4132450 2024-09-18T12:34:42.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:42.597+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e265 definitely_dead 0 2024-09-18T12:34:42.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.597+0000 7f00d0ff9640 1 -- 172.21.15.144:0/3359463837 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v265) v1 ==== 92+0+0 (secure 0 0 0) 0x7f00cc010cd0 con 0x7f00d4132450 2024-09-18T12:34:42.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.597+0000 7f00d0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.599 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.597+0000 7f00d0ff9640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-18T12:34:42.599 INFO:tasks.workunit.client.0.smithi144.stderr:osd.12 does not exist. 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00b8038440 msgr2=0x7f00b803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00b8038440 0x7f00b803a8f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f00c8006fd0 tx=0x7f00c8008040 comp rx=0 tx=0).stop 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 10 monclient: shutdown 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 msgr2=0x7f00d4098c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 0x7f00d4098c60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00ccc0 tx=0x7f00cc005370 comp rx=0 tx=0).stop 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 shutdown_connections 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f00b8038440 0x7f00b803a8f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f00d4132450 0x7f00d4098c60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.601 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00d4131b40 0x7f00d412ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 --2- 172.21.15.144:0/3359463837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00d409f240 0x7f00d4141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.599+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 >> 172.21.15.144:0/3359463837 conn(0x7f00d40a5100 msgr2=0x7f00d40a3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:42.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.600+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 shutdown_connections 2024-09-18T12:34:42.602 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.600+0000 7f00dab9d640 1 -- 172.21.15.144:0/3359463837 wait complete. 2024-09-18T12:34:42.609 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd rm 12 2024-09-18T12:34:42.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.680+0000 7f7d53f5d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:42.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.680+0000 7f7d53f5d640 10 monclient: build_initial_monmap 2024-09-18T12:34:42.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.680+0000 7f7d53f5d640 10 monclient: monmap: 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:42.681543+0000 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:42.681543+0000 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:42.682 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 1 Processor -- start 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 1 -- start start 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient: init 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d4c069a30 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d4c09bb40 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d4c09bf30 0x7f7d4c138320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c09bf30 addr v2:172.21.15.116:3300/0 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c09e6b0 con 0x7f7d4c09bb40 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c069e00 con 0x7f7d4c069a30 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.685 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c09cf00 con 0x7f7d4c09bf30 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d53f5d640 10 monclient(hunting): authenticate will time out at 1318.250890s 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d51cf9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7d4c069a30 0x7f7d4c12fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57486/0 (socket says 172.21.15.144:57486) 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.681+0000 7f7d51cf9640 1 -- 172.21.15.144:0/336337430 learned_addr learned my addr 172.21.15.144:0/336337430 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 1 --2- 172.21.15.144:0/336337430 >> v2:172.21.15.116:3300/0 conn(0x7f7d4c09bf30 0x7f7d4c138320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): get_auth_request con 0x7f7d4c09bf30 auth_method 0 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient(hunting): handle_auth_done global_id 104736 payload 274 2024-09-18T12:34:42.686 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 1 -- 172.21.15.144:0/336337430 >> v1:172.21.15.144:6789/0 conn(0x7f7d4c069a30 legacy=0x7f7d4c12fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 1 -- 172.21.15.144:0/336337430 >> v1:172.21.15.116:6789/0 conn(0x7f7d4c09bb40 legacy=0x7f7d4c134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient: _finish_hunting 0 2024-09-18T12:34:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 1 monclient: found mon.noname-a 2024-09-18T12:34:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:34:42.687 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.682+0000 7f7d524fa640 1 -- 172.21.15.144:0/336337430 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c09c390 con 0x7f7d4c09bf30 2024-09-18T12:34:42.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d524fa640 1 -- 172.21.15.144:0/336337430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bf30 msgr2=0x7f7d4c138320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:42.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d524fa640 1 --2- 172.21.15.144:0/336337430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bf30 0x7f7d4c138320 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009e60 tx=0x7f7d44009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972387d6decaafb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:42.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 1 -- 172.21.15.144:0/336337430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44007840 con 0x7f7d4c09bf30 2024-09-18T12:34:42.694 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: dump: 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.695 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: _finish_auth 0 2024-09-18T12:34:42.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:42.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d53f5d640 5 monclient: authenticate success, global_id 104736 2024-09-18T12:34:42.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 1 -- 172.21.15.144:0/336337430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4402d050 con 0x7f7d4c09bf30 2024-09-18T12:34:42.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d53f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:42.696 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d53f5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d53f5d640 4 set_mon_vals no callback set 2024-09-18T12:34:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 1 -- 172.21.15.144:0/336337430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44007bd0 con 0x7f7d4c09bf30 2024-09-18T12:34:42.697 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:34:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.685+0000 7f7d50cf7640 10 monclient: dump: 2024-09-18T12:34:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.698 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.699 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.699 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.699 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.699 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.699 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.699 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 10 monclient: shutdown 2024-09-18T12:34:42.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:42.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 1 -- 172.21.15.144:0/336337430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bf30 msgr2=0x7f7d4c138320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/336337430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bf30 0x7f7d4c138320 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009e60 tx=0x7f7d44009780 comp rx=0 tx=0).stop 2024-09-18T12:34:42.700 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 1 -- 172.21.15.144:0/336337430 shutdown_connections 2024-09-18T12:34:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/336337430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bf30 0x7f7d4c138320 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 1 -- 172.21.15.144:0/336337430 >> 172.21.15.144:0/336337430 conn(0x7f7d4c09ea70 msgr2=0x7f7d4c09ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 1 -- 172.21.15.144:0/336337430 shutdown_connections 2024-09-18T12:34:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 1 -- 172.21.15.144:0/336337430 wait complete. 2024-09-18T12:34:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 10 monclient: build_initial_monmap 2024-09-18T12:34:42.701 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.686+0000 7f7d53f5d640 10 monclient: monmap: 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:42.702 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:42.704 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 Processor -- start 2024-09-18T12:34:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 -- start start 2024-09-18T12:34:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient: init 2024-09-18T12:34:42.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:42.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:42.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 0x7f7d4c13ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d4c069a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:42.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bb40 0x7f7d4c12e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c09bb40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:42.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d4c09bf30 0x7f7d4c1185e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d4c09bf30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:42.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c13f250 con 0x7f7d4c09bb40 2024-09-18T12:34:42.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c09b3e0 con 0x7f7d4c069a30 2024-09-18T12:34:42.706 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:42.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c11c030 con 0x7f7d4c09bf30 2024-09-18T12:34:42.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:42.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d53f5d640 10 monclient(hunting): authenticate will time out at 1318.256534s 2024-09-18T12:34:42.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 0x7f7d4c13ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 0x7f7d4c13ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42174/0 (socket says 172.21.15.144:42174) 2024-09-18T12:34:42.707 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 -- 172.21.15.144:0/871436222 learned_addr learned my addr 172.21.15.144:0/871436222 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:42.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): get_auth_request con 0x7f7d4c069a30 auth_method 0 2024-09-18T12:34:42.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:42.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:42.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:42.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d524fa640 1 -- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d4c09bf30 msgr2=0x7f7d4c1185e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:42.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d524fa640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d4c09bf30 0x7f7d4c1185e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:42.708 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:42.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:42.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:42.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d514f8640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bb40 0x7f7d4c12e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d514f8640 10 monclient(hunting): get_auth_request con 0x7f7d4c09bb40 auth_method 0 2024-09-18T12:34:42.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d514f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:42.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d514f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:42.709 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d514f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:42.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient(hunting): handle_auth_done global_id 124237 payload 824 2024-09-18T12:34:42.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 -- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d4c09bf30 msgr2=0x7f7d4c1185e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:42.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d4c09bf30 0x7f7d4c1185e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 -- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bb40 msgr2=0x7f7d4c12e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bb40 0x7f7d4c12e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient: _finish_hunting 0 2024-09-18T12:34:42.710 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 monclient: found mon.noname-b 2024-09-18T12:34:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.687+0000 7f7d51cf9640 1 -- 172.21.15.144:0/871436222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d44009a20 con 0x7f7d4c069a30 2024-09-18T12:34:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d51cf9640 1 -- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 msgr2=0x7f7d4c13ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:42.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d51cf9640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 0x7f7d4c13ed10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800b540 tx=0x7f7d480090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48011840 con 0x7f7d4c069a30 2024-09-18T12:34:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: dump: 2024-09-18T12:34:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.712 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.713 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.714 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:42.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d53f5d640 5 monclient: authenticate success, global_id 124237 2024-09-18T12:34:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d480119a0 con 0x7f7d4c069a30 2024-09-18T12:34:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d53f5d640 10 monclient: _renew_subs 2024-09-18T12:34:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d53f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c11bd50 con 0x7f7d4c069a30 2024-09-18T12:34:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:42.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48011d60 con 0x7f7d4c069a30 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d41ffb640 10 monclient: dump: 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:42.716 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d43fff640 4 set_mon_vals no callback set 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d53f5d640 10 monclient: _renew_subs 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d53f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.690+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c11ab80 con 0x7f7d4c069a30 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7d48010590 con 0x7f7d4c069a30 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d41ffb640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d30038440 0x7f7d3003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d53f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d53f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d53f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(265..265 src has 192..265) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7d4804c420 con 0x7f7d4c069a30 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c119ed0 con 0x7f7d4c069a30 2024-09-18T12:34:42.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d514f8640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d30038440 0x7f7d3003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:42.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.691+0000 7f7d514f8640 10 monclient: get_auth_request con 0x7f7d30038440 auth_method 0 2024-09-18T12:34:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.694+0000 7f7d514f8640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d30038440 0x7f7d3003a8f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0037d0 tx=0x7f7d3c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.694+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:34:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.696+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7d48015030 con 0x7f7d4c069a30 2024-09-18T12:34:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.696+0000 7f7d41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:42.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.696+0000 7f7d41ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:42.860 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.859+0000 7f7d53f5d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.859+0000 7f7d53f5d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.859+0000 7f7d53f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.861 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.859+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f7d4c0636e0 con 0x7f7d4c069a30 2024-09-18T12:34:42.861 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:42.861+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e265 definitely_dead 0 2024-09-18T12:34:42.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.861+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v265) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7d48029080 con 0x7f7d4c069a30 2024-09-18T12:34:42.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.861+0000 7f7d41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.862 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.861+0000 7f7d41ffb640 10 monclient: _finish_command 2 = system:0 osd.12 does not exist. 2024-09-18T12:34:42.964 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.963+0000 7f7d53f5d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.963+0000 7f7d53f5d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.963+0000 7f7d53f5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:42.965 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.963+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) v1 -- 0x7f7d4c072640 con 0x7f7d4c069a30 2024-09-18T12:34:42.965 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:42.964+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e265 definitely_dead 0 2024-09-18T12:34:42.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.964+0000 7f7d41ffb640 1 -- 172.21.15.144:0/871436222 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v265) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7d48015020 con 0x7f7d4c069a30 2024-09-18T12:34:42.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.964+0000 7f7d41ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-18T12:34:42.966 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.964+0000 7f7d41ffb640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-18T12:34:42.966 INFO:tasks.workunit.client.0.smithi144.stderr:osd.12 does not exist. 2024-09-18T12:34:42.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d30038440 msgr2=0x7f7d3003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.969 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d30038440 0x7f7d3003a8f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0037d0 tx=0x7f7d3c009040 comp rx=0 tx=0).stop 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 10 monclient: shutdown 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 msgr2=0x7f7d4c13ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 0x7f7d4c13ed10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800b540 tx=0x7f7d480090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 shutdown_connections 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7d30038440 0x7f7d3003a8f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7d4c09bf30 0x7f7d4c1185e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d4c09bb40 0x7f7d4c12e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 --2- 172.21.15.144:0/871436222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7d4c069a30 0x7f7d4c13ed10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:42.970 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 >> 172.21.15.144:0/871436222 conn(0x7f7d4c09ea70 msgr2=0x7f7d4c132950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:42.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.966+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 shutdown_connections 2024-09-18T12:34:42.971 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:42.967+0000 7f7d53f5d640 1 -- 172.21.15.144:0/871436222 wait complete. 2024-09-18T12:34:42.976 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd rm 14 2024-09-18T12:34:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.047+0000 7f20e0b5d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:43.048 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.047+0000 7f20e0b5d640 10 monclient: build_initial_monmap 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.047+0000 7f20e0b5d640 10 monclient: monmap: 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:43.048034+0000 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:43.048034+0000 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:43.049 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.047+0000 7f20e0b5d640 1 Processor -- start 2024-09-18T12:34:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.047+0000 7f20e0b5d640 1 -- start start 2024-09-18T12:34:43.050 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.047+0000 7f20e0b5d640 10 monclient: init 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20dc131b40 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131f10 0x7f20dc132300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f20dc131f10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f20dc136e70 0x7f20dc139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f20dc136e70 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20dc132840 con 0x7f20dc131b40 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20dc0a0300 con 0x7f20dc136e70 2024-09-18T12:34:43.052 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0a00c0 con 0x7f20dc131f10 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20e0b5d640 10 monclient(hunting): authenticate will time out at 1318.617448s 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20d9d9b640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131f10 msgr2=0x7f20dc132300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20d9d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131f10 0x7f20dc132300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f20dc136e70 0x7f20dc139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f20dc136e70 0x7f20dc139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42184/0 (socket says 172.21.15.144:42184) 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 -- 172.21.15.144:0/869014602 learned_addr learned my addr 172.21.15.144:0/869014602 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): get_auth_request con 0x7f20dc136e70 auth_method 0 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:43.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient(hunting): handle_auth_done global_id 124240 payload 274 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 -- 172.21.15.144:0/869014602 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131f10 msgr2=0x7f20dc132300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 --2- 172.21.15.144:0/869014602 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131f10 0x7f20dc132300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 -- 172.21.15.144:0/869014602 >> v1:172.21.15.116:6789/0 conn(0x7f20dc131b40 legacy=0x7f20dc09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 monclient: found mon.noname-c 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:43.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.048+0000 7f20dad9d640 1 -- 172.21.15.144:0/869014602 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc09bee0 con 0x7f20dc136e70 2024-09-18T12:34:43.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20dad9d640 1 -- 172.21.15.144:0/869014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 msgr2=0x7f20dc139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:43.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20dad9d640 1 --2- 172.21.15.144:0/869014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc139250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009f20 tx=0x7f20d4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3b9cef4a8a0496e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:43.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 1 -- 172.21.15.144:0/869014602 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4007610 con 0x7f20dc136e70 2024-09-18T12:34:43.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:43.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:43.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: dump: 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: _finish_auth 0 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:43.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20e0b5d640 5 monclient: authenticate success, global_id 124240 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20e0b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 1 -- 172.21.15.144:0/869014602 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d402d050 con 0x7f20dc136e70 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20e0b5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20e0b5d640 4 set_mon_vals no callback set 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 1 -- 172.21.15.144:0/869014602 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d40079e0 con 0x7f20dc136e70 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.051+0000 7f20d959a640 10 monclient: dump: 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:43.062 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 10 monclient: shutdown 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 -- 172.21.15.144:0/869014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 msgr2=0x7f20dc139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/869014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc139250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009f20 tx=0x7f20d4009840 comp rx=0 tx=0).stop 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 -- 172.21.15.144:0/869014602 shutdown_connections 2024-09-18T12:34:43.063 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/869014602 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc139250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/869014602 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131f10 0x7f20dc132300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 -- 172.21.15.144:0/869014602 >> 172.21.15.144:0/869014602 conn(0x7f20dc0a5100 msgr2=0x7f20dc0a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 -- 172.21.15.144:0/869014602 shutdown_connections 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 1 -- 172.21.15.144:0/869014602 wait complete. 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 10 monclient: build_initial_monmap 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.052+0000 7f20e0b5d640 10 monclient: monmap: 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:43.064 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 Processor -- start 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 -- start start 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient: init 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131b40 0x7f20dc1418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f20dc131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc131f10 0x7f20dc0987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f20dc131f10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:43.065 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc098cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f20dc136e70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0a0a30 con 0x7f20dc131f10 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc130cf0 con 0x7f20dc136e70 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc09a850 con 0x7f20dc131b40 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20e0b5d640 10 monclient(hunting): authenticate will time out at 1318.622761s 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc098cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc098cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42190/0 (socket says 172.21.15.144:42190) 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20da59c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131b40 msgr2=0x7f20dc1418d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:43.066 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 1 -- 172.21.15.144:0/1675778873 learned_addr learned my addr 172.21.15.144:0/1675778873 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 10 monclient(hunting): get_auth_request con 0x7f20dc136e70 auth_method 0 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20da59c640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131b40 0x7f20dc1418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.053+0000 7f20d9d9b640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc131f10 0x7f20dc0987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20d9d9b640 10 monclient(hunting): get_auth_request con 0x7f20dc131f10 auth_method 0 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:43.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 10 monclient(hunting): handle_auth_done global_id 124243 payload 824 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 1 -- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131b40 msgr2=0x7f20dc1418d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131b40 0x7f20dc1418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 1 -- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc131f10 msgr2=0x7f20dc0987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc131f10 0x7f20dc0987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 1 monclient: found mon.noname-b 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20dad9d640 1 -- 172.21.15.144:0/1675778873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d4009ae0 con 0x7f20dc136e70 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.054+0000 7f20d9d9b640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc131f10 0x7f20dc0987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20dad9d640 1 -- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 msgr2=0x7f20dc098cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:43.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20dad9d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc098cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009f20 tx=0x7f20d4036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20dad9d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc098cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009f20 tx=0x7f20d4036000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f20d4031cd0 mon_map magic: 0 v1 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4031cd0 con 0x7f20dc136e70 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 10 monclient: dump: 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:43.069 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20e0b5d640 5 monclient: authenticate success, global_id 124243 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d4031e30 con 0x7f20dc136e70 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20e0b5d640 10 monclient: _renew_subs 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20e0b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc11bfe0 con 0x7f20dc136e70 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20d8d99640 4 set_mon_vals no callback set 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20e0b5d640 10 monclient: _renew_subs 2024-09-18T12:34:43.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20e0b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.056+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc09a340 con 0x7f20dc136e70 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d403c630 con 0x7f20dc136e70 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20ca7fc640 10 monclient: dump: 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:43.071 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20e0b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20e0b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20e0b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f20d403c9e0 con 0x7f20dc136e70 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc0636e0 con 0x7f20dc136e70 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20ca7fc640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f20bc0384c0 0x7f20bc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20da59c640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f20bc0384c0 0x7f20bc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:43.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.057+0000 7f20da59c640 10 monclient: get_auth_request con 0x7f20bc0384c0 auth_method 0 2024-09-18T12:34:43.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.060+0000 7f20da59c640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f20bc0384c0 0x7f20bc03a970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f20cc007fd0 tx=0x7f20cc007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:43.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.061+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(265..265 src has 192..265) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f20d406d5a0 con 0x7f20dc136e70 2024-09-18T12:34:43.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.680+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f20d4044720 con 0x7f20dc136e70 2024-09-18T12:34:43.680 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.680+0000 7f20ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:43.681 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.680+0000 7f20ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.843+0000 7f20e0b5d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-18T12:34:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.843+0000 7f20e0b5d640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-18T12:34:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.843+0000 7f20e0b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:43.844 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:43.843+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7f20dc064a70 con 0x7f20dc136e70 2024-09-18T12:34:43.845 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:43.844+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e265 definitely_dead 0 2024-09-18T12:34:44.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.511+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 removed osd.14 v266) v1 ==== 83+0+0 (secure 0 0 0) 0x7f20d404a080 con 0x7f20dc136e70 2024-09-18T12:34:44.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.511+0000 7f20ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-18T12:34:44.512 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.511+0000 7f20ca7fc640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-18T12:34:44.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.614+0000 7f20e0b5d640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-18T12:34:44.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.614+0000 7f20e0b5d640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-18T12:34:44.615 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.614+0000 7f20e0b5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.616 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.614+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) v1 -- 0x7f20dc0999f0 con 0x7f20dc136e70 2024-09-18T12:34:44.616 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:44.615+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e266 definitely_dead 0 2024-09-18T12:34:44.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.615+0000 7f20ca7fc640 1 -- 172.21.15.144:0/1675778873 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 osd.14 does not exist. v266) v1 ==== 92+0+0 (secure 0 0 0) 0x7f20d4040030 con 0x7f20dc136e70 2024-09-18T12:34:44.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.615+0000 7f20ca7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-18T12:34:44.617 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.615+0000 7f20ca7fc640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-18T12:34:44.617 INFO:tasks.workunit.client.0.smithi144.stderr:osd.14 does not exist. 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f20bc0384c0 msgr2=0x7f20bc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f20bc0384c0 0x7f20bc03a970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f20cc007fd0 tx=0x7f20cc007a70 comp rx=0 tx=0).stop 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 10 monclient: shutdown 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 msgr2=0x7f20dc098cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc098cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009f20 tx=0x7f20d4036000 comp rx=0 tx=0).stop 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 shutdown_connections 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f20bc0384c0 0x7f20bc03a970 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20dc136e70 0x7f20dc098cf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc131f10 0x7f20dc0987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.619 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 --2- 172.21.15.144:0/1675778873 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f20dc131b40 0x7f20dc1418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 >> 172.21.15.144:0/1675778873 conn(0x7f20dc0a5100 msgr2=0x7f20dc0a3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:44.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 shutdown_connections 2024-09-18T12:34:44.620 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.617+0000 7f20e0b5d640 1 -- 172.21.15.144:0/1675778873 wait complete. 2024-09-18T12:34:44.626 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph osd rm 1000 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.697+0000 7f7c5202b640 10 monclient: get_monmap_and_config 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.697+0000 7f7c5202b640 10 monclient: build_initial_monmap 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.697+0000 7f7c5202b640 10 monclient: monmap: 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:44.698862+0000 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:44.698862+0000 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:44.699 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:44.700 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:44.700 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:44.700 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:44.700 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:44.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.698+0000 7f7c5202b640 1 Processor -- start 2024-09-18T12:34:44.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.698+0000 7f7c5202b640 1 -- start start 2024-09-18T12:34:44.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.698+0000 7f7c5202b640 10 monclient: init 2024-09-18T12:34:44.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:44.702 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c4c09ecb0 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c4c069ec0 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c4c137d40 0x7f7c4c13a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c4c137d40 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4c06a2c0 con 0x7f7c4c09ecb0 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4c0a1840 con 0x7f7c4c069ec0 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4c09dad0 con 0x7f7c4c137d40 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c5202b640 10 monclient(hunting): authenticate will time out at 1320.268324s 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c4c137d40 0x7f7c4c13a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4affd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f7c4c069ec0 0x7f7c4c09f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57488/0 (socket says 172.21.15.144:57488) 2024-09-18T12:34:44.703 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4affd640 1 -- 172.21.15.144:0/968600490 learned_addr learned my addr 172.21.15.144:0/968600490 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): get_auth_request con 0x7f7c4c137d40 auth_method 0 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient(hunting): handle_auth_done global_id 124246 payload 274 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 1 -- 172.21.15.144:0/968600490 >> v1:172.21.15.144:6789/0 conn(0x7f7c4c069ec0 legacy=0x7f7c4c09f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 1 -- 172.21.15.144:0/968600490 >> v1:172.21.15.116:6789/0 conn(0x7f7c4c09ecb0 legacy=0x7f7c4c09d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 1 monclient: found mon.noname-c 2024-09-18T12:34:44.704 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.705 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.699+0000 7f7c4bfff640 1 -- 172.21.15.144:0/968600490 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c09f790 con 0x7f7c4c137d40 2024-09-18T12:34:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4bfff640 1 -- 172.21.15.144:0/968600490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c137d40 msgr2=0x7f7c4c13a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4bfff640 1 --2- 172.21.15.144:0/968600490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c137d40 0x7f7c4c13a120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009e60 tx=0x7f7c44009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d74b358bb83c930 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 1 -- 172.21.15.144:0/968600490 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44007840 con 0x7f7c4c137d40 2024-09-18T12:34:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:44.711 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: dump: 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:44.712 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 1 -- 172.21.15.144:0/968600490 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4402d050 con 0x7f7c4c137d40 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c5202b640 5 monclient: authenticate success, global_id 124246 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c5202b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 1 -- 172.21.15.144:0/968600490 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44007bd0 con 0x7f7c4c137d40 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:44.713 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c4a7fc640 10 monclient: dump: 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:44.714 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c5202b640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.702+0000 7f7c5202b640 4 set_mon_vals no callback set 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 10 monclient: shutdown 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 1 -- 172.21.15.144:0/968600490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c137d40 msgr2=0x7f7c4c13a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 1 --2- 172.21.15.144:0/968600490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c137d40 0x7f7c4c13a120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009e60 tx=0x7f7c44009780 comp rx=0 tx=0).stop 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 1 -- 172.21.15.144:0/968600490 shutdown_connections 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 1 --2- 172.21.15.144:0/968600490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c137d40 0x7f7c4c13a120 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 1 -- 172.21.15.144:0/968600490 >> 172.21.15.144:0/968600490 conn(0x7f7c4c0a5310 msgr2=0x7f7c4c0a5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 1 -- 172.21.15.144:0/968600490 shutdown_connections 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 1 -- 172.21.15.144:0/968600490 wait complete. 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 10 monclient: build_initial_monmap 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.703+0000 7f7c5202b640 10 monclient: monmap: 2024-09-18T12:34:44.715 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 Processor -- start 2024-09-18T12:34:44.716 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 -- start start 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient: init 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 0x7f7c4c140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4c069ec0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c4c09ecb0 0x7f7c4c1373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c4c09ecb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c4c137d40 0x7f7c4c11be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c4c137d40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c09bf90 con 0x7f7c4c09ecb0 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:44.717 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c11faf0 con 0x7f7c4c069ec0 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c11f780 con 0x7f7c4c137d40 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c5202b640 10 monclient(hunting): authenticate will time out at 1320.273885s 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c4bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c4c137d40 msgr2=0x7f7c4c11be30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c4b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 0x7f7c4c140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c4c137d40 0x7f7c4c11be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c4b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 0x7f7c4c140d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42204/0 (socket says 172.21.15.144:42204) 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c4b7fe640 1 -- 172.21.15.144:0/1650693032 learned_addr learned my addr 172.21.15.144:0/1650693032 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.704+0000 7f7c4b7fe640 10 monclient(hunting): get_auth_request con 0x7f7c4c069ec0 auth_method 0 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:44.718 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4affd640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c4c09ecb0 0x7f7c4c1373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4affd640 10 monclient(hunting): get_auth_request con 0x7f7c4c09ecb0 auth_method 0 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient(hunting): handle_auth_done global_id 124249 payload 824 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 1 -- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c4c137d40 msgr2=0x7f7c4c11be30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c4c137d40 0x7f7c4c11be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.719 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 1 -- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c4c09ecb0 msgr2=0x7f7c4c1373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c4c09ecb0 0x7f7c4c1373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 1 monclient: found mon.noname-b 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4b7fe640 1 -- 172.21.15.144:0/1650693032 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c44009a20 con 0x7f7c4c069ec0 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.705+0000 7f7c4affd640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c4c09ecb0 0x7f7c4c1373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c4b7fe640 1 -- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 msgr2=0x7f7c4c140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c4b7fe640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 0x7f7c4c140d60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00c610 tx=0x7f7c3c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c012840 con 0x7f7c4c069ec0 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:44.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: dump: 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: _finish_auth 0 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3c0129e0 con 0x7f7c4c069ec0 2024-09-18T12:34:44.721 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c5202b640 5 monclient: authenticate success, global_id 124249 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c5202b640 10 monclient: _renew_subs 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c5202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4c11f4a0 con 0x7f7c4c069ec0 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c012da0 con 0x7f7c4c069ec0 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c337fe640 10 monclient: dump: 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:44.722 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c5202b640 10 monclient: _renew_subs 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c5202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4c1412a0 con 0x7f7c4c069ec0 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.707+0000 7f7c49ffb640 4 set_mon_vals no callback set 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.708+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7c3c011590 con 0x7f7c4c069ec0 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.708+0000 7f7c5202b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.708+0000 7f7c5202b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:44.723 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.708+0000 7f7c5202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.708+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4c0636e0 con 0x7f7c4c069ec0 2024-09-18T12:34:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.708+0000 7f7c337fe640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c38038440 0x7f7c3803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.708+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(266..266 src has 192..266) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7c3c04bdf0 con 0x7f7c4c069ec0 2024-09-18T12:34:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.709+0000 7f7c4affd640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c38038440 0x7f7c3803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:44.724 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.709+0000 7f7c4affd640 10 monclient: get_auth_request con 0x7f7c38038440 auth_method 0 2024-09-18T12:34:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.711+0000 7f7c4affd640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c38038440 0x7f7c3803a8f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c11dc10 tx=0x7f7c40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.713+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7c3c04f030 con 0x7f7c4c069ec0 2024-09-18T12:34:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.713+0000 7f7c337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:44.725 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.713+0000 7f7c337fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:44.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.876+0000 7f7c5202b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T12:34:44.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.876+0000 7f7c5202b640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T12:34:44.877 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.876+0000 7f7c5202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.876+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f7c4c064b20 con 0x7f7c4c069ec0 2024-09-18T12:34:44.878 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:44.877+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e266 definitely_dead 0 2024-09-18T12:34:44.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.878+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v266) v1 ==== 96+0+0 (secure 0 0 0) 0x7f7c3c019910 con 0x7f7c4c069ec0 2024-09-18T12:34:44.878 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.878+0000 7f7c337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T12:34:44.879 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.878+0000 7f7c337fe640 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-18T12:34:44.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.978+0000 7f7c5202b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T12:34:44.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.978+0000 7f7c5202b640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T12:34:44.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.978+0000 7f7c5202b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:44.979 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.978+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f7c4c11e540 con 0x7f7c4c069ec0 2024-09-18T12:34:44.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:34:44.979+0000 7f3cb75f1640 -1 mon.a@0(leader).osd e266 definitely_dead 0 2024-09-18T12:34:44.980 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.979+0000 7f7c337fe640 1 -- 172.21.15.144:0/1650693032 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v266) v1 ==== 96+0+0 (secure 0 0 0) 0x7f7c3c019310 con 0x7f7c4c069ec0 2024-09-18T12:34:44.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.979+0000 7f7c337fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T12:34:44.981 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.979+0000 7f7c337fe640 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-18T12:34:44.981 INFO:tasks.workunit.client.0.smithi144.stderr:osd.1000 does not exist. 2024-09-18T12:34:44.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c38038440 msgr2=0x7f7c3803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:44.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c38038440 0x7f7c3803a8f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c11dc10 tx=0x7f7c40008040 comp rx=0 tx=0).stop 2024-09-18T12:34:44.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 10 monclient: shutdown 2024-09-18T12:34:44.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:44.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 msgr2=0x7f7c4c140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:44.983 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 0x7f7c4c140d60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00c610 tx=0x7f7c3c007ed0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 shutdown_connections 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c38038440 0x7f7c3803a8f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c4c137d40 0x7f7c4c11be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c4c09ecb0 0x7f7c4c1373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 --2- 172.21.15.144:0/1650693032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c4c069ec0 0x7f7c4c140d60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 >> 172.21.15.144:0/1650693032 conn(0x7f7c4c0a5310 msgr2=0x7f7c4c0a3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 shutdown_connections 2024-09-18T12:34:44.984 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:44.981+0000 7f7c5202b640 1 -- 172.21.15.144:0/1650693032 wait complete. 2024-09-18T12:34:44.991 INFO:tasks.workunit.client.0.smithi144.stderr:++ ceph osd create 4b751cce-d39e-4f85-9cff-56f6325f3ab2 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.063+0000 7f1647eef640 10 monclient: get_monmap_and_config 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.063+0000 7f1647eef640 10 monclient: build_initial_monmap 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.063+0000 7f1647eef640 10 monclient: monmap: 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:45.063988+0000 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:45.063988+0000 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:45.068 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.063+0000 7f1647eef640 1 Processor -- start 2024-09-18T12:34:45.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.063+0000 7f1647eef640 1 -- start start 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.063+0000 7f1647eef640 10 monclient: init 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1640131b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640131f10 0x7f1640132300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient(hunting): picked mon.noname-b con 0x7f1640131f10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1640136e70 0x7f1640139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient(hunting): picked mon.noname-c con 0x7f1640136e70 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:45.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1640132840 con 0x7f1640131b40 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16400a0300 con 0x7f1640136e70 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16400a00c0 con 0x7f1640131f10 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1647eef640 10 monclient(hunting): authenticate will time out at 1320.633412s 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164548a640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640131f10 msgr2=0x7f1640132300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164548a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640131f10 0x7f1640132300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1640136e70 0x7f1640139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1645c8b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f1640131b40 0x7f164009ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:57498/0 (socket says 172.21.15.144:57498) 2024-09-18T12:34:45.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f1645c8b640 1 -- 172.21.15.144:0/3081167789 learned_addr learned my addr 172.21.15.144:0/3081167789 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): get_auth_request con 0x7f1640136e70 auth_method 0 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient(hunting): handle_auth_done global_id 124252 payload 274 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 1 -- 172.21.15.144:0/3081167789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640131f10 msgr2=0x7f1640132300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 1 --2- 172.21.15.144:0/3081167789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640131f10 0x7f1640132300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 1 -- 172.21.15.144:0/3081167789 >> v1:172.21.15.144:6789/0 conn(0x7f1640131b40 legacy=0x7f164009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:45.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient: _finish_hunting 0 2024-09-18T12:34:45.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 1 monclient: found mon.noname-c 2024-09-18T12:34:45.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:45.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:45.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.064+0000 7f164648c640 1 -- 172.21.15.144:0/3081167789 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164009bee0 con 0x7f1640136e70 2024-09-18T12:34:45.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f164648c640 1 -- 172.21.15.144:0/3081167789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640136e70 msgr2=0x7f1640139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f164648c640 1 --2- 172.21.15.144:0/3081167789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640136e70 0x7f1640139250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e60 tx=0x7f1638009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b4d6a1d61af18de server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 1 -- 172.21.15.144:0/3081167789 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638007840 con 0x7f1640136e70 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: dump: 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.080 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: _finish_auth 0 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 1 -- 172.21.15.144:0/3081167789 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163802d050 con 0x7f1640136e70 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1647eef640 5 monclient: authenticate success, global_id 124252 2024-09-18T12:34:45.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1647eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1647eef640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 1 -- 172.21.15.144:0/3081167789 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638007bd0 con 0x7f1640136e70 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1647eef640 4 set_mon_vals no callback set 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.067+0000 7f1644c89640 10 monclient: dump: 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.082 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 10 monclient: shutdown 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 -- 172.21.15.144:0/3081167789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640136e70 msgr2=0x7f1640139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 --2- 172.21.15.144:0/3081167789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640136e70 0x7f1640139250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1638009e60 tx=0x7f1638009780 comp rx=0 tx=0).stop 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 -- 172.21.15.144:0/3081167789 shutdown_connections 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 --2- 172.21.15.144:0/3081167789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640136e70 0x7f1640139250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.083 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 --2- 172.21.15.144:0/3081167789 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640131f10 0x7f1640132300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 -- 172.21.15.144:0/3081167789 >> 172.21.15.144:0/3081167789 conn(0x7f16400a5100 msgr2=0x7f16400a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 -- 172.21.15.144:0/3081167789 shutdown_connections 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 1 -- 172.21.15.144:0/3081167789 wait complete. 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 10 monclient: build_initial_monmap 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.068+0000 7f1647eef640 10 monclient: monmap: 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:45.084 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 Processor -- start 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 -- start start 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient: init 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:45.085 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 0x7f1640098830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): picked mon.noname-b con 0x7f1640131b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1640131f10 0x7f1640098d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): picked mon.noname-a con 0x7f1640131f10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640136e70 0x7f16400992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): picked mon.noname-c con 0x7f1640136e70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16400a0a30 con 0x7f1640131f10 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1640130c60 con 0x7f1640131b40 2024-09-18T12:34:45.086 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164011cc00 con 0x7f1640136e70 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1647eef640 10 monclient(hunting): authenticate will time out at 1320.638757s 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 0x7f1640098830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 0x7f1640098830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42212/0 (socket says 172.21.15.144:42212) 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 1 -- 172.21.15.144:0/1310444612 learned_addr learned my addr 172.21.15.144:0/1310444612 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 10 monclient(hunting): get_auth_request con 0x7f1640131b40 auth_method 0 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:45.087 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f164648c640 1 -- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640136e70 msgr2=0x7f16400992b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f164648c640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640136e70 0x7f16400992b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.069+0000 7f1645c8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 10 monclient(hunting): handle_auth_done global_id 124255 payload 824 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f164548a640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1640131f10 0x7f1640098d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 1 -- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640136e70 msgr2=0x7f16400992b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640136e70 0x7f16400992b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 1 -- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1640131f10 msgr2=0x7f1640098d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1640131f10 0x7f1640098d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.088 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 10 monclient: _finish_hunting 0 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 1 monclient: found mon.noname-b 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f1645c8b640 1 -- 172.21.15.144:0/1310444612 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1638009a20 con 0x7f1640131b40 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f164548a640 10 monclient: get_auth_request con 0x7f1640131f10 auth_method 0 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.070+0000 7f164548a640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1640131f10 0x7f1640098d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1645c8b640 1 -- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 msgr2=0x7f1640098830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1645c8b640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 0x7f1640098830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c5e0 tx=0x7f163c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c005cb0 con 0x7f1640131b40 2024-09-18T12:34:45.089 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: dump: 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:45.090 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: _finish_auth 0 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c005e30 con 0x7f1640131b40 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1647eef640 5 monclient: authenticate success, global_id 124255 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1647eef640 10 monclient: _renew_subs 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1647eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164009aca0 con 0x7f1640131b40 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c007340 con 0x7f1640131b40 2024-09-18T12:34:45.091 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1635ffb640 10 monclient: dump: 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:45.092 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1647eef640 10 monclient: _renew_subs 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1637fff640 4 set_mon_vals no callback set 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1647eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.072+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164011b940 con 0x7f1640131b40 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f163c01b400 con 0x7f1640131b40 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f1635ffb640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16240384c0 0x7f162403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(266..266 src has 192..266) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f163c04bf10 con 0x7f1640131b40 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f1647eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f1647eef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f1647eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:45.093 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f164548a640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16240384c0 0x7f162403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:45.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164009a130 con 0x7f1640131b40 2024-09-18T12:34:45.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.073+0000 7f164548a640 10 monclient: get_auth_request con 0x7f16240384c0 auth_method 0 2024-09-18T12:34:45.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.076+0000 7f164548a640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16240384c0 0x7f162403a970 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f164009f340 tx=0x7f1630008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:45.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.078+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f163c01ab40 con 0x7f1640131b40 2024-09-18T12:34:45.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.078+0000 7f1635ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:45.094 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.078+0000 7f1635ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:45.241 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.240+0000 7f1647eef640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:45.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.240+0000 7f1647eef640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:45.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.240+0000 7f1647eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:45.242 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.240+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"} v 0) v1 -- 0x7f16400636e0 con 0x7f1640131b40 2024-09-18T12:34:45.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.520+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}]=0 v267) v1 ==== 106+0+3 (secure 0 0 0) 0x7f163c0290b0 con 0x7f1640131b40 2024-09-18T12:34:45.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.520+0000 7f1635ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:45.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.520+0000 7f1635ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:45.625 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.624+0000 7f1647eef640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:45.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.624+0000 7f1647eef640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:45.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.624+0000 7f1647eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:45.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.624+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"} v 0) v1 -- 0x7f164011bb60 con 0x7f1640131b40 2024-09-18T12:34:45.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.625+0000 7f1635ffb640 1 -- 172.21.15.144:0/1310444612 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}]=0 v267) v1 ==== 106+0+3 (secure 0 0 0) 0x7f163c01a360 con 0x7f1640131b40 2024-09-18T12:34:45.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.625+0000 7f1635ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "4b751cce-d39e-4f85-9cff-56f6325f3ab2"}] 2024-09-18T12:34:45.626 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.625+0000 7f1635ffb640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16240384c0 msgr2=0x7f162403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16240384c0 0x7f162403a970 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f164009f340 tx=0x7f1630008040 comp rx=0 tx=0).stop 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 10 monclient: shutdown 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 msgr2=0x7f1640098830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 0x7f1640098830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c5e0 tx=0x7f163c004960 comp rx=0 tx=0).stop 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 shutdown_connections 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f16240384c0 0x7f162403a970 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1640136e70 0x7f16400992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1640131f10 0x7f1640098d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.629 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 --2- 172.21.15.144:0/1310444612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1640131b40 0x7f1640098830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:45.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 >> 172.21.15.144:0/1310444612 conn(0x7f16400a5100 msgr2=0x7f16400a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:45.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 shutdown_connections 2024-09-18T12:34:45.630 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:45.627+0000 7f1647eef640 1 -- 172.21.15.144:0/1310444612 wait complete. 2024-09-18T12:34:45.637 INFO:tasks.workunit.client.0.smithi144.stderr:+ na2=12 2024-09-18T12:34:45.637 INFO:tasks.workunit.client.0.smithi144.stdout:OK 2024-09-18T12:34:45.638 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo OK 2024-09-18T12:34:45.638 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:34:45.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:34:45.703 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-18T12:34:45.705 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-18T12:34:45.705 DEBUG:teuthology.orchestra.run.smithi144:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-18T12:34:45.763 INFO:tasks.workunit.client.0.smithi144.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-18T12:34:45.764 INFO:tasks.workunit.client.0.smithi144.stderr:+ exit_on_error=1 2024-09-18T12:34:45.764 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ ! -z '' ]] 2024-09-18T12:34:45.764 INFO:tasks.workunit.client.0.smithi144.stderr:++ uname 2024-09-18T12:34:45.765 INFO:tasks.workunit.client.0.smithi144.stderr:+ '[' Linux = FreeBSD ']' 2024-09-18T12:34:45.765 INFO:tasks.workunit.client.0.smithi144.stderr:+ ETIMEDOUT=110 2024-09-18T12:34:45.765 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-18T12:34:45.765 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-18T12:34:45.765 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=0 2024-09-18T12:34:45.765 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-18T12:34:45.765 INFO:tasks.workunit.client.0.smithi144.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-18T12:34:45.766 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-18T12:34:46.237 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=0 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-18T12:34:46.238 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-18T12:34:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:34:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:34:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:46.314 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph auth del client.bazar 2024-09-18T12:34:46.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.385+0000 7f5b1e71d640 10 monclient: get_monmap_and_config 2024-09-18T12:34:46.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.385+0000 7f5b1e71d640 10 monclient: build_initial_monmap 2024-09-18T12:34:46.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.385+0000 7f5b1e71d640 10 monclient: monmap: 2024-09-18T12:34:46.387 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:46.387 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:46.387 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:34:46.386092+0000 2024-09-18T12:34:46.387 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:34:46.386092+0000 2024-09-18T12:34:46.387 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.385+0000 7f5b1e71d640 1 Processor -- start 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.385+0000 7f5b1e71d640 1 -- start start 2024-09-18T12:34:46.388 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.385+0000 7f5b1e71d640 10 monclient: init 2024-09-18T12:34:46.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:46.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T12:34:46.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1809ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:46.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b18131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:46.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b1809f240 addr v1:172.21.15.116:6789/0 2024-09-18T12:34:46.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5b18136e70 0x7f5b18139250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b18136e70 addr v2:172.21.15.144:3300/0 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1806a230 con 0x7f5b1809f240 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b180a0300 con 0x7f5b18136e70 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180a00c0 con 0x7f5b18131b40 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1e71d640 10 monclient(hunting): authenticate will time out at 1321.955475s 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5b18136e70 0x7f5b18139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5b18136e70 0x7f5b18139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42258/0 (socket says 172.21.15.144:42258) 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 -- 172.21.15.144:0/509549915 learned_addr learned my addr 172.21.15.144:0/509549915 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:46.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): get_auth_request con 0x7f5b18136e70 auth_method 0 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b17fff640 1 -- 172.21.15.144:0/509549915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 msgr2=0x7f5b1809ed00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b17fff640 1 --2- 172.21.15.144:0/509549915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1809ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_done global_id 124270 payload 274 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 -- 172.21.15.144:0/509549915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 msgr2=0x7f5b1809ed00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 --2- 172.21.15.144:0/509549915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1809ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 -- 172.21.15.144:0/509549915 >> v1:172.21.15.116:6789/0 conn(0x7f5b1809f240 legacy=0x7f5b18131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:34:46.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient: _finish_hunting 0 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 monclient: found mon.noname-c 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.386+0000 7f5b1ccba640 1 -- 172.21.15.144:0/509549915 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1809bee0 con 0x7f5b18136e70 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b1ccba640 1 -- 172.21.15.144:0/509549915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 msgr2=0x7f5b18139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b1ccba640 1 --2- 172.21.15.144:0/509549915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b18139250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009f20 tx=0x7f5b10009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee0c663d27ea8af2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 1 -- 172.21.15.144:0/509549915 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10007610 con 0x7f5b18136e70 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: dump: 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:46.404 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.405 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.405 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:46.405 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:46.405 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:46.405 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:46.405 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:46.405 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: _finish_auth 0 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b1e71d640 5 monclient: authenticate success, global_id 124270 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b1e71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 1 -- 172.21.15.144:0/509549915 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1002d050 con 0x7f5b18136e70 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b1e71d640 10 monclient: get_monmap_and_config success 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b1e71d640 4 set_mon_vals no callback set 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 1 -- 172.21.15.144:0/509549915 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b100079a0 con 0x7f5b18136e70 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:46.406 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.389+0000 7f5b16ffd640 10 monclient: dump: 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 10 monclient: shutdown 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 -- 172.21.15.144:0/509549915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 msgr2=0x7f5b18139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:46.407 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/509549915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b18139250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009f20 tx=0x7f5b10009840 comp rx=0 tx=0).stop 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 -- 172.21.15.144:0/509549915 shutdown_connections 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/509549915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b18139250 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/509549915 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1809ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 -- 172.21.15.144:0/509549915 >> 172.21.15.144:0/509549915 conn(0x7f5b180a5100 msgr2=0x7f5b180a5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 -- 172.21.15.144:0/509549915 shutdown_connections 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 1 -- 172.21.15.144:0/509549915 wait complete. 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 10 monclient: build_initial_monmap 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.390+0000 7f5b1e71d640 10 monclient: monmap: 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:34:46.408 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 Processor -- start 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 -- start start 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient: init 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient: _reopen_session rank -1 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b1809f240 0x7f5b1812f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:46.409 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): picked mon.noname-a con 0x7f5b1809f240 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1812f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): picked mon.noname-c con 0x7f5b18131b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b180987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): picked mon.noname-b con 0x7f5b18136e70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1811bcc0 con 0x7f5b1809f240 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1809a8c0 con 0x7f5b18136e70 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180a0a30 con 0x7f5b18131b40 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): _renew_subs 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.391+0000 7f5b1e71d640 10 monclient(hunting): authenticate will time out at 1321.960996s 2024-09-18T12:34:46.410 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b180987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b177fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 msgr2=0x7f5b1812f620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b180987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:42272/0 (socket says 172.21.15.144:42272) 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 -- 172.21.15.144:0/1877586751 learned_addr learned my addr 172.21.15.144:0/1877586751 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): get_auth_request con 0x7f5b18136e70 auth_method 0 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b177fe640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1812f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b1809f240 0x7f5b1812f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:46.411 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 10 monclient(hunting): get_auth_request con 0x7f5b1809f240 auth_method 0 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient(hunting): handle_auth_done global_id 124273 payload 824 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 -- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 msgr2=0x7f5b1812f620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1812f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 -- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b1809f240 msgr2=0x7f5b1812f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b1809f240 0x7f5b1812f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:46.412 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient: _finish_hunting 0 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 monclient: found mon.noname-b 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.392+0000 7f5b1ccba640 1 -- 172.21.15.144:0/1877586751 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10009ae0 con 0x7f5b18136e70 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1ccba640 1 -- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 msgr2=0x7f5b180987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1ccba640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b180987b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009f20 tx=0x7f5b10036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10031cd0 con 0x7f5b18136e70 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: dump: 2024-09-18T12:34:46.413 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: _finish_auth 0 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b10031e30 con 0x7f5b18136e70 2024-09-18T12:34:46.414 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1e71d640 5 monclient: authenticate success, global_id 124273 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1e71d640 10 monclient: _renew_subs 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b181309d0 con 0x7f5b18136e70 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1003c630 con 0x7f5b18136e70 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b0ffff640 10 monclient: dump: 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:34:46.415 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b167fc640 4 set_mon_vals no callback set 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1e71d640 10 monclient: _renew_subs 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.394+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1809af10 con 0x7f5b18136e70 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.395+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5b10003020 con 0x7f5b18136e70 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.395+0000 7f5b1e71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.395+0000 7f5b1e71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.395+0000 7f5b1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:46.416 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.395+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1811bfd0 con 0x7f5b18136e70 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.395+0000 7f5b0ffff640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5af8038440 0x7f5af803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.396+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5b1006cd50 con 0x7f5b18136e70 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.396+0000 7f5b17fff640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5af8038440 0x7f5af803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.396+0000 7f5b17fff640 10 monclient: get_auth_request con 0x7f5af8038440 auth_method 0 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.398+0000 7f5b17fff640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5af8038440 0x7f5af803a8f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b080038f0 tx=0x7f5b080090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.400+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5b1003a910 con 0x7f5b18136e70 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.400+0000 7f5b0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:34:46.417 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.400+0000 7f5b0ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:34:46.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.617+0000 7f5b1e71d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T12:34:46.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.617+0000 7f5b1e71d640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T12:34:46.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.617+0000 7f5b1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:46.618 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:46.617+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7f5b180636e0 con 0x7f5b18136e70 2024-09-18T12:34:47.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.300+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v17) v1 ==== 82+0+0 (secure 0 0 0) 0x7f5b1003a730 con 0x7f5b18136e70 2024-09-18T12:34:47.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.300+0000 7f5b0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T12:34:47.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.300+0000 7f5b0ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:34:47.450 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.449+0000 7f5b1e71d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T12:34:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.449+0000 7f5b1e71d640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T12:34:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.449+0000 7f5b1e71d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:34:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.449+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7f5b18072640 con 0x7f5b18136e70 2024-09-18T12:34:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.450+0000 7f5b0ffff640 1 -- 172.21.15.144:0/1877586751 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v17) v1 ==== 82+0+0 (secure 0 0 0) 0x7f5b1003a910 con 0x7f5b18136e70 2024-09-18T12:34:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.450+0000 7f5b0ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T12:34:47.451 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.450+0000 7f5b0ffff640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:34:47.453 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5af8038440 msgr2=0x7f5af803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5af8038440 0x7f5af803a8f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b080038f0 tx=0x7f5b080090f0 comp rx=0 tx=0).stop 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 10 monclient: shutdown 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 msgr2=0x7f5b180987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b180987b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009f20 tx=0x7f5b10036000 comp rx=0 tx=0).stop 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 shutdown_connections 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5af8038440 0x7f5af803a8f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5b18136e70 0x7f5b180987b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5b18131b40 0x7f5b1812f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 --2- 172.21.15.144:0/1877586751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b1809f240 0x7f5b1812f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 >> 172.21.15.144:0/1877586751 conn(0x7f5b180a5100 msgr2=0x7f5b180a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:34:47.454 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 shutdown_connections 2024-09-18T12:34:47.455 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:34:47.452+0000 7f5b1e71d640 1 -- 172.21.15.144:0/1877586751 wait complete. 2024-09-18T12:34:47.462 INFO:tasks.workunit.client.0.smithi144.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-18T12:34:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-18T12:34:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-18T12:34:47.463 INFO:tasks.workunit.client.0.smithi144.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-18T12:34:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=0 2024-09-18T12:34:47.463 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-18T12:34:47.464 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-18T12:34:47.930 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=0 2024-09-18T12:34:47.930 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T12:34:47.930 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:47.930 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-18T12:34:47.930 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-18T12:34:47.931 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=0 2024-09-18T12:34:47.931 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T12:34:47.931 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T12:34:47.931 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T12:34:48.396 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=0 2024-09-18T12:34:48.396 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T12:34:48.397 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:48.397 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-18T12:34:48.397 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-18T12:34:48.397 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=0 2024-09-18T12:34:48.397 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-18T12:34:48.397 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-18T12:34:48.397 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-18T12:34:48.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=0 2024-09-18T12:34:48.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T12:34:48.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:48.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-18T12:34:48.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-18T12:34:48.860 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:34:48.861 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-18T12:34:48.861 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-18T12:34:48.861 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-18T12:34:49.163 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-18T12:34:49.164 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-18T12:34:49.467 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:34:49.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:34:49.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:49.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-18T12:34:49.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-18T12:34:49.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:34:49.468 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-18T12:34:49.468 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-18T12:34:49.469 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-18T12:34:49.710 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=110 2024-09-18T12:34:49.711 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-18T12:34:49.712 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-18T12:34:49.712 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-18T12:34:53.179 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:34:53.179 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:34:53.179 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:34:53.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:34:53.183 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:34:53.183 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-18T12:34:53.183 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-18T12:34:53.184 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:35:08.185 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-18T12:35:08.185 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-18T12:35:08.186 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:35:08.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.290+0000 7fbcc5cdd640 10 monclient: get_monmap_and_config 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.290+0000 7fbcc5cdd640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.290+0000 7fbcc5cdd640 10 monclient: monmap: 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:08.290907+0000 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:08.290907+0000 2024-09-18T12:35:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:08.292 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.290+0000 7fbcc5cdd640 1 Processor -- start 2024-09-18T12:35:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 1 -- start start 2024-09-18T12:35:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient: init 2024-09-18T12:35:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbcc0131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbcc009e910 0x7fbcc009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient(hunting): picked mon.noname-a con 0x7fbcc009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient(hunting): picked mon.noname-b con 0x7fbcc009f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc009faf0 con 0x7fbcc0131ab0 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc0132590 con 0x7fbcc009e910 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc009fd30 con 0x7fbcc009f250 2024-09-18T12:35:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcc5cdd640 10 monclient(hunting): authenticate will time out at 1335.612521s 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbcc009e910 0x7fbcc009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbcc009e910 0x7fbcc009ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45242/0 (socket says 172.21.15.116:45242) 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54304/0 (socket says 172.21.15.116:54304) 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 1 -- 172.21.15.116:0/2652486289 learned_addr learned my addr 172.21.15.116:0/2652486289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): get_auth_request con 0x7fbcc009e910 auth_method 0 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 10 monclient(hunting): get_auth_request con 0x7fbcc009f250 auth_method 0 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 1 -- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 msgr2=0x7fbcc0138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 1 -- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 msgr2=0x7fbcc0138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 1 --2- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbffff640 1 --2- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 10 monclient(hunting): handle_auth_done global_id 104790 payload 274 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 1 -- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 msgr2=0x7fbcc0138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 1 --2- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.291+0000 7fbcbeffd640 1 -- 172.21.15.116:0/2652486289 >> v1:172.21.15.144:6789/0 conn(0x7fbcc0131ab0 legacy=0x7fbcc0131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.292+0000 7fbcbeffd640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.292+0000 7fbcbeffd640 1 monclient: found mon.noname-a 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.292+0000 7fbcbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.292+0000 7fbcbeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.292+0000 7fbcbeffd640 1 -- 172.21.15.116:0/2652486289 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00a1110 con 0x7fbcc009e910 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.294+0000 7fbcbeffd640 1 -- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc009e910 msgr2=0x7fbcc009ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.294+0000 7fbcbeffd640 1 --2- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc009e910 0x7fbcc009ed10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009910 tx=0x7fbcb00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c702197c99a9701c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 1 -- 172.21.15.116:0/2652486289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0032440 con 0x7fbcc009e910 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: dump: 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcc5cdd640 5 monclient: authenticate success, global_id 104790 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcc5cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 1 -- 172.21.15.116:0/2652486289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb00325a0 con 0x7fbcc009e910 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 1 -- 172.21.15.116:0/2652486289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0032980 con 0x7fbcc009e910 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcbe7fc640 10 monclient: dump: 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.305 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcc5cdd640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.297+0000 7fbcc5cdd640 4 set_mon_vals no callback set 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 10 monclient: shutdown 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc009e910 msgr2=0x7fbcc009ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc009e910 0x7fbcc009ed10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009910 tx=0x7fbcb00091d0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/2652486289 shutdown_connections 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/2652486289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc009e910 0x7fbcc009ed10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/2652486289 >> 172.21.15.116:0/2652486289 conn(0x7fbcc0067d30 msgr2=0x7fbcc00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/2652486289 shutdown_connections 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/2652486289 wait complete. 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.298+0000 7fbcc5cdd640 10 monclient: monmap: 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:08.307 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.308 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.308 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:08.308 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:08.308 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:08.308 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 Processor -- start 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 -- start start 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient: init 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc009e910 0x7fbcc0141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): picked mon.noname-b con 0x7fbcc009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:08.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): picked mon.noname-c con 0x7fbcc009f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc0131ab0 0x7fbcc0134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): picked mon.noname-a con 0x7fbcc0131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc0130c60 con 0x7fbcc0131ab0 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc0135b10 con 0x7fbcc009e910 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc0136650 con 0x7fbcc009f250 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.299+0000 7fbcc5cdd640 10 monclient(hunting): authenticate will time out at 1335.620954s 2024-09-18T12:35:08.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc0131ab0 0x7fbcc0134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0141d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54320/0 (socket says 172.21.15.116:54320) 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 1 -- 172.21.15.116:0/1087791468 learned_addr learned my addr 172.21.15.116:0/1087791468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 10 monclient(hunting): get_auth_request con 0x7fbcc009f250 auth_method 0 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): get_auth_request con 0x7fbcc0131ab0 auth_method 0 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 1 -- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 msgr2=0x7fbcc0141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:35:08.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 1 -- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 msgr2=0x7fbcc0141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbeffd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0141d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbf7fe640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc009e910 0x7fbcc0141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbf7fe640 10 monclient(hunting): get_auth_request con 0x7fbcc009e910 auth_method 0 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient(hunting): handle_auth_done global_id 104793 payload 824 2024-09-18T12:35:08.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 1 -- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 msgr2=0x7fbcc0141d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0141d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 1 -- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc009e910 msgr2=0x7fbcc0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc009e910 0x7fbcc0141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 1 monclient: found mon.noname-a 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbffff640 1 -- 172.21.15.116:0/1087791468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb0009450 con 0x7fbcc0131ab0 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.300+0000 7fbcbf7fe640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc009e910 0x7fbcc0141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcbffff640 1 -- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc0131ab0 msgr2=0x7fbcc0134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcbffff640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc0131ab0 0x7fbcc0134070 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8007e70 tx=0x7fbcb80097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 1 -- 172.21.15.116:0/1087791468 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8014420 con 0x7fbcc0131ab0 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: dump: 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.318 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcc5cdd640 5 monclient: authenticate success, global_id 104793 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 1 -- 172.21.15.116:0/1087791468 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb80145c0 con 0x7fbcc0131ab0 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcc5cdd640 10 monclient: _renew_subs 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcc5cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc009bde0 con 0x7fbcc0131ab0 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 1 -- 172.21.15.116:0/1087791468 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8014980 con 0x7fbcc0131ab0 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcb77fe640 10 monclient: dump: 2024-09-18T12:35:08.319 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcc5cdd640 10 monclient: _renew_subs 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcbdffb640 4 set_mon_vals no callback set 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcc5cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.302+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc009d820 con 0x7fbcc0131ab0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.303+0000 7fbcb77fe640 1 -- 172.21.15.116:0/1087791468 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbcb801e460 con 0x7fbcc0131ab0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.303+0000 7fbcc5cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.303+0000 7fbcc5cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.303+0000 7fbcc5cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.303+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc01379b0 con 0x7fbcc0131ab0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.303+0000 7fbcb77fe640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbca00384c0 0x7fbca003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.303+0000 7fbcb77fe640 1 -- 172.21.15.116:0/1087791468 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fbcb804c9e0 con 0x7fbcc0131ab0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.304+0000 7fbcbf7fe640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbca00384c0 0x7fbca003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.304+0000 7fbcbf7fe640 10 monclient: get_auth_request con 0x7fbca00384c0 auth_method 0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.306+0000 7fbcbf7fe640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbca00384c0 0x7fbca003a970 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbca8002860 tx=0x7fbca8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.308+0000 7fbcb77fe640 1 -- 172.21.15.116:0/1087791468 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbcb8013380 con 0x7fbcc0131ab0 2024-09-18T12:35:08.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.308+0000 7fbcb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.308+0000 7fbcb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.521+0000 7fbcc5cdd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.521+0000 7fbcc5cdd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.521+0000 7fbcc5cdd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.521+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbcc004bf60 con 0x7fbcc0131ab0 2024-09-18T12:35:08.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.521+0000 7fbcb77fe640 1 -- 172.21.15.116:0/1087791468 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fbcb801b310 con 0x7fbcc0131ab0 2024-09-18T12:35:08.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.521+0000 7fbcb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:08.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.521+0000 7fbcb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:08.523 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbca00384c0 msgr2=0x7fbca003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbca00384c0 0x7fbca003a970 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbca8002860 tx=0x7fbca8009040 comp rx=0 tx=0).stop 2024-09-18T12:35:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 10 monclient: shutdown 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc0131ab0 msgr2=0x7fbcc0134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc0131ab0 0x7fbcc0134070 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8007e70 tx=0x7fbcb80097e0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 shutdown_connections 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbca00384c0 0x7fbca003a970 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcc0131ab0 0x7fbcc0134070 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbcc009f250 0x7fbcc0141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 --2- 172.21.15.116:0/1087791468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbcc009e910 0x7fbcc0141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 >> 172.21.15.116:0/1087791468 conn(0x7fbcc0067d30 msgr2=0x7fbcc0068b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 shutdown_connections 2024-09-18T12:35:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.523+0000 7fbcc5cdd640 1 -- 172.21.15.116:0/1087791468 wait complete. 2024-09-18T12:35:08.535 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:35:08.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:35:08.535 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:35:08.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.628+0000 7f793b64f640 10 monclient: get_monmap_and_config 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.628+0000 7f793b64f640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.628+0000 7f793b64f640 10 monclient: monmap: 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:08.628732+0000 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:08.628732+0000 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:08.630 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.628+0000 7f793b64f640 1 Processor -- start 2024-09-18T12:35:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.628+0000 7f793b64f640 1 -- start start 2024-09-18T12:35:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.628+0000 7f793b64f640 10 monclient: init 2024-09-18T12:35:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T12:35:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f79341318f0 0x7f7934131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient(hunting): picked mon.noname-c con 0x7f79341318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f793409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient(hunting): picked mon.noname-b con 0x7f793409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7934068d90 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7934131cc0 con 0x7f7934068d90 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793409fbf0 con 0x7f79341318f0 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f793409fe30 con 0x7f793409ea50 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f793b64f640 10 monclient(hunting): authenticate will time out at 1335.950485s 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f793409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f793409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54336/0 (socket says 172.21.15.116:54336) 2024-09-18T12:35:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 -- 172.21.15.116:0/405040346 learned_addr learned my addr 172.21.15.116:0/405040346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): get_auth_request con 0x7f793409ea50 auth_method 0 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f79393eb640 1 --2- 172.21.15.116:0/405040346 >> v2:172.21.15.144:3300/0 conn(0x7f79341318f0 0x7f7934131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f79393eb640 10 monclient(hunting): get_auth_request con 0x7f79341318f0 auth_method 0 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f79393eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f79393eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f79393eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient(hunting): handle_auth_done global_id 134099 payload 274 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 -- 172.21.15.116:0/405040346 >> v2:172.21.15.144:3300/0 conn(0x7f79341318f0 msgr2=0x7f7934131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 --2- 172.21.15.116:0/405040346 >> v2:172.21.15.144:3300/0 conn(0x7f79341318f0 0x7f7934131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 -- 172.21.15.116:0/405040346 >> v1:172.21.15.144:6789/0 conn(0x7f7934068d90 legacy=0x7f79340691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 monclient: found mon.noname-b 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:35:08.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.629+0000 7f7938bea640 1 -- 172.21.15.116:0/405040346 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79340a1210 con 0x7f793409ea50 2024-09-18T12:35:08.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f7938bea640 1 -- 172.21.15.116:0/405040346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 msgr2=0x7f793409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f7938bea640 1 --2- 172.21.15.116:0/405040346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f793409ee20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7924009910 tx=0x7f79240091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f59c52d1fd9b9e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 1 -- 172.21.15.116:0/405040346 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792402ae90 con 0x7f793409ea50 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: dump: 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 1 -- 172.21.15.116:0/405040346 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7924032440 con 0x7f793409ea50 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f793b64f640 5 monclient: authenticate success, global_id 134099 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f793b64f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f793b64f640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 1 -- 172.21.15.116:0/405040346 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79240327e0 con 0x7f793409ea50 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f793b64f640 4 set_mon_vals no callback set 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.632+0000 7f792bfff640 10 monclient: dump: 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 10 monclient: shutdown 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 -- 172.21.15.116:0/405040346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 msgr2=0x7f793409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 --2- 172.21.15.116:0/405040346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f793409ee20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7924009910 tx=0x7f79240091d0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 -- 172.21.15.116:0/405040346 shutdown_connections 2024-09-18T12:35:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 --2- 172.21.15.116:0/405040346 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f793409ee20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 --2- 172.21.15.116:0/405040346 >> v2:172.21.15.144:3300/0 conn(0x7f79341318f0 0x7f7934131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 -- 172.21.15.116:0/405040346 >> 172.21.15.116:0/405040346 conn(0x7f7934067110 msgr2=0x7f7934067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 -- 172.21.15.116:0/405040346 shutdown_connections 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.633+0000 7f793b64f640 1 -- 172.21.15.116:0/405040346 wait complete. 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.634+0000 7f793b64f640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.634+0000 7f793b64f640 10 monclient: monmap: 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:08.646 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.634+0000 7f793b64f640 1 Processor -- start 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.634+0000 7f793b64f640 1 -- start start 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.634+0000 7f793b64f640 10 monclient: init 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.634+0000 7f793b64f640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.634+0000 7f793b64f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7934068d90 0x7f793409a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): picked mon.noname-b con 0x7f7934068d90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f7934130b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): picked mon.noname-c con 0x7f793409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79341318f0 0x7f79341310a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): picked mon.noname-a con 0x7f79341318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793409b5d0 con 0x7f79341318f0 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793409bf10 con 0x7f7934068d90 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79340a44b0 con 0x7f793409ea50 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f793b64f640 10 monclient(hunting): authenticate will time out at 1335.956071s 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f7934130b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f7934130b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54342/0 (socket says 172.21.15.116:54342) 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 -- 172.21.15.116:0/3452478986 learned_addr learned my addr 172.21.15.116:0/3452478986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): get_auth_request con 0x7f793409ea50 auth_method 0 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79341318f0 0x7f79341310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 10 monclient(hunting): get_auth_request con 0x7f79341318f0 auth_method 0 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7934068d90 0x7f793409a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 10 monclient(hunting): get_auth_request con 0x7f7934068d90 auth_method 0 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7939bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient(hunting): handle_auth_done global_id 134102 payload 824 2024-09-18T12:35:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7934068d90 msgr2=0x7f793409a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7934068d90 0x7f793409a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79341318f0 msgr2=0x7f79341310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79341318f0 0x7f79341310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 monclient: found mon.noname-c 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.635+0000 7f7938bea640 1 -- 172.21.15.116:0/3452478986 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7924009450 con 0x7f793409ea50 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f7938bea640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 msgr2=0x7f7934130b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f7938bea640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f7934130b60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7924009970 tx=0x7f7924032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924031c70 con 0x7f793409ea50 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: dump: 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792402ae90 con 0x7f793409ea50 2024-09-18T12:35:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f793b64f640 5 monclient: authenticate success, global_id 134102 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f793b64f640 10 monclient: _renew_subs 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f793b64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793411dc80 con 0x7f793409ea50 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792403d440 con 0x7f793409ea50 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f79297fa640 10 monclient: dump: 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f792b7fe640 4 set_mon_vals no callback set 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f793b64f640 10 monclient: _renew_subs 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f793b64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.637+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793411c630 con 0x7f793409ea50 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7924039040 con 0x7f793409ea50 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f793b64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f793b64f640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 0x7f7934140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f793b64f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7934065e00 con 0x7f793411bc30 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 0x7f7934140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient: get_auth_request con 0x7f793411bc30 auth_method 0 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79297fa640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79180384c0 0x7f791803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f792406be30 con 0x7f793409ea50 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.638+0000 7f79393eb640 10 monclient(hunting): handle_auth_done global_id 104799 payload 824 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.639+0000 7f79393eb640 1 -- 172.21.15.116:0/3452478986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f793404bf60 con 0x7f793411bc30 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.639+0000 7f7939bec640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79180384c0 0x7f791803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.639+0000 7f7939bec640 10 monclient: get_auth_request con 0x7f79180384c0 auth_method 0 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79393eb640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 msgr2=0x7f7934140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79393eb640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 0x7f7934140f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f79300108b0 tx=0x7f7930008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793000ed50 con 0x7f793411bc30 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 10 monclient: dump: 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.656 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f793000eeb0 con 0x7f793411bc30 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 msgr2=0x7f7934140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 0x7f7934140f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f79300108b0 tx=0x7f7930008040 comp rx=0 tx=0).stop 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 msgr2=0x7f7934140f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f79297fa640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f793411bc30 0x7f7934140f00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.641+0000 7f7939bec640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79180384c0 0x7f791803a970 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f792c007fd0 tx=0x7f792c0025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f793b64f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:08.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f793b64f640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 0x7f7934141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f793b64f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7934141d80 con 0x7f7934141440 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 0x7f7934141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient: get_auth_request con 0x7f7934141440 auth_method 0 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 10 monclient(hunting): handle_auth_done global_id 104802 payload 824 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.647+0000 7f79393eb640 1 -- 172.21.15.116:0/3452478986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f793411e2d0 con 0x7f7934141440 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.649+0000 7f79393eb640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 msgr2=0x7f7934141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.649+0000 7f79393eb640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 0x7f7934141840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7930019000 tx=0x7f793001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.649+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793002a070 con 0x7f7934141440 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.649+0000 7f79297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.649+0000 7f79297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.649+0000 7f79297fa640 10 monclient: dump: 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f7930022040 con 0x7f7934141440 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f79297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f79297fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 msgr2=0x7f7934141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f79297fa640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 0x7f7934141840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7930019000 tx=0x7f793001c000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f79297fa640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 msgr2=0x7f7934141840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f79297fa640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7934141440 0x7f7934141840 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7930019000 tx=0x7f793001c000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:08.662 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 98, 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:08.663 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.664 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:08.665 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:08.667 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.672 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.673 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:08.674 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79180384c0 msgr2=0x7f791803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79180384c0 0x7f791803a970 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f792c007fd0 tx=0x7f792c0025b0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 10 monclient: shutdown 2024-09-18T12:35:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 msgr2=0x7f7934130b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 --2- 172.21.15.116:0/3452478986 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f793409ea50 0x7f7934130b60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7924009970 tx=0x7f7924032000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f7939bec640 1 -- 172.21.15.116:0/3452478986 reap_dead start 2024-09-18T12:35:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f7939bec640 1 -- 172.21.15.116:0/3452478986 reap_dead start 2024-09-18T12:35:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 shutdown_connections 2024-09-18T12:35:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 >> 172.21.15.116:0/3452478986 conn(0x7f7934067110 msgr2=0x7f793409ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 shutdown_connections 2024-09-18T12:35:08.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.650+0000 7f793b64f640 1 -- 172.21.15.116:0/3452478986 wait complete. 2024-09-18T12:35:08.677 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:35:08.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:35:08.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:35:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.770+0000 7f0001709640 10 monclient: get_monmap_and_config 2024-09-18T12:35:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.770+0000 7f0001709640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.770+0000 7f0001709640 10 monclient: monmap: 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:08.771462+0000 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:08.771462+0000 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:08.772 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 1 Processor -- start 2024-09-18T12:35:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 1 -- start start 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient: init 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efffc131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efffc09e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efffc137bc0 0x7efffc139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient(hunting): picked mon.noname-a con 0x7efffc137bc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efffc09faf0 con 0x7efffc09e910 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efffc09fd30 con 0x7efffc131ab0 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efffc132590 con 0x7efffc137bc0 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7f0001709640 10 monclient(hunting): authenticate will time out at 1336.092864s 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7efffb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efffc137bc0 0x7efffc139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efffc137bc0 0x7efffc139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45274/0 (socket says 172.21.15.116:45274) 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 1 -- 172.21.15.116:0/110737709 learned_addr learned my addr 172.21.15.116:0/110737709 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): get_auth_request con 0x7efffc137bc0 auth_method 0 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.771+0000 7efffa7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efffc09e910 0x7efffc09ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41564/0 (socket says 172.21.15.116:41564) 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient(hunting): handle_auth_done global_id 104805 payload 274 2024-09-18T12:35:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 1 -- 172.21.15.116:0/110737709 >> v1:172.21.15.144:6789/0 conn(0x7efffc131ab0 legacy=0x7efffc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 1 -- 172.21.15.116:0/110737709 >> v1:172.21.15.116:6789/0 conn(0x7efffc09e910 legacy=0x7efffc09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 1 monclient: found mon.noname-a 2024-09-18T12:35:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.772+0000 7efffb7fe640 1 -- 172.21.15.116:0/110737709 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc0a1110 con 0x7efffc137bc0 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.774+0000 7efffb7fe640 1 -- 172.21.15.116:0/110737709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 msgr2=0x7efffc139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.774+0000 7efffb7fe640 1 --2- 172.21.15.116:0/110737709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc139fa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efff4009f50 tx=0x7efff4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3f151f060e6c6b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 1 -- 172.21.15.116:0/110737709 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4007610 con 0x7efffc137bc0 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: dump: 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 5 monclient: authenticate success, global_id 104805 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 1 -- 172.21.15.116:0/110737709 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff402d050 con 0x7efffc137bc0 2024-09-18T12:35:08.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:08.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 4 set_mon_vals no callback set 2024-09-18T12:35:08.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 1 -- 172.21.15.116:0/110737709 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff40079e0 con 0x7efffc137bc0 2024-09-18T12:35:08.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:08.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7efff9ffb640 10 monclient: dump: 2024-09-18T12:35:08.787 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 10 monclient: shutdown 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 1 -- 172.21.15.116:0/110737709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 msgr2=0x7efffc139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 1 --2- 172.21.15.116:0/110737709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc139fa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efff4009f50 tx=0x7efff4009840 comp rx=0 tx=0).stop 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 1 -- 172.21.15.116:0/110737709 shutdown_connections 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 1 --2- 172.21.15.116:0/110737709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc139fa0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 1 -- 172.21.15.116:0/110737709 >> 172.21.15.116:0/110737709 conn(0x7efffc067d30 msgr2=0x7efffc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 1 -- 172.21.15.116:0/110737709 shutdown_connections 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.775+0000 7f0001709640 1 -- 172.21.15.116:0/110737709 wait complete. 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.776+0000 7f0001709640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.776+0000 7f0001709640 10 monclient: monmap: 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.776+0000 7f0001709640 1 Processor -- start 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.776+0000 7f0001709640 1 -- start start 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.776+0000 7f0001709640 10 monclient: init 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.776+0000 7f0001709640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.776+0000 7f0001709640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09e910 0x7efffc12e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): picked mon.noname-c con 0x7efffc09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efffc131ab0 0x7efffc12e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): picked mon.noname-b con 0x7efffc131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc12ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): picked mon.noname-a con 0x7efffc137bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc09bde0 con 0x7efffc137bc0 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc0a54b0 con 0x7efffc131ab0 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efffc131410 con 0x7efffc09e910 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7f0001709640 10 monclient(hunting): authenticate will time out at 1336.098049s 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc12ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09e910 0x7efffc12e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc12ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45276/0 (socket says 172.21.15.116:45276) 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 1 -- 172.21.15.116:0/1620175270 learned_addr learned my addr 172.21.15.116:0/1620175270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 10 monclient(hunting): get_auth_request con 0x7efffc137bc0 auth_method 0 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): get_auth_request con 0x7efffc09e910 auth_method 0 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efffc131ab0 0x7efffc12e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 10 monclient(hunting): get_auth_request con 0x7efffc131ab0 auth_method 0 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient(hunting): handle_auth_done global_id 134105 payload 824 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efffc131ab0 msgr2=0x7efffc12e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efffc131ab0 0x7efffc12e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 msgr2=0x7efffc12ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc12ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 1 monclient: found mon.noname-c 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffaffd640 1 -- 172.21.15.116:0/1620175270 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff4009ae0 con 0x7efffc09e910 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.777+0000 7efffb7fe640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efffc137bc0 0x7efffc12ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7efffaffd640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09e910 msgr2=0x7efffc12e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7efffaffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09e910 0x7efffc12e1c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7effec00c4e0 tx=0x7effec007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec0117a0 con 0x7efffc09e910 2024-09-18T12:35:08.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: dump: 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec011920 con 0x7efffc09e910 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec011ce0 con 0x7efffc09e910 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7f0001709640 5 monclient: authenticate success, global_id 134105 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7f0001709640 10 monclient: _renew_subs 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7f0001709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efffc136710 con 0x7efffc09e910 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7effe2ffd640 10 monclient: dump: 2024-09-18T12:35:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7efff97fa640 4 set_mon_vals no callback set 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7f0001709640 10 monclient: _renew_subs 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7f0001709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.779+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efffc12f960 con 0x7efffc09e910 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7f0001709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7f0001709640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 0x7efffc12f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7f0001709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7effec012630 con 0x7efffc09e910 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efffc063740 con 0x7efffc1304b0 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 0x7efffc12f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 10 monclient: get_auth_request con 0x7efffc1304b0 auth_method 0 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7effe2ffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effe8038490 0x7effe803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.780+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.781+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7effec04bec0 con 0x7efffc09e910 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.781+0000 7efffa7fc640 10 monclient(hunting): handle_auth_done global_id 134108 payload 824 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.781+0000 7efffa7fc640 1 -- 172.21.15.116:0/1620175270 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efffc04bf60 con 0x7efffc1304b0 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.781+0000 7efffb7fe640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effe8038490 0x7effe803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.781+0000 7efffb7fe640 10 monclient: get_auth_request con 0x7effe8038490 auth_method 0 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7efffa7fc640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 msgr2=0x7efffc12f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7efffa7fc640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 0x7efffc12f180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efff000e740 tx=0x7efff0007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0005050 con 0x7efffc1304b0 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 10 monclient: dump: 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efff00051b0 con 0x7efffc1304b0 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:08.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 msgr2=0x7efffc12f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 0x7efffc12f180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efff000e740 tx=0x7efff0007e40 comp rx=0 tx=0).stop 2024-09-18T12:35:08.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 msgr2=0x7efffc12f180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7effe2ffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc1304b0 0x7efffc12f180 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efff000e740 tx=0x7efff0007e40 comp rx=0 tx=0).stop 2024-09-18T12:35:08.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.783+0000 7efffb7fe640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effe8038490 0x7effe803a940 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efff4009880 tx=0x7efff4036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7f0001709640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7f0001709640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 0x7efffc09da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7f0001709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efffc066280 con 0x7efffc09d690 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 0x7efffc09da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient: get_auth_request con 0x7efffc09d690 auth_method 0 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 10 monclient(hunting): handle_auth_done global_id 134111 payload 824 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.789+0000 7efffa7fc640 1 -- 172.21.15.116:0/1620175270 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efffc09dfd0 con 0x7efffc09d690 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.791+0000 7efffa7fc640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 msgr2=0x7efffc09da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7efffa7fc640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 0x7efffc09da90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efff000e740 tx=0x7efff0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0014e60 con 0x7efffc09d690 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 10 monclient: dump: 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7efff0025070 con 0x7efffc09d690 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 msgr2=0x7efffc09da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 0x7efffc09da90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efff000e740 tx=0x7efff0019000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 msgr2=0x7efffc09da90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7effe2ffd640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09d690 0x7efffc09da90 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efff000e740 tx=0x7efff0019000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 98, 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.806 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.807 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:08.809 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.810 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.813 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:08.814 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.816 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:08.817 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:08.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effe8038490 msgr2=0x7effe803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effe8038490 0x7effe803a940 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efff4009880 tx=0x7efff4036000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 10 monclient: shutdown 2024-09-18T12:35:08.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09e910 msgr2=0x7efffc12e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 --2- 172.21.15.116:0/1620175270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efffc09e910 0x7efffc12e1c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7effec00c4e0 tx=0x7effec007e10 comp rx=0 tx=0).stop 2024-09-18T12:35:08.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7efffb7fe640 1 -- 172.21.15.116:0/1620175270 reap_dead start 2024-09-18T12:35:08.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7efffb7fe640 1 -- 172.21.15.116:0/1620175270 reap_dead start 2024-09-18T12:35:08.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 shutdown_connections 2024-09-18T12:35:08.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 >> 172.21.15.116:0/1620175270 conn(0x7efffc067d30 msgr2=0x7efffc139160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 shutdown_connections 2024-09-18T12:35:08.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.792+0000 7f0001709640 1 -- 172.21.15.116:0/1620175270 wait complete. 2024-09-18T12:35:08.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:35:08.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:35:08.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:35:08.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.913+0000 7efead900640 10 monclient: get_monmap_and_config 2024-09-18T12:35:08.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.913+0000 7efead900640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.913+0000 7efead900640 10 monclient: monmap: 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:08.914041+0000 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:08.914041+0000 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.913+0000 7efead900640 1 Processor -- start 2024-09-18T12:35:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.913+0000 7efead900640 1 -- start start 2024-09-18T12:35:08.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.913+0000 7efead900640 10 monclient: init 2024-09-18T12:35:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 0x7efea8131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient(hunting): picked mon.noname-b con 0x7efea8131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efea809e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efea8136bb0 0x7efea8138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient(hunting): picked mon.noname-c con 0x7efea8136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea809faf0 con 0x7efea809e910 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea809fd30 con 0x7efea8136bb0 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea80a1110 con 0x7efea8131ab0 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efead900640 10 monclient(hunting): authenticate will time out at 1336.235467s 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 0x7efea8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea67fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efea809e910 0x7efea809ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41570/0 (socket says 172.21.15.116:41570) 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea67fc640 1 -- 172.21.15.116:0/3499480764 learned_addr learned my addr 172.21.15.116:0/3499480764 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 10 monclient(hunting): get_auth_request con 0x7efea8131ab0 auth_method 0 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 1 -- 172.21.15.116:0/3499480764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 msgr2=0x7efea8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 1 --2- 172.21.15.116:0/3499480764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 0x7efea8131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea6ffd640 1 --2- 172.21.15.116:0/3499480764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 0x7efea8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:08.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 1 --2- 172.21.15.116:0/3499480764 >> v2:172.21.15.144:3300/0 conn(0x7efea8136bb0 0x7efea8138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 10 monclient(hunting): get_auth_request con 0x7efea8136bb0 auth_method 0 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.914+0000 7efea77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 10 monclient(hunting): handle_auth_done global_id 124321 payload 274 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 1 -- 172.21.15.116:0/3499480764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 msgr2=0x7efea8131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 1 --2- 172.21.15.116:0/3499480764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 0x7efea8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 1 -- 172.21.15.116:0/3499480764 >> v1:172.21.15.116:6789/0 conn(0x7efea809e910 legacy=0x7efea809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 1 monclient: found mon.noname-c 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.915+0000 7efea77fe640 1 -- 172.21.15.116:0/3499480764 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea80a13f0 con 0x7efea8136bb0 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea77fe640 1 -- 172.21.15.116:0/3499480764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 msgr2=0x7efea8138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea77fe640 1 --2- 172.21.15.116:0/3499480764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 0x7efea8138f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efea0009e60 tx=0x7efea0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=571da8c5dbf17b31 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 1 -- 172.21.15.116:0/3499480764 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea0007760 con 0x7efea8136bb0 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: dump: 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 1 -- 172.21.15.116:0/3499480764 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea002d050 con 0x7efea8136bb0 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 5 monclient: authenticate success, global_id 124321 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 1 -- 172.21.15.116:0/3499480764 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea0007af0 con 0x7efea8136bb0 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efea5ffb640 10 monclient: dump: 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.926 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 4 set_mon_vals no callback set 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 10 monclient: shutdown 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 1 -- 172.21.15.116:0/3499480764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 msgr2=0x7efea8138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.918+0000 7efead900640 1 --2- 172.21.15.116:0/3499480764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 0x7efea8138f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efea0009e60 tx=0x7efea0009780 comp rx=0 tx=0).stop 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 1 -- 172.21.15.116:0/3499480764 shutdown_connections 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 1 --2- 172.21.15.116:0/3499480764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 0x7efea8138f90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 1 --2- 172.21.15.116:0/3499480764 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea8131ab0 0x7efea8131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 1 -- 172.21.15.116:0/3499480764 >> 172.21.15.116:0/3499480764 conn(0x7efea8067d30 msgr2=0x7efea80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 1 -- 172.21.15.116:0/3499480764 shutdown_connections 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 1 -- 172.21.15.116:0/3499480764 wait complete. 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 10 monclient: build_initial_monmap 2024-09-18T12:35:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.919+0000 7efead900640 10 monclient: monmap: 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 Processor -- start 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 -- start start 2024-09-18T12:35:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient: init 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea809e910 0x7efea8136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): picked mon.noname-c con 0x7efea809e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efea8131ab0 0x7efea812eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): picked mon.noname-a con 0x7efea8131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 0x7efea812f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): picked mon.noname-b con 0x7efea8136bb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea81330f0 con 0x7efea8131ab0 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea809be50 con 0x7efea8136bb0 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea809d2a0 con 0x7efea809e910 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efead900640 10 monclient(hunting): authenticate will time out at 1336.241556s 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efea8131ab0 0x7efea812eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea809e910 0x7efea8136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea809e910 0x7efea8136040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54386/0 (socket says 172.21.15.116:54386) 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 learned_addr learned my addr 172.21.15.116:0/1998716870 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 10 monclient(hunting): get_auth_request con 0x7efea809e910 auth_method 0 2024-09-18T12:35:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): get_auth_request con 0x7efea8131ab0 auth_method 0 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea77fe640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 0x7efea812f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient(hunting): handle_auth_done global_id 104811 payload 824 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea809e910 msgr2=0x7efea8136040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea809e910 0x7efea8136040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 msgr2=0x7efea812f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 0x7efea812f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient: _finish_hunting 0 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 1 monclient: found mon.noname-a 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.920+0000 7efea67fc640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0009a20 con 0x7efea8131ab0 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.921+0000 7efea77fe640 10 monclient: get_auth_request con 0x7efea8136bb0 auth_method 0 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.921+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efea809e910 0x7efea8136040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:35:08.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.921+0000 7efea77fe640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea8136bb0 0x7efea812f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efea67fc640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efea8131ab0 msgr2=0x7efea812eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efea67fc640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efea8131ab0 0x7efea812eed0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00c490 tx=0x7efe9c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c005cb0 con 0x7efea8131ab0 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: dump: 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: _finish_auth 0 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c005e30 con 0x7efea8131ab0 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efead900640 5 monclient: authenticate success, global_id 104811 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c007320 con 0x7efea8131ab0 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efead900640 10 monclient: _renew_subs 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efead900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea81307b0 con 0x7efea8131ab0 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efe8effd640 10 monclient: dump: 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efea57fa640 4 set_mon_vals no callback set 2024-09-18T12:35:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efead900640 10 monclient: _renew_subs 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efead900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.923+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea81354c0 con 0x7efea8131ab0 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7efe9c01b3b0 con 0x7efea8131ab0 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efead900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efead900640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 0x7efea812fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efead900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8063740 con 0x7efea812f950 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe94038490 0x7efe9403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efe9c04b800 con 0x7efea8131ab0 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 0x7efea812fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea77fe640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe94038490 0x7efe9403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea6ffd640 10 monclient: get_auth_request con 0x7efea812f950 auth_method 0 2024-09-18T12:35:08.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea77fe640 10 monclient: get_auth_request con 0x7efe94038490 auth_method 0 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.924+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.925+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.925+0000 7efea6ffd640 10 monclient(hunting): handle_auth_done global_id 124324 payload 824 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.925+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efea8133430 con 0x7efea812f950 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.927+0000 7efea77fe640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe94038490 0x7efe9403a940 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7efea80a5ec0 tx=0x7efea0037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.927+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 msgr2=0x7efea812fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.927+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 0x7efea812fd20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efe98007f40 tx=0x7efe98008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.927+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98014640 con 0x7efea812f950 2024-09-18T12:35:08.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.927+0000 7efe8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.927+0000 7efe8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.927+0000 7efe8effd640 10 monclient: dump: 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.928+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efe980147a0 con 0x7efea812f950 2024-09-18T12:35:08.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.928+0000 7efe8effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.928+0000 7efe8effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.928+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 msgr2=0x7efea812fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.928+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 0x7efea812fd20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efe98007f40 tx=0x7efe98008040 comp rx=0 tx=0).stop 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.928+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 msgr2=0x7efea812fd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.928+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812f950 0x7efea812fd20 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efe98007f40 tx=0x7efe98008040 comp rx=0 tx=0).stop 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.933+0000 7efead900640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.933+0000 7efead900640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 0x7efea812adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.933+0000 7efead900640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.933+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8130260 con 0x7efea812a9b0 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 0x7efea812adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient: get_auth_request con 0x7efea812a9b0 auth_method 0 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 10 monclient(hunting): handle_auth_done global_id 124327 payload 824 2024-09-18T12:35:08.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.934+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efea804bf60 con 0x7efea812a9b0 2024-09-18T12:35:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.936+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 msgr2=0x7efea812adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.936+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 0x7efea812adb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe9801a000 tx=0x7efe9801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.936+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 msgr2=0x7efea812adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:35:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.936+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 msgr2=0x7efea812adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.936+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 0x7efea812adb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe9801a000 tx=0x7efe9801d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 0x7efea812adb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe9801a000 tx=0x7efe9801d000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 msgr2=0x7efea812adb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 0x7efea812adb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 0x7efe940432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 msgr2=0x7efea812adb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efea812a9b0 0x7efea812adb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efe8effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe940438d0 con 0x7efe94040ee0 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 0x7efe940432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient: get_auth_request con 0x7efe94040ee0 auth_method 0 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 10 monclient(hunting): handle_auth_done global_id 124330 payload 824 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.937+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efe980147a0 con 0x7efe94040ee0 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.939+0000 7efea6ffd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 msgr2=0x7efe940432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.939+0000 7efea6ffd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 0x7efe940432c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efe98007ce0 tx=0x7efe9801f070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9801dea0 con 0x7efe94040ee0 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 10 monclient: dump: 2024-09-18T12:35:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7efe98023040 con 0x7efe94040ee0 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 msgr2=0x7efe940432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 0x7efe940432c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efe98007ce0 tx=0x7efe9801f070 comp rx=0 tx=0).stop 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 msgr2=0x7efe940432c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efe8effd640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe94040ee0 0x7efe940432c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efea77fe640 1 -- 172.21.15.116:0/1998716870 reap_dead start 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 98, 2024-09-18T12:35:08.952 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.953 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:08.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:08.956 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.957 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:08.958 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:08.961 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:08.962 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.963 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:08.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:08.965 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:08.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:08.965 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:08.965 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe94038490 msgr2=0x7efe9403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe94038490 0x7efe9403a940 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7efea80a5ec0 tx=0x7efea0037000 comp rx=0 tx=0).stop 2024-09-18T12:35:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 10 monclient: shutdown 2024-09-18T12:35:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efea8131ab0 msgr2=0x7efea812eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:08.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efea8131ab0 0x7efea812eed0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00c490 tx=0x7efe9c007f00 comp rx=0 tx=0).stop 2024-09-18T12:35:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 shutdown_connections 2024-09-18T12:35:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe94038490 0x7efe9403a940 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 --2- 172.21.15.116:0/1998716870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efea8131ab0 0x7efea812eed0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 >> 172.21.15.116:0/1998716870 conn(0x7efea8067d30 msgr2=0x7efea809d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 shutdown_connections 2024-09-18T12:35:08.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:08.940+0000 7efead900640 1 -- 172.21.15.116:0/1998716870 wait complete. 2024-09-18T12:35:08.967 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:35:08.967 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:35:08.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:35:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.059+0000 7f3d0679d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.059+0000 7f3d0679d640 10 monclient: build_initial_monmap 2024-09-18T12:35:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.059+0000 7f3d0679d640 10 monclient: monmap: 2024-09-18T12:35:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:09.060377+0000 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:09.060377+0000 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:09.061 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 1 Processor -- start 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 1 -- start start 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient: init 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d0009c860 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d0009e840 0x7f3d0009aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d0009e840 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d00136cb0 0x7f3d00139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d00136cb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0009ec40 con 0x7f3d0009c860 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0009a9d0 con 0x7f3d0009e840 2024-09-18T12:35:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006a320 con 0x7f3d00136cb0 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.060+0000 7f3d0679d640 10 monclient(hunting): authenticate will time out at 1336.381922s 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d00136cb0 0x7f3d00139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d0009e840 0x7f3d0009aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d00136cb0 0x7f3d00139090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54400/0 (socket says 172.21.15.116:54400) 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d0009e840 0x7f3d0009aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45300/0 (socket says 172.21.15.116:45300) 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 1 -- 172.21.15.116:0/471452134 learned_addr learned my addr 172.21.15.116:0/471452134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 10 monclient(hunting): get_auth_request con 0x7f3d00136cb0 auth_method 0 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cfffff640 1 --1- 172.21.15.116:0/471452134 >> v1:172.21.15.116:6789/0 conn(0x7f3d0009c860 0x7f3d0009cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41576/0 (socket says 172.21.15.116:41576) 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): get_auth_request con 0x7f3d0009e840 auth_method 0 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient(hunting): handle_auth_done global_id 104814 payload 274 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 1 -- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d00136cb0 msgr2=0x7f3d00139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:09.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 1 --2- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d00136cb0 0x7f3d00139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 1 -- 172.21.15.116:0/471452134 >> v1:172.21.15.116:6789/0 conn(0x7f3d0009c860 legacy=0x7f3d0009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:35:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 1 monclient: found mon.noname-a 2024-09-18T12:35:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3cff7fe640 1 -- 172.21.15.116:0/471452134 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d00068a20 con 0x7f3d0009e840 2024-09-18T12:35:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.061+0000 7f3d04d3a640 1 --2- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d00136cb0 0x7f3d00139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:35:09.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cff7fe640 1 -- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 msgr2=0x7f3d0009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:09.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cff7fe640 1 --2- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d0009aba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009850 tx=0x7f3cf40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e6dc9957a0641e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:09.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 1 -- 172.21.15.116:0/471452134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf402add0 con 0x7f3d0009e840 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: dump: 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:09.074 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: _finish_auth 0 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 1 -- 172.21.15.116:0/471452134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf4032440 con 0x7f3d0009e840 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 5 monclient: authenticate success, global_id 104814 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 1 -- 172.21.15.116:0/471452134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf40327e0 con 0x7f3d0009e840 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 4 set_mon_vals no callback set 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3cfeffd640 10 monclient: dump: 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 10 monclient: shutdown 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 -- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 msgr2=0x7f3d0009aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 --2- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d0009aba0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009850 tx=0x7f3cf40090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 -- 172.21.15.116:0/471452134 shutdown_connections 2024-09-18T12:35:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 --2- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d00136cb0 0x7f3d00139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 --2- 172.21.15.116:0/471452134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d0009aba0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 -- 172.21.15.116:0/471452134 >> 172.21.15.116:0/471452134 conn(0x7f3d000a2e40 msgr2=0x7f3d000a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 -- 172.21.15.116:0/471452134 shutdown_connections 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.064+0000 7f3d0679d640 1 -- 172.21.15.116:0/471452134 wait complete. 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.065+0000 7f3d0679d640 10 monclient: build_initial_monmap 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.065+0000 7f3d0679d640 10 monclient: monmap: 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.065+0000 7f3d0679d640 1 Processor -- start 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.065+0000 7f3d0679d640 1 -- start start 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.065+0000 7f3d0679d640 10 monclient: init 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 0x7f3d0006e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d0009c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d00135170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d0009e840 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d00136cb0 0x7f3d001356b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d00136cb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006c5e0 con 0x7f3d0009e840 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00099cc0 con 0x7f3d00136cb0 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001390a0 con 0x7f3d0009c860 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d0679d640 10 monclient(hunting): authenticate will time out at 1336.387219s 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 0x7f3d0006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d00135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 0x7f3d0006e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54404/0 (socket says 172.21.15.116:54404) 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 -- 172.21.15.116:0/25218464 learned_addr learned my addr 172.21.15.116:0/25218464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d00135170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45310/0 (socket says 172.21.15.116:45310) 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): get_auth_request con 0x7f3d0009c860 auth_method 0 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 10 monclient(hunting): get_auth_request con 0x7f3d0009e840 auth_method 0 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:09.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d04d3a640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d00136cb0 0x7f3d001356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d04d3a640 10 monclient(hunting): get_auth_request con 0x7f3d00136cb0 auth_method 0 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d04d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d04d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3d04d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient(hunting): handle_auth_done global_id 134120 payload 824 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 -- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d00136cb0 msgr2=0x7f3d001356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d00136cb0 0x7f3d001356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 -- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 msgr2=0x7f3d00135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d00135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient: _finish_hunting 0 2024-09-18T12:35:09.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 monclient: found mon.noname-c 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.066+0000 7f3cfffff640 1 -- 172.21.15.116:0/25218464 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf8009ae0 con 0x7f3d0009c860 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3cfffff640 1 -- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 msgr2=0x7f3d0006e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3cfffff640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 0x7f3d0006e200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b540 tx=0x7f3cf00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 1 -- 172.21.15.116:0/25218464 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0011840 con 0x7f3d0009c860 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 10 monclient: dump: 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 10 monclient: _finish_auth 0 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 1 -- 172.21.15.116:0/25218464 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf00119a0 con 0x7f3d0009c860 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3d0679d640 5 monclient: authenticate success, global_id 134120 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3d0679d640 10 monclient: _renew_subs 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3d0679d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf4009390 con 0x7f3d0009c860 2024-09-18T12:35:09.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 1 -- 172.21.15.116:0/25218464 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0011d60 con 0x7f3d0009c860 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3ce7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3ce7fff640 10 monclient: dump: 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:09.084 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.068+0000 7f3cfe7fc640 4 set_mon_vals no callback set 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3d0679d640 10 monclient: _renew_subs 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3d0679d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0006ba10 con 0x7f3d0009c860 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3ce7fff640 1 -- 172.21.15.116:0/25218464 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3cf0010590 con 0x7f3d0009c860 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3d0679d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3d0679d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3d0679d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.069+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0004bf60 con 0x7f3d0009c860 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.070+0000 7f3ce7fff640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cec038490 0x7f3cec03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.070+0000 7f3ce7fff640 1 -- 172.21.15.116:0/25218464 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3cf004c2d0 con 0x7f3d0009c860 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.070+0000 7f3cff7fe640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cec038490 0x7f3cec03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.070+0000 7f3cff7fe640 10 monclient: get_auth_request con 0x7f3cec038490 auth_method 0 2024-09-18T12:35:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.072+0000 7f3cff7fe640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cec038490 0x7f3cec03a940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3d00071740 tx=0x7f3cf4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.074+0000 7f3ce7fff640 1 -- 172.21.15.116:0/25218464 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3cf000f8a0 con 0x7f3d0009c860 2024-09-18T12:35:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.074+0000 7f3ce7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.074+0000 7f3ce7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:09.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.288+0000 7f3d0679d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:35:09.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.288+0000 7f3d0679d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:35:09.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.288+0000 7f3d0679d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:09.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.288+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3d0006d970 con 0x7f3d0009c860 2024-09-18T12:35:09.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.289+0000 7f3ce7fff640 1 -- 172.21.15.116:0/25218464 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3cf0015020 con 0x7f3d0009c860 2024-09-18T12:35:09.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.289+0000 7f3ce7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:35:09.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.289+0000 7f3ce7fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cec038490 msgr2=0x7f3cec03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cec038490 0x7f3cec03a940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3d00071740 tx=0x7f3cf4032000 comp rx=0 tx=0).stop 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 10 monclient: shutdown 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 msgr2=0x7f3d0006e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 0x7f3d0006e200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b540 tx=0x7f3cf00090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 shutdown_connections 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3cec038490 0x7f3cec03a940 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3d00136cb0 0x7f3d001356b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d0009e840 0x7f3d00135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 --2- 172.21.15.116:0/25218464 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3d0009c860 0x7f3d0006e200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 >> 172.21.15.116:0/25218464 conn(0x7f3d000a2e40 msgr2=0x7f3d00098a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 shutdown_connections 2024-09-18T12:35:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:09.292+0000 7f3d0679d640 1 -- 172.21.15.116:0/25218464 wait complete. 2024-09-18T12:35:09.295 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.293+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:09.295 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.293+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=1508494308,config=3140514009,health=1744959055,kv=2187923472,logm=3195460688}) 2024-09-18T12:35:09.295 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.293+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=795818418,config=3140514009,health=1744959055,kv=2187923472,logm=1789497744}) 2024-09-18T12:35:09.295 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.293+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:09.295 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.293+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=1508494308,config=3140514009,health=1744959055,kv=2187923472,logm=3195460688}) 2024-09-18T12:35:09.295 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.293+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=29942958,config=3140514009,health=1744959055,kv=2187923472,logm=1834720589}) 2024-09-18T12:35:09.303 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:35:09.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:09.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564029897}) 2024-09-18T12:35:09.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889755855}) 2024-09-18T12:35:09.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:09.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564029897}) 2024-09-18T12:35:09.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:09.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601067647}) 2024-09-18T12:35:10.137 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.136+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.137 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.136+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993707367}) 2024-09-18T12:35:10.137 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.136+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=208439095}) 2024-09-18T12:35:10.137 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.136+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.137 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.136+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993707367}) 2024-09-18T12:35:10.137 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.136+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3756634863}) 2024-09-18T12:35:10.148 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669999039}) 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918027241}) 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669999039}) 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2037637153}) 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995082328}) 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1740954915}) 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.149 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995082328}) 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.137+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515079194}) 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.138+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.138+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7905066}) 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.138+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2112202288}) 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.138+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.138+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7905066}) 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.138+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322196152}) 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.139+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.139+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762963712}) 2024-09-18T12:35:10.150 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.139+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2729101263}) 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.139+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.139+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762963712}) 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.139+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043585766}) 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.139+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389646451}) 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195514947}) 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389646451}) 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084846004}) 2024-09-18T12:35:10.151 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953575525}) 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3455340095}) 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953575525}) 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.140+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140234141}) 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.141+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.141+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853581021}) 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.141+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2062878206}) 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.141+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.152 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.141+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853581021}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.141+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171755877}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138993847}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733516950}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138993847}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.142+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2643308449}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597323629}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444942628}) 2024-09-18T12:35:10.153 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597323629}) 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300240992}) 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162292044}) 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=934992663}) 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162292044}) 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.143+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890633249}) 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287612273}) 2024-09-18T12:35:10.154 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603635507}) 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287612273}) 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.144+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1665312874}) 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394689831}) 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=269581781}) 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394689831}) 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.145+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=496182052}) 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.155 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730039591}) 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3883998588}) 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730039591}) 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1034718173}) 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643030604}) 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1799235933}) 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643030604}) 2024-09-18T12:35:10.156 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.146+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2358515797}) 2024-09-18T12:35:10.157 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.157 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532484167}) 2024-09-18T12:35:10.157 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4109676780}) 2024-09-18T12:35:10.157 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.157 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532484167}) 2024-09-18T12:35:10.157 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.147+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777950513}) 2024-09-18T12:35:10.168 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.148+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.168 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.148+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391410393}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.148+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2416576469}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.148+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.148+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391410393}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.148+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50415948}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475705874}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3700133110}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475705874}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=603597514}) 2024-09-18T12:35:10.169 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945817219}) 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2726251404}) 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945817219}) 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.149+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=284554649}) 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.150+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.150+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075262107}) 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.150+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3370426766}) 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.150+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.150+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075262107}) 2024-09-18T12:35:10.170 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.150+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=330048253}) 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820462812}) 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789697131}) 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820462812}) 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051428693}) 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809835111}) 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1857198048}) 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.171 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809835111}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.151+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2686171676}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.152+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.152+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945058967}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.152+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4029693082}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.152+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.152+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945058967}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.152+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972355102}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542260510}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1299231647}) 2024-09-18T12:35:10.172 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542260510}) 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321753532}) 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969293219}) 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614823852}) 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.153+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969293219}) 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959250232}) 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385080731}) 2024-09-18T12:35:10.173 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1124398200}) 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385080731}) 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.154+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913444127}) 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.155+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.155+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324373049}) 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.155+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128193249}) 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.155+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.155+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324373049}) 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.155+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1769963476}) 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.174 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466766252}) 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3912508015}) 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466766252}) 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=431429892}) 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163130523}) 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3090356383}) 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163130523}) 2024-09-18T12:35:10.175 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.156+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158399603}) 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.157+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.157+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705638913}) 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.157+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2341353219}) 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.157+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.157+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705638913}) 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.157+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291839149}) 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.158+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.158+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192535900}) 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.158+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2689159362}) 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.158+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.176 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.158+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192535900}) 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.158+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815467593}) 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400727195}) 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4223764767}) 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400727195}) 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516603987}) 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671985822}) 2024-09-18T12:35:10.177 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133495463}) 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671985822}) 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.159+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955521472}) 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.160+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.160+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772858007}) 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.160+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=892207453}) 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.160+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.160+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772858007}) 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.160+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541852597}) 2024-09-18T12:35:10.178 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103890169}) 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867284731}) 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103890169}) 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316697164}) 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571440082}) 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513206390}) 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.161+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.162+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571440082}) 2024-09-18T12:35:10.179 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.162+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534175528}) 2024-09-18T12:35:10.304 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:35:10.304 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.406+0000 7ff42995d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.406+0000 7ff42995d640 10 monclient: build_initial_monmap 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.406+0000 7ff42995d640 10 monclient: monmap: 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:10.406816+0000 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:10.406816+0000 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:10.408 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.408 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.406+0000 7ff42995d640 1 Processor -- start 2024-09-18T12:35:10.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258607810}) 2024-09-18T12:35:10.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830632845}) 2024-09-18T12:35:10.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258607810}) 2024-09-18T12:35:10.409 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.406+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1706757766}) 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.406+0000 7ff42995d640 1 -- start start 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.406+0000 7ff42995d640 10 monclient: init 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff424131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff42409e910 0x7ff42409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient(hunting): picked mon.noname-a con 0x7ff42409e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff42409f250 0x7ff424138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient(hunting): picked mon.noname-b con 0x7ff42409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42409faf0 con 0x7ff424131ab0 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff424132590 con 0x7ff42409e910 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42409fd30 con 0x7ff42409f250 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff42995d640 10 monclient(hunting): authenticate will time out at 1337.728456s 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff42409e910 0x7ff42409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff422ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff424131ab0 0x7ff424131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41590/0 (socket says 172.21.15.116:41590) 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): get_auth_request con 0x7ff42409e910 auth_method 0 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff422ffd640 1 -- 172.21.15.116:0/1237538925 learned_addr learned my addr 172.21.15.116:0/1237538925 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 1 --2- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff42409f250 0x7ff424138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 10 monclient(hunting): get_auth_request con 0x7ff42409f250 auth_method 0 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient(hunting): handle_auth_done global_id 104820 payload 274 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 1 -- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff42409f250 msgr2=0x7ff424138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 1 --2- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff42409f250 0x7ff424138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 1 -- 172.21.15.116:0/1237538925 >> v1:172.21.15.116:6789/0 conn(0x7ff424131ab0 legacy=0x7ff424131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:10.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient: _finish_hunting 0 2024-09-18T12:35:10.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 1 monclient: found mon.noname-a 2024-09-18T12:35:10.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:10.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.407+0000 7ff4227fc640 1 -- 172.21.15.116:0/1237538925 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240a1110 con 0x7ff42409e910 2024-09-18T12:35:10.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742477665}) 2024-09-18T12:35:10.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697565914}) 2024-09-18T12:35:10.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742477665}) 2024-09-18T12:35:10.416 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.409+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237578602}) 2024-09-18T12:35:10.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.410+0000 7ff4227fc640 1 -- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 msgr2=0x7ff42409ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.410+0000 7ff4227fc640 1 --2- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff42409ed10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff418009850 tx=0x7ff4180090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a45329460e3ef35 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560113568}) 2024-09-18T12:35:10.418 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612070523}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=560113568}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.410+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2503773063}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795126992}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4092047105}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795126992}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.411+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415211052}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446633205}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225558764}) 2024-09-18T12:35:10.419 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff421ffb640 1 -- 172.21.15.116:0/1237538925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41802ae60 con 0x7ff42409e910 2024-09-18T12:35:10.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff421ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.420 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446633205}) 2024-09-18T12:35:10.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff421ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:10.421 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702193077}) 2024-09-18T12:35:10.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff421ffb640 10 monclient: dump: 2024-09-18T12:35:10.421 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.421 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.421 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.421 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff421ffb640 10 monclient: _finish_auth 0 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff421ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff42995d640 5 monclient: authenticate success, global_id 104820 2024-09-18T12:35:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.412+0000 7ff42995d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:10.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042664143}) 2024-09-18T12:35:10.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348188711}) 2024-09-18T12:35:10.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.412+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042664143}) 2024-09-18T12:35:10.423 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119244009}) 2024-09-18T12:35:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff421ffb640 1 -- 172.21.15.116:0/1237538925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff418032440 con 0x7ff42409e910 2024-09-18T12:35:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff421ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff42995d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff421ffb640 1 -- 172.21.15.116:0/1237538925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4180327e0 con 0x7ff42409e910 2024-09-18T12:35:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff421ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff42995d640 4 set_mon_vals no callback set 2024-09-18T12:35:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff421ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:10.424 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.425 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806877937}) 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.413+0000 7ff421ffb640 10 monclient: dump: 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.426 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.426 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3862199066}) 2024-09-18T12:35:10.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806877937}) 2024-09-18T12:35:10.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.413+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228277030}) 2024-09-18T12:35:10.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 10 monclient: shutdown 2024-09-18T12:35:10.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:10.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 -- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 msgr2=0x7ff42409ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 --2- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff42409ed10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff418009850 tx=0x7ff4180090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 -- 172.21.15.116:0/1237538925 shutdown_connections 2024-09-18T12:35:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 --2- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff42409f250 0x7ff424138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 --2- 172.21.15.116:0/1237538925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff42409ed10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 -- 172.21.15.116:0/1237538925 >> 172.21.15.116:0/1237538925 conn(0x7ff424067d30 msgr2=0x7ff4240a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:10.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 -- 172.21.15.116:0/1237538925 shutdown_connections 2024-09-18T12:35:10.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731396651}) 2024-09-18T12:35:10.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1868492678}) 2024-09-18T12:35:10.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731396651}) 2024-09-18T12:35:10.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.414+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=303778288}) 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.414+0000 7ff42995d640 1 -- 172.21.15.116:0/1237538925 wait complete. 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.415+0000 7ff42995d640 10 monclient: build_initial_monmap 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.415+0000 7ff42995d640 10 monclient: monmap: 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:10.430 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:10.431 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.431 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.431 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250282278}) 2024-09-18T12:35:10.431 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160229984}) 2024-09-18T12:35:10.431 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.432 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250282278}) 2024-09-18T12:35:10.432 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.415+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781740770}) 2024-09-18T12:35:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.415+0000 7ff42995d640 1 Processor -- start 2024-09-18T12:35:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.415+0000 7ff42995d640 1 -- start start 2024-09-18T12:35:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.415+0000 7ff42995d640 10 monclient: init 2024-09-18T12:35:10.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425526406}) 2024-09-18T12:35:10.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821383011}) 2024-09-18T12:35:10.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425526406}) 2024-09-18T12:35:10.433 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.416+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291677877}) 2024-09-18T12:35:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:35:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff424134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): picked mon.noname-a con 0x7ff42409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff42409f250 0x7ff424134780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): picked mon.noname-b con 0x7ff42409f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff424131ab0 0x7ff424134cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): picked mon.noname-c con 0x7ff424131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff424131670 con 0x7ff42409e910 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42409bde0 con 0x7ff42409f250 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4241387d0 con 0x7ff424131ab0 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff42995d640 10 monclient(hunting): authenticate will time out at 1337.737597s 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff424134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff424131ab0 0x7ff424134cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff424134240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45342/0 (socket says 172.21.15.116:45342) 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 1 -- 172.21.15.116:0/3430533919 learned_addr learned my addr 172.21.15.116:0/3430533919 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 10 monclient(hunting): get_auth_request con 0x7ff42409e910 auth_method 0 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 10 monclient(hunting): get_auth_request con 0x7ff424131ab0 auth_method 0 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff42409f250 0x7ff424134780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 10 monclient(hunting): get_auth_request con 0x7ff42409f250 auth_method 0 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff422ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.416+0000 7ff4227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 10 monclient(hunting): handle_auth_done global_id 104823 payload 824 2024-09-18T12:35:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 1 -- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff424131ab0 msgr2=0x7ff424134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff424131ab0 0x7ff424134cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 1 -- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff42409f250 msgr2=0x7ff424134780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff42409f250 0x7ff424134780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329687609}) 2024-09-18T12:35:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:35:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 1 monclient: found mon.noname-a 2024-09-18T12:35:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff422ffd640 1 -- 172.21.15.116:0/3430533919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff418009390 con 0x7ff42409e910 2024-09-18T12:35:10.440 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916591682}) 2024-09-18T12:35:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.417+0000 7ff4237fe640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff424131ab0 0x7ff424134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:35:10.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329687609}) 2024-09-18T12:35:10.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935399411}) 2024-09-18T12:35:10.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472429738}) 2024-09-18T12:35:10.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1467689656}) 2024-09-18T12:35:10.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472429738}) 2024-09-18T12:35:10.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.417+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881116734}) 2024-09-18T12:35:10.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff422ffd640 1 -- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 msgr2=0x7ff424134240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff422ffd640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff424134240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff41400c5e0 tx=0x7ff414004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 1 -- 172.21.15.116:0/3430533919 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414005cb0 con 0x7ff42409e910 2024-09-18T12:35:10.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: dump: 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: _finish_auth 0 2024-09-18T12:35:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff42995d640 5 monclient: authenticate success, global_id 104823 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 1 -- 172.21.15.116:0/3430533919 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414005e30 con 0x7ff42409e910 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 1 -- 172.21.15.116:0/3430533919 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414007340 con 0x7ff42409e910 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff42995d640 10 monclient: _renew_subs 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff42995d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff424137650 con 0x7ff42409e910 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff40affd640 10 monclient: dump: 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff4217fa640 4 set_mon_vals no callback set 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff42995d640 10 monclient: _renew_subs 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff42995d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.419+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff424135200 con 0x7ff42409e910 2024-09-18T12:35:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff40affd640 1 -- 172.21.15.116:0/3430533919 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff41401b400 con 0x7ff42409e910 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff40affd640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff410038490 0x7ff41003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff40affd640 1 -- 172.21.15.116:0/3430533919 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff41404be50 con 0x7ff42409e910 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff42995d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff42995d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff42995d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42404bf60 con 0x7ff42409e910 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff4227fc640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff410038490 0x7ff41003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.420+0000 7ff4227fc640 10 monclient: get_auth_request con 0x7ff410038490 auth_method 0 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.423+0000 7ff4227fc640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff410038490 0x7ff41003a940 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff424076760 tx=0x7ff418032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.425+0000 7ff40affd640 1 -- 172.21.15.116:0/3430533919 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff41401b6b0 con 0x7ff42409e910 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.425+0000 7ff40affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.425+0000 7ff40affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:10.449 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.439+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.449 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.439+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2195746062,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1641904935}) 2024-09-18T12:35:10.449 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.439+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13} crc {logm=2165409888,mdsmap=1531151339,mgr=1447842296,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4155961732}) 2024-09-18T12:35:10.449 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.440+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.449 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.440+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=2195746062,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1641904935}) 2024-09-18T12:35:10.450 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.440+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=1} crc {logm=2897757008,mdsmap=1531151339,mgr=2854130551,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2517311440,mon_config_key=494987680}) 2024-09-18T12:35:10.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.614 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mgrstat=3220354832,mon_config_key=3655569570,monmap=3140514009,osd_metadata=687973059,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2070037479}) 2024-09-18T12:35:10.615 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=6,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mgrstat=110968184,mon_config_key=3844344687,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2320921181}) 2024-09-18T12:35:10.615 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.615 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=78} crc {mgrstat=3220354832,mon_config_key=3655569570,monmap=3140514009,osd_metadata=687973059,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2070037479}) 2024-09-18T12:35:10.615 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.613+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=76} crc {mgrstat=1779333896,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2298182905}) 2024-09-18T12:35:10.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.627+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.627+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=3459135503}) 2024-09-18T12:35:10.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.627+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=48} crc {osdmap=4247249094}) 2024-09-18T12:35:10.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.627+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:35:10.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.627+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=3459135503}) 2024-09-18T12:35:10.628 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:35:10.627+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=244150779}) 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.651+0000 7ff42995d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.651+0000 7ff42995d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.651+0000 7ff42995d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.651+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff4240a2620 con 0x7ff42409e910 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.651+0000 7ff40affd640 1 -- 172.21.15.116:0/3430533919 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7ff41401a360 con 0x7ff42409e910 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.651+0000 7ff40affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.651+0000 7ff40affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:10.652 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff410038490 msgr2=0x7ff41003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff410038490 0x7ff41003a940 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff424076760 tx=0x7ff418032000 comp rx=0 tx=0).stop 2024-09-18T12:35:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 10 monclient: shutdown 2024-09-18T12:35:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 msgr2=0x7ff424134240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff424134240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff41400c5e0 tx=0x7ff414004960 comp rx=0 tx=0).stop 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 shutdown_connections 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff410038490 0x7ff41003a940 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff424131ab0 0x7ff424134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff42409f250 0x7ff424134780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 --2- 172.21.15.116:0/3430533919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff42409e910 0x7ff424134240 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 >> 172.21.15.116:0/3430533919 conn(0x7ff424067d30 msgr2=0x7ff4240a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 shutdown_connections 2024-09-18T12:35:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.653+0000 7ff42995d640 1 -- 172.21.15.116:0/3430533919 wait complete. 2024-09-18T12:35:10.664 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:35:10.665 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:35:10.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:35:10.665 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:35:10.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.757+0000 7f13b6f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:10.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.757+0000 7f13b6f9d640 10 monclient: build_initial_monmap 2024-09-18T12:35:10.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.757+0000 7f13b6f9d640 10 monclient: monmap: 2024-09-18T12:35:10.758 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:10.758394+0000 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:10.758394+0000 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:10.759 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:10.760 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 1 Processor -- start 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 1 -- start start 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient: init 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13b0096e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f13b00971e0 0x7f13b00975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f13b00971e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13b0133b30 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0099620 con 0x7f13b0133b30 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0099340 con 0x7f13b0096e10 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b0099900 con 0x7f13b00971e0 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.758+0000 7f13b6f9d640 10 monclient(hunting): authenticate will time out at 1338.079932s 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13b553a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13b0133b30 0x7f13b0135f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41592/0 (socket says 172.21.15.116:41592) 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13b553a640 1 -- 172.21.15.116:0/2741561065 learned_addr learned my addr 172.21.15.116:0/2741561065 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 1 --2- 172.21.15.116:0/2741561065 >> v2:172.21.15.144:3300/0 conn(0x7f13b00971e0 0x7f13b00975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): get_auth_request con 0x7f13b00971e0 auth_method 0 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient(hunting): handle_auth_done global_id 124339 payload 274 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 1 -- 172.21.15.116:0/2741561065 >> v1:172.21.15.144:6789/0 conn(0x7f13b0096e10 legacy=0x7f13b009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 1 -- 172.21.15.116:0/2741561065 >> v1:172.21.15.116:6789/0 conn(0x7f13b0133b30 legacy=0x7f13b0135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient: _finish_hunting 0 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 1 monclient: found mon.noname-c 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:35:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.759+0000 7f13affff640 1 -- 172.21.15.116:0/2741561065 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b009ac70 con 0x7f13b00971e0 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13affff640 1 -- 172.21.15.116:0/2741561065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 msgr2=0x7f13b00975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13affff640 1 --2- 172.21.15.116:0/2741561065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 0x7f13b00975e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f139c009850 tx=0x7f139c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91aaec6b126578e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 1 -- 172.21.15.116:0/2741561065 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c02ae70 con 0x7f13b00971e0 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: dump: 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: _finish_auth 0 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 1 -- 172.21.15.116:0/2741561065 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c032440 con 0x7f13b00971e0 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13b6f9d640 5 monclient: authenticate success, global_id 124339 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13b6f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13b6f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 1 -- 172.21.15.116:0/2741561065 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c0327e0 con 0x7f13b00971e0 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13b6f9d640 4 set_mon_vals no callback set 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.762+0000 7f13af7fe640 10 monclient: dump: 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 10 monclient: shutdown 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 1 -- 172.21.15.116:0/2741561065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 msgr2=0x7f13b00975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 1 --2- 172.21.15.116:0/2741561065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 0x7f13b00975e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f139c009850 tx=0x7f139c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 1 -- 172.21.15.116:0/2741561065 shutdown_connections 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 1 --2- 172.21.15.116:0/2741561065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 0x7f13b00975e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 1 -- 172.21.15.116:0/2741561065 >> 172.21.15.116:0/2741561065 conn(0x7f13b00672b0 msgr2=0x7f13b00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 1 -- 172.21.15.116:0/2741561065 shutdown_connections 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 1 -- 172.21.15.116:0/2741561065 wait complete. 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 10 monclient: build_initial_monmap 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.763+0000 7f13b6f9d640 10 monclient: monmap: 2024-09-18T12:35:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 Processor -- start 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 -- start start 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient: init 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0096e10 0x7f13b0131750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f13b0096e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 0x7f13b0131c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f13b00971e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b0133b30 0x7f13b01321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f13b0133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b012b3e0 con 0x7f13b0096e10 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b01332b0 con 0x7f13b00971e0 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b009b050 con 0x7f13b0133b30 2024-09-18T12:35:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b6f9d640 10 monclient(hunting): authenticate will time out at 1338.085879s 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b4d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0096e10 0x7f13b0131750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b0133b30 0x7f13b01321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.764+0000 7f13b4d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0096e10 0x7f13b0131750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45344/0 (socket says 172.21.15.116:45344) 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 1 -- 172.21.15.116:0/3226816664 learned_addr learned my addr 172.21.15.116:0/3226816664 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): get_auth_request con 0x7f13b0096e10 auth_method 0 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 10 monclient(hunting): get_auth_request con 0x7f13b0133b30 auth_method 0 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13affff640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 0x7f13b0131c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient(hunting): handle_auth_done global_id 104826 payload 824 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b0133b30 msgr2=0x7f13b01321d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b0133b30 0x7f13b01321d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 msgr2=0x7f13b0131c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b00971e0 0x7f13b0131c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient: _finish_hunting 0 2024-09-18T12:35:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 1 monclient: found mon.noname-a 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b4d39640 1 -- 172.21.15.116:0/3226816664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c009390 con 0x7f13b0096e10 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.765+0000 7f13b553a640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b0133b30 0x7f13b01321d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b4d39640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0096e10 msgr2=0x7f13b0131750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b4d39640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0096e10 0x7f13b0131750 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c5e0 tx=0x7f13a4007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4012840 con 0x7f13b0096e10 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: dump: 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: _finish_auth 0 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a40129a0 con 0x7f13b0096e10 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b6f9d640 5 monclient: authenticate success, global_id 104826 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4012da0 con 0x7f13b0096e10 2024-09-18T12:35:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b6f9d640 10 monclient: _renew_subs 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b6f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b012b160 con 0x7f13b0096e10 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13aeffd640 4 set_mon_vals no callback set 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13acff9640 10 monclient: dump: 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b6f9d640 10 monclient: _renew_subs 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b6f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.767+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b012bf60 con 0x7f13b0096e10 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13b6f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13b6f9d640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 0x7f13b0132710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13b6f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b0065f20 con 0x7f13b012a1a0 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f13a4011590 con 0x7f13b0096e10 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 0x7f13b0132710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient: get_auth_request con 0x7f13b012a1a0 auth_method 0 2024-09-18T12:35:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13acff9640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1394038490 0x7f139403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f13a404c450 con 0x7f13b0096e10 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 10 monclient(hunting): handle_auth_done global_id 104829 payload 824 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.768+0000 7f13affff640 1 -- 172.21.15.116:0/3226816664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13b004bf60 con 0x7f13b012a1a0 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.769+0000 7f13b553a640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1394038490 0x7f139403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.769+0000 7f13b553a640 10 monclient: get_auth_request con 0x7f1394038490 auth_method 0 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13affff640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 msgr2=0x7f13b0132710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13affff640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 0x7f13b0132710 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f13b0076be0 tx=0x7f139c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c03d8a0 con 0x7f13b012a1a0 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 10 monclient: dump: 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f139c03da00 con 0x7f13b012a1a0 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 msgr2=0x7f13b0132710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 0x7f13b0132710 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f13b0076be0 tx=0x7f139c032000 comp rx=0 tx=0).stop 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 msgr2=0x7f13b0132710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13acff9640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012a1a0 0x7f13b0132710 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f13b0076be0 tx=0x7f139c032000 comp rx=0 tx=0).stop 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.771+0000 7f13b553a640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1394038490 0x7f139403a940 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f13a800d6a0 tx=0x7f13a8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13b6f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13b6f9d640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 0x7f13b01351d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13b6f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00638d0 con 0x7f13b012b620 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 0x7f13b01351d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient: get_auth_request con 0x7f13b012b620 auth_method 0 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 10 monclient(hunting): handle_auth_done global_id 104832 payload 824 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.777+0000 7f13affff640 1 -- 172.21.15.116:0/3226816664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f13b0135710 con 0x7f13b012b620 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13affff640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 msgr2=0x7f13b01351d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13affff640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 0x7f13b01351d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f139c009850 tx=0x7f139c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c03c8d0 con 0x7f13b012b620 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 10 monclient: dump: 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f139c03ca30 con 0x7f13b012b620 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 msgr2=0x7f13b01351d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 0x7f13b01351d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f139c009850 tx=0x7f139c03d000 comp rx=0 tx=0).stop 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 msgr2=0x7f13b01351d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.779+0000 7f13acff9640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b012b620 0x7f13b01351d0 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f139c009850 tx=0x7f139c03d000 comp rx=0 tx=0).stop 2024-09-18T12:35:10.791 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 98, 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:10.792 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:10.793 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:10.794 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:10.801 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.802 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.804 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:10.804 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1394038490 msgr2=0x7f139403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1394038490 0x7f139403a940 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f13a800d6a0 tx=0x7f13a8009040 comp rx=0 tx=0).stop 2024-09-18T12:35:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 10 monclient: shutdown 2024-09-18T12:35:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0096e10 msgr2=0x7f13b0131750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 --2- 172.21.15.116:0/3226816664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0096e10 0x7f13b0131750 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c5e0 tx=0x7f13a4007ed0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b553a640 1 -- 172.21.15.116:0/3226816664 reap_dead start 2024-09-18T12:35:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b553a640 1 -- 172.21.15.116:0/3226816664 reap_dead start 2024-09-18T12:35:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 shutdown_connections 2024-09-18T12:35:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 >> 172.21.15.116:0/3226816664 conn(0x7f13b00672b0 msgr2=0x7f13b009ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 shutdown_connections 2024-09-18T12:35:10.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.780+0000 7f13b6f9d640 1 -- 172.21.15.116:0/3226816664 wait complete. 2024-09-18T12:35:10.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:35:10.806 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:35:10.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.899+0000 7fdecb59d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.899+0000 7fdecb59d640 10 monclient: build_initial_monmap 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.899+0000 7fdecb59d640 10 monclient: monmap: 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:10.899778+0000 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:10.899778+0000 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:10.900 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:10.901 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.899+0000 7fdecb59d640 1 Processor -- start 2024-09-18T12:35:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.899+0000 7fdecb59d640 1 -- start start 2024-09-18T12:35:10.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.899+0000 7fdecb59d640 10 monclient: init 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdec41318f0 0x7fdec4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient(hunting): picked mon.noname-c con 0x7fdec41318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdec409ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 0x7fdec4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient(hunting): picked mon.noname-b con 0x7fdec4136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec4131cc0 con 0x7fdec409ea50 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec409f530 con 0x7fdec41318f0 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec409fbf0 con 0x7fdec4136c20 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdecb59d640 10 monclient(hunting): authenticate will time out at 1338.221535s 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 0x7fdec4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 0x7fdec4139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54444/0 (socket says 172.21.15.116:54444) 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 -- 172.21.15.116:0/1205597610 learned_addr learned my addr 172.21.15.116:0/1205597610 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): get_auth_request con 0x7fdec4136c20 auth_method 0 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9339640 1 --2- 172.21.15.116:0/1205597610 >> v2:172.21.15.144:3300/0 conn(0x7fdec41318f0 0x7fdec4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient(hunting): handle_auth_done global_id 134132 payload 274 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 -- 172.21.15.116:0/1205597610 >> v2:172.21.15.144:3300/0 conn(0x7fdec41318f0 msgr2=0x7fdec4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 --2- 172.21.15.116:0/1205597610 >> v2:172.21.15.144:3300/0 conn(0x7fdec41318f0 0x7fdec4131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 -- 172.21.15.116:0/1205597610 >> v1:172.21.15.144:6789/0 conn(0x7fdec409ea50 legacy=0x7fdec409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 monclient: found mon.noname-b 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9b3a640 1 -- 172.21.15.116:0/1205597610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec409fe30 con 0x7fdec4136c20 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9339640 10 monclient: get_auth_request con 0x7fdec41318f0 auth_method 0 2024-09-18T12:35:10.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.900+0000 7fdec9339640 1 --2- 172.21.15.116:0/1205597610 >> v2:172.21.15.144:3300/0 conn(0x7fdec41318f0 0x7fdec4131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdec9b3a640 1 -- 172.21.15.116:0/1205597610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 msgr2=0x7fdec4139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdec9b3a640 1 --2- 172.21.15.116:0/1205597610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 0x7fdec4139000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009f20 tx=0x7fdebc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49bbbc48619ed8b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 1 -- 172.21.15.116:0/1205597610 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc007610 con 0x7fdec4136c20 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: dump: 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.912 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: _finish_auth 0 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 1 -- 172.21.15.116:0/1205597610 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc02d050 con 0x7fdec4136c20 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdecb59d640 5 monclient: authenticate success, global_id 134132 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdecb59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:10.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdecb59d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdecb59d640 4 set_mon_vals no callback set 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 1 -- 172.21.15.116:0/1205597610 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0079a0 con 0x7fdec4136c20 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.903+0000 7fdebbfff640 10 monclient: dump: 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 10 monclient: shutdown 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 -- 172.21.15.116:0/1205597610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 msgr2=0x7fdec4139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 --2- 172.21.15.116:0/1205597610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 0x7fdec4139000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009f20 tx=0x7fdebc009840 comp rx=0 tx=0).stop 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 -- 172.21.15.116:0/1205597610 shutdown_connections 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 --2- 172.21.15.116:0/1205597610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec4136c20 0x7fdec4139000 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 --2- 172.21.15.116:0/1205597610 >> v2:172.21.15.144:3300/0 conn(0x7fdec41318f0 0x7fdec4131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 -- 172.21.15.116:0/1205597610 >> 172.21.15.116:0/1205597610 conn(0x7fdec4067110 msgr2=0x7fdec4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 -- 172.21.15.116:0/1205597610 shutdown_connections 2024-09-18T12:35:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 1 -- 172.21.15.116:0/1205597610 wait complete. 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 10 monclient: build_initial_monmap 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.904+0000 7fdecb59d640 10 monclient: monmap: 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 Processor -- start 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 -- start start 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient: init 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409ea50 0x7fdec411da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): picked mon.noname-c con 0x7fdec409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec41318f0 0x7fdec41306d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): picked mon.noname-a con 0x7fdec41318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec4136c20 0x7fdec4099420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): picked mon.noname-b con 0x7fdec4136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec411c780 con 0x7fdec41318f0 2024-09-18T12:35:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec409ddf0 con 0x7fdec4136c20 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec409bf10 con 0x7fdec409ea50 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.905+0000 7fdecb59d640 10 monclient(hunting): authenticate will time out at 1338.226956s 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409ea50 0x7fdec411da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec41318f0 0x7fdec41306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409ea50 0x7fdec411da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54448/0 (socket says 172.21.15.116:54448) 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 -- 172.21.15.116:0/1591637823 learned_addr learned my addr 172.21.15.116:0/1591637823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): get_auth_request con 0x7fdec409ea50 auth_method 0 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 10 monclient(hunting): get_auth_request con 0x7fdec41318f0 auth_method 0 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient(hunting): handle_auth_done global_id 134135 payload 824 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec4136c20 msgr2=0x7fdec4099420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdec4136c20 0x7fdec4099420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec41318f0 msgr2=0x7fdec41306d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec41318f0 0x7fdec41306d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient: _finish_hunting 0 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 monclient: found mon.noname-c 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec9339640 1 -- 172.21.15.116:0/1591637823 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc009ae0 con 0x7fdec409ea50 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.906+0000 7fdec8b38640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec41318f0 0x7fdec41306d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdec9339640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409ea50 msgr2=0x7fdec411da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdec9339640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409ea50 0x7fdec411da20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdec000c880 tx=0x7fdec0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec000ce10 con 0x7fdec409ea50 2024-09-18T12:35:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: dump: 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: _finish_auth 0 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdecb59d640 5 monclient: authenticate success, global_id 134135 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec0007aa0 con 0x7fdec409ea50 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdecb59d640 10 monclient: _renew_subs 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdecb59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec409afe0 con 0x7fdec409ea50 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0007e60 con 0x7fdec409ea50 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdeb97fa640 10 monclient: dump: 2024-09-18T12:35:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdebb7fe640 4 set_mon_vals no callback set 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdecb59d640 10 monclient: _renew_subs 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdecb59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.908+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec4130c10 con 0x7fdec409ea50 2024-09-18T12:35:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdecb59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdecb59d640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 0x7fdec40298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdecb59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec4065e00 con 0x7fdec409a120 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdec0011600 con 0x7fdec409ea50 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 0x7fdec40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient: get_auth_request con 0x7fdec409a120 auth_method 0 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 10 monclient(hunting): handle_auth_done global_id 134138 payload 824 2024-09-18T12:35:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdec8b38640 1 -- 172.21.15.116:0/1591637823 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdec404bf60 con 0x7fdec409a120 2024-09-18T12:35:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdeb97fa640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdea8038490 0x7fdea803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.909+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdec004c390 con 0x7fdec409ea50 2024-09-18T12:35:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.910+0000 7fdec9b3a640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdea8038490 0x7fdea803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.910+0000 7fdec9b3a640 10 monclient: get_auth_request con 0x7fdea8038490 auth_method 0 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdec8b38640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 msgr2=0x7fdec40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdec8b38640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 0x7fdec40298c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdec411b7d0 tx=0x7fdeb4007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb400dce0 con 0x7fdec409a120 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 10 monclient: dump: 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdeb400de40 con 0x7fdec409a120 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 msgr2=0x7fdec40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 0x7fdec40298c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdec411b7d0 tx=0x7fdeb4007e10 comp rx=0 tx=0).stop 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 msgr2=0x7fdec40298c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdeb97fa640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a120 0x7fdec40298c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.912+0000 7fdec9b3a640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdea8038490 0x7fdea803a940 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdec406f600 tx=0x7fdebc036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdecb59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdecb59d640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 0x7fdec4029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdecb59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec411b840 con 0x7fdec409a4f0 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 0x7fdec4029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient: get_auth_request con 0x7fdec409a4f0 auth_method 0 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 10 monclient(hunting): handle_auth_done global_id 134141 payload 824 2024-09-18T12:35:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.918+0000 7fdec8b38640 1 -- 172.21.15.116:0/1591637823 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdec411c9a0 con 0x7fdec409a4f0 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdec8b38640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 msgr2=0x7fdec4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdec8b38640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 0x7fdec4029e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdec40a44f0 tx=0x7fdeb4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb401ee10 con 0x7fdec409a4f0 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 10 monclient: dump: 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fdeb400dad0 con 0x7fdec409a4f0 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 msgr2=0x7fdec4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 0x7fdec4029e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdec40a44f0 tx=0x7fdeb4017000 comp rx=0 tx=0).stop 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 msgr2=0x7fdec4029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.920+0000 7fdeb97fa640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409a4f0 0x7fdec4029e00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:10.932 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 98, 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:35:10.933 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:10.934 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:10.935 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:10.936 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:10.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:10.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:10.942 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:10.943 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:10.944 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdea8038490 msgr2=0x7fdea803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdea8038490 0x7fdea803a940 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdec406f600 tx=0x7fdebc036000 comp rx=0 tx=0).stop 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 10 monclient: shutdown 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409ea50 msgr2=0x7fdec411da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 --2- 172.21.15.116:0/1591637823 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdec409ea50 0x7fdec411da20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdec000c880 tx=0x7fdec0008040 comp rx=0 tx=0).stop 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdec9b3a640 1 -- 172.21.15.116:0/1591637823 reap_dead start 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdec9b3a640 1 -- 172.21.15.116:0/1591637823 reap_dead start 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 shutdown_connections 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 >> 172.21.15.116:0/1591637823 conn(0x7fdec4067110 msgr2=0x7fdec40a3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 shutdown_connections 2024-09-18T12:35:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:10.921+0000 7fdecb59d640 1 -- 172.21.15.116:0/1591637823 wait complete. 2024-09-18T12:35:10.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:35:10.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:35:10.947 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.042+0000 7f4529f5d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.042+0000 7f4529f5d640 10 monclient: build_initial_monmap 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.042+0000 7f4529f5d640 10 monclient: monmap: 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:11.042910+0000 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:11.042910+0000 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:11.044 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 1 Processor -- start 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 1 -- start start 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient: init 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4524131ab0 0x7f4524131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4524131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f452409e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4524136bb0 0x7f4524138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f4524136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452409faf0 con 0x7f452409e910 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452409fd30 con 0x7f4524131ab0 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45240a1110 con 0x7f4524136bb0 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f4529f5d640 10 monclient(hunting): authenticate will time out at 1338.364722s 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4524131ab0 0x7f4524131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f451bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f452409e910 0x7f452409ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41594/0 (socket says 172.21.15.116:41594) 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 10 monclient(hunting): get_auth_request con 0x7f4524131ab0 auth_method 0 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f451bfff640 1 -- 172.21.15.116:0/398738940 learned_addr learned my addr 172.21.15.116:0/398738940 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.043+0000 7f45237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f4523fff640 1 --2- 172.21.15.116:0/398738940 >> v2:172.21.15.144:3300/0 conn(0x7f4524136bb0 0x7f4524138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 10 monclient(hunting): handle_auth_done global_id 104838 payload 274 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 1 -- 172.21.15.116:0/398738940 >> v2:172.21.15.144:3300/0 conn(0x7f4524136bb0 msgr2=0x7f4524138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 1 --2- 172.21.15.116:0/398738940 >> v2:172.21.15.144:3300/0 conn(0x7f4524136bb0 0x7f4524138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 1 -- 172.21.15.116:0/398738940 >> v1:172.21.15.116:6789/0 conn(0x7f452409e910 legacy=0x7f452409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 10 monclient: _finish_hunting 0 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 1 monclient: found mon.noname-a 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:11.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:11.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.044+0000 7f45237fe640 1 -- 172.21.15.116:0/398738940 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45240a13f0 con 0x7f4524131ab0 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.046+0000 7f45237fe640 1 -- 172.21.15.116:0/398738940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 msgr2=0x7f4524131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.046+0000 7f45237fe640 1 --2- 172.21.15.116:0/398738940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 0x7f4524131e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4514009850 tx=0x7f45140090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3859b71435f4d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.046+0000 7f4522ffd640 1 -- 172.21.15.116:0/398738940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514032440 con 0x7f4524131ab0 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.046+0000 7f4522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.046+0000 7f4522ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.046+0000 7f4522ffd640 10 monclient: dump: 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.055 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.046+0000 7f4522ffd640 10 monclient: _finish_auth 0 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4522ffd640 1 -- 172.21.15.116:0/398738940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45140325a0 con 0x7f4524131ab0 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 5 monclient: authenticate success, global_id 104838 2024-09-18T12:35:11.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4522ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 4 set_mon_vals no callback set 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4522ffd640 1 -- 172.21.15.116:0/398738940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514032980 con 0x7f4524131ab0 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4522ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4522ffd640 10 monclient: dump: 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.057 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 10 monclient: shutdown 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 -- 172.21.15.116:0/398738940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 msgr2=0x7f4524131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 --2- 172.21.15.116:0/398738940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 0x7f4524131e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4514009850 tx=0x7f45140090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 -- 172.21.15.116:0/398738940 shutdown_connections 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 --2- 172.21.15.116:0/398738940 >> v2:172.21.15.144:3300/0 conn(0x7f4524136bb0 0x7f4524138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 --2- 172.21.15.116:0/398738940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 0x7f4524131e80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 -- 172.21.15.116:0/398738940 >> 172.21.15.116:0/398738940 conn(0x7f4524067d30 msgr2=0x7f45240a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 -- 172.21.15.116:0/398738940 shutdown_connections 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.047+0000 7f4529f5d640 1 -- 172.21.15.116:0/398738940 wait complete. 2024-09-18T12:35:11.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient: build_initial_monmap 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient: monmap: 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 Processor -- start 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 -- start start 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient: init 2024-09-18T12:35:11.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 0x7f45240975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f452409e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 0x7f4524097b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4524131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524136bb0 0x7f4524098050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f4524136bb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524130c60 con 0x7f4524131ab0 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45240a2680 con 0x7f4524136bb0 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452409af90 con 0x7f452409e910 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:11.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f4529f5d640 10 monclient(hunting): authenticate will time out at 1338.369882s 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f451bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 0x7f4524097b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f45237fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 0x7f45240975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f45237fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 0x7f45240975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54476/0 (socket says 172.21.15.116:54476) 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.048+0000 7f451bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 0x7f4524097b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45386/0 (socket says 172.21.15.116:45386) 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 1 -- 172.21.15.116:0/3688413894 learned_addr learned my addr 172.21.15.116:0/3688413894 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): get_auth_request con 0x7f452409e910 auth_method 0 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f451bfff640 10 monclient(hunting): get_auth_request con 0x7f4524131ab0 auth_method 0 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f451bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f451bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f451bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f451bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f4523fff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524136bb0 0x7f4524098050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient(hunting): handle_auth_done global_id 134144 payload 824 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524136bb0 msgr2=0x7f4524098050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524136bb0 0x7f4524098050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 msgr2=0x7f4524097b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4524131ab0 0x7f4524097b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient: _finish_hunting 0 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 1 monclient: found mon.noname-c 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f45237fe640 1 -- 172.21.15.116:0/3688413894 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4514009390 con 0x7f452409e910 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f4523fff640 10 monclient: get_auth_request con 0x7f4524136bb0 auth_method 0 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.049+0000 7f4523fff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524136bb0 0x7f4524098050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f45237fe640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 msgr2=0x7f45240975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f45237fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 0x7f45240975d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4514009850 tx=0x7f4514032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514031c70 con 0x7f452409e910 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: dump: 2024-09-18T12:35:11.063 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4514031dd0 con 0x7f452409e910 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f4529f5d640 5 monclient: authenticate success, global_id 134144 2024-09-18T12:35:11.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f4529f5d640 10 monclient: _renew_subs 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f4529f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4524098ea0 con 0x7f452409e910 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451403d650 con 0x7f452409e910 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f451b7fe640 10 monclient: dump: 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.065 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f45227fc640 4 set_mon_vals no callback set 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f4529f5d640 10 monclient: _renew_subs 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f4529f5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.051+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45241315c0 con 0x7f452409e910 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4514039040 con 0x7f452409e910 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f451b7fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4504038490 0x7f450403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4529f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4529f5d640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 0x7f452402aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4529f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524063740 con 0x7f452409b690 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f451406c2b0 con 0x7f452409e910 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4523fff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 0x7f452402aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f451bfff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4504038490 0x7f450403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4523fff640 10 monclient: get_auth_request con 0x7f452409b690 auth_method 0 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4523fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f4523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.052+0000 7f451bfff640 10 monclient: get_auth_request con 0x7f4504038490 auth_method 0 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.053+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.053+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.053+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.053+0000 7f4523fff640 10 monclient(hunting): handle_auth_done global_id 124342 payload 824 2024-09-18T12:35:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.053+0000 7f4523fff640 1 -- 172.21.15.116:0/3688413894 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f452409b1d0 con 0x7f452409b690 2024-09-18T12:35:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.055+0000 7f451bfff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4504038490 0x7f450403a940 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f450c007fd0 tx=0x7f450c007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.055+0000 7f4523fff640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 msgr2=0x7f452402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.055+0000 7f4523fff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 0x7f452402aa40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f451c00dac0 tx=0x7f451c007950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.055+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c00e400 con 0x7f452409b690 2024-09-18T12:35:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.055+0000 7f451b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.055+0000 7f451b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.055+0000 7f451b7fe640 10 monclient: dump: 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.056+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f451c0163f0 con 0x7f452409b690 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.056+0000 7f451b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.056+0000 7f451b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.056+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 msgr2=0x7f452402aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.056+0000 7f451b7fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 0x7f452402aa40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f451c00dac0 tx=0x7f451c007950 comp rx=0 tx=0).stop 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.056+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 msgr2=0x7f452402aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.056+0000 7f451b7fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f452409b690 0x7f452402aa40 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f451c00dac0 tx=0x7f451c007950 comp rx=0 tx=0).stop 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.061+0000 7f4529f5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.061+0000 7f4529f5d640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 0x7f452405e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.061+0000 7f4529f5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.061+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452409ada0 con 0x7f4524098590 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 0x7f452405e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient: get_auth_request con 0x7f4524098590 auth_method 0 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 10 monclient(hunting): handle_auth_done global_id 124345 payload 824 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.062+0000 7f4523fff640 1 -- 172.21.15.116:0/3688413894 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f452404bf60 con 0x7f4524098590 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.064+0000 7f4523fff640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 msgr2=0x7f452405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.064+0000 7f4523fff640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 0x7f452405e310 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f451c00dac0 tx=0x7f451c015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c023440 con 0x7f4524098590 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 10 monclient: dump: 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.073 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f451c0153a0 con 0x7f4524098590 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 msgr2=0x7f452405e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 0x7f452405e310 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f451c00dac0 tx=0x7f451c015000 comp rx=0 tx=0).stop 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 msgr2=0x7f452405e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f451b7fe640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4524098590 0x7f452405e310 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f451c00dac0 tx=0x7f451c015000 comp rx=0 tx=0).stop 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 98, 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:35:11.075 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:11.076 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.077 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:11.078 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:11.079 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:11.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:11.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:11.082 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:11.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:11.085 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:11.086 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:11.087 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4504038490 msgr2=0x7f450403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4504038490 0x7f450403a940 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f450c007fd0 tx=0x7f450c007f00 comp rx=0 tx=0).stop 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 10 monclient: shutdown 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 msgr2=0x7f45240975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4523fff640 1 -- 172.21.15.116:0/3688413894 reap_dead start 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 0x7f45240975d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4514009850 tx=0x7f4514032000 comp rx=0 tx=0).stop 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 shutdown_connections 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 --2- 172.21.15.116:0/3688413894 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f452409e910 0x7f45240975d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 >> 172.21.15.116:0/3688413894 conn(0x7f4524067d30 msgr2=0x7f452409d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 shutdown_connections 2024-09-18T12:35:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.065+0000 7f4529f5d640 1 -- 172.21.15.116:0/3688413894 wait complete. 2024-09-18T12:35:11.090 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:35:11.090 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:35:11.090 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:35:11.090 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:35:11.090 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:35:11.090 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:35:11.091 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:35:11.101 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:35:11.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:35:11.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.232+0000 7effdaf1d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.232+0000 7effdaf1d640 10 monclient: build_initial_monmap 2024-09-18T12:35:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.232+0000 7effdaf1d640 10 monclient: monmap: 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:11.233412+0000 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:11.233412+0000 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:11.234 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.233+0000 7effdaf1d640 1 Processor -- start 2024-09-18T12:35:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.233+0000 7effdaf1d640 1 -- start start 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.233+0000 7effdaf1d640 10 monclient: init 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.233+0000 7effdaf1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.233+0000 7effdaf1d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.233+0000 7effdaf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7effd4096e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.233+0000 7effdaf1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 10 monclient(hunting): picked mon.noname-b con 0x7effd40971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7effd4133b30 0x7effd4135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 10 monclient(hunting): picked mon.noname-c con 0x7effd4133b30 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd4099620 con 0x7effd4096e10 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd4099340 con 0x7effd4133b30 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd4099900 con 0x7effd40971e0 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effdaf1d640 10 monclient(hunting): authenticate will time out at 1338.555053s 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd3fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 msgr2=0x7effd40975b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40975b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7effd4133b30 0x7effd4135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd8cb9640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7effd4096e10 0x7effd409a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:50514/0 (socket says 172.21.15.116:50514) 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd8cb9640 1 -- 172.21.15.116:0/2462851039 learned_addr learned my addr 172.21.15.116:0/2462851039 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): get_auth_request con 0x7effd4133b30 auth_method 0 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient(hunting): handle_auth_done global_id 124348 payload 274 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 1 -- 172.21.15.116:0/2462851039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 msgr2=0x7effd40975b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 1 --2- 172.21.15.116:0/2462851039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40975b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 1 -- 172.21.15.116:0/2462851039 >> v1:172.21.15.144:6789/0 conn(0x7effd4096e10 legacy=0x7effd409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient: _finish_hunting 0 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 1 monclient: found mon.noname-c 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:35:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.234+0000 7effd94ba640 1 -- 172.21.15.116:0/2462851039 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd409ac70 con 0x7effd4133b30 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.237+0000 7effd94ba640 1 -- 172.21.15.116:0/2462851039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4133b30 msgr2=0x7effd4135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.237+0000 7effd94ba640 1 --2- 172.21.15.116:0/2462851039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4133b30 0x7effd4135f10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7effd409aff0 tx=0x7effcc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f01b3768553c4622 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.237+0000 7effd37fe640 1 -- 172.21.15.116:0/2462851039 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc007840 con 0x7effd4133b30 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.237+0000 7effd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 10 monclient: dump: 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 10 monclient: _finish_auth 0 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 1 -- 172.21.15.116:0/2462851039 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc02d050 con 0x7effd4133b30 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 5 monclient: authenticate success, global_id 124348 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 4 set_mon_vals no callback set 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 1 -- 172.21.15.116:0/2462851039 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc007c10 con 0x7effd4133b30 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effd37fe640 10 monclient: dump: 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 10 monclient: shutdown 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 -- 172.21.15.116:0/2462851039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4133b30 msgr2=0x7effd4135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 --2- 172.21.15.116:0/2462851039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4133b30 0x7effd4135f10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7effd409aff0 tx=0x7effcc009780 comp rx=0 tx=0).stop 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 -- 172.21.15.116:0/2462851039 shutdown_connections 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 --2- 172.21.15.116:0/2462851039 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4133b30 0x7effd4135f10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 --2- 172.21.15.116:0/2462851039 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 -- 172.21.15.116:0/2462851039 >> 172.21.15.116:0/2462851039 conn(0x7effd40672b0 msgr2=0x7effd40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 -- 172.21.15.116:0/2462851039 shutdown_connections 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.238+0000 7effdaf1d640 1 -- 172.21.15.116:0/2462851039 wait complete. 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.239+0000 7effdaf1d640 10 monclient: build_initial_monmap 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.239+0000 7effdaf1d640 10 monclient: monmap: 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.239+0000 7effdaf1d640 1 Processor -- start 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.239+0000 7effdaf1d640 1 -- start start 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.239+0000 7effdaf1d640 10 monclient: init 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4096e10 0x7effd406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): picked mon.noname-b con 0x7effd4096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): picked mon.noname-c con 0x7effd40971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 0x7effd4069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): picked mon.noname-a con 0x7effd4133b30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd409b050 con 0x7effd4133b30 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4137f40 con 0x7effd4096e10 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd406ce10 con 0x7effd40971e0 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effdaf1d640 10 monclient(hunting): authenticate will time out at 1338.561174s 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd3fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 msgr2=0x7effd40696e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40696e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 0x7effd4069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 0x7effd4069c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45394/0 (socket says 172.21.15.116:45394) 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 -- 172.21.15.116:0/3451905857 learned_addr learned my addr 172.21.15.116:0/3451905857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): get_auth_request con 0x7effd4133b30 auth_method 0 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4096e10 0x7effd406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 10 monclient(hunting): get_auth_request con 0x7effd4096e10 auth_method 0 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd8cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient(hunting): handle_auth_done global_id 104844 payload 824 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 -- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 msgr2=0x7effd40696e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40696e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 -- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4096e10 msgr2=0x7effd406ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4096e10 0x7effd406ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient: _finish_hunting 0 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 monclient: found mon.noname-a 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.240+0000 7effd94ba640 1 -- 172.21.15.116:0/3451905857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc009a20 con 0x7effd4133b30 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.242+0000 7effd94ba640 1 -- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 msgr2=0x7effd4069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.242+0000 7effd94ba640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 0x7effd4069c20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7effcc0099f0 tx=0x7effcc037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc031cd0 con 0x7effd4133b30 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: dump: 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc040070 con 0x7effd4133b30 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effdaf1d640 5 monclient: authenticate success, global_id 104844 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc045640 con 0x7effd4133b30 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effdaf1d640 10 monclient: _renew_subs 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effdaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd406d3a0 con 0x7effd4133b30 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd2ffd640 4 set_mon_vals no callback set 2024-09-18T12:35:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 10 monclient: dump: 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effdaf1d640 10 monclient: _renew_subs 2024-09-18T12:35:11.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effdaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd406b250 con 0x7effd4133b30 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.243+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7effcc003020 con 0x7effd4133b30 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effdaf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effdaf1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effdaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd406cf80 con 0x7effd4133b30 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effd0ff9640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effb8038440 0x7effb803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7effcc06e000 con 0x7effd4133b30 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effd8cb9640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effb8038440 0x7effb803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:11.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.244+0000 7effd8cb9640 10 monclient: get_auth_request con 0x7effb8038440 auth_method 0 2024-09-18T12:35:11.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.246+0000 7effd8cb9640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effb8038440 0x7effb803a8f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7effc8007fd0 tx=0x7effc8007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:11.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.246+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:35:11.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.249+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7effcc031e30 con 0x7effd4133b30 2024-09-18T12:35:11.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.249+0000 7effd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:11.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.249+0000 7effd0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.461+0000 7effdaf1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.461+0000 7effdaf1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.461+0000 7effdaf1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.461+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7effd404bf60 con 0x7effd4133b30 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.461+0000 7effd0ff9640 1 -- 172.21.15.116:0/3451905857 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7effcc0393c0 con 0x7effd4133b30 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.461+0000 7effd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.461+0000 7effd0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:11.462 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effb8038440 msgr2=0x7effb803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effb8038440 0x7effb803a8f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7effc8007fd0 tx=0x7effc8007f00 comp rx=0 tx=0).stop 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 10 monclient: shutdown 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 msgr2=0x7effd4069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 0x7effd4069c20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7effcc0099f0 tx=0x7effcc037000 comp rx=0 tx=0).stop 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 shutdown_connections 2024-09-18T12:35:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7effb8038440 0x7effb803a8f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effd4133b30 0x7effd4069c20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7effd40971e0 0x7effd40696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 --2- 172.21.15.116:0/3451905857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7effd4096e10 0x7effd406ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 >> 172.21.15.116:0/3451905857 conn(0x7effd40672b0 msgr2=0x7effd409f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 shutdown_connections 2024-09-18T12:35:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:11.463+0000 7effdaf1d640 1 -- 172.21.15.116:0/3451905857 wait complete. 2024-09-18T12:35:14.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.576+0000 7fccffcc0640 10 monclient: get_monmap_and_config 2024-09-18T12:35:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.576+0000 7fccffcc0640 10 monclient: build_initial_monmap 2024-09-18T12:35:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.576+0000 7fccffcc0640 10 monclient: monmap: 2024-09-18T12:35:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:14.577 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:14.577339+0000 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:14.577339+0000 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:14.578 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:14.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 1 Processor -- start 2024-09-18T12:35:14.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 1 -- start start 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 10 monclient: init 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fccf80a1890 0x7fccf812eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 10 monclient(hunting): picked mon.noname-c con 0x7fccf80a1890 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.577+0000 7fccffcc0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccf80975c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf8136c80 0x7fccf8139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 10 monclient(hunting): picked mon.noname-a con 0x7fccf8136c80 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf806e900 con 0x7fccf80975c0 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf8098150 con 0x7fccf8136c80 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf80a1c60 con 0x7fccf80a1890 2024-09-18T12:35:14.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccffcc0640 10 monclient(hunting): authenticate will time out at 1341.899055s 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf8136c80 0x7fccf8139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf8136c80 0x7fccf8139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45404/0 (socket says 172.21.15.116:45404) 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 -- 172.21.15.116:0/2961711791 learned_addr learned my addr 172.21.15.116:0/2961711791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): get_auth_request con 0x7fccf8136c80 auth_method 0 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfda5c640 1 --2- 172.21.15.116:0/2961711791 >> v2:172.21.15.144:3300/0 conn(0x7fccf80a1890 0x7fccf812eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfda5c640 10 monclient(hunting): get_auth_request con 0x7fccf80a1890 auth_method 0 2024-09-18T12:35:14.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfda5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfda5c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfda5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient(hunting): handle_auth_done global_id 104847 payload 274 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 -- 172.21.15.116:0/2961711791 >> v2:172.21.15.144:3300/0 conn(0x7fccf80a1890 msgr2=0x7fccf812eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 --2- 172.21.15.116:0/2961711791 >> v2:172.21.15.144:3300/0 conn(0x7fccf80a1890 0x7fccf812eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 -- 172.21.15.116:0/2961711791 >> v1:172.21.15.144:6789/0 conn(0x7fccf80975c0 legacy=0x7fccf8097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient: _finish_hunting 0 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 monclient: found mon.noname-a 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.578+0000 7fccfe25d640 1 -- 172.21.15.116:0/2961711791 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf80641d0 con 0x7fccf8136c80 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfe25d640 1 -- 172.21.15.116:0/2961711791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 msgr2=0x7fccf8139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfe25d640 1 --2- 172.21.15.116:0/2961711791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8139060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fccf806ee60 tx=0x7fccf0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0d4327f93448a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 1 -- 172.21.15.116:0/2961711791 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0007610 con 0x7fccf8136c80 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: dump: 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:14.589 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: _finish_auth 0 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 1 -- 172.21.15.116:0/2961711791 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf002d050 con 0x7fccf8136c80 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccffcc0640 5 monclient: authenticate success, global_id 104847 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccffcc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccffcc0640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:14.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccffcc0640 4 set_mon_vals no callback set 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 1 -- 172.21.15.116:0/2961711791 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf00079a0 con 0x7fccf8136c80 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.581+0000 7fccfca5a640 10 monclient: dump: 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:14.591 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 10 monclient: shutdown 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 -- 172.21.15.116:0/2961711791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 msgr2=0x7fccf8139060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 --2- 172.21.15.116:0/2961711791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8139060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fccf806ee60 tx=0x7fccf0009840 comp rx=0 tx=0).stop 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 -- 172.21.15.116:0/2961711791 shutdown_connections 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 --2- 172.21.15.116:0/2961711791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8139060 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 --2- 172.21.15.116:0/2961711791 >> v2:172.21.15.144:3300/0 conn(0x7fccf80a1890 0x7fccf812eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 -- 172.21.15.116:0/2961711791 >> 172.21.15.116:0/2961711791 conn(0x7fccf8067b50 msgr2=0x7fccf80a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 -- 172.21.15.116:0/2961711791 shutdown_connections 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 1 -- 172.21.15.116:0/2961711791 wait complete. 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 10 monclient: build_initial_monmap 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.582+0000 7fccffcc0640 10 monclient: monmap: 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:14.592 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 Processor -- start 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 -- start start 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient: init 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:14.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf80975c0 0x7fccf8126250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): picked mon.noname-c con 0x7fccf80975c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf80a1890 0x7fccf8126790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): picked mon.noname-b con 0x7fccf80a1890 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfda5c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf80975c0 msgr2=0x7fccf8126250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8126cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): picked mon.noname-a con 0x7fccf8136c80 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81293d0 con 0x7fccf8136c80 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfda5c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf80975c0 0x7fccf8126250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf806ee90 con 0x7fccf80a1890 2024-09-18T12:35:14.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80a12d0 con 0x7fccf80975c0 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccffcc0640 10 monclient(hunting): authenticate will time out at 1341.904718s 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8126cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8126cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45420/0 (socket says 172.21.15.116:45420) 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 1 -- 172.21.15.116:0/1189173318 learned_addr learned my addr 172.21.15.116:0/1189173318 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 10 monclient(hunting): get_auth_request con 0x7fccf8136c80 auth_method 0 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:14.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfd25b640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf80a1890 0x7fccf8126790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfe25d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfd25b640 10 monclient(hunting): get_auth_request con 0x7fccf80a1890 auth_method 0 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfd25b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfd25b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.583+0000 7fccfd25b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfd25b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfd25b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfd25b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 10 monclient(hunting): handle_auth_done global_id 104850 payload 824 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 1 -- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf80975c0 msgr2=0x7fccf8126250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf80975c0 0x7fccf8126250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 1 -- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf80a1890 msgr2=0x7fccf8126790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:14.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf80a1890 0x7fccf8126790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 10 monclient: _finish_hunting 0 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 1 monclient: found mon.noname-a 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.584+0000 7fccfe25d640 1 -- 172.21.15.116:0/1189173318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf0009ae0 con 0x7fccf8136c80 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fccfe25d640 1 -- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 msgr2=0x7fccf8126cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fccfe25d640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8126cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009ab0 tx=0x7fccf0036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 1 -- 172.21.15.116:0/1189173318 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0032c60 con 0x7fccf8136c80 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: dump: 2024-09-18T12:35:14.597 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 1 -- 172.21.15.116:0/1189173318 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf0032dc0 con 0x7fccf8136c80 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fccffcc0640 5 monclient: authenticate success, global_id 104850 2024-09-18T12:35:14.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fccffcc0640 10 monclient: _renew_subs 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fccffcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf812a600 con 0x7fccf8136c80 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 1 -- 172.21.15.116:0/1189173318 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf003b640 con 0x7fccf8136c80 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fcce9ffb640 10 monclient: dump: 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:14.599 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.587+0000 7fccffcc0640 10 monclient: _renew_subs 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.586+0000 7fccebfff640 4 set_mon_vals no callback set 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.587+0000 7fccffcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.587+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf8128f30 con 0x7fccf8136c80 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fccffcc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fccffcc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fccffcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fcce9ffb640 1 -- 172.21.15.116:0/1189173318 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fccf0003020 con 0x7fccf8136c80 2024-09-18T12:35:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf804bf60 con 0x7fccf8136c80 2024-09-18T12:35:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fcce9ffb640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccdc038470 0x7fccdc03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fcce9ffb640 1 -- 172.21.15.116:0/1189173318 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fccf006cc60 con 0x7fccf8136c80 2024-09-18T12:35:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fccfda5c640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccdc038470 0x7fccdc03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.588+0000 7fccfda5c640 10 monclient: get_auth_request con 0x7fccdc038470 auth_method 0 2024-09-18T12:35:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.591+0000 7fccfda5c640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccdc038470 0x7fccdc03a920 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fccf4006f90 tx=0x7fccf4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.592+0000 7fcce9ffb640 1 -- 172.21.15.116:0/1189173318 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fccf0039ae0 con 0x7fccf8136c80 2024-09-18T12:35:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.592+0000 7fcce9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.592+0000 7fcce9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.804+0000 7fccffcc0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.804+0000 7fccffcc0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.804+0000 7fccffcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.804+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fccf809b590 con 0x7fccf8136c80 2024-09-18T12:35:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.804+0000 7fcce9ffb640 1 -- 172.21.15.116:0/1189173318 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fccf003f020 con 0x7fccf8136c80 2024-09-18T12:35:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.804+0000 7fcce9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.804+0000 7fcce9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:14.806 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccdc038470 msgr2=0x7fccdc03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccdc038470 0x7fccdc03a920 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fccf4006f90 tx=0x7fccf4008040 comp rx=0 tx=0).stop 2024-09-18T12:35:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 10 monclient: shutdown 2024-09-18T12:35:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 msgr2=0x7fccf8126cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8126cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009ab0 tx=0x7fccf0036000 comp rx=0 tx=0).stop 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 shutdown_connections 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fccdc038470 0x7fccdc03a920 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf8136c80 0x7fccf8126cd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccf80a1890 0x7fccf8126790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 --2- 172.21.15.116:0/1189173318 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fccf80975c0 0x7fccf8126250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 >> 172.21.15.116:0/1189173318 conn(0x7fccf8067b50 msgr2=0x7fccf8096c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 shutdown_connections 2024-09-18T12:35:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:14.806+0000 7fccffcc0640 1 -- 172.21.15.116:0/1189173318 wait complete. 2024-09-18T12:35:17.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.922+0000 7f2cb615d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.922+0000 7f2cb615d640 10 monclient: build_initial_monmap 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.922+0000 7f2cb615d640 10 monclient: monmap: 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:17.922998+0000 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:17.922998+0000 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:17.923 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:17.924 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:17.924 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:17.924 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:17.924 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:17.924 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:17.924 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:17.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 1 Processor -- start 2024-09-18T12:35:17.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 1 -- start start 2024-09-18T12:35:17.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient: init 2024-09-18T12:35:17.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:17.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:35:17.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2cb00981c0 0x7f2cb0098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:17.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb00981c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:17.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2cb0131da0 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2cb0132170 0x7f2cb0067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb0132170 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb0098ad0 con 0x7f2cb0131da0 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb0131bd0 con 0x7f2cb00981c0 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb009ca60 con 0x7f2cb0132170 2024-09-18T12:35:17.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2cb615d640 10 monclient(hunting): authenticate will time out at 1345.244703s 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2cb00981c0 0x7f2cb0098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2cb00981c0 0x7f2cb0098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49964/0 (socket says 172.21.15.116:49964) 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 1 -- 172.21.15.116:0/2730611773 learned_addr learned my addr 172.21.15.116:0/2730611773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 10 monclient(hunting): get_auth_request con 0x7f2cb00981c0 auth_method 0 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.923+0000 7f2caf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caffff640 1 --2- 172.21.15.116:0/2730611773 >> v2:172.21.15.144:3300/0 conn(0x7f2cb0132170 0x7f2cb0067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 10 monclient(hunting): handle_auth_done global_id 104853 payload 274 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 1 -- 172.21.15.116:0/2730611773 >> v2:172.21.15.144:3300/0 conn(0x7f2cb0132170 msgr2=0x7f2cb0067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 1 --2- 172.21.15.116:0/2730611773 >> v2:172.21.15.144:3300/0 conn(0x7f2cb0132170 0x7f2cb0067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 1 -- 172.21.15.116:0/2730611773 >> v1:172.21.15.144:6789/0 conn(0x7f2cb0131da0 legacy=0x7f2cb0067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 1 monclient: found mon.noname-a 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.924+0000 7f2caf7fe640 1 -- 172.21.15.116:0/2730611773 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb009cca0 con 0x7f2cb00981c0 2024-09-18T12:35:17.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2caf7fe640 1 -- 172.21.15.116:0/2730611773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb00981c0 msgr2=0x7f2cb0098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2caf7fe640 1 --2- 172.21.15.116:0/2730611773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb00981c0 0x7f2cb0098590 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2cb006f820 tx=0x7f2ca00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c0a2b2c6f31755 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 1 -- 172.21.15.116:0/2730611773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0032440 con 0x7f2cb00981c0 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: dump: 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:17.937 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: _finish_auth 0 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 5 monclient: authenticate success, global_id 104853 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 1 -- 172.21.15.116:0/2730611773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca00325a0 con 0x7f2cb00981c0 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 4 set_mon_vals no callback set 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 1 -- 172.21.15.116:0/2730611773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0032940 con 0x7f2cb00981c0 2024-09-18T12:35:17.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cae7fc640 10 monclient: dump: 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:17.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 10 monclient: shutdown 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 1 -- 172.21.15.116:0/2730611773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb00981c0 msgr2=0x7f2cb0098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 1 --2- 172.21.15.116:0/2730611773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb00981c0 0x7f2cb0098590 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2cb006f820 tx=0x7f2ca00091d0 comp rx=0 tx=0).stop 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 1 -- 172.21.15.116:0/2730611773 shutdown_connections 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 1 --2- 172.21.15.116:0/2730611773 >> v2:172.21.15.144:3300/0 conn(0x7f2cb0132170 0x7f2cb0067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 1 --2- 172.21.15.116:0/2730611773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb00981c0 0x7f2cb0098590 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 1 -- 172.21.15.116:0/2730611773 >> 172.21.15.116:0/2730611773 conn(0x7f2cb00a1e30 msgr2=0x7f2cb00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.927+0000 7f2cb615d640 1 -- 172.21.15.116:0/2730611773 shutdown_connections 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.928+0000 7f2cb615d640 1 -- 172.21.15.116:0/2730611773 wait complete. 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.928+0000 7f2cb615d640 10 monclient: build_initial_monmap 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.928+0000 7f2cb615d640 10 monclient: monmap: 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:17.940 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.928+0000 7f2cb615d640 1 Processor -- start 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 1 -- start start 2024-09-18T12:35:17.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient: init 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cb00981c0 0x7f2cb0141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb00981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cb0131da0 0x7f2cb01307d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb0131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 0x7f2cb011a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb0132170 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0098cb0 con 0x7f2cb0132170 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:17.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00a05b0 con 0x7f2cb0131da0 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb01416a0 con 0x7f2cb00981c0 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2cb615d640 10 monclient(hunting): authenticate will time out at 1345.250453s 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caf7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cb00981c0 msgr2=0x7f2cb0141160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caf7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cb00981c0 0x7f2cb0141160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 0x7f2cb011a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 0x7f2cb011a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49972/0 (socket says 172.21.15.116:49972) 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 -- 172.21.15.116:0/559204670 learned_addr learned my addr 172.21.15.116:0/559204670 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:17.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): get_auth_request con 0x7f2cb0132170 auth_method 0 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caeffd640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cb0131da0 0x7f2cb01307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caeffd640 10 monclient(hunting): get_auth_request con 0x7f2cb0131da0 auth_method 0 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:17.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient(hunting): handle_auth_done global_id 104856 payload 824 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 -- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cb00981c0 msgr2=0x7f2cb0141160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cb00981c0 0x7f2cb0141160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 -- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cb0131da0 msgr2=0x7f2cb01307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cb0131da0 0x7f2cb01307d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient: _finish_hunting 0 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 monclient: found mon.noname-a 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.929+0000 7f2caffff640 1 -- 172.21.15.116:0/559204670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0009450 con 0x7f2cb0132170 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2caffff640 1 -- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 msgr2=0x7f2cb011a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2caffff640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 0x7f2cb011a250 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800ceb0 tx=0x7f2ca8007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 1 -- 172.21.15.116:0/559204670 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca800e610 con 0x7f2cb0132170 2024-09-18T12:35:17.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: dump: 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:17.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: _finish_auth 0 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 1 -- 172.21.15.116:0/559204670 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca800e770 con 0x7f2cb0132170 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cb615d640 5 monclient: authenticate success, global_id 104856 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cb615d640 10 monclient: _renew_subs 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cb615d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb0141fa0 con 0x7f2cb0132170 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 1 -- 172.21.15.116:0/559204670 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca800eb30 con 0x7f2cb0132170 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2c977fe640 10 monclient: dump: 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:17.947 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cadffb640 4 set_mon_vals no callback set 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cb615d640 10 monclient: _renew_subs 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cb615d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.932+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb011be30 con 0x7f2cb0132170 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2c977fe640 1 -- 172.21.15.116:0/559204670 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2ca8009030 con 0x7f2cb0132170 2024-09-18T12:35:17.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2cb615d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2cb615d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2cb615d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb004bf60 con 0x7f2cb0132170 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2c977fe640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c9c038440 0x7f2c9c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2c977fe640 1 -- 172.21.15.116:0/559204670 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2ca8012070 con 0x7f2cb0132170 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2caf7fe640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c9c038440 0x7f2c9c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.933+0000 7f2caf7fe640 10 monclient: get_auth_request con 0x7f2c9c038440 auth_method 0 2024-09-18T12:35:17.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.936+0000 7f2caf7fe640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c9c038440 0x7f2c9c03a8f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0132870 tx=0x7f2ca0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:17.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.938+0000 7f2c977fe640 1 -- 172.21.15.116:0/559204670 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2ca80153d0 con 0x7f2cb0132170 2024-09-18T12:35:17.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.938+0000 7f2c977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:17.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:17.938+0000 7f2c977fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:18.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.149+0000 7f2cb615d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:18.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.149+0000 7f2cb615d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:18.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.149+0000 7f2cb615d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:18.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.149+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2cb0098ee0 con 0x7f2cb0132170 2024-09-18T12:35:18.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.150+0000 7f2c977fe640 1 -- 172.21.15.116:0/559204670 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2ca8014330 con 0x7f2cb0132170 2024-09-18T12:35:18.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.150+0000 7f2c977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:18.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.150+0000 7f2c977fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:18.152 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":98,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:18.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c9c038440 msgr2=0x7f2c9c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:18.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c9c038440 0x7f2c9c03a8f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0132870 tx=0x7f2ca0032000 comp rx=0 tx=0).stop 2024-09-18T12:35:18.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 10 monclient: shutdown 2024-09-18T12:35:18.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:18.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 msgr2=0x7f2cb011a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:18.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 0x7f2cb011a250 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800ceb0 tx=0x7f2ca8007870 comp rx=0 tx=0).stop 2024-09-18T12:35:18.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 shutdown_connections 2024-09-18T12:35:18.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c9c038440 0x7f2c9c03a8f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:18.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cb0132170 0x7f2cb011a250 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:18.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cb0131da0 0x7f2cb01307d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:18.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 --2- 172.21.15.116:0/559204670 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cb00981c0 0x7f2cb0141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:18.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 >> 172.21.15.116:0/559204670 conn(0x7f2cb00a1e30 msgr2=0x7f2cb0096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:18.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 shutdown_connections 2024-09-18T12:35:18.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:18.152+0000 7f2cb615d640 1 -- 172.21.15.116:0/559204670 wait complete. 2024-09-18T12:35:21.164 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.267+0000 7f00c2b5d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.267+0000 7f00c2b5d640 10 monclient: build_initial_monmap 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.267+0000 7f00c2b5d640 10 monclient: monmap: 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:21.268741+0000 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:21.268741+0000 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:21.269 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:21.270 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:21.270 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:21.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.267+0000 7f00c2b5d640 1 Processor -- start 2024-09-18T12:35:21.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.267+0000 7f00c2b5d640 1 -- start start 2024-09-18T12:35:21.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.267+0000 7f00c2b5d640 10 monclient: init 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00bc09a510 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00bc097370 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 0x7f00bc1380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f00bc097770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc09b1c0 con 0x7f00bc097370 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc09cb00 con 0x7f00bc09a510 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc09a930 con 0x7f00bc097770 2024-09-18T12:35:21.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c2b5d640 10 monclient(hunting): authenticate will time out at 1348.590489s 2024-09-18T12:35:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c10fa640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 msgr2=0x7f00bc1380d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00c10fa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 0x7f00bc1380d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00bbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f00bc097370 0x7f00bc1349a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36614/0 (socket says 172.21.15.116:36614) 2024-09-18T12:35:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.268+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 learned_addr learned my addr 172.21.15.116:0/3751186628 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:21.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.270+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v1:172.21.15.116:6789/0 conn(0x7f00bc097370 legacy=0x7f00bc1349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:21.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.271+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v1:172.21.15.144:6789/0 conn(0x7f00bc09a510 legacy=0x7f00bc096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.469+0000 7f00c10fa640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 msgr2=0x7f00bc1380d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.469+0000 7f00c10fa640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 0x7f00bc1380d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:35:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.870+0000 7f00c10fa640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 msgr2=0x7f00bc1380d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:21.870+0000 7f00c10fa640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 0x7f00bc1380d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:35:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:22.671+0000 7f00c10fa640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 msgr2=0x7f00bc1380d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:22.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:22.671+0000 7f00c10fa640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 0x7f00bc1380d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:35:24.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient(hunting): tick 2024-09-18T12:35:24.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 monclient(hunting): continuing hunt 2024-09-18T12:35:24.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:35:24.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 msgr2=0x7f00bc1380d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f00bc097770 0x7f00bc1380d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 >> v1:172.21.15.144:6789/0 conn(0x7f00bc09a510 legacy=0x7f00bc096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 >> v1:172.21.15.116:6789/0 conn(0x7f00bc097370 legacy=0x7f00bc1349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient: _add_conns ranks=[3,4] 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f009c003800 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f009c006360 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00bc09a930 con 0x7f009c003800 2024-09-18T12:35:24.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009c008ed0 con 0x7f009c006360 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00ba7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 10 monclient(hunting): get_auth_request con 0x7f009c003800 auth_method 0 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 10 monclient(hunting): get_auth_request con 0x7f009c006360 auth_method 0 2024-09-18T12:35:24.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:24.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:24.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:24.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:24.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:24.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:24.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:24.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.269+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:24.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:24.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 10 monclient(hunting): get_auth_request con 0x7f009c003800 auth_method 0 2024-09-18T12:35:24.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 10 monclient(hunting): get_auth_request con 0x7f009c006360 auth_method 0 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:35:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.470+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 10 monclient(hunting): get_auth_request con 0x7f009c003800 auth_method 0 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:24.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 10 monclient(hunting): get_auth_request con 0x7f009c006360 auth_method 0 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:24.871+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 10 monclient(hunting): get_auth_request con 0x7f009c003800 auth_method 0 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00c08f9640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00bbfff640 10 monclient(hunting): get_auth_request con 0x7f009c006360 auth_method 0 2024-09-18T12:35:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.672+0000 7f00bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.673+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:35:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.673+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.673+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:25.673+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:35:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 10 monclient(hunting): tick 2024-09-18T12:35:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 1 monclient(hunting): continuing hunt 2024-09-18T12:35:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:35:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 msgr2=0x7f009c008810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.144:3300/0 conn(0x7f009c006360 0x7f009c008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 msgr2=0x7f009c005c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c003800 0x7f009c005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00c10fa640 1 -- 172.21.15.116:0/3751186628 reap_dead start 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.268+0000 7f00ba7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f009c003c60 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c007f20 0x7f009c0082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f009c007f20 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c008830 0x7f009c00d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f009c008830 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc06ed80 con 0x7f009c003c60 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00bc09a930 con 0x7f009c007f20 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 1 -- 172.21.15.116:0/3751186628 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c008ed0 con 0x7f009c008830 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00ba7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00c10fa640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c008830 msgr2=0x7f009c00d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00c10fa640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c008830 0x7f009c00d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> v2:172.21.15.116:3300/0 conn(0x7f009c007f20 0x7f009c0082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): get_auth_request con 0x7f009c007f20 auth_method 0 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient(hunting): handle_auth_done global_id 104859 payload 274 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c008830 msgr2=0x7f009c00d260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c008830 0x7f009c00d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> v1:172.21.15.116:6789/0 conn(0x7f009c003c60 legacy=0x7f009c006800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 1 monclient: found mon.noname-a 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.269+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c00e990 con 0x7f009c007f20 2024-09-18T12:35:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bbfff640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c007f20 msgr2=0x7f009c0082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bbfff640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c007f20 0x7f009c0082f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f00a4007fd0 tx=0x7f00a4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec63d9df6d5e1db8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 1 -- 172.21.15.116:0/3751186628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a402b7a0 con 0x7f009c007f20 2024-09-18T12:35:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: dump: 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 1 -- 172.21.15.116:0/3751186628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a402f070 con 0x7f009c007f20 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 1 -- 172.21.15.116:0/3751186628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a402bb50 con 0x7f009c007f20 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00c2b5d640 5 monclient: authenticate success, global_id 104859 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00c2b5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00c2b5d640 4 set_mon_vals no callback set 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.272+0000 7f00bb7fe640 10 monclient: dump: 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.281 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 10 monclient: shutdown 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 -- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c007f20 msgr2=0x7f009c0082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c007f20 0x7f009c0082f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f00a4007fd0 tx=0x7f00a4009040 comp rx=0 tx=0).stop 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 -- 172.21.15.116:0/3751186628 shutdown_connections 2024-09-18T12:35:27.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c008830 0x7f009c00d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/3751186628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c007f20 0x7f009c0082f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 -- 172.21.15.116:0/3751186628 >> 172.21.15.116:0/3751186628 conn(0x7f00bc0672b0 msgr2=0x7f00bc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 -- 172.21.15.116:0/3751186628 shutdown_connections 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.273+0000 7f00c2b5d640 1 -- 172.21.15.116:0/3751186628 wait complete. 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.274+0000 7f00c2b5d640 10 monclient: build_initial_monmap 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.274+0000 7f00c2b5d640 10 monclient: monmap: 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:27.283 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.274+0000 7f00c2b5d640 1 Processor -- start 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 1 -- start start 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient: init 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c003c60 0x7f00bc13ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f009c003c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c007f20 0x7f00bc12e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f009c007f20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 0x7f00bc1184a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f009c008830 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc11a080 con 0x7f009c008830 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c08f9640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c003c60 msgr2=0x7f00bc13ee60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc12f930 con 0x7f009c007f20 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc13fd40 con 0x7f009c003c60 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c08f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c003c60 0x7f00bc13ee60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c2b5d640 10 monclient(hunting): authenticate will time out at 1354.597786s 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c10fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 0x7f00bc1184a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c10fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 0x7f00bc1184a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44534/0 (socket says 172.21.15.116:44534) 2024-09-18T12:35:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c10fa640 1 -- 172.21.15.116:0/2999657317 learned_addr learned my addr 172.21.15.116:0/2999657317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c10fa640 10 monclient(hunting): get_auth_request con 0x7f009c008830 auth_method 0 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c10fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c10fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.275+0000 7f00c10fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00bbfff640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c007f20 0x7f00bc12e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00bbfff640 10 monclient(hunting): get_auth_request con 0x7f009c007f20 auth_method 0 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 10 monclient(hunting): handle_auth_done global_id 104862 payload 824 2024-09-18T12:35:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 1 -- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c003c60 msgr2=0x7f00bc13ee60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c003c60 0x7f00bc13ee60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 1 -- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c007f20 msgr2=0x7f00bc12e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c007f20 0x7f00bc12e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 1 monclient: found mon.noname-a 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.276+0000 7f00c10fa640 1 -- 172.21.15.116:0/2999657317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a400f040 con 0x7f009c008830 2024-09-18T12:35:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c10fa640 1 -- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 msgr2=0x7f00bc1184a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c10fa640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 0x7f00bc1184a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f009c0062d0 tx=0x7f00b4009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 1 -- 172.21.15.116:0/2999657317 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4015070 con 0x7f009c008830 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: dump: 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c2b5d640 5 monclient: authenticate success, global_id 104862 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 1 -- 172.21.15.116:0/2999657317 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b400e510 con 0x7f009c008830 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c2b5d640 10 monclient: _renew_subs 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c2b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc11bd40 con 0x7f009c008830 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 1 -- 172.21.15.116:0/2999657317 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b400e8b0 con 0x7f009c008830 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00b8ff9640 10 monclient: dump: 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c2b5d640 10 monclient: _renew_subs 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00ba7fc640 4 set_mon_vals no callback set 2024-09-18T12:35:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c2b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.278+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc13f540 con 0x7f009c008830 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.279+0000 7f00b8ff9640 1 -- 172.21.15.116:0/2999657317 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f00b400ece0 con 0x7f009c008830 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00c2b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00c2b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00c2b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00bc04bf60 con 0x7f009c008830 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00b8ff9640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f009c03a2a0 0x7f009c03c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00b8ff9640 1 -- 172.21.15.116:0/2999657317 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f00b404b940 con 0x7f009c008830 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00c08f9640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f009c03a2a0 0x7f009c03c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.280+0000 7f00c08f9640 10 monclient: get_auth_request con 0x7f009c03a2a0 auth_method 0 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.282+0000 7f00c08f9640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f009c03a2a0 0x7f009c03c750 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f00ac000f60 tx=0x7f00ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.285+0000 7f00b8ff9640 1 -- 172.21.15.116:0/2999657317 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f00b401a410 con 0x7f009c008830 2024-09-18T12:35:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.285+0000 7f00b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.285+0000 7f00b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.500+0000 7f00c2b5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.500+0000 7f00c2b5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.500+0000 7f00c2b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.500+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f00bc0638d0 con 0x7f009c008830 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.501+0000 7f00b8ff9640 1 -- 172.21.15.116:0/2999657317 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f00b40172c0 con 0x7f009c008830 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.501+0000 7f00b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.501+0000 7f00b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:27.503 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":102,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f009c03a2a0 msgr2=0x7f009c03c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f009c03a2a0 0x7f009c03c750 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f00ac000f60 tx=0x7f00ac009040 comp rx=0 tx=0).stop 2024-09-18T12:35:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 10 monclient: shutdown 2024-09-18T12:35:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 msgr2=0x7f00bc1184a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 0x7f00bc1184a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f009c0062d0 tx=0x7f00b4009480 comp rx=0 tx=0).stop 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 shutdown_connections 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f009c03a2a0 0x7f009c03c750 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c008830 0x7f00bc1184a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f009c007f20 0x7f00bc12e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 --2- 172.21.15.116:0/2999657317 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f009c003c60 0x7f00bc13ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 >> 172.21.15.116:0/2999657317 conn(0x7f00bc0672b0 msgr2=0x7f00bc09fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 shutdown_connections 2024-09-18T12:35:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.503+0000 7f00c2b5d640 1 -- 172.21.15.116:0/2999657317 wait complete. 2024-09-18T12:35:27.516 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:35:27.516 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:35:27.516 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.608+0000 7f5553b09640 10 monclient: get_monmap_and_config 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.608+0000 7f5553b09640 10 monclient: build_initial_monmap 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.608+0000 7f5553b09640 10 monclient: monmap: 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:27.610033+0000 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:27.610033+0000 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:27.611 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.608+0000 7f5553b09640 1 Processor -- start 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 1 -- start start 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient: init 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f554c0974f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 0x7f554c09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient(hunting): picked mon.noname-b con 0x7f554c09a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f554c133070 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c12b210 con 0x7f554c0974f0 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c0978c0 con 0x7f554c133070 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c09ca20 con 0x7f554c09a8a0 2024-09-18T12:35:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f5553b09640 10 monclient(hunting): authenticate will time out at 1354.931601s 2024-09-18T12:35:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f55510a4640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 msgr2=0x7f554c09acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f55510a4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 0x7f554c09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f55518a5640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f554c0974f0 0x7f554c12ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57304/0 (socket says 172.21.15.116:57304) 2024-09-18T12:35:27.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.609+0000 7f55518a5640 1 -- 172.21.15.116:0/2556598265 learned_addr learned my addr 172.21.15.116:0/2556598265 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:27.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.611+0000 7f55518a5640 1 -- 172.21.15.116:0/2556598265 >> v1:172.21.15.116:6789/0 conn(0x7f554c0974f0 legacy=0x7f554c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726356868 0 0) 0x7f554c12b210 con 0x7f554c0974f0 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): my global_id is 104865 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55340045f0 con 0x7f554c0974f0 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 >> v1:172.21.15.144:6789/0 conn(0x7f554c133070 legacy=0x7f554c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 436705759 0 0) 0x7f55340045f0 con 0x7f554c0974f0 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 msgr2=0x7f554c09acb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 0x7f554c09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 >> v1:172.21.15.144:6789/0 conn(0x7f554c133070 legacy=0x7f554c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c09b1f0 con 0x7f554c0974f0 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f55440017a0 con 0x7f554c0974f0 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: dump: 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.626 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 >> v1:172.21.15.116:6789/0 conn(0x7f554c0974f0 legacy=0x7f554c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534004be0 0x7f5534004ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): picked mon.b con 0x7f5534004be0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 0x7f55340058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): picked mon.c con 0x7f554c09a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 0x7f553400cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): picked mon.a con 0x7f5534006030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c09ca20 con 0x7f5534006030 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534003400 con 0x7f5534004be0 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55340035c0 con 0x7f554c09a8a0 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55508a3640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55510a4640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 msgr2=0x7f55340058d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55510a4640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 0x7f55340058d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 0x7f553400cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): get_auth_request con 0x7f5534006030 auth_method 0 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534004be0 0x7f5534004ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 10 monclient(hunting): get_auth_request con 0x7f5534004be0 auth_method 0 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:35:27.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient(hunting): handle_auth_done global_id 104865 payload 293 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 msgr2=0x7f55340058d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f554c09a8a0 0x7f55340058d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534004be0 msgr2=0x7f5534004ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534004be0 0x7f5534004ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 monclient: found mon.a 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55340045f0 con 0x7f5534006030 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f55520a6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f5553b09640 5 monclient: authenticate success, global_id 104865 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.612+0000 7f5553b09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:27.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 msgr2=0x7f553400cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 0x7f553400cae0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f553c002200 tx=0x7f553c001890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa050e0adbdf9ea server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c005b00 con 0x7f5534006030 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient: dump: 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.631 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c02d070 con 0x7f5534006030 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f5553b09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 msgr2=0x7f553400cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 msgr2=0x7f553400cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 0x7f553400cae0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f553c002200 tx=0x7f553c001890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f5553b09640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f5553b09640 4 set_mon_vals no callback set 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 0x7f553400cae0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f553c002200 tx=0x7f553c001890 comp rx=0 tx=0).stop 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 reap_dead start 2024-09-18T12:35:27.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 msgr2=0x7f553400cae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534006030 0x7f553400cae0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534004ff0 0x7f5534005890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): picked mon.a con 0x7f5534004ff0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534003d20 0x7f553400f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): picked mon.b con 0x7f5534003d20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 0x7f5534011b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): picked mon.c con 0x7f553400f770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534012220 con 0x7f5534004ff0 2024-09-18T12:35:27.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534012360 con 0x7f5534003d20 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55340124f0 con 0x7f553400f770 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55508a3640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 msgr2=0x7f5534011b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55520a6640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 0x7f5534011b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.615+0000 7f55518a5640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534004ff0 0x7f5534005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): get_auth_request con 0x7f5534004ff0 auth_method 0 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55510a4640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534003d20 0x7f553400f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient(hunting): handle_auth_done global_id 104865 payload 293 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 msgr2=0x7f5534011b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 0x7f5534011b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534003d20 msgr2=0x7f553400f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534003d20 0x7f553400f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 1 monclient: found mon.a 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 10 monclient: shutdown 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 1 -- 172.21.15.116:0/2556598265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f554c12b210 con 0x7f5534004ff0 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55518a5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55510a4640 10 monclient: get_auth_request con 0x7f5534003d20 auth_method 0 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f55510a4640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534003d20 0x7f553400f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 -- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534004ff0 msgr2=0x7f5534005890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534004ff0 0x7f5534005890 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f5544001980 tx=0x7f554c0695f0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 -- 172.21.15.116:0/2556598265 shutdown_connections 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 0x7f5534011b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534003d20 0x7f553400f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 --2- 172.21.15.116:0/2556598265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534004ff0 0x7f5534005890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 -- 172.21.15.116:0/2556598265 >> 172.21.15.116:0/2556598265 conn(0x7f554c0672b0 msgr2=0x7f554c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 -- 172.21.15.116:0/2556598265 shutdown_connections 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.616+0000 7f5553b09640 1 -- 172.21.15.116:0/2556598265 wait complete. 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient: build_initial_monmap 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient: monmap: 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:27.637 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 Processor -- start 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 -- start start 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient: init 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534003d20 0x7f554c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): picked mon.noname-a con 0x7f5534003d20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534004ff0 0x7f554c132210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): picked mon.noname-b con 0x7f5534004ff0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 0x7f554c06cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): picked mon.noname-c con 0x7f553400f770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:27.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534012220 con 0x7f5534003d20 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c12a3d0 con 0x7f5534004ff0 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c12c400 con 0x7f553400f770 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f5553b09640 10 monclient(hunting): authenticate will time out at 1354.939877s 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55520a6640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 msgr2=0x7f554c06cd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534003d20 0x7f554c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55520a6640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 0x7f554c06cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534003d20 0x7f554c13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44554/0 (socket says 172.21.15.116:44554) 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 1 -- 172.21.15.116:0/3919772609 learned_addr learned my addr 172.21.15.116:0/3919772609 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:27.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 10 monclient(hunting): get_auth_request con 0x7f5534003d20 auth_method 0 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.617+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 10 monclient(hunting): handle_auth_done global_id 104868 payload 824 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 msgr2=0x7f554c06cd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553400f770 0x7f554c06cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534004ff0 msgr2=0x7f554c132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534004ff0 0x7f554c132210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 1 monclient: found mon.noname-a 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.618+0000 7f55518a5640 1 -- 172.21.15.116:0/3919772609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c12b210 con 0x7f5534003d20 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f55518a5640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534003d20 msgr2=0x7f554c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f55518a5640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534003d20 0x7f554c13c9b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5544016b80 tx=0x7f554c0695f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55440060d0 con 0x7f5534003d20 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: dump: 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.641 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5544006250 con 0x7f5534003d20 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5553b09640 5 monclient: authenticate success, global_id 104868 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5553b09640 10 monclient: _renew_subs 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5553b09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554c12c6e0 con 0x7f5534003d20 2024-09-18T12:35:27.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544006610 con 0x7f5534003d20 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5549ffb640 10 monclient: dump: 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.643 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5553b09640 10 monclient: _renew_subs 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f554bfff640 4 set_mon_vals no callback set 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5553b09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.620+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c13d890 con 0x7f5534003d20 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f5553b09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f5553b09640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 0x7f554c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f5553b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c065f20 con 0x7f554c06f370 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f55440049b0 con 0x7f5534003d20 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 0x7f554c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient: get_auth_request con 0x7f554c06f370 auth_method 0 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f5549ffb640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5538038440 0x7f553803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 10 monclient(hunting): handle_auth_done global_id 104871 payload 824 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f55510a4640 1 -- 172.21.15.116:0/3919772609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f554c12cc00 con 0x7f554c06f370 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.621+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f554404cdf0 con 0x7f5534003d20 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.622+0000 7f55520a6640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5538038440 0x7f553803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.622+0000 7f55520a6640 10 monclient: get_auth_request con 0x7f5538038440 auth_method 0 2024-09-18T12:35:27.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.623+0000 7f55510a4640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 msgr2=0x7f554c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f55510a4640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 0x7f554c06d270 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f55400059f0 tx=0x7f5540008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540003d90 con 0x7f554c06f370 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 10 monclient: dump: 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.648 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f55400050f0 con 0x7f554c06f370 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 msgr2=0x7f554c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 0x7f554c06d270 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f55400059f0 tx=0x7f5540008040 comp rx=0 tx=0).stop 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 msgr2=0x7f554c06d270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f5549ffb640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c06f370 0x7f554c06d270 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.624+0000 7f55520a6640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5538038440 0x7f553803a8f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f554c0980c0 tx=0x7f553c033000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.629+0000 7f5553b09640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.629+0000 7f5553b09640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 0x7f554c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f5553b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c13d890 con 0x7f554c132750 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 0x7f554c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient: get_auth_request con 0x7f554c132750 auth_method 0 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 10 monclient(hunting): handle_auth_done global_id 104874 payload 824 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.630+0000 7f55510a4640 1 -- 172.21.15.116:0/3919772609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f554c04bf60 con 0x7f554c132750 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f55510a4640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 msgr2=0x7f554c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f55510a4640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 0x7f554c06d7b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f554c0984e0 tx=0x7f5540018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540005a60 con 0x7f554c132750 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 10 monclient: dump: 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.651 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f5540003b80 con 0x7f554c132750 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 msgr2=0x7f554c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 0x7f554c06d7b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f554c0984e0 tx=0x7f5540018000 comp rx=0 tx=0).stop 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 msgr2=0x7f554c06d7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.632+0000 7f5549ffb640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f554c132750 0x7f554c06d7b0 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f554c0984e0 tx=0x7f5540018000 comp rx=0 tx=0).stop 2024-09-18T12:35:27.652 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 102, 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:27.653 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:27.654 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:27.655 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:27.656 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:27.657 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.658 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:27.660 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:27.661 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:27.662 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:27.664 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5538038440 msgr2=0x7f553803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5538038440 0x7f553803a8f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f554c0980c0 tx=0x7f553c033000 comp rx=0 tx=0).stop 2024-09-18T12:35:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 10 monclient: shutdown 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534003d20 msgr2=0x7f554c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 --2- 172.21.15.116:0/3919772609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5534003d20 0x7f554c13c9b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5544016b80 tx=0x7f554c0695f0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f55520a6640 1 -- 172.21.15.116:0/3919772609 reap_dead start 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f55520a6640 1 -- 172.21.15.116:0/3919772609 reap_dead start 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 shutdown_connections 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 >> 172.21.15.116:0/3919772609 conn(0x7f554c0672b0 msgr2=0x7f554c09b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 shutdown_connections 2024-09-18T12:35:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.633+0000 7f5553b09640 1 -- 172.21.15.116:0/3919772609 wait complete. 2024-09-18T12:35:27.666 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:35:27.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:35:27.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:35:27.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.757+0000 7f37eb11d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.757+0000 7f37eb11d640 10 monclient: build_initial_monmap 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.758+0000 7f37eb11d640 10 monclient: monmap: 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:27.759655+0000 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:27.759655+0000 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:27.760 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:27.761 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:27.761 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:27.761 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:27.761 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.758+0000 7f37eb11d640 1 Processor -- start 2024-09-18T12:35:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.758+0000 7f37eb11d640 1 -- start start 2024-09-18T12:35:27.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.758+0000 7f37eb11d640 10 monclient: init 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37e4131ab0 0x7f37e4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient(hunting): picked mon.noname-c con 0x7f37e4131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37e409e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37e4136bb0 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e409faf0 con 0x7f37e409e910 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e409fd30 con 0x7f37e4136bb0 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e40a1110 con 0x7f37e4131ab0 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37eb11d640 10 monclient(hunting): authenticate will time out at 1355.081336s 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f37e409e910 0x7f37e409ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57320/0 (socket says 172.21.15.116:57320) 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e3fff640 1 -- 172.21.15.116:0/1492710237 learned_addr learned my addr 172.21.15.116:0/1492710237 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 1 --2- 172.21.15.116:0/1492710237 >> v2:172.21.15.144:3300/0 conn(0x7f37e4131ab0 0x7f37e4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): get_auth_request con 0x7f37e4131ab0 auth_method 0 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_done global_id 124369 payload 274 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 1 -- 172.21.15.116:0/1492710237 >> v1:172.21.15.144:6789/0 conn(0x7f37e4136bb0 legacy=0x7f37e4138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 1 -- 172.21.15.116:0/1492710237 >> v1:172.21.15.116:6789/0 conn(0x7f37e409e910 legacy=0x7f37e409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 1 monclient: found mon.noname-c 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:35:27.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.759+0000 7f37e8eb9640 1 -- 172.21.15.116:0/1492710237 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40a13f0 con 0x7f37e4131ab0 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.762+0000 7f37e8eb9640 1 -- 172.21.15.116:0/1492710237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 msgr2=0x7f37e4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.762+0000 7f37e8eb9640 1 --2- 172.21.15.116:0/1492710237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 0x7f37e4131e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009880 tx=0x7f37dc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd4670da621fd1b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 1 -- 172.21.15.116:0/1492710237 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc032440 con 0x7f37e4131ab0 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: dump: 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 1 -- 172.21.15.116:0/1492710237 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc0325a0 con 0x7f37e4131ab0 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 1 -- 172.21.15.116:0/1492710237 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc032940 con 0x7f37e4131ab0 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 5 monclient: authenticate success, global_id 124369 2024-09-18T12:35:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 4 set_mon_vals no callback set 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37e37fe640 10 monclient: dump: 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.773 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 10 monclient: shutdown 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 1 -- 172.21.15.116:0/1492710237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 msgr2=0x7f37e4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 1 --2- 172.21.15.116:0/1492710237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 0x7f37e4131e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009880 tx=0x7f37dc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 1 -- 172.21.15.116:0/1492710237 shutdown_connections 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 1 --2- 172.21.15.116:0/1492710237 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 0x7f37e4131e80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 1 -- 172.21.15.116:0/1492710237 >> 172.21.15.116:0/1492710237 conn(0x7f37e4067d30 msgr2=0x7f37e40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 1 -- 172.21.15.116:0/1492710237 shutdown_connections 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.763+0000 7f37eb11d640 1 -- 172.21.15.116:0/1492710237 wait complete. 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.764+0000 7f37eb11d640 10 monclient: build_initial_monmap 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.764+0000 7f37eb11d640 10 monclient: monmap: 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.764+0000 7f37eb11d640 1 Processor -- start 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.764+0000 7f37eb11d640 1 -- start start 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.764+0000 7f37eb11d640 10 monclient: init 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:35:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e409e910 0x7f37e4135fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): picked mon.noname-a con 0x7f37e409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 0x7f37e4096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): picked mon.noname-b con 0x7f37e4131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37e4136bb0 0x7f37e4096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): picked mon.noname-c con 0x7f37e4136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e409bde0 con 0x7f37e409e910 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e413a410 con 0x7f37e4131ab0 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4099bf0 con 0x7f37e4136bb0 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:27.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37eb11d640 10 monclient(hunting): authenticate will time out at 1355.087183s 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e96ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37e4136bb0 msgr2=0x7f37e4096fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e96ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37e4136bb0 0x7f37e4096fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e409e910 0x7f37e4135fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e409e910 0x7f37e4135fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44576/0 (socket says 172.21.15.116:44576) 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 -- 172.21.15.116:0/2486399336 learned_addr learned my addr 172.21.15.116:0/2486399336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): get_auth_request con 0x7f37e409e910 auth_method 0 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 0x7f37e4096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 10 monclient(hunting): get_auth_request con 0x7f37e4131ab0 auth_method 0 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient(hunting): handle_auth_done global_id 104877 payload 824 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37e4136bb0 msgr2=0x7f37e4096fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37e4136bb0 0x7f37e4096fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 msgr2=0x7f37e4096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4131ab0 0x7f37e4096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient: _finish_hunting 0 2024-09-18T12:35:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 monclient: found mon.noname-a 2024-09-18T12:35:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.765+0000 7f37e8eb9640 1 -- 172.21.15.116:0/2486399336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc009390 con 0x7f37e409e910 2024-09-18T12:35:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.767+0000 7f37e8eb9640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e409e910 msgr2=0x7f37e4135fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.767+0000 7f37e8eb9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e409e910 0x7f37e4135fd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009360 tx=0x7f37dc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.767+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc031990 con 0x7f37e409e910 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: dump: 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: _finish_auth 0 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37eb11d640 5 monclient: authenticate success, global_id 104877 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc031af0 con 0x7f37e409e910 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37eb11d640 10 monclient: _renew_subs 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37eb11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e4097e50 con 0x7f37e409e910 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc03b400 con 0x7f37e409e910 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e0ff9640 10 monclient: dump: 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.785 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37eb11d640 10 monclient: _renew_subs 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37eb11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e409b850 con 0x7f37e409e910 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.769+0000 7f37e2ffd640 4 set_mon_vals no callback set 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37eb11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37eb11d640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 0x7f37e41326a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f37dc039040 con 0x7f37e409e910 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37eb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4063740 con 0x7f37e40974e0 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e0ff9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37c40384c0 0x7f37c403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e3fff640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 0x7f37e41326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e3fff640 10 monclient: get_auth_request con 0x7f37e40974e0 auth_method 0 2024-09-18T12:35:27.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.770+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f37dc06c2d0 con 0x7f37e409e910 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.771+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.771+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.771+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.771+0000 7f37e96ba640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37c40384c0 0x7f37c403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.771+0000 7f37e96ba640 10 monclient: get_auth_request con 0x7f37c40384c0 auth_method 0 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.771+0000 7f37e3fff640 10 monclient(hunting): handle_auth_done global_id 124375 payload 824 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.771+0000 7f37e3fff640 1 -- 172.21.15.116:0/2486399336 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37e4099e30 con 0x7f37e40974e0 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e3fff640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 msgr2=0x7f37e41326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e3fff640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 0x7f37e41326a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f37d40079a0 tx=0x7f37d4005330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e96ba640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37c40384c0 0x7f37c403a970 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f37cc007fd0 tx=0x7f37cc007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4007a90 con 0x7f37e40974e0 2024-09-18T12:35:27.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e0ff9640 10 monclient: dump: 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.773+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f37d400fd50 con 0x7f37e40974e0 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.774+0000 7f37e0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.774+0000 7f37e0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.774+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 msgr2=0x7f37e41326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.774+0000 7f37e0ff9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 0x7f37e41326a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f37d40079a0 tx=0x7f37d4005330 comp rx=0 tx=0).stop 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.774+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 msgr2=0x7f37e41326a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.774+0000 7f37e0ff9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e40974e0 0x7f37e41326a0 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f37d40079a0 tx=0x7f37d4005330 comp rx=0 tx=0).stop 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.779+0000 7f37eb11d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.779+0000 7f37eb11d640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 0x7f37e4132fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.779+0000 7f37eb11d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.779+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40978b0 con 0x7f37e4132be0 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.779+0000 7f37e3fff640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 0x7f37e4132fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient: get_auth_request con 0x7f37e4132be0 auth_method 0 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:27.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:27.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:27.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:27.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:27.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:27.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 10 monclient(hunting): handle_auth_done global_id 124378 payload 824 2024-09-18T12:35:27.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.780+0000 7f37e3fff640 1 -- 172.21.15.116:0/2486399336 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f37e404bf60 con 0x7f37e4132be0 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.782+0000 7f37e3fff640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 msgr2=0x7f37e4132fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.782+0000 7f37e3fff640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 0x7f37e4132fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f37d4017000 tx=0x7f37d4018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4005ea0 con 0x7f37e4132be0 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 10 monclient: dump: 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:27.791 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f37d4021880 con 0x7f37e4132be0 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 msgr2=0x7f37e4132fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 0x7f37e4132fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f37d4017000 tx=0x7f37d4018000 comp rx=0 tx=0).stop 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 msgr2=0x7f37e4132fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e0ff9640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37e4132be0 0x7f37e4132fe0 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f37d4017000 tx=0x7f37d4018000 comp rx=0 tx=0).stop 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 102, 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:35:27.793 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:27.794 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:27.795 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:35:27.796 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:35:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:35:27.798 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:27.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:35:27.800 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:35:27.801 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.802 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.803 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.804 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:35:27.805 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37c40384c0 msgr2=0x7f37c403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37c40384c0 0x7f37c403a970 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f37cc007fd0 tx=0x7f37cc007a70 comp rx=0 tx=0).stop 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 10 monclient: shutdown 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e409e910 msgr2=0x7f37e4135fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 --2- 172.21.15.116:0/2486399336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e409e910 0x7f37e4135fd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009360 tx=0x7f37dc032000 comp rx=0 tx=0).stop 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e96ba640 1 -- 172.21.15.116:0/2486399336 reap_dead start 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37e96ba640 1 -- 172.21.15.116:0/2486399336 reap_dead start 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 shutdown_connections 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 >> 172.21.15.116:0/2486399336 conn(0x7f37e4067d30 msgr2=0x7f37e409d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 shutdown_connections 2024-09-18T12:35:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:27.783+0000 7f37eb11d640 1 -- 172.21.15.116:0/2486399336 wait complete. 2024-09-18T12:35:27.808 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:35:27.808 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:35:47.808 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:35:47.809 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:35:47.809 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:35:47.810 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:35:47.814 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:35:47.814 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:35:47.815 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.918+0000 7f7b8c26f640 10 monclient: get_monmap_and_config 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.918+0000 7f7b8c26f640 10 monclient: build_initial_monmap 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.918+0000 7f7b8c26f640 10 monclient: monmap: 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:47.918860+0000 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:47.918860+0000 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:47.919 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:47.920 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:47.920 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:47.920 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:47.920 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:47.920 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.918+0000 7f7b8c26f640 1 Processor -- start 2024-09-18T12:35:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.918+0000 7f7b8c26f640 1 -- start start 2024-09-18T12:35:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.918+0000 7f7b8c26f640 10 monclient: init 2024-09-18T12:35:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T12:35:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b84131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient(hunting): picked mon.noname-b con 0x7f7b841318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7b8409ea50 0x7f7b8409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient(hunting): picked mon.noname-c con 0x7f7b8409ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b84068d90 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b84131cc0 con 0x7f7b84068d90 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8409fbf0 con 0x7f7b8409ea50 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8409fe30 con 0x7f7b841318f0 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8c26f640 10 monclient(hunting): authenticate will time out at 1375.240610s 2024-09-18T12:35:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b84131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a80c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7b84068d90 0x7f7b840691f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38928/0 (socket says 172.21.15.116:38928) 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a80c640 1 -- 172.21.15.116:0/3006099452 learned_addr learned my addr 172.21.15.116:0/3006099452 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 10 monclient(hunting): get_auth_request con 0x7f7b841318f0 auth_method 0 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 1 -- 172.21.15.116:0/3006099452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 msgr2=0x7f7b84131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 1 -- 172.21.15.116:0/3006099452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 msgr2=0x7f7b84131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 1 --2- 172.21.15.116:0/3006099452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b84131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8a00b640 1 --2- 172.21.15.116:0/3006099452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b84131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8980a640 1 --2- 172.21.15.116:0/3006099452 >> v2:172.21.15.144:3300/0 conn(0x7f7b8409ea50 0x7f7b8409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8980a640 10 monclient(hunting): get_auth_request con 0x7f7b8409ea50 auth_method 0 2024-09-18T12:35:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8980a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8980a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.919+0000 7f7b8980a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 10 monclient(hunting): handle_auth_done global_id 124381 payload 274 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 1 -- 172.21.15.116:0/3006099452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 msgr2=0x7f7b84131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 1 --2- 172.21.15.116:0/3006099452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b84131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 1 -- 172.21.15.116:0/3006099452 >> v1:172.21.15.116:6789/0 conn(0x7f7b84068d90 legacy=0x7f7b840691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 10 monclient: _finish_hunting 0 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 1 monclient: found mon.noname-c 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:35:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.920+0000 7f7b8980a640 1 -- 172.21.15.116:0/3006099452 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840a1210 con 0x7f7b8409ea50 2024-09-18T12:35:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.923+0000 7f7b8980a640 1 -- 172.21.15.116:0/3006099452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 msgr2=0x7f7b8409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.923+0000 7f7b8980a640 1 --2- 172.21.15.116:0/3006099452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 0x7f7b8409ee20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009850 tx=0x7f7b740090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e754cc85435398a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.923+0000 7f7b89009640 1 -- 172.21.15.116:0/3006099452 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7402add0 con 0x7f7b8409ea50 2024-09-18T12:35:47.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.923+0000 7f7b89009640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.923+0000 7f7b89009640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.923+0000 7f7b89009640 10 monclient: dump: 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:47.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 10 monclient: _finish_auth 0 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 5 monclient: authenticate success, global_id 124381 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 1 -- 172.21.15.116:0/3006099452 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b74032440 con 0x7f7b8409ea50 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 4 set_mon_vals no callback set 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 1 -- 172.21.15.116:0/3006099452 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b740327e0 con 0x7f7b8409ea50 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b89009640 10 monclient: dump: 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:47.935 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 10 monclient: shutdown 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 -- 172.21.15.116:0/3006099452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 msgr2=0x7f7b8409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 --2- 172.21.15.116:0/3006099452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 0x7f7b8409ee20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009850 tx=0x7f7b740090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 -- 172.21.15.116:0/3006099452 shutdown_connections 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 --2- 172.21.15.116:0/3006099452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 0x7f7b8409ee20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 --2- 172.21.15.116:0/3006099452 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b84131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 -- 172.21.15.116:0/3006099452 >> 172.21.15.116:0/3006099452 conn(0x7f7b84067110 msgr2=0x7f7b84067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 -- 172.21.15.116:0/3006099452 shutdown_connections 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.924+0000 7f7b8c26f640 1 -- 172.21.15.116:0/3006099452 wait complete. 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.925+0000 7f7b8c26f640 10 monclient: build_initial_monmap 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.925+0000 7f7b8c26f640 10 monclient: monmap: 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.925+0000 7f7b8c26f640 1 Processor -- start 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.925+0000 7f7b8c26f640 1 -- start start 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.925+0000 7f7b8c26f640 10 monclient: init 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 0x7f7b8409ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): picked mon.noname-a con 0x7f7b84068d90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 0x7f7b84130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): picked mon.noname-b con 0x7f7b8409ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b8411b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): picked mon.noname-c con 0x7f7b841318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b840a4d20 con 0x7f7b84068d90 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8411dd90 con 0x7f7b8409ea50 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8411e070 con 0x7f7b841318f0 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8c26f640 10 monclient(hunting): authenticate will time out at 1375.247200s 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 0x7f7b8409ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b8411b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 0x7f7b8409ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41706/0 (socket says 172.21.15.116:41706) 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 -- 172.21.15.116:0/1528915713 learned_addr learned my addr 172.21.15.116:0/1528915713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): get_auth_request con 0x7f7b84068d90 auth_method 0 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 10 monclient(hunting): get_auth_request con 0x7f7b841318f0 auth_method 0 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 1 -- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 msgr2=0x7f7b8411b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 1 -- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 msgr2=0x7f7b8411b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b8411b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 0x7f7b84130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 10 monclient(hunting): get_auth_request con 0x7f7b8409ea50 auth_method 0 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a80c640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b8411b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8980a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient(hunting): handle_auth_done global_id 104880 payload 824 2024-09-18T12:35:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 -- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 msgr2=0x7f7b8411b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b8411b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 -- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 msgr2=0x7f7b84130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 0x7f7b84130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient: _finish_hunting 0 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 monclient: found mon.noname-a 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.926+0000 7f7b8a00b640 1 -- 172.21.15.116:0/1528915713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74009390 con 0x7f7b84068d90 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.928+0000 7f7b8a00b640 1 -- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 msgr2=0x7f7b8409ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.928+0000 7f7b8a00b640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 0x7f7b8409ac80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000d890 tx=0x7f7b80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 1 -- 172.21.15.116:0/1528915713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000de20 con 0x7f7b84068d90 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: dump: 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:35:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b8c26f640 5 monclient: authenticate success, global_id 104880 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b8c26f640 10 monclient: _renew_subs 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b8c26f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 1 -- 172.21.15.116:0/1528915713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8000bae0 con 0x7f7b84068d90 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b8c26f640 1 -- 172.21.15.116:0/1528915713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8411e350 con 0x7f7b84068d90 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 1 -- 172.21.15.116:0/1528915713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000bea0 con 0x7f7b84068d90 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 10 monclient: dump: 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b88808640 4 set_mon_vals no callback set 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b8c26f640 10 monclient: _renew_subs 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b8c26f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b8c26f640 1 -- 172.21.15.116:0/1528915713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8411cea0 con 0x7f7b84068d90 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.929+0000 7f7b7a7fc640 1 -- 172.21.15.116:0/1528915713 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7b80014640 con 0x7f7b84068d90 2024-09-18T12:35:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b67fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b67fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b67fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b7a7fc640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b68038490 0x7f7b6803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8404bf60 con 0x7f7b84068d90 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b7a7fc640 1 -- 172.21.15.116:0/1528915713 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7b8004c8d0 con 0x7f7b84068d90 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b8980a640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b68038490 0x7f7b6803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.930+0000 7f7b8980a640 10 monclient: get_auth_request con 0x7f7b68038490 auth_method 0 2024-09-18T12:35:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.933+0000 7f7b8980a640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b68038490 0x7f7b6803a940 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7b74031000 tx=0x7f7b74032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.934+0000 7f7b7a7fc640 1 -- 172.21.15.116:0/1528915713 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7b8001e460 con 0x7f7b84068d90 2024-09-18T12:35:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.934+0000 7f7b7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:47.934+0000 7f7b7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.150+0000 7f7b67fff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.150+0000 7f7b67fff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.150+0000 7f7b67fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.150+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7b8411e540 con 0x7f7b84068d90 2024-09-18T12:35:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.151+0000 7f7b7a7fc640 1 -- 172.21.15.116:0/1528915713 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f7b80042610 con 0x7f7b84068d90 2024-09-18T12:35:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.151+0000 7f7b7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.151+0000 7f7b7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:48.152 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":102,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b68038490 msgr2=0x7f7b6803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b68038490 0x7f7b6803a940 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7b74031000 tx=0x7f7b74032000 comp rx=0 tx=0).stop 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 10 monclient: shutdown 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 msgr2=0x7f7b8409ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 0x7f7b8409ac80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000d890 tx=0x7f7b80008040 comp rx=0 tx=0).stop 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 shutdown_connections 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b68038490 0x7f7b6803a940 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b841318f0 0x7f7b8411b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8409ea50 0x7f7b84130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 --2- 172.21.15.116:0/1528915713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b84068d90 0x7f7b8409ac80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 >> 172.21.15.116:0/1528915713 conn(0x7f7b84067110 msgr2=0x7f7b8409dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 shutdown_connections 2024-09-18T12:35:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:48.153+0000 7f7b67fff640 1 -- 172.21.15.116:0/1528915713 wait complete. 2024-09-18T12:35:51.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.267+0000 7ffbae499640 10 monclient: get_monmap_and_config 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.267+0000 7ffbae499640 10 monclient: build_initial_monmap 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.267+0000 7ffbae499640 10 monclient: monmap: 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:51.268479+0000 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:51.268479+0000 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.268+0000 7ffbae499640 1 Processor -- start 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.268+0000 7ffbae499640 1 -- start start 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.268+0000 7ffbae499640 10 monclient: init 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffba809a510 0x7ffba8096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient(hunting): picked mon.noname-c con 0x7ffba809a510 addr v2:172.21.15.144:3300/0 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffba80971e0 0x7ffba80975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient(hunting): picked mon.noname-a con 0x7ffba80971e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffba812f670 addr v1:172.21.15.116:6789/0 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba8097b20 con 0x7ffba812f670 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba809cb00 con 0x7ffba80971e0 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba809a8e0 con 0x7ffba809a510 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbae499640 10 monclient(hunting): authenticate will time out at 1378.590174s 2024-09-18T12:35:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffba80971e0 0x7ffba80975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbaca36640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffba812f670 0x7ffba812fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38940/0 (socket says 172.21.15.116:38940) 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffbaca36640 1 -- 172.21.15.116:0/2504200958 learned_addr learned my addr 172.21.15.116:0/2504200958 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffba80971e0 0x7ffba80975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41746/0 (socket says 172.21.15.116:41746) 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): get_auth_request con 0x7ffba80971e0 auth_method 0 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient(hunting): handle_auth_done global_id 104883 payload 274 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 1 -- 172.21.15.116:0/2504200958 >> v2:172.21.15.144:3300/0 conn(0x7ffba809a510 msgr2=0x7ffba8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 1 --2- 172.21.15.116:0/2504200958 >> v2:172.21.15.144:3300/0 conn(0x7ffba809a510 0x7ffba8096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 1 -- 172.21.15.116:0/2504200958 >> v1:172.21.15.116:6789/0 conn(0x7ffba812f670 legacy=0x7ffba812fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:35:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 1 monclient: found mon.noname-a 2024-09-18T12:35:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.269+0000 7ffba77fe640 1 -- 172.21.15.116:0/2504200958 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba809d590 con 0x7ffba80971e0 2024-09-18T12:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba77fe640 1 -- 172.21.15.116:0/2504200958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba80971e0 msgr2=0x7ffba80975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba77fe640 1 --2- 172.21.15.116:0/2504200958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba80971e0 0x7ffba80975e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ffba809b450 tx=0x7ffb9c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15c9ae8280c422e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 1 -- 172.21.15.116:0/2504200958 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c032440 con 0x7ffba80971e0 2024-09-18T12:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: dump: 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 1 -- 172.21.15.116:0/2504200958 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9c0325a0 con 0x7ffba80971e0 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffbae499640 5 monclient: authenticate success, global_id 104883 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffbae499640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffbae499640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 1 -- 172.21.15.116:0/2504200958 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c032980 con 0x7ffba80971e0 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffbae499640 4 set_mon_vals no callback set 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.272+0000 7ffba6ffd640 10 monclient: dump: 2024-09-18T12:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 10 monclient: shutdown 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 -- 172.21.15.116:0/2504200958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba80971e0 msgr2=0x7ffba80975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 --2- 172.21.15.116:0/2504200958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba80971e0 0x7ffba80975e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ffba809b450 tx=0x7ffb9c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 -- 172.21.15.116:0/2504200958 shutdown_connections 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 --2- 172.21.15.116:0/2504200958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba80971e0 0x7ffba80975e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 --2- 172.21.15.116:0/2504200958 >> v2:172.21.15.144:3300/0 conn(0x7ffba809a510 0x7ffba8096ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 -- 172.21.15.116:0/2504200958 >> 172.21.15.116:0/2504200958 conn(0x7ffba80672b0 msgr2=0x7ffba80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 -- 172.21.15.116:0/2504200958 shutdown_connections 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 1 -- 172.21.15.116:0/2504200958 wait complete. 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 10 monclient: build_initial_monmap 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.273+0000 7ffbae499640 10 monclient: monmap: 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 Processor -- start 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 -- start start 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient: init 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba80971e0 0x7ffba8132f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): picked mon.noname-b con 0x7ffba80971e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 0x7ffba812e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): picked mon.noname-c con 0x7ffba809a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 0x7ffba81192c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): picked mon.noname-a con 0x7ffba812f670 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba811bae0 con 0x7ffba812f670 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba811ad60 con 0x7ffba80971e0 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba81334e0 con 0x7ffba809a510 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbae499640 10 monclient(hunting): authenticate will time out at 1378.595820s 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbaca36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 0x7ffba81192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffba77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 0x7ffba812e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbaca36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 0x7ffba81192c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41752/0 (socket says 172.21.15.116:41752) 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbaca36640 1 -- 172.21.15.116:0/3625552797 learned_addr learned my addr 172.21.15.116:0/3625552797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbaca36640 10 monclient(hunting): get_auth_request con 0x7ffba812f670 auth_method 0 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbaca36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbaca36640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffbaca36640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffba77fe640 10 monclient(hunting): get_auth_request con 0x7ffba809a510 auth_method 0 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffba77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffba77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.274+0000 7ffba77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba77fe640 1 -- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 msgr2=0x7ffba812e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba77fe640 1 -- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 msgr2=0x7ffba812e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba77fe640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 0x7ffba812e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba77fe640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 0x7ffba812e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba7fff640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba80971e0 0x7ffba8132f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba7fff640 10 monclient(hunting): get_auth_request con 0x7ffba80971e0 auth_method 0 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 10 monclient(hunting): handle_auth_done global_id 104886 payload 824 2024-09-18T12:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 1 -- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 msgr2=0x7ffba812e760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 0x7ffba812e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 1 -- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba80971e0 msgr2=0x7ffba8132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba80971e0 0x7ffba8132f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 10 monclient: _finish_hunting 0 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 1 monclient: found mon.noname-a 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.275+0000 7ffbaca36640 1 -- 172.21.15.116:0/3625552797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c009390 con 0x7ffba812f670 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbaca36640 1 -- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 msgr2=0x7ffba81192c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbaca36640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 0x7ffba81192c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ffba000ccc0 tx=0x7ffba0005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 1 -- 172.21.15.116:0/3625552797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba000ce80 con 0x7ffba812f670 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:35:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: dump: 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: _finish_auth 0 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 1 -- 172.21.15.116:0/3625552797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba000aa60 con 0x7ffba812f670 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbae499640 5 monclient: authenticate success, global_id 104886 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbae499640 10 monclient: _renew_subs 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbae499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba811c890 con 0x7ffba812f670 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 1 -- 172.21.15.116:0/3625552797 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba000ae20 con 0x7ffba812f670 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffb8ffff640 10 monclient: dump: 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbae499640 10 monclient: _renew_subs 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbae499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba811cc20 con 0x7ffba812f670 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.277+0000 7ffba67fc640 4 set_mon_vals no callback set 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffb8ffff640 1 -- 172.21.15.116:0/3625552797 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ffba0007300 con 0x7ffba812f670 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffbae499640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:51.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffbae499640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:51.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffbae499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:51.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba804bf60 con 0x7ffba812f670 2024-09-18T12:35:51.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffb8ffff640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb94038490 0x7ffb9403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:51.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffb8ffff640 1 -- 172.21.15.116:0/3625552797 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ffba004c4f0 con 0x7ffba812f670 2024-09-18T12:35:51.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.278+0000 7ffba7fff640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb94038490 0x7ffb9403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:51.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.279+0000 7ffba7fff640 10 monclient: get_auth_request con 0x7ffb94038490 auth_method 0 2024-09-18T12:35:51.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.281+0000 7ffba7fff640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb94038490 0x7ffb9403a940 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:51.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.283+0000 7ffb8ffff640 1 -- 172.21.15.116:0/3625552797 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ffba001a590 con 0x7ffba812f670 2024-09-18T12:35:51.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.283+0000 7ffb8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:51.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.283+0000 7ffb8ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:51.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.498+0000 7ffbae499640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:51.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.498+0000 7ffbae499640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:51.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.498+0000 7ffbae499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:35:51.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.498+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffba80638d0 con 0x7ffba812f670 2024-09-18T12:35:51.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.498+0000 7ffb8ffff640 1 -- 172.21.15.116:0/3625552797 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7ffba0010af0 con 0x7ffba812f670 2024-09-18T12:35:51.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.498+0000 7ffb8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:51.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.498+0000 7ffb8ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:51.500 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":102,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb94038490 msgr2=0x7ffb9403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb94038490 0x7ffb9403a940 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).stop 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 10 monclient: shutdown 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 msgr2=0x7ffba81192c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 0x7ffba81192c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ffba000ccc0 tx=0x7ffba0005370 comp rx=0 tx=0).stop 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 shutdown_connections 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffb94038490 0x7ffb9403a940 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba812f670 0x7ffba81192c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffba809a510 0x7ffba812e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 --2- 172.21.15.116:0/3625552797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffba80971e0 0x7ffba8132f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 >> 172.21.15.116:0/3625552797 conn(0x7ffba80672b0 msgr2=0x7ffba8132140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 shutdown_connections 2024-09-18T12:35:51.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:51.500+0000 7ffbae499640 1 -- 172.21.15.116:0/3625552797 wait complete. 2024-09-18T12:35:54.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.615+0000 7efd4bc20640 10 monclient: get_monmap_and_config 2024-09-18T12:35:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.615+0000 7efd4bc20640 10 monclient: build_initial_monmap 2024-09-18T12:35:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.615+0000 7efd4bc20640 10 monclient: monmap: 2024-09-18T12:35:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:54.615929+0000 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:54.615929+0000 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.615+0000 7efd4bc20640 1 Processor -- start 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 1 -- start start 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient: init 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient(hunting): picked mon.noname-b con 0x7efd44131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd4409e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient(hunting): picked mon.noname-a con 0x7efd44136bb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:35:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4409faf0 con 0x7efd4409e910 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd4409fd30 con 0x7efd44136bb0 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd440a1110 con 0x7efd44131ab0 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4bc20640 10 monclient(hunting): authenticate will time out at 1381.937581s 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45856/0 (socket says 172.21.15.116:45856) 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41756/0 (socket says 172.21.15.116:41756) 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 learned_addr learned my addr 172.21.15.116:0/2541159540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 10 monclient(hunting): get_auth_request con 0x7efd44131ab0 auth_method 0 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request con 0x7efd44136bb0 auth_method 0 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.616+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.619+0000 7efd491bb640 1 -- 172.21.15.116:0/2541159540 >> v1:172.21.15.144:6789/0 conn(0x7efd4409e910 legacy=0x7efd4409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 10 monclient(hunting): get_auth_request con 0x7efd44131ab0 auth_method 0 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request con 0x7efd44136bb0 auth_method 0 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:54.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:35:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:35:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:54.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:54.817+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:35:55.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 10 monclient(hunting): get_auth_request con 0x7efd44131ab0 auth_method 0 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request con 0x7efd44136bb0 auth_method 0 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:55.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:35:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:35:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:55.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:55.218+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:35:56.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 10 monclient(hunting): get_auth_request con 0x7efd44131ab0 auth_method 0 2024-09-18T12:35:56.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:56.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request con 0x7efd44136bb0 auth_method 0 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.019+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.020+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:35:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702826228 0 0) 0x7efd4409faf0 con 0x7efd4409e910 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): my global_id is 124390 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd300034e0 con 0x7efd4409e910 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2231236821 0 0) 0x7efd300034e0 con 0x7efd4409e910 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd44131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd44131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 msgr2=0x7efd44138f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 --2- 172.21.15.116:0/2541159540 >> v2:172.21.15.116:3300/0 conn(0x7efd44136bb0 0x7efd44138f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: _finish_hunting 0 2024-09-18T12:35:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd440a13f0 con 0x7efd4409e910 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: _finish_auth 0 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7efd340017f0 con 0x7efd4409e910 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd4bc20640 5 monclient: authenticate success, global_id 124390 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd4bc20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: dump: 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 >> v1:172.21.15.144:6789/0 conn(0x7efd4409e910 legacy=0x7efd4409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:35:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd30003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): picked mon.c con 0x7efd44131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd300041f0 0x7efd30004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): picked mon.a con 0x7efd300041f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd30004de0 0x7efd30005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): picked mon.b con 0x7efd30004de0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4409fd30 con 0x7efd300041f0 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd440a1110 con 0x7efd30004de0 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3000bd50 con 0x7efd44131ab0 2024-09-18T12:35:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd489ba640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.628+0000 7efd4bc20640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd30003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd30004de0 0x7efd30005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request con 0x7efd30004de0 auth_method 0 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 10 monclient(hunting): get_auth_request con 0x7efd44131ab0 auth_method 0 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd499bc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient(hunting): handle_auth_done global_id 124390 payload 293 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 msgr2=0x7efd30003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd44131ab0 0x7efd30003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd300041f0 msgr2=0x7efd30004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd300041f0 0x7efd30004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient: _finish_hunting 0 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 1 monclient: found mon.b 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:56.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:35:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd300034e0 con 0x7efd30004de0 2024-09-18T12:35:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 10 monclient: _finish_auth 0 2024-09-18T12:35:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.629+0000 7efd4a1bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:56.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.632+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd30004de0 msgr2=0x7efd30005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:56.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.632+0000 7efd4a1bd640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd30004de0 0x7efd30005250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009590 tx=0x7efd3c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed9f32ad2726e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:56.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.632+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c031480 con 0x7efd30004de0 2024-09-18T12:35:56.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.632+0000 7efd489ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:56.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.632+0000 7efd489ba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:56.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.632+0000 7efd489ba640 10 monclient: dump: 2024-09-18T12:35:56.644 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3c02e030 con 0x7efd30004de0 2024-09-18T12:35:56.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd489ba640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 10 monclient: get_monmap_and_config success 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd489ba640 1 -- 172.21.15.116:0/2541159540 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c031850 con 0x7efd30004de0 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 4 set_mon_vals no callback set 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd489ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd489ba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd489ba640 10 monclient: dump: 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:56.646 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 10 monclient: shutdown 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 1 -- 172.21.15.116:0/2541159540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd30004de0 msgr2=0x7efd30005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 1 --2- 172.21.15.116:0/2541159540 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd30004de0 0x7efd30005250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efd3c009590 tx=0x7efd3c00b040 comp rx=0 tx=0).stop 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4a1bd640 1 -- 172.21.15.116:0/2541159540 reap_dead start 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.633+0000 7efd4bc20640 1 -- 172.21.15.116:0/2541159540 shutdown_connections 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.634+0000 7efd4bc20640 1 -- 172.21.15.116:0/2541159540 >> 172.21.15.116:0/2541159540 conn(0x7efd44067d30 msgr2=0x7efd440a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.634+0000 7efd4bc20640 1 -- 172.21.15.116:0/2541159540 shutdown_connections 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.634+0000 7efd4bc20640 1 -- 172.21.15.116:0/2541159540 wait complete. 2024-09-18T12:35:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.634+0000 7efd4bc20640 10 monclient: build_initial_monmap 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.634+0000 7efd4bc20640 10 monclient: monmap: 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 Processor -- start 2024-09-18T12:35:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 -- start start 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient: init 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient: _reopen_session rank -1 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 0x7efd44141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): picked mon.noname-c con 0x7efd30004de0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd4411b840 0x7efd44141790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): picked mon.noname-b con 0x7efd4411b840 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd44099510 0x7efd44099990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): picked mon.noname-a con 0x7efd44099510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4409bde0 con 0x7efd44099510 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:56.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd440a41f0 con 0x7efd4411b840 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4411caf0 con 0x7efd30004de0 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): _renew_subs 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4bc20640 10 monclient(hunting): authenticate will time out at 1383.956746s 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd499bc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 0x7efd44141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4a1bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd44099510 0x7efd44099990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd499bc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 0x7efd44141250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36576/0 (socket says 172.21.15.116:36576) 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd499bc640 1 -- 172.21.15.116:0/3134191798 learned_addr learned my addr 172.21.15.116:0/3134191798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd499bc640 10 monclient(hunting): get_auth_request con 0x7efd30004de0 auth_method 0 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd499bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd499bc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd499bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request con 0x7efd44099510 auth_method 0 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4a1bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4a1bd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd4a1bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.635+0000 7efd491bb640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd4411b840 0x7efd44141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd499bc640 1 -- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 msgr2=0x7efd44141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): get_auth_request con 0x7efd4411b840 auth_method 0 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd499bc640 1 -- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 msgr2=0x7efd44141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd499bc640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 0x7efd44141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd499bc640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 0x7efd44141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:35:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd4a1bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd4a1bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd4a1bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient(hunting): handle_auth_done global_id 124393 payload 824 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 1 -- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 msgr2=0x7efd44141250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 0x7efd44141250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 1 -- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd44099510 msgr2=0x7efd44099990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd44099510 0x7efd44099990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient: _finish_hunting 0 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 1 monclient: found mon.noname-b 2024-09-18T12:35:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.636+0000 7efd491bb640 1 -- 172.21.15.116:0/3134191798 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4411c780 con 0x7efd4411b840 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.638+0000 7efd491bb640 1 -- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd4411b840 msgr2=0x7efd44141790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.638+0000 7efd491bb640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd4411b840 0x7efd44141790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efd34001520 tx=0x7efd34001180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 1 -- 172.21.15.116:0/3134191798 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340075e0 con 0x7efd4411b840 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: dump: 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: _finish_auth 0 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 1 -- 172.21.15.116:0/3134191798 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34005780 con 0x7efd4411b840 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd4bc20640 5 monclient: authenticate success, global_id 124393 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd4bc20640 10 monclient: _renew_subs 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 1 -- 172.21.15.116:0/3134191798 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34005b20 con 0x7efd4411b840 2024-09-18T12:35:56.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd4bc20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4409b0d0 con 0x7efd4411b840 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd3bfff640 4 set_mon_vals no callback set 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd39ffb640 10 monclient: dump: 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:35:56.655 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd4bc20640 10 monclient: _renew_subs 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd4bc20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.639+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd44141cd0 con 0x7efd4411b840 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd39ffb640 1 -- 172.21.15.116:0/3134191798 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7efd3401b3b0 con 0x7efd4411b840 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd39ffb640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd28038490 0x7efd2803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd39ffb640 1 -- 172.21.15.116:0/3134191798 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efd3404b950 con 0x7efd4411b840 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd4bc20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd4bc20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd4bc20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4404bf60 con 0x7efd4411b840 2024-09-18T12:35:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd499bc640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd28038490 0x7efd2803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:35:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.640+0000 7efd499bc640 10 monclient: get_auth_request con 0x7efd28038490 auth_method 0 2024-09-18T12:35:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.643+0000 7efd499bc640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd28038490 0x7efd2803a940 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7efd40012730 tx=0x7efd40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:35:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.646+0000 7efd39ffb640 1 -- 172.21.15.116:0/3134191798 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7efd34017070 con 0x7efd4411b840 2024-09-18T12:35:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.646+0000 7efd39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:35:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.646+0000 7efd39ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:35:56.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.863+0000 7efd4bc20640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:35:56.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.863+0000 7efd4bc20640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.863+0000 7efd4bc20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:35:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.863+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efd44066280 con 0x7efd4411b840 2024-09-18T12:35:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.864+0000 7efd39ffb640 1 -- 172.21.15.116:0/3134191798 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7efd3401be60 con 0x7efd4411b840 2024-09-18T12:35:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.864+0000 7efd39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:35:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.864+0000 7efd39ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:35:56.866 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":106,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:35:56.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd28038490 msgr2=0x7efd2803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:56.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd28038490 0x7efd2803a940 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7efd40012730 tx=0x7efd40008040 comp rx=0 tx=0).stop 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 10 monclient: shutdown 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd4411b840 msgr2=0x7efd44141790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd4411b840 0x7efd44141790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efd34001520 tx=0x7efd34001180 comp rx=0 tx=0).stop 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 shutdown_connections 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efd28038490 0x7efd2803a940 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd44099510 0x7efd44099990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efd4411b840 0x7efd44141790 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 --2- 172.21.15.116:0/3134191798 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efd30004de0 0x7efd44141250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 >> 172.21.15.116:0/3134191798 conn(0x7efd44067d30 msgr2=0x7efd4409d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 shutdown_connections 2024-09-18T12:35:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:56.866+0000 7efd4bc20640 1 -- 172.21.15.116:0/3134191798 wait complete. 2024-09-18T12:35:59.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:35:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.980+0000 7f385759d640 10 monclient: get_monmap_and_config 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.980+0000 7f385759d640 10 monclient: build_initial_monmap 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.980+0000 7f385759d640 10 monclient: monmap: 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:35:59.981584+0000 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:35:59.981584+0000 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:35:59.982 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:35:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:35:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:35:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:35:59.983 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.981+0000 7f385759d640 1 Processor -- start 2024-09-18T12:36:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.981+0000 7f385759d640 1 -- start start 2024-09-18T12:36:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.981+0000 7f385759d640 10 monclient: init 2024-09-18T12:36:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3850096c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f3850097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient(hunting): picked mon.noname-b con 0x7f3850097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3850133b40 0x7f3850135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient(hunting): picked mon.noname-c con 0x7f3850133b40 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385009d2e0 con 0x7f3850096c60 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385009d6b0 con 0x7f3850133b40 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38500992f0 con 0x7f3850097030 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f385759d640 10 monclient(hunting): authenticate will time out at 1387.303359s 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f3850097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f3850097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36580/0 (socket says 172.21.15.116:36580) 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 -- 172.21.15.116:0/358961709 learned_addr learned my addr 172.21.15.116:0/358961709 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): get_auth_request con 0x7f3850097030 auth_method 0 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3855b3a640 1 --2- 172.21.15.116:0/358961709 >> v2:172.21.15.144:3300/0 conn(0x7f3850133b40 0x7f3850135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3855b3a640 10 monclient(hunting): get_auth_request con 0x7f3850133b40 auth_method 0 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3855b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3855b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3855b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient(hunting): handle_auth_done global_id 144101 payload 274 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 -- 172.21.15.116:0/358961709 >> v2:172.21.15.144:3300/0 conn(0x7f3850133b40 msgr2=0x7f3850135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 --2- 172.21.15.116:0/358961709 >> v2:172.21.15.144:3300/0 conn(0x7f3850133b40 0x7f3850135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 -- 172.21.15.116:0/358961709 >> v1:172.21.15.144:6789/0 conn(0x7f3850096c60 legacy=0x7f385009a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 monclient: found mon.noname-b 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3854b38640 1 -- 172.21.15.116:0/358961709 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38500995d0 con 0x7f3850097030 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.982+0000 7f3855b3a640 1 --2- 172.21.15.116:0/358961709 >> v2:172.21.15.144:3300/0 conn(0x7f3850133b40 0x7f3850135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3854b38640 1 -- 172.21.15.116:0/358961709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 msgr2=0x7f3850097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3854b38640 1 --2- 172.21.15.116:0/358961709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f3850097430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3840009910 tx=0x7f38400091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73f0e5cb493b4e7f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 1 -- 172.21.15.116:0/358961709 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384002ae90 con 0x7f3850097030 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: dump: 2024-09-18T12:36:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 1 -- 172.21.15.116:0/358961709 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3840032440 con 0x7f3850097030 2024-09-18T12:36:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f385759d640 5 monclient: authenticate success, global_id 144101 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f385759d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f385759d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 1 -- 172.21.15.116:0/358961709 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38400327e0 con 0x7f3850097030 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f385759d640 4 set_mon_vals no callback set 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.985+0000 7f3847fff640 10 monclient: dump: 2024-09-18T12:36:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 10 monclient: shutdown 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 -- 172.21.15.116:0/358961709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 msgr2=0x7f3850097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 --2- 172.21.15.116:0/358961709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f3850097430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3840009910 tx=0x7f38400091d0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 -- 172.21.15.116:0/358961709 shutdown_connections 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 --2- 172.21.15.116:0/358961709 >> v2:172.21.15.144:3300/0 conn(0x7f3850133b40 0x7f3850135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 --2- 172.21.15.116:0/358961709 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f3850097430 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 -- 172.21.15.116:0/358961709 >> 172.21.15.116:0/358961709 conn(0x7f38500672b0 msgr2=0x7f38500676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 -- 172.21.15.116:0/358961709 shutdown_connections 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.986+0000 7f385759d640 1 -- 172.21.15.116:0/358961709 wait complete. 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.987+0000 7f385759d640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.987+0000 7f385759d640 10 monclient: monmap: 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.987+0000 7f385759d640 1 Processor -- start 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.987+0000 7f385759d640 1 -- start start 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.987+0000 7f385759d640 10 monclient: init 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3850096c60 0x7f385006e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): picked mon.noname-b con 0x7f3850096c60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f385012e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): picked mon.noname-c con 0x7f3850097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3850133b40 0x7f3850117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): picked mon.noname-a con 0x7f3850133b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385009add0 con 0x7f3850133b40 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385009fae0 con 0x7f3850096c60 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38501193a0 con 0x7f3850097030 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f385759d640 10 monclient(hunting): authenticate will time out at 1387.309200s 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f385012e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f385012e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36588/0 (socket says 172.21.15.116:36588) 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3850133b40 0x7f3850117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 1 -- 172.21.15.116:0/587152342 learned_addr learned my addr 172.21.15.116:0/587152342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 10 monclient(hunting): get_auth_request con 0x7f3850097030 auth_method 0 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): get_auth_request con 0x7f3850133b40 auth_method 0 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 1 -- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 msgr2=0x7f385012e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 1 -- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 msgr2=0x7f385012e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f385012e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3854b38640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f385012e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855339640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3850096c60 0x7f385006e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855339640 10 monclient(hunting): get_auth_request con 0x7f3850096c60 auth_method 0 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient(hunting): handle_auth_done global_id 104892 payload 824 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 1 -- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 msgr2=0x7f385012e570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f385012e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 1 -- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3850096c60 msgr2=0x7f385006e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3850096c60 0x7f385006e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 1 monclient: found mon.noname-a 2024-09-18T12:36:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.988+0000 7f3855b3a640 1 -- 172.21.15.116:0/587152342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3840009450 con 0x7f3850133b40 2024-09-18T12:36:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.990+0000 7f3855b3a640 1 -- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3850133b40 msgr2=0x7f3850117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.990+0000 7f3855b3a640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3850133b40 0x7f3850117480 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3848009fa0 tx=0x7f3848009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 1 -- 172.21.15.116:0/587152342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384800de50 con 0x7f3850133b40 2024-09-18T12:36:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: dump: 2024-09-18T12:36:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.020 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 1 -- 172.21.15.116:0/587152342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384800f040 con 0x7f3850133b40 2024-09-18T12:36:00.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f385759d640 5 monclient: authenticate success, global_id 104892 2024-09-18T12:36:00.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f385759d640 10 monclient: _renew_subs 2024-09-18T12:36:00.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f385759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3850119670 con 0x7f3850133b40 2024-09-18T12:36:00.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 1 -- 172.21.15.116:0/587152342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38480127b0 con 0x7f3850133b40 2024-09-18T12:36:00.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:00.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 10 monclient: dump: 2024-09-18T12:36:00.023 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.023 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.023 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.024 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.024 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.024 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.024 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.024 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.025 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.025 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38477fe640 4 set_mon_vals no callback set 2024-09-18T12:36:00.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f385759d640 10 monclient: _renew_subs 2024-09-18T12:36:00.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f385759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385013dcb0 con 0x7f3850133b40 2024-09-18T12:36:00.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.991+0000 7f38457fa640 1 -- 172.21.15.116:0/587152342 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3848012be0 con 0x7f3850133b40 2024-09-18T12:36:00.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f385759d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f385759d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f385759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3850119800 con 0x7f3850133b40 2024-09-18T12:36:00.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f38457fa640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3834038490 0x7f383403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f38457fa640 1 -- 172.21.15.116:0/587152342 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f384804c870 con 0x7f3850133b40 2024-09-18T12:36:00.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f3855339640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3834038490 0x7f383403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.992+0000 7f3855339640 10 monclient: get_auth_request con 0x7f3834038490 auth_method 0 2024-09-18T12:36:00.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.994+0000 7f3855339640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3834038490 0x7f383403a940 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f384c003740 tx=0x7f384c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.996+0000 7f38457fa640 1 -- 172.21.15.116:0/587152342 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3848002880 con 0x7f3850133b40 2024-09-18T12:36:00.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.996+0000 7f38457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:35:59.996+0000 7f38457fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.208+0000 7f385759d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.208+0000 7f385759d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.208+0000 7f385759d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.208+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f385004bf60 con 0x7f3850133b40 2024-09-18T12:36:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.208+0000 7f38457fa640 1 -- 172.21.15.116:0/587152342 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f384801a080 con 0x7f3850133b40 2024-09-18T12:36:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.208+0000 7f38457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.208+0000 7f38457fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:00.210 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3834038490 msgr2=0x7f383403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3834038490 0x7f383403a940 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f384c003740 tx=0x7f384c009040 comp rx=0 tx=0).stop 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 10 monclient: shutdown 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3850133b40 msgr2=0x7f3850117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3850133b40 0x7f3850117480 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3848009fa0 tx=0x7f3848009990 comp rx=0 tx=0).stop 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 shutdown_connections 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3834038490 0x7f383403a940 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3850133b40 0x7f3850117480 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3850097030 0x7f385012e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 --2- 172.21.15.116:0/587152342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3850096c60 0x7f385006e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 >> 172.21.15.116:0/587152342 conn(0x7f38500672b0 msgr2=0x7f38501320e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 shutdown_connections 2024-09-18T12:36:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.210+0000 7f385759d640 1 -- 172.21.15.116:0/587152342 wait complete. 2024-09-18T12:36:00.222 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:36:00.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:36:00.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:36:00.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.314+0000 7fc41288b640 10 monclient: get_monmap_and_config 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.314+0000 7fc41288b640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.314+0000 7fc41288b640 10 monclient: monmap: 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:00.314707+0000 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:00.314707+0000 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:00.315 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:00.316 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:00.316 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:00.316 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.314+0000 7fc41288b640 1 Processor -- start 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.314+0000 7fc41288b640 1 -- start start 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.314+0000 7fc41288b640 10 monclient: init 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c1318f0 0x7fc40c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient(hunting): picked mon.noname-b con 0x7fc40c1318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc40c09ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:00.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc40c136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40c131cc0 con 0x7fc40c09ea50 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40c09f530 con 0x7fc40c136c20 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c09fc40 con 0x7fc40c1318f0 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc41288b640 10 monclient(hunting): authenticate will time out at 1387.636432s 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c1318f0 0x7fc40c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc40c09ea50 0x7fc40c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34572/0 (socket says 172.21.15.116:34572) 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40b7fe640 1 -- 172.21.15.116:0/3527866505 learned_addr learned my addr 172.21.15.116:0/3527866505 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): get_auth_request con 0x7fc40c1318f0 auth_method 0 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient(hunting): handle_auth_done global_id 144107 payload 274 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 1 -- 172.21.15.116:0/3527866505 >> v1:172.21.15.144:6789/0 conn(0x7fc40c136c20 legacy=0x7fc40c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 1 -- 172.21.15.116:0/3527866505 >> v1:172.21.15.116:6789/0 conn(0x7fc40c09ea50 legacy=0x7fc40c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 1 monclient: found mon.noname-b 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.315+0000 7fc40bfff640 1 -- 172.21.15.116:0/3527866505 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c09fe30 con 0x7fc40c1318f0 2024-09-18T12:36:00.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40bfff640 1 -- 172.21.15.116:0/3527866505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c1318f0 msgr2=0x7fc40c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40bfff640 1 --2- 172.21.15.116:0/3527866505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c1318f0 0x7fc40c131f30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009880 tx=0x7fc3fc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fea9da3b07c95f0f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 1 -- 172.21.15.116:0/3527866505 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc02aea0 con 0x7fc40c1318f0 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: dump: 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 1 -- 172.21.15.116:0/3527866505 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc032440 con 0x7fc40c1318f0 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc41288b640 5 monclient: authenticate success, global_id 144107 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc41288b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc41288b640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:00.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 1 -- 172.21.15.116:0/3527866505 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0327e0 con 0x7fc40c1318f0 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc41288b640 4 set_mon_vals no callback set 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.318+0000 7fc40affd640 10 monclient: dump: 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 10 monclient: shutdown 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 1 -- 172.21.15.116:0/3527866505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c1318f0 msgr2=0x7fc40c131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 1 --2- 172.21.15.116:0/3527866505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c1318f0 0x7fc40c131f30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009880 tx=0x7fc3fc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 1 -- 172.21.15.116:0/3527866505 shutdown_connections 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 1 --2- 172.21.15.116:0/3527866505 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c1318f0 0x7fc40c131f30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 1 -- 172.21.15.116:0/3527866505 >> 172.21.15.116:0/3527866505 conn(0x7fc40c067110 msgr2=0x7fc40c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 1 -- 172.21.15.116:0/3527866505 shutdown_connections 2024-09-18T12:36:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.319+0000 7fc41288b640 1 -- 172.21.15.116:0/3527866505 wait complete. 2024-09-18T12:36:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 10 monclient: monmap: 2024-09-18T12:36:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 1 Processor -- start 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 1 -- start start 2024-09-18T12:36:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 10 monclient: init 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 0x7fc40c09b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.320+0000 7fc41288b640 10 monclient(hunting): picked mon.noname-a con 0x7fc40c09ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc40c1318f0 0x7fc40c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 10 monclient(hunting): picked mon.noname-b con 0x7fc40c1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c136c20 0x7fc40c096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 10 monclient(hunting): picked mon.noname-c con 0x7fc40c136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c13a4f0 con 0x7fc40c09ea50 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c099a90 con 0x7fc40c1318f0 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c097c80 con 0x7fc40c136c20 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc41288b640 10 monclient(hunting): authenticate will time out at 1387.642070s 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 0x7fc40c09b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 0x7fc40c09b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50148/0 (socket says 172.21.15.116:50148) 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 -- 172.21.15.116:0/175891042 learned_addr learned my addr 172.21.15.116:0/175891042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): get_auth_request con 0x7fc40c09ea50 auth_method 0 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c136c20 0x7fc40c096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 10 monclient(hunting): get_auth_request con 0x7fc40c136c20 auth_method 0 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc410e28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc40c1318f0 0x7fc40c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40b7fe640 10 monclient(hunting): get_auth_request con 0x7fc40c1318f0 auth_method 0 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient(hunting): handle_auth_done global_id 104895 payload 824 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c136c20 msgr2=0x7fc40c096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc40c136c20 0x7fc40c096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc40c1318f0 msgr2=0x7fc40c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc40c1318f0 0x7fc40c096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 monclient: found mon.noname-a 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.321+0000 7fc40bfff640 1 -- 172.21.15.116:0/175891042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc009390 con 0x7fc40c09ea50 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc40bfff640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 msgr2=0x7fc40c09b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc40bfff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 0x7fc40c09b660 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009880 tx=0x7fc3fc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc02aea0 con 0x7fc40c09ea50 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: dump: 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc031990 con 0x7fc40c09ea50 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc41288b640 5 monclient: authenticate success, global_id 104895 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc41288b640 10 monclient: _renew_subs 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc41288b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c0997b0 con 0x7fc40c09ea50 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc031d50 con 0x7fc40c09ea50 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc3f3fff640 10 monclient: dump: 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc41288b640 10 monclient: _renew_subs 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc41288b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc40a7fc640 4 set_mon_vals no callback set 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.323+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c09a610 con 0x7fc40c09ea50 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc3fc039040 con 0x7fc40c09ea50 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc41288b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc41288b640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 0x7fc40c0974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc41288b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c065e00 con 0x7fc40c098850 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc3f3fff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc3f8038490 0x7fc3f803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc3fc06c610 con 0x7fc40c09ea50 2024-09-18T12:36:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 0x7fc40c0974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 10 monclient: get_auth_request con 0x7fc40c098850 auth_method 0 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.324+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.325+0000 7fc410e28640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc3f8038490 0x7fc3f803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.325+0000 7fc410e28640 10 monclient: get_auth_request con 0x7fc3f8038490 auth_method 0 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.325+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_done global_id 104898 payload 824 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.325+0000 7fc40b7fe640 1 -- 172.21.15.116:0/175891042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc40c099c20 con 0x7fc40c098850 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc40b7fe640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 msgr2=0x7fc40c0974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 0x7fc40c0974e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e740 tx=0x7fc400009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc400004e60 con 0x7fc40c098850 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 10 monclient: dump: 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc410e28640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc3f8038490 0x7fc3f803a940 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc404007fd0 tx=0x7fc4040025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc400007170 con 0x7fc40c098850 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 msgr2=0x7fc40c0974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 0x7fc40c0974e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e740 tx=0x7fc400009040 comp rx=0 tx=0).stop 2024-09-18T12:36:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 msgr2=0x7fc40c0974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:00.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.327+0000 7fc3f3fff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c098850 0x7fc40c0974e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc41288b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc41288b640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 0x7fc40c0a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc41288b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c099f90 con 0x7fc40c0a38e0 2024-09-18T12:36:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 0x7fc40c0a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient: get_auth_request con 0x7fc40c0a38e0 auth_method 0 2024-09-18T12:36:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 10 monclient(hunting): handle_auth_done global_id 104901 payload 824 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.333+0000 7fc40b7fe640 1 -- 172.21.15.116:0/175891042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc40c04bf60 con 0x7fc40c0a38e0 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.335+0000 7fc40b7fe640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 msgr2=0x7fc40c0a3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.335+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 0x7fc40c0a3cb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc400017000 tx=0x7fc400018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.335+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc400020a50 con 0x7fc40c0a38e0 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.335+0000 7fc3f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.335+0000 7fc3f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.335+0000 7fc3f3fff640 10 monclient: dump: 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fc400004c50 con 0x7fc40c0a38e0 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc3f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc3f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 msgr2=0x7fc40c0a3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc3f3fff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 0x7fc40c0a3cb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc400017000 tx=0x7fc400018000 comp rx=0 tx=0).stop 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc3f3fff640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 msgr2=0x7fc40c0a3cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc3f3fff640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c0a38e0 0x7fc40c0a3cb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 108, 2024-09-18T12:36:00.350 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:00.351 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.352 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.352 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.352 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.352 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.352 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.352 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.353 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:00.354 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.355 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:00.357 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:00.360 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:00.361 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:00.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.362 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.362 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.362 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:00.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.362 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.362 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:36:00.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.364 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.364 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:00.364 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:00.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc3f8038490 msgr2=0x7fc3f803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc3f8038490 0x7fc3f803a940 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc404007fd0 tx=0x7fc4040025b0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 10 monclient: shutdown 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 msgr2=0x7fc40c09b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 0x7fc40c09b660 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009880 tx=0x7fc3fc032000 comp rx=0 tx=0).stop 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc410e28640 1 -- 172.21.15.116:0/175891042 reap_dead start 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 shutdown_connections 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 --2- 172.21.15.116:0/175891042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc40c09ea50 0x7fc40c09b660 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 >> 172.21.15.116:0/175891042 conn(0x7fc40c067110 msgr2=0x7fc40c09e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 shutdown_connections 2024-09-18T12:36:00.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.336+0000 7fc41288b640 1 -- 172.21.15.116:0/175891042 wait complete. 2024-09-18T12:36:00.366 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:00.367 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:36:00.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:36:00.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.460+0000 7f43cf59d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:00.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.460+0000 7f43cf59d640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.460+0000 7f43cf59d640 10 monclient: monmap: 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:00.460946+0000 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:00.460946+0000 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.460+0000 7f43cf59d640 1 Processor -- start 2024-09-18T12:36:00.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 1 -- start start 2024-09-18T12:36:00.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient: init 2024-09-18T12:36:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43c809a510 0x7f43c8096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient(hunting): picked mon.noname-a con 0x7f43c809a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43c80971a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80975a0 0x7f43c812fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient(hunting): picked mon.noname-b con 0x7f43c80975a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c809cb00 con 0x7f43c80971a0 2024-09-18T12:36:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c809a8e0 con 0x7f43c809a510 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c806ed80 con 0x7f43c80975a0 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cf59d640 10 monclient(hunting): authenticate will time out at 1387.782533s 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43c809a510 0x7f43c8096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80975a0 0x7f43c812fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43c809a510 0x7f43c8096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50164/0 (socket says 172.21.15.116:50164) 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43ccb38640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f43c80971a0 0x7f43c812f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34580/0 (socket says 172.21.15.116:34580) 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 1 -- 172.21.15.116:0/2853816194 learned_addr learned my addr 172.21.15.116:0/2853816194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): get_auth_request con 0x7f43c80975a0 auth_method 0 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 10 monclient(hunting): get_auth_request con 0x7f43c809a510 auth_method 0 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient(hunting): handle_auth_done global_id 144113 payload 274 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 1 -- 172.21.15.116:0/2853816194 >> v2:172.21.15.116:3300/0 conn(0x7f43c809a510 msgr2=0x7f43c8096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 1 --2- 172.21.15.116:0/2853816194 >> v2:172.21.15.116:3300/0 conn(0x7f43c809a510 0x7f43c8096c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 1 -- 172.21.15.116:0/2853816194 >> v1:172.21.15.116:6789/0 conn(0x7f43c80971a0 legacy=0x7f43c812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 1 monclient: found mon.noname-b 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cdb3a640 1 -- 172.21.15.116:0/2853816194 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c809c820 con 0x7f43c80975a0 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.461+0000 7f43cd339640 1 --2- 172.21.15.116:0/2853816194 >> v2:172.21.15.116:3300/0 conn(0x7f43c809a510 0x7f43c8096c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43cdb3a640 1 -- 172.21.15.116:0/2853816194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80975a0 msgr2=0x7f43c812fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43cdb3a640 1 --2- 172.21.15.116:0/2853816194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80975a0 0x7f43c812fd80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43c806ef70 tx=0x7f43c0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b0c6ee42e5758f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 1 -- 172.21.15.116:0/2853816194 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0007760 con 0x7f43c80975a0 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: dump: 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.477 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 1 -- 172.21.15.116:0/2853816194 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c002d050 con 0x7f43c80975a0 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43cf59d640 5 monclient: authenticate success, global_id 144113 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43cf59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 1 -- 172.21.15.116:0/2853816194 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0007af0 con 0x7f43c80975a0 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43cf59d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43cf59d640 4 set_mon_vals no callback set 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.464+0000 7f43bffff640 10 monclient: dump: 2024-09-18T12:36:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 10 monclient: shutdown 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 -- 172.21.15.116:0/2853816194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80975a0 msgr2=0x7f43c812fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 --2- 172.21.15.116:0/2853816194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80975a0 0x7f43c812fd80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43c806ef70 tx=0x7f43c0009780 comp rx=0 tx=0).stop 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 -- 172.21.15.116:0/2853816194 shutdown_connections 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 --2- 172.21.15.116:0/2853816194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80975a0 0x7f43c812fd80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 --2- 172.21.15.116:0/2853816194 >> v2:172.21.15.116:3300/0 conn(0x7f43c809a510 0x7f43c8096c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 -- 172.21.15.116:0/2853816194 >> 172.21.15.116:0/2853816194 conn(0x7f43c80672b0 msgr2=0x7f43c80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 -- 172.21.15.116:0/2853816194 shutdown_connections 2024-09-18T12:36:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 1 -- 172.21.15.116:0/2853816194 wait complete. 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.465+0000 7f43cf59d640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient: monmap: 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 Processor -- start 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 -- start start 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient: init 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80971a0 0x7f43c813ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): picked mon.noname-c con 0x7f43c80971a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80975a0 0x7f43c813bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): picked mon.noname-a con 0x7f43c80975a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43c809a510 0x7f43c813c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): picked mon.noname-b con 0x7f43c809a510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c811dac0 con 0x7f43c80975a0 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c809c370 con 0x7f43c809a510 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c8120240 con 0x7f43c80971a0 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.466+0000 7f43cf59d640 10 monclient(hunting): authenticate will time out at 1387.787977s 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80975a0 0x7f43c813bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80971a0 0x7f43c813ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80971a0 0x7f43c813ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36614/0 (socket says 172.21.15.116:36614) 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 -- 172.21.15.116:0/374304131 learned_addr learned my addr 172.21.15.116:0/374304131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): get_auth_request con 0x7f43c80971a0 auth_method 0 2024-09-18T12:36:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80975a0 0x7f43c813bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50172/0 (socket says 172.21.15.116:50172) 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 10 monclient(hunting): get_auth_request con 0x7f43c80975a0 auth_method 0 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cdb3a640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43c809a510 0x7f43c813c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cdb3a640 10 monclient(hunting): get_auth_request con 0x7f43c809a510 auth_method 0 2024-09-18T12:36:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cdb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cdb3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cdb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient(hunting): handle_auth_done global_id 144116 payload 824 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43c809a510 msgr2=0x7f43c813c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f43c809a510 0x7f43c813c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80975a0 msgr2=0x7f43c813bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80975a0 0x7f43c813bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 monclient: found mon.noname-c 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43cd339640 1 -- 172.21.15.116:0/374304131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4009450 con 0x7f43c80971a0 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.467+0000 7f43ccb38640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43c80975a0 0x7f43c813bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cd339640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80971a0 msgr2=0x7f43c813ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cd339640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80971a0 0x7f43c813ef30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f43c400ec80 tx=0x7f43c40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4009730 con 0x7f43c80971a0 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: dump: 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cf59d640 5 monclient: authenticate success, global_id 144116 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c400eec0 con 0x7f43c80971a0 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cf59d640 10 monclient: _renew_subs 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cf59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c0009a20 con 0x7f43c80971a0 2024-09-18T12:36:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4013470 con 0x7f43c80971a0 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bd7fa640 10 monclient: dump: 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cf59d640 10 monclient: _renew_subs 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43bf7fe640 4 set_mon_vals no callback set 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cf59d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.469+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c811fc80 con 0x7f43c80971a0 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43cf59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43cf59d640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 0x7f43c811c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43cf59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c8065f20 con 0x7f43c813f470 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f43c4013820 con 0x7f43c80971a0 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 0x7f43c811c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient: get_auth_request con 0x7f43c813f470 auth_method 0 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43bd7fa640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43ac038490 0x7f43ac03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f43c404cb50 con 0x7f43c80971a0 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 10 monclient(hunting): handle_auth_done global_id 144119 payload 824 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.470+0000 7f43ccb38640 1 -- 172.21.15.116:0/374304131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43c804bf60 con 0x7f43c813f470 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.471+0000 7f43cdb3a640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43ac038490 0x7f43ac03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.471+0000 7f43cdb3a640 10 monclient: get_auth_request con 0x7f43ac038490 auth_method 0 2024-09-18T12:36:00.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43ccb38640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 msgr2=0x7f43c811c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43ccb38640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 0x7f43c811c5e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43c806e250 tx=0x7f43b8007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b800fd50 con 0x7f43c813f470 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 10 monclient: dump: 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.495 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f43b800feb0 con 0x7f43c813f470 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 msgr2=0x7f43c811c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 0x7f43c811c5e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43c806e250 tx=0x7f43b8007e40 comp rx=0 tx=0).stop 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 msgr2=0x7f43c811c5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43bd7fa640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c813f470 0x7f43c811c5e0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43c806e250 tx=0x7f43b8007e40 comp rx=0 tx=0).stop 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.473+0000 7f43cdb3a640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43ac038490 0x7f43ac03a940 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f43c0009e60 tx=0x7f43c0037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43cf59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43cf59d640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 0x7f43c811cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43cf59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c811d460 con 0x7f43c811cb20 2024-09-18T12:36:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 0x7f43c811cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient: get_auth_request con 0x7f43c811cb20 auth_method 0 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 10 monclient(hunting): handle_auth_done global_id 144122 payload 824 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.479+0000 7f43ccb38640 1 -- 172.21.15.116:0/374304131 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f43c809ab00 con 0x7f43c811cb20 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.481+0000 7f43ccb38640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 msgr2=0x7f43c811cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.481+0000 7f43ccb38640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 0x7f43c811cf20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f43c806e250 tx=0x7f43b8015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.481+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8021870 con 0x7f43c811cb20 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.481+0000 7f43bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.481+0000 7f43bd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.481+0000 7f43bd7fa640 10 monclient: dump: 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f43b80219d0 con 0x7f43c811cb20 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43bd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43bd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 msgr2=0x7f43c811cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43bd7fa640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 0x7f43c811cf20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f43c806e250 tx=0x7f43b8015000 comp rx=0 tx=0).stop 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43bd7fa640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 msgr2=0x7f43c811cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43bd7fa640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c811cb20 0x7f43c811cf20 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f43c806e250 tx=0x7f43b8015000 comp rx=0 tx=0).stop 2024-09-18T12:36:00.499 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 108, 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:00.500 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.501 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:00.503 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.504 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.505 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.506 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:00.509 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:00.511 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:00.512 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43ac038490 msgr2=0x7f43ac03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f43ac038490 0x7f43ac03a940 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f43c0009e60 tx=0x7f43c0037000 comp rx=0 tx=0).stop 2024-09-18T12:36:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 10 monclient: shutdown 2024-09-18T12:36:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80971a0 msgr2=0x7f43c813ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 --2- 172.21.15.116:0/374304131 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f43c80971a0 0x7f43c813ef30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f43c400ec80 tx=0x7f43c40091d0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cdb3a640 1 -- 172.21.15.116:0/374304131 reap_dead start 2024-09-18T12:36:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cdb3a640 1 -- 172.21.15.116:0/374304131 reap_dead start 2024-09-18T12:36:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 shutdown_connections 2024-09-18T12:36:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 >> 172.21.15.116:0/374304131 conn(0x7f43c80672b0 msgr2=0x7f43c806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 shutdown_connections 2024-09-18T12:36:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.482+0000 7f43cf59d640 1 -- 172.21.15.116:0/374304131 wait complete. 2024-09-18T12:36:00.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:00.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:36:00.515 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:36:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.606+0000 7f2d00b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.606+0000 7f2d00b9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.606+0000 7f2d00b9d640 10 monclient: monmap: 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:00.607565+0000 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:00.607565+0000 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:00.609 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.607+0000 7f2d00b9d640 1 Processor -- start 2024-09-18T12:36:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.607+0000 7f2d00b9d640 1 -- start start 2024-09-18T12:36:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.607+0000 7f2d00b9d640 10 monclient: init 2024-09-18T12:36:00.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2cfc09a510 0x7f2cfc096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2cfc09a510 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc0975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2cfc0971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2cfc12f670 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc06ee10 con 0x7f2cfc12f670 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc09cb20 con 0x7f2cfc09a510 2024-09-18T12:36:00.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc097af0 con 0x7f2cfc0971e0 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2d00b9d640 10 monclient(hunting): authenticate will time out at 1387.929210s 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cfad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2cfc12f670 0x7f2cfc12fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34586/0 (socket says 172.21.15.116:34586) 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc0975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cfad9d640 1 -- 172.21.15.116:0/3926835604 learned_addr learned my addr 172.21.15.116:0/3926835604 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): get_auth_request con 0x7f2cfc0971e0 auth_method 0 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_done global_id 144125 payload 274 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3926835604 >> v2:172.21.15.144:3300/0 conn(0x7f2cfc09a510 msgr2=0x7f2cfc096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cfa59c640 1 --2- 172.21.15.116:0/3926835604 >> v2:172.21.15.144:3300/0 conn(0x7f2cfc09a510 0x7f2cfc096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 1 --2- 172.21.15.116:0/3926835604 >> v2:172.21.15.144:3300/0 conn(0x7f2cfc09a510 0x7f2cfc096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3926835604 >> v1:172.21.15.116:6789/0 conn(0x7f2cfc12f670 legacy=0x7f2cfc12fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 1 monclient: found mon.noname-b 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.608+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3926835604 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc09a8e0 con 0x7f2cfc0971e0 2024-09-18T12:36:00.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3926835604 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 msgr2=0x7f2cfc0975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf9d9b640 1 --2- 172.21.15.116:0/3926835604 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc0975b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009850 tx=0x7f2cf00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b74d553d8f221c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 1 -- 172.21.15.116:0/3926835604 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0032440 con 0x7f2cfc0971e0 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: dump: 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.623 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.624 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.624 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.624 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.624 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.624 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.625 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2d00b9d640 5 monclient: authenticate success, global_id 144125 2024-09-18T12:36:00.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2d00b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 1 -- 172.21.15.116:0/3926835604 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf00325a0 con 0x7f2cfc0971e0 2024-09-18T12:36:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2d00b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2d00b9d640 4 set_mon_vals no callback set 2024-09-18T12:36:00.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 1 -- 172.21.15.116:0/3926835604 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0032940 con 0x7f2cfc0971e0 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.611+0000 7f2cf959a640 10 monclient: dump: 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.627 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 10 monclient: shutdown 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3926835604 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 msgr2=0x7f2cfc0975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 --2- 172.21.15.116:0/3926835604 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc0975b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009850 tx=0x7f2cf00090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3926835604 shutdown_connections 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 --2- 172.21.15.116:0/3926835604 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc0975b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 --2- 172.21.15.116:0/3926835604 >> v2:172.21.15.144:3300/0 conn(0x7f2cfc09a510 0x7f2cfc096ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3926835604 >> 172.21.15.116:0/3926835604 conn(0x7f2cfc0672b0 msgr2=0x7f2cfc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3926835604 shutdown_connections 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3926835604 wait complete. 2024-09-18T12:36:00.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 10 monclient: monmap: 2024-09-18T12:36:00.629 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.629 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:00.629 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.612+0000 7f2d00b9d640 1 Processor -- start 2024-09-18T12:36:00.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 1 -- start start 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient: init 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2cfc0971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cfc09a510 0x7f2cfc12ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2cfc09a510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12f670 0x7f2cfc098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2cfc12f670 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc11f180 con 0x7f2cfc09a510 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc11ddd0 con 0x7f2cfc12f670 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc12f4b0 con 0x7f2cfc0971e0 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2d00b9d640 10 monclient(hunting): authenticate will time out at 1387.934471s 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cfc09a510 0x7f2cfc12ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36654/0 (socket says 172.21.15.116:36654) 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 -- 172.21.15.116:0/3174218458 learned_addr learned my addr 172.21.15.116:0/3174218458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): get_auth_request con 0x7f2cfc0971e0 auth_method 0 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 10 monclient(hunting): get_auth_request con 0x7f2cfc09a510 auth_method 0 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfad9d640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12f670 0x7f2cfc098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfad9d640 10 monclient(hunting): get_auth_request con 0x7f2cfc12f670 auth_method 0 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient(hunting): handle_auth_done global_id 144128 payload 824 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12f670 msgr2=0x7f2cfc098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12f670 0x7f2cfc098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cfc09a510 msgr2=0x7f2cfc12ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cfc09a510 0x7f2cfc12ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 monclient: found mon.noname-c 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cfa59c640 1 -- 172.21.15.116:0/3174218458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0009390 con 0x7f2cfc0971e0 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.613+0000 7f2cf9d9b640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cfc09a510 0x7f2cfc12ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cfa59c640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 msgr2=0x7f2cfc141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cfa59c640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc141010 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00c880 tx=0x7f2cec008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00ce10 con 0x7f2cfc0971e0 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: dump: 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec007a40 con 0x7f2cfc0971e0 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2d00b9d640 5 monclient: authenticate success, global_id 144128 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2d00b9d640 10 monclient: _renew_subs 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2d00b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc11ebc0 con 0x7f2cfc0971e0 2024-09-18T12:36:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec007de0 con 0x7f2cfc0971e0 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cea7fc640 10 monclient: dump: 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2cf8d99640 4 set_mon_vals no callback set 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2d00b9d640 10 monclient: _renew_subs 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2d00b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.616+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc099d10 con 0x7f2cfc0971e0 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2cec011600 con 0x7f2cfc0971e0 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2d00b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2d00b9d640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 0x7f2cfc098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2d00b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc141f50 con 0x7f2cfc141550 2024-09-18T12:36:00.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cea7fc640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cdc038490 0x7f2cdc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2cec04c110 con 0x7f2cfc0971e0 2024-09-18T12:36:00.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 0x7f2cfc098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cfad9d640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cdc038490 0x7f2cdc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient: get_auth_request con 0x7f2cfc141550 auth_method 0 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cfad9d640 10 monclient: get_auth_request con 0x7f2cdc038490 auth_method 0 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_done global_id 124411 payload 824 2024-09-18T12:36:00.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.617+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3174218458 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2cfc11efb0 con 0x7f2cfc141550 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cfad9d640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cdc038490 0x7f2cdc03a940 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc11daa0 tx=0x7f2cf4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 msgr2=0x7f2cfc098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cf9d9b640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 0x7f2cfc098840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0034000 tx=0x7f2cf0032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0032440 con 0x7f2cfc141550 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 10 monclient: dump: 2024-09-18T12:36:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2cf0040070 con 0x7f2cfc141550 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 msgr2=0x7f2cfc098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 0x7f2cfc098840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0034000 tx=0x7f2cf0032000 comp rx=0 tx=0).stop 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 msgr2=0x7f2cfc098840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.620+0000 7f2cea7fc640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc141550 0x7f2cfc098840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2d00b9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2d00b9d640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 0x7f2cfc098d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2d00b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc065f20 con 0x7f2cfc12efb0 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 0x7f2cfc098d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient: get_auth_request con 0x7f2cfc12efb0 auth_method 0 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 10 monclient(hunting): handle_auth_done global_id 124414 payload 824 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.626+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3174218458 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2cfc04bf60 con 0x7f2cfc12efb0 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cf9d9b640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 msgr2=0x7f2cfc098d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cf9d9b640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 0x7f2cfc098d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc11f570 tx=0x7f2cf003e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf00455c0 con 0x7f2cfc12efb0 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 10 monclient: dump: 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f2cf003c440 con 0x7f2cfc12efb0 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 msgr2=0x7f2cfc098d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 0x7f2cfc098d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc11f570 tx=0x7f2cf003e070 comp rx=0 tx=0).stop 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 msgr2=0x7f2cfc098d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.629+0000 7f2cea7fc640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cfc12efb0 0x7f2cfc098d80 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc11f570 tx=0x7f2cf003e070 comp rx=0 tx=0).stop 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:00.648 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 108, 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:00.649 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.650 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:00.651 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:00.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:00.653 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.655 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.655 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:00.655 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.655 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.660 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.661 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:00.663 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:00.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:00.663 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:00.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:00.663 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cdc038490 msgr2=0x7f2cdc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2cdc038490 0x7f2cdc03a940 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc11daa0 tx=0x7f2cf4009040 comp rx=0 tx=0).stop 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 10 monclient: shutdown 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 msgr2=0x7f2cfc141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 --2- 172.21.15.116:0/3174218458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2cfc0971e0 0x7f2cfc141010 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00c880 tx=0x7f2cec008040 comp rx=0 tx=0).stop 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2cfad9d640 1 -- 172.21.15.116:0/3174218458 reap_dead start 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2cfad9d640 1 -- 172.21.15.116:0/3174218458 reap_dead start 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 shutdown_connections 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 >> 172.21.15.116:0/3174218458 conn(0x7f2cfc0672b0 msgr2=0x7f2cfc09faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 shutdown_connections 2024-09-18T12:36:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.630+0000 7f2d00b9d640 1 -- 172.21.15.116:0/3174218458 wait complete. 2024-09-18T12:36:00.667 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:00.667 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:36:00.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:36:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.758+0000 7f8785d00640 10 monclient: get_monmap_and_config 2024-09-18T12:36:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.758+0000 7f8785d00640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.758+0000 7f8785d00640 10 monclient: monmap: 2024-09-18T12:36:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:00.759206+0000 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:00.759206+0000 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:00.760 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 1 Processor -- start 2024-09-18T12:36:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 1 -- start start 2024-09-18T12:36:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient: init 2024-09-18T12:36:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009c860 0x7f878009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient(hunting): picked mon.noname-b con 0x7f878009c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f878009aba0 0x7f878009d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient(hunting): picked mon.noname-a con 0x7f878009aba0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f878009af70 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878006a320 con 0x7f878009af70 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878009a9d0 con 0x7f878009aba0 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8780068a20 con 0x7f878009c860 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f8785d00640 10 monclient(hunting): authenticate will time out at 1388.080795s 2024-09-18T12:36:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009c860 0x7f878009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f878009aba0 0x7f878009d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f878009aba0 0x7f878009d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50186/0 (socket says 172.21.15.116:50186) 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877effd640 1 -- 172.21.15.116:0/3137399166 learned_addr learned my addr 172.21.15.116:0/3137399166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f878009af70 0x7f878009dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34600/0 (socket says 172.21.15.116:34600) 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009c860 0x7f878009cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36662/0 (socket says 172.21.15.116:36662) 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877effd640 10 monclient(hunting): get_auth_request con 0x7f878009aba0 auth_method 0 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877f7fe640 10 monclient(hunting): get_auth_request con 0x7f878009c860 auth_method 0 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.759+0000 7f877f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 10 monclient(hunting): handle_auth_done global_id 104913 payload 274 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 1 -- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009c860 msgr2=0x7f878009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 1 --2- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009c860 0x7f878009cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 1 -- 172.21.15.116:0/3137399166 >> v1:172.21.15.116:6789/0 conn(0x7f878009af70 legacy=0x7f878009dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 1 monclient: found mon.noname-a 2024-09-18T12:36:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877effd640 1 -- 172.21.15.116:0/3137399166 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780068e60 con 0x7f878009aba0 2024-09-18T12:36:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.760+0000 7f877f7fe640 1 --2- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009c860 0x7f878009cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.762+0000 7f877effd640 1 -- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009aba0 msgr2=0x7f878009d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.762+0000 7f877effd640 1 --2- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009aba0 0x7f878009d830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8768009910 tx=0x7f87680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5475cbb2b1d0b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 1 -- 172.21.15.116:0/3137399166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876802aea0 con 0x7f878009aba0 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: dump: 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 1 -- 172.21.15.116:0/3137399166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8768032440 con 0x7f878009aba0 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 5 monclient: authenticate success, global_id 104913 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 4 set_mon_vals no callback set 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 1 -- 172.21.15.116:0/3137399166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768032820 con 0x7f878009aba0 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f877e7fc640 10 monclient: dump: 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.774 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 10 monclient: shutdown 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 -- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009aba0 msgr2=0x7f878009d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 --2- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009aba0 0x7f878009d830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8768009910 tx=0x7f87680091d0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 -- 172.21.15.116:0/3137399166 shutdown_connections 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 --2- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009aba0 0x7f878009d830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 --2- 172.21.15.116:0/3137399166 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009c860 0x7f878009cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 -- 172.21.15.116:0/3137399166 >> 172.21.15.116:0/3137399166 conn(0x7f87800a2e40 msgr2=0x7f87800a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 -- 172.21.15.116:0/3137399166 shutdown_connections 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.763+0000 7f8785d00640 1 -- 172.21.15.116:0/3137399166 wait complete. 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.764+0000 7f8785d00640 10 monclient: build_initial_monmap 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.764+0000 7f8785d00640 10 monclient: monmap: 2024-09-18T12:36:00.775 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.764+0000 7f8785d00640 1 Processor -- start 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.764+0000 7f8785d00640 1 -- start start 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.764+0000 7f8785d00640 10 monclient: init 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:00.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f878009aba0 0x7f8780133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): picked mon.noname-b con 0x7f878009aba0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009af70 0x7f878011e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): picked mon.noname-c con 0x7f878009af70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009c860 0x7f878011ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): picked mon.noname-a con 0x7f878009c860 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780120240 con 0x7f878009c860 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780121d00 con 0x7f878009aba0 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8780121fe0 con 0x7f878009af70 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:00.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f8785d00640 10 monclient(hunting): authenticate will time out at 1388.086203s 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009af70 0x7f878011e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009c860 0x7f878011ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009af70 0x7f878011e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36664/0 (socket says 172.21.15.116:36664) 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 1 -- 172.21.15.116:0/2890037773 learned_addr learned my addr 172.21.15.116:0/2890037773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 10 monclient(hunting): get_auth_request con 0x7f878009af70 auth_method 0 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): get_auth_request con 0x7f878009c860 auth_method 0 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:00.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient(hunting): handle_auth_done global_id 104916 payload 824 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 1 -- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009af70 msgr2=0x7f878011e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877f7fe640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f878009aba0 0x7f8780133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009af70 0x7f878011e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 1 -- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f878009aba0 msgr2=0x7f8780133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f878009aba0 0x7f8780133bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:00.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 1 monclient: found mon.noname-a 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877ffff640 1 -- 172.21.15.116:0/2890037773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8770009390 con 0x7f878009c860 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.765+0000 7f877effd640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009af70 0x7f878011e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f877ffff640 1 -- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009c860 msgr2=0x7f878011ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f877ffff640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009c860 0x7f878011ecd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f877800c790 tx=0x7f8778004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 1 -- 172.21.15.116:0/2890037773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778005d60 con 0x7f878009c860 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: dump: 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.780 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: _finish_auth 0 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 1 -- 172.21.15.116:0/2890037773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877800a640 con 0x7f878009c860 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f8785d00640 5 monclient: authenticate success, global_id 104916 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:00.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 1 -- 172.21.15.116:0/2890037773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877800aa00 con 0x7f878009c860 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f8785d00640 10 monclient: _renew_subs 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f8785d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8768009450 con 0x7f878009c860 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f87777fe640 10 monclient: dump: 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.767+0000 7f877dffb640 4 set_mon_vals no callback set 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.768+0000 7f8785d00640 10 monclient: _renew_subs 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.768+0000 7f8785d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.768+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8780120da0 con 0x7f878009c860 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.768+0000 7f87777fe640 1 -- 172.21.15.116:0/2890037773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f87780070f0 con 0x7f878009c860 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f8785d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f8785d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f8785d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878004bf60 con 0x7f878009c860 2024-09-18T12:36:00.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f87777fe640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8760038490 0x7f876003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:00.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f87777fe640 1 -- 172.21.15.116:0/2890037773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f877804bfc0 con 0x7f878009c860 2024-09-18T12:36:00.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f877f7fe640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8760038490 0x7f876003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:00.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.769+0000 7f877f7fe640 10 monclient: get_auth_request con 0x7f8760038490 auth_method 0 2024-09-18T12:36:00.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.771+0000 7f877f7fe640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8760038490 0x7f876003a940 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8770006e70 tx=0x7f8770008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:00.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.773+0000 7f87777fe640 1 -- 172.21.15.116:0/2890037773 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f877801b400 con 0x7f878009c860 2024-09-18T12:36:00.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.773+0000 7f87777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:00.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.773+0000 7f87777fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:00.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.987+0000 7f8785d00640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:36:00.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.987+0000 7f8785d00640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:36:00.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.987+0000 7f8785d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:00.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.987+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f87801221a0 con 0x7f878009c860 2024-09-18T12:36:00.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.988+0000 7f87777fe640 1 -- 172.21.15.116:0/2890037773 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f8778010300 con 0x7f878009c860 2024-09-18T12:36:00.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.988+0000 7f87777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:36:00.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.988+0000 7f87777fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:00.989 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.989+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:00.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.989+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=1,health=11,kv=9,logm=56} crc {auth=2698826843,config=1228700967,health=1491445710,kv=2187923472,logm=3142080935}) 2024-09-18T12:36:00.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.989+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=11,kv=9,logm=55} crc {auth=2375170415,config=3140514009,health=4203599156,kv=3580939700,logm=2470125336}) 2024-09-18T12:36:00.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.989+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:00.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.989+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=1,health=11,kv=9,logm=56} crc {auth=2698826843,config=1228700967,health=1491445710,kv=2187923472,logm=3142080935}) 2024-09-18T12:36:00.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.989+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=8,logm=54} crc {auth=3543756603,config=3140514009,health=528802886,kv=2000523877,logm=3702231457}) 2024-09-18T12:36:00.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:00.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753559638}) 2024-09-18T12:36:00.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261082613}) 2024-09-18T12:36:00.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:00.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753559638}) 2024-09-18T12:36:00.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2905411090}) 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8760038490 msgr2=0x7f876003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8760038490 0x7f876003a940 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8770006e70 tx=0x7f8770008040 comp rx=0 tx=0).stop 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 10 monclient: shutdown 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009c860 msgr2=0x7f878011ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009c860 0x7f878011ecd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f877800c790 tx=0x7f8778004e20 comp rx=0 tx=0).stop 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 shutdown_connections 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8760038490 0x7f876003a940 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f878009c860 0x7f878011ecd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f878009af70 0x7f878011e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 --2- 172.21.15.116:0/2890037773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f878009aba0 0x7f8780133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:00.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 >> 172.21.15.116:0/2890037773 conn(0x7f87800a2e40 msgr2=0x7f8780098d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:00.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 shutdown_connections 2024-09-18T12:36:00.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:00.990+0000 7f8785d00640 1 -- 172.21.15.116:0/2890037773 wait complete. 2024-09-18T12:36:01.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.991+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.991+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=519228116}) 2024-09-18T12:36:01.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.991+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149084363}) 2024-09-18T12:36:01.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.991+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.991+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=519228116}) 2024-09-18T12:36:01.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.991+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298317370}) 2024-09-18T12:36:01.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.992+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.992+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559444853}) 2024-09-18T12:36:01.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.992+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199977758}) 2024-09-18T12:36:01.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.992+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.992+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559444853}) 2024-09-18T12:36:01.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.992+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064315924}) 2024-09-18T12:36:01.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.992+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389784587}) 2024-09-18T12:36:01.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843644073}) 2024-09-18T12:36:01.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389784587}) 2024-09-18T12:36:01.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61539589}) 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579252182}) 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3490327376}) 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579252182}) 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.993+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1101518969}) 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.994+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.994+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503043991}) 2024-09-18T12:36:01.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.994+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478483450}) 2024-09-18T12:36:01.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.994+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.994+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503043991}) 2024-09-18T12:36:01.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.994+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726931128}) 2024-09-18T12:36:01.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543960218}) 2024-09-18T12:36:01.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195457913}) 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543960218}) 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1405890062}) 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395825539}) 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129782196}) 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395825539}) 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.995+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1516502675}) 2024-09-18T12:36:01.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.996+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.996+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658277548}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.996+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2707726586}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.996+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.996+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658277548}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.996+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1242892888}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716662915}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919787487}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716662915}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1899958787}) 2024-09-18T12:36:01.007 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1380294521}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=901188018}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1380294521}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.997+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363703522}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048195064}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2830014625}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048195064}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813828366}) 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.008 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735112727}) 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272926693}) 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.998+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735112727}) 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:00.999+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737931139}) 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.000+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.000+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174476373}) 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.000+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2406044085}) 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.000+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.000+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174476373}) 2024-09-18T12:36:01.009 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.000+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1889882287}) 2024-09-18T12:36:01.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.002+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.002+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373485620}) 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.002+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2844515305}) 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.002+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.002+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373485620}) 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.002+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=307777865}) 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.003+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.003+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948948346}) 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.003+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274141005}) 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.003+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.003+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948948346}) 2024-09-18T12:36:01.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.003+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078335895}) 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.004+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.004+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232089843}) 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.004+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734589758}) 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.004+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.004+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232089843}) 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.004+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125470514}) 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171962487}) 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227489000}) 2024-09-18T12:36:01.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171962487}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367264276}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481628122}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489192343}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.005+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481628122}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178797090}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059141295}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1446867506}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059141295}) 2024-09-18T12:36:01.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.006+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1425627462}) 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.007+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.007+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803123145}) 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.007+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514309943}) 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.007+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.007+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803123145}) 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.007+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459179561}) 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.008+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.008+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822691871}) 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.008+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2859952938}) 2024-09-18T12:36:01.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.008+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.008+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822691871}) 2024-09-18T12:36:01.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.008+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042761264}) 2024-09-18T12:36:01.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.009+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.009+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597489286}) 2024-09-18T12:36:01.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.009+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=479846381}) 2024-09-18T12:36:01.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.009+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.009+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597489286}) 2024-09-18T12:36:01.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.009+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367520634}) 2024-09-18T12:36:01.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.010+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.010+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472248818}) 2024-09-18T12:36:01.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.010+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445384616}) 2024-09-18T12:36:01.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.010+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.010+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472248818}) 2024-09-18T12:36:01.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.010+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2136867683}) 2024-09-18T12:36:01.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.011+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.011+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460883182}) 2024-09-18T12:36:01.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.011+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3493205623}) 2024-09-18T12:36:01.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.011+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.011+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460883182}) 2024-09-18T12:36:01.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.011+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3059247598}) 2024-09-18T12:36:01.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.012+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.012+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946664434}) 2024-09-18T12:36:01.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.012+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478112527}) 2024-09-18T12:36:01.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.012+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.012+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946664434}) 2024-09-18T12:36:01.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.012+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4157702237}) 2024-09-18T12:36:01.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.013+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.013+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887494420}) 2024-09-18T12:36:01.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.013+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3967548795}) 2024-09-18T12:36:01.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.013+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.013+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887494420}) 2024-09-18T12:36:01.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.013+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783594735}) 2024-09-18T12:36:01.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.014+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.014+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733667471}) 2024-09-18T12:36:01.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.014+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086433826}) 2024-09-18T12:36:01.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.014+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.014+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733667471}) 2024-09-18T12:36:01.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.014+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077928490}) 2024-09-18T12:36:01.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.015+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.015+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260859672}) 2024-09-18T12:36:01.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.015+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3274102391}) 2024-09-18T12:36:01.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.015+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.015+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260859672}) 2024-09-18T12:36:01.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.015+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697966768}) 2024-09-18T12:36:01.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062189676}) 2024-09-18T12:36:01.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280287437}) 2024-09-18T12:36:01.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062189676}) 2024-09-18T12:36:01.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4231752615}) 2024-09-18T12:36:01.054 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.055 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032685690}) 2024-09-18T12:36:01.055 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648656358}) 2024-09-18T12:36:01.055 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.055 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032685690}) 2024-09-18T12:36:01.055 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.016+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3463753589}) 2024-09-18T12:36:01.056 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.017+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.056 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.017+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180610104}) 2024-09-18T12:36:01.056 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.017+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=802476552}) 2024-09-18T12:36:01.056 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.017+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.056 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.017+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180610104}) 2024-09-18T12:36:01.057 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.017+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3257514053}) 2024-09-18T12:36:01.057 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.018+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.057 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.018+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254377941}) 2024-09-18T12:36:01.057 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.018+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3255303109}) 2024-09-18T12:36:01.057 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.018+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.057 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.018+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254377941}) 2024-09-18T12:36:01.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.018+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744337851}) 2024-09-18T12:36:01.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.019+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.019+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760353554}) 2024-09-18T12:36:01.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.019+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2679333864}) 2024-09-18T12:36:01.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.019+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.019+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760353554}) 2024-09-18T12:36:01.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.019+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826037755}) 2024-09-18T12:36:01.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.020+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.020+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240385924}) 2024-09-18T12:36:01.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.020+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2672497463}) 2024-09-18T12:36:01.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.020+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.020+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240385924}) 2024-09-18T12:36:01.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.020+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423082775}) 2024-09-18T12:36:01.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.021+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.021+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749823169}) 2024-09-18T12:36:01.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.021+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512101972}) 2024-09-18T12:36:01.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.021+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.021+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749823169}) 2024-09-18T12:36:01.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.021+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472454536}) 2024-09-18T12:36:01.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.022+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.022+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823872231}) 2024-09-18T12:36:01.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.022+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2296590590}) 2024-09-18T12:36:01.062 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.022+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.062 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.022+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823872231}) 2024-09-18T12:36:01.062 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.022+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2209000891}) 2024-09-18T12:36:01.062 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.022+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.062 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276678301}) 2024-09-18T12:36:01.062 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432804810}) 2024-09-18T12:36:01.063 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.063 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276678301}) 2024-09-18T12:36:01.063 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1713733691}) 2024-09-18T12:36:01.063 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.063 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924711072}) 2024-09-18T12:36:01.064 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=357189320}) 2024-09-18T12:36:01.064 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.064 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924711072}) 2024-09-18T12:36:01.064 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.023+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3556516447}) 2024-09-18T12:36:01.064 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.064 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918971902}) 2024-09-18T12:36:01.065 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1610472399}) 2024-09-18T12:36:01.065 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.065 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918971902}) 2024-09-18T12:36:01.065 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264484678}) 2024-09-18T12:36:01.065 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946979179}) 2024-09-18T12:36:01.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3762482111}) 2024-09-18T12:36:01.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946979179}) 2024-09-18T12:36:01.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.024+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=599450899}) 2024-09-18T12:36:01.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.025+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.025+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280248929}) 2024-09-18T12:36:01.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.025+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3407891807}) 2024-09-18T12:36:01.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.025+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.025+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280248929}) 2024-09-18T12:36:01.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.025+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857453767}) 2024-09-18T12:36:01.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974738786}) 2024-09-18T12:36:01.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135173936}) 2024-09-18T12:36:01.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974738786}) 2024-09-18T12:36:01.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4091349857}) 2024-09-18T12:36:01.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547182375}) 2024-09-18T12:36:01.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307733983}) 2024-09-18T12:36:01.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547182375}) 2024-09-18T12:36:01.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.026+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1289027305}) 2024-09-18T12:36:01.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872045888}) 2024-09-18T12:36:01.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3364915059}) 2024-09-18T12:36:01.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872045888}) 2024-09-18T12:36:01.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568294144}) 2024-09-18T12:36:01.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181515905}) 2024-09-18T12:36:01.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210095386}) 2024-09-18T12:36:01.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181515905}) 2024-09-18T12:36:01.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.027+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2629825656}) 2024-09-18T12:36:01.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.028+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.028+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299832304}) 2024-09-18T12:36:01.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.028+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=264476921}) 2024-09-18T12:36:01.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.028+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.073 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.028+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299832304}) 2024-09-18T12:36:01.073 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.028+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2472157145}) 2024-09-18T12:36:01.073 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.073 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2473096981}) 2024-09-18T12:36:01.073 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2500804357}) 2024-09-18T12:36:01.074 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.074 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2473096981}) 2024-09-18T12:36:01.074 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31698168}) 2024-09-18T12:36:01.074 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.074 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3978689218}) 2024-09-18T12:36:01.074 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1577888870}) 2024-09-18T12:36:01.075 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.075 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3978689218}) 2024-09-18T12:36:01.075 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.029+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228535330}) 2024-09-18T12:36:01.075 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.031+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.075 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.031+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320701697}) 2024-09-18T12:36:01.076 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.031+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262834849}) 2024-09-18T12:36:01.076 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.031+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.076 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.031+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320701697}) 2024-09-18T12:36:01.076 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.031+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829889806}) 2024-09-18T12:36:01.076 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330649086}) 2024-09-18T12:36:01.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356520267}) 2024-09-18T12:36:01.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330649086}) 2024-09-18T12:36:01.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3899672371}) 2024-09-18T12:36:01.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522434333}) 2024-09-18T12:36:01.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2879923573}) 2024-09-18T12:36:01.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522434333}) 2024-09-18T12:36:01.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.032+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2063429042}) 2024-09-18T12:36:01.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.033+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.033+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151675123}) 2024-09-18T12:36:01.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.033+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3139833346}) 2024-09-18T12:36:01.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.033+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.033+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151675123}) 2024-09-18T12:36:01.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.033+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992466396}) 2024-09-18T12:36:01.080 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.034+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.080 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.034+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=2265201620,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=138568482,mon_config_key=3655569570,monmap=148614106,osd_metadata=2900866072,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1738050859}) 2024-09-18T12:36:01.080 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.034+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=60,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_snap=2,osdmap=6} crc {logm=3625881420,mdsmap=2826658659,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=138568482,mon_config_key=3655569570,monmap=3750608922,osd_metadata=2440437218,osd_snap=2033388227,osdmap=1738050859}) 2024-09-18T12:36:01.080 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.034+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.080 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.034+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=2265201620,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=138568482,mon_config_key=3655569570,monmap=148614106,osd_metadata=2900866072,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1738050859}) 2024-09-18T12:36:01.080 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.034+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=13} crc {logm=2345544594,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=138568482,mon_config_key=3655569570,monmap=148614106,osd_metadata=4175318930,osd_pg_creating=2312870804,osd_snap=3823785925,osdmap=3367206591}) 2024-09-18T12:36:01.081 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.035+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.081 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.035+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3063656503}) 2024-09-18T12:36:01.081 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.035+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2606542666}) 2024-09-18T12:36:01.081 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.035+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.081 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.035+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3063656503}) 2024-09-18T12:36:01.081 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.035+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3306103874}) 2024-09-18T12:36:01.082 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.040+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.082 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.040+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=4293436652}) 2024-09-18T12:36:01.082 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.040+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=21} crc {osdmap=17604436}) 2024-09-18T12:36:01.082 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.040+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:01.082 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.040+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=4293436652}) 2024-09-18T12:36:01.082 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:01.040+0000 7f3cb75f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=1560996123}) 2024-09-18T12:36:02.012 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:36:02.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.115+0000 7f553b289640 10 monclient: get_monmap_and_config 2024-09-18T12:36:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.115+0000 7f553b289640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.115+0000 7f553b289640 10 monclient: monmap: 2024-09-18T12:36:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:02.116161+0000 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:02.116161+0000 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:02.118 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 1 Processor -- start 2024-09-18T12:36:02.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 1 -- start start 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient: init 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f553409a7f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f553409abc0 0x7f553409afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient(hunting): picked mon.noname-a con 0x7f553409abc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 0x7f5534136e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient(hunting): picked mon.noname-b con 0x7f5534134a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553409c800 con 0x7f553409a7f0 2024-09-18T12:36:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55340a2050 con 0x7f553409abc0 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553409cfb0 con 0x7f5534134a30 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.116+0000 7f553b289640 10 monclient(hunting): authenticate will time out at 1389.437918s 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 0x7f5534136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f553409abc0 0x7f553409afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f553409abc0 0x7f553409afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50214/0 (socket says 172.21.15.116:50214) 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 0x7f5534136e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36672/0 (socket says 172.21.15.116:36672) 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 1 -- 172.21.15.116:0/3808045208 learned_addr learned my addr 172.21.15.116:0/3808045208 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:02.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 10 monclient(hunting): get_auth_request con 0x7f553409abc0 auth_method 0 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): get_auth_request con 0x7f5534134a30 auth_method 0 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient(hunting): handle_auth_done global_id 144137 payload 274 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 1 -- 172.21.15.116:0/3808045208 >> v2:172.21.15.116:3300/0 conn(0x7f553409abc0 msgr2=0x7f553409afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 1 --2- 172.21.15.116:0/3808045208 >> v2:172.21.15.116:3300/0 conn(0x7f553409abc0 0x7f553409afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 1 -- 172.21.15.116:0/3808045208 >> v1:172.21.15.144:6789/0 conn(0x7f553409a7f0 legacy=0x7f553409c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 1 monclient: found mon.noname-b 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5539826640 1 -- 172.21.15.116:0/3808045208 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553409d290 con 0x7f5534134a30 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.117+0000 7f5538824640 1 --2- 172.21.15.116:0/3808045208 >> v2:172.21.15.116:3300/0 conn(0x7f553409abc0 0x7f553409afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f5539826640 1 -- 172.21.15.116:0/3808045208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 msgr2=0x7f5534136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f5539826640 1 --2- 172.21.15.116:0/3808045208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 0x7f5534136e10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f552c00f300 tx=0x7f552c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fda50eeec871b01 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 1 -- 172.21.15.116:0/3808045208 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c00d4f0 con 0x7f5534134a30 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: dump: 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.126 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 1 -- 172.21.15.116:0/3808045208 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c00b750 con 0x7f5534134a30 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 5 monclient: authenticate success, global_id 144137 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 1 -- 172.21.15.116:0/3808045208 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c00baf0 con 0x7f5534134a30 2024-09-18T12:36:02.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 4 set_mon_vals no callback set 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f552bfff640 10 monclient: dump: 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.128 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 10 monclient: shutdown 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 1 -- 172.21.15.116:0/3808045208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 msgr2=0x7f5534136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 1 --2- 172.21.15.116:0/3808045208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 0x7f5534136e10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f552c00f300 tx=0x7f552c00db90 comp rx=0 tx=0).stop 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.120+0000 7f553b289640 1 -- 172.21.15.116:0/3808045208 shutdown_connections 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 1 --2- 172.21.15.116:0/3808045208 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5534134a30 0x7f5534136e10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 1 --2- 172.21.15.116:0/3808045208 >> v2:172.21.15.116:3300/0 conn(0x7f553409abc0 0x7f553409afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 1 -- 172.21.15.116:0/3808045208 >> 172.21.15.116:0/3808045208 conn(0x7f5534067b50 msgr2=0x7f55340a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 1 -- 172.21.15.116:0/3808045208 shutdown_connections 2024-09-18T12:36:02.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 1 -- 172.21.15.116:0/3808045208 wait complete. 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 10 monclient: monmap: 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.135 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 1 Processor -- start 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 1 -- start start 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.121+0000 7f553b289640 10 monclient: init 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 0x7f55340891f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): picked mon.noname-c con 0x7f553409a7f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553409abc0 0x7f5534089730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): picked mon.noname-a con 0x7f553409abc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:02.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534134a30 0x7f5534089c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): picked mon.noname-b con 0x7f5534134a30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553409b5a0 con 0x7f553409abc0 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534095900 con 0x7f5534134a30 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553408af00 con 0x7f553409a7f0 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f553b289640 10 monclient(hunting): authenticate will time out at 1389.443595s 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553409abc0 0x7f5534089730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 0x7f55340891f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553409abc0 0x7f5534089730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50220/0 (socket says 172.21.15.116:50220) 2024-09-18T12:36:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 1 -- 172.21.15.116:0/2222334335 learned_addr learned my addr 172.21.15.116:0/2222334335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 0x7f55340891f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36688/0 (socket says 172.21.15.116:36688) 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 10 monclient(hunting): get_auth_request con 0x7f553409abc0 auth_method 0 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): get_auth_request con 0x7f553409a7f0 auth_method 0 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5538824640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539826640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534134a30 0x7f5534089c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539826640 10 monclient(hunting): get_auth_request con 0x7f5534134a30 auth_method 0 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539826640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539826640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 10 monclient(hunting): handle_auth_done global_id 144140 payload 824 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 1 -- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534134a30 msgr2=0x7f5534089c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.122+0000 7f5539025640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534134a30 0x7f5534089c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5539025640 1 -- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553409abc0 msgr2=0x7f5534089730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5539025640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553409abc0 0x7f5534089730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5539025640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5539025640 1 monclient: found mon.noname-c 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5539025640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5539025640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5539025640 1 -- 172.21.15.116:0/2222334335 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5524009450 con 0x7f553409a7f0 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.123+0000 7f5538824640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553409abc0 0x7f5534089730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f5539025640 1 -- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 msgr2=0x7f55340891f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f5539025640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 0x7f55340891f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f553000c4e0 tx=0x7f5530007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 1 -- 172.21.15.116:0/2222334335 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55300117a0 con 0x7f553409a7f0 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: dump: 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f553b289640 5 monclient: authenticate success, global_id 144140 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 1 -- 172.21.15.116:0/2222334335 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5530011920 con 0x7f553409a7f0 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 1 -- 172.21.15.116:0/2222334335 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530011ce0 con 0x7f553409a7f0 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f553b289640 10 monclient: _renew_subs 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f553b289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c00de30 con 0x7f553409a7f0 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f55297fa640 10 monclient: dump: 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.142 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f552b7fe640 4 set_mon_vals no callback set 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f553b289640 10 monclient: _renew_subs 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f553b289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.125+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553408c7a0 con 0x7f553409a7f0 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f55297fa640 1 -- 172.21.15.116:0/2222334335 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5530012630 con 0x7f553409a7f0 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f553b289640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f553b289640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f553b289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553408c390 con 0x7f553409a7f0 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f55297fa640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5518038490 0x7f551803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f55297fa640 1 -- 172.21.15.116:0/2222334335 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f553004be40 con 0x7f553409a7f0 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f5538824640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5518038490 0x7f551803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.126+0000 7f5538824640 10 monclient: get_auth_request con 0x7f5518038490 auth_method 0 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.129+0000 7f5538824640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5518038490 0x7f551803a940 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5524009fd0 tx=0x7f55240091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.131+0000 7f55297fa640 1 -- 172.21.15.116:0/2222334335 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f553408c390 con 0x7f553409a7f0 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.131+0000 7f55297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.131+0000 7f55297fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.344+0000 7f553b289640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.344+0000 7f553b289640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.344+0000 7f553b289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.344+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f553408c390 con 0x7f553409a7f0 2024-09-18T12:36:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.344+0000 7f55297fa640 1 -- 172.21.15.116:0/2222334335 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f553408c390 con 0x7f553409a7f0 2024-09-18T12:36:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.344+0000 7f55297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:02.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.344+0000 7f55297fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:02.346 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:02.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5518038490 msgr2=0x7f551803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5518038490 0x7f551803a940 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5524009fd0 tx=0x7f55240091d0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 10 monclient: shutdown 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 msgr2=0x7f55340891f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 0x7f55340891f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f553000c4e0 tx=0x7f5530007e10 comp rx=0 tx=0).stop 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 shutdown_connections 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5518038490 0x7f551803a940 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5534134a30 0x7f5534089c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553409abc0 0x7f5534089730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 --2- 172.21.15.116:0/2222334335 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f553409a7f0 0x7f55340891f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 >> 172.21.15.116:0/2222334335 conn(0x7f5534067b50 msgr2=0x7f5534099660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 shutdown_connections 2024-09-18T12:36:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.346+0000 7f553b289640 1 -- 172.21.15.116:0/2222334335 wait complete. 2024-09-18T12:36:02.357 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:36:02.357 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:36:02.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:36:02.357 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:36:02.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.450+0000 7fa4b4ee8640 10 monclient: get_monmap_and_config 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.450+0000 7fa4b4ee8640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.450+0000 7fa4b4ee8640 10 monclient: monmap: 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:02.451382+0000 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:02.451382+0000 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:02.452 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:02.453 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:02.453 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:02.453 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 1 Processor -- start 2024-09-18T12:36:02.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 1 -- start start 2024-09-18T12:36:02.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient: init 2024-09-18T12:36:02.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4b01318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4b009ea50 0x7fa4b009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b009ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4b0136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0131cc0 con 0x7fa4b01318f0 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b009fbf0 con 0x7fa4b0136c20 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b009fe30 con 0x7fa4b009ea50 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.451+0000 7fa4b4ee8640 10 monclient(hunting): authenticate will time out at 1389.772940s 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4b009ea50 0x7fa4b009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4ae59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa4b01318f0 0x7fa4b0131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34612/0 (socket says 172.21.15.116:34612) 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): get_auth_request con 0x7fa4b009ea50 auth_method 0 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4ae59c640 1 -- 172.21.15.116:0/1457739451 learned_addr learned my addr 172.21.15.116:0/1457739451 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient(hunting): handle_auth_done global_id 104925 payload 274 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 1 -- 172.21.15.116:0/1457739451 >> v1:172.21.15.144:6789/0 conn(0x7fa4b0136c20 legacy=0x7fa4b0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 1 -- 172.21.15.116:0/1457739451 >> v1:172.21.15.116:6789/0 conn(0x7fa4b01318f0 legacy=0x7fa4b0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 1 monclient: found mon.noname-a 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.452+0000 7fa4add9b640 1 -- 172.21.15.116:0/1457739451 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00a1210 con 0x7fa4b009ea50 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4add9b640 1 -- 172.21.15.116:0/1457739451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b009ea50 msgr2=0x7fa4b009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4add9b640 1 --2- 172.21.15.116:0/1457739451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b009ea50 0x7fa4b009ee20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009850 tx=0x7fa4a40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601e58d3d03b9279 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 1 -- 172.21.15.116:0/1457739451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4032440 con 0x7fa4b009ea50 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: dump: 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.458 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 5 monclient: authenticate success, global_id 104925 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 1 -- 172.21.15.116:0/1457739451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a40325a0 con 0x7fa4b009ea50 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 4 set_mon_vals no callback set 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 1 -- 172.21.15.116:0/1457739451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4032980 con 0x7fa4b009ea50 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4ad59a640 10 monclient: dump: 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.460 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.461 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.461 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 10 monclient: shutdown 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/1457739451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b009ea50 msgr2=0x7fa4b009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 1 --2- 172.21.15.116:0/1457739451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b009ea50 0x7fa4b009ee20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009850 tx=0x7fa4a40090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/1457739451 shutdown_connections 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 1 --2- 172.21.15.116:0/1457739451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b009ea50 0x7fa4b009ee20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/1457739451 >> 172.21.15.116:0/1457739451 conn(0x7fa4b0067110 msgr2=0x7fa4b0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/1457739451 shutdown_connections 2024-09-18T12:36:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.455+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/1457739451 wait complete. 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 10 monclient: monmap: 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 1 Processor -- start 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 1 -- start start 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 10 monclient: init 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4b009ea50 0x7fa4b01329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 10 monclient(hunting): picked mon.noname-b con 0x7fa4b009ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01318f0 0x7fa4b0132f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.456+0000 7fa4b4ee8640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b01318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4b0136c20 0x7fa4b0133470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b0136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0130990 con 0x7fa4b01318f0 2024-09-18T12:36:02.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01361a0 con 0x7fa4b009ea50 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01345e0 con 0x7fa4b0136c20 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4b4ee8640 10 monclient(hunting): authenticate will time out at 1389.778040s 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01318f0 0x7fa4b0132f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01318f0 0x7fa4b0132f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50238/0 (socket says 172.21.15.116:50238) 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 -- 172.21.15.116:0/2181057238 learned_addr learned my addr 172.21.15.116:0/2181057238 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4b0136c20 0x7fa4b0133470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): get_auth_request con 0x7fa4b01318f0 auth_method 0 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 10 monclient(hunting): get_auth_request con 0x7fa4b0136c20 auth_method 0 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4ae59c640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4b009ea50 0x7fa4b01329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4ae59c640 10 monclient(hunting): get_auth_request con 0x7fa4b009ea50 auth_method 0 2024-09-18T12:36:02.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient(hunting): handle_auth_done global_id 104928 payload 824 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4b0136c20 msgr2=0x7fa4b0133470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4b0136c20 0x7fa4b0133470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4b009ea50 msgr2=0x7fa4b01329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4b009ea50 0x7fa4b01329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 monclient: found mon.noname-a 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4add9b640 1 -- 172.21.15.116:0/2181057238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4009390 con 0x7fa4b01318f0 2024-09-18T12:36:02.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4aed9d640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa4b0136c20 0x7fa4b0133470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.457+0000 7fa4ae59c640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4b009ea50 0x7fa4b01329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4add9b640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01318f0 msgr2=0x7fa4b0132f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4add9b640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01318f0 0x7fa4b0132f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009850 tx=0x7fa4a4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4031990 con 0x7fa4b01318f0 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: dump: 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4b4ee8640 5 monclient: authenticate success, global_id 104928 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a4031af0 con 0x7fa4b01318f0 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4b4ee8640 10 monclient: _renew_subs 2024-09-18T12:36:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4b4ee8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b0135ec0 con 0x7fa4b01318f0 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4031e90 con 0x7fa4b01318f0 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa49e7fc640 10 monclient: dump: 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4acd99640 4 set_mon_vals no callback set 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4b4ee8640 10 monclient: _renew_subs 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4b4ee8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.459+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b0135c90 con 0x7fa4b01318f0 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4b4ee8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa4a4039040 con 0x7fa4b01318f0 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4b4ee8640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 0x7fa4b009d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4b4ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0065e00 con 0x7fa4b01339b0 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 0x7fa4b009d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 10 monclient: get_auth_request con 0x7fa4b01339b0 auth_method 0 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa49e7fc640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa490038490 0x7fa49003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.460+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa4a406c6b0 con 0x7fa4b01318f0 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.461+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_done global_id 104931 payload 824 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.461+0000 7fa4ae59c640 1 -- 172.21.15.116:0/2181057238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4b004bf60 con 0x7fa4b01339b0 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.461+0000 7fa4aed9d640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa490038490 0x7fa49003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.461+0000 7fa4aed9d640 10 monclient: get_auth_request con 0x7fa490038490 auth_method 0 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa4ae59c640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 msgr2=0x7fa4b009d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa4ae59c640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 0x7fa4b009d890 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000e740 tx=0x7fa4a0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0005cb0 con 0x7fa4b01339b0 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 10 monclient: dump: 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa4aed9d640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa490038490 0x7fa49003a940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80075e0 tx=0x7fa4a8009780 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa4a0005e10 con 0x7fa4b01339b0 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 msgr2=0x7fa4b009d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 0x7fa4b009d890 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000e740 tx=0x7fa4a0009040 comp rx=0 tx=0).stop 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 msgr2=0x7fa4b009d890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.463+0000 7fa49e7fc640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01339b0 0x7fa4b009d890 secure :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000e740 tx=0x7fa4a0009040 comp rx=0 tx=0).stop 2024-09-18T12:36:02.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4b4ee8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:02.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4b4ee8640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 0x7fa4b009ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4b4ee8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0133d80 con 0x7fa4b0136390 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 0x7fa4b009ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient: get_auth_request con 0x7fa4b0136390 auth_method 0 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 10 monclient(hunting): handle_auth_done global_id 104934 payload 824 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.469+0000 7fa4ae59c640 1 -- 172.21.15.116:0/2181057238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa4b00638d0 con 0x7fa4b0136390 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.471+0000 7fa4ae59c640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 msgr2=0x7fa4b009ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.471+0000 7fa4ae59c640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 0x7fa4b009ddd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000e740 tx=0x7fa4a001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.471+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a000c610 con 0x7fa4b0136390 2024-09-18T12:36:02.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.471+0000 7fa49e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.471+0000 7fa49e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.471+0000 7fa49e7fc640 10 monclient: dump: 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fa4a000c770 con 0x7fa4b0136390 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa49e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa49e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 msgr2=0x7fa4b009ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa49e7fc640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 0x7fa4b009ddd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000e740 tx=0x7fa4a001a000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa49e7fc640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 msgr2=0x7fa4b009ddd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa49e7fc640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b0136390 0x7fa4b009ddd0 secure :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000e740 tx=0x7fa4a001a000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 108, 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:02.496 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa490038490 msgr2=0x7fa49003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa490038490 0x7fa49003a940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80075e0 tx=0x7fa4a8009780 comp rx=0 tx=0).stop 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 10 monclient: shutdown 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01318f0 msgr2=0x7fa4b0132f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 --2- 172.21.15.116:0/2181057238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4b01318f0 0x7fa4b0132f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009850 tx=0x7fa4a4032000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4aed9d640 1 -- 172.21.15.116:0/2181057238 reap_dead start 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4aed9d640 1 -- 172.21.15.116:0/2181057238 reap_dead start 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 shutdown_connections 2024-09-18T12:36:02.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 >> 172.21.15.116:0/2181057238 conn(0x7fa4b0067110 msgr2=0x7fa4b012bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 shutdown_connections 2024-09-18T12:36:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.472+0000 7fa4b4ee8640 1 -- 172.21.15.116:0/2181057238 wait complete. 2024-09-18T12:36:02.499 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:02.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:36:02.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:36:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.591+0000 7f37ac700640 10 monclient: get_monmap_and_config 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.591+0000 7f37ac700640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.591+0000 7f37ac700640 10 monclient: monmap: 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:02.591837+0000 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:02.591837+0000 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:02.593 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:02.593 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:02.593 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.591+0000 7f37ac700640 1 Processor -- start 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.591+0000 7f37ac700640 1 -- start start 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.591+0000 7f37ac700640 10 monclient: init 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37a4131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37a409e910 0x7f37a409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient(hunting): picked mon.noname-a con 0x7f37a409e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient(hunting): picked mon.noname-b con 0x7f37a409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a409faf0 con 0x7f37a4131ab0 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a4132590 con 0x7f37a409e910 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a409fd30 con 0x7f37a409f250 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37ac700640 10 monclient(hunting): authenticate will time out at 1389.913461s 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37a409e910 0x7f37a409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aa49c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f37a4131ab0 0x7f37a4131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34616/0 (socket says 172.21.15.116:34616) 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aa49c640 1 -- 172.21.15.116:0/1677522893 learned_addr learned my addr 172.21.15.116:0/1677522893 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 10 monclient(hunting): get_auth_request con 0x7f37a409e910 auth_method 0 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): get_auth_request con 0x7f37a409f250 auth_method 0 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient(hunting): handle_auth_done global_id 144146 payload 274 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 1 -- 172.21.15.116:0/1677522893 >> v2:172.21.15.116:3300/0 conn(0x7f37a409e910 msgr2=0x7f37a409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 1 --2- 172.21.15.116:0/1677522893 >> v2:172.21.15.116:3300/0 conn(0x7f37a409e910 0x7f37a409ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 1 -- 172.21.15.116:0/1677522893 >> v1:172.21.15.116:6789/0 conn(0x7f37a4131ab0 legacy=0x7f37a4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 1 monclient: found mon.noname-b 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37aac9d640 1 -- 172.21.15.116:0/1677522893 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a40a1110 con 0x7f37a409f250 2024-09-18T12:36:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.592+0000 7f37a9c9b640 1 --2- 172.21.15.116:0/1677522893 >> v2:172.21.15.116:3300/0 conn(0x7f37a409e910 0x7f37a409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37aac9d640 1 -- 172.21.15.116:0/1677522893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 msgr2=0x7f37a4138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37aac9d640 1 --2- 172.21.15.116:0/1677522893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4138bc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f379c009e60 tx=0x7f379c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c3931788d2b34 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 1 -- 172.21.15.116:0/1677522893 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c007760 con 0x7f37a409f250 2024-09-18T12:36:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: dump: 2024-09-18T12:36:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 1 -- 172.21.15.116:0/1677522893 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c02d050 con 0x7f37a409f250 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37ac700640 5 monclient: authenticate success, global_id 144146 2024-09-18T12:36:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37ac700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37ac700640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 1 -- 172.21.15.116:0/1677522893 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c007af0 con 0x7f37a409f250 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37ac700640 4 set_mon_vals no callback set 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.595+0000 7f37a949a640 10 monclient: dump: 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 10 monclient: shutdown 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 -- 172.21.15.116:0/1677522893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 msgr2=0x7f37a4138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 --2- 172.21.15.116:0/1677522893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4138bc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f379c009e60 tx=0x7f379c009780 comp rx=0 tx=0).stop 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 -- 172.21.15.116:0/1677522893 shutdown_connections 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 --2- 172.21.15.116:0/1677522893 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4138bc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 --2- 172.21.15.116:0/1677522893 >> v2:172.21.15.116:3300/0 conn(0x7f37a409e910 0x7f37a409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 -- 172.21.15.116:0/1677522893 >> 172.21.15.116:0/1677522893 conn(0x7f37a4067d30 msgr2=0x7f37a40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 -- 172.21.15.116:0/1677522893 shutdown_connections 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 1 -- 172.21.15.116:0/1677522893 wait complete. 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.596+0000 7f37ac700640 10 monclient: monmap: 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:02.606 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 Processor -- start 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 -- start start 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient: init 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a409e910 0x7f37a4130f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): picked mon.noname-a con 0x7f37a409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): picked mon.noname-c con 0x7f37a409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37a4131ab0 0x7f37a4096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): picked mon.noname-b con 0x7f37a4131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a409d240 con 0x7f37a409e910 2024-09-18T12:36:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4099aa0 con 0x7f37a4131ab0 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40997c0 con 0x7f37a409f250 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37ac700640 10 monclient(hunting): authenticate will time out at 1389.918785s 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a409e910 0x7f37a4130f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36722/0 (socket says 172.21.15.116:36722) 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 1 -- 172.21.15.116:0/1547754647 learned_addr learned my addr 172.21.15.116:0/1547754647 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 10 monclient(hunting): get_auth_request con 0x7f37a409f250 auth_method 0 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 10 monclient(hunting): get_auth_request con 0x7f37a409e910 auth_method 0 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37aa49c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.597+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 10 monclient(hunting): handle_auth_done global_id 104940 payload 824 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aac9d640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37a4131ab0 0x7f37a4096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 msgr2=0x7f37a4096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37a4131ab0 msgr2=0x7f37a4096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37a4131ab0 0x7f37a4096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 1 monclient: found mon.noname-a 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aa49c640 1 -- 172.21.15.116:0/1547754647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794009390 con 0x7f37a409e910 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aac9d640 10 monclient: get_auth_request con 0x7f37a4131ab0 auth_method 0 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37a9c9b640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a409f250 0x7f37a4096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.598+0000 7f37aac9d640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37a4131ab0 0x7f37a4096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.600+0000 7f37aa49c640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a409e910 msgr2=0x7f37a4130f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.600+0000 7f37aa49c640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a409e910 0x7f37a4130f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c490 tx=0x7f37a0007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0005cb0 con 0x7f37a409e910 2024-09-18T12:36:02.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:02.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: dump: 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a0005e30 con 0x7f37a409e910 2024-09-18T12:36:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37ac700640 5 monclient: authenticate success, global_id 104940 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37ac700640 10 monclient: _renew_subs 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37ac700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c009a20 con 0x7f37a409e910 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0007320 con 0x7f37a409e910 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f379a7fc640 10 monclient: dump: 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37a8c99640 4 set_mon_vals no callback set 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37ac700640 10 monclient: _renew_subs 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37ac700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.602+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a4097e40 con 0x7f37a409e910 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37ac700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37ac700640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 0x7f37a40974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4063740 con 0x7f37a4098580 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 0x7f37a40974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f37a001b3b0 con 0x7f37a409e910 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient: get_auth_request con 0x7f37a4098580 auth_method 0 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_done global_id 144152 payload 824 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f379a7fc640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37880384e0 0x7f378803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f37a9c9b640 1 -- 172.21.15.116:0/1547754647 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37a404bf60 con 0x7f37a4098580 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.603+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f37a004b7d0 con 0x7f37a409e910 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.604+0000 7f37aac9d640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37880384e0 0x7f378803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.604+0000 7f37aac9d640 10 monclient: get_auth_request con 0x7f37880384e0 auth_method 0 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f37a9c9b640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 msgr2=0x7f37a40974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f37a9c9b640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 0x7f37a40974e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f379400f4a0 tx=0x7f37940090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794013a40 con 0x7f37a4098580 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 10 monclient: dump: 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3794013ba0 con 0x7f37a4098580 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 msgr2=0x7f37a40974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 0x7f37a40974e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f379400f4a0 tx=0x7f37940090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 msgr2=0x7f37a40974e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f379a7fc640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a4098580 0x7f37a40974e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.606+0000 7f37aac9d640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37880384e0 0x7f378803a990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f37a4076760 tx=0x7f379c037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37ac700640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37ac700640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 0x7f37a412e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4066280 con 0x7f37a412df20 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 0x7f37a412e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient: get_auth_request con 0x7f37a412df20 auth_method 0 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 10 monclient(hunting): handle_auth_done global_id 144155 payload 824 2024-09-18T12:36:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.612+0000 7f37a9c9b640 1 -- 172.21.15.116:0/1547754647 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f37a40994e0 con 0x7f37a412df20 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.614+0000 7f37a9c9b640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 msgr2=0x7f37a412e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.614+0000 7f37a9c9b640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 0x7f37a412e320 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f379400f4a0 tx=0x7f379401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.614+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794024040 con 0x7f37a412df20 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.614+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.614+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.614+0000 7f379a7fc640 10 monclient: dump: 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.625 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f3794013830 con 0x7f37a412df20 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f379a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f379a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 msgr2=0x7f37a412e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f379a7fc640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 0x7f37a412e320 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f379400f4a0 tx=0x7f379401c000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f379a7fc640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 msgr2=0x7f37a412e320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:02.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f379a7fc640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37a412df20 0x7f37a412e320 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f379400f4a0 tx=0x7f379401c000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 108, 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:02.639 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37880384e0 msgr2=0x7f378803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37880384e0 0x7f378803a990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f37a4076760 tx=0x7f379c037000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 10 monclient: shutdown 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a409e910 msgr2=0x7f37a4130f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37aac9d640 1 -- 172.21.15.116:0/1547754647 reap_dead start 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a409e910 0x7f37a4130f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c490 tx=0x7f37a0007f00 comp rx=0 tx=0).stop 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 shutdown_connections 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 --2- 172.21.15.116:0/1547754647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a409e910 0x7f37a4130f00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 >> 172.21.15.116:0/1547754647 conn(0x7f37a4067d30 msgr2=0x7f37a409d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 shutdown_connections 2024-09-18T12:36:02.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.615+0000 7f37ac700640 1 -- 172.21.15.116:0/1547754647 wait complete. 2024-09-18T12:36:02.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:02.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:36:02.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.735+0000 7f13ba6d4640 10 monclient: get_monmap_and_config 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.735+0000 7f13ba6d4640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.735+0000 7f13ba6d4640 10 monclient: monmap: 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:02.735771+0000 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:02.735771+0000 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:02.737 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.735+0000 7f13ba6d4640 1 Processor -- start 2024-09-18T12:36:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.735+0000 7f13ba6d4640 1 -- start start 2024-09-18T12:36:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.735+0000 7f13ba6d4640 10 monclient: init 2024-09-18T12:36:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T12:36:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f13b4131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f13b409e910 0x7f13b409ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f13b409e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13b409f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b409faf0 con 0x7f13b409f220 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b409fd30 con 0x7f13b409e910 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40a1110 con 0x7f13b4131ab0 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13ba6d4640 10 monclient(hunting): authenticate will time out at 1390.057385s 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b8c71640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13b409f220 0x7f13b4068ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34624/0 (socket says 172.21.15.116:34624) 2024-09-18T12:36:02.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b8c71640 1 -- 172.21.15.116:0/3303120567 learned_addr learned my addr 172.21.15.116:0/3303120567 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 1 --2- 172.21.15.116:0/3303120567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b4131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36750/0 (socket says 172.21.15.116:36750) 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): get_auth_request con 0x7f13b4131ab0 auth_method 0 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b37fe640 1 --2- 172.21.15.116:0/3303120567 >> v2:172.21.15.144:3300/0 conn(0x7f13b409e910 0x7f13b409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b37fe640 10 monclient(hunting): get_auth_request con 0x7f13b409e910 auth_method 0 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient(hunting): handle_auth_done global_id 144158 payload 274 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 1 -- 172.21.15.116:0/3303120567 >> v2:172.21.15.144:3300/0 conn(0x7f13b409e910 msgr2=0x7f13b409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 1 --2- 172.21.15.116:0/3303120567 >> v2:172.21.15.144:3300/0 conn(0x7f13b409e910 0x7f13b409ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 1 -- 172.21.15.116:0/3303120567 >> v1:172.21.15.116:6789/0 conn(0x7f13b409f220 legacy=0x7f13b4068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 1 monclient: found mon.noname-b 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.736+0000 7f13b3fff640 1 -- 172.21.15.116:0/3303120567 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b40a13f0 con 0x7f13b4131ab0 2024-09-18T12:36:02.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b3fff640 1 -- 172.21.15.116:0/3303120567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 msgr2=0x7f13b4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b3fff640 1 --2- 172.21.15.116:0/3303120567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b4131e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009850 tx=0x7f13a40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f238a20fff50d338 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 1 -- 172.21.15.116:0/3303120567 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a402ae60 con 0x7f13b4131ab0 2024-09-18T12:36:02.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: dump: 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.749 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13ba6d4640 5 monclient: authenticate success, global_id 144158 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 1 -- 172.21.15.116:0/3303120567 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a4032440 con 0x7f13b4131ab0 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13ba6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13ba6d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13ba6d4640 4 set_mon_vals no callback set 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 1 -- 172.21.15.116:0/3303120567 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a40327e0 con 0x7f13b4131ab0 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.739+0000 7f13b2ffd640 10 monclient: dump: 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 10 monclient: shutdown 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 -- 172.21.15.116:0/3303120567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 msgr2=0x7f13b4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/3303120567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b4131e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009850 tx=0x7f13a40090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 -- 172.21.15.116:0/3303120567 shutdown_connections 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/3303120567 >> v2:172.21.15.144:3300/0 conn(0x7f13b409e910 0x7f13b409ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/3303120567 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b4131e80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 -- 172.21.15.116:0/3303120567 >> 172.21.15.116:0/3303120567 conn(0x7f13b4067d30 msgr2=0x7f13b40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 -- 172.21.15.116:0/3303120567 shutdown_connections 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 1 -- 172.21.15.116:0/3303120567 wait complete. 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.740+0000 7f13ba6d4640 10 monclient: monmap: 2024-09-18T12:36:02.752 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 Processor -- start 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 -- start start 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient: init 2024-09-18T12:36:02.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b409e910 0x7f13b409ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f13b409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b409f220 0x7f13b4130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f13b409f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b411b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f13b4131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b409b720 con 0x7f13b409e910 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b409bde0 con 0x7f13b409f220 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b409dd40 con 0x7f13b4131ab0 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13ba6d4640 10 monclient(hunting): authenticate will time out at 1390.062911s 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.741+0000 7f13b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b409e910 0x7f13b409ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b411b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b409e910 0x7f13b409ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50294/0 (socket says 172.21.15.116:50294) 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 1 -- 172.21.15.116:0/2073217853 learned_addr learned my addr 172.21.15.116:0/2073217853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 10 monclient(hunting): get_auth_request con 0x7f13b409e910 auth_method 0 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): get_auth_request con 0x7f13b4131ab0 auth_method 0 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b37fe640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b409f220 0x7f13b4130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b37fe640 10 monclient(hunting): get_auth_request con 0x7f13b409f220 auth_method 0 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient(hunting): handle_auth_done global_id 144161 payload 824 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b409f220 msgr2=0x7f13b4130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b409f220 0x7f13b4130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b409e910 msgr2=0x7f13b409ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b409e910 0x7f13b409ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 1 monclient: found mon.noname-c 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b8c71640 1 -- 172.21.15.116:0/2073217853 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a4009390 con 0x7f13b4131ab0 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b37fe640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b409f220 0x7f13b4130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.742+0000 7f13b3fff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b409e910 0x7f13b409ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13b8c71640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 msgr2=0x7f13b411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13b8c71640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b411b6b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00cb90 tx=0x7f13ac0052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac00a850 con 0x7f13b4131ab0 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: dump: 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac00a9d0 con 0x7f13b4131ab0 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac00ad90 con 0x7f13b4131ab0 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13ba6d4640 5 monclient: authenticate success, global_id 144161 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f139bfff640 10 monclient: dump: 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.759 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13ba6d4640 10 monclient: _renew_subs 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13ba6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13b27fc640 4 set_mon_vals no callback set 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b4141db0 con 0x7f13b4131ab0 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13ba6d4640 10 monclient: _renew_subs 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13ba6d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.744+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b411ce30 con 0x7f13b4131ab0 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13ba6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 0x7f13b411bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13ba6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b411e210 con 0x7f13b4141350 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f13ac007300 con 0x7f13b4131ab0 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f139bfff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a00384e0 0x7f13a003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 0x7f13b411bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f13ac04c300 con 0x7f13b4131ab0 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 10 monclient: get_auth_request con 0x7f13b4141350 auth_method 0 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.745+0000 7f13b37fe640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a00384e0 0x7f13a003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.746+0000 7f13b37fe640 10 monclient: get_auth_request con 0x7f13a00384e0 auth_method 0 2024-09-18T12:36:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.746+0000 7f13b3fff640 10 monclient(hunting): handle_auth_done global_id 124429 payload 824 2024-09-18T12:36:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.746+0000 7f13b3fff640 1 -- 172.21.15.116:0/2073217853 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13b404bf60 con 0x7f13b4141350 2024-09-18T12:36:02.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f13b3fff640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 msgr2=0x7f13b411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f13b3fff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 0x7f13b411bbf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13a4035040 tx=0x7f13a4032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f13b37fe640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a00384e0 0x7f13a003a990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f13b40a4f60 tx=0x7f13a8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a403d600 con 0x7f13b4141350 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 10 monclient: dump: 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.765 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f13a403d760 con 0x7f13b4141350 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 msgr2=0x7f13b411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 0x7f13b411bbf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13a4035040 tx=0x7f13a4032000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 msgr2=0x7f13b411bbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.748+0000 7f139bfff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b4141350 0x7f13b411bbf0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13a4035040 tx=0x7f13a4032000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13ba6d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 0x7f13b411c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13ba6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4063740 con 0x7f13b41311b0 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13b3fff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 0x7f13b411c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13b3fff640 10 monclient: get_auth_request con 0x7f13b41311b0 auth_method 0 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.754+0000 7f13b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.755+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.755+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.755+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.755+0000 7f13b3fff640 10 monclient(hunting): handle_auth_done global_id 124432 payload 824 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.755+0000 7f13b3fff640 1 -- 172.21.15.116:0/2073217853 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f13b411e3e0 con 0x7f13b41311b0 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.757+0000 7f13b3fff640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 msgr2=0x7f13b411c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.757+0000 7f13b3fff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 0x7f13b411c130 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f13b4068680 tx=0x7f13a403e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.757+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a404b040 con 0x7f13b41311b0 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.757+0000 7f139bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.757+0000 7f139bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.757+0000 7f139bfff640 10 monclient: dump: 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f13a4047e30 con 0x7f13b41311b0 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f139bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f139bfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 msgr2=0x7f13b411c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f139bfff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 0x7f13b411c130 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f13b4068680 tx=0x7f13a403e000 comp rx=0 tx=0).stop 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f139bfff640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 msgr2=0x7f13b411c130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f139bfff640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b41311b0 0x7f13b411c130 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.769 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 108, 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:02.782 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a00384e0 msgr2=0x7f13a003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13a00384e0 0x7f13a003a990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f13b40a4f60 tx=0x7f13a8008040 comp rx=0 tx=0).stop 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 10 monclient: shutdown 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 msgr2=0x7f13b411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b411b6b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00cb90 tx=0x7f13ac0052b0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13b8c71640 1 -- 172.21.15.116:0/2073217853 reap_dead start 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 shutdown_connections 2024-09-18T12:36:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 --2- 172.21.15.116:0/2073217853 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f13b4131ab0 0x7f13b411b6b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 >> 172.21.15.116:0/2073217853 conn(0x7f13b4067d30 msgr2=0x7f13b409df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 shutdown_connections 2024-09-18T12:36:02.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.758+0000 7f13ba6d4640 1 -- 172.21.15.116:0/2073217853 wait complete. 2024-09-18T12:36:02.785 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:02.785 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:36:02.785 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:36:02.785 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:36:02.786 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:36:02.786 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:36:02.786 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:36:02.799 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:36:02.799 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:36:02.799 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.931+0000 7f9795d5d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.931+0000 7f9795d5d640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.931+0000 7f9795d5d640 10 monclient: monmap: 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:02.931784+0000 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:02.931784+0000 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.932 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:02.933 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:02.933 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:02.933 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:02.933 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:02.933 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.931+0000 7f9795d5d640 1 Processor -- start 2024-09-18T12:36:02.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.931+0000 7f9795d5d640 1 -- start start 2024-09-18T12:36:02.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.931+0000 7f9795d5d640 10 monclient: init 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9790131ab0 0x7f9790131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f9790131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f979009e910 0x7f979009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f979009e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f979009f250 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979009faf0 con 0x7f979009f250 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979009fd30 con 0x7f9790131ab0 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97900a1110 con 0x7f979009e910 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f9795d5d640 10 monclient(hunting): authenticate will time out at 1390.253370s 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978ffff640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f979009f250 legacy=0x7f9790068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978f7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f9790131ab0 msgr2=0x7f9790131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9790131ab0 0x7f9790131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f979009e910 0x7f979009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f979009e910 0x7f979009ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:34884/0 (socket says 172.21.15.116:34884) 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 -- 172.21.15.116:0/923553089 learned_addr learned my addr 172.21.15.116:0/923553089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): get_auth_request con 0x7f979009e910 auth_method 0 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient(hunting): handle_auth_done global_id 124435 payload 274 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 -- 172.21.15.116:0/923553089 >> v2:172.21.15.116:3300/0 conn(0x7f9790131ab0 msgr2=0x7f9790131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 --2- 172.21.15.116:0/923553089 >> v2:172.21.15.116:3300/0 conn(0x7f9790131ab0 0x7f9790131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 -- 172.21.15.116:0/923553089 >> v1:172.21.15.116:6789/0 conn(0x7f979009f250 legacy=0x7f9790068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 monclient: found mon.noname-c 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:36:02.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.932+0000 7f978effd640 1 -- 172.21.15.116:0/923553089 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900a13f0 con 0x7f979009e910 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978effd640 1 -- 172.21.15.116:0/923553089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009e910 msgr2=0x7f979009ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978effd640 1 --2- 172.21.15.116:0/923553089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009e910 0x7f979009ed10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f979009bdb0 tx=0x7f97780091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aaf3f5a6413fcaf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 1 -- 172.21.15.116:0/923553089 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9778032440 con 0x7f979009e910 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: dump: 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 1 -- 172.21.15.116:0/923553089 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97780325a0 con 0x7f979009e910 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 5 monclient: authenticate success, global_id 124435 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 1 -- 172.21.15.116:0/923553089 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9778032980 con 0x7f979009e910 2024-09-18T12:36:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 4 set_mon_vals no callback set 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f978e7fc640 10 monclient: dump: 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.946 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 10 monclient: shutdown 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 -- 172.21.15.116:0/923553089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009e910 msgr2=0x7f979009ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 --2- 172.21.15.116:0/923553089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009e910 0x7f979009ed10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f979009bdb0 tx=0x7f97780091d0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 -- 172.21.15.116:0/923553089 shutdown_connections 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 --2- 172.21.15.116:0/923553089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009e910 0x7f979009ed10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 --2- 172.21.15.116:0/923553089 >> v2:172.21.15.116:3300/0 conn(0x7f9790131ab0 0x7f9790131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 -- 172.21.15.116:0/923553089 >> 172.21.15.116:0/923553089 conn(0x7f9790067d30 msgr2=0x7f97900a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 -- 172.21.15.116:0/923553089 shutdown_connections 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.936+0000 7f9795d5d640 1 -- 172.21.15.116:0/923553089 wait complete. 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.937+0000 7f9795d5d640 10 monclient: build_initial_monmap 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.937+0000 7f9795d5d640 10 monclient: monmap: 2024-09-18T12:36:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.937+0000 7f9795d5d640 1 Processor -- start 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.937+0000 7f9795d5d640 1 -- start start 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.937+0000 7f9795d5d640 10 monclient: init 2024-09-18T12:36:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f979009e910 0x7f979011e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f979009e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009f250 0x7f979011e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f979009f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 0x7f979011eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f9790131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9790121740 con 0x7f979009e910 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978f7fe640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f979009e910 msgr2=0x7f979011e040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979009bde0 con 0x7f979009f250 2024-09-18T12:36:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97900a44a0 con 0x7f9790131ab0 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f9795d5d640 10 monclient(hunting): authenticate will time out at 1390.259381s 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f979009e910 0x7f979011e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 0x7f979011eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 0x7f979011eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36800/0 (socket says 172.21.15.116:36800) 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 -- 172.21.15.116:0/1288568763 learned_addr learned my addr 172.21.15.116:0/1288568763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): get_auth_request con 0x7f9790131ab0 auth_method 0 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009f250 0x7f979011e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 10 monclient(hunting): get_auth_request con 0x7f979009f250 auth_method 0 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient(hunting): handle_auth_done global_id 144164 payload 824 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 -- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009f250 msgr2=0x7f979011e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009f250 0x7f979011e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 -- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f979009e910 msgr2=0x7f979011e040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f979009e910 0x7f979011e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 monclient: found mon.noname-c 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.938+0000 7f978ffff640 1 -- 172.21.15.116:0/1288568763 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778009450 con 0x7f9790131ab0 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.941+0000 7f978ffff640 1 -- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 msgr2=0x7f979011eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.941+0000 7f978ffff640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 0x7f979011eac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f978800bf80 tx=0x7f9788009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 1 -- 172.21.15.116:0/1288568763 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788016430 con 0x7f9790131ab0 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: dump: 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: _finish_auth 0 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 1 -- 172.21.15.116:0/1288568763 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97880165d0 con 0x7f9790131ab0 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f9795d5d640 5 monclient: authenticate success, global_id 144164 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f9795d5d640 10 monclient: _renew_subs 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f9795d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979011f9a0 con 0x7f9790131ab0 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 1 -- 172.21.15.116:0/1288568763 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788016780 con 0x7f9790131ab0 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f97877fe640 10 monclient: dump: 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f978d7fa640 4 set_mon_vals no callback set 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f9795d5d640 10 monclient: _renew_subs 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f9795d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.943+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97901314f0 con 0x7f9790131ab0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f97877fe640 1 -- 172.21.15.116:0/1288568763 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9788012020 con 0x7f9790131ab0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f9795d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f9795d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f9795d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979004bf60 con 0x7f9790131ab0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f97877fe640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9770038490 0x7f977003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f97877fe640 1 -- 172.21.15.116:0/1288568763 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f978804c820 con 0x7f9790131ab0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f978f7fe640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9770038490 0x7f977003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.944+0000 7f978f7fe640 10 monclient: get_auth_request con 0x7f9770038490 auth_method 0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.947+0000 7f978f7fe640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9770038490 0x7f977003a940 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9780006fd0 tx=0x7f9780006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:02.948+0000 7f978ffff640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 0x7f979011eac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f978800bf80 tx=0x7f9788009730 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f97880078f0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:36:03.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.777+0000 7f97877fe640 1 -- 172.21.15.116:0/1288568763 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f97880078f0 con 0x7f9790131ab0 2024-09-18T12:36:03.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.777+0000 7f97877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:03.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.777+0000 7f97877fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:03.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.990+0000 7f9795d5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:03.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.990+0000 7f9795d5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:03.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.990+0000 7f9795d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:03.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.990+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9790066280 con 0x7f9790131ab0 2024-09-18T12:36:03.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.990+0000 7f97877fe640 1 -- 172.21.15.116:0/1288568763 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f978801d300 con 0x7f9790131ab0 2024-09-18T12:36:03.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.990+0000 7f97877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:03.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.990+0000 7f97877fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:03.992 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9770038490 msgr2=0x7f977003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9770038490 0x7f977003a940 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9780006fd0 tx=0x7f9780006d30 comp rx=0 tx=0).stop 2024-09-18T12:36:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 10 monclient: shutdown 2024-09-18T12:36:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 msgr2=0x7f979011eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 0x7f979011eac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f978800bf80 tx=0x7f9788009730 comp rx=0 tx=0).stop 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 shutdown_connections 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9770038490 0x7f977003a940 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9790131ab0 0x7f979011eac0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f979009f250 0x7f979011e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 --2- 172.21.15.116:0/1288568763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f979009e910 0x7f979011e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 >> 172.21.15.116:0/1288568763 conn(0x7f9790067d30 msgr2=0x7f979009b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 shutdown_connections 2024-09-18T12:36:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:03.992+0000 7f9795d5d640 1 -- 172.21.15.116:0/1288568763 wait complete. 2024-09-18T12:36:07.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:07.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.107+0000 7fb3c019d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.107+0000 7fb3c019d640 10 monclient: build_initial_monmap 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.107+0000 7fb3c019d640 10 monclient: monmap: 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:07.107804+0000 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:07.107804+0000 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:07.108 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:07.109 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:07.109 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:07.109 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:07.109 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:07.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.107+0000 7fb3c019d640 1 Processor -- start 2024-09-18T12:36:07.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 1 -- start start 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient: init 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3b8131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3b809e910 0x7fb3b809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b809e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 0x7fb3b8138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b809faf0 con 0x7fb3b8131ab0 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b8132590 con 0x7fb3b809e910 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b809fd30 con 0x7fb3b809f250 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3bdf39640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7fb3b8131ab0 legacy=0x7fb3b8131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3c019d640 10 monclient(hunting): authenticate will time out at 1394.429597s 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3bd738640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fb3b809e910 msgr2=0x7fb3b809ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3bd738640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3b809e910 0x7fb3b809ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 0x7fb3b8138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 0x7fb3b8138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44796/0 (socket says 172.21.15.116:44796) 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 -- 172.21.15.116:0/3754199702 learned_addr learned my addr 172.21.15.116:0/3754199702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): get_auth_request con 0x7fb3b809f250 auth_method 0 2024-09-18T12:36:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient(hunting): handle_auth_done global_id 144167 payload 274 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 -- 172.21.15.116:0/3754199702 >> v2:172.21.15.116:3300/0 conn(0x7fb3b809e910 msgr2=0x7fb3b809ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 --2- 172.21.15.116:0/3754199702 >> v2:172.21.15.116:3300/0 conn(0x7fb3b809e910 0x7fb3b809ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 -- 172.21.15.116:0/3754199702 >> v1:172.21.15.116:6789/0 conn(0x7fb3b8131ab0 legacy=0x7fb3b8131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 monclient: found mon.noname-b 2024-09-18T12:36:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.108+0000 7fb3be73a640 1 -- 172.21.15.116:0/3754199702 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b80a1110 con 0x7fb3b809f250 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3be73a640 1 -- 172.21.15.116:0/3754199702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 msgr2=0x7fb3b8138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3be73a640 1 --2- 172.21.15.116:0/3754199702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 0x7fb3b8138bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009910 tx=0x7fb3a80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54c6212ae43ff4fc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 1 -- 172.21.15.116:0/3754199702 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a802ae20 con 0x7fb3b809f250 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 10 monclient: dump: 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.122 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 10 monclient: _finish_auth 0 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3c019d640 5 monclient: authenticate success, global_id 144167 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3c019d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:07.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 1 -- 172.21.15.116:0/3754199702 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a802f4b0 con 0x7fb3b809f250 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3c019d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3c019d640 4 set_mon_vals no callback set 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 1 -- 172.21.15.116:0/3754199702 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a802f890 con 0x7fb3b809f250 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.111+0000 7fb3bcf37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3bcf37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3bcf37640 10 monclient: dump: 2024-09-18T12:36:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 10 monclient: shutdown 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:07.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 -- 172.21.15.116:0/3754199702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 msgr2=0x7fb3b8138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 --2- 172.21.15.116:0/3754199702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 0x7fb3b8138bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009910 tx=0x7fb3a80091d0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 -- 172.21.15.116:0/3754199702 shutdown_connections 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 --2- 172.21.15.116:0/3754199702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b809f250 0x7fb3b8138bc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 --2- 172.21.15.116:0/3754199702 >> v2:172.21.15.116:3300/0 conn(0x7fb3b809e910 0x7fb3b809ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 -- 172.21.15.116:0/3754199702 >> 172.21.15.116:0/3754199702 conn(0x7fb3b8067d30 msgr2=0x7fb3b80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 -- 172.21.15.116:0/3754199702 shutdown_connections 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.112+0000 7fb3c019d640 1 -- 172.21.15.116:0/3754199702 wait complete. 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient: build_initial_monmap 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient: monmap: 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 Processor -- start 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 -- start start 2024-09-18T12:36:07.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient: init 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b809e910 0x7fb3b809a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b809e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b809f250 0x7fb3b8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b809f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 0x7fb3b8096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b8131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b8099e10 con 0x7fb3b809e910 2024-09-18T12:36:07.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b80998c0 con 0x7fb3b809f250 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b8098380 con 0x7fb3b8131ab0 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3c019d640 10 monclient(hunting): authenticate will time out at 1394.434928s 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3bdf39640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b809e910 msgr2=0x7fb3b809a6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.113+0000 7fb3bdf39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b809e910 0x7fb3b809a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 0x7fb3b8096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 0x7fb3b8096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44804/0 (socket says 172.21.15.116:44804) 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 -- 172.21.15.116:0/1711673294 learned_addr learned my addr 172.21.15.116:0/1711673294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): get_auth_request con 0x7fb3b8131ab0 auth_method 0 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient(hunting): handle_auth_done global_id 144170 payload 824 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3bd738640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b809f250 0x7fb3b8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 -- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b809f250 msgr2=0x7fb3b8096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b809f250 0x7fb3b8096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 -- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b809e910 msgr2=0x7fb3b809a6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b809e910 0x7fb3b809a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 monclient: found mon.noname-c 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3be73a640 1 -- 172.21.15.116:0/1711673294 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8009450 con 0x7fb3b8131ab0 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3bd738640 10 monclient: get_auth_request con 0x7fb3b809f250 auth_method 0 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.114+0000 7fb3bd738640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b809f250 0x7fb3b8096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3be73a640 1 -- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 msgr2=0x7fb3b8096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3be73a640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 0x7fb3b8096fa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8035040 tx=0x7fb3a8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 1 -- 172.21.15.116:0/1711673294 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8032d10 con 0x7fb3b8131ab0 2024-09-18T12:36:07.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 10 monclient: dump: 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:07.132 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3be73a640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 0x7fb3b8096fa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8035040 tx=0x7fb3a8032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fb3a803b650 mon_map magic: 0 v1 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3c019d640 5 monclient: authenticate success, global_id 144170 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3c019d640 10 monclient: _renew_subs 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3c019d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 1 -- 172.21.15.116:0/1711673294 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a8032e90 con 0x7fb3b8131ab0 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b8099b30 con 0x7fb3b8131ab0 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.117+0000 7fb3c019d640 10 monclient: _renew_subs 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.116+0000 7fb3b7fff640 4 set_mon_vals no callback set 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.117+0000 7fb3c019d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.117+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b8097f60 con 0x7fb3b8131ab0 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3b5ffb640 1 -- 172.21.15.116:0/1711673294 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a803b650 con 0x7fb3b8131ab0 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3b5ffb640 10 monclient: dump: 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3b5ffb640 1 -- 172.21.15.116:0/1711673294 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb3a8043030 con 0x7fb3b8131ab0 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3c019d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3c019d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3c019d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b8063700 con 0x7fb3b8131ab0 2024-09-18T12:36:07.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3b5ffb640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3a4038440 0x7fb3a403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:07.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3bdf39640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3a4038440 0x7fb3a403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:07.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.118+0000 7fb3bdf39640 10 monclient: get_auth_request con 0x7fb3a4038440 auth_method 0 2024-09-18T12:36:07.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.121+0000 7fb3bdf39640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3a4038440 0x7fb3a403a8f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0003cc0 tx=0x7fb3b0009310 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:07.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.122+0000 7fb3b5ffb640 1 -- 172.21.15.116:0/1711673294 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb3a8041070 con 0x7fb3b8131ab0 2024-09-18T12:36:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.739+0000 7fb3b5ffb640 1 -- 172.21.15.116:0/1711673294 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb3a80313a0 con 0x7fb3b8131ab0 2024-09-18T12:36:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.739+0000 7fb3b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:07.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.739+0000 7fb3b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.952+0000 7fb3c019d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.952+0000 7fb3c019d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.952+0000 7fb3c019d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.952+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb3b804bf60 con 0x7fb3b8131ab0 2024-09-18T12:36:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.952+0000 7fb3b5ffb640 1 -- 172.21.15.116:0/1711673294 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fb3a8037020 con 0x7fb3b8131ab0 2024-09-18T12:36:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.952+0000 7fb3b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.952+0000 7fb3b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:07.954 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3a4038440 msgr2=0x7fb3a403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3a4038440 0x7fb3a403a8f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0003cc0 tx=0x7fb3b0009310 comp rx=0 tx=0).stop 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 10 monclient: shutdown 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 msgr2=0x7fb3b8096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 0x7fb3b8096fa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8035040 tx=0x7fb3a8032000 comp rx=0 tx=0).stop 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 shutdown_connections 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3a4038440 0x7fb3a403a8f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3b8131ab0 0x7fb3b8096fa0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3b809f250 0x7fb3b8096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.954+0000 7fb3c019d640 1 --2- 172.21.15.116:0/1711673294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3b809e910 0x7fb3b809a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.955+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 >> 172.21.15.116:0/1711673294 conn(0x7fb3b8067d30 msgr2=0x7fb3b809d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.955+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 shutdown_connections 2024-09-18T12:36:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:07.955+0000 7fb3c019d640 1 -- 172.21.15.116:0/1711673294 wait complete. 2024-09-18T12:36:10.966 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.069+0000 7fa85ff1d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.069+0000 7fa85ff1d640 10 monclient: build_initial_monmap 2024-09-18T12:36:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.069+0000 7fa85ff1d640 10 monclient: monmap: 2024-09-18T12:36:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:11.070 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:11.070223+0000 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:11.070223+0000 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:11.071 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 1 Processor -- start 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 1 -- start start 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 10 monclient: init 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa858096ca0 0x7fa858098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa858096ca0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa85809b810 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa85809c320 0x7fa85813d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa85809c320 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.070+0000 7fa85ff1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa858097070 con 0x7fa85809b810 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85ff1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85ff1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8580993a0 con 0x7fa858096ca0 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85ff1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85ff1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa858097a30 con 0x7fa85809c320 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85ff1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85ff1d640 10 monclient(hunting): authenticate will time out at 1398.392026s 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85dcb9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7fa858096ca0 msgr2=0x7fa858098e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85dcb9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa858096ca0 0x7fa858098e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa85809c320 0x7fa85813d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85d4b8640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa85809b810 0x7fa85809bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:33016/0 (socket says 172.21.15.116:33016) 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85d4b8640 1 -- 172.21.15.116:0/156720852 learned_addr learned my addr 172.21.15.116:0/156720852 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:11.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): get_auth_request con 0x7fa85809c320 auth_method 0 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_done global_id 124441 payload 274 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 1 -- 172.21.15.116:0/156720852 >> v2:172.21.15.116:3300/0 conn(0x7fa858096ca0 msgr2=0x7fa858098e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 1 --2- 172.21.15.116:0/156720852 >> v2:172.21.15.116:3300/0 conn(0x7fa858096ca0 0x7fa858098e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 1 -- 172.21.15.116:0/156720852 >> v1:172.21.15.144:6789/0 conn(0x7fa85809b810 legacy=0x7fa85809bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient: _finish_hunting 0 2024-09-18T12:36:11.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 1 monclient: found mon.noname-c 2024-09-18T12:36:11.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:11.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:36:11.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.071+0000 7fa85e4ba640 1 -- 172.21.15.116:0/156720852 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa858099620 con 0x7fa85809c320 2024-09-18T12:36:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85e4ba640 1 -- 172.21.15.116:0/156720852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85809c320 msgr2=0x7fa85813d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85e4ba640 1 --2- 172.21.15.116:0/156720852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85809c320 0x7fa85813d250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa85809c7a0 tx=0x7fa850009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fa94b3ce871b16a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 1 -- 172.21.15.116:0/156720852 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850007840 con 0x7fa85809c320 2024-09-18T12:36:11.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: dump: 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:11.083 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: _finish_auth 0 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 1 -- 172.21.15.116:0/156720852 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85002d050 con 0x7fa85809c320 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ff1d640 5 monclient: authenticate success, global_id 124441 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ff1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 1 -- 172.21.15.116:0/156720852 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850007bd0 con 0x7fa85809c320 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ff1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ff1d640 4 set_mon_vals no callback set 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.074+0000 7fa85ccb7640 10 monclient: dump: 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 10 monclient: shutdown 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 -- 172.21.15.116:0/156720852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85809c320 msgr2=0x7fa85813d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/156720852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85809c320 0x7fa85813d250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa85809c7a0 tx=0x7fa850009780 comp rx=0 tx=0).stop 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 -- 172.21.15.116:0/156720852 shutdown_connections 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/156720852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa85809c320 0x7fa85813d250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/156720852 >> v2:172.21.15.116:3300/0 conn(0x7fa858096ca0 0x7fa858098e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 -- 172.21.15.116:0/156720852 >> 172.21.15.116:0/156720852 conn(0x7fa8580672b0 msgr2=0x7fa8580676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 -- 172.21.15.116:0/156720852 shutdown_connections 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.075+0000 7fa85ff1d640 1 -- 172.21.15.116:0/156720852 wait complete. 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient: build_initial_monmap 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient: monmap: 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 Processor -- start 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 -- start start 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient: init 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa858096ca0 0x7fa85811fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa858096ca0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa85809b810 0x7fa8581201d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa85809b810 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 0x7fa858120710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa85809c320 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858138f80 con 0x7fa85809b810 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858122f30 con 0x7fa858096ca0 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa858123180 con 0x7fa85809c320 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85ff1d640 10 monclient(hunting): authenticate will time out at 1398.397908s 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85d4b8640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa85809b810 msgr2=0x7fa8581201d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.076+0000 7fa85d4b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa85809b810 0x7fa8581201d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 0x7fa858120710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 0x7fa858120710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44820/0 (socket says 172.21.15.116:44820) 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 -- 172.21.15.116:0/722645833 learned_addr learned my addr 172.21.15.116:0/722645833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): get_auth_request con 0x7fa85809c320 auth_method 0 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85dcb9640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa858096ca0 0x7fa85811fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85dcb9640 10 monclient(hunting): get_auth_request con 0x7fa858096ca0 auth_method 0 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient(hunting): handle_auth_done global_id 144173 payload 824 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 -- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa858096ca0 msgr2=0x7fa85811fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa858096ca0 0x7fa85811fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 -- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa85809b810 msgr2=0x7fa8581201d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa85809b810 0x7fa8581201d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient: _finish_hunting 0 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 monclient: found mon.noname-c 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.077+0000 7fa85e4ba640 1 -- 172.21.15.116:0/722645833 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa850009a20 con 0x7fa85809c320 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa85e4ba640 1 -- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 msgr2=0x7fa858120710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa85e4ba640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 0x7fa858120710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa85002ab00 tx=0x7fa850037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 1 -- 172.21.15.116:0/722645833 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850031cd0 con 0x7fa85809c320 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: dump: 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: _finish_auth 0 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 1 -- 172.21.15.116:0/722645833 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa850040070 con 0x7fa85809c320 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa85ff1d640 5 monclient: authenticate success, global_id 144173 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa85ff1d640 10 monclient: _renew_subs 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa85ff1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85809d1d0 con 0x7fa85809c320 2024-09-18T12:36:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 1 -- 172.21.15.116:0/722645833 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850045640 con 0x7fa85809c320 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84dffb640 10 monclient: dump: 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:11.093 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.079+0000 7fa84ffff640 4 set_mon_vals no callback set 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.080+0000 7fa85ff1d640 10 monclient: _renew_subs 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.080+0000 7fa85ff1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.080+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85809ff80 con 0x7fa85809c320 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.080+0000 7fa84dffb640 1 -- 172.21.15.116:0/722645833 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa850042030 con 0x7fa85809c320 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa85ff1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa85ff1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa85ff1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85804bf60 con 0x7fa85809c320 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa84dffb640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa83c038440 0x7fa83c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa84dffb640 1 -- 172.21.15.116:0/722645833 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa85006dd10 con 0x7fa85809c320 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa85dcb9640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa83c038440 0x7fa83c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.081+0000 7fa85dcb9640 10 monclient: get_auth_request con 0x7fa83c038440 auth_method 0 2024-09-18T12:36:11.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.083+0000 7fa85dcb9640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa83c038440 0x7fa83c03a8f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa854003850 tx=0x7fa854006ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:11.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.085+0000 7fa85e4ba640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 0x7fa858120710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa85002ab00 tx=0x7fa850037000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fa850031e30 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:36:11.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.913+0000 7fa84dffb640 1 -- 172.21.15.116:0/722645833 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa850031e30 con 0x7fa85809c320 2024-09-18T12:36:11.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.913+0000 7fa84dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:11.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:11.913+0000 7fa84dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:12.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:12.133+0000 7fa85ff1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:12.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:12.133+0000 7fa85ff1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:12.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:12.133+0000 7fa85ff1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:12.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:12.133+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa8580638d0 con 0x7fa85809c320 2024-09-18T12:36:14.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:14.077+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:14.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:14.077+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:14.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:14.077+0000 7fa84cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:15.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:15.077+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:15.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:15.077+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:16.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:16.077+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:16.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:16.077+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:17.077+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:17.077+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:18.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:18.078+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:18.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:18.078+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:19.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:19.078+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:19.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:19.078+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:20.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:20.078+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:20.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:20.078+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.076+0000 7fa84cff9640 10 monclient: tick 2024-09-18T12:36:21.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.076+0000 7fa84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.935+0000 7fa84dffb640 1 -- 172.21.15.116:0/722645833 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa8500330c0 con 0x7fa85809c320 2024-09-18T12:36:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.935+0000 7fa84dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.935+0000 7fa84dffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:21.937 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.937+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa83c038440 msgr2=0x7fa83c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.937+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa83c038440 0x7fa83c03a8f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa854003850 tx=0x7fa854006ed0 comp rx=0 tx=0).stop 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.937+0000 7fa85ff1d640 10 monclient: shutdown 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.937+0000 7fa85ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.937+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 msgr2=0x7fa858120710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.937+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 0x7fa858120710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa85002ab00 tx=0x7fa850037000 comp rx=0 tx=0).stop 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 shutdown_connections 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa83c038440 0x7fa83c03a8f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa85809c320 0x7fa858120710 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa85809b810 0x7fa8581201d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 --2- 172.21.15.116:0/722645833 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa858096ca0 0x7fa85811fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 >> 172.21.15.116:0/722645833 conn(0x7fa8580672b0 msgr2=0x7fa85809ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 shutdown_connections 2024-09-18T12:36:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:21.938+0000 7fa85ff1d640 1 -- 172.21.15.116:0/722645833 wait complete. 2024-09-18T12:36:21.953 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:36:21.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:36:21.954 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:36:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.050+0000 7f37d7f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.050+0000 7f37d7f9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.050+0000 7f37d7f9d640 10 monclient: monmap: 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:22.052516+0000 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:22.052516+0000 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:22.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.051+0000 7f37d7f9d640 1 Processor -- start 2024-09-18T12:36:22.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.051+0000 7f37d7f9d640 1 -- start start 2024-09-18T12:36:22.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.051+0000 7f37d7f9d640 10 monclient: init 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37d00975c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37d012ea30 0x7f37d0136c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f37d012ea30 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37d012ee50 0x7f37d01391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f37d012ee50 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d006e9e0 con 0x7f37d00975c0 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d0063950 con 0x7f37d012ea30 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d009d1c0 con 0x7f37d012ee50 2024-09-18T12:36:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d7f9d640 10 monclient(hunting): authenticate will time out at 1409.374168s 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d5538640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f37d012ea30 msgr2=0x7f37d0136c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d5d39640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f37d00975c0 legacy=0x7f37d0097990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d5538640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f37d012ea30 0x7f37d0136c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37d012ee50 0x7f37d01391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f37d012ee50 0x7f37d01391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:47860/0 (socket says 172.21.15.116:47860) 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 -- 172.21.15.116:0/3058523119 learned_addr learned my addr 172.21.15.116:0/3058523119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): get_auth_request con 0x7f37d012ee50 auth_method 0 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient(hunting): handle_auth_done global_id 124447 payload 274 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 -- 172.21.15.116:0/3058523119 >> v2:172.21.15.116:3300/0 conn(0x7f37d012ea30 msgr2=0x7f37d0136c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 --2- 172.21.15.116:0/3058523119 >> v2:172.21.15.116:3300/0 conn(0x7f37d012ea30 0x7f37d0136c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 -- 172.21.15.116:0/3058523119 >> v1:172.21.15.116:6789/0 conn(0x7f37d00975c0 legacy=0x7f37d0097990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 monclient: found mon.noname-c 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:36:22.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.052+0000 7f37d653a640 1 -- 172.21.15.116:0/3058523119 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d009d4a0 con 0x7f37d012ee50 2024-09-18T12:36:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.055+0000 7f37d653a640 1 -- 172.21.15.116:0/3058523119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ee50 msgr2=0x7f37d01391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.055+0000 7f37d653a640 1 --2- 172.21.15.116:0/3058523119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ee50 0x7f37d01391d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f37d006f410 tx=0x7f37c8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d359200cb32d540a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 1 -- 172.21.15.116:0/3058523119 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8007610 con 0x7f37d012ee50 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: dump: 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: _finish_auth 0 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 1 -- 172.21.15.116:0/3058523119 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c802d050 con 0x7f37d012ee50 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 5 monclient: authenticate success, global_id 124447 2024-09-18T12:36:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 4 set_mon_vals no callback set 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 1 -- 172.21.15.116:0/3058523119 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c80079e0 con 0x7f37d012ee50 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d4d37640 10 monclient: dump: 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 10 monclient: shutdown 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 1 -- 172.21.15.116:0/3058523119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ee50 msgr2=0x7f37d01391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/3058523119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ee50 0x7f37d01391d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f37d006f410 tx=0x7f37c8009840 comp rx=0 tx=0).stop 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 1 -- 172.21.15.116:0/3058523119 shutdown_connections 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/3058523119 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ee50 0x7f37d01391d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/3058523119 >> v2:172.21.15.116:3300/0 conn(0x7f37d012ea30 0x7f37d0136c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.056+0000 7f37d7f9d640 1 -- 172.21.15.116:0/3058523119 >> 172.21.15.116:0/3058523119 conn(0x7f37d0067b50 msgr2=0x7f37d00a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.057+0000 7f37d7f9d640 1 -- 172.21.15.116:0/3058523119 shutdown_connections 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.057+0000 7f37d7f9d640 1 -- 172.21.15.116:0/3058523119 wait complete. 2024-09-18T12:36:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.057+0000 7f37d7f9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.057+0000 7f37d7f9d640 10 monclient: monmap: 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.057+0000 7f37d7f9d640 1 Processor -- start 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.057+0000 7f37d7f9d640 1 -- start start 2024-09-18T12:36:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.057+0000 7f37d7f9d640 10 monclient: init 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 0x7f37d0128af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f37d00975c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ea30 0x7f37d0129030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f37d012ea30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37d012ee50 0x7f37d0129570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f37d012ee50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d0096fe0 con 0x7f37d012ee50 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d009ce40 con 0x7f37d012ea30 2024-09-18T12:36:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d012b470 con 0x7f37d00975c0 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d7f9d640 10 monclient(hunting): authenticate will time out at 1409.380276s 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d653a640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37d012ee50 msgr2=0x7f37d0129570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d653a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37d012ee50 0x7f37d0129570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 0x7f37d0128af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 0x7f37d0128af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59292/0 (socket says 172.21.15.116:59292) 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 -- 172.21.15.116:0/1038771560 learned_addr learned my addr 172.21.15.116:0/1038771560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): get_auth_request con 0x7f37d00975c0 auth_method 0 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5538640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ea30 0x7f37d0129030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient(hunting): handle_auth_done global_id 144176 payload 824 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ea30 msgr2=0x7f37d0129030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ea30 0x7f37d0129030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37d012ee50 msgr2=0x7f37d0129570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37d012ee50 0x7f37d0129570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient: _finish_hunting 0 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 monclient: found mon.noname-c 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5d39640 1 -- 172.21.15.116:0/1038771560 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c8009ae0 con 0x7f37d00975c0 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5538640 10 monclient: get_auth_request con 0x7f37d012ea30 auth_method 0 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.058+0000 7f37d5538640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f37d012ea30 0x7f37d0129030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.060+0000 7f37d5d39640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 msgr2=0x7f37d0128af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.060+0000 7f37d5d39640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 0x7f37d0128af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00b5c0 tx=0x7f37cc009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc003d90 con 0x7f37d00975c0 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: dump: 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc006150 con 0x7f37d00975c0 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37d7f9d640 5 monclient: authenticate success, global_id 144176 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37d7f9d640 10 monclient: _renew_subs 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37d7f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d012c190 con 0x7f37d00975c0 2024-09-18T12:36:22.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc006510 con 0x7f37d00975c0 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 10 monclient: dump: 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c7fff640 4 set_mon_vals no callback set 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37d7f9d640 10 monclient: _renew_subs 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37d7f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d0139270 con 0x7f37d00975c0 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.061+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f37cc0103c0 con 0x7f37d00975c0 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d7f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 0x7f37d009a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d7f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d0063720 con 0x7f37d0129ab0 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37c5ffb640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37b4038440 0x7f37b403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f37cc04bb90 con 0x7f37d00975c0 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 0x7f37d009a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient: get_auth_request con 0x7f37d0129ab0 auth_method 0 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d653a640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37b4038440 0x7f37b403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d653a640 10 monclient: get_auth_request con 0x7f37b4038440 auth_method 0 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 10 monclient(hunting): handle_auth_done global_id 144179 payload 824 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.062+0000 7f37d5538640 1 -- 172.21.15.116:0/1038771560 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37d012f850 con 0x7f37d0129ab0 2024-09-18T12:36:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.064+0000 7f37d5538640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 msgr2=0x7f37d009a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.064+0000 7f37d5538640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 0x7f37d009a570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f37d012c0c0 tx=0x7f37c0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.064+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0003d90 con 0x7f37d0129ab0 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.064+0000 7f37c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.064+0000 7f37c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.064+0000 7f37c5ffb640 10 monclient: dump: 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.064+0000 7f37d653a640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37b4038440 0x7f37b403a8f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f37c8033000 tx=0x7f37c8036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.065+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f37c0006150 con 0x7f37d0129ab0 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.065+0000 7f37c5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.065+0000 7f37c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.065+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 msgr2=0x7f37d009a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.065+0000 7f37c5ffb640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 0x7f37d009a570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f37d012c0c0 tx=0x7f37c0008040 comp rx=0 tx=0).stop 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.065+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 msgr2=0x7f37d009a570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.065+0000 7f37c5ffb640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d0129ab0 0x7f37d009a570 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f37d012c0c0 tx=0x7f37c0008040 comp rx=0 tx=0).stop 2024-09-18T12:36:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d7f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 0x7f37d009aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d7f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d012fbe0 con 0x7f37d009aab0 2024-09-18T12:36:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 0x7f37d009aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 10 monclient: get_auth_request con 0x7f37d009aab0 auth_method 0 2024-09-18T12:36:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.070+0000 7f37d5538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.071+0000 7f37d5538640 10 monclient(hunting): handle_auth_done global_id 144182 payload 824 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.071+0000 7f37d5538640 1 -- 172.21.15.116:0/1038771560 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f37d012fd90 con 0x7f37d009aab0 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d5538640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 msgr2=0x7f37d009aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d5538640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 0x7f37d009aeb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f37d012c0c0 tx=0x7f37c0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c00085f0 con 0x7f37d009aab0 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 10 monclient: dump: 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.085 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f37c0008750 con 0x7f37d009aab0 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 msgr2=0x7f37d009aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 0x7f37d009aeb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f37d012c0c0 tx=0x7f37c0017000 comp rx=0 tx=0).stop 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 msgr2=0x7f37d009aeb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37c5ffb640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d009aab0 0x7f37d009aeb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:22.086 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 112, 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:22.087 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:22.089 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:22.090 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:22.091 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.092 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:22.102 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:22.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37b4038440 msgr2=0x7f37b403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f37b4038440 0x7f37b403a8f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f37c8033000 tx=0x7f37c8036000 comp rx=0 tx=0).stop 2024-09-18T12:36:22.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d7f9d640 10 monclient: shutdown 2024-09-18T12:36:22.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d7f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 msgr2=0x7f37d0128af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d653a640 1 -- 172.21.15.116:0/1038771560 reap_dead start 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.073+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 0x7f37d0128af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00b5c0 tx=0x7f37cc009120 comp rx=0 tx=0).stop 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.074+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 shutdown_connections 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.074+0000 7f37d7f9d640 1 --2- 172.21.15.116:0/1038771560 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f37d00975c0 0x7f37d0128af0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.074+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 >> 172.21.15.116:0/1038771560 conn(0x7f37d0067b50 msgr2=0x7f37d00a0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.074+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 shutdown_connections 2024-09-18T12:36:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.074+0000 7f37d7f9d640 1 -- 172.21.15.116:0/1038771560 wait complete. 2024-09-18T12:36:22.106 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:36:22.106 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:36:22.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:36:22.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.198+0000 7f2d24f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:22.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.198+0000 7f2d24f9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:22.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.198+0000 7f2d24f9d640 10 monclient: monmap: 2024-09-18T12:36:22.200 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:22.200 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.200 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:22.199694+0000 2024-09-18T12:36:22.200 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:22.199694+0000 2024-09-18T12:36:22.200 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:22.201 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.201 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:22.201 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:22.201 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:22.201 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:22.201 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:22.201 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.198+0000 7f2d24f9d640 1 Processor -- start 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.198+0000 7f2d24f9d640 1 -- start start 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.198+0000 7f2d24f9d640 10 monclient: init 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d20131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d2009e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d20137bc0 0x7f2d20139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d20137bc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2009faf0 con 0x7f2d2009e910 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2009fd30 con 0x7f2d20131ab0 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d20132590 con 0x7f2d20137bc0 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d24f9d640 10 monclient(hunting): authenticate will time out at 1409.521302s 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d1dd9b640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f2d2009e910 legacy=0x7f2d2009ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d1ed9d640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f2d20137bc0 msgr2=0x7f2d20139fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d1ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d20137bc0 0x7f2d20139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d1e59c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f2d20131ab0 0x7f2d20131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:41500/0 (socket says 172.21.15.116:41500) 2024-09-18T12:36:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.199+0000 7f2d1e59c640 1 -- 172.21.15.116:0/462098600 learned_addr learned my addr 172.21.15.116:0/462098600 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.201+0000 7f2d1e59c640 1 -- 172.21.15.116:0/462098600 >> v1:172.21.15.144:6789/0 conn(0x7f2d20131ab0 legacy=0x7f2d20131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.201+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011190912 0 0) 0x7f2d2009fd30 con 0x7f2d20131ab0 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.201+0000 7f2d1d59a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.201+0000 7f2d1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.201+0000 7f2d1d59a640 10 monclient(hunting): my global_id is 124450 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.201+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d080034e0 con 0x7f2d20131ab0 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2253429224 0 0) 0x7f2d080034e0 con 0x7f2d20131ab0 2024-09-18T12:36:22.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 >> v2:172.21.15.116:3300/0 conn(0x7f2d20137bc0 msgr2=0x7f2d20139fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 --2- 172.21.15.116:0/462098600 >> v2:172.21.15.116:3300/0 conn(0x7f2d20137bc0 0x7f2d20139fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 >> v1:172.21.15.116:6789/0 conn(0x7f2d2009e910 legacy=0x7f2d2009ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d200a1110 con 0x7f2d20131ab0 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: _finish_auth 0 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f2d10001be0 con 0x7f2d20131ab0 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:36:22.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: dump: 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.218 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 >> v1:172.21.15.144:6789/0 conn(0x7f2d20131ab0 legacy=0x7f2d20131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d080039d0 0x7f2d08003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): picked mon.a con 0x7f2d080039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d08004780 0x7f2d08004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): picked mon.b con 0x7f2d08004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): picked mon.c con 0x7f2d08005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d20132590 con 0x7f2d080039d0 2024-09-18T12:36:22.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0800c290 con 0x7f2d08004780 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0800c450 con 0x7f2d08005330 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1d59a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1e59c640 1 -- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d080039d0 msgr2=0x7f2d08003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1e59c640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d080039d0 0x7f2d08003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): get_auth_request con 0x7f2d08005330 auth_method 0 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d08004780 0x7f2d08004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 10 monclient(hunting): get_auth_request con 0x7f2d08004780 auth_method 0 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:36:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient(hunting): handle_auth_done global_id 124450 payload 293 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 1 -- 172.21.15.116:0/462098600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d08004780 msgr2=0x7f2d08004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d08004780 0x7f2d08004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 1 -- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d080039d0 msgr2=0x7f2d08003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d080039d0 0x7f2d08003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 1 monclient: found mon.c 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:22.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 1 -- 172.21.15.116:0/462098600 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080034e0 con 0x7f2d08005330 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 10 monclient: _finish_auth 0 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d1ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d24f9d640 5 monclient: authenticate success, global_id 124450 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.202+0000 7f2d24f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.203+0000 7f2d1ed9d640 1 -- 172.21.15.116:0/462098600 reap_dead start 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1ed9d640 1 -- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 msgr2=0x7f2d0800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d0800b9d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d180099c0 tx=0x7f2d18009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e833b59a94e44e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d18003b80 con 0x7f2d08005330 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 10 monclient: dump: 2024-09-18T12:36:22.223 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.224 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d24f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d18003d20 con 0x7f2d08005330 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 1 -- 172.21.15.116:0/462098600 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1802ab80 con 0x7f2d08005330 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d24f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d24f9d640 4 set_mon_vals no callback set 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.205+0000 7f2d1d59a640 10 monclient: dump: 2024-09-18T12:36:22.225 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 10 monclient: shutdown 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:22.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 1 -- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 msgr2=0x7f2d0800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d0800b9d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d180099c0 tx=0x7f2d18009060 comp rx=0 tx=0).stop 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 1 -- 172.21.15.116:0/462098600 shutdown_connections 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 1 --2- 172.21.15.116:0/462098600 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d0800b9d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 1 -- 172.21.15.116:0/462098600 >> 172.21.15.116:0/462098600 conn(0x7f2d20067d30 msgr2=0x7f2d200a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 1 -- 172.21.15.116:0/462098600 shutdown_connections 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 1 -- 172.21.15.116:0/462098600 wait complete. 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.206+0000 7f2d24f9d640 10 monclient: monmap: 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:22.227 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 Processor -- start 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 -- start start 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient: init 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d20141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2d08005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d20130ac0 0x7f2d20098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2d20130ac0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d20130eb0 0x7f2d20098f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2d20130eb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2011c210 con 0x7f2d20130ac0 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2009bde0 con 0x7f2d20130eb0 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2009d740 con 0x7f2d08005330 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d24f9d640 10 monclient(hunting): authenticate will time out at 1409.529650s 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1dd9b640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d20130ac0 msgr2=0x7f2d20098a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d20130ac0 0x7f2d20098a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:22.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d0e7fc640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d20141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d20141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59342/0 (socket says 172.21.15.116:59342) 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 -- 172.21.15.116:0/2110591090 learned_addr learned my addr 172.21.15.116:0/2110591090 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): get_auth_request con 0x7f2d08005330 auth_method 0 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient(hunting): handle_auth_done global_id 144185 payload 824 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d20130eb0 0x7f2d20098f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d20130eb0 msgr2=0x7f2d20098f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d20130eb0 0x7f2d20098f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d20130ac0 msgr2=0x7f2d20098a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d20130ac0 0x7f2d20098a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 monclient: found mon.noname-c 2024-09-18T12:36:22.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.207+0000 7f2d1e59c640 1 -- 172.21.15.116:0/2110591090 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2009aa20 con 0x7f2d08005330 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.208+0000 7f2d1ed9d640 10 monclient: get_auth_request con 0x7f2d20130eb0 auth_method 0 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.208+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d20130eb0 0x7f2d20098f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d1e59c640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 msgr2=0x7f2d20141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d1e59c640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d20141090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000cf90 tx=0x7f2d10001550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d10007e00 con 0x7f2d08005330 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: dump: 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:22.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1000aad0 con 0x7f2d08005330 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d24f9d640 5 monclient: authenticate success, global_id 144185 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1000ae90 con 0x7f2d08005330 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d24f9d640 10 monclient: _renew_subs 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d24f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2009a510 con 0x7f2d08005330 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d0e7fc640 10 monclient: dump: 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.234 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d1cd99640 4 set_mon_vals no callback set 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d24f9d640 10 monclient: _renew_subs 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d24f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.210+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d201415d0 con 0x7f2d08005330 2024-09-18T12:36:22.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d24f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d24f9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 0x7f2d200994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d24f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d20063740 con 0x7f2d2002aa40 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2d100073a0 con 0x7f2d08005330 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d0e7fc640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d00038440 0x7f2d0003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 0x7f2d200994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient: get_auth_request con 0x7f2d2002aa40 auth_method 0 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2d1004c5d0 con 0x7f2d08005330 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d00038440 0x7f2d0003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1ed9d640 10 monclient: get_auth_request con 0x7f2d00038440 auth_method 0 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_done global_id 124453 payload 824 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.211+0000 7f2d1dd9b640 1 -- 172.21.15.116:0/2110591090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d2002ae10 con 0x7f2d2002aa40 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d1ed9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d00038440 0x7f2d0003a8f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2d2006a280 tx=0x7f2d18030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d1dd9b640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 msgr2=0x7f2d200994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d1dd9b640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 0x7f2d200994a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d14003940 tx=0x7f2d14009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1400e950 con 0x7f2d2002aa40 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 10 monclient: dump: 2024-09-18T12:36:22.237 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2d1400eab0 con 0x7f2d2002aa40 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 msgr2=0x7f2d200994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 0x7f2d200994a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d14003940 tx=0x7f2d14009060 comp rx=0 tx=0).stop 2024-09-18T12:36:22.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 msgr2=0x7f2d200994a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:22.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.214+0000 7f2d0e7fc640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2002aa40 0x7f2d200994a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d24f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:22.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d24f9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 0x7f2d2005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d24f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2004bf60 con 0x7f2d2005e310 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 0x7f2d2005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 10 monclient: get_auth_request con 0x7f2d2005e310 auth_method 0 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.220+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.221+0000 7f2d1dd9b640 10 monclient(hunting): handle_auth_done global_id 124456 payload 824 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.221+0000 7f2d1dd9b640 1 -- 172.21.15.116:0/2110591090 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2d2009d210 con 0x7f2d2005e310 2024-09-18T12:36:22.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d1dd9b640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 msgr2=0x7f2d2005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d1dd9b640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 0x7f2d2005e710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2d14003a40 tx=0x7f2d14019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14022070 con 0x7f2d2005e310 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 10 monclient: dump: 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:22.242 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7f2d14006970 con 0x7f2d2005e310 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 msgr2=0x7f2d2005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 0x7f2d2005e710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2d14003a40 tx=0x7f2d14019000 comp rx=0 tx=0).stop 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 msgr2=0x7f2d2005e710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:22.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.223+0000 7f2d0e7fc640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d2005e310 0x7f2d2005e710 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 112, 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.244 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.245 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:22.246 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:22.247 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.248 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:22.249 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:22.250 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:22.251 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:22.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:22.254 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:22.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:22.256 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:22.257 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d00038440 msgr2=0x7f2d0003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d00038440 0x7f2d0003a8f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2d2006a280 tx=0x7f2d18030000 comp rx=0 tx=0).stop 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 10 monclient: shutdown 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 msgr2=0x7f2d20141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 --2- 172.21.15.116:0/2110591090 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d08005330 0x7f2d20141090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000cf90 tx=0x7f2d10001550 comp rx=0 tx=0).stop 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d1ed9d640 1 -- 172.21.15.116:0/2110591090 reap_dead start 2024-09-18T12:36:22.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d1ed9d640 1 -- 172.21.15.116:0/2110591090 reap_dead start 2024-09-18T12:36:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 shutdown_connections 2024-09-18T12:36:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 >> 172.21.15.116:0/2110591090 conn(0x7f2d20067d30 msgr2=0x7f2d200a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 shutdown_connections 2024-09-18T12:36:22.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:22.224+0000 7f2d24f9d640 1 -- 172.21.15.116:0/2110591090 wait complete. 2024-09-18T12:36:22.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:36:22.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:36:42.260 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:36:42.260 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:36:42.260 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:36:42.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:36:42.264 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:36:42.264 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:36:42.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.367+0000 7f1d9a4b8640 10 monclient: get_monmap_and_config 2024-09-18T12:36:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.367+0000 7f1d9a4b8640 10 monclient: build_initial_monmap 2024-09-18T12:36:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.367+0000 7f1d9a4b8640 10 monclient: monmap: 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:42.368309+0000 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:42.368309+0000 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:42.369 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.368+0000 7f1d9a4b8640 1 Processor -- start 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.368+0000 7f1d9a4b8640 1 -- start start 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.368+0000 7f1d9a4b8640 10 monclient: init 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d94131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d9409e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1d94137bc0 0x7f1d94139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient(hunting): picked mon.noname-c con 0x7f1d94137bc0 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9409faf0 con 0x7f1d94131ab0 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9409fd30 con 0x7f1d9409e910 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d94132590 con 0x7f1d94137bc0 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d9a4b8640 10 monclient(hunting): authenticate will time out at 1429.690185s 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d93fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1d94131ab0 0x7f1d94131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46382/0 (socket says 172.21.15.116:46382) 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d93fff640 1 -- 172.21.15.116:0/3844723434 learned_addr learned my addr 172.21.15.116:0/3844723434 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 1 --2- 172.21.15.116:0/3844723434 >> v2:172.21.15.144:3300/0 conn(0x7f1d94137bc0 0x7f1d94139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): get_auth_request con 0x7f1d94137bc0 auth_method 0 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient(hunting): handle_auth_done global_id 124459 payload 274 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 1 -- 172.21.15.116:0/3844723434 >> v1:172.21.15.144:6789/0 conn(0x7f1d9409e910 legacy=0x7f1d9409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 1 -- 172.21.15.116:0/3844723434 >> v1:172.21.15.116:6789/0 conn(0x7f1d94131ab0 legacy=0x7f1d94131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient: _finish_hunting 0 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 1 monclient: found mon.noname-c 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:36:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.369+0000 7f1d98a55640 1 -- 172.21.15.116:0/3844723434 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d940a1110 con 0x7f1d94137bc0 2024-09-18T12:36:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.372+0000 7f1d98a55640 1 -- 172.21.15.116:0/3844723434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94137bc0 msgr2=0x7f1d94139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.372+0000 7f1d98a55640 1 --2- 172.21.15.116:0/3844723434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94137bc0 0x7f1d94139fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009e90 tx=0x7f1d8c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=121de06e067b7e08 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 1 -- 172.21.15.116:0/3844723434 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c007840 con 0x7f1d94137bc0 2024-09-18T12:36:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: dump: 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: _finish_auth 0 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 1 -- 172.21.15.116:0/3844723434 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c02d050 con 0x7f1d94137bc0 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 5 monclient: authenticate success, global_id 124459 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 1 -- 172.21.15.116:0/3844723434 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c007bd0 con 0x7f1d94137bc0 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 4 set_mon_vals no callback set 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d92ffd640 10 monclient: dump: 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 10 monclient: shutdown 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3844723434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94137bc0 msgr2=0x7f1d94139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3844723434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94137bc0 0x7f1d94139fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009e90 tx=0x7f1d8c009780 comp rx=0 tx=0).stop 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3844723434 shutdown_connections 2024-09-18T12:36:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3844723434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94137bc0 0x7f1d94139fa0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3844723434 >> 172.21.15.116:0/3844723434 conn(0x7f1d94067d30 msgr2=0x7f1d940a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3844723434 shutdown_connections 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.373+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3844723434 wait complete. 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.374+0000 7f1d9a4b8640 10 monclient: build_initial_monmap 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.374+0000 7f1d9a4b8640 10 monclient: monmap: 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.374+0000 7f1d9a4b8640 1 Processor -- start 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.374+0000 7f1d9a4b8640 1 -- start start 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.374+0000 7f1d9a4b8640 10 monclient: init 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 0x7f1d9412de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): picked mon.noname-c con 0x7f1d9409e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94131ab0 0x7f1d9412e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): picked mon.noname-b con 0x7f1d94131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 0x7f1d9412e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): picked mon.noname-a con 0x7f1d94137bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9409bde0 con 0x7f1d94137bc0 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940a54b0 con 0x7f1d94131ab0 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d94137330 con 0x7f1d9409e910 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d9a4b8640 10 monclient(hunting): authenticate will time out at 1429.696254s 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 0x7f1d9412de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 0x7f1d9412e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 0x7f1d9412de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54466/0 (socket says 172.21.15.116:54466) 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 -- 172.21.15.116:0/3788772627 learned_addr learned my addr 172.21.15.116:0/3788772627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): get_auth_request con 0x7f1d9409e910 auth_method 0 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 10 monclient(hunting): get_auth_request con 0x7f1d94137bc0 auth_method 0 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94131ab0 0x7f1d9412e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 1 -- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 msgr2=0x7f1d9412e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 1 -- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 msgr2=0x7f1d9412e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 0x7f1d9412e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 10 monclient(hunting): get_auth_request con 0x7f1d94131ab0 auth_method 0 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:42.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d98a55640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 0x7f1d9412e880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient(hunting): handle_auth_done global_id 144188 payload 824 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 -- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94131ab0 msgr2=0x7f1d9412e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94131ab0 0x7f1d9412e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 -- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 msgr2=0x7f1d9412e880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 0x7f1d9412e880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 monclient: found mon.noname-c 2024-09-18T12:36:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.375+0000 7f1d93fff640 1 -- 172.21.15.116:0/3788772627 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c009a20 con 0x7f1d9409e910 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.377+0000 7f1d93fff640 1 -- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 msgr2=0x7f1d9412de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.377+0000 7f1d93fff640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 0x7f1d9412de00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400c610 tx=0x7f1d84004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 1 -- 172.21.15.116:0/3788772627 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84005cb0 con 0x7f1d9409e910 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: dump: 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: _finish_auth 0 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d9a4b8640 5 monclient: authenticate success, global_id 144188 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 1 -- 172.21.15.116:0/3788772627 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d84005e30 con 0x7f1d9409e910 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d9a4b8640 10 monclient: _renew_subs 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d9a4b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d94132b50 con 0x7f1d9409e910 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 1 -- 172.21.15.116:0/3788772627 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84007340 con 0x7f1d9409e910 2024-09-18T12:36:42.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 10 monclient: dump: 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d927fc640 4 set_mon_vals no callback set 2024-09-18T12:36:42.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d9a4b8640 10 monclient: _renew_subs 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d9a4b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d94130070 con 0x7f1d9409e910 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.378+0000 7f1d7bfff640 1 -- 172.21.15.116:0/3788772627 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1d8401b400 con 0x7f1d9409e910 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d7bfff640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d80038490 0x7f1d8003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d7bfff640 1 -- 172.21.15.116:0/3788772627 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1d8404bec0 con 0x7f1d9409e910 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d9a4b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d9a4b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d9a4b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d94130560 con 0x7f1d9409e910 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d937fe640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d80038490 0x7f1d8003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:42.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.379+0000 7f1d937fe640 10 monclient: get_auth_request con 0x7f1d80038490 auth_method 0 2024-09-18T12:36:42.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.382+0000 7f1d937fe640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d80038490 0x7f1d8003a940 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1d9409f3c0 tx=0x7f1d88007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:42.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.383+0000 7f1d7bfff640 1 -- 172.21.15.116:0/3788772627 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1d8401ab40 con 0x7f1d9409e910 2024-09-18T12:36:42.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.383+0000 7f1d7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:42.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.383+0000 7f1d7bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:42.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.597+0000 7f1d9a4b8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.597+0000 7f1d9a4b8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.597+0000 7f1d9a4b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.597+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1d9404bf60 con 0x7f1d9409e910 2024-09-18T12:36:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.598+0000 7f1d7bfff640 1 -- 172.21.15.116:0/3788772627 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f1d84029080 con 0x7f1d9409e910 2024-09-18T12:36:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.598+0000 7f1d7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.598+0000 7f1d7bfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:42.599 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d80038490 msgr2=0x7f1d8003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d80038490 0x7f1d8003a940 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1d9409f3c0 tx=0x7f1d88007e10 comp rx=0 tx=0).stop 2024-09-18T12:36:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 10 monclient: shutdown 2024-09-18T12:36:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 msgr2=0x7f1d9412de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 0x7f1d9412de00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400c610 tx=0x7f1d84004960 comp rx=0 tx=0).stop 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 shutdown_connections 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1d80038490 0x7f1d8003a940 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d94137bc0 0x7f1d9412e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d94131ab0 0x7f1d9412e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 --2- 172.21.15.116:0/3788772627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1d9409e910 0x7f1d9412de00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 >> 172.21.15.116:0/3788772627 conn(0x7f1d94067d30 msgr2=0x7f1d941390e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 shutdown_connections 2024-09-18T12:36:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:42.600+0000 7f1d9a4b8640 1 -- 172.21.15.116:0/3788772627 wait complete. 2024-09-18T12:36:45.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.718+0000 7fc79751d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.718+0000 7fc79751d640 10 monclient: build_initial_monmap 2024-09-18T12:36:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.718+0000 7fc79751d640 10 monclient: monmap: 2024-09-18T12:36:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:45.719 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:45.718932+0000 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:45.718932+0000 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:45.720 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:45.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.718+0000 7fc79751d640 1 Processor -- start 2024-09-18T12:36:45.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 1 -- start start 2024-09-18T12:36:45.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient: init 2024-09-18T12:36:45.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:45.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T12:36:45.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 0x7fc79012ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient(hunting): picked mon.noname-a con 0x7fc7900974f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc79009a8a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc790133070 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7900978c0 con 0x7fc790133070 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79009ca20 con 0x7fc79009a8a0 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc790098ad0 con 0x7fc7900974f0 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc79751d640 10 monclient(hunting): authenticate will time out at 1433.040659s 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 0x7fc79012ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 0x7fc79012ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43900/0 (socket says 172.21.15.116:43900) 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc795aba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc790133070 0x7fc790135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39036/0 (socket says 172.21.15.116:39036) 2024-09-18T12:36:45.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 1 -- 172.21.15.116:0/1769136408 learned_addr learned my addr 172.21.15.116:0/1769136408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 10 monclient(hunting): get_auth_request con 0x7fc7900974f0 auth_method 0 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 1 -- 172.21.15.116:0/1769136408 >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 msgr2=0x7fc79012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 1 -- 172.21.15.116:0/1769136408 >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 msgr2=0x7fc79012ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 1 --2- 172.21.15.116:0/1769136408 >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 0x7fc79012ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.719+0000 7fc7952b9640 1 --2- 172.21.15.116:0/1769136408 >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 0x7fc79012ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.721+0000 7fc795aba640 1 -- 172.21.15.116:0/1769136408 >> v1:172.21.15.116:6789/0 conn(0x7fc790133070 legacy=0x7fc790135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc794ab8640 1 -- 172.21.15.116:0/1769136408 >> v1:172.21.15.144:6789/0 conn(0x7fc79009a8a0 legacy=0x7fc79009ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119301204 0 0) 0x7fc79009ca20 con 0x7fc79009a8a0 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:45.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient(hunting): my global_id is 124465 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc77c0034e0 con 0x7fc79009a8a0 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3207685317 0 0) 0x7fc77c0034e0 con 0x7fc79009a8a0 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 msgr2=0x7fc79012ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 --2- 172.21.15.116:0/1769136408 >> v2:172.21.15.116:3300/0 conn(0x7fc7900974f0 0x7fc79012ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 >> v1:172.21.15.116:6789/0 conn(0x7fc790133070 legacy=0x7fc790135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc790098db0 con 0x7fc79009a8a0 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: _finish_auth 0 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:45.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fc78c0018b0 con 0x7fc79009a8a0 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc79751d640 5 monclient: authenticate success, global_id 124465 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc79751d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: dump: 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:45.730 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 >> v1:172.21.15.144:6789/0 conn(0x7fc79009a8a0 legacy=0x7fc79009ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc77c003af0 0x7fc77c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient(hunting): picked mon.b con 0x7fc77c003af0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:45.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc77c004780 0x7fc77c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient(hunting): picked mon.c con 0x7fc77c004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c005380 0x7fc77c00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient(hunting): picked mon.a con 0x7fc77c005380 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.722+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc790098ad0 con 0x7fc77c005380 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc787fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c00c350 con 0x7fc77c003af0 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc787fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c00c510 con 0x7fc77c004780 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc787fff640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc787fff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc79751d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:45.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc77c004780 0x7fc77c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): get_auth_request con 0x7fc77c004780 auth_method 0 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc77c003af0 0x7fc77c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c005380 0x7fc77c00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 10 monclient(hunting): get_auth_request con 0x7fc77c003af0 auth_method 0 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 10 monclient(hunting): get_auth_request con 0x7fc77c005380 auth_method 0 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:45.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc7952b9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient(hunting): handle_auth_done global_id 124465 payload 293 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 1 -- 172.21.15.116:0/1769136408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc77c003af0 msgr2=0x7fc77c003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc77c003af0 0x7fc77c003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 1 -- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c005380 msgr2=0x7fc77c00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:45.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c005380 0x7fc77c00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient: _finish_hunting 0 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 1 monclient: found mon.c 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 1 -- 172.21.15.116:0/1769136408 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c0034e0 con 0x7fc77c004780 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 10 monclient: _finish_auth 0 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc794ab8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c005380 0x7fc77c00ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.723+0000 7fc795aba640 1 -- 172.21.15.116:0/1769136408 reap_dead start 2024-09-18T12:36:45.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc794ab8640 1 -- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc77c004780 msgr2=0x7fc77c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc794ab8640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc77c004780 0x7fc77c004bf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c5c0 tx=0x7fc78c007a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2998e43d42fee54 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0070f0 con 0x7fc77c004780 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 10 monclient: dump: 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c007270 con 0x7fc77c004780 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 4 set_mon_vals no callback set 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 1 -- 172.21.15.116:0/1769136408 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c007420 con 0x7fc77c004780 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc787fff640 10 monclient: dump: 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:45.742 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 10 monclient: shutdown 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 1 -- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc77c004780 msgr2=0x7fc77c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc77c004780 0x7fc77c004bf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c5c0 tx=0x7fc78c007a60 comp rx=0 tx=0).stop 2024-09-18T12:36:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 1 -- 172.21.15.116:0/1769136408 shutdown_connections 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 1 --2- 172.21.15.116:0/1769136408 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc77c004780 0x7fc77c004bf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.726+0000 7fc79751d640 1 -- 172.21.15.116:0/1769136408 >> 172.21.15.116:0/1769136408 conn(0x7fc7900672b0 msgr2=0x7fc7900676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.727+0000 7fc79751d640 1 -- 172.21.15.116:0/1769136408 shutdown_connections 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.727+0000 7fc79751d640 1 -- 172.21.15.116:0/1769136408 wait complete. 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.727+0000 7fc79751d640 10 monclient: build_initial_monmap 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.727+0000 7fc79751d640 10 monclient: monmap: 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.727+0000 7fc79751d640 1 Processor -- start 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.727+0000 7fc79751d640 1 -- start start 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.727+0000 7fc79751d640 10 monclient: init 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 0x7fc79012fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): picked mon.noname-a con 0x7fc77c004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc790130260 0x7fc790130630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): picked mon.noname-b con 0x7fc790130260 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc790130b70 0x7fc790147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): picked mon.noname-c con 0x7fc790130b70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79006cbe0 con 0x7fc77c004780 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc790132370 con 0x7fc790130260 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc790098330 con 0x7fc790130b70 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc79751d640 10 monclient(hunting): authenticate will time out at 1433.049510s 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 0x7fc79012fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc790130b70 0x7fc790147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 0x7fc79012fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43922/0 (socket says 172.21.15.116:43922) 2024-09-18T12:36:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 1 -- 172.21.15.116:0/2185602650 learned_addr learned my addr 172.21.15.116:0/2185602650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 10 monclient(hunting): get_auth_request con 0x7fc77c004780 auth_method 0 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): get_auth_request con 0x7fc790130b70 auth_method 0 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 1 -- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 msgr2=0x7fc79012fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 1 -- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 msgr2=0x7fc79012fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 0x7fc79012fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc7952b9640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 0x7fc79012fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc794ab8640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc790130260 0x7fc790130630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient(hunting): handle_auth_done global_id 144191 payload 824 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 1 -- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc790130260 msgr2=0x7fc790130630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc790130260 0x7fc790130630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 1 -- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 msgr2=0x7fc79012fd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 0x7fc79012fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient: _finish_hunting 0 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 1 monclient: found mon.noname-c 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc795aba640 1 -- 172.21.15.116:0/2185602650 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79006c590 con 0x7fc790130b70 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc794ab8640 10 monclient: get_auth_request con 0x7fc790130260 auth_method 0 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.728+0000 7fc794ab8640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc790130260 0x7fc790130630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc795aba640 1 -- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc790130b70 msgr2=0x7fc790147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc795aba640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc790130b70 0x7fc790147550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc7900987b0 tx=0x7fc7880018d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 1 -- 172.21.15.116:0/2185602650 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788007a60 con 0x7fc790130b70 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: dump: 2024-09-18T12:36:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: _finish_auth 0 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 1 -- 172.21.15.116:0/2185602650 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc788012560 con 0x7fc790130b70 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc79751d640 5 monclient: authenticate success, global_id 144191 2024-09-18T12:36:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc79751d640 10 monclient: _renew_subs 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc79751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7901317a0 con 0x7fc790130b70 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 1 -- 172.21.15.116:0/2185602650 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788012900 con 0x7fc790130b70 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7857fa640 10 monclient: dump: 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc7877fe640 4 set_mon_vals no callback set 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc79751d640 10 monclient: _renew_subs 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc79751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.731+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc790130fc0 con 0x7fc790130b70 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc7857fa640 1 -- 172.21.15.116:0/2185602650 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc788012cf0 con 0x7fc790130b70 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc79751d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc79751d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc79751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc7857fa640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc774038490 0x7fc77403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7901311b0 con 0x7fc790130b70 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc7857fa640 1 -- 172.21.15.116:0/2185602650 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc78804bcb0 con 0x7fc790130b70 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.732+0000 7fc7952b9640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc774038490 0x7fc77403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.733+0000 7fc7952b9640 10 monclient: get_auth_request con 0x7fc774038490 auth_method 0 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.735+0000 7fc7952b9640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc774038490 0x7fc77403a940 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc780006a70 tx=0x7fc78000f040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.737+0000 7fc7857fa640 1 -- 172.21.15.116:0/2185602650 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc7880135b0 con 0x7fc790130b70 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.737+0000 7fc7857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.737+0000 7fc7857fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:45.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.947+0000 7fc79751d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:45.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.947+0000 7fc79751d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.947+0000 7fc79751d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.947+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc79012a470 con 0x7fc790130b70 2024-09-18T12:36:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.948+0000 7fc7857fa640 1 -- 172.21.15.116:0/2185602650 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fc7880112b0 con 0x7fc790130b70 2024-09-18T12:36:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.948+0000 7fc7857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.948+0000 7fc7857fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:45.949 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.949+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc774038490 msgr2=0x7fc77403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.949+0000 7fc79751d640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc774038490 0x7fc77403a940 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc780006a70 tx=0x7fc78000f040 comp rx=0 tx=0).stop 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.949+0000 7fc79751d640 10 monclient: shutdown 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.949+0000 7fc79751d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.949+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc790130b70 msgr2=0x7fc790147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.949+0000 7fc79751d640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc790130b70 0x7fc790147550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc7900987b0 tx=0x7fc7880018d0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 shutdown_connections 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc774038490 0x7fc77403a940 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc790130b70 0x7fc790147550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc790130260 0x7fc790130630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 --2- 172.21.15.116:0/2185602650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc77c004780 0x7fc79012fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 >> 172.21.15.116:0/2185602650 conn(0x7fc7900672b0 msgr2=0x7fc79009d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 shutdown_connections 2024-09-18T12:36:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:45.950+0000 7fc79751d640 1 -- 172.21.15.116:0/2185602650 wait complete. 2024-09-18T12:36:48.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:49.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.065+0000 7fa76962b640 10 monclient: get_monmap_and_config 2024-09-18T12:36:49.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.065+0000 7fa76962b640 10 monclient: build_initial_monmap 2024-09-18T12:36:49.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.065+0000 7fa76962b640 10 monclient: monmap: 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:49.066331+0000 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:49.066331+0000 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:49.067 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 1 Processor -- start 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 1 -- start start 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient: init 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 0x7fa764131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient(hunting): picked mon.noname-a con 0x7fa764131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa76409fe10 0x7fa76409ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient(hunting): picked mon.noname-b con 0x7fa76409fe10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:49.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa76409ef70 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7641325b0 con 0x7fa76409ef70 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7640a11f0 con 0x7fa764131980 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa764132290 con 0x7fa76409fe10 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.066+0000 7fa76962b640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa76962b640 10 monclient(hunting): authenticate will time out at 1436.388007s 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 0x7fa764131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa76409fe10 0x7fa76409ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 0x7fa764131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43936/0 (socket says 172.21.15.116:43936) 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 1 -- 172.21.15.116:0/1120423541 learned_addr learned my addr 172.21.15.116:0/1120423541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:49.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): get_auth_request con 0x7fa76409fe10 auth_method 0 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 10 monclient(hunting): get_auth_request con 0x7fa764131980 auth_method 0 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 1 -- 172.21.15.116:0/1120423541 >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 msgr2=0x7fa764131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:36:49.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 1 -- 172.21.15.116:0/1120423541 >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 msgr2=0x7fa764131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 1 --2- 172.21.15.116:0/1120423541 >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 0x7fa764131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa762ffd640 1 --2- 172.21.15.116:0/1120423541 >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 0x7fa764131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient(hunting): handle_auth_done global_id 144194 payload 274 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 1 -- 172.21.15.116:0/1120423541 >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 msgr2=0x7fa764131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 1 --2- 172.21.15.116:0/1120423541 >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 0x7fa764131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 1 -- 172.21.15.116:0/1120423541 >> v1:172.21.15.144:6789/0 conn(0x7fa76409ef70 legacy=0x7fa764068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 1 monclient: found mon.noname-b 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:49.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.067+0000 7fa75bfff640 1 -- 172.21.15.116:0/1120423541 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76409f390 con 0x7fa76409fe10 2024-09-18T12:36:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa75bfff640 1 -- 172.21.15.116:0/1120423541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa76409fe10 msgr2=0x7fa76409ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa75bfff640 1 --2- 172.21.15.116:0/1120423541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa76409fe10 0x7fa76409ea30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009850 tx=0x7fa74c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d8c37910b235bf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 1 -- 172.21.15.116:0/1120423541 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c02add0 con 0x7fa76409fe10 2024-09-18T12:36:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: dump: 2024-09-18T12:36:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 1 -- 172.21.15.116:0/1120423541 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c032440 con 0x7fa76409fe10 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 1 -- 172.21.15.116:0/1120423541 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c032820 con 0x7fa76409fe10 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa7627fc640 10 monclient: dump: 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa76962b640 5 monclient: authenticate success, global_id 144194 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa76962b640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.070+0000 7fa76962b640 4 set_mon_vals no callback set 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 10 monclient: shutdown 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 -- 172.21.15.116:0/1120423541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa76409fe10 msgr2=0x7fa76409ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 --2- 172.21.15.116:0/1120423541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa76409fe10 0x7fa76409ea30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009850 tx=0x7fa74c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 -- 172.21.15.116:0/1120423541 shutdown_connections 2024-09-18T12:36:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 --2- 172.21.15.116:0/1120423541 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa76409fe10 0x7fa76409ea30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 --2- 172.21.15.116:0/1120423541 >> v2:172.21.15.116:3300/0 conn(0x7fa764131980 0x7fa764131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 -- 172.21.15.116:0/1120423541 >> 172.21.15.116:0/1120423541 conn(0x7fa7640a5050 msgr2=0x7fa7640a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 -- 172.21.15.116:0/1120423541 shutdown_connections 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 -- 172.21.15.116:0/1120423541 wait complete. 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 10 monclient: build_initial_monmap 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 10 monclient: monmap: 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.071+0000 7fa76962b640 1 Processor -- start 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 1 -- start start 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient: init 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 0x7fa76409a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): picked mon.noname-a con 0x7fa76409ef70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa76409fe10 0x7fa764130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:49.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): picked mon.noname-b con 0x7fa76409fe10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa764131980 0x7fa764141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): picked mon.noname-c con 0x7fa764131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76409e370 con 0x7fa76409ef70 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76409b980 con 0x7fa76409fe10 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76411cc60 con 0x7fa764131980 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa76962b640 10 monclient(hunting): authenticate will time out at 1436.393568s 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 0x7fa76409a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 0x7fa76409a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43946/0 (socket says 172.21.15.116:43946) 2024-09-18T12:36:49.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 1 -- 172.21.15.116:0/2439867411 learned_addr learned my addr 172.21.15.116:0/2439867411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa764131980 0x7fa764141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 10 monclient(hunting): get_auth_request con 0x7fa76409ef70 auth_method 0 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): get_auth_request con 0x7fa764131980 auth_method 0 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 1 -- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 msgr2=0x7fa76409a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 1 -- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 msgr2=0x7fa76409a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:36:49.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 0x7fa76409a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa762ffd640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 0x7fa76409a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa75bfff640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa76409fe10 0x7fa764130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa75bfff640 10 monclient(hunting): get_auth_request con 0x7fa76409fe10 auth_method 0 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa75bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa75bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient(hunting): handle_auth_done global_id 144197 payload 824 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 1 -- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa76409fe10 msgr2=0x7fa764130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:49.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa76409fe10 0x7fa764130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 1 -- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 msgr2=0x7fa76409a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 0x7fa76409a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient: _finish_hunting 0 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 1 monclient: found mon.noname-c 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa7637fe640 1 -- 172.21.15.116:0/2439867411 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c009390 con 0x7fa764131980 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.072+0000 7fa75bfff640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa76409fe10 0x7fa764130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa7637fe640 1 -- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa764131980 msgr2=0x7fa764141230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa7637fe640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa764131980 0x7fa764141230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00b810 tx=0x7fa75c0092b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 1 -- 172.21.15.116:0/2439867411 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c00c940 con 0x7fa764131980 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: dump: 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: _finish_auth 0 2024-09-18T12:36:49.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa76962b640 5 monclient: authenticate success, global_id 144197 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 1 -- 172.21.15.116:0/2439867411 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c00cae0 con 0x7fa764131980 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa76962b640 10 monclient: _renew_subs 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa76962b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76411dbc0 con 0x7fa764131980 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 1 -- 172.21.15.116:0/2439867411 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c00cea0 con 0x7fa764131980 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa75affd640 10 monclient: dump: 2024-09-18T12:36:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa76962b640 10 monclient: _renew_subs 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa76962b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76409b470 con 0x7fa764131980 2024-09-18T12:36:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.075+0000 7fa761ffb640 4 set_mon_vals no callback set 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa75affd640 1 -- 172.21.15.116:0/2439867411 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa75c014620 con 0x7fa764131980 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa76962b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa76962b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa76962b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa764069f20 con 0x7fa764131980 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa75affd640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa75affd640 1 -- 172.21.15.116:0/2439867411 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa75c04bc50 con 0x7fa764131980 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa762ffd640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.076+0000 7fa762ffd640 10 monclient: get_auth_request con 0x7fa744038490 auth_method 0 2024-09-18T12:36:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.079+0000 7fa762ffd640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa754006f30 tx=0x7fa754008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.081+0000 7fa75affd640 1 -- 172.21.15.116:0/2439867411 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa764069f20 con 0x7fa764131980 2024-09-18T12:36:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.081+0000 7fa75affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.081+0000 7fa75affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.295+0000 7fa76962b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.295+0000 7fa76962b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.295+0000 7fa76962b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.295+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa764069f20 con 0x7fa764131980 2024-09-18T12:36:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.295+0000 7fa75affd640 1 -- 172.21.15.116:0/2439867411 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fa764069f20 con 0x7fa764131980 2024-09-18T12:36:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.296+0000 7fa75affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.296+0000 7fa75affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:49.297 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 msgr2=0x7fa74403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa754006f30 tx=0x7fa754008040 comp rx=0 tx=0).stop 2024-09-18T12:36:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 10 monclient: shutdown 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa764131980 msgr2=0x7fa764141230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa764131980 0x7fa764141230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00b810 tx=0x7fa75c0092b0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 shutdown_connections 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa764131980 0x7fa764141230 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa76409fe10 0x7fa764130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 --2- 172.21.15.116:0/2439867411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa76409ef70 0x7fa76409a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 >> 172.21.15.116:0/2439867411 conn(0x7fa7640a5050 msgr2=0x7fa76409e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 shutdown_connections 2024-09-18T12:36:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:49.297+0000 7fa76962b640 1 -- 172.21.15.116:0/2439867411 wait complete. 2024-09-18T12:36:52.310 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.413+0000 7f5cb0f54640 10 monclient: get_monmap_and_config 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.413+0000 7f5cb0f54640 10 monclient: build_initial_monmap 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.413+0000 7f5cb0f54640 10 monclient: monmap: 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:52.413858+0000 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:52.413858+0000 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:52.415 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:52.415 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:52.415 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:52.415 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:52.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.413+0000 7f5cb0f54640 1 Processor -- start 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.413+0000 7f5cb0f54640 1 -- start start 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.413+0000 7f5cb0f54640 10 monclient: init 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5cac1318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5cac09ea50 0x7f5cac09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient(hunting): picked mon.noname-c con 0x7f5cac09ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient(hunting): picked mon.noname-b con 0x7f5cac136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac132640 con 0x7f5cac1318f0 2024-09-18T12:36:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac131cc0 con 0x7f5cac09ea50 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac09fbf0 con 0x7f5cac136c20 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5cb0f54640 10 monclient(hunting): authenticate will time out at 1439.735524s 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caa59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5cac1318f0 0x7f5cac131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39048/0 (socket says 172.21.15.116:39048) 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:38732/0 (socket says 172.21.15.116:38732) 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caa59c640 1 -- 172.21.15.116:0/3557836585 learned_addr learned my addr 172.21.15.116:0/3557836585 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): get_auth_request con 0x7f5cac136c20 auth_method 0 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5ca9d9b640 1 --2- 172.21.15.116:0/3557836585 >> v2:172.21.15.144:3300/0 conn(0x7f5cac09ea50 0x7f5cac09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5ca9d9b640 10 monclient(hunting): get_auth_request con 0x7f5cac09ea50 auth_method 0 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5ca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5ca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5ca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient(hunting): handle_auth_done global_id 144200 payload 274 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 1 -- 172.21.15.116:0/3557836585 >> v2:172.21.15.144:3300/0 conn(0x7f5cac09ea50 msgr2=0x7f5cac09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 1 --2- 172.21.15.116:0/3557836585 >> v2:172.21.15.144:3300/0 conn(0x7f5cac09ea50 0x7f5cac09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 1 -- 172.21.15.116:0/3557836585 >> v1:172.21.15.116:6789/0 conn(0x7f5cac1318f0 legacy=0x7f5cac131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:52.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:36:52.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 1 monclient: found mon.noname-b 2024-09-18T12:36:52.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:52.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:52.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.414+0000 7f5caad9d640 1 -- 172.21.15.116:0/3557836585 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac09fe30 con 0x7f5cac136c20 2024-09-18T12:36:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5caad9d640 1 -- 172.21.15.116:0/3557836585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 msgr2=0x7f5cac139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5caad9d640 1 --2- 172.21.15.116:0/3557836585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac139000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009850 tx=0x7f5c9c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=138f0ca25ad12430 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5ca959a640 1 -- 172.21.15.116:0/3557836585 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c02add0 con 0x7f5cac136c20 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5ca959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5ca959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5ca959a640 10 monclient: dump: 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:52.428 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5ca959a640 10 monclient: _finish_auth 0 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5ca959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5ca959a640 1 -- 172.21.15.116:0/3557836585 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c032440 con 0x7f5cac136c20 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.417+0000 7f5cb0f54640 5 monclient: authenticate success, global_id 144200 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5ca959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 4 set_mon_vals no callback set 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5ca959a640 1 -- 172.21.15.116:0/3557836585 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c032820 con 0x7f5cac136c20 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5ca959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:52.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5ca959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5ca959a640 10 monclient: dump: 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 10 monclient: shutdown 2024-09-18T12:36:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 -- 172.21.15.116:0/3557836585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 msgr2=0x7f5cac139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/3557836585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac139000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009850 tx=0x7f5c9c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 -- 172.21.15.116:0/3557836585 shutdown_connections 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/3557836585 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac139000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/3557836585 >> v2:172.21.15.144:3300/0 conn(0x7f5cac09ea50 0x7f5cac09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 -- 172.21.15.116:0/3557836585 >> 172.21.15.116:0/3557836585 conn(0x7f5cac067110 msgr2=0x7f5cac067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 -- 172.21.15.116:0/3557836585 shutdown_connections 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.418+0000 7f5cb0f54640 1 -- 172.21.15.116:0/3557836585 wait complete. 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient: build_initial_monmap 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient: monmap: 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 1 Processor -- start 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 1 -- start start 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient: init 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:36:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac09ea50 0x7f5cac1362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient(hunting): picked mon.noname-b con 0x7f5cac09ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 0x7f5cac12f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient(hunting): picked mon.noname-a con 0x7f5cac1318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac12f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.419+0000 7f5cb0f54640 10 monclient(hunting): picked mon.noname-c con 0x7f5cac136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac09d360 con 0x7f5cac1318f0 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac1339d0 con 0x7f5cac09ea50 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac1309f0 con 0x7f5cac136c20 2024-09-18T12:36:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:52.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5cb0f54640 10 monclient(hunting): authenticate will time out at 1439.741024s 2024-09-18T12:36:52.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 0x7f5cac12f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac12f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac12f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:38736/0 (socket says 172.21.15.116:38736) 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 -- 172.21.15.116:0/1490260815 learned_addr learned my addr 172.21.15.116:0/1490260815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 10 monclient(hunting): get_auth_request con 0x7f5cac1318f0 auth_method 0 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): get_auth_request con 0x7f5cac136c20 auth_method 0 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 1 -- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 msgr2=0x7f5cac12f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 1 -- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 msgr2=0x7f5cac12f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 0x7f5cac12f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5ca9d9b640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 0x7f5cac12f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caa59c640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac09ea50 0x7f5cac1362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caa59c640 10 monclient(hunting): get_auth_request con 0x7f5cac09ea50 auth_method 0 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caa59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient(hunting): handle_auth_done global_id 144203 payload 824 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 -- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac09ea50 msgr2=0x7f5cac1362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac09ea50 0x7f5cac1362d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 -- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 msgr2=0x7f5cac12f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 0x7f5cac12f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 monclient: found mon.noname-c 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.420+0000 7f5caad9d640 1 -- 172.21.15.116:0/1490260815 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9c009390 con 0x7f5cac136c20 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5caad9d640 1 -- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 msgr2=0x7f5cac12f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5caad9d640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac12f5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009360 tx=0x7f5c9c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 1 -- 172.21.15.116:0/1490260815 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c031e90 con 0x7f5cac136c20 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: dump: 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 1 -- 172.21.15.116:0/1490260815 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c03d440 con 0x7f5cac136c20 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5cb0f54640 5 monclient: authenticate success, global_id 144203 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5cb0f54640 10 monclient: _renew_subs 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5cb0f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac133410 con 0x7f5cac136c20 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 1 -- 172.21.15.116:0/1490260815 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c03d7e0 con 0x7f5cac136c20 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca27fc640 10 monclient: dump: 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5ca8d99640 4 set_mon_vals no callback set 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5cb0f54640 10 monclient: _renew_subs 2024-09-18T12:36:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5cb0f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.422+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac1310b0 con 0x7f5cac136c20 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.423+0000 7f5ca27fc640 1 -- 172.21.15.116:0/1490260815 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5c9c039040 con 0x7f5cac136c20 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.423+0000 7f5ca27fc640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c8c038490 0x7f5c8c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.423+0000 7f5ca27fc640 1 -- 172.21.15.116:0/1490260815 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5c9c06c510 con 0x7f5cac136c20 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.423+0000 7f5cb0f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.423+0000 7f5cb0f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.423+0000 7f5cb0f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.423+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cac133650 con 0x7f5cac136c20 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.424+0000 7f5caa59c640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c8c038490 0x7f5c8c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.424+0000 7f5caa59c640 10 monclient: get_auth_request con 0x7f5c8c038490 auth_method 0 2024-09-18T12:36:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.426+0000 7f5caa59c640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c8c038490 0x7f5c8c03a940 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400e160 tx=0x7f5ca4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.428+0000 7f5ca27fc640 1 -- 172.21.15.116:0/1490260815 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5c9c037080 con 0x7f5cac136c20 2024-09-18T12:36:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.428+0000 7f5ca27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.428+0000 7f5ca27fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.639+0000 7f5cb0f54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.640+0000 7f5cb0f54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.640+0000 7f5cb0f54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.640+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5cac04bf60 con 0x7f5cac136c20 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.640+0000 7f5ca27fc640 1 -- 172.21.15.116:0/1490260815 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f5c9c042030 con 0x7f5cac136c20 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.640+0000 7f5ca27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.640+0000 7f5ca27fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:52.641 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":30,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:52.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c8c038490 msgr2=0x7f5c8c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:52.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c8c038490 0x7f5c8c03a940 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400e160 tx=0x7f5ca4009040 comp rx=0 tx=0).stop 2024-09-18T12:36:52.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 10 monclient: shutdown 2024-09-18T12:36:52.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:52.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 msgr2=0x7f5cac12f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:52.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac12f5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009360 tx=0x7f5c9c032000 comp rx=0 tx=0).stop 2024-09-18T12:36:52.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 shutdown_connections 2024-09-18T12:36:52.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c8c038490 0x7f5c8c03a940 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5cac136c20 0x7f5cac12f5c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cac1318f0 0x7f5cac12f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 --2- 172.21.15.116:0/1490260815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cac09ea50 0x7f5cac1362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:52.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 >> 172.21.15.116:0/1490260815 conn(0x7f5cac067110 msgr2=0x7f5cac12cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:52.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 shutdown_connections 2024-09-18T12:36:52.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:52.642+0000 7f5cb0f54640 1 -- 172.21.15.116:0/1490260815 wait complete. 2024-09-18T12:36:55.654 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.756+0000 7f3077300640 10 monclient: get_monmap_and_config 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.756+0000 7f3077300640 10 monclient: build_initial_monmap 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.756+0000 7f3077300640 10 monclient: monmap: 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:55.756843+0000 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:55.756843+0000 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:55.758 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.756+0000 7f3077300640 1 Processor -- start 2024-09-18T12:36:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 1 -- start start 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient: init 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3070096c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3070097030 0x7f3070097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient(hunting): picked mon.noname-a con 0x7f3070097030 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070133b40 0x7f3070135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient(hunting): picked mon.noname-b con 0x7f3070133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307009d2e0 con 0x7f3070096c60 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307009d6b0 con 0x7f3070097030 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30700992f0 con 0x7f3070133b40 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f3077300640 10 monclient(hunting): authenticate will time out at 1443.078643s 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070133b40 0x7f3070135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3070097030 0x7f3070097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070133b40 0x7f3070135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58158/0 (socket says 172.21.15.116:58158) 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 1 -- 172.21.15.116:0/1185738188 learned_addr learned my addr 172.21.15.116:0/1185738188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 10 monclient(hunting): get_auth_request con 0x7f3070097030 auth_method 0 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 10 monclient(hunting): get_auth_request con 0x7f3070133b40 auth_method 0 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307489b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.757+0000 7f307589d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 10 monclient(hunting): handle_auth_done global_id 154098 payload 274 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 1 -- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070133b40 msgr2=0x7f3070135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:55.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 1 --2- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070133b40 0x7f3070135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 1 -- 172.21.15.116:0/1185738188 >> v1:172.21.15.144:6789/0 conn(0x7f3070096c60 legacy=0x7f307009a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 10 monclient: _finish_hunting 0 2024-09-18T12:36:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 1 monclient: found mon.noname-a 2024-09-18T12:36:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307489b640 1 -- 172.21.15.116:0/1185738188 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30700995d0 con 0x7f3070097030 2024-09-18T12:36:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.758+0000 7f307589d640 1 --2- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070133b40 0x7f3070135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.760+0000 7f307489b640 1 -- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070097030 msgr2=0x7f3070097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.760+0000 7f307489b640 1 --2- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070097030 0x7f3070097430 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f306000baf0 tx=0x7f306000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da766e9940d7f573 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 1 -- 172.21.15.116:0/1185738188 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060031720 con 0x7f3070097030 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: dump: 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: _finish_auth 0 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 1 -- 172.21.15.116:0/1185738188 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306002f050 con 0x7f3070097030 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 5 monclient: authenticate success, global_id 154098 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 1 -- 172.21.15.116:0/1185738188 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060031ab0 con 0x7f3070097030 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 4 set_mon_vals no callback set 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3067fff640 10 monclient: dump: 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 10 monclient: shutdown 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 -- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070097030 msgr2=0x7f3070097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 --2- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070097030 0x7f3070097430 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f306000baf0 tx=0x7f306000b3b0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 -- 172.21.15.116:0/1185738188 shutdown_connections 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 --2- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070133b40 0x7f3070135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 --2- 172.21.15.116:0/1185738188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070097030 0x7f3070097430 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 -- 172.21.15.116:0/1185738188 >> 172.21.15.116:0/1185738188 conn(0x7f30700672b0 msgr2=0x7f30700676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 -- 172.21.15.116:0/1185738188 shutdown_connections 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.761+0000 7f3077300640 1 -- 172.21.15.116:0/1185738188 wait complete. 2024-09-18T12:36:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.762+0000 7f3077300640 10 monclient: build_initial_monmap 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.762+0000 7f3077300640 10 monclient: monmap: 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.762+0000 7f3077300640 1 Processor -- start 2024-09-18T12:36:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.762+0000 7f3077300640 1 -- start start 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.762+0000 7f3077300640 10 monclient: init 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 0x7f307006e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): picked mon.noname-c con 0x7f3070096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3070097030 0x7f307012e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): picked mon.noname-b con 0x7f3070097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070133b40 0x7f30701411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): picked mon.noname-a con 0x7f3070133b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307009add0 con 0x7f3070133b40 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307009fbe0 con 0x7f3070097030 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307011de60 con 0x7f3070096c60 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f3077300640 10 monclient(hunting): authenticate will time out at 1443.084282s 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 0x7f307006e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070133b40 0x7f30701411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 0x7f307006e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58160/0 (socket says 172.21.15.116:58160) 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 -- 172.21.15.116:0/2825828458 learned_addr learned my addr 172.21.15.116:0/2825828458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): get_auth_request con 0x7f3070096c60 auth_method 0 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 10 monclient(hunting): get_auth_request con 0x7f3070133b40 auth_method 0 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307489b640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3070097030 0x7f307012e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307489b640 10 monclient(hunting): get_auth_request con 0x7f3070097030 auth_method 0 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307489b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307489b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307489b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient(hunting): handle_auth_done global_id 144209 payload 824 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 -- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3070097030 msgr2=0x7f307012e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3070097030 0x7f307012e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 -- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070133b40 msgr2=0x7f30701411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070133b40 0x7f30701411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient: _finish_hunting 0 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 monclient: found mon.noname-c 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307509c640 1 -- 172.21.15.116:0/2825828458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068009ae0 con 0x7f3070096c60 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307489b640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3070097030 0x7f307012e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.763+0000 7f307589d640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070133b40 0x7f30701411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.765+0000 7f307509c640 1 -- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 msgr2=0x7f307006e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.765+0000 7f307509c640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 0x7f307006e5a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f306c00c5e0 tx=0x7f306c007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 1 -- 172.21.15.116:0/2825828458 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c012840 con 0x7f3070096c60 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: dump: 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: _finish_auth 0 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 1 -- 172.21.15.116:0/2825828458 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306c0129a0 con 0x7f3070096c60 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f3077300640 5 monclient: authenticate success, global_id 144209 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f3077300640 10 monclient: _renew_subs 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f3077300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306000b630 con 0x7f3070096c60 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 1 -- 172.21.15.116:0/2825828458 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c012da0 con 0x7f3070096c60 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30657fa640 10 monclient: dump: 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f30677fe640 4 set_mon_vals no callback set 2024-09-18T12:36:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f3077300640 10 monclient: _renew_subs 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f3077300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.766+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307011daf0 con 0x7f3070096c60 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f30657fa640 1 -- 172.21.15.116:0/2825828458 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f306c011590 con 0x7f3070096c60 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f3077300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f3077300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f3077300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307004bf60 con 0x7f3070096c60 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f30657fa640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3054038490 0x7f305403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f30657fa640 1 -- 172.21.15.116:0/2825828458 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f306c04c4d0 con 0x7f3070096c60 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f307489b640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3054038490 0x7f305403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.767+0000 7f307489b640 10 monclient: get_auth_request con 0x7f3054038490 auth_method 0 2024-09-18T12:36:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.770+0000 7f307489b640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3054038490 0x7f305403a940 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3060033000 tx=0x7f3060039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.771+0000 7f30657fa640 1 -- 172.21.15.116:0/2825828458 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f306c010920 con 0x7f3070096c60 2024-09-18T12:36:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.771+0000 7f30657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.771+0000 7f30657fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.986+0000 7f3077300640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.986+0000 7f3077300640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.986+0000 7f3077300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:55.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.986+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f307011d760 con 0x7f3070096c60 2024-09-18T12:36:55.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.986+0000 7f30657fa640 1 -- 172.21.15.116:0/2825828458 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f306c016080 con 0x7f3070096c60 2024-09-18T12:36:55.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.986+0000 7f30657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:55.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.986+0000 7f30657fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:55.988 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.988+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3054038490 msgr2=0x7f305403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.988+0000 7f3077300640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3054038490 0x7f305403a940 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3060033000 tx=0x7f3060039000 comp rx=0 tx=0).stop 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.988+0000 7f3077300640 10 monclient: shutdown 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 msgr2=0x7f307006e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 0x7f307006e5a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f306c00c5e0 tx=0x7f306c007ed0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 shutdown_connections 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3054038490 0x7f305403a940 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3070133b40 0x7f30701411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3070097030 0x7f307012e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 --2- 172.21.15.116:0/2825828458 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3070096c60 0x7f307006e5a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 >> 172.21.15.116:0/2825828458 conn(0x7f30700672b0 msgr2=0x7f3070132200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 shutdown_connections 2024-09-18T12:36:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:55.989+0000 7f3077300640 1 -- 172.21.15.116:0/2825828458 wait complete. 2024-09-18T12:36:56.000 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:36:56.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:36:56.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:36:56.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.092+0000 7fe3f239d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:56.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.092+0000 7fe3f239d640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.092+0000 7fe3f239d640 10 monclient: monmap: 2024-09-18T12:36:56.093 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:56.093389+0000 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:56.093389+0000 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:56.094 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:56.095 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:56.095 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 1 Processor -- start 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 1 -- start start 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 10 monclient: init 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3ec09a7f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3ec09abc0 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.093+0000 7fe3f239d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe3ec09afe0 0x7fe3ec138160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f239d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3ec09afe0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f239d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec09c800 con 0x7fe3ec09a7f0 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f239d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec0a2050 con 0x7fe3ec09abc0 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f239d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec09b440 con 0x7fe3ec09afe0 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f239d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f239d640 10 monclient(hunting): authenticate will time out at 1443.415059s 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f093a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe3ec09afe0 0x7fe3ec138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f093a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe3ec09afe0 0x7fe3ec138160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43376/0 (socket says 172.21.15.116:43376) 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f093a640 1 -- 172.21.15.116:0/2771336130 learned_addr learned my addr 172.21.15.116:0/2771336130 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f093a640 10 monclient(hunting): get_auth_request con 0x7fe3ec09afe0 auth_method 0 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f093a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.094+0000 7fe3f093a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3ebfff640 1 -- 172.21.15.116:0/2771336130 >> v1:172.21.15.116:6789/0 conn(0x7fe3ec09a7f0 legacy=0x7fe3ec09c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 10 monclient(hunting): handle_auth_done global_id 154104 payload 274 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 1 -- 172.21.15.116:0/2771336130 >> v1:172.21.15.144:6789/0 conn(0x7fe3ec09abc0 legacy=0x7fe3ec134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 1 -- 172.21.15.116:0/2771336130 >> v1:172.21.15.116:6789/0 conn(0x7fe3ec09a7f0 legacy=0x7fe3ec09c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 1 monclient: found mon.noname-a 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.096+0000 7fe3f093a640 1 -- 172.21.15.116:0/2771336130 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec09cfb0 con 0x7fe3ec09afe0 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f093a640 1 -- 172.21.15.116:0/2771336130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 msgr2=0x7fe3ec138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f093a640 1 --2- 172.21.15.116:0/2771336130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec138160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009fa0 tx=0x7fe3e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5de5e6b5cc7728 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 1 -- 172.21.15.116:0/2771336130 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4007610 con 0x7fe3ec09afe0 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: dump: 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.107 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 5 monclient: authenticate success, global_id 154104 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 1 -- 172.21.15.116:0/2771336130 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e402d050 con 0x7fe3ec09afe0 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:56.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 4 set_mon_vals no callback set 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 1 -- 172.21.15.116:0/2771336130 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40079a0 con 0x7fe3ec09afe0 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3eaffd640 10 monclient: dump: 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.109 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 10 monclient: shutdown 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 1 -- 172.21.15.116:0/2771336130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 msgr2=0x7fe3ec138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.099+0000 7fe3f239d640 1 --2- 172.21.15.116:0/2771336130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec138160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009fa0 tx=0x7fe3e4009840 comp rx=0 tx=0).stop 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 1 -- 172.21.15.116:0/2771336130 shutdown_connections 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 1 --2- 172.21.15.116:0/2771336130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec138160 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 1 -- 172.21.15.116:0/2771336130 >> 172.21.15.116:0/2771336130 conn(0x7fe3ec067b50 msgr2=0x7fe3ec0a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 1 -- 172.21.15.116:0/2771336130 shutdown_connections 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 1 -- 172.21.15.116:0/2771336130 wait complete. 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 10 monclient: monmap: 2024-09-18T12:36:56.110 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.100+0000 7fe3f239d640 1 Processor -- start 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 1 -- start start 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient: init 2024-09-18T12:36:56.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec09a7f0 0x7fe3ec140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3ec09a7f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3ec09abc0 0x7fe3ec131a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3ec09abc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec0951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3ec09afe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec132800 con 0x7fe3ec09afe0 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec12dc40 con 0x7fe3ec09a7f0 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec12df20 con 0x7fe3ec09abc0 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f239d640 10 monclient(hunting): authenticate will time out at 1443.422524s 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec0951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3ec09abc0 0x7fe3ec131a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec0951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43388/0 (socket says 172.21.15.116:43388) 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 -- 172.21.15.116:0/4049627086 learned_addr learned my addr 172.21.15.116:0/4049627086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): get_auth_request con 0x7fe3ec09afe0 auth_method 0 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 10 monclient(hunting): get_auth_request con 0x7fe3ec09abc0 auth_method 0 2024-09-18T12:36:56.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3ebfff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec09a7f0 0x7fe3ec140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3ebfff640 10 monclient(hunting): get_auth_request con 0x7fe3ec09a7f0 auth_method 0 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient(hunting): handle_auth_done global_id 154110 payload 824 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3ec09abc0 msgr2=0x7fe3ec131a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe3ec09abc0 0x7fe3ec131a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec09a7f0 msgr2=0x7fe3ec140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec09a7f0 0x7fe3ec140e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 monclient: found mon.noname-a 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.101+0000 7fe3f093a640 1 -- 172.21.15.116:0/4049627086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e4009ae0 con 0x7fe3ec09afe0 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f093a640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 msgr2=0x7fe3ec0951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f093a640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec0951c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009ab0 tx=0x7fe3e4036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4007170 con 0x7fe3ec09afe0 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: dump: 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.116 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e4032cc0 con 0x7fe3ec09afe0 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f239d640 5 monclient: authenticate success, global_id 154110 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f239d640 10 monclient: _renew_subs 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f239d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec12e200 con 0x7fe3ec09afe0 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e403b680 con 0x7fe3ec09afe0 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3d3fff640 10 monclient: dump: 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3ea7fc640 4 set_mon_vals no callback set 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f239d640 10 monclient: _renew_subs 2024-09-18T12:36:56.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f239d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.104+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec1413a0 con 0x7fe3ec09afe0 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3f239d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe3e4038050 con 0x7fe3ec09afe0 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3f239d640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 0x7fe3ec0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3f239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec141590 con 0x7fe3ec12cce0 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 0x7fe3ec0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient: get_auth_request con 0x7fe3ec12cce0 auth_method 0 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3d3fff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe3d8038490 0x7fe3d803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe3e406c750 con 0x7fe3ec09afe0 2024-09-18T12:36:56.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_done global_id 154113 payload 824 2024-09-18T12:36:56.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3ebfff640 1 -- 172.21.15.116:0/4049627086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3ec12e570 con 0x7fe3ec12cce0 2024-09-18T12:36:56.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3eb7fe640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe3d8038490 0x7fe3d803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.105+0000 7fe3eb7fe640 10 monclient: get_auth_request con 0x7fe3d8038490 auth_method 0 2024-09-18T12:36:56.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.107+0000 7fe3ebfff640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 msgr2=0x7fe3ec0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.107+0000 7fe3ebfff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 0x7fe3ec0298c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00e530 tx=0x7fe3dc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.107+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc0075c0 con 0x7fe3ec12cce0 2024-09-18T12:36:56.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.107+0000 7fe3d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.107+0000 7fe3d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.107+0000 7fe3d3fff640 10 monclient: dump: 2024-09-18T12:36:56.123 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.123 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe3dc005780 con 0x7fe3ec12cce0 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3d3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 msgr2=0x7fe3ec0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3d3fff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 0x7fe3ec0298c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00e530 tx=0x7fe3dc009040 comp rx=0 tx=0).stop 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 msgr2=0x7fe3ec0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3d3fff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec12cce0 0x7fe3ec0298c0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00e530 tx=0x7fe3dc009040 comp rx=0 tx=0).stop 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.108+0000 7fe3eb7fe640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe3d8038490 0x7fe3d803a940 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec12d550 tx=0x7fe3e0007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3f239d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3f239d640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 0x7fe3ec029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3f239d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec063720 con 0x7fe3ec131f70 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 0x7fe3ec029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 10 monclient: get_auth_request con 0x7fe3ec131f70 auth_method 0 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.113+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.114+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_done global_id 154116 payload 824 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.114+0000 7fe3ebfff640 1 -- 172.21.15.116:0/4049627086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe3ec04bf60 con 0x7fe3ec131f70 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3ebfff640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 msgr2=0x7fe3ec029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3ebfff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 0x7fe3ec029e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc017000 tx=0x7fe3dc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc020600 con 0x7fe3ec131f70 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 10 monclient: dump: 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fe3dc020760 con 0x7fe3ec131f70 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 msgr2=0x7fe3ec029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 0x7fe3ec029e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc017000 tx=0x7fe3dc01a000 comp rx=0 tx=0).stop 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 msgr2=0x7fe3ec029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3d3fff640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec131f70 0x7fe3ec029e00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 116, 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:56.128 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.129 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.130 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:56.131 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.132 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.133 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:56.134 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:56.135 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:56.136 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.138 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:56.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:56.140 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe3d8038490 msgr2=0x7fe3d803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe3d8038490 0x7fe3d803a940 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec12d550 tx=0x7fe3e0007e10 comp rx=0 tx=0).stop 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 10 monclient: shutdown 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 msgr2=0x7fe3ec0951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 1 --2- 172.21.15.116:0/4049627086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec09afe0 0x7fe3ec0951c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009ab0 tx=0x7fe3e4036000 comp rx=0 tx=0).stop 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f093a640 1 -- 172.21.15.116:0/4049627086 reap_dead start 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f093a640 1 -- 172.21.15.116:0/4049627086 reap_dead start 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 shutdown_connections 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.116+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 >> 172.21.15.116:0/4049627086 conn(0x7fe3ec067b50 msgr2=0x7fe3ec0a0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.117+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 shutdown_connections 2024-09-18T12:36:56.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.117+0000 7fe3f239d640 1 -- 172.21.15.116:0/4049627086 wait complete. 2024-09-18T12:36:56.143 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:56.143 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:36:56.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:36:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.236+0000 7f4e5999d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.236+0000 7f4e5999d640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.236+0000 7f4e5999d640 10 monclient: monmap: 2024-09-18T12:36:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:56.236849+0000 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:56.236849+0000 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.236+0000 7f4e5999d640 1 Processor -- start 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.236+0000 7f4e5999d640 1 -- start start 2024-09-18T12:36:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.236+0000 7f4e5999d640 10 monclient: init 2024-09-18T12:36:56.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e541318f0 0x7f4e54131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient(hunting): picked mon.noname-a con 0x7f4e541318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e5409ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e54136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5409f530 con 0x7f4e5409ea50 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5409fbf0 con 0x7f4e54136c20 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e54131d50 con 0x7f4e541318f0 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e5999d640 10 monclient(hunting): authenticate will time out at 1443.558312s 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e541318f0 0x7f4e54131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e541318f0 0x7f4e54131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43414/0 (socket says 172.21.15.116:43414) 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 1 -- 172.21.15.116:0/1619575502 learned_addr learned my addr 172.21.15.116:0/1619575502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 10 monclient(hunting): get_auth_request con 0x7f4e541318f0 auth_method 0 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.237+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e527fc640 1 -- 172.21.15.116:0/1619575502 >> v1:172.21.15.116:6789/0 conn(0x7f4e5409ea50 legacy=0x7f4e5409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_done global_id 154119 payload 274 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e537fe640 1 -- 172.21.15.116:0/1619575502 >> v1:172.21.15.144:6789/0 conn(0x7f4e54136c20 legacy=0x7f4e54139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 1 -- 172.21.15.116:0/1619575502 >> v1:172.21.15.144:6789/0 conn(0x7f4e54136c20 legacy=0x7f4e54139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 1 -- 172.21.15.116:0/1619575502 >> v1:172.21.15.116:6789/0 conn(0x7f4e5409ea50 legacy=0x7f4e5409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 1 monclient: found mon.noname-a 2024-09-18T12:36:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.239+0000 7f4e52ffd640 1 -- 172.21.15.116:0/1619575502 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5409fe30 con 0x7f4e541318f0 2024-09-18T12:36:56.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e52ffd640 1 -- 172.21.15.116:0/1619575502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e541318f0 msgr2=0x7f4e54131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e52ffd640 1 --2- 172.21.15.116:0/1619575502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e541318f0 0x7f4e54131f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400b0d0 tx=0x7f4e44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c742476c03b84619 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 1 -- 172.21.15.116:0/1619575502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44007610 con 0x7f4e541318f0 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: dump: 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.248 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 1 -- 172.21.15.116:0/1619575502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e44007770 con 0x7f4e541318f0 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e5999d640 5 monclient: authenticate success, global_id 154119 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e5999d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 1 -- 172.21.15.116:0/1619575502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44007b50 con 0x7f4e541318f0 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e5999d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e5999d640 4 set_mon_vals no callback set 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.242+0000 7f4e51ffb640 10 monclient: dump: 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 10 monclient: shutdown 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 1 -- 172.21.15.116:0/1619575502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e541318f0 msgr2=0x7f4e54131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 1 --2- 172.21.15.116:0/1619575502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e541318f0 0x7f4e54131f30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400b0d0 tx=0x7f4e44009840 comp rx=0 tx=0).stop 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 1 -- 172.21.15.116:0/1619575502 shutdown_connections 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 1 --2- 172.21.15.116:0/1619575502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e541318f0 0x7f4e54131f30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 1 -- 172.21.15.116:0/1619575502 >> 172.21.15.116:0/1619575502 conn(0x7f4e54067110 msgr2=0x7f4e54067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 1 -- 172.21.15.116:0/1619575502 shutdown_connections 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.243+0000 7f4e5999d640 1 -- 172.21.15.116:0/1619575502 wait complete. 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient: monmap: 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.251 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 1 Processor -- start 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 1 -- start start 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient: init 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5409ea50 0x7f4e54135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e541318f0 0x7f4e54096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient(hunting): picked mon.noname-b con 0x7f4e541318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e54136c20 0x7f4e54096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.244+0000 7f4e5999d640 10 monclient(hunting): picked mon.noname-a con 0x7f4e54136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5409bf10 con 0x7f4e54136c20 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5413a410 con 0x7f4e541318f0 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5409a900 con 0x7f4e5409ea50 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e5999d640 10 monclient(hunting): authenticate will time out at 1443.566027s 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5409ea50 0x7f4e54135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e54136c20 0x7f4e54096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e54136c20 0x7f4e54096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43428/0 (socket says 172.21.15.116:43428) 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 -- 172.21.15.116:0/3069652151 learned_addr learned my addr 172.21.15.116:0/3069652151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): get_auth_request con 0x7f4e54136c20 auth_method 0 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5409ea50 0x7f4e54135ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58190/0 (socket says 172.21.15.116:58190) 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 10 monclient(hunting): get_auth_request con 0x7f4e5409ea50 auth_method 0 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e527fc640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e541318f0 0x7f4e54096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e527fc640 10 monclient(hunting): get_auth_request con 0x7f4e541318f0 auth_method 0 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e527fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient(hunting): handle_auth_done global_id 154122 payload 824 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5409ea50 msgr2=0x7f4e54135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5409ea50 0x7f4e54135ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e541318f0 msgr2=0x7f4e54096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e541318f0 0x7f4e54096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 monclient: found mon.noname-a 2024-09-18T12:36:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e537fe640 1 -- 172.21.15.116:0/3069652151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e44009ae0 con 0x7f4e54136c20 2024-09-18T12:36:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.245+0000 7f4e52ffd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e5409ea50 0x7f4e54135ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e537fe640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e54136c20 msgr2=0x7f4e54096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e537fe640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e54136c20 0x7f4e54096fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00c560 tx=0x7f4e4c007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c0074f0 con 0x7f4e54136c20 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: dump: 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.261 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e5999d640 5 monclient: authenticate success, global_id 154122 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c005780 con 0x7f4e54136c20 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e5999d640 10 monclient: _renew_subs 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c005b40 con 0x7f4e54136c20 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e5999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e540996e0 con 0x7f4e54136c20 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e4affd640 10 monclient: dump: 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e517fa640 4 set_mon_vals no callback set 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e5999d640 10 monclient: _renew_subs 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e5999d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.247+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e540987f0 con 0x7f4e54136c20 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4e4c01b3b0 con 0x7f4e54136c20 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e5999d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e5999d640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 0x7f4e541327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e5999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54065e00 con 0x7f4e540974e0 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 0x7f4e541327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient: get_auth_request con 0x7f4e540974e0 auth_method 0 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e4affd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4e34038490 0x7f4e3403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f4e4c04b7b0 con 0x7f4e54136c20 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_done global_id 144218 payload 824 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.248+0000 7f4e52ffd640 1 -- 172.21.15.116:0/3069652151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e54099b80 con 0x7f4e540974e0 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.249+0000 7f4e527fc640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4e34038490 0x7f4e3403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.249+0000 7f4e527fc640 10 monclient: get_auth_request con 0x7f4e34038490 auth_method 0 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e52ffd640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 msgr2=0x7f4e541327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e52ffd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 0x7f4e541327d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e5406f600 tx=0x7f4e44037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44007610 con 0x7f4e540974e0 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 10 monclient: dump: 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4e44041070 con 0x7f4e540974e0 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 msgr2=0x7f4e541327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 0x7f4e541327d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e5406f600 tx=0x7f4e44037000 comp rx=0 tx=0).stop 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 msgr2=0x7f4e541327d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e4affd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e540974e0 0x7f4e541327d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.251+0000 7f4e527fc640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4e34038490 0x7f4e3403a940 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c001a30 tx=0x7f4e3c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e5999d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e5999d640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 0x7f4e54133110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e5999d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540978b0 con 0x7f4e54132d10 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 0x7f4e54133110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient: get_auth_request con 0x7f4e54132d10 auth_method 0 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 10 monclient(hunting): handle_auth_done global_id 144221 payload 824 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.257+0000 7f4e52ffd640 1 -- 172.21.15.116:0/3069652151 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4e54098a10 con 0x7f4e54132d10 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e52ffd640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 msgr2=0x7f4e54133110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e52ffd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 0x7f4e54133110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e54099950 tx=0x7f4e4403f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4404b070 con 0x7f4e54132d10 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 10 monclient: dump: 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.269 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f4e4403ddf0 con 0x7f4e54132d10 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 msgr2=0x7f4e54133110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 0x7f4e54133110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e54099950 tx=0x7f4e4403f070 comp rx=0 tx=0).stop 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 msgr2=0x7f4e54133110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.259+0000 7f4e4affd640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4e54132d10 0x7f4e54133110 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e54099950 tx=0x7f4e4403f070 comp rx=0 tx=0).stop 2024-09-18T12:36:56.270 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 116, 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:56.271 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.272 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:56.274 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.275 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:56.277 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:56.280 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.282 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.283 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.284 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:56.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.284 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.284 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:56.284 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4e34038490 msgr2=0x7f4e3403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4e34038490 0x7f4e3403a940 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c001a30 tx=0x7f4e3c009040 comp rx=0 tx=0).stop 2024-09-18T12:36:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 10 monclient: shutdown 2024-09-18T12:36:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e54136c20 msgr2=0x7f4e54096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 --2- 172.21.15.116:0/3069652151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e54136c20 0x7f4e54096fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00c560 tx=0x7f4e4c007cf0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e537fe640 1 -- 172.21.15.116:0/3069652151 reap_dead start 2024-09-18T12:36:56.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e537fe640 1 -- 172.21.15.116:0/3069652151 reap_dead start 2024-09-18T12:36:56.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 shutdown_connections 2024-09-18T12:36:56.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 >> 172.21.15.116:0/3069652151 conn(0x7f4e54067110 msgr2=0x7f4e540a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 shutdown_connections 2024-09-18T12:36:56.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.260+0000 7f4e5999d640 1 -- 172.21.15.116:0/3069652151 wait complete. 2024-09-18T12:36:56.286 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:56.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:36:56.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:36:56.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.379+0000 7fc4b3b09640 10 monclient: get_monmap_and_config 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.379+0000 7fc4b3b09640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.379+0000 7fc4b3b09640 10 monclient: monmap: 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:56.379767+0000 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:56.379767+0000 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:56.380 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:56.381 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:56.381 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:56.381 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:56.381 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.379+0000 7fc4b3b09640 1 Processor -- start 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.379+0000 7fc4b3b09640 1 -- start start 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.379+0000 7fc4b3b09640 10 monclient: init 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc4ac131ab0 0x7fc4ac131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ac131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4ac09e910 0x7fc4ac09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ac09e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:56.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4ac09f250 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac09faf0 con 0x7fc4ac09f250 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac09fd30 con 0x7fc4ac131ab0 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac0a1110 con 0x7fc4ac09e910 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b3b09640 10 monclient(hunting): authenticate will time out at 1443.701385s 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc4ac131ab0 0x7fc4ac131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc4ac131ab0 0x7fc4ac131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43430/0 (socket says 172.21.15.116:43430) 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 -- 172.21.15.116:0/4274894368 learned_addr learned my addr 172.21.15.116:0/4274894368 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): get_auth_request con 0x7fc4ac131ab0 auth_method 0 2024-09-18T12:36:56.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b10a4640 1 --2- 172.21.15.116:0/4274894368 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac09e910 0x7fc4ac09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b10a4640 10 monclient(hunting): get_auth_request con 0x7fc4ac09e910 auth_method 0 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b10a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b10a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b10a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient(hunting): handle_auth_done global_id 154125 payload 274 2024-09-18T12:36:56.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 -- 172.21.15.116:0/4274894368 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac09e910 msgr2=0x7fc4ac09ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:56.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 --2- 172.21.15.116:0/4274894368 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac09e910 0x7fc4ac09ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 -- 172.21.15.116:0/4274894368 >> v1:172.21.15.144:6789/0 conn(0x7fc4ac09f250 legacy=0x7fc4ac068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:56.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 monclient: found mon.noname-a 2024-09-18T12:36:56.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.380+0000 7fc4b18a5640 1 -- 172.21.15.116:0/4274894368 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0a13f0 con 0x7fc4ac131ab0 2024-09-18T12:36:56.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b18a5640 1 -- 172.21.15.116:0/4274894368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac131ab0 msgr2=0x7fc4ac131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b18a5640 1 --2- 172.21.15.116:0/4274894368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac131ab0 0x7fc4ac131e80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac09bdb0 tx=0x7fc49c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3af741e194fae2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 1 -- 172.21.15.116:0/4274894368 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c007610 con 0x7fc4ac131ab0 2024-09-18T12:36:56.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:56.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: dump: 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b3b09640 5 monclient: authenticate success, global_id 154125 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 1 -- 172.21.15.116:0/4274894368 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c02d050 con 0x7fc4ac131ab0 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b3b09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b3b09640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b3b09640 4 set_mon_vals no callback set 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 1 -- 172.21.15.116:0/4274894368 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c0079e0 con 0x7fc4ac131ab0 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.383+0000 7fc4b08a3640 10 monclient: dump: 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.395 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 10 monclient: shutdown 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 -- 172.21.15.116:0/4274894368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac131ab0 msgr2=0x7fc4ac131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/4274894368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac131ab0 0x7fc4ac131e80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac09bdb0 tx=0x7fc49c009840 comp rx=0 tx=0).stop 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 -- 172.21.15.116:0/4274894368 shutdown_connections 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/4274894368 >> v2:172.21.15.144:3300/0 conn(0x7fc4ac09e910 0x7fc4ac09ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/4274894368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac131ab0 0x7fc4ac131e80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 -- 172.21.15.116:0/4274894368 >> 172.21.15.116:0/4274894368 conn(0x7fc4ac067d30 msgr2=0x7fc4ac0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 -- 172.21.15.116:0/4274894368 shutdown_connections 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 1 -- 172.21.15.116:0/4274894368 wait complete. 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.384+0000 7fc4b3b09640 10 monclient: monmap: 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:56.397 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 Processor -- start 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 -- start start 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient: init 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4ac09e910 0x7fc4ac1416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ac09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 0x7fc4ac141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ac09f250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac131ab0 0x7fc4ac142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): picked mon.noname-b con 0x7fc4ac131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac11f880 con 0x7fc4ac09f250 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac09be10 con 0x7fc4ac131ab0 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac1219d0 con 0x7fc4ac09e910 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b3b09640 10 monclient(hunting): authenticate will time out at 1443.706578s 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 0x7fc4ac141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4ac09e910 0x7fc4ac1416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 0x7fc4ac141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43444/0 (socket says 172.21.15.116:43444) 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 -- 172.21.15.116:0/3777196911 learned_addr learned my addr 172.21.15.116:0/3777196911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): get_auth_request con 0x7fc4ac09f250 auth_method 0 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 10 monclient(hunting): get_auth_request con 0x7fc4ac09e910 auth_method 0 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient(hunting): handle_auth_done global_id 154128 payload 824 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4ac09e910 msgr2=0x7fc4ac1416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4ac09e910 0x7fc4ac1416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac131ab0 msgr2=0x7fc4ac142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b20a6640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac131ab0 0x7fc4ac142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac131ab0 0x7fc4ac142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 monclient: found mon.noname-a 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b10a4640 1 -- 172.21.15.116:0/3777196911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c009ae0 con 0x7fc4ac09f250 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.385+0000 7fc4b18a5640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc4ac09e910 0x7fc4ac1416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b10a4640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 msgr2=0x7fc4ac141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b10a4640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 0x7fc4ac141c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000d8c0 tx=0x7fc4a0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a000de50 con 0x7fc4ac09f250 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: dump: 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.402 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b3b09640 5 monclient: authenticate success, global_id 154128 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a000bae0 con 0x7fc4ac09f250 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b3b09640 10 monclient: _renew_subs 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b3b09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac120d00 con 0x7fc4ac09f250 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a000be80 con 0x7fc4ac09f250 2024-09-18T12:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4a9ffb640 10 monclient: dump: 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b3b09640 10 monclient: _renew_subs 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b3b09640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac11ec80 con 0x7fc4ac09f250 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.388+0000 7fc4abfff640 4 set_mon_vals no callback set 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc4a0014640 con 0x7fc4ac09f250 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4a9ffb640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc4980384e0 0x7fc49803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc4a004c8b0 con 0x7fc4ac09f250 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b3b09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 0x7fc4ac099e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b3b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac063740 con 0x7fc4ac099a80 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b18a5640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc4980384e0 0x7fc49803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b18a5640 10 monclient: get_auth_request con 0x7fc4980384e0 auth_method 0 2024-09-18T12:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 0x7fc4ac099e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 10 monclient: get_auth_request con 0x7fc4ac099a80 auth_method 0 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.389+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.390+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_done global_id 124489 payload 824 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.390+0000 7fc4b20a6640 1 -- 172.21.15.116:0/3777196911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4ac04bf60 con 0x7fc4ac099a80 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4b18a5640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc4980384e0 0x7fc49803a990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc49c033000 tx=0x7fc49c036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4b20a6640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 msgr2=0x7fc4ac099e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4b20a6640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 0x7fc4ac099e50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400d380 tx=0x7fc4a4009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4012810 con 0x7fc4ac099a80 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 10 monclient: dump: 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.407 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc4a4012970 con 0x7fc4ac099a80 2024-09-18T12:36:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 msgr2=0x7fc4ac099e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 0x7fc4ac099e50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400d380 tx=0x7fc4a4009040 comp rx=0 tx=0).stop 2024-09-18T12:36:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 msgr2=0x7fc4ac099e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:56.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.392+0000 7fc4a9ffb640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac099a80 0x7fc4ac099e50 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400d380 tx=0x7fc4a4009040 comp rx=0 tx=0).stop 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b3b09640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 0x7fc4ac11d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b3b09640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac120980 con 0x7fc4ac09a390 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 0x7fc4ac11d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 10 monclient: get_auth_request con 0x7fc4ac09a390 auth_method 0 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.398+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.399+0000 7fc4b20a6640 10 monclient(hunting): handle_auth_done global_id 124492 payload 824 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.399+0000 7fc4b20a6640 1 -- 172.21.15.116:0/3777196911 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc4ac0a2980 con 0x7fc4ac09a390 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.401+0000 7fc4b20a6640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 msgr2=0x7fc4ac11d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.401+0000 7fc4b20a6640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 0x7fc4ac11d0f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4017000 tx=0x7fc4a401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.401+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a401e900 con 0x7fc4ac09a390 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.401+0000 7fc4a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.401+0000 7fc4a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.401+0000 7fc4a9ffb640 10 monclient: dump: 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fc4a4029470 con 0x7fc4ac09a390 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4a9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4a9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 msgr2=0x7fc4ac11d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4a9ffb640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 0x7fc4ac11d0f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4017000 tx=0x7fc4a401a000 comp rx=0 tx=0).stop 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4a9ffb640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 msgr2=0x7fc4ac11d0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4a9ffb640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4ac09a390 0x7fc4ac11d0f0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4017000 tx=0x7fc4a401a000 comp rx=0 tx=0).stop 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 116, 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:56.413 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.414 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.415 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:56.416 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:56.417 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.418 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.419 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.420 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:56.421 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:56.423 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.424 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.425 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:56.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:56.427 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:56.427 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:56.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc4980384e0 msgr2=0x7fc49803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc4980384e0 0x7fc49803a990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc49c033000 tx=0x7fc49c036000 comp rx=0 tx=0).stop 2024-09-18T12:36:56.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 10 monclient: shutdown 2024-09-18T12:36:56.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 msgr2=0x7fc4ac141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b20a6640 1 -- 172.21.15.116:0/3777196911 reap_dead start 2024-09-18T12:36:56.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 0x7fc4ac141c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000d8c0 tx=0x7fc4a0008040 comp rx=0 tx=0).stop 2024-09-18T12:36:56.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 shutdown_connections 2024-09-18T12:36:56.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 --2- 172.21.15.116:0/3777196911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4ac09f250 0x7fc4ac141c00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 >> 172.21.15.116:0/3777196911 conn(0x7fc4ac067d30 msgr2=0x7fc4ac09c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 shutdown_connections 2024-09-18T12:36:56.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.402+0000 7fc4b3b09640 1 -- 172.21.15.116:0/3777196911 wait complete. 2024-09-18T12:36:56.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:56.429 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:36:56.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.522+0000 7f065e6e8640 10 monclient: get_monmap_and_config 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.522+0000 7f065e6e8640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.522+0000 7f065e6e8640 10 monclient: monmap: 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:56.522829+0000 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:56.522829+0000 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:56.524 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:56.524 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:56.524 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:56.524 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:56.524 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.522+0000 7f065e6e8640 1 Processor -- start 2024-09-18T12:36:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.522+0000 7f065e6e8640 1 -- start start 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.522+0000 7f065e6e8640 10 monclient: init 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0658131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f065809e910 0x7f065809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient(hunting): picked mon.noname-c con 0x7f065809e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f065809f250 0x7f0658138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient(hunting): picked mon.noname-a con 0x7f065809f250 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065809faf0 con 0x7f0658131ab0 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0658132590 con 0x7f065809f250 2024-09-18T12:36:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065809fd30 con 0x7f065809e910 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065e6e8640 10 monclient(hunting): authenticate will time out at 1443.844393s 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f065809f250 0x7f0658138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f065809f250 0x7f0658138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43458/0 (socket says 172.21.15.116:43458) 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 -- 172.21.15.116:0/526594871 learned_addr learned my addr 172.21.15.116:0/526594871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): get_auth_request con 0x7f065809f250 auth_method 0 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f06577fe640 1 --2- 172.21.15.116:0/526594871 >> v2:172.21.15.144:3300/0 conn(0x7f065809e910 0x7f065809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient(hunting): handle_auth_done global_id 154131 payload 274 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 -- 172.21.15.116:0/526594871 >> v2:172.21.15.144:3300/0 conn(0x7f065809e910 msgr2=0x7f065809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 --2- 172.21.15.116:0/526594871 >> v2:172.21.15.144:3300/0 conn(0x7f065809e910 0x7f065809ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 -- 172.21.15.116:0/526594871 >> v1:172.21.15.144:6789/0 conn(0x7f0658131ab0 legacy=0x7f0658131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 monclient: found mon.noname-a 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.523+0000 7f065cc85640 1 -- 172.21.15.116:0/526594871 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06580a1110 con 0x7f065809f250 2024-09-18T12:36:56.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f065cc85640 1 -- 172.21.15.116:0/526594871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f065809f250 msgr2=0x7f0658138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f065cc85640 1 --2- 172.21.15.116:0/526594871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f065809f250 0x7f0658138bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f065809bdb0 tx=0x7f0650009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ea1c31f61d8cf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 1 -- 172.21.15.116:0/526594871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650007610 con 0x7f065809f250 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: dump: 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f065e6e8640 5 monclient: authenticate success, global_id 154131 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 1 -- 172.21.15.116:0/526594871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065002d050 con 0x7f065809f250 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f065e6e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f065e6e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f065e6e8640 4 set_mon_vals no callback set 2024-09-18T12:36:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 1 -- 172.21.15.116:0/526594871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06500079a0 con 0x7f065809f250 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.526+0000 7f0656ffd640 10 monclient: dump: 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 10 monclient: shutdown 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 -- 172.21.15.116:0/526594871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f065809f250 msgr2=0x7f0658138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 --2- 172.21.15.116:0/526594871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f065809f250 0x7f0658138bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f065809bdb0 tx=0x7f0650009840 comp rx=0 tx=0).stop 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 -- 172.21.15.116:0/526594871 shutdown_connections 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 --2- 172.21.15.116:0/526594871 >> v2:172.21.15.144:3300/0 conn(0x7f065809e910 0x7f065809ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 --2- 172.21.15.116:0/526594871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f065809f250 0x7f0658138bc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 -- 172.21.15.116:0/526594871 >> 172.21.15.116:0/526594871 conn(0x7f0658067d30 msgr2=0x7f06580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 -- 172.21.15.116:0/526594871 shutdown_connections 2024-09-18T12:36:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 1 -- 172.21.15.116:0/526594871 wait complete. 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 10 monclient: build_initial_monmap 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.527+0000 7f065e6e8640 10 monclient: monmap: 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 Processor -- start 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 -- start start 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient: init 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 0x7f065809a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): picked mon.noname-c con 0x7f065809e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065809f250 0x7f0658096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): picked mon.noname-b con 0x7f065809f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0658131ab0 0x7f0658096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): picked mon.noname-a con 0x7f0658131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0658138ef0 con 0x7f0658131ab0 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065809be10 con 0x7f065809f250 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065809a200 con 0x7f065809e910 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.528+0000 7f065e6e8640 10 monclient(hunting): authenticate will time out at 1443.849946s 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 0x7f065809a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0658131ab0 0x7f0658096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 0x7f065809a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58216/0 (socket says 172.21.15.116:58216) 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 -- 172.21.15.116:0/2889090866 learned_addr learned my addr 172.21.15.116:0/2889090866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): get_auth_request con 0x7f065809e910 auth_method 0 2024-09-18T12:36:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 10 monclient(hunting): get_auth_request con 0x7f0658131ab0 auth_method 0 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f065cc85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f06577fe640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065809f250 0x7f0658096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f06577fe640 10 monclient(hunting): get_auth_request con 0x7f065809f250 auth_method 0 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f06577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f06577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f06577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient(hunting): handle_auth_done global_id 144227 payload 824 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 -- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065809f250 msgr2=0x7f0658096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065809f250 0x7f0658096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 -- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0658131ab0 msgr2=0x7f0658096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0658131ab0 0x7f0658096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 monclient: found mon.noname-c 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.529+0000 7f0657fff640 1 -- 172.21.15.116:0/2889090866 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650009ae0 con 0x7f065809e910 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f0657fff640 1 -- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 msgr2=0x7f065809a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f0657fff640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 0x7f065809a680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f064800c5e0 tx=0x7f0648007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 1 -- 172.21.15.116:0/2889090866 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480117a0 con 0x7f065809e910 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: dump: 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: _finish_auth 0 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f065e6e8640 5 monclient: authenticate success, global_id 144227 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 1 -- 172.21.15.116:0/2889090866 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648011920 con 0x7f065809e910 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f065e6e8640 10 monclient: _renew_subs 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f065e6e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065809b910 con 0x7f065809e910 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 1 -- 172.21.15.116:0/2889090866 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648011ce0 con 0x7f065809e910 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f063ffff640 10 monclient: dump: 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f065e6e8640 10 monclient: _renew_subs 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f065e6e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06580995c0 con 0x7f065809e910 2024-09-18T12:36:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.531+0000 7f06567fc640 4 set_mon_vals no callback set 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f063ffff640 1 -- 172.21.15.116:0/2889090866 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0648012630 con 0x7f065809e910 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f063ffff640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0644038490 0x7f064403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f065e6e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f065e6e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f065e6e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f063ffff640 1 -- 172.21.15.116:0/2889090866 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f064804c100 con 0x7f065809e910 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065804bf60 con 0x7f065809e910 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.532+0000 7f06577fe640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0644038490 0x7f064403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.533+0000 7f06577fe640 10 monclient: get_auth_request con 0x7f0644038490 auth_method 0 2024-09-18T12:36:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.535+0000 7f06577fe640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0644038490 0x7f064403a940 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0658131590 tx=0x7f064c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.537+0000 7f063ffff640 1 -- 172.21.15.116:0/2889090866 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f064801c460 con 0x7f065809e910 2024-09-18T12:36:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.537+0000 7f063ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.537+0000 7f063ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.752+0000 7f065e6e8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:36:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.752+0000 7f065e6e8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:36:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.752+0000 7f065e6e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.752+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0658066280 con 0x7f065809e910 2024-09-18T12:36:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.753+0000 7f063ffff640 1 -- 172.21.15.116:0/2889090866 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f064802a080 con 0x7f065809e910 2024-09-18T12:36:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.753+0000 7f063ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:36:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.753+0000 7f063ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:56.754 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.754+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.755 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.754+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=2855652215,config=3140514009,health=1675380437,kv=2187923472,logm=1137375023}) 2024-09-18T12:36:56.755 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.754+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=3326240825,config=3140514009,health=4198586779,kv=3604100458,logm=3198574609}) 2024-09-18T12:36:56.755 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.754+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.755 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.754+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=9,logm=54} crc {auth=2855652215,config=3140514009,health=1675380437,kv=2187923472,logm=1137375023}) 2024-09-18T12:36:56.755 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.754+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=7,logm=53} crc {auth=3326240825,config=3140514009,health=1675380437,kv=3288607513,logm=4009971306}) 2024-09-18T12:36:56.755 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.756 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550895639}) 2024-09-18T12:36:56.756 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044309487}) 2024-09-18T12:36:56.756 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.756 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550895639}) 2024-09-18T12:36:56.756 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112127525}) 2024-09-18T12:36:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0644038490 msgr2=0x7f064403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0644038490 0x7f064403a940 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0658131590 tx=0x7f064c008040 comp rx=0 tx=0).stop 2024-09-18T12:36:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 10 monclient: shutdown 2024-09-18T12:36:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 msgr2=0x7f065809a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 0x7f065809a680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f064800c5e0 tx=0x7f0648007ed0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 shutdown_connections 2024-09-18T12:36:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0644038490 0x7f064403a940 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0658131ab0 0x7f0658096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f065809f250 0x7f0658096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 --2- 172.21.15.116:0/2889090866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f065809e910 0x7f065809a680 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 >> 172.21.15.116:0/2889090866 conn(0x7f0658067d30 msgr2=0x7f065812caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:56.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 shutdown_connections 2024-09-18T12:36:56.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:56.755+0000 7f065e6e8640 1 -- 172.21.15.116:0/2889090866 wait complete. 2024-09-18T12:36:56.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.756+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.765 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.756+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510228549}) 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.756+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139467930}) 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.756+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.756+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510228549}) 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.756+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761722496}) 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.757+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.757+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246582693}) 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.757+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729078452}) 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.757+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.757+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246582693}) 2024-09-18T12:36:56.766 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.757+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098568008}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.758+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.758+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609812692}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.758+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584823522}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.758+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.758+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609812692}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.758+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063778292}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182965269}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855385931}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182965269}) 2024-09-18T12:36:56.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084360758}) 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280772767}) 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.759+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1069652013}) 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280772767}) 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174629320}) 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496121636}) 2024-09-18T12:36:56.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2308031891}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496121636}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.760+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=349738882}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.761+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.761+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839602510}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.761+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642148886}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.761+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.761+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839602510}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.761+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215745532}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059912271}) 2024-09-18T12:36:56.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484515098}) 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059912271}) 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1043680906}) 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157194945}) 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649043416}) 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157194945}) 2024-09-18T12:36:56.770 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.762+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1000190108}) 2024-09-18T12:36:56.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:36:56.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.769+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.769+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373017217}) 2024-09-18T12:36:56.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.769+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262671212}) 2024-09-18T12:36:56.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.769+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.769+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373017217}) 2024-09-18T12:36:56.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.769+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=65303072}) 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.770+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.770+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841908768}) 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.770+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1913755170}) 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.770+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.770+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841908768}) 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.770+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848067712}) 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.771+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.771+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234813516}) 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.771+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105472902}) 2024-09-18T12:36:56.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.771+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.771+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234813516}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.771+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=458379089}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412287642}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076632510}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412287642}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805360585}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972181614}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2660396035}) 2024-09-18T12:36:56.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972181614}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.772+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469681693}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438122331}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1115770054}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438122331}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3383681762}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287338596}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3701792986}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287338596}) 2024-09-18T12:36:56.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.773+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=559676809}) 2024-09-18T12:36:56.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.774+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.774+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434927851}) 2024-09-18T12:36:56.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.774+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586970952}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.774+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.774+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434927851}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.774+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808370745}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537920227}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71265129}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537920227}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3887678921}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594882737}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105149111}) 2024-09-18T12:36:56.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594882737}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.775+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137011620}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.776+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.776+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005688646}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.776+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358351781}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.776+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.776+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005688646}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.776+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109168646}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1779998007}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866479516}) 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.786 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1779998007}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265149485}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293858639}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418016671}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293858639}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.777+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347996722}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006665618}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002704390}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006665618}) 2024-09-18T12:36:56.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088430244}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928072252}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.778+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3393887749}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928072252}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421226119}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645921276}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451729255}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645921276}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.779+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3535715939}) 2024-09-18T12:36:56.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106800232}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561295032}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106800232}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=614610241}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385030734}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2276501790}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385030734}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.780+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2758805877}) 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.781+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.789 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.781+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022195056}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.781+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659327624}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.781+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.781+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022195056}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.781+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2207498574}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803321605}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1089884980}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803321605}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567094963}) 2024-09-18T12:36:56.790 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308689369}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758542458}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308689369}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.782+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627828809}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.783+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.783+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737763398}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.783+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3589674142}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.783+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.783+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737763398}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.783+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74229752}) 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453151610}) 2024-09-18T12:36:56.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787027926}) 2024-09-18T12:36:56.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453151610}) 2024-09-18T12:36:56.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1669340097}) 2024-09-18T12:36:56.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614387942}) 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3840989687}) 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614387942}) 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.784+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065649319}) 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821604342}) 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2931538086}) 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821604342}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2231909041}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677519328}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995716255}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677519328}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.785+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=271962792}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.786+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.786+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873078260}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.786+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3846951930}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.786+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.786+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873078260}) 2024-09-18T12:36:56.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.786+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2068626516}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882610087}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1917830958}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882610087}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1338143802}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524889766}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823636014}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524889766}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.787+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089832965}) 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73291556}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645089691}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73291556}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586836606}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441396215}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4005388341}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441396215}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579801959}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206654480}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3725369928}) 2024-09-18T12:36:56.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206654480}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627410126}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.790+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.790+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008850365}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.790+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266006906}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.790+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.790+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008850365}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.790+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107904818}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291688823}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=787050710}) 2024-09-18T12:36:56.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291688823}) 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=823037098}) 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428004029}) 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449977112}) 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428004029}) 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3730555760}) 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469352119}) 2024-09-18T12:36:56.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=205860403}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469352119}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141982455}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140516095}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=356582041}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140516095}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586590477}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770527578}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936573790}) 2024-09-18T12:36:56.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770527578}) 2024-09-18T12:36:56.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230535995}) 2024-09-18T12:36:56.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153177833}) 2024-09-18T12:36:56.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=67849571}) 2024-09-18T12:36:56.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153177833}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=388689149}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639166011}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3309153095}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639166011}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=951061201}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147199219}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521542546}) 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147199219}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348663816}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956398356}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748201305}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956398356}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372682990}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015193285}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1438346270}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015193285}) 2024-09-18T12:36:56.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328957358}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235614522}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536403626}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235614522}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3344220452}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498404779}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728801780}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498404779}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186753780}) 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.815 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445300246}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3028150985}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445300246}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320338458}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270293026}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135584652}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270293026}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910113878}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=365900216,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2661843067,mon_config_key=3655569570,monmap=148614106,osd_metadata=3813717626}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=2} crc {logm=3072453863,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3920340668,mon_config_key=3655569570,monmap=4208436772,osd_metadata=626683131}) 2024-09-18T12:36:56.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=365900216,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2661843067,mon_config_key=3655569570,monmap=148614106,osd_metadata=3813717626}) 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=4} crc {logm=1223398631,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2439998008,mon_config_key=3655569570,monmap=148614106,osd_metadata=3874183036}) 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_snap=2,osdmap=93} crc {osd_metadata=3127564625,osd_snap=2033388227,osdmap=1398397594}) 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=92} crc {osd_metadata=3127564625,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2999278873}) 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_snap=2,osdmap=93} crc {osd_metadata=3127564625,osd_snap=2033388227,osdmap=1398397594}) 2024-09-18T12:36:56.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=92} crc {osd_metadata=3127564625,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1266791023}) 2024-09-18T12:36:56.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=2208468814}) 2024-09-18T12:36:56.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=3141865673}) 2024-09-18T12:36:56.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:36:56.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=2208468814}) 2024-09-18T12:36:56.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:36:56.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=2118423159}) 2024-09-18T12:36:57.772 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:36:57.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.871+0000 7fa852d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.871+0000 7fa852d9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.871+0000 7fa852d9d640 10 monclient: monmap: 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:57.871869+0000 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:57.871869+0000 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:57.873 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.871+0000 7fa852d9d640 1 Processor -- start 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.871+0000 7fa852d9d640 1 -- start start 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.871+0000 7fa852d9d640 10 monclient: init 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa84c096e10 0x7fa84c09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa84c096e10 addr v2:172.21.15.116:3300/0 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa84c09aaa0 0x7fa84c0971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa84c09aaa0 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa84c097720 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c099620 con 0x7fa84c097720 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c099340 con 0x7fa84c096e10 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c099900 con 0x7fa84c09aaa0 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa852d9d640 10 monclient(hunting): authenticate will time out at 1445.193469s 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa850b39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa84c096e10 0x7fa84c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa850b39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa84c096e10 0x7fa84c09a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43466/0 (socket says 172.21.15.116:43466) 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa850b39640 1 -- 172.21.15.116:0/3060109275 learned_addr learned my addr 172.21.15.116:0/3060109275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa850b39640 10 monclient(hunting): get_auth_request con 0x7fa84c096e10 auth_method 0 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa850b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa850b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa850b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa84bfff640 1 --2- 172.21.15.116:0/3060109275 >> v2:172.21.15.144:3300/0 conn(0x7fa84c09aaa0 0x7fa84c0971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa84bfff640 10 monclient(hunting): get_auth_request con 0x7fa84c09aaa0 auth_method 0 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa84bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa84bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.872+0000 7fa84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 10 monclient(hunting): handle_auth_done global_id 124498 payload 274 2024-09-18T12:36:57.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 1 -- 172.21.15.116:0/3060109275 >> v2:172.21.15.116:3300/0 conn(0x7fa84c096e10 msgr2=0x7fa84c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 1 --2- 172.21.15.116:0/3060109275 >> v2:172.21.15.116:3300/0 conn(0x7fa84c096e10 0x7fa84c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 1 -- 172.21.15.116:0/3060109275 >> v1:172.21.15.116:6789/0 conn(0x7fa84c097720 legacy=0x7fa84c12e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 1 monclient: found mon.noname-c 2024-09-18T12:36:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:36:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.873+0000 7fa84bfff640 1 -- 172.21.15.116:0/3060109275 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c097b20 con 0x7fa84c09aaa0 2024-09-18T12:36:57.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84bfff640 1 -- 172.21.15.116:0/3060109275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 msgr2=0x7fa84c0971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84bfff640 1 --2- 172.21.15.116:0/3060109275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c0971e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa84c09aff0 tx=0x7fa8380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed04e78a05ea5126 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 1 -- 172.21.15.116:0/3060109275 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83802adf0 con 0x7fa84c09aaa0 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: dump: 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:57.886 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 1 -- 172.21.15.116:0/3060109275 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838032440 con 0x7fa84c09aaa0 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa852d9d640 5 monclient: authenticate success, global_id 124498 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa852d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa852d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 1 -- 172.21.15.116:0/3060109275 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838032820 con 0x7fa84c09aaa0 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa852d9d640 4 set_mon_vals no callback set 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.876+0000 7fa84b7fe640 10 monclient: dump: 2024-09-18T12:36:57.887 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 10 monclient: shutdown 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:57.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 -- 172.21.15.116:0/3060109275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 msgr2=0x7fa84c0971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 --2- 172.21.15.116:0/3060109275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c0971e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa84c09aff0 tx=0x7fa8380090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 -- 172.21.15.116:0/3060109275 shutdown_connections 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 --2- 172.21.15.116:0/3060109275 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c0971e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 --2- 172.21.15.116:0/3060109275 >> v2:172.21.15.116:3300/0 conn(0x7fa84c096e10 0x7fa84c09a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 -- 172.21.15.116:0/3060109275 >> 172.21.15.116:0/3060109275 conn(0x7fa84c0672b0 msgr2=0x7fa84c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 -- 172.21.15.116:0/3060109275 shutdown_connections 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 1 -- 172.21.15.116:0/3060109275 wait complete. 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 10 monclient: build_initial_monmap 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.877+0000 7fa852d9d640 10 monclient: monmap: 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:57.889 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 Processor -- start 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 -- start start 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient: init 2024-09-18T12:36:57.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84c096e10 0x7fa84c1363c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa84c096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 0x7fa84c136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa84c097720 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c131f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa84c09aaa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c135ab0 con 0x7fa84c097720 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c134500 con 0x7fa84c09aaa0 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:57.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c09b050 con 0x7fa84c096e10 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa852d9d640 10 monclient(hunting): authenticate will time out at 1445.199729s 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84c096e10 0x7fa84c1363c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 0x7fa84c136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84c096e10 0x7fa84c1363c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58244/0 (socket says 172.21.15.116:58244) 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 0x7fa84c136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43470/0 (socket says 172.21.15.116:43470) 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 1 -- 172.21.15.116:0/284179350 learned_addr learned my addr 172.21.15.116:0/284179350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 10 monclient(hunting): get_auth_request con 0x7fa84c096e10 auth_method 0 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:57.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 10 monclient(hunting): get_auth_request con 0x7fa84c097720 auth_method 0 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa850b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.878+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa85133a640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c131f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 10 monclient(hunting): handle_auth_done global_id 154140 payload 824 2024-09-18T12:36:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 1 -- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84c096e10 msgr2=0x7fa84c1363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84c096e10 0x7fa84c1363c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 1 -- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 msgr2=0x7fa84c131f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c131f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 1 monclient: found mon.noname-a 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa84bfff640 1 -- 172.21.15.116:0/284179350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa838009390 con 0x7fa84c097720 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa85133a640 10 monclient: get_auth_request con 0x7fa84c09aaa0 auth_method 0 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa85133a640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c131f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.879+0000 7fa850b39640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84c096e10 0x7fa84c1363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa84bfff640 1 -- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 msgr2=0x7fa84c136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:57.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa84bfff640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 0x7fa84c136f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa838031000 tx=0x7fa838032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 1 -- 172.21.15.116:0/284179350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83803bd90 con 0x7fa84c097720 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: dump: 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: _finish_auth 0 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa852d9d640 5 monclient: authenticate success, global_id 154140 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa852d9d640 10 monclient: _renew_subs 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 1 -- 172.21.15.116:0/284179350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8380453f0 con 0x7fa84c097720 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa852d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c09f370 con 0x7fa84c097720 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 1 -- 172.21.15.116:0/284179350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8380457b0 con 0x7fa84c097720 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa848ff9640 10 monclient: dump: 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa852d9d640 10 monclient: _renew_subs 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa84affd640 4 set_mon_vals no callback set 2024-09-18T12:36:57.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa852d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.881+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c136900 con 0x7fa84c097720 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.882+0000 7fa848ff9640 1 -- 172.21.15.116:0/284179350 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa8380313a0 con 0x7fa84c097720 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.882+0000 7fa848ff9640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa830038490 0x7fa83003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.882+0000 7fa852d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.882+0000 7fa852d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.882+0000 7fa852d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.882+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c04bf60 con 0x7fa84c097720 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.883+0000 7fa848ff9640 1 -- 172.21.15.116:0/284179350 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa838070a20 con 0x7fa84c097720 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.883+0000 7fa850b39640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa830038490 0x7fa83003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:57.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.883+0000 7fa850b39640 10 monclient: get_auth_request con 0x7fa830038490 auth_method 0 2024-09-18T12:36:57.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.885+0000 7fa850b39640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa830038490 0x7fa83003a940 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa840006cb0 tx=0x7fa840008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:57.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.887+0000 7fa848ff9640 1 -- 172.21.15.116:0/284179350 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa838037030 con 0x7fa84c097720 2024-09-18T12:36:57.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.887+0000 7fa848ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:57.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:57.887+0000 7fa848ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:58.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.100+0000 7fa852d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:36:58.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.100+0000 7fa852d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:58.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.100+0000 7fa852d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:58.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.100+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa84c04bf60 con 0x7fa84c097720 2024-09-18T12:36:58.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.101+0000 7fa848ff9640 1 -- 172.21.15.116:0/284179350 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa838037030 con 0x7fa84c097720 2024-09-18T12:36:58.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.101+0000 7fa848ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:36:58.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.101+0000 7fa848ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:58.102 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.102+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa830038490 msgr2=0x7fa83003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.102+0000 7fa852d9d640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa830038490 0x7fa83003a940 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa840006cb0 tx=0x7fa840008040 comp rx=0 tx=0).stop 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.102+0000 7fa852d9d640 10 monclient: shutdown 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.102+0000 7fa852d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.102+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 msgr2=0x7fa84c136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 0x7fa84c136f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa838031000 tx=0x7fa838032000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 shutdown_connections 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa830038490 0x7fa83003a940 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa84c09aaa0 0x7fa84c131f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa84c097720 0x7fa84c136f40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 --2- 172.21.15.116:0/284179350 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa84c096e10 0x7fa84c1363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 >> 172.21.15.116:0/284179350 conn(0x7fa84c0672b0 msgr2=0x7fa84c06e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:58.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 shutdown_connections 2024-09-18T12:36:58.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.103+0000 7fa852d9d640 1 -- 172.21.15.116:0/284179350 wait complete. 2024-09-18T12:36:58.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:36:58.114 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:36:58.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:36:58.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:36:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.207+0000 7fc0e6109640 10 monclient: get_monmap_and_config 2024-09-18T12:36:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.207+0000 7fc0e6109640 10 monclient: build_initial_monmap 2024-09-18T12:36:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.207+0000 7fc0e6109640 10 monclient: monmap: 2024-09-18T12:36:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:58.207831+0000 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:58.207831+0000 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.207+0000 7fc0e6109640 1 Processor -- start 2024-09-18T12:36:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.207+0000 7fc0e6109640 1 -- start start 2024-09-18T12:36:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.207+0000 7fc0e6109640 10 monclient: init 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0e01318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc0e009ea50 0x7fc0e009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e009ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0e0136c20 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e0131cc0 con 0x7fc0e01318f0 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e009fbf0 con 0x7fc0e0136c20 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e009fe30 con 0x7fc0e009ea50 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0e6109640 10 monclient(hunting): authenticate will time out at 1445.529413s 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0df7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc0e01318f0 0x7fc0e0131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42026/0 (socket says 172.21.15.116:42026) 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0df7fe640 1 -- 172.21.15.116:0/2761280013 learned_addr learned my addr 172.21.15.116:0/2761280013 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 1 --2- 172.21.15.116:0/2761280013 >> v2:172.21.15.144:3300/0 conn(0x7fc0e009ea50 0x7fc0e009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): get_auth_request con 0x7fc0e009ea50 auth_method 0 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.208+0000 7fc0deffd640 10 monclient(hunting): handle_auth_done global_id 124501 payload 274 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.209+0000 7fc0deffd640 1 -- 172.21.15.116:0/2761280013 >> v1:172.21.15.144:6789/0 conn(0x7fc0e0136c20 legacy=0x7fc0e0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.209+0000 7fc0deffd640 1 -- 172.21.15.116:0/2761280013 >> v1:172.21.15.116:6789/0 conn(0x7fc0e01318f0 legacy=0x7fc0e0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.209+0000 7fc0deffd640 10 monclient: _finish_hunting 0 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.209+0000 7fc0deffd640 1 monclient: found mon.noname-c 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.209+0000 7fc0deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.209+0000 7fc0deffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:36:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.209+0000 7fc0deffd640 1 -- 172.21.15.116:0/2761280013 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e00a1210 con 0x7fc0e009ea50 2024-09-18T12:36:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0deffd640 1 -- 172.21.15.116:0/2761280013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e009ea50 msgr2=0x7fc0e009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0deffd640 1 --2- 172.21.15.116:0/2761280013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e009ea50 0x7fc0e009ee20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009850 tx=0x7fc0d40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74307183e0191b95 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 1 -- 172.21.15.116:0/2761280013 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4032440 con 0x7fc0e009ea50 2024-09-18T12:36:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: dump: 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: _finish_auth 0 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0e6109640 5 monclient: authenticate success, global_id 124501 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 1 -- 172.21.15.116:0/2761280013 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d40325a0 con 0x7fc0e009ea50 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0e6109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0e6109640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0e6109640 4 set_mon_vals no callback set 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 1 -- 172.21.15.116:0/2761280013 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4032940 con 0x7fc0e009ea50 2024-09-18T12:36:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.212+0000 7fc0de7fc640 10 monclient: dump: 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 10 monclient: shutdown 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 1 -- 172.21.15.116:0/2761280013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e009ea50 msgr2=0x7fc0e009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 1 --2- 172.21.15.116:0/2761280013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e009ea50 0x7fc0e009ee20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009850 tx=0x7fc0d40090f0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 1 -- 172.21.15.116:0/2761280013 shutdown_connections 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 1 --2- 172.21.15.116:0/2761280013 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e009ea50 0x7fc0e009ee20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 1 -- 172.21.15.116:0/2761280013 >> 172.21.15.116:0/2761280013 conn(0x7fc0e0067110 msgr2=0x7fc0e0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 1 -- 172.21.15.116:0/2761280013 shutdown_connections 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 1 -- 172.21.15.116:0/2761280013 wait complete. 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 10 monclient: build_initial_monmap 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.213+0000 7fc0e6109640 10 monclient: monmap: 2024-09-18T12:36:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 Processor -- start 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 -- start start 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient: init 2024-09-18T12:36:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0e009ea50 0x7fc0e01327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e01318f0 0x7fc0e0132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e01318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0136c20 0x7fc0e0133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): picked mon.noname-a con 0x7fc0e0136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e013a460 con 0x7fc0e0136c20 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e01360c0 con 0x7fc0e01318f0 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0135de0 con 0x7fc0e009ea50 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0e6109640 10 monclient(hunting): authenticate will time out at 1445.535835s 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0df7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0e009ea50 0x7fc0e01327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0136c20 0x7fc0e0133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0df7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0e009ea50 0x7fc0e01327d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58250/0 (socket says 172.21.15.116:58250) 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0df7fe640 1 -- 172.21.15.116:0/3348980797 learned_addr learned my addr 172.21.15.116:0/3348980797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0dffff640 10 monclient(hunting): get_auth_request con 0x7fc0e0136c20 auth_method 0 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0df7fe640 10 monclient(hunting): get_auth_request con 0x7fc0e009ea50 auth_method 0 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.214+0000 7fc0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0deffd640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e01318f0 0x7fc0e0132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 10 monclient(hunting): handle_auth_done global_id 154143 payload 824 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0e009ea50 msgr2=0x7fc0e01327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0e009ea50 0x7fc0e01327d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e01318f0 msgr2=0x7fc0e0132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e01318f0 0x7fc0e0132d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 1 monclient: found mon.noname-a 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0dffff640 1 -- 172.21.15.116:0/3348980797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4009390 con 0x7fc0e0136c20 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0deffd640 10 monclient: get_auth_request con 0x7fc0e01318f0 auth_method 0 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0df7fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0e009ea50 0x7fc0e01327d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.215+0000 7fc0deffd640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc0e01318f0 0x7fc0e0132d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0dffff640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0136c20 msgr2=0x7fc0e0133250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0dffff640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0136c20 0x7fc0e0133250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c7e0 tx=0x7fc0d80025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d800a850 con 0x7fc0e0136c20 2024-09-18T12:36:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: dump: 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: _finish_auth 0 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0e6109640 5 monclient: authenticate success, global_id 154143 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d800a9d0 con 0x7fc0e0136c20 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0e6109640 10 monclient: _renew_subs 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0e6109640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e0134320 con 0x7fc0e0136c20 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d800ad90 con 0x7fc0e0136c20 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0c77fe640 10 monclient: dump: 2024-09-18T12:36:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.234 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0e6109640 10 monclient: _renew_subs 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0ddffb640 4 set_mon_vals no callback set 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0e6109640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.217+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e01366f0 con 0x7fc0e0136c20 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0e6109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0e6109640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 0x7fc0e0136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0e6109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0065e00 con 0x7fc0e0133790 2024-09-18T12:36:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 0x7fc0e0136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient: get_auth_request con 0x7fc0e0133790 auth_method 0 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc0d8012630 con 0x7fc0e0136c20 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_done global_id 154146 payload 824 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0df7fe640 1 -- 172.21.15.116:0/3348980797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0e0136280 con 0x7fc0e0133790 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0c77fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc0cc038490 0x7fc0cc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.218+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc0d804c570 con 0x7fc0e0136c20 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.219+0000 7fc0deffd640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc0cc038490 0x7fc0cc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.219+0000 7fc0deffd640 10 monclient: get_auth_request con 0x7fc0cc038490 auth_method 0 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0df7fe640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 msgr2=0x7fc0e0136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0df7fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 0x7fc0e0136ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007320 tx=0x7fc0d0004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d000fd50 con 0x7fc0e0133790 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 10 monclient: dump: 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc0d000feb0 con 0x7fc0e0133790 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 msgr2=0x7fc0e0136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 0x7fc0e0136ff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007320 tx=0x7fc0d0004960 comp rx=0 tx=0).stop 2024-09-18T12:36:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 msgr2=0x7fc0e0136ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0c77fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0133790 0x7fc0e0136ff0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007320 tx=0x7fc0d0004960 comp rx=0 tx=0).stop 2024-09-18T12:36:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.221+0000 7fc0deffd640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc0cc038490 0x7fc0cc03a940 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4031000 tx=0x7fc0d4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0e6109640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0e6109640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 0x7fc0e0137930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0e6109640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e004bf60 con 0x7fc0e0137530 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 0x7fc0e0137930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient: get_auth_request con 0x7fc0e0137530 auth_method 0 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 10 monclient(hunting): handle_auth_done global_id 154149 payload 824 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.227+0000 7fc0df7fe640 1 -- 172.21.15.116:0/3348980797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc0e00638d0 con 0x7fc0e0137530 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.229+0000 7fc0df7fe640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 msgr2=0x7fc0e0137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.229+0000 7fc0df7fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 0x7fc0e0137930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007320 tx=0x7fc0d0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0021870 con 0x7fc0e0137530 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 10 monclient: dump: 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fc0d00219d0 con 0x7fc0e0137530 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 msgr2=0x7fc0e0137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 0x7fc0e0137930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007320 tx=0x7fc0d0018000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 msgr2=0x7fc0e0137930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0c77fe640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0137530 0x7fc0e0137930 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007320 tx=0x7fc0d0018000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 116, 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:58.244 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.245 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.246 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:58.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.248 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.249 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.250 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.251 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.252 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.255 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:36:58.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.256 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.256 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:58.256 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc0cc038490 msgr2=0x7fc0cc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc0cc038490 0x7fc0cc03a940 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4031000 tx=0x7fc0d4032000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 10 monclient: shutdown 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0136c20 msgr2=0x7fc0e0133250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0dffff640 1 -- 172.21.15.116:0/3348980797 reap_dead start 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0136c20 0x7fc0e0133250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0d800c7e0 tx=0x7fc0d80025b0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 shutdown_connections 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 --2- 172.21.15.116:0/3348980797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e0136c20 0x7fc0e0133250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 >> 172.21.15.116:0/3348980797 conn(0x7fc0e0067110 msgr2=0x7fc0e012bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 shutdown_connections 2024-09-18T12:36:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.230+0000 7fc0e6109640 1 -- 172.21.15.116:0/3348980797 wait complete. 2024-09-18T12:36:58.258 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:58.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:36:58.259 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:36:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.350+0000 7f3441eef640 10 monclient: get_monmap_and_config 2024-09-18T12:36:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.350+0000 7f3441eef640 10 monclient: build_initial_monmap 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.350+0000 7f3441eef640 10 monclient: monmap: 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:58.351557+0000 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:58.351557+0000 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:58.353 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.351+0000 7f3441eef640 1 Processor -- start 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.351+0000 7f3441eef640 1 -- start start 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.351+0000 7f3441eef640 10 monclient: init 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f343c09d2f0 0x7f343c099360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient(hunting): picked mon.noname-c con 0x7f343c09d2f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f343c09b7a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f343c09c2f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0977f0 con 0x7f343c09c2f0 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c09d6c0 con 0x7f343c09b7a0 2024-09-18T12:36:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c0989f0 con 0x7f343c09d2f0 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f3441eef640 10 monclient(hunting): authenticate will time out at 1445.673283s 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f343c09c2f0 0x7f343c1355c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42034/0 (socket says 172.21.15.116:42034) 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343bfff640 1 -- 172.21.15.116:0/1601660259 learned_addr learned my addr 172.21.15.116:0/1601660259 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 1 --2- 172.21.15.116:0/1601660259 >> v2:172.21.15.144:3300/0 conn(0x7f343c09d2f0 0x7f343c099360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): get_auth_request con 0x7f343c09d2f0 auth_method 0 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient(hunting): handle_auth_done global_id 124504 payload 274 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 1 -- 172.21.15.116:0/1601660259 >> v1:172.21.15.144:6789/0 conn(0x7f343c09b7a0 legacy=0x7f343c09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 1 -- 172.21.15.116:0/1601660259 >> v1:172.21.15.116:6789/0 conn(0x7f343c09c2f0 legacy=0x7f343c1355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 1 monclient: found mon.noname-c 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.352+0000 7f343b7fe640 1 -- 172.21.15.116:0/1601660259 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c098710 con 0x7f343c09d2f0 2024-09-18T12:36:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.355+0000 7f343b7fe640 1 -- 172.21.15.116:0/1601660259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09d2f0 msgr2=0x7f343c099360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.355+0000 7f343b7fe640 1 --2- 172.21.15.116:0/1601660259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09d2f0 0x7f343c099360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f342c00a9f0 tx=0x7f342c00a2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b3d914c8127dbdc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 1 -- 172.21.15.116:0/1601660259 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c030800 con 0x7f343c09d2f0 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: dump: 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: _finish_auth 0 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 1 -- 172.21.15.116:0/1601660259 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c02e050 con 0x7f343c09d2f0 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 5 monclient: authenticate success, global_id 124504 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 4 set_mon_vals no callback set 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 1 -- 172.21.15.116:0/1601660259 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c030bd0 con 0x7f343c09d2f0 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f343affd640 10 monclient: dump: 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 10 monclient: shutdown 2024-09-18T12:36:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 1 -- 172.21.15.116:0/1601660259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09d2f0 msgr2=0x7f343c099360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 1 --2- 172.21.15.116:0/1601660259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09d2f0 0x7f343c099360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f342c00a9f0 tx=0x7f342c00a2b0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 1 -- 172.21.15.116:0/1601660259 shutdown_connections 2024-09-18T12:36:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 1 --2- 172.21.15.116:0/1601660259 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09d2f0 0x7f343c099360 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 1 -- 172.21.15.116:0/1601660259 >> 172.21.15.116:0/1601660259 conn(0x7f343c067d00 msgr2=0x7f343c0a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 1 -- 172.21.15.116:0/1601660259 shutdown_connections 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.356+0000 7f3441eef640 1 -- 172.21.15.116:0/1601660259 wait complete. 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient: build_initial_monmap 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient: monmap: 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:58.370 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 Processor -- start 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 -- start start 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient: init 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f343c09b7a0 0x7f343c13cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): picked mon.noname-a con 0x7f343c09b7a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09c2f0 0x7f343c12f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): picked mon.noname-b con 0x7f343c09c2f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c09d2f0 0x7f343c119360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): picked mon.noname-c con 0x7f343c09d2f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c06e840 con 0x7f343c09b7a0 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c130640 con 0x7f343c09c2f0 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c13da00 con 0x7f343c09d2f0 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.357+0000 7f3441eef640 10 monclient(hunting): authenticate will time out at 1445.678973s 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f343c09b7a0 0x7f343c13cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f343c09b7a0 0x7f343c13cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43496/0 (socket says 172.21.15.116:43496) 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 -- 172.21.15.116:0/1679030287 learned_addr learned my addr 172.21.15.116:0/1679030287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): get_auth_request con 0x7f343c09b7a0 auth_method 0 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c09d2f0 0x7f343c119360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 10 monclient(hunting): get_auth_request con 0x7f343c09d2f0 auth_method 0 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f3433fff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09c2f0 0x7f343c12f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient(hunting): handle_auth_done global_id 154152 payload 824 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c09d2f0 msgr2=0x7f343c119360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c09d2f0 0x7f343c119360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09c2f0 msgr2=0x7f343c12f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09c2f0 0x7f343c12f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 monclient: found mon.noname-a 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343b7fe640 1 -- 172.21.15.116:0/1679030287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c00a530 con 0x7f343c09b7a0 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f343bfff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c09d2f0 0x7f343c119360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f3433fff640 10 monclient: get_auth_request con 0x7f343c09c2f0 auth_method 0 2024-09-18T12:36:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.358+0000 7f3433fff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f343c09c2f0 0x7f343c12f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f343b7fe640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f343c09b7a0 msgr2=0x7f343c13cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f343b7fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f343c09b7a0 0x7f343c13cb90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f342c006e70 tx=0x7f342c038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c032da0 con 0x7f343c09b7a0 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: dump: 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: _finish_auth 0 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c03c410 con 0x7f343c09b7a0 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f3441eef640 5 monclient: authenticate success, global_id 154152 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f3441eef640 10 monclient: _renew_subs 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c03c7d0 con 0x7f343c09b7a0 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f3441eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c11c930 con 0x7f343c09b7a0 2024-09-18T12:36:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f34337fe640 10 monclient: dump: 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f343a7fc640 4 set_mon_vals no callback set 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f3441eef640 10 monclient: _renew_subs 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f3441eef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.360+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c13d0d0 con 0x7f343c09b7a0 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3441eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3441eef640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 0x7f343c1198a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3441eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c11cdb0 con 0x7f343c11b9d0 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f342c03cb80 con 0x7f343c09b7a0 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 0x7f343c1198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient: get_auth_request con 0x7f343c11b9d0 auth_method 0 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f34337fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 10 monclient(hunting): handle_auth_done global_id 144239 payload 824 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f3433fff640 1 -- 172.21.15.116:0/1679030287 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f343c13d2c0 con 0x7f343c11b9d0 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.361+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f342c06d380 con 0x7f343c09b7a0 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.362+0000 7f343bfff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.362+0000 7f343bfff640 10 monclient: get_auth_request con 0x7f341c038490 auth_method 0 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f3433fff640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 msgr2=0x7f343c1198a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f3433fff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 0x7f343c1198a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f342400d500 tx=0x7f3424009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424007030 con 0x7f343c11b9d0 2024-09-18T12:36:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 10 monclient: dump: 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3424007190 con 0x7f343c11b9d0 2024-09-18T12:36:58.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:58.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:58.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 msgr2=0x7f343c1198a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 0x7f343c1198a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f342400d500 tx=0x7f3424009040 comp rx=0 tx=0).stop 2024-09-18T12:36:58.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 msgr2=0x7f343c1198a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:58.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f34337fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c11b9d0 0x7f343c1198a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.364+0000 7f343bfff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f343400d6a0 tx=0x7f3434009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.369+0000 7f3441eef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3441eef640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 0x7f343c119de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3441eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c065f90 con 0x7f343c12fd50 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 0x7f343c119de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient: get_auth_request con 0x7f343c12fd50 auth_method 0 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 10 monclient(hunting): handle_auth_done global_id 144242 payload 824 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.370+0000 7f3433fff640 1 -- 172.21.15.116:0/1679030287 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f343c04bf60 con 0x7f343c12fd50 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f3433fff640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 msgr2=0x7f343c119de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f3433fff640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 0x7f343c119de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f342400d500 tx=0x7f342401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424020070 con 0x7f343c12fd50 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 10 monclient: dump: 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.384 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f34240076a0 con 0x7f343c12fd50 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 msgr2=0x7f343c119de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 0x7f343c119de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f342400d500 tx=0x7f342401a000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 msgr2=0x7f343c119de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:58.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.372+0000 7f34337fe640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f343c12fd50 0x7f343c119de0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 116, 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:58.386 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.387 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.388 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:58.389 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.390 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:58.391 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.392 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:58.395 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:58.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:58.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:58.398 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 msgr2=0x7f341c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f341c038490 0x7f341c03a940 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f343400d6a0 tx=0x7f3434009040 comp rx=0 tx=0).stop 2024-09-18T12:36:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 10 monclient: shutdown 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f343c09b7a0 msgr2=0x7f343c13cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 --2- 172.21.15.116:0/1679030287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f343c09b7a0 0x7f343c13cb90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f342c006e70 tx=0x7f342c038000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f343bfff640 1 -- 172.21.15.116:0/1679030287 reap_dead start 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f343bfff640 1 -- 172.21.15.116:0/1679030287 reap_dead start 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 shutdown_connections 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 >> 172.21.15.116:0/1679030287 conn(0x7f343c067d00 msgr2=0x7f343c130850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 shutdown_connections 2024-09-18T12:36:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.373+0000 7f3441eef640 1 -- 172.21.15.116:0/1679030287 wait complete. 2024-09-18T12:36:58.400 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:58.401 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:36:58.401 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:36:58.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.492+0000 7feb6be57640 10 monclient: get_monmap_and_config 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.492+0000 7feb6be57640 10 monclient: build_initial_monmap 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.492+0000 7feb6be57640 10 monclient: monmap: 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:36:58.493394+0000 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:36:58.493394+0000 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:36:58.494 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.493+0000 7feb6be57640 1 Processor -- start 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.493+0000 7feb6be57640 1 -- start start 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.493+0000 7feb6be57640 10 monclient: init 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.493+0000 7feb6be57640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.493+0000 7feb6be57640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb640974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb6409a8a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb64134080 0x7feb64136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 10 monclient(hunting): picked mon.noname-b con 0x7feb64134080 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6412b210 con 0x7feb6409a8a0 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb640978c0 con 0x7feb640974f0 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6409ca20 con 0x7feb64134080 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6be57640 10 monclient(hunting): authenticate will time out at 1445.815098s 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb693f2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feb6409a8a0 0x7feb6409ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42036/0 (socket says 172.21.15.116:42036) 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb693f2640 1 -- 172.21.15.116:0/1449634050 learned_addr learned my addr 172.21.15.116:0/1449634050 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 1 --2- 172.21.15.116:0/1449634050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb64134080 0x7feb64136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): get_auth_request con 0x7feb64134080 auth_method 0 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient(hunting): handle_auth_done global_id 144245 payload 274 2024-09-18T12:36:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 1 -- 172.21.15.116:0/1449634050 >> v1:172.21.15.144:6789/0 conn(0x7feb640974f0 legacy=0x7feb6412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 1 -- 172.21.15.116:0/1449634050 >> v1:172.21.15.116:6789/0 conn(0x7feb6409a8a0 legacy=0x7feb6409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:36:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient: _finish_hunting 0 2024-09-18T12:36:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 1 monclient: found mon.noname-b 2024-09-18T12:36:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:36:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.494+0000 7feb6a3f4640 1 -- 172.21.15.116:0/1449634050 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb64098ad0 con 0x7feb64134080 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb6a3f4640 1 -- 172.21.15.116:0/1449634050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb64134080 msgr2=0x7feb64136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb6a3f4640 1 --2- 172.21.15.116:0/1449634050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb64134080 0x7feb64136460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00afa0 tx=0x7feb5c00a890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cec5fe2b164ede1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 1 -- 172.21.15.116:0/1449634050 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c008660 con 0x7feb64134080 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: dump: 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.506 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: _finish_auth 0 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 1 -- 172.21.15.116:0/1449634050 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5c02e050 con 0x7feb64134080 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb6be57640 5 monclient: authenticate success, global_id 144245 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb6be57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:58.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb6be57640 10 monclient: get_monmap_and_config success 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 1 -- 172.21.15.116:0/1449634050 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c0089f0 con 0x7feb64134080 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb6be57640 4 set_mon_vals no callback set 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.497+0000 7feb68bf1640 10 monclient: dump: 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.508 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 10 monclient: shutdown 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 -- 172.21.15.116:0/1449634050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb64134080 msgr2=0x7feb64136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 --2- 172.21.15.116:0/1449634050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb64134080 0x7feb64136460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00afa0 tx=0x7feb5c00a890 comp rx=0 tx=0).stop 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 -- 172.21.15.116:0/1449634050 shutdown_connections 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 --2- 172.21.15.116:0/1449634050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb64134080 0x7feb64136460 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 -- 172.21.15.116:0/1449634050 >> 172.21.15.116:0/1449634050 conn(0x7feb640672b0 msgr2=0x7feb640676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 -- 172.21.15.116:0/1449634050 shutdown_connections 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 -- 172.21.15.116:0/1449634050 wait complete. 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 10 monclient: build_initial_monmap 2024-09-18T12:36:58.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 10 monclient: monmap: 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.498+0000 7feb6be57640 1 Processor -- start 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 1 -- start start 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient: init 2024-09-18T12:36:58.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient: _reopen_session rank -1 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb640974f0 0x7feb6406d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): picked mon.noname-c con 0x7feb640974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409a8a0 0x7feb6406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): picked mon.noname-b con 0x7feb6409a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb64134080 0x7feb640696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): picked mon.noname-a con 0x7feb64134080 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6406cbb0 con 0x7feb64134080 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6406c8d0 con 0x7feb6409a8a0 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6406b530 con 0x7feb640974f0 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): _renew_subs 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6be57640 10 monclient(hunting): authenticate will time out at 1445.820453s 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb640974f0 0x7feb6406d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb64134080 0x7feb640696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb640974f0 0x7feb6406d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58284/0 (socket says 172.21.15.116:58284) 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 -- 172.21.15.116:0/3175726046 learned_addr learned my addr 172.21.15.116:0/3175726046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): get_auth_request con 0x7feb640974f0 auth_method 0 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 10 monclient(hunting): get_auth_request con 0x7feb64134080 auth_method 0 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409a8a0 0x7feb6406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 10 monclient(hunting): get_auth_request con 0x7feb6409a8a0 auth_method 0 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient(hunting): handle_auth_done global_id 144248 payload 824 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409a8a0 msgr2=0x7feb6406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409a8a0 0x7feb6406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb64134080 msgr2=0x7feb640696a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb64134080 0x7feb640696a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient: _finish_hunting 0 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 monclient: found mon.noname-c 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb69bf3640 1 -- 172.21.15.116:0/3175726046 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c00ab30 con 0x7feb640974f0 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.499+0000 7feb6a3f4640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb64134080 0x7feb640696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:36:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb69bf3640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb640974f0 msgr2=0x7feb6406d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb69bf3640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb640974f0 0x7feb6406d2b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c5e0 tx=0x7feb60007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60012840 con 0x7feb640974f0 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: dump: 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: _finish_auth 0 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb600129a0 con 0x7feb640974f0 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb6be57640 5 monclient: authenticate success, global_id 144248 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb6be57640 10 monclient: _renew_subs 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb6be57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6406a9f0 con 0x7feb640974f0 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60012da0 con 0x7feb640974f0 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb59ffb640 10 monclient: dump: 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb5bfff640 4 set_mon_vals no callback set 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb6be57640 10 monclient: _renew_subs 2024-09-18T12:36:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb6be57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.502+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6406acf0 con 0x7feb640974f0 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7feb60011590 con 0x7feb640974f0 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb6be57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb6be57640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 0x7feb6406a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb6be57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb64065f20 con 0x7feb64069ea0 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb59ffb640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feb48038490 0x7feb4803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7feb6004c3c0 con 0x7feb640974f0 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 0x7feb6406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient: get_auth_request con 0x7feb64069ea0 auth_method 0 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb6a3f4640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feb48038490 0x7feb4803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb6a3f4640 10 monclient: get_auth_request con 0x7feb48038490 auth_method 0 2024-09-18T12:36:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 10 monclient(hunting): handle_auth_done global_id 124510 payload 824 2024-09-18T12:36:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.503+0000 7feb693f2640 1 -- 172.21.15.116:0/3175726046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb6406ca40 con 0x7feb64069ea0 2024-09-18T12:36:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb693f2640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 msgr2=0x7feb6406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb693f2640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 0x7feb6406a270 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb64128770 tx=0x7feb54007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb6a3f4640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feb48038490 0x7feb4803a940 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feb6409d040 tx=0x7feb5c037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54004b30 con 0x7feb64069ea0 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 10 monclient: dump: 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.522 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7feb54004c90 con 0x7feb64069ea0 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 msgr2=0x7feb6406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 0x7feb6406a270 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb64128770 tx=0x7feb54007f00 comp rx=0 tx=0).stop 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 msgr2=0x7feb6406a270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.506+0000 7feb59ffb640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb64069ea0 0x7feb6406a270 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb64128770 tx=0x7feb54007f00 comp rx=0 tx=0).stop 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb6be57640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:36:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb6be57640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 0x7feb6409ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb6be57640 10 monclient(hunting): start opening mon connection 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6404bf60 con 0x7feb6409ea60 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 0x7feb6409ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient: get_auth_request con 0x7feb6409ea60 auth_method 0 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 10 monclient(hunting): handle_auth_done global_id 124513 payload 824 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.512+0000 7feb693f2640 1 -- 172.21.15.116:0/3175726046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb64129090 con 0x7feb6409ea60 2024-09-18T12:36:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb693f2640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 msgr2=0x7feb6409ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb693f2640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 0x7feb6409ee60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb64128770 tx=0x7feb54019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb54015d30 con 0x7feb6409ea60 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 10 monclient: dump: 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:36:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7feb54015e90 con 0x7feb6409ea60 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 msgr2=0x7feb6409ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 0x7feb6409ee60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb64128770 tx=0x7feb54019000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 msgr2=0x7feb6409ee60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.515+0000 7feb59ffb640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7feb6409ea60 0x7feb6409ee60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:36:58.526 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 116, 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:36:58.527 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.528 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:36:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:36:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:36:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:36:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:36:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.537 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.538 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:36:58.539 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feb48038490 msgr2=0x7feb4803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feb48038490 0x7feb4803a940 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feb6409d040 tx=0x7feb5c037000 comp rx=0 tx=0).stop 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 10 monclient: shutdown 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb640974f0 msgr2=0x7feb6406d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 --2- 172.21.15.116:0/3175726046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7feb640974f0 0x7feb6406d2b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c5e0 tx=0x7feb60007ed0 comp rx=0 tx=0).stop 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6a3f4640 1 -- 172.21.15.116:0/3175726046 reap_dead start 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6a3f4640 1 -- 172.21.15.116:0/3175726046 reap_dead start 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 shutdown_connections 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 >> 172.21.15.116:0/3175726046 conn(0x7feb640672b0 msgr2=0x7feb641246c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 shutdown_connections 2024-09-18T12:36:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:36:58.516+0000 7feb6be57640 1 -- 172.21.15.116:0/3175726046 wait complete. 2024-09-18T12:36:58.542 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:36:58.542 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T12:36:58.542 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-18T12:36:58.542 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T12:36:58.542 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:36:58.542 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:36:58.543 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:36:58.561 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:36:58.561 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T12:36:58.562 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T12:36:58.562 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:37:13.562 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T12:37:13.563 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T12:37:13.563 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:37:13.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:37:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.665+0000 7f6757c6f640 10 monclient: get_monmap_and_config 2024-09-18T12:37:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.665+0000 7f6757c6f640 10 monclient: build_initial_monmap 2024-09-18T12:37:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.665+0000 7f6757c6f640 10 monclient: monmap: 2024-09-18T12:37:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:37:13.666333+0000 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:37:13.666333+0000 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:37:13.667 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:13.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 1 Processor -- start 2024-09-18T12:37:13.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 1 -- start start 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient: init 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient(hunting): picked mon.noname-b con 0x7f6750096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f675009aaa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient(hunting): picked mon.noname-a con 0x7f67500978f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675009d4a0 con 0x7f675009aaa0 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6750099620 con 0x7f67500978f0 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6750099340 con 0x7f6750096e10 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.666+0000 7f6757c6f640 10 monclient(hunting): authenticate will time out at 1460.987996s 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675520a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f675009aaa0 0x7f67500971e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43838/0 (socket says 172.21.15.116:43838) 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42212/0 (socket says 172.21.15.116:42212) 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 10 monclient(hunting): get_auth_request con 0x7f6750096e10 auth_method 0 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675520a640 1 -- 172.21.15.116:0/2925876518 learned_addr learned my addr 172.21.15.116:0/2925876518 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 10 monclient(hunting): get_auth_request con 0x7f67500978f0 auth_method 0 2024-09-18T12:37:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.667+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.669+0000 7f675520a640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.116:6789/0 conn(0x7f675009aaa0 legacy=0x7f67500971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:13.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:13.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 10 monclient(hunting): get_auth_request con 0x7f6750096e10 auth_method 0 2024-09-18T12:37:13.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 10 monclient(hunting): get_auth_request con 0x7f67500978f0 auth_method 0 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:37:13.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:13.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:13.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:13.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:13.868+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 10 monclient(hunting): get_auth_request con 0x7f6750096e10 auth_method 0 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 10 monclient(hunting): get_auth_request con 0x7f67500978f0 auth_method 0 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:37:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:14.269+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:37:15.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:15.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:15.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 10 monclient(hunting): get_auth_request con 0x7f67500978f0 auth_method 0 2024-09-18T12:37:15.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 10 monclient(hunting): get_auth_request con 0x7f6750096e10 auth_method 0 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:37:15.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:15.070+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:37:16.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 10 monclient(hunting): tick 2024-09-18T12:37:16.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:37:16.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:37:16.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 msgr2=0x7f675009a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:16.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6750096e10 0x7f675009a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:16.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 msgr2=0x7f6750135b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.116:3300/0 conn(0x7f67500978f0 0x7f6750135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.116:6789/0 conn(0x7f675009aaa0 legacy=0x7f67500971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 10 monclient: _add_conns ranks=[4,1] 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6734003700 addr v2:172.21.15.144:3300/0 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6734006240 addr v1:172.21.15.144:6789/0 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6750099900 con 0x7f6734006240 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6750099620 con 0x7f6734003700 2024-09-18T12:37:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f67477fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 msgr2=0x7f6734005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f675520a640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.144:6789/0 conn(0x7f6734006240 legacy=0x7f67340086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:37:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.667+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.868+0000 7f675520a640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.144:6789/0 conn(0x7f6734006240 legacy=0x7f67340086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:37:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.868+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 msgr2=0x7f6734005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:16.868+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:37:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:17.269+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 msgr2=0x7f6734005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:17.269+0000 7f675520a640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.144:6789/0 conn(0x7f6734006240 legacy=0x7f67340086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:37:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:17.269+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:37:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:18.070+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 msgr2=0x7f6734005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:18.070+0000 7f675520a640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.144:6789/0 conn(0x7f6734006240 legacy=0x7f67340086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:37:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:18.070+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:37:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): tick 2024-09-18T12:37:19.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 msgr2=0x7f6734005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.144:6789/0 conn(0x7f6734006240 legacy=0x7f67340086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 reap_dead start 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6734003700 addr v2:172.21.15.144:3300/0 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6734009fa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f673400a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): picked mon.noname-b con 0x7f673400a430 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:19.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673400afa0 con 0x7f6734009fa0 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6750099620 con 0x7f6734003700 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 1 -- 172.21.15.116:0/2925876518 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6750099340 con 0x7f673400a430 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f67477fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f6755a0b640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 msgr2=0x7f6734005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f6755a0b640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f673400a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 10 monclient(hunting): get_auth_request con 0x7f673400a430 auth_method 0 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:19.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.667+0000 7f675620c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 10 monclient(hunting): handle_auth_done global_id 144251 payload 274 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 msgr2=0x7f6734005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> v1:172.21.15.116:6789/0 conn(0x7f6734009fa0 legacy=0x7f67340086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 10 monclient: _finish_hunting 0 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 1 monclient: found mon.noname-b 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:37:19.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.668+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750099900 con 0x7f673400a430 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f675620c640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 msgr2=0x7f673400a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f673400a8d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f675009aff0 tx=0x7f674800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9570af96dc6f6b7b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f675620c640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f673400a8d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f675009aff0 tx=0x7f674800b040 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f6748031940 mon_map magic: 0 v1 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 1 -- 172.21.15.116:0/2925876518 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748031940 con 0x7f673400a430 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 10 monclient: dump: 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:19.775 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 10 monclient: _finish_auth 0 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6757c6f640 5 monclient: authenticate success, global_id 144251 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 1 -- 172.21.15.116:0/2925876518 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6748031ae0 con 0x7f673400a430 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6757c6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6754a09640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6757c6f640 10 monclient: get_monmap_and_config success 2024-09-18T12:37:19.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.671+0000 7f6757c6f640 4 set_mon_vals no callback set 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 10 monclient: shutdown 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 -- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 msgr2=0x7f673400a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f673400a8d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f675009aff0 tx=0x7f674800b040 comp rx=0 tx=0).stop 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6754a09640 1 -- 172.21.15.116:0/2925876518 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f6748031ea0 con 0x7f673400a430 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6754a09640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 -- 172.21.15.116:0/2925876518 shutdown_connections 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 --2- 172.21.15.116:0/2925876518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f673400a8d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 --2- 172.21.15.116:0/2925876518 >> v2:172.21.15.144:3300/0 conn(0x7f6734003700 0x7f6734005b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 -- 172.21.15.116:0/2925876518 >> 172.21.15.116:0/2925876518 conn(0x7f67500672b0 msgr2=0x7f67500676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 -- 172.21.15.116:0/2925876518 shutdown_connections 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 1 -- 172.21.15.116:0/2925876518 wait complete. 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 10 monclient: build_initial_monmap 2024-09-18T12:37:19.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.672+0000 7f6757c6f640 10 monclient: monmap: 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 Processor -- start 2024-09-18T12:37:19.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 -- start start 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient: init 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 0x7f6750140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): picked mon.noname-a con 0x7f6734003700 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6734009fa0 0x7f675012e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): picked mon.noname-b con 0x7f6734009fa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f6750118380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): picked mon.noname-c con 0x7f673400a430 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6750130700 con 0x7f6734003700 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675011bcd0 con 0x7f6734009fa0 2024-09-18T12:37:19.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675011b9f0 con 0x7f673400a430 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6757c6f640 10 monclient(hunting): authenticate will time out at 1466.994891s 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.673+0000 7f6755a0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 0x7f6750140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675520a640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6734009fa0 msgr2=0x7f675012e8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f6750118380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675520a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6734009fa0 0x7f675012e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 0x7f6750140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51936/0 (socket says 172.21.15.116:51936) 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 -- 172.21.15.116:0/1711618948 learned_addr learned my addr 172.21.15.116:0/1711618948 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): get_auth_request con 0x7f6734003700 auth_method 0 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:19.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 10 monclient(hunting): get_auth_request con 0x7f673400a430 auth_method 0 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient(hunting): handle_auth_done global_id 154158 payload 824 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 -- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 msgr2=0x7f6750118380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f6750118380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 -- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6734009fa0 msgr2=0x7f675012e8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6734009fa0 0x7f675012e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient: _finish_hunting 0 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 monclient: found mon.noname-a 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f6755a0b640 1 -- 172.21.15.116:0/1711618948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748009980 con 0x7f6734003700 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.674+0000 7f675620c640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f6750118380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.676+0000 7f6755a0b640 1 -- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 msgr2=0x7f6750140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.676+0000 7f6755a0b640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 0x7f6750140ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f674c00ddb0 tx=0x7f674c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 1 -- 172.21.15.116:0/1711618948 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c011710 con 0x7f6734003700 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:37:19.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: dump: 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: _finish_auth 0 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 1 -- 172.21.15.116:0/1711618948 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c011870 con 0x7f6734003700 2024-09-18T12:37:19.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6757c6f640 5 monclient: authenticate success, global_id 154158 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6757c6f640 10 monclient: _renew_subs 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6757c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6750141410 con 0x7f6734003700 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 1 -- 172.21.15.116:0/1711618948 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c011bf0 con 0x7f6734003700 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6745ffb640 10 monclient: dump: 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6757c6f640 10 monclient: _renew_subs 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6747fff640 4 set_mon_vals no callback set 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6757c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.677+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6750119ec0 con 0x7f6734003700 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f6745ffb640 1 -- 172.21.15.116:0/1711618948 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f674c017070 con 0x7f6734003700 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f6745ffb640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6734039a60 0x7f673403bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f6745ffb640 1 -- 172.21.15.116:0/1711618948 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f674c04ce60 con 0x7f6734003700 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f6757c6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f6757c6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f6757c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675004bf60 con 0x7f6734003700 2024-09-18T12:37:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f675520a640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6734039a60 0x7f673403bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.678+0000 7f675520a640 10 monclient: get_auth_request con 0x7f6734039a60 auth_method 0 2024-09-18T12:37:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.681+0000 7f675520a640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6734039a60 0x7f673403bf10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f6740007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.683+0000 7f6745ffb640 1 -- 172.21.15.116:0/1711618948 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f674c00bb40 con 0x7f6734003700 2024-09-18T12:37:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.683+0000 7f6745ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.683+0000 7f6745ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.903+0000 7f6757c6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.903+0000 7f6757c6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.903+0000 7f6757c6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.903+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f67500638d0 con 0x7f6734003700 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.903+0000 7f6745ffb640 1 -- 172.21.15.116:0/1711618948 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f674c00b360 con 0x7f6734003700 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.903+0000 7f6745ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.903+0000 7f6745ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:37:19.904 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":118,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:37:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6734039a60 msgr2=0x7f673403bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6734039a60 0x7f673403bf10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6740007fd0 tx=0x7f6740007a70 comp rx=0 tx=0).stop 2024-09-18T12:37:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 10 monclient: shutdown 2024-09-18T12:37:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 msgr2=0x7f6750140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 0x7f6750140ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f674c00ddb0 tx=0x7f674c00f730 comp rx=0 tx=0).stop 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 shutdown_connections 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6734039a60 0x7f673403bf10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f673400a430 0x7f6750118380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6734009fa0 0x7f675012e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 --2- 172.21.15.116:0/1711618948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6734003700 0x7f6750140ed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 >> 172.21.15.116:0/1711618948 conn(0x7f67500672b0 msgr2=0x7f675012f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 shutdown_connections 2024-09-18T12:37:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:19.905+0000 7f6757c6f640 1 -- 172.21.15.116:0/1711618948 wait complete. 2024-09-18T12:37:19.917 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:37:19.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:37:19.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.009+0000 7fb4550c0640 10 monclient: get_monmap_and_config 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.009+0000 7fb4550c0640 10 monclient: build_initial_monmap 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.009+0000 7fb4550c0640 10 monclient: monmap: 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:37:20.009817+0000 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:37:20.009817+0000 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.010 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:37:20.011 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:37:20.011 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:37:20.011 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:37:20.011 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:37:20.011 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.009+0000 7fb4550c0640 1 Processor -- start 2024-09-18T12:37:20.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.009+0000 7fb4550c0640 1 -- start start 2024-09-18T12:37:20.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.009+0000 7fb4550c0640 10 monclient: init 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb450131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): picked mon.noname-b con 0x7fb450131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb45009e910 0x7fb45009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): picked mon.noname-a con 0x7fb45009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb45009f220 0x7fb450068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): picked mon.noname-c con 0x7fb45009f220 addr v2:172.21.15.144:3300/0 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45009faf0 con 0x7fb45009e910 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45009fd30 con 0x7fb45009f220 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500a1110 con 0x7fb450131ab0 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb4550c0640 10 monclient(hunting): authenticate will time out at 1467.331271s 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb45009e910 0x7fb45009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb450131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb45009e910 0x7fb45009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51942/0 (socket says 172.21.15.116:51942) 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44f59e640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fb45009f220 msgr2=0x7fb450068ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 1 -- 172.21.15.116:0/522597274 learned_addr learned my addr 172.21.15.116:0/522597274 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb450131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59438/0 (socket says 172.21.15.116:59438) 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 10 monclient(hunting): get_auth_request con 0x7fb45009e910 auth_method 0 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): get_auth_request con 0x7fb450131ab0 auth_method 0 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44f59e640 1 --2- 172.21.15.116:0/522597274 >> v2:172.21.15.144:3300/0 conn(0x7fb45009f220 0x7fb450068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_done global_id 144257 payload 274 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 -- 172.21.15.116:0/522597274 >> v2:172.21.15.144:3300/0 conn(0x7fb45009f220 msgr2=0x7fb450068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:20.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/522597274 >> v2:172.21.15.144:3300/0 conn(0x7fb45009f220 0x7fb450068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 -- 172.21.15.116:0/522597274 >> v2:172.21.15.116:3300/0 conn(0x7fb45009e910 msgr2=0x7fb45009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/522597274 >> v2:172.21.15.116:3300/0 conn(0x7fb45009e910 0x7fb45009ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 monclient: found mon.noname-b 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44ed9d640 1 -- 172.21.15.116:0/522597274 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4500a13f0 con 0x7fb450131ab0 2024-09-18T12:37:20.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.010+0000 7fb44e59c640 1 --2- 172.21.15.116:0/522597274 >> v2:172.21.15.116:3300/0 conn(0x7fb45009e910 0x7fb45009ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:37:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44ed9d640 1 -- 172.21.15.116:0/522597274 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 msgr2=0x7fb450131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/522597274 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb450131e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb45009bd80 tx=0x7fb4400090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4cd591fd7f9e9e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 1 -- 172.21.15.116:0/522597274 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44002aea0 con 0x7fb450131ab0 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: dump: 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 1 -- 172.21.15.116:0/522597274 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb440032440 con 0x7fb450131ab0 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb4550c0640 5 monclient: authenticate success, global_id 144257 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb4550c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb4550c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 1 -- 172.21.15.116:0/522597274 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4400327e0 con 0x7fb450131ab0 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb4550c0640 4 set_mon_vals no callback set 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.013+0000 7fb44dd9b640 10 monclient: dump: 2024-09-18T12:37:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 10 monclient: shutdown 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 -- 172.21.15.116:0/522597274 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 msgr2=0x7fb450131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 --2- 172.21.15.116:0/522597274 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb450131e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb45009bd80 tx=0x7fb4400090f0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 -- 172.21.15.116:0/522597274 shutdown_connections 2024-09-18T12:37:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 --2- 172.21.15.116:0/522597274 >> v2:172.21.15.144:3300/0 conn(0x7fb45009f220 0x7fb450068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 --2- 172.21.15.116:0/522597274 >> v2:172.21.15.116:3300/0 conn(0x7fb45009e910 0x7fb45009ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 --2- 172.21.15.116:0/522597274 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb450131e80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 -- 172.21.15.116:0/522597274 >> 172.21.15.116:0/522597274 conn(0x7fb450067d30 msgr2=0x7fb4500a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 -- 172.21.15.116:0/522597274 shutdown_connections 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 1 -- 172.21.15.116:0/522597274 wait complete. 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 10 monclient: build_initial_monmap 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.014+0000 7fb4550c0640 10 monclient: monmap: 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:20.028 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 Processor -- start 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 -- start start 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient: init 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb45009e910 0x7fb45011dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): picked mon.noname-b con 0x7fb45009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45009f220 0x7fb45011e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): picked mon.noname-a con 0x7fb45009f220 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb45011e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): picked mon.noname-c con 0x7fb450131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45009bde0 con 0x7fb45009f220 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45009ddd0 con 0x7fb45009e910 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb450120650 con 0x7fb450131ab0 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb4550c0640 10 monclient(hunting): authenticate will time out at 1467.336595s 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45009f220 0x7fb45011e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb45011e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44ed9d640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb45009e910 msgr2=0x7fb45011dd90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45009f220 0x7fb45011e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51948/0 (socket says 172.21.15.116:51948) 2024-09-18T12:37:20.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb45011e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59454/0 (socket says 172.21.15.116:59454) 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 -- 172.21.15.116:0/44112653 learned_addr learned my addr 172.21.15.116:0/44112653 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): get_auth_request con 0x7fb45009f220 auth_method 0 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb45009e910 0x7fb45011dd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 10 monclient(hunting): get_auth_request con 0x7fb450131ab0 auth_method 0 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient(hunting): handle_auth_done global_id 154164 payload 824 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 msgr2=0x7fb45011e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb45011e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb45009e910 msgr2=0x7fb45011dd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb45009e910 0x7fb45011dd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 monclient: found mon.noname-a 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44e59c640 1 -- 172.21.15.116:0/44112653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438009450 con 0x7fb45009f220 2024-09-18T12:37:20.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.015+0000 7fb44f59e640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb450131ab0 0x7fb45011e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb44e59c640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45009f220 msgr2=0x7fb45011e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb44e59c640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45009f220 0x7fb45011e2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb43800ec80 tx=0x7fb4380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438009730 con 0x7fb45009f220 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: dump: 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.033 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: _finish_auth 0 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb4550c0640 5 monclient: authenticate success, global_id 154164 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43800eec0 con 0x7fb45009f220 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438013470 con 0x7fb45009f220 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb4550c0640 10 monclient: _renew_subs 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb4550c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb440009390 con 0x7fb45009f220 2024-09-18T12:37:20.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb44d59a640 4 set_mon_vals no callback set 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb446ffd640 10 monclient: dump: 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb4550c0640 10 monclient: _renew_subs 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb4550c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.018+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45011f970 con 0x7fb45009f220 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb4550c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb4550c0640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 0x7fb45011ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450063740 con 0x7fb45002aa40 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb438013820 con 0x7fb45009f220 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 0x7fb45011ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient: get_auth_request con 0x7fb45002aa40 auth_method 0 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb446ffd640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb430038440 0x7fb43003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb43804c6e0 con 0x7fb45009f220 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_done global_id 154167 payload 824 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44ed9d640 1 -- 172.21.15.116:0/44112653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb45002ae10 con 0x7fb45002aa40 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44f59e640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb430038440 0x7fb43003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.019+0000 7fb44f59e640 10 monclient: get_auth_request con 0x7fb430038440 auth_method 0 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.021+0000 7fb44ed9d640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 msgr2=0x7fb45011ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.021+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 0x7fb45011ed50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb440034000 tx=0x7fb440032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.021+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44002aea0 con 0x7fb45002aa40 2024-09-18T12:37:20.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.021+0000 7fb446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.021+0000 7fb446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.021+0000 7fb446ffd640 10 monclient: dump: 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb4400327c0 con 0x7fb45002aa40 2024-09-18T12:37:20.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb446ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:20.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb446ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:37:20.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 msgr2=0x7fb45011ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb446ffd640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 0x7fb45011ed50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb440034000 tx=0x7fb440032000 comp rx=0 tx=0).stop 2024-09-18T12:37:20.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 msgr2=0x7fb45011ed50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:37:20.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb446ffd640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45002aa40 0x7fb45011ed50 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb440034000 tx=0x7fb440032000 comp rx=0 tx=0).stop 2024-09-18T12:37:20.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.022+0000 7fb44f59e640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb430038440 0x7fb43003a8f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb448009fd0 tx=0x7fb44800d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.027+0000 7fb4550c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.027+0000 7fb4550c0640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 0x7fb45005e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.027+0000 7fb4550c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.027+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45004bf60 con 0x7fb45005e310 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.027+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 0x7fb45005e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient: get_auth_request con 0x7fb45005e310 auth_method 0 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 10 monclient(hunting): handle_auth_done global_id 154170 payload 824 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.028+0000 7fb44ed9d640 1 -- 172.21.15.116:0/44112653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb450121930 con 0x7fb45005e310 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb44ed9d640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 msgr2=0x7fb45005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb44ed9d640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 0x7fb45005e710 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb440009130 tx=0x7fb44003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440049070 con 0x7fb45005e310 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 10 monclient: dump: 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.042 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7fb44003b370 con 0x7fb45005e310 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 msgr2=0x7fb45005e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 0x7fb45005e710 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb440009130 tx=0x7fb44003c000 comp rx=0 tx=0).stop 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 msgr2=0x7fb45005e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:37:20.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.030+0000 7fb446ffd640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45005e310 0x7fb45005e710 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb440009130 tx=0x7fb44003c000 comp rx=0 tx=0).stop 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 118, 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:37:20.044 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.045 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:37:20.046 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:37:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:37:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:37:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:37:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:37:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:37:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.057 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:37:20.057 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb430038440 msgr2=0x7fb43003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb430038440 0x7fb43003a8f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb448009fd0 tx=0x7fb44800d040 comp rx=0 tx=0).stop 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 10 monclient: shutdown 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45009f220 msgr2=0x7fb45011e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 --2- 172.21.15.116:0/44112653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45009f220 0x7fb45011e2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb43800ec80 tx=0x7fb4380091d0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb44f59e640 1 -- 172.21.15.116:0/44112653 reap_dead start 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb44f59e640 1 -- 172.21.15.116:0/44112653 reap_dead start 2024-09-18T12:37:20.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 shutdown_connections 2024-09-18T12:37:20.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 >> 172.21.15.116:0/44112653 conn(0x7fb450067d30 msgr2=0x7fb45009e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:20.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 shutdown_connections 2024-09-18T12:37:20.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.031+0000 7fb4550c0640 1 -- 172.21.15.116:0/44112653 wait complete. 2024-09-18T12:37:20.059 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:37:20.059 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:37:20.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:37:20.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.152+0000 7fa95e59d640 10 monclient: get_monmap_and_config 2024-09-18T12:37:20.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.152+0000 7fa95e59d640 10 monclient: build_initial_monmap 2024-09-18T12:37:20.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.152+0000 7fa95e59d640 10 monclient: monmap: 2024-09-18T12:37:20.153 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:20.153 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.153 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:37:20.152946+0000 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:37:20.152946+0000 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:37:20.154 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 Processor -- start 2024-09-18T12:37:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 -- start start 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient: init 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa958131ab0 0x7fa958131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): picked mon.noname-c con 0x7fa958131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa95809e910 0x7fa95809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): picked mon.noname-a con 0x7fa95809e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809f250 0x7fa958068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): picked mon.noname-b con 0x7fa95809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95809faf0 con 0x7fa95809e910 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95809fd30 con 0x7fa958131ab0 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580a1110 con 0x7fa95809f250 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95e59d640 10 monclient(hunting): authenticate will time out at 1467.474550s 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809f250 0x7fa958068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa95809e910 0x7fa95809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa957fff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fa958131ab0 msgr2=0x7fa958131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809f250 0x7fa958068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59462/0 (socket says 172.21.15.116:59462) 2024-09-18T12:37:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): get_auth_request con 0x7fa95809e910 auth_method 0 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 1 -- 172.21.15.116:0/2109976071 learned_addr learned my addr 172.21.15.116:0/2109976071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa957fff640 1 --2- 172.21.15.116:0/2109976071 >> v2:172.21.15.144:3300/0 conn(0x7fa958131ab0 0x7fa958131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 10 monclient(hunting): get_auth_request con 0x7fa95809f250 auth_method 0 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa95cb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 10 monclient(hunting): handle_auth_done global_id 154173 payload 274 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 1 -- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809f250 msgr2=0x7fa958068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.153+0000 7fa9577fe640 1 --2- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809f250 0x7fa958068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.154+0000 7fa9577fe640 1 -- 172.21.15.116:0/2109976071 >> v2:172.21.15.144:3300/0 conn(0x7fa958131ab0 msgr2=0x7fa958131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.154+0000 7fa9577fe640 1 --2- 172.21.15.116:0/2109976071 >> v2:172.21.15.144:3300/0 conn(0x7fa958131ab0 0x7fa958131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.154+0000 7fa9577fe640 10 monclient: _finish_hunting 0 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.154+0000 7fa9577fe640 1 monclient: found mon.noname-a 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.154+0000 7fa9577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.154+0000 7fa9577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.154+0000 7fa9577fe640 1 -- 172.21.15.116:0/2109976071 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9580a13f0 con 0x7fa95809e910 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.156+0000 7fa9577fe640 1 -- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809e910 msgr2=0x7fa95809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.156+0000 7fa9577fe640 1 --2- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809e910 0x7fa95809ed10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa958076760 tx=0x7fa9400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3da38c208226357 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.156+0000 7fa956ffd640 1 -- 172.21.15.116:0/2109976071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94002aea0 con 0x7fa95809e910 2024-09-18T12:37:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.156+0000 7fa956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.156+0000 7fa956ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.156+0000 7fa956ffd640 10 monclient: dump: 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.167 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 10 monclient: _finish_auth 0 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 5 monclient: authenticate success, global_id 154173 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 1 -- 172.21.15.116:0/2109976071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa940032440 con 0x7fa95809e910 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 10 monclient: get_monmap_and_config success 2024-09-18T12:37:20.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 4 set_mon_vals no callback set 2024-09-18T12:37:20.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 1 -- 172.21.15.116:0/2109976071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9400327e0 con 0x7fa95809e910 2024-09-18T12:37:20.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:20.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa956ffd640 10 monclient: dump: 2024-09-18T12:37:20.169 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.169 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.169 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 10 monclient: shutdown 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 -- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809e910 msgr2=0x7fa95809ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 --2- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809e910 0x7fa95809ed10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa958076760 tx=0x7fa9400090f0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 -- 172.21.15.116:0/2109976071 shutdown_connections 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 --2- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809f250 0x7fa958068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 --2- 172.21.15.116:0/2109976071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809e910 0x7fa95809ed10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 --2- 172.21.15.116:0/2109976071 >> v2:172.21.15.144:3300/0 conn(0x7fa958131ab0 0x7fa958131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 -- 172.21.15.116:0/2109976071 >> 172.21.15.116:0/2109976071 conn(0x7fa958067d30 msgr2=0x7fa9580a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 -- 172.21.15.116:0/2109976071 shutdown_connections 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.157+0000 7fa95e59d640 1 -- 172.21.15.116:0/2109976071 wait complete. 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.158+0000 7fa95e59d640 10 monclient: build_initial_monmap 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.158+0000 7fa95e59d640 10 monclient: monmap: 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:20.171 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.158+0000 7fa95e59d640 1 Processor -- start 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.158+0000 7fa95e59d640 1 -- start start 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.158+0000 7fa95e59d640 10 monclient: init 2024-09-18T12:37:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.158+0000 7fa95e59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809e910 0x7fa95809abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): picked mon.noname-c con 0x7fa95809e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 0x7fa958130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): picked mon.noname-a con 0x7fa95809f250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa958131ab0 0x7fa958131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): picked mon.noname-b con 0x7fa958131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580a4430 con 0x7fa95809f250 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95811de70 con 0x7fa958131ab0 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95811e150 con 0x7fa95809e910 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95e59d640 10 monclient(hunting): authenticate will time out at 1467.480072s 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809e910 0x7fa95809abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 0x7fa958130d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809e910 0x7fa95809abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:59470/0 (socket says 172.21.15.116:59470) 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 1 -- 172.21.15.116:0/1326469459 learned_addr learned my addr 172.21.15.116:0/1326469459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 0x7fa958130d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51976/0 (socket says 172.21.15.116:51976) 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 10 monclient(hunting): get_auth_request con 0x7fa95809e910 auth_method 0 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): get_auth_request con 0x7fa95809f250 auth_method 0 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95cb3a640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa958131ab0 msgr2=0x7fa958131290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa95cb3a640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa958131ab0 0x7fa958131290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient(hunting): handle_auth_done global_id 154176 payload 824 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809e910 msgr2=0x7fa95809abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809e910 0x7fa95809abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa958131ab0 msgr2=0x7fa958131290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa958131ab0 0x7fa958131290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient: _finish_hunting 0 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 monclient: found mon.noname-a 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:37:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa9577fe640 1 -- 172.21.15.116:0/1326469459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa940009390 con 0x7fa95809f250 2024-09-18T12:37:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.159+0000 7fa957fff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95809e910 0x7fa95809abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.161+0000 7fa9577fe640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 msgr2=0x7fa958130d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.161+0000 7fa9577fe640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 0x7fa958130d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa940031000 tx=0x7fa940032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94002aea0 con 0x7fa95809f250 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: dump: 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: _finish_auth 0 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa940031c70 con 0x7fa95809f250 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 5 monclient: authenticate success, global_id 154176 2024-09-18T12:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 10 monclient: _renew_subs 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95811e430 con 0x7fa95809f250 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94003d440 con 0x7fa95809f250 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa94ffff640 10 monclient: dump: 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa9567fc640 4 set_mon_vals no callback set 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 10 monclient: _renew_subs 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa958141570 con 0x7fa95809f250 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.162+0000 7fa95e59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa95e59d640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 0x7fa95811ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa958063740 con 0x7fa95811b6b0 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa940039040 con 0x7fa95809f250 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 0x7fa95811ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient: get_auth_request con 0x7fa95811b6b0 auth_method 0 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 10 monclient(hunting): handle_auth_done global_id 144269 payload 824 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa94ffff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa938038440 0x7fa93803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa957fff640 1 -- 172.21.15.116:0/1326469459 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa95811dfe0 con 0x7fa95811b6b0 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa94006bb10 con 0x7fa95809f250 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa95cb3a640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa938038440 0x7fa93803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.163+0000 7fa95cb3a640 10 monclient: get_auth_request con 0x7fa938038440 auth_method 0 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa957fff640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 msgr2=0x7fa95811ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa957fff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 0x7fa95811ba80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9480063e0 tx=0x7fa9480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9480123f0 con 0x7fa95811b6b0 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 10 monclient: dump: 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.188 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa948012550 con 0x7fa95811b6b0 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 msgr2=0x7fa95811ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 0x7fa95811ba80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9480063e0 tx=0x7fa9480091a0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 msgr2=0x7fa95811ba80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:37:20.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.165+0000 7fa94ffff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811b6b0 0x7fa95811ba80 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa9480063e0 tx=0x7fa9480091a0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.166+0000 7fa95cb3a640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa938038440 0x7fa93803a8f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa9581349c0 tx=0x7fa95000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa95e59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa95e59d640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 0x7fa958096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa95e59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95804bf60 con 0x7fa95811bfc0 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 0x7fa958096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 10 monclient: get_auth_request con 0x7fa95811bfc0 auth_method 0 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:20.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.171+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.172+0000 7fa957fff640 10 monclient(hunting): handle_auth_done global_id 144272 payload 824 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.172+0000 7fa957fff640 1 -- 172.21.15.116:0/1326469459 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa9580a5270 con 0x7fa95811bfc0 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa957fff640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 msgr2=0x7fa958096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa957fff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 0x7fa958096a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9480063e0 tx=0x7fa948015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948025070 con 0x7fa95811bfc0 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 10 monclient: dump: 2024-09-18T12:37:20.191 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7fa9480202c0 con 0x7fa95811bfc0 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:37:20.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 msgr2=0x7fa958096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 0x7fa958096a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9480063e0 tx=0x7fa948015000 comp rx=0 tx=0).stop 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 msgr2=0x7fa958096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa94ffff640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa95811bfc0 0x7fa958096a60 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9480063e0 tx=0x7fa948015000 comp rx=0 tx=0).stop 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:37:20.193 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 118, 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:37:20.194 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:37:20.195 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:37:20.196 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:37:20.197 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:37:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:37:20.199 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:37:20.200 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:37:20.201 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:37:20.202 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:37:20.203 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:37:20.204 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:37:20.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:37:20.206 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:37:20.207 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:37:20.207 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:37:20.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:37:20.207 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:37:20.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:37:20.207 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:37:20.207 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa938038440 msgr2=0x7fa93803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa95e59d640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa938038440 0x7fa93803a8f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa9581349c0 tx=0x7fa95000b040 comp rx=0 tx=0).stop 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa95e59d640 10 monclient: shutdown 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa95e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 msgr2=0x7fa958130d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa95e59d640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 0x7fa958130d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa940031000 tx=0x7fa940032000 comp rx=0 tx=0).stop 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.174+0000 7fa95cb3a640 1 -- 172.21.15.116:0/1326469459 reap_dead start 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.175+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 shutdown_connections 2024-09-18T12:37:20.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.175+0000 7fa95e59d640 1 --2- 172.21.15.116:0/1326469459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa95809f250 0x7fa958130d50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:20.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.175+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 >> 172.21.15.116:0/1326469459 conn(0x7fa958067d30 msgr2=0x7fa95809d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:20.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.175+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 shutdown_connections 2024-09-18T12:37:20.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:20.175+0000 7fa95e59d640 1 -- 172.21.15.116:0/1326469459 wait complete. 2024-09-18T12:37:20.210 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:37:20.210 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:37:40.211 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:37:40.212 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T12:37:40.212 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T12:37:40.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T12:37:40.215 INFO:tasks.ceph.mon.b:Started 2024-09-18T12:37:40.215 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T12:37:40.216 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T12:37:40.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:37:55.217 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T12:37:55.219 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T12:37:55.219 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:37:55.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.321+0000 7ff08809f640 10 monclient: get_monmap_and_config 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.321+0000 7ff08809f640 10 monclient: build_initial_monmap 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.321+0000 7ff08809f640 10 monclient: monmap: 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:37:55.322591+0000 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:37:55.322591+0000 2024-09-18T12:37:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.322+0000 7ff08809f640 1 Processor -- start 2024-09-18T12:37:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.322+0000 7ff08809f640 1 -- start start 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.322+0000 7ff08809f640 10 monclient: init 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient(hunting): picked mon.noname-b con 0x7ff0800a1890 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0800975c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient(hunting): picked mon.noname-a con 0x7ff080136c80 addr v2:172.21.15.116:3300/0 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08006e900 con 0x7ff0800975c0 2024-09-18T12:37:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff080098120 con 0x7ff080136c80 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800a1c60 con 0x7ff0800a1890 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08809f640 10 monclient(hunting): authenticate will time out at 1502.644270s 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59298/0 (socket says 172.21.15.116:59298) 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36732/0 (socket says 172.21.15.116:36732) 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 learned_addr learned my addr 172.21.15.116:0/2472308342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 10 monclient(hunting): get_auth_request con 0x7ff080136c80 auth_method 0 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 10 monclient(hunting): get_auth_request con 0x7ff0800a1890 auth_method 0 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:55.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.323+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:55.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.325+0000 7ff08563a640 1 -- 172.21.15.116:0/2472308342 >> v1:172.21.15.144:6789/0 conn(0x7ff0800975c0 legacy=0x7ff080097a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:55.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:55.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:55.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 10 monclient(hunting): get_auth_request con 0x7ff080136c80 auth_method 0 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 10 monclient(hunting): get_auth_request con 0x7ff0800a1890 auth_method 0 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:55.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:55.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:55.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:37:55.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.524+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:37:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 10 monclient(hunting): get_auth_request con 0x7ff080136c80 auth_method 0 2024-09-18T12:37:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 10 monclient(hunting): get_auth_request con 0x7ff0800a1890 auth_method 0 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:37:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:55.925+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:37:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 10 monclient(hunting): get_auth_request con 0x7ff080136c80 auth_method 0 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 10 monclient(hunting): get_auth_request con 0x7ff0800a1890 auth_method 0 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:37:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:37:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:56.726+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 10 monclient(hunting): tick 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 msgr2=0x7ff08012eae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff0800a1890 0x7ff08012eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 msgr2=0x7ff080139060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.116:3300/0 conn(0x7ff080136c80 0x7ff080139060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 -- 172.21.15.116:0/2472308342 >> v1:172.21.15.144:6789/0 conn(0x7ff0800975c0 legacy=0x7ff080097a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 10 monclient: _add_conns ranks=[0,4] 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff064003700 addr v1:172.21.15.116:6789/0 2024-09-18T12:37:58.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff064007420 addr v2:172.21.15.144:3300/0 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 -- 172.21.15.116:0/2472308342 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0800641d0 con 0x7ff064003700 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 1 -- 172.21.15.116:0/2472308342 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff080098120 con 0x7ff064007420 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff0777fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 10 monclient(hunting): get_auth_request con 0x7ff064007420 auth_method 0 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 msgr2=0x7ff0640098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 msgr2=0x7ff0640098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.323+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.325+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> v1:172.21.15.116:6789/0 conn(0x7ff064003700 legacy=0x7ff064005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 10 monclient(hunting): get_auth_request con 0x7ff064007420 auth_method 0 2024-09-18T12:37:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 msgr2=0x7ff0640098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:37:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 msgr2=0x7ff0640098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.524+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:37:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 10 monclient(hunting): get_auth_request con 0x7ff064007420 auth_method 0 2024-09-18T12:37:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 msgr2=0x7ff0640098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:37:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 msgr2=0x7ff0640098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:37:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:58.925+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:37:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167170263 0 0) 0x7ff0800641d0 con 0x7ff064003700 2024-09-18T12:37:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): my global_id is 154179 2024-09-18T12:37:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff06c0034e0 con 0x7ff064003700 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224062567 0 0) 0x7ff06c0034e0 con 0x7ff064003700 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 msgr2=0x7ff0640098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 --2- 172.21.15.116:0/2472308342 >> v2:172.21.15.144:3300/0 conn(0x7ff064007420 0x7ff0640098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: _finish_hunting 0 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06400b000 con 0x7ff064003700 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: _finish_auth 0 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7ff07000d480 con 0x7ff064003700 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: dump: 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:37:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 >> v1:172.21.15.116:6789/0 conn(0x7ff064003700 legacy=0x7ff064005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff08663c640 1 -- 172.21.15.116:0/2472308342 reap_dead start 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c003ac0 0x7ff06c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): picked mon.a con 0x7ff06c003ac0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c004780 0x7ff06c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): picked mon.c con 0x7ff06c004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c005330 0x7ff06c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): picked mon.b con 0x7ff06c005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080098120 con 0x7ff06c003ac0 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c00c2f0 con 0x7ff06c005330 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c00c4b0 con 0x7ff06c004780 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff084e39640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c003ac0 0x7ff06c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.618+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c004780 0x7ff06c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): get_auth_request con 0x7ff06c003ac0 auth_method 0 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 10 monclient(hunting): get_auth_request con 0x7ff06c004780 auth_method 0 2024-09-18T12:37:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c005330 0x7ff06c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 10 monclient(hunting): get_auth_request con 0x7ff06c005330 auth_method 0 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08663c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient(hunting): handle_auth_done global_id 154179 payload 293 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c004780 msgr2=0x7ff06c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c004780 0x7ff06c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c005330 msgr2=0x7ff06c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c005330 0x7ff06c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient: _finish_hunting 0 2024-09-18T12:37:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 1 monclient: found mon.a 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c0034e0 con 0x7ff06c003ac0 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 10 monclient: _finish_auth 0 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff085e3b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08809f640 5 monclient: authenticate success, global_id 154179 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08809f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:37:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.619+0000 7ff08563a640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c004780 0x7ff06c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:37:59.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff085e3b640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c003ac0 msgr2=0x7ff06c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:59.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff085e3b640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c003ac0 0x7ff06c003eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff070006fd0 tx=0x7ff070008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574556a5b5635e1a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070034410 con 0x7ff06c003ac0 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 10 monclient: dump: 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff08809f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07002e030 con 0x7ff06c003ac0 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff08809f640 10 monclient: get_monmap_and_config success 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff08809f640 4 set_mon_vals no callback set 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 1 -- 172.21.15.116:0/2472308342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070034800 con 0x7ff06c003ac0 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.622+0000 7ff084e39640 10 monclient: dump: 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 10 monclient: shutdown 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 -- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c003ac0 msgr2=0x7ff06c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c003ac0 0x7ff06c003eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff070006fd0 tx=0x7ff070008040 comp rx=0 tx=0).stop 2024-09-18T12:37:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 -- 172.21.15.116:0/2472308342 shutdown_connections 2024-09-18T12:37:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c005330 0x7ff06c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c004780 0x7ff06c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 --2- 172.21.15.116:0/2472308342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c003ac0 0x7ff06c003eb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 -- 172.21.15.116:0/2472308342 >> 172.21.15.116:0/2472308342 conn(0x7ff080067b50 msgr2=0x7ff0800a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 -- 172.21.15.116:0/2472308342 shutdown_connections 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.623+0000 7ff08809f640 1 -- 172.21.15.116:0/2472308342 wait complete. 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.624+0000 7ff08809f640 10 monclient: build_initial_monmap 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.624+0000 7ff08809f640 10 monclient: monmap: 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.624+0000 7ff08809f640 1 Processor -- start 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.624+0000 7ff08809f640 1 -- start start 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.624+0000 7ff08809f640 10 monclient: init 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient: _reopen_session rank -1 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 0x7ff080127ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): picked mon.noname-b con 0x7ff06c003ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 0x7ff080128220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): picked mon.noname-a con 0x7ff06c004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 0x7ff080128760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): picked mon.noname-c con 0x7ff06c005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080135b10 con 0x7ff06c004780 2024-09-18T12:37:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08006ee90 con 0x7ff06c003ac0 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff080096690 con 0x7ff06c005330 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): _renew_subs 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08809f640 10 monclient(hunting): authenticate will time out at 1506.946110s 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 0x7ff080128220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 0x7ff080128220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50588/0 (socket says 172.21.15.116:50588) 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 0x7ff080128760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 -- 172.21.15.116:0/1645670791 learned_addr learned my addr 172.21.15.116:0/1645670791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): get_auth_request con 0x7ff06c004780 auth_method 0 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 10 monclient(hunting): get_auth_request con 0x7ff06c005330 auth_method 0 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 0x7ff080127ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 10 monclient(hunting): get_auth_request con 0x7ff06c003ac0 auth_method 0 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 msgr2=0x7ff080128760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:37:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 msgr2=0x7ff080128760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 0x7ff080128760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08663c640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 0x7ff080128760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 msgr2=0x7ff080127ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 msgr2=0x7ff080127ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 0x7ff080127ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff085e3b640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 0x7ff080127ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient(hunting): handle_auth_done global_id 154182 payload 824 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 msgr2=0x7ff080128760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 0x7ff080128760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 msgr2=0x7ff080127ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 0x7ff080127ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient: _finish_hunting 0 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 monclient: found mon.noname-a 2024-09-18T12:37:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.625+0000 7ff08563a640 1 -- 172.21.15.116:0/1645670791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080129780 con 0x7ff06c004780 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.627+0000 7ff08563a640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 msgr2=0x7ff080128220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.627+0000 7ff08563a640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 0x7ff080128220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0013a0 tx=0x7ff07c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 1 -- 172.21.15.116:0/1645670791 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c012e60 con 0x7ff06c004780 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: dump: 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: _finish_auth 0 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 1 -- 172.21.15.116:0/1645670791 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c0070d0 con 0x7ff06c004780 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff08809f640 5 monclient: authenticate success, global_id 154182 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff08809f640 10 monclient: _renew_subs 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff08809f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08012b2b0 con 0x7ff06c004780 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:37:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 1 -- 172.21.15.116:0/1645670791 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c007450 con 0x7ff06c004780 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff075ffb640 10 monclient: dump: 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff077fff640 4 set_mon_vals no callback set 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff08809f640 10 monclient: _renew_subs 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff08809f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.628+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08012bf70 con 0x7ff06c004780 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.629+0000 7ff075ffb640 1 -- 172.21.15.116:0/1645670791 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff07c005990 con 0x7ff06c004780 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.629+0000 7ff08809f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.629+0000 7ff08809f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.629+0000 7ff08809f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.629+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08004bf60 con 0x7ff06c004780 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.629+0000 7ff075ffb640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff064038f30 0x7ff06403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.629+0000 7ff075ffb640 1 -- 172.21.15.116:0/1645670791 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff07c04cb20 con 0x7ff06c004780 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.630+0000 7ff085e3b640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff064038f30 0x7ff06403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:37:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.630+0000 7ff085e3b640 10 monclient: get_auth_request con 0x7ff064038f30 auth_method 0 2024-09-18T12:37:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.632+0000 7ff085e3b640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff064038f30 0x7ff06403b3e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff070006ea0 tx=0x7ff070039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:37:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.634+0000 7ff075ffb640 1 -- 172.21.15.116:0/1645670791 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff07c016420 con 0x7ff06c004780 2024-09-18T12:37:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.634+0000 7ff075ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:37:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.634+0000 7ff075ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:37:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.850+0000 7ff08809f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:37:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.850+0000 7ff08809f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:37:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.850+0000 7ff08809f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:37:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.850+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff080063b60 con 0x7ff06c004780 2024-09-18T12:37:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.851+0000 7ff075ffb640 1 -- 172.21.15.116:0/1645670791 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7ff07c024020 con 0x7ff06c004780 2024-09-18T12:37:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.851+0000 7ff075ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:37:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.851+0000 7ff075ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:37:59.856 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":120,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff064038f30 msgr2=0x7ff06403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff064038f30 0x7ff06403b3e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff070006ea0 tx=0x7ff070039000 comp rx=0 tx=0).stop 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 10 monclient: shutdown 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 msgr2=0x7ff080128220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 0x7ff080128220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0013a0 tx=0x7ff07c009040 comp rx=0 tx=0).stop 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 shutdown_connections 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff064038f30 0x7ff06403b3e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff06c005330 0x7ff080128760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c004780 0x7ff080128220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 --2- 172.21.15.116:0/1645670791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff06c003ac0 0x7ff080127ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 >> 172.21.15.116:0/1645670791 conn(0x7ff080067b50 msgr2=0x7ff080095530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:37:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 shutdown_connections 2024-09-18T12:37:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:37:59.853+0000 7ff08809f640 1 -- 172.21.15.116:0/1645670791 wait complete. 2024-09-18T12:38:02.865 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.968+0000 7ff05e51d640 10 monclient: get_monmap_and_config 2024-09-18T12:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.968+0000 7ff05e51d640 10 monclient: build_initial_monmap 2024-09-18T12:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.968+0000 7ff05e51d640 10 monclient: monmap: 2024-09-18T12:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:02.969116+0000 2024-09-18T12:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:02.969116+0000 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 1 Processor -- start 2024-09-18T12:38:02.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 1 -- start start 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient: init 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff0580981c0 0x7ff058098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0580981c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff058131da0 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff058067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient(hunting): picked mon.noname-b con 0x7ff058132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff058098ad0 con 0x7ff058131da0 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff058131bd0 con 0x7ff0580981c0 2024-09-18T12:38:02.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05809ca60 con 0x7ff058132170 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05e51d640 10 monclient(hunting): authenticate will time out at 1510.290760s 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05caba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff058067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05caba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff058067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33134/0 (socket says 172.21.15.116:33134) 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05caba640 1 -- 172.21.15.116:0/1553257506 learned_addr learned my addr 172.21.15.116:0/1553257506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05caba640 10 monclient(hunting): get_auth_request con 0x7ff058132170 auth_method 0 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05caba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05caba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.969+0000 7ff05caba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:02.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff057fff640 1 --2- 172.21.15.116:0/1553257506 >> v2:172.21.15.144:3300/0 conn(0x7ff0580981c0 0x7ff058098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff057fff640 10 monclient(hunting): get_auth_request con 0x7ff0580981c0 auth_method 0 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 10 monclient(hunting): handle_auth_done global_id 144275 payload 274 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 1 -- 172.21.15.116:0/1553257506 >> v2:172.21.15.144:3300/0 conn(0x7ff0580981c0 msgr2=0x7ff058098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 1 --2- 172.21.15.116:0/1553257506 >> v2:172.21.15.144:3300/0 conn(0x7ff0580981c0 0x7ff058098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 1 -- 172.21.15.116:0/1553257506 >> v1:172.21.15.144:6789/0 conn(0x7ff058131da0 legacy=0x7ff058067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 10 monclient: _finish_hunting 0 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 1 monclient: found mon.noname-b 2024-09-18T12:38:02.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:02.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:38:02.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff05caba640 1 -- 172.21.15.116:0/1553257506 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05809cca0 con 0x7ff058132170 2024-09-18T12:38:02.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.970+0000 7ff057fff640 1 --2- 172.21.15.116:0/1553257506 >> v2:172.21.15.144:3300/0 conn(0x7ff0580981c0 0x7ff058098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:38:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05caba640 1 -- 172.21.15.116:0/1553257506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 msgr2=0x7ff058067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05caba640 1 --2- 172.21.15.116:0/1553257506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff058067820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff058066560 tx=0x7ff050009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49d3ba39af809372 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 1 -- 172.21.15.116:0/1553257506 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050007610 con 0x7ff058132170 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: dump: 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:02.984 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: _finish_auth 0 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 5 monclient: authenticate success, global_id 144275 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 1 -- 172.21.15.116:0/1553257506 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05002d050 con 0x7ff058132170 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 4 set_mon_vals no callback set 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 1 -- 172.21.15.116:0/1553257506 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0500079a0 con 0x7ff058132170 2024-09-18T12:38:02.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff056ffd640 10 monclient: dump: 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:02.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 10 monclient: shutdown 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 1 -- 172.21.15.116:0/1553257506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 msgr2=0x7ff058067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 1 --2- 172.21.15.116:0/1553257506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff058067820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff058066560 tx=0x7ff050009840 comp rx=0 tx=0).stop 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 1 -- 172.21.15.116:0/1553257506 shutdown_connections 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 1 --2- 172.21.15.116:0/1553257506 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff058067820 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 1 --2- 172.21.15.116:0/1553257506 >> v2:172.21.15.144:3300/0 conn(0x7ff0580981c0 0x7ff058098590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 1 -- 172.21.15.116:0/1553257506 >> 172.21.15.116:0/1553257506 conn(0x7ff0580a1e30 msgr2=0x7ff0580a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.973+0000 7ff05e51d640 1 -- 172.21.15.116:0/1553257506 shutdown_connections 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.974+0000 7ff05e51d640 1 -- 172.21.15.116:0/1553257506 wait complete. 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.974+0000 7ff05e51d640 10 monclient: build_initial_monmap 2024-09-18T12:38:02.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.974+0000 7ff05e51d640 10 monclient: monmap: 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:02.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.974+0000 7ff05e51d640 1 Processor -- start 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.974+0000 7ff05e51d640 1 -- start start 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient: init 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0580981c0 0x7ff058140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): picked mon.noname-a con 0x7ff0580981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff058131da0 0x7ff058130820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): picked mon.noname-b con 0x7ff058131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff05811a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): picked mon.noname-c con 0x7ff058132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:02.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05811dc20 con 0x7ff0580981c0 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0580a0a70 con 0x7ff058131da0 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05811d940 con 0x7ff058132170 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05e51d640 10 monclient(hunting): authenticate will time out at 1510.296463s 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff05811a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0580981c0 0x7ff058140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff05811a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33138/0 (socket says 172.21.15.116:33138) 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 -- 172.21.15.116:0/4260063173 learned_addr learned my addr 172.21.15.116:0/4260063173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): get_auth_request con 0x7ff058132170 auth_method 0 2024-09-18T12:38:02.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 10 monclient(hunting): get_auth_request con 0x7ff0580981c0 auth_method 0 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff058131da0 0x7ff058130820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 10 monclient(hunting): get_auth_request con 0x7ff058131da0 auth_method 0 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:02.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff0577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient(hunting): handle_auth_done global_id 144278 payload 824 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 -- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff058131da0 msgr2=0x7ff058130820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff058131da0 0x7ff058130820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 -- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0580981c0 msgr2=0x7ff058140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:02.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0580981c0 0x7ff058140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient: _finish_hunting 0 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 monclient: found mon.noname-c 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff05caba640 1 -- 172.21.15.116:0/4260063173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050009ae0 con 0x7ff058132170 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.975+0000 7ff057fff640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0580981c0 0x7ff058140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05caba640 1 -- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 msgr2=0x7ff05811a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05caba640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff05811a2d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff050009ab0 tx=0x7ff050036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 1 -- 172.21.15.116:0/4260063173 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050032ce0 con 0x7ff058132170 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: dump: 2024-09-18T12:38:02.993 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: _finish_auth 0 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05e51d640 5 monclient: authenticate success, global_id 144278 2024-09-18T12:38:02.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05e51d640 10 monclient: _renew_subs 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 1 -- 172.21.15.116:0/4260063173 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff050032e40 con 0x7ff058132170 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05e51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff058141310 con 0x7ff058132170 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 1 -- 172.21.15.116:0/4260063173 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05003b640 con 0x7ff058132170 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff03ffff640 10 monclient: dump: 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:02.995 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05e51d640 10 monclient: _renew_subs 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05e51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05811cb80 con 0x7ff058132170 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.978+0000 7ff0567fc640 4 set_mon_vals no callback set 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff03ffff640 1 -- 172.21.15.116:0/4260063173 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff050038050 con 0x7ff058132170 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff05e51d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff05e51d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff05e51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:02.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff058098d20 con 0x7ff058132170 2024-09-18T12:38:02.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff03ffff640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff044038490 0x7ff04403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:02.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff03ffff640 1 -- 172.21.15.116:0/4260063173 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff05006cdf0 con 0x7ff058132170 2024-09-18T12:38:02.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff057fff640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff044038490 0x7ff04403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:02.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.979+0000 7ff057fff640 10 monclient: get_auth_request con 0x7ff044038490 auth_method 0 2024-09-18T12:38:02.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.982+0000 7ff057fff640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff044038490 0x7ff04403a940 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff048006e70 tx=0x7ff048008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:02.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.983+0000 7ff03ffff640 1 -- 172.21.15.116:0/4260063173 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ff058098d20 con 0x7ff058132170 2024-09-18T12:38:02.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.983+0000 7ff03ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:02.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:02.983+0000 7ff03ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:03.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.197+0000 7ff05e51d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:38:03.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.197+0000 7ff05e51d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:38:03.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.197+0000 7ff05e51d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.197+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff058098d20 con 0x7ff058132170 2024-09-18T12:38:03.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.197+0000 7ff03ffff640 1 -- 172.21.15.116:0/4260063173 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7ff058098d20 con 0x7ff058132170 2024-09-18T12:38:03.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.197+0000 7ff03ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:38:03.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.197+0000 7ff03ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:03.199 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:38:03.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff044038490 msgr2=0x7ff04403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff044038490 0x7ff04403a940 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff048006e70 tx=0x7ff048008040 comp rx=0 tx=0).stop 2024-09-18T12:38:03.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 10 monclient: shutdown 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 msgr2=0x7ff05811a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff05811a2d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff050009ab0 tx=0x7ff050036000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 shutdown_connections 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff044038490 0x7ff04403a940 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff058132170 0x7ff05811a2d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff058131da0 0x7ff058130820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 --2- 172.21.15.116:0/4260063173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0580981c0 0x7ff058140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 >> 172.21.15.116:0/4260063173 conn(0x7ff0580a1e30 msgr2=0x7ff0581349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 shutdown_connections 2024-09-18T12:38:03.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.199+0000 7ff05e51d640 1 -- 172.21.15.116:0/4260063173 wait complete. 2024-09-18T12:38:03.211 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:38:03.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:38:03.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:38:03.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.304+0000 7efc97f1d640 10 monclient: get_monmap_and_config 2024-09-18T12:38:03.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.304+0000 7efc97f1d640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.304+0000 7efc97f1d640 10 monclient: monmap: 2024-09-18T12:38:03.305 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.305 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.305 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:03.305172+0000 2024-09-18T12:38:03.305 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:03.305172+0000 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:03.306 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 1 Processor -- start 2024-09-18T12:38:03.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 1 -- start start 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient: init 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc90067110 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc9009b010 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc9009b430 0x7efc901380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient(hunting): picked mon.noname-b con 0x7efc9009b430 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc90130540 con 0x7efc90067110 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc90099800 con 0x7efc9009b010 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc90067ed0 con 0x7efc9009b430 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc97f1d640 10 monclient(hunting): authenticate will time out at 1510.626759s 2024-09-18T12:38:03.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc9009b430 0x7efc901380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc95cb9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efc90067110 0x7efc9012f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45148/0 (socket says 172.21.15.116:45148) 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc95cb9640 1 -- 172.21.15.116:0/4064565413 learned_addr learned my addr 172.21.15.116:0/4064565413 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 10 monclient(hunting): get_auth_request con 0x7efc9009b430 auth_method 0 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.305+0000 7efc964ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 10 monclient(hunting): handle_auth_done global_id 144281 payload 274 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 1 -- 172.21.15.116:0/4064565413 >> v1:172.21.15.144:6789/0 conn(0x7efc9009b010 legacy=0x7efc901349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 1 -- 172.21.15.116:0/4064565413 >> v1:172.21.15.116:6789/0 conn(0x7efc90067110 legacy=0x7efc9012f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 1 monclient: found mon.noname-b 2024-09-18T12:38:03.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.306+0000 7efc964ba640 1 -- 172.21.15.116:0/4064565413 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc900674e0 con 0x7efc9009b430 2024-09-18T12:38:03.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.308+0000 7efc964ba640 1 -- 172.21.15.116:0/4064565413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc9009b430 msgr2=0x7efc901380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.308+0000 7efc964ba640 1 --2- 172.21.15.116:0/4064565413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc9009b430 0x7efc901380d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc80009880 tx=0x7efc800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84b8367e21248e1a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 1 -- 172.21.15.116:0/4064565413 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc80032440 con 0x7efc9009b430 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: dump: 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 5 monclient: authenticate success, global_id 144281 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 1 -- 172.21.15.116:0/4064565413 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc800325a0 con 0x7efc9009b430 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 4 set_mon_vals no callback set 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 1 -- 172.21.15.116:0/4064565413 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc80032980 con 0x7efc9009b430 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc94cb7640 10 monclient: dump: 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.315 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 10 monclient: shutdown 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 1 -- 172.21.15.116:0/4064565413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc9009b430 msgr2=0x7efc901380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 1 --2- 172.21.15.116:0/4064565413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc9009b430 0x7efc901380d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc80009880 tx=0x7efc800090f0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 1 -- 172.21.15.116:0/4064565413 shutdown_connections 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 1 --2- 172.21.15.116:0/4064565413 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc9009b430 0x7efc901380d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 1 -- 172.21.15.116:0/4064565413 >> 172.21.15.116:0/4064565413 conn(0x7efc900a1e30 msgr2=0x7efc900a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 1 -- 172.21.15.116:0/4064565413 shutdown_connections 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.309+0000 7efc97f1d640 1 -- 172.21.15.116:0/4064565413 wait complete. 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient: monmap: 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:03.317 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 Processor -- start 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 -- start start 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient: init 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 0x7efc9012f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient(hunting): picked mon.noname-c con 0x7efc90067110 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc9009b010 0x7efc90096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient(hunting): picked mon.noname-b con 0x7efc9009b010 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc9009b430 0x7efc90096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient(hunting): picked mon.noname-a con 0x7efc9009b430 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9011a440 con 0x7efc9009b430 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc90119a60 con 0x7efc9009b010 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.310+0000 7efc97f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc900a04c0 con 0x7efc90067110 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc97f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc97f1d640 10 monclient(hunting): authenticate will time out at 1510.632009s 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 0x7efc9012f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 0x7efc9012f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33148/0 (socket says 172.21.15.116:33148) 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 -- 172.21.15.116:0/1083669706 learned_addr learned my addr 172.21.15.116:0/1083669706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): get_auth_request con 0x7efc90067110 auth_method 0 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc9009b430 0x7efc90096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 10 monclient(hunting): get_auth_request con 0x7efc9009b430 auth_method 0 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc964ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient(hunting): handle_auth_done global_id 144284 payload 824 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc9009b010 msgr2=0x7efc90096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efc9009b010 0x7efc90096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc9009b430 msgr2=0x7efc90096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc9009b430 0x7efc90096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 monclient: found mon.noname-c 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.311+0000 7efc95cb9640 1 -- 172.21.15.116:0/1083669706 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc80009390 con 0x7efc90067110 2024-09-18T12:38:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc95cb9640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 msgr2=0x7efc9012f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc95cb9640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 0x7efc9012f790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efc8800c610 tx=0x7efc88004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc88005cb0 con 0x7efc90067110 2024-09-18T12:38:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: dump: 2024-09-18T12:38:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc88005e30 con 0x7efc90067110 2024-09-18T12:38:03.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc97f1d640 5 monclient: authenticate success, global_id 144284 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc97f1d640 10 monclient: _renew_subs 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc97f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9012ed40 con 0x7efc90067110 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc88007320 con 0x7efc90067110 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8dffb640 10 monclient: dump: 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.331 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc97f1d640 10 monclient: _renew_subs 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc8ffff640 4 set_mon_vals no callback set 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc97f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.313+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc90098430 con 0x7efc90067110 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc97f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc97f1d640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 0x7efc901306d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc97f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7efc8801b3b0 con 0x7efc90067110 2024-09-18T12:38:03.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc90119920 con 0x7efc900974e0 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 0x7efc901306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 10 monclient: get_auth_request con 0x7efc900974e0 auth_method 0 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc954b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc8dffb640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efc7c0384c0 0x7efc7c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.314+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efc8804b810 con 0x7efc90067110 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.315+0000 7efc954b8640 10 monclient(hunting): handle_auth_done global_id 154191 payload 824 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.315+0000 7efc954b8640 1 -- 172.21.15.116:0/1083669706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc900978b0 con 0x7efc900974e0 2024-09-18T12:38:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.315+0000 7efc964ba640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efc7c0384c0 0x7efc7c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.315+0000 7efc964ba640 10 monclient: get_auth_request con 0x7efc7c0384c0 auth_method 0 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc954b8640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 msgr2=0x7efc901306d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc954b8640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 0x7efc901306d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efc9009dee0 tx=0x7efc84002630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8400a940 con 0x7efc900974e0 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 10 monclient: dump: 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efc8400aaa0 con 0x7efc900974e0 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 msgr2=0x7efc901306d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 0x7efc901306d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efc9009dee0 tx=0x7efc84002630 comp rx=0 tx=0).stop 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 msgr2=0x7efc901306d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc8dffb640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc900974e0 0x7efc901306d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.317+0000 7efc964ba640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efc7c0384c0 0x7efc7c03a970 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efc900681f0 tx=0x7efc80032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc97f1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc97f1d640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 0x7efc90131010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc97f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc90065f90 con 0x7efc90130c10 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 0x7efc90131010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient: get_auth_request con 0x7efc90130c10 auth_method 0 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 10 monclient(hunting): handle_auth_done global_id 154194 payload 824 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.323+0000 7efc954b8640 1 -- 172.21.15.116:0/1083669706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efc90119ca0 con 0x7efc90130c10 2024-09-18T12:38:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.325+0000 7efc954b8640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 msgr2=0x7efc90131010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.325+0000 7efc954b8640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 0x7efc90131010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efc840028a0 tx=0x7efc84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc84012b10 con 0x7efc90130c10 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 10 monclient: dump: 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7efc84024070 con 0x7efc90130c10 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 msgr2=0x7efc90131010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 0x7efc90131010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efc840028a0 tx=0x7efc84017000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 msgr2=0x7efc90131010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc8dffb640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc90130c10 0x7efc90131010 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efc840028a0 tx=0x7efc84017000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 124, 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.339 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.340 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.341 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:03.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.343 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:03.344 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:03.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:38:03.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.352 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:03.353 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efc7c0384c0 msgr2=0x7efc7c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efc7c0384c0 0x7efc7c03a970 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efc900681f0 tx=0x7efc80032000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 10 monclient: shutdown 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 msgr2=0x7efc9012f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 0x7efc9012f790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efc8800c610 tx=0x7efc88004960 comp rx=0 tx=0).stop 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc964ba640 1 -- 172.21.15.116:0/1083669706 reap_dead start 2024-09-18T12:38:03.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 shutdown_connections 2024-09-18T12:38:03.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 --2- 172.21.15.116:0/1083669706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efc90067110 0x7efc9012f790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 >> 172.21.15.116:0/1083669706 conn(0x7efc900a1e30 msgr2=0x7efc9009c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 shutdown_connections 2024-09-18T12:38:03.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.326+0000 7efc97f1d640 1 -- 172.21.15.116:0/1083669706 wait complete. 2024-09-18T12:38:03.355 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:38:03.355 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:38:03.355 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.448+0000 7f06ac700640 10 monclient: get_monmap_and_config 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.448+0000 7f06ac700640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.448+0000 7f06ac700640 10 monclient: monmap: 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:03.448787+0000 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:03.448787+0000 2024-09-18T12:38:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.448+0000 7f06ac700640 1 Processor -- start 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.448+0000 7f06ac700640 1 -- start start 2024-09-18T12:38:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.448+0000 7f06ac700640 10 monclient: init 2024-09-18T12:38:03.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 0x7f06a4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient(hunting): picked mon.noname-b con 0x7f06a41318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06a409ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f06a4136c20 0x7f06a4139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient(hunting): picked mon.noname-c con 0x7f06a4136c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a4131cc0 con 0x7f06a409ea50 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a409f530 con 0x7f06a4136c20 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a409fbf0 con 0x7f06a41318f0 2024-09-18T12:38:03.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06ac700640 10 monclient(hunting): authenticate will time out at 1510.770345s 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 0x7f06a4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 0x7f06a4131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33160/0 (socket says 172.21.15.116:33160) 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 -- 172.21.15.116:0/158729744 learned_addr learned my addr 172.21.15.116:0/158729744 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): get_auth_request con 0x7f06a41318f0 auth_method 0 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06a9c9b640 1 --1- 172.21.15.116:0/158729744 >> v1:172.21.15.116:6789/0 conn(0x7f06a409ea50 0x7f06a409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45160/0 (socket says 172.21.15.116:45160) 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient(hunting): handle_auth_done global_id 144287 payload 274 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 -- 172.21.15.116:0/158729744 >> v2:172.21.15.144:3300/0 conn(0x7f06a4136c20 msgr2=0x7f06a4139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aac9d640 1 --2- 172.21.15.116:0/158729744 >> v2:172.21.15.144:3300/0 conn(0x7f06a4136c20 0x7f06a4139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 --2- 172.21.15.116:0/158729744 >> v2:172.21.15.144:3300/0 conn(0x7f06a4136c20 0x7f06a4139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 -- 172.21.15.116:0/158729744 >> v1:172.21.15.116:6789/0 conn(0x7f06a409ea50 legacy=0x7f06a409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 monclient: found mon.noname-b 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aa49c640 1 -- 172.21.15.116:0/158729744 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a409fe30 con 0x7f06a41318f0 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aac9d640 10 monclient: get_auth_request con 0x7f06a4136c20 auth_method 0 2024-09-18T12:38:03.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.449+0000 7f06aac9d640 1 --2- 172.21.15.116:0/158729744 >> v2:172.21.15.144:3300/0 conn(0x7f06a4136c20 0x7f06a4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06aa49c640 1 -- 172.21.15.116:0/158729744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 msgr2=0x7f06a4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06aa49c640 1 --2- 172.21.15.116:0/158729744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 0x7f06a4131f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009910 tx=0x7f06a00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96ca7685312e761 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 1 -- 172.21.15.116:0/158729744 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0032440 con 0x7f06a41318f0 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: dump: 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.463 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 1 -- 172.21.15.116:0/158729744 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a00325a0 con 0x7f06a41318f0 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06ac700640 5 monclient: authenticate success, global_id 144287 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06ac700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 1 -- 172.21.15.116:0/158729744 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0032940 con 0x7f06a41318f0 2024-09-18T12:38:03.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06ac700640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06ac700640 4 set_mon_vals no callback set 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.452+0000 7f06a949a640 10 monclient: dump: 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.465 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 10 monclient: shutdown 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 -- 172.21.15.116:0/158729744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 msgr2=0x7f06a4131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 --2- 172.21.15.116:0/158729744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 0x7f06a4131f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009910 tx=0x7f06a00091d0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 -- 172.21.15.116:0/158729744 shutdown_connections 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 --2- 172.21.15.116:0/158729744 >> v2:172.21.15.144:3300/0 conn(0x7f06a4136c20 0x7f06a4139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 --2- 172.21.15.116:0/158729744 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a41318f0 0x7f06a4131f30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 -- 172.21.15.116:0/158729744 >> 172.21.15.116:0/158729744 conn(0x7f06a4067110 msgr2=0x7f06a4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 -- 172.21.15.116:0/158729744 shutdown_connections 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 1 -- 172.21.15.116:0/158729744 wait complete. 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.453+0000 7f06ac700640 10 monclient: monmap: 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 Processor -- start 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 -- start start 2024-09-18T12:38:03.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient: init 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 0x7f06a41307f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): picked mon.noname-c con 0x7f06a409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06a41318f0 0x7f06a4099830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): picked mon.noname-a con 0x7f06a41318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a4136c20 0x7f06a4099d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): picked mon.noname-b con 0x7f06a4136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a409e090 con 0x7f06a41318f0 2024-09-18T12:38:03.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a41374d0 con 0x7f06a4136c20 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a411c610 con 0x7f06a409ea50 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06ac700640 10 monclient(hunting): authenticate will time out at 1510.775866s 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 0x7f06a41307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 0x7f06a41307f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33176/0 (socket says 172.21.15.116:33176) 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 1 -- 172.21.15.116:0/2293121460 learned_addr learned my addr 172.21.15.116:0/2293121460 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:03.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 10 monclient(hunting): get_auth_request con 0x7f06a409ea50 auth_method 0 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06a9c9b640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06a41318f0 0x7f06a4099830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06a9c9b640 10 monclient(hunting): get_auth_request con 0x7f06a41318f0 auth_method 0 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06a9c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06a9c9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06a9c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.454+0000 7f06aa49c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 10 monclient(hunting): handle_auth_done global_id 144290 payload 824 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aac9d640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a4136c20 0x7f06a4099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a4136c20 msgr2=0x7f06a4099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a4136c20 0x7f06a4099d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06a41318f0 msgr2=0x7f06a4099830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06a41318f0 0x7f06a4099830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 1 monclient: found mon.noname-c 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aa49c640 1 -- 172.21.15.116:0/2293121460 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a0009450 con 0x7f06a409ea50 2024-09-18T12:38:03.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aac9d640 10 monclient: get_auth_request con 0x7f06a4136c20 auth_method 0 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.455+0000 7f06aac9d640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f06a4136c20 0x7f06a4099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06aa49c640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 msgr2=0x7f06a41307f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06aa49c640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 0x7f06a41307f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06a0035040 tx=0x7f06a0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0000c40 con 0x7f06a409ea50 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: dump: 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.472 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a0031c70 con 0x7f06a409ea50 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06ac700640 5 monclient: authenticate success, global_id 144290 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06ac700640 10 monclient: _renew_subs 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06ac700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a411ba00 con 0x7f06a409ea50 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a003d440 con 0x7f06a409ea50 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f069a7fc640 10 monclient: dump: 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.474 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06a8c99640 4 set_mon_vals no callback set 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06ac700640 10 monclient: _renew_subs 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06ac700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.457+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a411cec0 con 0x7f06a409ea50 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06ac700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06ac700640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 0x7f06a40298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f06a0039040 con 0x7f06a409ea50 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a411cd80 con 0x7f06a4141cc0 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 0x7f06a40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient: get_auth_request con 0x7f06a4141cc0 auth_method 0 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f069a7fc640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0688038490 0x7f068803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f06a006bf90 con 0x7f06a409ea50 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_done global_id 144293 payload 824 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06aac9d640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0688038490 0x7f068803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06a9c9b640 1 -- 172.21.15.116:0/2293121460 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06a411ca60 con 0x7f06a4141cc0 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.458+0000 7f06aac9d640 10 monclient: get_auth_request con 0x7f0688038490 auth_method 0 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f06a9c9b640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 msgr2=0x7f06a40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f06a9c9b640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 0x7f06a40298c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a49a0 tx=0x7f0694007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694005050 con 0x7f06a4141cc0 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 10 monclient: dump: 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.477 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f06aac9d640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0688038490 0x7f068803a940 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a0240 tx=0x7f069c007670 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f06940051b0 con 0x7f06a4141cc0 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 msgr2=0x7f06a40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 0x7f06a40298c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a49a0 tx=0x7f0694007e40 comp rx=0 tx=0).stop 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 msgr2=0x7f06a40298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.461+0000 7f069a7fc640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4141cc0 0x7f06a40298c0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a49a0 tx=0x7f0694007e40 comp rx=0 tx=0).stop 2024-09-18T12:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.466+0000 7f06ac700640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.466+0000 7f06ac700640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 0x7f06a4029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06ac700640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a4065e00 con 0x7f06a4130d30 2024-09-18T12:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 0x7f06a4029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient: get_auth_request con 0x7f06a4130d30 auth_method 0 2024-09-18T12:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 10 monclient(hunting): handle_auth_done global_id 144296 payload 824 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.467+0000 7f06a9c9b640 1 -- 172.21.15.116:0/2293121460 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f06a404bf60 con 0x7f06a4130d30 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f06a9c9b640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 msgr2=0x7f06a4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f06a9c9b640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 0x7f06a4029e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a49a0 tx=0x7f0694019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069402a480 con 0x7f06a4130d30 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 10 monclient: dump: 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f0694025070 con 0x7f06a4130d30 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 msgr2=0x7f06a4029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 0x7f06a4029e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a49a0 tx=0x7f0694019000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 msgr2=0x7f06a4029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.469+0000 7f069a7fc640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a4130d30 0x7f06a4029e00 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a49a0 tx=0x7f0694019000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 124, 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.484 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.485 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.486 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:03.487 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.488 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.490 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.491 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:03.493 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:03.494 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.495 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:03.496 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0688038490 msgr2=0x7f068803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0688038490 0x7f068803a940 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f06a40a0240 tx=0x7f069c007670 comp rx=0 tx=0).stop 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 10 monclient: shutdown 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 msgr2=0x7f06a41307f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 0x7f06a41307f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06a0035040 tx=0x7f06a0032000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06aac9d640 1 -- 172.21.15.116:0/2293121460 reap_dead start 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 shutdown_connections 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 --2- 172.21.15.116:0/2293121460 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f06a409ea50 0x7f06a41307f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 >> 172.21.15.116:0/2293121460 conn(0x7f06a4067110 msgr2=0x7f06a40a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 shutdown_connections 2024-09-18T12:38:03.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.470+0000 7f06ac700640 1 -- 172.21.15.116:0/2293121460 wait complete. 2024-09-18T12:38:03.499 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:38:03.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:38:03.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:38:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.592+0000 7f1736cef640 10 monclient: get_monmap_and_config 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.592+0000 7f1736cef640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.592+0000 7f1736cef640 10 monclient: monmap: 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:03.592739+0000 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:03.592739+0000 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:03.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.592+0000 7f1736cef640 1 Processor -- start 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.592+0000 7f1736cef640 1 -- start start 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.592+0000 7f1736cef640 10 monclient: init 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f173009c860 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f173009e840 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1730137cc0 0x7f173013a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient(hunting): picked mon.noname-a con 0x7f1730137cc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173009ec40 con 0x7f173009c860 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173009b2b0 con 0x7f173009e840 2024-09-18T12:38:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173006a320 con 0x7f1730137cc0 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1736cef640 10 monclient(hunting): authenticate will time out at 1510.914305s 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1730137cc0 0x7f173013a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1734a8b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f173009c860 0x7f173009cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45162/0 (socket says 172.21.15.116:45162) 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f1734a8b640 1 -- 172.21.15.116:0/759021175 learned_addr learned my addr 172.21.15.116:0/759021175 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): get_auth_request con 0x7f1730137cc0 auth_method 0 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient(hunting): handle_auth_done global_id 154200 payload 274 2024-09-18T12:38:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 1 -- 172.21.15.116:0/759021175 >> v1:172.21.15.144:6789/0 conn(0x7f173009e840 legacy=0x7f173009aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 1 -- 172.21.15.116:0/759021175 >> v1:172.21.15.116:6789/0 conn(0x7f173009c860 legacy=0x7f173009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 1 monclient: found mon.noname-a 2024-09-18T12:38:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:38:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.593+0000 7f173528c640 1 -- 172.21.15.116:0/759021175 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730068a20 con 0x7f1730137cc0 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f173528c640 1 -- 172.21.15.116:0/759021175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 msgr2=0x7f173013a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f173528c640 1 --2- 172.21.15.116:0/759021175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173013a0a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1728009e60 tx=0x7f1728009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db2b10a0640b199 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 1 -- 172.21.15.116:0/759021175 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728007840 con 0x7f1730137cc0 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: dump: 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.605 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 1 -- 172.21.15.116:0/759021175 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172802d050 con 0x7f1730137cc0 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 1 -- 172.21.15.116:0/759021175 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728007bd0 con 0x7f1730137cc0 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f1736cef640 5 monclient: authenticate success, global_id 154200 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f1736cef640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f1736cef640 4 set_mon_vals no callback set 2024-09-18T12:38:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.596+0000 7f17277fe640 10 monclient: dump: 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 10 monclient: shutdown 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 1 -- 172.21.15.116:0/759021175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 msgr2=0x7f173013a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 1 --2- 172.21.15.116:0/759021175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173013a0a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1728009e60 tx=0x7f1728009780 comp rx=0 tx=0).stop 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 1 -- 172.21.15.116:0/759021175 shutdown_connections 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 1 --2- 172.21.15.116:0/759021175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173013a0a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 1 -- 172.21.15.116:0/759021175 >> 172.21.15.116:0/759021175 conn(0x7f17300a2e40 msgr2=0x7f17300a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 1 -- 172.21.15.116:0/759021175 shutdown_connections 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 1 -- 172.21.15.116:0/759021175 wait complete. 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.597+0000 7f1736cef640 10 monclient: monmap: 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 Processor -- start 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 -- start start 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient: init 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f173009c860 0x7f173006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): picked mon.noname-c con 0x7f173009c860 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173009e840 0x7f173006b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): picked mon.noname-b con 0x7f173009e840 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173006bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): picked mon.noname-a con 0x7f1730137cc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173006caf0 con 0x7f1730137cc0 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173009e090 con 0x7f173009e840 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1730099cc0 con 0x7f173009c860 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1736cef640 10 monclient(hunting): authenticate will time out at 1510.919904s 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f1734a8b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f173009c860 0x7f173006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.598+0000 7f173528c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173006bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f173009c860 0x7f173006ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33206/0 (socket says 172.21.15.116:33206) 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173006bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50666/0 (socket says 172.21.15.116:50666) 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 -- 172.21.15.116:0/2624035472 learned_addr learned my addr 172.21.15.116:0/2624035472 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): get_auth_request con 0x7f173009c860 auth_method 0 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 10 monclient(hunting): get_auth_request con 0x7f1730137cc0 auth_method 0 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1727fff640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173009e840 0x7f173006b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1727fff640 10 monclient(hunting): get_auth_request con 0x7f173009e840 auth_method 0 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient(hunting): handle_auth_done global_id 144299 payload 824 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173009e840 msgr2=0x7f173006b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173009e840 0x7f173006b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 msgr2=0x7f173006bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173006bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 monclient: found mon.noname-c 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1734a8b640 1 -- 172.21.15.116:0/2624035472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1728009a20 con 0x7f173009c860 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f173528c640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1730137cc0 0x7f173006bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.599+0000 7f1727fff640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173009e840 0x7f173006b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1734a8b640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f173009c860 msgr2=0x7f173006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1734a8b640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f173009c860 0x7f173006ee80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c5e0 tx=0x7f172c004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c005cb0 con 0x7f173009c860 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: dump: 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c005e30 con 0x7f173009c860 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1736cef640 5 monclient: authenticate success, global_id 144299 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c007340 con 0x7f173009c860 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1736cef640 10 monclient: _renew_subs 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1736cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173006cf00 con 0x7f173009c860 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1724ff9640 10 monclient: dump: 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1726ffd640 4 set_mon_vals no callback set 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1736cef640 10 monclient: _renew_subs 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1736cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.601+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1730099500 con 0x7f173009c860 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1736cef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1736cef640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 0x7f1730138090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1736cef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730065e00 con 0x7f173006c330 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f172c01b400 con 0x7f173009c860 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1727fff640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 0x7f1730138090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1724ff9640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f171c038490 0x7f171c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1727fff640 10 monclient: get_auth_request con 0x7f173006c330 auth_method 0 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.602+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f172c04be00 con 0x7f173009c860 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.603+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.603+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.603+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.603+0000 7f173528c640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f171c038490 0x7f171c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.603+0000 7f173528c640 10 monclient: get_auth_request con 0x7f171c038490 auth_method 0 2024-09-18T12:38:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.603+0000 7f1727fff640 10 monclient(hunting): handle_auth_done global_id 164107 payload 824 2024-09-18T12:38:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.603+0000 7f1727fff640 1 -- 172.21.15.116:0/2624035472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f173006c700 con 0x7f173006c330 2024-09-18T12:38:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.605+0000 7f1727fff640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 msgr2=0x7f1730138090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.605+0000 7f1727fff640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 0x7f1730138090 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f172000d740 tx=0x7f1720008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.605+0000 7f173528c640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f171c038490 0x7f171c03a940 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1730071740 tx=0x7f1728037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.605+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720009950 con 0x7f173006c330 2024-09-18T12:38:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.605+0000 7f1724ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.605+0000 7f1724ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.605+0000 7f1724ff9640 10 monclient: dump: 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.606+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1720009ab0 con 0x7f173006c330 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.606+0000 7f1724ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.606+0000 7f1724ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.606+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 msgr2=0x7f1730138090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.606+0000 7f1724ff9640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 0x7f1730138090 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f172000d740 tx=0x7f1720008040 comp rx=0 tx=0).stop 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.606+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 msgr2=0x7f1730138090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.606+0000 7f1724ff9640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f173006c330 0x7f1730138090 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f172000d740 tx=0x7f1720008040 comp rx=0 tx=0).stop 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.611+0000 7f1736cef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.611+0000 7f1736cef640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 0x7f17301389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.611+0000 7f1736cef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.611+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173006e1b0 con 0x7f17301385d0 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 0x7f17301389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient: get_auth_request con 0x7f17301385d0 auth_method 0 2024-09-18T12:38:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 10 monclient(hunting): handle_auth_done global_id 164110 payload 824 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.612+0000 7f1727fff640 1 -- 172.21.15.116:0/2624035472 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f173004bf60 con 0x7f17301385d0 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.614+0000 7f1727fff640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 msgr2=0x7f17301389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.614+0000 7f1727fff640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 0x7f17301389d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1720017000 tx=0x7f172001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.614+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172001e9e0 con 0x7f17301385d0 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.614+0000 7f1724ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.614+0000 7f1724ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.614+0000 7f1724ff9640 10 monclient: dump: 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f1720012400 con 0x7f17301385d0 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1724ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1724ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 msgr2=0x7f17301389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1724ff9640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 0x7f17301389d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1720017000 tx=0x7f172001a000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1724ff9640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 msgr2=0x7f17301389d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1724ff9640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17301385d0 0x7f17301389d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:03.625 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 124, 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.626 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.627 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:03.628 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:03.629 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:03.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.632 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:03.635 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.636 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:03.637 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f171c038490 msgr2=0x7f171c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f171c038490 0x7f171c03a940 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1730071740 tx=0x7f1728037000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 10 monclient: shutdown 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f173009c860 msgr2=0x7f173006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 --2- 172.21.15.116:0/2624035472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f173009c860 0x7f173006ee80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c5e0 tx=0x7f172c004960 comp rx=0 tx=0).stop 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f173528c640 1 -- 172.21.15.116:0/2624035472 reap_dead start 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f173528c640 1 -- 172.21.15.116:0/2624035472 reap_dead start 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 shutdown_connections 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 >> 172.21.15.116:0/2624035472 conn(0x7f17300a2e40 msgr2=0x7f1730131b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 shutdown_connections 2024-09-18T12:38:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.615+0000 7f1736cef640 1 -- 172.21.15.116:0/2624035472 wait complete. 2024-09-18T12:38:03.640 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:38:03.640 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:38:03.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:38:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.731+0000 7f1999060640 10 monclient: get_monmap_and_config 2024-09-18T12:38:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.731+0000 7f1999060640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.731+0000 7f1999060640 10 monclient: monmap: 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:03.732436+0000 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:03.732436+0000 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:03.733 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 1 Processor -- start 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 1 -- start start 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient: init 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 0x7f1994096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient(hunting): picked mon.noname-b con 0x7f199409a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19940971e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f19940975b0 0x7f199412fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient(hunting): picked mon.noname-a con 0x7f19940975b0 addr v2:172.21.15.116:3300/0 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199406ee10 con 0x7f19940971e0 2024-09-18T12:38:03.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199409cb20 con 0x7f19940975b0 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199409a8e0 con 0x7f199409a510 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.732+0000 7f1999060640 10 monclient(hunting): authenticate will time out at 1511.053922s 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 0x7f1994096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f19940975b0 0x7f199412fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 0x7f1994096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33222/0 (socket says 172.21.15.116:33222) 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 -- 172.21.15.116:0/1542368800 learned_addr learned my addr 172.21.15.116:0/1542368800 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): get_auth_request con 0x7f199409a510 auth_method 0 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 10 monclient(hunting): get_auth_request con 0x7f19940975b0 auth_method 0 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient(hunting): handle_auth_done global_id 144302 payload 274 2024-09-18T12:38:03.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 -- 172.21.15.116:0/1542368800 >> v2:172.21.15.116:3300/0 conn(0x7f19940975b0 msgr2=0x7f199412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 --2- 172.21.15.116:0/1542368800 >> v2:172.21.15.116:3300/0 conn(0x7f19940975b0 0x7f199412fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 -- 172.21.15.116:0/1542368800 >> v1:172.21.15.116:6789/0 conn(0x7f19940971e0 legacy=0x7f199412f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 monclient: found mon.noname-b 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f1992d9d640 1 -- 172.21.15.116:0/1542368800 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199409d590 con 0x7f199409a510 2024-09-18T12:38:03.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.733+0000 7f199359e640 1 --2- 172.21.15.116:0/1542368800 >> v2:172.21.15.116:3300/0 conn(0x7f19940975b0 0x7f199412fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1992d9d640 1 -- 172.21.15.116:0/1542368800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 msgr2=0x7f1994096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1992d9d640 1 --2- 172.21.15.116:0/1542368800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 0x7f1994096ca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1984009910 tx=0x7f19840091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f786ced388d6716 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 1 -- 172.21.15.116:0/1542368800 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984032440 con 0x7f199409a510 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: dump: 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.745 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 5 monclient: authenticate success, global_id 144302 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 1 -- 172.21.15.116:0/1542368800 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19840325a0 con 0x7f199409a510 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 4 set_mon_vals no callback set 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 1 -- 172.21.15.116:0/1542368800 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984032940 con 0x7f199409a510 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1991d9b640 10 monclient: dump: 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.747 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 10 monclient: shutdown 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 -- 172.21.15.116:0/1542368800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 msgr2=0x7f1994096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 --2- 172.21.15.116:0/1542368800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 0x7f1994096ca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1984009910 tx=0x7f19840091d0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 -- 172.21.15.116:0/1542368800 shutdown_connections 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 --2- 172.21.15.116:0/1542368800 >> v2:172.21.15.116:3300/0 conn(0x7f19940975b0 0x7f199412fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 --2- 172.21.15.116:0/1542368800 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f199409a510 0x7f1994096ca0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 -- 172.21.15.116:0/1542368800 >> 172.21.15.116:0/1542368800 conn(0x7f19940672b0 msgr2=0x7f19940676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 -- 172.21.15.116:0/1542368800 shutdown_connections 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.736+0000 7f1999060640 1 -- 172.21.15.116:0/1542368800 wait complete. 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.737+0000 7f1999060640 10 monclient: build_initial_monmap 2024-09-18T12:38:03.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.737+0000 7f1999060640 10 monclient: monmap: 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.737+0000 7f1999060640 1 Processor -- start 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.737+0000 7f1999060640 1 -- start start 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.737+0000 7f1999060640 10 monclient: init 2024-09-18T12:38:03.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 0x7f1994141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): picked mon.noname-c con 0x7f19940971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19940975b0 0x7f1994130ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): picked mon.noname-a con 0x7f19940975b0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f199409a510 0x7f19940983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): picked mon.noname-b con 0x7f199409a510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994099fd0 con 0x7f19940975b0 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199409c370 con 0x7f199409a510 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1994132860 con 0x7f19940971e0 2024-09-18T12:38:03.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1999060640 10 monclient(hunting): authenticate will time out at 1511.059188s 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 0x7f1994141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 0x7f1994141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33224/0 (socket says 172.21.15.116:33224) 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19940975b0 0x7f1994130ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 -- 172.21.15.116:0/3114107107 learned_addr learned my addr 172.21.15.116:0/3114107107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): get_auth_request con 0x7f19940971e0 auth_method 0 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 10 monclient(hunting): get_auth_request con 0x7f19940975b0 auth_method 0 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:03.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient(hunting): handle_auth_done global_id 144305 payload 824 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 -- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f199409a510 msgr2=0x7f19940983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f199409a510 0x7f19940983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 -- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19940975b0 msgr2=0x7f1994130ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19940975b0 0x7f1994130ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:38:03.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 monclient: found mon.noname-c 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f1992d9d640 1 -- 172.21.15.116:0/3114107107 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c009a20 con 0x7f19940971e0 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.738+0000 7f199259c640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19940975b0 0x7f1994130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f1992d9d640 1 -- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 msgr2=0x7f1994141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f1992d9d640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 0x7f1994141280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1984009210 tx=0x7f1984032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 1 -- 172.21.15.116:0/3114107107 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984000c40 con 0x7f19940971e0 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: dump: 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.753 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: _finish_auth 0 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 1 -- 172.21.15.116:0/3114107107 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1984031c70 con 0x7f19940971e0 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f1999060640 5 monclient: authenticate success, global_id 144305 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f1999060640 10 monclient: _renew_subs 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f1999060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1984009450 con 0x7f19940971e0 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 1 -- 172.21.15.116:0/3114107107 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198403b400 con 0x7f19940971e0 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f197affd640 10 monclient: dump: 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:03.755 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f1999060640 10 monclient: _renew_subs 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f1999060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.740+0000 7f199159a640 4 set_mon_vals no callback set 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19941200c0 con 0x7f19940971e0 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f197affd640 1 -- 172.21.15.116:0/3114107107 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1984039040 con 0x7f19940971e0 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f1999060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f1999060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f1999060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.741+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199404bf60 con 0x7f19940971e0 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.742+0000 7f197affd640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1980038490 0x7f198003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.742+0000 7f197affd640 1 -- 172.21.15.116:0/3114107107 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f198406c3c0 con 0x7f19940971e0 2024-09-18T12:38:03.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.742+0000 7f199259c640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1980038490 0x7f198003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:03.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.742+0000 7f199259c640 10 monclient: get_auth_request con 0x7f1980038490 auth_method 0 2024-09-18T12:38:03.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.744+0000 7f199259c640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1980038490 0x7f198003a940 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1988007fd0 tx=0x7f1988007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:03.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.746+0000 7f197affd640 1 -- 172.21.15.116:0/3114107107 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1984039ad0 con 0x7f19940971e0 2024-09-18T12:38:03.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.746+0000 7f197affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:03.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.746+0000 7f197affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:03.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.962+0000 7f1999060640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:38:03.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.962+0000 7f1999060640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:38:03.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.962+0000 7f1999060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:03.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.962+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f199406a8c0 con 0x7f19940971e0 2024-09-18T12:38:03.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f197affd640 1 -- 172.21.15.116:0/3114107107 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f1984037080 con 0x7f19940971e0 2024-09-18T12:38:03.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f197affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:38:03.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f197affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:03.965 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.965 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=3870919382,config=55734535,health=3420451498,kv=2187923472,logm=2671871990}) 2024-09-18T12:38:03.966 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=3490545852,config=3140514009,health=2760637573,kv=2187923472,logm=2498080948}) 2024-09-18T12:38:03.966 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.966 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=3870919382,config=55734535,health=3420451498,kv=2187923472,logm=2671871990}) 2024-09-18T12:38:03.966 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.964+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=8,logm=51} crc {auth=3643870872,config=3140514009,health=1015403528,kv=3604100458,logm=292219720}) 2024-09-18T12:38:03.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.965+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.965+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979410557}) 2024-09-18T12:38:03.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.965+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136032672}) 2024-09-18T12:38:03.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.965+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.965+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979410557}) 2024-09-18T12:38:03.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.965+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424353804}) 2024-09-18T12:38:03.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1980038490 msgr2=0x7f198003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1980038490 0x7f198003a940 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1988007fd0 tx=0x7f1988007e10 comp rx=0 tx=0).stop 2024-09-18T12:38:03.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 10 monclient: shutdown 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 msgr2=0x7f1994141280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 0x7f1994141280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1984009210 tx=0x7f1984032000 comp rx=0 tx=0).stop 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 shutdown_connections 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1980038490 0x7f198003a940 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f199409a510 0x7f19940983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19940975b0 0x7f1994130ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 --2- 172.21.15.116:0/3114107107 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19940971e0 0x7f1994141280 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 >> 172.21.15.116:0/3114107107 conn(0x7f19940672b0 msgr2=0x7f199406a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 shutdown_connections 2024-09-18T12:38:03.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f1999060640 1 -- 172.21.15.116:0/3114107107 wait complete. 2024-09-18T12:38:03.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.966+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.967+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360596233}) 2024-09-18T12:38:03.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.967+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895526380}) 2024-09-18T12:38:03.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.967+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.967+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360596233}) 2024-09-18T12:38:03.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.967+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2301885088}) 2024-09-18T12:38:03.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616080109}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4014309508}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616080109}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2207467844}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415817422}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551195720}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415817422}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.968+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753106704}) 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.969+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.976 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.969+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2133049534}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.969+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3677362304}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.969+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.969+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2133049534}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.969+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913487351}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.970+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.970+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138907757}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.970+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3583179413}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.970+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.970+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138907757}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.970+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744220990}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970646452}) 2024-09-18T12:38:03.977 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606614671}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970646452}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2763641339}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755802608}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.971+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=132425743}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755802608}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261172218}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791407830}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015735417}) 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.978 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791407830}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.972+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102896884}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.973+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.973+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443012975}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.973+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2914691444}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.973+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.973+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443012975}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.973+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3786865857}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486415941}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914099616}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486415941}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1736282269}) 2024-09-18T12:38:03.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3618126574}) 2024-09-18T12:38:03.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2203579050}) 2024-09-18T12:38:03.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3618126574}) 2024-09-18T12:38:03.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.974+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320535529}) 2024-09-18T12:38:03.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.975+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.975+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426569362}) 2024-09-18T12:38:03.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.975+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1365694049}) 2024-09-18T12:38:03.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.975+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.975+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426569362}) 2024-09-18T12:38:03.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.975+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4135798294}) 2024-09-18T12:38:03.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.976+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.976+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584070499}) 2024-09-18T12:38:03.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.976+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1888608668}) 2024-09-18T12:38:03.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.976+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.976+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584070499}) 2024-09-18T12:38:03.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.976+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=850555199}) 2024-09-18T12:38:03.983 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761493695}) 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729024994}) 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761493695}) 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172557008}) 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057330639}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2086254053}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057330639}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.977+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3986227062}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.978+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.978+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434321}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.978+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801139504}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.978+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.978+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696434321}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.978+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=452335909}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.979+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.979+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3360880938}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.979+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=932127203}) 2024-09-18T12:38:03.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.979+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.979+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3360880938}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.979+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935946940}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066463030}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603878392}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066463030}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909444044}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747565767}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036403075}) 2024-09-18T12:38:03.997 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747565767}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.980+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=930108336}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.981+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.981+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193219098}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.981+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1373832984}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.981+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.981+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193219098}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.981+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2517059250}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.982+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.982+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067093512}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.982+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839250536}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.982+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.982+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067093512}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.982+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984955864}) 2024-09-18T12:38:03.998 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237565469}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2380549150}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237565469}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=115941298}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929987363}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88154364}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929987363}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.983+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3203160099}) 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.984+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:03.999 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.984+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625846746}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.984+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950799399}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.984+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.984+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625846746}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.984+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4261431104}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242269447}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722434764}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242269447}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966437681}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481070885}) 2024-09-18T12:38:04.000 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2664108021}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481070885}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.985+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909308379}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.986+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.986+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=26013994}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.986+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=305807703}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.986+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.986+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=26013994}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.986+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626175345}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2889141578}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426095168}) 2024-09-18T12:38:04.001 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2889141578}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=704130500}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.987+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532279677}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3419153795}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532279677}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461850837}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952014568}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754539566}) 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.002 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952014568}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.988+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1452934230}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.989+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.989+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426761851}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.989+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256216578}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.989+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.989+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426761851}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.989+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479210757}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240821471}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3223113537}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240821471}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3560130788}) 2024-09-18T12:38:04.003 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630050809}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2618490010}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630050809}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.990+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1345375113}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.991+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.991+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091883936}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.991+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2741030049}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.991+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.991+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091883936}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.991+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018659997}) 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.004 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541476523}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3301017148}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541476523}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4191505849}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918650626}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2560176689}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918650626}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.992+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715764838}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.993+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.993+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557102935}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.993+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808624546}) 2024-09-18T12:38:04.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.993+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.993+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557102935}) 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.993+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616918776}) 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.994+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.994+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800792531}) 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.994+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=225348905}) 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.994+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.994+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800792531}) 2024-09-18T12:38:04.006 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.994+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3378205401}) 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1214055527}) 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302937517}) 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1214055527}) 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047643501}) 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962982620}) 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216639134}) 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962982620}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.995+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1774183570}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.996+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.996+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986193268}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.996+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=493035455}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.996+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.996+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986193268}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.996+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1927124703}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918256073}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=662406422}) 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918256073}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884560564}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953402447}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394378242}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953402447}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.997+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700292633}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.998+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.998+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639999188}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.998+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078554037}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.998+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.998+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639999188}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.998+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780855193}) 2024-09-18T12:38:04.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630693767}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013615080}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630693767}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=876850778}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706402918}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074907123}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:03.999+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706402918}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=331496423}) 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252235897}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155704600}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252235897}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.000+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1156215518}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.001+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.001+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784483204}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.001+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1026485582}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.001+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.001+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784483204}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.001+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355940900}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.001+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381690309}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=666725347}) 2024-09-18T12:38:04.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381690309}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982341489}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975940922}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462665744}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975940922}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.002+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065804509}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1531975095}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758260157}) 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1531975095}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1373177207}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832009492}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1029696138}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.003+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832009492}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.004+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605823523}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.004+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.004+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918001916}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.004+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404260445}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.004+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.004+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918001916}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.004+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2176001976}) 2024-09-18T12:38:04.025 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.005+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.005+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055909417}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.005+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465971001}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.005+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.005+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055909417}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.005+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1518439615}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272337662}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497136068}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272337662}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830193501}) 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.026 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691940236}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2675660242}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691940236}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.006+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1036453840}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.007+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.007+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429556978}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.007+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1714888015}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.007+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.007+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429556978}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.007+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476400729}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396934084}) 2024-09-18T12:38:04.027 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956030103}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396934084}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1635609987}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448003044}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3259612895}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448003044}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.008+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269057257}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.009+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.009+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906325486}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.009+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187583433}) 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.009+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.028 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.009+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906325486}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.009+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798290002}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.010+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.010+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=95,mdsmap=2,mgr=3} crc {logm=3572072680,mdsmap=3585761405,mgr=2466192}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.010+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=138783257,mdsmap=1531151339}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.010+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.010+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=95,mdsmap=2,mgr=3} crc {logm=3572072680,mdsmap=3585761405,mgr=2466192}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.010+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1322194419,mdsmap=1531151339,mgr=3533299129}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.011+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.011+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=64} crc {mgr=1556005868,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1937988460,mon_config_key=3655569571,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=3071141073}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.011+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgr=1556005868,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=767862764,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=3115523317}) 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.011+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.029 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.011+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=64} crc {mgr=1556005868,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1937988460,mon_config_key=3655569571,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=3071141073}) 2024-09-18T12:38:04.030 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.011+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=1556005868,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1937988460,mon_config_key=3655569570,monmap=3750608922,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=3033740927}) 2024-09-18T12:38:04.030 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.018+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.030 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.018+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=2426482855}) 2024-09-18T12:38:04.030 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.018+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=63} crc {osdmap=3030201059}) 2024-09-18T12:38:04.030 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.018+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:38:04.030 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.018+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=2426482855}) 2024-09-18T12:38:04.030 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:38:04.018+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=68} crc {osdmap=3110345096}) 2024-09-18T12:38:04.984 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:38:04.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:38:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.079+0000 7fb816034640 10 monclient: get_monmap_and_config 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.079+0000 7fb816034640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.080+0000 7fb816034640 10 monclient: monmap: 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:05.080638+0000 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:05.080638+0000 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:05.082 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.080+0000 7fb816034640 1 Processor -- start 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.080+0000 7fb816034640 1 -- start start 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.080+0000 7fb816034640 10 monclient: init 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb81006eb80 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb810068a20 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 0x7fb810139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient(hunting): picked mon.noname-b con 0x7fb810137ac0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb810069b80 con 0x7fb810068a20 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb810069e00 con 0x7fb81006eb80 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb810099f90 con 0x7fb810137ac0 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb816034640 10 monclient(hunting): authenticate will time out at 1512.402477s 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 0x7fb810139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb807fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb810068a20 0x7fb81009c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45180/0 (socket says 172.21.15.116:45180) 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 0x7fb810139ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33232/0 (socket says 172.21.15.116:33232) 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb807fff640 1 -- 172.21.15.116:0/698734990 learned_addr learned my addr 172.21.15.116:0/698734990 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): get_auth_request con 0x7fb810137ac0 auth_method 0 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient(hunting): handle_auth_done global_id 144308 payload 274 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 1 -- 172.21.15.116:0/698734990 >> v1:172.21.15.144:6789/0 conn(0x7fb81006eb80 legacy=0x7fb81006ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 1 -- 172.21.15.116:0/698734990 >> v1:172.21.15.116:6789/0 conn(0x7fb810068a20 legacy=0x7fb81009c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 1 monclient: found mon.noname-b 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.081+0000 7fb80ffff640 1 -- 172.21.15.116:0/698734990 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81006f660 con 0x7fb810137ac0 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80ffff640 1 -- 172.21.15.116:0/698734990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 msgr2=0x7fb810139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80ffff640 1 --2- 172.21.15.116:0/698734990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 0x7fb810139ea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb808009e90 tx=0x7fb808009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e7c9ace533e352e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 1 -- 172.21.15.116:0/698734990 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808007840 con 0x7fb810137ac0 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: dump: 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.093 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 1 -- 172.21.15.116:0/698734990 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80802d050 con 0x7fb810137ac0 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb816034640 5 monclient: authenticate success, global_id 144308 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb816034640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb816034640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 1 -- 172.21.15.116:0/698734990 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808007bd0 con 0x7fb810137ac0 2024-09-18T12:38:05.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb816034640 4 set_mon_vals no callback set 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.084+0000 7fb80effd640 10 monclient: dump: 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.095 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 10 monclient: shutdown 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 1 -- 172.21.15.116:0/698734990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 msgr2=0x7fb810139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 1 --2- 172.21.15.116:0/698734990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 0x7fb810139ea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb808009e90 tx=0x7fb808009780 comp rx=0 tx=0).stop 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 1 -- 172.21.15.116:0/698734990 shutdown_connections 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 1 --2- 172.21.15.116:0/698734990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810137ac0 0x7fb810139ea0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 1 -- 172.21.15.116:0/698734990 >> 172.21.15.116:0/698734990 conn(0x7fb810067cd0 msgr2=0x7fb8100a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 1 -- 172.21.15.116:0/698734990 shutdown_connections 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.085+0000 7fb816034640 1 -- 172.21.15.116:0/698734990 wait complete. 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient: monmap: 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 Processor -- start 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 -- start start 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient: init 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 0x7fb810137340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): picked mon.noname-c con 0x7fb810068a20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81006eb80 0x7fb81006cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): picked mon.noname-a con 0x7fb81006eb80 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb810137ac0 0x7fb81006d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): picked mon.noname-b con 0x7fb810137ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81009cf40 con 0x7fb81006eb80 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81011f8f0 con 0x7fb810137ac0 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81011dd10 con 0x7fb810068a20 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.086+0000 7fb816034640 10 monclient(hunting): authenticate will time out at 1512.407939s 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 0x7fb810137340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81006eb80 0x7fb81006cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 0x7fb810137340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33238/0 (socket says 172.21.15.116:33238) 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81006eb80 0x7fb81006cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50690/0 (socket says 172.21.15.116:50690) 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 -- 172.21.15.116:0/1981466045 learned_addr learned my addr 172.21.15.116:0/1981466045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): get_auth_request con 0x7fb810068a20 auth_method 0 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 10 monclient(hunting): get_auth_request con 0x7fb81006eb80 auth_method 0 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80ffff640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb810137ac0 0x7fb81006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80ffff640 10 monclient(hunting): get_auth_request con 0x7fb810137ac0 auth_method 0 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient(hunting): handle_auth_done global_id 144311 payload 824 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 -- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb810137ac0 msgr2=0x7fb81006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb810137ac0 0x7fb81006d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 -- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81006eb80 msgr2=0x7fb81006cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81006eb80 0x7fb81006cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:38:05.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb80f7fe640 1 -- 172.21.15.116:0/1981466045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808009a20 con 0x7fb810068a20 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.087+0000 7fb807fff640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81006eb80 0x7fb81006cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb80f7fe640 1 -- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 msgr2=0x7fb810137340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb80f7fe640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 0x7fb810137340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb80000b5e0 tx=0x7fb8000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 1 -- 172.21.15.116:0/1981466045 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80000f7a0 con 0x7fb810068a20 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: dump: 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.102 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb816034640 5 monclient: authenticate success, global_id 144311 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 1 -- 172.21.15.116:0/1981466045 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80000f920 con 0x7fb810068a20 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 1 -- 172.21.15.116:0/1981466045 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80000fce0 con 0x7fb810068a20 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb816034640 10 monclient: _renew_subs 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb816034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81011f610 con 0x7fb810068a20 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb8077fe640 10 monclient: dump: 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.104 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb80e7fc640 4 set_mon_vals no callback set 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb816034640 10 monclient: _renew_subs 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb816034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.089+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81011e570 con 0x7fb810068a20 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.090+0000 7fb8077fe640 1 -- 172.21.15.116:0/1981466045 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb800010630 con 0x7fb810068a20 2024-09-18T12:38:05.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.090+0000 7fb816034640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.090+0000 7fb816034640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.090+0000 7fb816034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.090+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81004bf60 con 0x7fb810068a20 2024-09-18T12:38:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.090+0000 7fb8077fe640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb7f0038490 0x7fb7f003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.090+0000 7fb8077fe640 1 -- 172.21.15.116:0/1981466045 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb80004bf10 con 0x7fb810068a20 2024-09-18T12:38:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.091+0000 7fb807fff640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb7f0038490 0x7fb7f003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.091+0000 7fb807fff640 10 monclient: get_auth_request con 0x7fb7f0038490 auth_method 0 2024-09-18T12:38:05.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.093+0000 7fb807fff640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb7f0038490 0x7fb7f003a940 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8007fd0 tx=0x7fb7f8007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.095+0000 7fb8077fe640 1 -- 172.21.15.116:0/1981466045 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb80000e2e0 con 0x7fb810068a20 2024-09-18T12:38:05.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.095+0000 7fb8077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.095+0000 7fb8077fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.310+0000 7fb816034640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:38:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.310+0000 7fb816034640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:38:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.310+0000 7fb816034640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.310+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb8100638d0 con 0x7fb810068a20 2024-09-18T12:38:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.310+0000 7fb8077fe640 1 -- 172.21.15.116:0/1981466045 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fb800017920 con 0x7fb810068a20 2024-09-18T12:38:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.310+0000 7fb8077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:38:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.310+0000 7fb8077fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:05.312 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":124,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:38:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb7f0038490 msgr2=0x7fb7f003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb7f0038490 0x7fb7f003a940 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8007fd0 tx=0x7fb7f8007f00 comp rx=0 tx=0).stop 2024-09-18T12:38:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 10 monclient: shutdown 2024-09-18T12:38:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 msgr2=0x7fb810137340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 0x7fb810137340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb80000b5e0 tx=0x7fb8000090f0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 shutdown_connections 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb7f0038490 0x7fb7f003a940 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb810137ac0 0x7fb81006d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81006eb80 0x7fb81006cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 --2- 172.21.15.116:0/1981466045 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb810068a20 0x7fb810137340 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 >> 172.21.15.116:0/1981466045 conn(0x7fb810067cd0 msgr2=0x7fb8100a15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 shutdown_connections 2024-09-18T12:38:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.312+0000 7fb816034640 1 -- 172.21.15.116:0/1981466045 wait complete. 2024-09-18T12:38:05.323 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:38:05.323 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:38:05.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:38:05.323 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.416+0000 7fc249d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.416+0000 7fc249d9d640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.416+0000 7fc249d9d640 10 monclient: monmap: 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:05.416823+0000 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:05.416823+0000 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:05.418 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 1 Processor -- start 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 1 -- start start 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient: init 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc244131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc24409e910 0x7fc24409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc24409e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 0x7fc244138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc24409f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24409faf0 con 0x7fc244131ab0 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc244132590 con 0x7fc24409e910 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24409fd30 con 0x7fc24409f250 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc249d9d640 10 monclient(hunting): authenticate will time out at 1512.738689s 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 0x7fc244138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 0x7fc244138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33244/0 (socket says 172.21.15.116:33244) 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 1 -- 172.21.15.116:0/2103472711 learned_addr learned my addr 172.21.15.116:0/2103472711 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 10 monclient(hunting): get_auth_request con 0x7fc24409f250 auth_method 0 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.417+0000 7fc243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc23bfff640 1 --2- 172.21.15.116:0/2103472711 >> v2:172.21.15.144:3300/0 conn(0x7fc24409e910 0x7fc24409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 10 monclient(hunting): handle_auth_done global_id 144314 payload 274 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 1 -- 172.21.15.116:0/2103472711 >> v2:172.21.15.144:3300/0 conn(0x7fc24409e910 msgr2=0x7fc24409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 1 --2- 172.21.15.116:0/2103472711 >> v2:172.21.15.144:3300/0 conn(0x7fc24409e910 0x7fc24409ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 1 -- 172.21.15.116:0/2103472711 >> v1:172.21.15.144:6789/0 conn(0x7fc244131ab0 legacy=0x7fc244131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 1 monclient: found mon.noname-b 2024-09-18T12:38:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.418+0000 7fc243fff640 1 -- 172.21.15.116:0/2103472711 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2440a1110 con 0x7fc24409f250 2024-09-18T12:38:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.420+0000 7fc243fff640 1 -- 172.21.15.116:0/2103472711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 msgr2=0x7fc244138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.420+0000 7fc243fff640 1 --2- 172.21.15.116:0/2103472711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 0x7fc244138bc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009f20 tx=0x7fc23c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7519f964248021 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.420+0000 7fc242ffd640 1 -- 172.21.15.116:0/2103472711 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c007610 con 0x7fc24409f250 2024-09-18T12:38:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: dump: 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.432 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 5 monclient: authenticate success, global_id 144314 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 1 -- 172.21.15.116:0/2103472711 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23c02d050 con 0x7fc24409f250 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 4 set_mon_vals no callback set 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 1 -- 172.21.15.116:0/2103472711 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0079a0 con 0x7fc24409f250 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc242ffd640 10 monclient: dump: 2024-09-18T12:38:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 10 monclient: shutdown 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 -- 172.21.15.116:0/2103472711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 msgr2=0x7fc244138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2103472711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 0x7fc244138bc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009f20 tx=0x7fc23c009840 comp rx=0 tx=0).stop 2024-09-18T12:38:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 -- 172.21.15.116:0/2103472711 shutdown_connections 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2103472711 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc24409f250 0x7fc244138bc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2103472711 >> v2:172.21.15.144:3300/0 conn(0x7fc24409e910 0x7fc24409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 -- 172.21.15.116:0/2103472711 >> 172.21.15.116:0/2103472711 conn(0x7fc244067d30 msgr2=0x7fc2440a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 -- 172.21.15.116:0/2103472711 shutdown_connections 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.421+0000 7fc249d9d640 1 -- 172.21.15.116:0/2103472711 wait complete. 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient: monmap: 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 Processor -- start 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 -- start start 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient: init 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24409e910 0x7fc244097fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc24409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc24409f250 0x7fc24409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc24409f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 0x7fc244096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc244131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc244099870 con 0x7fc24409e910 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc244099590 con 0x7fc24409f250 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc244099b50 con 0x7fc244131ab0 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.422+0000 7fc249d9d640 10 monclient(hunting): authenticate will time out at 1512.743948s 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 0x7fc244096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24409e910 0x7fc244097fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 0x7fc244096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:33258/0 (socket says 172.21.15.116:33258) 2024-09-18T12:38:05.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 -- 172.21.15.116:0/2698955403 learned_addr learned my addr 172.21.15.116:0/2698955403 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): get_auth_request con 0x7fc244131ab0 auth_method 0 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 10 monclient(hunting): get_auth_request con 0x7fc24409e910 auth_method 0 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc24409f250 0x7fc24409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 10 monclient(hunting): get_auth_request con 0x7fc24409f250 auth_method 0 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc23bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient(hunting): handle_auth_done global_id 144317 payload 824 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc24409f250 msgr2=0x7fc24409b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc24409f250 0x7fc24409b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24409e910 msgr2=0x7fc244097fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24409e910 0x7fc244097fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 monclient: found mon.noname-c 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc243fff640 1 -- 172.21.15.116:0/2698955403 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c009ae0 con 0x7fc244131ab0 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.423+0000 7fc2437fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24409e910 0x7fc244097fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc243fff640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 msgr2=0x7fc244096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc243fff640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 0x7fc244096a60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009ab0 tx=0x7fc23c036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c032cc0 con 0x7fc244131ab0 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:05.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 10 monclient: dump: 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23c032e20 con 0x7fc244131ab0 2024-09-18T12:38:05.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc249d9d640 5 monclient: authenticate success, global_id 144317 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc249d9d640 10 monclient: _renew_subs 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c03b640 con 0x7fc244131ab0 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc249d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc244099e30 con 0x7fc244131ab0 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc23b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.425+0000 7fc2427fc640 4 set_mon_vals no callback set 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc23b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc23b7fe640 10 monclient: dump: 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc249d9d640 10 monclient: _renew_subs 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc249d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24409a9a0 con 0x7fc244131ab0 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc249d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 0x7fc244097370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc249d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc244063740 con 0x7fc244096fa0 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc23c038050 con 0x7fc244131ab0 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 0x7fc244097370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 10 monclient: get_auth_request con 0x7fc244096fa0 auth_method 0 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.426+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.427+0000 7fc23b7fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc224038490 0x7fc22403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.427+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc23c06ceb0 con 0x7fc244131ab0 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.427+0000 7fc2437fe640 10 monclient(hunting): handle_auth_done global_id 154218 payload 824 2024-09-18T12:38:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.427+0000 7fc2437fe640 1 -- 172.21.15.116:0/2698955403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2440978b0 con 0x7fc244096fa0 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.427+0000 7fc23bfff640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc224038490 0x7fc22403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.427+0000 7fc23bfff640 10 monclient: get_auth_request con 0x7fc224038490 auth_method 0 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc2437fe640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 msgr2=0x7fc244097370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc2437fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 0x7fc244097370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc23400eb00 tx=0x7fc234009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23400ce80 con 0x7fc244096fa0 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 10 monclient: dump: 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc2340123f0 con 0x7fc244096fa0 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 msgr2=0x7fc244097370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 0x7fc244097370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc23400eb00 tx=0x7fc234009060 comp rx=0 tx=0).stop 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 msgr2=0x7fc244097370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23b7fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc244096fa0 0x7fc244097370 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc23400eb00 tx=0x7fc234009060 comp rx=0 tx=0).stop 2024-09-18T12:38:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.429+0000 7fc23bfff640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc224038490 0x7fc22403a940 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc24406a730 tx=0x7fc22c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc249d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 0x7fc24412e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc249d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24404bf60 con 0x7fc24412ddb0 2024-09-18T12:38:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 0x7fc24412e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient: get_auth_request con 0x7fc24412ddb0 auth_method 0 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 10 monclient(hunting): handle_auth_done global_id 154221 payload 824 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.435+0000 7fc2437fe640 1 -- 172.21.15.116:0/2698955403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc244066280 con 0x7fc24412ddb0 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.437+0000 7fc2437fe640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 msgr2=0x7fc24412e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.437+0000 7fc2437fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 0x7fc24412e1b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc23400eb00 tx=0x7fc234015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23401cdc0 con 0x7fc24412ddb0 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 10 monclient: dump: 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7fc23400cc70 con 0x7fc24412ddb0 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 msgr2=0x7fc24412e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 0x7fc24412e1b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc23400eb00 tx=0x7fc234015000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 msgr2=0x7fc24412e1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc23b7fe640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24412ddb0 0x7fc24412e1b0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc23400eb00 tx=0x7fc234015000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:05.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 124, 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.453 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:05.454 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:05.455 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:05.459 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:05.460 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.462 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.463 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:05.464 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc224038490 msgr2=0x7fc22403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc224038490 0x7fc22403a940 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc24406a730 tx=0x7fc22c008040 comp rx=0 tx=0).stop 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 10 monclient: shutdown 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 msgr2=0x7fc244096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 0x7fc244096a60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc23c009ab0 tx=0x7fc23c036000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc243fff640 1 -- 172.21.15.116:0/2698955403 reap_dead start 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 shutdown_connections 2024-09-18T12:38:05.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 --2- 172.21.15.116:0/2698955403 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc244131ab0 0x7fc244096a60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 >> 172.21.15.116:0/2698955403 conn(0x7fc244067d30 msgr2=0x7fc2440a3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 shutdown_connections 2024-09-18T12:38:05.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.438+0000 7fc249d9d640 1 -- 172.21.15.116:0/2698955403 wait complete. 2024-09-18T12:38:05.467 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:38:05.467 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:38:05.467 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:38:05.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.559+0000 7fea3e55d640 10 monclient: get_monmap_and_config 2024-09-18T12:38:05.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.559+0000 7fea3e55d640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.559+0000 7fea3e55d640 10 monclient: monmap: 2024-09-18T12:38:05.560 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:05.560253+0000 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:05.560253+0000 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:05.561 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 1 Processor -- start 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 1 -- start start 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient: init 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea38096c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fea38097030 0x7fea38097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient(hunting): picked mon.noname-c con 0x7fea38097030 addr v2:172.21.15.144:3300/0 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 0x7fea38135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient(hunting): picked mon.noname-b con 0x7fea38133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3809d2e0 con 0x7fea38096c60 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3809d6b0 con 0x7fea38097030 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea380992f0 con 0x7fea38133b40 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.560+0000 7fea3e55d640 10 monclient(hunting): authenticate will time out at 1512.881946s 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 0x7fea38135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 0x7fea38135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41378/0 (socket says 172.21.15.116:41378) 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 -- 172.21.15.116:0/2685603763 learned_addr learned my addr 172.21.15.116:0/2685603763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): get_auth_request con 0x7fea38133b40 auth_method 0 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 1 --2- 172.21.15.116:0/2685603763 >> v2:172.21.15.144:3300/0 conn(0x7fea38097030 0x7fea38097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 10 monclient(hunting): get_auth_request con 0x7fea38097030 auth_method 0 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient(hunting): handle_auth_done global_id 144320 payload 274 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 -- 172.21.15.116:0/2685603763 >> v2:172.21.15.144:3300/0 conn(0x7fea38097030 msgr2=0x7fea38097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 --2- 172.21.15.116:0/2685603763 >> v2:172.21.15.144:3300/0 conn(0x7fea38097030 0x7fea38097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 -- 172.21.15.116:0/2685603763 >> v1:172.21.15.144:6789/0 conn(0x7fea38096c60 legacy=0x7fea3809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 monclient: found mon.noname-b 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.561+0000 7fea3cafa640 1 -- 172.21.15.116:0/2685603763 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea380995d0 con 0x7fea38133b40 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3cafa640 1 -- 172.21.15.116:0/2685603763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 msgr2=0x7fea38135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3cafa640 1 --2- 172.21.15.116:0/2685603763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 0x7fea38135f20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fea30009f20 tx=0x7fea30009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d03a6c6c16669f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 1 -- 172.21.15.116:0/2685603763 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30007610 con 0x7fea38133b40 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: dump: 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 1 -- 172.21.15.116:0/2685603763 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3002d050 con 0x7fea38133b40 2024-09-18T12:38:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 5 monclient: authenticate success, global_id 144320 2024-09-18T12:38:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 1 -- 172.21.15.116:0/2685603763 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea300079a0 con 0x7fea38133b40 2024-09-18T12:38:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 4 set_mon_vals no callback set 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea36ffd640 10 monclient: dump: 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.572 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 10 monclient: shutdown 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 1 -- 172.21.15.116:0/2685603763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 msgr2=0x7fea38135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 1 --2- 172.21.15.116:0/2685603763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 0x7fea38135f20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fea30009f20 tx=0x7fea30009840 comp rx=0 tx=0).stop 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 1 -- 172.21.15.116:0/2685603763 shutdown_connections 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 1 --2- 172.21.15.116:0/2685603763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38133b40 0x7fea38135f20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 1 --2- 172.21.15.116:0/2685603763 >> v2:172.21.15.144:3300/0 conn(0x7fea38097030 0x7fea38097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.564+0000 7fea3e55d640 1 -- 172.21.15.116:0/2685603763 >> 172.21.15.116:0/2685603763 conn(0x7fea380672b0 msgr2=0x7fea380676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.565+0000 7fea3e55d640 1 -- 172.21.15.116:0/2685603763 shutdown_connections 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.565+0000 7fea3e55d640 1 -- 172.21.15.116:0/2685603763 wait complete. 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.565+0000 7fea3e55d640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.565+0000 7fea3e55d640 10 monclient: monmap: 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.565+0000 7fea3e55d640 1 Processor -- start 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 1 -- start start 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient: init 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea38096c60 0x7fea38132340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): picked mon.noname-a con 0x7fea38096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38097030 0x7fea38132880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): picked mon.noname-c con 0x7fea38097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38133b40 0x7fea38132dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): picked mon.noname-b con 0x7fea38133b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3806eb60 con 0x7fea38096c60 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3806aba0 con 0x7fea38133b40 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea381357c0 con 0x7fea38097030 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3e55d640 10 monclient(hunting): authenticate will time out at 1512.887478s 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea38096c60 0x7fea38132340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38097030 0x7fea38132880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38097030 0x7fea38132880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41394/0 (socket says 172.21.15.116:41394) 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea38096c60 0x7fea38132340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53684/0 (socket says 172.21.15.116:53684) 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 -- 172.21.15.116:0/1300789864 learned_addr learned my addr 172.21.15.116:0/1300789864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): get_auth_request con 0x7fea38097030 auth_method 0 2024-09-18T12:38:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 10 monclient(hunting): get_auth_request con 0x7fea38096c60 auth_method 0 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3cafa640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38133b40 0x7fea38132dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient(hunting): handle_auth_done global_id 144323 payload 824 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38133b40 msgr2=0x7fea38132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38133b40 0x7fea38132dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea38096c60 msgr2=0x7fea38132340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea38096c60 0x7fea38132340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 monclient: found mon.noname-c 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea377fe640 1 -- 172.21.15.116:0/1300789864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30009ae0 con 0x7fea38097030 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3cafa640 10 monclient: get_auth_request con 0x7fea38133b40 auth_method 0 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea3cafa640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38133b40 0x7fea38132dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.566+0000 7fea37fff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea38096c60 0x7fea38132340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:05.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea377fe640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38097030 msgr2=0x7fea38132880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea377fe640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38097030 0x7fea38132880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00ea50 tx=0x7fea2c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c00cc20 con 0x7fea38097030 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: dump: 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.586 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea3e55d640 5 monclient: authenticate success, global_id 144323 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c00cd80 con 0x7fea38097030 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea3e55d640 10 monclient: _renew_subs 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea3e55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3806a8c0 con 0x7fea38097030 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c013680 con 0x7fea38097030 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea1ffff640 10 monclient: dump: 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea367fc640 4 set_mon_vals no callback set 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea3e55d640 10 monclient: _renew_subs 2024-09-18T12:38:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea3e55d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.569+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3806a5e0 con 0x7fea38097030 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea3e55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea3e55d640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 0x7fea3812bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea3e55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fea2c013a70 con 0x7fea38097030 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea38065f20 con 0x7fea3809b9e0 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 0x7fea3812bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient: get_auth_request con 0x7fea3809b9e0 auth_method 0 2024-09-18T12:38:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea1ffff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea240384c0 0x7fea2403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fea2c04c670 con 0x7fea38097030 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 10 monclient(hunting): handle_auth_done global_id 144326 payload 824 2024-09-18T12:38:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea37fff640 1 -- 172.21.15.116:0/1300789864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea3804bf60 con 0x7fea3809b9e0 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea3cafa640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea240384c0 0x7fea2403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.570+0000 7fea3cafa640 10 monclient: get_auth_request con 0x7fea240384c0 auth_method 0 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.572+0000 7fea37fff640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 msgr2=0x7fea3812bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.572+0000 7fea37fff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 0x7fea3812bd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea2800fa50 tx=0x7fea28007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.572+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800dda0 con 0x7fea3809b9e0 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.572+0000 7fea1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.572+0000 7fea1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.572+0000 7fea1ffff640 10 monclient: dump: 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fea28014470 con 0x7fea3809b9e0 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea1ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 msgr2=0x7fea3812bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea1ffff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 0x7fea3812bd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea2800fa50 tx=0x7fea28007e10 comp rx=0 tx=0).stop 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 msgr2=0x7fea3812bd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea1ffff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3809b9e0 0x7fea3812bd20 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea2800fa50 tx=0x7fea28007e10 comp rx=0 tx=0).stop 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.573+0000 7fea3cafa640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea240384c0 0x7fea2403a970 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fea380637c0 tx=0x7fea30036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.578+0000 7fea3e55d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.578+0000 7fea3e55d640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 0x7fea3812c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.578+0000 7fea3e55d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.578+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea380983b0 con 0x7fea3812c260 2024-09-18T12:38:05.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 0x7fea3812c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient: get_auth_request con 0x7fea3812c260 auth_method 0 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 10 monclient(hunting): handle_auth_done global_id 144329 payload 824 2024-09-18T12:38:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.579+0000 7fea37fff640 1 -- 172.21.15.116:0/1300789864 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fea3806ad10 con 0x7fea3812c260 2024-09-18T12:38:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea37fff640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 msgr2=0x7fea3812c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea37fff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 0x7fea3812c660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea28019000 tx=0x7fea2801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2801ed20 con 0x7fea3812c260 2024-09-18T12:38:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 10 monclient: dump: 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fea28020410 con 0x7fea3812c260 2024-09-18T12:38:05.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 msgr2=0x7fea3812c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 0x7fea3812c660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea28019000 tx=0x7fea2801a000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 msgr2=0x7fea3812c660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.581+0000 7fea1ffff640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea3812c260 0x7fea3812c660 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea28019000 tx=0x7fea2801a000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:05.597 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 124, 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.598 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.599 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:05.601 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.602 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:05.605 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.606 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:05.607 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:05.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:38:05.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.610 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.610 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:05.610 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:05.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea240384c0 msgr2=0x7fea2403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea240384c0 0x7fea2403a970 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fea380637c0 tx=0x7fea30036000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 10 monclient: shutdown 2024-09-18T12:38:05.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38097030 msgr2=0x7fea38132880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 --2- 172.21.15.116:0/1300789864 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea38097030 0x7fea38132880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00ea50 tx=0x7fea2c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3cafa640 1 -- 172.21.15.116:0/1300789864 reap_dead start 2024-09-18T12:38:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3cafa640 1 -- 172.21.15.116:0/1300789864 reap_dead start 2024-09-18T12:38:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 shutdown_connections 2024-09-18T12:38:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 >> 172.21.15.116:0/1300789864 conn(0x7fea380672b0 msgr2=0x7fea38135cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 shutdown_connections 2024-09-18T12:38:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.582+0000 7fea3e55d640 1 -- 172.21.15.116:0/1300789864 wait complete. 2024-09-18T12:38:05.612 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:38:05.612 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:38:05.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:38:05.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.707+0000 7fe2150c0640 10 monclient: get_monmap_and_config 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.707+0000 7fe2150c0640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.707+0000 7fe2150c0640 10 monclient: monmap: 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:05.707714+0000 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:05.707714+0000 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:05.708 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:05.709 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:05.709 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.707+0000 7fe2150c0640 1 Processor -- start 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.707+0000 7fe2150c0640 1 -- start start 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.707+0000 7fe2150c0640 10 monclient: init 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe210131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe21009e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe210137bc0 0x7fe210139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient(hunting): picked mon.noname-a con 0x7fe210137bc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:38:05.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21009faf0 con 0x7fe210131ab0 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21009fd30 con 0x7fe21009e910 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe210132590 con 0x7fe210137bc0 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe2150c0640 10 monclient(hunting): authenticate will time out at 1513.029206s 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe210137bc0 0x7fe210139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe210131ab0 0x7fe210131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42892/0 (socket says 172.21.15.116:42892) 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20ed9d640 1 -- 172.21.15.116:0/15031329 learned_addr learned my addr 172.21.15.116:0/15031329 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): get_auth_request con 0x7fe210137bc0 auth_method 0 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient(hunting): handle_auth_done global_id 154227 payload 274 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 1 -- 172.21.15.116:0/15031329 >> v1:172.21.15.144:6789/0 conn(0x7fe21009e910 legacy=0x7fe21009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 1 -- 172.21.15.116:0/15031329 >> v1:172.21.15.116:6789/0 conn(0x7fe210131ab0 legacy=0x7fe210131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 1 monclient: found mon.noname-a 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:38:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.708+0000 7fe20f59e640 1 -- 172.21.15.116:0/15031329 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100a1110 con 0x7fe210137bc0 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20f59e640 1 -- 172.21.15.116:0/15031329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 msgr2=0x7fe210139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20f59e640 1 --2- 172.21.15.116:0/15031329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe210139fa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe208009e90 tx=0x7fe208009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58a7a06d67ba26e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 1 -- 172.21.15.116:0/15031329 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208007840 con 0x7fe210137bc0 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: dump: 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe2150c0640 5 monclient: authenticate success, global_id 154227 2024-09-18T12:38:05.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe2150c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 1 -- 172.21.15.116:0/15031329 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20802d050 con 0x7fe210137bc0 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe2150c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe2150c0640 4 set_mon_vals no callback set 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 1 -- 172.21.15.116:0/15031329 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208007bd0 con 0x7fe210137bc0 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.711+0000 7fe20dd9b640 10 monclient: dump: 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 10 monclient: shutdown 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 1 -- 172.21.15.116:0/15031329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 msgr2=0x7fe210139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 1 --2- 172.21.15.116:0/15031329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe210139fa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe208009e90 tx=0x7fe208009780 comp rx=0 tx=0).stop 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 1 -- 172.21.15.116:0/15031329 shutdown_connections 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 1 --2- 172.21.15.116:0/15031329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe210139fa0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 1 -- 172.21.15.116:0/15031329 >> 172.21.15.116:0/15031329 conn(0x7fe210067d30 msgr2=0x7fe2100a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 1 -- 172.21.15.116:0/15031329 shutdown_connections 2024-09-18T12:38:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 1 -- 172.21.15.116:0/15031329 wait complete. 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 10 monclient: build_initial_monmap 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.712+0000 7fe2150c0640 10 monclient: monmap: 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 Processor -- start 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 -- start start 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient: init 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21009e910 0x7fe210137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): picked mon.noname-b con 0x7fe21009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe210131ab0 0x7fe21012e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): picked mon.noname-c con 0x7fe210131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe21012e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): picked mon.noname-a con 0x7fe210137bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe210132e30 con 0x7fe210137bc0 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21009dab0 con 0x7fe21009e910 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21009bde0 con 0x7fe210131ab0 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe2150c0640 10 monclient(hunting): authenticate will time out at 1513.034597s 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe21012e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe210131ab0 0x7fe21012e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe21012e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53708/0 (socket says 172.21.15.116:53708) 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 10 monclient(hunting): get_auth_request con 0x7fe210131ab0 auth_method 0 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 1 -- 172.21.15.116:0/640575154 learned_addr learned my addr 172.21.15.116:0/640575154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): get_auth_request con 0x7fe210137bc0 auth_method 0 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21009e910 0x7fe210137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 10 monclient(hunting): get_auth_request con 0x7fe21009e910 auth_method 0 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.713+0000 7fe20f59e640 10 monclient(hunting): handle_auth_done global_id 154230 payload 824 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe210131ab0 msgr2=0x7fe21012e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe210131ab0 0x7fe21012e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21009e910 msgr2=0x7fe210137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe21009e910 0x7fe210137050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 10 monclient: _finish_hunting 0 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 1 monclient: found mon.noname-a 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20f59e640 1 -- 172.21.15.116:0/640575154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208009a20 con 0x7fe210137bc0 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.714+0000 7fe20e59c640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe210131ab0 0x7fe21012e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe20f59e640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 msgr2=0x7fe21012e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe20f59e640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe21012e5d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe208009de0 tx=0x7fe208037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208031cd0 con 0x7fe210137bc0 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: dump: 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.730 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.731 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe208040070 con 0x7fe210137bc0 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe2150c0640 5 monclient: authenticate success, global_id 154230 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe2150c0640 10 monclient: _renew_subs 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe2150c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe210131690 con 0x7fe210137bc0 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208045690 con 0x7fe210137bc0 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe1f6ffd640 10 monclient: dump: 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.734 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe20d59a640 4 set_mon_vals no callback set 2024-09-18T12:38:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe2150c0640 10 monclient: _renew_subs 2024-09-18T12:38:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe2150c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:38:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.716+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21012f8b0 con 0x7fe210137bc0 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe2150c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe2150c0640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 0x7fe21012eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe2150c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe210063740 con 0x7fe2101303c0 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe208042030 con 0x7fe210137bc0 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe1f6ffd640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1fc038490 0x7fe1fc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 0x7fe21012eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient: get_auth_request con 0x7fe2101303c0 auth_method 0 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe20806db60 con 0x7fe210137bc0 2024-09-18T12:38:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20e59c640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1fc038490 0x7fe1fc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20e59c640 10 monclient: get_auth_request con 0x7fe1fc038490 auth_method 0 2024-09-18T12:38:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_done global_id 164125 payload 824 2024-09-18T12:38:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.717+0000 7fe20ed9d640 1 -- 172.21.15.116:0/640575154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe21004bf60 con 0x7fe2101303c0 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe20ed9d640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 msgr2=0x7fe21012eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe20ed9d640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 0x7fe21012eb10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e710 tx=0x7fe200007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe20e59c640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1fc038490 0x7fe1fc03a940 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe204006fd0 tx=0x7fe204009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200004b50 con 0x7fe2101303c0 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 10 monclient: dump: 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe200004cb0 con 0x7fe2101303c0 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 msgr2=0x7fe21012eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 0x7fe21012eb10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e710 tx=0x7fe200007f00 comp rx=0 tx=0).stop 2024-09-18T12:38:05.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 msgr2=0x7fe21012eb10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.720+0000 7fe1f6ffd640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2101303c0 0x7fe21012eb10 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e710 tx=0x7fe200007f00 comp rx=0 tx=0).stop 2024-09-18T12:38:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.726+0000 7fe2150c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.726+0000 7fe2150c0640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 0x7fe210139180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.726+0000 7fe2150c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.726+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100008d0 con 0x7fe210138d80 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.726+0000 7fe20ed9d640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 0x7fe210139180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient: get_auth_request con 0x7fe210138d80 auth_method 0 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 10 monclient(hunting): handle_auth_done global_id 164128 payload 824 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.727+0000 7fe20ed9d640 1 -- 172.21.15.116:0/640575154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe21012fef0 con 0x7fe210138d80 2024-09-18T12:38:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.729+0000 7fe20ed9d640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 msgr2=0x7fe210139180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.729+0000 7fe20ed9d640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 0x7fe210139180 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e710 tx=0x7fe200019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200015d30 con 0x7fe210138d80 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 10 monclient: dump: 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:05.743 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fe200015e90 con 0x7fe210138d80 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 msgr2=0x7fe210139180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 0x7fe210139180 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e710 tx=0x7fe200019000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 msgr2=0x7fe210139180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe1f6ffd640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe210138d80 0x7fe210139180 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe20000e710 tx=0x7fe200019000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 124, 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:38:05.745 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:05.746 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:05.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.750 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:05.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.752 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:05.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:05.754 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:05.755 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:05.756 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1fc038490 msgr2=0x7fe1fc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe1fc038490 0x7fe1fc03a940 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe204006fd0 tx=0x7fe204009060 comp rx=0 tx=0).stop 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 10 monclient: shutdown 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 msgr2=0x7fe21012e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe20f59e640 1 -- 172.21.15.116:0/640575154 reap_dead start 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe21012e5d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe208009de0 tx=0x7fe208037000 comp rx=0 tx=0).stop 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 shutdown_connections 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 --2- 172.21.15.116:0/640575154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe210137bc0 0x7fe21012e5d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 >> 172.21.15.116:0/640575154 conn(0x7fe210067d30 msgr2=0x7fe21012bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 shutdown_connections 2024-09-18T12:38:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:05.730+0000 7fe2150c0640 1 -- 172.21.15.116:0/640575154 wait complete. 2024-09-18T12:38:05.759 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:38:05.759 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T12:38:05.759 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-18T12:38:05.759 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T12:38:05.759 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:38:05.759 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:38:05.760 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:38:05.773 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:38:05.773 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T12:38:05.773 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T12:38:05.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:38:20.774 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T12:38:20.775 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T12:38:20.775 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:38:20.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.878+0000 7fdfbd39d640 10 monclient: get_monmap_and_config 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.878+0000 7fdfbd39d640 10 monclient: build_initial_monmap 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.878+0000 7fdfbd39d640 10 monclient: monmap: 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:20.879555+0000 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:20.879555+0000 2024-09-18T12:38:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 1 Processor -- start 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 1 -- start start 2024-09-18T12:38:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 10 monclient: init 2024-09-18T12:38:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T12:38:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb8131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:38:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfb809fe10 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdfb8138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.879+0000 7fdfbd39d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb809f140 addr v2:172.21.15.144:3300/0 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfbd39d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb80a14d0 con 0x7fdfb809fe10 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfbd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfbd39d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb81325b0 con 0x7fdfb8131980 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfbd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfbd39d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb80a11f0 con 0x7fdfb809f140 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfbd39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfbd39d640 10 monclient(hunting): authenticate will time out at 1528.201028s 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb77fe640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 msgr2=0x7fdfb8138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38478/0 (socket says 172.21.15.116:38478) 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 learned_addr learned my addr 172.21.15.116:0/1590124525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdfb8131980 auth_method 0 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb67fc640 1 --1- 172.21.15.116:0/1590124525 >> v1:172.21.15.116:6789/0 conn(0x7fdfb809fe10 0x7fdfb809ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48448/0 (socket says 172.21.15.116:48448) 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb77fe640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdfb8138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.880+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:20.882+0000 7fdfb67fc640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.116:6789/0 conn(0x7fdfb809fe10 legacy=0x7fdfb809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb77fe640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 msgr2=0x7fdfb8138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb77fe640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdfb8138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdfb8131980 auth_method 0 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:21.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.081+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:38:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb77fe640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 msgr2=0x7fdfb8138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdfb8131980 auth_method 0 2024-09-18T12:38:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb77fe640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdfb8138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:38:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:38:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:21.481+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:38:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb77fe640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 msgr2=0x7fdfb8138cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb77fe640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdfb8138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:38:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdfb8131980 auth_method 0 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:22.282+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:38:23.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 10 monclient(hunting): tick 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 monclient(hunting): continuing hunt 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 msgr2=0x7fdfb8138cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdfb8138cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 msgr2=0x7fdfb8131d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdfb8131d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.116:6789/0 conn(0x7fdfb809fe10 legacy=0x7fdfb809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 10 monclient: _add_conns ranks=[2,1] 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fdf980036f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf98006260 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb8132290 con 0x7fdf98006260 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb81325b0 con 0x7fdf980036f0 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb4ff9640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb67fc640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.144:6789/0 conn(0x7fdf98006260 legacy=0x7fdf980086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdf980036f0 auth_method 0 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:38:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:23.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:23.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:23.879+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:24.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb67fc640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.144:6789/0 conn(0x7fdf98006260 legacy=0x7fdf980086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:38:24.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:24.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdf980036f0 auth_method 0 2024-09-18T12:38:24.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:24.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:24.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:24.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:38:24.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:24.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:24.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.080+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:38:24.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb67fc640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.144:6789/0 conn(0x7fdf98006260 legacy=0x7fdf980086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:38:24.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdf980036f0 auth_method 0 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:24.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:24.481+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:38:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb67fc640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.144:6789/0 conn(0x7fdf98006260 legacy=0x7fdf980086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:38:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdf980036f0 auth_method 0 2024-09-18T12:38:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:38:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:38:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:38:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:25.282+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:38:26.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): tick 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 monclient(hunting): continuing hunt 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 msgr2=0x7fdf98005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdf980036f0 0x7fdf98005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.144:6789/0 conn(0x7fdf98006260 legacy=0x7fdf980086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdf98009fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb8131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb77fe640 1 -- 172.21.15.116:0/1590124525 reap_dead start 2024-09-18T12:38:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdf9800a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb809f140 addr v2:172.21.15.144:3300/0 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf9800aa60 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9800b040 con 0x7fdf9800aa60 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb81325b0 con 0x7fdfb8131980 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 1 -- 172.21.15.116:0/1590124525 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb80a11f0 con 0x7fdfb809f140 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb4ff9640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb67fc640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 msgr2=0x7fdf9800a4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb67fc640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdf9800a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.116:3300/0 conn(0x7fdfb8131980 0x7fdf98009fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdfb8131980 auth_method 0 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 10 monclient(hunting): handle_auth_done global_id 154233 payload 274 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 msgr2=0x7fdf9800a4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdf9800a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.879+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> v1:172.21.15.116:6789/0 conn(0x7fdf9800aa60 legacy=0x7fdf98008ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.880+0000 7fdfb6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:38:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.880+0000 7fdfb6ffd640 1 monclient: found mon.noname-a 2024-09-18T12:38:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.880+0000 7fdfb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.880+0000 7fdfb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:38:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.880+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb8132290 con 0x7fdfb8131980 2024-09-18T12:38:26.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.882+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb8131980 msgr2=0x7fdf98009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:26.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.882+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb8131980 0x7fdf98009fb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a60a0 tx=0x7fdfb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175d1763e3e14886 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:26.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 1 -- 172.21.15.116:0/1590124525 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb002ca50 con 0x7fdfb8131980 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: dump: 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:26.892 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 1 -- 172.21.15.116:0/1590124525 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb000c040 con 0x7fdfb8131980 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 1 -- 172.21.15.116:0/1590124525 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb002ce20 con 0x7fdfb8131980 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 5 monclient: authenticate success, global_id 154233 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 4 set_mon_vals no callback set 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfb5ffb640 10 monclient: dump: 2024-09-18T12:38:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 10 monclient: shutdown 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 1 -- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb8131980 msgr2=0x7fdf98009fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb8131980 0x7fdf98009fb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80a60a0 tx=0x7fdfb0008040 comp rx=0 tx=0).stop 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 1 -- 172.21.15.116:0/1590124525 shutdown_connections 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/1590124525 >> v2:172.21.15.144:3300/0 conn(0x7fdfb809f140 0x7fdf9800a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/1590124525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb8131980 0x7fdf98009fb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.883+0000 7fdfbd39d640 1 -- 172.21.15.116:0/1590124525 >> 172.21.15.116:0/1590124525 conn(0x7fdfb80a5050 msgr2=0x7fdfb80a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.884+0000 7fdfbd39d640 1 -- 172.21.15.116:0/1590124525 shutdown_connections 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.884+0000 7fdfbd39d640 1 -- 172.21.15.116:0/1590124525 wait complete. 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.884+0000 7fdfbd39d640 10 monclient: build_initial_monmap 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.884+0000 7fdfbd39d640 10 monclient: monmap: 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 Processor -- start 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 -- start start 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient: init 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9800aa60 0x7fdfb809aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf9800aa60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdfb809f140 0x7fdfb8130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb809f140 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfb8131980 0x7fdfb8141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb8131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb8069eb0 con 0x7fdf9800aa60 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb80a4270 con 0x7fdfb8131980 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb811cc60 con 0x7fdfb809f140 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfbd39d640 10 monclient(hunting): authenticate will time out at 1534.207812s 2024-09-18T12:38:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9800aa60 0x7fdfb809aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb67fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdfb809f140 0x7fdfb8130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9800aa60 0x7fdfb809aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35154/0 (socket says 172.21.15.116:35154) 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb6ffd640 1 -- 172.21.15.116:0/520552772 learned_addr learned my addr 172.21.15.116:0/520552772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb67fc640 10 monclient(hunting): get_auth_request con 0x7fdfb809f140 auth_method 0 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb77fe640 1 -- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfb8131980 msgr2=0x7fdfb8141130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdf9800aa60 auth_method 0 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.885+0000 7fdfb77fe640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfb8131980 0x7fdfb8141130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 10 monclient(hunting): handle_auth_done global_id 144335 payload 824 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 1 -- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfb8131980 msgr2=0x7fdfb8141130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfb8131980 0x7fdfb8141130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 1 -- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9800aa60 msgr2=0x7fdfb809aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9800aa60 0x7fdfb809aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 10 monclient: _finish_hunting 0 2024-09-18T12:38:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 1 monclient: found mon.noname-c 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb67fc640 1 -- 172.21.15.116:0/520552772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb000e040 con 0x7fdfb809f140 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.886+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9800aa60 0x7fdfb809aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfb67fc640 1 -- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdfb809f140 msgr2=0x7fdfb8130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfb67fc640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdfb809f140 0x7fdfb8130aa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00014a0 tx=0x7fdfa0007c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 1 -- 172.21.15.116:0/520552772 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0007510 con 0x7fdfb809f140 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: dump: 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: _finish_auth 0 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 1 -- 172.21.15.116:0/520552772 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa00047a0 con 0x7fdfb809f140 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfbd39d640 5 monclient: authenticate success, global_id 144335 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfbd39d640 10 monclient: _renew_subs 2024-09-18T12:38:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfbd39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb811dbc0 con 0x7fdfb809f140 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 1 -- 172.21.15.116:0/520552772 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0007690 con 0x7fdfb809f140 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfaeffd640 10 monclient: dump: 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfb57fa640 4 set_mon_vals no callback set 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.888+0000 7fdfbd39d640 10 monclient: _renew_subs 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.889+0000 7fdfbd39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.889+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb811bb90 con 0x7fdfb809f140 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.889+0000 7fdfaeffd640 1 -- 172.21.15.116:0/520552772 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdfa0013400 con 0x7fdfb809f140 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfbd39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfbd39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfbd39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb804bf60 con 0x7fdfb809f140 2024-09-18T12:38:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfaeffd640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf980396b0 0x7fdf9803bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfaeffd640 1 -- 172.21.15.116:0/520552772 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdfa004b5b0 con 0x7fdfb809f140 2024-09-18T12:38:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf980396b0 0x7fdf9803bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.890+0000 7fdfb6ffd640 10 monclient: get_auth_request con 0x7fdf980396b0 auth_method 0 2024-09-18T12:38:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.893+0000 7fdfb6ffd640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf980396b0 0x7fdf9803bb60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0010300 tx=0x7fdfb0037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.894+0000 7fdfaeffd640 1 -- 172.21.15.116:0/520552772 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdfa000f840 con 0x7fdfb809f140 2024-09-18T12:38:26.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.894+0000 7fdfaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:26.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:26.894+0000 7fdfaeffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.111+0000 7fdfbd39d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:38:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.111+0000 7fdfbd39d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:38:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.111+0000 7fdfbd39d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:38:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.111+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdfb8067f00 con 0x7fdfb809f140 2024-09-18T12:38:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.111+0000 7fdfaeffd640 1 -- 172.21.15.116:0/520552772 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fdfa000f260 con 0x7fdfb809f140 2024-09-18T12:38:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.111+0000 7fdfaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:38:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.111+0000 7fdfaeffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:27.114 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":128,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf980396b0 msgr2=0x7fdf9803bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf980396b0 0x7fdf9803bb60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0010300 tx=0x7fdfb0037000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 10 monclient: shutdown 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdfb809f140 msgr2=0x7fdfb8130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdfb809f140 0x7fdfb8130aa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00014a0 tx=0x7fdfa0007c50 comp rx=0 tx=0).stop 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 shutdown_connections 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdf980396b0 0x7fdf9803bb60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdfb8131980 0x7fdfb8141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdfb809f140 0x7fdfb8130aa0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 --2- 172.21.15.116:0/520552772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9800aa60 0x7fdfb809aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 >> 172.21.15.116:0/520552772 conn(0x7fdfb80a5050 msgr2=0x7fdfb80a4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:27.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 shutdown_connections 2024-09-18T12:38:27.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.113+0000 7fdfbd39d640 1 -- 172.21.15.116:0/520552772 wait complete. 2024-09-18T12:38:27.126 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:38:27.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:38:27.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.218+0000 7efeb391d640 10 monclient: get_monmap_and_config 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.218+0000 7efeb391d640 10 monclient: build_initial_monmap 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.218+0000 7efeb391d640 10 monclient: monmap: 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:27.220103+0000 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:27.220103+0000 2024-09-18T12:38:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:27.221 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.218+0000 7efeb391d640 1 Processor -- start 2024-09-18T12:38:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 1 -- start start 2024-09-18T12:38:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient: init 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efeac0974f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efeac09a8a0 0x7efeac09ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient(hunting): picked mon.noname-c con 0x7efeac09a8a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efeac133070 addr v1:172.21.15.144:6789/0 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac12b210 con 0x7efeac0974f0 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeac0978c0 con 0x7efeac133070 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeac09b220 con 0x7efeac09a8a0 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb391d640 10 monclient(hunting): authenticate will time out at 1534.541698s 2024-09-18T12:38:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb0eb8640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7efeac09a8a0 msgr2=0x7efeac09ace0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb1eba640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7efeac133070 legacy=0x7efeac135450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:38:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb0eb8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efeac09a8a0 0x7efeac09ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb16b9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efeac0974f0 0x7efeac12ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46628/0 (socket says 172.21.15.116:46628) 2024-09-18T12:38:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.219+0000 7efeb16b9640 1 -- 172.21.15.116:0/1716985518 learned_addr learned my addr 172.21.15.116:0/1716985518 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:38:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.221+0000 7efeb16b9640 1 -- 172.21.15.116:0/1716985518 >> v1:172.21.15.116:6789/0 conn(0x7efeac0974f0 legacy=0x7efeac12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539007548 0 0) 0x7efeac12b210 con 0x7efeac0974f0 2024-09-18T12:38:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): my global_id is 154239 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe980034e0 con 0x7efeac0974f0 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436184921 0 0) 0x7efe980034e0 con 0x7efeac0974f0 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 >> v2:172.21.15.144:3300/0 conn(0x7efeac09a8a0 msgr2=0x7efeac09ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 --2- 172.21.15.116:0/1716985518 >> v2:172.21.15.144:3300/0 conn(0x7efeac09a8a0 0x7efeac09ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 >> v1:172.21.15.144:6789/0 conn(0x7efeac133070 legacy=0x7efeac135450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: _finish_hunting 0 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac09ca20 con 0x7efeac0974f0 2024-09-18T12:38:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: _finish_auth 0 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb391d640 5 monclient: authenticate success, global_id 154239 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7efea80017a0 con 0x7efeac0974f0 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb391d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: dump: 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.233 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 >> v1:172.21.15.116:6789/0 conn(0x7efeac0974f0 legacy=0x7efeac12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe980039d0 0x7efe98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): picked mon.a con 0x7efe980039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe98004780 0x7efe98004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): picked mon.c con 0x7efe98004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:27.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe98005330 0x7efe9800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): picked mon.b con 0x7efe98005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac09b220 con 0x7efe980039d0 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9800c2f0 con 0x7efe98005330 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9800c4b0 con 0x7efe98004780 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efea3fff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb391d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe980039d0 0x7efe98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): get_auth_request con 0x7efe980039d0 auth_method 0 2024-09-18T12:38:27.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe98004780 0x7efe98004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 10 monclient(hunting): get_auth_request con 0x7efe98004780 auth_method 0 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb1eba640 1 -- 172.21.15.116:0/1716985518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe98005330 msgr2=0x7efe9800ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb1eba640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe98005330 0x7efe9800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb0eb8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient(hunting): handle_auth_done global_id 154239 payload 293 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 1 -- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe98004780 msgr2=0x7efe98004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe98004780 0x7efe98004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 1 -- 172.21.15.116:0/1716985518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe98005330 msgr2=0x7efe9800ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe98005330 0x7efe9800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient: _finish_hunting 0 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 1 monclient: found mon.a 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb1eba640 1 -- 172.21.15.116:0/1716985518 reap_dead start 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 1 -- 172.21.15.116:0/1716985518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe980034e0 con 0x7efe980039d0 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 10 monclient: _finish_auth 0 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.222+0000 7efeb16b9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efeb16b9640 1 -- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe980039d0 msgr2=0x7efe98003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efeb16b9640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe980039d0 0x7efe98003dc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efea80014d0 tx=0x7efeac09cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233e484e8f01b9a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8005890 con 0x7efe980039d0 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 10 monclient: dump: 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efeb391d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea8005a10 con 0x7efe980039d0 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 1 -- 172.21.15.116:0/1716985518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8005dd0 con 0x7efe980039d0 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efeb391d640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efeb391d640 4 set_mon_vals no callback set 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.225+0000 7efea3fff640 10 monclient: dump: 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 10 monclient: shutdown 2024-09-18T12:38:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 1 -- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe980039d0 msgr2=0x7efe98003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe980039d0 0x7efe98003dc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efea80014d0 tx=0x7efeac09cbb0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 1 -- 172.21.15.116:0/1716985518 shutdown_connections 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 1 --2- 172.21.15.116:0/1716985518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe980039d0 0x7efe98003dc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 1 -- 172.21.15.116:0/1716985518 >> 172.21.15.116:0/1716985518 conn(0x7efeac0672b0 msgr2=0x7efeac0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 1 -- 172.21.15.116:0/1716985518 shutdown_connections 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 1 -- 172.21.15.116:0/1716985518 wait complete. 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 10 monclient: build_initial_monmap 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.226+0000 7efeb391d640 10 monclient: monmap: 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 Processor -- start 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 -- start start 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient: init 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe980039d0 0x7efeac06cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): picked mon.noname-c con 0x7efe980039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac06e400 0x7efeac1322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): picked mon.noname-b con 0x7efeac06e400 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139510 0x7efeac139980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): picked mon.noname-a con 0x7efeac139510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac06d8c0 con 0x7efeac139510 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac098300 con 0x7efeac06e400 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeac12a450 con 0x7efe980039d0 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb391d640 10 monclient(hunting): authenticate will time out at 1534.549828s 2024-09-18T12:38:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb16b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe980039d0 0x7efeac06cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb1eba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139510 0x7efeac139980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb16b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe980039d0 0x7efeac06cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39516/0 (socket says 172.21.15.116:39516) 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb16b9640 1 -- 172.21.15.116:0/2048960351 learned_addr learned my addr 172.21.15.116:0/2048960351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb16b9640 10 monclient(hunting): get_auth_request con 0x7efe980039d0 auth_method 0 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb16b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb16b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb0eb8640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac06e400 msgr2=0x7efeac1322a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb16b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb0eb8640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac06e400 0x7efeac1322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb1eba640 10 monclient(hunting): get_auth_request con 0x7efeac139510 auth_method 0 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb1eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb1eba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.227+0000 7efeb1eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 10 monclient(hunting): handle_auth_done global_id 154242 payload 824 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe980039d0 msgr2=0x7efeac06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe980039d0 0x7efeac06cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac06e400 msgr2=0x7efeac1322a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efeac06e400 0x7efeac1322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 10 monclient: _finish_hunting 0 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 1 monclient: found mon.noname-a 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb1eba640 1 -- 172.21.15.116:0/2048960351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac06f5d0 con 0x7efeac139510 2024-09-18T12:38:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.228+0000 7efeb16b9640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efe980039d0 0x7efeac06cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb1eba640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139510 msgr2=0x7efeac139980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb1eba640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139510 0x7efeac139980 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efeac0987b0 tx=0x7efea4009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea400c940 con 0x7efeac139510 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: dump: 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: _finish_auth 0 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb391d640 5 monclient: authenticate success, global_id 154242 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea4010070 con 0x7efeac139510 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea400ccf0 con 0x7efeac139510 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb391d640 10 monclient: _renew_subs 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb391d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac06f2f0 con 0x7efeac139510 2024-09-18T12:38:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea37fe640 4 set_mon_vals no callback set 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efea17fa640 10 monclient: dump: 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb391d640 10 monclient: _renew_subs 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb391d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.230+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac13cc50 con 0x7efeac139510 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb391d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb391d640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 0x7efeac126920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb391d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac065f20 con 0x7efeac139ec0 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7efea40063c0 con 0x7efeac139510 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 0x7efeac126920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient: get_auth_request con 0x7efeac139ec0 auth_method 0 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efea17fa640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe90038440 0x7efe9003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 10 monclient(hunting): handle_auth_done global_id 154245 payload 824 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efea404bd70 con 0x7efeac139510 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.231+0000 7efeb16b9640 1 -- 172.21.15.116:0/2048960351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efeac04bf60 con 0x7efeac139ec0 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.232+0000 7efeb0eb8640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe90038440 0x7efe9003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.232+0000 7efeb0eb8640 10 monclient: get_auth_request con 0x7efe90038440 auth_method 0 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efeb16b9640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 msgr2=0x7efeac126920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efeb16b9640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 0x7efeac126920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efea8007e80 tx=0x7efea8036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea80049b0 con 0x7efeac139ec0 2024-09-18T12:38:27.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 10 monclient: dump: 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.256 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efea802e4c0 con 0x7efeac139ec0 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 msgr2=0x7efeac126920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 0x7efeac126920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efea8007e80 tx=0x7efea8036000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 msgr2=0x7efeac126920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efea17fa640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139ec0 0x7efeac126920 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efea8007e80 tx=0x7efea8036000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.234+0000 7efeb0eb8640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe90038440 0x7efe9003a8f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efeac06d4b0 tx=0x7efe9c009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb391d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb391d640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 0x7efeac127260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb391d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac0638d0 con 0x7efeac126e60 2024-09-18T12:38:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 0x7efeac127260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient: get_auth_request con 0x7efeac126e60 auth_method 0 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 10 monclient(hunting): handle_auth_done global_id 154248 payload 824 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.240+0000 7efeb16b9640 1 -- 172.21.15.116:0/2048960351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efeac13a290 con 0x7efeac126e60 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efeb16b9640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 msgr2=0x7efeac127260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efeb16b9640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 0x7efeac127260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efea8044000 tx=0x7efea8042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8043b50 con 0x7efeac126e60 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 10 monclient: dump: 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.259 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7efea8054070 con 0x7efeac126e60 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 msgr2=0x7efeac127260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 0x7efeac127260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efea8044000 tx=0x7efea8042000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 msgr2=0x7efeac127260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.242+0000 7efea17fa640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac126e60 0x7efeac127260 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efea8044000 tx=0x7efea8042000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:27.260 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 128, 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:27.261 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:27.262 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:27.263 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:27.264 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.265 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:27.268 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.270 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:27.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.272 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:27.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.273 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:27.273 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:27.273 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:38:27.273 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.273 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.273 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:27.277 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe90038440 msgr2=0x7efe9003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efe90038440 0x7efe9003a8f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efeac06d4b0 tx=0x7efe9c009060 comp rx=0 tx=0).stop 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 10 monclient: shutdown 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139510 msgr2=0x7efeac139980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139510 0x7efeac139980 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efeac0987b0 tx=0x7efea4009660 comp rx=0 tx=0).stop 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb1eba640 1 -- 172.21.15.116:0/2048960351 reap_dead start 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 shutdown_connections 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 --2- 172.21.15.116:0/2048960351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeac139510 0x7efeac139980 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 >> 172.21.15.116:0/2048960351 conn(0x7efeac0672b0 msgr2=0x7efeac09d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 shutdown_connections 2024-09-18T12:38:27.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.243+0000 7efeb391d640 1 -- 172.21.15.116:0/2048960351 wait complete. 2024-09-18T12:38:27.280 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:38:27.280 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:38:27.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:38:27.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.372+0000 7f7275509640 10 monclient: get_monmap_and_config 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.372+0000 7f7275509640 10 monclient: build_initial_monmap 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.372+0000 7f7275509640 10 monclient: monmap: 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:38:27.374151+0000 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:38:27.374151+0000 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:38:27.375 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:38:27.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 1 Processor -- start 2024-09-18T12:38:27.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 1 -- start start 2024-09-18T12:38:27.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient: init 2024-09-18T12:38:27.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:27.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:38:27.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7270131ab0 0x7f7270131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient(hunting): picked mon.noname-a con 0x7f7270131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:38:27.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f727009e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270136bb0 0x7f7270138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient(hunting): picked mon.noname-b con 0x7f7270136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727009faf0 con 0x7f727009e910 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727009fd30 con 0x7f7270131ab0 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72700a1110 con 0x7f7270136bb0 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7275509640 10 monclient(hunting): authenticate will time out at 1534.695775s 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7270131ab0 0x7f7270131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7267fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f727009e910 0x7f727009ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46630/0 (socket says 172.21.15.116:46630) 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270136bb0 0x7f7270138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726effd640 10 monclient(hunting): get_auth_request con 0x7f7270131ab0 auth_method 0 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f7267fff640 1 -- 172.21.15.116:0/2830782113 learned_addr learned my addr 172.21.15.116:0/2830782113 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726f7fe640 10 monclient(hunting): get_auth_request con 0x7f7270136bb0 auth_method 0 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.373+0000 7f726effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 10 monclient(hunting): handle_auth_done global_id 154251 payload 274 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 1 -- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270136bb0 msgr2=0x7f7270138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 1 --2- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270136bb0 0x7f7270138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 1 -- 172.21.15.116:0/2830782113 >> v1:172.21.15.116:6789/0 conn(0x7f727009e910 legacy=0x7f727009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 10 monclient: _finish_hunting 0 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 1 monclient: found mon.noname-a 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:27.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726effd640 1 -- 172.21.15.116:0/2830782113 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72700a13f0 con 0x7f7270131ab0 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.374+0000 7f726f7fe640 1 --2- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270136bb0 0x7f7270138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.376+0000 7f726effd640 1 -- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7270131ab0 msgr2=0x7f7270131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.376+0000 7f726effd640 1 --2- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7270131ab0 0x7f7270131e80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7260009850 tx=0x7f72600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a85121e02bd55ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 1 -- 172.21.15.116:0/2830782113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726002ae60 con 0x7f7270131ab0 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: dump: 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 5 monclient: authenticate success, global_id 154251 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 1 -- 172.21.15.116:0/2830782113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7260032440 con 0x7f7270131ab0 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 10 monclient: get_monmap_and_config success 2024-09-18T12:38:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 4 set_mon_vals no callback set 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 1 -- 172.21.15.116:0/2830782113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72600327e0 con 0x7f7270131ab0 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f726e7fc640 10 monclient: dump: 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.438 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 10 monclient: shutdown 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 -- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7270131ab0 msgr2=0x7f7270131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 --2- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7270131ab0 0x7f7270131e80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7260009850 tx=0x7f72600090f0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 -- 172.21.15.116:0/2830782113 shutdown_connections 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 --2- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270136bb0 0x7f7270138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 --2- 172.21.15.116:0/2830782113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7270131ab0 0x7f7270131e80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 -- 172.21.15.116:0/2830782113 >> 172.21.15.116:0/2830782113 conn(0x7f7270067d30 msgr2=0x7f72700a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:27.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 -- 172.21.15.116:0/2830782113 shutdown_connections 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.377+0000 7f7275509640 1 -- 172.21.15.116:0/2830782113 wait complete. 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.378+0000 7f7275509640 10 monclient: build_initial_monmap 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.378+0000 7f7275509640 10 monclient: monmap: 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:38:27.440 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.378+0000 7f7275509640 1 Processor -- start 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.378+0000 7f7275509640 1 -- start start 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.378+0000 7f7275509640 10 monclient: init 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.378+0000 7f7275509640 10 monclient: _reopen_session rank -1 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f727009e910 0x7f72700973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): picked mon.noname-a con 0x7f727009e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131ab0 0x7f7270097930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): picked mon.noname-c con 0x7f7270131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7270136bb0 0x7f7270097e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): picked mon.noname-b con 0x7f7270136bb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7270099920 con 0x7f727009e910 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72700992d0 con 0x7f7270136bb0 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727009bde0 con 0x7f7270131ab0 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): _renew_subs 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7275509640 10 monclient(hunting): authenticate will time out at 1534.701069s 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f727009e910 0x7f72700973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131ab0 0x7f7270097930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131ab0 0x7f7270097930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:39530/0 (socket says 172.21.15.116:39530) 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): get_auth_request con 0x7f727009e910 auth_method 0 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 1 -- 172.21.15.116:0/2626849510 learned_addr learned my addr 172.21.15.116:0/2626849510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 10 monclient(hunting): get_auth_request con 0x7f7270131ab0 auth_method 0 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726f7fe640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7270136bb0 msgr2=0x7f7270097e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726f7fe640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7270136bb0 0x7f7270097e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient(hunting): handle_auth_done global_id 154254 payload 824 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131ab0 msgr2=0x7f7270097930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131ab0 0x7f7270097930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7270136bb0 msgr2=0x7f7270097e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7270136bb0 0x7f7270097e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient: _finish_hunting 0 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 1 monclient: found mon.noname-a 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f726effd640 1 -- 172.21.15.116:0/2626849510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7268009a20 con 0x7f727009e910 2024-09-18T12:38:27.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.379+0000 7f7267fff640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131ab0 0x7f7270097930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f726effd640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f727009e910 msgr2=0x7f72700973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f726effd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f727009e910 0x7f72700973f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f727006a2f0 tx=0x7f7260032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260031d40 con 0x7f727009e910 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: dump: 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.450 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: _finish_auth 0 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726003d440 con 0x7f727009e910 2024-09-18T12:38:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7275509640 5 monclient: authenticate success, global_id 154254 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7275509640 10 monclient: _renew_subs 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7275509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7260009390 con 0x7f727009e910 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260031ea0 con 0x7f727009e910 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7266ffd640 10 monclient: dump: 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7275509640 10 monclient: _renew_subs 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f726dffb640 4 set_mon_vals no callback set 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7275509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.381+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727009a810 con 0x7f727009e910 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7260039040 con 0x7f727009e910 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7275509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7275509640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 0x7f7270132690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7275509640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7270063740 con 0x7f72700983b0 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 0x7f7270132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 10 monclient: get_auth_request con 0x7f72700983b0 auth_method 0 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7266ffd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7250038440 0x7f725003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.382+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f726006bd20 con 0x7f727009e910 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.383+0000 7f7267fff640 10 monclient(hunting): handle_auth_done global_id 144347 payload 824 2024-09-18T12:38:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.383+0000 7f7267fff640 1 -- 172.21.15.116:0/2626849510 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f727004bf60 con 0x7f72700983b0 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.383+0000 7f726f7fe640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7250038440 0x7f725003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.383+0000 7f726f7fe640 10 monclient: get_auth_request con 0x7f7250038440 auth_method 0 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7267fff640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 msgr2=0x7f7270132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7267fff640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 0x7f7270132690 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f725800eb00 tx=0x7f72580090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725800cee0 con 0x7f72700983b0 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 10 monclient: dump: 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7258013470 con 0x7f72700983b0 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 msgr2=0x7f7270132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 0x7f7270132690 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f725800eb00 tx=0x7f72580090f0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 msgr2=0x7f7270132690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f7266ffd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f72700983b0 0x7f7270132690 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f725800eb00 tx=0x7f72580090f0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.385+0000 7f726f7fe640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7250038440 0x7f725003a8f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f727009b5a0 tx=0x7f726800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7275509640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7275509640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 0x7f7270132bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7275509640 10 monclient(hunting): start opening mon connection 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727009abc0 con 0x7f7270131490 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 0x7f7270132bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient: get_auth_request con 0x7f7270131490 auth_method 0 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:38:27.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 10 monclient(hunting): handle_auth_done global_id 144350 payload 824 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.391+0000 7f7267fff640 1 -- 172.21.15.116:0/2626849510 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f72700008d0 con 0x7f7270131490 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.393+0000 7f7267fff640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 msgr2=0x7f7270132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.393+0000 7f7267fff640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 0x7f7270132bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7258019000 tx=0x7f725801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258023070 con 0x7f7270131490 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 10 monclient: dump: 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:38:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f7258005e20 con 0x7f7270131490 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 msgr2=0x7f7270132bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 0x7f7270132bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7258019000 tx=0x7f725801a000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 msgr2=0x7f7270132bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:38:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7266ffd640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7270131490 0x7f7270132bd0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7258019000 tx=0x7f725801a000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 128, 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:38:27.460 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:38:27.461 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:38:27.462 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:38:27.463 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:38:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:38:27.468 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.469 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:38:27.470 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:38:27.471 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:38:27.472 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:38:27.472 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:38:27.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:38:27.472 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:38:27.472 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:38:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7250038440 msgr2=0x7f725003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7250038440 0x7f725003a8f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f727009b5a0 tx=0x7f726800b040 comp rx=0 tx=0).stop 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 10 monclient: shutdown 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f727009e910 msgr2=0x7f72700973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f727009e910 0x7f72700973f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f727006a2f0 tx=0x7f7260032000 comp rx=0 tx=0).stop 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f726f7fe640 1 -- 172.21.15.116:0/2626849510 reap_dead start 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 shutdown_connections 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 --2- 172.21.15.116:0/2626849510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f727009e910 0x7f72700973f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:38:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 >> 172.21.15.116:0/2626849510 conn(0x7f7270067d30 msgr2=0x7f72700a47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:38:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 shutdown_connections 2024-09-18T12:38:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:38:27.394+0000 7f7275509640 1 -- 172.21.15.116:0/2626849510 wait complete. 2024-09-18T12:38:27.475 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:38:27.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:38:47.476 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:38:47.477 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T12:38:47.477 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T12:38:47.477 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T12:38:47.481 INFO:tasks.ceph.mon.b:Started 2024-09-18T12:38:47.481 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T12:38:47.481 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T12:38:47.481 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:39:02.482 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T12:39:02.483 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T12:39:02.483 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:39:02.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:02.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.585+0000 7f5c5355d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:02.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.585+0000 7f5c5355d640 10 monclient: build_initial_monmap 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.585+0000 7f5c5355d640 10 monclient: monmap: 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:02.585725+0000 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:02.585725+0000 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:02.586 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.585+0000 7f5c5355d640 1 Processor -- start 2024-09-18T12:39:02.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.585+0000 7f5c5355d640 1 -- start start 2024-09-18T12:39:02.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.585+0000 7f5c5355d640 10 monclient: init 2024-09-18T12:39:02.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:02.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T12:39:02.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5c4c131ab0 0x7f5c4c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c4c131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 0x7f5c4c09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c4c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c4c09f220 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4c09faf0 con 0x7f5c4c09f220 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4c09fd30 con 0x7f5c4c131ab0 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c0a1110 con 0x7f5c4c09e910 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c5355d640 10 monclient(hunting): authenticate will time out at 1569.907387s 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 0x7f5c4c09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 0x7f5c4c09ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45872/0 (socket says 172.21.15.116:45872) 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 -- 172.21.15.116:0/3726182025 learned_addr learned my addr 172.21.15.116:0/3726182025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:02.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): get_auth_request con 0x7f5c4c09e910 auth_method 0 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c512f9640 1 --2- 172.21.15.116:0/3726182025 >> v2:172.21.15.144:3300/0 conn(0x7f5c4c131ab0 0x7f5c4c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c512f9640 10 monclient(hunting): get_auth_request con 0x7f5c4c131ab0 auth_method 0 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c512f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c512f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c512f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient(hunting): handle_auth_done global_id 144353 payload 274 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 -- 172.21.15.116:0/3726182025 >> v2:172.21.15.144:3300/0 conn(0x7f5c4c131ab0 msgr2=0x7f5c4c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:02.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 --2- 172.21.15.116:0/3726182025 >> v2:172.21.15.144:3300/0 conn(0x7f5c4c131ab0 0x7f5c4c131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 -- 172.21.15.116:0/3726182025 >> v1:172.21.15.144:6789/0 conn(0x7f5c4c09f220 legacy=0x7f5c4c068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient: _finish_hunting 0 2024-09-18T12:39:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 monclient: found mon.noname-b 2024-09-18T12:39:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.586+0000 7f5c50af8640 1 -- 172.21.15.116:0/3726182025 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c0a13f0 con 0x7f5c4c09e910 2024-09-18T12:39:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c50af8640 1 -- 172.21.15.116:0/3726182025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 msgr2=0x7f5c4c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c50af8640 1 --2- 172.21.15.116:0/3726182025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 0x7f5c4c09ece0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009910 tx=0x7f5c3c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25d777a7b283f708 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 1 -- 172.21.15.116:0/3726182025 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c032440 con 0x7f5c4c09e910 2024-09-18T12:39:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: dump: 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: _finish_auth 0 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 1 -- 172.21.15.116:0/3726182025 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c0325a0 con 0x7f5c4c09e910 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c5355d640 5 monclient: authenticate success, global_id 144353 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c5355d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c5355d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 1 -- 172.21.15.116:0/3726182025 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c032980 con 0x7f5c4c09e910 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c5355d640 4 set_mon_vals no callback set 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.589+0000 7f5c43fff640 10 monclient: dump: 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 10 monclient: shutdown 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 -- 172.21.15.116:0/3726182025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 msgr2=0x7f5c4c09ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 --2- 172.21.15.116:0/3726182025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 0x7f5c4c09ece0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009910 tx=0x7f5c3c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 -- 172.21.15.116:0/3726182025 shutdown_connections 2024-09-18T12:39:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 --2- 172.21.15.116:0/3726182025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c09e910 0x7f5c4c09ece0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 --2- 172.21.15.116:0/3726182025 >> v2:172.21.15.144:3300/0 conn(0x7f5c4c131ab0 0x7f5c4c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 -- 172.21.15.116:0/3726182025 >> 172.21.15.116:0/3726182025 conn(0x7f5c4c067d30 msgr2=0x7f5c4c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 -- 172.21.15.116:0/3726182025 shutdown_connections 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 1 -- 172.21.15.116:0/3726182025 wait complete. 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 10 monclient: build_initial_monmap 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.590+0000 7f5c5355d640 10 monclient: monmap: 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 Processor -- start 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 -- start start 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient: init 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c4c09e910 0x7f5c4c09ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): picked mon.noname-a con 0x7f5c4c09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c4c09f220 0x7f5c4c130ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c4c09f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 0x7f5c4c11b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c4c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c09bde0 con 0x7f5c4c09e910 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c11c850 con 0x7f5c4c09f220 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c09de00 con 0x7f5c4c131ab0 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.591+0000 7f5c5355d640 10 monclient(hunting): authenticate will time out at 1569.912959s 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c4c09e910 0x7f5c4c09ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 0x7f5c4c11b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 0x7f5c4c11b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45880/0 (socket says 172.21.15.116:45880) 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c4c09e910 0x7f5c4c09ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59878/0 (socket says 172.21.15.116:59878) 2024-09-18T12:39:02.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 -- 172.21.15.116:0/2771650610 learned_addr learned my addr 172.21.15.116:0/2771650610 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): get_auth_request con 0x7f5c4c131ab0 auth_method 0 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 10 monclient(hunting): get_auth_request con 0x7f5c4c09e910 auth_method 0 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c4c09f220 0x7f5c4c130ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 10 monclient(hunting): get_auth_request con 0x7f5c4c09f220 auth_method 0 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c50af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient(hunting): handle_auth_done global_id 144356 payload 824 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 -- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c4c09f220 msgr2=0x7f5c4c130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c4c09f220 0x7f5c4c130ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 -- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c4c09e910 msgr2=0x7f5c4c09ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c4c09e910 0x7f5c4c09ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient: _finish_hunting 0 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 monclient: found mon.noname-c 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c51afa640 1 -- 172.21.15.116:0/2771650610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c009450 con 0x7f5c4c131ab0 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.592+0000 7f5c512f9640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c4c09e910 0x7f5c4c09ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c51afa640 1 -- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 msgr2=0x7f5c4c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c51afa640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 0x7f5c4c11b6b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400c890 tx=0x7f5c44002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 1 -- 172.21.15.116:0/2771650610 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4400ce80 con 0x7f5c4c131ab0 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: dump: 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: _finish_auth 0 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 1 -- 172.21.15.116:0/2771650610 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4400aa00 con 0x7f5c4c131ab0 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c5355d640 5 monclient: authenticate success, global_id 144356 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c5355d640 10 monclient: _renew_subs 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 1 -- 172.21.15.116:0/2771650610 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4400ae00 con 0x7f5c4c131ab0 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c5355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c11de00 con 0x7f5c4c131ab0 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c417fa640 10 monclient: dump: 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c437fe640 4 set_mon_vals no callback set 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c5355d640 10 monclient: _renew_subs 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c5355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.594+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c11cf90 con 0x7f5c4c131ab0 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.595+0000 7f5c5355d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.595+0000 7f5c5355d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.595+0000 7f5c5355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.595+0000 7f5c417fa640 1 -- 172.21.15.116:0/2771650610 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5c44012630 con 0x7f5c4c131ab0 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.595+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4c04bf60 con 0x7f5c4c131ab0 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.595+0000 7f5c417fa640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c30038490 0x7f5c3003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.595+0000 7f5c417fa640 1 -- 172.21.15.116:0/2771650610 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5c4404bfd0 con 0x7f5c4c131ab0 2024-09-18T12:39:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.596+0000 7f5c512f9640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c30038490 0x7f5c3003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.596+0000 7f5c512f9640 10 monclient: get_auth_request con 0x7f5c30038490 auth_method 0 2024-09-18T12:39:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.598+0000 7f5c512f9640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c30038490 0x7f5c3003a940 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5c48007fa0 tx=0x7f5c48008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.600+0000 7f5c417fa640 1 -- 172.21.15.116:0/2771650610 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5c440188a0 con 0x7f5c4c131ab0 2024-09-18T12:39:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.600+0000 7f5c417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.600+0000 7f5c417fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:02.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.816+0000 7f5c5355d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:02.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.816+0000 7f5c5355d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:02.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.816+0000 7f5c5355d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.816+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5c4c062f40 con 0x7f5c4c131ab0 2024-09-18T12:39:02.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.816+0000 7f5c417fa640 1 -- 172.21.15.116:0/2771650610 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f5c44016020 con 0x7f5c4c131ab0 2024-09-18T12:39:02.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.816+0000 7f5c417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:02.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.816+0000 7f5c417fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:02.818 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:02.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c30038490 msgr2=0x7f5c3003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c30038490 0x7f5c3003a940 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5c48007fa0 tx=0x7f5c48008040 comp rx=0 tx=0).stop 2024-09-18T12:39:02.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 10 monclient: shutdown 2024-09-18T12:39:02.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 msgr2=0x7f5c4c11b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 0x7f5c4c11b6b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400c890 tx=0x7f5c44002640 comp rx=0 tx=0).stop 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 shutdown_connections 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5c30038490 0x7f5c3003a940 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5c4c131ab0 0x7f5c4c11b6b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5c4c09f220 0x7f5c4c130ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 --2- 172.21.15.116:0/2771650610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c4c09e910 0x7f5c4c09ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 >> 172.21.15.116:0/2771650610 conn(0x7f5c4c067d30 msgr2=0x7f5c4c09e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:02.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 shutdown_connections 2024-09-18T12:39:02.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.818+0000 7f5c5355d640 1 -- 172.21.15.116:0/2771650610 wait complete. 2024-09-18T12:39:02.830 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:39:02.831 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:39:02.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.924+0000 7fa767b14640 10 monclient: get_monmap_and_config 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.924+0000 7fa767b14640 10 monclient: build_initial_monmap 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.924+0000 7fa767b14640 10 monclient: monmap: 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:02.925176+0000 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:02.925176+0000 2024-09-18T12:39:02.925 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:02.926 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.926 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:02.926 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:02.926 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:02.926 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:02.926 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:02.926 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 1 Processor -- start 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 1 -- start start 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient: init 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 0x7fa760095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient(hunting): picked mon.noname-c con 0x7fa7600975f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7600979c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa760097dc0 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa760098200 con 0x7fa760097dc0 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76009d1b0 con 0x7fa7600979c0 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76009d420 con 0x7fa7600975f0 2024-09-18T12:39:02.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa767b14640 10 monclient(hunting): authenticate will time out at 1570.246781s 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa7660b1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa760097dc0 0x7fa76009c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57764/0 (socket says 172.21.15.116:57764) 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.925+0000 7fa7660b1640 1 -- 172.21.15.116:0/3824232490 learned_addr learned my addr 172.21.15.116:0/3824232490 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 1 --2- 172.21.15.116:0/3824232490 >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 0x7fa760095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 10 monclient(hunting): get_auth_request con 0x7fa7600975f0 auth_method 0 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 1 -- 172.21.15.116:0/3824232490 >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 msgr2=0x7fa760095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:39:02.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 1 -- 172.21.15.116:0/3824232490 >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 msgr2=0x7fa760095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 1 --2- 172.21.15.116:0/3824232490 >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 0x7fa760095ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:02.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.926+0000 7fa7658b0640 1 --2- 172.21.15.116:0/3824232490 >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 0x7fa760095ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 -- 172.21.15.116:0/3824232490 >> v1:172.21.15.116:6789/0 conn(0x7fa760097dc0 legacy=0x7fa76009c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801913561 0 0) 0x7fa760098200 con 0x7fa760097dc0 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): my global_id is 154260 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa74c0034e0 con 0x7fa760097dc0 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4082020628 0 0) 0x7fa74c0034e0 con 0x7fa760097dc0 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 msgr2=0x7fa760095ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:02.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 --2- 172.21.15.116:0/3824232490 >> v2:172.21.15.144:3300/0 conn(0x7fa7600975f0 0x7fa760095ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 >> v1:172.21.15.144:6789/0 conn(0x7fa7600979c0 legacy=0x7fa76009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: _finish_hunting 0 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76009d9e0 con 0x7fa760097dc0 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: _finish_auth 0 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa767b14640 5 monclient: authenticate success, global_id 154260 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fa758001b10 con 0x7fa760097dc0 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa767b14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: dump: 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:39:02.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 >> v1:172.21.15.116:6789/0 conn(0x7fa760097dc0 legacy=0x7fa76009c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c003b70 0x7fa74c003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): picked mon.b con 0x7fa74c003b70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c004780 0x7fa74c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): picked mon.c con 0x7fa74c004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa74c0099c0 0x7fa74c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): picked mon.a con 0x7fa74c0099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76009d420 con 0x7fa74c0099c0 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c00c3c0 con 0x7fa74c003b70 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c00c580 con 0x7fa74c004780 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7648ae640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa767b14640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa74c0099c0 0x7fa74c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c004780 0x7fa74c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): get_auth_request con 0x7fa74c0099c0 auth_method 0 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 10 monclient(hunting): get_auth_request con 0x7fa74c004780 auth_method 0 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c003b70 0x7fa74c003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 10 monclient(hunting): get_auth_request con 0x7fa74c003b70 auth_method 0 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:02.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7658b0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient(hunting): handle_auth_done global_id 154260 payload 293 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 -- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c004780 msgr2=0x7fa74c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c004780 0x7fa74c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 -- 172.21.15.116:0/3824232490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c003b70 msgr2=0x7fa74c003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa74c003b70 0x7fa74c003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient: _finish_hunting 0 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 monclient: found mon.a 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 1 -- 172.21.15.116:0/3824232490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c0034e0 con 0x7fa74c0099c0 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 10 monclient: _finish_auth 0 2024-09-18T12:39:02.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.928+0000 7fa7660b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.929+0000 7fa7660b1640 1 -- 172.21.15.116:0/3824232490 reap_dead start 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7660b1640 1 -- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa74c0099c0 msgr2=0x7fa74c00bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7660b1640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa74c0099c0 0x7fa74c00bda0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa758001cf0 tx=0x7fa75800a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e5e84cf856411a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75802b740 con 0x7fa74c0099c0 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 10 monclient: dump: 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.948 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa767b14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75802b900 con 0x7fa74c0099c0 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa767b14640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 1 -- 172.21.15.116:0/3824232490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75802bcc0 con 0x7fa74c0099c0 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa767b14640 4 set_mon_vals no callback set 2024-09-18T12:39:02.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.931+0000 7fa7648ae640 10 monclient: dump: 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 10 monclient: shutdown 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 1 -- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa74c0099c0 msgr2=0x7fa74c00bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa74c0099c0 0x7fa74c00bda0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa758001cf0 tx=0x7fa75800a330 comp rx=0 tx=0).stop 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 1 -- 172.21.15.116:0/3824232490 shutdown_connections 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 1 --2- 172.21.15.116:0/3824232490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa74c0099c0 0x7fa74c00bda0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 1 -- 172.21.15.116:0/3824232490 >> 172.21.15.116:0/3824232490 conn(0x7fa7600678a0 msgr2=0x7fa7600a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 1 -- 172.21.15.116:0/3824232490 shutdown_connections 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 1 -- 172.21.15.116:0/3824232490 wait complete. 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 10 monclient: build_initial_monmap 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.932+0000 7fa767b14640 10 monclient: monmap: 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:02.951 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 Processor -- start 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 -- start start 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient: init 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:39:02.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c0099c0 0x7fa7600942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): picked mon.noname-c con 0x7fa74c0099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760094820 0x7fa7601283f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): picked mon.noname-a con 0x7fa760094820 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa760094bf0 0x7fa760128930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): picked mon.noname-b con 0x7fa760094bf0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76012a690 con 0x7fa760094820 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600983c0 con 0x7fa760094bf0 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76012bca0 con 0x7fa74c0099c0 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:02.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa767b14640 10 monclient(hunting): authenticate will time out at 1570.254771s 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c0099c0 0x7fa7600942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760094820 0x7fa7601283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c0099c0 0x7fa7600942e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45896/0 (socket says 172.21.15.116:45896) 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760094820 0x7fa7601283f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59888/0 (socket says 172.21.15.116:59888) 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 1 -- 172.21.15.116:0/1429105339 learned_addr learned my addr 172.21.15.116:0/1429105339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 10 monclient(hunting): get_auth_request con 0x7fa74c0099c0 auth_method 0 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 10 monclient(hunting): get_auth_request con 0x7fa760094820 auth_method 0 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7658b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7660b1640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa760094bf0 0x7fa760128930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7660b1640 10 monclient(hunting): get_auth_request con 0x7fa760094bf0 auth_method 0 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7660b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7660b1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.933+0000 7fa7660b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7660b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7660b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7660b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 10 monclient(hunting): handle_auth_done global_id 144359 payload 824 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa760094bf0 msgr2=0x7fa760128930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa760094bf0 0x7fa760128930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760094820 msgr2=0x7fa7601283f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760094820 0x7fa7601283f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 10 monclient: _finish_hunting 0 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 1 monclient: found mon.noname-c 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7658b0640 1 -- 172.21.15.116:0/1429105339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76012b9c0 con 0x7fa74c0099c0 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.934+0000 7fa7650af640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760094820 0x7fa7601283f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa7658b0640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c0099c0 msgr2=0x7fa7600942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa7658b0640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c0099c0 0x7fa7600942e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa75c010980 tx=0x7fa75c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c00b580 con 0x7fa74c0099c0 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: dump: 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.957 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: _finish_auth 0 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c00eb70 con 0x7fa74c0099c0 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa767b14640 5 monclient: authenticate success, global_id 144359 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa767b14640 10 monclient: _renew_subs 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa767b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa760129e90 con 0x7fa74c0099c0 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c016480 con 0x7fa74c0099c0 2024-09-18T12:39:02.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa755ffb640 10 monclient: dump: 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa767b14640 10 monclient: _renew_subs 2024-09-18T12:39:02.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa767b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76012aa60 con 0x7fa74c0099c0 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.936+0000 7fa757fff640 4 set_mon_vals no callback set 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa767b14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa767b14640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 0x7fa7600298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa767b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760065e00 con 0x7fa760129630 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa75c016870 con 0x7fa74c0099c0 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 0x7fa7600298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient: get_auth_request con 0x7fa760129630 auth_method 0 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa755ffb640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 10 monclient(hunting): handle_auth_done global_id 154266 payload 824 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa7650af640 1 -- 172.21.15.116:0/1429105339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa76004bf60 con 0x7fa760129630 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.937+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa75c04cb10 con 0x7fa74c0099c0 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.938+0000 7fa7660b1640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.938+0000 7fa7660b1640 10 monclient: get_auth_request con 0x7fa744038490 auth_method 0 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa7650af640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 msgr2=0x7fa7600298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa7650af640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 0x7fa7600298c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa750003f10 tx=0x7fa750007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750015650 con 0x7fa760129630 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:02.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 10 monclient: dump: 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa7500157b0 con 0x7fa760129630 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:02.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 msgr2=0x7fa7600298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 0x7fa7600298c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa750003f10 tx=0x7fa750007ca0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 msgr2=0x7fa7600298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa755ffb640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760129630 0x7fa7600298c0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa750003f10 tx=0x7fa750007ca0 comp rx=0 tx=0).stop 2024-09-18T12:39:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.940+0000 7fa7660b1640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa760098190 tx=0x7fa758031000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.945+0000 7fa767b14640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa767b14640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 0x7fa76002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa767b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76012be10 con 0x7fa760029e00 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 0x7fa76002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient: get_auth_request con 0x7fa760029e00 auth_method 0 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:02.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 10 monclient(hunting): handle_auth_done global_id 154269 payload 824 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.946+0000 7fa7650af640 1 -- 172.21.15.116:0/1429105339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa76006deb0 con 0x7fa760029e00 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa7650af640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 msgr2=0x7fa76002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa7650af640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 0x7fa76002a200 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa750003f10 tx=0x7fa75001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75001ed20 con 0x7fa760029e00 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 10 monclient: dump: 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:02.966 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7fa7500283a0 con 0x7fa760029e00 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 msgr2=0x7fa76002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 0x7fa76002a200 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa750003f10 tx=0x7fa75001c000 comp rx=0 tx=0).stop 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 msgr2=0x7fa76002a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:02.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.948+0000 7fa755ffb640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760029e00 0x7fa76002a200 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa750003f10 tx=0x7fa75001c000 comp rx=0 tx=0).stop 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 136, 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:39:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:39:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:02.981 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:02.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 msgr2=0x7fa74403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa744038490 0x7fa74403a940 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa760098190 tx=0x7fa758031000 comp rx=0 tx=0).stop 2024-09-18T12:39:02.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 10 monclient: shutdown 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c0099c0 msgr2=0x7fa7600942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 --2- 172.21.15.116:0/1429105339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa74c0099c0 0x7fa7600942e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa75c010980 tx=0x7fa75c008040 comp rx=0 tx=0).stop 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa7660b1640 1 -- 172.21.15.116:0/1429105339 reap_dead start 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa7660b1640 1 -- 172.21.15.116:0/1429105339 reap_dead start 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 shutdown_connections 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 >> 172.21.15.116:0/1429105339 conn(0x7fa7600678a0 msgr2=0x7fa76009be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 shutdown_connections 2024-09-18T12:39:02.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:02.949+0000 7fa767b14640 1 -- 172.21.15.116:0/1429105339 wait complete. 2024-09-18T12:39:02.983 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:02.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:39:02.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:39:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.077+0000 7f630955d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.077+0000 7f630955d640 10 monclient: build_initial_monmap 2024-09-18T12:39:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.077+0000 7f630955d640 10 monclient: monmap: 2024-09-18T12:39:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:03.078060+0000 2024-09-18T12:39:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:03.078060+0000 2024-09-18T12:39:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:03.081 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.077+0000 7f630955d640 1 Processor -- start 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 1 -- start start 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient: init 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63040a57e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6304131e90 0x7f630409f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient(hunting): picked mon.noname-c con 0x7f6304131e90 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient(hunting): picked mon.noname-b con 0x7f6304136bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630409fbd0 con 0x7f63040a57e0 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040a13f0 con 0x7f6304131e90 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040a5bb0 con 0x7f6304136bd0 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f630955d640 10 monclient(hunting): authenticate will time out at 1570.399503s 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304138fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45910/0 (socket says 172.21.15.116:45910) 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 -- 172.21.15.116:0/2138105597 learned_addr learned my addr 172.21.15.116:0/2138105597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): get_auth_request con 0x7f6304136bd0 auth_method 0 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63027fc640 1 --2- 172.21.15.116:0/2138105597 >> v2:172.21.15.144:3300/0 conn(0x7f6304131e90 0x7f630409f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient(hunting): handle_auth_done global_id 144362 payload 274 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 -- 172.21.15.116:0/2138105597 >> v2:172.21.15.144:3300/0 conn(0x7f6304131e90 msgr2=0x7f630409f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 --2- 172.21.15.116:0/2138105597 >> v2:172.21.15.144:3300/0 conn(0x7f6304131e90 0x7f630409f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 -- 172.21.15.116:0/2138105597 >> v1:172.21.15.144:6789/0 conn(0x7f63040a57e0 legacy=0x7f630409e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 monclient: found mon.noname-b 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63037fe640 1 -- 172.21.15.116:0/2138105597 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040a1110 con 0x7f6304136bd0 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63027fc640 10 monclient: get_auth_request con 0x7f6304131e90 auth_method 0 2024-09-18T12:39:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.078+0000 7f63027fc640 1 --2- 172.21.15.116:0/2138105597 >> v2:172.21.15.144:3300/0 conn(0x7f6304131e90 0x7f630409f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f63037fe640 1 -- 172.21.15.116:0/2138105597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 msgr2=0x7f6304138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f63037fe640 1 --2- 172.21.15.116:0/2138105597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304138fb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009910 tx=0x7f62f40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1716cbc256dbc70f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 1 -- 172.21.15.116:0/2138105597 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4032440 con 0x7f6304136bd0 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: dump: 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: _finish_auth 0 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 1 -- 172.21.15.116:0/2138105597 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f40325a0 con 0x7f6304136bd0 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f630955d640 5 monclient: authenticate success, global_id 144362 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f630955d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f630955d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 1 -- 172.21.15.116:0/2138105597 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4032940 con 0x7f6304136bd0 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f630955d640 4 set_mon_vals no callback set 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.081+0000 7f6301ffb640 10 monclient: dump: 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 10 monclient: shutdown 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 -- 172.21.15.116:0/2138105597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 msgr2=0x7f6304138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 --2- 172.21.15.116:0/2138105597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304138fb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009910 tx=0x7f62f40091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 -- 172.21.15.116:0/2138105597 shutdown_connections 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 --2- 172.21.15.116:0/2138105597 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304138fb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 --2- 172.21.15.116:0/2138105597 >> v2:172.21.15.144:3300/0 conn(0x7f6304131e90 0x7f630409f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 -- 172.21.15.116:0/2138105597 >> 172.21.15.116:0/2138105597 conn(0x7f6304067760 msgr2=0x7f63040a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 -- 172.21.15.116:0/2138105597 shutdown_connections 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.082+0000 7f630955d640 1 -- 172.21.15.116:0/2138105597 wait complete. 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient: build_initial_monmap 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient: monmap: 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 Processor -- start 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 -- start start 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient: init 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63040a57e0 0x7f63041313b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): picked mon.noname-a con 0x7f63040a57e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6304131e90 0x7f6304097250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): picked mon.noname-b con 0x7f6304131e90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): picked mon.noname-c con 0x7f6304136bd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630409bae0 con 0x7f63040a57e0 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630409be50 con 0x7f6304131e90 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040a4460 con 0x7f6304136bd0 2024-09-18T12:39:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f630955d640 10 monclient(hunting): authenticate will time out at 1570.404893s 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f63037fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.083+0000 7f6302ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63040a57e0 0x7f63041313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304097790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45922/0 (socket says 172.21.15.116:45922) 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 -- 172.21.15.116:0/470195011 learned_addr learned my addr 172.21.15.116:0/470195011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): get_auth_request con 0x7f6304136bd0 auth_method 0 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 10 monclient(hunting): get_auth_request con 0x7f63040a57e0 auth_method 0 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63027fc640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6304131e90 0x7f6304097250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient(hunting): handle_auth_done global_id 144365 payload 824 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6304131e90 msgr2=0x7f6304097250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6304131e90 0x7f6304097250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63040a57e0 msgr2=0x7f63041313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63040a57e0 0x7f63041313b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 monclient: found mon.noname-c 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63037fe640 1 -- 172.21.15.116:0/470195011 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4009450 con 0x7f6304136bd0 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63027fc640 10 monclient: get_auth_request con 0x7f6304131e90 auth_method 0 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f6302ffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63040a57e0 0x7f63041313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.084+0000 7f63027fc640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6304131e90 0x7f6304097250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f63037fe640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 msgr2=0x7f6304097790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f63037fe640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304097790 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f62f4035040 tx=0x7f62f4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4031c70 con 0x7f6304136bd0 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: dump: 2024-09-18T12:39:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: _finish_auth 0 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4031df0 con 0x7f6304136bd0 2024-09-18T12:39:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f630955d640 5 monclient: authenticate success, global_id 144365 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f630955d640 10 monclient: _renew_subs 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f630955d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630409a8c0 con 0x7f6304136bd0 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f403d690 con 0x7f6304136bd0 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f62faffd640 10 monclient: dump: 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.102 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f63017fa640 4 set_mon_vals no callback set 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f630955d640 10 monclient: _renew_subs 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f630955d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.086+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6304099960 con 0x7f6304136bd0 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f630955d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f630955d640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 0x7f6304132770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f630955d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6304065e00 con 0x7f6304097cd0 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f62f4039040 con 0x7f6304136bd0 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 0x7f6304132770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient: get_auth_request con 0x7f6304097cd0 auth_method 0 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f62faffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62e4038490 0x7f62e403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 10 monclient(hunting): handle_auth_done global_id 144368 payload 824 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f62f406c330 con 0x7f6304136bd0 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.087+0000 7f6302ffd640 1 -- 172.21.15.116:0/470195011 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6304099c10 con 0x7f6304097cd0 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.088+0000 7f63027fc640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62e4038490 0x7f62e403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.088+0000 7f63027fc640 10 monclient: get_auth_request con 0x7f62e4038490 auth_method 0 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f6302ffd640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 msgr2=0x7f6304132770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f6302ffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 0x7f6304132770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e4f0 tx=0x7f62fc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc013900 con 0x7f6304097cd0 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 10 monclient: dump: 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f62fc00f040 con 0x7f6304097cd0 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 msgr2=0x7f6304132770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 0x7f6304132770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e4f0 tx=0x7f62fc009780 comp rx=0 tx=0).stop 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 msgr2=0x7f6304132770 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f62faffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304097cd0 0x7f6304132770 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e4f0 tx=0x7f62fc009780 comp rx=0 tx=0).stop 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.090+0000 7f63027fc640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62e4038490 0x7f62e403a940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f63040a57b0 tx=0x7f62ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f630955d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f630955d640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 0x7f63041330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f630955d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630409aad0 con 0x7f6304132cb0 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 0x7f63041330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient: get_auth_request con 0x7f6304132cb0 auth_method 0 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 10 monclient(hunting): handle_auth_done global_id 144371 payload 824 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.096+0000 7f6302ffd640 1 -- 172.21.15.116:0/470195011 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f630404bf60 con 0x7f6304132cb0 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f6302ffd640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 msgr2=0x7f63041330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f6302ffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 0x7f63041330b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f62fc018000 tx=0x7f62fc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc01f860 con 0x7f6304132cb0 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 10 monclient: dump: 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.110 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f62fc01f9c0 con 0x7f6304132cb0 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 msgr2=0x7f63041330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 0x7f63041330b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f62fc018000 tx=0x7f62fc016000 comp rx=0 tx=0).stop 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 msgr2=0x7f63041330b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.098+0000 7f62faffd640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304132cb0 0x7f63041330b0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f62fc018000 tx=0x7f62fc016000 comp rx=0 tx=0).stop 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 136, 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:03.115 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.116 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:03.117 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:03.118 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:03.119 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:03.120 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.121 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:03.122 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:03.123 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:03.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62e4038490 msgr2=0x7f62e403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f62e4038490 0x7f62e403a940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f63040a57b0 tx=0x7f62ec008040 comp rx=0 tx=0).stop 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 10 monclient: shutdown 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 msgr2=0x7f6304097790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304097790 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f62f4035040 tx=0x7f62f4032000 comp rx=0 tx=0).stop 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f63037fe640 1 -- 172.21.15.116:0/470195011 reap_dead start 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 shutdown_connections 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 --2- 172.21.15.116:0/470195011 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6304136bd0 0x7f6304097790 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 >> 172.21.15.116:0/470195011 conn(0x7f6304067760 msgr2=0x7f6304137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 shutdown_connections 2024-09-18T12:39:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.099+0000 7f630955d640 1 -- 172.21.15.116:0/470195011 wait complete. 2024-09-18T12:39:03.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:03.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:39:03.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:39:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.218+0000 7f228035d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.218+0000 7f228035d640 10 monclient: build_initial_monmap 2024-09-18T12:39:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.218+0000 7f228035d640 10 monclient: monmap: 2024-09-18T12:39:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:03.219178+0000 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:03.219178+0000 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 1 Processor -- start 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 1 -- start start 2024-09-18T12:39:03.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient: init 2024-09-18T12:39:03.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:03.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T12:39:03.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f227806eb80 0x7f227806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient(hunting): picked mon.noname-c con 0x7f227806eb80 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:03.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2278068a20 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:03.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f227806f490 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2278069b80 con 0x7f2278068a20 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2278069e00 con 0x7f227806f490 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2278099f90 con 0x7f227806eb80 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f228035d640 10 monclient(hunting): authenticate will time out at 1570.540765s 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f227d8f8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2278068a20 0x7f227809c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57768/0 (socket says 172.21.15.116:57768) 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.219+0000 7f227d8f8640 1 -- 172.21.15.116:0/436602150 learned_addr learned my addr 172.21.15.116:0/436602150 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 1 --2- 172.21.15.116:0/436602150 >> v2:172.21.15.144:3300/0 conn(0x7f227806eb80 0x7f227806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 10 monclient(hunting): get_auth_request con 0x7f227806eb80 auth_method 0 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.220+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 -- 172.21.15.116:0/436602150 >> v1:172.21.15.116:6789/0 conn(0x7f2278068a20 legacy=0x7f227809c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134192345 0 0) 0x7f2278069b80 con 0x7f2278068a20 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): my global_id is 154275 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22640034e0 con 0x7f2278068a20 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 1 -- 172.21.15.116:0/436602150 >> v1:172.21.15.144:6789/0 conn(0x7f227806f490 legacy=0x7f2278138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902398874 0 0) 0x7f22640034e0 con 0x7f2278068a20 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 >> v2:172.21.15.144:3300/0 conn(0x7f227806eb80 msgr2=0x7f227806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 --2- 172.21.15.116:0/436602150 >> v2:172.21.15.144:3300/0 conn(0x7f227806eb80 0x7f227806ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 >> v1:172.21.15.144:6789/0 conn(0x7f227806f490 legacy=0x7f2278138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: _finish_hunting 0 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227809cc50 con 0x7f2278068a20 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: _finish_auth 0 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f22680017a0 con 0x7f2278068a20 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f228035d640 5 monclient: authenticate success, global_id 154275 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f228035d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:39:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: dump: 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 >> v1:172.21.15.116:6789/0 conn(0x7f2278068a20 legacy=0x7f227809c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264003bc0 0x7f2264003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): picked mon.b con 0x7f2264003bc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2264004780 0x7f2264004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): picked mon.a con 0x7f2264004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22640099c0 0x7f226400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): picked mon.c con 0x7f22640099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2278099f90 con 0x7f2264004780 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226400c360 con 0x7f2264003bc0 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226400c520 con 0x7f22640099c0 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d0f7640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f228035d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2264004780 0x7f2264004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): get_auth_request con 0x7f2264004780 auth_method 0 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22640099c0 0x7f226400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264003bc0 0x7f2264003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 10 monclient(hunting): get_auth_request con 0x7f22640099c0 auth_method 0 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 10 monclient(hunting): get_auth_request con 0x7f2264003bc0 auth_method 0 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient(hunting): handle_auth_done global_id 154275 payload 293 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 -- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22640099c0 msgr2=0x7f226400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22640099c0 0x7f226400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 -- 172.21.15.116:0/436602150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264003bc0 msgr2=0x7f2264003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264003bc0 0x7f2264003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient: _finish_hunting 0 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 monclient: found mon.a 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 1 -- 172.21.15.116:0/436602150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22640034e0 con 0x7f2264004780 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 10 monclient: _finish_auth 0 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227d8f8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:03.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.222+0000 7f227e8fa640 1 -- 172.21.15.116:0/436602150 reap_dead start 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d8f8640 1 -- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2264004780 msgr2=0x7f2264004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d8f8640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2264004780 0x7f2264004bf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f22680014d0 tx=0x7f2268007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b303128851859ef server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268005890 con 0x7f2264004780 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 10 monclient: dump: 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.245 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2268005a10 con 0x7f2264004780 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 1 -- 172.21.15.116:0/436602150 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268005dd0 con 0x7f2264004780 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f228035d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f228035d640 4 set_mon_vals no callback set 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.225+0000 7f227d0f7640 10 monclient: dump: 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.246 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 10 monclient: shutdown 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 1 -- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2264004780 msgr2=0x7f2264004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2264004780 0x7f2264004bf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f22680014d0 tx=0x7f2268007a90 comp rx=0 tx=0).stop 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 1 -- 172.21.15.116:0/436602150 shutdown_connections 2024-09-18T12:39:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 1 --2- 172.21.15.116:0/436602150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2264004780 0x7f2264004bf0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 1 -- 172.21.15.116:0/436602150 >> 172.21.15.116:0/436602150 conn(0x7f2278067cd0 msgr2=0x7f22780a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 1 -- 172.21.15.116:0/436602150 shutdown_connections 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 1 -- 172.21.15.116:0/436602150 wait complete. 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 10 monclient: build_initial_monmap 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.226+0000 7f228035d640 10 monclient: monmap: 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 Processor -- start 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 -- start start 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient: init 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264004780 0x7f2278133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): picked mon.noname-b con 0x7f2264004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22781341c0 0x7f2278140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): picked mon.noname-c con 0x7f22781341c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:03.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2278141330 0x7f227811e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): picked mon.noname-a con 0x7f2278141330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22781345b0 con 0x7f2278141330 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227809cf40 con 0x7f2264004780 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22781417a0 con 0x7f22781341c0 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f228035d640 10 monclient(hunting): authenticate will time out at 1570.548698s 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22781341c0 0x7f2278140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2278141330 0x7f227811e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22781341c0 0x7f2278140df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45968/0 (socket says 172.21.15.116:45968) 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 1 -- 172.21.15.116:0/1525966636 learned_addr learned my addr 172.21.15.116:0/1525966636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 10 monclient(hunting): get_auth_request con 0x7f22781341c0 auth_method 0 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 10 monclient(hunting): get_auth_request con 0x7f2278141330 auth_method 0 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227d8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.227+0000 7f227e8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264004780 0x7f2278133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227e0f9640 10 monclient(hunting): get_auth_request con 0x7f2264004780 auth_method 0 2024-09-18T12:39:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227e0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 10 monclient(hunting): handle_auth_done global_id 144374 payload 824 2024-09-18T12:39:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264004780 msgr2=0x7f2278133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2264004780 0x7f2278133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2278141330 msgr2=0x7f227811e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2278141330 0x7f227811e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 10 monclient: _finish_hunting 0 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 1 monclient: found mon.noname-c 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227d8f8640 1 -- 172.21.15.116:0/1525966636 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2278122090 con 0x7f22781341c0 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.228+0000 7f227e8fa640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2278141330 0x7f227811e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f227d8f8640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22781341c0 msgr2=0x7f2278140df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f227d8f8640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22781341c0 0x7f2278140df0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f226802af80 tx=0x7f2268034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268007510 con 0x7f22781341c0 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: dump: 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f228035d640 5 monclient: authenticate success, global_id 144374 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22680076b0 con 0x7f22781341c0 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226803a660 con 0x7f22781341c0 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f228035d640 10 monclient: _renew_subs 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f228035d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:03.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2278121db0 con 0x7f22781341c0 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f226e7fc640 10 monclient: dump: 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f227c8f6640 4 set_mon_vals no callback set 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f228035d640 10 monclient: _renew_subs 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f228035d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.230+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227811ed30 con 0x7f22781341c0 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f228035d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f228035d640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f228035d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f226803a9d0 con 0x7f22781341c0 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2278065e00 con 0x7f227811f010 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f226e7fc640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f225c038490 0x7f225c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f226806c830 con 0x7f22781341c0 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient: get_auth_request con 0x7f227811f010 auth_method 0 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e8fa640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f225c038490 0x7f225c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e8fa640 10 monclient: get_auth_request con 0x7f225c038490 auth_method 0 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 10 monclient(hunting): handle_auth_done global_id 174118 payload 824 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.231+0000 7f227e0f9640 1 -- 172.21.15.116:0/1525966636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22781210f0 con 0x7f227811f010 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.232+0000 7f227e0f9640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 msgr2=0x7f227806b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.232+0000 7f227e0f9640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 msgr2=0x7f227806b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.232+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f22740134e0 tx=0x7f227400b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.232+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:03.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.234+0000 7f227e8fa640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f225c038490 0x7f225c03a940 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f227806a110 tx=0x7f2270007c80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: get_auth_request con 0x7f227811f010 auth_method 0 2024-09-18T12:39:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: _init_auth method 2 2024-09-18T12:39:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: _init_auth already have auth, reseting 2024-09-18T12:39:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: handle_auth_reply_more payload 9 2024-09-18T12:39:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:03.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.433+0000 7f227e0f9640 10 monclient: handle_auth_done global_id 174118 payload 293 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f227e0f9640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 msgr2=0x7f227806b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f22740134e0 tx=0x7f2274018040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227400e040 con 0x7f227811f010 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 10 monclient: dump: 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.437 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.438 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.438 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.438 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.438 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.438 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f227400f040 con 0x7f227811f010 2024-09-18T12:39:03.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 msgr2=0x7f227806b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f22740134e0 tx=0x7f2274018040 comp rx=0 tx=0).stop 2024-09-18T12:39:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 msgr2=0x7f227806b840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.436+0000 7f226e7fc640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f010 0x7f227806b840 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f22740134e0 tx=0x7f2274018040 comp rx=0 tx=0).stop 2024-09-18T12:39:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.442+0000 7f228035d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.442+0000 7f228035d640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 0x7f227806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.442+0000 7f228035d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.442+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227811fd00 con 0x7f227811f3e0 2024-09-18T12:39:03.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 0x7f227806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient: get_auth_request con 0x7f227811f3e0 auth_method 0 2024-09-18T12:39:03.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 10 monclient(hunting): handle_auth_done global_id 174121 payload 824 2024-09-18T12:39:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.443+0000 7f227e0f9640 1 -- 172.21.15.116:0/1525966636 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f22781222d0 con 0x7f227811f3e0 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.445+0000 7f227e0f9640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 msgr2=0x7f227806bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.445+0000 7f227e0f9640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 0x7f227806bd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f22740134e0 tx=0x7f2274020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.445+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274022600 con 0x7f227811f3e0 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.445+0000 7f226e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.445+0000 7f226e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.445+0000 7f226e7fc640 10 monclient: dump: 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f2274015a50 con 0x7f227811f3e0 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f226e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f226e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 msgr2=0x7f227806bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f226e7fc640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 0x7f227806bd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f22740134e0 tx=0x7f2274020000 comp rx=0 tx=0).stop 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f226e7fc640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 msgr2=0x7f227806bd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f226e7fc640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f227811f3e0 0x7f227806bd80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.448 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 136, 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:03.449 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:03.450 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:03.451 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:03.452 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:03.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:03.454 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:03.455 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.456 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:03.457 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:03.459 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:03.460 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:03.460 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:03.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f225c038490 msgr2=0x7f225c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f225c038490 0x7f225c03a940 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f227806a110 tx=0x7f2270007c80 comp rx=0 tx=0).stop 2024-09-18T12:39:03.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 10 monclient: shutdown 2024-09-18T12:39:03.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:03.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22781341c0 msgr2=0x7f2278140df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 --2- 172.21.15.116:0/1525966636 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f22781341c0 0x7f2278140df0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f226802af80 tx=0x7f2268034000 comp rx=0 tx=0).stop 2024-09-18T12:39:03.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f227e8fa640 1 -- 172.21.15.116:0/1525966636 reap_dead start 2024-09-18T12:39:03.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f227e8fa640 1 -- 172.21.15.116:0/1525966636 reap_dead start 2024-09-18T12:39:03.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 shutdown_connections 2024-09-18T12:39:03.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 >> 172.21.15.116:0/1525966636 conn(0x7f2278067cd0 msgr2=0x7f22780a1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:03.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 shutdown_connections 2024-09-18T12:39:03.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.446+0000 7f228035d640 1 -- 172.21.15.116:0/1525966636 wait complete. 2024-09-18T12:39:03.462 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:03.462 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:39:03.463 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:39:03.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.555+0000 7f614af9d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:03.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.555+0000 7f614af9d640 10 monclient: build_initial_monmap 2024-09-18T12:39:03.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.555+0000 7f614af9d640 10 monclient: monmap: 2024-09-18T12:39:03.557 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:03.557 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.557 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:03.555830+0000 2024-09-18T12:39:03.557 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:03.555830+0000 2024-09-18T12:39:03.558 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:03.558 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.558 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:03.558 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:03.558 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:03.559 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:03.559 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:03.559 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.555+0000 7f614af9d640 1 Processor -- start 2024-09-18T12:39:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.555+0000 7f614af9d640 1 -- start start 2024-09-18T12:39:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.555+0000 7f614af9d640 10 monclient: init 2024-09-18T12:39:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T12:39:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6144131ab0 0x7f6144131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient(hunting): picked mon.noname-a con 0x7f6144131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f614409e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6144136bb0 0x7f6144138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient(hunting): picked mon.noname-c con 0x7f6144136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614409faf0 con 0x7f614409e910 2024-09-18T12:39:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614409fd30 con 0x7f6144131ab0 2024-09-18T12:39:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61440a1110 con 0x7f6144136bb0 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614af9d640 10 monclient(hunting): authenticate will time out at 1570.877452s 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6144131ab0 0x7f6144131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6144131ab0 0x7f6144131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59944/0 (socket says 172.21.15.116:59944) 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 -- 172.21.15.116:0/64018133 learned_addr learned my addr 172.21.15.116:0/64018133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): get_auth_request con 0x7f6144131ab0 auth_method 0 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614953a640 1 --2- 172.21.15.116:0/64018133 >> v2:172.21.15.144:3300/0 conn(0x7f6144136bb0 0x7f6144138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614953a640 10 monclient(hunting): get_auth_request con 0x7f6144136bb0 auth_method 0 2024-09-18T12:39:03.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614953a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614953a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f614953a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient(hunting): handle_auth_done global_id 154281 payload 274 2024-09-18T12:39:03.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 -- 172.21.15.116:0/64018133 >> v2:172.21.15.144:3300/0 conn(0x7f6144136bb0 msgr2=0x7f6144138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:03.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 --2- 172.21.15.116:0/64018133 >> v2:172.21.15.144:3300/0 conn(0x7f6144136bb0 0x7f6144138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 -- 172.21.15.116:0/64018133 >> v1:172.21.15.144:6789/0 conn(0x7f614409e910 legacy=0x7f614409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient: _finish_hunting 0 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 monclient: found mon.noname-a 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.556+0000 7f6148d39640 1 -- 172.21.15.116:0/64018133 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440a13f0 con 0x7f6144131ab0 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f6148d39640 1 -- 172.21.15.116:0/64018133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6144131ab0 msgr2=0x7f6144131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f6148d39640 1 --2- 172.21.15.116:0/64018133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6144131ab0 0x7f6144131e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f613c009910 tx=0x7f613c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf640485a13d044f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 1 -- 172.21.15.116:0/64018133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c02aea0 con 0x7f6144131ab0 2024-09-18T12:39:03.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:03.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: dump: 2024-09-18T12:39:03.579 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.579 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:03.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:03.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 1 -- 172.21.15.116:0/64018133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c032440 con 0x7f6144131ab0 2024-09-18T12:39:03.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f614af9d640 5 monclient: authenticate success, global_id 154281 2024-09-18T12:39:03.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f614af9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:03.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:03.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f614af9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:03.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 1 -- 172.21.15.116:0/64018133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c0327e0 con 0x7f6144131ab0 2024-09-18T12:39:03.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f614af9d640 4 set_mon_vals no callback set 2024-09-18T12:39:03.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:03.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.559+0000 7f61437fe640 10 monclient: dump: 2024-09-18T12:39:03.583 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.583 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.583 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.583 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.583 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.583 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.584 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.584 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.584 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.584 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 10 monclient: shutdown 2024-09-18T12:39:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 -- 172.21.15.116:0/64018133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6144131ab0 msgr2=0x7f6144131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 --2- 172.21.15.116:0/64018133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6144131ab0 0x7f6144131e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f613c009910 tx=0x7f613c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 -- 172.21.15.116:0/64018133 shutdown_connections 2024-09-18T12:39:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 --2- 172.21.15.116:0/64018133 >> v2:172.21.15.144:3300/0 conn(0x7f6144136bb0 0x7f6144138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 --2- 172.21.15.116:0/64018133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6144131ab0 0x7f6144131e80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 -- 172.21.15.116:0/64018133 >> 172.21.15.116:0/64018133 conn(0x7f6144067d30 msgr2=0x7f61440a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 -- 172.21.15.116:0/64018133 shutdown_connections 2024-09-18T12:39:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 1 -- 172.21.15.116:0/64018133 wait complete. 2024-09-18T12:39:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 10 monclient: build_initial_monmap 2024-09-18T12:39:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.560+0000 7f614af9d640 10 monclient: monmap: 2024-09-18T12:39:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:03.588 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 Processor -- start 2024-09-18T12:39:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 -- start start 2024-09-18T12:39:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient: init 2024-09-18T12:39:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:39:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 0x7f614412fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): picked mon.noname-a con 0x7f614409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6144131ab0 0x7f6144130320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): picked mon.noname-b con 0x7f6144131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6144136bb0 0x7f6144097520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): picked mon.noname-c con 0x7f6144136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61440a2240 con 0x7f614409e910 2024-09-18T12:39:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614409ac40 con 0x7f6144131ab0 2024-09-18T12:39:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614409af20 con 0x7f6144136bb0 2024-09-18T12:39:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614af9d640 10 monclient(hunting): authenticate will time out at 1570.882823s 2024-09-18T12:39:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 0x7f614412fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 0x7f614412fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59954/0 (socket says 172.21.15.116:59954) 2024-09-18T12:39:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 1 -- 172.21.15.116:0/3174319181 learned_addr learned my addr 172.21.15.116:0/3174319181 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 10 monclient(hunting): get_auth_request con 0x7f614409e910 auth_method 0 2024-09-18T12:39:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614953a640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6144136bb0 0x7f6144097520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f6148d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614953a640 10 monclient(hunting): get_auth_request con 0x7f6144136bb0 auth_method 0 2024-09-18T12:39:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614953a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614953a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.561+0000 7f614953a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f614953a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f614953a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f614953a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 10 monclient(hunting): handle_auth_done global_id 154284 payload 824 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 1 -- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6144136bb0 msgr2=0x7f6144097520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6144136bb0 0x7f6144097520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 1 -- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6144131ab0 msgr2=0x7f6144130320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6143fff640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6144131ab0 0x7f6144130320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6144131ab0 0x7f6144130320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 10 monclient: _finish_hunting 0 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 1 monclient: found mon.noname-a 2024-09-18T12:39:03.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.562+0000 7f6148d39640 1 -- 172.21.15.116:0/3174319181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c009450 con 0x7f614409e910 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6148d39640 1 -- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 msgr2=0x7f614412fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6148d39640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 0x7f614412fde0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f613c035040 tx=0x7f613c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 1 -- 172.21.15.116:0/3174319181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c02aea0 con 0x7f614409e910 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: dump: 2024-09-18T12:39:03.597 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.598 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: _finish_auth 0 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 1 -- 172.21.15.116:0/3174319181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c031c70 con 0x7f614409e910 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f614af9d640 5 monclient: authenticate success, global_id 154284 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f614af9d640 10 monclient: _renew_subs 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 1 -- 172.21.15.116:0/3174319181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c03d440 con 0x7f614409e910 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f614af9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:03.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6144099180 con 0x7f614409e910 2024-09-18T12:39:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6140ff9640 10 monclient: dump: 2024-09-18T12:39:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.600 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:03.601 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:03.601 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f6142ffd640 4 set_mon_vals no callback set 2024-09-18T12:39:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f614af9d640 10 monclient: _renew_subs 2024-09-18T12:39:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f614af9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.564+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6144099d50 con 0x7f614409e910 2024-09-18T12:39:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.565+0000 7f6140ff9640 1 -- 172.21.15.116:0/3174319181 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f613c039040 con 0x7f614409e910 2024-09-18T12:39:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.565+0000 7f614af9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:03.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.565+0000 7f614af9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.565+0000 7f614af9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.565+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614404bf60 con 0x7f614409e910 2024-09-18T12:39:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.565+0000 7f6140ff9640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6124038490 0x7f612403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.565+0000 7f6140ff9640 1 -- 172.21.15.116:0/3174319181 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f613c06be00 con 0x7f614409e910 2024-09-18T12:39:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.566+0000 7f6143fff640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6124038490 0x7f612403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.566+0000 7f6143fff640 10 monclient: get_auth_request con 0x7f6124038490 auth_method 0 2024-09-18T12:39:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.568+0000 7f6143fff640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6124038490 0x7f612403a940 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6134007fd0 tx=0x7f6134009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.570+0000 7f6140ff9640 1 -- 172.21.15.116:0/3174319181 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f613c02f2a0 con 0x7f614409e910 2024-09-18T12:39:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.570+0000 7f6140ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.570+0000 7f6140ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:03.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.787+0000 7f614af9d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:39:03.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.787+0000 7f614af9d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:39:03.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.787+0000 7f614af9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:03.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.787+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f61440a26a0 con 0x7f614409e910 2024-09-18T12:39:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6140ff9640 1 -- 172.21.15.116:0/3174319181 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f613c030e60 con 0x7f614409e910 2024-09-18T12:39:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6140ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:39:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6140ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:03.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=11,kv=8,logm=49} crc {auth=254541418,config=3140514009,health=3462558291,kv=1885674990,logm=638113344}) 2024-09-18T12:39:03.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=9,logm=50} crc {auth=2913078065,config=3140514009,health=1085733778,kv=2187923472,logm=2593768307}) 2024-09-18T12:39:03.791 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=11,kv=8,logm=49} crc {auth=254541418,config=3140514009,health=3462558291,kv=1885674990,logm=638113344}) 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.788+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=1842296080,config=3140514009,health=514036659,kv=2187923472,logm=1225776219}) 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553035551}) 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792474076}) 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553035551}) 2024-09-18T12:39:03.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.789+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702831455}) 2024-09-18T12:39:03.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6124038490 msgr2=0x7f612403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6124038490 0x7f612403a940 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6134007fd0 tx=0x7f6134009040 comp rx=0 tx=0).stop 2024-09-18T12:39:03.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 10 monclient: shutdown 2024-09-18T12:39:03.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:03.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 msgr2=0x7f614412fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 0x7f614412fde0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f613c035040 tx=0x7f613c032000 comp rx=0 tx=0).stop 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 shutdown_connections 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6124038490 0x7f612403a940 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6144136bb0 0x7f6144097520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6144131ab0 0x7f6144130320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 --2- 172.21.15.116:0/3174319181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f614409e910 0x7f614412fde0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 >> 172.21.15.116:0/3174319181 conn(0x7f6144067d30 msgr2=0x7f614409d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 shutdown_connections 2024-09-18T12:39:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:03.790+0000 7f614af9d640 1 -- 172.21.15.116:0/3174319181 wait complete. 2024-09-18T12:39:03.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.798 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259676903}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355063183}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259676903}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.791+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240427681}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2954785346}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2242526425}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2954785346}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909829513}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197979643}) 2024-09-18T12:39:03.799 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265986788}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197979643}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.792+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=81384836}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199067248}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851971186}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199067248}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.793+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700625189}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3037894538}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945043456}) 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.800 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3037894538}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3053171052}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623115264}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680018661}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623115264}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.794+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003175013}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725649196}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2100925919}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725649196}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.795+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040943024}) 2024-09-18T12:39:03.801 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1855663389}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588777768}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1855663389}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1590109662}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531921532}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3329580900}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531921532}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.796+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235719152}) 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.802 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202743477}) 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39135182}) 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202743477}) 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.797+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1217394087}) 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713267773}) 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1676099961}) 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713267773}) 2024-09-18T12:39:03.803 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=973966834}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.798+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668338827}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247317597}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668338827}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2105767841}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2752856007}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618530812}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2752856007}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.799+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690499382}) 2024-09-18T12:39:03.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150613264}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502227543}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150613264}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.800+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2806972060}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.801+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.801+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969406242}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.801+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1562886858}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.801+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.801+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969406242}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.801+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484976592}) 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880530820}) 2024-09-18T12:39:03.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=637964667}) 2024-09-18T12:39:03.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880530820}) 2024-09-18T12:39:03.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.802+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342065023}) 2024-09-18T12:39:03.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.803+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.803+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018182341}) 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.803+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2402738771}) 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.803+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.803+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018182341}) 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.803+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3702201571}) 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123866791}) 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1414025372}) 2024-09-18T12:39:03.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123866791}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655531874}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396422065}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841905721}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396422065}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.804+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1190277930}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.805+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.805+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4259635690}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.805+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=456707861}) 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.805+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.805+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4259635690}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.805+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748143646}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3655473020}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322308842}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3655473020}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802886846}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063713468}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420857597}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063713468}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.806+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021724743}) 2024-09-18T12:39:03.820 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908663280}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1082029249}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908663280}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1499441384}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.807+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802985834}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088458064}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802985834}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021337063}) 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.821 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1267829840}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799578534}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1267829840}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.808+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1341905130}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.809+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.809+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253724419}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.809+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051404436}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.809+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.809+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253724419}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.809+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648230520}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.810+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.810+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272787325}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.810+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3182923955}) 2024-09-18T12:39:03.822 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.810+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.810+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272787325}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.810+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324841007}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.811+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.811+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059770945}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.811+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420184751}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.811+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.811+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059770945}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.811+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=700196977}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043204195}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451800165}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043204195}) 2024-09-18T12:39:03.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4034030425}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277419501}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1126583569}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277419501}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.812+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317836815}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.813+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.813+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279563396}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.813+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663781859}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.813+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.813+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279563396}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.813+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2474705460}) 2024-09-18T12:39:03.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214476761}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3690860591}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214476761}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993349829}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574335371}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789269914}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574335371}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.814+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188798982}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.815+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.815+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328473743}) 2024-09-18T12:39:03.825 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.815+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128697844}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.815+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.815+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328473743}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.815+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2718974469}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812417812}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1699685486}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812417812}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590630346}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281865118}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3001793998}) 2024-09-18T12:39:03.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281865118}) 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.816+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577176475}) 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.817+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.817+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2230006581}) 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.817+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376794395}) 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.817+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.817+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2230006581}) 2024-09-18T12:39:03.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.817+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047310194}) 2024-09-18T12:39:03.834 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.817+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.834 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385800117}) 2024-09-18T12:39:03.834 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=138971294}) 2024-09-18T12:39:03.834 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385800117}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=86551395}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710893769}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645348768}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710893769}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.818+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248555184}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364249904}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3356997887}) 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364249904}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=233209648}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884441493}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2440696604}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884441493}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.819+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707419738}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.820+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.820+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946609825}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.820+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787908256}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.820+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.820+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946609825}) 2024-09-18T12:39:03.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.820+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173997864}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869577273}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3111788287}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869577273}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=64791044}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525339710}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3000432651}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525339710}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.821+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885840932}) 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.822+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.822+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160243997}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.822+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=743484078}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.822+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.822+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160243997}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.822+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1135060743}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32843491}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692832479}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32843491}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173007638}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742310526}) 2024-09-18T12:39:03.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1770111732}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742310526}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.823+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1991236205}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.824+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.824+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246568448}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.824+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1263299762}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.824+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.824+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246568448}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.824+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=953100432}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864072765}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930053812}) 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.839 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864072765}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205768817}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151690132}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461979906}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151690132}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.825+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673750664}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.826+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.826+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616823140}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.826+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2396631396}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.826+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.826+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616823140}) 2024-09-18T12:39:03.840 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.826+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024760237}) 2024-09-18T12:39:03.841 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.827+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.841 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.827+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258255999}) 2024-09-18T12:39:03.841 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.827+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2952404374}) 2024-09-18T12:39:03.841 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.827+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:03.841 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.827+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258255999}) 2024-09-18T12:39:03.841 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:03.827+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030380554}) 2024-09-18T12:39:04.200 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.199+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.200 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.199+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=389419198}) 2024-09-18T12:39:04.200 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.199+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020687553}) 2024-09-18T12:39:04.200 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.199+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.200 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.199+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=389419198}) 2024-09-18T12:39:04.200 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.199+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3120570417}) 2024-09-18T12:39:04.202 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.200+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.202 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.200+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052782726}) 2024-09-18T12:39:04.202 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.200+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727790073}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.200+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.200+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052782726}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.200+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2488825254}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061388156}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953499315}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061388156}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3657194461}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315963234}) 2024-09-18T12:39:04.203 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3893729679}) 2024-09-18T12:39:04.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315963234}) 2024-09-18T12:39:04.204 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.201+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114685850}) 2024-09-18T12:39:04.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.202+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.202+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349407618}) 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.202+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352717845}) 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.202+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.202+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349407618}) 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.202+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122303888}) 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.203+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.203+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759293048}) 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.203+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122588716}) 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.203+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.203+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759293048}) 2024-09-18T12:39:04.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.203+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737088350}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.204+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.204+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740292970}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.204+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975957540}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.204+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.204+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740292970}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.204+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1529437369}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390075892}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002198507}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390075892}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1431483376}) 2024-09-18T12:39:04.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963588465}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=587587396}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963588465}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.205+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003198927}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.206+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.206+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2975844721}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.206+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3229787741}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.206+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.206+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2975844721}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.206+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617492166}) 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90190308}) 2024-09-18T12:39:04.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=395649927}) 2024-09-18T12:39:04.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90190308}) 2024-09-18T12:39:04.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980618437}) 2024-09-18T12:39:04.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443012603}) 2024-09-18T12:39:04.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850563972}) 2024-09-18T12:39:04.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443012603}) 2024-09-18T12:39:04.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.207+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867077655}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.208+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.208+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887223266}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.208+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2955557846}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.208+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.208+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887223266}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.208+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206490622}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.209+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.209+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=4,monmap=3,osd_metadata=1} crc {logm=1816429323,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1593986305,mon_config_key=3629230923,monmap=148614106,osd_metadata=3813717626}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.209+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=2} crc {logm=2984834759,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1593986305,mon_config_key=2947491177}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.209+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.209+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=4,monmap=3,osd_metadata=1} crc {logm=1816429323,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1593986305,mon_config_key=3629230923,monmap=148614106,osd_metadata=3813717626}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.209+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5} crc {logm=2572595680,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=675300955,mon_config_key=3655569570}) 2024-09-18T12:39:04.214 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.211+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.215 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.211+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=92} crc {osd_metadata=3127564625,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2898910636}) 2024-09-18T12:39:04.215 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.211+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=92} crc {osd_metadata=3127564625,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=4099979292}) 2024-09-18T12:39:04.215 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.211+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.215 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.211+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=92} crc {osd_metadata=3127564625,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2898910636}) 2024-09-18T12:39:04.215 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.211+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=92} crc {osd_metadata=1176557007,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=3113903274}) 2024-09-18T12:39:04.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.217+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.217+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=3907731838}) 2024-09-18T12:39:04.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.217+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=4087507896}) 2024-09-18T12:39:04.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.217+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:04.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.217+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=3907731838}) 2024-09-18T12:39:04.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:04.217+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=2359506841}) 2024-09-18T12:39:04.810 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:39:04.810 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:04.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.912+0000 7fcb4088b640 10 monclient: get_monmap_and_config 2024-09-18T12:39:04.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.912+0000 7fcb4088b640 10 monclient: build_initial_monmap 2024-09-18T12:39:04.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.912+0000 7fcb4088b640 10 monclient: monmap: 2024-09-18T12:39:04.913 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:04.913 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:04.913 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:04.913232+0000 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:04.913232+0000 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:04.914 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 1 Processor -- start 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 1 -- start start 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient: init 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb3c09c9c0 0x7fcb3c09d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb3c09c9c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09e340 0x7fcb3c09e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb3c09e340 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb3c12f770 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3c09bdd0 con 0x7fcb3c12f770 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3c09ec90 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c09a880 con 0x7fcb3c09e340 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.913+0000 7fcb4088b640 10 monclient(hunting): authenticate will time out at 1572.234942s 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09e340 0x7fcb3c09e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb3c09c9c0 0x7fcb3c09d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3ad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcb3c12f770 0x7fcb3c12fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57778/0 (socket says 172.21.15.116:57778) 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3ad9d640 1 -- 172.21.15.116:0/2021247025 learned_addr learned my addr 172.21.15.116:0/2021247025 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): get_auth_request con 0x7fcb3c09c9c0 auth_method 0 2024-09-18T12:39:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 10 monclient(hunting): get_auth_request con 0x7fcb3c09e340 auth_method 0 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_done global_id 154287 payload 274 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09e340 msgr2=0x7fcb3c09e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 1 --2- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09e340 0x7fcb3c09e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2021247025 >> v1:172.21.15.116:6789/0 conn(0x7fcb3c12f770 legacy=0x7fcb3c12fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 1 monclient: found mon.noname-a 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2021247025 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c09a440 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.914+0000 7fcb39d9b640 1 --2- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09e340 0x7fcb3c09e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09c9c0 msgr2=0x7fcb3c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3a59c640 1 --2- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09c9c0 0x7fcb3c09d830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009880 tx=0x7fcb2c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eec49b47a0665b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 1 -- 172.21.15.116:0/2021247025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c02add0 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: dump: 2024-09-18T12:39:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: _finish_auth 0 2024-09-18T12:39:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 1 -- 172.21.15.116:0/2021247025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c032440 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 5 monclient: authenticate success, global_id 154287 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 1 -- 172.21.15.116:0/2021247025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c0327e0 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 4 set_mon_vals no callback set 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb3959a640 10 monclient: dump: 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 10 monclient: shutdown 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 1 -- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09c9c0 msgr2=0x7fcb3c09d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09c9c0 0x7fcb3c09d830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009880 tx=0x7fcb2c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.917+0000 7fcb4088b640 1 -- 172.21.15.116:0/2021247025 shutdown_connections 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09e340 0x7fcb3c09e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2021247025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09c9c0 0x7fcb3c09d830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 1 -- 172.21.15.116:0/2021247025 >> 172.21.15.116:0/2021247025 conn(0x7fcb3c0a2e40 msgr2=0x7fcb3c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 1 -- 172.21.15.116:0/2021247025 shutdown_connections 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 1 -- 172.21.15.116:0/2021247025 wait complete. 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 10 monclient: build_initial_monmap 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 10 monclient: monmap: 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 1 Processor -- start 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 1 -- start start 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.918+0000 7fcb4088b640 10 monclient: init 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 0x7fcb3c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:04.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb3c09c9c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09e340 0x7fcb3c12e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb3c09e340 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb3c12f770 0x7fcb3c12ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb3c12f770 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c11a4e0 con 0x7fcb3c09e340 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c11ba20 con 0x7fcb3c12f770 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c11bd00 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb4088b640 10 monclient(hunting): authenticate will time out at 1572.240325s 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 0x7fcb3c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09e340 0x7fcb3c12e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09e340 0x7fcb3c12e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59972/0 (socket says 172.21.15.116:59972) 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 0x7fcb3c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:45990/0 (socket says 172.21.15.116:45990) 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 1 -- 172.21.15.116:0/2730826375 learned_addr learned my addr 172.21.15.116:0/2730826375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 10 monclient(hunting): get_auth_request con 0x7fcb3c09e340 auth_method 0 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): get_auth_request con 0x7fcb3c09c9c0 auth_method 0 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3ad9d640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb3c12f770 0x7fcb3c12ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3ad9d640 10 monclient(hunting): get_auth_request con 0x7fcb3c12f770 auth_method 0 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient(hunting): handle_auth_done global_id 144383 payload 824 2024-09-18T12:39:04.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb3c12f770 msgr2=0x7fcb3c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb3c12f770 0x7fcb3c12ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09e340 msgr2=0x7fcb3c12e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09e340 0x7fcb3c12e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 monclient: found mon.noname-c 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2730826375 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30009390 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.919+0000 7fcb39d9b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09e340 0x7fcb3c12e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb3a59c640 1 -- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 msgr2=0x7fcb3c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb3a59c640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 0x7fcb3c06e7d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c031000 tx=0x7fcb2c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 1 -- 172.21.15.116:0/2730826375 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c031d40 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: dump: 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:04.937 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 1 -- 172.21.15.116:0/2730826375 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c03d440 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb4088b640 5 monclient: authenticate success, global_id 144383 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb4088b640 10 monclient: _renew_subs 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb4088b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2c009390 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 1 -- 172.21.15.116:0/2730826375 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c031ea0 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb2a7fc640 10 monclient: dump: 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:04.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb38d99640 4 set_mon_vals no callback set 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb4088b640 10 monclient: _renew_subs 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb4088b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.922+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c11ab30 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb2a7fc640 1 -- 172.21.15.116:0/2730826375 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fcb2c039040 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb4088b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb4088b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb4088b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3c04bf60 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb2a7fc640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcb1c038490 0x7fcb1c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb2a7fc640 1 -- 172.21.15.116:0/2730826375 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fcb2c06bf50 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb39d9b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcb1c038490 0x7fcb1c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.923+0000 7fcb39d9b640 10 monclient: get_auth_request con 0x7fcb1c038490 auth_method 0 2024-09-18T12:39:04.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.926+0000 7fcb39d9b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcb1c038490 0x7fcb1c03a940 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c063630 tx=0x7fcb300090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:04.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.927+0000 7fcb2a7fc640 1 -- 172.21.15.116:0/2730826375 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fcb2c0307b0 con 0x7fcb3c09c9c0 2024-09-18T12:39:04.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.927+0000 7fcb2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:04.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:04.927+0000 7fcb2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.142+0000 7fcb4088b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.142+0000 7fcb4088b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.142+0000 7fcb4088b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.142+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcb3c099ef0 con 0x7fcb3c09c9c0 2024-09-18T12:39:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.142+0000 7fcb2a7fc640 1 -- 172.21.15.116:0/2730826375 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fcb2c041080 con 0x7fcb3c09c9c0 2024-09-18T12:39:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.142+0000 7fcb2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.142+0000 7fcb2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:05.144 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcb1c038490 msgr2=0x7fcb1c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcb1c038490 0x7fcb1c03a940 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c063630 tx=0x7fcb300090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 10 monclient: shutdown 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 msgr2=0x7fcb3c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 0x7fcb3c06e7d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c031000 tx=0x7fcb2c032000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 shutdown_connections 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcb1c038490 0x7fcb1c03a940 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcb3c12f770 0x7fcb3c12ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb3c09e340 0x7fcb3c12e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.144+0000 7fcb4088b640 1 --2- 172.21.15.116:0/2730826375 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcb3c09c9c0 0x7fcb3c06e7d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.145+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 >> 172.21.15.116:0/2730826375 conn(0x7fcb3c0a2e40 msgr2=0x7fcb3c132960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.145+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 shutdown_connections 2024-09-18T12:39:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.145+0000 7fcb4088b640 1 -- 172.21.15.116:0/2730826375 wait complete. 2024-09-18T12:39:05.156 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:39:05.156 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:39:05.156 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:39:05.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:39:05.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.248+0000 7f140dadd640 10 monclient: get_monmap_and_config 2024-09-18T12:39:05.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.248+0000 7f140dadd640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.248+0000 7f140dadd640 10 monclient: monmap: 2024-09-18T12:39:05.249 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:05.249332+0000 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:05.249332+0000 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:05.251 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 1 Processor -- start 2024-09-18T12:39:05.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 1 -- start start 2024-09-18T12:39:05.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 10 monclient: init 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14080974f0 0x7f140812ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 10 monclient(hunting): picked mon.noname-a con 0x7f14080974f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f140809a8a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.249+0000 7f140dadd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1408133070 0x7f1408135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 10 monclient(hunting): picked mon.noname-c con 0x7f1408133070 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14080978c0 con 0x7f140809a8a0 2024-09-18T12:39:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140809ca20 con 0x7f14080974f0 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1408098ad0 con 0x7f1408133070 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f140dadd640 10 monclient(hunting): authenticate will time out at 1572.571055s 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14080974f0 0x7f140812ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14080974f0 0x7f140812ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59986/0 (socket says 172.21.15.116:59986) 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 -- 172.21.15.116:0/2666503302 learned_addr learned my addr 172.21.15.116:0/2666503302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): get_auth_request con 0x7f14080974f0 auth_method 0 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f1407fff640 1 --2- 172.21.15.116:0/2666503302 >> v2:172.21.15.144:3300/0 conn(0x7f1408133070 0x7f1408135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f1407fff640 10 monclient(hunting): get_auth_request con 0x7f1408133070 auth_method 0 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f1407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f1407fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f1407fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient(hunting): handle_auth_done global_id 154293 payload 274 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 -- 172.21.15.116:0/2666503302 >> v2:172.21.15.144:3300/0 conn(0x7f1408133070 msgr2=0x7f1408135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:05.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 --2- 172.21.15.116:0/2666503302 >> v2:172.21.15.144:3300/0 conn(0x7f1408133070 0x7f1408135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 -- 172.21.15.116:0/2666503302 >> v1:172.21.15.144:6789/0 conn(0x7f140809a8a0 legacy=0x7f140809ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 monclient: found mon.noname-a 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.250+0000 7f14077fe640 1 -- 172.21.15.116:0/2666503302 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1408098db0 con 0x7f14080974f0 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14077fe640 1 -- 172.21.15.116:0/2666503302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 msgr2=0x7f140812ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14077fe640 1 --2- 172.21.15.116:0/2666503302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f140812ab00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1408098780 tx=0x7f13f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de52ec7eb303e21 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 1 -- 172.21.15.116:0/2666503302 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8032440 con 0x7f14080974f0 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: dump: 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 1 -- 172.21.15.116:0/2666503302 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f80325a0 con 0x7f14080974f0 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 5 monclient: authenticate success, global_id 154293 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 4 set_mon_vals no callback set 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 1 -- 172.21.15.116:0/2666503302 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8032940 con 0x7f14080974f0 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f14067fc640 10 monclient: dump: 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 10 monclient: shutdown 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 1 -- 172.21.15.116:0/2666503302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 msgr2=0x7f140812ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 1 --2- 172.21.15.116:0/2666503302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f140812ab00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1408098780 tx=0x7f13f80091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 1 -- 172.21.15.116:0/2666503302 shutdown_connections 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 1 --2- 172.21.15.116:0/2666503302 >> v2:172.21.15.144:3300/0 conn(0x7f1408133070 0x7f1408135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 1 --2- 172.21.15.116:0/2666503302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f140812ab00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.253+0000 7f140dadd640 1 -- 172.21.15.116:0/2666503302 >> 172.21.15.116:0/2666503302 conn(0x7f14080672b0 msgr2=0x7f14080676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.254+0000 7f140dadd640 1 -- 172.21.15.116:0/2666503302 shutdown_connections 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.254+0000 7f140dadd640 1 -- 172.21.15.116:0/2666503302 wait complete. 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.254+0000 7f140dadd640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.254+0000 7f140dadd640 10 monclient: monmap: 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.254+0000 7f140dadd640 1 Processor -- start 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.254+0000 7f140dadd640 1 -- start start 2024-09-18T12:39:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.254+0000 7f140dadd640 10 monclient: init 2024-09-18T12:39:05.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:39:05.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f1408128280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): picked mon.noname-a con 0x7f14080974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:05.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f140809a8a0 0x7f140806dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): picked mon.noname-c con 0x7f140809a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1408133070 0x7f140806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): picked mon.noname-b con 0x7f1408133070 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1408098300 con 0x7f14080974f0 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1408129fd0 con 0x7f1408133070 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1408127850 con 0x7f140809a8a0 2024-09-18T12:39:05.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f140dadd640 10 monclient(hunting): authenticate will time out at 1572.576454s 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f1408128280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f1408128280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59998/0 (socket says 172.21.15.116:59998) 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 -- 172.21.15.116:0/280502914 learned_addr learned my addr 172.21.15.116:0/280502914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): get_auth_request con 0x7f14080974f0 auth_method 0 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f140809a8a0 0x7f140806dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 10 monclient(hunting): get_auth_request con 0x7f140809a8a0 auth_method 0 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1407fff640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1408133070 0x7f140806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1407fff640 10 monclient(hunting): get_auth_request con 0x7f1408133070 auth_method 0 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1407fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f1407fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient(hunting): handle_auth_done global_id 154296 payload 824 2024-09-18T12:39:05.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f140809a8a0 msgr2=0x7f140806dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f140809a8a0 0x7f140806dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1408133070 msgr2=0x7f140806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1408133070 0x7f140806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 monclient: found mon.noname-a 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.255+0000 7f14077fe640 1 -- 172.21.15.116:0/280502914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8009450 con 0x7f14080974f0 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f14077fe640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 msgr2=0x7f1408128280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f14077fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f1408128280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f13f8035040 tx=0x7f13f8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8031a40 con 0x7f14080974f0 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: dump: 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.277 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f8031be0 con 0x7f14080974f0 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 5 monclient: authenticate success, global_id 154296 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 10 monclient: _renew_subs 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1408132570 con 0x7f14080974f0 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f803d420 con 0x7f14080974f0 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f13ef7fe640 10 monclient: dump: 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.279 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 10 monclient: _renew_subs 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140806ecf0 con 0x7f14080974f0 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f1405ffb640 4 set_mon_vals no callback set 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 0x7f140806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.258+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1408127680 con 0x7f1408126580 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 0x7f140806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient: get_auth_request con 0x7f1408126580 auth_method 0 2024-09-18T12:39:05.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f13f8039040 con 0x7f14080974f0 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 10 monclient(hunting): handle_auth_done global_id 154299 payload 824 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1406ffd640 1 -- 172.21.15.116:0/280502914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f140804bf60 con 0x7f1408126580 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f13ef7fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13f4038490 0x7f13f403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f13f806bef0 con 0x7f14080974f0 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1407fff640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13f4038490 0x7f13f403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.259+0000 7f1407fff640 10 monclient: get_auth_request con 0x7f13f4038490 auth_method 0 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f1406ffd640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 msgr2=0x7f140806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f1406ffd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 0x7f140806e530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1408126990 tx=0x7f13fc007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc00d910 con 0x7f1408126580 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 10 monclient: dump: 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.282 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f13fc00da70 con 0x7f1408126580 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 msgr2=0x7f140806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 0x7f140806e530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1408126990 tx=0x7f13fc007e10 comp rx=0 tx=0).stop 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 msgr2=0x7f140806e530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.261+0000 7f13ef7fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408126580 0x7f140806e530 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.262+0000 7f1407fff640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13f4038490 0x7f13f403a940 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f140809c940 tx=0x7f140000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f140dadd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:05.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f140dadd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 0x7f1408134850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f140dadd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1408065f20 con 0x7f1408134450 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 0x7f1408134850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient: get_auth_request con 0x7f1408134450 auth_method 0 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 10 monclient(hunting): handle_auth_done global_id 154302 payload 824 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.267+0000 7f1406ffd640 1 -- 172.21.15.116:0/280502914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1408127990 con 0x7f1408134450 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f1406ffd640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 msgr2=0x7f1408134850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f1406ffd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 0x7f1408134850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1408126990 tx=0x7f13fc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc014ac0 con 0x7f1408134450 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 10 monclient: dump: 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7f13fc00d700 con 0x7f1408134450 2024-09-18T12:39:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:05.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:05.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 msgr2=0x7f1408134850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 0x7f1408134850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1408126990 tx=0x7f13fc01a000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 msgr2=0x7f1408134850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:05.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f13ef7fe640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1408134450 0x7f1408134850 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1408126990 tx=0x7f13fc01a000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.288 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:39:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 136, 2024-09-18T12:39:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.291 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:05.292 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:05.293 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.300 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.301 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.303 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.304 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.306 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:05.306 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13f4038490 msgr2=0x7f13f403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f13f4038490 0x7f13f403a940 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f140809c940 tx=0x7f140000b040 comp rx=0 tx=0).stop 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 10 monclient: shutdown 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 msgr2=0x7f1408128280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 --2- 172.21.15.116:0/280502914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14080974f0 0x7f1408128280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f13f8035040 tx=0x7f13f8032000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f1407fff640 1 -- 172.21.15.116:0/280502914 reap_dead start 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f1407fff640 1 -- 172.21.15.116:0/280502914 reap_dead start 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 shutdown_connections 2024-09-18T12:39:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 >> 172.21.15.116:0/280502914 conn(0x7f14080672b0 msgr2=0x7f140809b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 shutdown_connections 2024-09-18T12:39:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.270+0000 7f140dadd640 1 -- 172.21.15.116:0/280502914 wait complete. 2024-09-18T12:39:05.309 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:05.309 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:39:05.309 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:39:05.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.401+0000 7f6ba3d1d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:05.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.401+0000 7f6ba3d1d640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.401+0000 7f6ba3d1d640 10 monclient: monmap: 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:05.402340+0000 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:05.402340+0000 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 1 Processor -- start 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 1 -- start start 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient: init 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b9c09a510 0x7f6b9c096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6b9c09a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b9c0971e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b9c0975e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c09cb00 con 0x7f6b9c0975e0 2024-09-18T12:39:05.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c09a8e0 con 0x7f6b9c0971e0 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c09d590 con 0x7f6b9c09a510 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.402+0000 7f6ba3d1d640 10 monclient(hunting): authenticate will time out at 1572.723999s 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b9c09a510 0x7f6b9c096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba22ba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6b9c0975e0 0x7f6b9c12fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57782/0 (socket says 172.21.15.116:57782) 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b9c09a510 0x7f6b9c096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60022/0 (socket says 172.21.15.116:60022) 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba22ba640 1 -- 172.21.15.116:0/4045320484 learned_addr learned my addr 172.21.15.116:0/4045320484 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): get_auth_request con 0x7f6b9c09a510 auth_method 0 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient(hunting): handle_auth_done global_id 154305 payload 274 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 1 -- 172.21.15.116:0/4045320484 >> v1:172.21.15.144:6789/0 conn(0x7f6b9c0971e0 legacy=0x7f6b9c12f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 1 -- 172.21.15.116:0/4045320484 >> v1:172.21.15.116:6789/0 conn(0x7f6b9c0975e0 legacy=0x7f6b9c12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 1 monclient: found mon.noname-a 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.403+0000 7f6ba1ab9640 1 -- 172.21.15.116:0/4045320484 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c09c0e0 con 0x7f6b9c09a510 2024-09-18T12:39:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba1ab9640 1 -- 172.21.15.116:0/4045320484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c09a510 msgr2=0x7f6b9c096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba1ab9640 1 --2- 172.21.15.116:0/4045320484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c09a510 0x7f6b9c096ca0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009850 tx=0x7f6b980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74360acfbf294f93 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 1 -- 172.21.15.116:0/4045320484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98032440 con 0x7f6b9c09a510 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: dump: 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.416 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 1 -- 172.21.15.116:0/4045320484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b980325a0 con 0x7f6b9c09a510 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba3d1d640 5 monclient: authenticate success, global_id 154305 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba3d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba3d1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 1 -- 172.21.15.116:0/4045320484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98032980 con 0x7f6b9c09a510 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba3d1d640 4 set_mon_vals no callback set 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.406+0000 7f6ba0ab7640 10 monclient: dump: 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 10 monclient: shutdown 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/4045320484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c09a510 msgr2=0x7f6b9c096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 --2- 172.21.15.116:0/4045320484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c09a510 0x7f6b9c096ca0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009850 tx=0x7f6b980090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/4045320484 shutdown_connections 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 --2- 172.21.15.116:0/4045320484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c09a510 0x7f6b9c096ca0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/4045320484 >> 172.21.15.116:0/4045320484 conn(0x7f6b9c0672b0 msgr2=0x7f6b9c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/4045320484 shutdown_connections 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/4045320484 wait complete. 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 10 monclient: monmap: 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.419 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.407+0000 7f6ba3d1d640 1 Processor -- start 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 1 -- start start 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient: init 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c0971e0 0x7f6b9c13ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f6b9c0971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0975e0 0x7f6b9c12e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6b9c0975e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c09a510 0x7f6b9c118390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6b9c09a510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c09c370 con 0x7f6b9c0975e0 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c12f7c0 con 0x7f6b9c09a510 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c11bff0 con 0x7f6b9c0971e0 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba3d1d640 10 monclient(hunting): authenticate will time out at 1572.729487s 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c0971e0 0x7f6b9c13ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0975e0 0x7f6b9c12e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c0971e0 0x7f6b9c13ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:46014/0 (socket says 172.21.15.116:46014) 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 1 -- 172.21.15.116:0/1396176154 learned_addr learned my addr 172.21.15.116:0/1396176154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 10 monclient(hunting): get_auth_request con 0x7f6b9c0971e0 auth_method 0 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): get_auth_request con 0x7f6b9c0975e0 auth_method 0 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c09a510 0x7f6b9c118390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 10 monclient(hunting): get_auth_request con 0x7f6b9c09a510 auth_method 0 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient(hunting): handle_auth_done global_id 154308 payload 824 2024-09-18T12:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c0971e0 msgr2=0x7f6b9c13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c0971e0 0x7f6b9c13ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c09a510 msgr2=0x7f6b9c118390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c09a510 0x7f6b9c118390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 1 monclient: found mon.noname-a 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba12b8640 1 -- 172.21.15.116:0/1396176154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b98009390 con 0x7f6b9c0975e0 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.408+0000 7f6ba1ab9640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c0971e0 0x7f6b9c13ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba12b8640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0975e0 msgr2=0x7f6b9c12e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba12b8640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0975e0 0x7f6b9c12e8e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00c4e0 tx=0x7f6b8c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0117a0 con 0x7f6b9c0975e0 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: dump: 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba3d1d640 5 monclient: authenticate success, global_id 154308 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c011920 con 0x7f6b9c0975e0 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c011ce0 con 0x7f6b9c0975e0 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba3d1d640 10 monclient: _renew_subs 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba3d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c13f440 con 0x7f6b9c0975e0 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b93fff640 4 set_mon_vals no callback set 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6b91ffb640 10 monclient: dump: 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba3d1d640 10 monclient: _renew_subs 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba3d1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.411+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c11ba00 con 0x7f6b9c0975e0 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6b8c012630 con 0x7f6b9c0975e0 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6b91ffb640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b800384c0 0x7f6b8003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba3d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba3d1d640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 0x7f6b9c1188d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba3d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c065f20 con 0x7f6b9c12ee20 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6b8c04bea0 con 0x7f6b9c0975e0 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 0x7f6b9c1188d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient: get_auth_request con 0x7f6b9c12ee20 auth_method 0 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba1ab9640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b800384c0 0x7f6b8003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_done global_id 144392 payload 824 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba22ba640 1 -- 172.21.15.116:0/1396176154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b9c04bf60 con 0x7f6b9c12ee20 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.412+0000 7f6ba1ab9640 10 monclient: get_auth_request con 0x7f6b800384c0 auth_method 0 2024-09-18T12:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.414+0000 7f6ba22ba640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 msgr2=0x7f6b9c1188d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.414+0000 7f6ba22ba640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 0x7f6b9c1188d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6b940079a0 tx=0x7f6b940052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.414+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94006e30 con 0x7f6b9c12ee20 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.414+0000 7f6b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.414+0000 7f6b91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.414+0000 7f6b91ffb640 10 monclient: dump: 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6ba1ab9640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b800384c0 0x7f6b8003a970 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009130 tx=0x7f6b98032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6b9400fd50 con 0x7f6b9c12ee20 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6b91ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6b91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 msgr2=0x7f6b9c1188d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6b91ffb640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 0x7f6b9c1188d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6b940079a0 tx=0x7f6b940052b0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 msgr2=0x7f6b9c1188d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.415+0000 7f6b91ffb640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c12ee20 0x7f6b9c1188d0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6b940079a0 tx=0x7f6b940052b0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.420+0000 7f6ba3d1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.420+0000 7f6ba3d1d640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 0x7f6b9c06ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.420+0000 7f6ba3d1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.420+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c11a440 con 0x7f6b9c118e10 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 0x7f6b9c06ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient: get_auth_request con 0x7f6b9c118e10 auth_method 0 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 10 monclient(hunting): handle_auth_done global_id 144395 payload 824 2024-09-18T12:39:05.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.421+0000 7f6ba22ba640 1 -- 172.21.15.116:0/1396176154 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6b9c119210 con 0x7f6b9c118e10 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6ba22ba640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 msgr2=0x7f6b9c06ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6ba22ba640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 0x7f6b9c06ae00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6b940079a0 tx=0x7f6b94018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94006400 con 0x7f6b9c118e10 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 10 monclient: dump: 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.435 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f6b9401c3e0 con 0x7f6b9c118e10 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 msgr2=0x7f6b9c06ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 0x7f6b9c06ae00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6b940079a0 tx=0x7f6b94018000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 msgr2=0x7f6b9c06ae00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.423+0000 7f6b91ffb640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c118e10 0x7f6b9c06ae00 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6b940079a0 tx=0x7f6b94018000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.436 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 136, 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:39:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.438 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:05.439 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:05.440 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.441 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:05.443 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:05.444 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.445 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.446 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.447 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.448 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:05.449 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b800384c0 msgr2=0x7f6b8003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b800384c0 0x7f6b8003a970 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009130 tx=0x7f6b98032000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 10 monclient: shutdown 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0975e0 msgr2=0x7f6b9c12e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 --2- 172.21.15.116:0/1396176154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0975e0 0x7f6b9c12e8e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00c4e0 tx=0x7f6b8c007e10 comp rx=0 tx=0).stop 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba22ba640 1 -- 172.21.15.116:0/1396176154 reap_dead start 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba22ba640 1 -- 172.21.15.116:0/1396176154 reap_dead start 2024-09-18T12:39:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 shutdown_connections 2024-09-18T12:39:05.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 >> 172.21.15.116:0/1396176154 conn(0x7f6b9c0672b0 msgr2=0x7f6b9c09fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 shutdown_connections 2024-09-18T12:39:05.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.424+0000 7f6ba3d1d640 1 -- 172.21.15.116:0/1396176154 wait complete. 2024-09-18T12:39:05.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:05.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:39:05.452 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:39:05.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.545+0000 7f377355d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:05.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.545+0000 7f377355d640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.545+0000 7f377355d640 10 monclient: monmap: 2024-09-18T12:39:05.546 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:05.546245+0000 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:05.546245+0000 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 1 Processor -- start 2024-09-18T12:39:05.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 1 -- start start 2024-09-18T12:39:05.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient: init 2024-09-18T12:39:05.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T12:39:05.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f376c096c60 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:05.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient(hunting): picked mon.noname-b con 0x7f376c097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f376c133b40 0x7f376c135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient(hunting): picked mon.noname-a con 0x7f376c133b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376c09d2e0 con 0x7f376c096c60 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376c09d6b0 con 0x7f376c133b40 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c0992f0 con 0x7f376c097030 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f377355d640 10 monclient(hunting): authenticate will time out at 1572.867885s 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f37712f9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f376c096c60 0x7f376c09a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43896/0 (socket says 172.21.15.116:43896) 2024-09-18T12:39:05.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f37712f9640 1 -- 172.21.15.116:0/1633530215 learned_addr learned my addr 172.21.15.116:0/1633530215 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f3771afa640 1 --2- 172.21.15.116:0/1633530215 >> v2:172.21.15.116:3300/0 conn(0x7f376c133b40 0x7f376c135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f3770af8640 1 --2- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f3771afa640 10 monclient(hunting): get_auth_request con 0x7f376c133b40 auth_method 0 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f3771afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f3771afa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.546+0000 7f3771afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 10 monclient(hunting): get_auth_request con 0x7f376c097030 auth_method 0 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 10 monclient(hunting): handle_auth_done global_id 154311 payload 274 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 1 -- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 msgr2=0x7f376c097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 1 --2- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 1 -- 172.21.15.116:0/1633530215 >> v1:172.21.15.116:6789/0 conn(0x7f376c096c60 legacy=0x7f376c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 1 monclient: found mon.noname-a 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3771afa640 1 -- 172.21.15.116:0/1633530215 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0995d0 con 0x7f376c133b40 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.547+0000 7f3770af8640 1 --2- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.549+0000 7f3771afa640 1 -- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c133b40 msgr2=0x7f376c135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.549+0000 7f3771afa640 1 --2- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c133b40 0x7f376c135f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f376400d0d0 tx=0x7f376400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e047334f43fd9f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 1 -- 172.21.15.116:0/1633530215 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764009550 con 0x7f376c133b40 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: dump: 2024-09-18T12:39:05.555 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 1 -- 172.21.15.116:0/1633530215 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376402f050 con 0x7f376c133b40 2024-09-18T12:39:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 5 monclient: authenticate success, global_id 154311 2024-09-18T12:39:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:05.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 1 -- 172.21.15.116:0/1633530215 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37640098e0 con 0x7f376c133b40 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 4 set_mon_vals no callback set 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f3763fff640 10 monclient: dump: 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.558 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 10 monclient: shutdown 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 -- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c133b40 msgr2=0x7f376c135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 --2- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c133b40 0x7f376c135f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f376400d0d0 tx=0x7f376400b990 comp rx=0 tx=0).stop 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 -- 172.21.15.116:0/1633530215 shutdown_connections 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 --2- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c133b40 0x7f376c135f20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 --2- 172.21.15.116:0/1633530215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 -- 172.21.15.116:0/1633530215 >> 172.21.15.116:0/1633530215 conn(0x7f376c0672b0 msgr2=0x7f376c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 -- 172.21.15.116:0/1633530215 shutdown_connections 2024-09-18T12:39:05.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.550+0000 7f377355d640 1 -- 172.21.15.116:0/1633530215 wait complete. 2024-09-18T12:39:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.551+0000 7f377355d640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.551+0000 7f377355d640 10 monclient: monmap: 2024-09-18T12:39:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.551+0000 7f377355d640 1 Processor -- start 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.551+0000 7f377355d640 1 -- start start 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.551+0000 7f377355d640 10 monclient: init 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c096c60 0x7f376c12e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): picked mon.noname-a con 0x7f376c096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): picked mon.noname-c con 0x7f376c097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c133b40 0x7f376c06e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): picked mon.noname-b con 0x7f376c133b40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c11c2c0 con 0x7f376c096c60 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c09add0 con 0x7f376c133b40 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c09f5b0 con 0x7f376c097030 2024-09-18T12:39:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f377355d640 10 monclient(hunting): authenticate will time out at 1572.873267s 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c096c60 0x7f376c12e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c06e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:52306/0 (socket says 172.21.15.116:52306) 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c096c60 0x7f376c12e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60974/0 (socket says 172.21.15.116:60974) 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 1 -- 172.21.15.116:0/1904313267 learned_addr learned my addr 172.21.15.116:0/1904313267 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): get_auth_request con 0x7f376c096c60 auth_method 0 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 10 monclient(hunting): get_auth_request con 0x7f376c097030 auth_method 0 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient(hunting): handle_auth_done global_id 154314 payload 824 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 msgr2=0x7f376c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f376c097030 0x7f376c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c133b40 msgr2=0x7f376c06e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c133b40 0x7f376c06e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 monclient: found mon.noname-a 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.552+0000 7f37712f9640 1 -- 172.21.15.116:0/1904313267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c009390 con 0x7f376c096c60 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37712f9640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c096c60 msgr2=0x7f376c12e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37712f9640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c096c60 0x7f376c12e6a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f376800c4c0 tx=0x7f3768007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768005cb0 con 0x7f376c096c60 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: dump: 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3768005e30 con 0x7f376c096c60 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f377355d640 5 monclient: authenticate success, global_id 154314 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f377355d640 10 monclient: _renew_subs 2024-09-18T12:39:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f377355d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376400bc30 con 0x7f376c096c60 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768007320 con 0x7f376c096c60 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37617fa640 10 monclient: dump: 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.572 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f377355d640 10 monclient: _renew_subs 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.554+0000 7f37637fe640 4 set_mon_vals no callback set 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f377355d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376c11c8f0 con 0x7f376c096c60 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f377355d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f377355d640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 0x7f376c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f377355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c11d980 con 0x7f376c12ebe0 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f376801b3b0 con 0x7f376c096c60 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.555+0000 7f37617fa640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3750038490 0x7f375003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f376804b7f0 con 0x7f376c096c60 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 0x7f376c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient: get_auth_request con 0x7f376c12ebe0 auth_method 0 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3771afa640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3750038490 0x7f375003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3771afa640 10 monclient: get_auth_request con 0x7f3750038490 auth_method 0 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 10 monclient(hunting): handle_auth_done global_id 174139 payload 824 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.556+0000 7f3770af8640 1 -- 172.21.15.116:0/1904313267 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f376c11bdd0 con 0x7f376c12ebe0 2024-09-18T12:39:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.558+0000 7f3770af8640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 msgr2=0x7f376c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.558+0000 7f3770af8640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 0x7f376c0298c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f375c011c70 tx=0x7f375c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.558+0000 7f3771afa640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3750038490 0x7f375003a940 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f376c0637c0 tx=0x7f3764036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c011eb0 con 0x7f376c12ebe0 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 10 monclient: dump: 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f375c014650 con 0x7f376c12ebe0 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 msgr2=0x7f376c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 0x7f376c0298c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f375c011c70 tx=0x7f375c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 msgr2=0x7f376c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.559+0000 7f37617fa640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c12ebe0 0x7f376c0298c0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f375c011c70 tx=0x7f375c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.564+0000 7f377355d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.564+0000 7f377355d640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 0x7f376c02a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.564+0000 7f377355d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.564+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c065f20 con 0x7f376c029e00 2024-09-18T12:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 0x7f376c02a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient: get_auth_request con 0x7f376c029e00 auth_method 0 2024-09-18T12:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 10 monclient(hunting): handle_auth_done global_id 174142 payload 824 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.565+0000 7f3770af8640 1 -- 172.21.15.116:0/1904313267 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f376c11e230 con 0x7f376c029e00 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.567+0000 7f3770af8640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 msgr2=0x7f376c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.567+0000 7f3770af8640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 0x7f376c02a200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f375c012010 tx=0x7f375c01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c029430 con 0x7f376c029e00 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 10 monclient: dump: 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f375c01c3e0 con 0x7f376c029e00 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 msgr2=0x7f376c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 0x7f376c02a200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f375c012010 tx=0x7f375c01d000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 msgr2=0x7f376c02a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.568+0000 7f37617fa640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f376c029e00 0x7f376c02a200 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f375c012010 tx=0x7f375c01d000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 136, 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:05.581 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.582 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.583 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:05.584 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:05.585 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.586 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.587 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.588 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.589 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.591 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:05.592 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:05.593 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:05.593 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:05.593 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:05.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:05.593 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:05.593 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3750038490 msgr2=0x7f375003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3750038490 0x7f375003a940 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f376c0637c0 tx=0x7f3764036000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 10 monclient: shutdown 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c096c60 msgr2=0x7f376c12e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 --2- 172.21.15.116:0/1904313267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c096c60 0x7f376c12e6a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f376800c4c0 tx=0x7f3768007f00 comp rx=0 tx=0).stop 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f3771afa640 1 -- 172.21.15.116:0/1904313267 reap_dead start 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f3771afa640 1 -- 172.21.15.116:0/1904313267 reap_dead start 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 shutdown_connections 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 >> 172.21.15.116:0/1904313267 conn(0x7f376c0672b0 msgr2=0x7f376c131ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 shutdown_connections 2024-09-18T12:39:05.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.569+0000 7f377355d640 1 -- 172.21.15.116:0/1904313267 wait complete. 2024-09-18T12:39:05.596 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:05.596 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:39:05.596 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:39:05.596 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:39:05.596 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:39:05.596 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:39:05.596 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:39:05.609 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:39:05.609 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:39:05.609 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.740+0000 7f5e24ec9640 10 monclient: get_monmap_and_config 2024-09-18T12:39:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.740+0000 7f5e24ec9640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.740+0000 7f5e24ec9640 10 monclient: monmap: 2024-09-18T12:39:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:05.741214+0000 2024-09-18T12:39:05.741 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:05.741214+0000 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:05.742 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 1 Processor -- start 2024-09-18T12:39:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 1 -- start start 2024-09-18T12:39:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient: init 2024-09-18T12:39:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:39:05.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e20096e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5e200971e0 0x7f5e200975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e200971e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e20133b30 0x7f5e20135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f5e20133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e20099620 con 0x7f5e20096e10 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e20099340 con 0x7f5e200971e0 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e20099900 con 0x7f5e20133b30 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e24ec9640 10 monclient(hunting): authenticate will time out at 1573.062905s 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e1ed9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e20133b30 msgr2=0x7f5e20135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.741+0000 7f5e1ed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e20133b30 0x7f5e20135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:05.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5e200971e0 0x7f5e200975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5e200971e0 0x7f5e200975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:43662/0 (socket says 172.21.15.116:43662) 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 1 -- 172.21.15.116:0/3905654366 learned_addr learned my addr 172.21.15.116:0/3905654366 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 10 monclient(hunting): get_auth_request con 0x7f5e200971e0 auth_method 0 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.742+0000 7f5e17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e1e59c640 1 -- 172.21.15.116:0/3905654366 >> v1:172.21.15.144:6789/0 conn(0x7f5e20096e10 legacy=0x7f5e2009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 10 monclient(hunting): handle_auth_done global_id 174145 payload 274 2024-09-18T12:39:05.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 1 -- 172.21.15.116:0/3905654366 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e20133b30 msgr2=0x7f5e20135f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:05.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 1 --2- 172.21.15.116:0/3905654366 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e20133b30 0x7f5e20135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 1 -- 172.21.15.116:0/3905654366 >> v1:172.21.15.144:6789/0 conn(0x7f5e20096e10 legacy=0x7f5e2009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 1 monclient: found mon.noname-c 2024-09-18T12:39:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:39:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.744+0000 7f5e17fff640 1 -- 172.21.15.116:0/3905654366 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2009ac70 con 0x7f5e200971e0 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.747+0000 7f5e17fff640 1 -- 172.21.15.116:0/3905654366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e200971e0 msgr2=0x7f5e200975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.747+0000 7f5e17fff640 1 --2- 172.21.15.116:0/3905654366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e200971e0 0x7f5e200975e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009990 tx=0x7f5e080091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26fabfe614aed5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 1 -- 172.21.15.116:0/3905654366 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0802f4b0 con 0x7f5e200971e0 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: dump: 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.750 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 5 monclient: authenticate success, global_id 174145 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 1 -- 172.21.15.116:0/3905654366 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0802f610 con 0x7f5e200971e0 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 1 -- 172.21.15.116:0/3905654366 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0802f9b0 con 0x7f5e200971e0 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 4 set_mon_vals no callback set 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e1dd9b640 10 monclient: dump: 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 10 monclient: shutdown 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 -- 172.21.15.116:0/3905654366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e200971e0 msgr2=0x7f5e200975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/3905654366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e200971e0 0x7f5e200975e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009990 tx=0x7f5e080091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 -- 172.21.15.116:0/3905654366 shutdown_connections 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/3905654366 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e20133b30 0x7f5e20135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/3905654366 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e200971e0 0x7f5e200975e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 -- 172.21.15.116:0/3905654366 >> 172.21.15.116:0/3905654366 conn(0x7f5e200672b0 msgr2=0x7f5e200676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 -- 172.21.15.116:0/3905654366 shutdown_connections 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.748+0000 7f5e24ec9640 1 -- 172.21.15.116:0/3905654366 wait complete. 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 10 monclient: build_initial_monmap 2024-09-18T12:39:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 10 monclient: monmap: 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 1 Processor -- start 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 1 -- start start 2024-09-18T12:39:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 10 monclient: init 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 0x7f5e2012e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f5e20096e10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e200971e0 0x7f5e20118050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.749+0000 7f5e24ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f5e200971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e20133b30 0x7f5e20118590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f5e20133b30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2009b050 con 0x7f5e20096e10 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e17fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e200971e0 msgr2=0x7f5e20118050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200a0010 con 0x7f5e20133b30 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2011a800 con 0x7f5e200971e0 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e200971e0 0x7f5e20118050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e24ec9640 10 monclient(hunting): authenticate will time out at 1573.071074s 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 0x7f5e2012e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 0x7f5e2012e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60992/0 (socket says 172.21.15.116:60992) 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 -- 172.21.15.116:0/2823148525 learned_addr learned my addr 172.21.15.116:0/2823148525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): get_auth_request con 0x7f5e20096e10 auth_method 0 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient(hunting): handle_auth_done global_id 154317 payload 824 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 -- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e200971e0 msgr2=0x7f5e20118050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e200971e0 0x7f5e20118050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 -- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e20133b30 msgr2=0x7f5e20118590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1ed9d640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e20133b30 0x7f5e20118590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e20133b30 0x7f5e20118590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 monclient: found mon.noname-a 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1e59c640 1 -- 172.21.15.116:0/2823148525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08009450 con 0x7f5e20096e10 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1ed9d640 10 monclient: get_auth_request con 0x7f5e20133b30 auth_method 0 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.750+0000 7f5e1ed9d640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e20133b30 0x7f5e20118590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e1e59c640 1 -- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 msgr2=0x7f5e2012e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e1e59c640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 0x7f5e2012e5d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000c590 tx=0x7f5e10007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 1 -- 172.21.15.116:0/2823148525 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10007510 con 0x7f5e20096e10 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: dump: 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 1 -- 172.21.15.116:0/2823148525 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e10005780 con 0x7f5e20096e10 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e24ec9640 5 monclient: authenticate success, global_id 154317 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e24ec9640 10 monclient: _renew_subs 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 1 -- 172.21.15.116:0/2823148525 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10005b40 con 0x7f5e20096e10 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e24ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2011a140 con 0x7f5e20096e10 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e167fc640 10 monclient: dump: 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e1d59a640 4 set_mon_vals no callback set 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e24ec9640 10 monclient: _renew_subs 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e24ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.752+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e201343d0 con 0x7f5e20096e10 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.753+0000 7f5e167fc640 1 -- 172.21.15.116:0/2823148525 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5e1001b3b0 con 0x7f5e20096e10 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.753+0000 7f5e24ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.753+0000 7f5e24ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.753+0000 7f5e24ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.753+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2004bf60 con 0x7f5e20096e10 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.753+0000 7f5e167fc640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e00038440 0x7f5e0003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.753+0000 7f5e167fc640 1 -- 172.21.15.116:0/2823148525 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5e1004b6e0 con 0x7f5e20096e10 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.754+0000 7f5e17fff640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e00038440 0x7f5e0003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.754+0000 7f5e17fff640 10 monclient: get_auth_request con 0x7f5e00038440 auth_method 0 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.756+0000 7f5e17fff640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e00038440 0x7f5e0003a8f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5e20076be0 tx=0x7f5e08032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.758+0000 7f5e167fc640 1 -- 172.21.15.116:0/2823148525 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5e10018820 con 0x7f5e20096e10 2024-09-18T12:39:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.758+0000 7f5e167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.758+0000 7f5e167fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.970+0000 7f5e24ec9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.970+0000 7f5e24ec9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.970+0000 7f5e24ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.970+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e2011ba60 con 0x7f5e20096e10 2024-09-18T12:39:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.970+0000 7f5e167fc640 1 -- 172.21.15.116:0/2823148525 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f5e10018260 con 0x7f5e20096e10 2024-09-18T12:39:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.970+0000 7f5e167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.970+0000 7f5e167fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:05.972 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e00038440 msgr2=0x7f5e0003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e00038440 0x7f5e0003a8f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5e20076be0 tx=0x7f5e08032000 comp rx=0 tx=0).stop 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 10 monclient: shutdown 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 msgr2=0x7f5e2012e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 0x7f5e2012e5d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000c590 tx=0x7f5e10007d60 comp rx=0 tx=0).stop 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 shutdown_connections 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e00038440 0x7f5e0003a8f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e20133b30 0x7f5e20118590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e200971e0 0x7f5e20118050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 --2- 172.21.15.116:0/2823148525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e20096e10 0x7f5e2012e5d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.972+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 >> 172.21.15.116:0/2823148525 conn(0x7f5e200672b0 msgr2=0x7f5e20132310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.973+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 shutdown_connections 2024-09-18T12:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:05.973+0000 7f5e24ec9640 1 -- 172.21.15.116:0/2823148525 wait complete. 2024-09-18T12:39:08.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.088+0000 7f2e25f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.088+0000 7f2e25f9d640 10 monclient: build_initial_monmap 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.088+0000 7f2e25f9d640 10 monclient: monmap: 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:09.088746+0000 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:09.088746+0000 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:09.090 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 1 Processor -- start 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 1 -- start start 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient: init 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e20131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e2009e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e20137bc0 0x7f2e20139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e20137bc0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2009faf0 con 0x7f2e2009e910 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2009fd30 con 0x7f2e20131ab0 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e20132590 con 0x7f2e20137bc0 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e25f9d640 10 monclient(hunting): authenticate will time out at 1576.410647s 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e17fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2e2009e910 0x7f2e2009ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43902/0 (socket says 172.21.15.116:43902) 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e17fff640 1 -- 172.21.15.116:0/286403250 learned_addr learned my addr 172.21.15.116:0/286403250 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 1 --2- 172.21.15.116:0/286403250 >> v2:172.21.15.116:3300/0 conn(0x7f2e20137bc0 0x7f2e20139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): get_auth_request con 0x7f2e20137bc0 auth_method 0 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.089+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_done global_id 154320 payload 274 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.090+0000 7f2e1ffff640 1 -- 172.21.15.116:0/286403250 >> v1:172.21.15.144:6789/0 conn(0x7f2e20131ab0 legacy=0x7f2e20131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.090+0000 7f2e1ffff640 1 -- 172.21.15.116:0/286403250 >> v1:172.21.15.116:6789/0 conn(0x7f2e2009e910 legacy=0x7f2e2009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.090+0000 7f2e1ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.090+0000 7f2e1ffff640 1 monclient: found mon.noname-a 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.090+0000 7f2e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.090+0000 7f2e1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.090+0000 7f2e1ffff640 1 -- 172.21.15.116:0/286403250 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e200a1110 con 0x7f2e20137bc0 2024-09-18T12:39:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.092+0000 7f2e1ffff640 1 -- 172.21.15.116:0/286403250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 msgr2=0x7f2e20139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.092+0000 7f2e1ffff640 1 --2- 172.21.15.116:0/286403250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e20139fa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009e90 tx=0x7f2e18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a80d77177cce6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.092+0000 7f2e1effd640 1 -- 172.21.15.116:0/286403250 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18007840 con 0x7f2e20137bc0 2024-09-18T12:39:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.092+0000 7f2e1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 10 monclient: dump: 2024-09-18T12:39:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 10 monclient: _finish_auth 0 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 1 -- 172.21.15.116:0/286403250 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1802d050 con 0x7f2e20137bc0 2024-09-18T12:39:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 5 monclient: authenticate success, global_id 154320 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 4 set_mon_vals no callback set 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 1 -- 172.21.15.116:0/286403250 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18007bd0 con 0x7f2e20137bc0 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e1effd640 10 monclient: dump: 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 10 monclient: shutdown 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 1 -- 172.21.15.116:0/286403250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 msgr2=0x7f2e20139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/286403250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e20139fa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009e90 tx=0x7f2e18009780 comp rx=0 tx=0).stop 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 1 -- 172.21.15.116:0/286403250 shutdown_connections 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/286403250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e20139fa0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 1 -- 172.21.15.116:0/286403250 >> 172.21.15.116:0/286403250 conn(0x7f2e20067d30 msgr2=0x7f2e200a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 1 -- 172.21.15.116:0/286403250 shutdown_connections 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.093+0000 7f2e25f9d640 1 -- 172.21.15.116:0/286403250 wait complete. 2024-09-18T12:39:09.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 10 monclient: build_initial_monmap 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 10 monclient: monmap: 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 1 Processor -- start 2024-09-18T12:39:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 1 -- start start 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 10 monclient: init 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2009e910 0x7f2e20137080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e2009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.094+0000 7f2e25f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e20131ab0 0x7f2e2012de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e20131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e2012e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e20137bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2009bde0 con 0x7f2e20137bc0 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e200a54d0 con 0x7f2e2009e910 2024-09-18T12:39:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e201326d0 con 0x7f2e20131ab0 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e25f9d640 10 monclient(hunting): authenticate will time out at 1576.416047s 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e17fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e20131ab0 msgr2=0x7f2e2012de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e17fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e20131ab0 0x7f2e2012de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e2012e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e2012e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32776/0 (socket says 172.21.15.116:32776) 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 -- 172.21.15.116:0/2563036231 learned_addr learned my addr 172.21.15.116:0/2563036231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): get_auth_request con 0x7f2e20137bc0 auth_method 0 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_done global_id 154323 payload 824 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1f7fe640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2009e910 0x7f2e20137080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 -- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e20131ab0 msgr2=0x7f2e2012de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e20131ab0 0x7f2e2012de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 -- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2009e910 msgr2=0x7f2e20137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2009e910 0x7f2e20137080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 monclient: found mon.noname-a 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:09.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1ffff640 1 -- 172.21.15.116:0/2563036231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009a20 con 0x7f2e20137bc0 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1f7fe640 10 monclient: get_auth_request con 0x7f2e2009e910 auth_method 0 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.095+0000 7f2e1f7fe640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2009e910 0x7f2e20137080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e1ffff640 1 -- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 msgr2=0x7f2e2012e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e1ffff640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e2012e340 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009de0 tx=0x7f2e18037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 1 -- 172.21.15.116:0/2563036231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18031db0 con 0x7f2e20137bc0 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 10 monclient: dump: 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e25f9d640 5 monclient: authenticate success, global_id 154323 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 1 -- 172.21.15.116:0/2563036231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e18040070 con 0x7f2e20137bc0 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e177fe640 1 -- 172.21.15.116:0/2563036231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18045690 con 0x7f2e20137bc0 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e25f9d640 10 monclient: _renew_subs 2024-09-18T12:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e25f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e201312f0 con 0x7f2e20137bc0 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.097+0000 7f2e1e7fc640 4 set_mon_vals no callback set 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e177fe640 10 monclient: dump: 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e25f9d640 10 monclient: _renew_subs 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e25f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e201300e0 con 0x7f2e20137bc0 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e177fe640 1 -- 172.21.15.116:0/2563036231 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2e18042030 con 0x7f2e20137bc0 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e25f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e25f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e25f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.098+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e201314d0 con 0x7f2e20137bc0 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.099+0000 7f2e177fe640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e00038490 0x7f2e0003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.099+0000 7f2e177fe640 1 -- 172.21.15.116:0/2563036231 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2e1806dc80 con 0x7f2e20137bc0 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.099+0000 7f2e1f7fe640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e00038490 0x7f2e0003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.099+0000 7f2e1f7fe640 10 monclient: get_auth_request con 0x7f2e00038490 auth_method 0 2024-09-18T12:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.101+0000 7f2e1f7fe640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e00038490 0x7f2e0003a940 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2e100027e0 tx=0x7f2e10009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.103+0000 7f2e177fe640 1 -- 172.21.15.116:0/2563036231 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2e201314d0 con 0x7f2e20137bc0 2024-09-18T12:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.103+0000 7f2e177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.103+0000 7f2e177fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.316+0000 7f2e25f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.316+0000 7f2e25f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.316+0000 7f2e25f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:09.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.316+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2e201314d0 con 0x7f2e20137bc0 2024-09-18T12:39:09.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.317+0000 7f2e177fe640 1 -- 172.21.15.116:0/2563036231 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2e201314d0 con 0x7f2e20137bc0 2024-09-18T12:39:09.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.317+0000 7f2e177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:09.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.317+0000 7f2e177fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:09.318 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:09.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e00038490 msgr2=0x7f2e0003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e00038490 0x7f2e0003a940 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2e100027e0 tx=0x7f2e10009040 comp rx=0 tx=0).stop 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 10 monclient: shutdown 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 msgr2=0x7f2e2012e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e2012e340 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009de0 tx=0x7f2e18037000 comp rx=0 tx=0).stop 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 shutdown_connections 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2e00038490 0x7f2e0003a940 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20137bc0 0x7f2e2012e340 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e20131ab0 0x7f2e2012de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 --2- 172.21.15.116:0/2563036231 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2e2009e910 0x7f2e20137080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 >> 172.21.15.116:0/2563036231 conn(0x7f2e20067d30 msgr2=0x7f2e2012bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.318+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 shutdown_connections 2024-09-18T12:39:09.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:09.319+0000 7f2e25f9d640 1 -- 172.21.15.116:0/2563036231 wait complete. 2024-09-18T12:39:12.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:12.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.433+0000 7f7b9291d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:12.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.433+0000 7f7b9291d640 10 monclient: build_initial_monmap 2024-09-18T12:39:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.433+0000 7f7b9291d640 10 monclient: monmap: 2024-09-18T12:39:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:12.434060+0000 2024-09-18T12:39:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:12.434060+0000 2024-09-18T12:39:12.435 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:12.436 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:12.437 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:12.437 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:12.437 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 1 Processor -- start 2024-09-18T12:39:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 1 -- start start 2024-09-18T12:39:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient: init 2024-09-18T12:39:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T12:39:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b8c096c60 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 0x7f7b8c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b8c097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b8c133b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8c09d2e0 con 0x7f7b8c096c60 2024-09-18T12:39:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8c09d6b0 con 0x7f7b8c133b40 2024-09-18T12:39:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c0992f0 con 0x7f7b8c097030 2024-09-18T12:39:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b9291d640 10 monclient(hunting): authenticate will time out at 1579.755738s 2024-09-18T12:39:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b8b7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 msgr2=0x7f7b8c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b8b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 0x7f7b8c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b8bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7b8c096c60 0x7f7b8c09a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43906/0 (socket says 172.21.15.116:43906) 2024-09-18T12:39:12.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.434+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3674348076 learned_addr learned my addr 172.21.15.116:0/3674348076 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:12.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.437+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3674348076 >> v1:172.21.15.116:6789/0 conn(0x7f7b8c096c60 legacy=0x7f7b8c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:12.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.437+0000 7f7b90eba640 1 -- 172.21.15.116:0/3674348076 >> v1:172.21.15.144:6789/0 conn(0x7f7b8c133b40 legacy=0x7f7b8c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.635+0000 7f7b8b7fe640 1 -- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 msgr2=0x7f7b8c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:12.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:12.635+0000 7f7b8b7fe640 1 --2- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 0x7f7b8c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:39:13.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:13.036+0000 7f7b8b7fe640 1 -- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 msgr2=0x7f7b8c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:13.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:13.036+0000 7f7b8b7fe640 1 --2- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 0x7f7b8c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:39:13.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:13.837+0000 7f7b8b7fe640 1 -- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 msgr2=0x7f7b8c097430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:13.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:13.837+0000 7f7b8b7fe640 1 --2- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 0x7f7b8c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:39:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient(hunting): tick 2024-09-18T12:39:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 monclient(hunting): continuing hunt 2024-09-18T12:39:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:39:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 -- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 msgr2=0x7f7b8c097430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 --2- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c097030 0x7f7b8c097430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 -- 172.21.15.116:0/3674348076 >> v1:172.21.15.144:6789/0 conn(0x7f7b8c133b40 legacy=0x7f7b8c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 -- 172.21.15.116:0/3674348076 >> v1:172.21.15.116:6789/0 conn(0x7f7b8c096c60 legacy=0x7f7b8c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient: _add_conns ranks=[3,4] 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 --2- 172.21.15.116:0/3674348076 >> v2:172.21.15.116:3300/0 conn(0x7f7b780037d0 0x7f7b78005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f7b780037d0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 --2- 172.21.15.116:0/3674348076 >> v2:172.21.15.144:3300/0 conn(0x7f7b78006330 0x7f7b780087e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f7b78006330 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 -- 172.21.15.116:0/3674348076 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c0992f0 con 0x7f7b780037d0 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 1 -- 172.21.15.116:0/3674348076 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b78008e80 con 0x7f7b78006330 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b89ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 1 --2- 172.21.15.116:0/3674348076 >> v2:172.21.15.116:3300/0 conn(0x7f7b780037d0 0x7f7b78005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request con 0x7f7b780037d0 auth_method 0 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8b7fe640 1 --2- 172.21.15.116:0/3674348076 >> v2:172.21.15.144:3300/0 conn(0x7f7b78006330 0x7f7b780087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8b7fe640 10 monclient(hunting): get_auth_request con 0x7f7b78006330 auth_method 0 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_done global_id 154326 payload 274 2024-09-18T12:39:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3674348076 >> v2:172.21.15.144:3300/0 conn(0x7f7b78006330 msgr2=0x7f7b780087e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 1 --2- 172.21.15.116:0/3674348076 >> v2:172.21.15.144:3300/0 conn(0x7f7b78006330 0x7f7b780087e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 1 monclient: found mon.noname-a 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.435+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3674348076 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c0995d0 con 0x7f7b780037d0 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b780037d0 msgr2=0x7f7b78005c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8bfff640 1 --2- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b780037d0 0x7f7b78005c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006300 tx=0x7f7b7c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e879f9a3a1566c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 1 -- 172.21.15.116:0/3674348076 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c02ae00 con 0x7f7b780037d0 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: dump: 2024-09-18T12:39:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: _finish_auth 0 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 1 -- 172.21.15.116:0/3674348076 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c005890 con 0x7f7b780037d0 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b9291d640 5 monclient: authenticate success, global_id 154326 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b9291d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b9291d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b9291d640 4 set_mon_vals no callback set 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 1 -- 172.21.15.116:0/3674348076 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c005c50 con 0x7f7b780037d0 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.438+0000 7f7b8affd640 10 monclient: dump: 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 10 monclient: shutdown 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 1 -- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b780037d0 msgr2=0x7f7b78005c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b780037d0 0x7f7b78005c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006300 tx=0x7f7b7c007e00 comp rx=0 tx=0).stop 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b90eba640 1 -- 172.21.15.116:0/3674348076 reap_dead start 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 1 -- 172.21.15.116:0/3674348076 shutdown_connections 2024-09-18T12:39:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3674348076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b780037d0 0x7f7b78005c40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 1 -- 172.21.15.116:0/3674348076 >> 172.21.15.116:0/3674348076 conn(0x7f7b8c0672b0 msgr2=0x7f7b8c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 1 -- 172.21.15.116:0/3674348076 shutdown_connections 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.439+0000 7f7b9291d640 1 -- 172.21.15.116:0/3674348076 wait complete. 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.440+0000 7f7b9291d640 10 monclient: build_initial_monmap 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.440+0000 7f7b9291d640 10 monclient: monmap: 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.440+0000 7f7b9291d640 1 Processor -- start 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.440+0000 7f7b9291d640 1 -- start start 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.440+0000 7f7b9291d640 10 monclient: init 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 0x7f7b8c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b8c09b8c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c06e3c0 0x7f7b8c06e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b8c06e3c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c143760 0x7f7b8c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b8c143760 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c09c030 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c136970 con 0x7f7b8c143760 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c09add0 con 0x7f7b8c06e3c0 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b9291d640 10 monclient(hunting): authenticate will time out at 1582.762533s 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8b7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c06e3c0 msgr2=0x7f7b8c06e790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c06e3c0 0x7f7b8c06e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 0x7f7b8c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 0x7f7b8c06de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53016/0 (socket says 172.21.15.116:53016) 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3721579031 learned_addr learned my addr 172.21.15.116:0/3721579031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request con 0x7f7b8c09b8c0 auth_method 0 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c143760 0x7f7b8c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 10 monclient(hunting): get_auth_request con 0x7f7b8c143760 auth_method 0 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.441+0000 7f7b90eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_done global_id 154329 payload 824 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c06e3c0 msgr2=0x7f7b8c06e790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c06e3c0 0x7f7b8c06e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c143760 msgr2=0x7f7b8c145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c143760 0x7f7b8c145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:15.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 1 monclient: found mon.noname-a 2024-09-18T12:39:15.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:15.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.442+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3721579031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c09d2e0 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b8bfff640 1 -- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 msgr2=0x7f7b8c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b8bfff640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 0x7f7b8c06de80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c02a6b0 tx=0x7f7b7c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 1 -- 172.21.15.116:0/3721579031 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c033890 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: dump: 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: _finish_auth 0 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b9291d640 5 monclient: authenticate success, global_id 154329 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 1 -- 172.21.15.116:0/3721579031 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c038070 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 1 -- 172.21.15.116:0/3721579031 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c033c20 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b73fff640 10 monclient: dump: 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b9291d640 10 monclient: _renew_subs 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b9291d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b8a7fc640 4 set_mon_vals no callback set 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c06ecd0 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b9291d640 10 monclient: _renew_subs 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b9291d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.444+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c118d60 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.445+0000 7f7b73fff640 1 -- 172.21.15.116:0/3721579031 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7b7c030070 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.445+0000 7f7b9291d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.445+0000 7f7b9291d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.445+0000 7f7b9291d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.445+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c1185c0 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.445+0000 7f7b73fff640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b78039280 0x7f7b7803b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.446+0000 7f7b73fff640 1 -- 172.21.15.116:0/3721579031 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7b7c06c5d0 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.446+0000 7f7b8b7fe640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b78039280 0x7f7b7803b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.446+0000 7f7b8b7fe640 10 monclient: get_auth_request con 0x7f7b78039280 auth_method 0 2024-09-18T12:39:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.448+0000 7f7b8b7fe640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b78039280 0x7f7b7803b730 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7b80003b90 tx=0x7f7b80005420 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.450+0000 7f7b73fff640 1 -- 172.21.15.116:0/3721579031 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7b7c033e00 con 0x7f7b8c09b8c0 2024-09-18T12:39:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.450+0000 7f7b73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.450+0000 7f7b73fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:15.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.669+0000 7f7b9291d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:15.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.669+0000 7f7b9291d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:15.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.670+0000 7f7b9291d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:15.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:15.670+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7b8c04bf60 con 0x7f7b8c09b8c0 2024-09-18T12:39:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:18.442+0000 7f7b72ffd640 10 monclient: tick 2024-09-18T12:39:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:18.442+0000 7f7b72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:18.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:18.442+0000 7f7b72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:19.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:19.441+0000 7f7b72ffd640 10 monclient: tick 2024-09-18T12:39:19.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:19.441+0000 7f7b72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:20.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.442+0000 7f7b72ffd640 10 monclient: tick 2024-09-18T12:39:20.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.442+0000 7f7b72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:20.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.593+0000 7f7b73fff640 1 -- 172.21.15.116:0/3721579031 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f7b7c0323d0 con 0x7f7b8c09b8c0 2024-09-18T12:39:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.593+0000 7f7b73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.593+0000 7f7b73fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:20.594 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":140,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b78039280 msgr2=0x7f7b7803b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b78039280 0x7f7b7803b730 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7b80003b90 tx=0x7f7b80005420 comp rx=0 tx=0).stop 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 10 monclient: shutdown 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 msgr2=0x7f7b8c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 0x7f7b8c06de80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c02a6b0 tx=0x7f7b7c030000 comp rx=0 tx=0).stop 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 shutdown_connections 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7b78039280 0x7f7b7803b730 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b8c143760 0x7f7b8c145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7b8c06e3c0 0x7f7b8c06e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 --2- 172.21.15.116:0/3721579031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c09b8c0 0x7f7b8c06de80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 >> 172.21.15.116:0/3721579031 conn(0x7f7b8c0672b0 msgr2=0x7f7b8c135130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:20.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 shutdown_connections 2024-09-18T12:39:20.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.596+0000 7f7b9291d640 1 -- 172.21.15.116:0/3721579031 wait complete. 2024-09-18T12:39:20.610 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:39:20.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:39:20.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:39:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.708+0000 7fdb6a8c0640 10 monclient: get_monmap_and_config 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.708+0000 7fdb6a8c0640 10 monclient: build_initial_monmap 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.709+0000 7fdb6a8c0640 10 monclient: monmap: 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:20.709674+0000 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:20.709674+0000 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:20.711 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:20.711 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:20.711 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:20.711 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:20.711 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.709+0000 7fdb6a8c0640 1 Processor -- start 2024-09-18T12:39:20.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.709+0000 7fdb6a8c0640 1 -- start start 2024-09-18T12:39:20.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.709+0000 7fdb6a8c0640 10 monclient: init 2024-09-18T12:39:20.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:20.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:39:20.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb64096e10 0x7fdb6409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fdb64096e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:20.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb6409aaa0 0x7fdb640971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6409aaa0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:20.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64097720 0x7fdb6412e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): picked mon.noname-b con 0x7fdb64097720 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:20.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb64099620 con 0x7fdb6409aaa0 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb64099340 con 0x7fdb64096e10 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb64099900 con 0x7fdb64097720 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb6a8c0640 10 monclient(hunting): authenticate will time out at 1588.031332s 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb68e5d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64097720 msgr2=0x7fdb6412e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb68e5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64097720 0x7fdb6412e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb6409aaa0 0x7fdb640971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb6409aaa0 0x7fdb640971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53032/0 (socket says 172.21.15.116:53032) 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 -- 172.21.15.116:0/2767804146 learned_addr learned my addr 172.21.15.116:0/2767804146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): get_auth_request con 0x7fdb6409aaa0 auth_method 0 2024-09-18T12:39:20.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient(hunting): handle_auth_done global_id 154332 payload 274 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 -- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64097720 msgr2=0x7fdb6412e5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 --2- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64097720 0x7fdb6412e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb63fff640 1 --2- 172.21.15.116:0/2767804146 >> v2:172.21.15.144:3300/0 conn(0x7fdb64096e10 0x7fdb6409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 -- 172.21.15.116:0/2767804146 >> v2:172.21.15.144:3300/0 conn(0x7fdb64096e10 msgr2=0x7fdb6409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 --2- 172.21.15.116:0/2767804146 >> v2:172.21.15.144:3300/0 conn(0x7fdb64096e10 0x7fdb6409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:20.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 monclient: found mon.noname-a 2024-09-18T12:39:20.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:20.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.710+0000 7fdb637fe640 1 -- 172.21.15.116:0/2767804146 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64097af0 con 0x7fdb6409aaa0 2024-09-18T12:39:20.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb637fe640 1 -- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 msgr2=0x7fdb640971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb637fe640 1 --2- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb640971e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb64076be0 tx=0x7fdb580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa1997c2441d2ee server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 1 -- 172.21.15.116:0/2767804146 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58032440 con 0x7fdb6409aaa0 2024-09-18T12:39:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: dump: 2024-09-18T12:39:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: _finish_auth 0 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 1 -- 172.21.15.116:0/2767804146 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb580325a0 con 0x7fdb6409aaa0 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb6a8c0640 5 monclient: authenticate success, global_id 154332 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb6a8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb6a8c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb6a8c0640 4 set_mon_vals no callback set 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 1 -- 172.21.15.116:0/2767804146 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58032940 con 0x7fdb6409aaa0 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.713+0000 7fdb62ffd640 10 monclient: dump: 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.727 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 10 monclient: shutdown 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 msgr2=0x7fdb640971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb640971e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb64076be0 tx=0x7fdb580091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2767804146 shutdown_connections 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64097720 0x7fdb6412e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2767804146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb640971e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2767804146 >> v2:172.21.15.144:3300/0 conn(0x7fdb64096e10 0x7fdb6409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2767804146 >> 172.21.15.116:0/2767804146 conn(0x7fdb640672b0 msgr2=0x7fdb640676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2767804146 shutdown_connections 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2767804146 wait complete. 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 10 monclient: build_initial_monmap 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.714+0000 7fdb6a8c0640 10 monclient: monmap: 2024-09-18T12:39:20.729 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:20.730 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:20.731 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 Processor -- start 2024-09-18T12:39:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 -- start start 2024-09-18T12:39:20.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient: init 2024-09-18T12:39:20.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:20.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:39:20.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64096e10 0x7fdb6409b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fdb64096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:20.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb64097720 0x7fdb6411ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): picked mon.noname-b con 0x7fdb64097720 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:20.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb6412db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6409aaa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6411b1a0 con 0x7fdb6409aaa0 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6411cf40 con 0x7fdb64097720 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6411b830 con 0x7fdb64096e10 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb63fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64096e10 msgr2=0x7fdb6409b890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb6a8c0640 10 monclient(hunting): authenticate will time out at 1588.036897s 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64096e10 0x7fdb6409b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:20.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.715+0000 7fdb68e5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb6412db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb6412db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53046/0 (socket says 172.21.15.116:53046) 2024-09-18T12:39:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 -- 172.21.15.116:0/2623884714 learned_addr learned my addr 172.21.15.116:0/2623884714 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): get_auth_request con 0x7fdb6409aaa0 auth_method 0 2024-09-18T12:39:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient(hunting): handle_auth_done global_id 154335 payload 824 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb637fe640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb64097720 0x7fdb6411ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64096e10 msgr2=0x7fdb6409b890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb64096e10 0x7fdb6409b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb64097720 msgr2=0x7fdb6411ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb64097720 0x7fdb6411ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient: _finish_hunting 0 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 monclient: found mon.noname-a 2024-09-18T12:39:20.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.716+0000 7fdb68e5d640 1 -- 172.21.15.116:0/2623884714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb58009450 con 0x7fdb6409aaa0 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb68e5d640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 msgr2=0x7fdb6412db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb68e5d640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb6412db20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00bf80 tx=0x7fdb5c009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c016430 con 0x7fdb6409aaa0 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: dump: 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.736 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: _finish_auth 0 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb6a8c0640 5 monclient: authenticate success, global_id 154335 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c0165b0 con 0x7fdb6409aaa0 2024-09-18T12:39:20.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb6a8c0640 10 monclient: _renew_subs 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb6a8c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6411be40 con 0x7fdb6409aaa0 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c016760 con 0x7fdb6409aaa0 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb4bfff640 10 monclient: dump: 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.738 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb6a8c0640 10 monclient: _renew_subs 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb6a8c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6412e060 con 0x7fdb6409aaa0 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.718+0000 7fdb627fc640 4 set_mon_vals no callback set 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb6a8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 0x7fdb6413f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6411d6a0 con 0x7fdb6412e2e0 2024-09-18T12:39:20.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 0x7fdb6413f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient: get_auth_request con 0x7fdb6412e2e0 auth_method 0 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdb5c012020 con 0x7fdb6409aaa0 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 10 monclient(hunting): handle_auth_done global_id 154338 payload 824 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb63fff640 1 -- 172.21.15.116:0/2623884714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb6404bf60 con 0x7fdb6412e2e0 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.719+0000 7fdb4bfff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb50038440 0x7fdb5003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.720+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdb5c04c560 con 0x7fdb6409aaa0 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.720+0000 7fdb637fe640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb50038440 0x7fdb5003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.720+0000 7fdb637fe640 10 monclient: get_auth_request con 0x7fdb50038440 auth_method 0 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb63fff640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 msgr2=0x7fdb6413f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb63fff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 0x7fdb6413f640 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c700 tx=0x7fdb54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54011810 con 0x7fdb6412e2e0 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 10 monclient: dump: 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.741 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdb54011970 con 0x7fdb6412e2e0 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 msgr2=0x7fdb6413f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 0x7fdb6413f640 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c700 tx=0x7fdb54008040 comp rx=0 tx=0).stop 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 msgr2=0x7fdb6413f640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb4bfff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6412e2e0 0x7fdb6413f640 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.722+0000 7fdb637fe640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb50038440 0x7fdb5003a8f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdb6409aff0 tx=0x7fdb58032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb6a8c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:20.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 0x7fdb64029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb6a8c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb64065f20 con 0x7fdb640298c0 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 0x7fdb64029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient: get_auth_request con 0x7fdb640298c0 auth_method 0 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 10 monclient(hunting): handle_auth_done global_id 154341 payload 824 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.728+0000 7fdb63fff640 1 -- 172.21.15.116:0/2623884714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdb6406cd20 con 0x7fdb640298c0 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb63fff640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 msgr2=0x7fdb64029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb63fff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 0x7fdb64029cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb54016000 tx=0x7fdb54019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb54020070 con 0x7fdb640298c0 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 10 monclient: dump: 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7fdb54011600 con 0x7fdb640298c0 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 msgr2=0x7fdb64029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 0x7fdb64029cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb54016000 tx=0x7fdb54019000 comp rx=0 tx=0).stop 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 msgr2=0x7fdb64029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.730+0000 7fdb4bfff640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb640298c0 0x7fdb64029cc0 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb54016000 tx=0x7fdb54019000 comp rx=0 tx=0).stop 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 140, 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:20.753 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:20.754 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.755 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:20.758 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.759 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.760 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb50038440 msgr2=0x7fdb5003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb50038440 0x7fdb5003a8f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdb6409aff0 tx=0x7fdb58032000 comp rx=0 tx=0).stop 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 10 monclient: shutdown 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 msgr2=0x7fdb6412db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 --2- 172.21.15.116:0/2623884714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb6409aaa0 0x7fdb6412db20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00bf80 tx=0x7fdb5c009730 comp rx=0 tx=0).stop 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb68e5d640 1 -- 172.21.15.116:0/2623884714 reap_dead start 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb68e5d640 1 -- 172.21.15.116:0/2623884714 reap_dead start 2024-09-18T12:39:20.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 shutdown_connections 2024-09-18T12:39:20.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 >> 172.21.15.116:0/2623884714 conn(0x7fdb640672b0 msgr2=0x7fdb64069f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:20.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 shutdown_connections 2024-09-18T12:39:20.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.731+0000 7fdb6a8c0640 1 -- 172.21.15.116:0/2623884714 wait complete. 2024-09-18T12:39:20.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:39:20.764 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:39:20.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:39:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.856+0000 7f90ee6d4640 10 monclient: get_monmap_and_config 2024-09-18T12:39:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.856+0000 7f90ee6d4640 10 monclient: build_initial_monmap 2024-09-18T12:39:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.856+0000 7f90ee6d4640 10 monclient: monmap: 2024-09-18T12:39:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:20.857014+0000 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:20.857014+0000 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:20.861 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 1 Processor -- start 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 1 -- start start 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient: init 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e80974f0 0x7f90e812ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f90e80974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90e809a8a0 0x7f90e809acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f90e809a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90e812dd40 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e80978c0 con 0x7f90e812dd40 2024-09-18T12:39:20.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e809ca20 con 0x7f90e809a8a0 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e809b1f0 con 0x7f90e80974f0 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90ee6d4640 10 monclient(hunting): authenticate will time out at 1588.178679s 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e7fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e80974f0 msgr2=0x7f90e812ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e80974f0 0x7f90e812ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90e809a8a0 0x7f90e809acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90e809a8a0 0x7f90e809acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53078/0 (socket says 172.21.15.116:53078) 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 1 -- 172.21.15.116:0/3983173245 learned_addr learned my addr 172.21.15.116:0/3983173245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 10 monclient(hunting): get_auth_request con 0x7f90e809a8a0 auth_method 0 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.857+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 10 monclient(hunting): handle_auth_done global_id 154344 payload 274 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 1 -- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e80974f0 msgr2=0x7f90e812ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 1 --2- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e80974f0 0x7f90e812ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 1 -- 172.21.15.116:0/3983173245 >> v1:172.21.15.144:6789/0 conn(0x7f90e812dd40 legacy=0x7f90e812e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 1 monclient: found mon.noname-a 2024-09-18T12:39:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.858+0000 7f90e77fe640 1 -- 172.21.15.116:0/3983173245 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e8098ad0 con 0x7f90e809a8a0 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e77fe640 1 -- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e809a8a0 msgr2=0x7f90e809acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e77fe640 1 --2- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e809a8a0 0x7f90e809acb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00bbb0 tx=0x7f90dc00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d054c0a61718082b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 1 -- 172.21.15.116:0/3983173245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc032770 con 0x7f90e809a8a0 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: dump: 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90ee6d4640 5 monclient: authenticate success, global_id 154344 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 1 -- 172.21.15.116:0/3983173245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90dc02f030 con 0x7f90e809a8a0 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90ee6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90ee6d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90ee6d4640 4 set_mon_vals no callback set 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 1 -- 172.21.15.116:0/3983173245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc032b40 con 0x7f90e809a8a0 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.860+0000 7f90e6ffd640 10 monclient: dump: 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 10 monclient: shutdown 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e809a8a0 msgr2=0x7f90e809acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e809a8a0 0x7f90e809acb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00bbb0 tx=0x7f90dc00b4b0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3983173245 shutdown_connections 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e809a8a0 0x7f90e809acb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3983173245 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e80974f0 0x7f90e812ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3983173245 >> 172.21.15.116:0/3983173245 conn(0x7f90e80672b0 msgr2=0x7f90e80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3983173245 shutdown_connections 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3983173245 wait complete. 2024-09-18T12:39:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 10 monclient: build_initial_monmap 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.861+0000 7f90ee6d4640 10 monclient: monmap: 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 Processor -- start 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 -- start start 2024-09-18T12:39:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient: init 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 0x7f90e806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f90e80974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e809a8a0 0x7f90e806d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f90e809a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e812dd40 0x7f90e806def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f90e812dd40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e809e080 con 0x7f90e80974f0 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e8098300 con 0x7f90e809a8a0 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e8118b30 con 0x7f90e812dd40 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ee6d4640 10 monclient(hunting): authenticate will time out at 1588.183915s 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ecc71640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e812dd40 msgr2=0x7f90e806def0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90ecc71640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e812dd40 0x7f90e806def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.862+0000 7f90e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 0x7f90e806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 0x7f90e806d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53090/0 (socket says 172.21.15.116:53090) 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 -- 172.21.15.116:0/3989216678 learned_addr learned my addr 172.21.15.116:0/3989216678 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): get_auth_request con 0x7f90e80974f0 auth_method 0 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e77fe640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e809a8a0 0x7f90e806d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e77fe640 10 monclient(hunting): get_auth_request con 0x7f90e809a8a0 auth_method 0 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient(hunting): handle_auth_done global_id 154347 payload 824 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e812dd40 msgr2=0x7f90e806def0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f90e812dd40 0x7f90e806def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e809a8a0 msgr2=0x7f90e806d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e809a8a0 0x7f90e806d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 monclient: found mon.noname-a 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.863+0000 7f90e7fff640 1 -- 172.21.15.116:0/3989216678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc00b6f0 con 0x7f90e80974f0 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90e7fff640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 msgr2=0x7f90e806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90e7fff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 0x7f90e806d470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f90d800b600 tx=0x7f90d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d8015070 con 0x7f90e80974f0 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: dump: 2024-09-18T12:39:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: _finish_auth 0 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8003d90 con 0x7f90e80974f0 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90ee6d4640 5 monclient: authenticate success, global_id 154347 2024-09-18T12:39:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90ee6d4640 10 monclient: _renew_subs 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90ee6d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e806ee00 con 0x7f90e80974f0 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d8006380 con 0x7f90e80974f0 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90cffff640 10 monclient: dump: 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90ee6d4640 10 monclient: _renew_subs 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90e67fc640 4 set_mon_vals no callback set 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90ee6d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.865+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e81182e0 con 0x7f90e80974f0 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90ee6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 0x7f90e80696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90ee6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e8065f20 con 0x7f90e806e430 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f90d8010440 con 0x7f90e80974f0 2024-09-18T12:39:20.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 0x7f90e80696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 10 monclient: get_auth_request con 0x7f90e806e430 auth_method 0 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90cffff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f90d4038440 0x7f90d403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 192..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f90d804bda0 con 0x7f90e80974f0 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90ecc71640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f90d4038440 0x7f90d403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.866+0000 7f90ecc71640 10 monclient: get_auth_request con 0x7f90d4038440 auth_method 0 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.867+0000 7f90e77fe640 10 monclient(hunting): handle_auth_done global_id 174160 payload 824 2024-09-18T12:39:20.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.867+0000 7f90e77fe640 1 -- 172.21.15.116:0/3989216678 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90e81198c0 con 0x7f90e806e430 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90e77fe640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 msgr2=0x7f90e80696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90e77fe640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 0x7f90e80696a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90e8065310 tx=0x7f90dc03a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90ecc71640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f90d4038440 0x7f90d403a8f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f90e0007fd0 tx=0x7f90e0007df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc03fa10 con 0x7f90e806e430 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 10 monclient: dump: 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f90dc043070 con 0x7f90e806e430 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 msgr2=0x7f90e80696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 0x7f90e80696a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90e8065310 tx=0x7f90dc03a000 comp rx=0 tx=0).stop 2024-09-18T12:39:20.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 msgr2=0x7f90e80696a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:20.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.869+0000 7f90cffff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e806e430 0x7f90e80696a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.875+0000 7f90ee6d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.875+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 0x7f90e8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.875+0000 7f90ee6d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.875+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e804bf60 con 0x7f90e8069be0 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 0x7f90e8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient: get_auth_request con 0x7f90e8069be0 auth_method 0 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:20.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 10 monclient(hunting): handle_auth_done global_id 174163 payload 824 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.876+0000 7f90e77fe640 1 -- 172.21.15.116:0/3989216678 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f90e80638d0 con 0x7f90e8069be0 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.878+0000 7f90e77fe640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 msgr2=0x7f90e8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.878+0000 7f90e77fe640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 0x7f90e8069fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f90dc032fd0 tx=0x7f90dc041070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.878+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc0359d0 con 0x7f90e8069be0 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.878+0000 7f90cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.878+0000 7f90cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.878+0000 7f90cffff640 10 monclient: dump: 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:20.892 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f90dc03f800 con 0x7f90e8069be0 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90cffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90cffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 msgr2=0x7f90e8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90cffff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 0x7f90e8069fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f90dc032fd0 tx=0x7f90dc041070 comp rx=0 tx=0).stop 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90cffff640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 msgr2=0x7f90e8069fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:20.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90cffff640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90e8069be0 0x7f90e8069fe0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f90dc032fd0 tx=0x7f90dc041070 comp rx=0 tx=0).stop 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 140, 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.894 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:20.895 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.896 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:20.897 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:20.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.901 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.902 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:20.903 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:20.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:20.905 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:20.906 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f90d4038440 msgr2=0x7f90d403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f90d4038440 0x7f90d403a8f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f90e0007fd0 tx=0x7f90e0007df0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 10 monclient: shutdown 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 msgr2=0x7f90e806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ecc71640 1 -- 172.21.15.116:0/3989216678 reap_dead start 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 0x7f90e806d470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f90d800b600 tx=0x7f90d80091a0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 shutdown_connections 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 --2- 172.21.15.116:0/3989216678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e80974f0 0x7f90e806d470 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:20.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 >> 172.21.15.116:0/3989216678 conn(0x7f90e80672b0 msgr2=0x7f90e80a06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:20.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 shutdown_connections 2024-09-18T12:39:20.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:20.879+0000 7f90ee6d4640 1 -- 172.21.15.116:0/3989216678 wait complete. 2024-09-18T12:39:20.908 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:39:20.908 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:39:40.909 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:39:40.909 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:39:40.909 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:39:40.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:39:40.913 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:39:40.913 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:39:40.914 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:41.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.016+0000 7f804339d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.016+0000 7f804339d640 10 monclient: build_initial_monmap 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.016+0000 7f804339d640 10 monclient: monmap: 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:41.017835+0000 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:41.017835+0000 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:41.018 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:41.019 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:41.019 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:41.019 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:41.019 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:41.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.016+0000 7f804339d640 1 Processor -- start 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.016+0000 7f804339d640 1 -- start start 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.016+0000 7f804339d640 10 monclient: init 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient(hunting): picked mon.noname-b con 0x7f803c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f803c09e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f803c136bb0 0x7f803c138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient(hunting): picked mon.noname-a con 0x7f803c136bb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c09faf0 con 0x7f803c09e910 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803c09fd30 con 0x7f803c136bb0 2024-09-18T12:39:41.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c0a1110 con 0x7f803c131ab0 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804339d640 10 monclient(hunting): authenticate will time out at 1608.339517s 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f803c136bb0 0x7f803c138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51462/0 (socket says 172.21.15.116:51462) 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 1 -- 172.21.15.116:0/2099005727 learned_addr learned my addr 172.21.15.116:0/2099005727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f803c136bb0 0x7f803c138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47494/0 (socket says 172.21.15.116:47494) 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): get_auth_request con 0x7f803c136bb0 auth_method 0 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:41.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 10 monclient(hunting): get_auth_request con 0x7f803c131ab0 auth_method 0 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 1 -- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 msgr2=0x7f803c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 1 -- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 msgr2=0x7f803c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 1 --2- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f8041139640 1 --2- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:41.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient(hunting): handle_auth_done global_id 154350 payload 274 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 1 -- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 msgr2=0x7f803c131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 1 --2- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 1 -- 172.21.15.116:0/2099005727 >> v1:172.21.15.116:6789/0 conn(0x7f803c09e910 legacy=0x7f803c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient: _finish_hunting 0 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 1 monclient: found mon.noname-a 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.017+0000 7f804193a640 1 -- 172.21.15.116:0/2099005727 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c0a13f0 con 0x7f803c136bb0 2024-09-18T12:39:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.020+0000 7f804193a640 1 -- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 msgr2=0x7f803c138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.020+0000 7f804193a640 1 --2- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c138f90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8034009e60 tx=0x7f8034009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eef95ac8ad92c65 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.020+0000 7f8033fff640 1 -- 172.21.15.116:0/2099005727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034007760 con 0x7f803c136bb0 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.020+0000 7f8033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.020+0000 7f8033fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.020+0000 7f8033fff640 10 monclient: dump: 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 10 monclient: _finish_auth 0 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 5 monclient: authenticate success, global_id 154350 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 1 -- 172.21.15.116:0/2099005727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803402d050 con 0x7f803c136bb0 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 4 set_mon_vals no callback set 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 1 -- 172.21.15.116:0/2099005727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034007b30 con 0x7f803c136bb0 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:41.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f8033fff640 10 monclient: dump: 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 10 monclient: shutdown 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 -- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 msgr2=0x7f803c138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 --2- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c138f90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8034009e60 tx=0x7f8034009780 comp rx=0 tx=0).stop 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 -- 172.21.15.116:0/2099005727 shutdown_connections 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 --2- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c138f90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 --2- 172.21.15.116:0/2099005727 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 -- 172.21.15.116:0/2099005727 >> 172.21.15.116:0/2099005727 conn(0x7f803c067d30 msgr2=0x7f803c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 -- 172.21.15.116:0/2099005727 shutdown_connections 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.021+0000 7f804339d640 1 -- 172.21.15.116:0/2099005727 wait complete. 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.022+0000 7f804339d640 10 monclient: build_initial_monmap 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.022+0000 7f804339d640 10 monclient: monmap: 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:41.037 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.022+0000 7f804339d640 1 Processor -- start 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.022+0000 7f804339d640 1 -- start start 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.022+0000 7f804339d640 10 monclient: init 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803c09e910 0x7f803c09b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): picked mon.noname-b con 0x7f803c09e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:41.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c1410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): picked mon.noname-c con 0x7f803c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c1415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): picked mon.noname-a con 0x7f803c136bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c09b960 con 0x7f803c136bb0 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c09b6f0 con 0x7f803c09e910 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c141b20 con 0x7f803c131ab0 2024-09-18T12:39:41.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804339d640 10 monclient(hunting): authenticate will time out at 1608.345105s 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c1415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c1410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c1415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47506/0 (socket says 172.21.15.116:47506) 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 -- 172.21.15.116:0/2850562095 learned_addr learned my addr 172.21.15.116:0/2850562095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): get_auth_request con 0x7f803c136bb0 auth_method 0 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:41.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 10 monclient(hunting): get_auth_request con 0x7f803c131ab0 auth_method 0 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 1 -- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 msgr2=0x7f803c1410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 1 -- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 msgr2=0x7f803c1410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c1410a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8040938640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c1410a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8041139640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803c09e910 0x7f803c09b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:41.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8041139640 10 monclient(hunting): get_auth_request con 0x7f803c09e910 auth_method 0 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8041139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8041139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8041139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient(hunting): handle_auth_done global_id 154353 payload 824 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 -- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 msgr2=0x7f803c1410a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c1410a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 -- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803c09e910 msgr2=0x7f803c09b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:41.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803c09e910 0x7f803c09b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient: _finish_hunting 0 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 monclient: found mon.noname-a 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f804193a640 1 -- 172.21.15.116:0/2850562095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8034009a20 con 0x7f803c136bb0 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.023+0000 7f8041139640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803c09e910 0x7f803c09b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.025+0000 7f804193a640 1 -- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 msgr2=0x7f803c1415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.025+0000 7f804193a640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c1415e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80340099f0 tx=0x7f8034037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 1 -- 172.21.15.116:0/2850562095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034031c10 con 0x7f803c136bb0 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: dump: 2024-09-18T12:39:41.043 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: _finish_auth 0 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:41.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 1 -- 172.21.15.116:0/2850562095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8034040070 con 0x7f803c136bb0 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f804339d640 5 monclient: authenticate success, global_id 154353 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f804339d640 10 monclient: _renew_subs 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f804339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c09bde0 con 0x7f803c136bb0 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 1 -- 172.21.15.116:0/2850562095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8034045430 con 0x7f803c136bb0 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80317fa640 10 monclient: dump: 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.045 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f80337fe640 4 set_mon_vals no callback set 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f804339d640 10 monclient: _renew_subs 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f804339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.026+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c09a7d0 con 0x7f803c136bb0 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f804339d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f804339d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f804339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f80317fa640 1 -- 172.21.15.116:0/2850562095 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8034042030 con 0x7f803c136bb0 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c098af0 con 0x7f803c136bb0 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f80317fa640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80200384e0 0x7f802003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f80317fa640 1 -- 172.21.15.116:0/2850562095 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f803406dd40 con 0x7f803c136bb0 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f8041139640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80200384e0 0x7f802003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.027+0000 7f8041139640 10 monclient: get_auth_request con 0x7f80200384e0 auth_method 0 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.030+0000 7f8041139640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80200384e0 0x7f802003a990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8038006fd0 tx=0x7f8038008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.031+0000 7f80317fa640 1 -- 172.21.15.116:0/2850562095 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f803c098af0 con 0x7f803c136bb0 2024-09-18T12:39:41.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.031+0000 7f80317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:41.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.031+0000 7f80317fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.245+0000 7f804339d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.245+0000 7f804339d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.245+0000 7f804339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.245+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f803c09cd00 con 0x7f803c136bb0 2024-09-18T12:39:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.245+0000 7f80317fa640 1 -- 172.21.15.116:0/2850562095 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f803403b5e0 con 0x7f803c136bb0 2024-09-18T12:39:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.245+0000 7f80317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.245+0000 7f80317fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:41.248 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":140,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80200384e0 msgr2=0x7f802003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80200384e0 0x7f802003a990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8038006fd0 tx=0x7f8038008040 comp rx=0 tx=0).stop 2024-09-18T12:39:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 10 monclient: shutdown 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 msgr2=0x7f803c1415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c1415e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80340099f0 tx=0x7f8034037000 comp rx=0 tx=0).stop 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 shutdown_connections 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80200384e0 0x7f802003a990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f803c136bb0 0x7f803c1415e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f803c131ab0 0x7f803c1410a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 --2- 172.21.15.116:0/2850562095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f803c09e910 0x7f803c09b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 >> 172.21.15.116:0/2850562095 conn(0x7f803c067d30 msgr2=0x7f803c0a4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 shutdown_connections 2024-09-18T12:39:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:41.247+0000 7f804339d640 1 -- 172.21.15.116:0/2850562095 wait complete. 2024-09-18T12:39:44.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:44.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.363+0000 7fe74dc34640 10 monclient: get_monmap_and_config 2024-09-18T12:39:44.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.363+0000 7fe74dc34640 10 monclient: build_initial_monmap 2024-09-18T12:39:44.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.363+0000 7fe74dc34640 10 monclient: monmap: 2024-09-18T12:39:44.364 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:44.364 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:44.364311+0000 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:44.364311+0000 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:44.365 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.364+0000 7fe74dc34640 1 Processor -- start 2024-09-18T12:39:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.364+0000 7fe74dc34640 1 -- start start 2024-09-18T12:39:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.364+0000 7fe74dc34640 10 monclient: init 2024-09-18T12:39:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:39:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe748096c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe748097030 0x7fe748097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient(hunting): picked mon.noname-c con 0x7fe748097030 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe748133b40 0x7fe748135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient(hunting): picked mon.noname-a con 0x7fe748133b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74809d2e0 con 0x7fe748096c60 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74809d6b0 con 0x7fe748133b40 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7480992f0 con 0x7fe748097030 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe74dc34640 10 monclient(hunting): authenticate will time out at 1611.686128s 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe748133b40 0x7fe748135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe748133b40 0x7fe748135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47548/0 (socket says 172.21.15.116:47548) 2024-09-18T12:39:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 -- 172.21.15.116:0/3833186583 learned_addr learned my addr 172.21.15.116:0/3833186583 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): get_auth_request con 0x7fe748133b40 auth_method 0 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe746ffd640 1 --2- 172.21.15.116:0/3833186583 >> v2:172.21.15.144:3300/0 conn(0x7fe748097030 0x7fe748097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe746ffd640 10 monclient(hunting): get_auth_request con 0x7fe748097030 auth_method 0 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient(hunting): handle_auth_done global_id 154356 payload 274 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 -- 172.21.15.116:0/3833186583 >> v2:172.21.15.144:3300/0 conn(0x7fe748097030 msgr2=0x7fe748097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 --2- 172.21.15.116:0/3833186583 >> v2:172.21.15.144:3300/0 conn(0x7fe748097030 0x7fe748097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 -- 172.21.15.116:0/3833186583 >> v1:172.21.15.144:6789/0 conn(0x7fe748096c60 legacy=0x7fe74809a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 monclient: found mon.noname-a 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.365+0000 7fe747fff640 1 -- 172.21.15.116:0/3833186583 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480995d0 con 0x7fe748133b40 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.368+0000 7fe747fff640 1 -- 172.21.15.116:0/3833186583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748133b40 msgr2=0x7fe748135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.368+0000 7fe747fff640 1 --2- 172.21.15.116:0/3833186583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748133b40 0x7fe748135f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe74809ecd0 tx=0x7fe740009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb5ce99154a1a3a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 1 -- 172.21.15.116:0/3833186583 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740007610 con 0x7fe748133b40 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: dump: 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 1 -- 172.21.15.116:0/3833186583 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74002d050 con 0x7fe748133b40 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe74dc34640 5 monclient: authenticate success, global_id 154356 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe74dc34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe74dc34640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 1 -- 172.21.15.116:0/3833186583 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7400079a0 con 0x7fe748133b40 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe74dc34640 4 set_mon_vals no callback set 2024-09-18T12:39:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.370+0000 7fe7467fc640 10 monclient: dump: 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 10 monclient: shutdown 2024-09-18T12:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 -- 172.21.15.116:0/3833186583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748133b40 msgr2=0x7fe748135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 --2- 172.21.15.116:0/3833186583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748133b40 0x7fe748135f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe74809ecd0 tx=0x7fe740009840 comp rx=0 tx=0).stop 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 -- 172.21.15.116:0/3833186583 shutdown_connections 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 --2- 172.21.15.116:0/3833186583 >> v2:172.21.15.144:3300/0 conn(0x7fe748097030 0x7fe748097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 --2- 172.21.15.116:0/3833186583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748133b40 0x7fe748135f20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 -- 172.21.15.116:0/3833186583 >> 172.21.15.116:0/3833186583 conn(0x7fe7480672b0 msgr2=0x7fe7480676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 -- 172.21.15.116:0/3833186583 shutdown_connections 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 1 -- 172.21.15.116:0/3833186583 wait complete. 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 10 monclient: build_initial_monmap 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.371+0000 7fe74dc34640 10 monclient: monmap: 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:44.376 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.372+0000 7fe74dc34640 1 Processor -- start 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.372+0000 7fe74dc34640 1 -- start start 2024-09-18T12:39:44.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.372+0000 7fe74dc34640 10 monclient: init 2024-09-18T12:39:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.372+0000 7fe74dc34640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.372+0000 7fe74dc34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 0x7fe748130290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): picked mon.noname-a con 0x7fe748096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe748097030 0x7fe7481307d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): picked mon.noname-b con 0x7fe748097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 0x7fe748130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): picked mon.noname-c con 0x7fe748133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe748133890 con 0x7fe748096c60 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe748069aa0 con 0x7fe748097030 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74809add0 con 0x7fe748133b40 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe74dc34640 10 monclient(hunting): authenticate will time out at 1611.694100s 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 0x7fe748130290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 0x7fe748130290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47556/0 (socket says 172.21.15.116:47556) 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 -- 172.21.15.116:0/684605195 learned_addr learned my addr 172.21.15.116:0/684605195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): get_auth_request con 0x7fe748096c60 auth_method 0 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 0x7fe748130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 10 monclient(hunting): get_auth_request con 0x7fe748133b40 auth_method 0 2024-09-18T12:39:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 1 -- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 msgr2=0x7fe748130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 1 -- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 msgr2=0x7fe748130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 0x7fe748130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe747fff640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 0x7fe748130d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe746ffd640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe748097030 0x7fe7481307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:44.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe746ffd640 10 monclient(hunting): get_auth_request con 0x7fe748097030 auth_method 0 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe746ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient(hunting): handle_auth_done global_id 154359 payload 824 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 -- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 msgr2=0x7fe748130d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 0x7fe748130d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 -- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe748097030 msgr2=0x7fe7481307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe748097030 0x7fe7481307d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 monclient: found mon.noname-a 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.373+0000 7fe7477fe640 1 -- 172.21.15.116:0/684605195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740009ae0 con 0x7fe748096c60 2024-09-18T12:39:44.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe7477fe640 1 -- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 msgr2=0x7fe748130290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe7477fe640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 0x7fe748130290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe73800e9a0 tx=0x7fe738007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 1 -- 172.21.15.116:0/684605195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73800ee60 con 0x7fe748096c60 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: dump: 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 1 -- 172.21.15.116:0/684605195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe738002af0 con 0x7fe748096c60 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe74dc34640 5 monclient: authenticate success, global_id 154359 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe74dc34640 10 monclient: _renew_subs 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe74dc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7481357e0 con 0x7fe748096c60 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 1 -- 172.21.15.116:0/684605195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73801a480 con 0x7fe748096c60 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe72f7fe640 10 monclient: dump: 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe74dc34640 10 monclient: _renew_subs 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe74dc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7481328a0 con 0x7fe748096c60 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.375+0000 7fe745ffb640 4 set_mon_vals no callback set 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.376+0000 7fe72f7fe640 1 -- 172.21.15.116:0/684605195 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe738011470 con 0x7fe748096c60 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.376+0000 7fe74dc34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.376+0000 7fe74dc34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.376+0000 7fe74dc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.376+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe748132b20 con 0x7fe748096c60 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.377+0000 7fe72f7fe640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe734038490 0x7fe73403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.377+0000 7fe72f7fe640 1 -- 172.21.15.116:0/684605195 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe738015070 con 0x7fe748096c60 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.377+0000 7fe746ffd640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe734038490 0x7fe73403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.377+0000 7fe746ffd640 10 monclient: get_auth_request con 0x7fe734038490 auth_method 0 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.380+0000 7fe746ffd640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe734038490 0x7fe73403a940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00bb40 tx=0x7fe73c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.381+0000 7fe72f7fe640 1 -- 172.21.15.116:0/684605195 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe73800fd90 con 0x7fe748096c60 2024-09-18T12:39:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.381+0000 7fe72f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.381+0000 7fe72f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.594+0000 7fe74dc34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.594+0000 7fe74dc34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.594+0000 7fe74dc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.594+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe74804bf60 con 0x7fe748096c60 2024-09-18T12:39:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.594+0000 7fe72f7fe640 1 -- 172.21.15.116:0/684605195 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fe73801f610 con 0x7fe748096c60 2024-09-18T12:39:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.594+0000 7fe72f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.594+0000 7fe72f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:44.596 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":140,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:44.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe734038490 msgr2=0x7fe73403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:44.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe734038490 0x7fe73403a940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00bb40 tx=0x7fe73c008040 comp rx=0 tx=0).stop 2024-09-18T12:39:44.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 10 monclient: shutdown 2024-09-18T12:39:44.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:44.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 msgr2=0x7fe748130290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:44.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 0x7fe748130290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe73800e9a0 tx=0x7fe738007f60 comp rx=0 tx=0).stop 2024-09-18T12:39:44.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 shutdown_connections 2024-09-18T12:39:44.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe734038490 0x7fe73403a940 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe748133b40 0x7fe748130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe748097030 0x7fe7481307d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 --2- 172.21.15.116:0/684605195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe748096c60 0x7fe748130290 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:44.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 >> 172.21.15.116:0/684605195 conn(0x7fe7480672b0 msgr2=0x7fe748135c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:44.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 shutdown_connections 2024-09-18T12:39:44.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:44.596+0000 7fe74dc34640 1 -- 172.21.15.116:0/684605195 wait complete. 2024-09-18T12:39:47.609 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.713+0000 7f3319868640 10 monclient: get_monmap_and_config 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.713+0000 7f3319868640 10 monclient: build_initial_monmap 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.713+0000 7f3319868640 10 monclient: monmap: 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:47.714027+0000 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:47.714027+0000 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:47.715 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.713+0000 7f3319868640 1 Processor -- start 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 1 -- start start 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient: init 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3314131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:47.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient(hunting): picked mon.noname-b con 0x7f331409e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient(hunting): picked mon.noname-c con 0x7f331409f220 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331409faf0 con 0x7f3314131ab0 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3314132590 con 0x7f331409f220 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331409fd30 con 0x7f331409e910 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f3319868640 10 monclient(hunting): authenticate will time out at 1615.035599s 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:37962/0 (socket says 172.21.15.116:37962) 2024-09-18T12:39:47.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 learned_addr learned my addr 172.21.15.116:0/2386288400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f331409e910 auth_method 0 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f33137fe640 10 monclient(hunting): get_auth_request con 0x7f331409f220 auth_method 0 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f33137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f33137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:47.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.714+0000 7f33137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:47.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.715+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:39:47.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.715+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:47.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.715+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:47.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.715+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:47.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.717+0000 7f3312ffd640 1 -- 172.21.15.116:0/2386288400 >> v1:172.21.15.144:6789/0 conn(0x7f3314131ab0 legacy=0x7f3314131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:47.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:47.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f331409e910 auth_method 0 2024-09-18T12:39:47.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:47.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:47.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:47.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:47.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:47.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:47.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.915+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 10 monclient(hunting): get_auth_request con 0x7f331409f220 auth_method 0 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:47.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:47.916+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:39:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f331409e910 auth_method 0 2024-09-18T12:39:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.316+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:39:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 10 monclient(hunting): get_auth_request con 0x7f331409f220 auth_method 0 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:48.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:48.317+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:39:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f331409e910 auth_method 0 2024-09-18T12:39:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:49.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.117+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 10 monclient(hunting): get_auth_request con 0x7f331409f220 auth_method 0 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:49.118+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:39:49.952 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=110 2024-09-18T12:39:49.952 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 110 -ne 110 ]] 2024-09-18T12:39:49.952 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:49.952 INFO:tasks.workunit.client.0.smithi144.stderr:+ export CEPH_ARGS= 2024-09-18T12:39:49.952 INFO:tasks.workunit.client.0.smithi144.stderr:+ CEPH_ARGS= 2024-09-18T12:39:49.953 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph auth del client.foo 2024-09-18T12:39:50.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.024+0000 7f09e3a60640 10 monclient: get_monmap_and_config 2024-09-18T12:39:50.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.024+0000 7f09e3a60640 10 monclient: build_initial_monmap 2024-09-18T12:39:50.026 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.024+0000 7f09e3a60640 10 monclient: monmap: 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:39:50.026016+0000 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:39:50.026016+0000 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:50.027 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:50.028 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:50.028 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:50.028 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:50.028 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:39:50.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.024+0000 7f09e3a60640 1 Processor -- start 2024-09-18T12:39:50.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 1 -- start start 2024-09-18T12:39:50.028 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient: init 2024-09-18T12:39:50.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:50.030 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09dc131bc0 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient(hunting): picked mon.noname-a con 0x7f09dc0684e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient(hunting): picked mon.noname-b con 0x7f09dc1324d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc0688e0 con 0x7f09dc131bc0 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09dc0a15e0 con 0x7f09dc0684e0 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc097b70 con 0x7f09dc1324d0 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:50.031 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e3a60640 10 monclient(hunting): authenticate will time out at 1625.595646s 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e1ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e1ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.144:41426/0 (socket says 172.21.15.144:41426) 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e0ffb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 learned_addr learned my addr 172.21.15.144:0/1536802793 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request con 0x7f09dc1324d0 auth_method 0 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e1ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.025+0000 7f09e1ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request con 0x7f09dc0684e0 auth_method 0 2024-09-18T12:39:50.032 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09dc138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09dc138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 msgr2=0x7f09dc131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 msgr2=0x7f09dc131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.033 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.026+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:50.034 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.028+0000 7f09e17fc640 1 -- 172.21.15.144:0/1536802793 >> v1:172.21.15.116:6789/0 conn(0x7f09dc131bc0 legacy=0x7f09dc097460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:50.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.229 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request con 0x7f09dc1324d0 auth_method 0 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request con 0x7f09dc0684e0 auth_method 0 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09dc138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09dc138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.230 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 msgr2=0x7f09dc131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:39:50.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 msgr2=0x7f09dc131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:39:50.231 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.227+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:39:50.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request con 0x7f09dc1324d0 auth_method 0 2024-09-18T12:39:50.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:50.631 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request con 0x7f09dc0684e0 auth_method 0 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09dc138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09dc138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 msgr2=0x7f09dc131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 msgr2=0x7f09dc131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.632 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:50.628+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 10 monclient(hunting): tick 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 monclient(hunting): continuing hunt 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331409ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331409ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 msgr2=0x7f3314138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.144:3300/0 conn(0x7f331409f220 0x7f3314138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 -- 172.21.15.116:0/2386288400 >> v1:172.21.15.144:6789/0 conn(0x7f3314131ab0 legacy=0x7f3314131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:50.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 10 monclient: _add_conns ranks=[0,3] 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32f4003700 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 10 monclient(hunting): picked mon.noname-a con 0x7f32f4007380 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 -- 172.21.15.116:0/2386288400 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33140a1110 con 0x7f32f4003700 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 1 -- 172.21.15.116:0/2386288400 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3314132590 con 0x7f32f4007380 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f33117fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.714+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.715+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f32f4007380 auth_method 0 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.715+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.715+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.715+0000 7f330a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:50.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.717+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 msgr2=0x7f32f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:39:50.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.717+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 msgr2=0x7f32f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.717+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.717+0000 7f3312ffd640 1 -- 172.21.15.116:0/2386288400 >> v1:172.21.15.116:6789/0 conn(0x7f32f4003700 legacy=0x7f32f4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:50.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.717+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f32f4007380 auth_method 0 2024-09-18T12:39:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 msgr2=0x7f32f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:39:50.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 msgr2=0x7f32f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:50.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:50.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:50.918+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:39:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.242+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806814681 0 0) 0x7f33140a1110 con 0x7f32f4003700 2024-09-18T12:39:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.242+0000 7f33127fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.242+0000 7f33127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.242+0000 7f33127fc640 10 monclient(hunting): my global_id is 154365 2024-09-18T12:39:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.242+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33000034e0 con 0x7f32f4003700 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1449778386 0 0) 0x7f33000034e0 con 0x7f32f4003700 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 msgr2=0x7f32f4009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 --2- 172.21.15.116:0/2386288400 >> v2:172.21.15.116:3300/0 conn(0x7f32f4007380 0x7f32f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: _finish_hunting 0 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f400af00 con 0x7f32f4003700 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f3304001830 con 0x7f32f4003700 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3319868640 5 monclient: authenticate success, global_id 154365 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3319868640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: dump: 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 >> v1:172.21.15.116:6789/0 conn(0x7f32f4003700 legacy=0x7f32f4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f3300003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): picked mon.c con 0x7f331409e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3300004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): picked mon.b con 0x7f3300004120 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33137fe640 1 -- 172.21.15.116:0/2386288400 reap_dead start 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f3300005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): picked mon.a con 0x7f3300004de0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3314132590 con 0x7f3300004de0 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330000bd70 con 0x7f3300004120 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330000bf30 con 0x7f331409e910 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33127fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3319868640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3312ffd640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f3300003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3312ffd640 10 monclient(hunting): get_auth_request con 0x7f331409e910 auth_method 0 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3312ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f3312ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f330a7fc640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3300004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.243+0000 7f33137fe640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f3300005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f3300004120 auth_method 0 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f330a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f33137fe640 10 monclient(hunting): get_auth_request con 0x7f3300004de0 auth_method 0 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f33137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f33137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f33137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f33137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f33137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f33137fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f330a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f330a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f330a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 10 monclient(hunting): handle_auth_done global_id 154365 payload 293 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 msgr2=0x7f3300004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3300004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 msgr2=0x7f3300005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f3300005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 1 monclient: found mon.c 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 1 -- 172.21.15.116:0/2386288400 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33000034e0 con 0x7f331409e910 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 10 monclient: _finish_auth 0 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.244+0000 7f3312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f3312ffd640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f3300003ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f3312ffd640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f3300003ac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3304001a10 tx=0x7f33040079e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54a64e2792122b58 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33040070f0 con 0x7f331409e910 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 10 monclient: dump: 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330402e070 con 0x7f331409e910 2024-09-18T12:39:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f3319868640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 1 -- 172.21.15.116:0/2386288400 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304007310 con 0x7f331409e910 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f3319868640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f3319868640 4 set_mon_vals no callback set 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.247+0000 7f33127fc640 10 monclient: dump: 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.268 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 10 monclient: shutdown 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 -- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f3300003ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f3300003ac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3304001a10 tx=0x7f33040079e0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 -- 172.21.15.116:0/2386288400 shutdown_connections 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3300004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f3300003ac0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 --2- 172.21.15.116:0/2386288400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f3300005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 -- 172.21.15.116:0/2386288400 >> 172.21.15.116:0/2386288400 conn(0x7f3314067d30 msgr2=0x7f33140a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 -- 172.21.15.116:0/2386288400 shutdown_connections 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.248+0000 7f3319868640 1 -- 172.21.15.116:0/2386288400 wait complete. 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.249+0000 7f3319868640 10 monclient: build_initial_monmap 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.249+0000 7f3319868640 10 monclient: monmap: 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:51.270 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 Processor -- start 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 -- start start 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient: init 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3314130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): picked mon.noname-b con 0x7f3300004120 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f331411d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): picked mon.noname-a con 0x7f3300004de0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331411da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): picked mon.noname-c con 0x7f331409e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:51.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331409dc10 con 0x7f3300004de0 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3314120b50 con 0x7f3300004120 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3314120e30 con 0x7f331409e910 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.250+0000 7f3319868640 10 monclient(hunting): authenticate will time out at 1618.571920s 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331411da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f331411d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331411da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:38016/0 (socket says 172.21.15.116:38016) 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 1 -- 172.21.15.116:0/1763212253 learned_addr learned my addr 172.21.15.116:0/1763212253 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:51.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 10 monclient(hunting): get_auth_request con 0x7f331409e910 auth_method 0 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): get_auth_request con 0x7f3300004de0 auth_method 0 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3314130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 10 monclient(hunting): get_auth_request con 0x7f3300004120 auth_method 0 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331411da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331411da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331411da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f33137fe640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331411da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f3312ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient(hunting): handle_auth_done global_id 154371 payload 824 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 msgr2=0x7f331411da70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331411da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 msgr2=0x7f3314130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3314130a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 1 monclient: found mon.noname-a 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.251+0000 7f330a7fc640 1 -- 172.21.15.116:0/1763212253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331411f020 con 0x7f3300004de0 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330a7fc640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 msgr2=0x7f331411d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330a7fc640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f331411d530 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32fc006fd0 tx=0x7f32fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330b7fe640 1 -- 172.21.15.116:0/1763212253 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc01d070 con 0x7f3300004de0 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330b7fe640 10 monclient: dump: 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f330b7fe640 1 -- 172.21.15.116:0/1763212253 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32fc00c040 con 0x7f3300004de0 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f3319868640 5 monclient: authenticate success, global_id 154371 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f3319868640 10 monclient: _renew_subs 2024-09-18T12:39:51.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f3319868640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.253+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33141415a0 con 0x7f3300004de0 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f330b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f330b7fe640 1 -- 172.21.15.116:0/1763212253 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc022420 con 0x7f3300004de0 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f330b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f330b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f330b7fe640 10 monclient: dump: 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f3311ffb640 4 set_mon_vals no callback set 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f3319868640 10 monclient: _renew_subs 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f3319868640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.254+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331411fbf0 con 0x7f3300004de0 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.255+0000 7f330b7fe640 1 -- 172.21.15.116:0/1763212253 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f32fc022790 con 0x7f3300004de0 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.255+0000 7f3319868640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.255+0000 7f3319868640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.255+0000 7f3319868640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.255+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3314120ff0 con 0x7f3300004de0 2024-09-18T12:39:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.255+0000 7f330b7fe640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.255+0000 7f330b7fe640 1 -- 172.21.15.116:0/1763212253 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f32fc04d390 con 0x7f3300004de0 2024-09-18T12:39:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.257+0000 7f3312ffd640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.257+0000 7f3312ffd640 10 monclient: get_auth_request con 0x7f32f4038dc0 auth_method 0 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.241+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590509264 0 0) 0x7f09dc0688e0 con 0x7f09dc131bc0 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.241+0000 7f09dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.241+0000 7f09dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.241+0000 7f09dbfff640 10 monclient(hunting): my global_id is 154362 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.241+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c40034e0 con 0x7f09dc131bc0 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666690554 0 0) 0x7f09c40034e0 con 0x7f09dc131bc0 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09dc138f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:51.280 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09dc138f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 msgr2=0x7f09dc131f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 --2- 172.21.15.144:0/1536802793 >> v2:172.21.15.116:3300/0 conn(0x7f09dc0684e0 0x7f09dc131f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc0a1300 con 0x7f09dc131bc0 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: _finish_auth 0 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f09d40017f0 con 0x7f09dc131bc0 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.281 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: dump: 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:51.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 >> v1:172.21.15.116:6789/0 conn(0x7f09dc131bc0 legacy=0x7f09dc097460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4003b70 0x7f09c4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): picked mon.a con 0x7f09c4003b70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c4004780 0x7f09c4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): picked mon.b con 0x7f09c4004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09c400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): picked mon.c con 0x7f09dc1324d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0a15e0 con 0x7f09c4003b70 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc097b70 con 0x7f09c4004780 2024-09-18T12:39:51.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c400bf10 con 0x7f09dc1324d0 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09dbfff640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c4004780 0x7f09c4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request con 0x7f09c4004780 auth_method 0 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4003b70 0x7f09c4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 10 monclient(hunting): get_auth_request con 0x7f09c4003b70 auth_method 0 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.284 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09c400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request con 0x7f09dc1324d0 auth_method 0 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e17fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.285 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e1ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_done global_id 154362 payload 293 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 msgr2=0x7f09c400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.242+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09dc1324d0 0x7f09c400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4003b70 msgr2=0x7f09c4003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c4003b70 0x7f09c4003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 1 monclient: found mon.b 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c40034e0 con 0x7f09c4004780 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 10 monclient: _finish_auth 0 2024-09-18T12:39:51.286 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e0ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e3a60640 5 monclient: authenticate success, global_id 154362 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.243+0000 7f09e3a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09e0ffb640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c4004780 msgr2=0x7f09c4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c4004780 0x7f09c4004bf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f09d0006ea0 tx=0x7f09d0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1389fa9dc121a15e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0032480 con 0x7f09c4004780 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 10 monclient: dump: 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.287 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d0032620 con 0x7f09c4004780 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09e3a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09e3a60640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09e3a60640 4 set_mon_vals no callback set 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 1 -- 172.21.15.144:0/1536802793 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d00329e0 con 0x7f09c4004780 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.288 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.246+0000 7f09dbfff640 10 monclient: dump: 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 10 monclient: shutdown 2024-09-18T12:39:51.289 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 1 -- 172.21.15.144:0/1536802793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c4004780 msgr2=0x7f09c4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 1 --2- 172.21.15.144:0/1536802793 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09c4004780 0x7f09c4004bf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f09d0006ea0 tx=0x7f09d0008040 comp rx=0 tx=0).stop 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e1ffd640 1 -- 172.21.15.144:0/1536802793 reap_dead start 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 1 -- 172.21.15.144:0/1536802793 shutdown_connections 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 1 -- 172.21.15.144:0/1536802793 >> 172.21.15.144:0/1536802793 conn(0x7f09dc0a51d0 msgr2=0x7f09dc0a55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 1 -- 172.21.15.144:0/1536802793 shutdown_connections 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 1 -- 172.21.15.144:0/1536802793 wait complete. 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 10 monclient: build_initial_monmap 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.247+0000 7f09e3a60640 10 monclient: monmap: 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:39:51.290 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.248+0000 7f09e3a60640 1 Processor -- start 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.248+0000 7f09e3a60640 1 -- start start 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.248+0000 7f09e3a60640 10 monclient: init 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 0x7f09dc1411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): picked mon.noname-c con 0x7f09c4004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:51.291 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 0x7f09dc141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): picked mon.noname-b con 0x7f09dc141720 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09dc1348a0 0x7f09dc134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): picked mon.noname-a con 0x7f09dc1348a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc11ce30 con 0x7f09dc1348a0 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0a2a20 con 0x7f09dc141720 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc142030 con 0x7f09c4004780 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e3a60640 10 monclient(hunting): authenticate will time out at 1626.819346s 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 0x7f09dc141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.292 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 0x7f09dc141af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:57300/0 (socket says 172.21.15.144:57300) 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 0x7f09dc1411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 -- 172.21.15.144:0/3464913630 learned_addr learned my addr 172.21.15.144:0/3464913630 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request con 0x7f09dc141720 auth_method 0 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 10 monclient(hunting): get_auth_request con 0x7f09c4004780 auth_method 0 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09dc1348a0 0x7f09dc134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.293 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request con 0x7f09dc1348a0 auth_method 0 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 1 -- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 msgr2=0x7f09dc1411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 1 -- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 msgr2=0x7f09dc1411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 0x7f09dc1411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e17fc640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 0x7f09dc1411e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e1ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient(hunting): handle_auth_done global_id 174175 payload 824 2024-09-18T12:39:51.294 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 -- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 msgr2=0x7f09dc1411e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 0x7f09dc1411e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 -- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09dc1348a0 msgr2=0x7f09dc134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09dc1348a0 0x7f09dc134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 monclient: found mon.noname-b 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.249+0000 7f09e0ffb640 1 -- 172.21.15.144:0/3464913630 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc11c870 con 0x7f09dc141720 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e0ffb640 1 -- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 msgr2=0x7f09dc141af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e0ffb640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 0x7f09dc141af0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09d0006ea0 tx=0x7f09d0038000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0031880 con 0x7f09dc141720 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:39:51.295 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: dump: 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: _finish_auth 0 2024-09-18T12:39:51.296 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d00319e0 con 0x7f09dc141720 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e3a60640 5 monclient: authenticate success, global_id 174175 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e3a60640 10 monclient: _renew_subs 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e3a60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc11cb50 con 0x7f09dc141720 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0031d80 con 0x7f09dc141720 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09d97fa640 10 monclient: dump: 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.297 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e3a60640 10 monclient: _renew_subs 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e3a60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc136390 con 0x7f09dc141720 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.252+0000 7f09db7fe640 4 set_mon_vals no callback set 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f09d003d420 con 0x7f09dc141720 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09d97fa640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09c8038490 0x7f09c803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09e3a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:51.298 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09e3a60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09e3a60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f09d006ded0 con 0x7f09dc141720 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc066600 con 0x7f09dc141720 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09e17fc640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09c8038490 0x7f09c803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.253+0000 7f09e17fc640 10 monclient: get_auth_request con 0x7f09c8038490 auth_method 0 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.256+0000 7f09e17fc640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09c8038490 0x7f09c803a940 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f09d4001920 tx=0x7f09d4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.258+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f09d0041080 con 0x7f09dc141720 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.258+0000 7f09d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:51.299 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.258+0000 7f09d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:51.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.260+0000 7f330b7fe640 1 -- 172.21.15.116:0/1763212253 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f32fc0147c0 con 0x7f3300004de0 2024-09-18T12:39:51.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.260+0000 7f330b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:51.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.260+0000 7f330b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:51.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.260+0000 7f3312ffd640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 msgr2=0x7f32f403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:51.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.260+0000 7f3312ffd640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 msgr2=0x7f32f403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:51.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.260+0000 7f3312ffd640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f3304001eb0 tx=0x7f3304034000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:51.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.260+0000 7f3312ffd640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:51.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.461+0000 7f3312ffd640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:51.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.461+0000 7f3312ffd640 10 monclient: get_auth_request con 0x7f32f4038dc0 auth_method 0 2024-09-18T12:39:51.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.463+0000 7f3312ffd640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3304041000 tx=0x7f3304040000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:51.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.475+0000 7f3319868640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:51.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.475+0000 7f3319868640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:51.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.475+0000 7f3319868640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:51.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.475+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f331404bf60 con 0x7f3300004de0 2024-09-18T12:39:51.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.475+0000 7f330b7fe640 1 -- 172.21.15.116:0/1763212253 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f32fc02ae90 con 0x7f3300004de0 2024-09-18T12:39:51.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.475+0000 7f330b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:51.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.475+0000 7f330b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:51.477 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":144,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 msgr2=0x7f32f403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3304041000 tx=0x7f3304040000 comp rx=0 tx=0).stop 2024-09-18T12:39:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 10 monclient: shutdown 2024-09-18T12:39:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 msgr2=0x7f331411d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f331411d530 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32fc006fd0 tx=0x7f32fc008040 comp rx=0 tx=0).stop 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 shutdown_connections 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f32f4038dc0 0x7f32f403b270 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f331409e910 0x7f331411da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3300004de0 0x7f331411d530 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 --2- 172.21.15.116:0/1763212253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3300004120 0x7f3314130a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 >> 172.21.15.116:0/1763212253 conn(0x7f3314067d30 msgr2=0x7f33140a31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 shutdown_connections 2024-09-18T12:39:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:51.477+0000 7f3319868640 1 -- 172.21.15.116:0/1763212253 wait complete. 2024-09-18T12:39:51.479 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.477+0000 7f09e3a60640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T12:39:51.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.477+0000 7f09e3a60640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T12:39:51.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.477+0000 7f09e3a60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:51.480 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.477+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7f09dc09fed0 con 0x7f09dc141720 2024-09-18T12:39:51.485 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.484+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f09d004f750 con 0x7f09dc141720 2024-09-18T12:39:51.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.484+0000 7f09d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T12:39:51.486 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.484+0000 7f09d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:51.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.637+0000 7f09e3a60640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T12:39:51.639 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.637+0000 7f09e3a60640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T12:39:51.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.637+0000 7f09e3a60640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:51.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.637+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7f09dc09fed0 con 0x7f09dc141720 2024-09-18T12:39:51.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.638+0000 7f09d97fa640 1 -- 172.21.15.144:0/3464913630 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f09d0049120 con 0x7f09dc141720 2024-09-18T12:39:51.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.638+0000 7f09d97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T12:39:51.640 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.638+0000 7f09d97fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:39:51.642 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09c8038490 msgr2=0x7f09c803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09c8038490 0x7f09c803a940 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f09d4001920 tx=0x7f09d4009040 comp rx=0 tx=0).stop 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 10 monclient: shutdown 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 msgr2=0x7f09dc141af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 0x7f09dc141af0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09d0006ea0 tx=0x7f09d0038000 comp rx=0 tx=0).stop 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 shutdown_connections 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f09c8038490 0x7f09c803a940 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09dc1348a0 0x7f09dc134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09dc141720 0x7f09dc141af0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 --2- 172.21.15.144:0/3464913630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f09c4004780 0x7f09dc1411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 >> 172.21.15.144:0/3464913630 conn(0x7f09dc0a51d0 msgr2=0x7f09dc097d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 shutdown_connections 2024-09-18T12:39:51.643 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:39:51.640+0000 7f09e3a60640 1 -- 172.21.15.144:0/3464913630 wait complete. 2024-09-18T12:39:51.651 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-18T12:39:51.651 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-18T12:39:51.651 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:39:51.652 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T12:39:51.652 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T12:39:51.652 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T12:39:54.490 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.592+0000 7fc549634640 10 monclient: get_monmap_and_config 2024-09-18T12:39:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.592+0000 7fc549634640 10 monclient: build_initial_monmap 2024-09-18T12:39:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.592+0000 7fc549634640 10 monclient: monmap: 2024-09-18T12:39:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:54.592961+0000 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:54.592961+0000 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.592+0000 7fc549634640 1 Processor -- start 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.592+0000 7fc549634640 1 -- start start 2024-09-18T12:39:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.592+0000 7fc549634640 10 monclient: init 2024-09-18T12:39:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc5441318f0 0x7fc544131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient(hunting): picked mon.noname-a con 0x7fc5441318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 0x7fc54409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient(hunting): picked mon.noname-b con 0x7fc54409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc544068d90 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc544131cc0 con 0x7fc544068d90 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54409fbf0 con 0x7fc5441318f0 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54409fe30 con 0x7fc54409ea50 2024-09-18T12:39:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc549634640 10 monclient(hunting): authenticate will time out at 1621.914478s 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc5441318f0 0x7fc544131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc5441318f0 0x7fc544131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45986/0 (socket says 172.21.15.116:45986) 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 -- 172.21.15.116:0/2893624954 learned_addr learned my addr 172.21.15.116:0/2893624954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 0x7fc54409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): get_auth_request con 0x7fc5441318f0 auth_method 0 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 10 monclient(hunting): get_auth_request con 0x7fc54409ea50 auth_method 0 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 1 -- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 msgr2=0x7fc54409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 1 -- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 msgr2=0x7fc54409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 1 --2- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 0x7fc54409ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc53a7fc640 1 --2- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 0x7fc54409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient(hunting): handle_auth_done global_id 154374 payload 274 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 -- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 msgr2=0x7fc54409ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 --2- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 0x7fc54409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 -- 172.21.15.116:0/2893624954 >> v1:172.21.15.144:6789/0 conn(0x7fc544068d90 legacy=0x7fc5440691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:39:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 monclient: found mon.noname-a 2024-09-18T12:39:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.593+0000 7fc542ffd640 1 -- 172.21.15.116:0/2893624954 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5440a1210 con 0x7fc5441318f0 2024-09-18T12:39:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc542ffd640 1 -- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5441318f0 msgr2=0x7fc544131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc542ffd640 1 --2- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5441318f0 0x7fc544131f30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc534009910 tx=0x7fc5340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def2e4c73d4bad59 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 1 -- 172.21.15.116:0/2893624954 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53402ae90 con 0x7fc5441318f0 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: dump: 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.606 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 1 -- 172.21.15.116:0/2893624954 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc534032440 con 0x7fc5441318f0 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc549634640 5 monclient: authenticate success, global_id 154374 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc549634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc549634640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc549634640 4 set_mon_vals no callback set 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 1 -- 172.21.15.116:0/2893624954 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc534032820 con 0x7fc5441318f0 2024-09-18T12:39:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.596+0000 7fc5427fc640 10 monclient: dump: 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.608 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 10 monclient: shutdown 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 -- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5441318f0 msgr2=0x7fc544131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 --2- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5441318f0 0x7fc544131f30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc534009910 tx=0x7fc5340091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 -- 172.21.15.116:0/2893624954 shutdown_connections 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 --2- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc54409ea50 0x7fc54409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 --2- 172.21.15.116:0/2893624954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5441318f0 0x7fc544131f30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 -- 172.21.15.116:0/2893624954 >> 172.21.15.116:0/2893624954 conn(0x7fc544067110 msgr2=0x7fc544067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 -- 172.21.15.116:0/2893624954 shutdown_connections 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.597+0000 7fc549634640 1 -- 172.21.15.116:0/2893624954 wait complete. 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.598+0000 7fc549634640 10 monclient: build_initial_monmap 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.598+0000 7fc549634640 10 monclient: monmap: 2024-09-18T12:39:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.598+0000 7fc549634640 1 Processor -- start 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.598+0000 7fc549634640 1 -- start start 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.598+0000 7fc549634640 10 monclient: init 2024-09-18T12:39:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 0x7fc54409ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): picked mon.noname-c con 0x7fc544068d90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 0x7fc544130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): picked mon.noname-a con 0x7fc54409ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5441318f0 0x7fc544141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): picked mon.noname-b con 0x7fc5441318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5440a4da0 con 0x7fc54409ea50 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54411c930 con 0x7fc5441318f0 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54409bf10 con 0x7fc544068d90 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc549634640 10 monclient(hunting): authenticate will time out at 1621.920171s 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 0x7fc544130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 0x7fc54409ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 0x7fc54409ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:38036/0 (socket says 172.21.15.116:38036) 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 0x7fc544130dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46000/0 (socket says 172.21.15.116:46000) 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 1 -- 172.21.15.116:0/884639223 learned_addr learned my addr 172.21.15.116:0/884639223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 10 monclient(hunting): get_auth_request con 0x7fc544068d90 auth_method 0 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): get_auth_request con 0x7fc54409ea50 auth_method 0 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 1 -- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 msgr2=0x7fc54409ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 1 -- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 msgr2=0x7fc54409ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 0x7fc54409ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc542ffd640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 0x7fc54409ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc5437fe640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5441318f0 0x7fc544141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc5437fe640 10 monclient(hunting): get_auth_request con 0x7fc5441318f0 auth_method 0 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc5437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc5437fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient(hunting): handle_auth_done global_id 154377 payload 824 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 -- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 msgr2=0x7fc54409ac00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 0x7fc54409ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 -- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5441318f0 msgr2=0x7fc544141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5441318f0 0x7fc544141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 monclient: found mon.noname-a 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.599+0000 7fc53a7fc640 1 -- 172.21.15.116:0/884639223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc534009450 con 0x7fc54409ea50 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53a7fc640 1 -- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 msgr2=0x7fc544130dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53a7fc640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 0x7fc544130dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc52c006ea0 tx=0x7fc52c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 1 -- 172.21.15.116:0/884639223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00b8d0 con 0x7fc54409ea50 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 10 monclient: dump: 2024-09-18T12:39:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 1 -- 172.21.15.116:0/884639223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c00ba30 con 0x7fc54409ea50 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc549634640 5 monclient: authenticate success, global_id 154377 2024-09-18T12:39:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc549634640 10 monclient: _renew_subs 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc549634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54411d050 con 0x7fc54409ea50 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.601+0000 7fc53b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc53b7fe640 1 -- 172.21.15.116:0/884639223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00bdf0 con 0x7fc54409ea50 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc53b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc53b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc53b7fe640 10 monclient: dump: 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc541ffb640 4 set_mon_vals no callback set 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc549634640 10 monclient: _renew_subs 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc549634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54411bc30 con 0x7fc54409ea50 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc53b7fe640 1 -- 172.21.15.116:0/884639223 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc52c014640 con 0x7fc54409ea50 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc549634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc549634640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc549634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.602+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54404bf60 con 0x7fc54409ea50 2024-09-18T12:39:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.603+0000 7fc53b7fe640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc524038490 0x7fc52403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.603+0000 7fc53b7fe640 1 -- 172.21.15.116:0/884639223 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc52c04c760 con 0x7fc54409ea50 2024-09-18T12:39:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.603+0000 7fc542ffd640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc524038490 0x7fc52403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.603+0000 7fc542ffd640 10 monclient: get_auth_request con 0x7fc524038490 auth_method 0 2024-09-18T12:39:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.605+0000 7fc542ffd640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc524038490 0x7fc52403a940 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc534009210 tx=0x7fc534032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.607+0000 7fc53b7fe640 1 -- 172.21.15.116:0/884639223 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc52c0148f0 con 0x7fc54409ea50 2024-09-18T12:39:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.607+0000 7fc53b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.607+0000 7fc53b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:54.724 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:39:54.724 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:39:54.724 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:54.724 INFO:tasks.workunit.client.0.smithi144.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-18T12:39:54.724 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-18T12:39:54.724 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-18T12:39:54.725 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=0 2024-09-18T12:39:54.725 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T12:39:54.725 INFO:tasks.workunit.client.0.smithi144.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T12:39:54.725 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T12:39:54.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.823+0000 7fc549634640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:54.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.823+0000 7fc549634640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:54.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.823+0000 7fc549634640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.823+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc5440a36b0 con 0x7fc54409ea50 2024-09-18T12:39:54.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.824+0000 7fc53b7fe640 1 -- 172.21.15.116:0/884639223 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fc52c000ed0 con 0x7fc54409ea50 2024-09-18T12:39:54.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.824+0000 7fc53b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:54.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.824+0000 7fc53b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:54.825 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":146,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:54.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc524038490 msgr2=0x7fc52403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc524038490 0x7fc52403a940 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc534009210 tx=0x7fc534032000 comp rx=0 tx=0).stop 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 10 monclient: shutdown 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 msgr2=0x7fc544130dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 0x7fc544130dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc52c006ea0 tx=0x7fc52c008040 comp rx=0 tx=0).stop 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 shutdown_connections 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.825+0000 7fc549634640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc524038490 0x7fc52403a940 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.826+0000 7fc549634640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc5441318f0 0x7fc544141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.826+0000 7fc549634640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc54409ea50 0x7fc544130dc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.826+0000 7fc549634640 1 --2- 172.21.15.116:0/884639223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc544068d90 0x7fc54409ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.826+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 >> 172.21.15.116:0/884639223 conn(0x7fc544067110 msgr2=0x7fc54409df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.826+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 shutdown_connections 2024-09-18T12:39:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.826+0000 7fc549634640 1 -- 172.21.15.116:0/884639223 wait complete. 2024-09-18T12:39:54.837 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:39:54.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:39:54.837 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:39:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.930+0000 7f9fd342b640 10 monclient: get_monmap_and_config 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.930+0000 7f9fd342b640 10 monclient: build_initial_monmap 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.930+0000 7f9fd342b640 10 monclient: monmap: 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:54.931372+0000 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:54.931372+0000 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:54.933 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 1 Processor -- start 2024-09-18T12:39:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 1 -- start start 2024-09-18T12:39:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient: init 2024-09-18T12:39:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T12:39:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 0x7f9fcc09c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient(hunting): picked mon.noname-b con 0x7f9fcc09a7f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fcc09abc0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9fcc09afb0 0x7f9fcc12fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient(hunting): picked mon.noname-a con 0x7f9fcc09afb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc09b410 con 0x7f9fcc09abc0 2024-09-18T12:39:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fcc0a2050 con 0x7f9fcc09afb0 2024-09-18T12:39:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc09cfb0 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd342b640 10 monclient(hunting): authenticate will time out at 1622.252879s 2024-09-18T12:39:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.931+0000 7f9fd11c7640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 0x7f9fcc09c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9fcc09afb0 0x7f9fcc12fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 0x7f9fcc09c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:38044/0 (socket says 172.21.15.116:38044) 2024-09-18T12:39:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 1 -- 172.21.15.116:0/3180020439 learned_addr learned my addr 172.21.15.116:0/3180020439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 10 monclient(hunting): get_auth_request con 0x7f9fcc09a7f0 auth_method 0 2024-09-18T12:39:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): get_auth_request con 0x7f9fcc09afb0 auth_method 0 2024-09-18T12:39:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:54.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:54.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 1 -- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 msgr2=0x7f9fcc09c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:39:54.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 1 -- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 msgr2=0x7f9fcc09c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:54.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 1 --2- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 0x7f9fcc09c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:54.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd11c7640 1 --2- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 0x7f9fcc09c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_done global_id 154389 payload 274 2024-09-18T12:39:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 1 -- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 msgr2=0x7f9fcc09c0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 0x7f9fcc09c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 1 -- 172.21.15.116:0/3180020439 >> v1:172.21.15.144:6789/0 conn(0x7f9fcc09abc0 legacy=0x7f9fcc12f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:54.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient: _finish_hunting 0 2024-09-18T12:39:54.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 1 monclient: found mon.noname-a 2024-09-18T12:39:54.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:54.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.932+0000 7f9fd19c8640 1 -- 172.21.15.116:0/3180020439 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc09d290 con 0x7f9fcc09afb0 2024-09-18T12:39:54.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd19c8640 1 -- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09afb0 msgr2=0x7f9fcc12fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:54.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09afb0 0x7f9fcc12fdf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4009e60 tx=0x7f9fc4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf4c0df4974ec11 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 1 -- 172.21.15.116:0/3180020439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4007760 con 0x7f9fcc09afb0 2024-09-18T12:39:54.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:54.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: dump: 2024-09-18T12:39:54.952 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.952 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.952 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.952 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.952 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.952 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.953 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.953 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.953 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.953 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: _finish_auth 0 2024-09-18T12:39:54.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:54.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 1 -- 172.21.15.116:0/3180020439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc402d050 con 0x7f9fcc09afb0 2024-09-18T12:39:54.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:54.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 1 -- 172.21.15.116:0/3180020439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4007af0 con 0x7f9fcc09afb0 2024-09-18T12:39:54.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 5 monclient: authenticate success, global_id 154389 2024-09-18T12:39:54.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 4 set_mon_vals no callback set 2024-09-18T12:39:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fc3fff640 10 monclient: dump: 2024-09-18T12:39:54.955 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.956 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.957 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 10 monclient: shutdown 2024-09-18T12:39:54.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 -- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09afb0 msgr2=0x7f9fcc12fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 --2- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09afb0 0x7f9fcc12fdf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4009e60 tx=0x7f9fc4009780 comp rx=0 tx=0).stop 2024-09-18T12:39:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 -- 172.21.15.116:0/3180020439 shutdown_connections 2024-09-18T12:39:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 --2- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09afb0 0x7f9fcc12fdf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 --2- 172.21.15.116:0/3180020439 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09a7f0 0x7f9fcc09c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 -- 172.21.15.116:0/3180020439 >> 172.21.15.116:0/3180020439 conn(0x7f9fcc067b50 msgr2=0x7f9fcc0a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 -- 172.21.15.116:0/3180020439 shutdown_connections 2024-09-18T12:39:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.935+0000 7f9fd342b640 1 -- 172.21.15.116:0/3180020439 wait complete. 2024-09-18T12:39:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.936+0000 7f9fd342b640 10 monclient: build_initial_monmap 2024-09-18T12:39:54.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.936+0000 7f9fd342b640 10 monclient: monmap: 2024-09-18T12:39:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:54.961 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.936+0000 7f9fd342b640 1 Processor -- start 2024-09-18T12:39:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.936+0000 7f9fd342b640 1 -- start start 2024-09-18T12:39:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.936+0000 7f9fd342b640 10 monclient: init 2024-09-18T12:39:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:39:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09a7f0 0x7f9fcc0942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): picked mon.noname-a con 0x7f9fcc09a7f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fcc09abc0 0x7f9fcc094820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): picked mon.noname-b con 0x7f9fcc09abc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 0x7f9fcc1283c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): picked mon.noname-c con 0x7f9fcc09afb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc12bcd0 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc12b9f0 con 0x7f9fcc09abc0 2024-09-18T12:39:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc12bfb0 con 0x7f9fcc09afb0 2024-09-18T12:39:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd342b640 10 monclient(hunting): authenticate will time out at 1622.258265s 2024-09-18T12:39:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09a7f0 0x7f9fcc0942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 0x7f9fcc1283c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09a7f0 0x7f9fcc0942e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46020/0 (socket says 172.21.15.116:46020) 2024-09-18T12:39:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 -- 172.21.15.116:0/2283432627 learned_addr learned my addr 172.21.15.116:0/2283432627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): get_auth_request con 0x7f9fcc09a7f0 auth_method 0 2024-09-18T12:39:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 10 monclient(hunting): get_auth_request con 0x7f9fcc09afb0 auth_method 0 2024-09-18T12:39:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fcc09abc0 0x7f9fcc094820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 msgr2=0x7f9fcc1283c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:39:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 msgr2=0x7f9fcc1283c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 10 monclient(hunting): get_auth_request con 0x7f9fcc09abc0 auth_method 0 2024-09-18T12:39:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 0x7f9fcc1283c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 0x7f9fcc1283c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd09c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient(hunting): handle_auth_done global_id 154392 payload 824 2024-09-18T12:39:54.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 msgr2=0x7f9fcc1283c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9fcc09afb0 0x7f9fcc1283c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fcc09abc0 msgr2=0x7f9fcc094820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9fcc09abc0 0x7f9fcc094820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient: _finish_hunting 0 2024-09-18T12:39:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 monclient: found mon.noname-a 2024-09-18T12:39:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.937+0000 7f9fd11c7640 1 -- 172.21.15.116:0/2283432627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc4009a20 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.939+0000 7f9fd11c7640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09a7f0 msgr2=0x7f9fcc0942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.939+0000 7f9fd11c7640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09a7f0 0x7f9fcc0942e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fc800e870 tx=0x7f9fc8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc800c6d0 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: dump: 2024-09-18T12:39:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.975 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: _finish_auth 0 2024-09-18T12:39:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc800c830 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fd342b640 5 monclient: authenticate success, global_id 154392 2024-09-18T12:39:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fd342b640 10 monclient: _renew_subs 2024-09-18T12:39:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fd342b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc12aa90 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc800cbf0 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:54.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 10 monclient: dump: 2024-09-18T12:39:54.977 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.977 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.977 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.978 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.978 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.978 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.978 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.978 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.979 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc37fe640 4 set_mon_vals no callback set 2024-09-18T12:39:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fd342b640 10 monclient: _renew_subs 2024-09-18T12:39:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fd342b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc094d60 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.940+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9fc8013410 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd342b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fc17fa640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9fb0038490 0x7f9fb003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd342b640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 0x7f9fcc128900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd342b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc063720 con 0x7f9fcc1325a0 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9fc804c1c0 con 0x7f9fcc09a7f0 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 0x7f9fcc128900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient: get_auth_request con 0x7f9fcc1325a0 auth_method 0 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd09c6640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9fb0038490 0x7f9fb003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:54.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:54.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:54.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd09c6640 10 monclient: get_auth_request con 0x7f9fb0038490 auth_method 0 2024-09-18T12:39:54.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_done global_id 154395 payload 824 2024-09-18T12:39:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.941+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fcc12bb60 con 0x7f9fcc1325a0 2024-09-18T12:39:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.943+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 msgr2=0x7f9fcc128900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.943+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 0x7f9fcc128900 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4034000 tx=0x7f9fc4037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.943+0000 7f9fd09c6640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9fb0038490 0x7f9fb003a940 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc007fd0 tx=0x7f9fbc007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.943+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc403dd60 con 0x7f9fcc1325a0 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.943+0000 7f9fc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.943+0000 7f9fc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.943+0000 7f9fc17fa640 10 monclient: dump: 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.944+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9fc403b440 con 0x7f9fcc1325a0 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.944+0000 7f9fc17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.944+0000 7f9fc17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.944+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 msgr2=0x7f9fcc128900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.944+0000 7f9fc17fa640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 0x7f9fcc128900 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4034000 tx=0x7f9fc4037000 comp rx=0 tx=0).stop 2024-09-18T12:39:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.944+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 msgr2=0x7f9fcc128900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.944+0000 7f9fc17fa640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc1325a0 0x7f9fcc128900 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4034000 tx=0x7f9fc4037000 comp rx=0 tx=0).stop 2024-09-18T12:39:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd342b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd342b640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 0x7f9fcc129240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd342b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc129780 con 0x7f9fcc128e40 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 0x7f9fcc129240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 10 monclient: get_auth_request con 0x7f9fcc128e40 auth_method 0 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.949+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.950+0000 7f9fd19c8640 10 monclient(hunting): handle_auth_done global_id 154398 payload 824 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.950+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9fcc04bf60 con 0x7f9fcc128e40 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 msgr2=0x7f9fcc129240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd19c8640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 0x7f9fcc129240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40097c0 tx=0x7f9fc403f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:54.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4031db0 con 0x7f9fcc128e40 2024-09-18T12:39:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 10 monclient: dump: 2024-09-18T12:39:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:54.989 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:54.991 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7f9fc40453f0 con 0x7f9fcc128e40 2024-09-18T12:39:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 msgr2=0x7f9fcc129240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:54.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 0x7f9fcc129240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40097c0 tx=0x7f9fc403f040 comp rx=0 tx=0).stop 2024-09-18T12:39:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 msgr2=0x7f9fcc129240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fc17fa640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc128e40 0x7f9fcc129240 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:54.992 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 146, 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:55.008 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9fb0038490 msgr2=0x7f9fb003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd342b640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9fb0038490 0x7f9fb003a940 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc007fd0 tx=0x7f9fbc007e10 comp rx=0 tx=0).stop 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd342b640 10 monclient: shutdown 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd342b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09a7f0 msgr2=0x7f9fcc0942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd342b640 1 --2- 172.21.15.116:0/2283432627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fcc09a7f0 0x7f9fcc0942e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fc800e870 tx=0x7f9fc8008040 comp rx=0 tx=0).stop 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 reap_dead start 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd19c8640 1 -- 172.21.15.116:0/2283432627 reap_dead start 2024-09-18T12:39:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.952+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 shutdown_connections 2024-09-18T12:39:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.953+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 >> 172.21.15.116:0/2283432627 conn(0x7f9fcc067b50 msgr2=0x7f9fcc099660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.953+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 shutdown_connections 2024-09-18T12:39:55.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:54.953+0000 7f9fd342b640 1 -- 172.21.15.116:0/2283432627 wait complete. 2024-09-18T12:39:55.011 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:55.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:39:55.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:39:55.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.103+0000 7f40db500640 10 monclient: get_monmap_and_config 2024-09-18T12:39:55.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.103+0000 7f40db500640 10 monclient: build_initial_monmap 2024-09-18T12:39:55.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.103+0000 7f40db500640 10 monclient: monmap: 2024-09-18T12:39:55.104 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:55.104195+0000 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:55.104195+0000 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:55.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 1 Processor -- start 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 1 -- start start 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient: init 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40d4096c60 0x7f40d409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient(hunting): picked mon.noname-a con 0x7f40d4096c60 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40d4097030 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient(hunting): picked mon.noname-b con 0x7f40d4133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d40696a0 con 0x7f40d4097030 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d409ac00 con 0x7f40d4096c60 2024-09-18T12:39:55.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d409d6b0 con 0x7f40d4133b40 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40db500640 10 monclient(hunting): authenticate will time out at 1622.425786s 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d929c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40d4096c60 0x7f40d409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d9a9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d8a9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f40d4097030 0x7f40d4097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39614/0 (socket says 172.21.15.116:39614) 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d9a9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:38064/0 (socket says 172.21.15.116:38064) 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d929c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40d4096c60 0x7f40d409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46058/0 (socket says 172.21.15.116:46058) 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d8a9b640 1 -- 172.21.15.116:0/1057395842 learned_addr learned my addr 172.21.15.116:0/1057395842 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d9a9d640 10 monclient(hunting): get_auth_request con 0x7f40d4133b40 auth_method 0 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d9a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d9a9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d9a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d929c640 10 monclient(hunting): get_auth_request con 0x7f40d4096c60 auth_method 0 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d929c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.104+0000 7f40d929c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d9a9d640 1 -- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 msgr2=0x7f40d4135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d9a9d640 1 -- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 msgr2=0x7f40d4135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d9a9d640 1 --2- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d9a9d640 1 --2- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 10 monclient(hunting): handle_auth_done global_id 154401 payload 274 2024-09-18T12:39:55.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 1 -- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 msgr2=0x7f40d4135f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:39:55.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 1 --2- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4135f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:55.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 1 -- 172.21.15.116:0/1057395842 >> v1:172.21.15.116:6789/0 conn(0x7f40d4097030 legacy=0x7f40d4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:55.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 10 monclient: _finish_hunting 0 2024-09-18T12:39:55.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 1 monclient: found mon.noname-a 2024-09-18T12:39:55.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:55.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:55.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.105+0000 7f40d929c640 1 -- 172.21.15.116:0/1057395842 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d409d2e0 con 0x7f40d4096c60 2024-09-18T12:39:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40d929c640 1 -- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4096c60 msgr2=0x7f40d409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40d929c640 1 --2- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4096c60 0x7f40d409a6c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009850 tx=0x7f40d00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a350731560c112 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 1 -- 172.21.15.116:0/1057395842 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d002add0 con 0x7f40d4096c60 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: dump: 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: _finish_auth 0 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 5 monclient: authenticate success, global_id 154401 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 1 -- 172.21.15.116:0/1057395842 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d0032440 con 0x7f40d4096c60 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 4 set_mon_vals no callback set 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 1 -- 172.21.15.116:0/1057395842 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d00327e0 con 0x7f40d4096c60 2024-09-18T12:39:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40cbfff640 10 monclient: dump: 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 10 monclient: shutdown 2024-09-18T12:39:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 1 -- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4096c60 msgr2=0x7f40d409a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 1 --2- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4096c60 0x7f40d409a6c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009850 tx=0x7f40d00090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 1 -- 172.21.15.116:0/1057395842 shutdown_connections 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.108+0000 7f40db500640 1 --2- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 1 --2- 172.21.15.116:0/1057395842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4096c60 0x7f40d409a6c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 1 -- 172.21.15.116:0/1057395842 >> 172.21.15.116:0/1057395842 conn(0x7f40d40672b0 msgr2=0x7f40d40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 1 -- 172.21.15.116:0/1057395842 shutdown_connections 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 1 -- 172.21.15.116:0/1057395842 wait complete. 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 10 monclient: build_initial_monmap 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 10 monclient: monmap: 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 1 Processor -- start 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 1 -- start start 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.109+0000 7f40db500640 10 monclient: init 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:39:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40d4096c60 0x7f40d413f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): picked mon.noname-b con 0x7f40d4096c60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4097030 0x7f40d412e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): picked mon.noname-a con 0x7f40d4097030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4118230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): picked mon.noname-c con 0x7f40d4133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d409add0 con 0x7f40d4097030 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d41337a0 con 0x7f40d4096c60 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d413f570 con 0x7f40d4133b40 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:55.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40db500640 10 monclient(hunting): authenticate will time out at 1622.431484s 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4097030 0x7f40d412e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4118230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4097030 0x7f40d412e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46060/0 (socket says 172.21.15.116:46060) 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4118230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:38080/0 (socket says 172.21.15.116:38080) 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 -- 172.21.15.116:0/3420643370 learned_addr learned my addr 172.21.15.116:0/3420643370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): get_auth_request con 0x7f40d4097030 auth_method 0 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 10 monclient(hunting): get_auth_request con 0x7f40d4133b40 auth_method 0 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40d4096c60 0x7f40d413f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 10 monclient(hunting): get_auth_request con 0x7f40d4096c60 auth_method 0 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 msgr2=0x7f40d4118230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 msgr2=0x7f40d4118230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4118230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d9a9d640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4118230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient(hunting): handle_auth_done global_id 154404 payload 824 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 msgr2=0x7f40d4118230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4133b40 0x7f40d4118230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40d4096c60 msgr2=0x7f40d413f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40d4096c60 0x7f40d413f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient: _finish_hunting 0 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 monclient: found mon.noname-a 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.110+0000 7f40d8a9b640 1 -- 172.21.15.116:0/3420643370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d0009390 con 0x7f40d4097030 2024-09-18T12:39:55.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40d8a9b640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4097030 msgr2=0x7f40d412e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40d8a9b640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4097030 0x7f40d412e7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c5e0 tx=0x7f40c4004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4005cb0 con 0x7f40d4097030 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: dump: 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:55.127 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: _finish_auth 0 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c4005e30 con 0x7f40d4097030 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40db500640 5 monclient: authenticate success, global_id 154404 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40db500640 10 monclient: _renew_subs 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40db500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d413fe70 con 0x7f40d4097030 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4007340 con 0x7f40d4097030 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:55.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40c97fa640 10 monclient: dump: 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40cb7fe640 4 set_mon_vals no callback set 2024-09-18T12:39:55.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40db500640 10 monclient: _renew_subs 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40db500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.113+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d4119e10 con 0x7f40d4097030 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f40c401b400 con 0x7f40d4097030 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40db500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40db500640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40db500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d411ba70 con 0x7f40d412ecf0 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40c97fa640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40b8038490 0x7f40b803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f40c404be20 con 0x7f40d4097030 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 10 monclient: get_auth_request con 0x7f40d412ecf0 auth_method 0 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d9a9d640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40b8038490 0x7f40b803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d9a9d640 10 monclient: get_auth_request con 0x7f40b8038490 auth_method 0 2024-09-18T12:39:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.114+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:55.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.116+0000 7f40d9a9d640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40b8038490 0x7f40b803a940 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009590 tx=0x7f40cc00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 10 monclient: get_auth_request con 0x7f40d412ecf0 auth_method 0 2024-09-18T12:39:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.315+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:39:55.401 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=0 2024-09-18T12:39:55.402 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T12:39:55.402 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:55.402 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-18T12:39:55.402 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-18T12:39:55.402 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=0 2024-09-18T12:39:55.402 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T12:39:55.402 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T12:39:55.403 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T12:39:55.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:55.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 10 monclient: get_auth_request con 0x7f40d412ecf0 auth_method 0 2024-09-18T12:39:55.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:55.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:55.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:55.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:55.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:55.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:55.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:55.716+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:39:55.856 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=0 2024-09-18T12:39:55.857 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T12:39:55.857 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:55.857 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-18T12:39:55.857 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-18T12:39:55.857 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:39:55.858 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-18T12:39:55.858 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-18T12:39:55.859 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-18T12:39:56.152 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:39:56.152 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:39:56.152 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:56.152 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-18T12:39:56.152 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-18T12:39:56.152 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:39:56.153 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-18T12:39:56.153 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-18T12:39:56.153 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-18T12:39:56.446 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:39:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:39:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-18T12:39:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-18T12:39:56.447 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:39:56.447 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-18T12:39:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-18T12:39:56.448 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-18T12:39:56.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient: get_auth_request con 0x7f40d412ecf0 auth_method 0 2024-09-18T12:39:56.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 10 monclient(hunting): handle_auth_done global_id 184109 payload 824 2024-09-18T12:39:56.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.517+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40d404bf60 con 0x7f40d412ecf0 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.519+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.519+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40d409ecd0 tx=0x7f40d0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0043070 con 0x7f40d412ecf0 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 10 monclient: dump: 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.521 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f40d00473e0 con 0x7f40d412ecf0 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:56.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40d409ecd0 tx=0x7f40d0032000 comp rx=0 tx=0).stop 2024-09-18T12:39:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 msgr2=0x7f40d40298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:56.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.520+0000 7f40c97fa640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d412ecf0 0x7f40d40298c0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40d409ecd0 tx=0x7f40d0032000 comp rx=0 tx=0).stop 2024-09-18T12:39:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40db500640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40db500640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 0x7f40d402a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40db500640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d4065f20 con 0x7f40d4029e00 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 0x7f40d402a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40d929c640 10 monclient: get_auth_request con 0x7f40d4029e00 auth_method 0 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40d929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40d929c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.526+0000 7f40d929c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.527+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.527+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.527+0000 7f40d929c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.527+0000 7f40d929c640 10 monclient(hunting): handle_auth_done global_id 184115 payload 824 2024-09-18T12:39:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.527+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f40d40983b0 con 0x7f40d4029e00 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40d929c640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 msgr2=0x7f40d402a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40d929c640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 0x7f40d402a200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40d4068680 tx=0x7f40d0043000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0048a90 con 0x7f40d4029e00 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 10 monclient: dump: 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f40d00543b0 con 0x7f40d4029e00 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 msgr2=0x7f40d402a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 0x7f40d402a200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40d4068680 tx=0x7f40d0043000 comp rx=0 tx=0).stop 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 msgr2=0x7f40d402a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.529+0000 7f40c97fa640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f40d4029e00 0x7f40d402a200 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40d4068680 tx=0x7f40d0043000 comp rx=0 tx=0).stop 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 146, 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:39:56.533 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.534 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:56.536 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:56.537 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:56.539 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:56.540 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:56.540 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:56.540 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:56.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.540 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:56.541 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:56.542 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.543 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.544 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:56.545 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40b8038490 msgr2=0x7f40b803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f40b8038490 0x7f40b803a940 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009590 tx=0x7f40cc00b040 comp rx=0 tx=0).stop 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 10 monclient: shutdown 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4097030 msgr2=0x7f40d412e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 --2- 172.21.15.116:0/3420643370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40d4097030 0x7f40d412e7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c5e0 tx=0x7f40c4004960 comp rx=0 tx=0).stop 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40d9a9d640 1 -- 172.21.15.116:0/3420643370 reap_dead start 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40d9a9d640 1 -- 172.21.15.116:0/3420643370 reap_dead start 2024-09-18T12:39:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 shutdown_connections 2024-09-18T12:39:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 >> 172.21.15.116:0/3420643370 conn(0x7f40d40672b0 msgr2=0x7f40d40a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 shutdown_connections 2024-09-18T12:39:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.530+0000 7f40db500640 1 -- 172.21.15.116:0/3420643370 wait complete. 2024-09-18T12:39:56.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:56.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:39:56.548 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:39:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.646+0000 7fe35e19d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.646+0000 7fe35e19d640 10 monclient: build_initial_monmap 2024-09-18T12:39:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.646+0000 7fe35e19d640 10 monclient: monmap: 2024-09-18T12:39:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:56.647 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:56.647209+0000 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:56.647209+0000 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:56.648 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 1 Processor -- start 2024-09-18T12:39:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 1 -- start start 2024-09-18T12:39:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient: init 2024-09-18T12:39:56.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe35809a7f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe35809abc0 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe35809afe0 0x7fe358138160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient(hunting): picked mon.noname-c con 0x7fe35809afe0 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35809c800 con 0x7fe35809a7f0 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580a2050 con 0x7fe35809abc0 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35809b440 con 0x7fe35809afe0 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe35e19d640 10 monclient(hunting): authenticate will time out at 1623.968768s 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe3577fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe35809a7f0 0x7fe35809c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44174/0 (socket says 172.21.15.116:44174) 2024-09-18T12:39:56.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.647+0000 7fe3577fe640 1 -- 172.21.15.116:0/3302176827 learned_addr learned my addr 172.21.15.116:0/3302176827 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 1 --2- 172.21.15.116:0/3302176827 >> v2:172.21.15.144:3300/0 conn(0x7fe35809afe0 0x7fe358138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): get_auth_request con 0x7fe35809afe0 auth_method 0 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient(hunting): handle_auth_done global_id 174220 payload 274 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 1 -- 172.21.15.116:0/3302176827 >> v1:172.21.15.144:6789/0 conn(0x7fe35809abc0 legacy=0x7fe358134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 1 -- 172.21.15.116:0/3302176827 >> v1:172.21.15.116:6789/0 conn(0x7fe35809a7f0 legacy=0x7fe35809c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:56.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 1 monclient: found mon.noname-c 2024-09-18T12:39:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:39:56.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.648+0000 7fe357fff640 1 -- 172.21.15.116:0/3302176827 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35809cfb0 con 0x7fe35809afe0 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe357fff640 1 -- 172.21.15.116:0/3302176827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809afe0 msgr2=0x7fe358138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe357fff640 1 --2- 172.21.15.116:0/3302176827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809afe0 0x7fe358138160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe350009e60 tx=0x7fe350009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73c7aa46a3db0ae2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 1 -- 172.21.15.116:0/3302176827 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350007840 con 0x7fe35809afe0 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: dump: 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.661 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 5 monclient: authenticate success, global_id 174220 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 1 -- 172.21.15.116:0/3302176827 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35002d050 con 0x7fe35809afe0 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:56.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 1 -- 172.21.15.116:0/3302176827 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350007c10 con 0x7fe35809afe0 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 4 set_mon_vals no callback set 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe3567fc640 10 monclient: dump: 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.663 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 10 monclient: shutdown 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 1 -- 172.21.15.116:0/3302176827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809afe0 msgr2=0x7fe358138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 1 --2- 172.21.15.116:0/3302176827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809afe0 0x7fe358138160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe350009e60 tx=0x7fe350009780 comp rx=0 tx=0).stop 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 1 -- 172.21.15.116:0/3302176827 shutdown_connections 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 1 --2- 172.21.15.116:0/3302176827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809afe0 0x7fe358138160 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 1 -- 172.21.15.116:0/3302176827 >> 172.21.15.116:0/3302176827 conn(0x7fe358067b50 msgr2=0x7fe3580a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 1 -- 172.21.15.116:0/3302176827 shutdown_connections 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.651+0000 7fe35e19d640 1 -- 172.21.15.116:0/3302176827 wait complete. 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.652+0000 7fe35e19d640 10 monclient: build_initial_monmap 2024-09-18T12:39:56.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.652+0000 7fe35e19d640 10 monclient: monmap: 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.652+0000 7fe35e19d640 1 Processor -- start 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.652+0000 7fe35e19d640 1 -- start start 2024-09-18T12:39:56.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.652+0000 7fe35e19d640 10 monclient: init 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe35809a7f0 0x7fe358140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): picked mon.noname-c con 0x7fe35809a7f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809abc0 0x7fe3581319a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): picked mon.noname-b con 0x7fe35809abc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe35809afe0 0x7fe3580951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): picked mon.noname-a con 0x7fe35809afe0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35812cb30 con 0x7fe35809afe0 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35812de90 con 0x7fe35809abc0 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35809b5f0 con 0x7fe35809a7f0 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe35e19d640 10 monclient(hunting): authenticate will time out at 1623.974348s 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe35809a7f0 0x7fe358140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe35809afe0 0x7fe3580951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe35809a7f0 0x7fe358140ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:50338/0 (socket says 172.21.15.116:50338) 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 1 -- 172.21.15.116:0/45248001 learned_addr learned my addr 172.21.15.116:0/45248001 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 10 monclient(hunting): get_auth_request con 0x7fe35809afe0 auth_method 0 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 10 monclient(hunting): get_auth_request con 0x7fe35809a7f0 auth_method 0 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809abc0 0x7fe3581319a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): get_auth_request con 0x7fe35809abc0 auth_method 0 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient(hunting): handle_auth_done global_id 174223 payload 824 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe35809a7f0 msgr2=0x7fe358140ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe35809a7f0 0x7fe358140ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe35809afe0 msgr2=0x7fe3580951c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe35809afe0 0x7fe3580951c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 1 monclient: found mon.noname-b 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe356ffd640 1 -- 172.21.15.116:0/45248001 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe350009a20 con 0x7fe35809abc0 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe35809a7f0 0x7fe358140ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.653+0000 7fe357fff640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe35809afe0 0x7fe3580951c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe356ffd640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809abc0 msgr2=0x7fe3581319a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe356ffd640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809abc0 0x7fe3581319a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c510 tx=0x7fe348007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480117a0 con 0x7fe35809abc0 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: dump: 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 5 monclient: authenticate success, global_id 174223 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 10 monclient: _renew_subs 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348011900 con 0x7fe35809abc0 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe358141840 con 0x7fe35809abc0 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348011ca0 con 0x7fe35809abc0 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe33f7fe640 10 monclient: dump: 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 10 monclient: _renew_subs 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:39:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe358096c60 con 0x7fe35809abc0 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe355ffb640 4 set_mon_vals no callback set 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 0x7fe358095700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.656+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe358063720 con 0x7fe358131ee0 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe356ffd640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809abc0 0x7fe3581319a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c510 tx=0x7fe348007e10 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fe34804bf40 osd_map(267..267 src has 242..267) v4 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe348012630 con 0x7fe35809abc0 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 0x7fe358095700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient: get_auth_request con 0x7fe358131ee0 auth_method 0 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe33f7fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe344038490 0x7fe34403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe357fff640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe344038490 0x7fe34403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe357fff640 10 monclient: get_auth_request con 0x7fe344038490 auth_method 0 2024-09-18T12:39:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 10 monclient(hunting): handle_auth_done global_id 174226 payload 824 2024-09-18T12:39:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.657+0000 7fe3577fe640 1 -- 172.21.15.116:0/45248001 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe35812e0d0 con 0x7fe358131ee0 2024-09-18T12:39:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.659+0000 7fe3577fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 msgr2=0x7fe358095700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.659+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 0x7fe358095700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe34c007320 tx=0x7fe34c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.659+0000 7fe357fff640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe344038490 0x7fe34403a940 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe350034000 tx=0x7fe350037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.659+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00fd50 con 0x7fe358131ee0 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.659+0000 7fe33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.659+0000 7fe33f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.659+0000 7fe33f7fe640 10 monclient: dump: 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.676 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.660+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe34c00feb0 con 0x7fe358131ee0 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.660+0000 7fe33f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.660+0000 7fe33f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.660+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 msgr2=0x7fe358095700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.660+0000 7fe33f7fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 0x7fe358095700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe34c007320 tx=0x7fe34c004960 comp rx=0 tx=0).stop 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.660+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 msgr2=0x7fe358095700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.660+0000 7fe33f7fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358131ee0 0x7fe358095700 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe34c007320 tx=0x7fe34c004960 comp rx=0 tx=0).stop 2024-09-18T12:39:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.665+0000 7fe35e19d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.665+0000 7fe35e19d640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.665+0000 7fe35e19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.665+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35809d320 con 0x7fe358095c40 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.665+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 10 monclient: get_auth_request con 0x7fe358095c40 auth_method 0 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 msgr2=0x7fe35806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 msgr2=0x7fe35806d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:39:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.666+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:39:56.748 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:39:56.748 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:39:56.748 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:56.748 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-18T12:39:56.748 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-18T12:39:56.748 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:39:56.748 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-18T12:39:56.749 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-18T12:39:56.749 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-18T12:39:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient: get_auth_request con 0x7fe358095c40 auth_method 0 2024-09-18T12:39:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:39:56.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 10 monclient(hunting): handle_auth_done global_id 174238 payload 824 2024-09-18T12:39:56.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.867+0000 7fe3577fe640 1 -- 172.21.15.116:0/45248001 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe358096040 con 0x7fe358095c40 2024-09-18T12:39:56.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.869+0000 7fe3577fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 msgr2=0x7fe35806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.869+0000 7fe3577fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe34c017000 tx=0x7fe34c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c022940 con 0x7fe358095c40 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 10 monclient: dump: 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fe34c00fb40 con 0x7fe358095c40 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 msgr2=0x7fe35806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe34c017000 tx=0x7fe34c018000 comp rx=0 tx=0).stop 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 msgr2=0x7fe35806d3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe358095c40 0x7fe35806d3f0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe34c017000 tx=0x7fe34c018000 comp rx=0 tx=0).stop 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 146, 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:39:56.875 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:56.876 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:56.877 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:56.878 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:56.879 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.886 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:56.887 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe344038490 msgr2=0x7fe34403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe344038490 0x7fe34403a940 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe350034000 tx=0x7fe350037000 comp rx=0 tx=0).stop 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 10 monclient: shutdown 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809abc0 msgr2=0x7fe3581319a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 --2- 172.21.15.116:0/45248001 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe35809abc0 0x7fe3581319a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c510 tx=0x7fe348007e10 comp rx=0 tx=0).stop 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe357fff640 1 -- 172.21.15.116:0/45248001 reap_dead start 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe357fff640 1 -- 172.21.15.116:0/45248001 reap_dead start 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe33f7fe640 1 -- 172.21.15.116:0/45248001 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7fe34804bf40 con 0x7fe35809abc0 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 shutdown_connections 2024-09-18T12:39:56.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 >> 172.21.15.116:0/45248001 conn(0x7fe358067b50 msgr2=0x7fe358132990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:56.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 shutdown_connections 2024-09-18T12:39:56.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.870+0000 7fe35e19d640 1 -- 172.21.15.116:0/45248001 wait complete. 2024-09-18T12:39:56.890 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:56.890 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:39:56.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:39:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.982+0000 7f6815f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.982+0000 7f6815f9d640 10 monclient: build_initial_monmap 2024-09-18T12:39:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.982+0000 7f6815f9d640 10 monclient: monmap: 2024-09-18T12:39:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:56.983417+0000 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:56.983417+0000 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:56.984 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 1 Processor -- start 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 1 -- start start 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient: init 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6810096e10 0x7f681009a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f6810096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f681009aaa0 0x7f68100971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f681009aaa0 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6810097720 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6810099620 con 0x7f6810097720 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6810099340 con 0x7f681009aaa0 2024-09-18T12:39:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6810099900 con 0x7f6810096e10 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.983+0000 7f6815f9d640 10 monclient(hunting): authenticate will time out at 1624.304995s 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6810096e10 0x7f681009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6810097720 0x7f681012e5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44176/0 (socket says 172.21.15.116:44176) 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680ffff640 1 -- 172.21.15.116:0/3170003671 learned_addr learned my addr 172.21.15.116:0/3170003671 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): get_auth_request con 0x7f6810096e10 auth_method 0 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient(hunting): handle_auth_done global_id 184121 payload 274 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680effd640 1 --2- 172.21.15.116:0/3170003671 >> v2:172.21.15.144:3300/0 conn(0x7f681009aaa0 0x7f68100971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 1 -- 172.21.15.116:0/3170003671 >> v2:172.21.15.144:3300/0 conn(0x7f681009aaa0 msgr2=0x7f68100971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 1 --2- 172.21.15.116:0/3170003671 >> v2:172.21.15.144:3300/0 conn(0x7f681009aaa0 0x7f68100971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 1 -- 172.21.15.116:0/3170003671 >> v1:172.21.15.116:6789/0 conn(0x7f6810097720 legacy=0x7f681012e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:39:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.984+0000 7f680f7fe640 1 -- 172.21.15.116:0/3170003671 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810097b10 con 0x7f6810096e10 2024-09-18T12:39:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680f7fe640 1 -- 172.21.15.116:0/3170003671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6810096e10 msgr2=0x7f681009a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680f7fe640 1 --2- 172.21.15.116:0/3170003671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6810096e10 0x7f681009a560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6800009850 tx=0x7f68000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2674ee8cf3dfc681 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:56.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 1 -- 172.21.15.116:0/3170003671 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800032440 con 0x7f6810096e10 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: dump: 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.996 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 1 -- 172.21.15.116:0/3170003671 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68000325a0 con 0x7f6810096e10 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 5 monclient: authenticate success, global_id 184121 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 1 -- 172.21.15.116:0/3170003671 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800032940 con 0x7f6810096e10 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f680e7fc640 10 monclient: dump: 2024-09-18T12:39:56.997 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 4 set_mon_vals no callback set 2024-09-18T12:39:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 10 monclient: shutdown 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 -- 172.21.15.116:0/3170003671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6810096e10 msgr2=0x7f681009a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 --2- 172.21.15.116:0/3170003671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6810096e10 0x7f681009a560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6800009850 tx=0x7f68000090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 -- 172.21.15.116:0/3170003671 shutdown_connections 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 --2- 172.21.15.116:0/3170003671 >> v2:172.21.15.144:3300/0 conn(0x7f681009aaa0 0x7f68100971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 --2- 172.21.15.116:0/3170003671 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6810096e10 0x7f681009a560 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 -- 172.21.15.116:0/3170003671 >> 172.21.15.116:0/3170003671 conn(0x7f68100672b0 msgr2=0x7f68100676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 -- 172.21.15.116:0/3170003671 shutdown_connections 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.987+0000 7f6815f9d640 1 -- 172.21.15.116:0/3170003671 wait complete. 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.988+0000 7f6815f9d640 10 monclient: build_initial_monmap 2024-09-18T12:39:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.988+0000 7f6815f9d640 10 monclient: monmap: 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.988+0000 7f6815f9d640 1 Processor -- start 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.988+0000 7f6815f9d640 1 -- start start 2024-09-18T12:39:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.988+0000 7f6815f9d640 10 monclient: init 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6810096e10 0x7f681013ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f6810096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6810097720 0x7f681012d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f6810097720 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 0x7f68101176a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f681009aaa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681011aa80 con 0x7f6810097720 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681011b0b0 con 0x7f6810096e10 2024-09-18T12:39:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681012e940 con 0x7f681009aaa0 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f6815f9d640 10 monclient(hunting): authenticate will time out at 1624.310419s 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 0x7f68101176a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6810097720 0x7f681012d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 0x7f68101176a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:50354/0 (socket says 172.21.15.116:50354) 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6810097720 0x7f681012d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47390/0 (socket says 172.21.15.116:47390) 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 -- 172.21.15.116:0/1399646472 learned_addr learned my addr 172.21.15.116:0/1399646472 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): get_auth_request con 0x7f681009aaa0 auth_method 0 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 10 monclient(hunting): get_auth_request con 0x7f6810097720 auth_method 0 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6810096e10 0x7f681013ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 10 monclient(hunting): get_auth_request con 0x7f6810096e10 auth_method 0 2024-09-18T12:39:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient(hunting): handle_auth_done global_id 184124 payload 824 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 -- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6810096e10 msgr2=0x7f681013ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6810096e10 0x7f681013ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 -- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6810097720 msgr2=0x7f681012d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6810097720 0x7f681012d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 monclient: found mon.noname-c 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.989+0000 7f680ffff640 1 -- 172.21.15.116:0/1399646472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800009390 con 0x7f681009aaa0 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f680ffff640 1 -- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 msgr2=0x7f68101176a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f680ffff640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 0x7f68101176a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f680800ccc0 tx=0x7f6808005370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 1 -- 172.21.15.116:0/1399646472 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680800ce80 con 0x7f681009aaa0 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: dump: 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:57.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 1 -- 172.21.15.116:0/1399646472 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680800aa60 con 0x7f681009aaa0 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f6815f9d640 5 monclient: authenticate success, global_id 184124 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f6815f9d640 10 monclient: _renew_subs 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f6815f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681013fcf0 con 0x7f681009aaa0 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 1 -- 172.21.15.116:0/1399646472 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680800ae20 con 0x7f681009aaa0 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 10 monclient: dump: 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:57.007 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f6815f9d640 10 monclient: _renew_subs 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f6815f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681013f560 con 0x7f681009aaa0 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f680dffb640 4 set_mon_vals no callback set 2024-09-18T12:39:57.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.992+0000 7f67f77fe640 1 -- 172.21.15.116:0/1399646472 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6808007300 con 0x7f681009aaa0 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f6815f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f6815f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f6815f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681004bf60 con 0x7f681009aaa0 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f67f77fe640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67fc038490 0x7f67fc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f67f77fe640 1 -- 172.21.15.116:0/1399646472 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f680804c430 con 0x7f681009aaa0 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f680f7fe640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67fc038490 0x7f67fc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:57.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.993+0000 7f680f7fe640 10 monclient: get_auth_request con 0x7f67fc038490 auth_method 0 2024-09-18T12:39:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.995+0000 7f680f7fe640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67fc038490 0x7f67fc03a940 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f681009aff0 tx=0x7f6800032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.997+0000 7f67f77fe640 1 -- 172.21.15.116:0/1399646472 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f68080075b0 con 0x7f681009aaa0 2024-09-18T12:39:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.997+0000 7f67f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:57.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:56.997+0000 7f67f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:57.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:39:57.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:39:57.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:39:57.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T12:39:57.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T12:39:57.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-18T12:39:57.189 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-18T12:39:57.190 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=110 2024-09-18T12:39:57.190 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-18T12:39:57.190 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-18T12:39:57.190 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-18T12:39:57.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.216+0000 7f6815f9d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:39:57.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.216+0000 7f6815f9d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:39:57.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.216+0000 7f6815f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:57.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.216+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f68100638d0 con 0x7f681009aaa0 2024-09-18T12:39:57.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.217+0000 7f67f77fe640 1 -- 172.21.15.116:0/1399646472 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f6808010af0 con 0x7f681009aaa0 2024-09-18T12:39:57.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.217+0000 7f67f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:39:57.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.217+0000 7f67f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:57.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.218+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.218+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=11,kv=8,logm=50} crc {auth=3685731978,config=1228700967,health=1182347415,kv=771457840,logm=226586104}) 2024-09-18T12:39:57.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.218+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=12,kv=7,logm=48} crc {auth=3052692363,config=3140514009,health=261395888,kv=96606817,logm=221812964}) 2024-09-18T12:39:57.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.218+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.218+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=11,kv=8,logm=50} crc {auth=3685731978,config=1228700967,health=1182347415,kv=771457840,logm=226586104}) 2024-09-18T12:39:57.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.218+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=33,config=2,health=11,kv=9,logm=45} crc {auth=2147789852,config=3140514009,health=18924727,kv=2187923472,logm=1059419132}) 2024-09-18T12:39:57.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.219+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.219+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513917570}) 2024-09-18T12:39:57.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.219+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459230531}) 2024-09-18T12:39:57.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.219+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.219+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513917570}) 2024-09-18T12:39:57.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.219+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571416238}) 2024-09-18T12:39:57.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950375663}) 2024-09-18T12:39:57.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1431669131}) 2024-09-18T12:39:57.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950375663}) 2024-09-18T12:39:57.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359903724}) 2024-09-18T12:39:57.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67fc038490 msgr2=0x7f67fc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:57.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67fc038490 0x7f67fc03a940 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f681009aff0 tx=0x7f6800032000 comp rx=0 tx=0).stop 2024-09-18T12:39:57.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 10 monclient: shutdown 2024-09-18T12:39:57.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 msgr2=0x7f68101176a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 0x7f68101176a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f680800ccc0 tx=0x7f6808005370 comp rx=0 tx=0).stop 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 shutdown_connections 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f67fc038490 0x7f67fc03a940 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f681009aaa0 0x7f68101176a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6810097720 0x7f681012d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 --2- 172.21.15.116:0/1399646472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6810096e10 0x7f681013ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.220+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 >> 172.21.15.116:0/1399646472 conn(0x7f68100672b0 msgr2=0x7f681012f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 shutdown_connections 2024-09-18T12:39:57.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6815f9d640 1 -- 172.21.15.116:0/1399646472 wait complete. 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841659551}) 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505091445}) 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841659551}) 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.221+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3154794168}) 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.222+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.222+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056357874}) 2024-09-18T12:39:57.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.222+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467846746}) 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.222+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.222+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056357874}) 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.222+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2179345770}) 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.223+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.223+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986823472}) 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.223+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540920281}) 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.223+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.223+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986823472}) 2024-09-18T12:39:57.227 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.223+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509434124}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.224+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.224+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194533502}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.224+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082321283}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.224+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.224+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194533502}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.224+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655318333}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.225+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.225+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452744393}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.225+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174630475}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.225+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.225+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452744393}) 2024-09-18T12:39:57.228 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.225+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2507289678}) 2024-09-18T12:39:57.235 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.226+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.235 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.226+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547693293}) 2024-09-18T12:39:57.235 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.226+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2601924516}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.226+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.226+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547693293}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.226+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640627541}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758199652}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188405586}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758199652}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788005248}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624471529}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3347773980}) 2024-09-18T12:39:57.236 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624471529}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.227+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2948385737}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.228+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.228+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650991237}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.228+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661461104}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.228+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.228+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650991237}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.228+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1216507688}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.229+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.229+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088675607}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.229+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125577308}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.229+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.229+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088675607}) 2024-09-18T12:39:57.237 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.229+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661999152}) 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.230+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.230+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=404748603}) 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.230+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2951525234}) 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.230+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.230+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=404748603}) 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.230+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728572420}) 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111474216}) 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=622601390}) 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.238 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111474216}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210080199}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.231+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187954432}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2253734541}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187954432}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=509166241}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963811712}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1842553362}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963811712}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.232+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3209054539}) 2024-09-18T12:39:57.239 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.233+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.233+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673869532}) 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.233+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1830003644}) 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.233+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.233+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673869532}) 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.233+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2037468920}) 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958248780}) 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3104038755}) 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=958248780}) 2024-09-18T12:39:57.240 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426476959}) 2024-09-18T12:39:57.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:39:57.242 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.242 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1079479218}) 2024-09-18T12:39:57.242 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181089134}) 2024-09-18T12:39:57.242 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.242 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1079479218}) 2024-09-18T12:39:57.243 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.234+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2194094627}) 2024-09-18T12:39:57.296 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.296 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058625804}) 2024-09-18T12:39:57.296 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1492419891}) 2024-09-18T12:39:57.296 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.296 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058625804}) 2024-09-18T12:39:57.296 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622375500}) 2024-09-18T12:39:57.300 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.300 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025652171}) 2024-09-18T12:39:57.300 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3498881485}) 2024-09-18T12:39:57.300 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.300 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025652171}) 2024-09-18T12:39:57.300 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.295+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428377404}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.296+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.296+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888490257}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.296+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170638542}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.296+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.296+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888490257}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.296+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010278926}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286689614}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=22498702}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286689614}) 2024-09-18T12:39:57.301 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2634058524}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158711413}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146683981}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158711413}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.297+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320447464}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.298+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.298+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463277289}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.298+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3696006466}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.298+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.298+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463277289}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.298+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334474126}) 2024-09-18T12:39:57.302 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.303 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329582554}) 2024-09-18T12:39:57.303 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=93962449}) 2024-09-18T12:39:57.303 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.303 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329582554}) 2024-09-18T12:39:57.303 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465876723}) 2024-09-18T12:39:57.304 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.304 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962955863}) 2024-09-18T12:39:57.304 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1029147223}) 2024-09-18T12:39:57.304 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.304 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962955863}) 2024-09-18T12:39:57.304 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.299+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3655057489}) 2024-09-18T12:39:57.308 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.308 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835862897}) 2024-09-18T12:39:57.308 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024163680}) 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835862897}) 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903491341}) 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093680681}) 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307837071}) 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093680681}) 2024-09-18T12:39:57.309 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.307+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661080195}) 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.308+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.308+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388240145}) 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.308+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=112942339}) 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.308+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.308+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388240145}) 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.308+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1921829591}) 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3451261128}) 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953657310}) 2024-09-18T12:39:57.315 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3451261128}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1400823250}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463058440}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=266813709}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463058440}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.309+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814155203}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.310+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.310+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561899048}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.310+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468984094}) 2024-09-18T12:39:57.316 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.310+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.310+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561899048}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.310+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121419371}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438195855}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537335381}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438195855}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814410750}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210492227}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3467349002}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210492227}) 2024-09-18T12:39:57.317 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.311+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456448322}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.312+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.312+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207677204}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.312+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801280465}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.312+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.312+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207677204}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.312+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3636713356}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174620710}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668814260}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174620710}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=800914212}) 2024-09-18T12:39:57.318 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819047521}) 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2201268322}) 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819047521}) 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.313+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4034237838}) 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.314+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.314+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223486476}) 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.314+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907177145}) 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.314+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.314+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223486476}) 2024-09-18T12:39:57.319 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.314+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2299572767}) 2024-09-18T12:39:57.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245602136}) 2024-09-18T12:39:57.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838039779}) 2024-09-18T12:39:57.323 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245602136}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495209025}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674899592}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3030452396}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674899592}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.315+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039230391}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.316+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.316+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627714552}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.316+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=137407357}) 2024-09-18T12:39:57.324 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.316+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.316+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627714552}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.316+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448711142}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344423144}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2756467466}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344423144}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=76501457}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273686548}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039078351}) 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.325 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273686548}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.317+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223562775}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.318+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.318+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223837310}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.318+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2751629071}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.318+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.318+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223837310}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.318+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194845295}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.319+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.319+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117327343}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.319+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007300}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.319+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.319+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117327343}) 2024-09-18T12:39:57.326 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.319+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015234533}) 2024-09-18T12:39:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287167842}) 2024-09-18T12:39:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1149370136}) 2024-09-18T12:39:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287167842}) 2024-09-18T12:39:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74526677}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221371656}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1542720849}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221371656}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.450+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1005423234}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.451+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.451+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449727875}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.451+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939633830}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.451+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.451+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449727875}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.451+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=819557177}) 2024-09-18T12:39:57.456 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217100439}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1209597987}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217100439}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2539545500}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948839237}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.452+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3986216855}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948839237}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585452550}) 2024-09-18T12:39:57.457 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466073283}) 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663532127}) 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466073283}) 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.453+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885736181}) 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.454+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.454+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054142859}) 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.454+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1450108497}) 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.454+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.454+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054142859}) 2024-09-18T12:39:57.458 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.454+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890344313}) 2024-09-18T12:39:57.469 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.469 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275074892}) 2024-09-18T12:39:57.469 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1413686944}) 2024-09-18T12:39:57.469 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.469 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275074892}) 2024-09-18T12:39:57.469 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975538362}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450981254}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2497140364}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.455+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450981254}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.456+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615304723}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.456+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.456+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485457141}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.456+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=184592924}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.456+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.456+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485457141}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.456+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423119560}) 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.457+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.470 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.457+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049322395}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.457+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160483163}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.457+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.457+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049322395}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.457+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041593930}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536879532}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3220847899}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536879532}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160106565}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906923245}) 2024-09-18T12:39:57.471 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1160297838}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906923245}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.458+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900171387}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.459+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.459+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105838811}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.459+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1328956975}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.459+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.459+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105838811}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.459+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4108170958}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.460+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.460+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200675805}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.460+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1714835145}) 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.460+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.472 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.460+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200675805}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.460+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4154406706}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.460+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436812386}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482372237}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436812386}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319200976}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678759084}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131682938}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678759084}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.461+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=279644404}) 2024-09-18T12:39:57.473 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.462+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.462+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080936723}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.462+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4226041141}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.462+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.462+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080936723}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.462+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479041872}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892834849}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3645955594}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892834849}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913857352}) 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.474 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660341733}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713571536}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660341733}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.463+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=47266123}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.464+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.464+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213993104}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.464+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3929741341}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.464+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.464+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213993104}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.464+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357152911}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058555119}) 2024-09-18T12:39:57.475 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=587052162}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058555119}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3169815185}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637875575}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2301607623}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.465+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637875575}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.466+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989611826}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.466+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.466+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435214780}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.466+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1795085600}) 2024-09-18T12:39:57.476 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.466+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.466+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435214780}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.466+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243428667}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.467+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.467+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942322245}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.467+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=865112156}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.467+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.467+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942322245}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.467+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752711362}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.468+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.468+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5} crc {logm=1078815937,mdsmap=3585761405,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2003381882,mon_config_key=3655569570}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.468+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=1553414541,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3454364142}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.468+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.468+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5} crc {logm=1078815937,mdsmap=3585761405,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2003381882,mon_config_key=3655569570}) 2024-09-18T12:39:57.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.468+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=1} crc {logm=2098528154,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2555671963,mon_config_key=3655569571,monmap=1305321694}) 2024-09-18T12:39:57.478 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.474+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.478 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.474+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1896442181}) 2024-09-18T12:39:57.478 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.474+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {monmap=3750608922,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2843437582}) 2024-09-18T12:39:57.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.474+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:39:57.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.474+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1896442181}) 2024-09-18T12:39:57.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:39:57.474+0000 7f6816ff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {monmap=4208436772,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2015871766}) 2024-09-18T12:39:58.242 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:39:58.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.335+0000 7f81b508b640 10 monclient: get_monmap_and_config 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.335+0000 7f81b508b640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.335+0000 7f81b508b640 10 monclient: monmap: 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:58.336628+0000 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:58.336628+0000 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 1 Processor -- start 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 1 -- start start 2024-09-18T12:39:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 10 monclient: init 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81b00981c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b0131da0 0x7f81b0136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.336+0000 7f81b508b640 10 monclient(hunting): picked mon.noname-b con 0x7f81b0131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81b508b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81b0132170 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81b508b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b009ca60 con 0x7f81b00981c0 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81b508b640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b009cca0 con 0x7f81b0132170 2024-09-18T12:39:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81b508b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81b508b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b0131bd0 con 0x7f81b0131da0 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81b508b640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81b508b640 10 monclient(hunting): authenticate will time out at 1625.658041s 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b0131da0 0x7f81b0136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81aed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f81b00981c0 0x7f81b0098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44186/0 (socket says 172.21.15.116:44186) 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81aed9d640 1 -- 172.21.15.116:0/2421124889 learned_addr learned my addr 172.21.15.116:0/2421124889 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): get_auth_request con 0x7f81b0131da0 auth_method 0 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient(hunting): handle_auth_done global_id 184130 payload 274 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 1 -- 172.21.15.116:0/2421124889 >> v1:172.21.15.144:6789/0 conn(0x7f81b0132170 legacy=0x7f81b01390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 1 -- 172.21.15.116:0/2421124889 >> v1:172.21.15.116:6789/0 conn(0x7f81b00981c0 legacy=0x7f81b0098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 1 monclient: found mon.noname-b 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.337+0000 7f81ae59c640 1 -- 172.21.15.116:0/2421124889 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b009e010 con 0x7f81b0131da0 2024-09-18T12:39:58.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81ae59c640 1 -- 172.21.15.116:0/2421124889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b0131da0 msgr2=0x7f81b0136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81ae59c640 1 --2- 172.21.15.116:0/2421124889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b0131da0 0x7f81b0136b70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009880 tx=0x7f81a00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75a7425e19510a19 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 1 -- 172.21.15.116:0/2421124889 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a002aea0 con 0x7f81b0131da0 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: dump: 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.349 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 1 -- 172.21.15.116:0/2421124889 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a0032440 con 0x7f81b0131da0 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 5 monclient: authenticate success, global_id 184130 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 1 -- 172.21.15.116:0/2421124889 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a0032820 con 0x7f81b0131da0 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 4 set_mon_vals no callback set 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81add9b640 10 monclient: dump: 2024-09-18T12:39:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 10 monclient: shutdown 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 1 -- 172.21.15.116:0/2421124889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b0131da0 msgr2=0x7f81b0136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 1 --2- 172.21.15.116:0/2421124889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b0131da0 0x7f81b0136b70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009880 tx=0x7f81a00090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 1 -- 172.21.15.116:0/2421124889 shutdown_connections 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 1 --2- 172.21.15.116:0/2421124889 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b0131da0 0x7f81b0136b70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 1 -- 172.21.15.116:0/2421124889 >> 172.21.15.116:0/2421124889 conn(0x7f81b00a1e30 msgr2=0x7f81b00a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 1 -- 172.21.15.116:0/2421124889 shutdown_connections 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.340+0000 7f81b508b640 1 -- 172.21.15.116:0/2421124889 wait complete. 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.341+0000 7f81b508b640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.341+0000 7f81b508b640 10 monclient: monmap: 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.341+0000 7f81b508b640 1 Processor -- start 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.341+0000 7f81b508b640 1 -- start start 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.341+0000 7f81b508b640 10 monclient: init 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 0x7f81b01314b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): picked mon.noname-c con 0x7f81b00981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b0131da0 0x7f81b0138090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): picked mon.noname-b con 0x7f81b0131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b0132170 0x7f81b01385d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): picked mon.noname-a con 0x7f81b0132170 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b0119d40 con 0x7f81b0132170 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b0098cb0 con 0x7f81b0131da0 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00a08d0 con 0x7f81b00981c0 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81b508b640 10 monclient(hunting): authenticate will time out at 1625.663244s 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 0x7f81b01314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 0x7f81b01314b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:50376/0 (socket says 172.21.15.116:50376) 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 -- 172.21.15.116:0/3351026033 learned_addr learned my addr 172.21.15.116:0/3351026033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): get_auth_request con 0x7f81b00981c0 auth_method 0 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b0132170 0x7f81b01385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 10 monclient(hunting): get_auth_request con 0x7f81b0132170 auth_method 0 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b0131da0 0x7f81b0138090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 10 monclient(hunting): get_auth_request con 0x7f81b0131da0 auth_method 0 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient(hunting): handle_auth_done global_id 184133 payload 824 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 -- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b0131da0 msgr2=0x7f81b0138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b0131da0 0x7f81b0138090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 -- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b0132170 msgr2=0x7f81b01385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b0132170 0x7f81b01385d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 monclient: found mon.noname-c 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.342+0000 7f81aed9d640 1 -- 172.21.15.116:0/3351026033 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0009390 con 0x7f81b00981c0 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81aed9d640 1 -- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 msgr2=0x7f81b01314b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81aed9d640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 0x7f81b01314b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f819800c610 tx=0x7f8198004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81a6ffd640 1 -- 172.21.15.116:0/3351026033 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8198005cb0 con 0x7f81b00981c0 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81a6ffd640 10 monclient: dump: 2024-09-18T12:39:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81a6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81b508b640 5 monclient: authenticate success, global_id 184133 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81a6ffd640 1 -- 172.21.15.116:0/3351026033 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8198005e30 con 0x7f81b00981c0 2024-09-18T12:39:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.344+0000 7f81b508b640 10 monclient: _renew_subs 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b0130ad0 con 0x7f81b00981c0 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81a6ffd640 1 -- 172.21.15.116:0/3351026033 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8198007340 con 0x7f81b00981c0 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81a6ffd640 10 monclient: dump: 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81ad59a640 4 set_mon_vals no callback set 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 10 monclient: _renew_subs 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b011a0f0 con 0x7f81b00981c0 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81a6ffd640 1 -- 172.21.15.116:0/3351026033 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f819801b400 con 0x7f81b00981c0 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:58.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81a6ffd640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8190038490 0x7f819003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.345+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b004bf60 con 0x7f81b00981c0 2024-09-18T12:39:58.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.346+0000 7f81a6ffd640 1 -- 172.21.15.116:0/3351026033 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f819804be70 con 0x7f81b00981c0 2024-09-18T12:39:58.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.346+0000 7f81ae59c640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8190038490 0x7f819003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.346+0000 7f81ae59c640 10 monclient: get_auth_request con 0x7f8190038490 auth_method 0 2024-09-18T12:39:58.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.348+0000 7f81ae59c640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8190038490 0x7f819003a940 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f81a0034000 tx=0x7f81a0032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.350+0000 7f81a6ffd640 1 -- 172.21.15.116:0/3351026033 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f819801ab40 con 0x7f81b00981c0 2024-09-18T12:39:58.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.350+0000 7f81a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:58.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.350+0000 7f81a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:58.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.564+0000 7f81b508b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:39:58.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.564+0000 7f81b508b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:58.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.564+0000 7f81b508b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.564+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f81b0098ee0 con 0x7f81b00981c0 2024-09-18T12:39:58.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.564+0000 7f81a6ffd640 1 -- 172.21.15.116:0/3351026033 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f819800b4a0 con 0x7f81b00981c0 2024-09-18T12:39:58.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.564+0000 7f81a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:39:58.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.564+0000 7f81a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:58.566 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":146,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:39:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8190038490 msgr2=0x7f819003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8190038490 0x7f819003a940 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f81a0034000 tx=0x7f81a0032000 comp rx=0 tx=0).stop 2024-09-18T12:39:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 10 monclient: shutdown 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 msgr2=0x7f81b01314b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 0x7f81b01314b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f819800c610 tx=0x7f8198004960 comp rx=0 tx=0).stop 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 shutdown_connections 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8190038490 0x7f819003a940 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b0132170 0x7f81b01385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81b0131da0 0x7f81b0138090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 --2- 172.21.15.116:0/3351026033 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f81b00981c0 0x7f81b01314b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 >> 172.21.15.116:0/3351026033 conn(0x7f81b00a1e30 msgr2=0x7f81b0096f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 shutdown_connections 2024-09-18T12:39:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.566+0000 7f81b508b640 1 -- 172.21.15.116:0/3351026033 wait complete. 2024-09-18T12:39:58.577 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:39:58.577 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:39:58.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:39:58.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:39:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.671+0000 7f1cc6f1d640 10 monclient: get_monmap_and_config 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.671+0000 7f1cc6f1d640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.671+0000 7f1cc6f1d640 10 monclient: monmap: 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:58.671742+0000 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:58.671742+0000 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:58.673 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.671+0000 7f1cc6f1d640 1 Processor -- start 2024-09-18T12:39:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.671+0000 7f1cc6f1d640 1 -- start start 2024-09-18T12:39:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.671+0000 7f1cc6f1d640 10 monclient: init 2024-09-18T12:39:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cc0131ab0 0x7f1cc0131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc0131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1cc009e910 0x7f1cc009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc009e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc009f250 0x7f1cc0068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc009f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc009faf0 con 0x7f1cc0131ab0 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc009fd30 con 0x7f1cc009e910 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc00a1110 con 0x7f1cc009f250 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc6f1d640 10 monclient(hunting): authenticate will time out at 1625.993312s 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cc0131ab0 0x7f1cc0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc009f250 0x7f1cc0068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cc0131ab0 0x7f1cc0131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47408/0 (socket says 172.21.15.116:47408) 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/98908290 learned_addr learned my addr 172.21.15.116:0/98908290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): get_auth_request con 0x7f1cc0131ab0 auth_method 0 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 10 monclient(hunting): get_auth_request con 0x7f1cc009f250 auth_method 0 2024-09-18T12:39:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 1 --2- 172.21.15.116:0/98908290 >> v2:172.21.15.144:3300/0 conn(0x7f1cc009e910 0x7f1cc009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 10 monclient(hunting): get_auth_request con 0x7f1cc009e910 auth_method 0 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_done global_id 154446 payload 274 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc009f250 msgr2=0x7f1cc0068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc009f250 0x7f1cc0068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/98908290 >> v2:172.21.15.144:3300/0 conn(0x7f1cc009e910 msgr2=0x7f1cc009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/98908290 >> v2:172.21.15.144:3300/0 conn(0x7f1cc009e910 0x7f1cc009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 monclient: found mon.noname-a 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/98908290 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc00a13f0 con 0x7f1cc0131ab0 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.672+0000 7f1cc54ba640 1 --2- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc009f250 0x7f1cc0068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0131ab0 msgr2=0x7f1cc0131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0131ab0 0x7f1cc0131e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1cc009bdb0 tx=0x7f1cb40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab57d200f7dbf43c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 1 -- 172.21.15.116:0/98908290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4032440 con 0x7f1cc0131ab0 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: dump: 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 1 -- 172.21.15.116:0/98908290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb40325a0 con 0x7f1cc0131ab0 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cc6f1d640 5 monclient: authenticate success, global_id 154446 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cc6f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cc6f1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 1 -- 172.21.15.116:0/98908290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4032940 con 0x7f1cc0131ab0 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cc6f1d640 4 set_mon_vals no callback set 2024-09-18T12:39:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.675+0000 7f1cbf7fe640 10 monclient: dump: 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 10 monclient: shutdown 2024-09-18T12:39:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0131ab0 msgr2=0x7f1cc0131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0131ab0 0x7f1cc0131e80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1cc009bdb0 tx=0x7f1cb40091d0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/98908290 shutdown_connections 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc009f250 0x7f1cc0068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/98908290 >> v2:172.21.15.144:3300/0 conn(0x7f1cc009e910 0x7f1cc009ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/98908290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0131ab0 0x7f1cc0131e80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/98908290 >> 172.21.15.116:0/98908290 conn(0x7f1cc0067d30 msgr2=0x7f1cc00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/98908290 shutdown_connections 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/98908290 wait complete. 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.676+0000 7f1cc6f1d640 10 monclient: monmap: 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 Processor -- start 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 -- start start 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient: init 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc009e910 0x7f1cc009b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f1cc009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc009f250 0x7f1cc011b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f1cc009f250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc0131ab0 0x7f1cc011bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f1cc0131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc011d060 con 0x7f1cc009f250 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc00a4480 con 0x7f1cc009e910 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc011e0e0 con 0x7f1cc0131ab0 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc6f1d640 10 monclient(hunting): authenticate will time out at 1625.998689s 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc0131ab0 0x7f1cc011bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc009f250 0x7f1cc011b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc0131ab0 0x7f1cc011bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:50394/0 (socket says 172.21.15.116:50394) 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 1 -- 172.21.15.116:0/3339730442 learned_addr learned my addr 172.21.15.116:0/3339730442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 10 monclient(hunting): get_auth_request con 0x7f1cc0131ab0 auth_method 0 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc009f250 0x7f1cc011b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47416/0 (socket says 172.21.15.116:47416) 2024-09-18T12:39:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 10 monclient(hunting): get_auth_request con 0x7f1cc009f250 auth_method 0 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc54ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc009e910 0x7f1cc009b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc4cb9640 10 monclient(hunting): get_auth_request con 0x7f1cc009e910 auth_method 0 2024-09-18T12:39:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc4cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.677+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 10 monclient(hunting): handle_auth_done global_id 184139 payload 824 2024-09-18T12:39:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc009e910 msgr2=0x7f1cc009b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1cc009e910 0x7f1cc009b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc009f250 msgr2=0x7f1cc011b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc009f250 0x7f1cc011b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 1 monclient: found mon.noname-c 2024-09-18T12:39:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cc54ba640 1 -- 172.21.15.116:0/3339730442 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8009a20 con 0x7f1cc0131ab0 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.678+0000 7f1cbffff640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc009f250 0x7f1cc011b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc54ba640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc0131ab0 msgr2=0x7f1cc011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc54ba640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc0131ab0 0x7f1cc011bbf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009fc0 tx=0x7f1cb800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800ce70 con 0x7f1cc0131ab0 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: dump: 2024-09-18T12:39:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc6f1d640 5 monclient: authenticate success, global_id 184139 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc6f1d640 10 monclient: _renew_subs 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc6f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb800f040 con 0x7f1cc0131ab0 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb4009450 con 0x7f1cc0131ab0 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80148e0 con 0x7f1cc0131ab0 2024-09-18T12:39:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbcff9640 10 monclient: dump: 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cbeffd640 4 set_mon_vals no callback set 2024-09-18T12:39:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc6f1d640 10 monclient: _renew_subs 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc6f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.680+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc011cb30 con 0x7f1cc0131ab0 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc6f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1cb8003020 con 0x7f1cc0131ab0 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 0x7f1cc0096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0063740 con 0x7f1cc011c130 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 0x7f1cc0096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient: get_auth_request con 0x7f1cc011c130 auth_method 0 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cbcff9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ca4038490 0x7f1ca403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1cb804cdf0 con 0x7f1cc0131ab0 2024-09-18T12:39:58.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_done global_id 154452 payload 824 2024-09-18T12:39:58.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/3339730442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cc0130700 con 0x7f1cc011c130 2024-09-18T12:39:58.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cbffff640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ca4038490 0x7f1ca403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.681+0000 7f1cbffff640 10 monclient: get_auth_request con 0x7f1ca4038490 auth_method 0 2024-09-18T12:39:58.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.683+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 msgr2=0x7f1cc0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.683+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 0x7f1cc0096a60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4034000 tx=0x7f1cb4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.683+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb403ca10 con 0x7f1cc011c130 2024-09-18T12:39:58.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.683+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:58.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 10 monclient: dump: 2024-09-18T12:39:58.707 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1cb403cb70 con 0x7f1cc011c130 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 msgr2=0x7f1cc0096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 0x7f1cc0096a60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4034000 tx=0x7f1cb4032000 comp rx=0 tx=0).stop 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 msgr2=0x7f1cc0096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbcff9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc011c130 0x7f1cc0096a60 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4034000 tx=0x7f1cb4032000 comp rx=0 tx=0).stop 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.684+0000 7f1cbffff640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ca4038490 0x7f1ca403a940 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0113e0 tx=0x7f1cac0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.689+0000 7f1cc6f1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.689+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 0x7f1cc0097370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.689+0000 7f1cc6f1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.689+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc004bf60 con 0x7f1cc0096fa0 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 0x7f1cc0097370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient: get_auth_request con 0x7f1cc0096fa0 auth_method 0 2024-09-18T12:39:58.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 10 monclient(hunting): handle_auth_done global_id 154455 payload 824 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.690+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/3339730442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1cc011e220 con 0x7f1cc0096fa0 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cc4cb9640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 msgr2=0x7f1cc0097370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cc4cb9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 0x7f1cc0097370 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1cc009be70 tx=0x7f1cb403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4046e40 con 0x7f1cc0096fa0 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:58.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 10 monclient: dump: 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4750 (secure 0 0 0) 0x7f1cb403b3d0 con 0x7f1cc0096fa0 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 msgr2=0x7f1cc0097370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 0x7f1cc0097370 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1cc009be70 tx=0x7f1cb403d000 comp rx=0 tx=0).stop 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 msgr2=0x7f1cc0097370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.692+0000 7f1cbcff9640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cc0096fa0 0x7f1cc0097370 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1cc009be70 tx=0x7f1cb403d000 comp rx=0 tx=0).stop 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 146, 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:58.712 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:58.713 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:58.714 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:58.715 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:58.716 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.717 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:58.718 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:58.719 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:58.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:58.722 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:58.724 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:58.725 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:58.726 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:58.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ca4038490 msgr2=0x7f1ca403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ca4038490 0x7f1ca403a940 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0113e0 tx=0x7f1cac0090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 10 monclient: shutdown 2024-09-18T12:39:58.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc0131ab0 msgr2=0x7f1cc011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 --2- 172.21.15.116:0/3339730442 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1cc0131ab0 0x7f1cc011bbf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009fc0 tx=0x7f1cb800b040 comp rx=0 tx=0).stop 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc54ba640 1 -- 172.21.15.116:0/3339730442 reap_dead start 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc54ba640 1 -- 172.21.15.116:0/3339730442 reap_dead start 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 shutdown_connections 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 >> 172.21.15.116:0/3339730442 conn(0x7f1cc0067d30 msgr2=0x7f1cc00a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 shutdown_connections 2024-09-18T12:39:58.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.693+0000 7f1cc6f1d640 1 -- 172.21.15.116:0/3339730442 wait complete. 2024-09-18T12:39:58.729 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:58.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:39:58.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:39:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.822+0000 7f87eba9a640 10 monclient: get_monmap_and_config 2024-09-18T12:39:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.822+0000 7f87eba9a640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.822+0000 7f87eba9a640 10 monclient: monmap: 2024-09-18T12:39:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:58.823087+0000 2024-09-18T12:39:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:58.823087+0000 2024-09-18T12:39:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:58.824 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:58.825 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:58.825 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.822+0000 7f87eba9a640 1 Processor -- start 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 1 -- start start 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient: init 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f87e40974f0 0x7f87e412ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient(hunting): picked mon.noname-c con 0x7f87e40974f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87e409a8a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:58.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87e4133070 addr v1:172.21.15.144:6789/0 2024-09-18T12:39:58.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e40978c0 con 0x7f87e409a8a0 2024-09-18T12:39:58.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e409ca20 con 0x7f87e4133070 2024-09-18T12:39:58.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e4098ad0 con 0x7f87e40974f0 2024-09-18T12:39:58.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87eba9a640 10 monclient(hunting): authenticate will time out at 1626.144684s 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87e9035640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f87e409a8a0 0x7f87e409ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44196/0 (socket says 172.21.15.116:44196) 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.823+0000 7f87e9035640 1 -- 172.21.15.116:0/477119139 learned_addr learned my addr 172.21.15.116:0/477119139 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 1 --2- 172.21.15.116:0/477119139 >> v2:172.21.15.144:3300/0 conn(0x7f87e40974f0 0x7f87e412ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): get_auth_request con 0x7f87e40974f0 auth_method 0 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient(hunting): handle_auth_done global_id 174259 payload 274 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 1 -- 172.21.15.116:0/477119139 >> v1:172.21.15.144:6789/0 conn(0x7f87e4133070 legacy=0x7f87e4135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 1 -- 172.21.15.116:0/477119139 >> v1:172.21.15.116:6789/0 conn(0x7f87e409a8a0 legacy=0x7f87e409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 1 monclient: found mon.noname-c 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.824+0000 7f87e9836640 1 -- 172.21.15.116:0/477119139 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4098db0 con 0x7f87e40974f0 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e9836640 1 -- 172.21.15.116:0/477119139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e40974f0 msgr2=0x7f87e412ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e9836640 1 --2- 172.21.15.116:0/477119139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e40974f0 0x7f87e412ab00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009850 tx=0x7f87e00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4f00b3da3267f10 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 1 -- 172.21.15.116:0/477119139 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0032440 con 0x7f87e40974f0 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: dump: 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87eba9a640 5 monclient: authenticate success, global_id 174259 2024-09-18T12:39:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87eba9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 1 -- 172.21.15.116:0/477119139 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e00325a0 con 0x7f87e40974f0 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87eba9a640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87eba9a640 4 set_mon_vals no callback set 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 1 -- 172.21.15.116:0/477119139 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0032980 con 0x7f87e40974f0 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.827+0000 7f87e8834640 10 monclient: dump: 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 10 monclient: shutdown 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 -- 172.21.15.116:0/477119139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e40974f0 msgr2=0x7f87e412ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 --2- 172.21.15.116:0/477119139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e40974f0 0x7f87e412ab00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009850 tx=0x7f87e00090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 -- 172.21.15.116:0/477119139 shutdown_connections 2024-09-18T12:39:58.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 --2- 172.21.15.116:0/477119139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e40974f0 0x7f87e412ab00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 -- 172.21.15.116:0/477119139 >> 172.21.15.116:0/477119139 conn(0x7f87e40672b0 msgr2=0x7f87e40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 -- 172.21.15.116:0/477119139 shutdown_connections 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 -- 172.21.15.116:0/477119139 wait complete. 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 10 monclient: monmap: 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:58.843 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.828+0000 7f87eba9a640 1 Processor -- start 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 1 -- start start 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient: init 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e40974f0 0x7f87e406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): picked mon.noname-c con 0x7f87e40974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e409a8a0 0x7f87e4125b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): picked mon.noname-b con 0x7f87e409a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87e4133070 0x7f87e4126080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): picked mon.noname-a con 0x7f87e4133070 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4098300 con 0x7f87e4133070 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4138860 con 0x7f87e409a8a0 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e406a650 con 0x7f87e40974f0 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87eba9a640 10 monclient(hunting): authenticate will time out at 1626.150479s 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e40974f0 0x7f87e406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87e4133070 0x7f87e4126080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87e4133070 0x7f87e4126080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47462/0 (socket says 172.21.15.116:47462) 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e40974f0 0x7f87e406e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:50396/0 (socket says 172.21.15.116:50396) 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 -- 172.21.15.116:0/2611082235 learned_addr learned my addr 172.21.15.116:0/2611082235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): get_auth_request con 0x7f87e4133070 auth_method 0 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 10 monclient(hunting): get_auth_request con 0x7f87e40974f0 auth_method 0 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e409a8a0 0x7f87e4125b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 10 monclient(hunting): get_auth_request con 0x7f87e409a8a0 auth_method 0 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient(hunting): handle_auth_done global_id 154458 payload 824 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e40974f0 msgr2=0x7f87e406e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e40974f0 0x7f87e406e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e409a8a0 msgr2=0x7f87e4125b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f87e409a8a0 0x7f87e4125b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 monclient: found mon.noname-a 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87ea037640 1 -- 172.21.15.116:0/2611082235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0009390 con 0x7f87e4133070 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.829+0000 7f87e9836640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e40974f0 0x7f87e406e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87ea037640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87e4133070 msgr2=0x7f87e4126080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87ea037640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87e4133070 0x7f87e4126080 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c810 tx=0x7f87dc002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00ce00 con 0x7f87e4133070 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:39:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: dump: 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87eba9a640 5 monclient: authenticate success, global_id 154458 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87eba9a640 10 monclient: _renew_subs 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87eba9a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e4127920 con 0x7f87e4133070 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc00aa60 con 0x7f87e4133070 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00ae20 con 0x7f87e4133070 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:39:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87d9ffb640 10 monclient: dump: 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87dbfff640 4 set_mon_vals no callback set 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87eba9a640 10 monclient: _renew_subs 2024-09-18T12:39:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87eba9a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.832+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e406a390 con 0x7f87e4133070 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87eba9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87eba9a640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 0x7f87e41265c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87eba9a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4065f20 con 0x7f87e406f250 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f87dc012630 con 0x7f87e4133070 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 0x7f87e41265c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient: get_auth_request con 0x7f87e406f250 auth_method 0 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87d9ffb640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f87c8038490 0x7f87c803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f87dc04c050 con 0x7f87e4133070 2024-09-18T12:39:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 10 monclient(hunting): handle_auth_done global_id 184145 payload 824 2024-09-18T12:39:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9836640 1 -- 172.21.15.116:0/2611082235 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87e404bf60 con 0x7f87e406f250 2024-09-18T12:39:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9035640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f87c8038490 0x7f87c803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.833+0000 7f87e9035640 10 monclient: get_auth_request con 0x7f87c8038490 auth_method 0 2024-09-18T12:39:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.835+0000 7f87e9836640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 msgr2=0x7f87e41265c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.835+0000 7f87e9836640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 0x7f87e41265c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87e409d040 tx=0x7f87e0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.835+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e003bed0 con 0x7f87e406f250 2024-09-18T12:39:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.835+0000 7f87d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.835+0000 7f87d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.835+0000 7f87d9ffb640 10 monclient: dump: 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87e9035640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f87c8038490 0x7f87c803a940 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f87d4007fd0 tx=0x7f87d4007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f87e0040070 con 0x7f87e406f250 2024-09-18T12:39:58.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87d9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 msgr2=0x7f87e41265c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87d9ffb640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 0x7f87e41265c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87e409d040 tx=0x7f87e0032000 comp rx=0 tx=0).stop 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 msgr2=0x7f87e41265c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.836+0000 7f87d9ffb640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e406f250 0x7f87e41265c0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87e409d040 tx=0x7f87e0032000 comp rx=0 tx=0).stop 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.841+0000 7f87eba9a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.841+0000 7f87eba9a640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 0x7f87e4134820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.841+0000 7f87eba9a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.841+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4127560 con 0x7f87e4134450 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.841+0000 7f87e9836640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 0x7f87e4134820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient: get_auth_request con 0x7f87e4134450 auth_method 0 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 10 monclient(hunting): handle_auth_done global_id 184148 payload 824 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.842+0000 7f87e9836640 1 -- 172.21.15.116:0/2611082235 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f87e406a7c0 con 0x7f87e4134450 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.844+0000 7f87e9836640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 msgr2=0x7f87e4134820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.844+0000 7f87e9836640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 0x7f87e4134820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009850 tx=0x7f87e003e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.844+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0046b60 con 0x7f87e4134450 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.844+0000 7f87d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.844+0000 7f87d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.844+0000 7f87d9ffb640 10 monclient: dump: 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.859 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f87e0046cc0 con 0x7f87e4134450 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87d9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87d9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 msgr2=0x7f87e4134820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87d9ffb640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 0x7f87e4134820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009850 tx=0x7f87e003e070 comp rx=0 tx=0).stop 2024-09-18T12:39:58.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87d9ffb640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 msgr2=0x7f87e4134820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87d9ffb640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f87e4134450 0x7f87e4134820 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009850 tx=0x7f87e003e070 comp rx=0 tx=0).stop 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 146, 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:58.861 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:58.862 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:58.863 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:58.864 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:58.864 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:58.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:58.864 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:58.864 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:58.864 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:58.865 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:58.866 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:58.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:58.874 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:58.874 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:58.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f87c8038490 msgr2=0x7f87c803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f87c8038490 0x7f87c803a940 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f87d4007fd0 tx=0x7f87d4007f00 comp rx=0 tx=0).stop 2024-09-18T12:39:58.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 10 monclient: shutdown 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87e4133070 msgr2=0x7f87e4126080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 --2- 172.21.15.116:0/2611082235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87e4133070 0x7f87e4126080 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c810 tx=0x7f87dc002620 comp rx=0 tx=0).stop 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87ea037640 1 -- 172.21.15.116:0/2611082235 reap_dead start 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87ea037640 1 -- 172.21.15.116:0/2611082235 reap_dead start 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 shutdown_connections 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 >> 172.21.15.116:0/2611082235 conn(0x7f87e40672b0 msgr2=0x7f87e409b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 shutdown_connections 2024-09-18T12:39:58.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.845+0000 7f87eba9a640 1 -- 172.21.15.116:0/2611082235 wait complete. 2024-09-18T12:39:58.876 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:58.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:39:58.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:39:58.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.970+0000 7fd6ce2dd640 10 monclient: get_monmap_and_config 2024-09-18T12:39:58.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.970+0000 7fd6ce2dd640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.971+0000 7fd6ce2dd640 10 monclient: monmap: 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:39:58.971658+0000 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:39:58.971658+0000 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:39:58.972 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:39:58.973 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.971+0000 7fd6ce2dd640 1 Processor -- start 2024-09-18T12:39:58.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.971+0000 7fd6ce2dd640 1 -- start start 2024-09-18T12:39:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.971+0000 7fd6ce2dd640 10 monclient: init 2024-09-18T12:39:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T12:39:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6c809c860 addr v1:172.21.15.116:6789/0 2024-09-18T12:39:58.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c809e840 0x7fd6c809aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient(hunting): picked mon.noname-b con 0x7fd6c809e840 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6c8136cb0 0x7fd6c8139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient(hunting): picked mon.noname-a con 0x7fd6c8136cb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c809ec40 con 0x7fd6c809c860 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c809a9d0 con 0x7fd6c8136cb0 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c806a320 con 0x7fd6c809e840 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6ce2dd640 10 monclient(hunting): authenticate will time out at 1626.293315s 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c809e840 0x7fd6c809aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd6c809c860 0x7fd6c809cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44206/0 (socket says 172.21.15.116:44206) 2024-09-18T12:39:58.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c7fff640 1 -- 172.21.15.116:0/2138333697 learned_addr learned my addr 172.21.15.116:0/2138333697 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): get_auth_request con 0x7fd6c809e840 auth_method 0 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 1 --2- 172.21.15.116:0/2138333697 >> v2:172.21.15.116:3300/0 conn(0x7fd6c8136cb0 0x7fd6c8139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 10 monclient(hunting): get_auth_request con 0x7fd6c8136cb0 auth_method 0 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient(hunting): handle_auth_done global_id 184151 payload 274 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 1 -- 172.21.15.116:0/2138333697 >> v2:172.21.15.116:3300/0 conn(0x7fd6c8136cb0 msgr2=0x7fd6c8139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 1 --2- 172.21.15.116:0/2138333697 >> v2:172.21.15.116:3300/0 conn(0x7fd6c8136cb0 0x7fd6c8139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 1 -- 172.21.15.116:0/2138333697 >> v1:172.21.15.116:6789/0 conn(0x7fd6c809c860 legacy=0x7fd6c809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 1 monclient: found mon.noname-b 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6c77fe640 1 -- 172.21.15.116:0/2138333697 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8068a20 con 0x7fd6c809e840 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.972+0000 7fd6cc87a640 1 --2- 172.21.15.116:0/2138333697 >> v2:172.21.15.116:3300/0 conn(0x7fd6c8136cb0 0x7fd6c8139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c77fe640 1 -- 172.21.15.116:0/2138333697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c809e840 msgr2=0x7fd6c809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c77fe640 1 --2- 172.21.15.116:0/2138333697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c809e840 0x7fd6c809aba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009850 tx=0x7fd6bc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c9c69decf641676 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 1 -- 172.21.15.116:0/2138333697 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc02add0 con 0x7fd6c809e840 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: dump: 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.981 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 1 -- 172.21.15.116:0/2138333697 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc032440 con 0x7fd6c809e840 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6ce2dd640 5 monclient: authenticate success, global_id 184151 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6ce2dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 1 -- 172.21.15.116:0/2138333697 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0327e0 con 0x7fd6c809e840 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6ce2dd640 10 monclient: get_monmap_and_config success 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6ce2dd640 4 set_mon_vals no callback set 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.975+0000 7fd6c6ffd640 10 monclient: dump: 2024-09-18T12:39:58.982 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.983 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 10 monclient: shutdown 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/2138333697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c809e840 msgr2=0x7fd6c809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 --2- 172.21.15.116:0/2138333697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c809e840 0x7fd6c809aba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009850 tx=0x7fd6bc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/2138333697 shutdown_connections 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 --2- 172.21.15.116:0/2138333697 >> v2:172.21.15.116:3300/0 conn(0x7fd6c8136cb0 0x7fd6c8139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 --2- 172.21.15.116:0/2138333697 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c809e840 0x7fd6c809aba0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/2138333697 >> 172.21.15.116:0/2138333697 conn(0x7fd6c80a2e40 msgr2=0x7fd6c80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/2138333697 shutdown_connections 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/2138333697 wait complete. 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 10 monclient: build_initial_monmap 2024-09-18T12:39:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.976+0000 7fd6ce2dd640 10 monclient: monmap: 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 Processor -- start 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 -- start start 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient: init 2024-09-18T12:39:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient: _reopen_session rank -1 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c809c860 0x7fd6c806e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): picked mon.noname-b con 0x7fd6c809c860 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c809e840 0x7fd6c806e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): picked mon.noname-a con 0x7fd6c809e840 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c8136cb0 0x7fd6c806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): picked mon.noname-c con 0x7fd6c8136cb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c8135f20 con 0x7fd6c809e840 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c811fac0 con 0x7fd6c809c860 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c8136870 con 0x7fd6c8136cb0 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): _renew_subs 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6ce2dd640 10 monclient(hunting): authenticate will time out at 1626.298860s 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c809e840 0x7fd6c806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c809e840 0x7fd6c806e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47478/0 (socket says 172.21.15.116:47478) 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6c77fe640 1 -- 172.21.15.116:0/1226150708 learned_addr learned my addr 172.21.15.116:0/1226150708 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6c77fe640 10 monclient(hunting): get_auth_request con 0x7fd6c809e840 auth_method 0 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6cc87a640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c8136cb0 0x7fd6c806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6cc87a640 10 monclient(hunting): get_auth_request con 0x7fd6c8136cb0 auth_method 0 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6cc87a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6cc87a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.977+0000 7fd6cc87a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 10 monclient(hunting): handle_auth_done global_id 184154 payload 824 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6c7fff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c809c860 0x7fd6c806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c809c860 msgr2=0x7fd6c806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c809c860 0x7fd6c806e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c809e840 msgr2=0x7fd6c806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c809e840 0x7fd6c806e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 10 monclient: _finish_hunting 0 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 1 monclient: found mon.noname-c 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6cc87a640 1 -- 172.21.15.116:0/1226150708 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c0009a20 con 0x7fd6c8136cb0 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.978+0000 7fd6c77fe640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c809e840 0x7fd6c806e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6cc87a640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c8136cb0 msgr2=0x7fd6c806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6cc87a640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c8136cb0 0x7fd6c806ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009ed0 tx=0x7fd6c000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0014480 con 0x7fd6c8136cb0 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: dump: 2024-09-18T12:39:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: _finish_auth 0 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6ce2dd640 5 monclient: authenticate success, global_id 184154 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c000f040 con 0x7fd6c8136cb0 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6ce2dd640 10 monclient: _renew_subs 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0014850 con 0x7fd6c8136cb0 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6ce2dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc009390 con 0x7fd6c8136cb0 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:39:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6affff640 10 monclient: dump: 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6c67fc640 4 set_mon_vals no callback set 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6ce2dd640 10 monclient: _renew_subs 2024-09-18T12:39:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6ce2dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.980+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c811dcd0 con 0x7fd6c8136cb0 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6ce2dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6ce2dd640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 0x7fd6c806f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6ce2dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c811f980 con 0x7fd6c811e880 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd6c0014bc0 con 0x7fd6c8136cb0 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6affff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd6b4038490 0x7fd6b403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 0x7fd6c806f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:59.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 10 monclient: get_auth_request con 0x7fd6c811e880 auth_method 0 2024-09-18T12:39:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd6c004d110 con 0x7fd6c8136cb0 2024-09-18T12:39:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c77fe640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd6b4038490 0x7fd6b403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.981+0000 7fd6c77fe640 10 monclient: get_auth_request con 0x7fd6b4038490 auth_method 0 2024-09-18T12:39:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.982+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_done global_id 174265 payload 824 2024-09-18T12:39:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.982+0000 7fd6c7fff640 1 -- 172.21.15.116:0/1226150708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6c811fd00 con 0x7fd6c811e880 2024-09-18T12:39:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6c7fff640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 msgr2=0x7fd6c806f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6c7fff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 0x7fd6c806f200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800d420 tx=0x7fd6b8009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6c77fe640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd6b4038490 0x7fd6b403a940 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009130 tx=0x7fd6bc032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8005cb0 con 0x7fd6c811e880 2024-09-18T12:39:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:59.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 10 monclient: dump: 2024-09-18T12:39:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:59.005 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd6b8005e10 con 0x7fd6c811e880 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 msgr2=0x7fd6c806f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:59.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 0x7fd6c806f200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800d420 tx=0x7fd6b8009040 comp rx=0 tx=0).stop 2024-09-18T12:39:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 msgr2=0x7fd6c806f200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.984+0000 7fd6affff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c811e880 0x7fd6c806f200 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800d420 tx=0x7fd6b8009040 comp rx=0 tx=0).stop 2024-09-18T12:39:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6ce2dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:39:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6ce2dd640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 0x7fd6c8137480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:39:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6ce2dd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:39:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c8065e00 con 0x7fd6c8137080 2024-09-18T12:39:59.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6c7fff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 0x7fd6c8137480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:39:59.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6c7fff640 10 monclient: get_auth_request con 0x7fd6c8137080 auth_method 0 2024-09-18T12:39:59.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:39:59.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:39:59.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.990+0000 7fd6c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:39:59.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.991+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:39:59.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.991+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:39:59.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.991+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:39:59.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.991+0000 7fd6c7fff640 10 monclient(hunting): handle_auth_done global_id 174268 payload 824 2024-09-18T12:39:59.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.991+0000 7fd6c7fff640 1 -- 172.21.15.116:0/1226150708 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6c806f740 con 0x7fd6c8137080 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.993+0000 7fd6c7fff640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 msgr2=0x7fd6c8137480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.993+0000 7fd6c7fff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 0x7fd6c8137480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8016000 tx=0x7fd6b8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.993+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8012d10 con 0x7fd6c8137080 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.993+0000 7fd6affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.993+0000 7fd6affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.993+0000 7fd6affff640 10 monclient: dump: 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:39:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fd6b8012e70 con 0x7fd6c8137080 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6affff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6affff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 msgr2=0x7fd6c8137480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6affff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 0x7fd6c8137480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8016000 tx=0x7fd6b8019000 comp rx=0 tx=0).stop 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6affff640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 msgr2=0x7fd6c8137480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:39:59.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6affff640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6c8137080 0x7fd6c8137480 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8016000 tx=0x7fd6b8019000 comp rx=0 tx=0).stop 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 146, 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:59.013 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:59.014 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:59.016 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:39:59.017 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.018 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:39:59.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:39:59.019 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:59.019 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:59.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:59.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:59.021 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:39:59.022 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:59.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:59.025 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:39:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:39:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:39:59.027 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:39:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd6b4038490 msgr2=0x7fd6b403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd6b4038490 0x7fd6b403a940 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009130 tx=0x7fd6bc032000 comp rx=0 tx=0).stop 2024-09-18T12:39:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 10 monclient: shutdown 2024-09-18T12:39:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:39:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c8136cb0 msgr2=0x7fd6c806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:39:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 --2- 172.21.15.116:0/1226150708 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd6c8136cb0 0x7fd6c806ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009ed0 tx=0x7fd6c000b040 comp rx=0 tx=0).stop 2024-09-18T12:39:59.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6cc87a640 1 -- 172.21.15.116:0/1226150708 reap_dead start 2024-09-18T12:39:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6cc87a640 1 -- 172.21.15.116:0/1226150708 reap_dead start 2024-09-18T12:39:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 shutdown_connections 2024-09-18T12:39:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 >> 172.21.15.116:0/1226150708 conn(0x7fd6c80a2e40 msgr2=0x7fd6c81389c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:39:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 shutdown_connections 2024-09-18T12:39:59.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:39:58.994+0000 7fd6ce2dd640 1 -- 172.21.15.116:0/1226150708 wait complete. 2024-09-18T12:39:59.029 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:39:59.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T12:39:59.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-18T12:39:59.030 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T12:39:59.030 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:39:59.031 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:39:59.031 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:39:59.044 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:39:59.045 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-18T12:39:59.045 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-18T12:39:59.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:40:14.046 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-18T12:40:14.046 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-18T12:40:14.046 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:40:14.047 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:40:14.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.151+0000 7f6ef9b00640 10 monclient: get_monmap_and_config 2024-09-18T12:40:14.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.151+0000 7f6ef9b00640 10 monclient: build_initial_monmap 2024-09-18T12:40:14.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.151+0000 7f6ef9b00640 10 monclient: monmap: 2024-09-18T12:40:14.152 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:40:14.152 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:14.152 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:40:14.152241+0000 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:40:14.152241+0000 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:40:14.153 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 1 Processor -- start 2024-09-18T12:40:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 1 -- start start 2024-09-18T12:40:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 10 monclient: init 2024-09-18T12:40:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T12:40:14.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef4096e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.152+0000 7f6ef9b00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ef409aaa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef9b00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ef40978f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef9b00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef409d4a0 con 0x7f6ef409aaa0 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef9b00640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef4099620 con 0x7f6ef40978f0 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef9b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef9b00640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef4099340 con 0x7f6ef4096e10 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef9b00640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef9b00640 10 monclient(hunting): authenticate will time out at 1641.474066s 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef2ffd640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f6ef409aaa0 legacy=0x7f6ef40971e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef37fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef37fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:43532/0 (socket says 172.21.15.116:43532) 2024-09-18T12:40:14.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 learned_addr learned my addr 172.21.15.116:0/2735800252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:40:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request con 0x7f6ef4096e10 auth_method 0 2024-09-18T12:40:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef3fff640 1 --1- 172.21.15.116:0/2735800252 >> v1:172.21.15.144:6789/0 conn(0x7f6ef40978f0 0x7f6ef4135b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:33806/0 (socket says 172.21.15.116:33806) 2024-09-18T12:40:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.153+0000 7f6ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.155+0000 7f6ef3fff640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.144:6789/0 conn(0x7f6ef40978f0 legacy=0x7f6ef4135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.155+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.155+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.155+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:14.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.155+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:14.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.354+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.116:6789/0 conn(0x7f6ef409aaa0 legacy=0x7f6ef40971e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:40:14.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:14.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request con 0x7f6ef4096e10 auth_method 0 2024-09-18T12:40:14.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:14.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:14.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:14.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:14.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:14.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:14.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.356+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:40:14.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.755+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.116:6789/0 conn(0x7f6ef409aaa0 legacy=0x7f6ef40971e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:40:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request con 0x7f6ef4096e10 auth_method 0 2024-09-18T12:40:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:14.757+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:40:15.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.556+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.116:6789/0 conn(0x7f6ef409aaa0 legacy=0x7f6ef40971e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:40:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.558+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.558+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request con 0x7f6ef4096e10 auth_method 0 2024-09-18T12:40:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.558+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.558+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.558+0000 7f6ef37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.559+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.559+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.559+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:15.559+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:40:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient(hunting): tick 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 monclient(hunting): continuing hunt 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 msgr2=0x7f6ef409a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.144:3300/0 conn(0x7f6ef4096e10 0x7f6ef409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.144:6789/0 conn(0x7f6ef40978f0 legacy=0x7f6ef4135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.116:6789/0 conn(0x7f6ef409aaa0 legacy=0x7f6ef40971e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient: _add_conns ranks=[3,2] 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 0x7f6ee0005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient(hunting): picked mon.noname-a con 0x7f6ee00036e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:40:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient(hunting): picked mon.noname-b con 0x7f6ee00061e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef4099340 con 0x7f6ee00036e0 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0008d70 con 0x7f6ee00061e0 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef17fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 msgr2=0x7f6ee0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 0x7f6ee0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f6ee00061e0 auth_method 0 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.153+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 msgr2=0x7f6ee0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 0x7f6ee0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f6ee00061e0 auth_method 0 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:17.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:17.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:17.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.354+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:40:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 msgr2=0x7f6ee0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 0x7f6ee0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:40:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f6ee00061e0 auth_method 0 2024-09-18T12:40:17.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:40:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:17.755+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:40:18.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 msgr2=0x7f6ee0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:18.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 0x7f6ee0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:40:18.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:40:18.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f6ee00061e0 auth_method 0 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:18.556+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:40:20.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient(hunting): tick 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 monclient(hunting): continuing hunt 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 msgr2=0x7f6ee0008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6ee00061e0 0x7f6ee0008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 msgr2=0x7f6ee0005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee00036e0 0x7f6ee0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee000b3a0 0x7f6ee000d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient(hunting): picked mon.noname-a con 0x7f6ee000b3a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef3fff640 1 -- 172.21.15.116:0/2735800252 reap_dead start 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ef409aaa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:40:20.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ef40978f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef4099900 con 0x7f6ef409aaa0 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef409d4a0 con 0x7f6ef40978f0 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 1 -- 172.21.15.116:0/2735800252 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef4099340 con 0x7f6ee000b3a0 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef17fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.116:6789/0 conn(0x7f6ef409aaa0 legacy=0x7f6ee000dd60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee000b3a0 msgr2=0x7f6ee000d780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.154+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee000b3a0 0x7f6ee000d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef3fff640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.144:6789/0 conn(0x7f6ef40978f0 legacy=0x7f6ee0011490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317252738 0 0) 0x7f6ef409d4a0 con 0x7f6ef40978f0 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient(hunting): my global_id is 174271 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6edc0034e0 con 0x7f6ef40978f0 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276652146 0 0) 0x7f6edc0034e0 con 0x7f6ef40978f0 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee000b3a0 msgr2=0x7f6ee000d780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 --2- 172.21.15.116:0/2735800252 >> v2:172.21.15.116:3300/0 conn(0x7f6ee000b3a0 0x7f6ee000d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.116:6789/0 conn(0x7f6ef409aaa0 legacy=0x7f6ee000dd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:20.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0005d10 con 0x7f6ef40978f0 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: _finish_auth 0 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f6eec0015e0 con 0x7f6ef40978f0 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: dump: 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.161 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 >> v1:172.21.15.144:6789/0 conn(0x7f6ef40978f0 legacy=0x7f6ee0011490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.156+0000 7f6ef27fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6edc0039d0 0x7f6edc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): picked mon.a con 0x7f6edc0039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:40:20.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6edc004780 0x7f6edc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): picked mon.b con 0x7f6edc004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc005230 0x7f6edc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): picked mon.c con 0x7f6edc005230 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef4099340 con 0x7f6edc0039d0 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef37fe640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6edc0039d0 msgr2=0x7f6edc003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc00c1f0 con 0x7f6edc004780 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc00c3b0 con 0x7f6edc005230 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef27fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:40:20.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6edc0039d0 0x7f6edc003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6edc004780 0x7f6edc004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc005230 0x7f6edc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request con 0x7f6edc004780 auth_method 0 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 10 monclient(hunting): get_auth_request con 0x7f6edc005230 auth_method 0 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:40:20.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient(hunting): handle_auth_done global_id 174271 payload 293 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc005230 msgr2=0x7f6edc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc005230 0x7f6edc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6edc0039d0 msgr2=0x7f6edc003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6edc0039d0 0x7f6edc003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 1 monclient: found mon.b 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0034e0 con 0x7f6edc004780 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:40:20.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:40:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef9b00640 5 monclient: authenticate success, global_id 174271 2024-09-18T12:40:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef9b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:40:20.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.157+0000 7f6ef3fff640 1 -- 172.21.15.116:0/2735800252 reap_dead start 2024-09-18T12:40:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.160+0000 7f6ef2ffd640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6edc004780 msgr2=0x7f6edc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.160+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6edc004780 0x7f6edc004bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8005130 tx=0x7f6ee8005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=118616b0ae5f3425 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.160+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee802bce0 con 0x7f6edc004780 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.160+0000 7f6ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.160+0000 7f6ef27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.160+0000 7f6ef27fc640 10 monclient: dump: 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.173 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee802be60 con 0x7f6edc004780 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef27fc640 1 -- 172.21.15.116:0/2735800252 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee800a280 con 0x7f6edc004780 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 10 monclient: get_monmap_and_config success 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 4 set_mon_vals no callback set 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef27fc640 10 monclient: dump: 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.174 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 10 monclient: shutdown 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 1 -- 172.21.15.116:0/2735800252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6edc004780 msgr2=0x7f6edc004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6edc004780 0x7f6edc004bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8005130 tx=0x7f6ee8005040 comp rx=0 tx=0).stop 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 1 -- 172.21.15.116:0/2735800252 shutdown_connections 2024-09-18T12:40:20.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/2735800252 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6edc004780 0x7f6edc004bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 1 -- 172.21.15.116:0/2735800252 >> 172.21.15.116:0/2735800252 conn(0x7f6ef40672b0 msgr2=0x7f6ef40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 1 -- 172.21.15.116:0/2735800252 shutdown_connections 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.161+0000 7f6ef9b00640 1 -- 172.21.15.116:0/2735800252 wait complete. 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.162+0000 7f6ef9b00640 10 monclient: build_initial_monmap 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.162+0000 7f6ef9b00640 10 monclient: monmap: 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:40:20.176 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 Processor -- start 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 -- start start 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient: init 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 0x7f6ef412e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): picked mon.noname-c con 0x7f6edc004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ef412fc00 0x7f6ef411a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef412fc00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ef412ebd0 0x7f6ef411a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef412ebd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef409b050 con 0x7f6ef412fc00 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef406f450 con 0x7f6ef412ebd0 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef411dab0 con 0x7f6edc004780 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef9b00640 10 monclient(hunting): authenticate will time out at 1647.484932s 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef2ffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ef412fc00 msgr2=0x7f6ef411a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.163+0000 7f6ef2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ef412fc00 0x7f6ef411a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 0x7f6ef412e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 0x7f6ef412e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51612/0 (socket says 172.21.15.116:51612) 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 -- 172.21.15.116:0/778468100 learned_addr learned my addr 172.21.15.116:0/778468100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request con 0x7f6edc004780 auth_method 0 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ef412ebd0 0x7f6ef411a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 10 monclient(hunting): get_auth_request con 0x7f6ef412ebd0 auth_method 0 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient(hunting): handle_auth_done global_id 184157 payload 824 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 -- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ef412ebd0 msgr2=0x7f6ef411a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ef412ebd0 0x7f6ef411a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 -- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ef412fc00 msgr2=0x7f6ef411a200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ef412fc00 0x7f6ef411a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 monclient: found mon.noname-c 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:40:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.164+0000 7f6ef37fe640 1 -- 172.21.15.116:0/778468100 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef411bd10 con 0x7f6edc004780 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6ef37fe640 1 -- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 msgr2=0x7f6ef412e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 0x7f6ef412e690 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4013ce0 tx=0x7f6ee4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 1 -- 172.21.15.116:0/778468100 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4013e40 con 0x7f6edc004780 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 10 monclient: dump: 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 10 monclient: _finish_auth 0 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6ef9b00640 5 monclient: authenticate success, global_id 184157 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 1 -- 172.21.15.116:0/778468100 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee400f040 con 0x7f6edc004780 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6ef9b00640 10 monclient: _renew_subs 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 1 -- 172.21.15.116:0/778468100 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4007e50 con 0x7f6edc004780 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6ef9b00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef413d590 con 0x7f6edc004780 2024-09-18T12:40:20.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6edb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.167+0000 7f6edb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.167+0000 7f6edb7fe640 10 monclient: dump: 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.183 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.166+0000 7f6ef1ffb640 4 set_mon_vals no callback set 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.167+0000 7f6ef9b00640 10 monclient: _renew_subs 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.167+0000 7f6ef9b00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.167+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef411c8e0 con 0x7f6edc004780 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.167+0000 7f6edb7fe640 1 -- 172.21.15.116:0/778468100 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6ee4003020 con 0x7f6edc004780 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6edb7fe640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ee003b7f0 0x7f6ee003dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6edb7fe640 1 -- 172.21.15.116:0/778468100 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6ee404c620 con 0x7f6edc004780 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6ef9b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6ef9b00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6ef9b00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef404bf60 con 0x7f6edc004780 2024-09-18T12:40:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ee003b7f0 0x7f6ee003dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.168+0000 7f6ef2ffd640 10 monclient: get_auth_request con 0x7f6ee003b7f0 auth_method 0 2024-09-18T12:40:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.171+0000 7f6ef2ffd640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ee003b7f0 0x7f6ee003dca0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8031000 tx=0x7f6ee8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.173+0000 7f6edb7fe640 1 -- 172.21.15.116:0/778468100 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6ee401eb30 con 0x7f6edc004780 2024-09-18T12:40:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.173+0000 7f6edb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:40:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.173+0000 7f6edb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:40:20.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.389+0000 7f6ef9b00640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:40:20.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.389+0000 7f6ef9b00640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:40:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.390+0000 7f6ef9b00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.390+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6ef40638d0 con 0x7f6edc004780 2024-09-18T12:40:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.390+0000 7f6edb7fe640 1 -- 172.21.15.116:0/778468100 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f6ee402d080 con 0x7f6edc004780 2024-09-18T12:40:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.390+0000 7f6edb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:40:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.390+0000 7f6edb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:40:20.393 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":148,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:40:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ee003b7f0 msgr2=0x7f6ee003dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ee003b7f0 0x7f6ee003dca0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8031000 tx=0x7f6ee8032000 comp rx=0 tx=0).stop 2024-09-18T12:40:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 10 monclient: shutdown 2024-09-18T12:40:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:40:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 msgr2=0x7f6ef412e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 0x7f6ef412e690 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4013ce0 tx=0x7f6ee4009040 comp rx=0 tx=0).stop 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 shutdown_connections 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6ee003b7f0 0x7f6ee003dca0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ef412ebd0 0x7f6ef411a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ef412fc00 0x7f6ef411a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 --2- 172.21.15.116:0/778468100 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6edc004780 0x7f6ef412e690 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 >> 172.21.15.116:0/778468100 conn(0x7f6ef40672b0 msgr2=0x7f6ef40a0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 shutdown_connections 2024-09-18T12:40:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.392+0000 7f6ef9b00640 1 -- 172.21.15.116:0/778468100 wait complete. 2024-09-18T12:40:20.404 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:40:20.404 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:40:20.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:40:20.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.497+0000 7f2c74b14640 10 monclient: get_monmap_and_config 2024-09-18T12:40:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.497+0000 7f2c74b14640 10 monclient: build_initial_monmap 2024-09-18T12:40:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.497+0000 7f2c74b14640 10 monclient: monmap: 2024-09-18T12:40:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:40:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:40:20.498740+0000 2024-09-18T12:40:20.499 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:40:20.498740+0000 2024-09-18T12:40:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:40:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:40:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:40:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:40:20.501 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:40:20.501 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:40:20.501 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.497+0000 7f2c74b14640 1 Processor -- start 2024-09-18T12:40:20.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.497+0000 7f2c74b14640 1 -- start start 2024-09-18T12:40:20.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.497+0000 7f2c74b14640 10 monclient: init 2024-09-18T12:40:20.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:20.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T12:40:20.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c70131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:40:20.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 0x7f2c7009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient(hunting): picked mon.noname-b con 0x7f2c7009e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:20.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c7009f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:40:20.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7009faf0 con 0x7f2c7009f220 2024-09-18T12:40:20.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7009fd30 con 0x7f2c70131ab0 2024-09-18T12:40:20.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70132590 con 0x7f2c7009e910 2024-09-18T12:40:20.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:20.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c74b14640 10 monclient(hunting): authenticate will time out at 1647.820340s 2024-09-18T12:40:20.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c6ed9d640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f2c7009f220 legacy=0x7f2c70138bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2024-09-18T12:40:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 0x7f2c7009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 0x7f2c7009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51620/0 (socket says 172.21.15.116:51620) 2024-09-18T12:40:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3768732578 learned_addr learned my addr 172.21.15.116:0/3768732578 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:40:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): get_auth_request con 0x7f2c7009e910 auth_method 0 2024-09-18T12:40:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_done global_id 184160 payload 274 2024-09-18T12:40:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3768732578 >> v1:172.21.15.144:6789/0 conn(0x7f2c70131ab0 legacy=0x7f2c70131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:40:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3768732578 >> v1:172.21.15.116:6789/0 conn(0x7f2c7009f220 legacy=0x7f2c70138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:40:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 1 monclient: found mon.noname-b 2024-09-18T12:40:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:40:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.498+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3768732578 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c700a1110 con 0x7f2c7009e910 2024-09-18T12:40:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3768732578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 msgr2=0x7f2c7009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c65d9b640 1 --2- 172.21.15.116:0/3768732578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 0x7f2c7009ece0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009940 tx=0x7f2c580091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b1bef62419307d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 1 -- 172.21.15.116:0/3768732578 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58032440 con 0x7f2c7009e910 2024-09-18T12:40:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:40:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: dump: 2024-09-18T12:40:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.533 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.533 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.533 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.533 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.533 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.533 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.534 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.534 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: _finish_auth 0 2024-09-18T12:40:20.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:40:20.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c74b14640 5 monclient: authenticate success, global_id 184160 2024-09-18T12:40:20.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 1 -- 172.21.15.116:0/3768732578 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c580325a0 con 0x7f2c7009e910 2024-09-18T12:40:20.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c74b14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:40:20.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:40:20.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c74b14640 10 monclient: get_monmap_and_config success 2024-09-18T12:40:20.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c74b14640 4 set_mon_vals no callback set 2024-09-18T12:40:20.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 1 -- 172.21.15.116:0/3768732578 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58032940 con 0x7f2c7009e910 2024-09-18T12:40:20.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:40:20.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.501+0000 7f2c6dd9b640 10 monclient: dump: 2024-09-18T12:40:20.536 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.536 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.537 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.537 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.537 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.537 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.537 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.538 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 10 monclient: shutdown 2024-09-18T12:40:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:40:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 1 -- 172.21.15.116:0/3768732578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 msgr2=0x7f2c7009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 1 --2- 172.21.15.116:0/3768732578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 0x7f2c7009ece0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009940 tx=0x7f2c580091d0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 1 -- 172.21.15.116:0/3768732578 shutdown_connections 2024-09-18T12:40:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 1 --2- 172.21.15.116:0/3768732578 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009e910 0x7f2c7009ece0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 1 -- 172.21.15.116:0/3768732578 >> 172.21.15.116:0/3768732578 conn(0x7f2c70067d30 msgr2=0x7f2c700a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:40:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 1 -- 172.21.15.116:0/3768732578 shutdown_connections 2024-09-18T12:40:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 1 -- 172.21.15.116:0/3768732578 wait complete. 2024-09-18T12:40:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 10 monclient: build_initial_monmap 2024-09-18T12:40:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.502+0000 7f2c74b14640 10 monclient: monmap: 2024-09-18T12:40:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:40:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:40:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:40:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:40:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:40:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:40:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:40:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:40:20.542 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 Processor -- start 2024-09-18T12:40:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 -- start start 2024-09-18T12:40:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient: init 2024-09-18T12:40:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:40:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7009e910 0x7f2c701340d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): picked mon.noname-b con 0x7f2c7009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:40:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009f220 0x7f2c70134610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): picked mon.noname-c con 0x7f2c7009f220 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c70131ab0 0x7f2c70134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): picked mon.noname-a con 0x7f2c70131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:40:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70137970 con 0x7f2c70131ab0 2024-09-18T12:40:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7009bde0 con 0x7f2c7009e910 2024-09-18T12:40:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70139970 con 0x7f2c7009f220 2024-09-18T12:40:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c74b14640 10 monclient(hunting): authenticate will time out at 1647.825617s 2024-09-18T12:40:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6ed9d640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c70131ab0 msgr2=0x7f2c70134b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c70131ab0 0x7f2c70134b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009f220 0x7f2c70134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009f220 0x7f2c70134610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51634/0 (socket says 172.21.15.116:51634) 2024-09-18T12:40:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3864062532 learned_addr learned my addr 172.21.15.116:0/3864062532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:40:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 10 monclient(hunting): get_auth_request con 0x7f2c7009f220 auth_method 0 2024-09-18T12:40:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7009e910 0x7f2c701340d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 10 monclient(hunting): get_auth_request con 0x7f2c7009e910 auth_method 0 2024-09-18T12:40:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.503+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 10 monclient(hunting): handle_auth_done global_id 184163 payload 824 2024-09-18T12:40:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7009e910 msgr2=0x7f2c701340d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c7009e910 0x7f2c701340d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c70131ab0 msgr2=0x7f2c70134b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:40:20.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c70131ab0 0x7f2c70134b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:40:20.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 1 monclient: found mon.noname-c 2024-09-18T12:40:20.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:40:20.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.504+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3864062532 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c58009450 con 0x7f2c7009f220 2024-09-18T12:40:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c65d9b640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009f220 msgr2=0x7f2c70134610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c65d9b640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009f220 0x7f2c70134610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009940 tx=0x7f2c58032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58031b10 con 0x7f2c7009f220 2024-09-18T12:40:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:40:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: dump: 2024-09-18T12:40:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: _finish_auth 0 2024-09-18T12:40:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c74b14640 5 monclient: authenticate success, global_id 184163 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c58031c90 con 0x7f2c7009f220 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c74b14640 10 monclient: _renew_subs 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c74b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c70135bd0 con 0x7f2c7009f220 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5803c6c0 con 0x7f2c7009f220 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c66ffd640 10 monclient: dump: 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c6d59a640 4 set_mon_vals no callback set 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c74b14640 10 monclient: _renew_subs 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c74b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.506+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c701367a0 con 0x7f2c7009f220 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c74b14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:40:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c74b14640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 0x7f2c70099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c74b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70063740 con 0x7f2c701385f0 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 0x7f2c70099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2c58044030 con 0x7f2c7009f220 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient: get_auth_request con 0x7f2c701385f0 auth_method 0 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_done global_id 184166 payload 824 2024-09-18T12:40:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c66ffd640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c50038440 0x7f2c5003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6e59c640 1 -- 172.21.15.116:0/3864062532 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c701389c0 con 0x7f2c701385f0 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2c58042070 con 0x7f2c7009f220 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6ed9d640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c50038440 0x7f2c5003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.507+0000 7f2c6ed9d640 10 monclient: get_auth_request con 0x7f2c50038440 auth_method 0 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.509+0000 7f2c6e59c640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 msgr2=0x7f2c70099a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.509+0000 7f2c6e59c640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 0x7f2c70099a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000f780 tx=0x7f2c60007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.509+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6000d8b0 con 0x7f2c701385f0 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.509+0000 7f2c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.509+0000 7f2c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.509+0000 7f2c66ffd640 10 monclient: dump: 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2c6000da10 con 0x7f2c701385f0 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 msgr2=0x7f2c70099a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c66ffd640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 0x7f2c70099a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000f780 tx=0x7f2c60007e10 comp rx=0 tx=0).stop 2024-09-18T12:40:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 msgr2=0x7f2c70099a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c66ffd640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c701385f0 0x7f2c70099a80 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000f780 tx=0x7f2c60007e10 comp rx=0 tx=0).stop 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.510+0000 7f2c6ed9d640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c50038440 0x7f2c5003a8f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2c70136530 tx=0x7f2c68009370 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c74b14640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c74b14640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 0x7f2c7009a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c74b14640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70137b90 con 0x7f2c70099fc0 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 0x7f2c7009a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 10 monclient: get_auth_request con 0x7f2c70099fc0 auth_method 0 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.515+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.516+0000 7f2c6e59c640 10 monclient(hunting): handle_auth_done global_id 184169 payload 824 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.516+0000 7f2c6e59c640 1 -- 172.21.15.116:0/3864062532 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2c7004bf60 con 0x7f2c70099fc0 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c6e59c640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 msgr2=0x7f2c7009a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c6e59c640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 0x7f2c7009a3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000f780 tx=0x7f2c60019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6001d400 con 0x7f2c70099fc0 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 10 monclient: dump: 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f2c6001d560 con 0x7f2c70099fc0 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 msgr2=0x7f2c7009a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 0x7f2c7009a3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000f780 tx=0x7f2c60019000 comp rx=0 tx=0).stop 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 msgr2=0x7f2c7009a3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c66ffd640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c70099fc0 0x7f2c7009a3c0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000f780 tx=0x7f2c60019000 comp rx=0 tx=0).stop 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 148, 2024-09-18T12:40:20.563 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:40:20.564 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:40:20.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:40:20.566 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:40:20.567 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:40:20.568 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:40:20.571 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:40:20.572 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:40:20.573 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:40:20.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:40:20.576 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:40:20.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c50038440 msgr2=0x7f2c5003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c74b14640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2c50038440 0x7f2c5003a8f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2c70136530 tx=0x7f2c68009370 comp rx=0 tx=0).stop 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c74b14640 10 monclient: shutdown 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c74b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009f220 msgr2=0x7f2c70134610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c74b14640 1 --2- 172.21.15.116:0/3864062532 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2c7009f220 0x7f2c70134610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009940 tx=0x7f2c58032000 comp rx=0 tx=0).stop 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.518+0000 7f2c6ed9d640 1 -- 172.21.15.116:0/3864062532 reap_dead start 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.519+0000 7f2c6ed9d640 1 -- 172.21.15.116:0/3864062532 reap_dead start 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.519+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 shutdown_connections 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.519+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 >> 172.21.15.116:0/3864062532 conn(0x7f2c70067d30 msgr2=0x7f2c7009cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.519+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 shutdown_connections 2024-09-18T12:40:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.519+0000 7f2c74b14640 1 -- 172.21.15.116:0/3864062532 wait complete. 2024-09-18T12:40:20.579 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:40:20.579 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:40:20.579 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:40:20.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.671+0000 7f290584f640 10 monclient: get_monmap_and_config 2024-09-18T12:40:20.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.671+0000 7f290584f640 10 monclient: build_initial_monmap 2024-09-18T12:40:20.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.671+0000 7f290584f640 10 monclient: monmap: 2024-09-18T12:40:20.673 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:40:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:40:20.673009+0000 2024-09-18T12:40:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:40:20.673009+0000 2024-09-18T12:40:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:40:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:40:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:40:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:40:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:40:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:40:20.675 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.671+0000 7f290584f640 1 Processor -- start 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.671+0000 7f290584f640 1 -- start start 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.671+0000 7f290584f640 10 monclient: init 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2900131ab0 0x7f2900131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): picked mon.noname-a con 0x7f2900131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f290009e910 0x7f290009ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): picked mon.noname-c con 0x7f290009e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): picked mon.noname-b con 0x7f290009f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290009faf0 con 0x7f2900131ab0 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290009fd30 con 0x7f290009e910 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000a1110 con 0x7f290009f250 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f290584f640 10 monclient(hunting): authenticate will time out at 1647.994495s 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28feffd640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f2900131ab0 msgr2=0x7f2900131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28feffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2900131ab0 0x7f2900131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51678/0 (socket says 172.21.15.116:51678) 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 -- 172.21.15.116:0/2909818521 learned_addr learned my addr 172.21.15.116:0/2909818521 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): get_auth_request con 0x7f290009f250 auth_method 0 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 1 --2- 172.21.15.116:0/2909818521 >> v2:172.21.15.144:3300/0 conn(0x7f290009e910 0x7f290009ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 10 monclient(hunting): get_auth_request con 0x7f290009e910 auth_method 0 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient(hunting): handle_auth_done global_id 184172 payload 274 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 -- 172.21.15.116:0/2909818521 >> v2:172.21.15.144:3300/0 conn(0x7f290009e910 msgr2=0x7f290009ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 --2- 172.21.15.116:0/2909818521 >> v2:172.21.15.144:3300/0 conn(0x7f290009e910 0x7f290009ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 -- 172.21.15.116:0/2909818521 >> v2:172.21.15.116:3300/0 conn(0x7f2900131ab0 msgr2=0x7f2900131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 --2- 172.21.15.116:0/2909818521 >> v2:172.21.15.116:3300/0 conn(0x7f2900131ab0 0x7f2900131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:40:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 monclient: found mon.noname-b 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.672+0000 7f28ff7fe640 1 -- 172.21.15.116:0/2909818521 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000a13f0 con 0x7f290009f250 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.675+0000 7f28ff7fe640 1 -- 172.21.15.116:0/2909818521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 msgr2=0x7f2900068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.675+0000 7f28ff7fe640 1 --2- 172.21.15.116:0/2909818521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900068ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f290009bdb0 tx=0x7f28f8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25272ed8ca4cfd7d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 1 -- 172.21.15.116:0/2909818521 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f8007610 con 0x7f290009f250 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: dump: 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: _finish_auth 0 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 1 -- 172.21.15.116:0/2909818521 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f802d050 con 0x7f290009f250 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 5 monclient: authenticate success, global_id 184172 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:40:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 10 monclient: get_monmap_and_config success 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 1 -- 172.21.15.116:0/2909818521 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f80079a0 con 0x7f290009f250 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 4 set_mon_vals no callback set 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f28fdffb640 10 monclient: dump: 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 10 monclient: shutdown 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 -- 172.21.15.116:0/2909818521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 msgr2=0x7f2900068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 --2- 172.21.15.116:0/2909818521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900068ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f290009bdb0 tx=0x7f28f8009840 comp rx=0 tx=0).stop 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 -- 172.21.15.116:0/2909818521 shutdown_connections 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 --2- 172.21.15.116:0/2909818521 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900068ce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 --2- 172.21.15.116:0/2909818521 >> v2:172.21.15.144:3300/0 conn(0x7f290009e910 0x7f290009ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 --2- 172.21.15.116:0/2909818521 >> v2:172.21.15.116:3300/0 conn(0x7f2900131ab0 0x7f2900131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 -- 172.21.15.116:0/2909818521 >> 172.21.15.116:0/2909818521 conn(0x7f2900067d30 msgr2=0x7f29000a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 -- 172.21.15.116:0/2909818521 shutdown_connections 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.676+0000 7f290584f640 1 -- 172.21.15.116:0/2909818521 wait complete. 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.677+0000 7f290584f640 10 monclient: build_initial_monmap 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.677+0000 7f290584f640 10 monclient: monmap: 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:40:20.693 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.677+0000 7f290584f640 1 Processor -- start 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.677+0000 7f290584f640 1 -- start start 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.677+0000 7f290584f640 10 monclient: init 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f290009e910 0x7f290009ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): picked mon.noname-a con 0x7f290009e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): picked mon.noname-c con 0x7f290009f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131ab0 0x7f290011b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): picked mon.noname-b con 0x7f2900131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:40:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290011e000 con 0x7f290009e910 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290009be50 con 0x7f2900131ab0 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290009db70 con 0x7f290009f250 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f290584f640 10 monclient(hunting): authenticate will time out at 1648.000147s 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28feffd640 1 -- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f290009e910 msgr2=0x7f290009ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28feffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f290009e910 0x7f290009ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900130c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51682/0 (socket says 172.21.15.116:51682) 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 -- 172.21.15.116:0/1585701142 learned_addr learned my addr 172.21.15.116:0/1585701142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): get_auth_request con 0x7f290009f250 auth_method 0 2024-09-18T12:40:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131ab0 0x7f290011b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 10 monclient(hunting): get_auth_request con 0x7f2900131ab0 auth_method 0 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient(hunting): handle_auth_done global_id 184175 payload 824 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131ab0 msgr2=0x7f290011b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131ab0 0x7f290011b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f290009e910 msgr2=0x7f290009ac40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f290009e910 0x7f290009ac40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 monclient: found mon.noname-c 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.678+0000 7f28fe7fc640 1 -- 172.21.15.116:0/1585701142 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8009ae0 con 0x7f290009f250 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28fe7fc640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 msgr2=0x7f2900130c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28fe7fc640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900130c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28e800ea80 tx=0x7f28e80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e800cbc0 con 0x7f290009f250 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28f6ffd640 10 monclient: dump: 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28f6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e800cd20 con 0x7f290009f250 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f290584f640 5 monclient: authenticate success, global_id 184175 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f290584f640 10 monclient: _renew_subs 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.680+0000 7f290584f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290011cdc0 con 0x7f290009f250 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8013680 con 0x7f290009f250 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:40:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 10 monclient: dump: 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 10 monclient: _renew_subs 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28fd7fa640 4 set_mon_vals no callback set 2024-09-18T12:40:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290011ddd0 con 0x7f290009f250 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f28e8013a70 con 0x7f290009f250 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 0x7f290011bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2900063740 con 0x7f2900131140 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28e0038440 0x7f28e003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.681+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f28e804c970 con 0x7f290009f250 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 0x7f290011bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient: get_auth_request con 0x7f2900131140 auth_method 0 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28ff7fe640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28e0038440 0x7f28e003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28ff7fe640 10 monclient: get_auth_request con 0x7f28e0038440 auth_method 0 2024-09-18T12:40:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 10 monclient(hunting): handle_auth_done global_id 174286 payload 824 2024-09-18T12:40:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.682+0000 7f28feffd640 1 -- 172.21.15.116:0/1585701142 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f290009b7c0 con 0x7f2900131140 2024-09-18T12:40:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.684+0000 7f28ff7fe640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28e0038440 0x7f28e003a8f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2900076760 tx=0x7f28f8036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.684+0000 7f28feffd640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 msgr2=0x7f290011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.684+0000 7f28feffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 0x7f290011bbf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b480 tx=0x7f28f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.684+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f00050f0 con 0x7f2900131140 2024-09-18T12:40:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.684+0000 7f28f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.684+0000 7f28f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.684+0000 7f28f6ffd640 10 monclient: dump: 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.685+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f28f0005250 con 0x7f2900131140 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.685+0000 7f28f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.685+0000 7f28f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.685+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 msgr2=0x7f290011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.685+0000 7f28f6ffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 0x7f290011bbf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b480 tx=0x7f28f00091a0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.685+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 msgr2=0x7f290011bbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.685+0000 7f28f6ffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2900131140 0x7f290011bbf0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b480 tx=0x7f28f00091a0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.690+0000 7f290584f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.690+0000 7f290584f640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 0x7f2900096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.690+0000 7f290584f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.690+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290009b180 con 0x7f290011c130 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 0x7f2900096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient: get_auth_request con 0x7f290011c130 auth_method 0 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 10 monclient(hunting): handle_auth_done global_id 174289 payload 824 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.691+0000 7f28feffd640 1 -- 172.21.15.116:0/1585701142 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f290011c530 con 0x7f290011c130 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.693+0000 7f28feffd640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 msgr2=0x7f2900096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.693+0000 7f28feffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 0x7f2900096a60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b480 tx=0x7f28f0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f001b650 con 0x7f290011c130 2024-09-18T12:40:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 10 monclient: dump: 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:40:20.708 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f28f001b7b0 con 0x7f290011c130 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 msgr2=0x7f2900096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 0x7f2900096a60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b480 tx=0x7f28f0015000 comp rx=0 tx=0).stop 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 msgr2=0x7f2900096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.694+0000 7f28f6ffd640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f290011c130 0x7f2900096a60 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b480 tx=0x7f28f0015000 comp rx=0 tx=0).stop 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:40:20.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 148, 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:40:20.710 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:40:20.711 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:40:20.712 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:40:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:40:20.714 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:40:20.715 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:40:20.716 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:40:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:40:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:40:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:40:20.722 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:40:20.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28e0038440 msgr2=0x7f28e003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f28e0038440 0x7f28e003a8f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2900076760 tx=0x7f28f8036000 comp rx=0 tx=0).stop 2024-09-18T12:40:20.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 10 monclient: shutdown 2024-09-18T12:40:20.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:40:20.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 msgr2=0x7f2900130c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:20.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900130c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28e800ea80 tx=0x7f28e80090f0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f28ff7fe640 1 -- 172.21.15.116:0/1585701142 reap_dead start 2024-09-18T12:40:20.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 shutdown_connections 2024-09-18T12:40:20.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 --2- 172.21.15.116:0/1585701142 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f290009f250 0x7f2900130c00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:20.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 >> 172.21.15.116:0/1585701142 conn(0x7f2900067d30 msgr2=0x7f29000a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:40:20.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 shutdown_connections 2024-09-18T12:40:20.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:20.695+0000 7f290584f640 1 -- 172.21.15.116:0/1585701142 wait complete. 2024-09-18T12:40:20.724 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:40:20.724 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:40:40.725 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T12:40:40.725 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T12:40:40.726 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T12:40:40.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T12:40:40.730 INFO:tasks.ceph.mon.a:Started 2024-09-18T12:40:40.730 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-18T12:40:40.731 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-18T12:40:40.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:40:55.731 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-18T12:40:55.732 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-18T12:40:55.732 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:40:55.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:40:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.835+0000 7f83a4299640 10 monclient: get_monmap_and_config 2024-09-18T12:40:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.835+0000 7f83a4299640 10 monclient: build_initial_monmap 2024-09-18T12:40:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.835+0000 7f83a4299640 10 monclient: monmap: 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:40:55.836476+0000 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:40:55.836476+0000 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:40:55.837 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.836+0000 7f83a4299640 1 Processor -- start 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.836+0000 7f83a4299640 1 -- start start 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.836+0000 7f83a4299640 10 monclient: init 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient: _reopen_session rank -1 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient(hunting): picked mon.noname-a con 0x7f839c096e10 addr v2:172.21.15.116:3300/0 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f839c09aaa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient(hunting): picked mon.noname-b con 0x7f839c0978f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:40:55.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c09d4a0 con 0x7f839c09aaa0 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c099620 con 0x7f839c096e10 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c099340 con 0x7f839c0978f0 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a4299640 10 monclient(hunting): authenticate will time out at 1683.158181s 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40078/0 (socket says 172.21.15.116:40078) 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 learned_addr learned my addr 172.21.15.116:0/3471856121 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 10 monclient(hunting): get_auth_request con 0x7f839c0978f0 auth_method 0 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 10 monclient(hunting): get_auth_request con 0x7f839c096e10 auth_method 0 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.837+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:55.839+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v1:172.21.15.116:6789/0 conn(0x7f839c09aaa0 legacy=0x7f839c0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:56.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:56.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:56.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 10 monclient(hunting): get_auth_request con 0x7f839c096e10 auth_method 0 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 10 monclient(hunting): get_auth_request con 0x7f839c0978f0 auth_method 0 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:56.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:40:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.038+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:40:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 10 monclient(hunting): get_auth_request con 0x7f839c096e10 auth_method 0 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 10 monclient(hunting): get_auth_request con 0x7f839c0978f0 auth_method 0 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:40:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:56.439+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:40:57.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:57.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 10 monclient(hunting): get_auth_request con 0x7f839c0978f0 auth_method 0 2024-09-18T12:40:57.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:57.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:57.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:57.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 10 monclient(hunting): get_auth_request con 0x7f839c096e10 auth_method 0 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:40:57.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:57.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:57.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:57.240+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:40:58.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 10 monclient(hunting): tick 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 monclient(hunting): continuing hunt 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 msgr2=0x7f839c135b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f839c0978f0 0x7f839c135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 msgr2=0x7f839c09a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.116:3300/0 conn(0x7f839c096e10 0x7f839c09a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 -- 172.21.15.116:0/3471856121 >> v1:172.21.15.116:6789/0 conn(0x7f839c09aaa0 legacy=0x7f839c0971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 10 monclient: _add_conns ranks=[1,4] 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8388005910 addr v1:172.21.15.144:6789/0 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f8388009650 addr v2:172.21.15.144:3300/0 2024-09-18T12:40:58.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 -- 172.21.15.116:0/3471856121 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c099900 con 0x7f8388005910 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 1 -- 172.21.15.116:0/3471856121 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c099620 con 0x7f8388009650 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f839bfff640 10 monclient(hunting): _renew_subs 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f83a1834640 10 monclient(hunting): get_auth_request con 0x7f8388009650 auth_method 0 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f83a1834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f83a1834640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.837+0000 7f83a1834640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.838+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.838+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.838+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:58.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.838+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:40:58.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:58.840+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> v1:172.21.15.144:6789/0 conn(0x7f8388005910 legacy=0x7f8388007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:40:59.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:59.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 10 monclient(hunting): get_auth_request con 0x7f8388009650 auth_method 0 2024-09-18T12:40:59.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:59.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:59.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:59.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:40:59.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:59.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:59.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.039+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:40:59.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:40:59.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 10 monclient(hunting): get_auth_request con 0x7f8388009650 auth_method 0 2024-09-18T12:40:59.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:40:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:40:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:40:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:40:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:40:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:40:59.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:40:59.440+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 10 monclient(hunting): get_auth_request con 0x7f8388009650 auth_method 0 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.241+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:00.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.242+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:41:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.743+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140330770 0 0) 0x7f839c099900 con 0x7f8388005910 2024-09-18T12:41:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.743+0000 7f83a1033640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.743+0000 7f83a1033640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.743+0000 7f83a1033640 10 monclient(hunting): my global_id is 174292 2024-09-18T12:41:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.743+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83840034e0 con 0x7f8388005910 2024-09-18T12:41:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.743+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 423515400 0 0) 0x7f83840034e0 con 0x7f8388005910 2024-09-18T12:41:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 msgr2=0x7f838800bae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 --2- 172.21.15.116:0/3471856121 >> v2:172.21.15.144:3300/0 conn(0x7f8388009650 0x7f838800bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: _finish_hunting 0 2024-09-18T12:41:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:41:00.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838800d1b0 con 0x7f8388005910 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: _finish_auth 0 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f838c00d480 con 0x7f8388005910 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: dump: 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 >> v1:172.21.15.144:6789/0 conn(0x7f8388005910 legacy=0x7f8388007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 1 -- 172.21.15.116:0/3471856121 reap_dead start 2024-09-18T12:41:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384003ac0 0x7f8384003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): picked mon.a con 0x7f8384003ac0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384004780 0x7f8384004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): picked mon.b con 0x7f8384004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384005330 0x7f838400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): picked mon.c con 0x7f8384005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c099620 con 0x7f8384003ac0 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838400c2f0 con 0x7f8384004780 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838400c4b0 con 0x7f8384005330 2024-09-18T12:41:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1033640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384003ac0 0x7f8384003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384005330 0x7f838400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): get_auth_request con 0x7f8384003ac0 auth_method 0 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 10 monclient(hunting): get_auth_request con 0x7f8384005330 auth_method 0 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384004780 0x7f8384004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 10 monclient(hunting): get_auth_request con 0x7f8384004780 auth_method 0 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2836640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient(hunting): handle_auth_done global_id 174292 payload 293 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384005330 msgr2=0x7f838400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384005330 0x7f838400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384004780 msgr2=0x7f8384004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384004780 0x7f8384004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient: _finish_hunting 0 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 1 monclient: found mon.a 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840034e0 con 0x7f8384003ac0 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 10 monclient: _finish_auth 0 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a2035640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a4299640 5 monclient: authenticate success, global_id 174292 2024-09-18T12:41:00.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a4299640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:00.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.744+0000 7f83a1834640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384004780 0x7f8384004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a2035640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384003ac0 msgr2=0x7f8384003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a2035640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384003ac0 0x7f8384003eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f838c006fd0 tx=0x7f838c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f50cb4ca564f619 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c034410 con 0x7f8384003ac0 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 10 monclient: dump: 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c02e030 con 0x7f8384003ac0 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a4299640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 1 -- 172.21.15.116:0/3471856121 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c034800 con 0x7f8384003ac0 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a4299640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a4299640 4 set_mon_vals no callback set 2024-09-18T12:41:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.747+0000 7f83a1033640 10 monclient: dump: 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 10 monclient: shutdown 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 -- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384003ac0 msgr2=0x7f8384003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384003ac0 0x7f8384003eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f838c006fd0 tx=0x7f838c008040 comp rx=0 tx=0).stop 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 -- 172.21.15.116:0/3471856121 shutdown_connections 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384005330 0x7f838400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384004780 0x7f8384004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 --2- 172.21.15.116:0/3471856121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384003ac0 0x7f8384003eb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 -- 172.21.15.116:0/3471856121 >> 172.21.15.116:0/3471856121 conn(0x7f839c0672b0 msgr2=0x7f839c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 -- 172.21.15.116:0/3471856121 shutdown_connections 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.748+0000 7f83a4299640 1 -- 172.21.15.116:0/3471856121 wait complete. 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.749+0000 7f83a4299640 10 monclient: build_initial_monmap 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.749+0000 7f83a4299640 10 monclient: monmap: 2024-09-18T12:41:00.763 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.749+0000 7f83a4299640 1 Processor -- start 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.749+0000 7f83a4299640 1 -- start start 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.749+0000 7f83a4299640 10 monclient: init 2024-09-18T12:41:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384003ac0 0x7f839c138d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): picked mon.noname-b con 0x7f8384003ac0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 0x7f839c1383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): picked mon.noname-c con 0x7f8384004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 0x7f839c13c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): picked mon.noname-a con 0x7f8384005330 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c11dd00 con 0x7f8384005330 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c11d740 con 0x7f8384003ac0 2024-09-18T12:41:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c11b1d0 con 0x7f8384004780 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a4299640 10 monclient(hunting): authenticate will time out at 1688.071474s 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 0x7f839c1383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 0x7f839c13c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 0x7f839c1383c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40148/0 (socket says 172.21.15.116:40148) 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 0x7f839c13c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37162/0 (socket says 172.21.15.116:37162) 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 -- 172.21.15.116:0/1409151499 learned_addr learned my addr 172.21.15.116:0/1409151499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): get_auth_request con 0x7f8384004780 auth_method 0 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 10 monclient(hunting): get_auth_request con 0x7f8384005330 auth_method 0 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 1 -- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 msgr2=0x7f839c13c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 1 -- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 msgr2=0x7f839c13c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 0x7f839c13c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2035640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384003ac0 0x7f839c138d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2836640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 0x7f839c13c280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:00.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2035640 10 monclient(hunting): get_auth_request con 0x7f8384003ac0 auth_method 0 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2035640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a2035640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient(hunting): handle_auth_done global_id 184178 payload 824 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 -- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384003ac0 msgr2=0x7f839c138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384003ac0 0x7f839c138d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 -- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 msgr2=0x7f839c13c280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 0x7f839c13c280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient: _finish_hunting 0 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 monclient: found mon.noname-c 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:41:00.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.750+0000 7f83a1834640 1 -- 172.21.15.116:0/1409151499 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c11d460 con 0x7f8384004780 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a1834640 1 -- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 msgr2=0x7f839c1383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a1834640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 0x7f839c1383c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83900018a0 tx=0x7f8390009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 1 -- 172.21.15.116:0/1409151499 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390016d60 con 0x7f8384004780 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: dump: 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 1 -- 172.21.15.116:0/1409151499 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390016ec0 con 0x7f8384004780 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 1 -- 172.21.15.116:0/1409151499 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390017680 con 0x7f8384004780 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f839a7fc640 10 monclient: dump: 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a4299640 5 monclient: authenticate success, global_id 184178 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a4299640 10 monclient: _renew_subs 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a4299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c11da20 con 0x7f8384004780 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a0832640 4 set_mon_vals no callback set 2024-09-18T12:41:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a4299640 10 monclient: _renew_subs 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a4299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.753+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c138b00 con 0x7f8384004780 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f839a7fc640 1 -- 172.21.15.116:0/1409151499 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8390017a30 con 0x7f8384004780 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f839a7fc640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f838803b0e0 0x7f838803d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f839a7fc640 1 -- 172.21.15.116:0/1409151499 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f839004ce70 con 0x7f8384004780 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f83a4299640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f83a4299640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f83a4299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c138b00 con 0x7f8384004780 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f83a2035640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f838803b0e0 0x7f838803d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.754+0000 7f83a2035640 10 monclient: get_auth_request con 0x7f838803b0e0 auth_method 0 2024-09-18T12:41:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.757+0000 7f83a2035640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f838803b0e0 0x7f838803d590 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f838c006ea0 tx=0x7f838c039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.759+0000 7f839a7fc640 1 -- 172.21.15.116:0/1409151499 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f839c138b00 con 0x7f8384004780 2024-09-18T12:41:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.759+0000 7f839a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.759+0000 7f839a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.979+0000 7f83a4299640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.979+0000 7f83a4299640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.979+0000 7f83a4299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.979+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f839c138b00 con 0x7f8384004780 2024-09-18T12:41:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.979+0000 7f839a7fc640 1 -- 172.21.15.116:0/1409151499 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f839c138b00 con 0x7f8384004780 2024-09-18T12:41:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.979+0000 7f839a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.979+0000 7f839a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:00.981 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":152,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f838803b0e0 msgr2=0x7f838803d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f838803b0e0 0x7f838803d590 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f838c006ea0 tx=0x7f838c039000 comp rx=0 tx=0).stop 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 10 monclient: shutdown 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 msgr2=0x7f839c1383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 0x7f839c1383c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83900018a0 tx=0x7f8390009040 comp rx=0 tx=0).stop 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 shutdown_connections 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f838803b0e0 0x7f838803d590 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384005330 0x7f839c13c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8384004780 0x7f839c1383c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 --2- 172.21.15.116:0/1409151499 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8384003ac0 0x7f839c138d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 >> 172.21.15.116:0/1409151499 conn(0x7f839c0672b0 msgr2=0x7f839c09f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 shutdown_connections 2024-09-18T12:41:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:00.981+0000 7f83a4299640 1 -- 172.21.15.116:0/1409151499 wait complete. 2024-09-18T12:41:03.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:04.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.100+0000 7f5f1a15d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.100+0000 7f5f1a15d640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.100+0000 7f5f1a15d640 10 monclient: monmap: 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:04.100817+0000 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:04.100817+0000 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:04.101 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:04.102 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:04.102 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.100+0000 7f5f1a15d640 1 Processor -- start 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.100+0000 7f5f1a15d640 1 -- start start 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.100+0000 7f5f1a15d640 10 monclient: init 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f14131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5f1409e910 0x7f5f1409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1409e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f1409f250 0x7f5f14138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1409f250 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1409faf0 con 0x7f5f14131ab0 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f14132590 con 0x7f5f1409f250 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f1409fd30 con 0x7f5f1409e910 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f1a15d640 10 monclient(hunting): authenticate will time out at 1691.422465s 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f1409f250 0x7f5f14138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f137fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5f14131ab0 0x7f5f14131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33492/0 (socket says 172.21.15.116:33492) 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f137fe640 1 -- 172.21.15.116:0/3884260384 learned_addr learned my addr 172.21.15.116:0/3884260384 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): get_auth_request con 0x7f5f1409f250 auth_method 0 2024-09-18T12:41:04.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f12ffd640 1 --2- 172.21.15.116:0/3884260384 >> v2:172.21.15.144:3300/0 conn(0x7f5f1409e910 0x7f5f1409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f12ffd640 10 monclient(hunting): get_auth_request con 0x7f5f1409e910 auth_method 0 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient(hunting): handle_auth_done global_id 194097 payload 274 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 1 -- 172.21.15.116:0/3884260384 >> v2:172.21.15.144:3300/0 conn(0x7f5f1409e910 msgr2=0x7f5f1409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 1 --2- 172.21.15.116:0/3884260384 >> v2:172.21.15.144:3300/0 conn(0x7f5f1409e910 0x7f5f1409ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 1 -- 172.21.15.116:0/3884260384 >> v1:172.21.15.116:6789/0 conn(0x7f5f14131ab0 legacy=0x7f5f14131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 1 monclient: found mon.noname-a 2024-09-18T12:41:04.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.101+0000 7f5f13fff640 1 -- 172.21.15.116:0/3884260384 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f140a1110 con 0x7f5f1409f250 2024-09-18T12:41:04.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.104+0000 7f5f13fff640 1 -- 172.21.15.116:0/3884260384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 msgr2=0x7f5f14138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.104+0000 7f5f13fff640 1 --2- 172.21.15.116:0/3884260384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14138bc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5f1409bdb0 tx=0x7f5f0c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9b214e9ae4236b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 1 -- 172.21.15.116:0/3884260384 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c02adf0 con 0x7f5f1409f250 2024-09-18T12:41:04.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: dump: 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 1 -- 172.21.15.116:0/3884260384 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0c032440 con 0x7f5f1409f250 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 5 monclient: authenticate success, global_id 194097 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 1 -- 172.21.15.116:0/3884260384 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c0327e0 con 0x7f5f1409f250 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 4 set_mon_vals no callback set 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f127fc640 10 monclient: dump: 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.114 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 10 monclient: shutdown 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 -- 172.21.15.116:0/3884260384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 msgr2=0x7f5f14138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/3884260384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14138bc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5f1409bdb0 tx=0x7f5f0c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 -- 172.21.15.116:0/3884260384 shutdown_connections 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/3884260384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14138bc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/3884260384 >> v2:172.21.15.144:3300/0 conn(0x7f5f1409e910 0x7f5f1409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 -- 172.21.15.116:0/3884260384 >> 172.21.15.116:0/3884260384 conn(0x7f5f14067d30 msgr2=0x7f5f140a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 -- 172.21.15.116:0/3884260384 shutdown_connections 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.106+0000 7f5f1a15d640 1 -- 172.21.15.116:0/3884260384 wait complete. 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.107+0000 7f5f1a15d640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.107+0000 7f5f1a15d640 10 monclient: monmap: 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:04.116 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.107+0000 7f5f1a15d640 1 Processor -- start 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.107+0000 7f5f1a15d640 1 -- start start 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.107+0000 7f5f1a15d640 10 monclient: init 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f1409e910 0x7f5f14130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1409e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1409f250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f14131ab0 0x7f5f14096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f14131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1409bde0 con 0x7f5f1409f250 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140a3960 con 0x7f5f14131ab0 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1409a330 con 0x7f5f1409e910 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f1a15d640 10 monclient(hunting): authenticate will time out at 1691.429301s 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37198/0 (socket says 172.21.15.116:37198) 2024-09-18T12:41:04.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f1409e910 0x7f5f14130fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 -- 172.21.15.116:0/4025371382 learned_addr learned my addr 172.21.15.116:0/4025371382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): get_auth_request con 0x7f5f1409f250 auth_method 0 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 10 monclient(hunting): get_auth_request con 0x7f5f1409e910 auth_method 0 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f14131ab0 0x7f5f14096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 10 monclient(hunting): get_auth_request con 0x7f5f14131ab0 auth_method 0 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient(hunting): handle_auth_done global_id 194100 payload 824 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 -- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f1409e910 msgr2=0x7f5f14130fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f1409e910 0x7f5f14130fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 -- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f14131ab0 msgr2=0x7f5f14096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f14131ab0 0x7f5f14096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 monclient: found mon.noname-a 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f12ffd640 1 -- 172.21.15.116:0/4025371382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c009390 con 0x7f5f1409f250 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.108+0000 7f5f137fe640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f1409e910 0x7f5f14130fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f12ffd640 1 -- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 msgr2=0x7f5f14096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f12ffd640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14096a60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00d890 tx=0x7f5efc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/4025371382 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc00de20 con 0x7f5f1409f250 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:04.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: dump: 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/4025371382 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc00ba60 con 0x7f5f1409f250 2024-09-18T12:41:04.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f1a15d640 5 monclient: authenticate success, global_id 194100 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/4025371382 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc00be00 con 0x7f5f1409f250 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f0b7fe640 10 monclient: dump: 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.123 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f1a15d640 10 monclient: _renew_subs 2024-09-18T12:41:04.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f11ffb640 4 set_mon_vals no callback set 2024-09-18T12:41:04.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f1a15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f14099670 con 0x7f5f1409f250 2024-09-18T12:41:04.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f1a15d640 10 monclient: _renew_subs 2024-09-18T12:41:04.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f1a15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.111+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1409aa50 con 0x7f5f1409f250 2024-09-18T12:41:04.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/4025371382 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5efc014640 con 0x7f5f1409f250 2024-09-18T12:41:04.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f1a15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f1a15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f1a15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f0b7fe640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ef4038490 0x7f5ef403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f14099aa0 con 0x7f5f1409f250 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/4025371382 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5efc04c930 con 0x7f5f1409f250 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f137fe640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ef4038490 0x7f5ef403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.112+0000 7f5f137fe640 10 monclient: get_auth_request con 0x7f5ef4038490 auth_method 0 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.115+0000 7f5f137fe640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ef4038490 0x7f5ef403a940 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5f04005bc0 tx=0x7f5f04009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.117+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/4025371382 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5efc01eb40 con 0x7f5f1409f250 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.117+0000 7f5f0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.117+0000 7f5f0b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.333+0000 7f5f1a15d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.333+0000 7f5f1a15d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.333+0000 7f5f1a15d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.333+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5f1404bf60 con 0x7f5f1409f250 2024-09-18T12:41:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.333+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/4025371382 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f5efc01dbb0 con 0x7f5f1409f250 2024-09-18T12:41:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.333+0000 7f5f0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.333+0000 7f5f0b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:04.335 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:04.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ef4038490 msgr2=0x7f5ef403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ef4038490 0x7f5ef403a940 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5f04005bc0 tx=0x7f5f04009040 comp rx=0 tx=0).stop 2024-09-18T12:41:04.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 10 monclient: shutdown 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 msgr2=0x7f5f14096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14096a60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00d890 tx=0x7f5efc008040 comp rx=0 tx=0).stop 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 shutdown_connections 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5ef4038490 0x7f5ef403a940 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5f14131ab0 0x7f5f14096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f1409f250 0x7f5f14096a60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 --2- 172.21.15.116:0/4025371382 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5f1409e910 0x7f5f14130fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 >> 172.21.15.116:0/4025371382 conn(0x7f5f14067d30 msgr2=0x7f5f140a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 shutdown_connections 2024-09-18T12:41:04.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.335+0000 7f5f1a15d640 1 -- 172.21.15.116:0/4025371382 wait complete. 2024-09-18T12:41:04.346 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:41:04.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:41:04.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:41:04.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.439+0000 7f8533f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:04.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.439+0000 7f8533f9d640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.439+0000 7f8533f9d640 10 monclient: monmap: 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:04.440292+0000 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:04.440292+0000 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 1 Processor -- start 2024-09-18T12:41:04.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 1 -- start start 2024-09-18T12:41:04.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient: init 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f852c09a510 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f852c097370 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f852c097770 0x7f852c1380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f852c097770 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852c09cb00 con 0x7f852c097370 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852c09a8e0 con 0x7f852c09a510 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852c06ed80 con 0x7f852c097770 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.440+0000 7f8533f9d640 10 monclient(hunting): authenticate will time out at 1691.761907s 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f8531538640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f852c097370 0x7f852c1349a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33496/0 (socket says 172.21.15.116:33496) 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f8531538640 1 -- 172.21.15.116:0/820300389 learned_addr learned my addr 172.21.15.116:0/820300389 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 1 --2- 172.21.15.116:0/820300389 >> v2:172.21.15.144:3300/0 conn(0x7f852c097770 0x7f852c1380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): get_auth_request con 0x7f852c097770 auth_method 0 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient(hunting): handle_auth_done global_id 174304 payload 274 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 1 -- 172.21.15.116:0/820300389 >> v1:172.21.15.144:6789/0 conn(0x7f852c09a510 legacy=0x7f852c096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 1 -- 172.21.15.116:0/820300389 >> v1:172.21.15.116:6789/0 conn(0x7f852c097370 legacy=0x7f852c1349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 1 monclient: found mon.noname-c 2024-09-18T12:41:04.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:41:04.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.441+0000 7f853253a640 1 -- 172.21.15.116:0/820300389 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c09c820 con 0x7f852c097770 2024-09-18T12:41:04.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f853253a640 1 -- 172.21.15.116:0/820300389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c097770 msgr2=0x7f852c1380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f853253a640 1 --2- 172.21.15.116:0/820300389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c097770 0x7f852c1380d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8524009880 tx=0x7f85240090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f03bc04a04cb2fa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 1 -- 172.21.15.116:0/820300389 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524032440 con 0x7f852c097770 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: dump: 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.453 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 1 -- 172.21.15.116:0/820300389 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85240325a0 con 0x7f852c097770 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8533f9d640 5 monclient: authenticate success, global_id 174304 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8533f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8533f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8533f9d640 4 set_mon_vals no callback set 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 1 -- 172.21.15.116:0/820300389 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524032940 con 0x7f852c097770 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:04.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.444+0000 7f8530d37640 10 monclient: dump: 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 10 monclient: shutdown 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 1 -- 172.21.15.116:0/820300389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c097770 msgr2=0x7f852c1380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 1 --2- 172.21.15.116:0/820300389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c097770 0x7f852c1380d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8524009880 tx=0x7f85240090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 1 -- 172.21.15.116:0/820300389 shutdown_connections 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 1 --2- 172.21.15.116:0/820300389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c097770 0x7f852c1380d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 1 -- 172.21.15.116:0/820300389 >> 172.21.15.116:0/820300389 conn(0x7f852c0672b0 msgr2=0x7f852c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 1 -- 172.21.15.116:0/820300389 shutdown_connections 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 1 -- 172.21.15.116:0/820300389 wait complete. 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.445+0000 7f8533f9d640 10 monclient: monmap: 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.456 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 Processor -- start 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 -- start start 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient: init 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c097370 0x7f852c12fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f852c097370 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f852c097770 0x7f852c1300d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f852c097770 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c09a510 0x7f852c140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f852c09a510 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c09c370 con 0x7f852c097370 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c0997f0 con 0x7f852c09a510 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1202a0 con 0x7f852c097770 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8533f9d640 10 monclient(hunting): authenticate will time out at 1691.767799s 2024-09-18T12:41:04.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c097370 0x7f852c12fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531538640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f852c097770 0x7f852c1300d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c097370 0x7f852c12fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37210/0 (socket says 172.21.15.116:37210) 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 1 -- 172.21.15.116:0/1941726592 learned_addr learned my addr 172.21.15.116:0/1941726592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 10 monclient(hunting): get_auth_request con 0x7f852c097370 auth_method 0 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531538640 10 monclient(hunting): get_auth_request con 0x7f852c097770 auth_method 0 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.446+0000 7f8531d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f853253a640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c09a510 0x7f852c140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f853253a640 10 monclient(hunting): get_auth_request con 0x7f852c09a510 auth_method 0 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f853253a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f853253a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f853253a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 10 monclient(hunting): handle_auth_done global_id 194103 payload 824 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f852c097770 msgr2=0x7f852c1300d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f852c097770 0x7f852c1300d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c09a510 msgr2=0x7f852c140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c09a510 0x7f852c140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 1 monclient: found mon.noname-a 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531d39640 1 -- 172.21.15.116:0/1941726592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524009390 con 0x7f852c097370 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f8531538640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f852c097770 0x7f852c1300d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.447+0000 7f853253a640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f852c09a510 0x7f852c140e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8531d39640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c097370 msgr2=0x7f852c12fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8531d39640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c097370 0x7f852c12fb90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f852800c8b0 tx=0x7f85280026c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852800cea0 con 0x7f852c097370 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:04.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: dump: 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8533f9d640 5 monclient: authenticate success, global_id 194103 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8533f9d640 10 monclient: _renew_subs 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8533f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c11ffc0 con 0x7f852c097370 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852800aa00 con 0x7f852c097370 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852800adc0 con 0x7f852c097370 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8521ffb640 10 monclient: dump: 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.463 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8523fff640 4 set_mon_vals no callback set 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8533f9d640 10 monclient: _renew_subs 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8533f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.449+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c141340 con 0x7f852c097370 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8533f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8533f9d640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 0x7f852c098380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8533f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c11fe80 con 0x7f852c141530 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8528012630 con 0x7f852c097370 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 0x7f852c098380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient: get_auth_request con 0x7f852c141530 auth_method 0 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8521ffb640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8510038490 0x7f851003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f852804c700 con 0x7f852c097370 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.450+0000 7f8531538640 10 monclient(hunting): handle_auth_done global_id 194106 payload 824 2024-09-18T12:41:04.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.451+0000 7f8531538640 1 -- 172.21.15.116:0/1941726592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f852c04bf60 con 0x7f852c141530 2024-09-18T12:41:04.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.451+0000 7f853253a640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8510038490 0x7f851003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.451+0000 7f853253a640 10 monclient: get_auth_request con 0x7f8510038490 auth_method 0 2024-09-18T12:41:04.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8531538640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 msgr2=0x7f852c098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8531538640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 0x7f852c098380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f852c141e50 tx=0x7f851c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c00fcc0 con 0x7f852c141530 2024-09-18T12:41:04.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:04.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 10 monclient: dump: 2024-09-18T12:41:04.468 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f851c00fe20 con 0x7f852c141530 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 msgr2=0x7f852c098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 0x7f852c098380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f852c141e50 tx=0x7f851c007f00 comp rx=0 tx=0).stop 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 msgr2=0x7f852c098380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f8521ffb640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c141530 0x7f852c098380 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f852c141e50 tx=0x7f851c007f00 comp rx=0 tx=0).stop 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.453+0000 7f853253a640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8510038490 0x7f851003a940 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8524031000 tx=0x7f8524032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8533f9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8533f9d640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 0x7f852c098cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8533f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c065f20 con 0x7f852c0988c0 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 0x7f852c098cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient: get_auth_request con 0x7f852c0988c0 auth_method 0 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 10 monclient(hunting): handle_auth_done global_id 194109 payload 824 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.459+0000 7f8531538640 1 -- 172.21.15.116:0/1941726592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f852c099200 con 0x7f852c0988c0 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.461+0000 7f8531538640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 msgr2=0x7f852c098cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.461+0000 7f8531538640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 0x7f852c098cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f851c017000 tx=0x7f851c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c021880 con 0x7f852c0988c0 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 10 monclient: dump: 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.471 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f851c01c3e0 con 0x7f852c0988c0 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 msgr2=0x7f852c098cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 0x7f852c098cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f851c017000 tx=0x7f851c018000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 msgr2=0x7f852c098cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:04.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8521ffb640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c0988c0 0x7f852c098cc0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f851c017000 tx=0x7f851c018000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 156, 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:04.473 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.475 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.477 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.478 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:04.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.480 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.482 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.483 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.484 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.484 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:41:04.484 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.484 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.484 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:04.484 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8510038490 msgr2=0x7f851003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8510038490 0x7f851003a940 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8524031000 tx=0x7f8524032000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 10 monclient: shutdown 2024-09-18T12:41:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c097370 msgr2=0x7f852c12fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 --2- 172.21.15.116:0/1941726592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c097370 0x7f852c12fb90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f852800c8b0 tx=0x7f85280026c0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f853253a640 1 -- 172.21.15.116:0/1941726592 reap_dead start 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f853253a640 1 -- 172.21.15.116:0/1941726592 reap_dead start 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 shutdown_connections 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 >> 172.21.15.116:0/1941726592 conn(0x7f852c0672b0 msgr2=0x7f852c06ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 shutdown_connections 2024-09-18T12:41:04.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.462+0000 7f8533f9d640 1 -- 172.21.15.116:0/1941726592 wait complete. 2024-09-18T12:41:04.486 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:04.486 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:41:04.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:41:04.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.580+0000 7f4840e0f640 10 monclient: get_monmap_and_config 2024-09-18T12:41:04.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.580+0000 7f4840e0f640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.580+0000 7f4840e0f640 10 monclient: monmap: 2024-09-18T12:41:04.581 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.581 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.581 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:04.581328+0000 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:04.581328+0000 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:04.582 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 1 Processor -- start 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 1 -- start start 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient: init 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f483c09a510 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f483c06e920 0x7f483c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f483c06e920 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c136d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f483c1349a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c09cb70 con 0x7f483c09a510 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c09b120 con 0x7f483c06e920 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c09a8e0 con 0x7f483c1349a0 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4840e0f640 10 monclient(hunting): authenticate will time out at 1691.902893s 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.581+0000 7f4839d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f483c06e920 0x7f483c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f483c06e920 0x7f483c06ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37242/0 (socket says 172.21.15.116:37242) 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 1 -- 172.21.15.116:0/1161267685 learned_addr learned my addr 172.21.15.116:0/1161267685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 10 monclient(hunting): get_auth_request con 0x7f483c06e920 auth_method 0 2024-09-18T12:41:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): get_auth_request con 0x7f483c1349a0 auth_method 0 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient(hunting): handle_auth_done global_id 184187 payload 274 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 1 -- 172.21.15.116:0/1161267685 >> v2:172.21.15.116:3300/0 conn(0x7f483c06e920 msgr2=0x7f483c06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 1 --2- 172.21.15.116:0/1161267685 >> v2:172.21.15.116:3300/0 conn(0x7f483c06e920 0x7f483c06ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 1 -- 172.21.15.116:0/1161267685 >> v1:172.21.15.144:6789/0 conn(0x7f483c09a510 legacy=0x7f483c12fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 1 monclient: found mon.noname-b 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f483ad9d640 1 -- 172.21.15.116:0/1161267685 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c09bb40 con 0x7f483c1349a0 2024-09-18T12:41:04.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.582+0000 7f4839d9b640 1 --2- 172.21.15.116:0/1161267685 >> v2:172.21.15.116:3300/0 conn(0x7f483c06e920 0x7f483c06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483ad9d640 1 -- 172.21.15.116:0/1161267685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 msgr2=0x7f483c136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483ad9d640 1 --2- 172.21.15.116:0/1161267685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c136d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4834009e60 tx=0x7f4834009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fc27c6bb3e5aaa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 1 -- 172.21.15.116:0/1161267685 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834007760 con 0x7f483c1349a0 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: dump: 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.595 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 5 monclient: authenticate success, global_id 184187 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 1 -- 172.21.15.116:0/1161267685 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483402d050 con 0x7f483c1349a0 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 1 -- 172.21.15.116:0/1161267685 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834007af0 con 0x7f483c1349a0 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 4 set_mon_vals no callback set 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f483959a640 10 monclient: dump: 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 10 monclient: shutdown 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 -- 172.21.15.116:0/1161267685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 msgr2=0x7f483c136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 --2- 172.21.15.116:0/1161267685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c136d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4834009e60 tx=0x7f4834009780 comp rx=0 tx=0).stop 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 -- 172.21.15.116:0/1161267685 shutdown_connections 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 --2- 172.21.15.116:0/1161267685 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c136d80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 --2- 172.21.15.116:0/1161267685 >> v2:172.21.15.116:3300/0 conn(0x7f483c06e920 0x7f483c06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 -- 172.21.15.116:0/1161267685 >> 172.21.15.116:0/1161267685 conn(0x7f483c0672b0 msgr2=0x7f483c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 -- 172.21.15.116:0/1161267685 shutdown_connections 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.585+0000 7f4840e0f640 1 -- 172.21.15.116:0/1161267685 wait complete. 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.586+0000 7f4840e0f640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.586+0000 7f4840e0f640 10 monclient: monmap: 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.586+0000 7f4840e0f640 1 Processor -- start 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.586+0000 7f4840e0f640 1 -- start start 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.586+0000 7f4840e0f640 10 monclient: init 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:41:04.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c06e920 0x7f483c06be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f483c06e920 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483c09a510 0x7f483c06c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f483c09a510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f483c1349a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c118250 con 0x7f483c09a510 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c06e0e0 con 0x7f483c06e920 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c118730 con 0x7f483c1349a0 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4840e0f640 10 monclient(hunting): authenticate will time out at 1691.908099s 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483c09a510 0x7f483c06c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483c09a510 0x7f483c06c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37252/0 (socket says 172.21.15.116:37252) 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40194/0 (socket says 172.21.15.116:40194) 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 1 -- 172.21.15.116:0/3700226285 learned_addr learned my addr 172.21.15.116:0/3700226285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 10 monclient(hunting): get_auth_request con 0x7f483c09a510 auth_method 0 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): get_auth_request con 0x7f483c1349a0 auth_method 0 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c06e920 0x7f483c06be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 10 monclient(hunting): get_auth_request con 0x7f483c06e920 auth_method 0 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient(hunting): handle_auth_done global_id 184190 payload 824 2024-09-18T12:41:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c06e920 msgr2=0x7f483c06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f483c06e920 0x7f483c06be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483c09a510 msgr2=0x7f483c06c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483c09a510 0x7f483c06c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 monclient: found mon.noname-c 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f483ad9d640 1 -- 172.21.15.116:0/3700226285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4830009450 con 0x7f483c1349a0 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.587+0000 7f4839d9b640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f483c09a510 0x7f483c06c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f483ad9d640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 msgr2=0x7f483c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f483ad9d640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c06c910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4834031000 tx=0x7f4834037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834042070 con 0x7f483c1349a0 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: dump: 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483403dd80 con 0x7f483c1349a0 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f4840e0f640 5 monclient: authenticate success, global_id 184190 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f4840e0f640 10 monclient: _renew_subs 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834048690 con 0x7f483c1349a0 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f4840e0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4834009a20 con 0x7f483c1349a0 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f482a7fc640 10 monclient: dump: 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.589+0000 7f4838d99640 4 set_mon_vals no callback set 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f4840e0f640 10 monclient: _renew_subs 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f4840e0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c06f5f0 con 0x7f483c1349a0 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f4840e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483ad9d640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c06c910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4834031000 tx=0x7f4834037000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f4834070f20 osd_map(267..267 src has 242..267) v4 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f4840e0f640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 0x7f483c06ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f4840e0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c065f20 con 0x7f483c02aa40 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4834046070 con 0x7f483c1349a0 2024-09-18T12:41:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 0x7f483c06ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 10 monclient: get_auth_request con 0x7f483c02aa40 auth_method 0 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.590+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.591+0000 7f482a7fc640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f481c038490 0x7f481c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.591+0000 7f483a59c640 10 monclient(hunting): handle_auth_done global_id 184193 payload 824 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.591+0000 7f483a59c640 1 -- 172.21.15.116:0/3700226285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f483c02ae10 con 0x7f483c02aa40 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.591+0000 7f4839d9b640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f481c038490 0x7f481c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.591+0000 7f4839d9b640 10 monclient: get_auth_request con 0x7f481c038490 auth_method 0 2024-09-18T12:41:04.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f483a59c640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 msgr2=0x7f483c06ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f483a59c640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 0x7f483c06ce50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f482c00f780 tx=0x7f482c007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c00d8b0 con 0x7f483c02aa40 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 10 monclient: dump: 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f482c00da10 con 0x7f483c02aa40 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 msgr2=0x7f483c06ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 0x7f483c06ce50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f482c00f780 tx=0x7f482c007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 msgr2=0x7f483c06ce50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f482a7fc640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c02aa40 0x7f483c06ce50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.593+0000 7f4839d9b640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f481c038490 0x7f481c03a940 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4830009fd0 tx=0x7f48300091d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f4840e0f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f4840e0f640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 0x7f483c05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f4840e0f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c09a310 con 0x7f483c05e310 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 0x7f483c05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient: get_auth_request con 0x7f483c05e310 auth_method 0 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 10 monclient(hunting): handle_auth_done global_id 184196 payload 824 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.599+0000 7f483a59c640 1 -- 172.21.15.116:0/3700226285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f483c09fc30 con 0x7f483c05e310 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f483a59c640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 msgr2=0x7f483c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f483a59c640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 0x7f483c05e710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f482c00f780 tx=0x7f482c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c007240 con 0x7f483c05e310 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 10 monclient: dump: 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f482c00fd20 con 0x7f483c05e310 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 msgr2=0x7f483c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 0x7f483c05e710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f482c00f780 tx=0x7f482c019000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 msgr2=0x7f483c05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:04.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c05e310 0x7f483c05e710 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f482c00f780 tx=0x7f482c019000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 156, 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:04.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.624 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.625 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.627 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.628 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:04.629 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.630 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.630 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:04.630 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f481c038490 msgr2=0x7f481c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f481c038490 0x7f481c03a940 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4830009fd0 tx=0x7f48300091d0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 10 monclient: shutdown 2024-09-18T12:41:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 msgr2=0x7f483c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 --2- 172.21.15.116:0/3700226285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f483c1349a0 0x7f483c06c910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4834031000 tx=0x7f4834037000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f483ad9d640 1 -- 172.21.15.116:0/3700226285 reap_dead start 2024-09-18T12:41:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f483ad9d640 1 -- 172.21.15.116:0/3700226285 reap_dead start 2024-09-18T12:41:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f482a7fc640 1 -- 172.21.15.116:0/3700226285 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7f4834070f20 con 0x7f483c1349a0 2024-09-18T12:41:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 shutdown_connections 2024-09-18T12:41:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 >> 172.21.15.116:0/3700226285 conn(0x7f483c0672b0 msgr2=0x7f483c134e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 shutdown_connections 2024-09-18T12:41:04.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.602+0000 7f4840e0f640 1 -- 172.21.15.116:0/3700226285 wait complete. 2024-09-18T12:41:04.632 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:04.632 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:41:04.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:41:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.725+0000 7f055ab5d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.725+0000 7f055ab5d640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.725+0000 7f055ab5d640 10 monclient: monmap: 2024-09-18T12:41:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:04.726418+0000 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:04.726418+0000 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 1 Processor -- start 2024-09-18T12:41:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 1 -- start start 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 10 monclient: init 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 0x7f055409c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7f055409ad90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f055409b160 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f055409b530 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05540692c0 con 0x7f055409b530 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.726+0000 7f055ab5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055409b990 con 0x7f055409b160 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f055ab5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f055ab5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055409db70 con 0x7f055409ad90 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f055ab5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f055ab5d640 10 monclient(hunting): authenticate will time out at 1692.048019s 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 0x7f055409c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 0x7f055409c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40220/0 (socket says 172.21.15.116:40220) 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 1 -- 172.21.15.116:0/3426850674 learned_addr learned my addr 172.21.15.116:0/3426850674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): get_auth_request con 0x7f055409ad90 auth_method 0 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05590fa640 1 --1- 172.21.15.116:0/3426850674 >> v1:172.21.15.116:6789/0 conn(0x7f055409b530 0x7f0554132070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33508/0 (socket says 172.21.15.116:33508) 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient(hunting): handle_auth_done global_id 184199 payload 274 2024-09-18T12:41:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 1 -- 172.21.15.116:0/3426850674 >> v1:172.21.15.144:6789/0 conn(0x7f055409b160 legacy=0x7f0554131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 1 -- 172.21.15.116:0/3426850674 >> v1:172.21.15.116:6789/0 conn(0x7f055409b530 legacy=0x7f0554132070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 1 monclient: found mon.noname-b 2024-09-18T12:41:04.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:41:04.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.727+0000 7f05588f9640 1 -- 172.21.15.116:0/3426850674 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055409d5b0 con 0x7f055409ad90 2024-09-18T12:41:04.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05588f9640 1 -- 172.21.15.116:0/3426850674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 msgr2=0x7f055409c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05588f9640 1 --2- 172.21.15.116:0/3426850674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 0x7f055409c5d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0548009940 tx=0x7f05480091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1145e9e344e559a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 1 -- 172.21.15.116:0/3426850674 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548032440 con 0x7f055409ad90 2024-09-18T12:41:04.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:41:04.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: dump: 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 5 monclient: authenticate success, global_id 184199 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 1 -- 172.21.15.116:0/3426850674 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05480325a0 con 0x7f055409ad90 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 4 set_mon_vals no callback set 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 1 -- 172.21.15.116:0/3426850674 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548032940 con 0x7f055409ad90 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f05537fe640 10 monclient: dump: 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 10 monclient: shutdown 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 1 -- 172.21.15.116:0/3426850674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 msgr2=0x7f055409c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 1 --2- 172.21.15.116:0/3426850674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 0x7f055409c5d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0548009940 tx=0x7f05480091d0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 1 -- 172.21.15.116:0/3426850674 shutdown_connections 2024-09-18T12:41:04.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 1 --2- 172.21.15.116:0/3426850674 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409ad90 0x7f055409c5d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 1 -- 172.21.15.116:0/3426850674 >> 172.21.15.116:0/3426850674 conn(0x7f05540678a0 msgr2=0x7f05540a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 1 -- 172.21.15.116:0/3426850674 shutdown_connections 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.730+0000 7f055ab5d640 1 -- 172.21.15.116:0/3426850674 wait complete. 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.731+0000 7f055ab5d640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.731+0000 7f055ab5d640 10 monclient: monmap: 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:04.743 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.731+0000 7f055ab5d640 1 Processor -- start 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.731+0000 7f055ab5d640 1 -- start start 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.731+0000 7f055ab5d640 10 monclient: init 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409ad90 0x7f0554141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): picked mon.noname-a con 0x7f055409ad90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409b160 0x7f0554130d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): picked mon.noname-c con 0x7f055409b160 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f055409b530 0x7f0554127a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): picked mon.noname-b con 0x7f055409b530 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055412b230 con 0x7f055409ad90 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055412af50 con 0x7f055409b530 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055412b510 con 0x7f055409b160 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f055ab5d640 10 monclient(hunting): authenticate will time out at 1692.053409s 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409ad90 0x7f0554141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409b160 0x7f0554130d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409ad90 0x7f0554141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37268/0 (socket says 172.21.15.116:37268) 2024-09-18T12:41:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 -- 172.21.15.116:0/1282464785 learned_addr learned my addr 172.21.15.116:0/1282464785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): get_auth_request con 0x7f055409ad90 auth_method 0 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 10 monclient(hunting): get_auth_request con 0x7f055409b160 auth_method 0 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05590fa640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f055409b530 0x7f0554127a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05590fa640 10 monclient(hunting): get_auth_request con 0x7f055409b530 auth_method 0 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05590fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05590fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05590fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient(hunting): handle_auth_done global_id 194118 payload 824 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409b160 msgr2=0x7f0554130d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f055409b160 0x7f0554130d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f055409b530 msgr2=0x7f0554127a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f055409b530 0x7f0554127a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 monclient: found mon.noname-a 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.732+0000 7f05588f9640 1 -- 172.21.15.116:0/1282464785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0548009450 con 0x7f055409ad90 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f05588f9640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409ad90 msgr2=0x7f0554141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f05588f9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409ad90 0x7f0554141150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0548035040 tx=0x7f0548032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054803b8f0 con 0x7f055409ad90 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: dump: 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.748 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f055ab5d640 5 monclient: authenticate success, global_id 194118 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054803ba50 con 0x7f055409ad90 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054803be10 con 0x7f055409ad90 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f055ab5d640 10 monclient: _renew_subs 2024-09-18T12:41:04.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f055ab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0554129920 con 0x7f055409ad90 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0552ffd640 4 set_mon_vals no callback set 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f0550ff9640 10 monclient: dump: 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.750 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f055ab5d640 10 monclient: _renew_subs 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f055ab5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.735+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05541312a0 con 0x7f055409ad90 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f055ab5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f055ab5d640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 0x7f0554127f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0548039040 con 0x7f055409ad90 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f055ab5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554065e00 con 0x7f0554141690 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0550ff9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0538038490 0x7f053803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 0x7f0554127f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient: get_auth_request con 0x7f0554141690 auth_method 0 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f054806c360 con 0x7f055409ad90 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f05590fa640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0538038490 0x7f053803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f05590fa640 10 monclient: get_auth_request con 0x7f0538038490 auth_method 0 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 10 monclient(hunting): handle_auth_done global_id 174316 payload 824 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.736+0000 7f0553fff640 1 -- 172.21.15.116:0/1282464785 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f055412b0c0 con 0x7f0554141690 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.738+0000 7f0553fff640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 msgr2=0x7f0554127f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.738+0000 7f0553fff640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 0x7f0554127f80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f054000f780 tx=0x7f0540007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.738+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054000d8b0 con 0x7f0554141690 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.738+0000 7f0550ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.738+0000 7f0550ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.738+0000 7f05590fa640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0538038490 0x7f053803a940 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f054c00abd0 tx=0x7f054c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.738+0000 7f0550ff9640 10 monclient: dump: 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.754 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.754 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.739+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f054000da10 con 0x7f0554141690 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.739+0000 7f0550ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.739+0000 7f0550ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.739+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 msgr2=0x7f0554127f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.739+0000 7f0550ff9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 0x7f0554127f80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f054000f780 tx=0x7f0540007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.739+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 msgr2=0x7f0554127f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.739+0000 7f0550ff9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0554141690 0x7f0554127f80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f055ab5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f055ab5d640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 0x7f0554095430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f055ab5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554141a60 con 0x7f05541284c0 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 0x7f0554095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient: get_auth_request con 0x7f05541284c0 auth_method 0 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 10 monclient(hunting): handle_auth_done global_id 174319 payload 824 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.745+0000 7f0553fff640 1 -- 172.21.15.116:0/1282464785 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f05541288c0 con 0x7f05541284c0 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.747+0000 7f0553fff640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 msgr2=0x7f0554095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.747+0000 7f0553fff640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 0x7f0554095430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f054000f780 tx=0x7f0540019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540014e40 con 0x7f05541284c0 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 10 monclient: dump: 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.758 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f054000fd20 con 0x7f05541284c0 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 msgr2=0x7f0554095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 0x7f0554095430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f054000f780 tx=0x7f0540019000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 msgr2=0x7f0554095430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f0550ff9640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f05541284c0 0x7f0554095430 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f054000f780 tx=0x7f0540019000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:04.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 156, 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:04.766 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:04.769 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.770 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.771 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:04.772 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0538038490 msgr2=0x7f053803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0538038490 0x7f053803a940 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f054c00abd0 tx=0x7f054c009040 comp rx=0 tx=0).stop 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 10 monclient: shutdown 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409ad90 msgr2=0x7f0554141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 --2- 172.21.15.116:0/1282464785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f055409ad90 0x7f0554141150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0548035040 tx=0x7f0548032000 comp rx=0 tx=0).stop 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f05590fa640 1 -- 172.21.15.116:0/1282464785 reap_dead start 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f05590fa640 1 -- 172.21.15.116:0/1282464785 reap_dead start 2024-09-18T12:41:04.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 shutdown_connections 2024-09-18T12:41:04.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 >> 172.21.15.116:0/1282464785 conn(0x7f05540678a0 msgr2=0x7f055409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 shutdown_connections 2024-09-18T12:41:04.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.748+0000 7f055ab5d640 1 -- 172.21.15.116:0/1282464785 wait complete. 2024-09-18T12:41:04.775 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:04.775 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:41:04.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:41:04.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.867+0000 7f992d00f640 10 monclient: get_monmap_and_config 2024-09-18T12:41:04.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.867+0000 7f992d00f640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.867+0000 7f992d00f640 10 monclient: monmap: 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:04.868458+0000 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:04.868458+0000 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:04.869 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 Processor -- start 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 -- start start 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient: init 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f99280974f0 0x7f992812ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): picked mon.noname-a con 0x7f99280974f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992809acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): picked mon.noname-b con 0x7f992809a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f992812dd40 0x7f992812e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): picked mon.noname-c con 0x7f992812dd40 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99280978c0 con 0x7f99280974f0 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992809ca20 con 0x7f992812dd40 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992809b1f0 con 0x7f992809a8a0 2024-09-18T12:41:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.868+0000 7f992d00f640 10 monclient(hunting): authenticate will time out at 1692.189990s 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992809acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f99280974f0 0x7f992812ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992809acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40232/0 (socket says 172.21.15.116:40232) 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 -- 172.21.15.116:0/1368916536 learned_addr learned my addr 172.21.15.116:0/1368916536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): get_auth_request con 0x7f992809a8a0 auth_method 0 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 10 monclient(hunting): get_auth_request con 0x7f99280974f0 auth_method 0 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992759e640 1 --2- 172.21.15.116:0/1368916536 >> v2:172.21.15.144:3300/0 conn(0x7f992812dd40 0x7f992812e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992759e640 10 monclient(hunting): get_auth_request con 0x7f992812dd40 auth_method 0 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient(hunting): handle_auth_done global_id 184205 payload 274 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 -- 172.21.15.116:0/1368916536 >> v2:172.21.15.144:3300/0 conn(0x7f992812dd40 msgr2=0x7f992812e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 --2- 172.21.15.116:0/1368916536 >> v2:172.21.15.144:3300/0 conn(0x7f992812dd40 0x7f992812e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 -- 172.21.15.116:0/1368916536 >> v2:172.21.15.116:3300/0 conn(0x7f99280974f0 msgr2=0x7f992812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 --2- 172.21.15.116:0/1368916536 >> v2:172.21.15.116:3300/0 conn(0x7f99280974f0 0x7f992812ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 monclient: found mon.noname-b 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f992659c640 1 -- 172.21.15.116:0/1368916536 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928098ad0 con 0x7f992809a8a0 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.869+0000 7f9926d9d640 1 --2- 172.21.15.116:0/1368916536 >> v2:172.21.15.116:3300/0 conn(0x7f99280974f0 0x7f992812ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992659c640 1 -- 172.21.15.116:0/1368916536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 msgr2=0x7f992809acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992659c640 1 --2- 172.21.15.116:0/1368916536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992809acb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9928065310 tx=0x7f991c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db97c21f2237ad58 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 1 -- 172.21.15.116:0/1368916536 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c032440 con 0x7f992809a8a0 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: dump: 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 1 -- 172.21.15.116:0/1368916536 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c0325a0 con 0x7f992809a8a0 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 5 monclient: authenticate success, global_id 184205 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 1 -- 172.21.15.116:0/1368916536 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c032980 con 0x7f992809a8a0 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 4 set_mon_vals no callback set 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f9925d9b640 10 monclient: dump: 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 10 monclient: shutdown 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 -- 172.21.15.116:0/1368916536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 msgr2=0x7f992809acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 --2- 172.21.15.116:0/1368916536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992809acb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9928065310 tx=0x7f991c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 -- 172.21.15.116:0/1368916536 shutdown_connections 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 --2- 172.21.15.116:0/1368916536 >> v2:172.21.15.144:3300/0 conn(0x7f992812dd40 0x7f992812e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 --2- 172.21.15.116:0/1368916536 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992809acb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 --2- 172.21.15.116:0/1368916536 >> v2:172.21.15.116:3300/0 conn(0x7f99280974f0 0x7f992812ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 -- 172.21.15.116:0/1368916536 >> 172.21.15.116:0/1368916536 conn(0x7f99280672b0 msgr2=0x7f99280676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 -- 172.21.15.116:0/1368916536 shutdown_connections 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.872+0000 7f992d00f640 1 -- 172.21.15.116:0/1368916536 wait complete. 2024-09-18T12:41:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.873+0000 7f992d00f640 10 monclient: build_initial_monmap 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.873+0000 7f992d00f640 10 monclient: monmap: 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.873+0000 7f992d00f640 1 Processor -- start 2024-09-18T12:41:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.873+0000 7f992d00f640 1 -- start start 2024-09-18T12:41:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.873+0000 7f992d00f640 10 monclient: init 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99280974f0 0x7f9928128930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): picked mon.noname-b con 0x7f99280974f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): picked mon.noname-c con 0x7f992809a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 0x7f992806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): picked mon.noname-a con 0x7f992812dd40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992809bf70 con 0x7f992812dd40 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9928127f70 con 0x7f99280974f0 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9928118bf0 con 0x7f992809a8a0 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992d00f640 10 monclient(hunting): authenticate will time out at 1692.195335s 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 0x7f992806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992806d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40246/0 (socket says 172.21.15.116:40246) 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 0x7f992806dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37290/0 (socket says 172.21.15.116:37290) 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 1 -- 172.21.15.116:0/949060861 learned_addr learned my addr 172.21.15.116:0/949060861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 10 monclient(hunting): get_auth_request con 0x7f992809a8a0 auth_method 0 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): get_auth_request con 0x7f992812dd40 auth_method 0 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99280974f0 0x7f9928128930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 10 monclient(hunting): get_auth_request con 0x7f99280974f0 auth_method 0 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient(hunting): handle_auth_done global_id 194124 payload 824 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 -- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 msgr2=0x7f992806d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992806d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 -- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99280974f0 msgr2=0x7f9928128930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99280974f0 0x7f9928128930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient: _finish_hunting 0 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 monclient: found mon.noname-a 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992759e640 1 -- 172.21.15.116:0/949060861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918009a20 con 0x7f992812dd40 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f9926d9d640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99280974f0 0x7f9928128930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.874+0000 7f992659c640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992806d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.876+0000 7f992759e640 1 -- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 msgr2=0x7f992806dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.876+0000 7f992759e640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 0x7f992806dc20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99200079a0 tx=0x7f9920008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 1 -- 172.21.15.116:0/949060861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920014070 con 0x7f992812dd40 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: dump: 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: _finish_auth 0 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 1 -- 172.21.15.116:0/949060861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9920009850 con 0x7f992812dd40 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992d00f640 5 monclient: authenticate success, global_id 194124 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 1 -- 172.21.15.116:0/949060861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920009c10 con 0x7f992812dd40 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992d00f640 10 monclient: _renew_subs 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992d00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c009450 con 0x7f992812dd40 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 10 monclient: dump: 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992559a640 4 set_mon_vals no callback set 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992d00f640 10 monclient: _renew_subs 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992d00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99281195d0 con 0x7f992812dd40 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.877+0000 7f990effd640 1 -- 172.21.15.116:0/949060861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9920018070 con 0x7f992812dd40 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f992d00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f992d00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f992d00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992806f1d0 con 0x7f992812dd40 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f990effd640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9914038490 0x7f991403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f990effd640 1 -- 172.21.15.116:0/949060861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f992004fad0 con 0x7f992812dd40 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f9926d9d640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9914038490 0x7f991403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.878+0000 7f9926d9d640 10 monclient: get_auth_request con 0x7f9914038490 auth_method 0 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.880+0000 7f9926d9d640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9914038490 0x7f991403a940 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f991800eae0 tx=0x7f991800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.882+0000 7f990effd640 1 -- 172.21.15.116:0/949060861 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f992806f1d0 con 0x7f992812dd40 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.882+0000 7f990effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:04.882+0000 7f990effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.095+0000 7f992d00f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:41:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.095+0000 7f992d00f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:41:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.095+0000 7f992d00f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.095+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f992806f1a0 con 0x7f992812dd40 2024-09-18T12:41:05.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.096+0000 7f990effd640 1 -- 172.21.15.116:0/949060861 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f9920019380 con 0x7f992812dd40 2024-09-18T12:41:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.096+0000 7f990effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:41:05.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.096+0000 7f990effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9914038490 msgr2=0x7f991403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9914038490 0x7f991403a940 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f991800eae0 tx=0x7f991800b040 comp rx=0 tx=0).stop 2024-09-18T12:41:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 10 monclient: shutdown 2024-09-18T12:41:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 msgr2=0x7f992806dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 0x7f992806dc20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99200079a0 tx=0x7f9920008040 comp rx=0 tx=0).stop 2024-09-18T12:41:05.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 shutdown_connections 2024-09-18T12:41:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9914038490 0x7f991403a940 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f992812dd40 0x7f992806dc20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f992809a8a0 0x7f992806d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 --2- 172.21.15.116:0/949060861 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f99280974f0 0x7f9928128930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 >> 172.21.15.116:0/949060861 conn(0x7f99280672b0 msgr2=0x7f99281296d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 shutdown_connections 2024-09-18T12:41:05.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:05.098+0000 7f992d00f640 1 -- 172.21.15.116:0/949060861 wait complete. 2024-09-18T12:41:05.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:41:05.530 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.530 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=3326263654,config=3140514009,health=3236339364,kv=2930496797,logm=1165113992}) 2024-09-18T12:41:05.530 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=3181131378,config=3140514009,health=1369586469,kv=2187923472,logm=4038784628}) 2024-09-18T12:41:05.531 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.531 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=8,logm=53} crc {auth=3326263654,config=3140514009,health=3236339364,kv=2930496797,logm=1165113992}) 2024-09-18T12:41:05.531 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=2470313490,config=3140514009,health=1307291893,kv=83634473,logm=499338336}) 2024-09-18T12:41:05.533 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.533 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297365429}) 2024-09-18T12:41:05.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=231747354}) 2024-09-18T12:41:05.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297365429}) 2024-09-18T12:41:05.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884405014}) 2024-09-18T12:41:05.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384349957}) 2024-09-18T12:41:05.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3569141883}) 2024-09-18T12:41:05.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384349957}) 2024-09-18T12:41:05.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385728510}) 2024-09-18T12:41:05.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.664+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.664+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010005986}) 2024-09-18T12:41:05.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.664+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2824016769}) 2024-09-18T12:41:05.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.664+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.664+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010005986}) 2024-09-18T12:41:05.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.664+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2095585162}) 2024-09-18T12:41:05.666 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.665+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.665+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994618992}) 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.665+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4046002845}) 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.665+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.665+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994618992}) 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.665+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=993873026}) 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2767827919}) 2024-09-18T12:41:05.667 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994997613}) 2024-09-18T12:41:05.668 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.668 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2767827919}) 2024-09-18T12:41:05.668 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349021394}) 2024-09-18T12:41:05.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465613350}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451502195}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465613350}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.666+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=667181010}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.667+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.667+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104138358}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.667+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224582967}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.667+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.667+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104138358}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.667+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726674834}) 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.668+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.668+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990226028}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.668+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133260210}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.668+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.668+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990226028}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.668+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743487438}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.669+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.669+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886014216}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.669+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1178857691}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.669+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.669+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886014216}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.669+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275108986}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965909182}) 2024-09-18T12:41:05.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1322595457}) 2024-09-18T12:41:05.673 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.673 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965909182}) 2024-09-18T12:41:05.673 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1000659025}) 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760606661}) 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942809970}) 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760606661}) 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.670+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014862856}) 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.671+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.676 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.671+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813496334}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.671+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725005980}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.671+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.671+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813496334}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.671+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2989334112}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.672+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.672+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773757544}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.672+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2188747728}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.672+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.672+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773757544}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.672+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=980587095}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.673+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.673+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888177492}) 2024-09-18T12:41:05.677 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.673+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3144787718}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.673+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.673+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888177492}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.673+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=381303111}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.674+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.674+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156162427}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.674+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2286632071}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.674+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.674+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156162427}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.674+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391258045}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.675+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.675+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368623885}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.675+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1605523425}) 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.675+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.678 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.675+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368623885}) 2024-09-18T12:41:05.679 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.675+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241962440}) 2024-09-18T12:41:05.684 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.676+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.684 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.676+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724013083}) 2024-09-18T12:41:05.684 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.676+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583166831}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.676+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.676+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724013083}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.676+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618375994}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144136414}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409489026}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144136414}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3542460453}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350402636}) 2024-09-18T12:41:05.685 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3338769922}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350402636}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.677+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=299009259}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.678+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.678+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403099340}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.678+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654615885}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.678+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.678+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403099340}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.678+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611389261}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.679+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.679+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329195174}) 2024-09-18T12:41:05.686 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.679+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384804163}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.679+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.679+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329195174}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.679+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334179718}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570470959}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148510609}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570470959}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442114442}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534599550}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=54287448}) 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.680+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.687 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534599550}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2995036719}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3140942725}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3353355152}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3140942725}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.681+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278925744}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.682+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.682+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153460735}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.682+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2653773314}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.682+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.682+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153460735}) 2024-09-18T12:41:05.688 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.682+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2479199035}) 2024-09-18T12:41:05.689 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.683+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.689 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.683+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321504357}) 2024-09-18T12:41:05.689 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.683+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3749468738}) 2024-09-18T12:41:05.689 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.683+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.689 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.683+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321504357}) 2024-09-18T12:41:05.689 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.683+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=625064275}) 2024-09-18T12:41:05.693 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.693+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.693 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.693+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422965738}) 2024-09-18T12:41:05.694 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.693+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3977303361}) 2024-09-18T12:41:05.694 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.693+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.694 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.693+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422965738}) 2024-09-18T12:41:05.694 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.693+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1131612006}) 2024-09-18T12:41:05.694 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.693+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.695 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099725668}) 2024-09-18T12:41:05.695 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002518468}) 2024-09-18T12:41:05.695 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.695 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099725668}) 2024-09-18T12:41:05.695 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197575330}) 2024-09-18T12:41:05.696 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.696 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583063297}) 2024-09-18T12:41:05.696 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1242234461}) 2024-09-18T12:41:05.696 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.694+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.696 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583063297}) 2024-09-18T12:41:05.696 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157451322}) 2024-09-18T12:41:05.697 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.697 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350511033}) 2024-09-18T12:41:05.697 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845453898}) 2024-09-18T12:41:05.697 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.697 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350511033}) 2024-09-18T12:41:05.697 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.695+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617344388}) 2024-09-18T12:41:05.698 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.696+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.698 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.696+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029623585}) 2024-09-18T12:41:05.698 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.696+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=605925103}) 2024-09-18T12:41:05.698 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.696+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.698 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.696+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029623585}) 2024-09-18T12:41:05.698 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.696+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137393069}) 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.697+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.697+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386967615}) 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.697+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898544778}) 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.697+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.697+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386967615}) 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.697+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=23947109}) 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.698+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.699 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.698+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990774170}) 2024-09-18T12:41:05.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.698+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420602511}) 2024-09-18T12:41:05.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.698+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.698+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990774170}) 2024-09-18T12:41:05.700 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.698+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1454466810}) 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.699+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.699+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820876500}) 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.699+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089837920}) 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.699+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.699+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820876500}) 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.699+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892048991}) 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.700+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.701 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.700+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240386402}) 2024-09-18T12:41:05.702 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.700+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341986973}) 2024-09-18T12:41:05.702 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.700+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.702 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.700+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240386402}) 2024-09-18T12:41:05.702 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.700+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246393660}) 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.701+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.701+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977534710}) 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.701+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559364105}) 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.701+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.701+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977534710}) 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.701+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3328981206}) 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.702+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.702+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680849264}) 2024-09-18T12:41:05.703 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.702+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390146612}) 2024-09-18T12:41:05.704 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.702+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.704 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.702+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680849264}) 2024-09-18T12:41:05.704 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.702+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925749415}) 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.703+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.703+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132274235}) 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.703+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2980834195}) 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.703+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.703+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132274235}) 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.703+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052654858}) 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.704+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.705 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.704+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219001742}) 2024-09-18T12:41:05.706 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.704+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4095789545}) 2024-09-18T12:41:05.706 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.704+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.706 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.704+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219001742}) 2024-09-18T12:41:05.706 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.704+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506542076}) 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.705+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.705+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593390875}) 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.705+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=492533833}) 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.705+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.705+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593390875}) 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.705+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1916638422}) 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.706+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.707 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.706+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578449773}) 2024-09-18T12:41:05.708 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.706+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2553610275}) 2024-09-18T12:41:05.708 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.706+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.708 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.706+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578449773}) 2024-09-18T12:41:05.708 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.706+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2147333815}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3043538517}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2806235833}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3043538517}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359026913}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165823181}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=473801005}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165823181}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.707+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3592857902}) 2024-09-18T12:41:05.710 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.708+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.711 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.708+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569773512}) 2024-09-18T12:41:05.711 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.708+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1511645664}) 2024-09-18T12:41:05.711 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.708+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.711 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.708+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569773512}) 2024-09-18T12:41:05.711 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.708+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3117874061}) 2024-09-18T12:41:05.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.731+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.731+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520436875}) 2024-09-18T12:41:05.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.731+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58311040}) 2024-09-18T12:41:05.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.731+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.731+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520436875}) 2024-09-18T12:41:05.732 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.731+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443763832}) 2024-09-18T12:41:05.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.744+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.744+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781164248}) 2024-09-18T12:41:05.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.744+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882091621}) 2024-09-18T12:41:05.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.744+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.744+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781164248}) 2024-09-18T12:41:05.745 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.744+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452456537}) 2024-09-18T12:41:05.746 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.745+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.746 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.745+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552032160}) 2024-09-18T12:41:05.746 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.745+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2054848665}) 2024-09-18T12:41:05.746 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.745+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:05.746 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.745+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552032160}) 2024-09-18T12:41:05.747 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:05.745+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330290833}) 2024-09-18T12:41:06.110 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:41:06.110 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:06.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.208+0000 7f65f8709640 10 monclient: get_monmap_and_config 2024-09-18T12:41:06.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.208+0000 7f65f8709640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.208+0000 7f65f8709640 10 monclient: monmap: 2024-09-18T12:41:06.209 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.209 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.209 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:06.209201+0000 2024-09-18T12:41:06.209 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:06.209201+0000 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:06.210 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 1 Processor -- start 2024-09-18T12:41:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 1 -- start start 2024-09-18T12:41:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient: init 2024-09-18T12:41:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:41:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65f0096e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f00975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient(hunting): picked mon.noname-b con 0x7f65f00971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f65f0133b30 0x7f65f0135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient(hunting): picked mon.noname-c con 0x7f65f0133b30 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f0099620 con 0x7f65f0096e10 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f0099340 con 0x7f65f0133b30 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0099900 con 0x7f65f00971e0 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f8709640 10 monclient(hunting): authenticate will time out at 1693.530835s 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f5ca4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f00975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f5ca4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f00975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40726/0 (socket says 172.21.15.116:40726) 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.209+0000 7f65f5ca4640 1 -- 172.21.15.116:0/2030113316 learned_addr learned my addr 172.21.15.116:0/2030113316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): get_auth_request con 0x7f65f00971e0 auth_method 0 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f6ca6640 1 --2- 172.21.15.116:0/2030113316 >> v2:172.21.15.144:3300/0 conn(0x7f65f0133b30 0x7f65f0135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_done global_id 184211 payload 274 2024-09-18T12:41:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 1 -- 172.21.15.116:0/2030113316 >> v2:172.21.15.144:3300/0 conn(0x7f65f0133b30 msgr2=0x7f65f0135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 1 --2- 172.21.15.116:0/2030113316 >> v2:172.21.15.144:3300/0 conn(0x7f65f0133b30 0x7f65f0135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 1 -- 172.21.15.116:0/2030113316 >> v1:172.21.15.144:6789/0 conn(0x7f65f0096e10 legacy=0x7f65f009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 1 monclient: found mon.noname-b 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f5ca4640 1 -- 172.21.15.116:0/2030113316 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f009ac70 con 0x7f65f00971e0 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f6ca6640 10 monclient: get_auth_request con 0x7f65f0133b30 auth_method 0 2024-09-18T12:41:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.210+0000 7f65f6ca6640 1 --2- 172.21.15.116:0/2030113316 >> v2:172.21.15.144:3300/0 conn(0x7f65f0133b30 0x7f65f0135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:41:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f5ca4640 1 -- 172.21.15.116:0/2030113316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 msgr2=0x7f65f00975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f5ca4640 1 --2- 172.21.15.116:0/2030113316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f00975b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65f009aff0 tx=0x7f65e40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e858a4a085e27f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 1 -- 172.21.15.116:0/2030113316 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4032440 con 0x7f65f00971e0 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: dump: 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 5 monclient: authenticate success, global_id 184211 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 1 -- 172.21.15.116:0/2030113316 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e40325a0 con 0x7f65f00971e0 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 4 set_mon_vals no callback set 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 1 -- 172.21.15.116:0/2030113316 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4032940 con 0x7f65f00971e0 2024-09-18T12:41:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f54a3640 10 monclient: dump: 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 10 monclient: shutdown 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 1 -- 172.21.15.116:0/2030113316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 msgr2=0x7f65f00975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.213+0000 7f65f8709640 1 --2- 172.21.15.116:0/2030113316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f00975b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65f009aff0 tx=0x7f65e40091d0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 -- 172.21.15.116:0/2030113316 shutdown_connections 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 --2- 172.21.15.116:0/2030113316 >> v2:172.21.15.144:3300/0 conn(0x7f65f0133b30 0x7f65f0135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 --2- 172.21.15.116:0/2030113316 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f00975b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 -- 172.21.15.116:0/2030113316 >> 172.21.15.116:0/2030113316 conn(0x7f65f00672b0 msgr2=0x7f65f00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 -- 172.21.15.116:0/2030113316 shutdown_connections 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 -- 172.21.15.116:0/2030113316 wait complete. 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 10 monclient: monmap: 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:06.226 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 Processor -- start 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 1 -- start start 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.214+0000 7f65f8709640 10 monclient: init 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f0096e10 0x7f65f0141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): picked mon.noname-b con 0x7f65f0096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:06.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f0141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): picked mon.noname-c con 0x7f65f00971e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65f0133b30 0x7f65f0141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): picked mon.noname-a con 0x7f65f0133b30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0133450 con 0x7f65f0133b30 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00696a0 con 0x7f65f0096e10 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01327b0 con 0x7f65f00971e0 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f8709640 10 monclient(hunting): authenticate will time out at 1693.536341s 2024-09-18T12:41:06.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f0141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f0141a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40740/0 (socket says 172.21.15.116:40740) 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 -- 172.21.15.116:0/3589101760 learned_addr learned my addr 172.21.15.116:0/3589101760 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): get_auth_request con 0x7f65f00971e0 auth_method 0 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65f0133b30 0x7f65f0141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f0096e10 0x7f65f0141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 10 monclient(hunting): get_auth_request con 0x7f65f0096e10 auth_method 0 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 10 monclient(hunting): get_auth_request con 0x7f65f0133b30 auth_method 0 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f64a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient(hunting): handle_auth_done global_id 184214 payload 824 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 -- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f0096e10 msgr2=0x7f65f0141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f0096e10 0x7f65f0141500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 -- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65f0133b30 msgr2=0x7f65f0141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65f0133b30 0x7f65f0141f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 monclient: found mon.noname-c 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f5ca4640 1 -- 172.21.15.116:0/3589101760 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e4009450 con 0x7f65f00971e0 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.215+0000 7f65f6ca6640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65f0133b30 0x7f65f0141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.217+0000 7f65f5ca4640 1 -- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 msgr2=0x7f65f0141a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.217+0000 7f65f5ca4640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f0141a40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65e4035040 tx=0x7f65e4032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 1 -- 172.21.15.116:0/3589101760 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4031a40 con 0x7f65f00971e0 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: dump: 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65f8709640 5 monclient: authenticate success, global_id 184214 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 1 -- 172.21.15.116:0/3589101760 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e4031be0 con 0x7f65f00971e0 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 1 -- 172.21.15.116:0/3589101760 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e403d440 con 0x7f65f00971e0 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65f8709640 10 monclient: _renew_subs 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65f8709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f006a8f0 con 0x7f65f00971e0 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 10 monclient: dump: 2024-09-18T12:41:06.233 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65effff640 4 set_mon_vals no callback set 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65f8709640 10 monclient: _renew_subs 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65f8709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f0131c20 con 0x7f65f00971e0 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.218+0000 7f65ee7fc640 1 -- 172.21.15.116:0/3589101760 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f65e4039040 con 0x7f65f00971e0 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65f8709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65f8709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65f8709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f004bf60 con 0x7f65f00971e0 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65ee7fc640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65dc0384c0 0x7f65dc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65ee7fc640 1 -- 172.21.15.116:0/3589101760 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f65e406c230 con 0x7f65f00971e0 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65f64a5640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65dc0384c0 0x7f65dc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.219+0000 7f65f64a5640 10 monclient: get_auth_request con 0x7f65dc0384c0 auth_method 0 2024-09-18T12:41:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.222+0000 7f65f64a5640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65dc0384c0 0x7f65dc03a970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f65e8007fd0 tx=0x7f65e8007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.224+0000 7f65ee7fc640 1 -- 172.21.15.116:0/3589101760 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f65e4030b20 con 0x7f65f00971e0 2024-09-18T12:41:06.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.224+0000 7f65ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.224+0000 7f65ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:06.328 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.328 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991818804}) 2024-09-18T12:41:06.328 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005346234}) 2024-09-18T12:41:06.328 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.328 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991818804}) 2024-09-18T12:41:06.328 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4221013184}) 2024-09-18T12:41:06.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.328+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.328+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=414100650}) 2024-09-18T12:41:06.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.328+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2459640363}) 2024-09-18T12:41:06.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.328+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.328+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=414100650}) 2024-09-18T12:41:06.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.328+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=86847444}) 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3328323189}) 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2861841442}) 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3328323189}) 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878556309}) 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046586328}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1446931014}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046586328}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.329+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903886791}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.330+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.330+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948633456}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.330+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1217732555}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.330+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.330+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948633456}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.330+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382298735}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130564337}) 2024-09-18T12:41:06.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279307185}) 2024-09-18T12:41:06.334 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.334 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130564337}) 2024-09-18T12:41:06.334 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2476152652}) 2024-09-18T12:41:06.334 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.335 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074263859}) 2024-09-18T12:41:06.335 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2723499145}) 2024-09-18T12:41:06.335 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.335 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074263859}) 2024-09-18T12:41:06.335 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=432535394}) 2024-09-18T12:41:06.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f65f8709640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:06.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f65f8709640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:06.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f65f8709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f65f006f700 con 0x7f65f00971e0 2024-09-18T12:41:06.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f65ee7fc640 1 -- 172.21.15.116:0/3589101760 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f65e4042080 con 0x7f65f00971e0 2024-09-18T12:41:06.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f65ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:06.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f65ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:06.441 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:06.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345868435}) 2024-09-18T12:41:06.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195435987}) 2024-09-18T12:41:06.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345868435}) 2024-09-18T12:41:06.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4235683567}) 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65dc0384c0 msgr2=0x7f65dc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65dc0384c0 0x7f65dc03a970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f65e8007fd0 tx=0x7f65e8007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 10 monclient: shutdown 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 msgr2=0x7f65f0141a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f0141a40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65e4035040 tx=0x7f65e4032000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 shutdown_connections 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f65dc0384c0 0x7f65dc03a970 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65f0133b30 0x7f65f0141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f65f00971e0 0x7f65f0141a40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 --2- 172.21.15.116:0/3589101760 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f65f0096e10 0x7f65f0141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 >> 172.21.15.116:0/3589101760 conn(0x7f65f00672b0 msgr2=0x7f65f0128700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 shutdown_connections 2024-09-18T12:41:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.441+0000 7f65f8709640 1 -- 172.21.15.116:0/3589101760 wait complete. 2024-09-18T12:41:06.452 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:41:06.452 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:41:06.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:41:06.452 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:41:06.459 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.458+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.459 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.458+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799490187}) 2024-09-18T12:41:06.459 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.458+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523231694}) 2024-09-18T12:41:06.459 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.458+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.459 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.458+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799490187}) 2024-09-18T12:41:06.460 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.458+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512240266}) 2024-09-18T12:41:06.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806471260}) 2024-09-18T12:41:06.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937659802}) 2024-09-18T12:41:06.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806471260}) 2024-09-18T12:41:06.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3368250149}) 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.546+0000 7f749e91d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.546+0000 7f749e91d640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.546+0000 7f749e91d640 10 monclient: monmap: 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:06.546958+0000 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:06.546958+0000 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:06.551 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.547+0000 7f749e91d640 1 Processor -- start 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.547+0000 7f749e91d640 1 -- start start 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.547+0000 7f749e91d640 10 monclient: init 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.547+0000 7f749e91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.547+0000 7f749e91d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.547+0000 7f749e91d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f74981318f0 0x7f7498131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.547+0000 7f749e91d640 10 monclient(hunting): picked mon.noname-c con 0x7f74981318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f749809ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7498136c20 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749809f530 con 0x7f7498136c20 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749809fbf0 con 0x7f749809ea50 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7498131d50 con 0x7f74981318f0 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749e91d640 10 monclient(hunting): authenticate will time out at 1693.869085s 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749ceba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7498136c20 0x7f7498139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39514/0 (socket says 172.21.15.116:39514) 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f749ceba640 1 -- 172.21.15.116:0/1638500370 learned_addr learned my addr 172.21.15.116:0/1638500370 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 1 --2- 172.21.15.116:0/1638500370 >> v2:172.21.15.144:3300/0 conn(0x7f74981318f0 0x7f7498131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): get_auth_request con 0x7f74981318f0 auth_method 0 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient(hunting): handle_auth_done global_id 174331 payload 274 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 1 -- 172.21.15.116:0/1638500370 >> v1:172.21.15.144:6789/0 conn(0x7f749809ea50 legacy=0x7f749809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 1 -- 172.21.15.116:0/1638500370 >> v1:172.21.15.116:6789/0 conn(0x7f7498136c20 legacy=0x7f7498139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 1 monclient: found mon.noname-c 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:41:06.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.548+0000 7f7497fff640 1 -- 172.21.15.116:0/1638500370 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749809fe30 con 0x7f74981318f0 2024-09-18T12:41:06.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493910903}) 2024-09-18T12:41:06.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262706321}) 2024-09-18T12:41:06.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493910903}) 2024-09-18T12:41:06.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151568016}) 2024-09-18T12:41:06.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7497fff640 1 -- 172.21.15.116:0/1638500370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 msgr2=0x7f7498131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7497fff640 1 --2- 172.21.15.116:0/1638500370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 0x7f7498131f30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7488009850 tx=0x7f74880090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=847cea9db82cbd78 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 1 -- 172.21.15.116:0/1638500370 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488032440 con 0x7f74981318f0 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: dump: 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.564 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f749e91d640 5 monclient: authenticate success, global_id 174331 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 1 -- 172.21.15.116:0/1638500370 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74880325a0 con 0x7f74981318f0 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f749e91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f749e91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:06.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f749e91d640 4 set_mon_vals no callback set 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 1 -- 172.21.15.116:0/1638500370 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488032980 con 0x7f74981318f0 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.551+0000 7f7496ffd640 10 monclient: dump: 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.566 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 10 monclient: shutdown 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 1 -- 172.21.15.116:0/1638500370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 msgr2=0x7f7498131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 1 --2- 172.21.15.116:0/1638500370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 0x7f7498131f30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7488009850 tx=0x7f74880090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 1 -- 172.21.15.116:0/1638500370 shutdown_connections 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 1 --2- 172.21.15.116:0/1638500370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 0x7f7498131f30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 1 -- 172.21.15.116:0/1638500370 >> 172.21.15.116:0/1638500370 conn(0x7f7498067110 msgr2=0x7f7498067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 1 -- 172.21.15.116:0/1638500370 shutdown_connections 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 1 -- 172.21.15.116:0/1638500370 wait complete. 2024-09-18T12:41:06.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.552+0000 7f749e91d640 10 monclient: monmap: 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:06.568 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 Processor -- start 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 -- start start 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient: init 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 0x7f74981327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): picked mon.noname-a con 0x7f749809ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 0x7f7498132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): picked mon.noname-b con 0x7f74981318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7498136c20 0x7f7498133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): picked mon.noname-c con 0x7f7498136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7498135f80 con 0x7f749809ea50 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7498131460 con 0x7f74981318f0 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7498135ca0 con 0x7f7498136c20 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f749e91d640 10 monclient(hunting): authenticate will time out at 1693.874905s 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.553+0000 7f7497fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 0x7f74981327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 0x7f74981327d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49320/0 (socket says 172.21.15.116:49320) 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7498136c20 0x7f7498133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 -- 172.21.15.116:0/2104221055 learned_addr learned my addr 172.21.15.116:0/2104221055 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:06.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): get_auth_request con 0x7f749809ea50 auth_method 0 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 10 monclient(hunting): get_auth_request con 0x7f7498136c20 auth_method 0 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 0x7f7498132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f749ceba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 10 monclient(hunting): get_auth_request con 0x7f74981318f0 auth_method 0 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient(hunting): handle_auth_done global_id 194130 payload 824 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7498136c20 msgr2=0x7f7498133250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7498136c20 0x7f7498133250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 msgr2=0x7f7498132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74981318f0 0x7f7498132d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 monclient: found mon.noname-a 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.554+0000 7f7497fff640 1 -- 172.21.15.116:0/2104221055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7488009390 con 0x7f749809ea50 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f7497fff640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 msgr2=0x7f74981327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f7497fff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 0x7f74981327d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7488009360 tx=0x7f7488032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488031990 con 0x7f749809ea50 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: dump: 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.573 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488031af0 con 0x7f749809ea50 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f749e91d640 5 monclient: authenticate success, global_id 194130 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f749e91d640 10 monclient: _renew_subs 2024-09-18T12:41:06.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f749e91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7498136260 con 0x7f749809ea50 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488031e70 con 0x7f749809ea50 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f747ffff640 10 monclient: dump: 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.575 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f749e91d640 10 monclient: _renew_subs 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f749e91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74981365b0 con 0x7f749809ea50 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.556+0000 7f74967fc640 4 set_mon_vals no callback set 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f749e91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f749e91d640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 0x7f74980a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f749e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7498065e00 con 0x7f7498133790 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 0x7f74980a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient: get_auth_request con 0x7f7498133790 auth_method 0 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f7497fff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 0x7f74981327d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7488009360 tx=0x7f7488032000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f748806c5f0 osd_map(267..267 src has 242..267) v4 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7488039040 con 0x7f749809ea50 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 10 monclient(hunting): handle_auth_done global_id 194133 payload 824 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f74977fe640 1 -- 172.21.15.116:0/2104221055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74981360c0 con 0x7f7498133790 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.557+0000 7f747ffff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74840384c0 0x7f748403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.558+0000 7f749ceba640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74840384c0 0x7f748403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.558+0000 7f749ceba640 10 monclient: get_auth_request con 0x7f74840384c0 auth_method 0 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f74977fe640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 msgr2=0x7f74980a3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f74977fe640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 0x7f74980a3d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f748c00f780 tx=0x7f748c007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c00d8b0 con 0x7f7498133790 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 10 monclient: dump: 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f748c00da10 con 0x7f7498133790 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 msgr2=0x7f74980a3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 0x7f74980a3d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f748c00f780 tx=0x7f748c007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 msgr2=0x7f74980a3d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f747ffff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7498133790 0x7f74980a3d00 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f748c00f780 tx=0x7f748c007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:06.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.560+0000 7f749ceba640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74840384c0 0x7f748403a970 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f74981344f0 tx=0x7f7490009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f749e91d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f749e91d640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 0x7f74980a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f749e91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74981363f0 con 0x7f74980a4240 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 0x7f74980a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient: get_auth_request con 0x7f74980a4240 auth_method 0 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 10 monclient(hunting): handle_auth_done global_id 194136 payload 824 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.566+0000 7f74977fe640 1 -- 172.21.15.116:0/2104221055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f749804bf60 con 0x7f74980a4240 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.568+0000 7f74977fe640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 msgr2=0x7f74980a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.568+0000 7f74977fe640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 0x7f74980a4640 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f748c00f780 tx=0x7f748c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748c007030 con 0x7f74980a4240 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 10 monclient: dump: 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.583 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f748c007190 con 0x7f74980a4240 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 msgr2=0x7f74980a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 0x7f74980a4640 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f748c00f780 tx=0x7f748c019000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 msgr2=0x7f74980a4640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74980a4240 0x7f74980a4640 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f748c00f780 tx=0x7f748c019000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 156, 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:06.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.586 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.587 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:06.588 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.589 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:06.591 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:06.592 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.593 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:06.594 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.595 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.596 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.597 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:06.598 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:06.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74840384c0 msgr2=0x7f748403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74840384c0 0x7f748403a970 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f74981344f0 tx=0x7f7490009040 comp rx=0 tx=0).stop 2024-09-18T12:41:06.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 10 monclient: shutdown 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 msgr2=0x7f74981327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 0x7f74981327d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7488009360 tx=0x7f7488032000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749ceba640 1 -- 172.21.15.116:0/2104221055 reap_dead start 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f747ffff640 1 -- 172.21.15.116:0/2104221055 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (unknown 0 0 0) 0x7f748806c5f0 con 0x7f749809ea50 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 shutdown_connections 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 --2- 172.21.15.116:0/2104221055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f749809ea50 0x7f74981327d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 >> 172.21.15.116:0/2104221055 conn(0x7f7498067110 msgr2=0x7f74980a1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 shutdown_connections 2024-09-18T12:41:06.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.569+0000 7f749e91d640 1 -- 172.21.15.116:0/2104221055 wait complete. 2024-09-18T12:41:06.601 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:06.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:41:06.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:41:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.732+0000 7f5e07714640 10 monclient: get_monmap_and_config 2024-09-18T12:41:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.732+0000 7f5e07714640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.732+0000 7f5e07714640 10 monclient: monmap: 2024-09-18T12:41:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.733 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:06.733376+0000 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:06.733376+0000 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:06.734 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 1 Processor -- start 2024-09-18T12:41:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 1 -- start start 2024-09-18T12:41:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient: init 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00131ab0 0x7f5e00131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient(hunting): picked mon.noname-b con 0x7f5e00131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e0009e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e00136bb0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0009faf0 con 0x7f5e00136bb0 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0009fd30 con 0x7f5e0009e910 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e000a1110 con 0x7f5e00131ab0 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.733+0000 7f5e07714640 10 monclient(hunting): authenticate will time out at 1694.054981s 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00131ab0 0x7f5e00131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e05cb1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5e00136bb0 0x7f5e00138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39528/0 (socket says 172.21.15.116:39528) 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e05cb1640 1 -- 172.21.15.116:0/2720795329 learned_addr learned my addr 172.21.15.116:0/2720795329 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): get_auth_request con 0x7f5e00131ab0 auth_method 0 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient(hunting): handle_auth_done global_id 184220 payload 274 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 1 -- 172.21.15.116:0/2720795329 >> v1:172.21.15.144:6789/0 conn(0x7f5e0009e910 legacy=0x7f5e0009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 1 -- 172.21.15.116:0/2720795329 >> v1:172.21.15.116:6789/0 conn(0x7f5e00136bb0 legacy=0x7f5e00138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 1 monclient: found mon.noname-b 2024-09-18T12:41:06.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.734+0000 7f5e054b0640 1 -- 172.21.15.116:0/2720795329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e000a13f0 con 0x7f5e00131ab0 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e054b0640 1 -- 172.21.15.116:0/2720795329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00131ab0 msgr2=0x7f5e00131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e054b0640 1 --2- 172.21.15.116:0/2720795329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00131ab0 0x7f5e00131e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009880 tx=0x7f5dfc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b89a652a5e74d5d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 1 -- 172.21.15.116:0/2720795329 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc032440 con 0x7f5e00131ab0 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: dump: 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 5 monclient: authenticate success, global_id 184220 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 1 -- 172.21.15.116:0/2720795329 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc0325a0 con 0x7f5e00131ab0 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 4 set_mon_vals no callback set 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 1 -- 172.21.15.116:0/2720795329 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc032980 con 0x7f5e00131ab0 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5df7fff640 10 monclient: dump: 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 10 monclient: shutdown 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 1 -- 172.21.15.116:0/2720795329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00131ab0 msgr2=0x7f5e00131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 1 --2- 172.21.15.116:0/2720795329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00131ab0 0x7f5e00131e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009880 tx=0x7f5dfc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 1 -- 172.21.15.116:0/2720795329 shutdown_connections 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 1 --2- 172.21.15.116:0/2720795329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00131ab0 0x7f5e00131e80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.737+0000 7f5e07714640 1 -- 172.21.15.116:0/2720795329 >> 172.21.15.116:0/2720795329 conn(0x7f5e00067d30 msgr2=0x7f5e000a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.738+0000 7f5e07714640 1 -- 172.21.15.116:0/2720795329 shutdown_connections 2024-09-18T12:41:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.738+0000 7f5e07714640 1 -- 172.21.15.116:0/2720795329 wait complete. 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.738+0000 7f5e07714640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.738+0000 7f5e07714640 10 monclient: monmap: 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.738+0000 7f5e07714640 1 Processor -- start 2024-09-18T12:41:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.738+0000 7f5e07714640 1 -- start start 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.738+0000 7f5e07714640 10 monclient: init 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e0009e910 0x7f5e00135d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0009e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e00131ab0 0x7f5e00096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): picked mon.noname-a con 0x7f5e00131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e00136bb0 0x7f5e00096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): picked mon.noname-b con 0x7f5e00136bb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0009bde0 con 0x7f5e00131ab0 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0013a3f0 con 0x7f5e00136bb0 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00099aa0 con 0x7f5e0009e910 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e07714640 10 monclient(hunting): authenticate will time out at 1694.060456s 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e0009e910 0x7f5e00135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e0009e910 0x7f5e00135d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40778/0 (socket says 172.21.15.116:40778) 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 -- 172.21.15.116:0/4046133838 learned_addr learned my addr 172.21.15.116:0/4046133838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): get_auth_request con 0x7f5e0009e910 auth_method 0 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e00131ab0 0x7f5e00096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 10 monclient(hunting): get_auth_request con 0x7f5e00131ab0 auth_method 0 2024-09-18T12:41:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e00136bb0 0x7f5e00096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 10 monclient(hunting): get_auth_request con 0x7f5e00136bb0 auth_method 0 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e05cb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient(hunting): handle_auth_done global_id 184223 payload 824 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e00136bb0 msgr2=0x7f5e00096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e00136bb0 0x7f5e00096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e00131ab0 msgr2=0x7f5e00096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e00131ab0 0x7f5e00096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 monclient: found mon.noname-c 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.739+0000 7f5e054b0640 1 -- 172.21.15.116:0/4046133838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc009390 con 0x7f5e0009e910 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e054b0640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e0009e910 msgr2=0x7f5e00135d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e054b0640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e0009e910 0x7f5e00135d60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009880 tx=0x7f5dfc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc031990 con 0x7f5e0009e910 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: dump: 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.756 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 5 monclient: authenticate success, global_id 184223 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc031b10 con 0x7f5e0009e910 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc03b400 con 0x7f5e0009e910 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 10 monclient: _renew_subs 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e00098a40 con 0x7f5e0009e910 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df77fe640 4 set_mon_vals no callback set 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 10 monclient: dump: 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 10 monclient: _renew_subs 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0009b700 con 0x7f5e0009e910 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5dfc043030 con 0x7f5e0009e910 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.742+0000 7f5e07714640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e07714640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 0x7f5e00132660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e07714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00099960 con 0x7f5e000974e0 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 0x7f5e00132660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient: get_auth_request con 0x7f5e000974e0 auth_method 0 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5df57fa640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5de40384c0 0x7f5de403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5dfc041070 con 0x7f5e0009e910 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 10 monclient(hunting): handle_auth_done global_id 184226 payload 824 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e04caf640 1 -- 172.21.15.116:0/4046133838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e000978b0 con 0x7f5e000974e0 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e05cb1640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5de40384c0 0x7f5de403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.743+0000 7f5e05cb1640 10 monclient: get_auth_request con 0x7f5de40384c0 auth_method 0 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.745+0000 7f5e04caf640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 msgr2=0x7f5e00132660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.745+0000 7f5e04caf640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 0x7f5e00132660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5df000f780 tx=0x7f5df0007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.745+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df000d8b0 con 0x7f5e000974e0 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.745+0000 7f5df57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.745+0000 7f5df57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.745+0000 7f5df57fa640 10 monclient: dump: 2024-09-18T12:41:06.760 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5e05cb1640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5de40384c0 0x7f5de403a970 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5e00066040 tx=0x7f5df80052b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5df000da10 con 0x7f5e000974e0 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5df57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5df57fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:06.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 msgr2=0x7f5e00132660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5df57fa640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 0x7f5e00132660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5df000f780 tx=0x7f5df0007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:06.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 msgr2=0x7f5e00132660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:06.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.746+0000 7f5df57fa640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e000974e0 0x7f5e00132660 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5df000f780 tx=0x7f5df0007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.751+0000 7f5e07714640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.751+0000 7f5e07714640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 0x7f5e00132fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.751+0000 7f5e07714640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.751+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00063740 con 0x7f5e00132ba0 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 0x7f5e00132fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient: get_auth_request con 0x7f5e00132ba0 auth_method 0 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 10 monclient(hunting): handle_auth_done global_id 184229 payload 824 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.752+0000 7f5e04caf640 1 -- 172.21.15.116:0/4046133838 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5e0009b890 con 0x7f5e00132ba0 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5e04caf640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 msgr2=0x7f5e00132fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5e04caf640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 0x7f5e00132fa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5df000f780 tx=0x7f5df0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df001d400 con 0x7f5e00132ba0 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 10 monclient: dump: 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f5df001d560 con 0x7f5e00132ba0 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 msgr2=0x7f5e00132fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 0x7f5e00132fa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5df000f780 tx=0x7f5df0019000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 msgr2=0x7f5e00132fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f5df57fa640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e00132ba0 0x7f5e00132fa0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5df000f780 tx=0x7f5df0019000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 156, 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:06.767 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.768 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:06.770 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.771 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:06.772 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.773 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:06.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.777 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.778 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:06.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.779 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.779 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:06.779 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:06.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.754+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029678784}) 2024-09-18T12:41:06.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2020327136}) 2024-09-18T12:41:06.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029678784}) 2024-09-18T12:41:06.780 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3542440821}) 2024-09-18T12:41:06.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5de40384c0 msgr2=0x7f5de403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5de40384c0 0x7f5de403a970 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5e00066040 tx=0x7f5df80052b0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 10 monclient: shutdown 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e0009e910 msgr2=0x7f5e00135d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 --2- 172.21.15.116:0/4046133838 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e0009e910 0x7f5e00135d60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009880 tx=0x7f5dfc032000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e05cb1640 1 -- 172.21.15.116:0/4046133838 reap_dead start 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e05cb1640 1 -- 172.21.15.116:0/4046133838 reap_dead start 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 shutdown_connections 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 >> 172.21.15.116:0/4046133838 conn(0x7f5e00067d30 msgr2=0x7f5e000a4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 shutdown_connections 2024-09-18T12:41:06.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.755+0000 7f5e07714640 1 -- 172.21.15.116:0/4046133838 wait complete. 2024-09-18T12:41:06.783 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:06.783 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:41:06.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:41:06.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.876+0000 7f367619d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:06.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.876+0000 7f367619d640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.876+0000 7f367619d640 10 monclient: monmap: 2024-09-18T12:41:06.878 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.878 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.878 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:06.877444+0000 2024-09-18T12:41:06.878 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:06.877444+0000 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:06.879 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 1 Processor -- start 2024-09-18T12:41:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 1 -- start start 2024-09-18T12:41:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 10 monclient: init 2024-09-18T12:41:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f367009ad90 0x7f367009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 10 monclient(hunting): picked mon.noname-c con 0x7f367009ad90 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f367009b160 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f367009b530 0x7f3670132070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 10 monclient(hunting): picked mon.noname-a con 0x7f367009b530 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.877+0000 7f367619d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36700692c0 con 0x7f367009b160 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f367619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f367619d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367009b990 con 0x7f367009b530 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f367619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f367619d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367009db70 con 0x7f367009ad90 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f367619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f367619d640 10 monclient(hunting): authenticate will time out at 1694.199016s 2024-09-18T12:41:06.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f367009b530 0x7f3670132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f367009b530 0x7f3670132070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49360/0 (socket says 172.21.15.116:49360) 2024-09-18T12:41:06.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 -- 172.21.15.116:0/2192492311 learned_addr learned my addr 172.21.15.116:0/2192492311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): get_auth_request con 0x7f367009b530 auth_method 0 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366f7fe640 1 --2- 172.21.15.116:0/2192492311 >> v2:172.21.15.144:3300/0 conn(0x7f367009ad90 0x7f367009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366f7fe640 10 monclient(hunting): get_auth_request con 0x7f367009ad90 auth_method 0 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient(hunting): handle_auth_done global_id 194142 payload 274 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 -- 172.21.15.116:0/2192492311 >> v2:172.21.15.144:3300/0 conn(0x7f367009ad90 msgr2=0x7f367009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 --2- 172.21.15.116:0/2192492311 >> v2:172.21.15.144:3300/0 conn(0x7f367009ad90 0x7f367009c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 -- 172.21.15.116:0/2192492311 >> v1:172.21.15.144:6789/0 conn(0x7f367009b160 legacy=0x7f3670131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 monclient: found mon.noname-a 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.878+0000 7f366ffff640 1 -- 172.21.15.116:0/2192492311 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367009d5b0 con 0x7f367009b530 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366ffff640 1 -- 172.21.15.116:0/2192492311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 msgr2=0x7f3670132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366ffff640 1 --2- 172.21.15.116:0/2192492311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670132070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3670068070 tx=0x7f36680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab65ffd1bd66e40f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 1 -- 172.21.15.116:0/2192492311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668032440 con 0x7f367009b530 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: dump: 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 1 -- 172.21.15.116:0/2192492311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36680325a0 con 0x7f367009b530 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 5 monclient: authenticate success, global_id 194142 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 1 -- 172.21.15.116:0/2192492311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668032940 con 0x7f367009b530 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 4 set_mon_vals no callback set 2024-09-18T12:41:06.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f366e7fc640 10 monclient: dump: 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 10 monclient: shutdown 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 -- 172.21.15.116:0/2192492311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 msgr2=0x7f3670132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 --2- 172.21.15.116:0/2192492311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670132070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3670068070 tx=0x7f36680091d0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 -- 172.21.15.116:0/2192492311 shutdown_connections 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 --2- 172.21.15.116:0/2192492311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670132070 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 --2- 172.21.15.116:0/2192492311 >> v2:172.21.15.144:3300/0 conn(0x7f367009ad90 0x7f367009c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 -- 172.21.15.116:0/2192492311 >> 172.21.15.116:0/2192492311 conn(0x7f36700678a0 msgr2=0x7f36700a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 -- 172.21.15.116:0/2192492311 shutdown_connections 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.881+0000 7f367619d640 1 -- 172.21.15.116:0/2192492311 wait complete. 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.882+0000 7f367619d640 10 monclient: build_initial_monmap 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.882+0000 7f367619d640 10 monclient: monmap: 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.882+0000 7f367619d640 1 Processor -- start 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.882+0000 7f367619d640 1 -- start start 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.882+0000 7f367619d640 10 monclient: init 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367009ad90 0x7f3670140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): picked mon.noname-b con 0x7f367009ad90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367009b160 0x7f3670094600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): picked mon.noname-c con 0x7f367009b160 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670094b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): picked mon.noname-a con 0x7f367009b530 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367008d8b0 con 0x7f367009b530 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367008de50 con 0x7f367009ad90 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367008e140 con 0x7f367009b160 2024-09-18T12:41:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f367619d640 10 monclient(hunting): authenticate will time out at 1694.204284s 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670094b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367009b160 0x7f3670094600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670094b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49374/0 (socket says 172.21.15.116:49374) 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367009b160 0x7f3670094600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40796/0 (socket says 172.21.15.116:40796) 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 1 -- 172.21.15.116:0/1041129034 learned_addr learned my addr 172.21.15.116:0/1041129034 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 10 monclient(hunting): get_auth_request con 0x7f367009b530 auth_method 0 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): get_auth_request con 0x7f367009b160 auth_method 0 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367009ad90 0x7f3670140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 10 monclient(hunting): get_auth_request con 0x7f367009ad90 auth_method 0 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient(hunting): handle_auth_done global_id 184232 payload 824 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367009ad90 msgr2=0x7f3670140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367009ad90 0x7f3670140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 msgr2=0x7f3670094b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670094b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 monclient: found mon.noname-c 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366effd640 1 -- 172.21.15.116:0/1041129034 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3668009450 con 0x7f367009b160 2024-09-18T12:41:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.883+0000 7f366ffff640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367009b530 0x7f3670094b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.885+0000 7f366effd640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367009b160 msgr2=0x7f3670094600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.885+0000 7f366effd640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367009b160 0x7f3670094600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f366400c540 tx=0x7f3664007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640117a0 con 0x7f367009b160 2024-09-18T12:41:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: dump: 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 5 monclient: authenticate success, global_id 184232 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 10 monclient: _renew_subs 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3664011920 con 0x7f367009b160 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3670141ad0 con 0x7f367009b160 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664011ce0 con 0x7f367009b160 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 10 monclient: dump: 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 10 monclient: _renew_subs 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367008fdc0 con 0x7f367009b160 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f366dffb640 4 set_mon_vals no callback set 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3664012630 con 0x7f367009b160 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 0x7f367008c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.886+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670065e00 con 0x7f36701415e0 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f36577fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f365c038490 0x7f365c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 0x7f367008c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient: get_auth_request con 0x7f36701415e0 auth_method 0 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f366404bfa0 con 0x7f367009b160 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366ffff640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f365c038490 0x7f365c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366ffff640 10 monclient: get_auth_request con 0x7f365c038490 auth_method 0 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 10 monclient(hunting): handle_auth_done global_id 174346 payload 824 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.887+0000 7f366f7fe640 1 -- 172.21.15.116:0/1041129034 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f367004bf60 con 0x7f36701415e0 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.889+0000 7f366f7fe640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 msgr2=0x7f367008c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.889+0000 7f366f7fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 0x7f367008c3d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36600024e0 tx=0x7f36600097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.889+0000 7f366ffff640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f365c038490 0x7f365c03a940 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3668009210 tx=0x7f3668032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.889+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660014880 con 0x7f36701415e0 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.889+0000 7f36577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.889+0000 7f36577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.889+0000 7f36577fe640 10 monclient: dump: 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.890+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f36600149e0 con 0x7f36701415e0 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.890+0000 7f36577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.890+0000 7f36577fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.890+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 msgr2=0x7f367008c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.890+0000 7f36577fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 0x7f367008c3d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36600024e0 tx=0x7f36600097e0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.890+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 msgr2=0x7f367008c3d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.890+0000 7f36577fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36701415e0 0x7f367008c3d0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36600024e0 tx=0x7f36600097e0 comp rx=0 tx=0).stop 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.895+0000 7f367619d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.895+0000 7f367619d640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 0x7f367008cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.895+0000 7f367619d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.895+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367008e280 con 0x7f367008c910 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 0x7f367008cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient: get_auth_request con 0x7f367008c910 auth_method 0 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 10 monclient(hunting): handle_auth_done global_id 174349 payload 824 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.896+0000 7f366f7fe640 1 -- 172.21.15.116:0/1041129034 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f367008d250 con 0x7f367008c910 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.898+0000 7f366f7fe640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 msgr2=0x7f367008cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.898+0000 7f366f7fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 0x7f367008cd10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3660019000 tx=0x7f3660017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.898+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366001e830 con 0x7f367008c910 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.898+0000 7f36577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.898+0000 7f36577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.898+0000 7f36577fe640 10 monclient: dump: 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f366001e990 con 0x7f367008c910 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f36577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f36577fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 msgr2=0x7f367008cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f36577fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 0x7f367008cd10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3660019000 tx=0x7f3660017000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f36577fe640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 msgr2=0x7f367008cd10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:06.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f36577fe640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f367008c910 0x7f367008cd10 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3660019000 tx=0x7f3660017000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 156, 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:06.913 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.914 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:06.916 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:06.917 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:06.926 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f365c038490 msgr2=0x7f365c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f365c038490 0x7f365c03a940 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3668009210 tx=0x7f3668032000 comp rx=0 tx=0).stop 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 10 monclient: shutdown 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367009b160 msgr2=0x7f3670094600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 --2- 172.21.15.116:0/1041129034 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f367009b160 0x7f3670094600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f366400c540 tx=0x7f3664007e10 comp rx=0 tx=0).stop 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f366ffff640 1 -- 172.21.15.116:0/1041129034 reap_dead start 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f366ffff640 1 -- 172.21.15.116:0/1041129034 reap_dead start 2024-09-18T12:41:06.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 shutdown_connections 2024-09-18T12:41:06.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 >> 172.21.15.116:0/1041129034 conn(0x7f36700678a0 msgr2=0x7f3670134b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:06.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 shutdown_connections 2024-09-18T12:41:06.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:06.899+0000 7f367619d640 1 -- 172.21.15.116:0/1041129034 wait complete. 2024-09-18T12:41:06.929 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:06.929 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:41:06.929 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:41:06.929 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:41:06.929 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:41:06.930 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:41:06.930 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:41:06.934 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.933+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.934 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.933+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210439467}) 2024-09-18T12:41:06.934 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.933+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4008226809}) 2024-09-18T12:41:06.934 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.933+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:06.934 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.933+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210439467}) 2024-09-18T12:41:06.934 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:06.933+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1895271814}) 2024-09-18T12:41:06.941 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:41:06.941 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:41:06.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.074+0000 7f86c951d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.074+0000 7f86c951d640 10 monclient: build_initial_monmap 2024-09-18T12:41:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.074+0000 7f86c951d640 10 monclient: monmap: 2024-09-18T12:41:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:07.074972+0000 2024-09-18T12:41:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:07.074972+0000 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:07.076 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:07.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.074+0000 7f86c951d640 1 Processor -- start 2024-09-18T12:41:07.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 1 -- start start 2024-09-18T12:41:07.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient: init 2024-09-18T12:41:07.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:07.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86c40981c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f86c4131da0 0x7f86c4136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient(hunting): picked mon.noname-a con 0x7f86c4131da0 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4132170 0x7f86c41390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient(hunting): picked mon.noname-b con 0x7f86c4132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c409ca60 con 0x7f86c40981c0 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c4131bd0 con 0x7f86c4131da0 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c409cca0 con 0x7f86c4132170 2024-09-18T12:41:07.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c951d640 10 monclient(hunting): authenticate will time out at 1694.396523s 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c37fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4132170 msgr2=0x7f86c41390c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4132170 0x7f86c41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c2ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f86c40981c0 0x7f86c4098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39532/0 (socket says 172.21.15.116:39532) 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c2ffd640 1 -- 172.21.15.116:0/1148525479 learned_addr learned my addr 172.21.15.116:0/1148525479 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 1 --2- 172.21.15.116:0/1148525479 >> v2:172.21.15.116:3300/0 conn(0x7f86c4131da0 0x7f86c4136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): get_auth_request con 0x7f86c4131da0 auth_method 0 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:07.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient(hunting): handle_auth_done global_id 194148 payload 274 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 1 -- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4132170 msgr2=0x7f86c41390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 1 --2- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4132170 0x7f86c41390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 1 -- 172.21.15.116:0/1148525479 >> v1:172.21.15.116:6789/0 conn(0x7f86c40981c0 legacy=0x7f86c4098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 1 monclient: found mon.noname-a 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.075+0000 7f86c27fc640 1 -- 172.21.15.116:0/1148525479 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c409e010 con 0x7f86c4131da0 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c27fc640 1 -- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c4131da0 msgr2=0x7f86c4136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:07.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c27fc640 1 --2- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c4131da0 0x7f86c4136b70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86c4132870 tx=0x7f86ac0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62ecea79d4a3afb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 1 -- 172.21.15.116:0/1148525479 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac02aea0 con 0x7f86c4131da0 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: dump: 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:07.087 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 1 -- 172.21.15.116:0/1148525479 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ac032440 con 0x7f86c4131da0 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c951d640 5 monclient: authenticate success, global_id 194148 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c951d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c951d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 1 -- 172.21.15.116:0/1148525479 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0327e0 con 0x7f86c4131da0 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c951d640 4 set_mon_vals no callback set 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:07.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.078+0000 7f86c1ffb640 10 monclient: dump: 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:07.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 10 monclient: shutdown 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 -- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c4131da0 msgr2=0x7f86c4136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 --2- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c4131da0 0x7f86c4136b70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86c4132870 tx=0x7f86ac0090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 -- 172.21.15.116:0/1148525479 shutdown_connections 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 --2- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4132170 0x7f86c41390c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 --2- 172.21.15.116:0/1148525479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c4131da0 0x7f86c4136b70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 -- 172.21.15.116:0/1148525479 >> 172.21.15.116:0/1148525479 conn(0x7f86c40a1e30 msgr2=0x7f86c40a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 -- 172.21.15.116:0/1148525479 shutdown_connections 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.079+0000 7f86c951d640 1 -- 172.21.15.116:0/1148525479 wait complete. 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient: build_initial_monmap 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient: monmap: 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:07.090 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 Processor -- start 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 -- start start 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient: init 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 0x7f86c41381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): picked mon.noname-a con 0x7f86c40981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4131da0 0x7f86c4118490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): picked mon.noname-c con 0x7f86c4131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c4132170 0x7f86c41189d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): picked mon.noname-b con 0x7f86c4132170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c411a960 con 0x7f86c40981c0 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4098cb0 con 0x7f86c4132170 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40a0a70 con 0x7f86c4131da0 2024-09-18T12:41:07.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c951d640 10 monclient(hunting): authenticate will time out at 1694.401937s 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c27fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4131da0 msgr2=0x7f86c4118490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c27fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4131da0 0x7f86c4118490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.080+0000 7f86c2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 0x7f86c41381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 0x7f86c41381e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49432/0 (socket says 172.21.15.116:49432) 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2357484409 learned_addr learned my addr 172.21.15.116:0/2357484409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): get_auth_request con 0x7f86c40981c0 auth_method 0 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c37fe640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c4132170 0x7f86c41189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c37fe640 10 monclient(hunting): get_auth_request con 0x7f86c4132170 auth_method 0 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient(hunting): handle_auth_done global_id 194151 payload 824 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4131da0 msgr2=0x7f86c4118490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4131da0 0x7f86c4118490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c4132170 msgr2=0x7f86c41189d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c4132170 0x7f86c41189d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 monclient: found mon.noname-a 2024-09-18T12:41:07.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:07.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:07.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.081+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2357484409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac009390 con 0x7f86c40981c0 2024-09-18T12:41:07.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c2ffd640 1 -- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 msgr2=0x7f86c41381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:07.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c2ffd640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 0x7f86c41381e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f86b400cca0 tx=0x7f86b40053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:07.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 1 -- 172.21.15.116:0/2357484409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b400aa60 con 0x7f86c40981c0 2024-09-18T12:41:07.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:07.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: dump: 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:07.098 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: _finish_auth 0 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c951d640 5 monclient: authenticate success, global_id 194151 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 1 -- 172.21.15.116:0/2357484409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b400abe0 con 0x7f86c40981c0 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 1 -- 172.21.15.116:0/2357484409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40070f0 con 0x7f86c40981c0 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c951d640 10 monclient: _renew_subs 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c4119830 con 0x7f86c40981c0 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:07.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86baffd640 10 monclient: dump: 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:07.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c17fa640 4 set_mon_vals no callback set 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c951d640 10 monclient: _renew_subs 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.083+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c4131780 con 0x7f86c40981c0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86baffd640 1 -- 172.21.15.116:0/2357484409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f86b4007520 con 0x7f86c40981c0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86c951d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86c951d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86c951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c404bf60 con 0x7f86c40981c0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86baffd640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86a4038440 0x7f86a403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86baffd640 1 -- 172.21.15.116:0/2357484409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f86b404c610 con 0x7f86c40981c0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86c27fc640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86a4038440 0x7f86a403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.084+0000 7f86c27fc640 10 monclient: get_auth_request con 0x7f86a4038440 auth_method 0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.087+0000 7f86c27fc640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86a4038440 0x7f86a403a8f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f86c4066560 tx=0x7f86ac032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.089+0000 7f86baffd640 1 -- 172.21.15.116:0/2357484409 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f86b401a360 con 0x7f86c40981c0 2024-09-18T12:41:07.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.089+0000 7f86baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:07.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.089+0000 7f86baffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.303+0000 7f86c951d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.303+0000 7f86c951d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.303+0000 7f86c951d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.303+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f86c4098ee0 con 0x7f86c40981c0 2024-09-18T12:41:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.303+0000 7f86baffd640 1 -- 172.21.15.116:0/2357484409 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f86b4010af0 con 0x7f86c40981c0 2024-09-18T12:41:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.303+0000 7f86baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.303+0000 7f86baffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:07.305 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86a4038440 msgr2=0x7f86a403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86a4038440 0x7f86a403a8f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f86c4066560 tx=0x7f86ac032000 comp rx=0 tx=0).stop 2024-09-18T12:41:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 10 monclient: shutdown 2024-09-18T12:41:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 msgr2=0x7f86c41381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 0x7f86c41381e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f86b400cca0 tx=0x7f86b40053f0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 shutdown_connections 2024-09-18T12:41:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86a4038440 0x7f86a403a8f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f86c4132170 0x7f86c41189d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86c4131da0 0x7f86c4118490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 --2- 172.21.15.116:0/2357484409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86c40981c0 0x7f86c41381e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 >> 172.21.15.116:0/2357484409 conn(0x7f86c40a1e30 msgr2=0x7f86c4096f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 shutdown_connections 2024-09-18T12:41:07.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:07.305+0000 7f86c951d640 1 -- 172.21.15.116:0/2357484409 wait complete. 2024-09-18T12:41:10.318 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.421+0000 7fa07b19d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.421+0000 7fa07b19d640 10 monclient: build_initial_monmap 2024-09-18T12:41:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.421+0000 7fa07b19d640 10 monclient: monmap: 2024-09-18T12:41:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:10.422 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:10.422197+0000 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:10.422197+0000 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:10.423 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:10.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 1 Processor -- start 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 1 -- start start 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient: init 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0740974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa07409a8a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 0x7fa074136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient(hunting): picked mon.noname-b con 0x7fa074134080 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07412b210 con 0x7fa07409a8a0 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0740978c0 con 0x7fa0740974f0 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07409ca20 con 0x7fa074134080 2024-09-18T12:41:10.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:10.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07b19d640 10 monclient(hunting): authenticate will time out at 1697.743893s 2024-09-18T12:41:10.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07973a640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 msgr2=0x7fa074136460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:10.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.422+0000 7fa07973a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 0x7fa074136460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:10.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.423+0000 7fa06bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa07409a8a0 0x7fa07409ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39544/0 (socket says 172.21.15.116:39544) 2024-09-18T12:41:10.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.423+0000 7fa06bfff640 1 -- 172.21.15.116:0/3097094411 learned_addr learned my addr 172.21.15.116:0/3097094411 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:10.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06bfff640 1 -- 172.21.15.116:0/3097094411 >> v1:172.21.15.116:6789/0 conn(0x7fa07409a8a0 legacy=0x7fa07409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 1 -- 172.21.15.116:0/3097094411 >> v1:172.21.15.144:6789/0 conn(0x7fa0740974f0 legacy=0x7fa07412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459299671 0 0) 0x7fa07412b210 con 0x7fa07409a8a0 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): my global_id is 194154 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05c0034e0 con 0x7fa07409a8a0 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754545959 0 0) 0x7fa05c0034e0 con 0x7fa07409a8a0 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 msgr2=0x7fa074136460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 0x7fa074136460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 >> v1:172.21.15.144:6789/0 conn(0x7fa0740974f0 legacy=0x7fa07412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:41:10.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa074098ad0 con 0x7fa07409a8a0 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fa0640017a0 con 0x7fa07409a8a0 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: dump: 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.435 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 >> v1:172.21.15.116:6789/0 conn(0x7fa07409a8a0 legacy=0x7fa07409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa05c003ad0 0x7fa05c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): picked mon.a con 0x7fa05c003ad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:10.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 0x7fa05c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): picked mon.c con 0x7fa074134080 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c004f20 0x7fa05c00cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): picked mon.b con 0x7fa05c004f20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07409ca20 con 0x7fa05c003ad0 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c00d020 con 0x7fa05c004f20 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c00d160 con 0x7fa074134080 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa05c003ad0 0x7fa05c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06bfff640 1 -- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 msgr2=0x7fa05c0047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:10.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): get_auth_request con 0x7fa05c003ad0 auth_method 0 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa06bfff640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 0x7fa05c0047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa07973a640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c004f20 0x7fa05c00cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 10 monclient(hunting): handle_auth_done global_id 194154 payload 293 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 1 -- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 msgr2=0x7fa05c0047c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.425+0000 7fa078f39640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa074134080 0x7fa05c0047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 1 -- 172.21.15.116:0/3097094411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c004f20 msgr2=0x7fa05c00cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:10.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c004f20 0x7fa05c00cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 10 monclient: _finish_hunting 0 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 1 monclient: found mon.a 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 1 -- 172.21.15.116:0/3097094411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c0034e0 con 0x7fa05c003ad0 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 10 monclient: _finish_auth 0 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa078f39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa07973a640 10 monclient: get_auth_request con 0x7fa05c004f20 auth_method 0 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa07b19d640 5 monclient: authenticate success, global_id 194154 2024-09-18T12:41:10.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa07973a640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c004f20 0x7fa05c00cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.426+0000 7fa07b19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa078f39640 1 -- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa05c003ad0 msgr2=0x7fa05c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa078f39640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa05c003ad0 0x7fa05c003ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa070001b40 tx=0x7fa070007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a37f01c9442b49 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0700070f0 con 0x7fa05c003ad0 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 10 monclient: dump: 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:10.440 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07002f070 con 0x7fa05c003ad0 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa07b19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:10.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa07b19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 1 -- 172.21.15.116:0/3097094411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070007290 con 0x7fa05c003ad0 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa07b19d640 4 set_mon_vals no callback set 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.428+0000 7fa06b7fe640 10 monclient: dump: 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:10.442 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:10.444 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 10 monclient: shutdown 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:10.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 1 -- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa05c003ad0 msgr2=0x7fa05c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 1 --2- 172.21.15.116:0/3097094411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa05c003ad0 0x7fa05c003ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa070001b40 tx=0x7fa070007c80 comp rx=0 tx=0).stop 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07973a640 1 -- 172.21.15.116:0/3097094411 reap_dead start 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 1 -- 172.21.15.116:0/3097094411 shutdown_connections 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 1 -- 172.21.15.116:0/3097094411 >> 172.21.15.116:0/3097094411 conn(0x7fa0740672b0 msgr2=0x7fa0740676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 1 -- 172.21.15.116:0/3097094411 shutdown_connections 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.429+0000 7fa07b19d640 1 -- 172.21.15.116:0/3097094411 wait complete. 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.430+0000 7fa07b19d640 10 monclient: build_initial_monmap 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.430+0000 7fa07b19d640 10 monclient: monmap: 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:10.446 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.430+0000 7fa07b19d640 1 Processor -- start 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.430+0000 7fa07b19d640 1 -- start start 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.430+0000 7fa07b19d640 10 monclient: init 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c003ad0 0x7fa0741251d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): picked mon.noname-b con 0x7fa05c003ad0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0740696a0 0x7fa074069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0740696a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 0x7fa074143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): picked mon.noname-a con 0x7fa074069fd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa074098300 con 0x7fa074069fd0 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07412a4a0 con 0x7fa05c003ad0 2024-09-18T12:41:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07406b280 con 0x7fa0740696a0 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07b19d640 10 monclient(hunting): authenticate will time out at 1697.752106s 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa06bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0740696a0 msgr2=0x7fa074069a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa06bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0740696a0 0x7fa074069a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 0x7fa074143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 0x7fa074143510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49462/0 (socket says 172.21.15.116:49462) 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 -- 172.21.15.116:0/3776263514 learned_addr learned my addr 172.21.15.116:0/3776263514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): get_auth_request con 0x7fa074069fd0 auth_method 0 2024-09-18T12:41:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa078f39640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c003ad0 0x7fa0741251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient(hunting): handle_auth_done global_id 194157 payload 824 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 -- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0740696a0 msgr2=0x7fa074069a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0740696a0 0x7fa074069a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 -- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c003ad0 msgr2=0x7fa0741251d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:10.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c003ad0 0x7fa0741251d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient: _finish_hunting 0 2024-09-18T12:41:10.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 monclient: found mon.noname-a 2024-09-18T12:41:10.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:10.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:10.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa07973a640 1 -- 172.21.15.116:0/3776263514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07406c5e0 con 0x7fa074069fd0 2024-09-18T12:41:10.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa078f39640 10 monclient: get_auth_request con 0x7fa05c003ad0 auth_method 0 2024-09-18T12:41:10.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.431+0000 7fa078f39640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c003ad0 0x7fa0741251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:41:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07973a640 1 -- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 msgr2=0x7fa074143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07973a640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 0x7fa074143510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa074065310 tx=0x7fa06c007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 1 -- 172.21.15.116:0/3776263514 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c00f680 con 0x7fa074069fd0 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: dump: 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:10.454 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: _finish_auth 0 2024-09-18T12:41:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07b19d640 5 monclient: authenticate success, global_id 194157 2024-09-18T12:41:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 1 -- 172.21.15.116:0/3776263514 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c00f820 con 0x7fa074069fd0 2024-09-18T12:41:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 1 -- 172.21.15.116:0/3776263514 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c00fbe0 con 0x7fa074069fd0 2024-09-18T12:41:10.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07b19d640 10 monclient: _renew_subs 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07b19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07406ab60 con 0x7fa074069fd0 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa068ff9640 10 monclient: dump: 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.456 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa06affd640 4 set_mon_vals no callback set 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07b19d640 10 monclient: _renew_subs 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07b19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.433+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07406f390 con 0x7fa074069fd0 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.434+0000 7fa068ff9640 1 -- 172.21.15.116:0/3776263514 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa06c00e5e0 con 0x7fa074069fd0 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.434+0000 7fa07b19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.434+0000 7fa07b19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:10.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.434+0000 7fa07b19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.434+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07404bf60 con 0x7fa074069fd0 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.435+0000 7fa068ff9640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa060038440 0x7fa06003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.435+0000 7fa068ff9640 1 -- 172.21.15.116:0/3776263514 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa06c006e40 con 0x7fa074069fd0 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.435+0000 7fa078f39640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa060038440 0x7fa06003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.435+0000 7fa078f39640 10 monclient: get_auth_request con 0x7fa060038440 auth_method 0 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.437+0000 7fa078f39640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa060038440 0x7fa06003a8f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa070001b10 tx=0x7fa070035000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.439+0000 7fa068ff9640 1 -- 172.21.15.116:0/3776263514 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa06c016af0 con 0x7fa074069fd0 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.439+0000 7fa068ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:10.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.439+0000 7fa068ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.655+0000 7fa07b19d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.655+0000 7fa07b19d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.655+0000 7fa07b19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.655+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa07412b180 con 0x7fa074069fd0 2024-09-18T12:41:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.655+0000 7fa068ff9640 1 -- 172.21.15.116:0/3776263514 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa06c016310 con 0x7fa074069fd0 2024-09-18T12:41:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.655+0000 7fa068ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:10.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.655+0000 7fa068ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:10.657 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:10.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa060038440 msgr2=0x7fa06003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:10.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa060038440 0x7fa06003a8f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa070001b10 tx=0x7fa070035000 comp rx=0 tx=0).stop 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 10 monclient: shutdown 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 msgr2=0x7fa074143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 0x7fa074143510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa074065310 tx=0x7fa06c007600 comp rx=0 tx=0).stop 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 shutdown_connections 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa060038440 0x7fa06003a8f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa074069fd0 0x7fa074143510 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa0740696a0 0x7fa074069a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 --2- 172.21.15.116:0/3776263514 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa05c003ad0 0x7fa0741251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 >> 172.21.15.116:0/3776263514 conn(0x7fa0740672b0 msgr2=0x7fa07409d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 shutdown_connections 2024-09-18T12:41:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:10.657+0000 7fa07b19d640 1 -- 172.21.15.116:0/3776263514 wait complete. 2024-09-18T12:41:13.669 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.772+0000 7fc24f499640 10 monclient: get_monmap_and_config 2024-09-18T12:41:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.772+0000 7fc24f499640 10 monclient: build_initial_monmap 2024-09-18T12:41:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.772+0000 7fc24f499640 10 monclient: monmap: 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:13.773513+0000 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:13.773513+0000 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:13.774 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.773+0000 7fc24f499640 1 Processor -- start 2024-09-18T12:41:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.773+0000 7fc24f499640 1 -- start start 2024-09-18T12:41:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.773+0000 7fc24f499640 10 monclient: init 2024-09-18T12:41:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-18T12:41:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc2480981c0 0x7fc248098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient(hunting): picked mon.noname-c con 0x7fc2480981c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248131da0 0x7fc248067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient(hunting): picked mon.noname-b con 0x7fc248131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc248132170 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc248098ad0 con 0x7fc248132170 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc248131bd0 con 0x7fc2480981c0 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24809ca60 con 0x7fc248131da0 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24f499640 10 monclient(hunting): authenticate will time out at 1701.095156s 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24ca34640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248131da0 msgr2=0x7fc248067110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24ca34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248131da0 0x7fc248067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24da36640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc248132170 0x7fc248067650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39556/0 (socket says 172.21.15.116:39556) 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24da36640 1 -- 172.21.15.116:0/469775830 learned_addr learned my addr 172.21.15.116:0/469775830 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 1 --2- 172.21.15.116:0/469775830 >> v2:172.21.15.144:3300/0 conn(0x7fc2480981c0 0x7fc248098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): get_auth_request con 0x7fc2480981c0 auth_method 0 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient(hunting): handle_auth_done global_id 174355 payload 274 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 1 -- 172.21.15.116:0/469775830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248131da0 msgr2=0x7fc248067110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:13.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 1 --2- 172.21.15.116:0/469775830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248131da0 0x7fc248067110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:13.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 1 -- 172.21.15.116:0/469775830 >> v1:172.21.15.116:6789/0 conn(0x7fc248132170 legacy=0x7fc248067650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:13.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient: _finish_hunting 0 2024-09-18T12:41:13.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 1 monclient: found mon.noname-c 2024-09-18T12:41:13.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:13.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:41:13.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.774+0000 7fc24d235640 1 -- 172.21.15.116:0/469775830 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24809cca0 con 0x7fc2480981c0 2024-09-18T12:41:13.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.777+0000 7fc24d235640 1 -- 172.21.15.116:0/469775830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2480981c0 msgr2=0x7fc248098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:13.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.777+0000 7fc24d235640 1 --2- 172.21.15.116:0/469775830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2480981c0 0x7fc248098590 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc244009850 tx=0x7fc2440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bec4b8f3cf906bf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:13.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 1 -- 172.21.15.116:0/469775830 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244032440 con 0x7fc2480981c0 2024-09-18T12:41:13.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: dump: 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: _finish_auth 0 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 1 -- 172.21.15.116:0/469775830 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2440325a0 con 0x7fc2480981c0 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 5 monclient: authenticate success, global_id 174355 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 4 set_mon_vals no callback set 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 1 -- 172.21.15.116:0/469775830 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244032980 con 0x7fc2480981c0 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc23ffff640 10 monclient: dump: 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 10 monclient: shutdown 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 -- 172.21.15.116:0/469775830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2480981c0 msgr2=0x7fc248098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 --2- 172.21.15.116:0/469775830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2480981c0 0x7fc248098590 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc244009850 tx=0x7fc2440090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:13.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 -- 172.21.15.116:0/469775830 shutdown_connections 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 --2- 172.21.15.116:0/469775830 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248131da0 0x7fc248067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 --2- 172.21.15.116:0/469775830 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2480981c0 0x7fc248098590 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 -- 172.21.15.116:0/469775830 >> 172.21.15.116:0/469775830 conn(0x7fc2480a1e30 msgr2=0x7fc2480a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 -- 172.21.15.116:0/469775830 shutdown_connections 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.778+0000 7fc24f499640 1 -- 172.21.15.116:0/469775830 wait complete. 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.779+0000 7fc24f499640 10 monclient: build_initial_monmap 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.779+0000 7fc24f499640 10 monclient: monmap: 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.779+0000 7fc24f499640 1 Processor -- start 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.779+0000 7fc24f499640 1 -- start start 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.779+0000 7fc24f499640 10 monclient: init 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 0x7fc248141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): picked mon.noname-a con 0x7fc2480981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc248131da0 0x7fc248141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): picked mon.noname-b con 0x7fc248131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248132170 0x7fc248142100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): picked mon.noname-c con 0x7fc248132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248098cb0 con 0x7fc2480981c0 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24811fa20 con 0x7fc248131da0 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2480a0c00 con 0x7fc248132170 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24f499640 10 monclient(hunting): authenticate will time out at 1701.101441s 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24da36640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248132170 msgr2=0x7fc248142100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24da36640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248132170 0x7fc248142100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 0x7fc248141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 0x7fc248141680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49464/0 (socket says 172.21.15.116:49464) 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 -- 172.21.15.116:0/1258728773 learned_addr learned my addr 172.21.15.116:0/1258728773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): get_auth_request con 0x7fc2480981c0 auth_method 0 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24ca34640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc248131da0 0x7fc248141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24ca34640 10 monclient(hunting): get_auth_request con 0x7fc248131da0 auth_method 0 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24ca34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24ca34640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24ca34640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient(hunting): handle_auth_done global_id 194160 payload 824 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 -- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248132170 msgr2=0x7fc248142100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248132170 0x7fc248142100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 -- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc248131da0 msgr2=0x7fc248141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc248131da0 0x7fc248141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient: _finish_hunting 0 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 monclient: found mon.noname-a 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.780+0000 7fc24d235640 1 -- 172.21.15.116:0/1258728773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc244009390 con 0x7fc2480981c0 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24d235640 1 -- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 msgr2=0x7fc248141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24d235640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 0x7fc248141680 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc244009360 tx=0x7fc244032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244031990 con 0x7fc2480981c0 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: dump: 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc244031af0 con 0x7fc2480981c0 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24f499640 5 monclient: authenticate success, global_id 194160 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244031e70 con 0x7fc2480981c0 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24f499640 10 monclient: _renew_subs 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24f499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc248122810 con 0x7fc2480981c0 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23d7fa640 10 monclient: dump: 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc23f7fe640 4 set_mon_vals no callback set 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24f499640 10 monclient: _renew_subs 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24f499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.783+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc248120320 con 0x7fc2480981c0 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc244039040 con 0x7fc2480981c0 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc23d7fa640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc22c038440 0x7fc22c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc24406c0d0 con 0x7fc2480981c0 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc24f499640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc24f499640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc24f499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24804bf60 con 0x7fc2480981c0 2024-09-18T12:41:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc24ca34640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc22c038440 0x7fc22c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.784+0000 7fc24ca34640 10 monclient: get_auth_request con 0x7fc22c038440 auth_method 0 2024-09-18T12:41:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.787+0000 7fc24ca34640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc22c038440 0x7fc22c03a8f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc24811f620 tx=0x7fc238006ef0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.787+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:41:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.789+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc24403f020 con 0x7fc2480981c0 2024-09-18T12:41:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.789+0000 7fc23d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:13.789+0000 7fc23d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:14.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.003+0000 7fc24f499640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:14.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.003+0000 7fc24f499640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:14.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.003+0000 7fc24f499640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:14.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.003+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc248098ee0 con 0x7fc2480981c0 2024-09-18T12:41:14.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.004+0000 7fc23d7fa640 1 -- 172.21.15.116:0/1258728773 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fc244037070 con 0x7fc2480981c0 2024-09-18T12:41:14.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.004+0000 7fc23d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:14.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.004+0000 7fc23d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:14.005 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:14.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc22c038440 msgr2=0x7fc22c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:14.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc22c038440 0x7fc22c03a8f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc24811f620 tx=0x7fc238006ef0 comp rx=0 tx=0).stop 2024-09-18T12:41:14.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 10 monclient: shutdown 2024-09-18T12:41:14.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:14.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 msgr2=0x7fc248141680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 0x7fc248141680 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc244009360 tx=0x7fc244032000 comp rx=0 tx=0).stop 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 shutdown_connections 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc22c038440 0x7fc22c03a8f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc248132170 0x7fc248142100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc248131da0 0x7fc248141bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 --2- 172.21.15.116:0/1258728773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2480981c0 0x7fc248141680 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 >> 172.21.15.116:0/1258728773 conn(0x7fc2480a1e30 msgr2=0x7fc24809bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 shutdown_connections 2024-09-18T12:41:14.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:14.006+0000 7fc24f499640 1 -- 172.21.15.116:0/1258728773 wait complete. 2024-09-18T12:41:17.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:17.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.119+0000 7f12fcd1d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.119+0000 7f12fcd1d640 10 monclient: build_initial_monmap 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.119+0000 7f12fcd1d640 10 monclient: monmap: 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:17.120619+0000 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:17.120619+0000 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:17.121 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:17.122 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:17.122 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.120+0000 7f12fcd1d640 1 Processor -- start 2024-09-18T12:41:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.120+0000 7f12fcd1d640 1 -- start start 2024-09-18T12:41:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.120+0000 7f12fcd1d640 10 monclient: init 2024-09-18T12:41:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient(hunting): picked mon.noname-a con 0x7f12f809a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12f80971e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 0x7f12f812fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient(hunting): picked mon.noname-b con 0x7f12f80975e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f806ee10 con 0x7f12f80971e0 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f809cb00 con 0x7f12f809a510 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f809a8e0 con 0x7f12f80975e0 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12fcd1d640 10 monclient(hunting): authenticate will time out at 1704.442273s 2024-09-18T12:41:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f6d9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 msgr2=0x7f12f812fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f6d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 0x7f12f812fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47676/0 (socket says 172.21.15.116:47676) 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 learned_addr learned my addr 172.21.15.116:0/3285491667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12f809a510 auth_method 0 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.121+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.123+0000 7f12effff640 1 -- 172.21.15.116:0/3285491667 >> v1:172.21.15.144:6789/0 conn(0x7f12f80971e0 legacy=0x7f12f812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:17.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f6d9d640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 msgr2=0x7f12f812fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:17.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f6d9d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 0x7f12f812fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:41:17.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12f809a510 auth_method 0 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:17.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.322+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f6d9d640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 msgr2=0x7f12f812fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f6d9d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 0x7f12f812fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12f809a510 auth_method 0 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:17.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:17.723+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f6d9d640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 msgr2=0x7f12f812fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f6d9d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 0x7f12f812fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12f809a510 auth_method 0 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:18.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:18.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:18.524+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 10 monclient(hunting): tick 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 monclient(hunting): continuing hunt 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 msgr2=0x7f12f812fd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12f80975e0 0x7f12f812fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 msgr2=0x7f12f8096ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.116:3300/0 conn(0x7f12f809a510 0x7f12f8096ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 -- 172.21.15.116:0/3285491667 >> v1:172.21.15.144:6789/0 conn(0x7f12f80971e0 legacy=0x7f12f812f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 10 monclient: _add_conns ranks=[4,0] 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 10 monclient(hunting): picked mon.noname-c con 0x7f12d8003720 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12d80061c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 -- 172.21.15.116:0/3285491667 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f809d590 con 0x7f12d80061c0 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 1 -- 172.21.15.116:0/3285491667 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f809cb00 con 0x7f12d8003720 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f4d99640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12d8003720 auth_method 0 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.120+0000 7f12f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.121+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.121+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.121+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.121+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.123+0000 7f12effff640 1 -- 172.21.15.116:0/3285491667 >> v1:172.21.15.116:6789/0 conn(0x7f12d80061c0 legacy=0x7f12d8008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12d8003720 auth_method 0 2024-09-18T12:41:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.322+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:41:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12d8003720 auth_method 0 2024-09-18T12:41:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:20.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:20.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:20.723+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:41:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12d8003720 auth_method 0 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.525+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:41:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.558+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564548361 0 0) 0x7f12f809d590 con 0x7f12d80061c0 2024-09-18T12:41:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.558+0000 7f12f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.558+0000 7f12f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.558+0000 7f12f5d9b640 10 monclient(hunting): my global_id is 194163 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.558+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e40034e0 con 0x7f12d80061c0 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867086075 0 0) 0x7f12e40034e0 con 0x7f12d80061c0 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 msgr2=0x7f12d8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 --2- 172.21.15.116:0/3285491667 >> v2:172.21.15.144:3300/0 conn(0x7f12d8003720 0x7f12d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d800af20 con 0x7f12d80061c0 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: _finish_auth 0 2024-09-18T12:41:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f12e00017b0 con 0x7f12d80061c0 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: dump: 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 >> v1:172.21.15.116:6789/0 conn(0x7f12d80061c0 legacy=0x7f12d8008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 1 -- 172.21.15.116:0/3285491667 reap_dead start 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e40039d0 0x7f12e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): picked mon.a con 0x7f12e40039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e4004780 0x7f12e4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): picked mon.c con 0x7f12e4004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): picked mon.b con 0x7f12e4005250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f809cb00 con 0x7f12e40039d0 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e400c1c0 con 0x7f12e4005250 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e400c380 con 0x7f12e4004780 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f5d9b640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12effff640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e4004780 msgr2=0x7f12e4004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12effff640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e4004780 0x7f12e4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e40039d0 0x7f12e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): get_auth_request con 0x7f12e40039d0 auth_method 0 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 10 monclient(hunting): get_auth_request con 0x7f12e4005250 auth_method 0 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 10 monclient(hunting): handle_auth_done global_id 194163 payload 293 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e4004780 msgr2=0x7f12e4004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e4004780 0x7f12e4004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 msgr2=0x7f12e400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.559+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12e400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12f659c640 10 monclient: _finish_hunting 0 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12f659c640 1 monclient: found mon.a 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12f659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e40034e0 con 0x7f12e40039d0 2024-09-18T12:41:21.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12f659c640 10 monclient: _finish_auth 0 2024-09-18T12:41:21.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12f659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:21.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12fcd1d640 5 monclient: authenticate success, global_id 194163 2024-09-18T12:41:21.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.560+0000 7f12fcd1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.564+0000 7f12f659c640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e40039d0 msgr2=0x7f12e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.564+0000 7f12f659c640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e40039d0 0x7f12e4003dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006f60 tx=0x7f12e8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43bb4e7c6d4f6fea server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e802c890 con 0x7f12e40039d0 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 10 monclient: dump: 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12fcd1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e800c040 con 0x7f12e40039d0 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 1 -- 172.21.15.116:0/3285491667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e802cc60 con 0x7f12e40039d0 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12f5d9b640 10 monclient: dump: 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12fcd1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.565+0000 7f12fcd1d640 4 set_mon_vals no callback set 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 10 monclient: shutdown 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 -- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e40039d0 msgr2=0x7f12e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e40039d0 0x7f12e4003dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006f60 tx=0x7f12e8008040 comp rx=0 tx=0).stop 2024-09-18T12:41:21.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 -- 172.21.15.116:0/3285491667 shutdown_connections 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12e400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e4004780 0x7f12e4004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/3285491667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e40039d0 0x7f12e4003dc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 -- 172.21.15.116:0/3285491667 >> 172.21.15.116:0/3285491667 conn(0x7f12f80672b0 msgr2=0x7f12f80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 -- 172.21.15.116:0/3285491667 shutdown_connections 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.566+0000 7f12fcd1d640 1 -- 172.21.15.116:0/3285491667 wait complete. 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.567+0000 7f12fcd1d640 10 monclient: build_initial_monmap 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.567+0000 7f12fcd1d640 10 monclient: monmap: 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:21.581 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.567+0000 7f12fcd1d640 1 Processor -- start 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.567+0000 7f12fcd1d640 1 -- start start 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.567+0000 7f12fcd1d640 10 monclient: init 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e40039d0 0x7f12f813d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): picked mon.noname-c con 0x7f12e40039d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 0x7f12f813ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): picked mon.noname-a con 0x7f12e4004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12f659c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e40039d0 msgr2=0x7f12f813d610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12f659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e40039d0 0x7f12f813d610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 0x7f12f813ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12f8140b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): picked mon.noname-b con 0x7f12e4005250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f81209b0 con 0x7f12e4004780 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 0x7f12f813ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47718/0 (socket says 172.21.15.116:47718) 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 1 -- 172.21.15.116:0/1234987300 learned_addr learned my addr 172.21.15.116:0/1234987300 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80997a0 con 0x7f12e4005250 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f8121d60 con 0x7f12e40039d0 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12fcd1d640 10 monclient(hunting): authenticate will time out at 1708.890584s 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 10 monclient(hunting): get_auth_request con 0x7f12e4004780 auth_method 0 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12f6d9d640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12f8140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12f6d9d640 10 monclient(hunting): get_auth_request con 0x7f12e4005250 auth_method 0 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.568+0000 7f12f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 10 monclient(hunting): handle_auth_done global_id 194166 payload 824 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 1 -- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e40039d0 msgr2=0x7f12f813d610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e40039d0 0x7f12f813d610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 1 -- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 msgr2=0x7f12f8140b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12f8140b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:21.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 1 monclient: found mon.noname-a 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.569+0000 7f12effff640 1 -- 172.21.15.116:0/1234987300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8122040 con 0x7f12e4004780 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12effff640 1 -- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 msgr2=0x7f12f813ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12effff640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 0x7f12f813ccc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000f60 tx=0x7f12e0001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 1 -- 172.21.15.116:0/1234987300 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e00075e0 con 0x7f12e4004780 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: dump: 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 1 -- 172.21.15.116:0/1234987300 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0005780 con 0x7f12e4004780 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 1 -- 172.21.15.116:0/1234987300 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0005b40 con 0x7f12e4004780 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12fcd1d640 5 monclient: authenticate success, global_id 194166 2024-09-18T12:41:21.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12fcd1d640 10 monclient: _renew_subs 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12fcd1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f8122320 con 0x7f12e4004780 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12ee7fc640 10 monclient: dump: 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12f559a640 4 set_mon_vals no callback set 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12fcd1d640 10 monclient: _renew_subs 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12fcd1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.571+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f813d3a0 con 0x7f12e4004780 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.572+0000 7f12ee7fc640 1 -- 172.21.15.116:0/1234987300 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f12e0012400 con 0x7f12e4004780 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.572+0000 7f12fcd1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.572+0000 7f12fcd1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.572+0000 7f12fcd1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.572+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f804bf60 con 0x7f12e4004780 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.573+0000 7f12ee7fc640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f12d8038cd0 0x7f12d803b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.573+0000 7f12ee7fc640 1 -- 172.21.15.116:0/1234987300 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f12e004b8c0 con 0x7f12e4004780 2024-09-18T12:41:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.573+0000 7f12f659c640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f12d8038cd0 0x7f12d803b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.573+0000 7f12f659c640 10 monclient: get_auth_request con 0x7f12d8038cd0 auth_method 0 2024-09-18T12:41:21.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.575+0000 7f12f659c640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f12d8038cd0 0x7f12d803b180 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006fd0 tx=0x7f12e8036000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.577+0000 7f12ee7fc640 1 -- 172.21.15.116:0/1234987300 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f12e001b3b0 con 0x7f12e4004780 2024-09-18T12:41:21.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.577+0000 7f12ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:21.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.577+0000 7f12ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:21.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.792+0000 7f12fcd1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.792+0000 7f12fcd1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.792+0000 7f12fcd1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.792+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f12f806e640 con 0x7f12e4004780 2024-09-18T12:41:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.793+0000 7f12ee7fc640 1 -- 172.21.15.116:0/1234987300 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f12e0017070 con 0x7f12e4004780 2024-09-18T12:41:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.793+0000 7f12ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.793+0000 7f12ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:21.795 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":160,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f12d8038cd0 msgr2=0x7f12d803b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f12d8038cd0 0x7f12d803b180 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006fd0 tx=0x7f12e8036000 comp rx=0 tx=0).stop 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 10 monclient: shutdown 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 msgr2=0x7f12f813ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 0x7f12f813ccc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000f60 tx=0x7f12e0001e40 comp rx=0 tx=0).stop 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 shutdown_connections 2024-09-18T12:41:21.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f12d8038cd0 0x7f12d803b180 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e4005250 0x7f12f8140b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004780 0x7f12f813ccc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 --2- 172.21.15.116:0/1234987300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f12e40039d0 0x7f12f813d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 >> 172.21.15.116:0/1234987300 conn(0x7f12f80672b0 msgr2=0x7f12f8069ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:21.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 shutdown_connections 2024-09-18T12:41:21.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.795+0000 7f12fcd1d640 1 -- 172.21.15.116:0/1234987300 wait complete. 2024-09-18T12:41:21.807 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:41:21.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:41:21.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.900+0000 7fa98539d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.900+0000 7fa98539d640 10 monclient: build_initial_monmap 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.900+0000 7fa98539d640 10 monclient: monmap: 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:21.901820+0000 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:21.901820+0000 2024-09-18T12:41:21.902 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.900+0000 7fa98539d640 1 Processor -- start 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.900+0000 7fa98539d640 1 -- start start 2024-09-18T12:41:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.900+0000 7fa98539d640 10 monclient: init 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9801318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 0x7fa98009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient(hunting): picked mon.noname-b con 0x7fa98009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa980136c20 0x7fa980139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient(hunting): picked mon.noname-c con 0x7fa980136c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa980132640 con 0x7fa9801318f0 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa980131d00 con 0x7fa980136c20 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98009fbf0 con 0x7fa98009ea50 2024-09-18T12:41:21.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:21.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa98539d640 10 monclient(hunting): authenticate will time out at 1709.223342s 2024-09-18T12:41:21.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa97e7fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 msgr2=0x7fa98009ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:21.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa97e7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 0x7fa98009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:21.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa97effd640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa9801318f0 0x7fa980131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:42028/0 (socket says 172.21.15.116:42028) 2024-09-18T12:41:21.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.901+0000 7fa97effd640 1 -- 172.21.15.116:0/1314166509 learned_addr learned my addr 172.21.15.116:0/1314166509 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:21.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.903+0000 7fa97effd640 1 -- 172.21.15.116:0/1314166509 >> v1:172.21.15.144:6789/0 conn(0x7fa9801318f0 legacy=0x7fa980131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:21.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97f7fe640 1 --2- 172.21.15.116:0/1314166509 >> v2:172.21.15.144:3300/0 conn(0x7fa980136c20 0x7fa980139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97f7fe640 10 monclient(hunting): get_auth_request con 0x7fa980136c20 auth_method 0 2024-09-18T12:41:21.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825560251 0 0) 0x7fa980132640 con 0x7fa9801318f0 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): my global_id is 174364 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9680034e0 con 0x7fa9801318f0 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721913735 0 0) 0x7fa9680034e0 con 0x7fa9801318f0 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 msgr2=0x7fa98009ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 0x7fa98009ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 >> v2:172.21.15.144:3300/0 conn(0x7fa980136c20 msgr2=0x7fa980139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 --2- 172.21.15.116:0/1314166509 >> v2:172.21.15.144:3300/0 conn(0x7fa980136c20 0x7fa980139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: _finish_hunting 0 2024-09-18T12:41:21.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98009fe30 con 0x7fa9801318f0 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: _finish_auth 0 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fa970001860 con 0x7fa9801318f0 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa98539d640 5 monclient: authenticate success, global_id 174364 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa98539d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: dump: 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.923 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 >> v1:172.21.15.144:6789/0 conn(0x7fa9801318f0 legacy=0x7fa980131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:41:21.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9680039d0 0x7fa968003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): picked mon.a con 0x7fa9680039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa968004780 0x7fa968004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): picked mon.b con 0x7fa968004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 0x7fa96800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): picked mon.c con 0x7fa98009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa980131d00 con 0x7fa9680039d0 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98009fbf0 con 0x7fa968004780 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa968005560 con 0x7fa98009ea50 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97dffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:41:21.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa98539d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97f7fe640 1 -- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 msgr2=0x7fa96800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97f7fe640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 0x7fa96800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9680039d0 0x7fa968003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): get_auth_request con 0x7fa9680039d0 auth_method 0 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient(hunting): handle_auth_done global_id 174364 payload 293 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 1 -- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 msgr2=0x7fa96800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa98009ea50 0x7fa96800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 1 -- 172.21.15.116:0/1314166509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa968004780 msgr2=0x7fa968004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa968004780 0x7fa968004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 1 monclient: found mon.a 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 1 -- 172.21.15.116:0/1314166509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680034e0 con 0x7fa9680039d0 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 10 monclient: _finish_auth 0 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.904+0000 7fa97effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97effd640 1 -- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9680039d0 msgr2=0x7fa968003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97effd640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9680039d0 0x7fa968003de0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c5c0 tx=0x7fa970007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72248f62b3f2974c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970005890 con 0x7fa9680039d0 2024-09-18T12:41:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 10 monclient: dump: 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa98539d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa970005a10 con 0x7fa9680039d0 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa98539d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 1 -- 172.21.15.116:0/1314166509 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970005dd0 con 0x7fa9680039d0 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa98539d640 4 set_mon_vals no callback set 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.907+0000 7fa97dffb640 10 monclient: dump: 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 10 monclient: shutdown 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 1 -- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9680039d0 msgr2=0x7fa968003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 1 --2- 172.21.15.116:0/1314166509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9680039d0 0x7fa968003de0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c5c0 tx=0x7fa970007b90 comp rx=0 tx=0).stop 2024-09-18T12:41:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa97f7fe640 1 -- 172.21.15.116:0/1314166509 reap_dead start 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 1 -- 172.21.15.116:0/1314166509 shutdown_connections 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 1 -- 172.21.15.116:0/1314166509 >> 172.21.15.116:0/1314166509 conn(0x7fa980067110 msgr2=0x7fa980067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 1 -- 172.21.15.116:0/1314166509 shutdown_connections 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.908+0000 7fa98539d640 1 -- 172.21.15.116:0/1314166509 wait complete. 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient: build_initial_monmap 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient: monmap: 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 Processor -- start 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 -- start start 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient: init 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9680039d0 0x7fa9801308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9680039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa980130e10 0x7fa980099670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): picked mon.noname-c con 0x7fa980130e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa980099bb0 0x7fa98009a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): picked mon.noname-a con 0x7fa980099bb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98011d0a0 con 0x7fa980099bb0 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98011c9e0 con 0x7fa9680039d0 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98009bf10 con 0x7fa980130e10 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa98539d640 10 monclient(hunting): authenticate will time out at 1709.231971s 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa97e7fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa980130e10 msgr2=0x7fa980099670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.909+0000 7fa97e7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa980130e10 0x7fa980099670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa980099bb0 0x7fa98009a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa980099bb0 0x7fa98009a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47738/0 (socket says 172.21.15.116:47738) 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 -- 172.21.15.116:0/650518320 learned_addr learned my addr 172.21.15.116:0/650518320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): get_auth_request con 0x7fa980099bb0 auth_method 0 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9680039d0 0x7fa9801308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 10 monclient(hunting): get_auth_request con 0x7fa9680039d0 auth_method 0 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient(hunting): handle_auth_done global_id 194169 payload 824 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa980130e10 msgr2=0x7fa980099670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa980130e10 0x7fa980099670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9680039d0 msgr2=0x7fa9801308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa9680039d0 0x7fa9801308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 monclient: found mon.noname-a 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.910+0000 7fa97f7fe640 1 -- 172.21.15.116:0/650518320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9800a49c0 con 0x7fa980099bb0 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa97f7fe640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa980099bb0 msgr2=0x7fa98009a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa97f7fe640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa980099bb0 0x7fa98009a050 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa980068d40 tx=0x7fa97800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978010070 con 0x7fa980099bb0 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: dump: 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: _finish_auth 0 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97800b4f0 con 0x7fa980099bb0 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa98539d640 5 monclient: authenticate success, global_id 194169 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa98539d640 10 monclient: _renew_subs 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa98539d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98009b9a0 con 0x7fa980099bb0 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780098d0 con 0x7fa980099bb0 2024-09-18T12:41:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa966ffd640 10 monclient: dump: 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa97d7fa640 4 set_mon_vals no callback set 2024-09-18T12:41:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa98539d640 10 monclient: _renew_subs 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa98539d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.912+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98011baf0 con 0x7fa980099bb0 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa98539d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa98539d640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 0x7fa98009a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa98539d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa980065e00 con 0x7fa98002aa40 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa978003020 con 0x7fa980099bb0 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 0x7fa98009a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 10 monclient: get_auth_request con 0x7fa98002aa40 auth_method 0 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.913+0000 7fa966ffd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa96c038490 0x7fa96c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.914+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa97804b160 con 0x7fa980099bb0 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.914+0000 7fa97effd640 10 monclient(hunting): handle_auth_done global_id 194172 payload 824 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.914+0000 7fa97effd640 1 -- 172.21.15.116:0/650518320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa98002ae10 con 0x7fa98002aa40 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.914+0000 7fa97e7fc640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa96c038490 0x7fa96c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.914+0000 7fa97e7fc640 10 monclient: get_auth_request con 0x7fa96c038490 auth_method 0 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa97effd640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 msgr2=0x7fa98009a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa97effd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 0x7fa98009a590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa970033000 tx=0x7fa970034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97003f030 con 0x7fa98002aa40 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 10 monclient: dump: 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa970040040 con 0x7fa98002aa40 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:21.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 msgr2=0x7fa98009a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 0x7fa98009a590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa970033000 tx=0x7fa970034000 comp rx=0 tx=0).stop 2024-09-18T12:41:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 msgr2=0x7fa98009a590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.916+0000 7fa966ffd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98002aa40 0x7fa98009a590 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa970033000 tx=0x7fa970034000 comp rx=0 tx=0).stop 2024-09-18T12:41:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.917+0000 7fa97e7fc640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa96c038490 0x7fa96c03a940 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa98009ae50 tx=0x7fa974006cb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa98539d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa98539d640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 0x7fa98005e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa98539d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98004bf60 con 0x7fa98005e310 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 0x7fa98005e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient: get_auth_request con 0x7fa98005e310 auth_method 0 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 10 monclient(hunting): handle_auth_done global_id 194175 payload 824 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.922+0000 7fa97effd640 1 -- 172.21.15.116:0/650518320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa98009aad0 con 0x7fa98005e310 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.924+0000 7fa97effd640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 msgr2=0x7fa98005e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.924+0000 7fa97effd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 0x7fa98005e6e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c5c0 tx=0x7fa970042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.924+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97002e440 con 0x7fa98005e310 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.924+0000 7fa966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.924+0000 7fa966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.924+0000 7fa966ffd640 10 monclient: dump: 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fa97002e5a0 con 0x7fa98005e310 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa966ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa966ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 msgr2=0x7fa98005e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa966ffd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 0x7fa98005e6e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c5c0 tx=0x7fa970042000 comp rx=0 tx=0).stop 2024-09-18T12:41:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa966ffd640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 msgr2=0x7fa98005e6e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa966ffd640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98005e310 0x7fa98005e6e0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c5c0 tx=0x7fa970042000 comp rx=0 tx=0).stop 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 160, 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:21.947 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:21.948 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:21.949 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:21.950 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:21.951 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:21.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:21.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:21.954 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.955 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:21.956 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:21.958 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:21.959 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:21.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:21.959 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:21.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:21.959 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:21.959 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:21.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa96c038490 msgr2=0x7fa96c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa96c038490 0x7fa96c03a940 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa98009ae50 tx=0x7fa974006cb0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 10 monclient: shutdown 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa980099bb0 msgr2=0x7fa98009a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 --2- 172.21.15.116:0/650518320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa980099bb0 0x7fa98009a050 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa980068d40 tx=0x7fa97800b7e0 comp rx=0 tx=0).stop 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa97f7fe640 1 -- 172.21.15.116:0/650518320 reap_dead start 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa97f7fe640 1 -- 172.21.15.116:0/650518320 reap_dead start 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 shutdown_connections 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 >> 172.21.15.116:0/650518320 conn(0x7fa980067110 msgr2=0x7fa9800a33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 shutdown_connections 2024-09-18T12:41:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:21.925+0000 7fa98539d640 1 -- 172.21.15.116:0/650518320 wait complete. 2024-09-18T12:41:21.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:41:21.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:41:21.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:41:22.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.054+0000 7fa6fd620640 10 monclient: get_monmap_and_config 2024-09-18T12:41:22.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.054+0000 7fa6fd620640 10 monclient: build_initial_monmap 2024-09-18T12:41:22.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.054+0000 7fa6fd620640 10 monclient: monmap: 2024-09-18T12:41:22.056 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:22.056015+0000 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:22.056015+0000 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:22.057 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.054+0000 7fa6fd620640 1 Processor -- start 2024-09-18T12:41:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 1 -- start start 2024-09-18T12:41:22.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient: init 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa6f8131980 0x7fa6f8131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f8131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6f809fe10 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa6f809f140 0x7fa6f8138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f809f140 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f80a14d0 con 0x7fa6f809fe10 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f81325b0 con 0x7fa6f8131980 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:22.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f80a11f0 con 0x7fa6f809f140 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6fd620640 10 monclient(hunting): authenticate will time out at 1709.377498s 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa6f8131980 0x7fa6f8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6ee7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa6f809fe10 0x7fa6f809ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56092/0 (socket says 172.21.15.116:56092) 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6ee7fc640 1 -- 172.21.15.116:0/3339104163 learned_addr learned my addr 172.21.15.116:0/3339104163 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): get_auth_request con 0x7fa6f8131980 auth_method 0 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:22.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_done global_id 194178 payload 274 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3339104163 >> v2:172.21.15.144:3300/0 conn(0x7fa6f809f140 msgr2=0x7fa6f8138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f77fe640 1 --2- 172.21.15.116:0/3339104163 >> v2:172.21.15.144:3300/0 conn(0x7fa6f809f140 0x7fa6f8138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 1 --2- 172.21.15.116:0/3339104163 >> v2:172.21.15.144:3300/0 conn(0x7fa6f809f140 0x7fa6f8138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3339104163 >> v1:172.21.15.116:6789/0 conn(0x7fa6f809fe10 legacy=0x7fa6f809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 1 monclient: found mon.noname-a 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:22.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.055+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3339104163 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f8132290 con 0x7fa6f8131980 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.058+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3339104163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 msgr2=0x7fa6f8131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.058+0000 7fa6f6ffd640 1 --2- 172.21.15.116:0/3339104163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8131d50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806a9f0 tx=0x7fa6e80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf728525f13fc08d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 1 -- 172.21.15.116:0/3339104163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8032440 con 0x7fa6f8131980 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: dump: 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:22.066 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:22.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 1 -- 172.21.15.116:0/3339104163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e80325a0 con 0x7fa6f8131980 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 5 monclient: authenticate success, global_id 194178 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 1 -- 172.21.15.116:0/3339104163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8032980 con 0x7fa6f8131980 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6f67fc640 10 monclient: dump: 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.068 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 4 set_mon_vals no callback set 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 10 monclient: shutdown 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:22.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 -- 172.21.15.116:0/3339104163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 msgr2=0x7fa6f8131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3339104163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8131d50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806a9f0 tx=0x7fa6e80090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 -- 172.21.15.116:0/3339104163 shutdown_connections 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3339104163 >> v2:172.21.15.144:3300/0 conn(0x7fa6f809f140 0x7fa6f8138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3339104163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8131d50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 -- 172.21.15.116:0/3339104163 >> 172.21.15.116:0/3339104163 conn(0x7fa6f80a5050 msgr2=0x7fa6f80a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 -- 172.21.15.116:0/3339104163 shutdown_connections 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.059+0000 7fa6fd620640 1 -- 172.21.15.116:0/3339104163 wait complete. 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.060+0000 7fa6fd620640 10 monclient: build_initial_monmap 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.060+0000 7fa6fd620640 10 monclient: monmap: 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:22.070 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.060+0000 7fa6fd620640 1 Processor -- start 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.060+0000 7fa6fd620640 1 -- start start 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.060+0000 7fa6fd620640 10 monclient: init 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f809f140 0x7fa6f81347c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f809f140 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f809fe10 0x7fa6f8134d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f809fe10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f8131980 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8136600 con 0x7fa6f8131980 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8069eb0 con 0x7fa6f809fe10 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f6ffd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f809f140 msgr2=0x7fa6f81347c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f80a47d0 con 0x7fa6f809f140 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f809f140 0x7fa6f81347c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:22.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6fd620640 10 monclient(hunting): authenticate will time out at 1709.383273s 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8135240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47776/0 (socket says 172.21.15.116:47776) 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 -- 172.21.15.116:0/3467613154 learned_addr learned my addr 172.21.15.116:0/3467613154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): get_auth_request con 0x7fa6f8131980 auth_method 0 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:22.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6ee7fc640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f809fe10 0x7fa6f8134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa6f809fe10 auth_method 0 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient(hunting): handle_auth_done global_id 194181 payload 824 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f809f140 msgr2=0x7fa6f81347c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa6f809f140 0x7fa6f81347c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f809fe10 msgr2=0x7fa6f8134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f809fe10 0x7fa6f8134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 monclient: found mon.noname-a 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.061+0000 7fa6f77fe640 1 -- 172.21.15.116:0/3467613154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e8009390 con 0x7fa6f8131980 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.063+0000 7fa6f77fe640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 msgr2=0x7fa6f8135240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.063+0000 7fa6f77fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8135240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cdb0 tx=0x7fa6f00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:22.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.063+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:41:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.065+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000ae80 con 0x7fa6f8131980 2024-09-18T12:41:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.065+0000 7fa6ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.065+0000 7fa6ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.065+0000 7fa6ef7fe640 10 monclient: dump: 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f0018480 con 0x7fa6f8131980 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0018820 con 0x7fa6f8131980 2024-09-18T12:41:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6fd620640 5 monclient: authenticate success, global_id 194181 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6fd620640 10 monclient: _renew_subs 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6fd620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f81378c0 con 0x7fa6f8131980 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6f5ffb640 4 set_mon_vals no callback set 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 10 monclient: dump: 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6fd620640 10 monclient: _renew_subs 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6fd620640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f8137c30 con 0x7fa6f8131980 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.066+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa6f000d040 con 0x7fa6f8131980 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6fd620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 0x7fa6f802aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6fd620640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8065e00 con 0x7fa6f8135780 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6ef7fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa6d8038490 0x7fa6d803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa6f0013070 con 0x7fa6f8131980 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 0x7fa6f802aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient: get_auth_request con 0x7fa6f8135780 auth_method 0 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_done global_id 174376 payload 824 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.067+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3467613154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6f804bf60 con 0x7fa6f8135780 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.068+0000 7fa6ee7fc640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa6d8038490 0x7fa6d803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:22.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.068+0000 7fa6ee7fc640 10 monclient: get_auth_request con 0x7fa6d8038490 auth_method 0 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.069+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 msgr2=0x7fa6f802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.069+0000 7fa6f6ffd640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 0x7fa6f802aa40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa6f809c9c0 tx=0x7fa6e8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8032440 con 0x7fa6f8135780 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 10 monclient: dump: 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:22.085 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa6e8032960 con 0x7fa6f8135780 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 msgr2=0x7fa6f802aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 0x7fa6f802aa40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa6f809c9c0 tx=0x7fa6e8032000 comp rx=0 tx=0).stop 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 msgr2=0x7fa6f802aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.070+0000 7fa6ef7fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f8135780 0x7fa6f802aa40 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa6f809c9c0 tx=0x7fa6e8032000 comp rx=0 tx=0).stop 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.071+0000 7fa6ee7fc640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa6d8038490 0x7fa6d803a940 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80a2980 tx=0x7fa6e0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6fd620640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 0x7fa6f805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6fd620640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8067f00 con 0x7fa6f805e310 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 0x7fa6f805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient: get_auth_request con 0x7fa6f805e310 auth_method 0 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:22.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:22.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:22.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 10 monclient(hunting): handle_auth_done global_id 174379 payload 824 2024-09-18T12:41:22.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.076+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3467613154 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa6f809a140 con 0x7fa6f805e310 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6f6ffd640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 msgr2=0x7fa6f805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6f6ffd640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 0x7fa6f805e710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8068680 tx=0x7fa6e803c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8047a00 con 0x7fa6f805e310 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 10 monclient: dump: 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:22.091 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7fa6e8047b60 con 0x7fa6f805e310 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:22.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 msgr2=0x7fa6f805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 0x7fa6f805e710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8068680 tx=0x7fa6e803c000 comp rx=0 tx=0).stop 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 msgr2=0x7fa6f805e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.079+0000 7fa6ef7fe640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa6f805e310 0x7fa6f805e710 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8068680 tx=0x7fa6e803c000 comp rx=0 tx=0).stop 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 160, 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:22.093 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:22.094 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:22.095 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:22.096 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:22.097 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:22.098 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:22.099 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.100 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:22.101 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:22.102 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:22.103 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:22.104 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:22.107 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa6d8038490 msgr2=0x7fa6d803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa6d8038490 0x7fa6d803a940 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80a2980 tx=0x7fa6e0008040 comp rx=0 tx=0).stop 2024-09-18T12:41:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 10 monclient: shutdown 2024-09-18T12:41:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 msgr2=0x7fa6f8135240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:22.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6f77fe640 1 -- 172.21.15.116:0/3467613154 reap_dead start 2024-09-18T12:41:22.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8135240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cdb0 tx=0x7fa6f00077d0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 shutdown_connections 2024-09-18T12:41:22.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 --2- 172.21.15.116:0/3467613154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6f8131980 0x7fa6f8135240 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:22.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 >> 172.21.15.116:0/3467613154 conn(0x7fa6f80a5050 msgr2=0x7fa6f809dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:22.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 shutdown_connections 2024-09-18T12:41:22.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:22.080+0000 7fa6fd620640 1 -- 172.21.15.116:0/3467613154 wait complete. 2024-09-18T12:41:22.110 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:41:22.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:41:42.111 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:41:42.111 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:41:42.112 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:41:42.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:41:42.117 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:41:42.117 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:41:42.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.221+0000 7f82c635d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.221+0000 7f82c635d640 10 monclient: build_initial_monmap 2024-09-18T12:41:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.221+0000 7f82c635d640 10 monclient: monmap: 2024-09-18T12:41:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:42.222302+0000 2024-09-18T12:41:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:42.222302+0000 2024-09-18T12:41:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:42.225 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.222+0000 7f82c635d640 1 Processor -- start 2024-09-18T12:41:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.222+0000 7f82c635d640 1 -- start start 2024-09-18T12:41:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.222+0000 7f82c635d640 10 monclient: init 2024-09-18T12:41:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T12:41:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82c01318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f82c009ea50 0x7f82c009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient(hunting): picked mon.noname-c con 0x7f82c009ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 0x7f82c0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient(hunting): picked mon.noname-b con 0x7f82c0136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c0132640 con 0x7f82c01318f0 2024-09-18T12:41:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c0131cc0 con 0x7f82c009ea50 2024-09-18T12:41:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c009fbf0 con 0x7f82c0136c20 2024-09-18T12:41:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c635d640 10 monclient(hunting): authenticate will time out at 1729.544102s 2024-09-18T12:41:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f82c01318f0 0x7f82c0131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32892/0 (socket says 172.21.15.116:32892) 2024-09-18T12:41:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bffff640 1 -- 172.21.15.116:0/1865959202 learned_addr learned my addr 172.21.15.116:0/1865959202 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 0x7f82c0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 10 monclient(hunting): get_auth_request con 0x7f82c0136c20 auth_method 0 2024-09-18T12:41:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 1 -- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 msgr2=0x7f82c0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:41:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 1 -- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 msgr2=0x7f82c0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 0x7f82c0139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 1 --2- 172.21.15.116:0/1865959202 >> v2:172.21.15.144:3300/0 conn(0x7f82c009ea50 0x7f82c009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): get_auth_request con 0x7f82c009ea50 auth_method 0 2024-09-18T12:41:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82c48fa640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 0x7f82c0139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:42.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_done global_id 174382 payload 274 2024-09-18T12:41:42.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 msgr2=0x7f82c0139000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 0x7f82c0139000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1865959202 >> v1:172.21.15.116:6789/0 conn(0x7f82c01318f0 legacy=0x7f82c0131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:41:42.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 1 monclient: found mon.noname-c 2024-09-18T12:41:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:41:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.223+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1865959202 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c009fe30 con 0x7f82c009ea50 2024-09-18T12:41:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1865959202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c009ea50 msgr2=0x7f82c009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:42.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82bf7fe640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c009ea50 0x7f82c009ee20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009850 tx=0x7f82b40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a98ba18b762ae4a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 1 -- 172.21.15.116:0/1865959202 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b402add0 con 0x7f82c009ea50 2024-09-18T12:41:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: dump: 2024-09-18T12:41:42.254 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:42.256 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: _finish_auth 0 2024-09-18T12:41:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 1 -- 172.21.15.116:0/1865959202 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b4032440 con 0x7f82c009ea50 2024-09-18T12:41:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82c635d640 5 monclient: authenticate success, global_id 174382 2024-09-18T12:41:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82c635d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82c635d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 1 -- 172.21.15.116:0/1865959202 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40327e0 con 0x7f82c009ea50 2024-09-18T12:41:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82c635d640 4 set_mon_vals no callback set 2024-09-18T12:41:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:42.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.226+0000 7f82beffd640 10 monclient: dump: 2024-09-18T12:41:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 10 monclient: shutdown 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 -- 172.21.15.116:0/1865959202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c009ea50 msgr2=0x7f82c009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c009ea50 0x7f82c009ee20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009850 tx=0x7f82b40090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 -- 172.21.15.116:0/1865959202 shutdown_connections 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c0136c20 0x7f82c0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 --2- 172.21.15.116:0/1865959202 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c009ea50 0x7f82c009ee20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 -- 172.21.15.116:0/1865959202 >> 172.21.15.116:0/1865959202 conn(0x7f82c0067110 msgr2=0x7f82c0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:42.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 -- 172.21.15.116:0/1865959202 shutdown_connections 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 1 -- 172.21.15.116:0/1865959202 wait complete. 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 10 monclient: build_initial_monmap 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.227+0000 7f82c635d640 10 monclient: monmap: 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:42.262 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 Processor -- start 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 -- start start 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient: init 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 0x7f82c009b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): picked mon.noname-c con 0x7f82c009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 0x7f82c0097250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): picked mon.noname-a con 0x7f82c01318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c0136c20 0x7f82c0097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:42.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): picked mon.noname-b con 0x7f82c0136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c009bf10 con 0x7f82c01318f0 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c009db90 con 0x7f82c0136c20 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c009abd0 con 0x7f82c009ea50 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.228+0000 7f82c635d640 10 monclient(hunting): authenticate will time out at 1729.549957s 2024-09-18T12:41:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 0x7f82c009b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 0x7f82c0097250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 0x7f82c0097250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54756/0 (socket says 172.21.15.116:54756) 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1065936062 learned_addr learned my addr 172.21.15.116:0/1065936062 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 0x7f82c009b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:60330/0 (socket says 172.21.15.116:60330) 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): get_auth_request con 0x7f82c01318f0 auth_method 0 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:42.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 10 monclient(hunting): get_auth_request con 0x7f82c009ea50 auth_method 0 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 1 -- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 msgr2=0x7f82c009b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 1 -- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 msgr2=0x7f82c009b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 0x7f82c009b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c0136c20 0x7f82c0097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:42.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bffff640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 0x7f82c009b520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 10 monclient(hunting): get_auth_request con 0x7f82c0136c20 auth_method 0 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82c48fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient(hunting): handle_auth_done global_id 194184 payload 824 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 msgr2=0x7f82c009b520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:42.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 0x7f82c009b520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c0136c20 msgr2=0x7f82c0097790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c0136c20 0x7f82c0097790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 monclient: found mon.noname-a 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.229+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1065936062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4009390 con 0x7f82c01318f0 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82bf7fe640 1 -- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 msgr2=0x7f82c0097250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82bf7fe640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 0x7f82c0097250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f82b40098b0 tx=0x7f82b4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 1 -- 172.21.15.116:0/1065936062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4031c70 con 0x7f82c01318f0 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:42.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: dump: 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:42.269 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: _finish_auth 0 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82c635d640 5 monclient: authenticate success, global_id 194184 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82c635d640 10 monclient: _renew_subs 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82c635d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c0098e00 con 0x7f82c01318f0 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82c635d640 10 monclient: _renew_subs 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82c635d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c0099970 con 0x7f82c01318f0 2024-09-18T12:41:42.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 1 -- 172.21.15.116:0/1065936062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b4031dd0 con 0x7f82c01318f0 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 1 -- 172.21.15.116:0/1065936062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b403c610 con 0x7f82c01318f0 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82a7fff640 10 monclient: dump: 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:42.271 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.231+0000 7f82be7fc640 4 set_mon_vals no callback set 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.232+0000 7f82a7fff640 1 -- 172.21.15.116:0/1065936062 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f82b403ca00 con 0x7f82c01318f0 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.232+0000 7f82a7fff640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82ac038490 0x7f82ac03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.232+0000 7f82a7fff640 1 -- 172.21.15.116:0/1065936062 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f82b406ced0 con 0x7f82c01318f0 2024-09-18T12:41:42.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.232+0000 7f82c635d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.233+0000 7f82c635d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.233+0000 7f82c635d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.233+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a0005010 con 0x7f82c01318f0 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.233+0000 7f82bffff640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82ac038490 0x7f82ac03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.233+0000 7f82bffff640 10 monclient: get_auth_request con 0x7f82ac038490 auth_method 0 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.235+0000 7f82bffff640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82ac038490 0x7f82ac03a940 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f82b0007d10 tx=0x7f82b0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.237+0000 7f82a7fff640 1 -- 172.21.15.116:0/1065936062 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f82b4037020 con 0x7f82c01318f0 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.237+0000 7f82a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:42.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.237+0000 7f82a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:42.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.454+0000 7f82c635d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.454+0000 7f82c635d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.454+0000 7f82c635d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.454+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f82a0005840 con 0x7f82c01318f0 2024-09-18T12:41:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.454+0000 7f82a7fff640 1 -- 172.21.15.116:0/1065936062 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f82b4037610 con 0x7f82c01318f0 2024-09-18T12:41:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.454+0000 7f82a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.454+0000 7f82a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:42.456 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":160,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:42.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82ac038490 msgr2=0x7f82ac03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:42.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82ac038490 0x7f82ac03a940 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f82b0007d10 tx=0x7f82b0009040 comp rx=0 tx=0).stop 2024-09-18T12:41:42.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 10 monclient: shutdown 2024-09-18T12:41:42.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 msgr2=0x7f82c0097250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 0x7f82c0097250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f82b40098b0 tx=0x7f82b4032000 comp rx=0 tx=0).stop 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 shutdown_connections 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f82ac038490 0x7f82ac03a940 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f82c0136c20 0x7f82c0097790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c01318f0 0x7f82c0097250 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 --2- 172.21.15.116:0/1065936062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f82c009ea50 0x7f82c009b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 >> 172.21.15.116:0/1065936062 conn(0x7f82c0067110 msgr2=0x7f82c00a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 shutdown_connections 2024-09-18T12:41:42.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:42.456+0000 7f82c635d640 1 -- 172.21.15.116:0/1065936062 wait complete. 2024-09-18T12:41:45.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.571+0000 7f3e31c20640 10 monclient: get_monmap_and_config 2024-09-18T12:41:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.571+0000 7f3e31c20640 10 monclient: build_initial_monmap 2024-09-18T12:41:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.571+0000 7f3e31c20640 10 monclient: monmap: 2024-09-18T12:41:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:45.572357+0000 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:45.572357+0000 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:45.573 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.572+0000 7f3e31c20640 1 Processor -- start 2024-09-18T12:41:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.572+0000 7f3e31c20640 1 -- start start 2024-09-18T12:41:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.572+0000 7f3e31c20640 10 monclient: init 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e2c096c60 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e2c097030 0x7f3e2c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient(hunting): picked mon.noname-a con 0x7f3e2c097030 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e2c133b40 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c09d2e0 con 0x7f3e2c096c60 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c09d6b0 con 0x7f3e2c133b40 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2c0992f0 con 0x7f3e2c097030 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e31c20640 10 monclient(hunting): authenticate will time out at 1732.894127s 2024-09-18T12:41:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e2c097030 0x7f3e2c097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3e2c096c60 0x7f3e2c09a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46848/0 (socket says 172.21.15.116:46848) 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): get_auth_request con 0x7f3e2c097030 auth_method 0 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/4267383379 learned_addr learned my addr 172.21.15.116:0/4267383379 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient(hunting): handle_auth_done global_id 194187 payload 274 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 1 -- 172.21.15.116:0/4267383379 >> v1:172.21.15.144:6789/0 conn(0x7f3e2c133b40 legacy=0x7f3e2c135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 1 -- 172.21.15.116:0/4267383379 >> v1:172.21.15.116:6789/0 conn(0x7f3e2c096c60 legacy=0x7f3e2c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 1 monclient: found mon.noname-a 2024-09-18T12:41:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.573+0000 7f3e2affd640 1 -- 172.21.15.116:0/4267383379 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c0995d0 con 0x7f3e2c097030 2024-09-18T12:41:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2affd640 1 -- 172.21.15.116:0/4267383379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c097030 msgr2=0x7f3e2c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2affd640 1 --2- 172.21.15.116:0/4267383379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c097030 0x7f3e2c097430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000baf0 tx=0x7f3e2000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677ebd0bc56240e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 1 -- 172.21.15.116:0/4267383379 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031800 con 0x7f3e2c097030 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: dump: 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:45.582 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 1 -- 172.21.15.116:0/4267383379 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2002f050 con 0x7f3e2c097030 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 1 -- 172.21.15.116:0/4267383379 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20031bd0 con 0x7f3e2c097030 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e31c20640 5 monclient: authenticate success, global_id 194187 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e31c20640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e31c20640 4 set_mon_vals no callback set 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.576+0000 7f3e2a7fc640 10 monclient: dump: 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 10 monclient: shutdown 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 1 -- 172.21.15.116:0/4267383379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c097030 msgr2=0x7f3e2c097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 1 --2- 172.21.15.116:0/4267383379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c097030 0x7f3e2c097430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000baf0 tx=0x7f3e2000b3b0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 1 -- 172.21.15.116:0/4267383379 shutdown_connections 2024-09-18T12:41:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 1 --2- 172.21.15.116:0/4267383379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c097030 0x7f3e2c097430 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 1 -- 172.21.15.116:0/4267383379 >> 172.21.15.116:0/4267383379 conn(0x7f3e2c0672b0 msgr2=0x7f3e2c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 1 -- 172.21.15.116:0/4267383379 shutdown_connections 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 1 -- 172.21.15.116:0/4267383379 wait complete. 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 10 monclient: build_initial_monmap 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.577+0000 7f3e31c20640 10 monclient: monmap: 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 Processor -- start 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 -- start start 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient: init 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 0x7f3e2c13d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): picked mon.noname-a con 0x7f3e2c096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e2c097030 0x7f3e2c06c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): picked mon.noname-b con 0x7f3e2c097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 0x7f3e2c06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): picked mon.noname-c con 0x7f3e2c133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c1336e0 con 0x7f3e2c096c60 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c1343e0 con 0x7f3e2c097030 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c117ad0 con 0x7f3e2c133b40 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e31c20640 10 monclient(hunting): authenticate will time out at 1732.899824s 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 0x7f3e2c13d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 0x7f3e2c13d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36070/0 (socket says 172.21.15.116:36070) 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/1615355723 learned_addr learned my addr 172.21.15.116:0/1615355723 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 10 monclient(hunting): get_auth_request con 0x7f3e2c096c60 auth_method 0 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:45.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2bfff640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 0x7f3e2c06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.578+0000 7f3e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 10 monclient(hunting): get_auth_request con 0x7f3e2c133b40 auth_method 0 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 1 -- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 msgr2=0x7f3e2c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 1 -- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 msgr2=0x7f3e2c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2affd640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e2c097030 0x7f3e2c06c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 0x7f3e2c06cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2affd640 10 monclient(hunting): get_auth_request con 0x7f3e2c097030 auth_method 0 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2bfff640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 0x7f3e2c06cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 10 monclient(hunting): handle_auth_done global_id 194190 payload 824 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 msgr2=0x7f3e2c06cb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 0x7f3e2c06cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e2c097030 msgr2=0x7f3e2c06c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e2c097030 0x7f3e2c06c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 1 monclient: found mon.noname-a 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:45.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.579+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/1615355723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2000b630 con 0x7f3e2c096c60 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 msgr2=0x7f3e2c13d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e2b7fe640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 0x7f3e2c13d640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00ccd0 tx=0x7f3e1c0053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 1 -- 172.21.15.116:0/1615355723 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c00aa60 con 0x7f3e2c096c60 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: dump: 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.592 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 1 -- 172.21.15.116:0/1615355723 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c00abe0 con 0x7f3e2c096c60 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e31c20640 5 monclient: authenticate success, global_id 194190 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 1 -- 172.21.15.116:0/1615355723 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c011560 con 0x7f3e2c096c60 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:45.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e137fe640 10 monclient: dump: 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e31c20640 10 monclient: _renew_subs 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e31c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c06e140 con 0x7f3e2c096c60 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e29ffb640 4 set_mon_vals no callback set 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e31c20640 10 monclient: _renew_subs 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e31c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.581+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c117480 con 0x7f3e2c096c60 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.582+0000 7f3e137fe640 1 -- 172.21.15.116:0/1615355723 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3e1c017070 con 0x7f3e2c096c60 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.582+0000 7f3e31c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.582+0000 7f3e31c20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.582+0000 7f3e31c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.582+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c04bf60 con 0x7f3e2c096c60 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.582+0000 7f3e137fe640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3e18038490 0x7f3e1803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.582+0000 7f3e137fe640 1 -- 172.21.15.116:0/1615355723 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3e1c050400 con 0x7f3e2c096c60 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.583+0000 7f3e2affd640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3e18038490 0x7f3e1803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.583+0000 7f3e2affd640 10 monclient: get_auth_request con 0x7f3e18038490 auth_method 0 2024-09-18T12:41:45.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.585+0000 7f3e2affd640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3e18038490 0x7f3e1803a940 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0637c0 tx=0x7f3e20039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:45.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.587+0000 7f3e137fe640 1 -- 172.21.15.116:0/1615355723 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3e1c017550 con 0x7f3e2c096c60 2024-09-18T12:41:45.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.587+0000 7f3e137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:45.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.587+0000 7f3e137fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.804+0000 7f3e31c20640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.804+0000 7f3e31c20640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.804+0000 7f3e31c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.804+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3e2c0983b0 con 0x7f3e2c096c60 2024-09-18T12:41:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.804+0000 7f3e137fe640 1 -- 172.21.15.116:0/1615355723 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f3e1c01f020 con 0x7f3e2c096c60 2024-09-18T12:41:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.804+0000 7f3e137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:45.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.804+0000 7f3e137fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:45.806 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":160,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3e18038490 msgr2=0x7f3e1803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3e18038490 0x7f3e1803a940 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0637c0 tx=0x7f3e20039000 comp rx=0 tx=0).stop 2024-09-18T12:41:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 10 monclient: shutdown 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 msgr2=0x7f3e2c13d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 0x7f3e2c13d640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00ccd0 tx=0x7f3e1c0053f0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 shutdown_connections 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3e18038490 0x7f3e1803a940 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3e2c133b40 0x7f3e2c06cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e2c097030 0x7f3e2c06c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 --2- 172.21.15.116:0/1615355723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e2c096c60 0x7f3e2c13d640 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 >> 172.21.15.116:0/1615355723 conn(0x7f3e2c0672b0 msgr2=0x7f3e2c0a0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 shutdown_connections 2024-09-18T12:41:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:45.806+0000 7f3e31c20640 1 -- 172.21.15.116:0/1615355723 wait complete. 2024-09-18T12:41:48.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:48.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.921+0000 7f736f309640 10 monclient: get_monmap_and_config 2024-09-18T12:41:48.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.921+0000 7f736f309640 10 monclient: build_initial_monmap 2024-09-18T12:41:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.921+0000 7f736f309640 10 monclient: monmap: 2024-09-18T12:41:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:48.922300+0000 2024-09-18T12:41:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:48.922300+0000 2024-09-18T12:41:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:48.925 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 1 Processor -- start 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 1 -- start start 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient: init 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7368096e10 0x7f736809a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient(hunting): picked mon.noname-c con 0x7f7368096e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f736809aaa0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:48.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient(hunting): picked mon.noname-b con 0x7f73680978f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:48.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736809d4a0 con 0x7f736809aaa0 2024-09-18T12:41:48.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7368099620 con 0x7f7368096e10 2024-09-18T12:41:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7368099340 con 0x7f73680978f0 2024-09-18T12:41:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.922+0000 7f736f309640 10 monclient(hunting): authenticate will time out at 1736.243946s 2024-09-18T12:41:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736c8a4640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f736809aaa0 0x7f73680971e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46854/0 (socket says 172.21.15.116:46854) 2024-09-18T12:41:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736c8a4640 1 -- 172.21.15.116:0/3314588094 learned_addr learned my addr 172.21.15.116:0/3314588094 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 10 monclient(hunting): get_auth_request con 0x7f73680978f0 auth_method 0 2024-09-18T12:41:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 1 -- 172.21.15.116:0/3314588094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 msgr2=0x7f7368135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 1 -- 172.21.15.116:0/3314588094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 msgr2=0x7f7368135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 1 --2- 172.21.15.116:0/3314588094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 1 --2- 172.21.15.116:0/3314588094 >> v2:172.21.15.144:3300/0 conn(0x7f7368096e10 0x7f736809a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): get_auth_request con 0x7f7368096e10 auth_method 0 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d8a6640 1 --2- 172.21.15.116:0/3314588094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient(hunting): handle_auth_done global_id 174391 payload 274 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 1 -- 172.21.15.116:0/3314588094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 msgr2=0x7f7368135b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 1 --2- 172.21.15.116:0/3314588094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 1 -- 172.21.15.116:0/3314588094 >> v1:172.21.15.116:6789/0 conn(0x7f736809aaa0 legacy=0x7f73680971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient: _finish_hunting 0 2024-09-18T12:41:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 1 monclient: found mon.noname-c 2024-09-18T12:41:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:41:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.923+0000 7f736d0a5640 1 -- 172.21.15.116:0/3314588094 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7368099900 con 0x7f7368096e10 2024-09-18T12:41:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f736d0a5640 1 -- 172.21.15.116:0/3314588094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 msgr2=0x7f736809a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f736d0a5640 1 --2- 172.21.15.116:0/3314588094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f736809a560 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7364009850 tx=0x7f73640090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=558d264b756712ad server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 1 -- 172.21.15.116:0/3314588094 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736402add0 con 0x7f7368096e10 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: dump: 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:48.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: _finish_auth 0 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 1 -- 172.21.15.116:0/3314588094 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7364032440 con 0x7f7368096e10 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f736f309640 5 monclient: authenticate success, global_id 174391 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f736f309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f736f309640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f736f309640 4 set_mon_vals no callback set 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 1 -- 172.21.15.116:0/3314588094 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7364032820 con 0x7f7368096e10 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.926+0000 7f735ffff640 10 monclient: dump: 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:48.959 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 10 monclient: shutdown 2024-09-18T12:41:48.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 -- 172.21.15.116:0/3314588094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 msgr2=0x7f736809a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 --2- 172.21.15.116:0/3314588094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f736809a560 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7364009850 tx=0x7f73640090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 -- 172.21.15.116:0/3314588094 shutdown_connections 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 --2- 172.21.15.116:0/3314588094 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368135b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 --2- 172.21.15.116:0/3314588094 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f736809a560 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 -- 172.21.15.116:0/3314588094 >> 172.21.15.116:0/3314588094 conn(0x7f73680672b0 msgr2=0x7f73680676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 -- 172.21.15.116:0/3314588094 shutdown_connections 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 1 -- 172.21.15.116:0/3314588094 wait complete. 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 10 monclient: build_initial_monmap 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.927+0000 7f736f309640 10 monclient: monmap: 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 Processor -- start 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 -- start start 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient: init 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f7368141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): picked mon.noname-b con 0x7f7368096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): picked mon.noname-c con 0x7f73680978f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 0x7f7368133fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): picked mon.noname-a con 0x7f736809aaa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7368135730 con 0x7f736809aaa0 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736806bd10 con 0x7f7368096e10 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736811b930 con 0x7f73680978f0 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736f309640 10 monclient(hunting): authenticate will time out at 1736.249796s 2024-09-18T12:41:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736c8a4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736d8a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 0x7f7368133fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736c8a4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41852/0 (socket says 172.21.15.116:41852) 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736c8a4640 1 -- 172.21.15.116:0/1302321329 learned_addr learned my addr 172.21.15.116:0/1302321329 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736c8a4640 10 monclient(hunting): get_auth_request con 0x7f73680978f0 auth_method 0 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736c8a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736c8a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736c8a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736d8a6640 10 monclient(hunting): get_auth_request con 0x7f736809aaa0 auth_method 0 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736d8a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736d8a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.928+0000 7f736d8a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736c8a4640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 msgr2=0x7f7368141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:41:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736c8a4640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 msgr2=0x7f7368141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736c8a4640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368141bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f7368141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736c8a4640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368141bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 10 monclient(hunting): get_auth_request con 0x7f7368096e10 auth_method 0 2024-09-18T12:41:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 10 monclient(hunting): handle_auth_done global_id 194193 payload 824 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 msgr2=0x7f7368141bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f7368141bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 msgr2=0x7f7368141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f7368141690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 10 monclient: _finish_hunting 0 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 1 monclient: found mon.noname-a 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.929+0000 7f736d8a6640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7364009390 con 0x7f736809aaa0 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736d8a6640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 msgr2=0x7f7368133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736d8a6640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 0x7f7368133fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7360007e40 tx=0x7f73600097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360014420 con 0x7f736809aaa0 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: dump: 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7360014580 con 0x7f736809aaa0 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736f309640 5 monclient: authenticate success, global_id 194193 2024-09-18T12:41:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736f309640 10 monclient: _renew_subs 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736f309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736811bc10 con 0x7f736809aaa0 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360014940 con 0x7f736809aaa0 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: dump: 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735f7fe640 4 set_mon_vals no callback set 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736f309640 10 monclient: _renew_subs 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736f309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7368142110 con 0x7f736809aaa0 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736d8a6640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 msgr2=0x7f7368133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736d8a6640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 msgr2=0x7f7368133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736d8a6640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 0x7f7368133fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7360007e40 tx=0x7f73600097e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f736d8a6640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 0x7f7368133fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7360007e40 tx=0x7f73600097e0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 msgr2=0x7f7368133fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f736809aaa0 0x7f7368133fc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f734c0057a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient(hunting): picked mon.b con 0x7f7368096e10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f734c0060a0 0x7f734c008520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient(hunting): picked mon.a con 0x7f734c0060a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f734c00b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient(hunting): picked mon.c con 0x7f73680978f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c00b9c0 con 0x7f734c0060a0 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c00bb00 con 0x7f7368096e10 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c00bc40 con 0x7f73680978f0 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.931+0000 7f735d7fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f734c0060a0 0x7f734c008520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): get_auth_request con 0x7f734c0060a0 auth_method 0 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f734c00b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 10 monclient(hunting): get_auth_request con 0x7f73680978f0 auth_method 0 2024-09-18T12:41:48.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f734c0057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 10 monclient(hunting): get_auth_request con 0x7f7368096e10 auth_method 0 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:48.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d8a6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736d0a5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient(hunting): handle_auth_done global_id 194193 payload 293 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 msgr2=0x7f734c00b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f734c00b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 msgr2=0x7f734c0057a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f734c0057a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient: _finish_hunting 0 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 1 monclient: found mon.a 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f734c00bf20 con 0x7f734c0060a0 2024-09-18T12:41:48.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 10 monclient: _finish_auth 0 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736c8a4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736f309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736f309640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736f309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.932+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736804bf60 con 0x7f734c0060a0 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f736c8a4640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f734c0060a0 msgr2=0x7f734c008520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f736c8a4640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f734c0060a0 0x7f734c008520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f736806ad60 tx=0x7f7358009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:48.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358004b70 con 0x7f734c0060a0 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f735d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f735d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f735d7fa640 10 monclient: dump: 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:48.976 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7358004cf0 con 0x7f734c0060a0 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f735d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.934+0000 7f735effd640 4 set_mon_vals no callback set 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.935+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7358007170 con 0x7f734c0060a0 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.935+0000 7f735d7fa640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f734c040ed0 0x7f734c043380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.935+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 4 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7358049be0 con 0x7f734c0060a0 2024-09-18T12:41:48.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.936+0000 7f736d0a5640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f734c040ed0 0x7f734c043380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.936+0000 7f736d0a5640 10 monclient: get_auth_request con 0x7f734c040ed0 auth_method 0 2024-09-18T12:41:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.939+0000 7f736d0a5640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f734c040ed0 0x7f734c043380 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7364034000 tx=0x7f7364032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.939+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7358004e60 con 0x7f734c0060a0 2024-09-18T12:41:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.939+0000 7f735d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:48.939+0000 7f735d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.154+0000 7f736f309640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.154+0000 7f736f309640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.154+0000 7f736f309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.154+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f736806a4e0 con 0x7f734c0060a0 2024-09-18T12:41:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.155+0000 7f735d7fa640 1 -- 172.21.15.116:0/1302321329 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f735801b270 con 0x7f734c0060a0 2024-09-18T12:41:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.155+0000 7f735d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.155+0000 7f735d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:49.156 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f734c040ed0 msgr2=0x7f734c043380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f734c040ed0 0x7f734c043380 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7364034000 tx=0x7f7364032000 comp rx=0 tx=0).stop 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 10 monclient: shutdown 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f734c0060a0 msgr2=0x7f734c008520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f734c0060a0 0x7f734c008520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f736806ad60 tx=0x7f7358009040 comp rx=0 tx=0).stop 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 shutdown_connections 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f734c040ed0 0x7f734c043380 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73680978f0 0x7f734c00b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f734c0060a0 0x7f734c008520 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 --2- 172.21.15.116:0/1302321329 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7368096e10 0x7f734c0057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 >> 172.21.15.116:0/1302321329 conn(0x7f73680672b0 msgr2=0x7f73680a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 shutdown_connections 2024-09-18T12:41:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.157+0000 7f736f309640 1 -- 172.21.15.116:0/1302321329 wait complete. 2024-09-18T12:41:49.168 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:41:49.169 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:41:49.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:41:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.263+0000 7f0cc8b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.263+0000 7f0cc8b9d640 10 monclient: build_initial_monmap 2024-09-18T12:41:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.263+0000 7f0cc8b9d640 10 monclient: monmap: 2024-09-18T12:41:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:49.263791+0000 2024-09-18T12:41:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:49.263791+0000 2024-09-18T12:41:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.263+0000 7f0cc8b9d640 1 Processor -- start 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.263+0000 7f0cc8b9d640 1 -- start start 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.263+0000 7f0cc8b9d640 10 monclient: init 2024-09-18T12:41:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 0x7f0cc4131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc41318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cc409ea50 0x7f0cc409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc409ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0cc4068d90 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc4131cc0 con 0x7f0cc4068d90 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc409fbf0 con 0x7f0cc409ea50 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc409fe30 con 0x7f0cc41318f0 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc8b9d640 10 monclient(hunting): authenticate will time out at 1736.585242s 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 0x7f0cc4131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cc409ea50 0x7f0cc409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cc409ea50 0x7f0cc409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36096/0 (socket says 172.21.15.116:36096) 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 0x7f0cc4131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41878/0 (socket says 172.21.15.116:41878) 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2218446468 learned_addr learned my addr 172.21.15.116:0/2218446468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request con 0x7f0cc409ea50 auth_method 0 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 10 monclient(hunting): get_auth_request con 0x7f0cc41318f0 auth_method 0 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 1 -- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 msgr2=0x7f0cc4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 1 -- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 msgr2=0x7f0cc4131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 1 --2- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 0x7f0cc4131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cc259c640 1 --2- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 0x7f0cc4131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:49.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient(hunting): handle_auth_done global_id 194196 payload 274 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 msgr2=0x7f0cc4131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 --2- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 0x7f0cc4131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2218446468 >> v1:172.21.15.144:6789/0 conn(0x7f0cc4068d90 legacy=0x7f0cc40691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 monclient: found mon.noname-a 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.264+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2218446468 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc40a1210 con 0x7f0cc409ea50 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc409ea50 msgr2=0x7f0cc409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cb9d9b640 1 --2- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc409ea50 0x7f0cc409ee20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009910 tx=0x7f0cac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b9c76038bc1676 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 1 -- 172.21.15.116:0/2218446468 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac02ae90 con 0x7f0cc409ea50 2024-09-18T12:41:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: dump: 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.295 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: _finish_auth 0 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 1 -- 172.21.15.116:0/2218446468 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac032440 con 0x7f0cc409ea50 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc8b9d640 5 monclient: authenticate success, global_id 194196 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc8b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc8b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 1 -- 172.21.15.116:0/2218446468 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac032820 con 0x7f0cc409ea50 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc8b9d640 4 set_mon_vals no callback set 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.267+0000 7f0cc1d9b640 10 monclient: dump: 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 10 monclient: shutdown 2024-09-18T12:41:49.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc409ea50 msgr2=0x7f0cc409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 --2- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc409ea50 0x7f0cc409ee20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009910 tx=0x7f0cac0091d0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2218446468 shutdown_connections 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 --2- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc409ea50 0x7f0cc409ee20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 --2- 172.21.15.116:0/2218446468 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc41318f0 0x7f0cc4131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2218446468 >> 172.21.15.116:0/2218446468 conn(0x7f0cc4067110 msgr2=0x7f0cc4067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2218446468 shutdown_connections 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2218446468 wait complete. 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 10 monclient: build_initial_monmap 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.268+0000 7f0cc8b9d640 10 monclient: monmap: 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 Processor -- start 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 -- start start 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient: init 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cc4068d90 0x7f0cc409ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc4068d90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 0x7f0cc4130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41318f0 0x7f0cc411b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc41318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc40a4d20 con 0x7f0cc41318f0 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc411dd90 con 0x7f0cc4068d90 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc411e070 con 0x7f0cc409ea50 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc8b9d640 10 monclient(hunting): authenticate will time out at 1736.590557s 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 0x7f0cc4130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41318f0 0x7f0cc411b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 0x7f0cc4130c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41882/0 (socket says 172.21.15.116:41882) 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): get_auth_request con 0x7f0cc41318f0 auth_method 0 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2010766712 learned_addr learned my addr 172.21.15.116:0/2010766712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request con 0x7f0cc409ea50 auth_method 0 2024-09-18T12:41:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cc4068d90 0x7f0cc409ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 10 monclient(hunting): get_auth_request con 0x7f0cc4068d90 auth_method 0 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 msgr2=0x7f0cc4130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 msgr2=0x7f0cc4130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 0x7f0cc4130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cb9d9b640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 0x7f0cc4130c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient(hunting): handle_auth_done global_id 194199 payload 824 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 msgr2=0x7f0cc4130c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0cc409ea50 0x7f0cc4130c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cc4068d90 msgr2=0x7f0cc409ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0cc4068d90 0x7f0cc409ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 1 monclient: found mon.noname-a 2024-09-18T12:41:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.269+0000 7f0cc2d9d640 1 -- 172.21.15.116:0/2010766712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac009450 con 0x7f0cc41318f0 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc2d9d640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41318f0 msgr2=0x7f0cc411b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc2d9d640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41318f0 0x7f0cc411b720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c7e0 tx=0x7f0cbc0025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00a850 con 0x7f0cc41318f0 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: dump: 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: _finish_auth 0 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc00a9d0 con 0x7f0cc41318f0 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc8b9d640 5 monclient: authenticate success, global_id 194199 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc8b9d640 10 monclient: _renew_subs 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc8b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc411e350 con 0x7f0cc41318f0 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00ad90 con 0x7f0cc41318f0 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cbaffd640 10 monclient: dump: 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc8b9d640 10 monclient: _renew_subs 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc8b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc159a640 4 set_mon_vals no callback set 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.272+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc411cea0 con 0x7f0cc41318f0 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc8b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc8b9d640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 0x7f0cc411bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc4065e00 con 0x7f0cc41311b0 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0cbc012630 con 0x7f0cc41318f0 2024-09-18T12:41:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 0x7f0cc411bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient: get_auth_request con 0x7f0cc41311b0 auth_method 0 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cbaffd640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca4038490 0x7f0ca403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 10 monclient(hunting): handle_auth_done global_id 194202 payload 824 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0cbc04c5e0 con 0x7f0cc41318f0 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cc259c640 1 -- 172.21.15.116:0/2010766712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cc404bf60 con 0x7f0cc41311b0 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cb9d9b640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca4038490 0x7f0ca403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.273+0000 7f0cb9d9b640 10 monclient: get_auth_request con 0x7f0ca4038490 auth_method 0 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cc259c640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 msgr2=0x7f0cc411bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cc259c640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 0x7f0cc411bc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4006ea0 tx=0x7f0cb4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb400ed50 con 0x7f0cc41311b0 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 10 monclient: dump: 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0cb400eeb0 con 0x7f0cc41311b0 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 msgr2=0x7f0cc411bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.275+0000 7f0cbaffd640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 0x7f0cc411bc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4006ea0 tx=0x7f0cb4008040 comp rx=0 tx=0).stop 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.276+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 msgr2=0x7f0cc411bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.276+0000 7f0cbaffd640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41311b0 0x7f0cc411bc60 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4006ea0 tx=0x7f0cb4008040 comp rx=0 tx=0).stop 2024-09-18T12:41:49.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.276+0000 7f0cb9d9b640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca4038490 0x7f0ca403a940 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009210 tx=0x7f0cac032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc8b9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc8b9d640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 0x7f0cc4096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc8b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc4131580 con 0x7f0cc411c1a0 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 0x7f0cc4096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 10 monclient: get_auth_request con 0x7f0cc411c1a0 auth_method 0 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.281+0000 7f0cc259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.282+0000 7f0cc259c640 10 monclient(hunting): handle_auth_done global_id 194205 payload 824 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.282+0000 7f0cc259c640 1 -- 172.21.15.116:0/2010766712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0cc411c5a0 con 0x7f0cc411c1a0 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc259c640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 msgr2=0x7f0cc4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc259c640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 0x7f0cc4096a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4019000 tx=0x7f0cb401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb401ed20 con 0x7f0cc411c1a0 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 10 monclient: dump: 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f0cb4020410 con 0x7f0cc411c1a0 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 msgr2=0x7f0cc4096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 0x7f0cc4096a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4019000 tx=0x7f0cb401c000 comp rx=0 tx=0).stop 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 msgr2=0x7f0cc4096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cbaffd640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc411c1a0 0x7f0cc4096a60 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4019000 tx=0x7f0cb401c000 comp rx=0 tx=0).stop 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 162, 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:41:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.317 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:49.318 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:49.319 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:49.320 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:49.321 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:49.323 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:49.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:49.326 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:49.327 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:49.327 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:49.327 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:49.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:49.327 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:49.327 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca4038490 msgr2=0x7f0ca403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc8b9d640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0ca4038490 0x7f0ca403a940 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009210 tx=0x7f0cac032000 comp rx=0 tx=0).stop 2024-09-18T12:41:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc8b9d640 10 monclient: shutdown 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc8b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41318f0 msgr2=0x7f0cc411b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc8b9d640 1 --2- 172.21.15.116:0/2010766712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cc41318f0 0x7f0cc411b720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c7e0 tx=0x7f0cbc0025b0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc2d9d640 1 -- 172.21.15.116:0/2010766712 reap_dead start 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.284+0000 7f0cc2d9d640 1 -- 172.21.15.116:0/2010766712 reap_dead start 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.285+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 shutdown_connections 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.285+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 >> 172.21.15.116:0/2010766712 conn(0x7f0cc4067110 msgr2=0x7f0cc409dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.285+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 shutdown_connections 2024-09-18T12:41:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.285+0000 7f0cc8b9d640 1 -- 172.21.15.116:0/2010766712 wait complete. 2024-09-18T12:41:49.329 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:49.330 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:41:49.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:41:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.422+0000 7f871399d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.422+0000 7f871399d640 10 monclient: build_initial_monmap 2024-09-18T12:41:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.422+0000 7f871399d640 10 monclient: monmap: 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:49.423411+0000 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:49.423411+0000 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:49.424 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 1 Processor -- start 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 1 -- start start 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient: init 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f870c0974f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f870c09a8a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f870c134080 0x7f870c136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient(hunting): picked mon.noname-a con 0x7f870c134080 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870c12b210 con 0x7f870c0974f0 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870c0978c0 con 0x7f870c09a8a0 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f870c09ca20 con 0x7f870c134080 2024-09-18T12:41:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.423+0000 7f871399d640 10 monclient(hunting): authenticate will time out at 1736.744965s 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711739640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f870c0974f0 0x7f870c12ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46870/0 (socket says 172.21.15.116:46870) 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711739640 1 -- 172.21.15.116:0/1278989923 learned_addr learned my addr 172.21.15.116:0/1278989923 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 1 --2- 172.21.15.116:0/1278989923 >> v2:172.21.15.116:3300/0 conn(0x7f870c134080 0x7f870c136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): get_auth_request con 0x7f870c134080 auth_method 0 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient(hunting): handle_auth_done global_id 194208 payload 274 2024-09-18T12:41:49.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 1 -- 172.21.15.116:0/1278989923 >> v1:172.21.15.144:6789/0 conn(0x7f870c09a8a0 legacy=0x7f870c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:49.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 1 -- 172.21.15.116:0/1278989923 >> v1:172.21.15.116:6789/0 conn(0x7f870c0974f0 legacy=0x7f870c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:49.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient: _finish_hunting 0 2024-09-18T12:41:49.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 1 monclient: found mon.noname-a 2024-09-18T12:41:49.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:49.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.424+0000 7f8711f3a640 1 -- 172.21.15.116:0/1278989923 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c098ad0 con 0x7f870c134080 2024-09-18T12:41:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8711f3a640 1 -- 172.21.15.116:0/1278989923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c134080 msgr2=0x7f870c136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8711f3a640 1 --2- 172.21.15.116:0/1278989923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c134080 0x7f870c136460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8704009e60 tx=0x7f8704009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd176b6de4c1cb12 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 1 -- 172.21.15.116:0/1278989923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704007840 con 0x7f870c134080 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: dump: 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.437 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: _finish_auth 0 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 1 -- 172.21.15.116:0/1278989923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870402d050 con 0x7f870c134080 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 5 monclient: authenticate success, global_id 194208 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 4 set_mon_vals no callback set 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 1 -- 172.21.15.116:0/1278989923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704007c10 con 0x7f870c134080 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f8703fff640 10 monclient: dump: 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 10 monclient: shutdown 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 1 -- 172.21.15.116:0/1278989923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c134080 msgr2=0x7f870c136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 1 --2- 172.21.15.116:0/1278989923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c134080 0x7f870c136460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8704009e60 tx=0x7f8704009780 comp rx=0 tx=0).stop 2024-09-18T12:41:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 1 -- 172.21.15.116:0/1278989923 shutdown_connections 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 1 --2- 172.21.15.116:0/1278989923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c134080 0x7f870c136460 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 1 -- 172.21.15.116:0/1278989923 >> 172.21.15.116:0/1278989923 conn(0x7f870c0672b0 msgr2=0x7f870c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 1 -- 172.21.15.116:0/1278989923 shutdown_connections 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.427+0000 7f871399d640 1 -- 172.21.15.116:0/1278989923 wait complete. 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.428+0000 7f871399d640 10 monclient: build_initial_monmap 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.428+0000 7f871399d640 10 monclient: monmap: 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.428+0000 7f871399d640 1 Processor -- start 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.428+0000 7f871399d640 1 -- start start 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.428+0000 7f871399d640 10 monclient: init 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 0x7f870c1297d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): picked mon.noname-a con 0x7f870c0974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 0x7f870c129d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): picked mon.noname-c con 0x7f870c09a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f870c134080 0x7f870c06d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): picked mon.noname-b con 0x7f870c134080 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c12cf70 con 0x7f870c0974f0 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c098300 con 0x7f870c134080 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c12b9c0 con 0x7f870c09a8a0 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f871399d640 10 monclient(hunting): authenticate will time out at 1736.750328s 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 0x7f870c1297d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 0x7f870c129d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 0x7f870c1297d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36150/0 (socket says 172.21.15.116:36150) 2024-09-18T12:41:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 -- 172.21.15.116:0/483919062 learned_addr learned my addr 172.21.15.116:0/483919062 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): get_auth_request con 0x7f870c0974f0 auth_method 0 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 10 monclient(hunting): get_auth_request con 0x7f870c09a8a0 auth_method 0 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 msgr2=0x7f870c129d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 msgr2=0x7f870c129d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 0x7f870c129d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 0x7f870c129d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient(hunting): handle_auth_done global_id 194211 payload 824 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 msgr2=0x7f870c129d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c09a8a0 0x7f870c129d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f870c134080 msgr2=0x7f870c06d700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f870c134080 0x7f870c06d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient: _finish_hunting 0 2024-09-18T12:41:49.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 monclient: found mon.noname-a 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.429+0000 7f8711739640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8704009a20 con 0x7f870c0974f0 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.431+0000 7f8711739640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 msgr2=0x7f870c1297d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.431+0000 7f8711739640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 0x7f870c1297d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f870800c610 tx=0x7f8708004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708005cb0 con 0x7f870c0974f0 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: dump: 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.445 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: _finish_auth 0 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8708005e30 con 0x7f870c0974f0 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 5 monclient: authenticate success, global_id 194211 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 10 monclient: _renew_subs 2024-09-18T12:41:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870c09e010 con 0x7f870c0974f0 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708007340 con 0x7f870c0974f0 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 10 monclient: dump: 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:49.447 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87037fe640 4 set_mon_vals no callback set 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 10 monclient: _renew_subs 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870c06f2e0 con 0x7f870c0974f0 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f870801b400 con 0x7f870c0974f0 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:49.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c065f20 con 0x7f870c06dc40 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.432+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 10 monclient: get_auth_request con 0x7f870c06dc40 auth_method 0 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f87017fa640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86f00384c0 0x7f86f003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f870804bf20 con 0x7f870c0974f0 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8711f3a640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86f00384c0 0x7f86f003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.433+0000 7f8711f3a640 10 monclient: get_auth_request con 0x7f86f00384c0 auth_method 0 2024-09-18T12:41:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.436+0000 7f8711f3a640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86f00384c0 0x7f86f003a970 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8704034000 tx=0x7f8704037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 10 monclient: get_auth_request con 0x7f870c06dc40 auth_method 0 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:49.634+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:41:50.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 10 monclient: get_auth_request con 0x7f870c06dc40 auth_method 0 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:50.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.035+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:41:50.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:50.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 10 monclient: get_auth_request con 0x7f870c06dc40 auth_method 0 2024-09-18T12:41:50.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:50.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:41:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:50.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:50.836+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:41:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.429+0000 7f86f7fff640 10 monclient: tick 2024-09-18T12:41:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.429+0000 7f86f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.429+0000 7f86f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 10 monclient: get_auth_request con 0x7f870c06dc40 auth_method 0 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:41:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:52.438+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-18T12:41:53.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 10 monclient: tick 2024-09-18T12:41:53.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:53.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-18T12:41:53.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:53.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 0x7f86e40059b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 msgr2=0x7f870c06e010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06dc40 0x7f870c06e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.429+0000 7f86f7fff640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e4006060 con 0x7f86e4003500 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 0x7f86e40059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient: get_auth_request con 0x7f86e4003500 auth_method 0 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 10 monclient(hunting): handle_auth_done global_id 204104 payload 824 2024-09-18T12:41:53.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.430+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86e40061d0 con 0x7f86e4003500 2024-09-18T12:41:53.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.432+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 msgr2=0x7f86e40059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.432+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 0x7f86e40059b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f870c06eee0 tx=0x7f86fc009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.432+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc014e80 con 0x7f86e4003500 2024-09-18T12:41:53.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.432+0000 7f87017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.432+0000 7f87017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:53.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.432+0000 7f87017fa640 10 monclient: dump: 2024-09-18T12:41:53.433 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.434 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.433+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f86fc007030 con 0x7f86e4003500 2024-09-18T12:41:53.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.433+0000 7f87017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:53.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.433+0000 7f87017fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:53.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.433+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 msgr2=0x7f86e40059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.433+0000 7f87017fa640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 0x7f86e40059b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f870c06eee0 tx=0x7f86fc009040 comp rx=0 tx=0).stop 2024-09-18T12:41:53.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.433+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 msgr2=0x7f86e40059b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:53.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.433+0000 7f87017fa640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f86e4003500 0x7f86e40059b0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f870c06eee0 tx=0x7f86fc009040 comp rx=0 tx=0).stop 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.439+0000 7f871399d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.439+0000 7f871399d640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 0x7f870c0696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.439+0000 7f871399d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.439+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c04bf60 con 0x7f870c06e550 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 0x7f870c0696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient: get_auth_request con 0x7f870c06e550 auth_method 0 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 10 monclient(hunting): handle_auth_done global_id 204107 payload 824 2024-09-18T12:41:53.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.440+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f870c12d1b0 con 0x7f870c06e550 2024-09-18T12:41:53.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.442+0000 7f8710f38640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 msgr2=0x7f870c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.442+0000 7f8710f38640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 0x7f870c0696a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f870c06eee0 tx=0x7f86fc023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.442+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc02f660 con 0x7f870c06e550 2024-09-18T12:41:53.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.442+0000 7f87017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:53.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 10 monclient: dump: 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.444 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f86fc022440 con 0x7f870c06e550 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 msgr2=0x7f870c0696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 0x7f870c0696a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f870c06eee0 tx=0x7f86fc023000 comp rx=0 tx=0).stop 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 msgr2=0x7f870c0696a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f87017fa640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f870c06e550 0x7f870c0696a0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f870c06eee0 tx=0x7f86fc023000 comp rx=0 tx=0).stop 2024-09-18T12:41:53.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f8711f3a640 1 -- 172.21.15.116:0/483919062 reap_dead start 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 162, 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:53.446 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:53.447 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:53.448 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:53.449 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:53.450 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:53.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:53.453 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:53.454 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.455 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:53.456 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:53.457 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86f00384c0 msgr2=0x7f86f003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86f00384c0 0x7f86f003a970 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8704034000 tx=0x7f8704037000 comp rx=0 tx=0).stop 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 10 monclient: shutdown 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 msgr2=0x7f870c1297d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 0x7f870c1297d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f870800c610 tx=0x7f8708004960 comp rx=0 tx=0).stop 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 shutdown_connections 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f86f00384c0 0x7f86f003a970 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.443+0000 7f871399d640 1 --2- 172.21.15.116:0/483919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f870c0974f0 0x7f870c1297d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.444+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 >> 172.21.15.116:0/483919062 conn(0x7f870c0672b0 msgr2=0x7f870c09c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.444+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 shutdown_connections 2024-09-18T12:41:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.444+0000 7f871399d640 1 -- 172.21.15.116:0/483919062 wait complete. 2024-09-18T12:41:53.460 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:53.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:41:53.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:41:53.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.562+0000 7f2d85e6f640 10 monclient: get_monmap_and_config 2024-09-18T12:41:53.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.562+0000 7f2d85e6f640 10 monclient: build_initial_monmap 2024-09-18T12:41:53.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.562+0000 7f2d85e6f640 10 monclient: monmap: 2024-09-18T12:41:53.563 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:53.563425+0000 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:53.563425+0000 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:53.564 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.563+0000 7f2d85e6f640 1 Processor -- start 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.563+0000 7f2d85e6f640 1 -- start start 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.563+0000 7f2d85e6f640 10 monclient: init 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d800a1ef0 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 0x7f2d8009b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient(hunting): picked mon.noname-a con 0x7f2d8009c6a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d80136d00 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8006f010 con 0x7f2d80136d00 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8009dcc0 con 0x7f2d800a1ef0 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8009da00 con 0x7f2d8009c6a0 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d85e6f640 10 monclient(hunting): authenticate will time out at 1740.885225s 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d77fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 0x7f2d8009b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d77fff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 msgr2=0x7f2d8009b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d77fff640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 msgr2=0x7f2d8009b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d77fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 0x7f2d8009b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d77fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 0x7f2d8009b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d7ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2d80136d00 0x7f2d801390e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46882/0 (socket says 172.21.15.116:46882) 2024-09-18T12:41:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.564+0000 7f2d7ffff640 1 -- 172.21.15.116:0/784941374 learned_addr learned my addr 172.21.15.116:0/784941374 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7ffff640 1 -- 172.21.15.116:0/784941374 >> v1:172.21.15.116:6789/0 conn(0x7f2d80136d00 legacy=0x7f2d801390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813469940 0 0) 0x7f2d8006f010 con 0x7f2d80136d00 2024-09-18T12:41:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 10 monclient(hunting): my global_id is 194214 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d6c0034e0 con 0x7f2d80136d00 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 964992904 0 0) 0x7f2d6c0034e0 con 0x7f2d80136d00 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 msgr2=0x7f2d8009b2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7f7fe640 1 -- 172.21.15.116:0/784941374 >> v1:172.21.15.144:6789/0 conn(0x7f2d800a1ef0 legacy=0x7f2d8009abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 --2- 172.21.15.116:0/784941374 >> v2:172.21.15.116:3300/0 conn(0x7f2d8009c6a0 0x7f2d8009b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 >> v1:172.21.15.144:6789/0 conn(0x7f2d800a1ef0 legacy=0x7f2d8009abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d800a22c0 con 0x7f2d80136d00 2024-09-18T12:41:53.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.566+0000 7f2d7effd640 10 monclient: _finish_auth 0 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d85e6f640 5 monclient: authenticate success, global_id 194214 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f2d78001b10 con 0x7f2d80136d00 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d85e6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient: dump: 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.579 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 >> v1:172.21.15.116:6789/0 conn(0x7f2d80136d00 legacy=0x7f2d801390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d6c003bc0 0x7f2d6c003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): picked mon.c con 0x7f2d6c003bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d6c004d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): picked mon.a con 0x7f2d6c0048a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d6c0099c0 0x7f2d6c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): picked mon.b con 0x7f2d6c0099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c00c470 con 0x7f2d6c0048a0 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c00c5b0 con 0x7f2d6c0099c0 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c00c770 con 0x7f2d6c003bc0 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7effd640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d85e6f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d6c004d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d6c003bc0 0x7f2d6c003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): get_auth_request con 0x7f2d6c0048a0 auth_method 0 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d6c003bc0 auth_method 0 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient(hunting): handle_auth_done global_id 194214 payload 293 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 1 -- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d6c003bc0 msgr2=0x7f2d6c003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:53.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d6c003bc0 0x7f2d6c003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 1 -- 172.21.15.116:0/784941374 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d6c0099c0 msgr2=0x7f2d6c00bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d6c0099c0 0x7f2d6c00bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 1 monclient: found mon.a 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7ffff640 1 -- 172.21.15.116:0/784941374 reap_dead start 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 1 -- 172.21.15.116:0/784941374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c0034e0 con 0x7f2d6c0048a0 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 10 monclient: _finish_auth 0 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.567+0000 7f2d7f7fe640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d6c003bc0 0x7f2d6c003fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d77fff640 1 -- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 msgr2=0x7f2d6c004d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d77fff640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d6c004d10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800aa50 tx=0x7f2d68008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8efa0c2948aca9c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6802ab80 con 0x7f2d6c0048a0 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 10 monclient: dump: 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6802ad00 con 0x7f2d6c0048a0 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d85e6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d85e6f640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 1 -- 172.21.15.116:0/784941374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68032690 con 0x7f2d6c0048a0 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d85e6f640 4 set_mon_vals no callback set 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d7effd640 10 monclient: dump: 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d85e6f640 10 monclient: shutdown 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d85e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d85e6f640 1 -- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 msgr2=0x7f2d6c004d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.570+0000 7f2d85e6f640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d6c004d10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800aa50 tx=0x7f2d68008040 comp rx=0 tx=0).stop 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 1 -- 172.21.15.116:0/784941374 shutdown_connections 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 1 --2- 172.21.15.116:0/784941374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d6c004d10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 1 -- 172.21.15.116:0/784941374 >> 172.21.15.116:0/784941374 conn(0x7f2d800678a0 msgr2=0x7f2d800a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 1 -- 172.21.15.116:0/784941374 shutdown_connections 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 1 -- 172.21.15.116:0/784941374 wait complete. 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 10 monclient: build_initial_monmap 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 10 monclient: monmap: 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:53.587 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 1 Processor -- start 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 1 -- start start 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.571+0000 7f2d85e6f640 10 monclient: init 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d80126630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6c0048a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80126b70 0x7f2d80126f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f2d80126b70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d800940b0 0x7f2d80094520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): picked mon.noname-c con 0x7f2d800940b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8009b910 con 0x7f2d6c0048a0 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80099180 con 0x7f2d80126b70 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d801274a0 con 0x7f2d800940b0 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d85e6f640 10 monclient(hunting): authenticate will time out at 1740.893310s 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d80126630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d800940b0 0x7f2d80094520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d80126630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36168/0 (socket says 172.21.15.116:36168) 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): get_auth_request con 0x7f2d800940b0 auth_method 0 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 1 -- 172.21.15.116:0/2683128046 learned_addr learned my addr 172.21.15.116:0/2683128046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d6c0048a0 auth_method 0 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d77fff640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80126b70 0x7f2d80126f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d77fff640 10 monclient(hunting): get_auth_request con 0x7f2d80126b70 auth_method 0 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient(hunting): handle_auth_done global_id 204110 payload 824 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80126b70 msgr2=0x7f2d80126f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80126b70 0x7f2d80126f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 msgr2=0x7f2d80126630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d80126630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 1 monclient: found mon.noname-c 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7ffff640 1 -- 172.21.15.116:0/2683128046 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d801277f0 con 0x7f2d800940b0 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.572+0000 7f2d7f7fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6c0048a0 0x7f2d80126630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.574+0000 7f2d7ffff640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d800940b0 msgr2=0x7f2d80094520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.574+0000 7f2d7ffff640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d800940b0 0x7f2d80094520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d80069f40 tx=0x7f2d78001b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78004b20 con 0x7f2d800940b0 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: dump: 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 5 monclient: authenticate success, global_id 204110 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d78004ce0 con 0x7f2d800940b0 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 10 monclient: _renew_subs 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d80129320 con 0x7f2d800940b0 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d780073e0 con 0x7f2d800940b0 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:53.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 10 monclient: dump: 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 10 monclient: _renew_subs 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d7e7fc640 4 set_mon_vals no callback set 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:53.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8012a5a0 con 0x7f2d800940b0 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 0x7f2d801317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80065e00 con 0x7f2d80094a60 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.575+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2d78007790 con 0x7f2d800940b0 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d777fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d600384c0 0x7f2d6003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2d7804b640 con 0x7f2d800940b0 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 0x7f2d801317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient: get_auth_request con 0x7f2d80094a60 auth_method 0 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d77fff640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d600384c0 0x7f2d6003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d77fff640 10 monclient: get_auth_request con 0x7f2d600384c0 auth_method 0 2024-09-18T12:41:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_done global_id 174403 payload 824 2024-09-18T12:41:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.576+0000 7f2d7f7fe640 1 -- 172.21.15.116:0/2683128046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d80128560 con 0x7f2d80094a60 2024-09-18T12:41:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.578+0000 7f2d77fff640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d600384c0 0x7f2d6003a970 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800aa50 tx=0x7f2d68035000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.578+0000 7f2d7f7fe640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 msgr2=0x7f2d801317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.578+0000 7f2d7f7fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 0x7f2d801317a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000c00 tx=0x7f2d70009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d700076a0 con 0x7f2d80094a60 2024-09-18T12:41:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 10 monclient: dump: 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2d70010950 con 0x7f2d80094a60 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 msgr2=0x7f2d801317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 0x7f2d801317a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000c00 tx=0x7f2d70009040 comp rx=0 tx=0).stop 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 msgr2=0x7f2d801317a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.579+0000 7f2d777fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80094a60 0x7f2d801317a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d85e6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d85e6f640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 0x7f2d801320e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d85e6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80128090 con 0x7f2d80131ce0 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d7f7fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 0x7f2d801320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d7f7fe640 10 monclient: get_auth_request con 0x7f2d80131ce0 auth_method 0 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.585+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.586+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.586+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.586+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.586+0000 7f2d7f7fe640 10 monclient(hunting): handle_auth_done global_id 174406 payload 824 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.586+0000 7f2d7f7fe640 1 -- 172.21.15.116:0/2683128046 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2d8012a790 con 0x7f2d80131ce0 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.588+0000 7f2d7f7fe640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 msgr2=0x7f2d801320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.588+0000 7f2d7f7fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 0x7f2d801320e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000c00 tx=0x7f2d7001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7001fc50 con 0x7f2d80131ce0 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 10 monclient: dump: 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f2d7001fdb0 con 0x7f2d80131ce0 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 msgr2=0x7f2d801320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 0x7f2d801320e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000c00 tx=0x7f2d7001b000 comp rx=0 tx=0).stop 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 msgr2=0x7f2d801320e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.589+0000 7f2d777fe640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d80131ce0 0x7f2d801320e0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d70000c00 tx=0x7f2d7001b000 comp rx=0 tx=0).stop 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 162, 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:53.603 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 6, 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:53.604 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:53.605 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:53.606 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:53.607 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:53.608 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:53.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:53.610 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:53.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:53.613 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:53.614 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:53.615 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d600384c0 msgr2=0x7f2d6003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2d600384c0 0x7f2d6003a970 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800aa50 tx=0x7f2d68035000 comp rx=0 tx=0).stop 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 10 monclient: shutdown 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d800940b0 msgr2=0x7f2d80094520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 --2- 172.21.15.116:0/2683128046 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2d800940b0 0x7f2d80094520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2d80069f40 tx=0x7f2d78001b60 comp rx=0 tx=0).stop 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d7ffff640 1 -- 172.21.15.116:0/2683128046 reap_dead start 2024-09-18T12:41:53.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d7ffff640 1 -- 172.21.15.116:0/2683128046 reap_dead start 2024-09-18T12:41:53.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 shutdown_connections 2024-09-18T12:41:53.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 >> 172.21.15.116:0/2683128046 conn(0x7f2d800678a0 msgr2=0x7f2d800a0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:53.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 shutdown_connections 2024-09-18T12:41:53.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.590+0000 7f2d85e6f640 1 -- 172.21.15.116:0/2683128046 wait complete. 2024-09-18T12:41:53.618 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:53.618 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:41:53.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.710+0000 7f5693f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.710+0000 7f5693f9d640 10 monclient: build_initial_monmap 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.710+0000 7f5693f9d640 10 monclient: monmap: 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:53.710919+0000 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:53.710919+0000 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:53.711 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:53.712 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:53.712 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:53.712 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:53.712 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.710+0000 7f5693f9d640 1 Processor -- start 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 1 -- start start 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient: init 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f568c131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f568c09e910 0x7f568c09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f568c09e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f568c09f250 0x7f568c138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f568c09f250 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568c09faf0 con 0x7f568c131ab0 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c132590 con 0x7f568c09f250 2024-09-18T12:41:53.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c09fd30 con 0x7f568c09e910 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f5693f9d640 10 monclient(hunting): authenticate will time out at 1741.032493s 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f568c09f250 0x7f568c138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f568c09f250 0x7f568c138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36176/0 (socket says 172.21.15.116:36176) 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 -- 172.21.15.116:0/902506362 learned_addr learned my addr 172.21.15.116:0/902506362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): get_auth_request con 0x7f568c09f250 auth_method 0 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient(hunting): handle_auth_done global_id 194220 payload 274 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 -- 172.21.15.116:0/902506362 >> v2:172.21.15.144:3300/0 conn(0x7f568c09e910 msgr2=0x7f568c09ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 --2- 172.21.15.116:0/902506362 >> v2:172.21.15.144:3300/0 conn(0x7f568c09e910 0x7f568c09ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 -- 172.21.15.116:0/902506362 >> v1:172.21.15.144:6789/0 conn(0x7f568c131ab0 legacy=0x7f568c131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient: _finish_hunting 0 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 monclient: found mon.noname-a 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:53.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.711+0000 7f569253a640 1 -- 172.21.15.116:0/902506362 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0a1110 con 0x7f568c09f250 2024-09-18T12:41:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f569253a640 1 -- 172.21.15.116:0/902506362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09f250 msgr2=0x7f568c138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f569253a640 1 --2- 172.21.15.116:0/902506362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09f250 0x7f568c138bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f568c09bdb0 tx=0x7f567c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a71c3aa3c1bb97 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 1 -- 172.21.15.116:0/902506362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c007610 con 0x7f568c09f250 2024-09-18T12:41:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: dump: 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: _finish_auth 0 2024-09-18T12:41:53.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 1 -- 172.21.15.116:0/902506362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567c02d050 con 0x7f568c09f250 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5693f9d640 5 monclient: authenticate success, global_id 194220 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5693f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5693f9d640 4 set_mon_vals no callback set 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 1 -- 172.21.15.116:0/902506362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c0079e0 con 0x7f568c09f250 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.714+0000 7f5690d37640 10 monclient: dump: 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 10 monclient: shutdown 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 -- 172.21.15.116:0/902506362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09f250 msgr2=0x7f568c138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 --2- 172.21.15.116:0/902506362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09f250 0x7f568c138bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f568c09bdb0 tx=0x7f567c009840 comp rx=0 tx=0).stop 2024-09-18T12:41:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 -- 172.21.15.116:0/902506362 shutdown_connections 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 --2- 172.21.15.116:0/902506362 >> v2:172.21.15.144:3300/0 conn(0x7f568c09e910 0x7f568c09ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 --2- 172.21.15.116:0/902506362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09f250 0x7f568c138bc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 -- 172.21.15.116:0/902506362 >> 172.21.15.116:0/902506362 conn(0x7f568c067d30 msgr2=0x7f568c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 -- 172.21.15.116:0/902506362 shutdown_connections 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 1 -- 172.21.15.116:0/902506362 wait complete. 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 10 monclient: build_initial_monmap 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.715+0000 7f5693f9d640 10 monclient: monmap: 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:53.726 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 Processor -- start 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 -- start start 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient: init 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 0x7f568c145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f568c09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f568c09f250 0x7f568c145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f568c09f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:53.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f568c131ab0 0x7f568c1461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f568c131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c06e240 con 0x7f568c09e910 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c06d930 con 0x7f568c131ab0 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c06d650 con 0x7f568c09f250 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5693f9d640 10 monclient(hunting): authenticate will time out at 1741.037730s 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 0x7f568c145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 0x7f568c145730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36192/0 (socket says 172.21.15.116:36192) 2024-09-18T12:41:53.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 1 -- 172.21.15.116:0/2475705138 learned_addr learned my addr 172.21.15.116:0/2475705138 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 10 monclient(hunting): get_auth_request con 0x7f568c09e910 auth_method 0 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f568c09f250 0x7f568c145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 10 monclient(hunting): get_auth_request con 0x7f568c09f250 auth_method 0 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.716+0000 7f5691538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 10 monclient(hunting): handle_auth_done global_id 194223 payload 824 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 1 -- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f568c09f250 msgr2=0x7f568c145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f568c09f250 0x7f568c145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 1 -- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f568c131ab0 msgr2=0x7f568c1461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f568c131ab0 0x7f568c1461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 10 monclient: _finish_hunting 0 2024-09-18T12:41:53.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 1 monclient: found mon.noname-a 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.717+0000 7f5691d39640 1 -- 172.21.15.116:0/2475705138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c009ae0 con 0x7f568c09e910 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5691d39640 1 -- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 msgr2=0x7f568c145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5691d39640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 0x7f568c145730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f568400b5e0 tx=0x7f56840090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 1 -- 172.21.15.116:0/2475705138 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568400f7a0 con 0x7f568c09e910 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: dump: 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.731 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: _finish_auth 0 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 1 -- 172.21.15.116:0/2475705138 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568400f920 con 0x7f568c09e910 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5693f9d640 5 monclient: authenticate success, global_id 194223 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5693f9d640 10 monclient: _renew_subs 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5693f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:53.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c06dc10 con 0x7f568c09e910 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 1 -- 172.21.15.116:0/2475705138 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568400fce0 con 0x7f568c09e910 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5689ffb640 10 monclient: dump: 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:53.733 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f568bfff640 4 set_mon_vals no callback set 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5693f9d640 10 monclient: _renew_subs 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5693f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.719+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c06d240 con 0x7f568c09e910 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5689ffb640 1 -- 172.21.15.116:0/2475705138 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5684010630 con 0x7f568c09e910 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5693f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5693f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5693f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c04bf60 con 0x7f568c09e910 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5689ffb640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5678038490 0x7f567803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5689ffb640 1 -- 172.21.15.116:0/2475705138 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f568404bec0 con 0x7f568c09e910 2024-09-18T12:41:53.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.720+0000 7f5691538640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5678038490 0x7f567803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:53.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.721+0000 7f5691538640 10 monclient: get_auth_request con 0x7f5678038490 auth_method 0 2024-09-18T12:41:53.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.723+0000 7f5691538640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5678038490 0x7f567803a940 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f568c131700 tx=0x7f5680009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:53.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:53.724+0000 7f5691d39640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 0x7f568c145730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f568400b5e0 tx=0x7f56840090f0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f5684017d30 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T12:41:54.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.553+0000 7f5689ffb640 1 -- 172.21.15.116:0/2475705138 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5684017d30 con 0x7f568c09e910 2024-09-18T12:41:54.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.553+0000 7f5689ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:54.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.553+0000 7f5689ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.770+0000 7f5693f9d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:41:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.770+0000 7f5693f9d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:41:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.770+0000 7f5693f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.770+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f568c066280 con 0x7f568c09e910 2024-09-18T12:41:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f5689ffb640 1 -- 172.21.15.116:0/2475705138 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f5684019ed0 con 0x7f568c09e910 2024-09-18T12:41:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f5689ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:41:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f5689ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:54.772 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.772 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=9,logm=51} crc {auth=648736922,config=3140514009,health=3291967371,kv=2187923472,logm=3374440750}) 2024-09-18T12:41:54.772 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=9,logm=51} crc {auth=542013477,config=3140514009,health=1100982214,kv=2187923472,logm=1496926326}) 2024-09-18T12:41:54.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=9,logm=51} crc {auth=648736922,config=3140514009,health=3291967371,kv=2187923472,logm=3374440750}) 2024-09-18T12:41:54.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=912498208,config=3140514009,health=1083522342,kv=771457840,logm=3598218257}) 2024-09-18T12:41:54.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.772+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.772+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301200185}) 2024-09-18T12:41:54.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.772+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=692480552}) 2024-09-18T12:41:54.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.772+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.772+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301200185}) 2024-09-18T12:41:54.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.772+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4176734993}) 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5678038490 msgr2=0x7f567803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5678038490 0x7f567803a940 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f568c131700 tx=0x7f5680009060 comp rx=0 tx=0).stop 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 10 monclient: shutdown 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 msgr2=0x7f568c145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 0x7f568c145730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f568400b5e0 tx=0x7f56840090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 shutdown_connections 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5678038490 0x7f567803a940 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:54.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f568c131ab0 0x7f568c1461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:54.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f568c09f250 0x7f568c145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:54.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 --2- 172.21.15.116:0/2475705138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f568c09e910 0x7f568c145730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:54.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 >> 172.21.15.116:0/2475705138 conn(0x7f568c067d30 msgr2=0x7f568c068ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:54.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 shutdown_connections 2024-09-18T12:41:54.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:54.773+0000 7f5693f9d640 1 -- 172.21.15.116:0/2475705138 wait complete. 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.774+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.774+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557630467}) 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.774+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640770442}) 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.774+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.774+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557630467}) 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.774+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479264965}) 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.775+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.775+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2513823541}) 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.775+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349445106}) 2024-09-18T12:41:54.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.775+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.775+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2513823541}) 2024-09-18T12:41:54.778 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.775+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162796569}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.777+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.777+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649872460}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.777+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443922149}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.777+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.777+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649872460}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.777+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590395561}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.778+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.778+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341889547}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.778+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52539005}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.778+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.778+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341889547}) 2024-09-18T12:41:54.779 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.778+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195760008}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.779+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.779+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3978630162}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.779+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893804594}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.779+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.779+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3978630162}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.779+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2370172223}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521086442}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1137966695}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521086442}) 2024-09-18T12:41:54.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3604337457}) 2024-09-18T12:41:54.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227857875}) 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434312913}) 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2227857875}) 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575532534}) 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1403582491}) 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=432831196}) 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1403582491}) 2024-09-18T12:41:54.783 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=764911647}) 2024-09-18T12:41:54.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293707283}) 2024-09-18T12:41:54.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4148004215}) 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293707283}) 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4120026400}) 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.783+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.783+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689011367}) 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.783+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678617447}) 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.783+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.783+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689011367}) 2024-09-18T12:41:54.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.783+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240082429}) 2024-09-18T12:41:54.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.784+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.784+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416830049}) 2024-09-18T12:41:54.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.784+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984860263}) 2024-09-18T12:41:54.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.784+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.784+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416830049}) 2024-09-18T12:41:54.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.784+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112462199}) 2024-09-18T12:41:54.787 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79509595}) 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1553453468}) 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79509595}) 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448177202}) 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1143859633}) 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3304952408}) 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.785+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1143859633}) 2024-09-18T12:41:54.788 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=127375629}) 2024-09-18T12:41:54.789 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:41:54.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.792 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774191794}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751698472}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774191794}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.786+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3093553770}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.787+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.787+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552885571}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.787+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596590320}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.787+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.787+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552885571}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.787+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206574615}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.788+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.788+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608310154}) 2024-09-18T12:41:54.793 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.788+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034902342}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.788+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.788+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608310154}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.788+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067879568}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.790+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.790+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161855385}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.790+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2456334738}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.790+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.790+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161855385}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.790+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1015954473}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156343229}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843816540}) 2024-09-18T12:41:54.794 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4156343229}) 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989811656}) 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975526465}) 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3939797990}) 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975526465}) 2024-09-18T12:41:54.795 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.791+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163796926}) 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.792+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.792+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3739079691}) 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.792+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3593732547}) 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.792+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.792+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3739079691}) 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.792+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=13476202}) 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.793+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.793+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565931048}) 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.793+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612814278}) 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.793+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.804 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.793+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565931048}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.793+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1155839513}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029219547}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722021645}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029219547}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2446740943}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461492210}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246678395}) 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.805 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461492210}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.794+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1304247916}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.795+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.795+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939018197}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.795+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4053787101}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.795+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.795+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939018197}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.795+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3848258302}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.796+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.796+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770475166}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.796+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1312758484}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.796+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.796+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770475166}) 2024-09-18T12:41:54.806 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.796+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=341233824}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.796+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205951753}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1847189224}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205951753}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2523725334}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726813154}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1322441420}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726813154}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.797+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812081623}) 2024-09-18T12:41:54.807 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.798+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.798+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893059950}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.798+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=820337217}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.798+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.798+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893059950}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.798+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=883142283}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366268704}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3085063297}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366268704}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255695220}) 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.808 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366005532}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642472374}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366005532}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.799+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197242357}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.800+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.800+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429736009}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.800+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4110080359}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.800+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.800+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429736009}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.800+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417560788}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.801+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.801+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638075869}) 2024-09-18T12:41:54.809 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.801+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=239530827}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.801+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.801+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638075869}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.801+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=540886518}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.802+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.802+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517065618}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.802+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513782177}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.802+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.802+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517065618}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.802+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4155673604}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743004035}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3157867822}) 2024-09-18T12:41:54.810 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743004035}) 2024-09-18T12:41:54.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782856698}) 2024-09-18T12:41:54.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.803+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879311220}) 2024-09-18T12:41:54.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.804+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566314496}) 2024-09-18T12:41:54.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.804+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:54.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.804+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879311220}) 2024-09-18T12:41:54.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:54.804+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266537198}) 2024-09-18T12:41:55.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601284981}) 2024-09-18T12:41:55.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2321057695}) 2024-09-18T12:41:55.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601284981}) 2024-09-18T12:41:55.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575453657}) 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.049+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.049+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372806842}) 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.049+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025877391}) 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.049+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.049+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372806842}) 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.049+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579559022}) 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.050+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.050+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234566486}) 2024-09-18T12:41:55.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.050+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=935997283}) 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.050+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.050+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234566486}) 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.050+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425062677}) 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.051+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.051+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471970629}) 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.051+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872998168}) 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.051+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.051+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471970629}) 2024-09-18T12:41:55.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.051+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177955165}) 2024-09-18T12:41:55.057 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.052+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.052+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845437642}) 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.052+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1244035323}) 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.052+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.052+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845437642}) 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.052+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248468456}) 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764678455}) 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461198358}) 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.058 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764678455}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394593182}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404813702}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3437317148}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404813702}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.053+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1601871100}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.054+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.054+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766024288}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.054+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327841915}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.054+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.054+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766024288}) 2024-09-18T12:41:55.059 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.054+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612634796}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.055+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.055+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455187863}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.055+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615452875}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.055+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.055+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455187863}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.055+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839170096}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.056+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.056+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861994406}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.056+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3519477697}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.056+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.056+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861994406}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.056+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=992644891}) 2024-09-18T12:41:55.060 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833425952}) 2024-09-18T12:41:55.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3803485812}) 2024-09-18T12:41:55.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833425952}) 2024-09-18T12:41:55.061 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606218831}) 2024-09-18T12:41:55.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.066 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379698686}) 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070132703}) 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379698686}) 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.057+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941843582}) 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.058+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.058+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526653737}) 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.058+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388463266}) 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.058+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.067 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.058+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526653737}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.058+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902690021}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.059+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.059+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203349083}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.059+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=525872134}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.059+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.059+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203349083}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.060+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649380473}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.060+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.060+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1144576838}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.060+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572485032}) 2024-09-18T12:41:55.068 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.060+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.060+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1144576838}) 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.060+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684115844}) 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.061+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.061+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2549749370}) 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.061+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2487461972}) 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.061+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.061+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2549749370}) 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.061+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196094549}) 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.062+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.062+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1538617309}) 2024-09-18T12:41:55.069 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.062+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3550787925}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.062+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.062+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1538617309}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.062+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098127053}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.063+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.063+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933187677}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.063+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339340716}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.063+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.063+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933187677}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.063+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287507077}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.064+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.064+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429822535}) 2024-09-18T12:41:55.070 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.064+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991390341}) 2024-09-18T12:41:55.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.064+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.064+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429822535}) 2024-09-18T12:41:55.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.064+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=521090866}) 2024-09-18T12:41:55.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543416682}) 2024-09-18T12:41:55.071 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584916902}) 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543416682}) 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43951721}) 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263128881}) 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3869924007}) 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263128881}) 2024-09-18T12:41:55.072 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.065+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3599607016}) 2024-09-18T12:41:55.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.066+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.066+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832089453}) 2024-09-18T12:41:55.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.066+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043486116}) 2024-09-18T12:41:55.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.066+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.066+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832089453}) 2024-09-18T12:41:55.077 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.066+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3223081745}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.067+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.067+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533964368}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.067+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4180950114}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.067+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.067+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533964368}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.067+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309923416}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278781660}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2804008600}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278781660}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2478282682}) 2024-09-18T12:41:55.078 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239353277}) 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.068+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3360815566}) 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239353277}) 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452651026}) 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514300306}) 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1609283719}) 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514300306}) 2024-09-18T12:41:55.079 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.069+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964265799}) 2024-09-18T12:41:55.328 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790680455}) 2024-09-18T12:41:55.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4285949594}) 2024-09-18T12:41:55.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790680455}) 2024-09-18T12:41:55.329 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.327+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579525103}) 2024-09-18T12:41:55.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420976603}) 2024-09-18T12:41:55.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3404159163}) 2024-09-18T12:41:55.332 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420976603}) 2024-09-18T12:41:55.333 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.331+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458777131}) 2024-09-18T12:41:55.344 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.343+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.344 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.343+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812976380}) 2024-09-18T12:41:55.344 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.343+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8080584}) 2024-09-18T12:41:55.345 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.343+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.345 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.343+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812976380}) 2024-09-18T12:41:55.345 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.343+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3148308756}) 2024-09-18T12:41:55.426 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.426+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.426+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210247072}) 2024-09-18T12:41:55.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.426+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1408349641}) 2024-09-18T12:41:55.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.426+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.426+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210247072}) 2024-09-18T12:41:55.427 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.426+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619018573}) 2024-09-18T12:41:55.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.427+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.429 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.427+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440769477}) 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.427+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890151480}) 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.427+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.427+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440769477}) 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.427+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2474769822}) 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.428+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.428+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90457438}) 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.428+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146212317}) 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.428+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.428+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90457438}) 2024-09-18T12:41:55.430 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.428+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=672267500}) 2024-09-18T12:41:55.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.435+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.436 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.435+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651021260}) 2024-09-18T12:41:55.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.435+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=909346293}) 2024-09-18T12:41:55.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.435+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.435+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651021260}) 2024-09-18T12:41:55.437 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.435+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2107281200}) 2024-09-18T12:41:55.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.436+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.436+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252689476}) 2024-09-18T12:41:55.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.436+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984377896}) 2024-09-18T12:41:55.438 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.436+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.436+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252689476}) 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.436+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570184794}) 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.437+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.437+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687900878}) 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.437+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234948449}) 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.437+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.437+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687900878}) 2024-09-18T12:41:55.439 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.437+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=355822123}) 2024-09-18T12:41:55.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.438+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.438+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012630287}) 2024-09-18T12:41:55.441 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.438+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833657381}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.438+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.438+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012630287}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.438+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193116876}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046143310}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=634425069}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046143310}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.439+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394460797}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.440+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.440+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1787005334}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.440+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3939132116}) 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.440+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.442 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.440+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1787005334}) 2024-09-18T12:41:55.443 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.440+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111485940}) 2024-09-18T12:41:55.790 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:41:55.790 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:55.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803769036}) 2024-09-18T12:41:55.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1951569574}) 2024-09-18T12:41:55.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803769036}) 2024-09-18T12:41:55.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166778349}) 2024-09-18T12:41:55.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235734637}) 2024-09-18T12:41:55.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975427403}) 2024-09-18T12:41:55.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:55.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235734637}) 2024-09-18T12:41:55.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:55.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1596735359}) 2024-09-18T12:41:55.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.897+0000 7f835a19d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:55.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.897+0000 7f835a19d640 10 monclient: build_initial_monmap 2024-09-18T12:41:55.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.897+0000 7f835a19d640 10 monclient: monmap: 2024-09-18T12:41:55.898 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:55.898 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:55.898 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:55.898104+0000 2024-09-18T12:41:55.898 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:55.898104+0000 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:55.899 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:55.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 1 Processor -- start 2024-09-18T12:41:55.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 1 -- start start 2024-09-18T12:41:55.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient: init 2024-09-18T12:41:55.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:55.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8354096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83540971e0 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8354134b40 0x7f8354136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient(hunting): picked mon.noname-c con 0x7f8354134b40 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835409d4a0 con 0x7f8354096e10 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8354099620 con 0x7f83540971e0 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8354099340 con 0x7f8354134b40 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f835a19d640 10 monclient(hunting): authenticate will time out at 1743.219769s 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f83537fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8354096e10 0x7f835409a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39900/0 (socket says 172.21.15.116:39900) 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.898+0000 7f83537fe640 1 -- 172.21.15.116:0/794695304 learned_addr learned my addr 172.21.15.116:0/794695304 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:55.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 1 --2- 172.21.15.116:0/794695304 >> v2:172.21.15.144:3300/0 conn(0x7f8354134b40 0x7f8354136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 10 monclient(hunting): get_auth_request con 0x7f8354134b40 auth_method 0 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.899+0000 7f8353fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 1 -- 172.21.15.116:0/794695304 >> v1:172.21.15.116:6789/0 conn(0x7f8354096e10 legacy=0x7f835409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991689041 0 0) 0x7f835409d4a0 con 0x7f8354096e10 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): my global_id is 194226 2024-09-18T12:41:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c0034e0 con 0x7f8354096e10 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572958735 0 0) 0x7f833c0034e0 con 0x7f8354096e10 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 >> v2:172.21.15.144:3300/0 conn(0x7f8354134b40 msgr2=0x7f8354136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 --2- 172.21.15.116:0/794695304 >> v2:172.21.15.144:3300/0 conn(0x7f8354134b40 0x7f8354136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 -- 172.21.15.116:0/794695304 >> v1:172.21.15.144:6789/0 conn(0x7f83540971e0 legacy=0x7f83540975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 >> v1:172.21.15.144:6789/0 conn(0x7f83540971e0 legacy=0x7f83540975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: _finish_hunting 0 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8354099900 con 0x7f8354096e10 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f8344001850 con 0x7f8354096e10 2024-09-18T12:41:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f835a19d640 5 monclient: authenticate success, global_id 194226 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f835a19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: dump: 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:55.908 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 >> v1:172.21.15.116:6789/0 conn(0x7f8354096e10 legacy=0x7f835409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c003b20 0x7f833c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): picked mon.c con 0x7f833c003b20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f833c004780 0x7f833c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): picked mon.a con 0x7f833c004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f833c005330 0x7f833c00b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): picked mon.b con 0x7f833c005330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8354099340 con 0x7f833c004780 2024-09-18T12:41:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c00c2d0 con 0x7f833c005330 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c00c490 con 0x7f833c003b20 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83527fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f835a19d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f833c004780 0x7f833c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c003b20 0x7f833c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 10 monclient(hunting): get_auth_request con 0x7f833c003b20 auth_method 0 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): get_auth_request con 0x7f833c004780 auth_method 0 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient(hunting): handle_auth_done global_id 194226 payload 293 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8353fff640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f833c005330 0x7f833c00b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 -- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c003b20 msgr2=0x7f833c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c003b20 0x7f833c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 -- 172.21.15.116:0/794695304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f833c005330 msgr2=0x7f833c00b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f833c005330 0x7f833c00b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 monclient: found mon.a 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 1 -- 172.21.15.116:0/794695304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0034e0 con 0x7f833c004780 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 10 monclient: _finish_auth 0 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8352ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8353fff640 10 monclient: get_auth_request con 0x7f833c005330 auth_method 0 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8353fff640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f833c005330 0x7f833c00b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:41:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.901+0000 7f8353fff640 1 -- 172.21.15.116:0/794695304 reap_dead start 2024-09-18T12:41:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f8352ffd640 1 -- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f833c004780 msgr2=0x7f833c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f8352ffd640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f833c004780 0x7f833c004bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8348001b10 tx=0x7f8348007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e192f62c6877663c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834802ab10 con 0x7f833c004780 2024-09-18T12:41:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 10 monclient: dump: 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f835a19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834802ac70 con 0x7f833c004780 2024-09-18T12:41:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 1 -- 172.21.15.116:0/794695304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83480070f0 con 0x7f833c004780 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f835a19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f835a19d640 4 set_mon_vals no callback set 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.904+0000 7f83527fc640 10 monclient: dump: 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 10 monclient: shutdown 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 1 -- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f833c004780 msgr2=0x7f833c004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f833c004780 0x7f833c004bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8348001b10 tx=0x7f8348007c80 comp rx=0 tx=0).stop 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 1 -- 172.21.15.116:0/794695304 shutdown_connections 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 1 --2- 172.21.15.116:0/794695304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f833c004780 0x7f833c004bf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 1 -- 172.21.15.116:0/794695304 >> 172.21.15.116:0/794695304 conn(0x7f83540672b0 msgr2=0x7f83540676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 1 -- 172.21.15.116:0/794695304 shutdown_connections 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 1 -- 172.21.15.116:0/794695304 wait complete. 2024-09-18T12:41:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 10 monclient: build_initial_monmap 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.905+0000 7f835a19d640 10 monclient: monmap: 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 Processor -- start 2024-09-18T12:41:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 -- start start 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient: init 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c004780 0x7f835409b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): picked mon.noname-c con 0x7f833c004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835412e760 0x7f835412eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): picked mon.noname-b con 0x7f835412e760 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f835406e1c0 0x7f835406e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): picked mon.noname-a con 0x7f835406e1c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835409b050 con 0x7f835406e1c0 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83541343b0 con 0x7f835412e760 2024-09-18T12:41:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8354119980 con 0x7f833c004780 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f835a19d640 10 monclient(hunting): authenticate will time out at 1743.227760s 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c004780 0x7f835409b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f835406e1c0 0x7f835406e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c004780 0x7f835409b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44448/0 (socket says 172.21.15.116:44448) 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 1 -- 172.21.15.116:0/210887805 learned_addr learned my addr 172.21.15.116:0/210887805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 10 monclient(hunting): get_auth_request con 0x7f833c004780 auth_method 0 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 10 monclient(hunting): get_auth_request con 0x7f835406e1c0 auth_method 0 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f8353fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.906+0000 7f83537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8352ffd640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835412e760 0x7f835412eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 10 monclient(hunting): handle_auth_done global_id 194229 payload 824 2024-09-18T12:41:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 1 -- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c004780 msgr2=0x7f835409b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c004780 0x7f835409b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 1 -- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835412e760 msgr2=0x7f835412eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835412e760 0x7f835412eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 10 monclient: _finish_hunting 0 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 1 monclient: found mon.noname-a 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f8353fff640 1 -- 172.21.15.116:0/210887805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835411a870 con 0x7f835406e1c0 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.907+0000 7f83537fe640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c004780 0x7f835409b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f8353fff640 1 -- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f835406e1c0 msgr2=0x7f835406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f8353fff640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f835406e1c0 0x7f835406e660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8354098510 tx=0x7f834c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 1 -- 172.21.15.116:0/210887805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c0147f0 con 0x7f835406e1c0 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: dump: 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:55.927 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 1 -- 172.21.15.116:0/210887805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c014950 con 0x7f835406e1c0 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f835a19d640 5 monclient: authenticate success, global_id 194229 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f835a19d640 10 monclient: _renew_subs 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f835a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835411ab50 con 0x7f835406e1c0 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 1 -- 172.21.15.116:0/210887805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c014cf0 con 0x7f835406e1c0 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f833b7fe640 10 monclient: dump: 2024-09-18T12:41:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f835a19d640 10 monclient: _renew_subs 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f835a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8354135420 con 0x7f835406e1c0 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.909+0000 7f8351ffb640 4 set_mon_vals no callback set 2024-09-18T12:41:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.910+0000 7f833b7fe640 1 -- 172.21.15.116:0/210887805 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f834c016630 con 0x7f835406e1c0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.910+0000 7f835a19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.910+0000 7f835a19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.910+0000 7f835a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.910+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835404bf60 con 0x7f835406e1c0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.910+0000 7f833b7fe640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8340038490 0x7f834003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.910+0000 7f833b7fe640 1 -- 172.21.15.116:0/210887805 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f834c04c6e0 con 0x7f835406e1c0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.911+0000 7f83537fe640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8340038490 0x7f834003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.911+0000 7f83537fe640 10 monclient: get_auth_request con 0x7f8340038490 auth_method 0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.913+0000 7f83537fe640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8340038490 0x7f834003a940 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f83440014a0 tx=0x7f8344007b10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.914+0000 7f833b7fe640 1 -- 172.21.15.116:0/210887805 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f834c01f410 con 0x7f835406e1c0 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.914+0000 7f833b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:55.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:55.914+0000 7f833b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.126+0000 7f835a19d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.126+0000 7f835a19d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.126+0000 7f835a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.126+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f835406f740 con 0x7f835406e1c0 2024-09-18T12:41:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.126+0000 7f833b7fe640 1 -- 172.21.15.116:0/210887805 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f834c01e370 con 0x7f835406e1c0 2024-09-18T12:41:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.126+0000 7f833b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:56.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.126+0000 7f833b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:56.128 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:56.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8340038490 msgr2=0x7f834003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8340038490 0x7f834003a940 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f83440014a0 tx=0x7f8344007b10 comp rx=0 tx=0).stop 2024-09-18T12:41:56.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 10 monclient: shutdown 2024-09-18T12:41:56.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f835406e1c0 msgr2=0x7f835406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f835406e1c0 0x7f835406e660 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8354098510 tx=0x7f834c00b040 comp rx=0 tx=0).stop 2024-09-18T12:41:56.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 shutdown_connections 2024-09-18T12:41:56.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8340038490 0x7f834003a940 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f835406e1c0 0x7f835406e660 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f835412e760 0x7f835412eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 --2- 172.21.15.116:0/210887805 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f833c004780 0x7f835409b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 >> 172.21.15.116:0/210887805 conn(0x7f83540672b0 msgr2=0x7f835412fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 shutdown_connections 2024-09-18T12:41:56.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.128+0000 7f835a19d640 1 -- 172.21.15.116:0/210887805 wait complete. 2024-09-18T12:41:56.139 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:41:56.139 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:41:56.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:41:56.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:41:56.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.217+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.217+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717521468}) 2024-09-18T12:41:56.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.217+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027168581}) 2024-09-18T12:41:56.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.217+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.217+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717521468}) 2024-09-18T12:41:56.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.217+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4289587142}) 2024-09-18T12:41:56.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.218+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.218+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177140281}) 2024-09-18T12:41:56.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.218+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=894644684}) 2024-09-18T12:41:56.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.218+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.218+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177140281}) 2024-09-18T12:41:56.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.218+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407444769}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.219+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.219+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597558164}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.219+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370348934}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.219+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.219+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597558164}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.219+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1383260762}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.220+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.220+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689285496}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.220+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=990889693}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.220+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.220+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689285496}) 2024-09-18T12:41:56.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.220+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=452511927}) 2024-09-18T12:41:56.222 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.221+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.221+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648539008}) 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.221+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2080424933}) 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.221+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.221+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648539008}) 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.221+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091915467}) 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.221+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100798487}) 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513609436}) 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100798487}) 2024-09-18T12:41:56.223 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552245169}) 2024-09-18T12:41:56.224 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.224 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175429509}) 2024-09-18T12:41:56.224 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106161740}) 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175429509}) 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.222+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885387226}) 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.223+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.223+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390174127}) 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.223+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585507907}) 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.223+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.223+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390174127}) 2024-09-18T12:41:56.225 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.223+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212870551}) 2024-09-18T12:41:56.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.225+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.225+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=38,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=3021244353,mdsmap=1531151339,mgr=2854130551,mgr_metadata=3239289319,mgrstat=3407756119,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1072996941}) 2024-09-18T12:41:56.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.225+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=25} crc {logm=1046060244,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=957494953,mon_config_key=3655569570,monmap=148614107,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=3207626191}) 2024-09-18T12:41:56.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.225+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.225+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=38,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=3021244353,mdsmap=1531151339,mgr=2854130551,mgr_metadata=3239289319,mgrstat=3407756119,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1072996941}) 2024-09-18T12:41:56.226 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.225+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=39,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=24} crc {logm=1941082474,mdsmap=3585761405,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=957494953,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=4183220990}) 2024-09-18T12:41:56.231 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.230+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.231 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.230+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=300438542}) 2024-09-18T12:41:56.231 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.230+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=875757298}) 2024-09-18T12:41:56.231 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.230+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:41:56.231 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.230+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=300438542}) 2024-09-18T12:41:56.231 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:41:56.230+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=875757298}) 2024-09-18T12:41:56.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.275+0000 7f63b0457640 10 monclient: get_monmap_and_config 2024-09-18T12:41:56.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.275+0000 7f63b0457640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.275+0000 7f63b0457640 10 monclient: monmap: 2024-09-18T12:41:56.276 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.276 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:56.275692+0000 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:56.275692+0000 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.275+0000 7f63b0457640 1 Processor -- start 2024-09-18T12:41:56.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.275+0000 7f63b0457640 1 -- start start 2024-09-18T12:41:56.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.275+0000 7f63b0457640 10 monclient: init 2024-09-18T12:41:56.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T12:41:56.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63a809ec60 0x7f63a809f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient(hunting): picked mon.noname-a con 0x7f63a809ec60 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:56.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63a809d1a0 0x7f63a8068d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient(hunting): picked mon.noname-c con 0x7f63a809d1a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:56.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63a809d5a0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a80a14b0 con 0x7f63a809d5a0 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a809c820 con 0x7f63a809ec60 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a809f570 con 0x7f63a809d1a0 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63b0457640 10 monclient(hunting): authenticate will time out at 1743.597276s 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae1f3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63a809ec60 0x7f63a809f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae1f3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63a809ec60 0x7f63a809f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54554/0 (socket says 172.21.15.116:54554) 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae9f4640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f63a809d5a0 0x7f63a80692d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39906/0 (socket says 172.21.15.116:39906) 2024-09-18T12:41:56.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae1f3640 1 -- 172.21.15.116:0/1298892102 learned_addr learned my addr 172.21.15.116:0/1298892102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:56.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae1f3640 10 monclient(hunting): get_auth_request con 0x7f63a809ec60 auth_method 0 2024-09-18T12:41:56.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae1f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ae1f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 1 --2- 172.21.15.116:0/1298892102 >> v2:172.21.15.144:3300/0 conn(0x7f63a809d1a0 0x7f63a8068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): get_auth_request con 0x7f63a809d1a0 auth_method 0 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_done global_id 174412 payload 274 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 1 -- 172.21.15.116:0/1298892102 >> v2:172.21.15.116:3300/0 conn(0x7f63a809ec60 msgr2=0x7f63a809f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 1 --2- 172.21.15.116:0/1298892102 >> v2:172.21.15.116:3300/0 conn(0x7f63a809ec60 0x7f63a809f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 1 -- 172.21.15.116:0/1298892102 >> v1:172.21.15.116:6789/0 conn(0x7f63a809d5a0 legacy=0x7f63a80692d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 1 monclient: found mon.noname-c 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.276+0000 7f63ad9f2640 1 -- 172.21.15.116:0/1298892102 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a809ffe0 con 0x7f63a809d1a0 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.279+0000 7f63ad9f2640 1 -- 172.21.15.116:0/1298892102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809d1a0 msgr2=0x7f63a8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.279+0000 7f63ad9f2640 1 --2- 172.21.15.116:0/1298892102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809d1a0 0x7f63a8068d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6398009850 tx=0x7f63980090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=960c7cab436a3188 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 1 -- 172.21.15.116:0/1298892102 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639802adf0 con 0x7f63a809d1a0 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: dump: 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.292 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 1 -- 172.21.15.116:0/1298892102 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6398032440 con 0x7f63a809d1a0 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 5 monclient: authenticate success, global_id 174412 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 1 -- 172.21.15.116:0/1298892102 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63980327e0 con 0x7f63a809d1a0 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 4 set_mon_vals no callback set 2024-09-18T12:41:56.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63ad1f1640 10 monclient: dump: 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 10 monclient: shutdown 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 -- 172.21.15.116:0/1298892102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809d1a0 msgr2=0x7f63a8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 --2- 172.21.15.116:0/1298892102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809d1a0 0x7f63a8068d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6398009850 tx=0x7f63980090f0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 -- 172.21.15.116:0/1298892102 shutdown_connections 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 --2- 172.21.15.116:0/1298892102 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809d1a0 0x7f63a8068d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 --2- 172.21.15.116:0/1298892102 >> v2:172.21.15.116:3300/0 conn(0x7f63a809ec60 0x7f63a809f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 -- 172.21.15.116:0/1298892102 >> 172.21.15.116:0/1298892102 conn(0x7f63a8067110 msgr2=0x7f63a8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 -- 172.21.15.116:0/1298892102 shutdown_connections 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.280+0000 7f63b0457640 1 -- 172.21.15.116:0/1298892102 wait complete. 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 10 monclient: monmap: 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:56.295 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 1 Processor -- start 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 1 -- start start 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 10 monclient: init 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63a809d1a0 0x7f63a8138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 10 monclient(hunting): picked mon.noname-c con 0x7f63a809d1a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809d5a0 0x7f63a8099b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.281+0000 7f63b0457640 10 monclient(hunting): picked mon.noname-a con 0x7f63a809d5a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809ec60 0x7f63a809a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 10 monclient(hunting): picked mon.noname-b con 0x7f63a809ec60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8139200 con 0x7f63a809d5a0 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a81240c0 con 0x7f63a809ec60 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a806ed10 con 0x7f63a809d1a0 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63b0457640 10 monclient(hunting): authenticate will time out at 1743.603036s 2024-09-18T12:41:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809d5a0 0x7f63a8099b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809d5a0 0x7f63a8099b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54564/0 (socket says 172.21.15.116:54564) 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 -- 172.21.15.116:0/4212241927 learned_addr learned my addr 172.21.15.116:0/4212241927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): get_auth_request con 0x7f63a809d5a0 auth_method 0 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63a809d1a0 0x7f63a8138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 10 monclient(hunting): get_auth_request con 0x7f63a809d1a0 auth_method 0 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae9f4640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809ec60 0x7f63a809a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient(hunting): handle_auth_done global_id 194235 payload 824 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63a809d1a0 msgr2=0x7f63a8138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f63a809d1a0 0x7f63a8138310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809ec60 msgr2=0x7f63a809a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809ec60 0x7f63a809a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 monclient: found mon.noname-a 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ad9f2640 1 -- 172.21.15.116:0/4212241927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6398009390 con 0x7f63a809d5a0 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae9f4640 10 monclient: get_auth_request con 0x7f63a809ec60 auth_method 0 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.282+0000 7f63ae9f4640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63a809ec60 0x7f63a809a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63ad9f2640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809d5a0 msgr2=0x7f63a8099b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63ad9f2640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809d5a0 0x7f63a8099b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6398009850 tx=0x7f6398032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398031b10 con 0x7f63a809d5a0 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: dump: 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63b0457640 5 monclient: authenticate success, global_id 194235 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6398031c70 con 0x7f63a809d5a0 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63b0457640 10 monclient: _renew_subs 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63b0457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a8125430 con 0x7f63a809d5a0 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639803d440 con 0x7f63a809d5a0 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f639e7fc640 10 monclient: dump: 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63ac9f0640 4 set_mon_vals no callback set 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63b0457640 10 monclient: _renew_subs 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63b0457640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.284+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a809b640 con 0x7f63a809d5a0 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63b0457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6398039040 con 0x7f63a809d5a0 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63b0457640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 0x7f63a809a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63b0457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a81257d0 con 0x7f63a8138850 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 0x7f63a809a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 10 monclient: get_auth_request con 0x7f63a8138850 auth_method 0 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f639e7fc640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f638c038490 0x7f638c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.285+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f639806c200 con 0x7f63a809d5a0 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.286+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_done global_id 194238 payload 824 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.286+0000 7f63ae1f3640 1 -- 172.21.15.116:0/4212241927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63a804bf60 con 0x7f63a8138850 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.286+0000 7f63ae9f4640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f638c038490 0x7f638c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.286+0000 7f63ae9f4640 10 monclient: get_auth_request con 0x7f638c038490 auth_method 0 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f63ae1f3640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 msgr2=0x7f63a809a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f63ae1f3640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 0x7f63a809a5d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f63a4010bb0 tx=0x7f63a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a400edd0 con 0x7f63a8138850 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 10 monclient: dump: 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f63a401d450 con 0x7f63a8138850 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 msgr2=0x7f63a809a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 0x7f63a809a5d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f63a4010bb0 tx=0x7f63a4008040 comp rx=0 tx=0).stop 2024-09-18T12:41:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 msgr2=0x7f63a809a5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f639e7fc640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a8138850 0x7f63a809a5d0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f63a4010bb0 tx=0x7f63a4008040 comp rx=0 tx=0).stop 2024-09-18T12:41:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.288+0000 7f63ae9f4640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f638c038490 0x7f638c03a940 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f63a809ba70 tx=0x7f63a0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63b0457640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:56.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63b0457640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 0x7f63a8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63b0457640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8065e00 con 0x7f63a809ab10 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 0x7f63a8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient: get_auth_request con 0x7f63a809ab10 auth_method 0 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 10 monclient(hunting): handle_auth_done global_id 194241 payload 824 2024-09-18T12:41:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.294+0000 7f63ae1f3640 1 -- 172.21.15.116:0/4212241927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f63a809af10 con 0x7f63a809ab10 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.296+0000 7f63ae1f3640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 msgr2=0x7f63a8096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.296+0000 7f63ae1f3640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 0x7f63a8096a60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f63a400b820 tx=0x7f63a401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.296+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4022070 con 0x7f63a809ab10 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.296+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.296+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.296+0000 7f639e7fc640 10 monclient: dump: 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f63a4014cd0 con 0x7f63a809ab10 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f639e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f639e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 msgr2=0x7f63a8096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f639e7fc640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 0x7f63a8096a60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f63a400b820 tx=0x7f63a401d000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f639e7fc640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 msgr2=0x7f63a8096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f639e7fc640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809ab10 0x7f63a8096a60 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f63a400b820 tx=0x7f63a401d000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:56.310 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 162, 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 8, 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:56.311 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.312 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.319 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:56.320 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.321 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:56.322 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:56.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f638c038490 msgr2=0x7f638c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f638c038490 0x7f638c03a940 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f63a809ba70 tx=0x7f63a0009040 comp rx=0 tx=0).stop 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 10 monclient: shutdown 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809d5a0 msgr2=0x7f63a8099b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 --2- 172.21.15.116:0/4212241927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a809d5a0 0x7f63a8099b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6398009850 tx=0x7f6398032000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63ae9f4640 1 -- 172.21.15.116:0/4212241927 reap_dead start 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63ae9f4640 1 -- 172.21.15.116:0/4212241927 reap_dead start 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 shutdown_connections 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 >> 172.21.15.116:0/4212241927 conn(0x7f63a8067110 msgr2=0x7f63a8139410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 shutdown_connections 2024-09-18T12:41:56.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.297+0000 7f63b0457640 1 -- 172.21.15.116:0/4212241927 wait complete. 2024-09-18T12:41:56.325 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:56.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:41:56.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:41:56.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.456+0000 7f9c8ae6f640 10 monclient: get_monmap_and_config 2024-09-18T12:41:56.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.457+0000 7f9c8ae6f640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.457+0000 7f9c8ae6f640 10 monclient: monmap: 2024-09-18T12:41:56.458 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.458 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.458 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:56.457678+0000 2024-09-18T12:41:56.458 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:56.457678+0000 2024-09-18T12:41:56.458 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.459 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.459 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:56.459 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:56.459 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:56.459 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:56.459 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:56.459 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.457+0000 7f9c8ae6f640 1 Processor -- start 2024-09-18T12:41:56.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.457+0000 7f9c8ae6f640 1 -- start start 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.457+0000 7f9c8ae6f640 10 monclient: init 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c84131ab0 0x7f9c84131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient(hunting): picked mon.noname-a con 0x7f9c84131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c8409e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c84138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient(hunting): picked mon.noname-b con 0x7f9c84136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8409faf0 con 0x7f9c8409e910 2024-09-18T12:41:56.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c8409fd30 con 0x7f9c84131ab0 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840a1110 con 0x7f9c84136bb0 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8ae6f640 10 monclient(hunting): authenticate will time out at 1743.779325s 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c84131ab0 0x7f9c84131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c84138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c84131ab0 0x7f9c84131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54582/0 (socket says 172.21.15.116:54582) 2024-09-18T12:41:56.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c84138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44470/0 (socket says 172.21.15.116:44470) 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 1 -- 172.21.15.116:0/1800342663 learned_addr learned my addr 172.21.15.116:0/1800342663 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 10 monclient(hunting): get_auth_request con 0x7f9c84131ab0 auth_method 0 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): get_auth_request con 0x7f9c84136bb0 auth_method 0 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c88c0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient(hunting): handle_auth_done global_id 204122 payload 274 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 1 -- 172.21.15.116:0/1800342663 >> v2:172.21.15.116:3300/0 conn(0x7f9c84131ab0 msgr2=0x7f9c84131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 1 --2- 172.21.15.116:0/1800342663 >> v2:172.21.15.116:3300/0 conn(0x7f9c84131ab0 0x7f9c84131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 1 -- 172.21.15.116:0/1800342663 >> v1:172.21.15.116:6789/0 conn(0x7f9c8409e910 legacy=0x7f9c8409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 1 monclient: found mon.noname-b 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:41:56.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.458+0000 7f9c8940c640 1 -- 172.21.15.116:0/1800342663 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c840a13f0 con 0x7f9c84136bb0 2024-09-18T12:41:56.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c8940c640 1 -- 172.21.15.116:0/1800342663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 msgr2=0x7f9c84138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c8940c640 1 --2- 172.21.15.116:0/1800342663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c84138f90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009e60 tx=0x7f9c7c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e6231bbe7d7f1f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 1 -- 172.21.15.116:0/1800342663 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c007760 con 0x7f9c84136bb0 2024-09-18T12:41:56.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: dump: 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.474 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 1 -- 172.21.15.116:0/1800342663 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c02d050 con 0x7f9c84136bb0 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c8ae6f640 5 monclient: authenticate success, global_id 204122 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c8ae6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c8ae6f640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 1 -- 172.21.15.116:0/1800342663 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c007af0 con 0x7f9c84136bb0 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c8ae6f640 4 set_mon_vals no callback set 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.461+0000 7f9c837fe640 10 monclient: dump: 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.476 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 10 monclient: shutdown 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/1800342663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 msgr2=0x7f9c84138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/1800342663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c84138f90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009e60 tx=0x7f9c7c009780 comp rx=0 tx=0).stop 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/1800342663 shutdown_connections 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/1800342663 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c84138f90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/1800342663 >> v2:172.21.15.116:3300/0 conn(0x7f9c84131ab0 0x7f9c84131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/1800342663 >> 172.21.15.116:0/1800342663 conn(0x7f9c84067d30 msgr2=0x7f9c840a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/1800342663 shutdown_connections 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/1800342663 wait complete. 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.462+0000 7f9c8ae6f640 10 monclient: monmap: 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 Processor -- start 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 -- start start 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient: init 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c8409e910 0x7f9c8412ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): picked mon.noname-b con 0x7f9c8409e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c84131ab0 0x7f9c8412f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): picked mon.noname-a con 0x7f9c84131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c8412f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): picked mon.noname-c con 0x7f9c84136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8409db00 con 0x7f9c84131ab0 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84133bb0 con 0x7f9c8409e910 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8409bde0 con 0x7f9c84136bb0 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.463+0000 7f9c8ae6f640 10 monclient(hunting): authenticate will time out at 1743.784950s 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c84131ab0 0x7f9c8412f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c8412f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c8412f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44486/0 (socket says 172.21.15.116:44486) 2024-09-18T12:41:56.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c84131ab0 0x7f9c8412f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54598/0 (socket says 172.21.15.116:54598) 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 -- 172.21.15.116:0/3063102992 learned_addr learned my addr 172.21.15.116:0/3063102992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): get_auth_request con 0x7f9c84136bb0 auth_method 0 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 10 monclient(hunting): get_auth_request con 0x7f9c84131ab0 auth_method 0 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c88c0b640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c8409e910 0x7f9c8412ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c88c0b640 10 monclient(hunting): get_auth_request con 0x7f9c8409e910 auth_method 0 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c88c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c88c0b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c88c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient(hunting): handle_auth_done global_id 204125 payload 824 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c8409e910 msgr2=0x7f9c8412ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c8409e910 0x7f9c8412ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c84131ab0 msgr2=0x7f9c8412f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c84131ab0 0x7f9c8412f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 monclient: found mon.noname-c 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c8940c640 1 -- 172.21.15.116:0/3063102992 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c009a20 con 0x7f9c84136bb0 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.464+0000 7f9c83fff640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c84131ab0 0x7f9c8412f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8940c640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 msgr2=0x7f9c8412f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8940c640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c8412f990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0099f0 tx=0x7f9c7c037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c031ac0 con 0x7f9c84136bb0 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: dump: 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.483 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c040070 con 0x7f9c84136bb0 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8ae6f640 5 monclient: authenticate success, global_id 204125 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8ae6f640 10 monclient: _renew_subs 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8ae6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:56.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c84130970 con 0x7f9c84136bb0 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c031e90 con 0x7f9c84136bb0 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c80ff9640 10 monclient: dump: 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.485 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8ae6f640 10 monclient: _renew_subs 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8ae6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c84131540 con 0x7f9c84136bb0 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.466+0000 7f9c82ffd640 4 set_mon_vals no callback set 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9c7c042030 con 0x7f9c84136bb0 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c80ff9640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c70038490 0x7f9c7003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c8ae6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9c7c06db60 con 0x7f9c84136bb0 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 0x7f9c8409c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c8ae6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c84063740 con 0x7f9c8412fed0 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 0x7f9c8409c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 10 monclient: get_auth_request con 0x7f9c8412fed0 auth_method 0 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c88c0b640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c70038490 0x7f9c7003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.467+0000 7f9c88c0b640 10 monclient: get_auth_request con 0x7f9c70038490 auth_method 0 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.468+0000 7f9c83fff640 10 monclient(hunting): handle_auth_done global_id 204128 payload 824 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.468+0000 7f9c83fff640 1 -- 172.21.15.116:0/3063102992 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c84133420 con 0x7f9c8412fed0 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c83fff640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 msgr2=0x7f9c8409c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c83fff640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 0x7f9c8409c8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800eb00 tx=0x7f9c78006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78002d80 con 0x7f9c8412fed0 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 10 monclient: dump: 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.488 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9c7801b450 con 0x7f9c8412fed0 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 msgr2=0x7f9c8409c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 0x7f9c8409c8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800eb00 tx=0x7f9c78006f50 comp rx=0 tx=0).stop 2024-09-18T12:41:56.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 msgr2=0x7f9c8409c8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:56.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c80ff9640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8412fed0 0x7f9c8409c8d0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800eb00 tx=0x7f9c78006f50 comp rx=0 tx=0).stop 2024-09-18T12:41:56.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.470+0000 7f9c88c0b640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c70038490 0x7f9c7003a940 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9c74006cb0 tx=0x7f9c74008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c8ae6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 0x7f9c8409d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c8ae6f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8404bf60 con 0x7f9c8409ce10 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 0x7f9c8409d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient: get_auth_request con 0x7f9c8409ce10 auth_method 0 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 10 monclient(hunting): handle_auth_done global_id 204131 payload 824 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.476+0000 7f9c83fff640 1 -- 172.21.15.116:0/3063102992 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9c841302a0 con 0x7f9c8409ce10 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.478+0000 7f9c83fff640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 msgr2=0x7f9c8409d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.478+0000 7f9c83fff640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 0x7f9c8409d210 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800a010 tx=0x7f9c78019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78012e20 con 0x7f9c8409ce10 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 10 monclient: dump: 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.493 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f9c780253a0 con 0x7f9c8409ce10 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 msgr2=0x7f9c8409d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 0x7f9c8409d210 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800a010 tx=0x7f9c78019000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 msgr2=0x7f9c8409d210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c80ff9640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c8409ce10 0x7f9c8409d210 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800a010 tx=0x7f9c78019000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 162, 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:56.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 9, 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.496 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.497 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:56.498 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.499 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.500 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.501 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:56.504 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.506 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:56.507 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:56.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:56.509 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c70038490 msgr2=0x7f9c7003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9c70038490 0x7f9c7003a940 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9c74006cb0 tx=0x7f9c74008040 comp rx=0 tx=0).stop 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 10 monclient: shutdown 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 msgr2=0x7f9c8412f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c8412f990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0099f0 tx=0x7f9c7c037000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8940c640 1 -- 172.21.15.116:0/3063102992 reap_dead start 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 shutdown_connections 2024-09-18T12:41:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 --2- 172.21.15.116:0/3063102992 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9c84136bb0 0x7f9c8412f990 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 >> 172.21.15.116:0/3063102992 conn(0x7f9c84067d30 msgr2=0x7f9c8412cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 shutdown_connections 2024-09-18T12:41:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.479+0000 7f9c8ae6f640 1 -- 172.21.15.116:0/3063102992 wait complete. 2024-09-18T12:41:56.512 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:56.512 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:41:56.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:41:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.605+0000 7f6d1051d640 10 monclient: get_monmap_and_config 2024-09-18T12:41:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.605+0000 7f6d1051d640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.605+0000 7f6d1051d640 10 monclient: monmap: 2024-09-18T12:41:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:56.606411+0000 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:56.606411+0000 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:56.607 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 Processor -- start 2024-09-18T12:41:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 -- start start 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient: init 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d08131ab0 0x7f6d08131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d08131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6d0809e910 0x7f6d0809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d0809e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d0809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0809faf0 con 0x7f6d08131ab0 2024-09-18T12:41:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0809fd30 con 0x7f6d0809e910 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080a1110 con 0x7f6d0809f250 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.606+0000 7f6d1051d640 10 monclient(hunting): authenticate will time out at 1743.927941s 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d08131ab0 0x7f6d08131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d08131ab0 0x7f6d08131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54608/0 (socket says 172.21.15.116:54608) 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44518/0 (socket says 172.21.15.116:44518) 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2358026994 learned_addr learned my addr 172.21.15.116:0/2358026994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): get_auth_request con 0x7f6d08131ab0 auth_method 0 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 10 monclient(hunting): get_auth_request con 0x7f6d0809f250 auth_method 0 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 msgr2=0x7f6d08068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 msgr2=0x7f6d08068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 1 --2- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0eaba640 1 --2- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0dab8640 1 --2- 172.21.15.116:0/2358026994 >> v2:172.21.15.144:3300/0 conn(0x7f6d0809e910 0x7f6d0809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_done global_id 194250 payload 274 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 msgr2=0x7f6d08068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2358026994 >> v2:172.21.15.144:3300/0 conn(0x7f6d0809e910 msgr2=0x7f6d0809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2358026994 >> v2:172.21.15.144:3300/0 conn(0x7f6d0809e910 0x7f6d0809ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 monclient: found mon.noname-a 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2358026994 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080a13f0 con 0x7f6d08131ab0 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0dab8640 10 monclient: get_auth_request con 0x7f6d0809e910 auth_method 0 2024-09-18T12:41:56.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.607+0000 7f6d0dab8640 1 --2- 172.21.15.116:0/2358026994 >> v2:172.21.15.144:3300/0 conn(0x7f6d0809e910 0x7f6d0809ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 msgr2=0x7f6d08131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 0x7f6d08131e80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d0809bdb0 tx=0x7f6d040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fbf317f23ce989 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 1 -- 172.21.15.116:0/2358026994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04032440 con 0x7f6d08131ab0 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: dump: 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.621 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 1 -- 172.21.15.116:0/2358026994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d040325a0 con 0x7f6d08131ab0 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 5 monclient: authenticate success, global_id 194250 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 1 -- 172.21.15.116:0/2358026994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04032940 con 0x7f6d08131ab0 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 4 set_mon_vals no callback set 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d0d2b7640 10 monclient: dump: 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.623 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 10 monclient: shutdown 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 -- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 msgr2=0x7f6d08131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 0x7f6d08131e80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d0809bdb0 tx=0x7f6d040091d0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 -- 172.21.15.116:0/2358026994 shutdown_connections 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2358026994 >> v2:172.21.15.144:3300/0 conn(0x7f6d0809e910 0x7f6d0809ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2358026994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 0x7f6d08131e80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 -- 172.21.15.116:0/2358026994 >> 172.21.15.116:0/2358026994 conn(0x7f6d08067d30 msgr2=0x7f6d080a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 -- 172.21.15.116:0/2358026994 shutdown_connections 2024-09-18T12:41:56.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.610+0000 7f6d1051d640 1 -- 172.21.15.116:0/2358026994 wait complete. 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.611+0000 7f6d1051d640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.611+0000 7f6d1051d640 10 monclient: monmap: 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.611+0000 7f6d1051d640 1 Processor -- start 2024-09-18T12:41:56.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.611+0000 7f6d1051d640 1 -- start start 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.611+0000 7f6d1051d640 10 monclient: init 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d0809e910 0x7f6d0809aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d0809e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d0809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 0x7f6d08140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d08131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080a43a0 con 0x7f6d08131ab0 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0811cd80 con 0x7f6d0809e910 2024-09-18T12:41:56.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0811dce0 con 0x7f6d0809f250 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d1051d640 10 monclient(hunting): authenticate will time out at 1743.933227s 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 0x7f6d08140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:44532/0 (socket says 172.21.15.116:44532) 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 1 -- 172.21.15.116:0/2650632934 learned_addr learned my addr 172.21.15.116:0/2650632934 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 10 monclient(hunting): get_auth_request con 0x7f6d0809f250 auth_method 0 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): get_auth_request con 0x7f6d08131ab0 auth_method 0 2024-09-18T12:41:56.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d0809e910 0x7f6d0809aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 10 monclient(hunting): get_auth_request con 0x7f6d0809e910 auth_method 0 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient(hunting): handle_auth_done global_id 194253 payload 824 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 msgr2=0x7f6d08130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d0809e910 msgr2=0x7f6d0809aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d0809e910 0x7f6d0809aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 1 monclient: found mon.noname-a 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2650632934 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d04009450 con 0x7f6d08131ab0 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.612+0000 7f6d0dab8640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6d0809f250 0x7f6d08130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 msgr2=0x7f6d08140d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6d0eaba640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 0x7f6d08140d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6d00007e40 tx=0x7f6d000097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00014420 con 0x7f6d08131ab0 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6cfe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6cfe7fc640 10 monclient: dump: 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.630 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6cfe7fc640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6cfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6d1051d640 5 monclient: authenticate success, global_id 194253 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d00014580 con 0x7f6d08131ab0 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6d1051d640 10 monclient: _renew_subs 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6d1051d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.614+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0811dfc0 con 0x7f6d08131ab0 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6cfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00014940 con 0x7f6d08131ab0 2024-09-18T12:41:56.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6cfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6cfe7fc640 10 monclient: dump: 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.632 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d1051d640 10 monclient: _renew_subs 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d0cab6640 4 set_mon_vals no callback set 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d1051d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0811bb80 con 0x7f6d08131ab0 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d1051d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 0x7f6d08141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08063740 con 0x7f6d08131100 2024-09-18T12:41:56.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.615+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6d00003020 con 0x7f6d08131ab0 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6cfe7fc640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6cec038490 0x7f6cec03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 0x7f6d08141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6d0004c810 con 0x7f6d08131ab0 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient: get_auth_request con 0x7f6d08131100 auth_method 0 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0dab8640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6cec038490 0x7f6cec03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0dab8640 10 monclient: get_auth_request con 0x7f6cec038490 auth_method 0 2024-09-18T12:41:56.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_done global_id 174421 payload 824 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.616+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2650632934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d0809be50 con 0x7f6d08131100 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.618+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 msgr2=0x7f6d08141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.618+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 0x7f6d08141290 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6d04034000 tx=0x7f6d04032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.618+0000 7f6d0dab8640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6cec038490 0x7f6cec03a940 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6d08068990 tx=0x7f6cf8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0403ca10 con 0x7f6d08131100 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 10 monclient: dump: 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.635 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6d0403cb70 con 0x7f6d08131100 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 msgr2=0x7f6d08141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 0x7f6d08141290 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6d04034000 tx=0x7f6d04032000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 msgr2=0x7f6d08141290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:56.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.619+0000 7f6cfe7fc640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d08131100 0x7f6d08141290 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.624+0000 7f6d1051d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:41:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.624+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 0x7f6d08096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.624+0000 7f6d1051d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.624+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0811e150 con 0x7f6d081417d0 2024-09-18T12:41:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 0x7f6d08096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient: get_auth_request con 0x7f6d081417d0 auth_method 0 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 10 monclient(hunting): handle_auth_done global_id 174424 payload 824 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.625+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2650632934 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6d08141bd0 con 0x7f6d081417d0 2024-09-18T12:41:56.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.627+0000 7f6d0e2b9640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 msgr2=0x7f6d08096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.627+0000 7f6d0e2b9640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 0x7f6d08096a60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d0811df00 tx=0x7f6d0403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04055070 con 0x7f6d081417d0 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 10 monclient: dump: 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.640 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f6d0403c800 con 0x7f6d081417d0 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 msgr2=0x7f6d08096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 0x7f6d08096a60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d0811df00 tx=0x7f6d0403d000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 msgr2=0x7f6d08096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6cfe7fc640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d081417d0 0x7f6d08096a60 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d0811df00 tx=0x7f6d0403d000 comp rx=0 tx=0).stop 2024-09-18T12:41:56.641 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 162, 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 9, 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:41:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:41:56.643 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.644 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:41:56.645 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:41:56.646 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:41:56.647 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.649 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:41:56.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.652 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.653 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:41:56.654 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6cec038490 msgr2=0x7f6cec03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6cec038490 0x7f6cec03a940 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6d08068990 tx=0x7f6cf8008040 comp rx=0 tx=0).stop 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 10 monclient: shutdown 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 msgr2=0x7f6d08140d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 1 --2- 172.21.15.116:0/2650632934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d08131ab0 0x7f6d08140d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6d00007e40 tx=0x7f6d000097e0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2650632934 reap_dead start 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d0eaba640 1 -- 172.21.15.116:0/2650632934 reap_dead start 2024-09-18T12:41:56.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 shutdown_connections 2024-09-18T12:41:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.628+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 >> 172.21.15.116:0/2650632934 conn(0x7f6d08067d30 msgr2=0x7f6d081347f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.629+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 shutdown_connections 2024-09-18T12:41:56.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.629+0000 7f6d1051d640 1 -- 172.21.15.116:0/2650632934 wait complete. 2024-09-18T12:41:56.657 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:41:56.657 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:41:56.657 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:41:56.657 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:41:56.657 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:41:56.658 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:41:56.658 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:41:56.668 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:41:56.668 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:41:56.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:41:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.799+0000 7fa3b9154640 10 monclient: get_monmap_and_config 2024-09-18T12:41:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.799+0000 7fa3b9154640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.799+0000 7fa3b9154640 10 monclient: monmap: 2024-09-18T12:41:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:41:56.800458+0000 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:41:56.800458+0000 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:41:56.801 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 1 Processor -- start 2024-09-18T12:41:56.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 1 -- start start 2024-09-18T12:41:56.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient: init 2024-09-18T12:41:56.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3b40974f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 0x7fa3b409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient(hunting): picked mon.noname-b con 0x7fa3b409a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3b4096c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b412b210 con 0x7fa3b40974f0 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b40978c0 con 0x7fa3b4096c60 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b409ca20 con 0x7fa3b409a8a0 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b9154640 10 monclient(hunting): authenticate will time out at 1744.121864s 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b259c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 msgr2=0x7fa3b409acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:56.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 0x7fa3b409acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b2d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa3b40974f0 0x7fa3b412ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39926/0 (socket says 172.21.15.116:39926) 2024-09-18T12:41:56.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.800+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/1997874067 learned_addr learned my addr 172.21.15.116:0/1997874067 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:41:56.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/1997874067 >> v1:172.21.15.116:6789/0 conn(0x7fa3b40974f0 legacy=0x7fa3b412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b359e640 1 -- 172.21.15.116:0/1997874067 >> v1:172.21.15.144:6789/0 conn(0x7fa3b4096c60 legacy=0x7fa3b4136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707163209 0 0) 0x7fa3b412b210 con 0x7fa3b40974f0 2024-09-18T12:41:56.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): my global_id is 194256 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa39c0034e0 con 0x7fa3b40974f0 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149644777 0 0) 0x7fa3b40978c0 con 0x7fa3b4096c60 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): my global_id is 174427 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b412b210 con 0x7fa3b4096c60 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3218119584 0 0) 0x7fa39c0034e0 con 0x7fa3b40974f0 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 msgr2=0x7fa3b409acb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 0x7fa3b409acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 >> v1:172.21.15.144:6789/0 conn(0x7fa3b4096c60 legacy=0x7fa3b4136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b409b1f0 con 0x7fa3b40974f0 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fa3a40017a0 con 0x7fa3b40974f0 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b9154640 5 monclient: authenticate success, global_id 194256 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b9154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: dump: 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.809 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 >> v1:172.21.15.116:6789/0 conn(0x7fa3b40974f0 legacy=0x7fa3b412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:41:56.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 0x7fa39c003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): picked mon.c con 0x7fa3b409a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa39c004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): picked mon.a con 0x7fa39c004200 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa39c004ec0 0x7fa39c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): picked mon.b con 0x7fa39c004ec0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b409ca20 con 0x7fa39c004200 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c00bf10 con 0x7fa39c004ec0 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c00c050 con 0x7fa3b409a8a0 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 msgr2=0x7fa39c003c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b1d9b640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b9154640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b2d9d640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 0x7fa39c003c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa39c004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 10 monclient(hunting): get_auth_request con 0x7fa39c004200 auth_method 0 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.803+0000 7fa3b359e640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa39c004ec0 0x7fa39c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b359e640 10 monclient(hunting): get_auth_request con 0x7fa39c004ec0 auth_method 0 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b359e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 10 monclient(hunting): handle_auth_done global_id 194256 payload 293 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 1 -- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 msgr2=0x7fa39c003c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b409a8a0 0x7fa39c003c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 1 -- 172.21.15.116:0/1997874067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa39c004ec0 msgr2=0x7fa39c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa39c004ec0 0x7fa39c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 1 monclient: found mon.a 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 1 -- 172.21.15.116:0/1997874067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0034e0 con 0x7fa39c004200 2024-09-18T12:41:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.804+0000 7fa3b359e640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa39c004ec0 0x7fa39c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:41:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.806+0000 7fa3b259c640 1 -- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 msgr2=0x7fa39c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.806+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:41:56.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.806+0000 7fa3b259c640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa39c004780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80099f0 tx=0x7fa3a8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f18d971147f05b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.808+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8003b80 con 0x7fa39c004200 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.808+0000 7fa3b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.808+0000 7fa3b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.808+0000 7fa3b1d9b640 10 monclient: dump: 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.820 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a8003d20 con 0x7fa39c004200 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b1d9b640 1 -- 172.21.15.116:0/1997874067 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a802ab80 con 0x7fa39c004200 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b1d9b640 10 monclient: dump: 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.821 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 10 monclient: get_monmap_and_config success 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 4 set_mon_vals no callback set 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 10 monclient: shutdown 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 1 -- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 msgr2=0x7fa39c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 1 --2- 172.21.15.116:0/1997874067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa39c004780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80099f0 tx=0x7fa3a8009060 comp rx=0 tx=0).stop 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b359e640 1 -- 172.21.15.116:0/1997874067 reap_dead start 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 1 -- 172.21.15.116:0/1997874067 shutdown_connections 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 1 -- 172.21.15.116:0/1997874067 >> 172.21.15.116:0/1997874067 conn(0x7fa3b40672b0 msgr2=0x7fa3b40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 1 -- 172.21.15.116:0/1997874067 shutdown_connections 2024-09-18T12:41:56.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.809+0000 7fa3b9154640 1 -- 172.21.15.116:0/1997874067 wait complete. 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.810+0000 7fa3b9154640 10 monclient: build_initial_monmap 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.810+0000 7fa3b9154640 10 monclient: monmap: 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.810+0000 7fa3b9154640 1 Processor -- start 2024-09-18T12:41:56.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.810+0000 7fa3b9154640 1 -- start start 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.810+0000 7fa3b9154640 10 monclient: init 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient: _reopen_session rank -1 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa3b4141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): picked mon.noname-a con 0x7fa39c004200 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b4139f60 0x7fa3b413a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): picked mon.noname-b con 0x7fa3b4139f60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b412bb90 0x7fa3b412c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): picked mon.noname-c con 0x7fa3b412bb90 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b412f100 con 0x7fa39c004200 2024-09-18T12:41:56.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b412dbc0 con 0x7fa3b4139f60 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): start opening mon connection 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b409e3c0 con 0x7fa3b412bb90 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): _renew_subs 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b9154640 10 monclient(hunting): authenticate will time out at 1744.132116s 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b359e640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b412bb90 msgr2=0x7fa3b412c010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b359e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b412bb90 0x7fa3b412c010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa3b4141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa3b4141070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54648/0 (socket says 172.21.15.116:54648) 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/3889950050 learned_addr learned my addr 172.21.15.116:0/3889950050 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): get_auth_request con 0x7fa39c004200 auth_method 0 2024-09-18T12:41:56.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b259c640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b4139f60 0x7fa3b413a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient(hunting): handle_auth_done global_id 194259 payload 824 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b412bb90 msgr2=0x7fa3b412c010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b412bb90 0x7fa3b412c010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b4139f60 msgr2=0x7fa3b413a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b4139f60 0x7fa3b413a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:41:56.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 monclient: found mon.noname-a 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/3889950050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b4098300 con 0x7fa39c004200 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b259c640 10 monclient: get_auth_request con 0x7fa3b4139f60 auth_method 0 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.811+0000 7fa3b259c640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b4139f60 0x7fa3b413a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b2d9d640 1 -- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 msgr2=0x7fa3b4141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b2d9d640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa3b4141070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400c710 tx=0x7fa3b40695f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 1 -- 172.21.15.116:0/3889950050 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4007570 con 0x7fa39c004200 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: dump: 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: _finish_auth 0 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 1 -- 172.21.15.116:0/3889950050 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a40047a0 con 0x7fa39c004200 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b9154640 5 monclient: authenticate success, global_id 194259 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b9154640 10 monclient: _renew_subs 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b9154640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b4141ec0 con 0x7fa39c004200 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 1 -- 172.21.15.116:0/3889950050 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4004b60 con 0x7fa39c004200 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa39affd640 10 monclient: dump: 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:41:56.829 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b9154640 10 monclient: _renew_subs 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b9154640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b159a640 4 set_mon_vals no callback set 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.813+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b412d640 con 0x7fa39c004200 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.814+0000 7fa39affd640 1 -- 172.21.15.116:0/3889950050 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa3a4014420 con 0x7fa39c004200 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.814+0000 7fa3b9154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.814+0000 7fa3b9154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.814+0000 7fa3b9154640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.814+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b404bf60 con 0x7fa39c004200 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.814+0000 7fa39affd640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3a0038440 0x7fa3a003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.814+0000 7fa39affd640 1 -- 172.21.15.116:0/3889950050 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa3a404c240 con 0x7fa39c004200 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.815+0000 7fa3b259c640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3a0038440 0x7fa3a003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:41:56.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.815+0000 7fa3b259c640 10 monclient: get_auth_request con 0x7fa3a0038440 auth_method 0 2024-09-18T12:41:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.817+0000 7fa3b259c640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3a0038440 0x7fa3a003a8f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80099c0 tx=0x7fa3a8030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:41:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.819+0000 7fa39affd640 1 -- 172.21.15.116:0/3889950050 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fa3a400a940 con 0x7fa39c004200 2024-09-18T12:41:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.819+0000 7fa39affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:41:56.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:56.819+0000 7fa39affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:41:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.034+0000 7fa3b9154640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:41:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.034+0000 7fa3b9154640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.034+0000 7fa3b9154640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:41:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.034+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa3b412f340 con 0x7fa39c004200 2024-09-18T12:41:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.035+0000 7fa39affd640 1 -- 172.21.15.116:0/3889950050 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa3a400a760 con 0x7fa39c004200 2024-09-18T12:41:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.035+0000 7fa39affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:41:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.035+0000 7fa39affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:41:57.036 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.036+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3a0038440 msgr2=0x7fa3a003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.036+0000 7fa3b9154640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3a0038440 0x7fa3a003a8f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80099c0 tx=0x7fa3a8030000 comp rx=0 tx=0).stop 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.036+0000 7fa3b9154640 10 monclient: shutdown 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.036+0000 7fa3b9154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.036+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 msgr2=0x7fa3b4141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.036+0000 7fa3b9154640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa3b4141070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400c710 tx=0x7fa3b40695f0 comp rx=0 tx=0).stop 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 shutdown_connections 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3a0038440 0x7fa3a003a8f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa3b412bb90 0x7fa3b412c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa3b4139f60 0x7fa3b413a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 --2- 172.21.15.116:0/3889950050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa39c004200 0x7fa3b4141070 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:41:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 >> 172.21.15.116:0/3889950050 conn(0x7fa3b40672b0 msgr2=0x7fa3b409b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:41:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 shutdown_connections 2024-09-18T12:41:57.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:41:57.037+0000 7fa3b9154640 1 -- 172.21.15.116:0/3889950050 wait complete. 2024-09-18T12:42:00.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.151+0000 7f48bdd9d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.151+0000 7f48bdd9d640 10 monclient: build_initial_monmap 2024-09-18T12:42:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.151+0000 7f48bdd9d640 10 monclient: monmap: 2024-09-18T12:42:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:00.152418+0000 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:00.152418+0000 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:00.153 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.152+0000 7f48bdd9d640 1 Processor -- start 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.152+0000 7f48bdd9d640 1 -- start start 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.152+0000 7f48bdd9d640 10 monclient: init 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b8131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f48b8131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48b809e910 0x7f48b809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f48b809e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48b809f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b809faf0 con 0x7f48b809f220 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b809fd30 con 0x7f48b809e910 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80a1110 con 0x7f48b8131ab0 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48bdd9d640 10 monclient(hunting): authenticate will time out at 1747.474100s 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b77fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 msgr2=0x7f48b8131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48b809e910 0x7f48b809ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48b809e910 0x7f48b809ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54650/0 (socket says 172.21.15.116:54650) 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b6ffd640 1 -- 172.21.15.116:0/2625364206 learned_addr learned my addr 172.21.15.116:0/2625364206 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:00.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b6ffd640 10 monclient(hunting): get_auth_request con 0x7f48b809e910 auth_method 0 2024-09-18T12:42:00.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:00.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:00.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:00.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.153+0000 7f48b7fff640 1 --1- 172.21.15.116:0/2625364206 >> v1:172.21.15.116:6789/0 conn(0x7f48b809f220 0x7f48b8068ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39934/0 (socket says 172.21.15.116:39934) 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b7fff640 1 -- 172.21.15.116:0/2625364206 >> v1:172.21.15.116:6789/0 conn(0x7f48b809f220 legacy=0x7f48b8068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 10 monclient(hunting): handle_auth_done global_id 194262 payload 274 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 1 -- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 msgr2=0x7f48b8131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 1 --2- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b8131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 1 -- 172.21.15.116:0/2625364206 >> v1:172.21.15.116:6789/0 conn(0x7f48b809f220 legacy=0x7f48b8068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 1 monclient: found mon.noname-a 2024-09-18T12:42:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:00.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:00.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.155+0000 7f48b6ffd640 1 -- 172.21.15.116:0/2625364206 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b80a13f0 con 0x7f48b809e910 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b6ffd640 1 -- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 msgr2=0x7f48b809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b6ffd640 1 --2- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b809ece0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009990 tx=0x7f48ac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6926f283539aa44 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 1 -- 172.21.15.116:0/2625364206 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac02f4b0 con 0x7f48b809e910 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: dump: 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: _finish_auth 0 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 1 -- 172.21.15.116:0/2625364206 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac02f610 con 0x7f48b809e910 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48bdd9d640 5 monclient: authenticate success, global_id 194262 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48bdd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48bdd9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 1 -- 172.21.15.116:0/2625364206 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac02f9b0 con 0x7f48b809e910 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48bdd9d640 4 set_mon_vals no callback set 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.158+0000 7f48b67fc640 10 monclient: dump: 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:00.163 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 10 monclient: shutdown 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 -- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 msgr2=0x7f48b809ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b809ece0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48ac009990 tx=0x7f48ac0091d0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 -- 172.21.15.116:0/2625364206 shutdown_connections 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b809ece0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/2625364206 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b8131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 -- 172.21.15.116:0/2625364206 >> 172.21.15.116:0/2625364206 conn(0x7f48b8067d30 msgr2=0x7f48b80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 -- 172.21.15.116:0/2625364206 shutdown_connections 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 1 -- 172.21.15.116:0/2625364206 wait complete. 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 10 monclient: build_initial_monmap 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.159+0000 7f48bdd9d640 10 monclient: monmap: 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:00.164 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 Processor -- start 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 -- start start 2024-09-18T12:42:00.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient: init 2024-09-18T12:42:00.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:00.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b811cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f48b809e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48b809f220 0x7f48b811d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f48b809f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b811d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f48b8131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b811f320 con 0x7f48b809e910 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b809bde0 con 0x7f48b809f220 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80a4510 con 0x7f48b8131ab0 2024-09-18T12:42:00.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48bdd9d640 10 monclient(hunting): authenticate will time out at 1747.481901s 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48b7fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 msgr2=0x7f48b811d810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48b7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b811d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.160+0000 7f48b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b811cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b811cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54656/0 (socket says 172.21.15.116:54656) 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 -- 172.21.15.116:0/3193569361 learned_addr learned my addr 172.21.15.116:0/3193569361 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): get_auth_request con 0x7f48b809e910 auth_method 0 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:00.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient(hunting): handle_auth_done global_id 194268 payload 824 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 -- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 msgr2=0x7f48b811d810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b811d810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 -- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48b809f220 msgr2=0x7f48b811d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48b809f220 0x7f48b811d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 monclient: found mon.noname-a 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.161+0000 7f48b77fe640 1 -- 172.21.15.116:0/3193569361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac009450 con 0x7f48b809e910 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48b77fe640 1 -- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 msgr2=0x7f48b811cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48b77fe640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b811cd90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f48a800b600 tx=0x7f48a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:00.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 1 -- 172.21.15.116:0/3193569361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8015070 con 0x7f48b809e910 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: dump: 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:42:00.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48bdd9d640 5 monclient: authenticate success, global_id 194268 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48bdd9d640 10 monclient: _renew_subs 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 1 -- 172.21.15.116:0/3193569361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a8003d90 con 0x7f48b809e910 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48bdd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b81203e0 con 0x7f48b809e910 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 1 -- 172.21.15.116:0/3193569361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8006380 con 0x7f48b809e910 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f489f7fe640 10 monclient: dump: 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.175 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48bdd9d640 10 monclient: _renew_subs 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48bdd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b8130470 con 0x7f48b809e910 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.163+0000 7f48b5ffb640 4 set_mon_vals no callback set 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.164+0000 7f489f7fe640 1 -- 172.21.15.116:0/3193569361 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f48a8010440 con 0x7f48b809e910 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.164+0000 7f48bdd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:00.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.164+0000 7f48bdd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:00.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.164+0000 7f48bdd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:00.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.164+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b804bf60 con 0x7f48b809e910 2024-09-18T12:42:00.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.164+0000 7f489f7fe640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48a4038440 0x7f48a403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:00.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.164+0000 7f489f7fe640 1 -- 172.21.15.116:0/3193569361 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f48a804be60 con 0x7f48b809e910 2024-09-18T12:42:00.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.165+0000 7f48b6ffd640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48a4038440 0x7f48a403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:00.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.165+0000 7f48b6ffd640 10 monclient: get_auth_request con 0x7f48a4038440 auth_method 0 2024-09-18T12:42:00.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.167+0000 7f48b6ffd640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48a4038440 0x7f48a403a8f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48b809bd80 tx=0x7f48ac032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:00.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.169+0000 7f489f7fe640 1 -- 172.21.15.116:0/3193569361 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f48a8018ab0 con 0x7f48b809e910 2024-09-18T12:42:00.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.169+0000 7f489f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:00.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.169+0000 7f489f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.382+0000 7f48bdd9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.382+0000 7f48bdd9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.382+0000 7f48bdd9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.382+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f48b809a140 con 0x7f48b809e910 2024-09-18T12:42:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.382+0000 7f489f7fe640 1 -- 172.21.15.116:0/3193569361 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f48a8018690 con 0x7f48b809e910 2024-09-18T12:42:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.382+0000 7f489f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.382+0000 7f489f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:00.384 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.384+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48a4038440 msgr2=0x7f48a403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.384+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48a4038440 0x7f48a403a8f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48b809bd80 tx=0x7f48ac032000 comp rx=0 tx=0).stop 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.384+0000 7f48bdd9d640 10 monclient: shutdown 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.384+0000 7f48bdd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.384+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 msgr2=0x7f48b811cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.384+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b811cd90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f48a800b600 tx=0x7f48a80091a0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.384+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 shutdown_connections 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.385+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f48a4038440 0x7f48a403a8f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.385+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f48b8131ab0 0x7f48b811d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.385+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48b809f220 0x7f48b811d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.385+0000 7f48bdd9d640 1 --2- 172.21.15.116:0/3193569361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b809e910 0x7f48b811cd90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.385+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 >> 172.21.15.116:0/3193569361 conn(0x7f48b8067d30 msgr2=0x7f48b81310a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.385+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 shutdown_connections 2024-09-18T12:42:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:00.385+0000 7f48bdd9d640 1 -- 172.21.15.116:0/3193569361 wait complete. 2024-09-18T12:42:03.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:03.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.498+0000 7f6c42060640 10 monclient: get_monmap_and_config 2024-09-18T12:42:03.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.498+0000 7f6c42060640 10 monclient: build_initial_monmap 2024-09-18T12:42:03.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.498+0000 7f6c42060640 10 monclient: monmap: 2024-09-18T12:42:03.524 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:03.524 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:03.524 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:03.498736+0000 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:03.498736+0000 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:03.525 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:03.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.498+0000 7f6c42060640 1 Processor -- start 2024-09-18T12:42:03.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.498+0000 7f6c42060640 1 -- start start 2024-09-18T12:42:03.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.498+0000 7f6c42060640 10 monclient: init 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c3c131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c3c09e910 0x7f6c3c09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c09e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6c3c09f250 0x7f6c3c138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c09f250 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c09faf0 con 0x7f6c3c131ab0 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c132590 con 0x7f6c3c09e910 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c09fd30 con 0x7f6c3c09f250 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:03.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c42060640 10 monclient(hunting): authenticate will time out at 1750.820517s 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c3affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c3c09e910 0x7f6c3c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c3affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c3c09e910 0x7f6c3c09ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54662/0 (socket says 172.21.15.116:54662) 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c3affd640 1 -- 172.21.15.116:0/2085015242 learned_addr learned my addr 172.21.15.116:0/2085015242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c3affd640 10 monclient(hunting): get_auth_request con 0x7f6c3c09e910 auth_method 0 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.499+0000 7f6c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 1 --2- 172.21.15.116:0/2085015242 >> v2:172.21.15.144:3300/0 conn(0x7f6c3c09f250 0x7f6c3c138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 10 monclient(hunting): get_auth_request con 0x7f6c3c09f250 auth_method 0 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:03.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3b7fe640 1 -- 172.21.15.116:0/2085015242 >> v1:172.21.15.116:6789/0 conn(0x7f6c3c131ab0 legacy=0x7f6c3c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.501+0000 7f6c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 10 monclient(hunting): handle_auth_done global_id 194271 payload 274 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 1 -- 172.21.15.116:0/2085015242 >> v2:172.21.15.144:3300/0 conn(0x7f6c3c09f250 msgr2=0x7f6c3c138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 1 --2- 172.21.15.116:0/2085015242 >> v2:172.21.15.144:3300/0 conn(0x7f6c3c09f250 0x7f6c3c138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 1 -- 172.21.15.116:0/2085015242 >> v1:172.21.15.116:6789/0 conn(0x7f6c3c131ab0 legacy=0x7f6c3c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 10 monclient: _finish_hunting 0 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 1 monclient: found mon.noname-a 2024-09-18T12:42:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.502+0000 7f6c3affd640 1 -- 172.21.15.116:0/2085015242 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c0a1110 con 0x7f6c3c09e910 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.504+0000 7f6c3affd640 1 -- 172.21.15.116:0/2085015242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c09e910 msgr2=0x7f6c3c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.504+0000 7f6c3affd640 1 --2- 172.21.15.116:0/2085015242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c09e910 0x7f6c3c09ed10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c06a250 tx=0x7f6c300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d8c985a9ca3c3c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 1 -- 172.21.15.116:0/2085015242 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c30032440 con 0x7f6c3c09e910 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: dump: 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 5 monclient: authenticate success, global_id 194271 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 1 -- 172.21.15.116:0/2085015242 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c300325a0 con 0x7f6c3c09e910 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 1 -- 172.21.15.116:0/2085015242 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c30032920 con 0x7f6c3c09e910 2024-09-18T12:42:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 4 set_mon_vals no callback set 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c3a7fc640 10 monclient: dump: 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:03.539 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 10 monclient: shutdown 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 1 -- 172.21.15.116:0/2085015242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c09e910 msgr2=0x7f6c3c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 1 --2- 172.21.15.116:0/2085015242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c09e910 0x7f6c3c09ed10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c06a250 tx=0x7f6c300091d0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 1 -- 172.21.15.116:0/2085015242 shutdown_connections 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 1 --2- 172.21.15.116:0/2085015242 >> v2:172.21.15.144:3300/0 conn(0x7f6c3c09f250 0x7f6c3c138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 1 --2- 172.21.15.116:0/2085015242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c09e910 0x7f6c3c09ed10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 1 -- 172.21.15.116:0/2085015242 >> 172.21.15.116:0/2085015242 conn(0x7f6c3c067d30 msgr2=0x7f6c3c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.505+0000 7f6c42060640 1 -- 172.21.15.116:0/2085015242 shutdown_connections 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.506+0000 7f6c42060640 1 -- 172.21.15.116:0/2085015242 wait complete. 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.506+0000 7f6c42060640 10 monclient: build_initial_monmap 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.506+0000 7f6c42060640 10 monclient: monmap: 2024-09-18T12:42:03.540 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 Processor -- start 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 -- start start 2024-09-18T12:42:03.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient: init 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c3c09e910 0x7f6c3c130fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c09f250 0x7f6c3c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): picked mon.noname-b con 0x7f6c3c09f250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 0x7f6c3c096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c09bde0 con 0x7f6c3c131ab0 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3b7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c3c09e910 msgr2=0x7f6c3c130fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:03.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c09d480 con 0x7f6c3c09f250 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c099780 con 0x7f6c3c09e910 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c42060640 10 monclient(hunting): authenticate will time out at 1750.828576s 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c3c09e910 0x7f6c3c130fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 0x7f6c3c096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 0x7f6c3c096fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54666/0 (socket says 172.21.15.116:54666) 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 -- 172.21.15.116:0/3057432433 learned_addr learned my addr 172.21.15.116:0/3057432433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): get_auth_request con 0x7f6c3c131ab0 auth_method 0 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3affd640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c09f250 0x7f6c3c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient(hunting): handle_auth_done global_id 194274 payload 824 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 -- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c3c09e910 msgr2=0x7f6c3c130fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c3c09e910 0x7f6c3c130fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 -- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c09f250 msgr2=0x7f6c3c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c09f250 0x7f6c3c096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 monclient: found mon.noname-a 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3bfff640 1 -- 172.21.15.116:0/3057432433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c30009450 con 0x7f6c3c131ab0 2024-09-18T12:42:03.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3affd640 10 monclient: get_auth_request con 0x7f6c3c09f250 auth_method 0 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.507+0000 7f6c3affd640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c09f250 0x7f6c3c096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c3bfff640 1 -- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 msgr2=0x7f6c3c096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c3bfff640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 0x7f6c3c096fa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400ea50 tx=0x7f6c340090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 1 -- 172.21.15.116:0/3057432433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400cb50 con 0x7f6c3c131ab0 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: dump: 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: _finish_auth 0 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 1 -- 172.21.15.116:0/3057432433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3400ccb0 con 0x7f6c3c131ab0 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c42060640 5 monclient: authenticate success, global_id 194274 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c42060640 10 monclient: _renew_subs 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c42060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c097cc0 con 0x7f6c3c131ab0 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 1 -- 172.21.15.116:0/3057432433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34013470 con 0x7f6c3c131ab0 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:03.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c237fe640 10 monclient: dump: 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c42060640 10 monclient: _renew_subs 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c42060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:03.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c098890 con 0x7f6c3c131ab0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.510+0000 7f6c39ffb640 4 set_mon_vals no callback set 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c237fe640 1 -- 172.21.15.116:0/3057432433 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6c34013840 con 0x7f6c3c131ab0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c42060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c42060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c42060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c099c20 con 0x7f6c3c131ab0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c237fe640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c28038440 0x7f6c2803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c237fe640 1 -- 172.21.15.116:0/3057432433 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6c3404c630 con 0x7f6c3c131ab0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c3b7fe640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c28038440 0x7f6c2803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.511+0000 7f6c3b7fe640 10 monclient: get_auth_request con 0x7f6c28038440 auth_method 0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.514+0000 7f6c3b7fe640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c28038440 0x7f6c2803a8f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007fd0 tx=0x7f6c2c007dd0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.515+0000 7f6c237fe640 1 -- 172.21.15.116:0/3057432433 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6c340236b0 con 0x7f6c3c131ab0 2024-09-18T12:42:03.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.515+0000 7f6c237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:03.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.515+0000 7f6c237fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.730+0000 7f6c42060640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.730+0000 7f6c42060640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.730+0000 7f6c42060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.730+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6c3c04bf60 con 0x7f6c3c131ab0 2024-09-18T12:42:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.731+0000 7f6c237fe640 1 -- 172.21.15.116:0/3057432433 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f6c340232f0 con 0x7f6c3c131ab0 2024-09-18T12:42:03.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.731+0000 7f6c237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:03.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.731+0000 7f6c237fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:03.733 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:03.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c28038440 msgr2=0x7f6c2803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:03.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c28038440 0x7f6c2803a8f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007fd0 tx=0x7f6c2c007dd0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 10 monclient: shutdown 2024-09-18T12:42:03.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:03.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 msgr2=0x7f6c3c096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 0x7f6c3c096fa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400ea50 tx=0x7f6c340090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 shutdown_connections 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6c28038440 0x7f6c2803a8f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c3c131ab0 0x7f6c3c096fa0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6c3c09f250 0x7f6c3c096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 --2- 172.21.15.116:0/3057432433 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6c3c09e910 0x7f6c3c130fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 >> 172.21.15.116:0/3057432433 conn(0x7f6c3c067d30 msgr2=0x7f6c3c0a1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 shutdown_connections 2024-09-18T12:42:03.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:03.733+0000 7f6c42060640 1 -- 172.21.15.116:0/3057432433 wait complete. 2024-09-18T12:42:06.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.848+0000 7fac03b1d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.848+0000 7fac03b1d640 10 monclient: build_initial_monmap 2024-09-18T12:42:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.848+0000 7fac03b1d640 10 monclient: monmap: 2024-09-18T12:42:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:06.848830+0000 2024-09-18T12:42:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:06.848830+0000 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:06.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.848+0000 7fac03b1d640 1 Processor -- start 2024-09-18T12:42:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 1 -- start start 2024-09-18T12:42:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient: init 2024-09-18T12:42:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:06.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabfc096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fabfc0971e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 0x7fabfc135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fabfc133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc099620 con 0x7fabfc096e10 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc099340 con 0x7fabfc0971e0 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc099900 con 0x7fabfc133b30 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac03b1d640 10 monclient(hunting): authenticate will time out at 1754.170534s 2024-09-18T12:42:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac020ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 msgr2=0x7fabfc135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac020ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 0x7fabfc135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac018b9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fabfc096e10 0x7fabfc09a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37806/0 (socket says 172.21.15.116:37806) 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac018b9640 1 -- 172.21.15.116:0/2552294223 learned_addr learned my addr 172.21.15.116:0/2552294223 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabfc0971e0 auth_method 0 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.849+0000 7fac010b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.850+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.850+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.850+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.850+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:06.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:06.851+0000 7fac018b9640 1 -- 172.21.15.116:0/2552294223 >> v1:172.21.15.116:6789/0 conn(0x7fabfc096e10 legacy=0x7fabfc09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:07.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.050+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 msgr2=0x7fabfc135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.050+0000 7fac020ba640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 0x7fabfc135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:42:07.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:07.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabfc0971e0 auth_method 0 2024-09-18T12:42:07.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:07.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:07.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:07.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:42:07.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:07.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:07.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.052+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:42:07.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.451+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 msgr2=0x7fabfc135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:07.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.451+0000 7fac020ba640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 0x7fabfc135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:42:07.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.453+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:07.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.453+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabfc0971e0 auth_method 0 2024-09-18T12:42:07.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.453+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:07.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.453+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:07.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.453+0000 7fac010b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:07.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.454+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:42:07.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.454+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:07.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.454+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:07.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:07.454+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:42:08.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.252+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 msgr2=0x7fabfc135f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:08.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.252+0000 7fac020ba640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 0x7fabfc135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:42:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabfc0971e0 auth_method 0 2024-09-18T12:42:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:08.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:42:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:08.255+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:42:09.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 10 monclient(hunting): tick 2024-09-18T12:42:09.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:42:09.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:42:09.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 msgr2=0x7fabfc135f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabfc133b30 0x7fabfc135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 msgr2=0x7fabfc0975e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.144:3300/0 conn(0x7fabfc0971e0 0x7fabfc0975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 -- 172.21.15.116:0/2552294223 >> v1:172.21.15.116:6789/0 conn(0x7fabfc096e10 legacy=0x7fabfc09a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 10 monclient: _add_conns ranks=[1,3] 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabe0005930 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 10 monclient(hunting): picked mon.noname-a con 0x7fabe0009650 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 -- 172.21.15.116:0/2552294223 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc09ac70 con 0x7fabe0005930 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 1 -- 172.21.15.116:0/2552294223 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabfc099340 con 0x7fabe0009650 2024-09-18T12:42:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.849+0000 7fabf37fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabe0009650 auth_method 0 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.850+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:09.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:09.852+0000 7fac018b9640 1 -- 172.21.15.116:0/2552294223 >> v1:172.21.15.144:6789/0 conn(0x7fabe0005930 legacy=0x7fabe0007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabe0009650 auth_method 0 2024-09-18T12:42:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.051+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:42:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabe0009650 auth_method 0 2024-09-18T12:42:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:10.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:10.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:10.452+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:42:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabe0009650 auth_method 0 2024-09-18T12:42:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.253+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:42:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.351+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291338826 0 0) 0x7fabfc09ac70 con 0x7fabe0005930 2024-09-18T12:42:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.351+0000 7fac008b7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.351+0000 7fac008b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.351+0000 7fac008b7640 10 monclient(hunting): my global_id is 174433 2024-09-18T12:42:11.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.351+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe80034e0 con 0x7fabe0005930 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2621350069 0 0) 0x7fabe80034e0 con 0x7fabe0005930 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 msgr2=0x7fabe000bb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 --2- 172.21.15.116:0/2552294223 >> v2:172.21.15.116:3300/0 conn(0x7fabe0009650 0x7fabe000bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: _finish_hunting 0 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:42:11.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe000d1b0 con 0x7fabe0005930 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: _finish_auth 0 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fabec001770 con 0x7fabe0005930 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac03b1d640 5 monclient: authenticate success, global_id 174433 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac03b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: dump: 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.361 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 >> v1:172.21.15.144:6789/0 conn(0x7fabe0005930 legacy=0x7fabe0007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 reap_dead start 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabe8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): picked mon.a con 0x7fabe80039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8004780 0x7fabe8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): picked mon.c con 0x7fabe8004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8005250 0x7fabe800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): picked mon.b con 0x7fabe8005250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc099340 con 0x7fabe80039d0 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe800c1c0 con 0x7fabe8005250 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe800c380 con 0x7fabe8004780 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac008b7640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:42:11.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac03b1d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac010b8640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8004780 msgr2=0x7fabe8004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac010b8640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8004780 0x7fabe8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabe8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8005250 0x7fabe800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): get_auth_request con 0x7fabe8005250 auth_method 0 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 10 monclient(hunting): get_auth_request con 0x7fabe80039d0 auth_method 0 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:11.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac018b9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient(hunting): handle_auth_done global_id 174433 payload 293 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8004780 msgr2=0x7fabe8004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8004780 0x7fabe8004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 msgr2=0x7fabe8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabe8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient: _finish_hunting 0 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 monclient: found mon.b 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:42:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.356+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80034e0 con 0x7fabe8005250 2024-09-18T12:42:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.357+0000 7fac020ba640 10 monclient: _finish_auth 0 2024-09-18T12:42:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.357+0000 7fac020ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac020ba640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8005250 msgr2=0x7fabe800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac020ba640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8005250 0x7fabe800b9d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fabfc09aff0 tx=0x7fabf4007530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23ee61b55ee88579 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf402aaf0 con 0x7fabe8005250 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 10 monclient: dump: 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.374 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf402ac90 con 0x7fabe8005250 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac03b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac03b1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 1 -- 172.21.15.116:0/2552294223 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf40336c0 con 0x7fabe8005250 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac03b1d640 4 set_mon_vals no callback set 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.360+0000 7fac008b7640 10 monclient: dump: 2024-09-18T12:42:11.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 10 monclient: shutdown 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 -- 172.21.15.116:0/2552294223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8005250 msgr2=0x7fabe800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8005250 0x7fabe800b9d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fabfc09aff0 tx=0x7fabf4007530 comp rx=0 tx=0).stop 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 -- 172.21.15.116:0/2552294223 shutdown_connections 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8005250 0x7fabe800b9d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8004780 0x7fabe8004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 --2- 172.21.15.116:0/2552294223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabe8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 -- 172.21.15.116:0/2552294223 >> 172.21.15.116:0/2552294223 conn(0x7fabfc0672b0 msgr2=0x7fabfc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 -- 172.21.15.116:0/2552294223 shutdown_connections 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.361+0000 7fac03b1d640 1 -- 172.21.15.116:0/2552294223 wait complete. 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 10 monclient: build_initial_monmap 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 10 monclient: monmap: 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 1 Processor -- start 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 1 -- start start 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 10 monclient: init 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabfc141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fabe80039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8004780 0x7fabfc141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.362+0000 7fac03b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fabe8004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8005250 0x7fabfc1311b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fabe8005250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc1179d0 con 0x7fabe80039d0 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc09b050 con 0x7fabe8004780 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc06f330 con 0x7fabe8005250 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac03b1d640 10 monclient(hunting): authenticate will time out at 1758.684048s 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac020ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8005250 msgr2=0x7fabfc1311b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac020ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8005250 0x7fabfc1311b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabfc141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabfc141200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42546/0 (socket says 172.21.15.116:42546) 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 -- 172.21.15.116:0/371096024 learned_addr learned my addr 172.21.15.116:0/371096024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): get_auth_request con 0x7fabe80039d0 auth_method 0 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac010b8640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8004780 0x7fabfc141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac010b8640 10 monclient(hunting): get_auth_request con 0x7fabe8004780 auth_method 0 2024-09-18T12:42:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac010b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac010b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac010b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient(hunting): handle_auth_done global_id 194277 payload 824 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 -- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8005250 msgr2=0x7fabfc1311b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8005250 0x7fabfc1311b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 -- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8004780 msgr2=0x7fabfc141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8004780 0x7fabfc141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient: _finish_hunting 0 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 monclient: found mon.noname-a 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac018b9640 1 -- 172.21.15.116:0/371096024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc06e020 con 0x7fabe80039d0 2024-09-18T12:42:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.363+0000 7fac010b8640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8004780 0x7fabfc141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.365+0000 7fac018b9640 1 -- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 msgr2=0x7fabfc141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.365+0000 7fac018b9640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabfc141200 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fabec001350 tx=0x7fabec0079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.365+0000 7fabf1ffb640 1 -- 172.21.15.116:0/371096024 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec018030 con 0x7fabe80039d0 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.365+0000 7fabf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 10 monclient: dump: 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 1 -- 172.21.15.116:0/371096024 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec00f9c0 con 0x7fabe80039d0 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fac03b1d640 5 monclient: authenticate success, global_id 194277 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fac03b1d640 10 monclient: _renew_subs 2024-09-18T12:42:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fac03b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc06db80 con 0x7fabe80039d0 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 1 -- 172.21.15.116:0/371096024 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabec00fd60 con 0x7fabe80039d0 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf1ffb640 10 monclient: dump: 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fac03b1d640 10 monclient: _renew_subs 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fabf3fff640 4 set_mon_vals no callback set 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fac03b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.366+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc118510 con 0x7fabe80039d0 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fabf1ffb640 1 -- 172.21.15.116:0/371096024 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fabec01b070 con 0x7fabe80039d0 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fac03b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fabf1ffb640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fabe0038d20 0x7fabe003b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fac03b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fac03b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc04bf60 con 0x7fabe80039d0 2024-09-18T12:42:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fabf1ffb640 1 -- 172.21.15.116:0/371096024 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fabec019070 con 0x7fabe80039d0 2024-09-18T12:42:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.367+0000 7fac010b8640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fabe0038d20 0x7fabe003b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.368+0000 7fac010b8640 10 monclient: get_auth_request con 0x7fabe0038d20 auth_method 0 2024-09-18T12:42:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.370+0000 7fac010b8640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fabe0038d20 0x7fabe003b1d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006d30 tx=0x7fabf8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.372+0000 7fabf1ffb640 1 -- 172.21.15.116:0/371096024 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fabec01bb00 con 0x7fabe80039d0 2024-09-18T12:42:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.372+0000 7fabf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.372+0000 7fabf1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.588+0000 7fac03b1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.588+0000 7fac03b1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.588+0000 7fac03b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.589+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fabfc068a60 con 0x7fabe80039d0 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.589+0000 7fabf1ffb640 1 -- 172.21.15.116:0/371096024 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fabec01b320 con 0x7fabe80039d0 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.589+0000 7fabf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.589+0000 7fabf1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:11.590 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":166,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fabe0038d20 msgr2=0x7fabe003b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fabe0038d20 0x7fabe003b1d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006d30 tx=0x7fabf8008040 comp rx=0 tx=0).stop 2024-09-18T12:42:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 10 monclient: shutdown 2024-09-18T12:42:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 msgr2=0x7fabfc141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabfc141200 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fabec001350 tx=0x7fabec0079a0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 shutdown_connections 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fabe0038d20 0x7fabe003b1d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fabe8005250 0x7fabfc1311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fabe8004780 0x7fabfc141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 --2- 172.21.15.116:0/371096024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe80039d0 0x7fabfc141200 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 >> 172.21.15.116:0/371096024 conn(0x7fabfc0672b0 msgr2=0x7fabfc09d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 shutdown_connections 2024-09-18T12:42:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.591+0000 7fac03b1d640 1 -- 172.21.15.116:0/371096024 wait complete. 2024-09-18T12:42:11.602 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:42:11.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:42:11.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:42:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.694+0000 7fd973b1d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.695+0000 7fd973b1d640 10 monclient: build_initial_monmap 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.695+0000 7fd973b1d640 10 monclient: monmap: 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:11.695678+0000 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:11.695678+0000 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:11.697 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.695+0000 7fd973b1d640 1 Processor -- start 2024-09-18T12:42:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.695+0000 7fd973b1d640 1 -- start start 2024-09-18T12:42:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.695+0000 7fd973b1d640 10 monclient: init 2024-09-18T12:42:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd96c09a510 0x7fd96c096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd96c09a510 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd96c0971a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c0975a0 0x7fd96c12fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd96c0975a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96c09cb00 con 0x7fd96c0971a0 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c09a8e0 con 0x7fd96c09a510 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c06ed80 con 0x7fd96c0975a0 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd973b1d640 10 monclient(hunting): authenticate will time out at 1759.017339s 2024-09-18T12:42:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9720ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c0975a0 msgr2=0x7fd96c12fd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9720ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c0975a0 0x7fd96c12fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd96c09a510 0x7fd96c096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd96c09a510 0x7fd96c096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:57138/0 (socket says 172.21.15.116:57138) 2024-09-18T12:42:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 1 -- 172.21.15.116:0/1736508161 learned_addr learned my addr 172.21.15.116:0/1736508161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 10 monclient(hunting): get_auth_request con 0x7fd96c09a510 auth_method 0 2024-09-18T12:42:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.696+0000 7fd9718b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.698+0000 7fd9710b8640 1 -- 172.21.15.116:0/1736508161 >> v1:172.21.15.144:6789/0 conn(0x7fd96c0971a0 legacy=0x7fd96c12f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 10 monclient(hunting): handle_auth_done global_id 174439 payload 274 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 1 -- 172.21.15.116:0/1736508161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c0975a0 msgr2=0x7fd96c12fd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 1 --2- 172.21.15.116:0/1736508161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c0975a0 0x7fd96c12fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 1 -- 172.21.15.116:0/1736508161 >> v1:172.21.15.144:6789/0 conn(0x7fd96c0971a0 legacy=0x7fd96c12f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 10 monclient: _finish_hunting 0 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 1 monclient: found mon.noname-c 2024-09-18T12:42:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:42:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.699+0000 7fd9718b9640 1 -- 172.21.15.116:0/1736508161 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c09c820 con 0x7fd96c09a510 2024-09-18T12:42:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.702+0000 7fd9718b9640 1 -- 172.21.15.116:0/1736508161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c09a510 msgr2=0x7fd96c096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.702+0000 7fd9718b9640 1 --2- 172.21.15.116:0/1736508161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c09a510 0x7fd96c096c60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd96c06ef70 tx=0x7fd9680091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec57e6cacc861a99 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 1 -- 172.21.15.116:0/1736508161 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96802ae90 con 0x7fd96c09a510 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: dump: 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: _finish_auth 0 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 1 -- 172.21.15.116:0/1736508161 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96802f4b0 con 0x7fd96c09a510 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 1 -- 172.21.15.116:0/1736508161 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96802f850 con 0x7fd96c09a510 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd9708b7640 10 monclient: dump: 2024-09-18T12:42:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd973b1d640 5 monclient: authenticate success, global_id 174439 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd973b1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:11.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.703+0000 7fd973b1d640 4 set_mon_vals no callback set 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 10 monclient: shutdown 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 -- 172.21.15.116:0/1736508161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c09a510 msgr2=0x7fd96c096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 --2- 172.21.15.116:0/1736508161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c09a510 0x7fd96c096c60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd96c06ef70 tx=0x7fd9680091d0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 -- 172.21.15.116:0/1736508161 shutdown_connections 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 --2- 172.21.15.116:0/1736508161 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c0975a0 0x7fd96c12fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 --2- 172.21.15.116:0/1736508161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c09a510 0x7fd96c096c60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 -- 172.21.15.116:0/1736508161 >> 172.21.15.116:0/1736508161 conn(0x7fd96c0672b0 msgr2=0x7fd96c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 -- 172.21.15.116:0/1736508161 shutdown_connections 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 -- 172.21.15.116:0/1736508161 wait complete. 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 10 monclient: build_initial_monmap 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 10 monclient: monmap: 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.704+0000 7fd973b1d640 1 Processor -- start 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 1 -- start start 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient: init 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 0x7fd96c141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): picked mon.noname-a con 0x7fd96c0971a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c0975a0 0x7fd96c12fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): picked mon.noname-b con 0x7fd96c0975a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c09a510 0x7fd96c098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): picked mon.noname-c con 0x7fd96c09a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c11f220 con 0x7fd96c0971a0 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c06db80 con 0x7fd96c0975a0 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c141f80 con 0x7fd96c09a510 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd973b1d640 10 monclient(hunting): authenticate will time out at 1759.026509s 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9720ba640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c09a510 msgr2=0x7fd96c098300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9720ba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c09a510 0x7fd96c098300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 0x7fd96c141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 0x7fd96c141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42554/0 (socket says 172.21.15.116:42554) 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 -- 172.21.15.116:0/4146053081 learned_addr learned my addr 172.21.15.116:0/4146053081 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): get_auth_request con 0x7fd96c0971a0 auth_method 0 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9710b8640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c0975a0 0x7fd96c12fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9710b8640 10 monclient(hunting): get_auth_request con 0x7fd96c0975a0 auth_method 0 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9710b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9710b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9710b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient(hunting): handle_auth_done global_id 194280 payload 824 2024-09-18T12:42:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c09a510 msgr2=0x7fd96c098300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd96c09a510 0x7fd96c098300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c0975a0 msgr2=0x7fd96c12fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd96c0975a0 0x7fd96c12fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient: _finish_hunting 0 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 monclient: found mon.noname-a 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.705+0000 7fd9718b9640 1 -- 172.21.15.116:0/4146053081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd968009450 con 0x7fd96c0971a0 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd9718b9640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 msgr2=0x7fd96c141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd9718b9640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 0x7fd96c141080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd968009210 tx=0x7fd968032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9680328e0 con 0x7fd96c0971a0 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: dump: 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: _finish_auth 0 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96802ae90 con 0x7fd96c0971a0 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd973b1d640 5 monclient: authenticate success, global_id 194280 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd973b1d640 10 monclient: _renew_subs 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968032cb0 con 0x7fd96c0971a0 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd973b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c099f30 con 0x7fd96c0971a0 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd961ffb640 10 monclient: dump: 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd963fff640 4 set_mon_vals no callback set 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd973b1d640 10 monclient: _renew_subs 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd973b1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.708+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c11dc70 con 0x7fd96c0971a0 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd973b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd973b1d640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 0x7fd96c098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd973b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c11efe0 con 0x7fd96c1415c0 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd968039070 con 0x7fd96c0971a0 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 0x7fd96c098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient: get_auth_request con 0x7fd96c1415c0 auth_method 0 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd961ffb640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd950038440 0x7fd95003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd96806c720 con 0x7fd96c0971a0 2024-09-18T12:42:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 10 monclient(hunting): handle_auth_done global_id 194283 payload 824 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9710b8640 1 -- 172.21.15.116:0/4146053081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd96c11ec70 con 0x7fd96c1415c0 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9720ba640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd950038440 0x7fd95003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.709+0000 7fd9720ba640 10 monclient: get_auth_request con 0x7fd950038440 auth_method 0 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.711+0000 7fd9710b8640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 msgr2=0x7fd96c098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.711+0000 7fd9710b8640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 0x7fd96c098840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd96c1305e0 tx=0x7fd95c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0075c0 con 0x7fd96c1415c0 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 10 monclient: dump: 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.723 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd95c005780 con 0x7fd96c1415c0 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 msgr2=0x7fd96c098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 0x7fd96c098840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd96c1305e0 tx=0x7fd95c009040 comp rx=0 tx=0).stop 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 msgr2=0x7fd96c098840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd961ffb640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c1415c0 0x7fd96c098840 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd96c1305e0 tx=0x7fd95c009040 comp rx=0 tx=0).stop 2024-09-18T12:42:11.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.712+0000 7fd9720ba640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd950038440 0x7fd95003a8f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd96c11f1d0 tx=0x7fd964009370 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd973b1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:11.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd973b1d640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 0x7fd96c098d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd973b1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c065f20 con 0x7fd96c12ff60 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 0x7fd96c098d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient: get_auth_request con 0x7fd96c12ff60 auth_method 0 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 10 monclient(hunting): handle_auth_done global_id 194286 payload 824 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.718+0000 7fd9710b8640 1 -- 172.21.15.116:0/4146053081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd96c04bf60 con 0x7fd96c12ff60 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd9710b8640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 msgr2=0x7fd96c098d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd9710b8640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 0x7fd96c098d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd96c1305e0 tx=0x7fd95c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0135f0 con 0x7fd96c12ff60 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 10 monclient: dump: 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.728 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fd95c013750 con 0x7fd96c12ff60 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 msgr2=0x7fd96c098d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 0x7fd96c098d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd96c1305e0 tx=0x7fd95c017000 comp rx=0 tx=0).stop 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 msgr2=0x7fd96c098d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.720+0000 7fd961ffb640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c12ff60 0x7fd96c098d80 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd96c1305e0 tx=0x7fd95c017000 comp rx=0 tx=0).stop 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 166, 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:42:11.730 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.731 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:11.732 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:11.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:11.734 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:11.735 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.737 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:11.738 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:11.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.741 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:42:11.742 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.744 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:11.744 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:11.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd950038440 msgr2=0x7fd95003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd950038440 0x7fd95003a8f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd96c11f1d0 tx=0x7fd964009370 comp rx=0 tx=0).stop 2024-09-18T12:42:11.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 10 monclient: shutdown 2024-09-18T12:42:11.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:11.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 msgr2=0x7fd96c141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 0x7fd96c141080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd968009210 tx=0x7fd968032000 comp rx=0 tx=0).stop 2024-09-18T12:42:11.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd9720ba640 1 -- 172.21.15.116:0/4146053081 reap_dead start 2024-09-18T12:42:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 shutdown_connections 2024-09-18T12:42:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 --2- 172.21.15.116:0/4146053081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd96c0971a0 0x7fd96c141080 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 >> 172.21.15.116:0/4146053081 conn(0x7fd96c0672b0 msgr2=0x7fd96c09fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 shutdown_connections 2024-09-18T12:42:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.721+0000 7fd973b1d640 1 -- 172.21.15.116:0/4146053081 wait complete. 2024-09-18T12:42:11.746 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:42:11.746 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:42:11.746 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:42:11.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.839+0000 7f47bdf9d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.839+0000 7f47bdf9d640 10 monclient: build_initial_monmap 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.839+0000 7f47bdf9d640 10 monclient: monmap: 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:11.839802+0000 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:11.839802+0000 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:11.840 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:11.841 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:11.841 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.839+0000 7f47bdf9d640 1 Processor -- start 2024-09-18T12:42:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.839+0000 7f47bdf9d640 1 -- start start 2024-09-18T12:42:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.839+0000 7f47bdf9d640 10 monclient: init 2024-09-18T12:42:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T12:42:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f47b8131ab0 0x7f47b8131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f47b8131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809e910 0x7f47b809ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f47b809e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47b809f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b809faf0 con 0x7f47b809f220 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b809fd30 con 0x7f47b8131ab0 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b80a1110 con 0x7f47b809e910 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47bdf9d640 10 monclient(hunting): authenticate will time out at 1759.161401s 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47aeffd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809e910 msgr2=0x7f47b809ece0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:11.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47aeffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809e910 0x7f47b809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f47b809f220 0x7f47b8068ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37814/0 (socket says 172.21.15.116:37814) 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b7fff640 1 -- 172.21.15.116:0/4102775302 learned_addr learned my addr 172.21.15.116:0/4102775302 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 1 --2- 172.21.15.116:0/4102775302 >> v2:172.21.15.144:3300/0 conn(0x7f47b8131ab0 0x7f47b8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): get_auth_request con 0x7f47b8131ab0 auth_method 0 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient(hunting): handle_auth_done global_id 174448 payload 274 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 1 -- 172.21.15.116:0/4102775302 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809e910 msgr2=0x7f47b809ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 1 --2- 172.21.15.116:0/4102775302 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809e910 0x7f47b809ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 1 -- 172.21.15.116:0/4102775302 >> v1:172.21.15.116:6789/0 conn(0x7f47b809f220 legacy=0x7f47b8068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:42:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 1 monclient: found mon.noname-c 2024-09-18T12:42:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:42:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.840+0000 7f47b77fe640 1 -- 172.21.15.116:0/4102775302 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b80a13f0 con 0x7f47b8131ab0 2024-09-18T12:42:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b77fe640 1 -- 172.21.15.116:0/4102775302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 msgr2=0x7f47b8131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b77fe640 1 --2- 172.21.15.116:0/4102775302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 0x7f47b8131e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009850 tx=0x7f47a80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408dd48dbcc52237 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 1 -- 172.21.15.116:0/4102775302 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8032440 con 0x7f47b8131ab0 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: dump: 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 1 -- 172.21.15.116:0/4102775302 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a80325a0 con 0x7f47b8131ab0 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 1 -- 172.21.15.116:0/4102775302 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8032980 con 0x7f47b8131ab0 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47bdf9d640 5 monclient: authenticate success, global_id 174448 2024-09-18T12:42:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47bdf9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47bdf9d640 4 set_mon_vals no callback set 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.844+0000 7f47b6ffd640 10 monclient: dump: 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.855 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 10 monclient: shutdown 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4102775302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 msgr2=0x7f47b8131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 --2- 172.21.15.116:0/4102775302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 0x7f47b8131e80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009850 tx=0x7f47a80090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4102775302 shutdown_connections 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 --2- 172.21.15.116:0/4102775302 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809e910 0x7f47b809ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 --2- 172.21.15.116:0/4102775302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 0x7f47b8131e80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4102775302 >> 172.21.15.116:0/4102775302 conn(0x7f47b8067d30 msgr2=0x7f47b80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4102775302 shutdown_connections 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.845+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4102775302 wait complete. 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.846+0000 7f47bdf9d640 10 monclient: build_initial_monmap 2024-09-18T12:42:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.846+0000 7f47bdf9d640 10 monclient: monmap: 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.846+0000 7f47bdf9d640 1 Processor -- start 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.846+0000 7f47bdf9d640 1 -- start start 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.846+0000 7f47bdf9d640 10 monclient: init 2024-09-18T12:42:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.846+0000 7f47bdf9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.846+0000 7f47bdf9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b809e910 0x7f47b81416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f47b809e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809f220 0x7f47b8141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f47b809f220 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 0x7f47b8142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f47b8131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b811fc30 con 0x7f47b809e910 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b809bde0 con 0x7f47b8131ab0 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b809de20 con 0x7f47b809f220 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47bdf9d640 10 monclient(hunting): authenticate will time out at 1759.168075s 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47aeffd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809f220 msgr2=0x7f47b8141c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47aeffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809f220 0x7f47b8141c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b809e910 0x7f47b81416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b809e910 0x7f47b81416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42584/0 (socket says 172.21.15.116:42584) 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 -- 172.21.15.116:0/4067915756 learned_addr learned my addr 172.21.15.116:0/4067915756 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): get_auth_request con 0x7f47b809e910 auth_method 0 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b7fff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 0x7f47b8142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b7fff640 10 monclient(hunting): get_auth_request con 0x7f47b8131ab0 auth_method 0 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient(hunting): handle_auth_done global_id 194289 payload 824 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809f220 msgr2=0x7f47b8141c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47b809f220 0x7f47b8141c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 msgr2=0x7f47b8142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 0x7f47b8142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 monclient: found mon.noname-a 2024-09-18T12:42:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b77fe640 1 -- 172.21.15.116:0/4067915756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009390 con 0x7f47b809e910 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.847+0000 7f47b7fff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b8131ab0 0x7f47b8142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47b77fe640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b809e910 msgr2=0x7f47b81416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47b77fe640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b809e910 0x7f47b81416c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009850 tx=0x7f47a8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8031940 con 0x7f47b809e910 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: dump: 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: _finish_auth 0 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47bdf9d640 5 monclient: authenticate success, global_id 194289 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a8031aa0 con 0x7f47b809e910 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8031e60 con 0x7f47b809e910 2024-09-18T12:42:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47bdf9d640 10 monclient: _renew_subs 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47bdf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b8122a20 con 0x7f47b809e910 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47affff640 10 monclient: dump: 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.849+0000 7f47b67fc640 4 set_mon_vals no callback set 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47bdf9d640 10 monclient: _renew_subs 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47bdf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b8120530 con 0x7f47b809e910 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47bdf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47bdf9d640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 0x7f47b805e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47bdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8063740 con 0x7f47b811f260 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f47a8039070 con 0x7f47b809e910 2024-09-18T12:42:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47aeffd640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 0x7f47b805e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47aeffd640 10 monclient: get_auth_request con 0x7f47b811f260 auth_method 0 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.850+0000 7f47affff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4798038440 0x7f479803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f47a806c950 con 0x7f47b809e910 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47b7fff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4798038440 0x7f479803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47b7fff640 10 monclient: get_auth_request con 0x7f4798038440 auth_method 0 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47aeffd640 10 monclient(hunting): handle_auth_done global_id 174454 payload 824 2024-09-18T12:42:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.851+0000 7f47aeffd640 1 -- 172.21.15.116:0/4067915756 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47b804bf60 con 0x7f47b811f260 2024-09-18T12:42:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.853+0000 7f47aeffd640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 msgr2=0x7f47b805e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.853+0000 7f47aeffd640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 0x7f47b805e310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b440 tx=0x7f47a0009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.853+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0003e70 con 0x7f47b811f260 2024-09-18T12:42:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.853+0000 7f47affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.853+0000 7f47affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.853+0000 7f47b7fff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4798038440 0x7f479803a8f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f47b000e1b0 tx=0x7f47b0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.853+0000 7f47affff640 10 monclient: dump: 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.854+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f47a00050f0 con 0x7f47b811f260 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.854+0000 7f47affff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.854+0000 7f47affff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.854+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 msgr2=0x7f47b805e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.854+0000 7f47affff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 0x7f47b805e310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b440 tx=0x7f47a0009120 comp rx=0 tx=0).stop 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.854+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 msgr2=0x7f47b805e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.854+0000 7f47affff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b811f260 0x7f47b805e310 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b440 tx=0x7f47a0009120 comp rx=0 tx=0).stop 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.859+0000 7f47bdf9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.859+0000 7f47bdf9d640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 0x7f47b80298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.859+0000 7f47bdf9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.859+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8062f40 con 0x7f47b805e850 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 0x7f47b80298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient: get_auth_request con 0x7f47b805e850 auth_method 0 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 10 monclient(hunting): handle_auth_done global_id 174457 payload 824 2024-09-18T12:42:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.860+0000 7f47aeffd640 1 -- 172.21.15.116:0/4067915756 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f47b8121a60 con 0x7f47b805e850 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.862+0000 7f47aeffd640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 msgr2=0x7f47b80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.862+0000 7f47aeffd640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 0x7f47b80298c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f47a0003fd0 tx=0x7f47a0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.862+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a001e620 con 0x7f47b805e850 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.862+0000 7f47affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.862+0000 7f47affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.862+0000 7f47affff640 10 monclient: dump: 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:11.872 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f47a001e780 con 0x7f47b805e850 2024-09-18T12:42:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47affff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47affff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 msgr2=0x7f47b80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47affff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 0x7f47b80298c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f47a0003fd0 tx=0x7f47a0015000 comp rx=0 tx=0).stop 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47affff640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 msgr2=0x7f47b80298c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47affff640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b805e850 0x7f47b80298c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 166, 2024-09-18T12:42:11.877 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:11.878 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:11.879 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:11.880 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:11.881 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:11.882 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:11.885 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:42:11.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:11.889 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:11.890 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4798038440 msgr2=0x7f479803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4798038440 0x7f479803a8f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f47b000e1b0 tx=0x7f47b0009040 comp rx=0 tx=0).stop 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 10 monclient: shutdown 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b809e910 msgr2=0x7f47b81416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 --2- 172.21.15.116:0/4067915756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b809e910 0x7f47b81416c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009850 tx=0x7f47a8032000 comp rx=0 tx=0).stop 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47b7fff640 1 -- 172.21.15.116:0/4067915756 reap_dead start 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47b7fff640 1 -- 172.21.15.116:0/4067915756 reap_dead start 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 shutdown_connections 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 >> 172.21.15.116:0/4067915756 conn(0x7f47b8067d30 msgr2=0x7f47b809b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 shutdown_connections 2024-09-18T12:42:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:11.863+0000 7f47bdf9d640 1 -- 172.21.15.116:0/4067915756 wait complete. 2024-09-18T12:42:11.892 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:42:11.893 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:42:31.893 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:42:31.894 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:42:31.894 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:42:31.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:42:31.898 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:42:31.898 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:42:31.898 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.000+0000 7f73cfeaf640 10 monclient: get_monmap_and_config 2024-09-18T12:42:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.000+0000 7f73cfeaf640 10 monclient: build_initial_monmap 2024-09-18T12:42:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.000+0000 7f73cfeaf640 10 monclient: monmap: 2024-09-18T12:42:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:32.002327+0000 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:32.002327+0000 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:32.003 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 1 Processor -- start 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 1 -- start start 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient: init 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73c8131980 0x7f73c8131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient(hunting): picked mon.noname-a con 0x7f73c8131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c809ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient(hunting): picked mon.noname-b con 0x7f73c809fe10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73c809ef70 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c81325b0 con 0x7f73c809ef70 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c80a11f0 con 0x7f73c8131980 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8132290 con 0x7f73c809fe10 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.001+0000 7f73cfeaf640 10 monclient(hunting): authenticate will time out at 1779.323967s 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c809ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73c8131980 0x7f73c8131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c809ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34080/0 (socket says 172.21.15.116:34080) 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): get_auth_request con 0x7f73c8131980 auth_method 0 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 1 -- 172.21.15.116:0/3628496859 learned_addr learned my addr 172.21.15.116:0/3628496859 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 10 monclient(hunting): get_auth_request con 0x7f73c809fe10 auth_method 0 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 1 -- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 msgr2=0x7f73c809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 1 -- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 msgr2=0x7f73c809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 1 --2- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c809ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cd44a640 1 --2- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c809ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_done global_id 194292 payload 274 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 1 -- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 msgr2=0x7f73c809ea30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:32.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 1 --2- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c809ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 1 -- 172.21.15.116:0/3628496859 >> v1:172.21.15.144:6789/0 conn(0x7f73c809ef70 legacy=0x7f73c8068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient: _finish_hunting 0 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 1 monclient: found mon.noname-a 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.002+0000 7f73cdc4b640 1 -- 172.21.15.116:0/3628496859 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c809f390 con 0x7f73c8131980 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cdc4b640 1 -- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c8131980 msgr2=0x7f73c8131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cdc4b640 1 --2- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c8131980 0x7f73c8131d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009850 tx=0x7f73c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770b1bb01862d33a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 1 -- 172.21.15.116:0/3628496859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c402add0 con 0x7f73c8131980 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: dump: 2024-09-18T12:42:32.014 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: _finish_auth 0 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 1 -- 172.21.15.116:0/3628496859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c4032440 con 0x7f73c8131980 2024-09-18T12:42:32.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 5 monclient: authenticate success, global_id 194292 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 4 set_mon_vals no callback set 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 1 -- 172.21.15.116:0/3628496859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c40327e0 con 0x7f73c8131980 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73ccc49640 10 monclient: dump: 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.016 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 10 monclient: shutdown 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 1 -- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c8131980 msgr2=0x7f73c8131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c8131980 0x7f73c8131d50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009850 tx=0x7f73c40090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 1 -- 172.21.15.116:0/3628496859 shutdown_connections 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c809ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/3628496859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c8131980 0x7f73c8131d50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.005+0000 7f73cfeaf640 1 -- 172.21.15.116:0/3628496859 >> 172.21.15.116:0/3628496859 conn(0x7f73c80a5050 msgr2=0x7f73c80a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.006+0000 7f73cfeaf640 1 -- 172.21.15.116:0/3628496859 shutdown_connections 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.006+0000 7f73cfeaf640 1 -- 172.21.15.116:0/3628496859 wait complete. 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.006+0000 7f73cfeaf640 10 monclient: build_initial_monmap 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.006+0000 7f73cfeaf640 10 monclient: monmap: 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:32.018 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.006+0000 7f73cfeaf640 1 Processor -- start 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.006+0000 7f73cfeaf640 1 -- start start 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.006+0000 7f73cfeaf640 10 monclient: init 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 0x7f73c809aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): picked mon.noname-a con 0x7f73c809ef70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c8130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): picked mon.noname-c con 0x7f73c809fe10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c8131980 0x7f73c811b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): picked mon.noname-b con 0x7f73c8131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:32.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c809e4e0 con 0x7f73c809ef70 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8069eb0 con 0x7f73c8131980 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c811e070 con 0x7f73c809fe10 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cfeaf640 10 monclient(hunting): authenticate will time out at 1779.329312s 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 0x7f73c809aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c8130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 0x7f73c809aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43466/0 (socket says 172.21.15.116:43466) 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 -- 172.21.15.116:0/4067937947 learned_addr learned my addr 172.21.15.116:0/4067937947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): get_auth_request con 0x7f73c809ef70 auth_method 0 2024-09-18T12:42:32.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 10 monclient(hunting): get_auth_request con 0x7f73c809fe10 auth_method 0 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 1 -- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 msgr2=0x7f73c8130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 1 -- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 msgr2=0x7f73c8130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c8130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cd44a640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c8130c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73ce44c640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c8131980 0x7f73c811b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient(hunting): handle_auth_done global_id 194295 payload 824 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 -- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 msgr2=0x7f73c8130c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c8130c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 -- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c8131980 msgr2=0x7f73c811b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c8131980 0x7f73c811b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient: _finish_hunting 0 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 monclient: found mon.noname-a 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73cdc4b640 1 -- 172.21.15.116:0/4067937947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c4009390 con 0x7f73c809ef70 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73ce44c640 10 monclient: get_auth_request con 0x7f73c8131980 auth_method 0 2024-09-18T12:42:32.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.007+0000 7f73ce44c640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c8131980 0x7f73c811b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.009+0000 7f73cdc4b640 1 -- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 msgr2=0x7f73c809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.009+0000 7f73cdc4b640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 0x7f73c809aba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009360 tx=0x7f73c4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 1 -- 172.21.15.116:0/4067937947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4031d40 con 0x7f73c809ef70 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: dump: 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: _finish_auth 0 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73cfeaf640 5 monclient: authenticate success, global_id 194295 2024-09-18T12:42:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 1 -- 172.21.15.116:0/4067937947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c403d440 con 0x7f73c809ef70 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73cfeaf640 10 monclient: _renew_subs 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73cfeaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c81419b0 con 0x7f73c809ef70 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 1 -- 172.21.15.116:0/4067937947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4031ea0 con 0x7f73c809ef70 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bdffb640 10 monclient: dump: 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73cfeaf640 10 monclient: _renew_subs 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73cfeaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c811dd90 con 0x7f73c809ef70 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.010+0000 7f73bffff640 4 set_mon_vals no callback set 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73bdffb640 1 -- 172.21.15.116:0/4067937947 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f73c4039040 con 0x7f73c809ef70 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73cfeaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73cfeaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73cfeaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c804bf60 con 0x7f73c809ef70 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73bdffb640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f73ac038490 0x7f73ac03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73bdffb640 1 -- 172.21.15.116:0/4067937947 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f73c406c370 con 0x7f73c809ef70 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73cd44a640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f73ac038490 0x7f73ac03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.011+0000 7f73cd44a640 10 monclient: get_auth_request con 0x7f73ac038490 auth_method 0 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.014+0000 7f73cd44a640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f73ac038490 0x7f73ac03a940 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f73c8063630 tx=0x7f73b8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.015+0000 7f73bdffb640 1 -- 172.21.15.116:0/4067937947 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f73c4030db0 con 0x7f73c809ef70 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.015+0000 7f73bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.015+0000 7f73bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:32.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.229+0000 7f73cfeaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.229+0000 7f73cfeaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.229+0000 7f73cfeaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.229+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f73c8067f00 con 0x7f73c809ef70 2024-09-18T12:42:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.230+0000 7f73bdffb640 1 -- 172.21.15.116:0/4067937947 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f73c4042610 con 0x7f73c809ef70 2024-09-18T12:42:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.230+0000 7f73bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.230+0000 7f73bdffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:32.232 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":166,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f73ac038490 msgr2=0x7f73ac03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f73ac038490 0x7f73ac03a940 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f73c8063630 tx=0x7f73b8008040 comp rx=0 tx=0).stop 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 10 monclient: shutdown 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 msgr2=0x7f73c809aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 0x7f73c809aba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009360 tx=0x7f73c4032000 comp rx=0 tx=0).stop 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 shutdown_connections 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f73ac038490 0x7f73ac03a940 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f73c8131980 0x7f73c811b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f73c809fe10 0x7f73c8130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 --2- 172.21.15.116:0/4067937947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c809ef70 0x7f73c809aba0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 >> 172.21.15.116:0/4067937947 conn(0x7f73c80a5050 msgr2=0x7f73c809e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 shutdown_connections 2024-09-18T12:42:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:32.232+0000 7f73cfeaf640 1 -- 172.21.15.116:0/4067937947 wait complete. 2024-09-18T12:42:35.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:35.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.346+0000 7f695d99d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:35.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.347+0000 7f695d99d640 10 monclient: build_initial_monmap 2024-09-18T12:42:35.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.347+0000 7f695d99d640 10 monclient: monmap: 2024-09-18T12:42:35.349 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:35.348676+0000 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:35.348676+0000 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.347+0000 7f695d99d640 1 Processor -- start 2024-09-18T12:42:35.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.347+0000 7f695d99d640 1 -- start start 2024-09-18T12:42:35.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.347+0000 7f695d99d640 10 monclient: init 2024-09-18T12:42:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T12:42:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69581318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f695809ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6958137c30 0x7f695813a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f6958137c30 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6958131cc0 con 0x7f695809ea50 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695809f530 con 0x7f69581318f0 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695809fbf0 con 0x7f6958137c30 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f695d99d640 10 monclient(hunting): authenticate will time out at 1782.670245s 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69567fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f695809ea50 0x7f695809ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46516/0 (socket says 172.21.15.116:46516) 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69567fc640 1 -- 172.21.15.116:0/3627111704 learned_addr learned my addr 172.21.15.116:0/3627111704 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 1 --2- 172.21.15.116:0/3627111704 >> v2:172.21.15.144:3300/0 conn(0x7f6958137c30 0x7f695813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 10 monclient(hunting): get_auth_request con 0x7f6958137c30 auth_method 0 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.348+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f69567fc640 1 -- 172.21.15.116:0/3627111704 >> v1:172.21.15.116:6789/0 conn(0x7f695809ea50 legacy=0x7f695809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105082095 0 0) 0x7f6958131cc0 con 0x7f695809ea50 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 10 monclient(hunting): my global_id is 194298 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69400034e0 con 0x7f695809ea50 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6956ffd640 1 -- 172.21.15.116:0/3627111704 >> v1:172.21.15.144:6789/0 conn(0x7f69581318f0 legacy=0x7f6958131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470201178 0 0) 0x7f69400034e0 con 0x7f695809ea50 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 >> v2:172.21.15.144:3300/0 conn(0x7f6958137c30 msgr2=0x7f695813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 1 --2- 172.21.15.116:0/3627111704 >> v2:172.21.15.144:3300/0 conn(0x7f6958137c30 0x7f695813a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.350+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 >> v1:172.21.15.144:6789/0 conn(0x7f69581318f0 legacy=0x7f6958131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695809fe30 con 0x7f695809ea50 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: _finish_auth 0 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f695d99d640 5 monclient: authenticate success, global_id 194298 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f694c0017a0 con 0x7f695809ea50 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f695d99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:42:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: dump: 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 >> v1:172.21.15.116:6789/0 conn(0x7f695809ea50 legacy=0x7f695809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6940003bc0 0x7f6940003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): picked mon.a con 0x7f6940003bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6940004780 0x7f6940004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): picked mon.b con 0x7f6940004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69400099c0 0x7f694000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): picked mon.c con 0x7f69400099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695809fbf0 con 0x7f6940003bc0 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694000c360 con 0x7f6940004780 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694000c520 con 0x7f69400099c0 2024-09-18T12:42:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6955ffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f695d99d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 1 --2- 172.21.15.116:0/3627111704 >> v2:172.21.15.144:3300/0 conn(0x7f6958137c30 0x7f695813a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6940003bc0 0x7f6940003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): get_auth_request con 0x7f6940003bc0 auth_method 0 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69400099c0 0x7f694000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 10 monclient(hunting): get_auth_request con 0x7f69400099c0 auth_method 0 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69567fc640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6940004780 0x7f6940004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69567fc640 10 monclient(hunting): get_auth_request con 0x7f6940004780 auth_method 0 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69567fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient(hunting): handle_auth_done global_id 194298 payload 293 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 1 -- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69400099c0 msgr2=0x7f694000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69400099c0 0x7f694000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 1 -- 172.21.15.116:0/3627111704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6940004780 msgr2=0x7f6940004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6940004780 0x7f6940004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 1 monclient: found mon.a 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 1 -- 172.21.15.116:0/3627111704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400034e0 con 0x7f6940003bc0 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 10 monclient: _finish_auth 0 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f6956ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f69400099c0 0x7f694000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.351+0000 7f69577fe640 1 -- 172.21.15.116:0/3627111704 reap_dead start 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6956ffd640 1 -- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6940003bc0 msgr2=0x7f6940003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6940003bc0 0x7f6940003fd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6948002230 tx=0x7f6948001890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd64c13bce056566 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948005b00 con 0x7f6940003bc0 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 10 monclient: dump: 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694802d070 con 0x7f6940003bc0 2024-09-18T12:42:35.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 1 -- 172.21.15.116:0/3627111704 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948005cc0 con 0x7f6940003bc0 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 4 set_mon_vals no callback set 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f6955ffb640 10 monclient: dump: 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 10 monclient: shutdown 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 1 -- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6940003bc0 msgr2=0x7f6940003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6940003bc0 0x7f6940003fd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6948002230 tx=0x7f6948001890 comp rx=0 tx=0).stop 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 1 -- 172.21.15.116:0/3627111704 shutdown_connections 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 1 --2- 172.21.15.116:0/3627111704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6940003bc0 0x7f6940003fd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.354+0000 7f695d99d640 1 -- 172.21.15.116:0/3627111704 >> 172.21.15.116:0/3627111704 conn(0x7f6958067110 msgr2=0x7f6958067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.355+0000 7f695d99d640 1 -- 172.21.15.116:0/3627111704 shutdown_connections 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.355+0000 7f695d99d640 1 -- 172.21.15.116:0/3627111704 wait complete. 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.355+0000 7f695d99d640 10 monclient: build_initial_monmap 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.355+0000 7f695d99d640 10 monclient: monmap: 2024-09-18T12:42:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.355+0000 7f695d99d640 1 Processor -- start 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 1 -- start start 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient: init 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 0x7f6958141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f6940003bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695811cb20 0x7f6958141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): picked mon.noname-b con 0x7f695811cb20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69581342c0 0x7f6958134690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f69581342c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695809bf10 con 0x7f69581342c0 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695809dc30 con 0x7f695811cb20 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695811be10 con 0x7f6940003bc0 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f695d99d640 10 monclient(hunting): authenticate will time out at 1782.678491s 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 0x7f6958141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 0x7f6958141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:34092/0 (socket says 172.21.15.116:34092) 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69581342c0 0x7f6958134690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 1 -- 172.21.15.116:0/3063680591 learned_addr learned my addr 172.21.15.116:0/3063680591 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 10 monclient(hunting): get_auth_request con 0x7f6940003bc0 auth_method 0 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): get_auth_request con 0x7f69581342c0 auth_method 0 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 1 -- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 msgr2=0x7f6958141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 1 -- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 msgr2=0x7f6958141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 0x7f6958141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695811cb20 0x7f6958141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 0x7f6958141430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 10 monclient(hunting): get_auth_request con 0x7f695811cb20 auth_method 0 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient(hunting): handle_auth_done global_id 194301 payload 824 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 1 -- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 msgr2=0x7f6958141430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 0x7f6958141430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 1 -- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695811cb20 msgr2=0x7f6958141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695811cb20 0x7f6958141970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient: _finish_hunting 0 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 1 monclient: found mon.noname-a 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.356+0000 7f69577fe640 1 -- 172.21.15.116:0/3063680591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6958136360 con 0x7f69581342c0 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f69577fe640 1 -- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69581342c0 msgr2=0x7f6958134690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f69577fe640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69581342c0 0x7f6958134690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f695809c980 tx=0x7f6950008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 1 -- 172.21.15.116:0/3063680591 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950011e20 con 0x7f69581342c0 2024-09-18T12:42:35.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: dump: 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: _finish_auth 0 2024-09-18T12:42:35.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 1 -- 172.21.15.116:0/3063680591 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6950015460 con 0x7f69581342c0 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f695d99d640 5 monclient: authenticate success, global_id 194301 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f695d99d640 10 monclient: _renew_subs 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f695d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6958135e50 con 0x7f69581342c0 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 1 -- 172.21.15.116:0/3063680591 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950015820 con 0x7f69581342c0 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f693effd640 10 monclient: dump: 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:35.382 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f695d99d640 10 monclient: _renew_subs 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f695d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6958142050 con 0x7f69581342c0 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.359+0000 7f69557fa640 4 set_mon_vals no callback set 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f695d99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f695d99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f695d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f693effd640 1 -- 172.21.15.116:0/3063680591 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6950015c90 con 0x7f69581342c0 2024-09-18T12:42:35.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695804bf60 con 0x7f69581342c0 2024-09-18T12:42:35.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f693effd640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f69440384e0 0x7f694403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:35.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f693effd640 1 -- 172.21.15.116:0/3063680591 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f695004bc20 con 0x7f69581342c0 2024-09-18T12:42:35.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f69440384e0 0x7f694403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:35.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.360+0000 7f6956ffd640 10 monclient: get_auth_request con 0x7f69440384e0 auth_method 0 2024-09-18T12:42:35.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.363+0000 7f6956ffd640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f69440384e0 0x7f694403a990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6948001c80 tx=0x7f6948033000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:35.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.364+0000 7f693effd640 1 -- 172.21.15.116:0/3063680591 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f695001b2c0 con 0x7f69581342c0 2024-09-18T12:42:35.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.364+0000 7f693effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:35.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.364+0000 7f693effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:35.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.577+0000 7f695d99d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.577+0000 7f695d99d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.577+0000 7f695d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.577+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f69581316d0 con 0x7f69581342c0 2024-09-18T12:42:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.578+0000 7f693effd640 1 -- 172.21.15.116:0/3063680591 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f6950019020 con 0x7f69581342c0 2024-09-18T12:42:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.578+0000 7f693effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.578+0000 7f693effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:35.580 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":166,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f69440384e0 msgr2=0x7f694403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f69440384e0 0x7f694403a990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6948001c80 tx=0x7f6948033000 comp rx=0 tx=0).stop 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 10 monclient: shutdown 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69581342c0 msgr2=0x7f6958134690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69581342c0 0x7f6958134690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f695809c980 tx=0x7f6950008040 comp rx=0 tx=0).stop 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 shutdown_connections 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f69440384e0 0x7f694403a990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69581342c0 0x7f6958134690 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f695811cb20 0x7f6958141970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 --2- 172.21.15.116:0/3063680591 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6940003bc0 0x7f6958141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 >> 172.21.15.116:0/3063680591 conn(0x7f6958067110 msgr2=0x7f69580a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 shutdown_connections 2024-09-18T12:42:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:35.580+0000 7f695d99d640 1 -- 172.21.15.116:0/3063680591 wait complete. 2024-09-18T12:42:38.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.696+0000 7f5e61cf8640 10 monclient: get_monmap_and_config 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.696+0000 7f5e61cf8640 10 monclient: build_initial_monmap 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.696+0000 7f5e61cf8640 10 monclient: monmap: 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:38.698017+0000 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:38.698017+0000 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:38.699 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 1 Processor -- start 2024-09-18T12:42:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 1 -- start start 2024-09-18T12:42:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient: init 2024-09-18T12:42:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e5c09c860 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f5e5c09e840 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient(hunting): picked mon.noname-c con 0x7f5e5c136cb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c09ec40 con 0x7f5e5c09c860 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c06a320 con 0x7f5e5c09e840 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c068a20 con 0x7f5e5c136cb0 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e61cf8640 10 monclient(hunting): authenticate will time out at 1786.019656s 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41902/0 (socket says 172.21.15.116:41902) 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 learned_addr learned my addr 172.21.15.116:0/3237870451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e5c09e840 auth_method 0 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request con 0x7f5e5c136cb0 auth_method 0 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.697+0000 7f5e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.698+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.698+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.698+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.698+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.700+0000 7f5e5b7fe640 1 -- 172.21.15.116:0/3237870451 >> v1:172.21.15.144:6789/0 conn(0x7f5e5c09c860 legacy=0x7f5e5c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e5c09e840 auth_method 0 2024-09-18T12:42:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request con 0x7f5e5c136cb0 auth_method 0 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:38.899+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:42:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e5c09e840 auth_method 0 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request con 0x7f5e5c136cb0 auth_method 0 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:39.300+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e5c09e840 auth_method 0 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request con 0x7f5e5c136cb0 auth_method 0 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:40.101+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:42:41.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 10 monclient(hunting): tick 2024-09-18T12:42:41.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 monclient(hunting): continuing hunt 2024-09-18T12:42:41.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 msgr2=0x7f5e5c139090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.144:3300/0 conn(0x7f5e5c136cb0 0x7f5e5c139090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 -- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 msgr2=0x7f5e5c09aba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 --2- 172.21.15.116:0/3237870451 >> v2:172.21.15.116:3300/0 conn(0x7f5e5c09e840 0x7f5e5c09aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 -- 172.21.15.116:0/3237870451 >> v1:172.21.15.144:6789/0 conn(0x7f5e5c09c860 legacy=0x7f5e5c09cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 10 monclient: _add_conns ranks=[0,2] 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e480036d0 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 10 monclient(hunting): picked mon.noname-b con 0x7f5e480073b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 -- 172.21.15.116:0/3237870451 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c068e60 con 0x7f5e480036d0 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 1 -- 172.21.15.116:0/3237870451 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06a320 con 0x7f5e480073b0 2024-09-18T12:42:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.698+0000 7f5e597fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e480073b0 auth_method 0 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 msgr2=0x7f5e48009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 msgr2=0x7f5e48009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:41.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.699+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:41.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.701+0000 7f5e5b7fe640 1 -- 172.21.15.116:0/3237870451 >> v1:172.21.15.116:6789/0 conn(0x7f5e480036d0 legacy=0x7f5e48005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e480073b0 auth_method 0 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 msgr2=0x7f5e48009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 msgr2=0x7f5e48009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:41.900+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:42:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e480073b0 auth_method 0 2024-09-18T12:42:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 msgr2=0x7f5e48009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:42.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 msgr2=0x7f5e48009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:42.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:42.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.301+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:42:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.594+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290427390 0 0) 0x7f5e5c068e60 con 0x7f5e480036d0 2024-09-18T12:42:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.594+0000 7f5e5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.594+0000 7f5e5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:42.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.594+0000 7f5e5a7fc640 10 monclient(hunting): my global_id is 194304 2024-09-18T12:42:42.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.594+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e440034e0 con 0x7f5e480036d0 2024-09-18T12:42:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4025453573 0 0) 0x7f5e440034e0 con 0x7f5e480036d0 2024-09-18T12:42:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 msgr2=0x7f5e48009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e480073b0 0x7f5e48009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:42:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:42:42.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4800b020 con 0x7f5e480036d0 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f5e4c001830 con 0x7f5e480036d0 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:42:42.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: dump: 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:42.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 >> v1:172.21.15.116:6789/0 conn(0x7f5e480036d0 legacy=0x7f5e48005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5bfff640 1 -- 172.21.15.116:0/3237870451 reap_dead start 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e44003a70 0x7f5e44003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient(hunting): picked mon.a con 0x7f5e44003a70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e44004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient(hunting): picked mon.c con 0x7f5e44004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e440052e0 0x7f5e4400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient(hunting): picked mon.b con 0x7f5e440052e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:42.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06a320 con 0x7f5e44003a70 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4400c2b0 con 0x7f5e440052e0 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4400c470 con 0x7f5e44004780 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.595+0000 7f5e5a7fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e44003a70 0x7f5e44003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e44003a70 auth_method 0 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e44004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e44004780 auth_method 0 2024-09-18T12:42:42.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e440052e0 0x7f5e4400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request con 0x7f5e440052e0 auth_method 0 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:42.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient(hunting): handle_auth_done global_id 194304 payload 293 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 msgr2=0x7f5e44004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e44004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e440052e0 msgr2=0x7f5e4400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e440052e0 0x7f5e4400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 1 monclient: found mon.a 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:42.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:42.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 1 -- 172.21.15.116:0/3237870451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440034e0 con 0x7f5e44003a70 2024-09-18T12:42:42.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:42:42.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:42.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e61cf8640 5 monclient: authenticate success, global_id 194304 2024-09-18T12:42:42.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.596+0000 7f5e61cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:42.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5b7fe640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e44003a70 msgr2=0x7f5e44003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:42.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5b7fe640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e44003a70 0x7f5e44003e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c001a10 tx=0x7f5e4c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4137d7f9edaa641d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:42.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c02d070 con 0x7f5e44003a70 2024-09-18T12:42:42.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 10 monclient: dump: 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:42.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c0070f0 con 0x7f5e44003a70 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e61cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 1 -- 172.21.15.116:0/3237870451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c0074b0 con 0x7f5e44003a70 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e61cf8640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e61cf8640 4 set_mon_vals no callback set 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.599+0000 7f5e5a7fc640 10 monclient: dump: 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 10 monclient: shutdown 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 -- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e44003a70 msgr2=0x7f5e44003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e44003a70 0x7f5e44003e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c001a10 tx=0x7f5e4c007a00 comp rx=0 tx=0).stop 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 -- 172.21.15.116:0/3237870451 shutdown_connections 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e440052e0 0x7f5e4400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e44004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/3237870451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e44003a70 0x7f5e44003e60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 -- 172.21.15.116:0/3237870451 >> 172.21.15.116:0/3237870451 conn(0x7f5e5c0a2e40 msgr2=0x7f5e5c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 -- 172.21.15.116:0/3237870451 shutdown_connections 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.600+0000 7f5e61cf8640 1 -- 172.21.15.116:0/3237870451 wait complete. 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.601+0000 7f5e61cf8640 10 monclient: build_initial_monmap 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.601+0000 7f5e61cf8640 10 monclient: monmap: 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 Processor -- start 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 -- start start 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient: init 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e44003a70 0x7f5e5c099100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): picked mon.noname-b con 0x7f5e44003a70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e5c06b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): picked mon.noname-c con 0x7f5e44004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e440052e0 0x7f5e5c06bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f5e440052e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06d280 con 0x7f5e440052e0 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0a21f0 con 0x7f5e44003a70 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06e970 con 0x7f5e44004780 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e61cf8640 10 monclient(hunting): authenticate will time out at 1789.923780s 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e5c06b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e44003a70 0x7f5e5c099100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e5c06b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:40686/0 (socket says 172.21.15.116:40686) 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5affd640 1 -- 172.21.15.116:0/566822706 learned_addr learned my addr 172.21.15.116:0/566822706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e440052e0 0x7f5e5c06bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5affd640 10 monclient(hunting): get_auth_request con 0x7f5e44004780 auth_method 0 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e44003a70 auth_method 0 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:42.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request con 0x7f5e440052e0 auth_method 0 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.602+0000 7f5e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5affd640 1 -- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 msgr2=0x7f5e5c06b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5affd640 1 -- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 msgr2=0x7f5e5c06b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5affd640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e5c06b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:42.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5affd640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e5c06b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 10 monclient(hunting): handle_auth_done global_id 194307 payload 824 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 1 -- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 msgr2=0x7f5e5c06b8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e5c06b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 1 -- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e44003a70 msgr2=0x7f5e5c099100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e44003a70 0x7f5e5c099100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 1 monclient: found mon.noname-a 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5bfff640 1 -- 172.21.15.116:0/566822706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c06cd90 con 0x7f5e440052e0 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.603+0000 7f5e5b7fe640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e44003a70 0x7f5e5c099100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e5bfff640 1 -- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e440052e0 msgr2=0x7f5e5c06bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:42.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e5bfff640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e440052e0 0x7f5e5c06bdf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c09d3c0 tx=0x7f5e5400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 1 -- 172.21.15.116:0/566822706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54018910 con 0x7f5e440052e0 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: dump: 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:42.623 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: _finish_auth 0 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 1 -- 172.21.15.116:0/566822706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5400f040 con 0x7f5e440052e0 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e61cf8640 5 monclient: authenticate success, global_id 194307 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e61cf8640 10 monclient: _renew_subs 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e61cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c099cc0 con 0x7f5e440052e0 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 1 -- 172.21.15.116:0/566822706 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54018cc0 con 0x7f5e440052e0 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:42.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e437fe640 10 monclient: dump: 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e61cf8640 10 monclient: _renew_subs 2024-09-18T12:42:42.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e61cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c06e740 con 0x7f5e440052e0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.605+0000 7f5e59ffb640 4 set_mon_vals no callback set 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.606+0000 7f5e437fe640 1 -- 172.21.15.116:0/566822706 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5e54003040 con 0x7f5e440052e0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.606+0000 7f5e61cf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.607+0000 7f5e61cf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.607+0000 7f5e61cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.607+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c04bf60 con 0x7f5e440052e0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.607+0000 7f5e437fe640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e48039020 0x7f5e4803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.607+0000 7f5e437fe640 1 -- 172.21.15.116:0/566822706 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5e5404d1d0 con 0x7f5e440052e0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.607+0000 7f5e5b7fe640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e48039020 0x7f5e4803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.607+0000 7f5e5b7fe640 10 monclient: get_auth_request con 0x7f5e48039020 auth_method 0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.610+0000 7f5e5b7fe640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e48039020 0x7f5e4803b4d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c001eb0 tx=0x7f5e4c034000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.611+0000 7f5e437fe640 1 -- 172.21.15.116:0/566822706 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f5e54028560 con 0x7f5e440052e0 2024-09-18T12:42:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.611+0000 7f5e437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:42.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.611+0000 7f5e437fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:42.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.829+0000 7f5e61cf8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:42.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.829+0000 7f5e61cf8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:42.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.829+0000 7f5e61cf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:42.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.829+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e5c0638d0 con 0x7f5e440052e0 2024-09-18T12:42:42.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.829+0000 7f5e437fe640 1 -- 172.21.15.116:0/566822706 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f5e540230c0 con 0x7f5e440052e0 2024-09-18T12:42:42.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.829+0000 7f5e437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:42.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.829+0000 7f5e437fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:42.831 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":170,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.831+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e48039020 msgr2=0x7f5e4803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.831+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e48039020 0x7f5e4803b4d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c001eb0 tx=0x7f5e4c034000 comp rx=0 tx=0).stop 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.831+0000 7f5e61cf8640 10 monclient: shutdown 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.831+0000 7f5e61cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.831+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e440052e0 msgr2=0x7f5e5c06bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.831+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e440052e0 0x7f5e5c06bdf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c09d3c0 tx=0x7f5e5400b040 comp rx=0 tx=0).stop 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 shutdown_connections 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5e48039020 0x7f5e4803b4d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e440052e0 0x7f5e5c06bdf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5e44004780 0x7f5e5c06b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 --2- 172.21.15.116:0/566822706 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e44003a70 0x7f5e5c099100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:42.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 >> 172.21.15.116:0/566822706 conn(0x7f5e5c0a2e40 msgr2=0x7f5e5c0a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:42.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 shutdown_connections 2024-09-18T12:42:42.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:42.832+0000 7f5e61cf8640 1 -- 172.21.15.116:0/566822706 wait complete. 2024-09-18T12:42:45.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.944+0000 7f65754e8640 10 monclient: get_monmap_and_config 2024-09-18T12:42:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.944+0000 7f65754e8640 10 monclient: build_initial_monmap 2024-09-18T12:42:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.944+0000 7f65754e8640 10 monclient: monmap: 2024-09-18T12:42:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:45.945346+0000 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:45.945346+0000 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:45.946 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 Processor -- start 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 -- start start 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient: init 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6570131980 0x7f6570131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): picked mon.noname-c con 0x7f6570131980 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f657009fe10 0x7f657009ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): picked mon.noname-a con 0x7f657009fe10 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 0x7f6570068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:45.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): picked mon.noname-b con 0x7f657009ef70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65701325b0 con 0x7f657009fe10 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65700a11f0 con 0x7f6570131980 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6570132290 con 0x7f657009ef70 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.945+0000 7f65754e8640 10 monclient(hunting): authenticate will time out at 1793.266918s 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 0x7f6570068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f657009fe10 0x7f657009ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 0x7f6570068ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41284/0 (socket says 172.21.15.116:41284) 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f657009fe10 0x7f657009ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54478/0 (socket says 172.21.15.116:54478) 2024-09-18T12:42:45.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 1 -- 172.21.15.116:0/1803151372 learned_addr learned my addr 172.21.15.116:0/1803151372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 10 monclient(hunting): get_auth_request con 0x7f657009ef70 auth_method 0 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): get_auth_request con 0x7f657009fe10 auth_method 0 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 1 -- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 msgr2=0x7f6570068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 1 --2- 172.21.15.116:0/1803151372 >> v2:172.21.15.144:3300/0 conn(0x7f6570131980 0x7f6570131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 1 -- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 msgr2=0x7f6570068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:45.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 1 --2- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 0x7f6570068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 10 monclient(hunting): get_auth_request con 0x7f6570131980 auth_method 0 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656f7fe640 1 --2- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 0x7f6570068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f656effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient(hunting): handle_auth_done global_id 194310 payload 274 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 -- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 msgr2=0x7f6570068ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:45.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 --2- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 0x7f6570068ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 -- 172.21.15.116:0/1803151372 >> v2:172.21.15.144:3300/0 conn(0x7f6570131980 msgr2=0x7f6570131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 --2- 172.21.15.116:0/1803151372 >> v2:172.21.15.144:3300/0 conn(0x7f6570131980 0x7f6570131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient: _finish_hunting 0 2024-09-18T12:42:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 monclient: found mon.noname-a 2024-09-18T12:42:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:45.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.946+0000 7f6567fff640 1 -- 172.21.15.116:0/1803151372 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657009f390 con 0x7f657009fe10 2024-09-18T12:42:45.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f6567fff640 1 -- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009fe10 msgr2=0x7f657009ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:45.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f6567fff640 1 --2- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009fe10 0x7f657009ea30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f657006f930 tx=0x7f65580090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b571d8e1b65f93 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 1 -- 172.21.15.116:0/1803151372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655802adf0 con 0x7f657009fe10 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: dump: 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:45.960 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 1 -- 172.21.15.116:0/1803151372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6558032440 con 0x7f657009fe10 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 1 -- 172.21.15.116:0/1803151372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65580327e0 con 0x7f657009fe10 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f65754e8640 5 monclient: authenticate success, global_id 194310 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f65754e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f65754e8640 4 set_mon_vals no callback set 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f656e7fc640 10 monclient: dump: 2024-09-18T12:42:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f65754e8640 10 monclient: shutdown 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f65754e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f65754e8640 1 -- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009fe10 msgr2=0x7f657009ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.949+0000 7f65754e8640 1 --2- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009fe10 0x7f657009ea30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f657006f930 tx=0x7f65580090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 -- 172.21.15.116:0/1803151372 shutdown_connections 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 --2- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009ef70 0x7f6570068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 --2- 172.21.15.116:0/1803151372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009fe10 0x7f657009ea30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 --2- 172.21.15.116:0/1803151372 >> v2:172.21.15.144:3300/0 conn(0x7f6570131980 0x7f6570131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 -- 172.21.15.116:0/1803151372 >> 172.21.15.116:0/1803151372 conn(0x7f65700a5050 msgr2=0x7f65700a5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 -- 172.21.15.116:0/1803151372 shutdown_connections 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 -- 172.21.15.116:0/1803151372 wait complete. 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 10 monclient: build_initial_monmap 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 10 monclient: monmap: 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 Processor -- start 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 1 -- start start 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.950+0000 7f65754e8640 10 monclient: init 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 0x7f657011bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): picked mon.noname-a con 0x7f657009ef70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 0x7f6570130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): picked mon.noname-c con 0x7f657009fe10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6570131980 0x7f65701410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): picked mon.noname-b con 0x7f6570131980 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657011e210 con 0x7f657009ef70 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657009b610 con 0x7f6570131980 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657011c0c0 con 0x7f657009fe10 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f65754e8640 10 monclient(hunting): authenticate will time out at 1793.272366s 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 0x7f657011bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 0x7f6570130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 0x7f657011bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54494/0 (socket says 172.21.15.116:54494) 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 -- 172.21.15.116:0/3973753133 learned_addr learned my addr 172.21.15.116:0/3973753133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): get_auth_request con 0x7f657009ef70 auth_method 0 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 10 monclient(hunting): get_auth_request con 0x7f657009fe10 auth_method 0 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 1 -- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 msgr2=0x7f6570130b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 1 -- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 msgr2=0x7f6570130b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 0x7f6570130b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f6567fff640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 0x7f6570130b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6570131980 0x7f65701410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 10 monclient(hunting): get_auth_request con 0x7f6570131980 auth_method 0 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient(hunting): handle_auth_done global_id 194313 payload 824 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 -- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 msgr2=0x7f6570130b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 0x7f6570130b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 -- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6570131980 msgr2=0x7f65701410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6570131980 0x7f65701410a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient: _finish_hunting 0 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 monclient: found mon.noname-a 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.951+0000 7f656effd640 1 -- 172.21.15.116:0/3973753133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6558009390 con 0x7f657009ef70 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.953+0000 7f656effd640 1 -- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 msgr2=0x7f657011bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.953+0000 7f656effd640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 0x7f657011bb80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f656000ea50 tx=0x7f65600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 1 -- 172.21.15.116:0/3973753133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560009600 con 0x7f657009ef70 2024-09-18T12:42:45.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: dump: 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:45.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: _finish_auth 0 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 1 -- 172.21.15.116:0/3973753133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656000cc20 con 0x7f657009ef70 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f65754e8640 5 monclient: authenticate success, global_id 194313 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f65754e8640 10 monclient: _renew_subs 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 1 -- 172.21.15.116:0/3973753133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560013470 con 0x7f657009ef70 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f65754e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657011df30 con 0x7f657009ef70 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f656dffb640 4 set_mon_vals no callback set 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f6566ffd640 10 monclient: dump: 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:45.970 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f65754e8640 10 monclient: _renew_subs 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f65754e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.954+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657009abe0 con 0x7f657009ef70 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f6566ffd640 1 -- 172.21.15.116:0/3973753133 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6560013860 con 0x7f657009ef70 2024-09-18T12:42:45.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f65754e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f65754e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f65754e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657004bf60 con 0x7f657009ef70 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f6566ffd640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6550038490 0x7f655003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f6566ffd640 1 -- 172.21.15.116:0/3973753133 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f656004c630 con 0x7f657009ef70 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f6567fff640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6550038490 0x7f655003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.955+0000 7f6567fff640 10 monclient: get_auth_request con 0x7f6550038490 auth_method 0 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.958+0000 7f6567fff640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6550038490 0x7f655003a940 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f657009c9c0 tx=0x7f6558032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:45.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.959+0000 7f6566ffd640 1 -- 172.21.15.116:0/3973753133 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f65600236b0 con 0x7f657009ef70 2024-09-18T12:42:45.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.959+0000 7f6566ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:45.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:45.959+0000 7f6566ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.173+0000 7f65754e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.173+0000 7f65754e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.173+0000 7f65754e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.173+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f657011d8c0 con 0x7f657009ef70 2024-09-18T12:42:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.173+0000 7f6566ffd640 1 -- 172.21.15.116:0/3973753133 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f65600232f0 con 0x7f657009ef70 2024-09-18T12:42:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.173+0000 7f6566ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.173+0000 7f6566ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:46.175 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":172,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6550038490 msgr2=0x7f655003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6550038490 0x7f655003a940 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f657009c9c0 tx=0x7f6558032000 comp rx=0 tx=0).stop 2024-09-18T12:42:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 10 monclient: shutdown 2024-09-18T12:42:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 msgr2=0x7f657011bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 0x7f657011bb80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f656000ea50 tx=0x7f65600090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 shutdown_connections 2024-09-18T12:42:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6550038490 0x7f655003a940 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6570131980 0x7f65701410a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f657009fe10 0x7f6570130b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 --2- 172.21.15.116:0/3973753133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657009ef70 0x7f657011bb80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 >> 172.21.15.116:0/3973753133 conn(0x7f65700a5050 msgr2=0x7f657009e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:46.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 shutdown_connections 2024-09-18T12:42:46.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.175+0000 7f65754e8640 1 -- 172.21.15.116:0/3973753133 wait complete. 2024-09-18T12:42:46.186 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:42:46.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:42:46.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.279+0000 7f80ec99d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.279+0000 7f80ec99d640 10 monclient: build_initial_monmap 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.279+0000 7f80ec99d640 10 monclient: monmap: 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:46.279684+0000 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:46.279684+0000 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.279+0000 7f80ec99d640 1 Processor -- start 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.279+0000 7f80ec99d640 1 -- start start 2024-09-18T12:42:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.279+0000 7f80ec99d640 10 monclient: init 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 0x7f80e8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient(hunting): picked mon.noname-b con 0x7f80e81318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f80e809ea50 0x7f80e809ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient(hunting): picked mon.noname-c con 0x7f80e809ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80e8068d90 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e8131cc0 con 0x7f80e8068d90 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e809fbf0 con 0x7f80e809ea50 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e809fe30 con 0x7f80e81318f0 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80ec99d640 10 monclient(hunting): authenticate will time out at 1793.601360s 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 0x7f80e8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 0x7f80e8131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41308/0 (socket says 172.21.15.116:41308) 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 1 -- 172.21.15.116:0/1599393434 learned_addr learned my addr 172.21.15.116:0/1599393434 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 10 monclient(hunting): get_auth_request con 0x7f80e81318f0 auth_method 0 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 1 -- 172.21.15.116:0/1599393434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 msgr2=0x7f80e8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 1 -- 172.21.15.116:0/1599393434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 msgr2=0x7f80e8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 1 --2- 172.21.15.116:0/1599393434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 0x7f80e8131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 1 --2- 172.21.15.116:0/1599393434 >> v2:172.21.15.144:3300/0 conn(0x7f80e809ea50 0x7f80e809ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e659c640 1 --2- 172.21.15.116:0/1599393434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 0x7f80e8131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 10 monclient(hunting): get_auth_request con 0x7f80e809ea50 auth_method 0 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.280+0000 7f80e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.282+0000 7f80e6d9d640 1 -- 172.21.15.116:0/1599393434 >> v1:172.21.15.144:6789/0 conn(0x7f80e8068d90 legacy=0x7f80e80691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 10 monclient(hunting): handle_auth_done global_id 174478 payload 274 2024-09-18T12:42:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 1 -- 172.21.15.116:0/1599393434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 msgr2=0x7f80e8131f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 1 --2- 172.21.15.116:0/1599393434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 0x7f80e8131f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 1 -- 172.21.15.116:0/1599393434 >> v1:172.21.15.144:6789/0 conn(0x7f80e8068d90 legacy=0x7f80e80691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 1 monclient: found mon.noname-c 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.283+0000 7f80e5d9b640 1 -- 172.21.15.116:0/1599393434 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e80a1210 con 0x7f80e809ea50 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e5d9b640 1 -- 172.21.15.116:0/1599393434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e809ea50 msgr2=0x7f80e809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e5d9b640 1 --2- 172.21.15.116:0/1599393434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e809ea50 0x7f80e809ee20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80dc0098d0 tx=0x7f80dc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9096502fb67ddc15 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 1 -- 172.21.15.116:0/1599393434 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc032440 con 0x7f80e809ea50 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: dump: 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: _finish_auth 0 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80ec99d640 5 monclient: authenticate success, global_id 174478 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80ec99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 1 -- 172.21.15.116:0/1599393434 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc0325a0 con 0x7f80e809ea50 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80ec99d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80ec99d640 4 set_mon_vals no callback set 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 1 -- 172.21.15.116:0/1599393434 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc032940 con 0x7f80e809ea50 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:46.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.286+0000 7f80e559a640 10 monclient: dump: 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 10 monclient: shutdown 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:46.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 -- 172.21.15.116:0/1599393434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e809ea50 msgr2=0x7f80e809ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 --2- 172.21.15.116:0/1599393434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e809ea50 0x7f80e809ee20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80dc0098d0 tx=0x7f80dc0090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 -- 172.21.15.116:0/1599393434 shutdown_connections 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 --2- 172.21.15.116:0/1599393434 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e809ea50 0x7f80e809ee20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 --2- 172.21.15.116:0/1599393434 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e81318f0 0x7f80e8131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 -- 172.21.15.116:0/1599393434 >> 172.21.15.116:0/1599393434 conn(0x7f80e8067110 msgr2=0x7f80e8067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 -- 172.21.15.116:0/1599393434 shutdown_connections 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 1 -- 172.21.15.116:0/1599393434 wait complete. 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 10 monclient: build_initial_monmap 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.287+0000 7f80ec99d640 10 monclient: monmap: 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:46.293 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 Processor -- start 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 -- start start 2024-09-18T12:42:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient: init 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e8068d90 0x7f80e809a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): picked mon.noname-b con 0x7f80e8068d90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 0x7f80e8130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): picked mon.noname-c con 0x7f80e809ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e81318f0 0x7f80e8140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): picked mon.noname-a con 0x7f80e81318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e809b990 con 0x7f80e81318f0 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e811dbc0 con 0x7f80e8068d90 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e811dea0 con 0x7f80e809ea50 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80ec99d640 10 monclient(hunting): authenticate will time out at 1793.609663s 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 0x7f80e8130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e81318f0 0x7f80e8140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 0x7f80e8130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41324/0 (socket says 172.21.15.116:41324) 2024-09-18T12:42:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 1 -- 172.21.15.116:0/410859632 learned_addr learned my addr 172.21.15.116:0/410859632 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 10 monclient(hunting): get_auth_request con 0x7f80e809ea50 auth_method 0 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 10 monclient(hunting): get_auth_request con 0x7f80e81318f0 auth_method 0 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 msgr2=0x7f80e8130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 msgr2=0x7f80e8130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 0x7f80e8130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e8068d90 0x7f80e809a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 10 monclient(hunting): get_auth_request con 0x7f80e8068d90 auth_method 0 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e5d9b640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 0x7f80e8130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.288+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 10 monclient(hunting): handle_auth_done global_id 194316 payload 824 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 msgr2=0x7f80e8130aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80e809ea50 0x7f80e8130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e8068d90 msgr2=0x7f80e809a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80e8068d90 0x7f80e809a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 1 monclient: found mon.noname-a 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.289+0000 7f80e6d9d640 1 -- 172.21.15.116:0/410859632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc009390 con 0x7f80e81318f0 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80e6d9d640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e81318f0 msgr2=0x7f80e8140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80e6d9d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e81318f0 0x7f80e8140e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f80e000c810 tx=0x7f80e0001830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e000ce00 con 0x7f80e81318f0 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: dump: 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: _finish_auth 0 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e000a640 con 0x7f80e81318f0 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80ec99d640 5 monclient: authenticate success, global_id 194316 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80ec99d640 10 monclient: _renew_subs 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80ec99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e811c800 con 0x7f80e81318f0 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e000aa00 con 0x7f80e81318f0 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80d67fc640 10 monclient: dump: 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80e4d99640 4 set_mon_vals no callback set 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80ec99d640 10 monclient: _renew_subs 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80ec99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.291+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e811ce70 con 0x7f80e81318f0 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80ec99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80ec99d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 0x7f80e81413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80ec99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8065e00 con 0x7f80e811bc00 2024-09-18T12:42:46.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f80e00070f0 con 0x7f80e81318f0 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 0x7f80e81413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient: get_auth_request con 0x7f80e811bc00 auth_method 0 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80d67fc640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c80384c0 0x7f80c803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f80e004b8f0 con 0x7f80e81318f0 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 10 monclient(hunting): handle_auth_done global_id 194319 payload 824 2024-09-18T12:42:46.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e659c640 1 -- 172.21.15.116:0/410859632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80e811dd30 con 0x7f80e811bc00 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e5d9b640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c80384c0 0x7f80c803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.292+0000 7f80e5d9b640 10 monclient: get_auth_request con 0x7f80c80384c0 auth_method 0 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.294+0000 7f80e659c640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 msgr2=0x7f80e81413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.294+0000 7f80e659c640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 0x7f80e81413b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80d8006fd0 tx=0x7f80d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.294+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d800c070 con 0x7f80e811bc00 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.294+0000 7f80d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.294+0000 7f80d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.294+0000 7f80d67fc640 10 monclient: dump: 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.309 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f80d8011e50 con 0x7f80e811bc00 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80d67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80d67fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 msgr2=0x7f80e81413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80d67fc640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 0x7f80e81413b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80d8006fd0 tx=0x7f80d8008040 comp rx=0 tx=0).stop 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 msgr2=0x7f80e81413b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80d67fc640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e811bc00 0x7f80e81413b0 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80d8006fd0 tx=0x7f80d8008040 comp rx=0 tx=0).stop 2024-09-18T12:42:46.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.295+0000 7f80e5d9b640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c80384c0 0x7f80c803a970 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009130 tx=0x7f80dc032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80ec99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80ec99d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 0x7f80e81418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80ec99d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e804bf60 con 0x7f80e8130fe0 2024-09-18T12:42:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80e659c640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 0x7f80e81418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80e659c640 10 monclient: get_auth_request con 0x7f80e8130fe0 auth_method 0 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80e659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.300+0000 7f80e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.301+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.301+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.301+0000 7f80e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.301+0000 7f80e659c640 10 monclient(hunting): handle_auth_done global_id 194322 payload 824 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.301+0000 7f80e659c640 1 -- 172.21.15.116:0/410859632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f80e80638d0 con 0x7f80e8130fe0 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80e659c640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 msgr2=0x7f80e81418f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80e659c640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 0x7f80e81418f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f80d8006fd0 tx=0x7f80d801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8023070 con 0x7f80e8130fe0 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 10 monclient: dump: 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f80d801e740 con 0x7f80e8130fe0 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 msgr2=0x7f80e81418f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 0x7f80e81418f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f80d8006fd0 tx=0x7f80d801e000 comp rx=0 tx=0).stop 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 msgr2=0x7f80e81418f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.303+0000 7f80d67fc640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e8130fe0 0x7f80e81418f0 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f80d8006fd0 tx=0x7f80d801e000 comp rx=0 tx=0).stop 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:46.315 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 172, 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:46.316 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:46.317 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:46.318 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:46.319 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.320 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:46.321 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:46.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:46.323 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:46.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.325 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:46.326 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:46.327 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:46.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:46.329 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:46.329 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c80384c0 msgr2=0x7f80c803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c80384c0 0x7f80c803a970 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009130 tx=0x7f80dc032000 comp rx=0 tx=0).stop 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 10 monclient: shutdown 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e81318f0 msgr2=0x7f80e8140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e81318f0 0x7f80e8140e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f80e000c810 tx=0x7f80e0001830 comp rx=0 tx=0).stop 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80e6d9d640 1 -- 172.21.15.116:0/410859632 reap_dead start 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 shutdown_connections 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 --2- 172.21.15.116:0/410859632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e81318f0 0x7f80e8140e70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 >> 172.21.15.116:0/410859632 conn(0x7f80e8067110 msgr2=0x7f80e8134a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 shutdown_connections 2024-09-18T12:42:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.304+0000 7f80ec99d640 1 -- 172.21.15.116:0/410859632 wait complete. 2024-09-18T12:42:46.331 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:42:46.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:42:46.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:42:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.424+0000 7f524351d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.424+0000 7f524351d640 10 monclient: build_initial_monmap 2024-09-18T12:42:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.424+0000 7f524351d640 10 monclient: monmap: 2024-09-18T12:42:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:46.424701+0000 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:46.424701+0000 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.424+0000 7f524351d640 1 Processor -- start 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.424+0000 7f524351d640 1 -- start start 2024-09-18T12:42:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.424+0000 7f524351d640 10 monclient: init 2024-09-18T12:42:46.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f523c1318f0 0x7f523c131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): picked mon.noname-c con 0x7f523c1318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): picked mon.noname-b con 0x7f523c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f523c068d90 0x7f523c0691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): picked mon.noname-a con 0x7f523c068d90 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c131cc0 con 0x7f523c068d90 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c09fbf0 con 0x7f523c1318f0 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c09fe30 con 0x7f523c09ea50 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f524351d640 10 monclient(hunting): authenticate will time out at 1793.746373s 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f523c068d90 0x7f523c0691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41330/0 (socket says 172.21.15.116:41330) 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 1 -- 172.21.15.116:0/2026676575 learned_addr learned my addr 172.21.15.116:0/2026676575 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:46.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 10 monclient(hunting): get_auth_request con 0x7f523c09ea50 auth_method 0 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): get_auth_request con 0x7f523c068d90 auth_method 0 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 1 -- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 msgr2=0x7f523c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:42:46.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 1 -- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 msgr2=0x7f523c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 1 --2- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c09ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5240ab8640 1 --2- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c09ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f52412b9640 1 --2- 172.21.15.116:0/2026676575 >> v2:172.21.15.144:3300/0 conn(0x7f523c1318f0 0x7f523c131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f52412b9640 10 monclient(hunting): get_auth_request con 0x7f523c1318f0 auth_method 0 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f52412b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f52412b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f52412b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient(hunting): handle_auth_done global_id 194325 payload 274 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 1 -- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 msgr2=0x7f523c09ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 1 --2- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c09ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 1 -- 172.21.15.116:0/2026676575 >> v2:172.21.15.144:3300/0 conn(0x7f523c1318f0 msgr2=0x7f523c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:46.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 1 --2- 172.21.15.116:0/2026676575 >> v2:172.21.15.144:3300/0 conn(0x7f523c1318f0 0x7f523c131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient: _finish_hunting 0 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 1 monclient: found mon.noname-a 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.425+0000 7f5241aba640 1 -- 172.21.15.116:0/2026676575 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0a1210 con 0x7f523c068d90 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5241aba640 1 -- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 msgr2=0x7f523c0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5241aba640 1 --2- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c0691f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f523c076600 tx=0x7f5234009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9588b3e42d94527b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 1 -- 172.21.15.116:0/2026676575 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234007760 con 0x7f523c068d90 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: dump: 2024-09-18T12:42:46.437 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.438 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: _finish_auth 0 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f524351d640 5 monclient: authenticate success, global_id 194325 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 1 -- 172.21.15.116:0/2026676575 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523402d050 con 0x7f523c068d90 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f524351d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f524351d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f524351d640 4 set_mon_vals no callback set 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 1 -- 172.21.15.116:0/2026676575 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234007af0 con 0x7f523c068d90 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.428+0000 7f5233fff640 10 monclient: dump: 2024-09-18T12:42:46.439 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 10 monclient: shutdown 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 -- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 msgr2=0x7f523c0691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 --2- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c0691f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f523c076600 tx=0x7f5234009780 comp rx=0 tx=0).stop 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 -- 172.21.15.116:0/2026676575 shutdown_connections 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 --2- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c0691f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 --2- 172.21.15.116:0/2026676575 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 --2- 172.21.15.116:0/2026676575 >> v2:172.21.15.144:3300/0 conn(0x7f523c1318f0 0x7f523c131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 -- 172.21.15.116:0/2026676575 >> 172.21.15.116:0/2026676575 conn(0x7f523c067110 msgr2=0x7f523c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 -- 172.21.15.116:0/2026676575 shutdown_connections 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 1 -- 172.21.15.116:0/2026676575 wait complete. 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 10 monclient: build_initial_monmap 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.429+0000 7f524351d640 10 monclient: monmap: 2024-09-18T12:42:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 Processor -- start 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 -- start start 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient: init 2024-09-18T12:42:46.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): picked mon.noname-a con 0x7f523c068d90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): picked mon.noname-c con 0x7f523c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f523c1318f0 0x7f523c11b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): picked mon.noname-b con 0x7f523c1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c09e550 con 0x7f523c068d90 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c11e210 con 0x7f523c1318f0 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c09b590 con 0x7f523c09ea50 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f524351d640 10 monclient(hunting): authenticate will time out at 1793.751775s 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c09aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54532/0 (socket says 172.21.15.116:54532) 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 1 -- 172.21.15.116:0/3564768773 learned_addr learned my addr 172.21.15.116:0/3564768773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 10 monclient(hunting): get_auth_request con 0x7f523c068d90 auth_method 0 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 10 monclient(hunting): get_auth_request con 0x7f523c09ea50 auth_method 0 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f52412b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 msgr2=0x7f523c130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 msgr2=0x7f523c130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.430+0000 7f5241aba640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f523c1318f0 0x7f523c11b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5241aba640 10 monclient(hunting): get_auth_request con 0x7f523c1318f0 auth_method 0 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5241aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5241aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5241aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5241aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5241aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f5241aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 10 monclient(hunting): handle_auth_done global_id 194328 payload 824 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 msgr2=0x7f523c130aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c09ea50 0x7f523c130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f523c1318f0 msgr2=0x7f523c11b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f523c1318f0 0x7f523c11b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 10 monclient: _finish_hunting 0 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 1 monclient: found mon.noname-a 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.431+0000 7f52412b9640 1 -- 172.21.15.116:0/3564768773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5234009a20 con 0x7f523c068d90 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52412b9640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 msgr2=0x7f523c09aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52412b9640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c09aa30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f523800d8c0 tx=0x7f5238008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800de50 con 0x7f523c068d90 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: dump: 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: _finish_auth 0 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523800bae0 con 0x7f523c068d90 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f524351d640 5 monclient: authenticate success, global_id 194328 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f524351d640 10 monclient: _renew_subs 2024-09-18T12:42:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f524351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c11dc20 con 0x7f523c068d90 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800be80 con 0x7f523c068d90 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52317fa640 10 monclient: dump: 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f52337fe640 4 set_mon_vals no callback set 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f524351d640 10 monclient: _renew_subs 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f524351d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.433+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c1410a0 con 0x7f523c068d90 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f524351d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f524351d640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c065e00 con 0x7f523c130fe0 2024-09-18T12:42:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5238014640 con 0x7f523c068d90 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 10 monclient: get_auth_request con 0x7f523c130fe0 auth_method 0 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 msgr2=0x7f523c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 msgr2=0x7f523c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f52317fa640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52200384c0 0x7f522003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f523804c980 con 0x7f523c068d90 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5241aba640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52200384c0 0x7f522003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.434+0000 7f5241aba640 10 monclient: get_auth_request con 0x7f52200384c0 auth_method 0 2024-09-18T12:42:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.437+0000 7f5241aba640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52200384c0 0x7f522003a970 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f523c064050 tx=0x7f5234037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 10 monclient: get_auth_request con 0x7f523c130fe0 auth_method 0 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 msgr2=0x7f523c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 msgr2=0x7f523c11bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:42:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:42:46.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:46.635+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:42:47.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient: get_auth_request con 0x7f523c130fe0 auth_method 0 2024-09-18T12:42:47.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:42:47.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 10 monclient(hunting): handle_auth_done global_id 214100 payload 824 2024-09-18T12:42:47.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.036+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f523c11e010 con 0x7f523c130fe0 2024-09-18T12:42:47.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.038+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 msgr2=0x7f523c11bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.038+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f523c066560 tx=0x7f522c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.038+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c012b10 con 0x7f523c130fe0 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.038+0000 7f52317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.038+0000 7f52317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.038+0000 7f52317fa640 10 monclient: dump: 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.040 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.039+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f522c00c040 con 0x7f523c130fe0 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.039+0000 7f52317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.039+0000 7f52317fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.039+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 msgr2=0x7f523c11bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.039+0000 7f52317fa640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f523c066560 tx=0x7f522c008040 comp rx=0 tx=0).stop 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.039+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 msgr2=0x7f523c11bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:47.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.039+0000 7f52317fa640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c130fe0 0x7f523c11bc60 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f523c066560 tx=0x7f522c008040 comp rx=0 tx=0).stop 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f524351d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f524351d640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 0x7f523c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f524351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c141320 con 0x7f523c11c1a0 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 0x7f523c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient: get_auth_request con 0x7f523c11c1a0 auth_method 0 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 10 monclient(hunting): handle_auth_done global_id 214103 payload 824 2024-09-18T12:42:47.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.045+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f523c11c5a0 con 0x7f523c11c1a0 2024-09-18T12:42:47.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f5240ab8640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 msgr2=0x7f523c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f5240ab8640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 0x7f523c096a60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f522c01e000 tx=0x7f522c021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c027070 con 0x7f523c11c1a0 2024-09-18T12:42:47.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 10 monclient: dump: 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f522c02d420 con 0x7f523c11c1a0 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 msgr2=0x7f523c096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 0x7f523c096a60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f522c01e000 tx=0x7f522c021000 comp rx=0 tx=0).stop 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 msgr2=0x7f523c096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.047+0000 7f52317fa640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f523c11c1a0 0x7f523c096a60 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f522c01e000 tx=0x7f522c021000 comp rx=0 tx=0).stop 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:47.050 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 172, 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:47.051 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:47.052 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:47.053 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:47.054 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:47.055 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.057 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:47.059 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:47.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:47.062 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:47.062 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:47.062 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:47.062 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.063 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:47.064 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:47.064 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:47.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.064 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.064 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:47.064 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:47.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52200384c0 msgr2=0x7f522003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f52200384c0 0x7f522003a970 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f523c064050 tx=0x7f5234037000 comp rx=0 tx=0).stop 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 10 monclient: shutdown 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 msgr2=0x7f523c09aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 --2- 172.21.15.116:0/3564768773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f523c068d90 0x7f523c09aa30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f523800d8c0 tx=0x7f5238008040 comp rx=0 tx=0).stop 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f5241aba640 1 -- 172.21.15.116:0/3564768773 reap_dead start 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f5241aba640 1 -- 172.21.15.116:0/3564768773 reap_dead start 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 shutdown_connections 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 >> 172.21.15.116:0/3564768773 conn(0x7f523c067110 msgr2=0x7f523c134790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 shutdown_connections 2024-09-18T12:42:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.048+0000 7f524351d640 1 -- 172.21.15.116:0/3564768773 wait complete. 2024-09-18T12:42:47.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:42:47.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:42:47.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:42:47.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.160+0000 7f36f4c68640 10 monclient: get_monmap_and_config 2024-09-18T12:42:47.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.160+0000 7f36f4c68640 10 monclient: build_initial_monmap 2024-09-18T12:42:47.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.160+0000 7f36f4c68640 10 monclient: monmap: 2024-09-18T12:42:47.161 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:47.161 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.161 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:47.160907+0000 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:47.160907+0000 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.160+0000 7f36f4c68640 1 Processor -- start 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.160+0000 7f36f4c68640 1 -- start start 2024-09-18T12:42:47.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.160+0000 7f36f4c68640 10 monclient: init 2024-09-18T12:42:47.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f0131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient(hunting): picked mon.noname-b con 0x7f36f0131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f36f009e910 0x7f36f009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient(hunting): picked mon.noname-a con 0x7f36f009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36f009f220 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f009faf0 con 0x7f36f009f220 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f009fd30 con 0x7f36f009e910 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f00a1110 con 0x7f36f0131ab0 2024-09-18T12:42:47.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36f4c68640 10 monclient(hunting): authenticate will time out at 1794.482565s 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f0131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36edd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f36f009e910 0x7f36f009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36edd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f36f009e910 0x7f36f009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54534/0 (socket says 172.21.15.116:54534) 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f0131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41386/0 (socket says 172.21.15.116:41386) 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36edd9b640 1 -- 172.21.15.116:0/1725480276 learned_addr learned my addr 172.21.15.116:0/1725480276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36edd9b640 10 monclient(hunting): get_auth_request con 0x7f36f009e910 auth_method 0 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): get_auth_request con 0x7f36f0131ab0 auth_method 0 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient(hunting): handle_auth_done global_id 214106 payload 274 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 1 -- 172.21.15.116:0/1725480276 >> v2:172.21.15.116:3300/0 conn(0x7f36f009e910 msgr2=0x7f36f009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 1 --2- 172.21.15.116:0/1725480276 >> v2:172.21.15.116:3300/0 conn(0x7f36f009e910 0x7f36f009ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 1 -- 172.21.15.116:0/1725480276 >> v1:172.21.15.116:6789/0 conn(0x7f36f009f220 legacy=0x7f36f0068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient: _finish_hunting 0 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 1 monclient: found mon.noname-b 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:42:47.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.161+0000 7f36ee59c640 1 -- 172.21.15.116:0/1725480276 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f00a13f0 con 0x7f36f0131ab0 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.164+0000 7f36ee59c640 1 -- 172.21.15.116:0/1725480276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 msgr2=0x7f36f0131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.164+0000 7f36ee59c640 1 --2- 172.21.15.116:0/1725480276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f0131e80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009850 tx=0x7f36e80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4fe2237dcfb5308 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 1 -- 172.21.15.116:0/1725480276 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e802ae60 con 0x7f36f0131ab0 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: dump: 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: _finish_auth 0 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 1 -- 172.21.15.116:0/1725480276 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e8032440 con 0x7f36f0131ab0 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 5 monclient: authenticate success, global_id 214106 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 4 set_mon_vals no callback set 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 1 -- 172.21.15.116:0/1725480276 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e80327e0 con 0x7f36f0131ab0 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36ed59a640 10 monclient: dump: 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 10 monclient: shutdown 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 -- 172.21.15.116:0/1725480276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 msgr2=0x7f36f0131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 --2- 172.21.15.116:0/1725480276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f0131e80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009850 tx=0x7f36e80090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 -- 172.21.15.116:0/1725480276 shutdown_connections 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 --2- 172.21.15.116:0/1725480276 >> v2:172.21.15.116:3300/0 conn(0x7f36f009e910 0x7f36f009ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 --2- 172.21.15.116:0/1725480276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f0131e80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 -- 172.21.15.116:0/1725480276 >> 172.21.15.116:0/1725480276 conn(0x7f36f0067d30 msgr2=0x7f36f00a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 -- 172.21.15.116:0/1725480276 shutdown_connections 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.165+0000 7f36f4c68640 1 -- 172.21.15.116:0/1725480276 wait complete. 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.166+0000 7f36f4c68640 10 monclient: build_initial_monmap 2024-09-18T12:42:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.166+0000 7f36f4c68640 10 monclient: monmap: 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.166+0000 7f36f4c68640 1 Processor -- start 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.166+0000 7f36f4c68640 1 -- start start 2024-09-18T12:42:47.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.166+0000 7f36f4c68640 10 monclient: init 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f009e910 0x7f36f009ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): picked mon.noname-b con 0x7f36f009e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f009f220 0x7f36f0130c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): picked mon.noname-a con 0x7f36f009f220 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f011b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): picked mon.noname-c con 0x7f36f0131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f009b720 con 0x7f36f009f220 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f009bde0 con 0x7f36f009e910 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f009dd40 con 0x7f36f0131ab0 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36f4c68640 10 monclient(hunting): authenticate will time out at 1794.488119s 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f009f220 0x7f36f0130c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f011b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f011b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41396/0 (socket says 172.21.15.116:41396) 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f009f220 0x7f36f0130c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54546/0 (socket says 172.21.15.116:54546) 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 -- 172.21.15.116:0/2956965190 learned_addr learned my addr 172.21.15.116:0/2956965190 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): get_auth_request con 0x7f36f0131ab0 auth_method 0 2024-09-18T12:42:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 10 monclient(hunting): get_auth_request con 0x7f36f009f220 auth_method 0 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36ee59c640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f009e910 0x7f36f009ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36ee59c640 10 monclient(hunting): get_auth_request con 0x7f36f009e910 auth_method 0 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient(hunting): handle_auth_done global_id 214109 payload 824 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f009e910 msgr2=0x7f36f009ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f009e910 0x7f36f009ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f009f220 msgr2=0x7f36f0130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f009f220 0x7f36f0130c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient: _finish_hunting 0 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 monclient: found mon.noname-c 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36eed9d640 1 -- 172.21.15.116:0/2956965190 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e8009390 con 0x7f36f0131ab0 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.167+0000 7f36edd9b640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f009f220 0x7f36f0130c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36eed9d640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 msgr2=0x7f36f011b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36eed9d640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f011b6b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c8c0 tx=0x7f36e0002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e000aa00 con 0x7f36f0131ab0 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: dump: 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: _finish_auth 0 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36f4c68640 5 monclient: authenticate success, global_id 214109 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e000ab80 con 0x7f36f0131ab0 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36f4c68640 10 monclient: _renew_subs 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36f4c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f0141db0 con 0x7f36f0131ab0 2024-09-18T12:42:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0012420 con 0x7f36f0131ab0 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36e67fc640 10 monclient: dump: 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.186 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36f4c68640 10 monclient: _renew_subs 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36ecd99640 4 set_mon_vals no callback set 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36f4c68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.169+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f011ce30 con 0x7f36f0131ab0 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.170+0000 7f36f4c68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.170+0000 7f36f4c68640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 0x7f36f011bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.170+0000 7f36f4c68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.170+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f011df30 con 0x7f36f0141350 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.170+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f36e0012890 con 0x7f36f0131ab0 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.170+0000 7f36e67fc640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f36d0038490 0x7f36d003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.170+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f36e004c6e0 con 0x7f36f0131ab0 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 0x7f36f011bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient: get_auth_request con 0x7f36f0141350 auth_method 0 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36edd9b640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f36d0038490 0x7f36d003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36edd9b640 10 monclient: get_auth_request con 0x7f36d0038490 auth_method 0 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 10 monclient(hunting): handle_auth_done global_id 174496 payload 824 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.171+0000 7f36ee59c640 1 -- 172.21.15.116:0/2956965190 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36f004bf60 con 0x7f36f0141350 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.173+0000 7f36edd9b640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f36d0038490 0x7f36d003a940 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f36f00a4f60 tx=0x7f36d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.173+0000 7f36ee59c640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 msgr2=0x7f36f011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.173+0000 7f36ee59c640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 0x7f36f011bbf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f36e8034000 tx=0x7f36e8032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8031d30 con 0x7f36f0141350 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 10 monclient: dump: 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f36e8031e90 con 0x7f36f0141350 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 msgr2=0x7f36f011bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 0x7f36f011bbf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f36e8034000 tx=0x7f36e8032000 comp rx=0 tx=0).stop 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 msgr2=0x7f36f011bbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.174+0000 7f36e67fc640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f0141350 0x7f36f011bbf0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f36e8034000 tx=0x7f36e8032000 comp rx=0 tx=0).stop 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.179+0000 7f36f4c68640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.179+0000 7f36f4c68640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 0x7f36f011c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.179+0000 7f36f4c68640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.179+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f011e210 con 0x7f36f01311b0 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 0x7f36f011c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient: get_auth_request con 0x7f36f01311b0 auth_method 0 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 10 monclient(hunting): handle_auth_done global_id 174499 payload 824 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.180+0000 7f36ee59c640 1 -- 172.21.15.116:0/2956965190 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f36f011e3e0 con 0x7f36f01311b0 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.182+0000 7f36ee59c640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 msgr2=0x7f36f011c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.182+0000 7f36ee59c640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 0x7f36f011c130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36f011de20 tx=0x7f36e803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8046d40 con 0x7f36f01311b0 2024-09-18T12:42:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 10 monclient: dump: 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f36e803c3e0 con 0x7f36f01311b0 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 msgr2=0x7f36f011c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 0x7f36f011c130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36f011de20 tx=0x7f36e803d000 comp rx=0 tx=0).stop 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 msgr2=0x7f36f011c130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36e67fc640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36f01311b0 0x7f36f011c130 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:47.195 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 172, 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:47.196 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:47.197 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:47.199 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:47.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:47.201 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:47.202 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:47.205 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.206 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:47.207 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:47.208 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:47.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f36d0038490 msgr2=0x7f36d003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f36d0038490 0x7f36d003a940 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f36f00a4f60 tx=0x7f36d8008040 comp rx=0 tx=0).stop 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 10 monclient: shutdown 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 msgr2=0x7f36f011b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f011b6b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c8c0 tx=0x7f36e0002640 comp rx=0 tx=0).stop 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36eed9d640 1 -- 172.21.15.116:0/2956965190 reap_dead start 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 shutdown_connections 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 --2- 172.21.15.116:0/2956965190 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f36f0131ab0 0x7f36f011b6b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 >> 172.21.15.116:0/2956965190 conn(0x7f36f0067d30 msgr2=0x7f36f009df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 shutdown_connections 2024-09-18T12:42:47.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.183+0000 7f36f4c68640 1 -- 172.21.15.116:0/2956965190 wait complete. 2024-09-18T12:42:47.211 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:42:47.211 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:42:47.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:42:47.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.303+0000 7f747c19d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.303+0000 7f747c19d640 10 monclient: build_initial_monmap 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.303+0000 7f747c19d640 10 monclient: monmap: 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:47.303769+0000 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:47.303769+0000 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:47.304 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:47.305 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:47.305 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:47.305 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:47.305 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.303+0000 7f747c19d640 1 Processor -- start 2024-09-18T12:42:47.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.303+0000 7f747c19d640 1 -- start start 2024-09-18T12:42:47.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.303+0000 7f747c19d640 10 monclient: init 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74740975c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f747412ea30 0x7f7474136c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient(hunting): picked mon.noname-a con 0x7f747412ea30 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f747412ee50 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747406e9e0 con 0x7f747412ee50 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747409d1c0 con 0x7f74740975c0 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7474063950 con 0x7f747412ea30 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f747c19d640 10 monclient(hunting): authenticate will time out at 1794.625340s 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f747412ea30 0x7f7474136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f747412ea30 0x7f7474136c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54562/0 (socket says 172.21.15.116:54562) 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 1 -- 172.21.15.116:0/3343313916 learned_addr learned my addr 172.21.15.116:0/3343313916 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 10 monclient(hunting): get_auth_request con 0x7f747412ea30 auth_method 0 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.304+0000 7f7479738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f747a73a640 1 -- 172.21.15.116:0/3343313916 >> v1:172.21.15.116:6789/0 conn(0x7f747412ee50 legacy=0x7f74741391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479f39640 1 -- 172.21.15.116:0/3343313916 >> v1:172.21.15.144:6789/0 conn(0x7f74740975c0 legacy=0x7f7474097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 10 monclient(hunting): handle_auth_done global_id 194337 payload 274 2024-09-18T12:42:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 1 -- 172.21.15.116:0/3343313916 >> v1:172.21.15.144:6789/0 conn(0x7f74740975c0 legacy=0x7f7474097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 1 -- 172.21.15.116:0/3343313916 >> v1:172.21.15.116:6789/0 conn(0x7f747412ee50 legacy=0x7f74741391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 10 monclient: _finish_hunting 0 2024-09-18T12:42:47.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 1 monclient: found mon.noname-a 2024-09-18T12:42:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:47.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.306+0000 7f7479738640 1 -- 172.21.15.116:0/3343313916 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747409d4a0 con 0x7f747412ea30 2024-09-18T12:42:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7479738640 1 -- 172.21.15.116:0/3343313916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ea30 msgr2=0x7f7474136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7479738640 1 --2- 172.21.15.116:0/3343313916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ea30 0x7f7474136c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7464009a10 tx=0x7f74640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90064b365a510088 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 1 -- 172.21.15.116:0/3343313916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746402f4b0 con 0x7f747412ea30 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: dump: 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: _finish_auth 0 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f747c19d640 5 monclient: authenticate success, global_id 194337 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 1 -- 172.21.15.116:0/3343313916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746402f610 con 0x7f747412ea30 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f747c19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f747c19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f747c19d640 4 set_mon_vals no callback set 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 1 -- 172.21.15.116:0/3343313916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746402f9b0 con 0x7f747412ea30 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.309+0000 7f7478f37640 10 monclient: dump: 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 10 monclient: shutdown 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 1 -- 172.21.15.116:0/3343313916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ea30 msgr2=0x7f7474136c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 1 --2- 172.21.15.116:0/3343313916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ea30 0x7f7474136c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7464009a10 tx=0x7f74640091d0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 1 -- 172.21.15.116:0/3343313916 shutdown_connections 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 1 --2- 172.21.15.116:0/3343313916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ea30 0x7f7474136c80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 1 -- 172.21.15.116:0/3343313916 >> 172.21.15.116:0/3343313916 conn(0x7f7474067b50 msgr2=0x7f74740a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 1 -- 172.21.15.116:0/3343313916 shutdown_connections 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.310+0000 7f747c19d640 1 -- 172.21.15.116:0/3343313916 wait complete. 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient: build_initial_monmap 2024-09-18T12:42:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient: monmap: 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 Processor -- start 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 -- start start 2024-09-18T12:42:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient: init 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74740975c0 0x7f7474130330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): picked mon.noname-b con 0x7f74740975c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747412ea30 0x7f7474128d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): picked mon.noname-c con 0x7f747412ea30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 0x7f74741292b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): picked mon.noname-a con 0x7f747412ee50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747412c690 con 0x7f747412ee50 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747412fc30 con 0x7f74740975c0 2024-09-18T12:42:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7474130960 con 0x7f747412ea30 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747c19d640 10 monclient(hunting): authenticate will time out at 1794.632806s 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f7479738640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747412ea30 0x7f7474128d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747a73a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 0x7f74741292b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f7479738640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747412ea30 0x7f7474128d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41406/0 (socket says 172.21.15.116:41406) 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f7479738640 1 -- 172.21.15.116:0/1355178988 learned_addr learned my addr 172.21.15.116:0/1355178988 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747a73a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 0x7f74741292b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54580/0 (socket says 172.21.15.116:54580) 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f7479738640 10 monclient(hunting): get_auth_request con 0x7f747412ea30 auth_method 0 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f7479738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f7479738640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f7479738640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747a73a640 10 monclient(hunting): get_auth_request con 0x7f747412ee50 auth_method 0 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747a73a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747a73a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.311+0000 7f747a73a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479f39640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74740975c0 0x7f7474130330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479f39640 10 monclient(hunting): get_auth_request con 0x7f74740975c0 auth_method 0 2024-09-18T12:42:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 10 monclient(hunting): handle_auth_done global_id 194340 payload 824 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 1 -- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747412ea30 msgr2=0x7f7474128d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747412ea30 0x7f7474128d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 1 -- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74740975c0 msgr2=0x7f7474130330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74740975c0 0x7f7474130330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 10 monclient: _finish_hunting 0 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 1 monclient: found mon.noname-a 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f747a73a640 1 -- 172.21.15.116:0/1355178988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7464009450 con 0x7f747412ee50 2024-09-18T12:42:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.312+0000 7f7479738640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747412ea30 0x7f7474128d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:47.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f747a73a640 1 -- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 msgr2=0x7f74741292b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:47.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f747a73a640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 0x7f74741292b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f746c00c810 tx=0x7f746c001830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 1 -- 172.21.15.116:0/1355178988 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c00ce00 con 0x7f747412ee50 2024-09-18T12:42:47.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:47.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: dump: 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: _finish_auth 0 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f747c19d640 5 monclient: authenticate success, global_id 194340 2024-09-18T12:42:47.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 1 -- 172.21.15.116:0/1355178988 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c00a640 con 0x7f747412ee50 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f747c19d640 10 monclient: _renew_subs 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f747c19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747412f950 con 0x7f747412ee50 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 1 -- 172.21.15.116:0/1355178988 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c00aa00 con 0x7f747412ee50 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f7469ffb640 10 monclient: dump: 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.315+0000 7f747c19d640 10 monclient: _renew_subs 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.314+0000 7f746bfff640 4 set_mon_vals no callback set 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.315+0000 7f747c19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.315+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7474129990 con 0x7f747412ee50 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.315+0000 7f7469ffb640 1 -- 172.21.15.116:0/1355178988 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f746c0070f0 con 0x7f747412ee50 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.315+0000 7f7469ffb640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74580384c0 0x7f745803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.315+0000 7f7469ffb640 1 -- 172.21.15.116:0/1355178988 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f746c04b980 con 0x7f747412ee50 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.316+0000 7f747c19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.316+0000 7f747c19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.316+0000 7f747c19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.316+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747404bf60 con 0x7f747412ee50 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.316+0000 7f7479f39640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74580384c0 0x7f745803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.316+0000 7f7479f39640 10 monclient: get_auth_request con 0x7f74580384c0 auth_method 0 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.318+0000 7f7479f39640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74580384c0 0x7f745803a970 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7470001a30 tx=0x7f7470009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.320+0000 7f7469ffb640 1 -- 172.21.15.116:0/1355178988 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f746c017020 con 0x7f747412ee50 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.320+0000 7f7469ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:47.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.320+0000 7f7469ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:47.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.531+0000 7f747c19d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:42:47.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.531+0000 7f747c19d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:42:47.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.531+0000 7f747c19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:47.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.531+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f747412b490 con 0x7f747412ee50 2024-09-18T12:42:47.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f7469ffb640 1 -- 172.21.15.116:0/1355178988 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f746c018a40 con 0x7f747412ee50 2024-09-18T12:42:47.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f7469ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:42:47.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f7469ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:47.533 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=8,logm=52} crc {auth=1259142338,config=3140514009,health=1969285066,kv=3604100458,logm=3390675794}) 2024-09-18T12:42:47.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=4133684831,config=3140514009,health=2881375469,kv=2187923472,logm=3384989403}) 2024-09-18T12:42:47.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=8,logm=52} crc {auth=1259142338,config=3140514009,health=1969285066,kv=3604100458,logm=3390675794}) 2024-09-18T12:42:47.534 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=1,health=12,kv=9,logm=48} crc {auth=3852835195,config=1228700967,health=3086468925,kv=2187923472,logm=2851402835}) 2024-09-18T12:42:47.535 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.535 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938614534}) 2024-09-18T12:42:47.535 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678846284}) 2024-09-18T12:42:47.535 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.535 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938614534}) 2024-09-18T12:42:47.535 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1692175907}) 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74580384c0 msgr2=0x7f745803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74580384c0 0x7f745803a970 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7470001a30 tx=0x7f7470009040 comp rx=0 tx=0).stop 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 10 monclient: shutdown 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 msgr2=0x7f74741292b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 0x7f74741292b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f746c00c810 tx=0x7f746c001830 comp rx=0 tx=0).stop 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 shutdown_connections 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74580384c0 0x7f745803a970 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f747412ee50 0x7f74741292b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f747412ea30 0x7f7474128d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 --2- 172.21.15.116:0/1355178988 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74740975c0 0x7f7474130330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 >> 172.21.15.116:0/1355178988 conn(0x7f7474067b50 msgr2=0x7f74740955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 shutdown_connections 2024-09-18T12:42:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:47.534+0000 7f747c19d640 1 -- 172.21.15.116:0/1355178988 wait complete. 2024-09-18T12:42:47.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1302510693}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109960501}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1302510693}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1954815120}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251347537}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1453670788}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251347537}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929600607}) 2024-09-18T12:42:47.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=241653465}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3878537216}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=241653465}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642340185}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589239157}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1151351864}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589239157}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243087746}) 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159608564}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801766918}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159608564}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421256528}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178695091}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2215824635}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178695091}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358049768}) 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603923642}) 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655041234}) 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603923642}) 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046017830}) 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2758648604}) 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=789786643}) 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2758648604}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=353748986}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693516898}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=421292204}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693516898}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3923567485}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396470343}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3155820188}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396470343}) 2024-09-18T12:42:47.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15743524}) 2024-09-18T12:42:47.549 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.549 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569586382}) 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707911067}) 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569586382}) 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=959918431}) 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945304143}) 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3356409201}) 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.550 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945304143}) 2024-09-18T12:42:47.551 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246832198}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792085326}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=418786595}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792085326}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3660280344}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952858262}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=629075271}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952858262}) 2024-09-18T12:42:47.552 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565383868}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617686333}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987571650}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617686333}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015097523}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110660380}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=454888288}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110660380}) 2024-09-18T12:42:47.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3220024684}) 2024-09-18T12:42:47.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.545+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.545+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378445645}) 2024-09-18T12:42:47.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.545+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245589599}) 2024-09-18T12:42:47.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.545+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.545+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378445645}) 2024-09-18T12:42:47.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.545+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1795445452}) 2024-09-18T12:42:47.555 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741169406}) 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797737873}) 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741169406}) 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2637882006}) 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353209780}) 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771407176}) 2024-09-18T12:42:47.581 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353209780}) 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.546+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2946202758}) 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.547+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.547+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024861307}) 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.547+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885801865}) 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.547+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.547+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024861307}) 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.547+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751827963}) 2024-09-18T12:42:47.582 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.547+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329175912}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388844366}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329175912}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061636724}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783956137}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3050793472}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783956137}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.548+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2416573041}) 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.583 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637874995}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108003496}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637874995}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=188644116}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250655764}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825750876}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250655764}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218922442}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.550+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.550+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916254483}) 2024-09-18T12:42:47.584 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.550+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=966615145}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.550+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.550+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2916254483}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.550+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358546612}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289753647}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843053415}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289753647}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887293245}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797168812}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=838876018}) 2024-09-18T12:42:47.585 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797168812}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.551+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1959838262}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879253649}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167438723}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879253649}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970576377}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.552+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954702388}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779972752}) 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.586 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954702388}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612934955}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259913956}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=117241335}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259913956}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.553+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613139062}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372078945}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2270039945}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372078945}) 2024-09-18T12:42:47.587 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965802269}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299646977}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524014945}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299646977}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.554+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530432236}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.555+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.555+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686021008}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.555+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2980067972}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.555+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.555+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686021008}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.555+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2800415437}) 2024-09-18T12:42:47.588 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101155579}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625301141}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101155579}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946714507}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366733342}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834342948}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366733342}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274514564}) 2024-09-18T12:42:47.589 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.557+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.557+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696528437}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.557+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4236615675}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.557+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.557+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696528437}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.557+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746638641}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468019170}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=99098195}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468019170}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218637088}) 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.590 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142056507}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1510798106}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142056507}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.558+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3386308553}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.559+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.559+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756337325}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.559+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3420411694}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.559+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.559+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756337325}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.559+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531278073}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182795740}) 2024-09-18T12:42:47.591 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399359626}) 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182795740}) 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321997880}) 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333785103}) 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4287189546}) 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333785103}) 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.560+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392211341}) 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.561+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.592 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.561+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279543273}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.561+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=961404656}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.561+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.561+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279543273}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.561+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1023086257}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.562+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.562+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588299852}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.562+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196056860}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.562+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.562+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588299852}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.562+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924001676}) 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.593 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458943639}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=87804883}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458943639}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474027142}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707282082}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135353206}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707282082}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.563+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492967896}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.564+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.564+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579041926}) 2024-09-18T12:42:47.594 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.564+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3235418540}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.564+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.564+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579041926}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.564+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=367622360}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151148248}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3264931113}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151148248}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080975846}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757812240}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=463656130}) 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.595 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757812240}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.565+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4221006747}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.566+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.566+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749754901}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.566+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911368138}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.566+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.566+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749754901}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.566+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171794437}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274776152}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1964321409}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274776152}) 2024-09-18T12:42:47.596 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617706919}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849711459}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3538732227}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849711459}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.567+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268744982}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694598056}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066555389}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694598056}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915765}) 2024-09-18T12:42:47.597 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397003609}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.568+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2431206842}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397003609}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906670914}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4189091345}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2453104007}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4189091345}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.569+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909026430}) 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.598 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008642060}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376377604}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008642060}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=155658811}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522639848}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251721956}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522639848}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.570+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323383793}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.571+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.571+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2089444812}) 2024-09-18T12:42:47.599 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.571+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2267193811}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.571+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.571+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2089444812}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.571+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017520889}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079800113}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047887436}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079800113}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617299979}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3515527456}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=192713176}) 2024-09-18T12:42:47.600 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3515527456}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.572+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147052919}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.573+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.573+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=904792499}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.573+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3920786100}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.573+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.573+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=904792499}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.573+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010245897}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627204053}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3855835334}) 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.601 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627204053}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80928518}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462014219}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275998730}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462014219}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.574+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189371421}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.575+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.575+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049329615}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.575+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=874863458}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.575+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.575+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049329615}) 2024-09-18T12:42:47.602 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.575+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291765995}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.575+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760349036}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=838930540}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760349036}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3772135283}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206326134}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774324210}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206326134}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.576+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1012984386}) 2024-09-18T12:42:47.603 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674184202}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550455126}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674184202}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380349304}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714775534}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=905517605}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714775534}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.577+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541888221}) 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.578+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.604 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.578+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943294912}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.578+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581820742}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.578+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.578+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943294912}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.578+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3694885200}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114092249}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067047291}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114092249}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095857503}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248290301}) 2024-09-18T12:42:47.605 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1499467311}) 2024-09-18T12:42:47.606 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.606 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248290301}) 2024-09-18T12:42:47.606 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.579+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461228426}) 2024-09-18T12:42:47.655 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.656 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129239438}) 2024-09-18T12:42:47.656 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601035651}) 2024-09-18T12:42:47.656 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.656 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129239438}) 2024-09-18T12:42:47.656 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263361136}) 2024-09-18T12:42:47.658 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.658 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331286358}) 2024-09-18T12:42:47.658 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727187424}) 2024-09-18T12:42:47.658 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.658 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331286358}) 2024-09-18T12:42:47.658 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926418301}) 2024-09-18T12:42:47.771 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.771 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775516503}) 2024-09-18T12:42:47.771 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3947903370}) 2024-09-18T12:42:47.771 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.771 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775516503}) 2024-09-18T12:42:47.772 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425146311}) 2024-09-18T12:42:47.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3811398640}) 2024-09-18T12:42:47.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481241881}) 2024-09-18T12:42:47.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:47.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3811398640}) 2024-09-18T12:42:47.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:47.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941436128}) 2024-09-18T12:42:48.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765331539}) 2024-09-18T12:42:48.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2775977776}) 2024-09-18T12:42:48.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=765331539}) 2024-09-18T12:42:48.005 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291201527}) 2024-09-18T12:42:48.039 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.039 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559806266}) 2024-09-18T12:42:48.039 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=810021724}) 2024-09-18T12:42:48.039 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.039 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559806266}) 2024-09-18T12:42:48.039 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1425759061}) 2024-09-18T12:42:48.040 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.040 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271554175}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=334634317}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271554175}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.038+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2789324774}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.039+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.039+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656905904}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.039+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4083770972}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.039+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.039+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656905904}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.039+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706955915}) 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.041 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504622342}) 2024-09-18T12:42:48.042 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953144816}) 2024-09-18T12:42:48.042 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.042 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504622342}) 2024-09-18T12:42:48.042 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616786575}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071896724}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728875261}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071896724}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.040+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962863929}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.041+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.041+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814466341}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.041+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4269392188}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.041+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.041+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814466341}) 2024-09-18T12:42:48.048 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.041+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106269893}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405197045}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2087075403}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405197045}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3929508125}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389337469}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2757059870}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389337469}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.042+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2075369160}) 2024-09-18T12:42:48.049 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.043+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.043+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851304803}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.043+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3587198095}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.043+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.043+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851304803}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.043+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639595210}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059987434}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680434412}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059987434}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038474364}) 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.050 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063168360}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312961540}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063168360}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.044+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3273412018}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516961033}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162588951}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516961033}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885338724}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147302829}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1082333895}) 2024-09-18T12:42:48.051 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147302829}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.045+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413820413}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684968133}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3022538457}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684968133}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040979667}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918353696}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2523895131}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918353696}) 2024-09-18T12:42:48.052 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.046+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1708898979}) 2024-09-18T12:42:48.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.053 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {logm=750032210,mdsmap=1531151339,mgr=3426601819,mgr_metadata=3239289319,mgrstat=4277366270,mon_config_key=1611758519,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=4037138792}) 2024-09-18T12:42:48.054 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=12,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=50} crc {logm=3553887977,mdsmap=3585761405,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4224416570,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=3955024689}) 2024-09-18T12:42:48.054 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:42:48.054 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {logm=750032210,mdsmap=1531151339,mgr=3426601819,mgr_metadata=3239289319,mgrstat=4277366270,mon_config_key=1611758519,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=4037138792}) 2024-09-18T12:42:48.054 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:42:48.048+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {logm=617908793,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=780939174,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2816006276}) 2024-09-18T12:42:48.556 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:42:48.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:42:48.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.658+0000 7f68d351d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:48.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.658+0000 7f68d351d640 10 monclient: build_initial_monmap 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.658+0000 7f68d351d640 10 monclient: monmap: 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:48.659366+0000 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:48.659366+0000 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:48.660 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:48.661 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:48.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.659+0000 7f68d351d640 1 Processor -- start 2024-09-18T12:42:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.659+0000 7f68d351d640 1 -- start start 2024-09-18T12:42:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.659+0000 7f68d351d640 10 monclient: init 2024-09-18T12:42:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.659+0000 7f68d351d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.659+0000 7f68d351d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T12:42:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.659+0000 7f68d351d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 0x7f68cc096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:48.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.659+0000 7f68d351d640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc09a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68cc0971a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68cc097570 0x7f68cc12fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc097570 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc09cb00 con 0x7f68cc0971a0 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68cc09a8e0 con 0x7f68cc097570 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc06ed80 con 0x7f68cc09a510 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d351d640 10 monclient(hunting): authenticate will time out at 1795.981117s 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 0x7f68cc096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 0x7f68cc096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41418/0 (socket says 172.21.15.116:41418) 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 -- 172.21.15.116:0/4240954590 learned_addr learned my addr 172.21.15.116:0/4240954590 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:48.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): get_auth_request con 0x7f68cc09a510 auth_method 0 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 1 --2- 172.21.15.116:0/4240954590 >> v2:172.21.15.116:3300/0 conn(0x7f68cc097570 0x7f68cc12fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 10 monclient(hunting): get_auth_request con 0x7f68cc097570 auth_method 0 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:48.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d1aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient(hunting): handle_auth_done global_id 214115 payload 274 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 -- 172.21.15.116:0/4240954590 >> v2:172.21.15.116:3300/0 conn(0x7f68cc097570 msgr2=0x7f68cc12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 --2- 172.21.15.116:0/4240954590 >> v2:172.21.15.116:3300/0 conn(0x7f68cc097570 0x7f68cc12fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 -- 172.21.15.116:0/4240954590 >> v1:172.21.15.144:6789/0 conn(0x7f68cc0971a0 legacy=0x7f68cc12f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient: _finish_hunting 0 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 monclient: found mon.noname-b 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:42:48.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.660+0000 7f68d12b9640 1 -- 172.21.15.116:0/4240954590 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc09c820 con 0x7f68cc09a510 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d12b9640 1 -- 172.21.15.116:0/4240954590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 msgr2=0x7f68cc096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d12b9640 1 --2- 172.21.15.116:0/4240954590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 0x7f68cc096c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68cc06ef70 tx=0x7f68c80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e3c8307948d141c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 1 -- 172.21.15.116:0/4240954590 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c802ae90 con 0x7f68cc09a510 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: dump: 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.672 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: _finish_auth 0 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 1 -- 172.21.15.116:0/4240954590 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c8032440 con 0x7f68cc09a510 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 5 monclient: authenticate success, global_id 214115 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 1 -- 172.21.15.116:0/4240954590 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c80327e0 con 0x7f68cc09a510 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 4 set_mon_vals no callback set 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68c3fff640 10 monclient: dump: 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 10 monclient: shutdown 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 1 -- 172.21.15.116:0/4240954590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 msgr2=0x7f68cc096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 1 --2- 172.21.15.116:0/4240954590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 0x7f68cc096c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68cc06ef70 tx=0x7f68c80091d0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 1 -- 172.21.15.116:0/4240954590 shutdown_connections 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 1 --2- 172.21.15.116:0/4240954590 >> v2:172.21.15.116:3300/0 conn(0x7f68cc097570 0x7f68cc12fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 1 --2- 172.21.15.116:0/4240954590 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc09a510 0x7f68cc096c60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.663+0000 7f68d351d640 1 -- 172.21.15.116:0/4240954590 >> 172.21.15.116:0/4240954590 conn(0x7f68cc0672b0 msgr2=0x7f68cc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.664+0000 7f68d351d640 1 -- 172.21.15.116:0/4240954590 shutdown_connections 2024-09-18T12:42:48.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.664+0000 7f68d351d640 1 -- 172.21.15.116:0/4240954590 wait complete. 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.664+0000 7f68d351d640 10 monclient: build_initial_monmap 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.664+0000 7f68d351d640 10 monclient: monmap: 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:48.676 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.664+0000 7f68d351d640 1 Processor -- start 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.664+0000 7f68d351d640 1 -- start start 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.664+0000 7f68d351d640 10 monclient: init 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 0x7f68cc141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc0971a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68cc097570 0x7f68cc12fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc097570 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68cc09a510 0x7f68cc098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc09a510 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc11f2d0 con 0x7f68cc09a510 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc06da90 con 0x7f68cc097570 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc11de70 con 0x7f68cc0971a0 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d351d640 10 monclient(hunting): authenticate will time out at 1795.986535s 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 0x7f68cc141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 0x7f68cc141130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41424/0 (socket says 172.21.15.116:41424) 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 -- 172.21.15.116:0/2242658339 learned_addr learned my addr 172.21.15.116:0/2242658339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): get_auth_request con 0x7f68cc0971a0 auth_method 0 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:48.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68cc09a510 0x7f68cc098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 10 monclient(hunting): get_auth_request con 0x7f68cc09a510 auth_method 0 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d0ab8640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68cc097570 0x7f68cc12fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient(hunting): handle_auth_done global_id 214118 payload 824 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 -- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68cc097570 msgr2=0x7f68cc12fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68cc097570 0x7f68cc12fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 -- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68cc09a510 msgr2=0x7f68cc098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68cc09a510 0x7f68cc098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient: _finish_hunting 0 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 monclient: found mon.noname-c 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d12b9640 1 -- 172.21.15.116:0/2242658339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c8009450 con 0x7f68cc0971a0 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d0ab8640 10 monclient: get_auth_request con 0x7f68cc097570 auth_method 0 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d0ab8640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68cc097570 0x7f68cc12fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:42:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.665+0000 7f68d1aba640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68cc09a510 0x7f68cc098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d12b9640 1 -- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 msgr2=0x7f68cc141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d12b9640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 0x7f68cc141130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009210 tx=0x7f68c8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 1 -- 172.21.15.116:0/2242658339 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8000c40 con 0x7f68cc0971a0 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: dump: 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: _finish_auth 0 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 1 -- 172.21.15.116:0/2242658339 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c802ae90 con 0x7f68cc0971a0 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d351d640 5 monclient: authenticate success, global_id 214118 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d351d640 10 monclient: _renew_subs 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d351d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc11efc0 con 0x7f68cc0971a0 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 1 -- 172.21.15.116:0/2242658339 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c803d420 con 0x7f68cc0971a0 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c17fa640 10 monclient: dump: 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68c37fe640 4 set_mon_vals no callback set 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d351d640 10 monclient: _renew_subs 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d351d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.668+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc141ed0 con 0x7f68cc0971a0 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68c17fa640 1 -- 172.21.15.116:0/2242658339 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f68c8039040 con 0x7f68cc0971a0 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68d351d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68d351d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68d351d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc04bf60 con 0x7f68cc0971a0 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68c17fa640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f68b00384e0 0x7f68b003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68c17fa640 1 -- 172.21.15.116:0/2242658339 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f68c806c120 con 0x7f68cc0971a0 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68d0ab8640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f68b00384e0 0x7f68b003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.669+0000 7f68d0ab8640 10 monclient: get_auth_request con 0x7f68b00384e0 auth_method 0 2024-09-18T12:42:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.672+0000 7f68d0ab8640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f68b00384e0 0x7f68b003a990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f68cc09a4e0 tx=0x7f68bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.674+0000 7f68c17fa640 1 -- 172.21.15.116:0/2242658339 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f68c803d810 con 0x7f68cc0971a0 2024-09-18T12:42:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.674+0000 7f68c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.674+0000 7f68c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.889+0000 7f68d351d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.889+0000 7f68d351d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.889+0000 7f68d351d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.889+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f68cc11f410 con 0x7f68cc0971a0 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.889+0000 7f68c17fa640 1 -- 172.21.15.116:0/2242658339 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f68c8042020 con 0x7f68cc0971a0 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.889+0000 7f68c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.889+0000 7f68c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:48.892 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":172,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:42:48.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f68b00384e0 msgr2=0x7f68b003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:48.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f68b00384e0 0x7f68b003a990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f68cc09a4e0 tx=0x7f68bc009040 comp rx=0 tx=0).stop 2024-09-18T12:42:48.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 10 monclient: shutdown 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 msgr2=0x7f68cc141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 0x7f68cc141130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009210 tx=0x7f68c8032000 comp rx=0 tx=0).stop 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 shutdown_connections 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f68b00384e0 0x7f68b003a990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68cc09a510 0x7f68cc098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f68cc097570 0x7f68cc12fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 --2- 172.21.15.116:0/2242658339 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f68cc0971a0 0x7f68cc141130 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 >> 172.21.15.116:0/2242658339 conn(0x7f68cc0672b0 msgr2=0x7f68cc09fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 shutdown_connections 2024-09-18T12:42:48.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.891+0000 7f68d351d640 1 -- 172.21.15.116:0/2242658339 wait complete. 2024-09-18T12:42:48.903 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:42:48.903 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:42:48.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:42:48.903 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:42:48.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.995+0000 7fea5b246640 10 monclient: get_monmap_and_config 2024-09-18T12:42:48.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.995+0000 7fea5b246640 10 monclient: build_initial_monmap 2024-09-18T12:42:48.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.995+0000 7fea5b246640 10 monclient: monmap: 2024-09-18T12:42:48.996 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:48.996 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:48.996255+0000 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:48.996255+0000 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:48.997 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 1 Processor -- start 2024-09-18T12:42:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 1 -- start start 2024-09-18T12:42:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient: init 2024-09-18T12:42:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T12:42:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fea54131ab0 0x7fea54131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient(hunting): picked mon.noname-a con 0x7fea54131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea5409e910 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea54136bb0 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5409faf0 con 0x7fea54136bb0 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5409fd30 con 0x7fea5409e910 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea540a1110 con 0x7fea54131ab0 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.996+0000 7fea5b246640 10 monclient(hunting): authenticate will time out at 1796.317960s 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.997+0000 7fea58fe2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fea54131ab0 0x7fea54131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.997+0000 7fea58fe2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fea54131ab0 0x7fea54131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54614/0 (socket says 172.21.15.116:54614) 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.997+0000 7fea58fe2640 1 -- 172.21.15.116:0/3949021320 learned_addr learned my addr 172.21.15.116:0/3949021320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.997+0000 7fea58fe2640 10 monclient(hunting): get_auth_request con 0x7fea54131ab0 auth_method 0 2024-09-18T12:42:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.997+0000 7fea58fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.997+0000 7fea58fe2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.997+0000 7fea58fe2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea597e3640 1 -- 172.21.15.116:0/3949021320 >> v1:172.21.15.116:6789/0 conn(0x7fea54136bb0 legacy=0x7fea54138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea53fff640 1 -- 172.21.15.116:0/3949021320 >> v1:172.21.15.144:6789/0 conn(0x7fea5409e910 legacy=0x7fea5409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 1 -- 172.21.15.116:0/3949021320 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769951825 0 0) 0x7fea5409faf0 con 0x7fea54136bb0 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 10 monclient(hunting): my global_id is 194352 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 1 -- 172.21.15.116:0/3949021320 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea440034e0 con 0x7fea54136bb0 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 1 -- 172.21.15.116:0/3949021320 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496218032 0 0) 0x7fea5409fd30 con 0x7fea5409e910 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 10 monclient(hunting): my global_id is 174505 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea537fe640 1 -- 172.21.15.116:0/3949021320 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5409faf0 con 0x7fea5409e910 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 10 monclient(hunting): handle_auth_done global_id 194349 payload 274 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 1 -- 172.21.15.116:0/3949021320 >> v1:172.21.15.144:6789/0 conn(0x7fea5409e910 legacy=0x7fea5409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 1 -- 172.21.15.116:0/3949021320 >> v1:172.21.15.116:6789/0 conn(0x7fea54136bb0 legacy=0x7fea54138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 10 monclient: _finish_hunting 0 2024-09-18T12:42:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 1 monclient: found mon.noname-a 2024-09-18T12:42:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:48.999+0000 7fea58fe2640 1 -- 172.21.15.116:0/3949021320 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea540a13f0 con 0x7fea54131ab0 2024-09-18T12:42:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea58fe2640 1 -- 172.21.15.116:0/3949021320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54131ab0 msgr2=0x7fea54131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea58fe2640 1 --2- 172.21.15.116:0/3949021320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54131ab0 0x7fea54131e80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea48009a40 tx=0x7fea480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff673f01a95f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea537fe640 1 -- 172.21.15.116:0/3949021320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4802f4b0 con 0x7fea54131ab0 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea537fe640 10 monclient: dump: 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.005 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.006 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea537fe640 10 monclient: _finish_auth 0 2024-09-18T12:42:49.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:49.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea5b246640 5 monclient: authenticate success, global_id 194349 2024-09-18T12:42:49.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.002+0000 7fea5b246640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea537fe640 1 -- 172.21.15.116:0/3949021320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4802f610 con 0x7fea54131ab0 2024-09-18T12:42:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea537fe640 1 -- 172.21.15.116:0/3949021320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4802f990 con 0x7fea54131ab0 2024-09-18T12:42:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea537fe640 10 monclient: dump: 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea5b246640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.007+0000 7fea5b246640 4 set_mon_vals no callback set 2024-09-18T12:42:49.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.008+0000 7fea5b246640 10 monclient: shutdown 2024-09-18T12:42:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.008+0000 7fea5b246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.008+0000 7fea5b246640 1 -- 172.21.15.116:0/3949021320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54131ab0 msgr2=0x7fea54131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.008+0000 7fea5b246640 1 --2- 172.21.15.116:0/3949021320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54131ab0 0x7fea54131e80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea48009a40 tx=0x7fea480091d0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.008+0000 7fea5b246640 1 -- 172.21.15.116:0/3949021320 shutdown_connections 2024-09-18T12:42:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.008+0000 7fea5b246640 1 --2- 172.21.15.116:0/3949021320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54131ab0 0x7fea54131e80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.008+0000 7fea5b246640 1 -- 172.21.15.116:0/3949021320 >> 172.21.15.116:0/3949021320 conn(0x7fea54067d30 msgr2=0x7fea540a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.009+0000 7fea5b246640 1 -- 172.21.15.116:0/3949021320 shutdown_connections 2024-09-18T12:42:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.009+0000 7fea5b246640 1 -- 172.21.15.116:0/3949021320 wait complete. 2024-09-18T12:42:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.009+0000 7fea5b246640 10 monclient: build_initial_monmap 2024-09-18T12:42:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.009+0000 7fea5b246640 10 monclient: monmap: 2024-09-18T12:42:49.022 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:49.023 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:49.023 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:49.023 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:49.023 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:49.023 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.024 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:49.024 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:49.024 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:49.024 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.009+0000 7fea5b246640 1 Processor -- start 2024-09-18T12:42:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 1 -- start start 2024-09-18T12:42:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient: init 2024-09-18T12:42:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea5409e910 0x7fea541303c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): picked mon.noname-a con 0x7fea5409e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea54131ab0 0x7fea54132680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): picked mon.noname-c con 0x7fea54131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54136bb0 0x7fea54132bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): picked mon.noname-b con 0x7fea54136bb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:49.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea54134a80 con 0x7fea5409e910 2024-09-18T12:42:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5413a130 con 0x7fea54136bb0 2024-09-18T12:42:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea54134100 con 0x7fea54131ab0 2024-09-18T12:42:49.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:49.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea5b246640 10 monclient(hunting): authenticate will time out at 1796.331488s 2024-09-18T12:42:49.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea54131ab0 0x7fea54132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea54131ab0 0x7fea54132680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41440/0 (socket says 172.21.15.116:41440) 2024-09-18T12:42:49.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea5409e910 0x7fea541303c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 1 -- 172.21.15.116:0/3932545344 learned_addr learned my addr 172.21.15.116:0/3932545344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:49.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 10 monclient(hunting): get_auth_request con 0x7fea54131ab0 auth_method 0 2024-09-18T12:42:49.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): get_auth_request con 0x7fea5409e910 auth_method 0 2024-09-18T12:42:49.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea597e3640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54136bb0 0x7fea54132bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea597e3640 10 monclient(hunting): get_auth_request con 0x7fea54136bb0 auth_method 0 2024-09-18T12:42:49.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea597e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea597e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea597e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient(hunting): handle_auth_done global_id 194355 payload 824 2024-09-18T12:42:49.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea54131ab0 msgr2=0x7fea54132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea54131ab0 0x7fea54132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54136bb0 msgr2=0x7fea54132bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea54136bb0 0x7fea54132bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient: _finish_hunting 0 2024-09-18T12:42:49.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 1 monclient: found mon.noname-a 2024-09-18T12:42:49.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:49.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea58fe2640 1 -- 172.21.15.116:0/3932545344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea48009450 con 0x7fea5409e910 2024-09-18T12:42:49.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.010+0000 7fea53fff640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fea54131ab0 0x7fea54132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:49.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea58fe2640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea5409e910 msgr2=0x7fea541303c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea58fe2640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea5409e910 0x7fea541303c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea48031000 tx=0x7fea48032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4802f4b0 con 0x7fea5409e910 2024-09-18T12:42:49.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:49.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: dump: 2024-09-18T12:42:49.035 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.035 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.036 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.037 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: _finish_auth 0 2024-09-18T12:42:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea48006b30 con 0x7fea5409e910 2024-09-18T12:42:49.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea5b246640 5 monclient: authenticate success, global_id 194355 2024-09-18T12:42:49.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea5b246640 10 monclient: _renew_subs 2024-09-18T12:42:49.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea5b246640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5409bde0 con 0x7fea5409e910 2024-09-18T12:42:49.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:49.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4802f990 con 0x7fea5409e910 2024-09-18T12:42:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea50ff9640 10 monclient: dump: 2024-09-18T12:42:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.041 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea5b246640 10 monclient: _renew_subs 2024-09-18T12:42:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea52ffd640 4 set_mon_vals no callback set 2024-09-18T12:42:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea5b246640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.013+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea541347d0 con 0x7fea5409e910 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea5b246640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea5b246640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 0x7fea54136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea5b246640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea54063740 con 0x7fea54133100 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fea48039040 con 0x7fea5409e910 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 0x7fea54136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient: get_auth_request con 0x7fea54133100 auth_method 0 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea50ff9640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea380384c0 0x7fea3803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 10 monclient(hunting): handle_auth_done global_id 194358 payload 824 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fea4806b910 con 0x7fea5409e910 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea53fff640 1 -- 172.21.15.116:0/3932545344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea54136370 con 0x7fea54133100 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea597e3640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea380384c0 0x7fea3803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.014+0000 7fea597e3640 10 monclient: get_auth_request con 0x7fea380384c0 auth_method 0 2024-09-18T12:42:49.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.016+0000 7fea53fff640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 msgr2=0x7fea54136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.016+0000 7fea53fff640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 0x7fea54136f80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fea4000fed0 tx=0x7fea40009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.016+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4000de60 con 0x7fea54133100 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.016+0000 7fea50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.016+0000 7fea50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.016+0000 7fea50ff9640 10 monclient: dump: 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fea400070d0 con 0x7fea54133100 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea50ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 msgr2=0x7fea54136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea50ff9640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 0x7fea54136f80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fea4000fed0 tx=0x7fea40009040 comp rx=0 tx=0).stop 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 msgr2=0x7fea54136f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea50ff9640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea54133100 0x7fea54136f80 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fea4000fed0 tx=0x7fea40009040 comp rx=0 tx=0).stop 2024-09-18T12:42:49.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.017+0000 7fea597e3640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea380384c0 0x7fea3803a970 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fea5412cf50 tx=0x7fea4c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea5b246640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea5b246640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 0x7fea541378c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea5b246640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea541334d0 con 0x7fea541374c0 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea53fff640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 0x7fea541378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea53fff640 10 monclient: get_auth_request con 0x7fea541374c0 auth_method 0 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea53fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.022+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.023+0000 7fea53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.023+0000 7fea53fff640 10 monclient(hunting): handle_auth_done global_id 194361 payload 824 2024-09-18T12:42:49.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.023+0000 7fea53fff640 1 -- 172.21.15.116:0/3932545344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fea5404bf60 con 0x7fea541374c0 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea53fff640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 msgr2=0x7fea541378c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea53fff640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 0x7fea541378c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea40017000 tx=0x7fea40018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40013960 con 0x7fea541374c0 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 10 monclient: dump: 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.049 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fea40013ac0 con 0x7fea541374c0 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 msgr2=0x7fea541378c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 0x7fea541378c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea40017000 tx=0x7fea40018000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 msgr2=0x7fea541378c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:49.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.025+0000 7fea50ff9640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea541374c0 0x7fea541378c0 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea40017000 tx=0x7fea40018000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 172, 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:42:49.051 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:49.052 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:49.053 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:49.054 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.055 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.056 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:49.057 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.062 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:49.063 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:49.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea380384c0 msgr2=0x7fea3803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fea380384c0 0x7fea3803a970 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fea5412cf50 tx=0x7fea4c009040 comp rx=0 tx=0).stop 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 10 monclient: shutdown 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea5409e910 msgr2=0x7fea541303c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 --2- 172.21.15.116:0/3932545344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea5409e910 0x7fea541303c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea48031000 tx=0x7fea48032000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea597e3640 1 -- 172.21.15.116:0/3932545344 reap_dead start 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea597e3640 1 -- 172.21.15.116:0/3932545344 reap_dead start 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 shutdown_connections 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 >> 172.21.15.116:0/3932545344 conn(0x7fea54067d30 msgr2=0x7fea540a3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 shutdown_connections 2024-09-18T12:42:49.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.026+0000 7fea5b246640 1 -- 172.21.15.116:0/3932545344 wait complete. 2024-09-18T12:42:49.066 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:42:49.066 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:42:49.066 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.159+0000 7f4c0679d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.159+0000 7f4c0679d640 10 monclient: build_initial_monmap 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.159+0000 7f4c0679d640 10 monclient: monmap: 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:49.159857+0000 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:49.159857+0000 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:49.161 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.159+0000 7f4c0679d640 1 Processor -- start 2024-09-18T12:42:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.159+0000 7f4c0679d640 1 -- start start 2024-09-18T12:42:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.159+0000 7f4c0679d640 10 monclient: init 2024-09-18T12:42:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T12:42:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0009c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c000a1ef0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c0009abd0 addr v1:172.21.15.144:6789/0 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4c00136d00 0x7f4c001390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient(hunting): picked mon.noname-a con 0x7f4c00136d00 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0006f010 con 0x7f4c0009abd0 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0009d9e0 con 0x7f4c00136d00 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0009dcc0 con 0x7f4c000a1ef0 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c0679d640 10 monclient(hunting): authenticate will time out at 1796.481448s 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4c00136d00 0x7f4c001390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4c00136d00 0x7f4c001390e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54654/0 (socket says 172.21.15.116:54654) 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0009c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41448/0 (socket says 172.21.15.116:41448) 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 1 -- 172.21.15.116:0/1172107987 learned_addr learned my addr 172.21.15.116:0/1172107987 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 10 monclient(hunting): get_auth_request con 0x7f4c00136d00 auth_method 0 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): get_auth_request con 0x7f4c000a1ef0 auth_method 0 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient(hunting): handle_auth_done global_id 214124 payload 274 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 1 -- 172.21.15.116:0/1172107987 >> v2:172.21.15.116:3300/0 conn(0x7f4c00136d00 msgr2=0x7f4c001390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 1 --2- 172.21.15.116:0/1172107987 >> v2:172.21.15.116:3300/0 conn(0x7f4c00136d00 0x7f4c001390e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 1 -- 172.21.15.116:0/1172107987 >> v1:172.21.15.144:6789/0 conn(0x7f4c0009abd0 legacy=0x7f4c0009afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient: _finish_hunting 0 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 1 monclient: found mon.noname-b 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4bfffff640 1 -- 172.21.15.116:0/1172107987 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c000a22c0 con 0x7f4c000a1ef0 2024-09-18T12:42:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.160+0000 7f4c04d3a640 1 --2- 172.21.15.116:0/1172107987 >> v2:172.21.15.116:3300/0 conn(0x7f4c00136d00 0x7f4c001390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfffff640 1 -- 172.21.15.116:0/1172107987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 msgr2=0x7f4c0009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfffff640 1 --2- 172.21.15.116:0/1172107987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0009c6a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009850 tx=0x7f4bf00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53383e6dd635b1c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 1 -- 172.21.15.116:0/1172107987 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf002ae60 con 0x7f4c000a1ef0 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: dump: 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: _finish_auth 0 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 1 -- 172.21.15.116:0/1172107987 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf0032440 con 0x7f4c000a1ef0 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 1 -- 172.21.15.116:0/1172107987 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf00327e0 con 0x7f4c000a1ef0 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4c0679d640 5 monclient: authenticate success, global_id 214124 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4c0679d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4c0679d640 4 set_mon_vals no callback set 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.163+0000 7f4bfeffd640 10 monclient: dump: 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 10 monclient: shutdown 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 -- 172.21.15.116:0/1172107987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 msgr2=0x7f4c0009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 --2- 172.21.15.116:0/1172107987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0009c6a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009850 tx=0x7f4bf00090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 -- 172.21.15.116:0/1172107987 shutdown_connections 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 --2- 172.21.15.116:0/1172107987 >> v2:172.21.15.116:3300/0 conn(0x7f4c00136d00 0x7f4c001390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 --2- 172.21.15.116:0/1172107987 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0009c6a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 -- 172.21.15.116:0/1172107987 >> 172.21.15.116:0/1172107987 conn(0x7f4c000678a0 msgr2=0x7f4c000a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 -- 172.21.15.116:0/1172107987 shutdown_connections 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 1 -- 172.21.15.116:0/1172107987 wait complete. 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 10 monclient: build_initial_monmap 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.164+0000 7f4c0679d640 10 monclient: monmap: 2024-09-18T12:42:49.177 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 Processor -- start 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 -- start start 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient: init 2024-09-18T12:42:49.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c0009abd0 0x7f4c001413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): picked mon.noname-a con 0x7f4c0009abd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0012ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): picked mon.noname-c con 0x7f4c000a1ef0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c00136d00 0x7f4c001337e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): picked mon.noname-b con 0x7f4c00136d00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0009b910 con 0x7f4c0009abd0 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0012fcc0 con 0x7f4c00136d00 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c00135f40 con 0x7f4c000a1ef0 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4c0679d640 10 monclient(hunting): authenticate will time out at 1796.486871s 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bfffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c0009abd0 0x7f4c001413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bfffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c0009abd0 0x7f4c001413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54658/0 (socket says 172.21.15.116:54658) 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bfffff640 1 -- 172.21.15.116:0/1927776432 learned_addr learned my addr 172.21.15.116:0/1927776432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bfffff640 10 monclient(hunting): get_auth_request con 0x7f4c0009abd0 auth_method 0 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bfffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.165+0000 7f4bff7fe640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0012ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bff7fe640 10 monclient(hunting): get_auth_request con 0x7f4c000a1ef0 auth_method 0 2024-09-18T12:42:49.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4c04d3a640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c00136d00 0x7f4c001337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4c04d3a640 10 monclient(hunting): get_auth_request con 0x7f4c00136d00 auth_method 0 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4c04d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4c04d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4c04d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 10 monclient(hunting): handle_auth_done global_id 194367 payload 824 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 msgr2=0x7f4c0012ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c000a1ef0 0x7f4c0012ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c00136d00 msgr2=0x7f4c001337e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c00136d00 0x7f4c001337e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 10 monclient: _finish_hunting 0 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 1 monclient: found mon.noname-a 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4bfffff640 1 -- 172.21.15.116:0/1927776432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf8009ae0 con 0x7f4c0009abd0 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.166+0000 7f4c04d3a640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c00136d00 0x7f4c001337e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bfffff640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c0009abd0 msgr2=0x7f4c001413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bfffff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c0009abd0 0x7f4c001413f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009850 tx=0x7f4bf0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0031d40 con 0x7f4c0009abd0 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: dump: 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: _finish_auth 0 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:49.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf003d440 con 0x7f4c0009abd0 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4c0679d640 5 monclient: authenticate success, global_id 194367 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4c0679d640 10 monclient: _renew_subs 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4c0679d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf0009390 con 0x7f4c0009abd0 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0031ea0 con 0x7f4c0009abd0 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bf7fff640 10 monclient: dump: 2024-09-18T12:42:49.185 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.186 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.186 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.186 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4bfe7fc640 4 set_mon_vals no callback set 2024-09-18T12:42:49.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4c0679d640 10 monclient: _renew_subs 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4c0679d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.168+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c00135370 con 0x7f4c0009abd0 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4bf0039040 con 0x7f4c0009abd0 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4c0679d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4c0679d640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 0x7f4c00133d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4c0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c00065e00 con 0x7f4c0012f1b0 2024-09-18T12:42:49.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bf7fff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4be00384c0 0x7f4be003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f4bf006bf70 con 0x7f4c0009abd0 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 0x7f4c00133d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient: get_auth_request con 0x7f4c0012f1b0 auth_method 0 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4c04d3a640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4be00384c0 0x7f4be003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_done global_id 214130 payload 824 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4bff7fe640 1 -- 172.21.15.116:0/1927776432 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c0004bf60 con 0x7f4c0012f1b0 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.169+0000 7f4c04d3a640 10 monclient: get_auth_request con 0x7f4be00384c0 auth_method 0 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bff7fe640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 msgr2=0x7f4c00133d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bff7fe640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 0x7f4c00133d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e7b0 tx=0x7f4be8006f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be800cac0 con 0x7f4c0012f1b0 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 10 monclient: dump: 2024-09-18T12:42:49.190 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4c04d3a640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4be00384c0 0x7f4be003a970 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009fd0 tx=0x7f4bf800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4be800cc20 con 0x7f4c0012f1b0 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:49.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 msgr2=0x7f4c00133d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 0x7f4c00133d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e7b0 tx=0x7f4be8006f50 comp rx=0 tx=0).stop 2024-09-18T12:42:49.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 msgr2=0x7f4c00133d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:49.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.172+0000 7f4bf7fff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c0012f1b0 0x7f4c00133d20 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e7b0 tx=0x7f4be8006f50 comp rx=0 tx=0).stop 2024-09-18T12:42:49.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4c0679d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4c0679d640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 0x7f4c00099650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4c0679d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0012f580 con 0x7f4c00134260 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 0x7f4c00099650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient: get_auth_request con 0x7f4c00134260 auth_method 0 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 10 monclient(hunting): handle_auth_done global_id 214133 payload 824 2024-09-18T12:42:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.178+0000 7f4bff7fe640 1 -- 172.21.15.116:0/1927776432 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c00134660 con 0x7f4c00134260 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bff7fe640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 msgr2=0x7f4c00099650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bff7fe640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 0x7f4c00099650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e7b0 tx=0x7f4be8015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be8021a10 con 0x7f4c00134260 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 10 monclient: dump: 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.195 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f4be8029410 con 0x7f4c00134260 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 msgr2=0x7f4c00099650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 0x7f4c00099650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e7b0 tx=0x7f4be8015000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 msgr2=0x7f4c00099650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.180+0000 7f4bf7fff640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c00134260 0x7f4c00099650 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4be800e7b0 tx=0x7f4be8015000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 172, 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:49.197 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.199 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.200 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:49.200 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:49.200 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:49.200 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:49.201 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:49.201 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:49.201 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:49.201 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:49.201 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:49.202 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:49.202 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:49.202 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:49.202 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:49.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:49.202 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:49.202 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:49.203 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:49.203 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.203 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.203 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.203 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.203 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.203 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:49.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.205 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:49.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:49.205 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.205 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.206 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:49.206 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.207 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:49.207 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.207 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.207 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:49.208 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:49.208 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.208 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.208 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.210 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.211 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.212 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:49.213 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:49.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4be00384c0 msgr2=0x7f4be003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4be00384c0 0x7f4be003a970 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009fd0 tx=0x7f4bf800b040 comp rx=0 tx=0).stop 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 10 monclient: shutdown 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c0009abd0 msgr2=0x7f4c001413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 --2- 172.21.15.116:0/1927776432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c0009abd0 0x7f4c001413f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0009850 tx=0x7f4bf0032000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c04d3a640 1 -- 172.21.15.116:0/1927776432 reap_dead start 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c04d3a640 1 -- 172.21.15.116:0/1927776432 reap_dead start 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 shutdown_connections 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 >> 172.21.15.116:0/1927776432 conn(0x7f4c000678a0 msgr2=0x7f4c000942c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 shutdown_connections 2024-09-18T12:42:49.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.181+0000 7f4c0679d640 1 -- 172.21.15.116:0/1927776432 wait complete. 2024-09-18T12:42:49.215 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:42:49.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:42:49.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:42:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.308+0000 7f9a4719d640 10 monclient: get_monmap_and_config 2024-09-18T12:42:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.308+0000 7f9a4719d640 10 monclient: build_initial_monmap 2024-09-18T12:42:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.308+0000 7f9a4719d640 10 monclient: monmap: 2024-09-18T12:42:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:42:49.309078+0000 2024-09-18T12:42:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:42:49.309078+0000 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:42:49.310 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 1 Processor -- start 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 1 -- start start 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient: init 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9a4009c860 0x7f9a4009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a4009c860 addr v2:172.21.15.144:3300/0 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a4009aba0 addr v1:172.21.15.116:6789/0 2024-09-18T12:42:49.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a4009afa0 0x7f9a4009df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a4009afa0 addr v2:172.21.15.116:3300/0 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4006a320 con 0x7f9a4009aba0 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a40068a20 con 0x7f9a4009afa0 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a40068e60 con 0x7f9a4009c860 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4719d640 10 monclient(hunting): authenticate will time out at 1796.630693s 2024-09-18T12:42:49.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4573a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a4009afa0 0x7f9a4009df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4573a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a4009afa0 0x7f9a4009df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54664/0 (socket says 172.21.15.116:54664) 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4573a640 1 -- 172.21.15.116:0/3765021089 learned_addr learned my addr 172.21.15.116:0/3765021089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4573a640 10 monclient(hunting): get_auth_request con 0x7f9a4009afa0 auth_method 0 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4573a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4573a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.309+0000 7f9a4573a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 1 --2- 172.21.15.116:0/3765021089 >> v2:172.21.15.144:3300/0 conn(0x7f9a4009c860 0x7f9a4009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): get_auth_request con 0x7f9a4009c860 auth_method 0 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient(hunting): handle_auth_done global_id 174514 payload 274 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 1 -- 172.21.15.116:0/3765021089 >> v2:172.21.15.116:3300/0 conn(0x7f9a4009afa0 msgr2=0x7f9a4009df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 1 --2- 172.21.15.116:0/3765021089 >> v2:172.21.15.116:3300/0 conn(0x7f9a4009afa0 0x7f9a4009df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 1 -- 172.21.15.116:0/3765021089 >> v1:172.21.15.116:6789/0 conn(0x7f9a4009aba0 legacy=0x7f9a4009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient: _finish_hunting 0 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 1 monclient: found mon.noname-c 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:42:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.310+0000 7f9a44f39640 1 -- 172.21.15.116:0/3765021089 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4006a040 con 0x7f9a4009c860 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a44f39640 1 -- 172.21.15.116:0/3765021089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 msgr2=0x7f9a4009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a44f39640 1 --2- 172.21.15.116:0/3765021089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 0x7f9a4009cc30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a40071740 tx=0x7f9a380090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e966d0679460f5f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 1 -- 172.21.15.116:0/3765021089 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3802adf0 con 0x7f9a4009c860 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: dump: 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 1 -- 172.21.15.116:0/3765021089 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38032440 con 0x7f9a4009c860 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a4719d640 5 monclient: authenticate success, global_id 174514 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a4719d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a4719d640 10 monclient: get_monmap_and_config success 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 1 -- 172.21.15.116:0/3765021089 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38032820 con 0x7f9a4009c860 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a4719d640 4 set_mon_vals no callback set 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.313+0000 7f9a3f7fe640 10 monclient: dump: 2024-09-18T12:42:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 10 monclient: shutdown 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 -- 172.21.15.116:0/3765021089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 msgr2=0x7f9a4009cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 --2- 172.21.15.116:0/3765021089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 0x7f9a4009cc30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a40071740 tx=0x7f9a380090f0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 -- 172.21.15.116:0/3765021089 shutdown_connections 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 --2- 172.21.15.116:0/3765021089 >> v2:172.21.15.116:3300/0 conn(0x7f9a4009afa0 0x7f9a4009df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 --2- 172.21.15.116:0/3765021089 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 0x7f9a4009cc30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 -- 172.21.15.116:0/3765021089 >> 172.21.15.116:0/3765021089 conn(0x7f9a400a2e40 msgr2=0x7f9a400a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 -- 172.21.15.116:0/3765021089 shutdown_connections 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 -- 172.21.15.116:0/3765021089 wait complete. 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 10 monclient: build_initial_monmap 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 10 monclient: monmap: 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:42:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.314+0000 7f9a4719d640 1 Processor -- start 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 1 -- start start 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient: init 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient: _reopen_session rank -1 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a4009aba0 0x7f9a40141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a4009aba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a4009afa0 0x7f9a4013dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a4009afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 0x7f9a4013e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a4009c860 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a401226b0 con 0x7f9a4009aba0 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4011fec0 con 0x7f9a4009c860 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a40121170 con 0x7f9a4009afa0 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): _renew_subs 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4719d640 10 monclient(hunting): authenticate will time out at 1796.636644s 2024-09-18T12:42:49.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a4009aba0 0x7f9a40141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a4009aba0 0x7f9a40141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54678/0 (socket says 172.21.15.116:54678) 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a4009afa0 0x7f9a4013dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 1 -- 172.21.15.116:0/4204077222 learned_addr learned my addr 172.21.15.116:0/4204077222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 10 monclient(hunting): get_auth_request con 0x7f9a4009aba0 auth_method 0 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request con 0x7f9a4009afa0 auth_method 0 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 0x7f9a4013e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a44f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 10 monclient(hunting): get_auth_request con 0x7f9a4009c860 auth_method 0 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.315+0000 7f9a4573a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 10 monclient(hunting): handle_auth_done global_id 194373 payload 824 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a4009afa0 msgr2=0x7f9a4013dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a4009afa0 0x7f9a4013dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 msgr2=0x7f9a4013e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a4009c860 0x7f9a4013e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 10 monclient: _finish_hunting 0 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 1 monclient: found mon.noname-a 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a44f39640 1 -- 172.21.15.116:0/4204077222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a38009390 con 0x7f9a4009aba0 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.316+0000 7f9a3ffff640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9a4009afa0 0x7f9a4013dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a44f39640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a4009aba0 msgr2=0x7f9a40141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a44f39640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a4009aba0 0x7f9a40141320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a38035040 tx=0x7f9a38032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38031b10 con 0x7f9a4009aba0 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: dump: 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: _finish_auth 0 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38031c70 con 0x7f9a4009aba0 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a4719d640 5 monclient: authenticate success, global_id 194373 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a4719d640 10 monclient: _renew_subs 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a4719d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a40122060 con 0x7f9a4009aba0 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3803d440 con 0x7f9a4009aba0 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3cff9640 10 monclient: dump: 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a3effd640 4 set_mon_vals no callback set 2024-09-18T12:42:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a4719d640 10 monclient: _renew_subs 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a4719d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.318+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4013ea50 con 0x7f9a4009aba0 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a4719d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a4719d640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 0x7f9a4011f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a401224e0 con 0x7f9a40141860 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9a38039040 con 0x7f9a4009aba0 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 0x7f9a4011f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient: get_auth_request con 0x7f9a40141860 auth_method 0 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3cff9640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a20038490 0x7f9a2003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9a3806c160 con 0x7f9a4009aba0 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_done global_id 174520 payload 824 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a3ffff640 1 -- 172.21.15.116:0/4204077222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a4004bf60 con 0x7f9a40141860 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a4573a640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a20038490 0x7f9a2003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.319+0000 7f9a4573a640 10 monclient: get_auth_request con 0x7f9a20038490 auth_method 0 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3ffff640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 msgr2=0x7f9a4011f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3ffff640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 0x7f9a4011f210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120230 tx=0x7f9a30002630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30014680 con 0x7f9a40141860 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 10 monclient: dump: 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a4573a640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a20038490 0x7f9a2003a940 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120c80 tx=0x7f9a28008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9a300147e0 con 0x7f9a40141860 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 msgr2=0x7f9a4011f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 0x7f9a4011f210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120230 tx=0x7f9a30002630 comp rx=0 tx=0).stop 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 msgr2=0x7f9a4011f210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.322+0000 7f9a3cff9640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a40141860 0x7f9a4011f210 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120230 tx=0x7f9a30002630 comp rx=0 tx=0).stop 2024-09-18T12:42:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a4719d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:42:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a4719d640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 0x7f9a40029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:42:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a4719d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:42:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a40065e00 con 0x7f9a400298c0 2024-09-18T12:42:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 0x7f9a40029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:42:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient: get_auth_request con 0x7f9a400298c0 auth_method 0 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 10 monclient(hunting): handle_auth_done global_id 174523 payload 824 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.328+0000 7f9a3ffff640 1 -- 172.21.15.116:0/4204077222 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9a400a1b60 con 0x7f9a400298c0 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3ffff640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 msgr2=0x7f9a40029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3ffff640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 0x7f9a40029cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120230 tx=0x7f9a30019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3002b480 con 0x7f9a400298c0 2024-09-18T12:42:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 10 monclient: dump: 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:42:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f9a30026070 con 0x7f9a400298c0 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 msgr2=0x7f9a40029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 0x7f9a40029cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120230 tx=0x7f9a30019000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 msgr2=0x7f9a40029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a3cff9640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9a400298c0 0x7f9a40029cc0 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120230 tx=0x7f9a30019000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:49.344 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 172, 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 5, 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.345 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.346 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:42:49.347 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:42:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.349 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.350 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:42:49.351 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:42:49.353 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.354 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.355 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.356 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:42:49.357 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a20038490 msgr2=0x7f9a2003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9a20038490 0x7f9a2003a940 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9a40120c80 tx=0x7f9a28008040 comp rx=0 tx=0).stop 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 10 monclient: shutdown 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a4009aba0 msgr2=0x7f9a40141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 1 --2- 172.21.15.116:0/4204077222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a4009aba0 0x7f9a40141320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a38035040 tx=0x7f9a38032000 comp rx=0 tx=0).stop 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4573a640 1 -- 172.21.15.116:0/4204077222 reap_dead start 2024-09-18T12:42:49.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4573a640 1 -- 172.21.15.116:0/4204077222 reap_dead start 2024-09-18T12:42:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 shutdown_connections 2024-09-18T12:42:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.331+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 >> 172.21.15.116:0/4204077222 conn(0x7f9a400a2e40 msgr2=0x7f9a40098db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:42:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.332+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 shutdown_connections 2024-09-18T12:42:49.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:42:49.332+0000 7f9a4719d640 1 -- 172.21.15.116:0/4204077222 wait complete. 2024-09-18T12:42:49.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:42:49.360 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:42:49.360 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'b'] 2024-09-18T12:42:49.361 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:42:49.361 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:42:49.361 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:42:49.361 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:42:49.371 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:42:49.371 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T12:42:49.371 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T12:42:49.371 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-18T12:42:49.372 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-18T12:42:49.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:43:04.373 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T12:43:04.374 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T12:43:04.374 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-18T12:43:04.374 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-18T12:43:04.374 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:43:04.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:43:04.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.476+0000 7fe518ed4640 10 monclient: get_monmap_and_config 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.476+0000 7fe518ed4640 10 monclient: build_initial_monmap 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.476+0000 7fe518ed4640 10 monclient: monmap: 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:04.477596+0000 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:04.477596+0000 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:04.478 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:04.479 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.477+0000 7fe518ed4640 1 Processor -- start 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.477+0000 7fe518ed4640 1 -- start start 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.477+0000 7fe518ed4640 10 monclient: init 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): picked mon.noname-a con 0x7fe514096e10 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): picked mon.noname-c con 0x7fe51409aaa0 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 0x7fe51412e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): picked mon.noname-b con 0x7fe514097720 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe514099620 con 0x7fe514096e10 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe514099340 con 0x7fe51409aaa0 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe514099900 con 0x7fe514097720 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe518ed4640 10 monclient(hunting): authenticate will time out at 1811.799328s 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe512d9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 msgr2=0x7fe51412e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe512d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 0x7fe51412e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58772/0 (socket says 172.21.15.116:58772) 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 learned_addr learned my addr 172.21.15.116:0/3176599781 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 10 monclient(hunting): get_auth_request con 0x7fe514096e10 auth_method 0 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 10 monclient(hunting): get_auth_request con 0x7fe51409aaa0 auth_method 0 2024-09-18T12:43:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.478+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 msgr2=0x7fe51412e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe512d9d640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 0x7fe51412e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 10 monclient(hunting): get_auth_request con 0x7fe514096e10 auth_method 0 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 10 monclient(hunting): get_auth_request con 0x7fe51409aaa0 auth_method 0 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:04.679+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 msgr2=0x7fe51412e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe512d9d640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 0x7fe51412e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 10 monclient(hunting): get_auth_request con 0x7fe514096e10 auth_method 0 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 10 monclient(hunting): get_auth_request con 0x7fe51409aaa0 auth_method 0 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.080+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 msgr2=0x7fe51412e5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe512d9d640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 0x7fe51412e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 10 monclient(hunting): get_auth_request con 0x7fe514096e10 auth_method 0 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 10 monclient(hunting): get_auth_request con 0x7fe51409aaa0 auth_method 0 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:05.881+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:43:07.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 10 monclient(hunting): tick 2024-09-18T12:43:07.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 monclient(hunting): continuing hunt 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 msgr2=0x7fe51412e5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe514097720 0x7fe51412e5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 msgr2=0x7fe5140971e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.144:3300/0 conn(0x7fe51409aaa0 0x7fe5140971e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 -- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 msgr2=0x7fe51409a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 --2- 172.21.15.116:0/3176599781 >> v2:172.21.15.116:3300/0 conn(0x7fe514096e10 0x7fe51409a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 10 monclient: _add_conns ranks=[1,0] 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4f4003670 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4f4007390 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:07.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 -- 172.21.15.116:0/3176599781 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe514097af0 con 0x7fe4f4007390 2024-09-18T12:43:07.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 1 -- 172.21.15.116:0/3176599781 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f400c0c0 con 0x7fe4f4003670 2024-09-18T12:43:07.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.478+0000 7fe503fff640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:07.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.481+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> v1:172.21.15.116:6789/0 conn(0x7fe4f4007390 legacy=0x7fe4f4009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:07.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:07.481+0000 7fe51259c640 1 -- 172.21.15.116:0/3176599781 >> v1:172.21.15.144:6789/0 conn(0x7fe4f4003670 legacy=0x7fe4f4005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520597284 0 0) 0x7fe514097af0 con 0x7fe4f4007390 2024-09-18T12:43:09.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): my global_id is 194376 2024-09-18T12:43:09.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc0034e0 con 0x7fe4f4007390 2024-09-18T12:43:09.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1564469635 0 0) 0x7fe4fc0034e0 con 0x7fe4f4007390 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 >> v1:172.21.15.144:6789/0 conn(0x7fe4f4003670 legacy=0x7fe4f4005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f400d260 con 0x7fe4f4007390 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fe50800d480 con 0x7fe4f4007390 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:43:09.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: dump: 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 >> v1:172.21.15.116:6789/0 conn(0x7fe4f4007390 legacy=0x7fe4f4009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 reap_dead start 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc003a00 0x7fe4fc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): picked mon.b con 0x7fe4fc003a00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc004780 0x7fe4fc004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): picked mon.c con 0x7fe4fc004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc005380 0x7fe4fc00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): picked mon.a con 0x7fe4fc005380 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00c2b0 con 0x7fe4fc005380 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00c420 con 0x7fe4fc003a00 2024-09-18T12:43:09.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00c5e0 con 0x7fe4fc004780 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51159a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe511d9b640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc004780 msgr2=0x7fe4fc004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe511d9b640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc004780 0x7fe4fc004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe512d9d640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc005380 0x7fe4fc00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe51259c640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc003a00 0x7fe4fc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe512d9d640 10 monclient(hunting): get_auth_request con 0x7fe4fc005380 auth_method 0 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe512d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe512d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.385+0000 7fe512d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe51259c640 10 monclient(hunting): get_auth_request con 0x7fe4fc003a00 auth_method 0 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe51259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe51259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe51259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe51259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe51259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe51259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 10 monclient(hunting): handle_auth_done global_id 194376 payload 293 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc004780 msgr2=0x7fe4fc004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc004780 0x7fe4fc004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc003a00 msgr2=0x7fe4fc003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc003a00 0x7fe4fc003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 1 monclient: found mon.a 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc0034e0 con 0x7fe4fc005380 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe512d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe518ed4640 5 monclient: authenticate success, global_id 194376 2024-09-18T12:43:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.386+0000 7fe518ed4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe512d9d640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc005380 msgr2=0x7fe4fc00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe512d9d640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc005380 0x7fe4fc00ba30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe51409aea0 tx=0x7fe50c007530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=386671ccb9da901 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c02aaf0 con 0x7fe4fc005380 2024-09-18T12:43:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 10 monclient: dump: 2024-09-18T12:43:09.401 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe518ed4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50c02ac90 con 0x7fe4fc005380 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 1 -- 172.21.15.116:0/3176599781 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0336c0 con 0x7fe4fc005380 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe518ed4640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe518ed4640 4 set_mon_vals no callback set 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.389+0000 7fe51159a640 10 monclient: dump: 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 10 monclient: shutdown 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 -- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc005380 msgr2=0x7fe4fc00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc005380 0x7fe4fc00ba30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe51409aea0 tx=0x7fe50c007530 comp rx=0 tx=0).stop 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 -- 172.21.15.116:0/3176599781 shutdown_connections 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc005380 0x7fe4fc00ba30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc004780 0x7fe4fc004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 --2- 172.21.15.116:0/3176599781 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc003a00 0x7fe4fc003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 -- 172.21.15.116:0/3176599781 >> 172.21.15.116:0/3176599781 conn(0x7fe5140672b0 msgr2=0x7fe5140676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 -- 172.21.15.116:0/3176599781 shutdown_connections 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.390+0000 7fe518ed4640 1 -- 172.21.15.116:0/3176599781 wait complete. 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.391+0000 7fe518ed4640 10 monclient: build_initial_monmap 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.391+0000 7fe518ed4640 10 monclient: monmap: 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.391+0000 7fe518ed4640 1 Processor -- start 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.391+0000 7fe518ed4640 1 -- start start 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.391+0000 7fe518ed4640 10 monclient: init 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 0x7fe514117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): picked mon.noname-a con 0x7fe4fc003a00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc004780 0x7fe5141179c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): picked mon.noname-b con 0x7fe4fc004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 0x7fe514117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 0x7fe514117480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52908/0 (socket says 172.21.15.116:52908) 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc004780 0x7fe5141179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc004780 0x7fe5141179c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:51388/0 (socket says 172.21.15.116:51388) 2024-09-18T12:43:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc005380 0x7fe514117f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): picked mon.noname-c con 0x7fe4fc005380 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 1 -- 172.21.15.116:0/4117782364 learned_addr learned my addr 172.21.15.116:0/4117782364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51409b050 con 0x7fe4fc003a00 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51406ec50 con 0x7fe4fc004780 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe514119960 con 0x7fe4fc005380 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe518ed4640 10 monclient(hunting): authenticate will time out at 1816.713815s 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 10 monclient(hunting): get_auth_request con 0x7fe4fc003a00 auth_method 0 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 10 monclient(hunting): get_auth_request con 0x7fe4fc004780 auth_method 0 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe512d9d640 1 -- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc005380 msgr2=0x7fe514117f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe512d9d640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc005380 0x7fe514117f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe51259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.392+0000 7fe511d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 10 monclient(hunting): handle_auth_done global_id 194379 payload 824 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 1 -- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc005380 msgr2=0x7fe514117f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc005380 0x7fe514117f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 1 -- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc004780 msgr2=0x7fe5141179c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc004780 0x7fe5141179c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 1 monclient: found mon.noname-a 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe51259c640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51411a550 con 0x7fe4fc003a00 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.393+0000 7fe511d9b640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc004780 0x7fe5141179c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe51259c640 1 -- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 msgr2=0x7fe514117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe51259c640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 0x7fe514117480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe504017950 tx=0x7fe504008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 1 -- 172.21.15.116:0/4117782364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50400c8c0 con 0x7fe4fc003a00 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: dump: 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 1 -- 172.21.15.116:0/4117782364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50401c070 con 0x7fe4fc003a00 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe518ed4640 5 monclient: authenticate success, global_id 194379 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe518ed4640 10 monclient: _renew_subs 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe518ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51406e610 con 0x7fe4fc003a00 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 1 -- 172.21.15.116:0/4117782364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe504015da0 con 0x7fe4fc003a00 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe5027fc640 10 monclient: dump: 2024-09-18T12:43:09.411 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe510d99640 4 set_mon_vals no callback set 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe518ed4640 10 monclient: _renew_subs 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe518ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.395+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5141184d0 con 0x7fe4fc003a00 2024-09-18T12:43:09.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.396+0000 7fe5027fc640 1 -- 172.21.15.116:0/4117782364 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe50401e030 con 0x7fe4fc003a00 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.396+0000 7fe518ed4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.396+0000 7fe518ed4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.396+0000 7fe518ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.396+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51404bf60 con 0x7fe4fc003a00 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.396+0000 7fe5027fc640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe4f4038f30 0x7fe4f403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.397+0000 7fe5027fc640 1 -- 172.21.15.116:0/4117782364 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe50404cd30 con 0x7fe4fc003a00 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.397+0000 7fe511d9b640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe4f4038f30 0x7fe4f403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.397+0000 7fe511d9b640 10 monclient: get_auth_request con 0x7fe4f4038f30 auth_method 0 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.399+0000 7fe511d9b640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe4f4038f30 0x7fe4f403b3e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe50800d310 tx=0x7fe508008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.401+0000 7fe5027fc640 1 -- 172.21.15.116:0/4117782364 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe504029580 con 0x7fe4fc003a00 2024-09-18T12:43:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.401+0000 7fe5027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:09.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.401+0000 7fe5027fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.615+0000 7fe518ed4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:43:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.615+0000 7fe518ed4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.615+0000 7fe518ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.615+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe51406e610 con 0x7fe4fc003a00 2024-09-18T12:43:09.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.615+0000 7fe5027fc640 1 -- 172.21.15.116:0/4117782364 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fe51406e610 con 0x7fe4fc003a00 2024-09-18T12:43:09.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.615+0000 7fe5027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:09.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.615+0000 7fe5027fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:09.617 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":176,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:43:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe4f4038f30 msgr2=0x7fe4f403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe4f4038f30 0x7fe4f403b3e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe50800d310 tx=0x7fe508008040 comp rx=0 tx=0).stop 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 10 monclient: shutdown 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 msgr2=0x7fe514117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 0x7fe514117480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe504017950 tx=0x7fe504008040 comp rx=0 tx=0).stop 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 shutdown_connections 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe4f4038f30 0x7fe4f403b3e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe4fc005380 0x7fe514117f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4fc004780 0x7fe5141179c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 --2- 172.21.15.116:0/4117782364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4fc003a00 0x7fe514117480 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 >> 172.21.15.116:0/4117782364 conn(0x7fe5140672b0 msgr2=0x7fe51409ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 shutdown_connections 2024-09-18T12:43:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.617+0000 7fe518ed4640 1 -- 172.21.15.116:0/4117782364 wait complete. 2024-09-18T12:43:09.629 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:43:09.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:43:09.629 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.722+0000 7fa5b8c2b640 10 monclient: get_monmap_and_config 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.722+0000 7fa5b8c2b640 10 monclient: build_initial_monmap 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.722+0000 7fa5b8c2b640 10 monclient: monmap: 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:09.723003+0000 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:09.723003+0000 2024-09-18T12:43:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:09.724 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.722+0000 7fa5b8c2b640 1 Processor -- start 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.722+0000 7fa5b8c2b640 1 -- start start 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.722+0000 7fa5b8c2b640 10 monclient: init 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5b4131ab0 0x7fa5b4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b4131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa5b409e910 0x7fa5b409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b409e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5b409f250 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b409faf0 con 0x7fa5b409f250 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b409fd30 con 0x7fa5b409e910 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b40a1110 con 0x7fa5b4131ab0 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b8c2b640 10 monclient(hunting): authenticate will time out at 1817.044581s 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa5b409e910 0x7fa5b409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa5b4131ab0 0x7fa5b4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa5b409e910 0x7fa5b409ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52922/0 (socket says 172.21.15.116:52922) 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 1 -- 172.21.15.116:0/616292900 learned_addr learned my addr 172.21.15.116:0/616292900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 10 monclient(hunting): get_auth_request con 0x7fa5b409e910 auth_method 0 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): get_auth_request con 0x7fa5b4131ab0 auth_method 0 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.723+0000 7fa5b259c640 10 monclient(hunting): handle_auth_done global_id 174529 payload 274 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 1 -- 172.21.15.116:0/616292900 >> v2:172.21.15.116:3300/0 conn(0x7fa5b409e910 msgr2=0x7fa5b409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 1 --2- 172.21.15.116:0/616292900 >> v2:172.21.15.116:3300/0 conn(0x7fa5b409e910 0x7fa5b409ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 1 -- 172.21.15.116:0/616292900 >> v1:172.21.15.116:6789/0 conn(0x7fa5b409f250 legacy=0x7fa5b4068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 1 monclient: found mon.noname-c 2024-09-18T12:43:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:43:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.724+0000 7fa5b259c640 1 -- 172.21.15.116:0/616292900 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b40a13f0 con 0x7fa5b4131ab0 2024-09-18T12:43:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.726+0000 7fa5b259c640 1 -- 172.21.15.116:0/616292900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b4131ab0 msgr2=0x7fa5b4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.726+0000 7fa5b259c640 1 --2- 172.21.15.116:0/616292900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b4131ab0 0x7fa5b4131e80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4076760 tx=0x7fa5a40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d64d054d5c18070 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 1 -- 172.21.15.116:0/616292900 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a402adf0 con 0x7fa5b4131ab0 2024-09-18T12:43:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: dump: 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 5 monclient: authenticate success, global_id 174529 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 1 -- 172.21.15.116:0/616292900 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4032440 con 0x7fa5b4131ab0 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 1 -- 172.21.15.116:0/616292900 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40327e0 con 0x7fa5b4131ab0 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 4 set_mon_vals no callback set 2024-09-18T12:43:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b159a640 10 monclient: dump: 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 10 monclient: shutdown 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/616292900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b4131ab0 msgr2=0x7fa5b4131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/616292900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b4131ab0 0x7fa5b4131e80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4076760 tx=0x7fa5a40090f0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/616292900 shutdown_connections 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/616292900 >> v2:172.21.15.116:3300/0 conn(0x7fa5b409e910 0x7fa5b409ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/616292900 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b4131ab0 0x7fa5b4131e80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/616292900 >> 172.21.15.116:0/616292900 conn(0x7fa5b4067d30 msgr2=0x7fa5b40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/616292900 shutdown_connections 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.727+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/616292900 wait complete. 2024-09-18T12:43:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.728+0000 7fa5b8c2b640 10 monclient: build_initial_monmap 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.728+0000 7fa5b8c2b640 10 monclient: monmap: 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:09.745 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.728+0000 7fa5b8c2b640 1 Processor -- start 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.728+0000 7fa5b8c2b640 1 -- start start 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.728+0000 7fa5b8c2b640 10 monclient: init 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b409e910 0x7fa5b411ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b409e910 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 0x7fa5b411d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b409f250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5b4131ab0 0x7fa5b411d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b4131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b409bde0 con 0x7fa5b409f250 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b409de40 con 0x7fa5b409e910 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4120630 con 0x7fa5b4131ab0 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b8c2b640 10 monclient(hunting): authenticate will time out at 1817.050116s 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b2d9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5b4131ab0 msgr2=0x7fa5b411d8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b2d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5b4131ab0 0x7fa5b411d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 0x7fa5b411d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 0x7fa5b411d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52926/0 (socket says 172.21.15.116:52926) 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 -- 172.21.15.116:0/9009624 learned_addr learned my addr 172.21.15.116:0/9009624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): get_auth_request con 0x7fa5b409f250 auth_method 0 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient(hunting): handle_auth_done global_id 194385 payload 824 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5b4131ab0 msgr2=0x7fa5b411d8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b259c640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b409e910 0x7fa5b411ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa5b4131ab0 0x7fa5b411d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b409e910 msgr2=0x7fa5b411ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa5b409e910 0x7fa5b411ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 monclient: found mon.noname-a 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.729+0000 7fa5b1d9b640 1 -- 172.21.15.116:0/9009624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a4009390 con 0x7fa5b409f250 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.731+0000 7fa5b1d9b640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 msgr2=0x7fa5b411d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.731+0000 7fa5b1d9b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 0x7fa5b411d360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ea20 tx=0x7fa59c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.731+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c00cb50 con 0x7fa5b409f250 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: dump: 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c00ccb0 con 0x7fa5b409f250 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c013470 con 0x7fa5b409f250 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5aa7fc640 10 monclient: dump: 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b8c2b640 5 monclient: authenticate success, global_id 194385 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b8c2b640 10 monclient: _renew_subs 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b8c2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b411e820 con 0x7fa5b409f250 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b0d99640 4 set_mon_vals no callback set 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b8c2b640 10 monclient: _renew_subs 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b8c2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.733+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b411f3f0 con 0x7fa5b409f250 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b8c2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 0x7fa5b4098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b8c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4063740 con 0x7fa5b411dde0 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 0x7fa5b4098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient: get_auth_request con 0x7fa5b411dde0 auth_method 0 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa59c013860 con 0x7fa5b409f250 2024-09-18T12:43:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 10 monclient(hunting): handle_auth_done global_id 194388 payload 824 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.734+0000 7fa5b259c640 1 -- 172.21.15.116:0/9009624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5b411f670 con 0x7fa5b411dde0 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.735+0000 7fa5aa7fc640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa594038490 0x7fa59403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.735+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa59c04c600 con 0x7fa5b409f250 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.735+0000 7fa5b2d9d640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa594038490 0x7fa59403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.735+0000 7fa5b2d9d640 10 monclient: get_auth_request con 0x7fa594038490 auth_method 0 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5b259c640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 msgr2=0x7fa5b4098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5b259c640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 0x7fa5b4098a70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4034000 tx=0x7fa5a4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a403b9d0 con 0x7fa5b411dde0 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 10 monclient: dump: 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.756 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa5a403bb30 con 0x7fa5b411dde0 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 msgr2=0x7fa5b4098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 0x7fa5b4098a70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4034000 tx=0x7fa5a4032000 comp rx=0 tx=0).stop 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 msgr2=0x7fa5b4098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5aa7fc640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b411dde0 0x7fa5b4098a70 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4034000 tx=0x7fa5a4032000 comp rx=0 tx=0).stop 2024-09-18T12:43:09.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.737+0000 7fa5b2d9d640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa594038490 0x7fa59403a940 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac007df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b8c2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 0x7fa5b40993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b8c2b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b41211b0 con 0x7fa5b4098fb0 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 0x7fa5b40993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient: get_auth_request con 0x7fa5b4098fb0 auth_method 0 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 10 monclient(hunting): handle_auth_done global_id 194391 payload 824 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.743+0000 7fa5b259c640 1 -- 172.21.15.116:0/9009624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa5b404bf60 con 0x7fa5b4098fb0 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.745+0000 7fa5b259c640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 msgr2=0x7fa5b40993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.745+0000 7fa5b259c640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 0x7fa5b40993b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4068720 tx=0x7fa5a403b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.745+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40448f0 con 0x7fa5b4098fb0 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.745+0000 7fa5aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.745+0000 7fa5aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.745+0000 7fa5aa7fc640 10 monclient: dump: 2024-09-18T12:43:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.745+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fa5a4044a50 con 0x7fa5b4098fb0 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5aa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5aa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 msgr2=0x7fa5b40993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5aa7fc640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 0x7fa5b40993b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4068720 tx=0x7fa5a403b000 comp rx=0 tx=0).stop 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5aa7fc640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 msgr2=0x7fa5b40993b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5aa7fc640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b4098fb0 0x7fa5b40993b0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4068720 tx=0x7fa5a403b000 comp rx=0 tx=0).stop 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:09.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 176, 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:09.764 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:09.765 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:09.766 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:09.767 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:09.768 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:09.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.770 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:09.771 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.772 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:09.773 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:09.774 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.776 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.778 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:09.778 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa594038490 msgr2=0x7fa59403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa594038490 0x7fa59403a940 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac007df0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 10 monclient: shutdown 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 msgr2=0x7fa5b411d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 0x7fa5b411d360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ea20 tx=0x7fa59c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b2d9d640 1 -- 172.21.15.116:0/9009624 reap_dead start 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 shutdown_connections 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 --2- 172.21.15.116:0/9009624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b409f250 0x7fa5b411d360 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 >> 172.21.15.116:0/9009624 conn(0x7fa5b4067d30 msgr2=0x7fa5b41310a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 shutdown_connections 2024-09-18T12:43:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.746+0000 7fa5b8c2b640 1 -- 172.21.15.116:0/9009624 wait complete. 2024-09-18T12:43:09.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:43:09.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:43:09.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.874+0000 7f798b19d640 10 monclient: get_monmap_and_config 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.874+0000 7f798b19d640 10 monclient: build_initial_monmap 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.874+0000 7f798b19d640 10 monclient: monmap: 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:09.874830+0000 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:09.874830+0000 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.875 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:09.876 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:09.876 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:09.876 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:09.876 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:09.876 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.874+0000 7f798b19d640 1 Processor -- start 2024-09-18T12:43:09.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.874+0000 7f798b19d640 1 -- start start 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.874+0000 7f798b19d640 10 monclient: init 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 0x7f7984131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient(hunting): picked mon.noname-b con 0x7f7984131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f798409e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7984136bb0 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798409faf0 con 0x7f798409e910 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798409fd30 con 0x7f7984136bb0 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79840a1110 con 0x7f7984131ab0 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:09.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f798b19d640 10 monclient(hunting): authenticate will time out at 1817.196380s 2024-09-18T12:43:09.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f7988f39640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 msgr2=0x7f7984131e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:09.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f7988f39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 0x7f7984131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:09.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f797bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f798409e910 0x7f798409ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37822/0 (socket says 172.21.15.116:37822) 2024-09-18T12:43:09.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.875+0000 7f797bfff640 1 -- 172.21.15.116:0/486254641 learned_addr learned my addr 172.21.15.116:0/486254641 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:43:09.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.877+0000 7f797bfff640 1 -- 172.21.15.116:0/486254641 >> v1:172.21.15.116:6789/0 conn(0x7f798409e910 legacy=0x7f798409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.877+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152035101 0 0) 0x7f798409faf0 con 0x7f798409e910 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.877+0000 7f797b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.877+0000 7f797b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.877+0000 7f797b7fe640 10 monclient(hunting): my global_id is 194394 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.877+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f796c0034e0 con 0x7f798409e910 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.877+0000 7f798973a640 1 -- 172.21.15.116:0/486254641 >> v1:172.21.15.144:6789/0 conn(0x7f7984136bb0 legacy=0x7f7984138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281254004 0 0) 0x7f796c0034e0 con 0x7f798409e910 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 msgr2=0x7f7984131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 0x7f7984131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 >> v1:172.21.15.144:6789/0 conn(0x7f7984136bb0 legacy=0x7f7984138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:43:09.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:43:09.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79840a13f0 con 0x7f798409e910 2024-09-18T12:43:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f7974001860 con 0x7f798409e910 2024-09-18T12:43:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798b19d640 5 monclient: authenticate success, global_id 194394 2024-09-18T12:43:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798b19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: dump: 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 >> v1:172.21.15.116:6789/0 conn(0x7f798409e910 legacy=0x7f798409ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 0x7f796c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): picked mon.c con 0x7f7984131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f796c004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): picked mon.a con 0x7f796c0040f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f796c004de0 0x7f796c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): picked mon.b con 0x7f796c004de0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79840a1110 con 0x7f796c0040f0 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c00bcb0 con 0x7f796c004de0 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c00be70 con 0x7f7984131ab0 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f7988f39640 1 -- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 msgr2=0x7f796c003a00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798b19d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f7988f39640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 0x7f796c003a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f796c004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): get_auth_request con 0x7f796c0040f0 auth_method 0 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f796c004de0 0x7f796c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 10 monclient(hunting): get_auth_request con 0x7f796c004de0 auth_method 0 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f798973a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient(hunting): handle_auth_done global_id 194394 payload 293 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 1 -- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 msgr2=0x7f796c003a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984131ab0 0x7f796c003a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 1 -- 172.21.15.116:0/486254641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f796c004de0 msgr2=0x7f796c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f796c004de0 0x7f796c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 1 monclient: found mon.a 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 1 -- 172.21.15.116:0/486254641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c0034e0 con 0x7f796c0040f0 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.878+0000 7f797bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797bfff640 1 -- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 msgr2=0x7f796c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797bfff640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f796c004780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f797400c5c0 tx=0x7f7974007a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c040cfdcbd22a39 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974005890 con 0x7f796c0040f0 2024-09-18T12:43:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 10 monclient: dump: 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f798b19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7974005a10 con 0x7f796c0040f0 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f798b19d640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f798b19d640 4 set_mon_vals no callback set 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 1 -- 172.21.15.116:0/486254641 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974005e10 con 0x7f796c0040f0 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f797b7fe640 10 monclient: dump: 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.881+0000 7f798b19d640 10 monclient: shutdown 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 1 -- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 msgr2=0x7f796c004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 1 --2- 172.21.15.116:0/486254641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f796c004780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f797400c5c0 tx=0x7f7974007a70 comp rx=0 tx=0).stop 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798973a640 1 -- 172.21.15.116:0/486254641 reap_dead start 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 1 -- 172.21.15.116:0/486254641 shutdown_connections 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 1 -- 172.21.15.116:0/486254641 >> 172.21.15.116:0/486254641 conn(0x7f7984067d30 msgr2=0x7f79840a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 1 -- 172.21.15.116:0/486254641 shutdown_connections 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 1 -- 172.21.15.116:0/486254641 wait complete. 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 10 monclient: build_initial_monmap 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 10 monclient: monmap: 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:09.901 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.882+0000 7f798b19d640 1 Processor -- start 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 1 -- start start 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient: init 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f7984130710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): picked mon.noname-a con 0x7f796c0040f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984141c30 0x7f7984099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): picked mon.noname-c con 0x7f7984141c30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7984130c50 0x7f7984099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): picked mon.noname-b con 0x7f7984130c50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798409b920 con 0x7f796c0040f0 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798409b1d0 con 0x7f7984130c50 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798411c820 con 0x7f7984141c30 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798b19d640 10 monclient(hunting): authenticate will time out at 1817.204553s 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f797bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984141c30 msgr2=0x7f7984099570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f797bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984141c30 0x7f7984099570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f7984130710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f7984130710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52964/0 (socket says 172.21.15.116:52964) 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 -- 172.21.15.116:0/2465318274 learned_addr learned my addr 172.21.15.116:0/2465318274 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): get_auth_request con 0x7f796c0040f0 auth_method 0 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient(hunting): handle_auth_done global_id 194397 payload 824 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984141c30 msgr2=0x7f7984099570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7984141c30 0x7f7984099570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798973a640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7984130c50 0x7f7984099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7984130c50 msgr2=0x7f7984099ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7984130c50 0x7f7984099ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient: _finish_hunting 0 2024-09-18T12:43:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 monclient: found mon.noname-a 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f7988f39640 1 -- 172.21.15.116:0/2465318274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798411cb00 con 0x7f796c0040f0 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798973a640 10 monclient: get_auth_request con 0x7f7984130c50 auth_method 0 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.883+0000 7f798973a640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7984130c50 0x7f7984099ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7988f39640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 msgr2=0x7f7984130710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7988f39640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f7984130710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f798000b7e0 tx=0x7f7980009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980003de0 con 0x7f796c0040f0 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: dump: 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: _finish_auth 0 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 5 monclient: authenticate success, global_id 194397 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79800050f0 con 0x7f796c0040f0 2024-09-18T12:43:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 10 monclient: _renew_subs 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79800054b0 con 0x7f796c0040f0 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798411c530 con 0x7f796c0040f0 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f7978ff9640 10 monclient: dump: 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f797affd640 4 set_mon_vals no callback set 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 10 monclient: _renew_subs 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7984099ff0 con 0x7f796c0040f0 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.886+0000 7f798b19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f798b19d640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 0x7f7984029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f798b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984063740 con 0x7f79840298c0 2024-09-18T12:43:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7980012070 con 0x7f796c0040f0 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f7978ff9640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7970038440 0x7f797003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f798004b990 con 0x7f796c0040f0 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 0x7f7984029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient: get_auth_request con 0x7f79840298c0 auth_method 0 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f798973a640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7970038440 0x7f797003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f798973a640 10 monclient: get_auth_request con 0x7f7970038440 auth_method 0 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 10 monclient(hunting): handle_auth_done global_id 174535 payload 824 2024-09-18T12:43:09.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.887+0000 7f797bfff640 1 -- 172.21.15.116:0/2465318274 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f798411ce70 con 0x7f79840298c0 2024-09-18T12:43:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f798973a640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7970038440 0x7f797003a8f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7984068b50 tx=0x7f797c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f797bfff640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 msgr2=0x7f7984029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f797bfff640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 0x7f7984029c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79740014d0 tx=0x7f7974034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79740011e0 con 0x7f79840298c0 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 10 monclient: dump: 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.913 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7974041070 con 0x7f79840298c0 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 msgr2=0x7f7984029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 0x7f7984029c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79740014d0 tx=0x7f7974034000 comp rx=0 tx=0).stop 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 msgr2=0x7f7984029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.890+0000 7f7978ff9640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840298c0 0x7f7984029c90 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79740014d0 tx=0x7f7974034000 comp rx=0 tx=0).stop 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f798b19d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f798b19d640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 0x7f798402a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f798b19d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798411cfe0 con 0x7f798402a1d0 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 0x7f798402a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient: get_auth_request con 0x7f798402a1d0 auth_method 0 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 10 monclient(hunting): handle_auth_done global_id 174538 payload 824 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.896+0000 7f797bfff640 1 -- 172.21.15.116:0/2465318274 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f798404bf60 con 0x7f798402a1d0 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f797bfff640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 msgr2=0x7f798402a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f797bfff640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 0x7f798402a5d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f797400c5c0 tx=0x7f797403f070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974005820 con 0x7f798402a1d0 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 10 monclient: dump: 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:09.916 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f797403d440 con 0x7f798402a1d0 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 msgr2=0x7f798402a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 0x7f798402a5d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f797400c5c0 tx=0x7f797403f070 comp rx=0 tx=0).stop 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 msgr2=0x7f798402a5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f7978ff9640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f798402a1d0 0x7f798402a5d0 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f797400c5c0 tx=0x7f797403f070 comp rx=0 tx=0).stop 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:09.917 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 176, 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:09.918 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:09.919 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:09.920 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:09.921 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:09.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:09.923 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.924 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:09.925 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.926 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:09.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:09.928 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:09.930 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:09.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:09.931 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:09.931 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:09.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7970038440 msgr2=0x7f797003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7970038440 0x7f797003a8f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7984068b50 tx=0x7f797c009040 comp rx=0 tx=0).stop 2024-09-18T12:43:09.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 10 monclient: shutdown 2024-09-18T12:43:09.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:09.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 msgr2=0x7f7984130710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:09.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f7984130710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f798000b7e0 tx=0x7f7980009220 comp rx=0 tx=0).stop 2024-09-18T12:43:09.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798973a640 1 -- 172.21.15.116:0/2465318274 reap_dead start 2024-09-18T12:43:09.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 shutdown_connections 2024-09-18T12:43:09.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 --2- 172.21.15.116:0/2465318274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796c0040f0 0x7f7984130710 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:09.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 >> 172.21.15.116:0/2465318274 conn(0x7f7984067d30 msgr2=0x7f798409d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:09.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 shutdown_connections 2024-09-18T12:43:09.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:09.899+0000 7f798b19d640 1 -- 172.21.15.116:0/2465318274 wait complete. 2024-09-18T12:43:09.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:43:09.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:43:29.934 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:43:29.934 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:43:29.935 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:43:29.935 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:43:29.938 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:43:29.938 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T12:43:29.938 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T12:43:29.939 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-18T12:43:29.939 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-18T12:43:29.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-18T12:43:44.940 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T12:43:44.941 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T12:43:44.941 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-18T12:43:44.941 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-18T12:43:44.941 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:43:44.942 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:43:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.044+0000 7fab0df9d640 10 monclient: get_monmap_and_config 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.044+0000 7fab0df9d640 10 monclient: build_initial_monmap 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.044+0000 7fab0df9d640 10 monclient: monmap: 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:45.044683+0000 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:45.044683+0000 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:45.046 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:45.046 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.044+0000 7fab0df9d640 1 Processor -- start 2024-09-18T12:43:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.044+0000 7fab0df9d640 1 -- start start 2024-09-18T12:43:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.044+0000 7fab0df9d640 10 monclient: init 2024-09-18T12:43:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab08096e10 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient(hunting): picked mon.noname-a con 0x7fab080971e0 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient(hunting): picked mon.noname-c con 0x7fab08133b30 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab08099620 con 0x7fab08096e10 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab08098510 con 0x7fab080971e0 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab08099340 con 0x7fab08133b30 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab0df9d640 10 monclient(hunting): authenticate will time out at 1852.366364s 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab077fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fab08096e10 0x7fab0809a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:50224/0 (socket says 172.21.15.116:50224) 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 learned_addr learned my addr 172.21.15.116:0/717192990 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab06ffd640 10 monclient(hunting): get_auth_request con 0x7fab080971e0 auth_method 0 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 10 monclient(hunting): get_auth_request con 0x7fab08133b30 auth_method 0 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.045+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.047+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.047+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> v1:172.21.15.116:6789/0 conn(0x7fab08096e10 legacy=0x7fab0809a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.047+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.047+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:45.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.047+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 10 monclient(hunting): get_auth_request con 0x7fab08133b30 auth_method 0 2024-09-18T12:43:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:45.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:45.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.246+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 10 monclient(hunting): get_auth_request con 0x7fab080971e0 auth_method 0 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:45.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.248+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:43:45.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:45.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 10 monclient(hunting): get_auth_request con 0x7fab08133b30 auth_method 0 2024-09-18T12:43:45.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:45.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:45.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:45.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:45.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:45.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:45.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.647+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 10 monclient(hunting): get_auth_request con 0x7fab080971e0 auth_method 0 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:45.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:45.649+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 10 monclient(hunting): get_auth_request con 0x7fab08133b30 auth_method 0 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.449+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:43:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 10 monclient(hunting): get_auth_request con 0x7fab080971e0 auth_method 0 2024-09-18T12:43:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:43:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:46.450+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 10 monclient(hunting): tick 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 monclient(hunting): continuing hunt 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 msgr2=0x7fab08135f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.144:3300/0 conn(0x7fab08133b30 0x7fab08135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 -- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 msgr2=0x7fab080975e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 --2- 172.21.15.116:0/717192990 >> v2:172.21.15.116:3300/0 conn(0x7fab080971e0 0x7fab080975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:48.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 -- 172.21.15.116:0/717192990 >> v1:172.21.15.116:6789/0 conn(0x7fab08096e10 legacy=0x7fab0809a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 10 monclient: _add_conns ranks=[2,1] 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 10 monclient(hunting): picked mon.noname-b con 0x7faaf40036f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaf4006220 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 -- 172.21.15.116:0/717192990 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab08099900 con 0x7faaf4006220 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 1 -- 172.21.15.116:0/717192990 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab08098510 con 0x7faaf40036f0 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab057fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab077fe640 10 monclient(hunting): get_auth_request con 0x7faaf40036f0 auth_method 0 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:48.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.045+0000 7fab077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.046+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:43:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.046+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.046+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.046+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:48.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.048+0000 7fab06ffd640 1 -- 172.21.15.116:0/717192990 >> v1:172.21.15.144:6789/0 conn(0x7faaf4006220 legacy=0x7faaf40086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 10 monclient(hunting): get_auth_request con 0x7faaf40036f0 auth_method 0 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:48.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:48.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.247+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:43:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 10 monclient(hunting): get_auth_request con 0x7faaf40036f0 auth_method 0 2024-09-18T12:43:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:43:48.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:48.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:48.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:48.648+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:43:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 10 monclient(hunting): get_auth_request con 0x7faaf40036f0 auth_method 0 2024-09-18T12:43:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:49.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:43:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:43:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:49.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.449+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.963+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197672149 0 0) 0x7fab08099900 con 0x7faaf4006220 2024-09-18T12:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.963+0000 7fab067fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.963+0000 7fab067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.963+0000 7fab067fc640 10 monclient(hunting): my global_id is 174541 2024-09-18T12:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.963+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf00034e0 con 0x7faaf4006220 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401677070 0 0) 0x7faaf00034e0 con 0x7faaf4006220 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: _finish_hunting 0 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:43:49.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf400b000 con 0x7faaf4006220 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: _finish_auth 0 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7faafc00d480 con 0x7faaf4006220 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: dump: 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:49.979 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 >> v1:172.21.15.144:6789/0 conn(0x7faaf4006220 legacy=0x7faaf40086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 reap_dead start 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf0003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): picked mon.c con 0x7faaf40036f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf00040d0 0x7faaf0004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): picked mon.b con 0x7faaf00040d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0004d90 0x7faaf0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): picked mon.a con 0x7faaf0004d90 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab08098510 con 0x7faaf0004d90 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf000bd20 con 0x7faaf00040d0 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf000bee0 con 0x7faaf40036f0 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab067fc640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf0003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 10 monclient(hunting): get_auth_request con 0x7faaf40036f0 auth_method 0 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0004d90 0x7faaf0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): get_auth_request con 0x7faaf0004d90 auth_method 0 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf00040d0 0x7faaf0004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 10 monclient(hunting): get_auth_request con 0x7faaf00040d0 auth_method 0 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient(hunting): handle_auth_done global_id 174541 payload 293 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7faaf0003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf0003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf00040d0 msgr2=0x7faaf0004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf00040d0 0x7faaf0004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 monclient: found mon.a 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00034e0 con 0x7faaf0004d90 2024-09-18T12:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 10 monclient: _finish_auth 0 2024-09-18T12:43:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab0df9d640 5 monclient: authenticate success, global_id 174541 2024-09-18T12:43:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab0df9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.975+0000 7fab077fe640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf0003a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.978+0000 7fab07fff640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0004d90 msgr2=0x7faaf0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.978+0000 7fab07fff640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0004d90 0x7faaf0005200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fab0809d8e0 tx=0x7fab0000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc96bc94264bac9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00031650 con 0x7faaf0004d90 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 10 monclient: dump: 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab0df9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab000317b0 con 0x7faaf0004d90 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 1 -- 172.21.15.116:0/717192990 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00031b50 con 0x7faaf0004d90 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab0df9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab0df9d640 4 set_mon_vals no callback set 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab067fc640 10 monclient: dump: 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab0df9d640 10 monclient: shutdown 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab0df9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.980+0000 7fab0df9d640 1 -- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0004d90 msgr2=0x7faaf0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0004d90 0x7faaf0005200 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fab0809d8e0 tx=0x7fab0000b040 comp rx=0 tx=0).stop 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 -- 172.21.15.116:0/717192990 shutdown_connections 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf0004d90 0x7faaf0005200 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf00040d0 0x7faaf0004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 --2- 172.21.15.116:0/717192990 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7faaf0003a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 -- 172.21.15.116:0/717192990 >> 172.21.15.116:0/717192990 conn(0x7fab080672b0 msgr2=0x7fab080676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 -- 172.21.15.116:0/717192990 shutdown_connections 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 1 -- 172.21.15.116:0/717192990 wait complete. 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 10 monclient: build_initial_monmap 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.981+0000 7fab0df9d640 10 monclient: monmap: 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:49.993 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 Processor -- start 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 -- start start 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient: init 2024-09-18T12:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 0x7fab0812afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): picked mon.noname-a con 0x7faaf00040d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf0004d90 0x7fab0806bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): picked mon.noname-b con 0x7faaf0004d90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7fab0806c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): picked mon.noname-c con 0x7faaf40036f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab08117c50 con 0x7faaf00040d0 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0809b080 con 0x7faaf0004d90 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0806d820 con 0x7faaf40036f0 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.982+0000 7fab0df9d640 10 monclient(hunting): authenticate will time out at 1857.303934s 2024-09-18T12:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 0x7fab0812afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7fab0806c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 0x7fab0812afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58616/0 (socket says 172.21.15.116:58616) 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 -- 172.21.15.116:0/3284284025 learned_addr learned my addr 172.21.15.116:0/3284284025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): get_auth_request con 0x7faaf00040d0 auth_method 0 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 10 monclient(hunting): get_auth_request con 0x7faaf40036f0 auth_method 0 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:49.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf0004d90 0x7fab0806bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 10 monclient(hunting): get_auth_request con 0x7faaf0004d90 auth_method 0 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 1 -- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7fab0806c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 1 -- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7fab0806c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7fab0806c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab07fff640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7fab0806c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient(hunting): handle_auth_done global_id 194400 payload 824 2024-09-18T12:43:49.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 -- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 msgr2=0x7fab0806c160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7fab0806c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 -- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf0004d90 msgr2=0x7fab0806bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf0004d90 0x7fab0806bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient: _finish_hunting 0 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 monclient: found mon.noname-a 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.983+0000 7fab077fe640 1 -- 172.21.15.116:0/3284284025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0812a5b0 con 0x7faaf00040d0 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab077fe640 1 -- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 msgr2=0x7fab0812afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab077fe640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 0x7fab0812afc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faaf80059b0 tx=0x7faaf8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 1 -- 172.21.15.116:0/3284284025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf8005520 con 0x7faaf00040d0 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: dump: 2024-09-18T12:43:49.998 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: _finish_auth 0 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 1 -- 172.21.15.116:0/3284284025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf801c070 con 0x7faaf00040d0 2024-09-18T12:43:49.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab0df9d640 5 monclient: authenticate success, global_id 194400 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab0df9d640 10 monclient: _renew_subs 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab0df9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0806dcc0 con 0x7faaf00040d0 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 1 -- 172.21.15.116:0/3284284025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf8005680 con 0x7faaf00040d0 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7faaef7fe640 10 monclient: dump: 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.000 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab0df9d640 10 monclient: _renew_subs 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab0df9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0806f320 con 0x7faaf00040d0 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.985+0000 7fab05ffb640 4 set_mon_vals no callback set 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.986+0000 7faaef7fe640 1 -- 172.21.15.116:0/3284284025 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7faaf80049b0 con 0x7faaf00040d0 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7fab0df9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7fab0df9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7fab0df9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0804bf60 con 0x7faaf00040d0 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7faaef7fe640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faaf4038eb0 0x7faaf403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7faaef7fe640 1 -- 172.21.15.116:0/3284284025 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7faaf804cd00 con 0x7faaf00040d0 2024-09-18T12:43:50.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7fab06ffd640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faaf4038eb0 0x7faaf403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.987+0000 7fab06ffd640 10 monclient: get_auth_request con 0x7faaf4038eb0 auth_method 0 2024-09-18T12:43:50.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.990+0000 7fab06ffd640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faaf4038eb0 0x7faaf403b360 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e010 tx=0x7faafc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.991+0000 7faaef7fe640 1 -- 172.21.15.116:0/3284284025 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7faaf8020980 con 0x7faaf00040d0 2024-09-18T12:43:50.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.992+0000 7faaef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:49.992+0000 7faaef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.207+0000 7fab0df9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.207+0000 7fab0df9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.207+0000 7fab0df9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.207+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab08132d70 con 0x7faaf00040d0 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.207+0000 7faaef7fe640 1 -- 172.21.15.116:0/3284284025 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7faaf8014370 con 0x7faaf00040d0 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.207+0000 7faaef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.207+0000 7faaef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:50.208 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:43:50.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faaf4038eb0 msgr2=0x7faaf403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faaf4038eb0 0x7faaf403b360 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e010 tx=0x7faafc008040 comp rx=0 tx=0).stop 2024-09-18T12:43:50.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 10 monclient: shutdown 2024-09-18T12:43:50.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 msgr2=0x7fab0812afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 0x7fab0812afc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faaf80059b0 tx=0x7faaf8009040 comp rx=0 tx=0).stop 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 shutdown_connections 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faaf4038eb0 0x7faaf403b360 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faaf40036f0 0x7fab0806c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faaf0004d90 0x7fab0806bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 --2- 172.21.15.116:0/3284284025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf00040d0 0x7fab0812afc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 >> 172.21.15.116:0/3284284025 conn(0x7fab080672b0 msgr2=0x7fab0809f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 shutdown_connections 2024-09-18T12:43:50.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.209+0000 7fab0df9d640 1 -- 172.21.15.116:0/3284284025 wait complete. 2024-09-18T12:43:50.220 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:43:50.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:43:50.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.313+0000 7fbac0d54640 10 monclient: get_monmap_and_config 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.313+0000 7fbac0d54640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.313+0000 7fbac0d54640 10 monclient: monmap: 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:50.313799+0000 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:50.313799+0000 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:50.315 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.313+0000 7fbac0d54640 1 Processor -- start 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.313+0000 7fbac0d54640 1 -- start start 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.313+0000 7fbac0d54640 10 monclient: init 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbabc131ab0 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fbabc09e910 0x7fbabc09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient(hunting): picked mon.noname-c con 0x7fbabc09e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbabc09f250 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbabc09faf0 con 0x7fbabc131ab0 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbabc09fd30 con 0x7fbabc09f250 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbabc132590 con 0x7fbabc09e910 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbac0d54640 10 monclient(hunting): authenticate will time out at 1857.635283s 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbaba59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbabc131ab0 0x7fbabc131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40668/0 (socket says 172.21.15.116:40668) 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbaba59c640 1 -- 172.21.15.116:0/2021548092 learned_addr learned my addr 172.21.15.116:0/2021548092 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 1 --2- 172.21.15.116:0/2021548092 >> v2:172.21.15.144:3300/0 conn(0x7fbabc09e910 0x7fbabc09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): get_auth_request con 0x7fbabc09e910 auth_method 0 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_done global_id 174547 payload 274 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 1 -- 172.21.15.116:0/2021548092 >> v1:172.21.15.144:6789/0 conn(0x7fbabc09f250 legacy=0x7fbabc138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 1 -- 172.21.15.116:0/2021548092 >> v1:172.21.15.116:6789/0 conn(0x7fbabc131ab0 legacy=0x7fbabc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 1 monclient: found mon.noname-c 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:43:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.314+0000 7fbab1d9b640 1 -- 172.21.15.116:0/2021548092 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc0a1110 con 0x7fbabc09e910 2024-09-18T12:43:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.317+0000 7fbab1d9b640 1 -- 172.21.15.116:0/2021548092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc09e910 msgr2=0x7fbabc09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.317+0000 7fbab1d9b640 1 --2- 172.21.15.116:0/2021548092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc09e910 0x7fbabc09ed10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009850 tx=0x7fbaa40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5fc5b39dda426b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 1 -- 172.21.15.116:0/2021548092 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4032440 con 0x7fbabc09e910 2024-09-18T12:43:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: dump: 2024-09-18T12:43:50.326 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 1 -- 172.21.15.116:0/2021548092 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa40325a0 con 0x7fbabc09e910 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 5 monclient: authenticate success, global_id 174547 2024-09-18T12:43:50.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 4 set_mon_vals no callback set 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 1 -- 172.21.15.116:0/2021548092 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4032940 con 0x7fbabc09e910 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbab9d9b640 10 monclient: dump: 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.328 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 10 monclient: shutdown 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 1 -- 172.21.15.116:0/2021548092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc09e910 msgr2=0x7fbabc09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 1 --2- 172.21.15.116:0/2021548092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc09e910 0x7fbabc09ed10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009850 tx=0x7fbaa40090f0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 1 -- 172.21.15.116:0/2021548092 shutdown_connections 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 1 --2- 172.21.15.116:0/2021548092 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc09e910 0x7fbabc09ed10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 1 -- 172.21.15.116:0/2021548092 >> 172.21.15.116:0/2021548092 conn(0x7fbabc067d30 msgr2=0x7fbabc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 1 -- 172.21.15.116:0/2021548092 shutdown_connections 2024-09-18T12:43:50.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.318+0000 7fbac0d54640 1 -- 172.21.15.116:0/2021548092 wait complete. 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 10 monclient: monmap: 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 1 Processor -- start 2024-09-18T12:43:50.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 1 -- start start 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 10 monclient: init 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc09e910 0x7fbabc130070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 10 monclient(hunting): picked mon.noname-a con 0x7fbabc09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.319+0000 7fbac0d54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbabc09f250 0x7fbabc1305b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 10 monclient(hunting): picked mon.noname-c con 0x7fbabc09f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc131ab0 0x7fbabc130af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 10 monclient(hunting): picked mon.noname-b con 0x7fbabc131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc098d10 con 0x7fbabc09e910 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc099370 con 0x7fbabc131ab0 2024-09-18T12:43:50.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc097310 con 0x7fbabc09f250 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbac0d54640 10 monclient(hunting): authenticate will time out at 1857.641028s 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc09e910 0x7fbabc130070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc09e910 0x7fbabc130070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58630/0 (socket says 172.21.15.116:58630) 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 -- 172.21.15.116:0/1883781320 learned_addr learned my addr 172.21.15.116:0/1883781320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): get_auth_request con 0x7fbabc09e910 auth_method 0 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbabc09f250 0x7fbabc1305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 10 monclient(hunting): get_auth_request con 0x7fbabc09f250 auth_method 0 2024-09-18T12:43:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbabad9d640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc131ab0 0x7fbabc130af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient(hunting): handle_auth_done global_id 194403 payload 824 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbabc09f250 msgr2=0x7fbabc1305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbabc09f250 0x7fbabc1305b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc131ab0 msgr2=0x7fbabc130af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc131ab0 0x7fbabc130af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 monclient: found mon.noname-a 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbaba59c640 1 -- 172.21.15.116:0/1883781320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa4009390 con 0x7fbabc09e910 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbabad9d640 10 monclient: get_auth_request con 0x7fbabc131ab0 auth_method 0 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbabad9d640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbabc131ab0 0x7fbabc130af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.320+0000 7fbab1d9b640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbabc09f250 0x7fbabc1305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbaba59c640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc09e910 msgr2=0x7fbabc130070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbaba59c640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc09e910 0x7fbabc130070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00c5e0 tx=0x7fbaac004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac005cb0 con 0x7fbabc09e910 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: dump: 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac005e30 con 0x7fbabc09e910 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbac0d54640 5 monclient: authenticate success, global_id 194403 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac007340 con 0x7fbabc09e910 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbac0d54640 10 monclient: _renew_subs 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbac0d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbabc1397e0 con 0x7fbabc09e910 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab2ffd640 10 monclient: dump: 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbab959a640 4 set_mon_vals no callback set 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbac0d54640 10 monclient: _renew_subs 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbac0d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.322+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc09aaf0 con 0x7fbabc09e910 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbac0d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbac0d54640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 0x7fbabc131030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbac0d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc063740 con 0x7fbabc097750 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbaac01b400 con 0x7fbabc09e910 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 0x7fbabc131030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 10 monclient: get_auth_request con 0x7fbabc097750 auth_method 0 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.323+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.324+0000 7fbab2ffd640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fba9c038490 0x7fba9c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.324+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_done global_id 194406 payload 824 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.324+0000 7fbab1d9b640 1 -- 172.21.15.116:0/1883781320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbabc04bf60 con 0x7fbabc097750 2024-09-18T12:43:50.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.324+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fbaac04be90 con 0x7fbabc09e910 2024-09-18T12:43:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.324+0000 7fbabad9d640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fba9c038490 0x7fba9c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.324+0000 7fbabad9d640 10 monclient: get_auth_request con 0x7fba9c038490 auth_method 0 2024-09-18T12:43:50.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab1d9b640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 msgr2=0x7fbabc131030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab1d9b640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 0x7fbabc131030 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbabc076760 tx=0x7fbaa4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4031e90 con 0x7fbabc097750 2024-09-18T12:43:50.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:50.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 10 monclient: dump: 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.344 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbaa4041070 con 0x7fbabc097750 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 msgr2=0x7fbabc131030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 0x7fbabc131030 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbabc076760 tx=0x7fbaa4032000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 msgr2=0x7fbabc131030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbab2ffd640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc097750 0x7fbabc131030 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbabc076760 tx=0x7fbaa4032000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.326+0000 7fbabad9d640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fba9c038490 0x7fba9c03a940 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fbab4007fd0 tx=0x7fbab4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbac0d54640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbac0d54640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 0x7fbabc0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbac0d54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc066280 con 0x7fbabc0a3cc0 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 0x7fbabc0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient: get_auth_request con 0x7fbabc0a3cc0 auth_method 0 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 10 monclient(hunting): handle_auth_done global_id 194409 payload 824 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.332+0000 7fbab1d9b640 1 -- 172.21.15.116:0/1883781320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbabc0a4600 con 0x7fbabc0a3cc0 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab1d9b640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 msgr2=0x7fbabc0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab1d9b640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 0x7fbabc0a40c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4044000 tx=0x7fbaa403f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4048af0 con 0x7fbabc0a3cc0 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 10 monclient: dump: 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fbaa4048c50 con 0x7fbabc0a3cc0 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:50.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 msgr2=0x7fbabc0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 0x7fbabc0a40c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4044000 tx=0x7fbaa403f070 comp rx=0 tx=0).stop 2024-09-18T12:43:50.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 msgr2=0x7fbabc0a40c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:50.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbab2ffd640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc0a3cc0 0x7fbabc0a40c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.349 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:50.349 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 182, 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:50.350 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.351 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.352 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:50.353 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.355 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.357 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:50.360 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.362 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.363 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.364 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.365 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:50.365 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:50.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fba9c038490 msgr2=0x7fba9c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fba9c038490 0x7fba9c03a940 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fbab4007fd0 tx=0x7fbab4009040 comp rx=0 tx=0).stop 2024-09-18T12:43:50.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 10 monclient: shutdown 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc09e910 msgr2=0x7fbabc130070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 --2- 172.21.15.116:0/1883781320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbabc09e910 0x7fbabc130070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00c5e0 tx=0x7fbaac004960 comp rx=0 tx=0).stop 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbabad9d640 1 -- 172.21.15.116:0/1883781320 reap_dead start 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbabad9d640 1 -- 172.21.15.116:0/1883781320 reap_dead start 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 shutdown_connections 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 >> 172.21.15.116:0/1883781320 conn(0x7fbabc067d30 msgr2=0x7fbabc09c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 shutdown_connections 2024-09-18T12:43:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.335+0000 7fbac0d54640 1 -- 172.21.15.116:0/1883781320 wait complete. 2024-09-18T12:43:50.368 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:43:50.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:43:50.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.461+0000 7f5119ef8640 10 monclient: get_monmap_and_config 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.461+0000 7f5119ef8640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.461+0000 7f5119ef8640 10 monclient: monmap: 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:50.461915+0000 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:50.461915+0000 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:50.463 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.461+0000 7f5119ef8640 1 Processor -- start 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 1 -- start start 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient: init 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51141318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f511409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f511409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5114136c20 0x7f5114139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f5114136c20 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5114132640 con 0x7f51141318f0 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5114131d00 con 0x7f5114136c20 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511409fbf0 con 0x7f511409ea50 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5119ef8640 10 monclient(hunting): authenticate will time out at 1857.783475s 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f511409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5113fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5114136c20 0x7f5114139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f511409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42126/0 (socket says 172.21.15.116:42126) 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5113fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5114136c20 0x7f5114139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58656/0 (socket says 172.21.15.116:58656) 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 -- 172.21.15.116:0/1445031791 learned_addr learned my addr 172.21.15.116:0/1445031791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): get_auth_request con 0x7f511409ea50 auth_method 0 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5113fff640 10 monclient(hunting): get_auth_request con 0x7f5114136c20 auth_method 0 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5113fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5113fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5113fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient(hunting): handle_auth_done global_id 224102 payload 274 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 -- 172.21.15.116:0/1445031791 >> v2:172.21.15.116:3300/0 conn(0x7f5114136c20 msgr2=0x7f5114139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 --2- 172.21.15.116:0/1445031791 >> v2:172.21.15.116:3300/0 conn(0x7f5114136c20 0x7f5114139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 -- 172.21.15.116:0/1445031791 >> v1:172.21.15.116:6789/0 conn(0x7f51141318f0 legacy=0x7f5114131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 monclient: found mon.noname-b 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.462+0000 7f5112ffd640 1 -- 172.21.15.116:0/1445031791 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511409fe30 con 0x7f511409ea50 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.465+0000 7f5112ffd640 1 -- 172.21.15.116:0/1445031791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 msgr2=0x7f511409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.465+0000 7f5112ffd640 1 --2- 172.21.15.116:0/1445031791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f511409ee20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5108009f20 tx=0x7f5108009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2058866ca69ed2a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 1 -- 172.21.15.116:0/1445031791 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108007610 con 0x7f511409ea50 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: dump: 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.478 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 1 -- 172.21.15.116:0/1445031791 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510802d050 con 0x7f511409ea50 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 5 monclient: authenticate success, global_id 224102 2024-09-18T12:43:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 1 -- 172.21.15.116:0/1445031791 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080079a0 con 0x7f511409ea50 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 4 set_mon_vals no callback set 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f51127fc640 10 monclient: dump: 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 10 monclient: shutdown 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 -- 172.21.15.116:0/1445031791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 msgr2=0x7f511409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 --2- 172.21.15.116:0/1445031791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f511409ee20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5108009f20 tx=0x7f5108009840 comp rx=0 tx=0).stop 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 -- 172.21.15.116:0/1445031791 shutdown_connections 2024-09-18T12:43:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 --2- 172.21.15.116:0/1445031791 >> v2:172.21.15.116:3300/0 conn(0x7f5114136c20 0x7f5114139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 --2- 172.21.15.116:0/1445031791 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f511409ee20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 -- 172.21.15.116:0/1445031791 >> 172.21.15.116:0/1445031791 conn(0x7f5114067110 msgr2=0x7f5114067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 -- 172.21.15.116:0/1445031791 shutdown_connections 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.466+0000 7f5119ef8640 1 -- 172.21.15.116:0/1445031791 wait complete. 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.467+0000 7f5119ef8640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.467+0000 7f5119ef8640 10 monclient: monmap: 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.467+0000 7f5119ef8640 1 Processor -- start 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.467+0000 7f5119ef8640 1 -- start start 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.467+0000 7f5119ef8640 10 monclient: init 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f51140971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f511409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51141318f0 0x7f51140976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f51141318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5114136c20 0x7f5114097c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f5114136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114099230 con 0x7f5114136c20 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511409bf10 con 0x7f51141318f0 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511409daf0 con 0x7f511409ea50 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5119ef8640 10 monclient(hunting): authenticate will time out at 1857.789254s 2024-09-18T12:43:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f51140971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f51140971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42134/0 (socket says 172.21.15.116:42134) 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 -- 172.21.15.116:0/3193284538 learned_addr learned my addr 172.21.15.116:0/3193284538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): get_auth_request con 0x7f511409ea50 auth_method 0 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5114136c20 0x7f5114097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 10 monclient(hunting): get_auth_request con 0x7f5114136c20 auth_method 0 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5113fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5112ffd640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51141318f0 0x7f51140976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5112ffd640 10 monclient(hunting): get_auth_request con 0x7f51141318f0 auth_method 0 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f5112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient(hunting): handle_auth_done global_id 224105 payload 824 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51141318f0 msgr2=0x7f51140976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f51141318f0 0x7f51140976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5114136c20 msgr2=0x7f5114097c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5114136c20 0x7f5114097c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 monclient: found mon.noname-c 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.468+0000 7f51137fe640 1 -- 172.21.15.116:0/3193284538 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5108009ae0 con 0x7f511409ea50 2024-09-18T12:43:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f51137fe640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 msgr2=0x7f51140971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f51137fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f51140971b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f510400c610 tx=0x7f5104007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104012840 con 0x7f511409ea50 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: dump: 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51040129e0 con 0x7f511409ea50 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104012da0 con 0x7f511409ea50 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f5119ef8640 5 monclient: authenticate success, global_id 224105 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f50fb7fe640 10 monclient: dump: 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f5119ef8640 10 monclient: _renew_subs 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f5119ef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511409a770 con 0x7f511409ea50 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.470+0000 7f5111ffb640 4 set_mon_vals no callback set 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5119ef8640 10 monclient: _renew_subs 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5119ef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51141314e0 con 0x7f511409ea50 2024-09-18T12:43:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5119ef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5119ef8640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 0x7f51141337e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5119ef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5114065e00 con 0x7f5114098170 2024-09-18T12:43:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5104011590 con 0x7f511409ea50 2024-09-18T12:43:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.471+0000 7f5112ffd640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 0x7f51141337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient: get_auth_request con 0x7f5114098170 auth_method 0 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f50fb7fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5100038490 0x7f510003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 10 monclient(hunting): handle_auth_done global_id 224108 payload 824 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5112ffd640 1 -- 172.21.15.116:0/3193284538 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f511409abe0 con 0x7f5114098170 2024-09-18T12:43:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f510404be30 con 0x7f511409ea50 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5113fff640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5100038490 0x7f510003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.472+0000 7f5113fff640 10 monclient: get_auth_request con 0x7f5100038490 auth_method 0 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f5112ffd640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 msgr2=0x7f51141337e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f5112ffd640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 0x7f51141337e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5108033000 tx=0x7f5108036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080319d0 con 0x7f5114098170 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 10 monclient: dump: 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5108031b30 con 0x7f5114098170 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 msgr2=0x7f51141337e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 0x7f51141337e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5108033000 tx=0x7f5108036000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 msgr2=0x7f51141337e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.474+0000 7f50fb7fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114098170 0x7f51141337e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.475+0000 7f5113fff640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5100038490 0x7f510003a940 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5114099930 tx=0x7f510c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5119ef8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5119ef8640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 0x7f5114134120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5119ef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511404bf60 con 0x7f5114133d20 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 0x7f5114134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient: get_auth_request con 0x7f5114133d20 auth_method 0 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 10 monclient(hunting): handle_auth_done global_id 224111 payload 824 2024-09-18T12:43:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.480+0000 7f5112ffd640 1 -- 172.21.15.116:0/3193284538 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5114098540 con 0x7f5114133d20 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5112ffd640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 msgr2=0x7f5114134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5112ffd640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 0x7f5114134120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5114099bf0 tx=0x7f510803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108049070 con 0x7f5114133d20 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 10 monclient: dump: 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f51080317c0 con 0x7f5114133d20 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 msgr2=0x7f5114134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 0x7f5114134120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5114099bf0 tx=0x7f510803d000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 msgr2=0x7f5114134120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f50fb7fe640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5114133d20 0x7f5114134120 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5114099bf0 tx=0x7f510803d000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 182, 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:50.502 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.503 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.504 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:50.505 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.506 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:50.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.508 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.509 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.511 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.512 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:50.513 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.514 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:50.515 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5100038490 msgr2=0x7f510003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5100038490 0x7f510003a940 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5114099930 tx=0x7f510c008040 comp rx=0 tx=0).stop 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 10 monclient: shutdown 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 msgr2=0x7f51140971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 --2- 172.21.15.116:0/3193284538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f511409ea50 0x7f51140971b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f510400c610 tx=0x7f5104007ed0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5113fff640 1 -- 172.21.15.116:0/3193284538 reap_dead start 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5113fff640 1 -- 172.21.15.116:0/3193284538 reap_dead start 2024-09-18T12:43:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 shutdown_connections 2024-09-18T12:43:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 >> 172.21.15.116:0/3193284538 conn(0x7f5114067110 msgr2=0x7f51140a1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 shutdown_connections 2024-09-18T12:43:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.483+0000 7f5119ef8640 1 -- 172.21.15.116:0/3193284538 wait complete. 2024-09-18T12:43:50.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:43:50.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:43:50.518 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:43:50.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.611+0000 7f3cb0060640 10 monclient: get_monmap_and_config 2024-09-18T12:43:50.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.611+0000 7f3cb0060640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.611+0000 7f3cb0060640 10 monclient: monmap: 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:50.612380+0000 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:50.612380+0000 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:50.613 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:50.614 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:50.614 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:50.614 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:50.614 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 1 Processor -- start 2024-09-18T12:43:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 1 -- start start 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 10 monclient: init 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ca8096ca0 0x7f3ca8098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 10 monclient(hunting): picked mon.noname-a con 0x7f3ca8096ca0 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca809bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.612+0000 7f3cb0060640 10 monclient(hunting): picked mon.noname-b con 0x7f3ca809b810 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ca809c120 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca8097070 con 0x7f3ca809c120 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca80993a0 con 0x7f3ca8096ca0 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8097a30 con 0x7f3ca809b810 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cb0060640 10 monclient(hunting): authenticate will time out at 1857.934044s 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ca8096ca0 0x7f3ca8098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca809bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca809bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42152/0 (socket says 172.21.15.116:42152) 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ca8096ca0 0x7f3ca8098e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58670/0 (socket says 172.21.15.116:58670) 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 -- 172.21.15.116:0/2748390247 learned_addr learned my addr 172.21.15.116:0/2748390247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): get_auth_request con 0x7f3ca809b810 auth_method 0 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 10 monclient(hunting): get_auth_request con 0x7f3ca8096ca0 auth_method 0 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_done global_id 224114 payload 274 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 -- 172.21.15.116:0/2748390247 >> v2:172.21.15.116:3300/0 conn(0x7f3ca8096ca0 msgr2=0x7f3ca8098e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 --2- 172.21.15.116:0/2748390247 >> v2:172.21.15.116:3300/0 conn(0x7f3ca8096ca0 0x7f3ca8098e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 -- 172.21.15.116:0/2748390247 >> v1:172.21.15.144:6789/0 conn(0x7f3ca809c120 legacy=0x7f3ca8135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 monclient: found mon.noname-b 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3cad5fb640 1 -- 172.21.15.116:0/2748390247 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8099620 con 0x7f3ca809b810 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.613+0000 7f3caddfc640 1 --2- 172.21.15.116:0/2748390247 >> v2:172.21.15.116:3300/0 conn(0x7f3ca8096ca0 0x7f3ca8098e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cad5fb640 1 -- 172.21.15.116:0/2748390247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 msgr2=0x7f3ca809bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cad5fb640 1 --2- 172.21.15.116:0/2748390247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca809bbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009910 tx=0x7f3c980091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c704bfb1f57adc16 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 1 -- 172.21.15.116:0/2748390247 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9802ae90 con 0x7f3ca809b810 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: dump: 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 1 -- 172.21.15.116:0/2748390247 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98032440 con 0x7f3ca809b810 2024-09-18T12:43:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 5 monclient: authenticate success, global_id 224114 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 1 -- 172.21.15.116:0/2748390247 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98032820 con 0x7f3ca809b810 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 4 set_mon_vals no callback set 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cacdfa640 10 monclient: dump: 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 10 monclient: shutdown 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 1 -- 172.21.15.116:0/2748390247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 msgr2=0x7f3ca809bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 1 --2- 172.21.15.116:0/2748390247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca809bbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009910 tx=0x7f3c980091d0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 1 -- 172.21.15.116:0/2748390247 shutdown_connections 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 1 --2- 172.21.15.116:0/2748390247 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca809bbe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 1 --2- 172.21.15.116:0/2748390247 >> v2:172.21.15.116:3300/0 conn(0x7f3ca8096ca0 0x7f3ca8098e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.616+0000 7f3cb0060640 1 -- 172.21.15.116:0/2748390247 >> 172.21.15.116:0/2748390247 conn(0x7f3ca80672b0 msgr2=0x7f3ca80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.617+0000 7f3cb0060640 1 -- 172.21.15.116:0/2748390247 shutdown_connections 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.617+0000 7f3cb0060640 1 -- 172.21.15.116:0/2748390247 wait complete. 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.617+0000 7f3cb0060640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.617+0000 7f3cb0060640 10 monclient: monmap: 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.617+0000 7f3cb0060640 1 Processor -- start 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.617+0000 7f3cb0060640 1 -- start start 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.617+0000 7f3cb0060640 10 monclient: init 2024-09-18T12:43:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8096ca0 0x7f3ca8145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): picked mon.noname-b con 0x7f3ca8096ca0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca81380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): picked mon.noname-c con 0x7f3ca809b810 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ca809c120 0x7f3ca8121be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): picked mon.noname-a con 0x7f3ca809c120 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8125810 con 0x7f3ca809c120 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca809d1d0 con 0x7f3ca8096ca0 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8138f30 con 0x7f3ca809b810 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cb0060640 10 monclient(hunting): authenticate will time out at 1857.939232s 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca81380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca81380a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42156/0 (socket says 172.21.15.116:42156) 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 -- 172.21.15.116:0/3941289609 learned_addr learned my addr 172.21.15.116:0/3941289609 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): get_auth_request con 0x7f3ca809b810 auth_method 0 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ca809c120 0x7f3ca8121be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 10 monclient(hunting): get_auth_request con 0x7f3ca809c120 auth_method 0 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3caddfc640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8096ca0 0x7f3ca8145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3caddfc640 10 monclient(hunting): get_auth_request con 0x7f3ca8096ca0 auth_method 0 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3caddfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3caddfc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3caddfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient(hunting): handle_auth_done global_id 224117 payload 824 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8096ca0 msgr2=0x7f3ca8145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8096ca0 0x7f3ca8145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ca809c120 msgr2=0x7f3ca8121be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ca809c120 0x7f3ca8121be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 monclient: found mon.noname-c 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cad5fb640 1 -- 172.21.15.116:0/3941289609 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca4009390 con 0x7f3ca809b810 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.618+0000 7f3cae5fd640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ca809c120 0x7f3ca8121be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3cad5fb640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 msgr2=0x7f3ca81380a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3cad5fb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca81380a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009970 tx=0x7f3c98032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98031c70 con 0x7f3ca809b810 2024-09-18T12:43:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3c9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3c9dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3c9dffb640 10 monclient: dump: 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3c9dffb640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3c9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.620+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9802ae90 con 0x7f3ca809b810 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3cb0060640 5 monclient: authenticate success, global_id 224117 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3cb0060640 10 monclient: _renew_subs 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3cb0060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c98009450 con 0x7f3ca809b810 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3c9dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9803d440 con 0x7f3ca809b810 2024-09-18T12:43:50.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3c9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3c9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3c9dffb640 10 monclient: dump: 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.642 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3c9ffff640 4 set_mon_vals no callback set 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3cb0060640 10 monclient: _renew_subs 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3cb0060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca81464a0 con 0x7f3ca809b810 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.621+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3c98039040 con 0x7f3ca809b810 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3cb0060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3cb0060640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 0x7f3ca8122120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3cb0060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8065f20 con 0x7f3ca81385e0 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3c9dffb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c8c0384e0 0x7f3c8c03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3c9806c1e0 con 0x7f3ca809b810 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 0x7f3ca8122120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient: get_auth_request con 0x7f3ca81385e0 auth_method 0 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3cae5fd640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c8c0384e0 0x7f3c8c03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3cae5fd640 10 monclient: get_auth_request con 0x7f3c8c0384e0 auth_method 0 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 10 monclient(hunting): handle_auth_done global_id 174556 payload 824 2024-09-18T12:43:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.622+0000 7f3caddfc640 1 -- 172.21.15.116:0/3941289609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ca81252e0 con 0x7f3ca81385e0 2024-09-18T12:43:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.624+0000 7f3cae5fd640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c8c0384e0 0x7f3c8c03a990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0007fd0 tx=0x7f3ca00025b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3caddfc640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 msgr2=0x7f3ca8122120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3caddfc640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 0x7f3ca8122120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4011790 tx=0x7f3ca4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4011e60 con 0x7f3ca81385e0 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 10 monclient: dump: 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3ca400fd60 con 0x7f3ca81385e0 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 msgr2=0x7f3ca8122120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 0x7f3ca8122120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4011790 tx=0x7f3ca4008040 comp rx=0 tx=0).stop 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 msgr2=0x7f3ca8122120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.625+0000 7f3c9dffb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca81385e0 0x7f3ca8122120 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4011790 tx=0x7f3ca4008040 comp rx=0 tx=0).stop 2024-09-18T12:43:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3cb0060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3cb0060640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 0x7f3ca806cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3cb0060640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca81389b0 con 0x7f3ca8122660 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 0x7f3ca806cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient: get_auth_request con 0x7f3ca8122660 auth_method 0 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 10 monclient(hunting): handle_auth_done global_id 174559 payload 824 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.631+0000 7f3caddfc640 1 -- 172.21.15.116:0/3941289609 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3ca8122a60 con 0x7f3ca8122660 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3caddfc640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 msgr2=0x7f3ca806cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3caddfc640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 0x7f3ca806cf30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ca401b000 tx=0x7f3ca401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4022070 con 0x7f3ca8122660 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 10 monclient: dump: 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f3ca40253b0 con 0x7f3ca8122660 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 msgr2=0x7f3ca806cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 0x7f3ca806cf30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ca401b000 tx=0x7f3ca401c000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 msgr2=0x7f3ca806cf30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3c9dffb640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3ca8122660 0x7f3ca806cf30 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ca401b000 tx=0x7f3ca401c000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 182, 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:50.653 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.654 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.655 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:50.656 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:50.657 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:50.657 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:50.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:50.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:50.657 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:50.657 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:50.657 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:50.658 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.660 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:50.661 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.662 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:50.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:50.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.666 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:43:50.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:50.668 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c8c0384e0 msgr2=0x7f3c8c03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3c8c0384e0 0x7f3c8c03a990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0007fd0 tx=0x7f3ca00025b0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 10 monclient: shutdown 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 msgr2=0x7f3ca81380a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 1 --2- 172.21.15.116:0/3941289609 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3ca809b810 0x7f3ca81380a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009970 tx=0x7f3c98032000 comp rx=0 tx=0).stop 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cae5fd640 1 -- 172.21.15.116:0/3941289609 reap_dead start 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cae5fd640 1 -- 172.21.15.116:0/3941289609 reap_dead start 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 shutdown_connections 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.634+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 >> 172.21.15.116:0/3941289609 conn(0x7f3ca80672b0 msgr2=0x7f3ca8139140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.635+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 shutdown_connections 2024-09-18T12:43:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.635+0000 7f3cb0060640 1 -- 172.21.15.116:0/3941289609 wait complete. 2024-09-18T12:43:50.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:43:50.671 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:43:50.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:43:50.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.762+0000 7f61fd554640 10 monclient: get_monmap_and_config 2024-09-18T12:43:50.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.762+0000 7f61fd554640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.762+0000 7f61fd554640 10 monclient: monmap: 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:50.763399+0000 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:50.763399+0000 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 1 Processor -- start 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 1 -- start start 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient: init 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61f8131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f61f809e910 0x7f61f809ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient(hunting): picked mon.noname-c con 0x7f61f809e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient(hunting): picked mon.noname-b con 0x7f61f809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f809faf0 con 0x7f61f8131ab0 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f8132590 con 0x7f61f809e910 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f809fd30 con 0x7f61f809f250 2024-09-18T12:43:50.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.763+0000 7f61fd554640 10 monclient(hunting): authenticate will time out at 1858.084952s 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42160/0 (socket says 172.21.15.116:42160) 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 -- 172.21.15.116:0/879756546 learned_addr learned my addr 172.21.15.116:0/879756546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): get_auth_request con 0x7f61f809f250 auth_method 0 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61effff640 1 --2- 172.21.15.116:0/879756546 >> v2:172.21.15.144:3300/0 conn(0x7f61f809e910 0x7f61f809ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61effff640 10 monclient(hunting): get_auth_request con 0x7f61f809e910 auth_method 0 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient(hunting): handle_auth_done global_id 224120 payload 274 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 -- 172.21.15.116:0/879756546 >> v2:172.21.15.144:3300/0 conn(0x7f61f809e910 msgr2=0x7f61f809ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 --2- 172.21.15.116:0/879756546 >> v2:172.21.15.144:3300/0 conn(0x7f61f809e910 0x7f61f809ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 -- 172.21.15.116:0/879756546 >> v1:172.21.15.144:6789/0 conn(0x7f61f8131ab0 legacy=0x7f61f8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 monclient: found mon.noname-b 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:43:50.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.764+0000 7f61f77fe640 1 -- 172.21.15.116:0/879756546 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80a1110 con 0x7f61f809f250 2024-09-18T12:43:50.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f77fe640 1 -- 172.21.15.116:0/879756546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 msgr2=0x7f61f8138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f77fe640 1 --2- 172.21.15.116:0/879756546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8138bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009f20 tx=0x7f61f0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec627992b8c14a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 1 -- 172.21.15.116:0/879756546 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0007610 con 0x7f61f809f250 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: dump: 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.777 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 1 -- 172.21.15.116:0/879756546 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f002d050 con 0x7f61f809f250 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 1 -- 172.21.15.116:0/879756546 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f00079a0 con 0x7f61f809f250 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 5 monclient: authenticate success, global_id 224120 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 4 set_mon_vals no callback set 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:50.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61f67fc640 10 monclient: dump: 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 10 monclient: shutdown 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:50.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 -- 172.21.15.116:0/879756546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 msgr2=0x7f61f8138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 --2- 172.21.15.116:0/879756546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8138bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009f20 tx=0x7f61f0009840 comp rx=0 tx=0).stop 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 -- 172.21.15.116:0/879756546 shutdown_connections 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 --2- 172.21.15.116:0/879756546 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8138bc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 --2- 172.21.15.116:0/879756546 >> v2:172.21.15.144:3300/0 conn(0x7f61f809e910 0x7f61f809ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 -- 172.21.15.116:0/879756546 >> 172.21.15.116:0/879756546 conn(0x7f61f8067d30 msgr2=0x7f61f80a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 -- 172.21.15.116:0/879756546 shutdown_connections 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.767+0000 7f61fd554640 1 -- 172.21.15.116:0/879756546 wait complete. 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.768+0000 7f61fd554640 10 monclient: build_initial_monmap 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.768+0000 7f61fd554640 10 monclient: monmap: 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.768+0000 7f61fd554640 1 Processor -- start 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.768+0000 7f61fd554640 1 -- start start 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.768+0000 7f61fd554640 10 monclient: init 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 0x7f61f8130f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): picked mon.noname-a con 0x7f61f809e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): picked mon.noname-c con 0x7f61f809f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f8131ab0 0x7f61f8096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): picked mon.noname-b con 0x7f61f8131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f8099e40 con 0x7f61f809e910 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f8099610 con 0x7f61f8131ab0 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f8099b60 con 0x7f61f809f250 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61fd554640 10 monclient(hunting): authenticate will time out at 1858.090252s 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 0x7f61f8130f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 0x7f61f8130f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58684/0 (socket says 172.21.15.116:58684) 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 -- 172.21.15.116:0/175795225 learned_addr learned my addr 172.21.15.116:0/175795225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): get_auth_request con 0x7f61f809e910 auth_method 0 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42174/0 (socket says 172.21.15.116:42174) 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 10 monclient(hunting): get_auth_request con 0x7f61f809f250 auth_method 0 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f8131ab0 0x7f61f8096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 10 monclient(hunting): get_auth_request con 0x7f61f8131ab0 auth_method 0 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient(hunting): handle_auth_done global_id 194424 payload 824 2024-09-18T12:43:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 -- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 msgr2=0x7f61f8096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 -- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f8131ab0 msgr2=0x7f61f8096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f8131ab0 0x7f61f8096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 monclient: found mon.noname-a 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61f6ffd640 1 -- 172.21.15.116:0/175795225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f0009ae0 con 0x7f61f809e910 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.769+0000 7f61effff640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.771+0000 7f61f6ffd640 1 -- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 msgr2=0x7f61f8130f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.771+0000 7f61f6ffd640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 0x7f61f8130f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61e800c640 tx=0x7f61e8007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 1 -- 172.21.15.116:0/175795225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e8012840 con 0x7f61f809e910 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: dump: 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.786 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: _finish_auth 0 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 1 -- 172.21.15.116:0/175795225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e80129a0 con 0x7f61f809e910 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61fd554640 5 monclient: authenticate success, global_id 194424 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61fd554640 10 monclient: _renew_subs 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61fd554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f8099880 con 0x7f61f809e910 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 1 -- 172.21.15.116:0/175795225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e8012d60 con 0x7f61f809e910 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 10 monclient: dump: 2024-09-18T12:43:50.787 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61fd554640 10 monclient: _renew_subs 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61f5ffb640 4 set_mon_vals no callback set 2024-09-18T12:43:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61fd554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f809a260 con 0x7f61f809e910 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.772+0000 7f61eeffd640 1 -- 172.21.15.116:0/175795225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f61e8011590 con 0x7f61f809e910 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61fd554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61fd554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61fd554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f804bf60 con 0x7f61f809e910 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61eeffd640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f61d8038490 0x7f61d803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61eeffd640 1 -- 172.21.15.116:0/175795225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f61e804c3e0 con 0x7f61f809e910 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61effff640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f61d8038490 0x7f61d803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.773+0000 7f61effff640 10 monclient: get_auth_request con 0x7f61d8038490 auth_method 0 2024-09-18T12:43:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.775+0000 7f61effff640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f61d8038490 0x7f61d803a940 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f61f8131660 tx=0x7f61e0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.777+0000 7f61eeffd640 1 -- 172.21.15.116:0/175795225 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f61e8010ac0 con 0x7f61f809e910 2024-09-18T12:43:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.777+0000 7f61eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.777+0000 7f61eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:50.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.992+0000 7f61fd554640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:43:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.992+0000 7f61fd554640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:43:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.992+0000 7f61fd554640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.992+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f61f8066280 con 0x7f61f809e910 2024-09-18T12:43:51.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.993+0000 7f61eeffd640 1 -- 172.21.15.116:0/175795225 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f61e8016080 con 0x7f61f809e910 2024-09-18T12:43:51.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.993+0000 7f61eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.993+0000 7f61eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f61d8038490 msgr2=0x7f61d803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f61d8038490 0x7f61d803a940 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f61f8131660 tx=0x7f61e0008040 comp rx=0 tx=0).stop 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 10 monclient: shutdown 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 msgr2=0x7f61f8130f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 0x7f61f8130f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f61e800c640 tx=0x7f61e8007ed0 comp rx=0 tx=0).stop 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 shutdown_connections 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f61d8038490 0x7f61d803a940 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f61f8131ab0 0x7f61f8096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f61f809f250 0x7f61f8096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 --2- 172.21.15.116:0/175795225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f809e910 0x7f61f8130f30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 >> 172.21.15.116:0/175795225 conn(0x7f61f8067d30 msgr2=0x7f61f812c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:51.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 shutdown_connections 2024-09-18T12:43:51.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:50.995+0000 7f61fd554640 1 -- 172.21.15.116:0/175795225 wait complete. 2024-09-18T12:43:51.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:43:51.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.476+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.476+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=742849458,config=3140514009,health=2797650238,kv=2187923472,logm=2505359316}) 2024-09-18T12:43:51.477 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.476+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=1541874720,config=3140514009,health=2797650238,kv=2187923472,logm=1743904682}) 2024-09-18T12:43:51.478 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.476+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.478 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.476+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=742849458,config=3140514009,health=2797650238,kv=2187923472,logm=2505359316}) 2024-09-18T12:43:51.478 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.476+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=1,health=12,kv=9,logm=50} crc {auth=3772750202,config=1228700967,health=2797650238,kv=2187923472,logm=4066160220}) 2024-09-18T12:43:51.478 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.477+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.477+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324711369}) 2024-09-18T12:43:51.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.477+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=701505950}) 2024-09-18T12:43:51.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.477+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.477+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2324711369}) 2024-09-18T12:43:51.479 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.477+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646802827}) 2024-09-18T12:43:51.480 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.478+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.480 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.478+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898113383}) 2024-09-18T12:43:51.480 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.479+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=437439498}) 2024-09-18T12:43:51.480 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.479+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.480 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.479+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898113383}) 2024-09-18T12:43:51.480 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.479+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4268182422}) 2024-09-18T12:43:51.481 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.480+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.481 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.480+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613182173}) 2024-09-18T12:43:51.481 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.480+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=508951524}) 2024-09-18T12:43:51.481 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.480+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.481 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.480+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613182173}) 2024-09-18T12:43:51.481 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.480+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4026516631}) 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626919937}) 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654164411}) 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626919937}) 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4072215987}) 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395480169}) 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907837495}) 2024-09-18T12:43:51.483 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395480169}) 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.481+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4008795502}) 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.482+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.482+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107253493}) 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.482+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465333347}) 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.482+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.482+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107253493}) 2024-09-18T12:43:51.484 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.482+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4236366935}) 2024-09-18T12:43:51.486 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.486 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289739342}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388892539}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289739342}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357437702}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069106264}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=554395238}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069106264}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.483+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1257104432}) 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.484+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.487 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.484+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933459260}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.484+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011167805}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.484+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.484+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933459260}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.484+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342682250}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526751921}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3507261879}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526751921}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3211601544}) 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.488 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382152821}) 2024-09-18T12:43:51.489 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413533041}) 2024-09-18T12:43:51.489 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.489 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382152821}) 2024-09-18T12:43:51.489 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.485+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761829343}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.486+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.486+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323611819}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.486+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1096079157}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.486+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.486+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323611819}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.486+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163344357}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.487+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.487+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991028860}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.487+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2749180552}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.487+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.487+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991028860}) 2024-09-18T12:43:51.493 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.487+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458405525}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.488+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.488+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029348492}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.488+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060051709}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.488+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.488+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029348492}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.488+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983681222}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132261417}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2155071793}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132261417}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4197886211}) 2024-09-18T12:43:51.494 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127673366}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.489+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2459836334}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127673366}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697187288}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15975179}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4267847571}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15975179}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.490+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218146636}) 2024-09-18T12:43:51.495 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.491+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.491+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855644178}) 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.491+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3472060407}) 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.491+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.491+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855644178}) 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.491+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890939791}) 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499206917}) 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273018307}) 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499206917}) 2024-09-18T12:43:51.496 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1024724244}) 2024-09-18T12:43:51.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356734636}) 2024-09-18T12:43:51.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2876504324}) 2024-09-18T12:43:51.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356734636}) 2024-09-18T12:43:51.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.492+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732949673}) 2024-09-18T12:43:51.504 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.493+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.493+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212332986}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.493+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3817855296}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.493+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.493+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212332986}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.493+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1439427001}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554748181}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879262203}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554748181}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987685306}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737472037}) 2024-09-18T12:43:51.505 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1751517739}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737472037}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.494+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=185983993}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.495+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.495+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914707067}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.495+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1402722640}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.495+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.495+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914707067}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.495+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2385718269}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.496+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.496+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652085941}) 2024-09-18T12:43:51.506 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.496+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885836316}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.496+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.496+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652085941}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.496+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=962078159}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.496+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418554730}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1990236854}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418554730}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=366308868}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260929651}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3578245909}) 2024-09-18T12:43:51.507 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260929651}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.497+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766676810}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.498+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.498+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403772038}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.498+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171978000}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.498+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.498+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403772038}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.498+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1811666346}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295737626}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029134303}) 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.508 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295737626}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2351155083}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201697980}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430178244}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201697980}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.499+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972432385}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.500+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.500+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733396099}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.500+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3888600810}) 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.500+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.509 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.500+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733396099}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.500+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260389664}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956109858}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3325995681}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956109858}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781607024}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1768641239}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1980790516}) 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.510 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1768641239}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.501+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492463595}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.502+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.502+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271721168}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.502+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2155538478}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.502+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.502+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271721168}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.502+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2832744284}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302508509}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461510892}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302508509}) 2024-09-18T12:43:51.511 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1400826892}) 2024-09-18T12:43:51.512 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.512 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2567071880}) 2024-09-18T12:43:51.512 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877980562}) 2024-09-18T12:43:51.512 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.512 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2567071880}) 2024-09-18T12:43:51.512 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.503+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=881277035}) 2024-09-18T12:43:51.517 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.517 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1873252684}) 2024-09-18T12:43:51.517 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36456544}) 2024-09-18T12:43:51.517 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.517 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1873252684}) 2024-09-18T12:43:51.517 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961014290}) 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850714069}) 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.516+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291027930}) 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850714069}) 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494326223}) 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2010671444}) 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195137819}) 2024-09-18T12:43:51.519 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2010671444}) 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.517+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2110319621}) 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037857820}) 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=413365803}) 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037857820}) 2024-09-18T12:43:51.520 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=997696972}) 2024-09-18T12:43:51.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703649221}) 2024-09-18T12:43:51.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2396624468}) 2024-09-18T12:43:51.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703649221}) 2024-09-18T12:43:51.525 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.518+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983111546}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.519+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.519+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429721081}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.519+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=83002894}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.519+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.519+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429721081}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.519+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078530493}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.520+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.520+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168931874}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.520+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1540159141}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.520+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.520+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168931874}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.520+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2177173689}) 2024-09-18T12:43:51.526 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2580322907}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=506247086}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2580322907}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582487561}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941932944}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262375545}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941932944}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.521+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125909313}) 2024-09-18T12:43:51.527 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.522+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.522+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120878477}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.522+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3155479503}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.522+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.522+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4120878477}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.522+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223430871}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722393058}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285441444}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722393058}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606957778}) 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.528 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967214605}) 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101256918}) 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967214605}) 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.523+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3249286539}) 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.524+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.524+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408442988}) 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.524+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899914464}) 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.524+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.524+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408442988}) 2024-09-18T12:43:51.529 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.524+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077678263}) 2024-09-18T12:43:51.539 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.539 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716616306}) 2024-09-18T12:43:51.539 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782868608}) 2024-09-18T12:43:51.539 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.539 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716616306}) 2024-09-18T12:43:51.539 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440466100}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151803205}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3326510568}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151803205}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.525+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304865566}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.526+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.526+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=508997405}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.526+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3495501167}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.526+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.526+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=508997405}) 2024-09-18T12:43:51.540 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.526+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426690612}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.527+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.527+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881292034}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.527+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=314941761}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.527+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.527+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881292034}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.527+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474485}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677444209}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=713019063}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1677444209}) 2024-09-18T12:43:51.541 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3418589763}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284024910}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379792183}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.528+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284024910}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319057607}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423041284}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=560019921}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423041284}) 2024-09-18T12:43:51.542 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.529+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039670092}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290496371}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1928438666}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290496371}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3310622151}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362499894}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406432979}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362499894}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.530+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1884000103}) 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.531+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.543 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.531+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2925968931}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.531+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4189701114}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.531+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.531+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2925968931}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.531+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3216023417}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388009621}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505529367}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388009621}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622729793}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528852760}) 2024-09-18T12:43:51.544 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192638019}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528852760}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.532+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2055427200}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338920812}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3908957674}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338920812}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783028793}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318840142}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=844490933}) 2024-09-18T12:43:51.545 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318840142}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.533+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219808814}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.534+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.534+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198178123}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.534+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2787245909}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.534+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.534+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198178123}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.534+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=831529721}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860319072}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4079209606}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860319072}) 2024-09-18T12:43:51.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.535+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241350637}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323126337}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2117871245}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323126337}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545068010}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553372455}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848290164}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553372455}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.536+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367136787}) 2024-09-18T12:43:51.547 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433455612}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292972702}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433455612}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1641476448}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518455688}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2415923400}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518455688}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.537+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74176071}) 2024-09-18T12:43:51.548 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.549 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273955007}) 2024-09-18T12:43:51.549 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3085624519}) 2024-09-18T12:43:51.549 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.549 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273955007}) 2024-09-18T12:43:51.549 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.538+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2041365637}) 2024-09-18T12:43:51.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224830046}) 2024-09-18T12:43:51.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=646629556}) 2024-09-18T12:43:51.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224830046}) 2024-09-18T12:43:51.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.539+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813825259}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885953801}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=166807222}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885953801}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4120448145}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.540+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152832197}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687266112}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152832197}) 2024-09-18T12:43:51.554 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2206059398}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414723161}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412633496}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414723161}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.541+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127049162}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987276910}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747775144}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987276910}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1430718720}) 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.555 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101190185}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=586625707}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.542+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101190185}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3711287692}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609663686}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3842891476}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609663686}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.543+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1418552088}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095636853}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057473480}) 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.556 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095636853}) 2024-09-18T12:43:51.557 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.544+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791240525}) 2024-09-18T12:43:51.811 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.811+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.811+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307281272}) 2024-09-18T12:43:51.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.811+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2943339941}) 2024-09-18T12:43:51.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.811+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.811+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307281272}) 2024-09-18T12:43:51.812 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.811+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=852550524}) 2024-09-18T12:43:51.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895369757}) 2024-09-18T12:43:51.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1156167057}) 2024-09-18T12:43:51.813 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895369757}) 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704933507}) 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373072130}) 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956394974}) 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373072130}) 2024-09-18T12:43:51.814 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.812+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633970671}) 2024-09-18T12:43:51.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.813+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.813+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=310148655}) 2024-09-18T12:43:51.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.813+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2378875983}) 2024-09-18T12:43:51.816 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.813+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.813+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=310148655}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.813+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=746403248}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320202573}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835727510}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320202573}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=950489667}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634176783}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009176437}) 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.817 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634176783}) 2024-09-18T12:43:51.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.814+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=198058969}) 2024-09-18T12:43:51.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.815+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.815+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386378344}) 2024-09-18T12:43:51.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.815+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769827538}) 2024-09-18T12:43:51.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.815+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.815+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386378344}) 2024-09-18T12:43:51.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.815+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019468316}) 2024-09-18T12:43:51.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.822+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.822+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329977897}) 2024-09-18T12:43:51.823 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.822+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551419124}) 2024-09-18T12:43:51.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.822+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.822+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329977897}) 2024-09-18T12:43:51.824 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.822+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663668174}) 2024-09-18T12:43:51.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027154807}) 2024-09-18T12:43:51.826 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1052258214}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027154807}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.823+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263565514}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.824+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.824+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619798604}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.824+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1977652810}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.824+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.824+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619798604}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.824+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=109822712}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312988681}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=288091040}) 2024-09-18T12:43:51.827 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312988681}) 2024-09-18T12:43:51.828 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.825+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037114114}) 2024-09-18T12:43:51.833 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.832+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.833 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.832+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874714036}) 2024-09-18T12:43:51.833 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.832+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2872235981}) 2024-09-18T12:43:51.833 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.832+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.833 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.832+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874714036}) 2024-09-18T12:43:51.833 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.832+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890151781}) 2024-09-18T12:43:51.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639142358}) 2024-09-18T12:43:51.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615920828}) 2024-09-18T12:43:51.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639142358}) 2024-09-18T12:43:51.835 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3848769562}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286360642}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.833+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=876194850}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286360642}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2274560089}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917915576}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1682448471}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917915576}) 2024-09-18T12:43:51.836 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.834+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=625177787}) 2024-09-18T12:43:51.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.836+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.836+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=42} crc {logm=625354243,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1307639742,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=603354297}) 2024-09-18T12:43:51.837 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.836+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=45} crc {logm=3796494009,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1931022319,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2471711570}) 2024-09-18T12:43:51.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.836+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.836+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=42} crc {logm=625354243,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1307639742,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=603354297}) 2024-09-18T12:43:51.838 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.836+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=17,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=43} crc {logm=625354243,mdsmap=1531151339,mgr=3801181680,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1931022319,mon_config_key=3655569570,monmap=3140514009,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=88410224}) 2024-09-18T12:43:51.842 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.842+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:43:51.843 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.842+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=13} crc {osdmap=4027644544}) 2024-09-18T12:43:51.843 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:43:51.842+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=12} crc {osdmap=3208057862}) 2024-09-18T12:43:52.254 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:43:52.254 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:43:52.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.357+0000 7f45e62a8640 10 monclient: get_monmap_and_config 2024-09-18T12:43:52.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.357+0000 7f45e62a8640 10 monclient: build_initial_monmap 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.357+0000 7f45e62a8640 10 monclient: monmap: 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:52.357728+0000 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:52.357728+0000 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:52.358 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.357+0000 7f45e62a8640 1 Processor -- start 2024-09-18T12:43:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 1 -- start start 2024-09-18T12:43:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient: init 2024-09-18T12:43:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T12:43:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45e01318f0 0x7f45e0131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient(hunting): picked mon.noname-a con 0x7f45e01318f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45e009ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f45e0136c20 0x7f45e0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient(hunting): picked mon.noname-c con 0x7f45e0136c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e009f530 con 0x7f45e009ea50 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e0131d50 con 0x7f45e01318f0 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e009fbf0 con 0x7f45e0136c20 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e62a8640 10 monclient(hunting): authenticate will time out at 1859.679653s 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45e01318f0 0x7f45e0131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45e01318f0 0x7f45e0131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58692/0 (socket says 172.21.15.116:58692) 2024-09-18T12:43:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 1 -- 172.21.15.116:0/417356517 learned_addr learned my addr 172.21.15.116:0/417356517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 10 monclient(hunting): get_auth_request con 0x7f45e01318f0 auth_method 0 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e4845640 1 --2- 172.21.15.116:0/417356517 >> v2:172.21.15.144:3300/0 conn(0x7f45e0136c20 0x7f45e0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e4845640 10 monclient(hunting): get_auth_request con 0x7f45e0136c20 auth_method 0 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e4845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e4845640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.358+0000 7f45e4845640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 10 monclient(hunting): handle_auth_done global_id 194427 payload 274 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 1 -- 172.21.15.116:0/417356517 >> v2:172.21.15.144:3300/0 conn(0x7f45e0136c20 msgr2=0x7f45e0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 1 --2- 172.21.15.116:0/417356517 >> v2:172.21.15.144:3300/0 conn(0x7f45e0136c20 0x7f45e0139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 1 -- 172.21.15.116:0/417356517 >> v1:172.21.15.144:6789/0 conn(0x7f45e009ea50 legacy=0x7f45e009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 1 monclient: found mon.noname-a 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.359+0000 7f45dffff640 1 -- 172.21.15.116:0/417356517 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e009fe30 con 0x7f45e01318f0 2024-09-18T12:43:52.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.361+0000 7f45dffff640 1 -- 172.21.15.116:0/417356517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 msgr2=0x7f45e0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.361+0000 7f45dffff640 1 --2- 172.21.15.116:0/417356517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0131f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45e009c9e0 tx=0x7f45d00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccda198e5222b14f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 1 -- 172.21.15.116:0/417356517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0032440 con 0x7f45e01318f0 2024-09-18T12:43:52.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:52.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: dump: 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: _finish_auth 0 2024-09-18T12:43:52.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 1 -- 172.21.15.116:0/417356517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d00325a0 con 0x7f45e01318f0 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 5 monclient: authenticate success, global_id 194427 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 1 -- 172.21.15.116:0/417356517 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0032940 con 0x7f45e01318f0 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 4 set_mon_vals no callback set 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45df7fe640 10 monclient: dump: 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.372 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 10 monclient: shutdown 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:52.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 -- 172.21.15.116:0/417356517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 msgr2=0x7f45e0131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 --2- 172.21.15.116:0/417356517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0131f30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45e009c9e0 tx=0x7f45d00091d0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 -- 172.21.15.116:0/417356517 shutdown_connections 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 --2- 172.21.15.116:0/417356517 >> v2:172.21.15.144:3300/0 conn(0x7f45e0136c20 0x7f45e0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 --2- 172.21.15.116:0/417356517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0131f30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 -- 172.21.15.116:0/417356517 >> 172.21.15.116:0/417356517 conn(0x7f45e0067110 msgr2=0x7f45e0067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 -- 172.21.15.116:0/417356517 shutdown_connections 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.362+0000 7f45e62a8640 1 -- 172.21.15.116:0/417356517 wait complete. 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient: build_initial_monmap 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient: monmap: 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:52.374 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 Processor -- start 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 -- start start 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient: init 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45e009ea50 0x7f45e0130990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient(hunting): picked mon.noname-c con 0x7f45e009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:52.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0130ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient(hunting): picked mon.noname-a con 0x7f45e01318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45e0136c20 0x7f45e00997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient(hunting): picked mon.noname-b con 0x7f45e0136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e009e090 con 0x7f45e01318f0 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e011cda0 con 0x7f45e0136c20 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e0141ed0 con 0x7f45e009ea50 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.363+0000 7f45e62a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45e62a8640 10 monclient(hunting): authenticate will time out at 1859.685007s 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0130ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45e009ea50 0x7f45e0130990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0130ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58706/0 (socket says 172.21.15.116:58706) 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 1 -- 172.21.15.116:0/4122110652 learned_addr learned my addr 172.21.15.116:0/4122110652 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 10 monclient(hunting): get_auth_request con 0x7f45e01318f0 auth_method 0 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): get_auth_request con 0x7f45e009ea50 auth_method 0 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient(hunting): handle_auth_done global_id 224126 payload 824 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45e4845640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45e0136c20 0x7f45e00997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 1 -- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45e0136c20 msgr2=0x7f45e00997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45e0136c20 0x7f45e00997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 1 -- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 msgr2=0x7f45e0130ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0130ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:52.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 1 monclient: found mon.noname-c 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45dffff640 1 -- 172.21.15.116:0/4122110652 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d0009450 con 0x7f45e009ea50 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.364+0000 7f45d7fff640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0130ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dffff640 1 -- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45e009ea50 msgr2=0x7f45e0130990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dffff640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45e009ea50 0x7f45e0130990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f45d0031000 tx=0x7f45d0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 1 -- 172.21.15.116:0/4122110652 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0031b10 con 0x7f45e009ea50 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: dump: 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.379 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: _finish_auth 0 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 1 -- 172.21.15.116:0/4122110652 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0031c90 con 0x7f45e009ea50 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45e62a8640 5 monclient: authenticate success, global_id 224126 2024-09-18T12:43:52.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 1 -- 172.21.15.116:0/4122110652 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d003d690 con 0x7f45e009ea50 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45e62a8640 10 monclient: _renew_subs 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45e62a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e011bb60 con 0x7f45e009ea50 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45dcff9640 10 monclient: dump: 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.381 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45deffd640 4 set_mon_vals no callback set 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45e62a8640 10 monclient: _renew_subs 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45e62a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.366+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e009b2c0 con 0x7f45e009ea50 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.367+0000 7f45dcff9640 1 -- 172.21.15.116:0/4122110652 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f45d0039040 con 0x7f45e009ea50 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.367+0000 7f45dcff9640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f45c0038490 0x7f45c003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.367+0000 7f45dcff9640 1 -- 172.21.15.116:0/4122110652 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f45d006bf90 con 0x7f45e009ea50 2024-09-18T12:43:52.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.367+0000 7f45e62a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:52.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.367+0000 7f45e62a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:52.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.367+0000 7f45e62a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.367+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e004bf60 con 0x7f45e009ea50 2024-09-18T12:43:52.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.368+0000 7f45d7fff640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f45c0038490 0x7f45c003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.368+0000 7f45d7fff640 10 monclient: get_auth_request con 0x7f45c0038490 auth_method 0 2024-09-18T12:43:52.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.370+0000 7f45d7fff640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f45c0038490 0x7f45c003a940 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f45e009de40 tx=0x7f45c8007ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.372+0000 7f45dcff9640 1 -- 172.21.15.116:0/4122110652 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f45d0037070 con 0x7f45e009ea50 2024-09-18T12:43:52.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.372+0000 7f45dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:52.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.372+0000 7f45dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.584+0000 7f45e62a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.584+0000 7f45e62a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.584+0000 7f45e62a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.584+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f45e00638d0 con 0x7f45e009ea50 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.585+0000 7f45dcff9640 1 -- 172.21.15.116:0/4122110652 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f45d0031e00 con 0x7f45e009ea50 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.585+0000 7f45dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.585+0000 7f45dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:52.586 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f45c0038490 msgr2=0x7f45c003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f45c0038490 0x7f45c003a940 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f45e009de40 tx=0x7f45c8007ed0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 10 monclient: shutdown 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45e009ea50 msgr2=0x7f45e0130990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45e009ea50 0x7f45e0130990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f45d0031000 tx=0x7f45d0032000 comp rx=0 tx=0).stop 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 shutdown_connections 2024-09-18T12:43:52.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f45c0038490 0x7f45c003a940 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f45e0136c20 0x7f45e00997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e01318f0 0x7f45e0130ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 --2- 172.21.15.116:0/4122110652 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f45e009ea50 0x7f45e0130990 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 >> 172.21.15.116:0/4122110652 conn(0x7f45e0067110 msgr2=0x7f45e00a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:52.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 shutdown_connections 2024-09-18T12:43:52.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.587+0000 7f45e62a8640 1 -- 172.21.15.116:0/4122110652 wait complete. 2024-09-18T12:43:52.598 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:43:52.598 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:43:52.598 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:43:52.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:43:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.689+0000 7f0e69e34640 10 monclient: get_monmap_and_config 2024-09-18T12:43:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.689+0000 7f0e69e34640 10 monclient: build_initial_monmap 2024-09-18T12:43:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.689+0000 7f0e69e34640 10 monclient: monmap: 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:52.690550+0000 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:52.690550+0000 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:52.691 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.690+0000 7f0e69e34640 1 Processor -- start 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.690+0000 7f0e69e34640 1 -- start start 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.690+0000 7f0e69e34640 10 monclient: init 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e641318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e6409ea50 0x7f0e6409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient(hunting): picked mon.noname-c con 0x7f0e6409ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0e64136c20 0x7f0e64139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient(hunting): picked mon.noname-a con 0x7f0e64136c20 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e64131cc0 con 0x7f0e641318f0 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6409fbf0 con 0x7f0e64136c20 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6409fe30 con 0x7f0e6409ea50 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e69e34640 10 monclient(hunting): authenticate will time out at 1860.012251s 2024-09-18T12:43:52.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e637fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0e641318f0 0x7f0e64131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40684/0 (socket says 172.21.15.116:40684) 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e637fe640 1 -- 172.21.15.116:0/2646465368 learned_addr learned my addr 172.21.15.116:0/2646465368 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e63fff640 1 --2- 172.21.15.116:0/2646465368 >> v2:172.21.15.116:3300/0 conn(0x7f0e64136c20 0x7f0e64139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e63fff640 10 monclient(hunting): get_auth_request con 0x7f0e64136c20 auth_method 0 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e63fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 1 --2- 172.21.15.116:0/2646465368 >> v2:172.21.15.144:3300/0 conn(0x7f0e6409ea50 0x7f0e6409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): get_auth_request con 0x7f0e6409ea50 auth_method 0 2024-09-18T12:43:52.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient(hunting): handle_auth_done global_id 174571 payload 274 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 1 -- 172.21.15.116:0/2646465368 >> v2:172.21.15.116:3300/0 conn(0x7f0e64136c20 msgr2=0x7f0e64139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 1 --2- 172.21.15.116:0/2646465368 >> v2:172.21.15.116:3300/0 conn(0x7f0e64136c20 0x7f0e64139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 1 -- 172.21.15.116:0/2646465368 >> v1:172.21.15.116:6789/0 conn(0x7f0e641318f0 legacy=0x7f0e64131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:52.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:43:52.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 1 monclient: found mon.noname-c 2024-09-18T12:43:52.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.691+0000 7f0e62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:52.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.692+0000 7f0e62ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:43:52.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.692+0000 7f0e62ffd640 1 -- 172.21.15.116:0/2646465368 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640a1210 con 0x7f0e6409ea50 2024-09-18T12:43:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.694+0000 7f0e62ffd640 1 -- 172.21.15.116:0/2646465368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6409ea50 msgr2=0x7f0e6409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.694+0000 7f0e62ffd640 1 --2- 172.21.15.116:0/2646465368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6409ea50 0x7f0e6409ee20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e6406f600 tx=0x7f0e580090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1136255a2ec42d52 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 1 -- 172.21.15.116:0/2646465368 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5802adf0 con 0x7f0e6409ea50 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: dump: 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: _finish_auth 0 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 1 -- 172.21.15.116:0/2646465368 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e58032440 con 0x7f0e6409ea50 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 1 -- 172.21.15.116:0/2646465368 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e580327e0 con 0x7f0e6409ea50 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e69e34640 5 monclient: authenticate success, global_id 174571 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e69e34640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e69e34640 4 set_mon_vals no callback set 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.695+0000 7f0e627fc640 10 monclient: dump: 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 10 monclient: shutdown 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 -- 172.21.15.116:0/2646465368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6409ea50 msgr2=0x7f0e6409ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 --2- 172.21.15.116:0/2646465368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6409ea50 0x7f0e6409ee20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e6406f600 tx=0x7f0e580090f0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 -- 172.21.15.116:0/2646465368 shutdown_connections 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 --2- 172.21.15.116:0/2646465368 >> v2:172.21.15.116:3300/0 conn(0x7f0e64136c20 0x7f0e64139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 --2- 172.21.15.116:0/2646465368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e6409ea50 0x7f0e6409ee20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 -- 172.21.15.116:0/2646465368 >> 172.21.15.116:0/2646465368 conn(0x7f0e64067110 msgr2=0x7f0e64067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 -- 172.21.15.116:0/2646465368 shutdown_connections 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 1 -- 172.21.15.116:0/2646465368 wait complete. 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 10 monclient: build_initial_monmap 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.696+0000 7f0e69e34640 10 monclient: monmap: 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 Processor -- start 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 -- start start 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient: init 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:43:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6409ea50 0x7f0e6409aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6409ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e641318f0 0x7f0e64096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): picked mon.noname-b con 0x7f0e641318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e64136c20 0x7f0e64096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): picked mon.noname-c con 0x7f0e64136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6409bf10 con 0x7f0e6409ea50 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6409df40 con 0x7f0e641318f0 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6409af60 con 0x7f0e64136c20 2024-09-18T12:43:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e69e34640 10 monclient(hunting): authenticate will time out at 1860.018827s 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6409ea50 0x7f0e6409aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6409ea50 0x7f0e6409aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58718/0 (socket says 172.21.15.116:58718) 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e637fe640 1 -- 172.21.15.116:0/1715455490 learned_addr learned my addr 172.21.15.116:0/1715455490 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e637fe640 10 monclient(hunting): get_auth_request con 0x7f0e6409ea50 auth_method 0 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e637fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e63fff640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e64136c20 0x7f0e64096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e63fff640 10 monclient(hunting): get_auth_request con 0x7f0e64136c20 auth_method 0 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e63fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.697+0000 7f0e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e62ffd640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e641318f0 0x7f0e64096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 10 monclient(hunting): handle_auth_done global_id 224129 payload 824 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e641318f0 msgr2=0x7f0e64096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e641318f0 0x7f0e64096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6409ea50 msgr2=0x7f0e6409aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6409ea50 0x7f0e6409aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 1 monclient: found mon.noname-c 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e63fff640 1 -- 172.21.15.116:0/1715455490 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58009390 con 0x7f0e64136c20 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.698+0000 7f0e637fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6409ea50 0x7f0e6409aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e63fff640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e64136c20 msgr2=0x7f0e64096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e63fff640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e64136c20 0x7f0e64096fa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c007ed0 tx=0x7f0e5c009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00ee30 con 0x7f0e64136c20 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: dump: 2024-09-18T12:43:52.716 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: _finish_auth 0 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c013590 con 0x7f0e64136c20 2024-09-18T12:43:52.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e69e34640 5 monclient: authenticate success, global_id 224129 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c013950 con 0x7f0e64136c20 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e69e34640 10 monclient: _renew_subs 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e69e34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6406e010 con 0x7f0e64136c20 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e4b7fe640 10 monclient: dump: 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.718 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e61ffb640 4 set_mon_vals no callback set 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e69e34640 10 monclient: _renew_subs 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e69e34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.700+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e64098a40 con 0x7f0e64136c20 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e69e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:52.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e69e34640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 0x7f0e6406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e69e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64065e00 con 0x7f0e640974e0 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0e5c003020 con 0x7f0e64136c20 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 0x7f0e6406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient: get_auth_request con 0x7f0e640974e0 auth_method 0 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e4b7fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e50038490 0x7f0e5003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 10 monclient(hunting): handle_auth_done global_id 194439 payload 824 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e637fe640 1 -- 172.21.15.116:0/1715455490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e640978b0 con 0x7f0e640974e0 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.701+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0e5c04c820 con 0x7f0e64136c20 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.702+0000 7f0e62ffd640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e50038490 0x7f0e5003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.702+0000 7f0e62ffd640 10 monclient: get_auth_request con 0x7f0e50038490 auth_method 0 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.703+0000 7f0e637fe640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 msgr2=0x7f0e6406ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.703+0000 7f0e637fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 0x7f0e6406ab10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0e54007320 tx=0x7f0e54004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400fd50 con 0x7f0e640974e0 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 10 monclient: dump: 2024-09-18T12:43:52.721 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0e5400feb0 con 0x7f0e640974e0 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:52.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 msgr2=0x7f0e6406ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 0x7f0e6406ab10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0e54007320 tx=0x7f0e54004960 comp rx=0 tx=0).stop 2024-09-18T12:43:52.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 msgr2=0x7f0e6406ab10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:52.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e4b7fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e640974e0 0x7f0e6406ab10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.704+0000 7f0e62ffd640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e50038490 0x7f0e5003a940 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0e58031000 tx=0x7f0e58032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.709+0000 7f0e69e34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.709+0000 7f0e69e34640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 0x7f0e6406b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.709+0000 7f0e69e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.709+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6404bf60 con 0x7f0e6406b050 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 0x7f0e6406b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient: get_auth_request con 0x7f0e6406b050 auth_method 0 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 10 monclient(hunting): handle_auth_done global_id 194442 payload 824 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.710+0000 7f0e637fe640 1 -- 172.21.15.116:0/1715455490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0e641316d0 con 0x7f0e6406b050 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e637fe640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 msgr2=0x7f0e6406b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e637fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 0x7f0e6406b450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e54007320 tx=0x7f0e54018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5401caa0 con 0x7f0e6406b050 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 10 monclient: dump: 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.726 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f0e5400fb40 con 0x7f0e6406b050 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:52.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 msgr2=0x7f0e6406b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 0x7f0e6406b450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e54007320 tx=0x7f0e54018000 comp rx=0 tx=0).stop 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 msgr2=0x7f0e6406b450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.712+0000 7f0e4b7fe640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e6406b050 0x7f0e6406b450 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e54007320 tx=0x7f0e54018000 comp rx=0 tx=0).stop 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 182, 2024-09-18T12:43:52.728 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:52.729 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:52.730 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:52.731 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:52.732 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.733 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:52.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:52.736 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:52.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.738 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.739 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.740 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:52.741 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:52.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e50038490 msgr2=0x7f0e5003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0e50038490 0x7f0e5003a940 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0e58031000 tx=0x7f0e58032000 comp rx=0 tx=0).stop 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 10 monclient: shutdown 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e64136c20 msgr2=0x7f0e64096fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 --2- 172.21.15.116:0/1715455490 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e64136c20 0x7f0e64096fa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c007ed0 tx=0x7f0e5c009850 comp rx=0 tx=0).stop 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e63fff640 1 -- 172.21.15.116:0/1715455490 reap_dead start 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e63fff640 1 -- 172.21.15.116:0/1715455490 reap_dead start 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 shutdown_connections 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 >> 172.21.15.116:0/1715455490 conn(0x7f0e64067110 msgr2=0x7f0e640a1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 shutdown_connections 2024-09-18T12:43:52.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.713+0000 7f0e69e34640 1 -- 172.21.15.116:0/1715455490 wait complete. 2024-09-18T12:43:52.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:43:52.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:43:52.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:43:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.837+0000 7ff76f59d640 10 monclient: get_monmap_and_config 2024-09-18T12:43:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.837+0000 7ff76f59d640 10 monclient: build_initial_monmap 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.837+0000 7ff76f59d640 10 monclient: monmap: 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:52.838426+0000 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:52.838426+0000 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:52.839 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.838+0000 7ff76f59d640 1 Processor -- start 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.838+0000 7ff76f59d640 1 -- start start 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.838+0000 7ff76f59d640 10 monclient: init 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.838+0000 7ff76f59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.838+0000 7ff76f59d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff7680975f0 0x7ff768095ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 10 monclient(hunting): picked mon.noname-c con 0x7ff7680975f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7680979c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 0x7ff76809c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 10 monclient(hunting): picked mon.noname-b con 0x7ff768097dc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff768098200 con 0x7ff7680979c0 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76809d1b0 con 0x7ff7680975f0 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76809d420 con 0x7ff768097dc0 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76f59d640 10 monclient(hunting): authenticate will time out at 1860.160109s 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 0x7ff76809c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 0x7ff76809c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42228/0 (socket says 172.21.15.116:42228) 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 -- 172.21.15.116:0/715512330 learned_addr learned my addr 172.21.15.116:0/715512330 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): get_auth_request con 0x7ff768097dc0 auth_method 0 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient(hunting): handle_auth_done global_id 224132 payload 274 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76d339640 1 --2- 172.21.15.116:0/715512330 >> v2:172.21.15.144:3300/0 conn(0x7ff7680975f0 0x7ff768095ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 -- 172.21.15.116:0/715512330 >> v2:172.21.15.144:3300/0 conn(0x7ff7680975f0 msgr2=0x7ff768095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 --2- 172.21.15.116:0/715512330 >> v2:172.21.15.144:3300/0 conn(0x7ff7680975f0 0x7ff768095ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 -- 172.21.15.116:0/715512330 >> v1:172.21.15.144:6789/0 conn(0x7ff7680979c0 legacy=0x7ff76809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient: _finish_hunting 0 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 monclient: found mon.noname-b 2024-09-18T12:43:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:43:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.839+0000 7ff76db3a640 1 -- 172.21.15.116:0/715512330 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76809d9e0 con 0x7ff768097dc0 2024-09-18T12:43:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76db3a640 1 -- 172.21.15.116:0/715512330 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 msgr2=0x7ff76809c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76db3a640 1 --2- 172.21.15.116:0/715512330 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 0x7ff76809c660 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff76806a3f0 tx=0x7ff760009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=269127db6d5b82cf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 1 -- 172.21.15.116:0/715512330 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760007610 con 0x7ff768097dc0 2024-09-18T12:43:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:43:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: dump: 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: _finish_auth 0 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 1 -- 172.21.15.116:0/715512330 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76002d050 con 0x7ff768097dc0 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 5 monclient: authenticate success, global_id 224132 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 1 -- 172.21.15.116:0/715512330 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7600079a0 con 0x7ff768097dc0 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff75bfff640 10 monclient: dump: 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.855 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 4 set_mon_vals no callback set 2024-09-18T12:43:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 10 monclient: shutdown 2024-09-18T12:43:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 1 -- 172.21.15.116:0/715512330 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 msgr2=0x7ff76809c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 1 --2- 172.21.15.116:0/715512330 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 0x7ff76809c660 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff76806a3f0 tx=0x7ff760009840 comp rx=0 tx=0).stop 2024-09-18T12:43:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 1 -- 172.21.15.116:0/715512330 shutdown_connections 2024-09-18T12:43:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 1 --2- 172.21.15.116:0/715512330 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff768097dc0 0x7ff76809c660 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 1 --2- 172.21.15.116:0/715512330 >> v2:172.21.15.144:3300/0 conn(0x7ff7680975f0 0x7ff768095ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.842+0000 7ff76f59d640 1 -- 172.21.15.116:0/715512330 >> 172.21.15.116:0/715512330 conn(0x7ff7680678a0 msgr2=0x7ff7680a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:52.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.843+0000 7ff76f59d640 1 -- 172.21.15.116:0/715512330 shutdown_connections 2024-09-18T12:43:52.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.843+0000 7ff76f59d640 1 -- 172.21.15.116:0/715512330 wait complete. 2024-09-18T12:43:52.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.843+0000 7ff76f59d640 10 monclient: build_initial_monmap 2024-09-18T12:43:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.843+0000 7ff76f59d640 10 monclient: monmap: 2024-09-18T12:43:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:52.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.843+0000 7ff76f59d640 1 Processor -- start 2024-09-18T12:43:52.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.843+0000 7ff76f59d640 1 -- start start 2024-09-18T12:43:52.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.843+0000 7ff76f59d640 10 monclient: init 2024-09-18T12:43:52.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:52.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:43:52.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7680975f0 0x7ff7680940b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7680975f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:52.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7680979c0 0x7ff7680945f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): picked mon.noname-c con 0x7ff7680979c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:52.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff768097dc0 0x7ff768094b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): picked mon.noname-a con 0x7ff768097dc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:52.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff768098430 con 0x7ff768097dc0 2024-09-18T12:43:52.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76812bd00 con 0x7ff7680975f0 2024-09-18T12:43:52.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76812cd50 con 0x7ff7680979c0 2024-09-18T12:43:52.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:52.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76f59d640 10 monclient(hunting): authenticate will time out at 1860.165498s 2024-09-18T12:43:52.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7680979c0 0x7ff7680945f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff768097dc0 0x7ff768094b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff768097dc0 0x7ff768094b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58752/0 (socket says 172.21.15.116:58752) 2024-09-18T12:43:52.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 -- 172.21.15.116:0/2433009417 learned_addr learned my addr 172.21.15.116:0/2433009417 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:52.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): get_auth_request con 0x7ff768097dc0 auth_method 0 2024-09-18T12:43:52.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7680979c0 0x7ff7680945f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42238/0 (socket says 172.21.15.116:42238) 2024-09-18T12:43:52.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 10 monclient(hunting): get_auth_request con 0x7ff7680979c0 auth_method 0 2024-09-18T12:43:52.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7680975f0 0x7ff7680940b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 10 monclient(hunting): get_auth_request con 0x7ff7680975f0 auth_method 0 2024-09-18T12:43:52.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient(hunting): handle_auth_done global_id 194445 payload 824 2024-09-18T12:43:52.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7680979c0 msgr2=0x7ff7680945f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7680979c0 0x7ff7680945f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7680975f0 msgr2=0x7ff7680940b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7680975f0 0x7ff7680940b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient: _finish_hunting 0 2024-09-18T12:43:52.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 monclient: found mon.noname-a 2024-09-18T12:43:52.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76db3a640 1 -- 172.21.15.116:0/2433009417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff760009ae0 con 0x7ff768097dc0 2024-09-18T12:43:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.844+0000 7ff76cb38640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff7680979c0 0x7ff7680945f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76db3a640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff768097dc0 msgr2=0x7ff768094b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76db3a640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff768097dc0 0x7ff768094b30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff760009880 tx=0x7ff760036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7600094e0 con 0x7ff768097dc0 2024-09-18T12:43:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: dump: 2024-09-18T12:43:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.872 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.872 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.872 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.872 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.872 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.872 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.873 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.873 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: _finish_auth 0 2024-09-18T12:43:52.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:52.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76f59d640 5 monclient: authenticate success, global_id 194445 2024-09-18T12:43:52.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff760007e10 con 0x7ff768097dc0 2024-09-18T12:43:52.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76f59d640 10 monclient: _renew_subs 2024-09-18T12:43:52.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76f59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:52.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76812ba80 con 0x7ff768097dc0 2024-09-18T12:43:52.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:52.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76003c420 con 0x7ff768097dc0 2024-09-18T12:43:52.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:52.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff7597fa640 10 monclient: dump: 2024-09-18T12:43:52.875 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.875 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.875 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.876 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.876 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.876 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.876 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.876 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.877 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.877 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76f59d640 10 monclient: _renew_subs 2024-09-18T12:43:52.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76f59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:52.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff75b7fe640 4 set_mon_vals no callback set 2024-09-18T12:43:52.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.847+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7681325d0 con 0x7ff768097dc0 2024-09-18T12:43:52.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76f59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:52.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76f59d640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 0x7ff76806dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76f59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76812c180 con 0x7ff76806d7f0 2024-09-18T12:43:52.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 0x7ff76806dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient: get_auth_request con 0x7ff76806d7f0 auth_method 0 2024-09-18T12:43:52.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ff76003c790 con 0x7ff768097dc0 2024-09-18T12:43:52.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 10 monclient(hunting): handle_auth_done global_id 224138 payload 824 2024-09-18T12:43:52.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76d339640 1 -- 172.21.15.116:0/2433009417 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff76804bf60 con 0x7ff76806d7f0 2024-09-18T12:43:52.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff7597fa640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff74c038490 0x7ff74c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ff76006d3d0 con 0x7ff768097dc0 2024-09-18T12:43:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76cb38640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff74c038490 0x7ff74c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.848+0000 7ff76cb38640 10 monclient: get_auth_request con 0x7ff74c038490 auth_method 0 2024-09-18T12:43:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff76d339640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 msgr2=0x7ff76806dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff76d339640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 0x7ff76806dbc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff76400f880 tx=0x7ff764008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76400fe70 con 0x7ff76806d7f0 2024-09-18T12:43:52.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:52.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 10 monclient: dump: 2024-09-18T12:43:52.882 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.882 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.883 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.883 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.883 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.883 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.883 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.884 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff76400c040 con 0x7ff76806d7f0 2024-09-18T12:43:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:52.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 msgr2=0x7ff76806dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 0x7ff76806dbc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff76400f880 tx=0x7ff764008040 comp rx=0 tx=0).stop 2024-09-18T12:43:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 msgr2=0x7ff76806dbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.850+0000 7ff7597fa640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806d7f0 0x7ff76806dbc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:52.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.851+0000 7ff76cb38640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff74c038490 0x7ff74c03a940 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff75c006fd0 tx=0x7ff75c009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76f59d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76f59d640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 0x7ff7681284a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76f59d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff768065e00 con 0x7ff76806e100 2024-09-18T12:43:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 0x7ff7681284a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient: get_auth_request con 0x7ff76806e100 auth_method 0 2024-09-18T12:43:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.856+0000 7ff76d339640 10 monclient(hunting): handle_auth_done global_id 224141 payload 824 2024-09-18T12:43:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.857+0000 7ff76d339640 1 -- 172.21.15.116:0/2433009417 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff7680638d0 con 0x7ff76806e100 2024-09-18T12:43:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76d339640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 msgr2=0x7ff7681284a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76d339640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 0x7ff7681284a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff76400f880 tx=0x7ff76401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76401bc20 con 0x7ff76806e100 2024-09-18T12:43:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:52.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 10 monclient: dump: 2024-09-18T12:43:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.893 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:52.894 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:52.894 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:52.894 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:52.894 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:52.894 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:52.894 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:52.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7ff76400db30 con 0x7ff76806e100 2024-09-18T12:43:52.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:52.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:52.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 msgr2=0x7ff7681284a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 0x7ff7681284a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff76400f880 tx=0x7ff76401c000 comp rx=0 tx=0).stop 2024-09-18T12:43:52.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 msgr2=0x7ff7681284a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:52.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff7597fa640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff76806e100 0x7ff7681284a0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff76400f880 tx=0x7ff76401c000 comp rx=0 tx=0).stop 2024-09-18T12:43:52.896 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:52.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:52.897 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:52.897 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:43:52.897 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 182, 2024-09-18T12:43:52.897 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:52.897 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:52.897 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:43:52.898 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:43:52.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.898 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:43:52.898 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:52.898 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:52.898 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:52.899 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:52.899 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:52.899 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:52.899 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:52.899 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:52.899 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:52.900 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:52.900 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:52.900 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:52.900 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.900 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:52.900 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:52.901 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:52.901 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:52.901 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:52.901 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:52.901 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:52.901 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:52.902 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:52.902 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:52.902 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:52.902 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.902 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:52.903 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:52.903 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:52.903 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:52.903 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:52.903 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:52.903 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:52.904 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:52.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:52.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:52.904 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:52.904 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:52.904 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:52.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:52.905 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:52.905 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:52.905 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:52.905 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:52.906 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:52.906 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:52.906 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:52.906 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:52.906 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:52.906 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:52.907 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:52.907 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:52.907 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.907 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:52.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.907 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:52.908 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.908 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:52.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:52.908 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:52.908 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:52.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:52.909 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:52.909 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:52.910 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:52.910 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.910 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.910 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.911 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:52.911 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:52.911 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:52.911 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:52.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:52.919 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff74c038490 msgr2=0x7ff74c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76f59d640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ff74c038490 0x7ff74c03a940 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff75c006fd0 tx=0x7ff75c009060 comp rx=0 tx=0).stop 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76f59d640 10 monclient: shutdown 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76f59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff768097dc0 msgr2=0x7ff768094b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76f59d640 1 --2- 172.21.15.116:0/2433009417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff768097dc0 0x7ff768094b30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff760009880 tx=0x7ff760036000 comp rx=0 tx=0).stop 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76db3a640 1 -- 172.21.15.116:0/2433009417 reap_dead start 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.859+0000 7ff76db3a640 1 -- 172.21.15.116:0/2433009417 reap_dead start 2024-09-18T12:43:52.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.860+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 shutdown_connections 2024-09-18T12:43:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.860+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 >> 172.21.15.116:0/2433009417 conn(0x7ff7680678a0 msgr2=0x7ff768132900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.860+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 shutdown_connections 2024-09-18T12:43:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:52.860+0000 7ff76f59d640 1 -- 172.21.15.116:0/2433009417 wait complete. 2024-09-18T12:43:52.922 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:43:52.922 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:43:52.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:43:53.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.018+0000 7fdb980c9640 10 monclient: get_monmap_and_config 2024-09-18T12:43:53.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.018+0000 7fdb980c9640 10 monclient: build_initial_monmap 2024-09-18T12:43:53.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.018+0000 7fdb980c9640 10 monclient: monmap: 2024-09-18T12:43:53.019 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:53.019 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.019 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:53.018813+0000 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:53.018813+0000 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.018+0000 7fdb980c9640 1 Processor -- start 2024-09-18T12:43:53.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.018+0000 7fdb980c9640 1 -- start start 2024-09-18T12:43:53.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.018+0000 7fdb980c9640 10 monclient: init 2024-09-18T12:43:53.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb901318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb9009ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb90137c30 0x7fdb9013a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient(hunting): picked mon.noname-a con 0x7fdb90137c30 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb90131cc0 con 0x7fdb9009ea50 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9009f530 con 0x7fdb901318f0 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb9009fbf0 con 0x7fdb90137c30 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb980c9640 10 monclient(hunting): authenticate will time out at 1860.340449s 2024-09-18T12:43:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb90137c30 0x7fdb9013a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb90137c30 0x7fdb9013a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58758/0 (socket says 172.21.15.116:58758) 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 1 -- 172.21.15.116:0/3232644518 learned_addr learned my addr 172.21.15.116:0/3232644518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 10 monclient(hunting): get_auth_request con 0x7fdb90137c30 auth_method 0 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb95664640 1 --1- 172.21.15.116:0/3232644518 >> v1:172.21.15.116:6789/0 conn(0x7fdb9009ea50 0x7fdb9009ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40690/0 (socket says 172.21.15.116:40690) 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.019+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb95664640 1 -- 172.21.15.116:0/3232644518 >> v1:172.21.15.116:6789/0 conn(0x7fdb9009ea50 legacy=0x7fdb9009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 10 monclient(hunting): handle_auth_done global_id 194448 payload 274 2024-09-18T12:43:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 1 -- 172.21.15.116:0/3232644518 >> v1:172.21.15.144:6789/0 conn(0x7fdb901318f0 legacy=0x7fdb90131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb95e65640 1 -- 172.21.15.116:0/3232644518 >> v1:172.21.15.144:6789/0 conn(0x7fdb901318f0 legacy=0x7fdb90131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 1 -- 172.21.15.116:0/3232644518 >> v1:172.21.15.116:6789/0 conn(0x7fdb9009ea50 legacy=0x7fdb9009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 10 monclient: _finish_hunting 0 2024-09-18T12:43:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 1 monclient: found mon.noname-a 2024-09-18T12:43:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.021+0000 7fdb96666640 1 -- 172.21.15.116:0/3232644518 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9009fe30 con 0x7fdb90137c30 2024-09-18T12:43:53.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb96666640 1 -- 172.21.15.116:0/3232644518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb90137c30 msgr2=0x7fdb9013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb96666640 1 --2- 172.21.15.116:0/3232644518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb90137c30 0x7fdb9013a010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009a10 tx=0x7fdb880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da36c5ac9b001b55 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 1 -- 172.21.15.116:0/3232644518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8802f4b0 con 0x7fdb90137c30 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: dump: 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.033 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: _finish_auth 0 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb980c9640 5 monclient: authenticate success, global_id 194448 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 1 -- 172.21.15.116:0/3232644518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8802f610 con 0x7fdb90137c30 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb980c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb980c9640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb980c9640 4 set_mon_vals no callback set 2024-09-18T12:43:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 1 -- 172.21.15.116:0/3232644518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8802f9f0 con 0x7fdb90137c30 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.024+0000 7fdb94e63640 10 monclient: dump: 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.035 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 10 monclient: shutdown 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 1 -- 172.21.15.116:0/3232644518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb90137c30 msgr2=0x7fdb9013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 1 --2- 172.21.15.116:0/3232644518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb90137c30 0x7fdb9013a010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009a10 tx=0x7fdb880091d0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 1 -- 172.21.15.116:0/3232644518 shutdown_connections 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 1 --2- 172.21.15.116:0/3232644518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb90137c30 0x7fdb9013a010 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 1 -- 172.21.15.116:0/3232644518 >> 172.21.15.116:0/3232644518 conn(0x7fdb90067110 msgr2=0x7fdb90067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 1 -- 172.21.15.116:0/3232644518 shutdown_connections 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.025+0000 7fdb980c9640 1 -- 172.21.15.116:0/3232644518 wait complete. 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient: build_initial_monmap 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient: monmap: 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 Processor -- start 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 -- start start 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient: init 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9009ea50 0x7fdb9009a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): picked mon.noname-b con 0x7fdb9009ea50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 0x7fdb9012deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): picked mon.noname-a con 0x7fdb901318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb90137c30 0x7fdb9012e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): picked mon.noname-c con 0x7fdb90137c30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90096b90 con 0x7fdb901318f0 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9013a240 con 0x7fdb9009ea50 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90097200 con 0x7fdb90137c30 2024-09-18T12:43:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb980c9640 10 monclient(hunting): authenticate will time out at 1860.347854s 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb95664640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 0x7fdb9012deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb95664640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 0x7fdb9012deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58760/0 (socket says 172.21.15.116:58760) 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb95664640 1 -- 172.21.15.116:0/2692384746 learned_addr learned my addr 172.21.15.116:0/2692384746 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb95664640 10 monclient(hunting): get_auth_request con 0x7fdb901318f0 auth_method 0 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb95664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb95664640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb95664640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb96666640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb90137c30 0x7fdb9012e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.026+0000 7fdb96666640 10 monclient(hunting): get_auth_request con 0x7fdb90137c30 auth_method 0 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb96666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb96666640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb96666640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95e65640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9009ea50 0x7fdb9009a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95e65640 10 monclient(hunting): get_auth_request con 0x7fdb9009ea50 auth_method 0 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95e65640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 10 monclient(hunting): handle_auth_done global_id 194451 payload 824 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb90137c30 msgr2=0x7fdb9012e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb90137c30 0x7fdb9012e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9009ea50 msgr2=0x7fdb9009a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9009ea50 0x7fdb9009a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 10 monclient: _finish_hunting 0 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 1 monclient: found mon.noname-a 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95664640 1 -- 172.21.15.116:0/2692384746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88009450 con 0x7fdb901318f0 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.027+0000 7fdb95e65640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9009ea50 0x7fdb9009a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb95664640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 msgr2=0x7fdb9012deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb95664640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 0x7fdb9012deb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400c590 tx=0x7fdb84007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84007510 con 0x7fdb901318f0 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: dump: 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: _finish_auth 0 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb840047a0 con 0x7fdb901318f0 2024-09-18T12:43:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb980c9640 5 monclient: authenticate success, global_id 194451 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb980c9640 10 monclient: _renew_subs 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb980c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9012fda0 con 0x7fdb901318f0 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84007690 con 0x7fdb901318f0 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8dffb640 10 monclient: dump: 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb980c9640 10 monclient: _renew_subs 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb980c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9009b610 con 0x7fdb901318f0 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.029+0000 7fdb8ffff640 4 set_mon_vals no callback set 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdb84013400 con 0x7fdb901318f0 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb8dffb640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb7c038490 0x7fdb7c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdb8404b5e0 con 0x7fdb901318f0 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb980c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb980c9640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 0x7fdb90129900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb980c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90065e00 con 0x7fdb9012e930 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb95e65640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb7c038490 0x7fdb7c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb95e65640 10 monclient: get_auth_request con 0x7fdb7c038490 auth_method 0 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb96666640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 0x7fdb90129900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb96666640 10 monclient: get_auth_request con 0x7fdb9012e930 auth_method 0 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb96666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb96666640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.030+0000 7fdb96666640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.031+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.031+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.031+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.031+0000 7fdb96666640 10 monclient(hunting): handle_auth_done global_id 174580 payload 824 2024-09-18T12:43:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.031+0000 7fdb96666640 1 -- 172.21.15.116:0/2692384746 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb9012ed00 con 0x7fdb9012e930 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.033+0000 7fdb95e65640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb7c038490 0x7fdb7c03a940 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fdb80002150 tx=0x7fdb80009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.033+0000 7fdb96666640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 msgr2=0x7fdb90129900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.033+0000 7fdb96666640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 0x7fdb90129900 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdb9006f600 tx=0x7fdb88032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.033+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88032c10 con 0x7fdb9012e930 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.033+0000 7fdb8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.033+0000 7fdb8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.033+0000 7fdb8dffb640 10 monclient: dump: 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.048 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.034+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdb88040070 con 0x7fdb9012e930 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.034+0000 7fdb8dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.034+0000 7fdb8dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.034+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 msgr2=0x7fdb90129900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.034+0000 7fdb8dffb640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 0x7fdb90129900 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdb9006f600 tx=0x7fdb88032000 comp rx=0 tx=0).stop 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.034+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 msgr2=0x7fdb90129900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.034+0000 7fdb8dffb640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb9012e930 0x7fdb90129900 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdb9006f600 tx=0x7fdb88032000 comp rx=0 tx=0).stop 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.039+0000 7fdb980c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:43:53.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.039+0000 7fdb980c9640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 0x7fdb9012a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.039+0000 7fdb980c9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.039+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9009bfc0 con 0x7fdb90129e40 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 0x7fdb9012a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient: get_auth_request con 0x7fdb90129e40 auth_method 0 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 10 monclient(hunting): handle_auth_done global_id 174583 payload 824 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.040+0000 7fdb96666640 1 -- 172.21.15.116:0/2692384746 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdb9004bf60 con 0x7fdb90129e40 2024-09-18T12:43:53.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb96666640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 msgr2=0x7fdb9012a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb96666640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 0x7fdb9012a240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009210 tx=0x7fdb8803e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88046c30 con 0x7fdb90129e40 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 10 monclient: dump: 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.051 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fdb8804a070 con 0x7fdb90129e40 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 msgr2=0x7fdb9012a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 0x7fdb9012a240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009210 tx=0x7fdb8803e070 comp rx=0 tx=0).stop 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 msgr2=0x7fdb9012a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:43:53.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.043+0000 7fdb8dffb640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb90129e40 0x7fdb9012a240 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 182, 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:53.053 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:53.054 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:43:53.055 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:43:53.056 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.057 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:53.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:53.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:43:53.060 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:53.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:53.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.063 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:43:53.064 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:43:53.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:43:53.065 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:43:53.065 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:43:53.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb7c038490 msgr2=0x7fdb7c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdb7c038490 0x7fdb7c03a940 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fdb80002150 tx=0x7fdb80009040 comp rx=0 tx=0).stop 2024-09-18T12:43:53.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 10 monclient: shutdown 2024-09-18T12:43:53.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 msgr2=0x7fdb9012deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb96666640 1 -- 172.21.15.116:0/2692384746 reap_dead start 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 0x7fdb9012deb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400c590 tx=0x7fdb84007dc0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 shutdown_connections 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 --2- 172.21.15.116:0/2692384746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb901318f0 0x7fdb9012deb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 >> 172.21.15.116:0/2692384746 conn(0x7fdb90067110 msgr2=0x7fdb9009d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 shutdown_connections 2024-09-18T12:43:53.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.044+0000 7fdb980c9640 1 -- 172.21.15.116:0/2692384746 wait complete. 2024-09-18T12:43:53.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:43:53.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T12:43:53.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:43:53.068 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T12:43:53.068 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:43:53.068 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:43:53.068 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:43:53.087 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:43:53.087 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:43:53.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:43:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.178+0000 7f25e371d640 10 monclient: get_monmap_and_config 2024-09-18T12:43:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.178+0000 7f25e371d640 10 monclient: build_initial_monmap 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.178+0000 7f25e371d640 10 monclient: monmap: 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:53.179445+0000 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:53.179445+0000 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:53.180 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.179+0000 7f25e371d640 1 Processor -- start 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.179+0000 7f25e371d640 1 -- start start 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.179+0000 7f25e371d640 10 monclient: init 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.179+0000 7f25e371d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.179+0000 7f25e371d640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.179+0000 7f25e371d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25dc0974f0 0x7f25dc12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.179+0000 7f25e371d640 10 monclient(hunting): picked mon.noname-a con 0x7f25dc0974f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25dc09a8a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25dc133070 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0978c0 con 0x7f25dc133070 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc09ca20 con 0x7f25dc09a8a0 2024-09-18T12:43:53.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc098ad0 con 0x7f25dc0974f0 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e371d640 10 monclient(hunting): authenticate will time out at 1860.501055s 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25dc0974f0 0x7f25dc12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25dc0974f0 0x7f25dc12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58786/0 (socket says 172.21.15.116:58786) 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 1 -- 172.21.15.116:0/3620539652 learned_addr learned my addr 172.21.15.116:0/3620539652 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): get_auth_request con 0x7f25dc0974f0 auth_method 0 2024-09-18T12:43:53.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e1cba640 1 --1- 172.21.15.116:0/3620539652 >> v1:172.21.15.116:6789/0 conn(0x7f25dc133070 0x7f25dc135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40696/0 (socket says 172.21.15.116:40696) 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e0cb8640 1 -- 172.21.15.116:0/3620539652 >> v1:172.21.15.144:6789/0 conn(0x7f25dc09a8a0 legacy=0x7f25dc09ace0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient(hunting): handle_auth_done global_id 194454 payload 274 2024-09-18T12:43:53.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 1 -- 172.21.15.116:0/3620539652 >> v1:172.21.15.144:6789/0 conn(0x7f25dc09a8a0 legacy=0x7f25dc09ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:53.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 1 -- 172.21.15.116:0/3620539652 >> v1:172.21.15.116:6789/0 conn(0x7f25dc133070 legacy=0x7f25dc135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:53.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient: _finish_hunting 0 2024-09-18T12:43:53.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 1 monclient: found mon.noname-a 2024-09-18T12:43:53.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:53.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.180+0000 7f25e14b9640 1 -- 172.21.15.116:0/3620539652 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc098db0 con 0x7f25dc0974f0 2024-09-18T12:43:53.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25e14b9640 1 -- 172.21.15.116:0/3620539652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc0974f0 msgr2=0x7f25dc12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25e14b9640 1 --2- 172.21.15.116:0/3620539652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc0974f0 0x7f25dc12ab00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009910 tx=0x7f25d80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41e2b99210b3ee2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 1 -- 172.21.15.116:0/3620539652 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8032440 con 0x7f25dc0974f0 2024-09-18T12:43:53.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: dump: 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: _finish_auth 0 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:53.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 1 -- 172.21.15.116:0/3620539652 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d80325a0 con 0x7f25dc0974f0 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25e371d640 5 monclient: authenticate success, global_id 194454 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25e371d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25e371d640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 1 -- 172.21.15.116:0/3620539652 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8032940 con 0x7f25dc0974f0 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25e371d640 4 set_mon_vals no callback set 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.183+0000 7f25d3fff640 10 monclient: dump: 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.193 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 10 monclient: shutdown 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 1 -- 172.21.15.116:0/3620539652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc0974f0 msgr2=0x7f25dc12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 1 --2- 172.21.15.116:0/3620539652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc0974f0 0x7f25dc12ab00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009910 tx=0x7f25d80091d0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 1 -- 172.21.15.116:0/3620539652 shutdown_connections 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 1 --2- 172.21.15.116:0/3620539652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc0974f0 0x7f25dc12ab00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 1 -- 172.21.15.116:0/3620539652 >> 172.21.15.116:0/3620539652 conn(0x7f25dc0672b0 msgr2=0x7f25dc0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 1 -- 172.21.15.116:0/3620539652 shutdown_connections 2024-09-18T12:43:53.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 1 -- 172.21.15.116:0/3620539652 wait complete. 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 10 monclient: build_initial_monmap 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.184+0000 7f25e371d640 10 monclient: monmap: 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 Processor -- start 2024-09-18T12:43:53.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 -- start start 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient: init 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25dc0974f0 0x7f25dc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): picked mon.noname-c con 0x7f25dc0974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25dc09a8a0 0x7f25dc06f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): picked mon.noname-b con 0x7f25dc09a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc133070 0x7f25dc13ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): picked mon.noname-a con 0x7f25dc133070 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc098300 con 0x7f25dc133070 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc129fa0 con 0x7f25dc09a8a0 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc06cf60 con 0x7f25dc0974f0 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e371d640 10 monclient(hunting): authenticate will time out at 1860.506683s 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25dc0974f0 0x7f25dc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc133070 0x7f25dc13ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25dc0974f0 0x7f25dc06ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:42284/0 (socket says 172.21.15.116:42284) 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 1 -- 172.21.15.116:0/5449270 learned_addr learned my addr 172.21.15.116:0/5449270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 10 monclient(hunting): get_auth_request con 0x7f25dc0974f0 auth_method 0 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 10 monclient(hunting): get_auth_request con 0x7f25dc133070 auth_method 0 2024-09-18T12:43:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e0cb8640 1 -- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25dc09a8a0 msgr2=0x7f25dc06f250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e0cb8640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25dc09a8a0 0x7f25dc06f250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e14b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.185+0000 7f25e1cba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 10 monclient(hunting): handle_auth_done global_id 194457 payload 824 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 1 -- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25dc0974f0 msgr2=0x7f25dc06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25dc0974f0 0x7f25dc06ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 1 -- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25dc09a8a0 msgr2=0x7f25dc06f250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25dc09a8a0 0x7f25dc06f250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 10 monclient: _finish_hunting 0 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 1 monclient: found mon.noname-a 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e1cba640 1 -- 172.21.15.116:0/5449270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d8009450 con 0x7f25dc133070 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.186+0000 7f25e14b9640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25dc0974f0 0x7f25dc06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e1cba640 1 -- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc133070 msgr2=0x7f25dc13ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e1cba640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc133070 0x7f25dc13ef80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c760 tx=0x7f25d4004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 1 -- 172.21.15.116:0/5449270 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d40070f0 con 0x7f25dc133070 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: dump: 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: _finish_auth 0 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 1 -- 172.21.15.116:0/5449270 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d4007270 con 0x7f25dc133070 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e371d640 5 monclient: authenticate success, global_id 194457 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e371d640 10 monclient: _renew_subs 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc06d240 con 0x7f25dc133070 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 1 -- 172.21.15.116:0/5449270 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4007630 con 0x7f25dc133070 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d37fe640 4 set_mon_vals no callback set 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25d17fa640 10 monclient: dump: 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e371d640 10 monclient: _renew_subs 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.188+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc129430 con 0x7f25dc133070 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25d17fa640 1 -- 172.21.15.116:0/5449270 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f25d401b400 con 0x7f25dc133070 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25e371d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25e371d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25e371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc04bf60 con 0x7f25dc133070 2024-09-18T12:43:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25d17fa640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038440 0x7f25c003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25d17fa640 1 -- 172.21.15.116:0/5449270 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f25d404bf40 con 0x7f25dc133070 2024-09-18T12:43:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25e14b9640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038440 0x7f25c003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.189+0000 7f25e14b9640 10 monclient: get_auth_request con 0x7f25c0038440 auth_method 0 2024-09-18T12:43:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.192+0000 7f25e14b9640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038440 0x7f25c003a8f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009420 tx=0x7f25d8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.193+0000 7f25d17fa640 1 -- 172.21.15.116:0/5449270 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f25d4010950 con 0x7f25dc133070 2024-09-18T12:43:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.193+0000 7f25d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:53.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.193+0000 7f25d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:53.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.406+0000 7f25e371d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:43:53.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.406+0000 7f25e371d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:53.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.406+0000 7f25e371d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:53.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.406+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f25dc0638d0 con 0x7f25dc133070 2024-09-18T12:43:53.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.406+0000 7f25d17fa640 1 -- 172.21.15.116:0/5449270 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f25d4010770 con 0x7f25dc133070 2024-09-18T12:43:53.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.406+0000 7f25d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:53.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.406+0000 7f25d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:53.408 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038440 msgr2=0x7f25c003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038440 0x7f25c003a8f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009420 tx=0x7f25d8032000 comp rx=0 tx=0).stop 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 10 monclient: shutdown 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc133070 msgr2=0x7f25dc13ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc133070 0x7f25dc13ef80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f25d400c760 tx=0x7f25d4004e20 comp rx=0 tx=0).stop 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 shutdown_connections 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f25c0038440 0x7f25c003a8f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc133070 0x7f25dc13ef80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25dc09a8a0 0x7f25dc06f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 --2- 172.21.15.116:0/5449270 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25dc0974f0 0x7f25dc06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 >> 172.21.15.116:0/5449270 conn(0x7f25dc0672b0 msgr2=0x7f25dc133440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 shutdown_connections 2024-09-18T12:43:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:53.408+0000 7f25e371d640 1 -- 172.21.15.116:0/5449270 wait complete. 2024-09-18T12:43:56.420 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:43:56.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.523+0000 7fafbe91d640 10 monclient: get_monmap_and_config 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.523+0000 7fafbe91d640 10 monclient: build_initial_monmap 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.523+0000 7fafbe91d640 10 monclient: monmap: 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:56.524261+0000 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:56.524261+0000 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:56.525 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:56.526 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:56.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.524+0000 7fafbe91d640 1 Processor -- start 2024-09-18T12:43:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.524+0000 7fafbe91d640 1 -- start start 2024-09-18T12:43:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.524+0000 7fafbe91d640 10 monclient: init 2024-09-18T12:43:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafb80974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fafb809a8a0 0x7fafb809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient(hunting): picked mon.noname-c con 0x7fafb809a8a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafb8133070 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb812b210 con 0x7fafb8133070 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb80978c0 con 0x7fafb80974f0 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb809b220 con 0x7fafb809a8a0 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbe91d640 10 monclient(hunting): authenticate will time out at 1863.846159s 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafaffff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fafb809a8a0 msgr2=0x7fafb809ace0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbceba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fafb8133070 0x7fafb8135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52160/0 (socket says 172.21.15.116:52160) 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafb7fff640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fafb80974f0 legacy=0x7fafb812ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:43:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafbceba640 1 -- 172.21.15.116:0/905789678 learned_addr learned my addr 172.21.15.116:0/905789678 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:43:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.525+0000 7fafaffff640 1 --2- 172.21.15.116:0/905789678 >> v2:172.21.15.144:3300/0 conn(0x7fafb809a8a0 0x7fafb809ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafbceba640 1 -- 172.21.15.116:0/905789678 >> v1:172.21.15.116:6789/0 conn(0x7fafb8133070 legacy=0x7fafb8135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820433528 0 0) 0x7fafb812b210 con 0x7fafb8133070 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient(hunting): my global_id is 194460 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa40034e0 con 0x7fafb8133070 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1371823823 0 0) 0x7fafa40034e0 con 0x7fafb8133070 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 >> v2:172.21.15.144:3300/0 conn(0x7fafb809a8a0 msgr2=0x7fafb809ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 --2- 172.21.15.116:0/905789678 >> v2:172.21.15.144:3300/0 conn(0x7fafb809a8a0 0x7fafb809ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 >> v1:172.21.15.144:6789/0 conn(0x7fafb80974f0 legacy=0x7fafb812ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb809ca20 con 0x7fafb8133070 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: _finish_auth 0 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fafb0001b20 con 0x7fafb8133070 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafbe91d640 5 monclient: authenticate success, global_id 194460 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafbe91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:43:56.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: dump: 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 >> v1:172.21.15.116:6789/0 conn(0x7fafb8133070 legacy=0x7fafb8135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa40039d0 0x7fafa4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 10 monclient(hunting): picked mon.b con 0x7fafa40039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.527+0000 7fafb77fe640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 0x7fafa4004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 10 monclient(hunting): picked mon.a con 0x7fafa4004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4005330 0x7fafa400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 10 monclient(hunting): picked mon.c con 0x7fafa4005330 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb809b220 con 0x7fafa4004780 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa400c2f0 con 0x7fafa40039d0 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa400c4b0 con 0x7fafa4005330 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb77fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbe91d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 0x7fafa4004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb7fff640 1 -- 172.21.15.116:0/905789678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa40039d0 msgr2=0x7fafa4003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): get_auth_request con 0x7fafa4004780 auth_method 0 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafb7fff640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa40039d0 0x7fafa4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4005330 0x7fafa400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 10 monclient(hunting): get_auth_request con 0x7fafa4005330 auth_method 0 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient(hunting): handle_auth_done global_id 194460 payload 293 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 1 -- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4005330 msgr2=0x7fafa400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4005330 0x7fafa400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 1 -- 172.21.15.116:0/905789678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa40039d0 msgr2=0x7fafa4003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafa40039d0 0x7fafa4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 1 monclient: found mon.a 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 1 -- 172.21.15.116:0/905789678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa40034e0 con 0x7fafa4004780 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 10 monclient: _finish_auth 0 2024-09-18T12:43:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4005330 0x7fafa400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.528+0000 7fafbceba640 1 -- 172.21.15.116:0/905789678 reap_dead start 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafaffff640 1 -- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 msgr2=0x7fafa4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafaffff640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 0x7fafa4004bf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fafa00099c0 tx=0x7fafa0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94c55803fd495f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0003b80 con 0x7fafa4004780 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 10 monclient: dump: 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafaffff640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 0x7fafa4004bf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fafa00099c0 tx=0x7fafa0009060 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fafa002ab80 mon_map magic: 0 v1 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa0003d20 con 0x7fafa4004780 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 4 set_mon_vals no callback set 2024-09-18T12:43:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 10 monclient: shutdown 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 1 -- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 msgr2=0x7fafa4004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 0x7fafa4004bf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fafa00099c0 tx=0x7fafa0009060 comp rx=0 tx=0).stop 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 1 -- 172.21.15.116:0/905789678 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7fafa002ab80 con 0x7fafa4004780 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafb77fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 1 -- 172.21.15.116:0/905789678 shutdown_connections 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 1 --2- 172.21.15.116:0/905789678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4004780 0x7fafa4004bf0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.531+0000 7fafbe91d640 1 -- 172.21.15.116:0/905789678 >> 172.21.15.116:0/905789678 conn(0x7fafb80672b0 msgr2=0x7fafb80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.532+0000 7fafbe91d640 1 -- 172.21.15.116:0/905789678 shutdown_connections 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.532+0000 7fafbe91d640 1 -- 172.21.15.116:0/905789678 wait complete. 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.532+0000 7fafbe91d640 10 monclient: build_initial_monmap 2024-09-18T12:43:56.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.532+0000 7fafbe91d640 10 monclient: monmap: 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.532+0000 7fafbe91d640 1 Processor -- start 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.532+0000 7fafbe91d640 1 -- start start 2024-09-18T12:43:56.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.532+0000 7fafbe91d640 10 monclient: init 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 0x7fafb81322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): picked mon.noname-c con 0x7fafa4004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb81327e0 0x7fafb813ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): picked mon.noname-a con 0x7fafb81327e0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb806e3d0 0x7fafb806be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): picked mon.noname-b con 0x7fafb806e3d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:56.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb806e030 con 0x7fafb81327e0 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb809e410 con 0x7fafb806e3d0 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb812bad0 con 0x7fafa4004780 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbe91d640 10 monclient(hunting): authenticate will time out at 1863.854400s 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 0x7fafb81322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 0x7fafb81322a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47328/0 (socket says 172.21.15.116:47328) 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 -- 172.21.15.116:0/770911925 learned_addr learned my addr 172.21.15.116:0/770911925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): get_auth_request con 0x7fafa4004780 auth_method 0 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:56.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb81327e0 0x7fafb813ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 10 monclient(hunting): get_auth_request con 0x7fafb81327e0 auth_method 0 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbceba640 1 -- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb806e3d0 msgr2=0x7fafb806be70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafbceba640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb806e3d0 0x7fafb806be70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:56.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient(hunting): handle_auth_done global_id 224150 payload 824 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 -- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb806e3d0 msgr2=0x7fafb806be70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb806e3d0 0x7fafb806be70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 -- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb81327e0 msgr2=0x7fafb813ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb81327e0 0x7fafb813ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 monclient: found mon.noname-c 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.533+0000 7fafb7fff640 1 -- 172.21.15.116:0/770911925 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8098300 con 0x7fafa4004780 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.535+0000 7fafb7fff640 1 -- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 msgr2=0x7fafb81322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:56.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.535+0000 7fafb7fff640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 0x7fafb81322a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fafa800b600 tx=0x7fafa80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 1 -- 172.21.15.116:0/770911925 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8003d90 con 0x7fafa4004780 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: dump: 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:56.553 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: _finish_auth 0 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 1 -- 172.21.15.116:0/770911925 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa8006150 con 0x7fafa4004780 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 1 -- 172.21.15.116:0/770911925 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8006510 con 0x7fafa4004780 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 10 monclient: dump: 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:56.554 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafbe91d640 5 monclient: authenticate success, global_id 224150 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafbe91d640 10 monclient: _renew_subs 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafbe91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb813f970 con 0x7fafa4004780 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb6ffd640 4 set_mon_vals no callback set 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafbe91d640 10 monclient: _renew_subs 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafbe91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb813f2c0 con 0x7fafa4004780 2024-09-18T12:43:56.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.536+0000 7fafb4ff9640 1 -- 172.21.15.116:0/770911925 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fafa80103e0 con 0x7fafa4004780 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafb4ff9640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faf98038440 0x7faf9803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafbe91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafbe91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafbe91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafb4ff9640 1 -- 172.21.15.116:0/770911925 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fafa804bb10 con 0x7fafa4004780 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb804bf60 con 0x7fafa4004780 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafaffff640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faf98038440 0x7faf9803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:56.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.537+0000 7fafaffff640 10 monclient: get_auth_request con 0x7faf98038440 auth_method 0 2024-09-18T12:43:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.540+0000 7fafaffff640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faf98038440 0x7faf9803a8f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fafa00038f0 tx=0x7fafa0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.541+0000 7fafb4ff9640 1 -- 172.21.15.116:0/770911925 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fafa8017030 con 0x7fafa4004780 2024-09-18T12:43:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.541+0000 7fafb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:56.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.541+0000 7fafb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:43:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.754+0000 7fafbe91d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:43:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.754+0000 7fafbe91d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:56.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.754+0000 7fafbe91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:43:56.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.754+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fafb80638d0 con 0x7fafa4004780 2024-09-18T12:43:56.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.754+0000 7fafb4ff9640 1 -- 172.21.15.116:0/770911925 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fafa8017030 con 0x7fafa4004780 2024-09-18T12:43:56.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.754+0000 7fafb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:43:56.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.754+0000 7fafb4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:43:56.756 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faf98038440 msgr2=0x7faf9803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faf98038440 0x7faf9803a8f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fafa00038f0 tx=0x7fafa0030000 comp rx=0 tx=0).stop 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 10 monclient: shutdown 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 msgr2=0x7fafb81322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 0x7fafb81322a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fafa800b600 tx=0x7fafa80091a0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 shutdown_connections 2024-09-18T12:43:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faf98038440 0x7faf9803a8f0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb806e3d0 0x7fafb806be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafb81327e0 0x7fafb813ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 --2- 172.21.15.116:0/770911925 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fafa4004780 0x7fafb81322a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 >> 172.21.15.116:0/770911925 conn(0x7fafb80672b0 msgr2=0x7fafb8129400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 shutdown_connections 2024-09-18T12:43:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:56.756+0000 7fafbe91d640 1 -- 172.21.15.116:0/770911925 wait complete. 2024-09-18T12:43:59.769 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:43:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.872+0000 7f32ada34640 10 monclient: get_monmap_and_config 2024-09-18T12:43:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.872+0000 7f32ada34640 10 monclient: build_initial_monmap 2024-09-18T12:43:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.872+0000 7f32ada34640 10 monclient: monmap: 2024-09-18T12:43:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:43:59.873170+0000 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:43:59.873170+0000 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:43:59.874 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 1 Processor -- start 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 1 -- start start 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient: init 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f32a809c860 0x7f32a809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient(hunting): picked mon.noname-a con 0x7f32a809c860 addr v2:172.21.15.116:3300/0 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32a809aba0 addr v1:172.21.15.116:6789/0 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809afa0 0x7f32a809df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient(hunting): picked mon.noname-b con 0x7f32a809afa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a806a320 con 0x7f32a809aba0 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a809a9d0 con 0x7f32a809c860 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8068a20 con 0x7f32a809afa0 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32ada34640 10 monclient(hunting): authenticate will time out at 1867.194845s 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32a77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809afa0 0x7f32a809df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.873+0000 7f32a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f32a809c860 0x7f32a809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f32a809c860 0x7f32a809cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51742/0 (socket says 172.21.15.116:51742) 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809afa0 0x7f32a809df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47344/0 (socket says 172.21.15.116:47344) 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 1 -- 172.21.15.116:0/899684302 learned_addr learned my addr 172.21.15.116:0/899684302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f329ffff640 1 --1- 172.21.15.116:0/899684302 >> v1:172.21.15.116:6789/0 conn(0x7f32a809aba0 0x7f32a809d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52168/0 (socket says 172.21.15.116:52168) 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): get_auth_request con 0x7f32a809c860 auth_method 0 2024-09-18T12:43:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 10 monclient(hunting): get_auth_request con 0x7f32a809afa0 auth_method 0 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient(hunting): handle_auth_done global_id 194466 payload 274 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 1 -- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809afa0 msgr2=0x7f32a809df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 1 --2- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809afa0 0x7f32a809df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 1 -- 172.21.15.116:0/899684302 >> v1:172.21.15.116:6789/0 conn(0x7f32a809aba0 legacy=0x7f32a809d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 1 monclient: found mon.noname-a 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a6ffd640 1 -- 172.21.15.116:0/899684302 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8068e60 con 0x7f32a809c860 2024-09-18T12:43:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.874+0000 7f32a77fe640 1 --2- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809afa0 0x7f32a809df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a6ffd640 1 -- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809c860 msgr2=0x7f32a809cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a6ffd640 1 --2- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809c860 0x7f32a809cc30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3298009910 tx=0x7f32980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184149b96b9996a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 1 -- 172.21.15.116:0/899684302 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298032440 con 0x7f32a809c860 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: dump: 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: _finish_auth 0 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 1 -- 172.21.15.116:0/899684302 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32980325a0 con 0x7f32a809c860 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 5 monclient: authenticate success, global_id 194466 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:59.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 10 monclient: get_monmap_and_config success 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 1 -- 172.21.15.116:0/899684302 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298032980 con 0x7f32a809c860 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 4 set_mon_vals no callback set 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32a67fc640 10 monclient: dump: 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:59.888 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 10 monclient: shutdown 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 -- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809c860 msgr2=0x7f32a809cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 --2- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809c860 0x7f32a809cc30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3298009910 tx=0x7f32980091d0 comp rx=0 tx=0).stop 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 -- 172.21.15.116:0/899684302 shutdown_connections 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 --2- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809afa0 0x7f32a809df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 --2- 172.21.15.116:0/899684302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809c860 0x7f32a809cc30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 -- 172.21.15.116:0/899684302 >> 172.21.15.116:0/899684302 conn(0x7f32a80a2e40 msgr2=0x7f32a80a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 -- 172.21.15.116:0/899684302 shutdown_connections 2024-09-18T12:43:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.877+0000 7f32ada34640 1 -- 172.21.15.116:0/899684302 wait complete. 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.878+0000 7f32ada34640 10 monclient: build_initial_monmap 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.878+0000 7f32ada34640 10 monclient: monmap: 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.878+0000 7f32ada34640 1 Processor -- start 2024-09-18T12:43:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.878+0000 7f32ada34640 1 -- start start 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient: init 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient: _reopen_session rank -1 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809aba0 0x7f32a8141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): picked mon.noname-c con 0x7f32a809aba0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 0x7f32a8133bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): picked mon.noname-a con 0x7f32a809afa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a809c860 0x7f32a811d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): picked mon.noname-b con 0x7f32a809c860 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8120fc0 con 0x7f32a809afa0 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8141f20 con 0x7f32a809c860 2024-09-18T12:43:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8120ce0 con 0x7f32a809aba0 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): _renew_subs 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32ada34640 10 monclient(hunting): authenticate will time out at 1867.200447s 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 0x7f32a8133bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 0x7f32a8133bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51744/0 (socket says 172.21.15.116:51744) 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809aba0 0x7f32a8141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 -- 172.21.15.116:0/2905312692 learned_addr learned my addr 172.21.15.116:0/2905312692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): get_auth_request con 0x7f32a809afa0 auth_method 0 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 10 monclient(hunting): get_auth_request con 0x7f32a809aba0 auth_method 0 2024-09-18T12:43:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a77fe640 1 -- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a809c860 msgr2=0x7f32a811d8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a77fe640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a809c860 0x7f32a811d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f32a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient(hunting): handle_auth_done global_id 194469 payload 824 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 -- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809aba0 msgr2=0x7f32a8141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:43:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809aba0 0x7f32a8141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 -- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a809c860 msgr2=0x7f32a811d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a809c860 0x7f32a811d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 monclient: found mon.noname-a 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.879+0000 7f329ffff640 1 -- 172.21.15.116:0/2905312692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0009a20 con 0x7f32a809afa0 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329ffff640 1 -- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 msgr2=0x7f32a8133bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329ffff640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 0x7f32a8133bc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f329000c5b0 tx=0x7f3290007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 1 -- 172.21.15.116:0/2905312692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290007510 con 0x7f32a809afa0 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:43:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: dump: 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: _finish_auth 0 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 1 -- 172.21.15.116:0/2905312692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32900057d0 con 0x7f32a809afa0 2024-09-18T12:43:59.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32ada34640 5 monclient: authenticate success, global_id 194469 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 1 -- 172.21.15.116:0/2905312692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290007690 con 0x7f32a809afa0 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32ada34640 10 monclient: _renew_subs 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32ada34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3298009450 con 0x7f32a809afa0 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f329effd640 10 monclient: dump: 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:43:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32a5ffb640 4 set_mon_vals no callback set 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32ada34640 10 monclient: _renew_subs 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32ada34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.882+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a811fe60 con 0x7f32a809afa0 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f329effd640 1 -- 172.21.15.116:0/2905312692 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f329001b400 con 0x7f32a809afa0 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f32ada34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f32ada34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f32ada34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a811f300 con 0x7f32a809afa0 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f329effd640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3288038440 0x7f328803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f329effd640 1 -- 172.21.15.116:0/2905312692 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f329004bd70 con 0x7f32a809afa0 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f32a6ffd640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3288038440 0x7f328803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.883+0000 7f32a6ffd640 10 monclient: get_auth_request con 0x7f3288038440 auth_method 0 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.886+0000 7f32a6ffd640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3288038440 0x7f328803a8f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3298034000 tx=0x7f3298032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.887+0000 7f329effd640 1 -- 172.21.15.116:0/2905312692 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f329001a540 con 0x7f32a809afa0 2024-09-18T12:43:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.887+0000 7f329effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:43:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:43:59.887+0000 7f329effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:00.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.101+0000 7f32ada34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:00.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.101+0000 7f32ada34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:00.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.101+0000 7f32ada34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:00.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.101+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f32a804bf60 con 0x7f32a809afa0 2024-09-18T12:44:00.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.102+0000 7f329effd640 1 -- 172.21.15.116:0/2905312692 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f329001a360 con 0x7f32a809afa0 2024-09-18T12:44:00.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.102+0000 7f329effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:00.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.102+0000 7f329effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:00.103 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":182,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3288038440 msgr2=0x7f328803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3288038440 0x7f328803a8f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3298034000 tx=0x7f3298032000 comp rx=0 tx=0).stop 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 10 monclient: shutdown 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 msgr2=0x7f32a8133bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 0x7f32a8133bc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f329000c5b0 tx=0x7f3290007f00 comp rx=0 tx=0).stop 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 shutdown_connections 2024-09-18T12:44:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3288038440 0x7f328803a8f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:00.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a809c860 0x7f32a811d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:00.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32a809afa0 0x7f32a8133bc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:00.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 --2- 172.21.15.116:0/2905312692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f32a809aba0 0x7f32a8141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:00.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 >> 172.21.15.116:0/2905312692 conn(0x7f32a80a2e40 msgr2=0x7f32a8134d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:00.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 shutdown_connections 2024-09-18T12:44:00.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:00.104+0000 7f32ada34640 1 -- 172.21.15.116:0/2905312692 wait complete. 2024-09-18T12:44:03.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.218+0000 7f4cbf91d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.218+0000 7f4cbf91d640 10 monclient: build_initial_monmap 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.218+0000 7f4cbf91d640 10 monclient: monmap: 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:03.219081+0000 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:03.219081+0000 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:03.219 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:03.220 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:03.220 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 1 Processor -- start 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 1 -- start start 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient: init 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cb809ad90 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb809b160 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:03.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cb8136c90 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb80692c0 con 0x7f4cb8136c90 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb809cce0 con 0x7f4cb809ad90 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb809db70 con 0x7f4cb809b160 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbf91d640 10 monclient(hunting): authenticate will time out at 1870.540858s 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbd6b9640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f4cb809ad90 legacy=0x7f4cb809c5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.219+0000 7f4cbceb8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbdeba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4cb8136c90 0x7f4cb8139070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52176/0 (socket says 172.21.15.116:52176) 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbdeba640 1 -- 172.21.15.116:0/640613406 learned_addr learned my addr 172.21.15.116:0/640613406 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4cb809b160 auth_method 0 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:03.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:44:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.220+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:03.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.222+0000 7f4cbdeba640 1 -- 172.21.15.116:0/640613406 >> v1:172.21.15.116:6789/0 conn(0x7f4cb8136c90 legacy=0x7f4cb8139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.420+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v1:172.21.15.144:6789/0 conn(0x7f4cb809ad90 legacy=0x7f4cb809c5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4cb809b160 auth_method 0 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.421+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:44:03.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.821+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v1:172.21.15.144:6789/0 conn(0x7f4cb809ad90 legacy=0x7f4cb809c5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4cb809b160 auth_method 0 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:03.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:03.822+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:44:04.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.622+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v1:172.21.15.144:6789/0 conn(0x7f4cb809ad90 legacy=0x7f4cb809c5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4cb809b160 auth_method 0 2024-09-18T12:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:04.623+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:44:06.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient(hunting): tick 2024-09-18T12:44:06.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 monclient(hunting): continuing hunt 2024-09-18T12:44:06.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:44:06.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4cb809b530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:06.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4cb809b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:06.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 >> v1:172.21.15.144:6789/0 conn(0x7f4cb809ad90 legacy=0x7f4cb809c5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 >> v1:172.21.15.116:6789/0 conn(0x7f4cb8136c90 legacy=0x7f4cb8139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient: _add_conns ranks=[4,2] 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 0x7f4c9c005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f4c9c003700 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f4c9c0061e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb809db70 con 0x7f4c9c003700 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c008d70 con 0x7f4c9c0061e0 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4caeffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 msgr2=0x7f4c9c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4c9c0061e0 auth_method 0 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 0x7f4c9c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.220+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 msgr2=0x7f4c9c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4c9c0061e0 auth_method 0 2024-09-18T12:44:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:06.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 0x7f4c9c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:44:06.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:44:06.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:06.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:06.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.421+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:44:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 msgr2=0x7f4c9c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 0x7f4c9c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:44:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4c9c0061e0 auth_method 0 2024-09-18T12:44:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:06.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:44:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:06.822+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:44:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 msgr2=0x7f4c9c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 0x7f4c9c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:44:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4c9c0061e0 auth_method 0 2024-09-18T12:44:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:07.623+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:44:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): tick 2024-09-18T12:44:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 monclient(hunting): continuing hunt 2024-09-18T12:44:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:44:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0061e0 0x7f4c9c008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 msgr2=0x7f4c9c005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.144:3300/0 conn(0x7f4c9c003700 0x7f4c9c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 1 -- 172.21.15.116:0/640613406 reap_dead start 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4c9c00b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f4c9c0091c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cb8136c90 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4c9c008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb809b160 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb809d5b0 con 0x7f4cb8136c90 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb809db70 con 0x7f4cb809b160 2024-09-18T12:44:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 1 -- 172.21.15.116:0/640613406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c008d70 con 0x7f4c9c0091c0 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4caeffd640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4c9c00b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4c9c008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): get_auth_request con 0x7f4c9c0091c0 auth_method 0 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 10 monclient(hunting): get_auth_request con 0x7f4cb809b160 auth_method 0 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbdeba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_done global_id 224159 payload 274 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 msgr2=0x7f4c9c008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4c9c008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> v1:172.21.15.116:6789/0 conn(0x7f4cb8136c90 legacy=0x7f4c9c00bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient: _finish_hunting 0 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 1 monclient: found mon.noname-b 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:44:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.220+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb809cce0 con 0x7f4c9c0091c0 2024-09-18T12:44:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.223+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 msgr2=0x7f4c9c00b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.223+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4c9c00b600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4006fd0 tx=0x7f4cb4009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b6a9bcdfa8bd152 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.223+0000 7f4caffff640 1 -- 172.21.15.116:0/640613406 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb402e070 con 0x7f4c9c0091c0 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.223+0000 7f4caffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 10 monclient: dump: 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.237 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 10 monclient: _finish_auth 0 2024-09-18T12:44:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 1 -- 172.21.15.116:0/640613406 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4007700 con 0x7f4c9c0091c0 2024-09-18T12:44:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 5 monclient: authenticate success, global_id 224159 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 4 set_mon_vals no callback set 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 1 -- 172.21.15.116:0/640613406 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4007b20 con 0x7f4c9c0091c0 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4caffff640 10 monclient: dump: 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 10 monclient: shutdown 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 1 -- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 msgr2=0x7f4c9c00b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4c9c00b600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4006fd0 tx=0x7f4cb4009720 comp rx=0 tx=0).stop 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 1 -- 172.21.15.116:0/640613406 shutdown_connections 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/640613406 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4c9c00b600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/640613406 >> v2:172.21.15.116:3300/0 conn(0x7f4cb809b160 0x7f4c9c008690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.224+0000 7f4cbf91d640 1 -- 172.21.15.116:0/640613406 >> 172.21.15.116:0/640613406 conn(0x7f4cb80678a0 msgr2=0x7f4cb80a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.225+0000 7f4cbf91d640 1 -- 172.21.15.116:0/640613406 shutdown_connections 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.225+0000 7f4cbf91d640 1 -- 172.21.15.116:0/640613406 wait complete. 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.225+0000 7f4cbf91d640 10 monclient: build_initial_monmap 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.225+0000 7f4cbf91d640 10 monclient: monmap: 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:09.240 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.225+0000 7f4cbf91d640 1 Processor -- start 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 1 -- start start 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient: init 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4cb8135950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): picked mon.noname-c con 0x7f4c9c0091c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb809b160 0x7f4cb8135e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb809b160 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb8136c90 0x7f4cb808d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb8136c90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:09.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb808f5b0 con 0x7f4cb809b160 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb808f0c0 con 0x7f4cb8136c90 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb806f020 con 0x7f4c9c0091c0 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbf91d640 10 monclient(hunting): authenticate will time out at 1876.547861s 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbd6b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4cb8135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.226+0000 7f4cbceb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb809b160 0x7f4cb8135e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbdeba640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb8136c90 msgr2=0x7f4cb808d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4cb8135950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53608/0 (socket says 172.21.15.116:53608) 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/1094047792 learned_addr learned my addr 172.21.15.116:0/1094047792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:09.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): get_auth_request con 0x7f4c9c0091c0 auth_method 0 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb809b160 0x7f4cb8135e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47008/0 (socket says 172.21.15.116:47008) 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbdeba640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb8136c90 0x7f4cb808d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request con 0x7f4cb809b160 auth_method 0 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient(hunting): handle_auth_done global_id 224162 payload 824 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb8136c90 msgr2=0x7f4cb808d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb8136c90 0x7f4cb808d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb809b160 msgr2=0x7f4cb8135e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb809b160 0x7f4cb8135e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient: _finish_hunting 0 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 monclient: found mon.noname-c 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/1094047792 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb4009a00 con 0x7f4c9c0091c0 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.227+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb809b160 0x7f4cb8135e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cbd6b9640 1 -- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 msgr2=0x7f4cb8135950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cbd6b9640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4cb8135950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4000c00 tx=0x7f4cb4036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 1 -- 172.21.15.116:0/1094047792 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4033ae0 con 0x7f4c9c0091c0 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: dump: 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: _finish_auth 0 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 1 -- 172.21.15.116:0/1094047792 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4033c60 con 0x7f4c9c0091c0 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cbf91d640 5 monclient: authenticate success, global_id 224162 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cbf91d640 10 monclient: _renew_subs 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 1 -- 172.21.15.116:0/1094047792 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb403f400 con 0x7f4c9c0091c0 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cbf91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb8098b40 con 0x7f4c9c0091c0 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4cad7fa640 10 monclient: dump: 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.229+0000 7f4caf7fe640 4 set_mon_vals no callback set 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.230+0000 7f4cbf91d640 10 monclient: _renew_subs 2024-09-18T12:44:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.230+0000 7f4cbf91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.230+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb808f780 con 0x7f4c9c0091c0 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.230+0000 7f4cad7fa640 1 -- 172.21.15.116:0/1094047792 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4cb403a070 con 0x7f4c9c0091c0 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cad7fa640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c9c039850 0x7f4c9c03bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cbf91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cad7fa640 1 -- 172.21.15.116:0/1094047792 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f4cb406d470 con 0x7f4c9c0091c0 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cbf91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cbf91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb804bf60 con 0x7f4c9c0091c0 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c9c039850 0x7f4c9c03bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.231+0000 7f4cbceb8640 10 monclient: get_auth_request con 0x7f4c9c039850 auth_method 0 2024-09-18T12:44:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.234+0000 7f4cbceb8640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c9c039850 0x7f4c9c03bd00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8094d10 tx=0x7f4ca8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.236+0000 7f4cad7fa640 1 -- 172.21.15.116:0/1094047792 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f4cb4048e30 con 0x7f4c9c0091c0 2024-09-18T12:44:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.236+0000 7f4cad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:09.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.236+0000 7f4cad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.454+0000 7f4cbf91d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.454+0000 7f4cbf91d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.454+0000 7f4cbf91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.454+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4cb804bf60 con 0x7f4c9c0091c0 2024-09-18T12:44:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.454+0000 7f4cad7fa640 1 -- 172.21.15.116:0/1094047792 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f4cb4048830 con 0x7f4c9c0091c0 2024-09-18T12:44:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.454+0000 7f4cad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.454+0000 7f4cad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:09.456 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":186,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c9c039850 msgr2=0x7f4c9c03bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c9c039850 0x7f4c9c03bd00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8094d10 tx=0x7f4ca8008040 comp rx=0 tx=0).stop 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 10 monclient: shutdown 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 msgr2=0x7f4cb8135950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4cb8135950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4000c00 tx=0x7f4cb4036000 comp rx=0 tx=0).stop 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 shutdown_connections 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4c9c039850 0x7f4c9c03bd00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cb8136c90 0x7f4cb808d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb809b160 0x7f4cb8135e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 --2- 172.21.15.116:0/1094047792 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c9c0091c0 0x7f4cb8135950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 >> 172.21.15.116:0/1094047792 conn(0x7f4cb80678a0 msgr2=0x7f4cb809f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 shutdown_connections 2024-09-18T12:44:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.456+0000 7f4cbf91d640 1 -- 172.21.15.116:0/1094047792 wait complete. 2024-09-18T12:44:09.467 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:44:09.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:44:09.468 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:44:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.561+0000 7f6befd5d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.561+0000 7f6befd5d640 10 monclient: build_initial_monmap 2024-09-18T12:44:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.561+0000 7f6befd5d640 10 monclient: monmap: 2024-09-18T12:44:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:09.562058+0000 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:09.562058+0000 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.561+0000 7f6befd5d640 1 Processor -- start 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 1 -- start start 2024-09-18T12:44:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient: init 2024-09-18T12:44:09.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6be8096c60 0x7f6be809a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6be8096c60 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6be8097030 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6be8133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be80696a0 con 0x7f6be8097030 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be809ac00 con 0x7f6be8096c60 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be809d6b0 con 0x7f6be8133b40 2024-09-18T12:44:09.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6befd5d640 10 monclient(hunting): authenticate will time out at 1876.883633s 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bed2f8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6be8097030 0x7f6be8097430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36854/0 (socket says 172.21.15.116:36854) 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bedaf9640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7f6be8096c60 msgr2=0x7f6be809a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bed2f8640 1 -- 172.21.15.116:0/2123833630 learned_addr learned my addr 172.21.15.116:0/2123833630 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bedaf9640 1 --2- 172.21.15.116:0/2123833630 >> v2:172.21.15.144:3300/0 conn(0x7f6be8096c60 0x7f6be809a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 10 monclient(hunting): get_auth_request con 0x7f6be8133b40 auth_method 0 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.562+0000 7f6bee2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 10 monclient(hunting): handle_auth_done global_id 224165 payload 274 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 1 -- 172.21.15.116:0/2123833630 >> v2:172.21.15.144:3300/0 conn(0x7f6be8096c60 msgr2=0x7f6be809a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 1 --2- 172.21.15.116:0/2123833630 >> v2:172.21.15.144:3300/0 conn(0x7f6be8096c60 0x7f6be809a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 1 -- 172.21.15.116:0/2123833630 >> v1:172.21.15.116:6789/0 conn(0x7f6be8097030 legacy=0x7f6be8097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 10 monclient: _finish_hunting 0 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 1 monclient: found mon.noname-b 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:44:09.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.563+0000 7f6bee2fa640 1 -- 172.21.15.116:0/2123833630 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be809d2e0 con 0x7f6be8133b40 2024-09-18T12:44:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.565+0000 7f6bee2fa640 1 -- 172.21.15.116:0/2123833630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 msgr2=0x7f6be8135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.565+0000 7f6bee2fa640 1 --2- 172.21.15.116:0/2123833630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8135f20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009e60 tx=0x7f6be0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92935f33c85382ed server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.565+0000 7f6becaf7640 1 -- 172.21.15.116:0/2123833630 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be0007840 con 0x7f6be8133b40 2024-09-18T12:44:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: dump: 2024-09-18T12:44:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: _finish_auth 0 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 1 -- 172.21.15.116:0/2123833630 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be002d050 con 0x7f6be8133b40 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 5 monclient: authenticate success, global_id 224165 2024-09-18T12:44:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 1 -- 172.21.15.116:0/2123833630 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be0007c10 con 0x7f6be8133b40 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 4 set_mon_vals no callback set 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6becaf7640 10 monclient: dump: 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 10 monclient: shutdown 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 -- 172.21.15.116:0/2123833630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 msgr2=0x7f6be8135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 --2- 172.21.15.116:0/2123833630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8135f20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009e60 tx=0x7f6be0009780 comp rx=0 tx=0).stop 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 -- 172.21.15.116:0/2123833630 shutdown_connections 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 --2- 172.21.15.116:0/2123833630 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8135f20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 --2- 172.21.15.116:0/2123833630 >> v2:172.21.15.144:3300/0 conn(0x7f6be8096c60 0x7f6be809a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 -- 172.21.15.116:0/2123833630 >> 172.21.15.116:0/2123833630 conn(0x7f6be80672b0 msgr2=0x7f6be80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 -- 172.21.15.116:0/2123833630 shutdown_connections 2024-09-18T12:44:09.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.566+0000 7f6befd5d640 1 -- 172.21.15.116:0/2123833630 wait complete. 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient: build_initial_monmap 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient: monmap: 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 Processor -- start 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 -- start start 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient: init 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be8096c60 0x7f6be813f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f6be8096c60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 0x7f6be812e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f6be8097030 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8118360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f6be8133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8119e50 con 0x7f6be8097030 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be809add0 con 0x7f6be8096c60 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be81335c0 con 0x7f6be8133b40 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.567+0000 7f6befd5d640 10 monclient(hunting): authenticate will time out at 1876.888932s 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 0x7f6be812e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8118360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 0x7f6be812e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47014/0 (socket says 172.21.15.116:47014) 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bedaf9640 1 -- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be8096c60 msgr2=0x7f6be813f030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 -- 172.21.15.116:0/1516815969 learned_addr learned my addr 172.21.15.116:0/1516815969 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): get_auth_request con 0x7f6be8097030 auth_method 0 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bedaf9640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be8096c60 0x7f6be813f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 10 monclient(hunting): get_auth_request con 0x7f6be8133b40 auth_method 0 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient(hunting): handle_auth_done global_id 194478 payload 824 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 msgr2=0x7f6be8118360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8118360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be8096c60 msgr2=0x7f6be813f030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6be8096c60 0x7f6be813f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient: _finish_hunting 0 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 monclient: found mon.noname-a 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bed2f8640 1 -- 172.21.15.116:0/1516815969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be0009a20 con 0x7f6be8097030 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.568+0000 7f6bee2fa640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6be8133b40 0x7f6be8118360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bed2f8640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 msgr2=0x7f6be812e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bed2f8640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 0x7f6be812e7d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c610 tx=0x7f6bd8004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8005cb0 con 0x7f6be8097030 2024-09-18T12:44:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: dump: 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8005e30 con 0x7f6be8097030 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6befd5d640 5 monclient: authenticate success, global_id 194478 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6befd5d640 10 monclient: _renew_subs 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6befd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be811b8f0 con 0x7f6be8097030 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8007340 con 0x7f6be8097030 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bddffb640 10 monclient: dump: 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6befd5d640 10 monclient: _renew_subs 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6bdffff640 4 set_mon_vals no callback set 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6befd5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.570+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be811a990 con 0x7f6be8097030 2024-09-18T12:44:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6befd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6bd801b400 con 0x7f6be8097030 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6befd5d640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 0x7f6be80298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6befd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8065f20 con 0x7f6be813f570 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 0x7f6be80298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 10 monclient: get_auth_request con 0x7f6be813f570 auth_method 0 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bddffb640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6bcc038440 0x7f6bcc03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.571+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.572+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6bd804be40 con 0x7f6be8097030 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.572+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_done global_id 194481 payload 824 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.572+0000 7f6bedaf9640 1 -- 172.21.15.116:0/1516815969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6be811c080 con 0x7f6be813f570 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.572+0000 7f6bee2fa640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6bcc038440 0x7f6bcc03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.572+0000 7f6bee2fa640 10 monclient: get_auth_request con 0x7f6bcc038440 auth_method 0 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bedaf9640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 msgr2=0x7f6be80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bedaf9640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 0x7f6be80298c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b480 tx=0x7f6be40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be40050f0 con 0x7f6be813f570 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 10 monclient: dump: 2024-09-18T12:44:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6be4005250 con 0x7f6be813f570 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 msgr2=0x7f6be80298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 0x7f6be80298c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b480 tx=0x7f6be40091a0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 msgr2=0x7f6be80298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bddffb640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be813f570 0x7f6be80298c0 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b480 tx=0x7f6be40091a0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.574+0000 7f6bee2fa640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6bcc038440 0x7f6bcc03a8f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6be0034000 tx=0x7f6be0037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6befd5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6befd5d640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 0x7f6be8029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6befd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be811abb0 con 0x7f6be812ed10 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 0x7f6be8029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient: get_auth_request con 0x7f6be812ed10 auth_method 0 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 10 monclient(hunting): handle_auth_done global_id 194484 payload 824 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.580+0000 7f6bedaf9640 1 -- 172.21.15.116:0/1516815969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6be804bf60 con 0x7f6be812ed10 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bedaf9640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 msgr2=0x7f6be8029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bedaf9640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 0x7f6be8029e00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b480 tx=0x7f6be4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be401b650 con 0x7f6be812ed10 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 10 monclient: dump: 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f6be401b7b0 con 0x7f6be812ed10 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 msgr2=0x7f6be8029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 0x7f6be8029e00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b480 tx=0x7f6be4015000 comp rx=0 tx=0).stop 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 msgr2=0x7f6be8029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bddffb640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be812ed10 0x7f6be8029e00 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b480 tx=0x7f6be4015000 comp rx=0 tx=0).stop 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 186, 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.600 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:09.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:09.602 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:09.604 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.605 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.606 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:09.607 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6bcc038440 msgr2=0x7f6bcc03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6bcc038440 0x7f6bcc03a8f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6be0034000 tx=0x7f6be0037000 comp rx=0 tx=0).stop 2024-09-18T12:44:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 10 monclient: shutdown 2024-09-18T12:44:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 msgr2=0x7f6be812e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 0x7f6be812e7d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c610 tx=0x7f6bd8004960 comp rx=0 tx=0).stop 2024-09-18T12:44:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6bee2fa640 1 -- 172.21.15.116:0/1516815969 reap_dead start 2024-09-18T12:44:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 shutdown_connections 2024-09-18T12:44:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 --2- 172.21.15.116:0/1516815969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6be8097030 0x7f6be812e7d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 >> 172.21.15.116:0/1516815969 conn(0x7f6be80672b0 msgr2=0x7f6be809c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 shutdown_connections 2024-09-18T12:44:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.583+0000 7f6befd5d640 1 -- 172.21.15.116:0/1516815969 wait complete. 2024-09-18T12:44:09.609 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:44:09.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T12:44:09.609 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:44:09.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.701+0000 7f1f0fd54640 10 monclient: get_monmap_and_config 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.701+0000 7f1f0fd54640 10 monclient: build_initial_monmap 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.702+0000 7f1f0fd54640 10 monclient: monmap: 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:09.702668+0000 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:09.702668+0000 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:09.703 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.702+0000 7f1f0fd54640 1 Processor -- start 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.702+0000 7f1f0fd54640 1 -- start start 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.702+0000 7f1f0fd54640 10 monclient: init 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f080974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f0809a8a0 0x7f1f0809ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient(hunting): picked mon.noname-a con 0x7f1f0809a8a0 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:09.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f08133070 0x7f1f08135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient(hunting): picked mon.noname-b con 0x7f1f08133070 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0812b210 con 0x7f1f080974f0 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f080978c0 con 0x7f1f0809a8a0 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0809b220 con 0x7f1f08133070 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0fd54640 10 monclient(hunting): authenticate will time out at 1877.024280s 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f0809a8a0 0x7f1f0809ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f08133070 0x7f1f08135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f0809a8a0 0x7f1f0809ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47020/0 (socket says 172.21.15.116:47020) 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0daf0640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7f1f080974f0 legacy=0x7f1f0812ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-09-18T12:44:09.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/2767367134 learned_addr learned my addr 172.21.15.116:0/2767367134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): get_auth_request con 0x7f1f0809a8a0 auth_method 0 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 10 monclient(hunting): get_auth_request con 0x7f1f08133070 auth_method 0 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_done global_id 194487 payload 274 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f08133070 msgr2=0x7f1f08135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 --2- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f08133070 0x7f1f08135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/2767367134 >> v1:172.21.15.144:6789/0 conn(0x7f1f080974f0 legacy=0x7f1f0812ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient: _finish_hunting 0 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 monclient: found mon.noname-a 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/2767367134 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0809ca20 con 0x7f1f0809a8a0 2024-09-18T12:44:09.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.703+0000 7f1f0e2f1640 1 --2- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f08133070 0x7f1f08135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:09.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 msgr2=0x7f1f0809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0d2ef640 1 --2- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 0x7f1f0809ace0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009910 tx=0x7f1efc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9703b7dc0e2b200a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 1 -- 172.21.15.116:0/2767367134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc02ae90 con 0x7f1f0809a8a0 2024-09-18T12:44:09.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: dump: 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: _finish_auth 0 2024-09-18T12:44:09.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 1 -- 172.21.15.116:0/2767367134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc032440 con 0x7f1f0809a8a0 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0fd54640 5 monclient: authenticate success, global_id 194487 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0fd54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0fd54640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 1 -- 172.21.15.116:0/2767367134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc0327e0 con 0x7f1f0809a8a0 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0fd54640 4 set_mon_vals no callback set 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.706+0000 7f1f0caee640 10 monclient: dump: 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.718 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 10 monclient: shutdown 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 -- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 msgr2=0x7f1f0809ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 --2- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 0x7f1f0809ace0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009910 tx=0x7f1efc0091d0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 -- 172.21.15.116:0/2767367134 shutdown_connections 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 --2- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f08133070 0x7f1f08135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 --2- 172.21.15.116:0/2767367134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 0x7f1f0809ace0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 -- 172.21.15.116:0/2767367134 >> 172.21.15.116:0/2767367134 conn(0x7f1f080672b0 msgr2=0x7f1f080676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 -- 172.21.15.116:0/2767367134 shutdown_connections 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 1 -- 172.21.15.116:0/2767367134 wait complete. 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 10 monclient: build_initial_monmap 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.707+0000 7f1f0fd54640 10 monclient: monmap: 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:09.720 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 Processor -- start 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 -- start start 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient: init 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f080974f0 0x7f1f0806efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): picked mon.noname-c con 0x7f1f080974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 0x7f1f0806b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): picked mon.noname-a con 0x7f1f0809a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f08133070 0x7f1f0806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): picked mon.noname-b con 0x7f1f08133070 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0806c480 con 0x7f1f0809a8a0 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0806dfb0 con 0x7f1f08133070 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0806e290 con 0x7f1f080974f0 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0fd54640 10 monclient(hunting): authenticate will time out at 1877.029532s 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f080974f0 0x7f1f0806efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 0x7f1f0806b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f080974f0 0x7f1f0806efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:53634/0 (socket says 172.21.15.116:53634) 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 1 -- 172.21.15.116:0/1826975646 learned_addr learned my addr 172.21.15.116:0/1826975646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 10 monclient(hunting): get_auth_request con 0x7f1f080974f0 auth_method 0 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0e2f1640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f08133070 msgr2=0x7f1f0806b760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.144:3300/0 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): get_auth_request con 0x7f1f0809a8a0 auth_method 0 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0e2f1640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f08133070 0x7f1f0806b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient(hunting): handle_auth_done global_id 194490 payload 824 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f080974f0 msgr2=0x7f1f0806efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f080974f0 0x7f1f0806efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f08133070 msgr2=0x7f1f0806b760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f08133070 0x7f1f0806b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient: _finish_hunting 0 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 1 monclient: found mon.noname-a 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/1826975646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00009390 con 0x7f1f0809a8a0 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.708+0000 7f1f0daf0640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f080974f0 0x7f1f0806efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0d2ef640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 msgr2=0x7f1f0806b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0d2ef640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 0x7f1f0806b220 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1efc031000 tx=0x7f1efc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc031c70 con 0x7f1f0809a8a0 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: dump: 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.725 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc02ae90 con 0x7f1f0809a8a0 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0fd54640 5 monclient: authenticate success, global_id 194490 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0fd54640 10 monclient: _renew_subs 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0fd54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1efc009450 con 0x7f1f0809a8a0 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc03d420 con 0x7f1f0809a8a0 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f05ffb640 10 monclient: dump: 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0fd54640 10 monclient: _renew_subs 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f07fff640 4 set_mon_vals no callback set 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0fd54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.711+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0806f4e0 con 0x7f1f0809a8a0 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1efc039040 con 0x7f1f0809a8a0 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f05ffb640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ef4038440 0x7f1ef403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0fd54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1efc06b430 con 0x7f1f0809a8a0 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0fd54640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 0x7f1f080298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0fd54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f08065f20 con 0x7f1f0813f540 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 0x7f1f080298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient: get_auth_request con 0x7f1f0813f540 auth_method 0 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_done global_id 224177 payload 824 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0daf0640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ef4038440 0x7f1ef403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0e2f1640 1 -- 172.21.15.116:0/1826975646 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f0804bf60 con 0x7f1f0813f540 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.712+0000 7f1f0daf0640 10 monclient: get_auth_request con 0x7f1ef4038440 auth_method 0 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.714+0000 7f1f0e2f1640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 msgr2=0x7f1f080298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.714+0000 7f1f0e2f1640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 0x7f1f080298c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000e850 tx=0x7f1f000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0000ccd0 con 0x7f1f0813f540 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 10 monclient: dump: 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1f0000ce30 con 0x7f1f0813f540 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f0daf0640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ef4038440 0x7f1ef403a8f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80066f0 tx=0x7f1ef80097f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 msgr2=0x7f1f080298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 0x7f1f080298c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000e850 tx=0x7f1f000090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 msgr2=0x7f1f080298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.715+0000 7f1f05ffb640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f540 0x7f1f080298c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.720+0000 7f1f0fd54640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.720+0000 7f1f0fd54640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 0x7f1f08029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.720+0000 7f1f0fd54640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.720+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f08134b10 con 0x7f1f0813f910 2024-09-18T12:44:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 0x7f1f08029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient: get_auth_request con 0x7f1f0813f910 auth_method 0 2024-09-18T12:44:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 10 monclient(hunting): handle_auth_done global_id 224180 payload 824 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.721+0000 7f1f0e2f1640 1 -- 172.21.15.116:0/1826975646 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1f080638b0 con 0x7f1f0813f910 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f0e2f1640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 msgr2=0x7f1f08029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f0e2f1640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 0x7f1f08029e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000e850 tx=0x7f1f00019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00005970 con 0x7f1f0813f910 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 10 monclient: dump: 2024-09-18T12:44:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f1f0000cac0 con 0x7f1f0813f910 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:09.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 msgr2=0x7f1f08029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 0x7f1f08029e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000e850 tx=0x7f1f00019000 comp rx=0 tx=0).stop 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 msgr2=0x7f1f08029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.723+0000 7f1f05ffb640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f0813f910 0x7f1f08029e00 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000e850 tx=0x7f1f00019000 comp rx=0 tx=0).stop 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:44:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 186, 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 1, 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:09.739 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:09.740 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:09.742 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:09.743 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.744 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.745 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:44:09.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:09.747 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:09.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:09.748 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:09.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:09.748 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:09.748 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ef4038440 msgr2=0x7f1ef403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1ef4038440 0x7f1ef403a8f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80066f0 tx=0x7f1ef80097f0 comp rx=0 tx=0).stop 2024-09-18T12:44:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 10 monclient: shutdown 2024-09-18T12:44:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 msgr2=0x7f1f0806b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 --2- 172.21.15.116:0/1826975646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f0809a8a0 0x7f1f0806b220 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1efc031000 tx=0x7f1efc032000 comp rx=0 tx=0).stop 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0e2f1640 1 -- 172.21.15.116:0/1826975646 reap_dead start 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0e2f1640 1 -- 172.21.15.116:0/1826975646 reap_dead start 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 shutdown_connections 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 >> 172.21.15.116:0/1826975646 conn(0x7f1f080672b0 msgr2=0x7f1f081297f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 shutdown_connections 2024-09-18T12:44:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:09.724+0000 7f1f0fd54640 1 -- 172.21.15.116:0/1826975646 wait complete. 2024-09-18T12:44:09.750 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:44:09.750 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:44:29.752 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T12:44:29.752 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T12:44:29.752 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T12:44:29.752 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T12:44:29.755 INFO:tasks.ceph.mon.b:Started 2024-09-18T12:44:29.756 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:44:29.756 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.856+0000 7f80d98af640 10 monclient: get_monmap_and_config 2024-09-18T12:44:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.856+0000 7f80d98af640 10 monclient: build_initial_monmap 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.856+0000 7f80d98af640 10 monclient: monmap: 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:29.858530+0000 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:29.858530+0000 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:29.859 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.857+0000 7f80d98af640 1 Processor -- start 2024-09-18T12:44:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.857+0000 7f80d98af640 1 -- start start 2024-09-18T12:44:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.857+0000 7f80d98af640 10 monclient: init 2024-09-18T12:44:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.857+0000 7f80d98af640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T12:44:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80d4131980 0x7f80d4131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient(hunting): picked mon.noname-a con 0x7f80d4131980 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f80d409fda0 0x7f80d4132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient(hunting): picked mon.noname-c con 0x7f80d409fda0 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80d409e9c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d409f510 con 0x7f80d409e9c0 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d40a1180 con 0x7f80d4131980 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d40a1460 con 0x7f80d409fda0 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d98af640 10 monclient(hunting): authenticate will time out at 1897.180111s 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80d4131980 0x7f80d4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f80d409e9c0 0x7f80d409ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42606/0 (socket says 172.21.15.116:42606) 2024-09-18T12:44:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f80d4131980 0x7f80d4131d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56030/0 (socket says 172.21.15.116:56030) 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d37fe640 1 -- 172.21.15.116:0/1614796832 learned_addr learned my addr 172.21.15.116:0/1614796832 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): get_auth_request con 0x7f80d4131980 auth_method 0 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d27fc640 1 --2- 172.21.15.116:0/1614796832 >> v2:172.21.15.144:3300/0 conn(0x7f80d409fda0 0x7f80d4132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d27fc640 10 monclient(hunting): get_auth_request con 0x7f80d409fda0 auth_method 0 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:29.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient(hunting): handle_auth_done global_id 194493 payload 274 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 1 -- 172.21.15.116:0/1614796832 >> v2:172.21.15.144:3300/0 conn(0x7f80d409fda0 msgr2=0x7f80d4132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 1 --2- 172.21.15.116:0/1614796832 >> v2:172.21.15.144:3300/0 conn(0x7f80d409fda0 0x7f80d4132290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 1 -- 172.21.15.116:0/1614796832 >> v1:172.21.15.116:6789/0 conn(0x7f80d409e9c0 legacy=0x7f80d409ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 1 monclient: found mon.noname-a 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.858+0000 7f80d2ffd640 1 -- 172.21.15.116:0/1614796832 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d4139cd0 con 0x7f80d4131980 2024-09-18T12:44:29.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d2ffd640 1 -- 172.21.15.116:0/1614796832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d4131980 msgr2=0x7f80d4131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d2ffd640 1 --2- 172.21.15.116:0/1614796832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d4131980 0x7f80d4131d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80d4064050 tx=0x7f80c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ba308a26aa62ca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 1 -- 172.21.15.116:0/1614796832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c402adf0 con 0x7f80d4131980 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: dump: 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 5 monclient: authenticate success, global_id 194493 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 1 -- 172.21.15.116:0/1614796832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c4032440 con 0x7f80d4131980 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 4 set_mon_vals no callback set 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 1 -- 172.21.15.116:0/1614796832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4032820 con 0x7f80d4131980 2024-09-18T12:44:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d1ffb640 10 monclient: dump: 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 10 monclient: shutdown 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 1 -- 172.21.15.116:0/1614796832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d4131980 msgr2=0x7f80d4131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.861+0000 7f80d98af640 1 --2- 172.21.15.116:0/1614796832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d4131980 0x7f80d4131d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80d4064050 tx=0x7f80c40090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 1 -- 172.21.15.116:0/1614796832 shutdown_connections 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 1 --2- 172.21.15.116:0/1614796832 >> v2:172.21.15.144:3300/0 conn(0x7f80d409fda0 0x7f80d4132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 1 --2- 172.21.15.116:0/1614796832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d4131980 0x7f80d4131d50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 1 -- 172.21.15.116:0/1614796832 >> 172.21.15.116:0/1614796832 conn(0x7f80d4067bd0 msgr2=0x7f80d40a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 1 -- 172.21.15.116:0/1614796832 shutdown_connections 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 1 -- 172.21.15.116:0/1614796832 wait complete. 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 10 monclient: build_initial_monmap 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.862+0000 7f80d98af640 10 monclient: monmap: 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 Processor -- start 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 -- start start 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient: init 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 0x7f80d41381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): picked mon.noname-b con 0x7f80d409e9c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 0x7f80d4140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): picked mon.noname-a con 0x7f80d409fda0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d4131980 0x7f80d4141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): picked mon.noname-c con 0x7f80d4131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d409bf80 con 0x7f80d409fda0 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d41374c0 con 0x7f80d409e9c0 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d409e450 con 0x7f80d4131980 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d98af640 10 monclient(hunting): authenticate will time out at 1897.185576s 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 0x7f80d4140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d4131980 0x7f80d4141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 0x7f80d4140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56034/0 (socket says 172.21.15.116:56034) 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 1 -- 172.21.15.116:0/453592356 learned_addr learned my addr 172.21.15.116:0/453592356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d4131980 0x7f80d4141480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58180/0 (socket says 172.21.15.116:58180) 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): get_auth_request con 0x7f80d409fda0 auth_method 0 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 10 monclient(hunting): get_auth_request con 0x7f80d4131980 auth_method 0 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 0x7f80d41381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:29.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 10 monclient(hunting): get_auth_request con 0x7f80d409e9c0 auth_method 0 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 1 -- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 msgr2=0x7f80d41381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 1 -- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 msgr2=0x7f80d41381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 0x7f80d41381d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d2ffd640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 0x7f80d41381d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 10 monclient(hunting): handle_auth_done global_id 194496 payload 824 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 1 -- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d4131980 msgr2=0x7f80d4141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d4131980 0x7f80d4141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 1 -- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 msgr2=0x7f80d41381d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.863+0000 7f80d27fc640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 0x7f80d41381d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.864+0000 7f80d27fc640 10 monclient: _finish_hunting 0 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.864+0000 7f80d27fc640 1 monclient: found mon.noname-a 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.864+0000 7f80d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.864+0000 7f80d27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.864+0000 7f80d27fc640 1 -- 172.21.15.116:0/453592356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c4009390 con 0x7f80d409fda0 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.864+0000 7f80d37fe640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d4131980 0x7f80d4141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d27fc640 1 -- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 msgr2=0x7f80d4140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d27fc640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 0x7f80d4140f40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80c800d890 tx=0x7f80c8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 1 -- 172.21.15.116:0/453592356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c800de20 con 0x7f80d409fda0 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: dump: 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: _finish_auth 0 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d98af640 5 monclient: authenticate success, global_id 194496 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 1 -- 172.21.15.116:0/453592356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c800bae0 con 0x7f80d409fda0 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d98af640 10 monclient: _renew_subs 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 1 -- 172.21.15.116:0/453592356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c800be80 con 0x7f80d409fda0 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d98af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d41371e0 con 0x7f80d409fda0 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80baffd640 10 monclient: dump: 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:29.882 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d17fa640 4 set_mon_vals no callback set 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d98af640 10 monclient: _renew_subs 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d98af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.866+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d4138710 con 0x7f80d409fda0 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80baffd640 1 -- 172.21.15.116:0/453592356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f80c8014640 con 0x7f80d409fda0 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80d98af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80d98af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80d98af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d404bf60 con 0x7f80d409fda0 2024-09-18T12:44:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80baffd640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c00384c0 0x7f80c003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80baffd640 1 -- 172.21.15.116:0/453592356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f80c804c850 con 0x7f80d409fda0 2024-09-18T12:44:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80d2ffd640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c00384c0 0x7f80c003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.867+0000 7f80d2ffd640 10 monclient: get_auth_request con 0x7f80c00384c0 auth_method 0 2024-09-18T12:44:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.870+0000 7f80d2ffd640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c00384c0 0x7f80c003a970 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f80c4034000 tx=0x7f80c4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.871+0000 7f80baffd640 1 -- 172.21.15.116:0/453592356 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f80c801dde0 con 0x7f80d409fda0 2024-09-18T12:44:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.871+0000 7f80baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:29.871+0000 7f80baffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:30.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.091+0000 7f80d98af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:30.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.091+0000 7f80d98af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:30.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.091+0000 7f80d98af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:30.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.091+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f80d40638d0 con 0x7f80d409fda0 2024-09-18T12:44:30.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.091+0000 7f80baffd640 1 -- 172.21.15.116:0/453592356 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f80c8012cb0 con 0x7f80d409fda0 2024-09-18T12:44:30.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.091+0000 7f80baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:30.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.091+0000 7f80baffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:30.094 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":186,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:30.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c00384c0 msgr2=0x7f80c003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c00384c0 0x7f80c003a970 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f80c4034000 tx=0x7f80c4032000 comp rx=0 tx=0).stop 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 10 monclient: shutdown 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 msgr2=0x7f80d4140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 0x7f80d4140f40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80c800d890 tx=0x7f80c8008040 comp rx=0 tx=0).stop 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 shutdown_connections 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f80c00384c0 0x7f80c003a970 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f80d4131980 0x7f80d4141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d409fda0 0x7f80d4140f40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 --2- 172.21.15.116:0/453592356 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80d409e9c0 0x7f80d41381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:30.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 >> 172.21.15.116:0/453592356 conn(0x7f80d4067bd0 msgr2=0x7f80d40a3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:30.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.093+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 shutdown_connections 2024-09-18T12:44:30.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:30.094+0000 7f80d98af640 1 -- 172.21.15.116:0/453592356 wait complete. 2024-09-18T12:44:33.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.210+0000 7fdd20f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.210+0000 7fdd20f9d640 10 monclient: build_initial_monmap 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.210+0000 7fdd20f9d640 10 monclient: monmap: 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:33.211987+0000 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:33.211987+0000 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:33.213 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.210+0000 7fdd20f9d640 1 Processor -- start 2024-09-18T12:44:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.210+0000 7fdd20f9d640 1 -- start start 2024-09-18T12:44:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.210+0000 7fdd20f9d640 10 monclient: init 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 0x7fdd1c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c0974f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd1c09a8a0 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd1c133070 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c0978c0 con 0x7fdd1c133070 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c09ca20 con 0x7fdd1c09a8a0 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1c098ad0 con 0x7fdd1c0974f0 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd20f9d640 10 monclient(hunting): authenticate will time out at 1900.533613s 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd1ad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdd1c133070 0x7fdd1c135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42616/0 (socket says 172.21.15.116:42616) 2024-09-18T12:44:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd1ad9d640 1 -- 172.21.15.116:0/3667626599 learned_addr learned my addr 172.21.15.116:0/3667626599 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/3667626599 >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 0x7fdd1c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.211+0000 7fdd1a59c640 10 monclient(hunting): get_auth_request con 0x7fdd1c0974f0 auth_method 0 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.212+0000 7fdd1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.212+0000 7fdd1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.212+0000 7fdd1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.212+0000 7fdd1a59c640 1 -- 172.21.15.116:0/3667626599 >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 msgr2=0x7fdd1c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.212+0000 7fdd1a59c640 1 -- 172.21.15.116:0/3667626599 >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 msgr2=0x7fdd1c12ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.212+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/3667626599 >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 0x7fdd1c12ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.212+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/3667626599 >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 0x7fdd1c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:33.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.213+0000 7fdd1ad9d640 1 -- 172.21.15.116:0/3667626599 >> v1:172.21.15.116:6789/0 conn(0x7fdd1c133070 legacy=0x7fdd1c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:33.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981337775 0 0) 0x7fdd1c0978c0 con 0x7fdd1c133070 2024-09-18T12:44:33.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): my global_id is 194499 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd080034e0 con 0x7fdd1c133070 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 -- 172.21.15.116:0/3667626599 >> v1:172.21.15.144:6789/0 conn(0x7fdd1c09a8a0 legacy=0x7fdd1c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1600990976 0 0) 0x7fdd080034e0 con 0x7fdd1c133070 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 msgr2=0x7fdd1c12ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 --2- 172.21.15.116:0/3667626599 >> v2:172.21.15.144:3300/0 conn(0x7fdd1c0974f0 0x7fdd1c12ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 >> v1:172.21.15.144:6789/0 conn(0x7fdd1c09a8a0 legacy=0x7fdd1c09ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: _finish_hunting 0 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c098db0 con 0x7fdd1c133070 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: _finish_auth 0 2024-09-18T12:44:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd20f9d640 5 monclient: authenticate success, global_id 194499 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd20f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fdd0c0017a0 con 0x7fdd1c133070 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: dump: 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 >> v1:172.21.15.116:6789/0 conn(0x7fdd1c133070 legacy=0x7fdd1c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08003bc0 0x7fdd08003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): picked mon.a con 0x7fdd08003bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd08004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): picked mon.c con 0x7fdd08004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd080099c0 0x7fdd0800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): picked mon.b con 0x7fdd080099c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c098ad0 con 0x7fdd08003bc0 2024-09-18T12:44:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0800c420 con 0x7fdd080099c0 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0800c5e0 con 0x7fdd08004780 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1959a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd20f9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08003bc0 0x7fdd08003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd08004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 10 monclient(hunting): get_auth_request con 0x7fdd08003bc0 auth_method 0 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): get_auth_request con 0x7fdd08004780 auth_method 0 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd080099c0 0x7fdd0800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 10 monclient(hunting): get_auth_request con 0x7fdd080099c0 auth_method 0 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:33.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient(hunting): handle_auth_done global_id 194499 payload 293 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 -- 172.21.15.116:0/3667626599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd080099c0 msgr2=0x7fdd0800bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd080099c0 0x7fdd0800bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 -- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08003bc0 msgr2=0x7fdd08003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08003bc0 0x7fdd08003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 monclient: found mon.c 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1ad9d640 1 -- 172.21.15.116:0/3667626599 reap_dead start 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 1 -- 172.21.15.116:0/3667626599 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd080034e0 con 0x7fdd08004780 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 10 monclient: _finish_auth 0 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd19d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:33.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.214+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08003bc0 0x7fdd08003fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd19d9b640 1 -- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 msgr2=0x7fdd08004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd19d9b640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd08004bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd04001b10 tx=0x7fdd04007c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92b1ff317d0c7ad3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0402ab30 con 0x7fdd08004780 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 10 monclient: dump: 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0402ac90 con 0x7fdd08004780 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd20f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 1 -- 172.21.15.116:0/3667626599 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04007340 con 0x7fdd08004780 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd20f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd20f9d640 4 set_mon_vals no callback set 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.217+0000 7fdd1959a640 10 monclient: dump: 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 10 monclient: shutdown 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 1 -- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 msgr2=0x7fdd08004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd08004bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd04001b10 tx=0x7fdd04007c80 comp rx=0 tx=0).stop 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 1 -- 172.21.15.116:0/3667626599 shutdown_connections 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/3667626599 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd08004bf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 1 -- 172.21.15.116:0/3667626599 >> 172.21.15.116:0/3667626599 conn(0x7fdd1c0672b0 msgr2=0x7fdd1c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 1 -- 172.21.15.116:0/3667626599 shutdown_connections 2024-09-18T12:44:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.218+0000 7fdd20f9d640 1 -- 172.21.15.116:0/3667626599 wait complete. 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient: build_initial_monmap 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient: monmap: 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 Processor -- start 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 -- start start 2024-09-18T12:44:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient: init 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd1c06f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd08004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd1c130310 0x7fdd1c132160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c130310 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd1c069750 0x7fdd1c069bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1c069750 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1317c0 con 0x7fdd1c130310 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c129e40 con 0x7fdd1c069750 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1314e0 con 0x7fdd08004780 2024-09-18T12:44:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd20f9d640 10 monclient(hunting): authenticate will time out at 1900.541925s 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd1a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd1c06f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.219+0000 7fdd19d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd1c130310 0x7fdd1c132160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd1c06f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58196/0 (socket says 172.21.15.116:58196) 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd1c130310 0x7fdd1c132160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56062/0 (socket says 172.21.15.116:56062) 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 -- 172.21.15.116:0/1587487938 learned_addr learned my addr 172.21.15.116:0/1587487938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): get_auth_request con 0x7fdd08004780 auth_method 0 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 10 monclient(hunting): get_auth_request con 0x7fdd1c130310 auth_method 0 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1ad9d640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd1c069750 0x7fdd1c069bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1ad9d640 10 monclient(hunting): get_auth_request con 0x7fdd1c069750 auth_method 0 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient(hunting): handle_auth_done global_id 224186 payload 824 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 -- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd1c069750 msgr2=0x7fdd1c069bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd1c069750 0x7fdd1c069bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 -- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd1c130310 msgr2=0x7fdd1c132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd1c130310 0x7fdd1c132160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 monclient: found mon.noname-c 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd1a59c640 1 -- 172.21.15.116:0/1587487938 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c131200 con 0x7fdd08004780 2024-09-18T12:44:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.220+0000 7fdd19d9b640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd1c130310 0x7fdd1c132160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd1a59c640 1 -- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 msgr2=0x7fdd1c06f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd1a59c640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd1c06f000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400f5f0 tx=0x7fdd14009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 1 -- 172.21.15.116:0/1587487938 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1400fec0 con 0x7fdd08004780 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: dump: 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: _finish_auth 0 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 1 -- 172.21.15.116:0/1587487938 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd140147f0 con 0x7fdd08004780 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd20f9d640 5 monclient: authenticate success, global_id 224186 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd20f9d640 10 monclient: _renew_subs 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd20f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c06f540 con 0x7fdd08004780 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 1 -- 172.21.15.116:0/1587487938 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14014b70 con 0x7fdd08004780 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd127fc640 10 monclient: dump: 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:33.241 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd20f9d640 10 monclient: _renew_subs 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd20f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd18d99640 4 set_mon_vals no callback set 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.222+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c06a110 con 0x7fdd08004780 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.223+0000 7fdd127fc640 1 -- 172.21.15.116:0/1587487938 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fdd1400ad50 con 0x7fdd08004780 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.223+0000 7fdd20f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.223+0000 7fdd20f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.223+0000 7fdd20f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.223+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c04bf60 con 0x7fdd08004780 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.223+0000 7fdd127fc640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdcfc0384e0 0x7fdcfc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.223+0000 7fdd127fc640 1 -- 172.21.15.116:0/1587487938 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fdd1404cd80 con 0x7fdd08004780 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.224+0000 7fdd19d9b640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdcfc0384e0 0x7fdcfc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.224+0000 7fdd19d9b640 10 monclient: get_auth_request con 0x7fdcfc0384e0 auth_method 0 2024-09-18T12:44:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.226+0000 7fdd19d9b640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdcfc0384e0 0x7fdcfc03a990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdd04007fd0 tx=0x7fdd04030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.228+0000 7fdd127fc640 1 -- 172.21.15.116:0/1587487938 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fdd1401db50 con 0x7fdd08004780 2024-09-18T12:44:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.228+0000 7fdd127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.228+0000 7fdd127fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.440+0000 7fdd20f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.440+0000 7fdd20f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.440+0000 7fdd20f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.440+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdd1c134b10 con 0x7fdd08004780 2024-09-18T12:44:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.440+0000 7fdd127fc640 1 -- 172.21.15.116:0/1587487938 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fdd1401d370 con 0x7fdd08004780 2024-09-18T12:44:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.440+0000 7fdd127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.440+0000 7fdd127fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:33.443 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":186,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:33.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdcfc0384e0 msgr2=0x7fdcfc03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:33.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdcfc0384e0 0x7fdcfc03a990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fdd04007fd0 tx=0x7fdd04030000 comp rx=0 tx=0).stop 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 10 monclient: shutdown 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 msgr2=0x7fdd1c06f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd1c06f000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400f5f0 tx=0x7fdd14009720 comp rx=0 tx=0).stop 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 shutdown_connections 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fdcfc0384e0 0x7fdcfc03a990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd1c069750 0x7fdd1c069bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd1c130310 0x7fdd1c132160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 --2- 172.21.15.116:0/1587487938 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdd08004780 0x7fdd1c06f000 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 >> 172.21.15.116:0/1587487938 conn(0x7fdd1c0672b0 msgr2=0x7fdd1c129640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 shutdown_connections 2024-09-18T12:44:33.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:33.442+0000 7fdd20f9d640 1 -- 172.21.15.116:0/1587487938 wait complete. 2024-09-18T12:44:36.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:36.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.556+0000 7faccb31d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:36.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.556+0000 7faccb31d640 10 monclient: build_initial_monmap 2024-09-18T12:44:36.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.556+0000 7faccb31d640 10 monclient: monmap: 2024-09-18T12:44:36.558 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:36.558 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:36.558443+0000 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:36.558443+0000 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:36.559 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.557+0000 7faccb31d640 1 Processor -- start 2024-09-18T12:44:36.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.557+0000 7faccb31d640 1 -- start start 2024-09-18T12:44:36.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.557+0000 7faccb31d640 10 monclient: init 2024-09-18T12:44:36.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.557+0000 7faccb31d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:36.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facc4096c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7facc4097030 0x7facc4097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient(hunting): picked mon.noname-c con 0x7facc4097030 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7facc4133b40 0x7facc4135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient(hunting): picked mon.noname-a con 0x7facc4133b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc409d2e0 con 0x7facc4096c60 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc409d6b0 con 0x7facc4133b40 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc40992f0 con 0x7facc4097030 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:36.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7faccb31d640 10 monclient(hunting): authenticate will time out at 1903.880122s 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7facc4133b40 0x7facc4135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7facc4133b40 0x7facc4135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58598/0 (socket says 172.21.15.116:58598) 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 -- 172.21.15.116:0/638038275 learned_addr learned my addr 172.21.15.116:0/638038275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): get_auth_request con 0x7facc4133b40 auth_method 0 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc88b8640 1 --2- 172.21.15.116:0/638038275 >> v2:172.21.15.144:3300/0 conn(0x7facc4097030 0x7facc4097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc88b8640 10 monclient(hunting): get_auth_request con 0x7facc4097030 auth_method 0 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc88b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc88b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc88b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient(hunting): handle_auth_done global_id 194505 payload 274 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 -- 172.21.15.116:0/638038275 >> v2:172.21.15.144:3300/0 conn(0x7facc4097030 msgr2=0x7facc4097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 --2- 172.21.15.116:0/638038275 >> v2:172.21.15.144:3300/0 conn(0x7facc4097030 0x7facc4097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 -- 172.21.15.116:0/638038275 >> v1:172.21.15.144:6789/0 conn(0x7facc4096c60 legacy=0x7facc409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient: _finish_hunting 0 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 monclient: found mon.noname-a 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.558+0000 7facc98ba640 1 -- 172.21.15.116:0/638038275 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40995d0 con 0x7facc4133b40 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facc98ba640 1 -- 172.21.15.116:0/638038275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 msgr2=0x7facc4135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facc98ba640 1 --2- 172.21.15.116:0/638038275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4135f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facc409ecd0 tx=0x7facbc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01e12e5ac5f9845 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 1 -- 172.21.15.116:0/638038275 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc007610 con 0x7facc4133b40 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: dump: 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.571 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: _finish_auth 0 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 1 -- 172.21.15.116:0/638038275 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc02d050 con 0x7facc4133b40 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7faccb31d640 5 monclient: authenticate success, global_id 194505 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7faccb31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7faccb31d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 1 -- 172.21.15.116:0/638038275 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc0079a0 con 0x7facc4133b40 2024-09-18T12:44:36.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7faccb31d640 4 set_mon_vals no callback set 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.561+0000 7facbbfff640 10 monclient: dump: 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.573 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 10 monclient: shutdown 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 -- 172.21.15.116:0/638038275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 msgr2=0x7facc4135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 --2- 172.21.15.116:0/638038275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4135f20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facc409ecd0 tx=0x7facbc009840 comp rx=0 tx=0).stop 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 -- 172.21.15.116:0/638038275 shutdown_connections 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 --2- 172.21.15.116:0/638038275 >> v2:172.21.15.144:3300/0 conn(0x7facc4097030 0x7facc4097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 --2- 172.21.15.116:0/638038275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4135f20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 -- 172.21.15.116:0/638038275 >> 172.21.15.116:0/638038275 conn(0x7facc40672b0 msgr2=0x7facc40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 -- 172.21.15.116:0/638038275 shutdown_connections 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 1 -- 172.21.15.116:0/638038275 wait complete. 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 10 monclient: build_initial_monmap 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.562+0000 7faccb31d640 10 monclient: monmap: 2024-09-18T12:44:36.574 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 Processor -- start 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 -- start start 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient: init 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:36.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc4096c60 0x7facc406f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient(hunting): picked mon.noname-b con 0x7facc4096c60 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc4097030 0x7facc412ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient(hunting): picked mon.noname-c con 0x7facc4097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4130480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient(hunting): picked mon.noname-a con 0x7facc4133b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc406e850 con 0x7facc4133b40 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41356b0 con 0x7facc4096c60 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.563+0000 7faccb31d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc409add0 con 0x7facc4097030 2024-09-18T12:44:36.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7faccb31d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7faccb31d640 10 monclient(hunting): authenticate will time out at 1903.886016s 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4130480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4130480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58602/0 (socket says 172.21.15.116:58602) 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 -- 172.21.15.116:0/3384673904 learned_addr learned my addr 172.21.15.116:0/3384673904 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): get_auth_request con 0x7facc4133b40 auth_method 0 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc4097030 0x7facc412ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 10 monclient(hunting): get_auth_request con 0x7facc4097030 auth_method 0 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc4096c60 0x7facc406f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 10 monclient(hunting): get_auth_request con 0x7facc4096c60 auth_method 0 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc90b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient(hunting): handle_auth_done global_id 194508 payload 824 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 -- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc4097030 msgr2=0x7facc412ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc4097030 0x7facc412ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 -- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc4096c60 msgr2=0x7facc406f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc4096c60 0x7facc406f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient: _finish_hunting 0 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 monclient: found mon.noname-a 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc98ba640 1 -- 172.21.15.116:0/3384673904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc009ae0 con 0x7facc4133b40 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.564+0000 7facc88b8640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc4097030 0x7facc412ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facc98ba640 1 -- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 msgr2=0x7facc4130480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:36.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facc98ba640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4130480 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7facbc009ab0 tx=0x7facbc036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 1 -- 172.21.15.116:0/3384673904 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc032cc0 con 0x7facc4133b40 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: dump: 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.580 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: _finish_auth 0 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 1 -- 172.21.15.116:0/3384673904 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc032e40 con 0x7facc4133b40 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7faccb31d640 5 monclient: authenticate success, global_id 194508 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7faccb31d640 10 monclient: _renew_subs 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7faccb31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc4069fa0 con 0x7facc4133b40 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 1 -- 172.21.15.116:0/3384673904 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc03b660 con 0x7facc4133b40 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:36.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facb97fa640 10 monclient: dump: 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7facbb7fe640 4 set_mon_vals no callback set 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7faccb31d640 10 monclient: _renew_subs 2024-09-18T12:44:36.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7faccb31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.566+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc4131b10 con 0x7facc4133b40 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.567+0000 7facb97fa640 1 -- 172.21.15.116:0/3384673904 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7facbc038050 con 0x7facc4133b40 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.567+0000 7faccb31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.567+0000 7faccb31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.567+0000 7faccb31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.568+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc404bf60 con 0x7facc4133b40 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.568+0000 7facb97fa640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faca8038490 0x7faca803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.568+0000 7facb97fa640 1 -- 172.21.15.116:0/3384673904 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7facbc06ce10 con 0x7facc4133b40 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.568+0000 7facc90b9640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faca8038490 0x7faca803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.568+0000 7facc90b9640 10 monclient: get_auth_request con 0x7faca8038490 auth_method 0 2024-09-18T12:44:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.571+0000 7facc90b9640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faca8038490 0x7faca803a940 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7facc0007fd0 tx=0x7facc0007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.572+0000 7facb97fa640 1 -- 172.21.15.116:0/3384673904 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7facbc03b9d0 con 0x7facc4133b40 2024-09-18T12:44:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.572+0000 7facb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:36.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.572+0000 7facb97fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.785+0000 7faccb31d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.785+0000 7faccb31d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.785+0000 7faccb31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.785+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7facc40983b0 con 0x7facc4133b40 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.785+0000 7facb97fa640 1 -- 172.21.15.116:0/3384673904 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7facbc06e020 con 0x7facc4133b40 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.785+0000 7facb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.785+0000 7facb97fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:36.787 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faca8038490 msgr2=0x7faca803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faca8038490 0x7faca803a940 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7facc0007fd0 tx=0x7facc0007e10 comp rx=0 tx=0).stop 2024-09-18T12:44:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 10 monclient: shutdown 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 msgr2=0x7facc4130480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4130480 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7facbc009ab0 tx=0x7facbc036000 comp rx=0 tx=0).stop 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 shutdown_connections 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faca8038490 0x7faca803a940 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facc4133b40 0x7facc4130480 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7facc4097030 0x7facc412ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 --2- 172.21.15.116:0/3384673904 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7facc4096c60 0x7facc406f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 >> 172.21.15.116:0/3384673904 conn(0x7facc40672b0 msgr2=0x7facc4135b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 shutdown_connections 2024-09-18T12:44:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.787+0000 7faccb31d640 1 -- 172.21.15.116:0/3384673904 wait complete. 2024-09-18T12:44:36.799 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:44:36.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:44:36.799 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.891+0000 7f7c3b91d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.891+0000 7f7c3b91d640 10 monclient: build_initial_monmap 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.891+0000 7f7c3b91d640 10 monclient: monmap: 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:36.893463+0000 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:36.893463+0000 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.895 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:36.895 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:36.895 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:36.895 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:36.895 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:36.895 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.892+0000 7f7c3b91d640 1 Processor -- start 2024-09-18T12:44:36.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.892+0000 7f7c3b91d640 1 -- start start 2024-09-18T12:44:36.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.892+0000 7f7c3b91d640 10 monclient: init 2024-09-18T12:44:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T12:44:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c3409a7f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c3409abc0 0x7f7c3409afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3409abc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7c34134a30 0x7f7c34136e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c34134a30 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3409c800 con 0x7f7c3409a7f0 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c340a2050 con 0x7f7c34134a30 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3409cfb0 con 0x7f7c3409abc0 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c3b91d640 10 monclient(hunting): authenticate will time out at 1904.215147s 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c396b9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7c3409a7f0 0x7f7c3409c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51792/0 (socket says 172.21.15.116:51792) 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c396b9640 1 -- 172.21.15.116:0/3066239847 learned_addr learned my addr 172.21.15.116:0/3066239847 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 1 --2- 172.21.15.116:0/3066239847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c3409abc0 0x7f7c3409afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): get_auth_request con 0x7f7c3409abc0 auth_method 0 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c39eba640 1 --2- 172.21.15.116:0/3066239847 >> v2:172.21.15.144:3300/0 conn(0x7f7c34134a30 0x7f7c34136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c39eba640 10 monclient(hunting): get_auth_request con 0x7f7c34134a30 auth_method 0 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c39eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c39eba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c39eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_done global_id 224192 payload 274 2024-09-18T12:44:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 1 -- 172.21.15.116:0/3066239847 >> v2:172.21.15.144:3300/0 conn(0x7f7c34134a30 msgr2=0x7f7c34136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 1 --2- 172.21.15.116:0/3066239847 >> v2:172.21.15.144:3300/0 conn(0x7f7c34134a30 0x7f7c34136e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 1 -- 172.21.15.116:0/3066239847 >> v1:172.21.15.116:6789/0 conn(0x7f7c3409a7f0 legacy=0x7f7c3409c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient: _finish_hunting 0 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 1 monclient: found mon.noname-b 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.893+0000 7f7c38eb8640 1 -- 172.21.15.116:0/3066239847 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3409d290 con 0x7f7c3409abc0 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c38eb8640 1 -- 172.21.15.116:0/3066239847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c3409abc0 msgr2=0x7f7c3409afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c38eb8640 1 --2- 172.21.15.116:0/3066239847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c3409abc0 0x7f7c3409afe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009850 tx=0x7f7c240090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73de76e25564130 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 1 -- 172.21.15.116:0/3066239847 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2402add0 con 0x7f7c3409abc0 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: dump: 2024-09-18T12:44:36.914 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: _finish_auth 0 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 5 monclient: authenticate success, global_id 224192 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:36.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 1 -- 172.21.15.116:0/3066239847 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c24032440 con 0x7f7c3409abc0 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 4 set_mon_vals no callback set 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 1 -- 172.21.15.116:0/3066239847 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c240327e0 con 0x7f7c3409abc0 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c2bfff640 10 monclient: dump: 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.916 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 10 monclient: shutdown 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 1 -- 172.21.15.116:0/3066239847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c3409abc0 msgr2=0x7f7c3409afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 1 --2- 172.21.15.116:0/3066239847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c3409abc0 0x7f7c3409afe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009850 tx=0x7f7c240090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 1 -- 172.21.15.116:0/3066239847 shutdown_connections 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 1 --2- 172.21.15.116:0/3066239847 >> v2:172.21.15.144:3300/0 conn(0x7f7c34134a30 0x7f7c34136e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 1 --2- 172.21.15.116:0/3066239847 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c3409abc0 0x7f7c3409afe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.896+0000 7f7c3b91d640 1 -- 172.21.15.116:0/3066239847 >> 172.21.15.116:0/3066239847 conn(0x7f7c34067b50 msgr2=0x7f7c340a1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.897+0000 7f7c3b91d640 1 -- 172.21.15.116:0/3066239847 shutdown_connections 2024-09-18T12:44:36.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.897+0000 7f7c3b91d640 1 -- 172.21.15.116:0/3066239847 wait complete. 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.897+0000 7f7c3b91d640 10 monclient: build_initial_monmap 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.897+0000 7f7c3b91d640 10 monclient: monmap: 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.897+0000 7f7c3b91d640 1 Processor -- start 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.897+0000 7f7c3b91d640 1 -- start start 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.897+0000 7f7c3b91d640 10 monclient: init 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c3409a7f0 0x7f7c34126bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3409a7f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3409abc0 0x7f7c34127110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3409abc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c34134a30 0x7f7c34127650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c34134a30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34129d30 con 0x7f7c3409abc0 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3409b5a0 con 0x7f7c3409a7f0 2024-09-18T12:44:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34135570 con 0x7f7c34134a30 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c3b91d640 10 monclient(hunting): authenticate will time out at 1904.220469s 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3409abc0 0x7f7c34127110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c34134a30 0x7f7c34127650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3409abc0 0x7f7c34127110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58614/0 (socket says 172.21.15.116:58614) 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 -- 172.21.15.116:0/446732101 learned_addr learned my addr 172.21.15.116:0/446732101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): get_auth_request con 0x7f7c3409abc0 auth_method 0 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c34134a30 0x7f7c34127650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35912/0 (socket says 172.21.15.116:35912) 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 10 monclient(hunting): get_auth_request con 0x7f7c34134a30 auth_method 0 2024-09-18T12:44:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c39eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient(hunting): handle_auth_done global_id 194511 payload 824 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c396b9640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c3409a7f0 0x7f7c34126bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c34134a30 msgr2=0x7f7c34127650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7c34134a30 0x7f7c34127650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c3409a7f0 msgr2=0x7f7c34126bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7c3409a7f0 0x7f7c34126bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient: _finish_hunting 0 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 monclient: found mon.noname-a 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.898+0000 7f7c38eb8640 1 -- 172.21.15.116:0/446732101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c24009390 con 0x7f7c3409abc0 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c38eb8640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3409abc0 msgr2=0x7f7c34127110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c38eb8640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3409abc0 0x7f7c34127110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009850 tx=0x7f7c24032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24031d40 con 0x7f7c3409abc0 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: dump: 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: _finish_auth 0 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2403d440 con 0x7f7c3409abc0 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c3b91d640 5 monclient: authenticate success, global_id 194511 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c3b91d640 10 monclient: _renew_subs 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c3b91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c34129f60 con 0x7f7c3409abc0 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:36.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24031ea0 con 0x7f7c3409abc0 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c297fa640 10 monclient: dump: 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c2b7fe640 4 set_mon_vals no callback set 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c3b91d640 10 monclient: _renew_subs 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c3b91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.901+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3412a270 con 0x7f7c3409abc0 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c3b91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c3b91d640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 0x7f7c34127b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c3b91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34063720 con 0x7f7c3412a610 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 0x7f7c34127b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient: get_auth_request con 0x7f7c3412a610 auth_method 0 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7c24039040 con 0x7f7c3409abc0 2024-09-18T12:44:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c297fa640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c18038490 0x7f7c1803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 10 monclient(hunting): handle_auth_done global_id 194514 payload 824 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c396b9640 1 -- 172.21.15.116:0/446732101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c3404bf60 con 0x7f7c3412a610 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7c2406c440 con 0x7f7c3409abc0 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c39eba640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c18038490 0x7f7c1803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.902+0000 7f7c39eba640 10 monclient: get_auth_request con 0x7f7c18038490 auth_method 0 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.904+0000 7f7c396b9640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 msgr2=0x7f7c34127b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.904+0000 7f7c396b9640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 0x7f7c34127b90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d400 tx=0x7f7c30009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.904+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30005cb0 con 0x7f7c3412a610 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.904+0000 7f7c297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.904+0000 7f7c297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.904+0000 7f7c297fa640 10 monclient: dump: 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7c30005e10 con 0x7f7c3412a610 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c297fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 msgr2=0x7f7c34127b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c297fa640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 0x7f7c34127b90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d400 tx=0x7f7c30009040 comp rx=0 tx=0).stop 2024-09-18T12:44:36.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 msgr2=0x7f7c34127b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:36.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c297fa640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3412a610 0x7f7c34127b90 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d400 tx=0x7f7c30009040 comp rx=0 tx=0).stop 2024-09-18T12:44:36.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.905+0000 7f7c39eba640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c18038490 0x7f7c1803a940 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7c3409b570 tx=0x7f7c2c009850 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c3b91d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:36.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c3b91d640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 0x7f7c3406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c3b91d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34129380 con 0x7f7c3406c3e0 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 0x7f7c3406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 10 monclient: get_auth_request con 0x7f7c3406c3e0 auth_method 0 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.910+0000 7f7c396b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.911+0000 7f7c396b9640 10 monclient(hunting): handle_auth_done global_id 194517 payload 824 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.911+0000 7f7c396b9640 1 -- 172.21.15.116:0/446732101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7c34063b60 con 0x7f7c3406c3e0 2024-09-18T12:44:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c396b9640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 msgr2=0x7f7c3406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c396b9640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 0x7f7c3406c7e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d400 tx=0x7f7c30019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30012c30 con 0x7f7c3406c3e0 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 10 monclient: dump: 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:36.931 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f7c30012d90 con 0x7f7c3406c3e0 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 msgr2=0x7f7c3406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 0x7f7c3406c7e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d400 tx=0x7f7c30019000 comp rx=0 tx=0).stop 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 msgr2=0x7f7c3406c7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c297fa640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3406c3e0 0x7f7c3406c7e0 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000d400 tx=0x7f7c30019000 comp rx=0 tx=0).stop 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:36.932 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 188, 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:36.933 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:36.934 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:36.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:36.936 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:36.938 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:36.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:36.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:36.942 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:44:36.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:36.944 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:36.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:36.944 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:36.944 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:36.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c18038490 msgr2=0x7f7c1803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c3b91d640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7c18038490 0x7f7c1803a940 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7c3409b570 tx=0x7f7c2c009850 comp rx=0 tx=0).stop 2024-09-18T12:44:36.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c3b91d640 10 monclient: shutdown 2024-09-18T12:44:36.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c3b91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:36.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3409abc0 msgr2=0x7f7c34127110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:36.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c3b91d640 1 --2- 172.21.15.116:0/446732101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c3409abc0 0x7f7c34127110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009850 tx=0x7f7c24032000 comp rx=0 tx=0).stop 2024-09-18T12:44:36.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.913+0000 7f7c39eba640 1 -- 172.21.15.116:0/446732101 reap_dead start 2024-09-18T12:44:36.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.914+0000 7f7c39eba640 1 -- 172.21.15.116:0/446732101 reap_dead start 2024-09-18T12:44:36.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.914+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 shutdown_connections 2024-09-18T12:44:36.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.914+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 >> 172.21.15.116:0/446732101 conn(0x7f7c34067b50 msgr2=0x7f7c340a0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:36.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.914+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 shutdown_connections 2024-09-18T12:44:36.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:36.914+0000 7f7c3b91d640 1 -- 172.21.15.116:0/446732101 wait complete. 2024-09-18T12:44:36.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:44:36.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:44:36.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.037+0000 7f0d0b6af640 10 monclient: get_monmap_and_config 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.037+0000 7f0d0b6af640 10 monclient: build_initial_monmap 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.037+0000 7f0d0b6af640 10 monclient: monmap: 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:37.039220+0000 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:37.039220+0000 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.040 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:37.040 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:37.040 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:37.040 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:37.040 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:37.040 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 1 Processor -- start 2024-09-18T12:44:37.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 1 -- start start 2024-09-18T12:44:37.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient: init 2024-09-18T12:44:37.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:37.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0d04131980 0x7f0d04131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient(hunting): picked mon.noname-c con 0x7f0d04131980 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d0409fda0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04132290 0x7f0d0409f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient(hunting): picked mon.noname-b con 0x7f0d04132290 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d040a1180 con 0x7f0d0409fda0 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d040a1460 con 0x7f0d04131980 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d04139cd0 con 0x7f0d04132290 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d0b6af640 10 monclient(hunting): authenticate will time out at 1904.360802s 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d08c4a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0d0409fda0 0x7f0d0409e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51802/0 (socket says 172.21.15.116:51802) 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d08c4a640 1 -- 172.21.15.116:0/508042457 learned_addr learned my addr 172.21.15.116:0/508042457 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d09c4c640 1 --2- 172.21.15.116:0/508042457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04132290 0x7f0d0409f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d09c4c640 10 monclient(hunting): get_auth_request con 0x7f0d04132290 auth_method 0 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d09c4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d09c4c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.038+0000 7f0d09c4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d0944b640 1 --2- 172.21.15.116:0/508042457 >> v2:172.21.15.144:3300/0 conn(0x7f0d04131980 0x7f0d04131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d0944b640 10 monclient(hunting): get_auth_request con 0x7f0d04131980 auth_method 0 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d0944b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d0944b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d0944b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 10 monclient(hunting): handle_auth_done global_id 224198 payload 274 2024-09-18T12:44:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 1 -- 172.21.15.116:0/508042457 >> v2:172.21.15.144:3300/0 conn(0x7f0d04131980 msgr2=0x7f0d04131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 1 --2- 172.21.15.116:0/508042457 >> v2:172.21.15.144:3300/0 conn(0x7f0d04131980 0x7f0d04131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 1 -- 172.21.15.116:0/508042457 >> v1:172.21.15.116:6789/0 conn(0x7f0d0409fda0 legacy=0x7f0d0409e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 10 monclient: _finish_hunting 0 2024-09-18T12:44:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 1 monclient: found mon.noname-b 2024-09-18T12:44:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:44:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.039+0000 7f0d09c4c640 1 -- 172.21.15.116:0/508042457 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d04139e10 con 0x7f0d04132290 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d09c4c640 1 -- 172.21.15.116:0/508042457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04132290 msgr2=0x7f0d0409f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d09c4c640 1 --2- 172.21.15.116:0/508042457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04132290 0x7f0d0409f0d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009850 tx=0x7f0cf40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5df4740b4045812 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 1 -- 172.21.15.116:0/508042457 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf402ae60 con 0x7f0d04132290 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: dump: 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: _finish_auth 0 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 5 monclient: authenticate success, global_id 224198 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 1 -- 172.21.15.116:0/508042457 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf4032440 con 0x7f0d04132290 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 4 set_mon_vals no callback set 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 1 -- 172.21.15.116:0/508042457 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf40327e0 con 0x7f0d04132290 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d03fff640 10 monclient: dump: 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 10 monclient: shutdown 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 -- 172.21.15.116:0/508042457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04132290 msgr2=0x7f0d0409f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/508042457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04132290 0x7f0d0409f0d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009850 tx=0x7f0cf40090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 -- 172.21.15.116:0/508042457 shutdown_connections 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/508042457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04132290 0x7f0d0409f0d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/508042457 >> v2:172.21.15.144:3300/0 conn(0x7f0d04131980 0x7f0d04131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 -- 172.21.15.116:0/508042457 >> 172.21.15.116:0/508042457 conn(0x7f0d04067bd0 msgr2=0x7f0d040a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 -- 172.21.15.116:0/508042457 shutdown_connections 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.042+0000 7f0d0b6af640 1 -- 172.21.15.116:0/508042457 wait complete. 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.043+0000 7f0d0b6af640 10 monclient: build_initial_monmap 2024-09-18T12:44:37.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.043+0000 7f0d0b6af640 10 monclient: monmap: 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.043+0000 7f0d0b6af640 1 Processor -- start 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.043+0000 7f0d0b6af640 1 -- start start 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.043+0000 7f0d0b6af640 10 monclient: init 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:37.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d0409fda0 0x7f0d04137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0409fda0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 0x7f0d04133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): picked mon.noname-c con 0x7f0d04131980 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d04132290 0x7f0d04133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): picked mon.noname-b con 0x7f0d04132290 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041371f0 con 0x7f0d0409fda0 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0409bf10 con 0x7f0d04132290 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d040a4760 con 0x7f0d04131980 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0b6af640 10 monclient(hunting): authenticate will time out at 1904.366273s 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d0409fda0 0x7f0d04137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 0x7f0d04133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d0409fda0 0x7f0d04137b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58636/0 (socket says 172.21.15.116:58636) 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 0x7f0d04133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35936/0 (socket says 172.21.15.116:35936) 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 1 -- 172.21.15.116:0/1309521391 learned_addr learned my addr 172.21.15.116:0/1309521391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 10 monclient(hunting): get_auth_request con 0x7f0d0409fda0 auth_method 0 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): get_auth_request con 0x7f0d04131980 auth_method 0 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d09c4c640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d04132290 0x7f0d04133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d09c4c640 10 monclient(hunting): get_auth_request con 0x7f0d04132290 auth_method 0 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d09c4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d09c4c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d09c4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient(hunting): handle_auth_done global_id 224201 payload 824 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d04132290 msgr2=0x7f0d04133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d04132290 0x7f0d04133ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d0409fda0 msgr2=0x7f0d04137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d0409fda0 0x7f0d04137b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient: _finish_hunting 0 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 monclient: found mon.noname-c 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d08c4a640 1 -- 172.21.15.116:0/1309521391 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf4009390 con 0x7f0d04131980 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d09c4c640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d04132290 0x7f0d04133ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.044+0000 7f0d0944b640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d0409fda0 0x7f0d04137b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.046+0000 7f0d08c4a640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 msgr2=0x7f0d04133760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.046+0000 7f0d08c4a640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 0x7f0d04133760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c4c0 tx=0x7f0cf8007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.046+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8005cb0 con 0x7f0d04131980 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:37.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: dump: 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: _finish_auth 0 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:37.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf8005e30 con 0x7f0d04131980 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 5 monclient: authenticate success, global_id 224201 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 10 monclient: _renew_subs 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d041355f0 con 0x7f0d04131980 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8007320 con 0x7f0d04131980 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 10 monclient: dump: 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.062 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 10 monclient: _renew_subs 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d04135100 con 0x7f0d04131980 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d037fe640 4 set_mon_vals no callback set 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 0x7f0d04130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d04065e00 con 0x7f0d041341e0 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0cf801b3b0 con 0x7f0d04131980 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.047+0000 7f0d0944b640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 0x7f0d04130fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient: get_auth_request con 0x7f0d041341e0 auth_method 0 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d017fa640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0cf0038490 0x7f0cf003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 10 monclient(hunting): handle_auth_done global_id 224204 payload 824 2024-09-18T12:44:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d0944b640 1 -- 172.21.15.116:0/1309521391 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d04136ef0 con 0x7f0d041341e0 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0cf804b7b0 con 0x7f0d04131980 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d09c4c640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0cf0038490 0x7f0cf003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.048+0000 7f0d09c4c640 10 monclient: get_auth_request con 0x7f0cf0038490 auth_method 0 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d0944b640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 msgr2=0x7f0d04130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d0944b640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 0x7f0d04130fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0027b0 tx=0x7f0cfc009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc002e80 con 0x7f0d041341e0 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 10 monclient: dump: 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0cfc010ce0 con 0x7f0d041341e0 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 msgr2=0x7f0d04130fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 0x7f0d04130fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0027b0 tx=0x7f0cfc009920 comp rx=0 tx=0).stop 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 msgr2=0x7f0d04130fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.050+0000 7f0d017fa640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d041341e0 0x7f0d04130fc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.051+0000 7f0d09c4c640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0cf0038490 0x7f0cf003a940 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4034000 tx=0x7f0cf4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0b6af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 0x7f0d04029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0b6af640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041345b0 con 0x7f0d040298c0 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 0x7f0d04029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient: get_auth_request con 0x7f0d040298c0 auth_method 0 2024-09-18T12:44:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 10 monclient(hunting): handle_auth_done global_id 224207 payload 824 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.056+0000 7f0d0944b640 1 -- 172.21.15.116:0/1309521391 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0d0404bf60 con 0x7f0d040298c0 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0944b640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 msgr2=0x7f0d04029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0944b640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 0x7f0d04029cc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0027b0 tx=0x7f0cfc01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc015de0 con 0x7f0d040298c0 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 10 monclient: dump: 2024-09-18T12:44:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f0cfc010ad0 con 0x7f0d040298c0 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 msgr2=0x7f0d04029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 0x7f0d04029cc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0027b0 tx=0x7f0cfc01b000 comp rx=0 tx=0).stop 2024-09-18T12:44:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 msgr2=0x7f0d04029cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d017fa640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d040298c0 0x7f0d04029cc0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0027b0 tx=0x7f0cfc01b000 comp rx=0 tx=0).stop 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 188, 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:37.071 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:37.072 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:37.074 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:37.076 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:37.077 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.079 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:37.080 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.081 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:37.082 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:37.083 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:37.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.083 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.083 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:37.083 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0cf0038490 msgr2=0x7f0cf003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0cf0038490 0x7f0cf003a940 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4034000 tx=0x7f0cf4032000 comp rx=0 tx=0).stop 2024-09-18T12:44:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 10 monclient: shutdown 2024-09-18T12:44:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 msgr2=0x7f0d04133760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 0x7f0d04133760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800c4c0 tx=0x7f0cf8007f00 comp rx=0 tx=0).stop 2024-09-18T12:44:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d09c4c640 1 -- 172.21.15.116:0/1309521391 reap_dead start 2024-09-18T12:44:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 shutdown_connections 2024-09-18T12:44:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 --2- 172.21.15.116:0/1309521391 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0d04131980 0x7f0d04133760 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 >> 172.21.15.116:0/1309521391 conn(0x7f0d04067bd0 msgr2=0x7f0d040a4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 shutdown_connections 2024-09-18T12:44:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.059+0000 7f0d0b6af640 1 -- 172.21.15.116:0/1309521391 wait complete. 2024-09-18T12:44:37.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:44:37.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:44:37.086 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.177+0000 7f5d51080640 10 monclient: get_monmap_and_config 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.177+0000 7f5d51080640 10 monclient: build_initial_monmap 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.177+0000 7f5d51080640 10 monclient: monmap: 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:37.178767+0000 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:37.178767+0000 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.177+0000 7f5d51080640 1 Processor -- start 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.177+0000 7f5d51080640 1 -- start start 2024-09-18T12:44:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.177+0000 7f5d51080640 10 monclient: init 2024-09-18T12:44:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T12:44:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d4c1318f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d4c09ea50 0x7f5d4c09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4c09ea50 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5d4c136c20 0x7f5d4c139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c136c20 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4c131cc0 con 0x7f5d4c1318f0 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4c09fbf0 con 0x7f5d4c09ea50 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4c09fe30 con 0x7f5d4c136c20 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d51080640 10 monclient(hunting): authenticate will time out at 1904.500259s 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4ad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5d4c1318f0 0x7f5d4c131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51818/0 (socket says 172.21.15.116:51818) 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4ad9d640 1 -- 172.21.15.116:0/2029720941 learned_addr learned my addr 172.21.15.116:0/2029720941 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d4c09ea50 0x7f5d4c09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request con 0x7f5d4c09ea50 auth_method 0 2024-09-18T12:44:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4b59e640 1 --2- 172.21.15.116:0/2029720941 >> v2:172.21.15.144:3300/0 conn(0x7f5d4c136c20 0x7f5d4c139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4b59e640 10 monclient(hunting): get_auth_request con 0x7f5d4c136c20 auth_method 0 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4b59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_done global_id 194523 payload 274 2024-09-18T12:44:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 1 -- 172.21.15.116:0/2029720941 >> v2:172.21.15.144:3300/0 conn(0x7f5d4c136c20 msgr2=0x7f5d4c139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 1 --2- 172.21.15.116:0/2029720941 >> v2:172.21.15.144:3300/0 conn(0x7f5d4c136c20 0x7f5d4c139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 1 -- 172.21.15.116:0/2029720941 >> v1:172.21.15.116:6789/0 conn(0x7f5d4c1318f0 legacy=0x7f5d4c131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient: _finish_hunting 0 2024-09-18T12:44:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 1 monclient: found mon.noname-a 2024-09-18T12:44:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.178+0000 7f5d4a59c640 1 -- 172.21.15.116:0/2029720941 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c0a1210 con 0x7f5d4c09ea50 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d4a59c640 1 -- 172.21.15.116:0/2029720941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c09ea50 msgr2=0x7f5d4c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d4a59c640 1 --2- 172.21.15.116:0/2029720941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c09ea50 0x7f5d4c09ee20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c09c9e0 tx=0x7f5d340090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47012b2a9f6e8937 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 1 -- 172.21.15.116:0/2029720941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3402adf0 con 0x7f5d4c09ea50 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: dump: 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.193 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: _finish_auth 0 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 1 -- 172.21.15.116:0/2029720941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d34032440 con 0x7f5d4c09ea50 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d51080640 5 monclient: authenticate success, global_id 194523 2024-09-18T12:44:37.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d51080640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 1 -- 172.21.15.116:0/2029720941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d340327e0 con 0x7f5d4c09ea50 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d49d9b640 10 monclient: dump: 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.195 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d51080640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.181+0000 7f5d51080640 4 set_mon_vals no callback set 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 10 monclient: shutdown 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 -- 172.21.15.116:0/2029720941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c09ea50 msgr2=0x7f5d4c09ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 --2- 172.21.15.116:0/2029720941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c09ea50 0x7f5d4c09ee20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c09c9e0 tx=0x7f5d340090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 -- 172.21.15.116:0/2029720941 shutdown_connections 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 --2- 172.21.15.116:0/2029720941 >> v2:172.21.15.144:3300/0 conn(0x7f5d4c136c20 0x7f5d4c139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 --2- 172.21.15.116:0/2029720941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c09ea50 0x7f5d4c09ee20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 -- 172.21.15.116:0/2029720941 >> 172.21.15.116:0/2029720941 conn(0x7f5d4c067110 msgr2=0x7f5d4c067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 -- 172.21.15.116:0/2029720941 shutdown_connections 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.182+0000 7f5d51080640 1 -- 172.21.15.116:0/2029720941 wait complete. 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient: build_initial_monmap 2024-09-18T12:44:37.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient: monmap: 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 Processor -- start 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 -- start start 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient: init 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:37.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4c09ea50 0x7f5d4c1308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c09ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c1318f0 0x7f5d4c099780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4c1318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c136c20 0x7f5d4c099cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4c136c20 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c11c9e0 con 0x7f5d4c136c20 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c09bf10 con 0x7f5d4c1318f0 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c09e080 con 0x7f5d4c09ea50 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.183+0000 7f5d51080640 10 monclient(hunting): authenticate will time out at 1904.505978s 2024-09-18T12:44:37.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4c09ea50 0x7f5d4c1308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c136c20 0x7f5d4c099cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4c09ea50 0x7f5d4c1308d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35958/0 (socket says 172.21.15.116:35958) 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 1 -- 172.21.15.116:0/553181250 learned_addr learned my addr 172.21.15.116:0/553181250 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 10 monclient(hunting): get_auth_request con 0x7f5d4c09ea50 auth_method 0 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): get_auth_request con 0x7f5d4c136c20 auth_method 0 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c1318f0 0x7f5d4c099780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request con 0x7f5d4c1318f0 auth_method 0 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient(hunting): handle_auth_done global_id 194526 payload 824 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4c09ea50 msgr2=0x7f5d4c1308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4c09ea50 0x7f5d4c1308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c1318f0 msgr2=0x7f5d4c099780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c1318f0 0x7f5d4c099780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient: _finish_hunting 0 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 1 monclient: found mon.noname-a 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4b59e640 1 -- 172.21.15.116:0/553181250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d34009390 con 0x7f5d4c136c20 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.184+0000 7f5d4ad9d640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5d4c09ea50 0x7f5d4c1308d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d4b59e640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c136c20 msgr2=0x7f5d4c099cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d4b59e640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c136c20 0x7f5d4c099cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c007f30 tx=0x7f5d3c009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c018070 con 0x7f5d4c136c20 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: dump: 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: _finish_auth 0 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c014420 con 0x7f5d4c136c20 2024-09-18T12:44:37.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d51080640 5 monclient: authenticate success, global_id 194526 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d51080640 10 monclient: _renew_subs 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d51080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c11ba80 con 0x7f5d4c136c20 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c0147c0 con 0x7f5d4c136c20 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d42ffd640 10 monclient: dump: 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.203 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d51080640 10 monclient: _renew_subs 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d51080640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c141e60 con 0x7f5d4c136c20 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.186+0000 7f5d4959a640 4 set_mon_vals no callback set 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.187+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f5d3c003020 con 0x7f5d4c136c20 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.187+0000 7f5d51080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.187+0000 7f5d42ffd640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d2c038490 0x7f5d2c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.187+0000 7f5d51080640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 0x7f5d4c0298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.187+0000 7f5d51080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.187+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c065e00 con 0x7f5d4c130e10 2024-09-18T12:44:37.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.187+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f5d3c04c5d0 con 0x7f5d4c136c20 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 0x7f5d4c0298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient: get_auth_request con 0x7f5d4c130e10 auth_method 0 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4ad9d640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d2c038490 0x7f5d2c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4ad9d640 10 monclient: get_auth_request con 0x7f5d2c038490 auth_method 0 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_done global_id 234121 payload 824 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.188+0000 7f5d4a59c640 1 -- 172.21.15.116:0/553181250 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d4c04bf60 con 0x7f5d4c130e10 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.190+0000 7f5d4ad9d640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d2c038490 0x7f5d2c03a940 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5d44006fd0 tx=0x7f5d44004960 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.190+0000 7f5d4a59c640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 msgr2=0x7f5d4c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.190+0000 7f5d4a59c640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 0x7f5d4c0298c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c076600 tx=0x7f5d34032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.190+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34031a70 con 0x7f5d4c130e10 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.190+0000 7f5d42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.190+0000 7f5d42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.190+0000 7f5d42ffd640 10 monclient: dump: 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.206 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.191+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5d34031bd0 con 0x7f5d4c130e10 2024-09-18T12:44:37.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.191+0000 7f5d42ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:37.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.191+0000 7f5d42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:37.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.191+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 msgr2=0x7f5d4c0298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.191+0000 7f5d42ffd640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 0x7f5d4c0298c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c076600 tx=0x7f5d34032000 comp rx=0 tx=0).stop 2024-09-18T12:44:37.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.191+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 msgr2=0x7f5d4c0298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:37.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.191+0000 7f5d42ffd640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c130e10 0x7f5d4c0298c0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c076600 tx=0x7f5d34032000 comp rx=0 tx=0).stop 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d51080640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d51080640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 0x7f5d4c02a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d51080640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c1311e0 con 0x7f5d4c029e00 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 0x7f5d4c02a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient: get_auth_request con 0x7f5d4c029e00 auth_method 0 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_done global_id 234124 payload 824 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.197+0000 7f5d4a59c640 1 -- 172.21.15.116:0/553181250 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5d4c09b290 con 0x7f5d4c029e00 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d4a59c640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 msgr2=0x7f5d4c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d4a59c640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 0x7f5d4c02a200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c09c0a0 tx=0x7f5d3403e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3403d8e0 con 0x7f5d4c029e00 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 10 monclient: dump: 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.210 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f5d3403da40 con 0x7f5d4c029e00 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 msgr2=0x7f5d4c02a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 0x7f5d4c02a200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c09c0a0 tx=0x7f5d3403e000 comp rx=0 tx=0).stop 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 msgr2=0x7f5d4c02a200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:37.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.200+0000 7f5d42ffd640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d4c029e00 0x7f5d4c02a200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 188, 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:37.212 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:37.214 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:37.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:37.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.217 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:37.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:37.222 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:37.223 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:37.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d2c038490 msgr2=0x7f5d2c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5d2c038490 0x7f5d2c03a940 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5d44006fd0 tx=0x7f5d44004960 comp rx=0 tx=0).stop 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 10 monclient: shutdown 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c136c20 msgr2=0x7f5d4c099cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 --2- 172.21.15.116:0/553181250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d4c136c20 0x7f5d4c099cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c007f30 tx=0x7f5d3c009850 comp rx=0 tx=0).stop 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d4b59e640 1 -- 172.21.15.116:0/553181250 reap_dead start 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d4b59e640 1 -- 172.21.15.116:0/553181250 reap_dead start 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 shutdown_connections 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 >> 172.21.15.116:0/553181250 conn(0x7f5d4c067110 msgr2=0x7f5d4c09e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 shutdown_connections 2024-09-18T12:44:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.201+0000 7f5d51080640 1 -- 172.21.15.116:0/553181250 wait complete. 2024-09-18T12:44:37.226 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:44:37.226 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:44:37.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:44:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.317+0000 7f7a0895d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.317+0000 7f7a0895d640 10 monclient: build_initial_monmap 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.317+0000 7f7a0895d640 10 monclient: monmap: 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:37.318726+0000 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:37.318726+0000 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:37.320 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.317+0000 7f7a0895d640 1 Processor -- start 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.317+0000 7f7a0895d640 1 -- start start 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.317+0000 7f7a0895d640 10 monclient: init 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a041318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a0409ea50 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a04137c30 0x7f7a0413a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient(hunting): picked mon.noname-a con 0x7f7a04137c30 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a04131cc0 con 0x7f7a0409ea50 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0409f530 con 0x7f7a041318f0 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0409fbf0 con 0x7f7a04137c30 2024-09-18T12:44:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a0895d640 10 monclient(hunting): authenticate will time out at 1904.640256s 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a04137c30 0x7f7a0413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a04137c30 0x7f7a0413a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58670/0 (socket says 172.21.15.116:58670) 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f79fbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7a0409ea50 0x7f7a0409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51834/0 (socket says 172.21.15.116:51834) 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 1 -- 172.21.15.116:0/3390289208 learned_addr learned my addr 172.21.15.116:0/3390289208 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): get_auth_request con 0x7f7a04137c30 auth_method 0 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient(hunting): handle_auth_done global_id 194529 payload 274 2024-09-18T12:44:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 1 -- 172.21.15.116:0/3390289208 >> v1:172.21.15.144:6789/0 conn(0x7f7a041318f0 legacy=0x7f7a04131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:37.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 1 -- 172.21.15.116:0/3390289208 >> v1:172.21.15.116:6789/0 conn(0x7f7a0409ea50 legacy=0x7f7a0409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:37.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:44:37.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 1 monclient: found mon.noname-a 2024-09-18T12:44:37.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:37.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.318+0000 7f7a02d9d640 1 -- 172.21.15.116:0/3390289208 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0409fe30 con 0x7f7a04137c30 2024-09-18T12:44:37.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a02d9d640 1 -- 172.21.15.116:0/3390289208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a04137c30 msgr2=0x7f7a0413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a02d9d640 1 --2- 172.21.15.116:0/3390289208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a04137c30 0x7f7a0413a010 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79fc009f20 tx=0x7f79fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac3805fa73653fc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 1 -- 172.21.15.116:0/3390289208 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc007610 con 0x7f7a04137c30 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: dump: 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.330 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: _finish_auth 0 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 1 -- 172.21.15.116:0/3390289208 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79fc02d050 con 0x7f7a04137c30 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a0895d640 5 monclient: authenticate success, global_id 194529 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a0895d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 1 -- 172.21.15.116:0/3390289208 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc0079a0 con 0x7f7a04137c30 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a01d9b640 10 monclient: dump: 2024-09-18T12:44:37.331 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a0895d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.321+0000 7f7a0895d640 4 set_mon_vals no callback set 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 10 monclient: shutdown 2024-09-18T12:44:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 1 -- 172.21.15.116:0/3390289208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a04137c30 msgr2=0x7f7a0413a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 1 --2- 172.21.15.116:0/3390289208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a04137c30 0x7f7a0413a010 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79fc009f20 tx=0x7f79fc009840 comp rx=0 tx=0).stop 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 1 -- 172.21.15.116:0/3390289208 shutdown_connections 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 1 --2- 172.21.15.116:0/3390289208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a04137c30 0x7f7a0413a010 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 1 -- 172.21.15.116:0/3390289208 >> 172.21.15.116:0/3390289208 conn(0x7f7a04067110 msgr2=0x7f7a04067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 1 -- 172.21.15.116:0/3390289208 shutdown_connections 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 1 -- 172.21.15.116:0/3390289208 wait complete. 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 10 monclient: build_initial_monmap 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.322+0000 7f7a0895d640 10 monclient: monmap: 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 Processor -- start 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 -- start start 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient: init 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a0409ea50 0x7f7a0409b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a0409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 0x7f7a041327a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): picked mon.noname-a con 0x7f7a041318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a04137c30 0x7f7a04132ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a04137c30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a041354d0 con 0x7f7a041318f0 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a04136120 con 0x7f7a04137c30 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a04135a00 con 0x7f7a0409ea50 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0895d640 10 monclient(hunting): authenticate will time out at 1904.645673s 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 0x7f7a041327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a0409ea50 0x7f7a0409b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 0x7f7a041327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58676/0 (socket says 172.21.15.116:58676) 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a0409ea50 0x7f7a0409b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35960/0 (socket says 172.21.15.116:35960) 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 1 -- 172.21.15.116:0/522659324 learned_addr learned my addr 172.21.15.116:0/522659324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 10 monclient(hunting): get_auth_request con 0x7f7a041318f0 auth_method 0 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 10 monclient(hunting): get_auth_request con 0x7f7a0409ea50 auth_method 0 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f79fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.323+0000 7f7a0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 10 monclient(hunting): handle_auth_done global_id 194532 payload 824 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f7a02d9d640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a04137c30 0x7f7a04132ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 1 -- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a0409ea50 msgr2=0x7f7a0409b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a0409ea50 0x7f7a0409b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 1 -- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a04137c30 msgr2=0x7f7a04132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a04137c30 0x7f7a04132ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 1 monclient: found mon.noname-a 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.324+0000 7f79fbfff640 1 -- 172.21.15.116:0/522659324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79fc009ae0 con 0x7f7a041318f0 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fbfff640 1 -- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 msgr2=0x7f7a041327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fbfff640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 0x7f7a041327a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00c5e0 tx=0x7f79ec007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 1 -- 172.21.15.116:0/522659324 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec007510 con 0x7f7a041318f0 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: dump: 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.338 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: _finish_auth 0 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 1 -- 172.21.15.116:0/522659324 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec0057d0 con 0x7f7a041318f0 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0895d640 5 monclient: authenticate success, global_id 194532 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0895d640 10 monclient: _renew_subs 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0895d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0409d790 con 0x7f7a041318f0 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 1 -- 172.21.15.116:0/522659324 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec007690 con 0x7f7a041318f0 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f79fa7fc640 10 monclient: dump: 2024-09-18T12:44:37.339 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0159a640 4 set_mon_vals no callback set 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0895d640 10 monclient: _renew_subs 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0895d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.326+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a04136890 con 0x7f7a041318f0 2024-09-18T12:44:37.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f79fa7fc640 1 -- 172.21.15.116:0/522659324 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f79ec01b400 con 0x7f7a041318f0 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f7a0895d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f7a0895d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f7a0895d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0404bf60 con 0x7f7a041318f0 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f79fa7fc640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79e4038490 0x7f79e403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f79fa7fc640 1 -- 172.21.15.116:0/522659324 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f79ec04bd50 con 0x7f7a041318f0 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f7a0259c640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79e4038490 0x7f79e403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.327+0000 7f7a0259c640 10 monclient: get_auth_request con 0x7f79e4038490 auth_method 0 2024-09-18T12:44:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.330+0000 7f7a0259c640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79e4038490 0x7f79e403a940 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f79f4007fd0 tx=0x7f79f4007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:37.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.331+0000 7f79fa7fc640 1 -- 172.21.15.116:0/522659324 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f79ec005e20 con 0x7f7a041318f0 2024-09-18T12:44:37.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.331+0000 7f79fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:37.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.331+0000 7f79fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.546+0000 7f7a0895d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:44:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.546+0000 7f7a0895d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:44:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.546+0000 7f7a0895d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.546+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7a04135720 con 0x7f7a041318f0 2024-09-18T12:44:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.547+0000 7f79fa7fc640 1 -- 172.21.15.116:0/522659324 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f79ec01a360 con 0x7f7a041318f0 2024-09-18T12:44:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.547+0000 7f79fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:44:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.547+0000 7f79fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79e4038490 msgr2=0x7f79e403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79e4038490 0x7f79e403a940 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f79f4007fd0 tx=0x7f79f4007e10 comp rx=0 tx=0).stop 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 10 monclient: shutdown 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 msgr2=0x7f7a041327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 0x7f7a041327a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00c5e0 tx=0x7f79ec007f00 comp rx=0 tx=0).stop 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 shutdown_connections 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f79e4038490 0x7f79e403a940 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7a04137c30 0x7f7a04132ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a041318f0 0x7f7a041327a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 --2- 172.21.15.116:0/522659324 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7a0409ea50 0x7f7a0409b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 >> 172.21.15.116:0/522659324 conn(0x7f7a04067110 msgr2=0x7f7a040a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 shutdown_connections 2024-09-18T12:44:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f7a0895d640 1 -- 172.21.15.116:0/522659324 wait complete. 2024-09-18T12:44:37.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=99006111,config=3140514009,health=2509884219,kv=2187923472,logm=2712253243}) 2024-09-18T12:44:37.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=10,kv=9,logm=54} crc {auth=4173838585,config=3140514009,health=2712057815,kv=1496160121,logm=1038570746}) 2024-09-18T12:44:37.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=99006111,config=3140514009,health=2509884219,kv=2187923472,logm=2712253243}) 2024-09-18T12:44:37.553 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.549+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=7,logm=54} crc {auth=1244994785,config=3140514009,health=2509884219,kv=771457840,logm=1840308048}) 2024-09-18T12:44:37.558 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.558 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256693262}) 2024-09-18T12:44:37.558 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=213999515}) 2024-09-18T12:44:37.558 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.558 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256693262}) 2024-09-18T12:44:37.558 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.556+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4198327648}) 2024-09-18T12:44:37.562 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:44:37.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.925+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.925+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165324112}) 2024-09-18T12:44:37.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.925+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2344290083}) 2024-09-18T12:44:37.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.925+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.925+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165324112}) 2024-09-18T12:44:37.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.925+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496621537}) 2024-09-18T12:44:37.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.926+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.926+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240141993}) 2024-09-18T12:44:37.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.926+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3130276639}) 2024-09-18T12:44:37.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.926+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.926+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240141993}) 2024-09-18T12:44:37.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.926+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907236985}) 2024-09-18T12:44:37.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.927+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.927+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402787509}) 2024-09-18T12:44:37.930 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.927+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2158498315}) 2024-09-18T12:44:37.930 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.927+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.930 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.927+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402787509}) 2024-09-18T12:44:37.930 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.928+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=901933216}) 2024-09-18T12:44:37.930 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.928+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.931 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.928+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477656777}) 2024-09-18T12:44:37.931 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.928+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4214826367}) 2024-09-18T12:44:37.931 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.928+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.931 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.928+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477656777}) 2024-09-18T12:44:37.931 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.928+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=249088151}) 2024-09-18T12:44:37.938 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.936+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.938 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.936+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521553014}) 2024-09-18T12:44:37.938 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.936+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133799187}) 2024-09-18T12:44:37.938 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.936+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.938 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.936+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521553014}) 2024-09-18T12:44:37.938 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.936+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=695713214}) 2024-09-18T12:44:37.939 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.939 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547439273}) 2024-09-18T12:44:37.939 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502487372}) 2024-09-18T12:44:37.939 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.939 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547439273}) 2024-09-18T12:44:37.939 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171220325}) 2024-09-18T12:44:37.939 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.940 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488564993}) 2024-09-18T12:44:37.940 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=258022917}) 2024-09-18T12:44:37.940 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.940 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488564993}) 2024-09-18T12:44:37.940 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.937+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982142728}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.938+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.938+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565002959}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.938+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513523692}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.938+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.938+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565002959}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.938+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2041797848}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207314654}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948796009}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207314654}) 2024-09-18T12:44:37.942 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373785224}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49307382}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4175630442}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49307382}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.939+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988542155}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932578342}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=526727181}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932578342}) 2024-09-18T12:44:37.943 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883339762}) 2024-09-18T12:44:37.946 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.946 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327051125}) 2024-09-18T12:44:37.946 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190950930}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.940+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327051125}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.941+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1391847966}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.941+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.941+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936154576}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.941+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489079923}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.941+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.941+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936154576}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.941+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635065810}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590815446}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273137839}) 2024-09-18T12:44:37.947 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590815446}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458204633}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209528030}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2964531631}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.942+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209528030}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.943+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993696209}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.943+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.943+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416227019}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.943+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773911003}) 2024-09-18T12:44:37.948 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.943+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.943+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416227019}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.943+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3759046624}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392392039}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3377408116}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392392039}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3463121005}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3368340406}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1392967310}) 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.949 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3368340406}) 2024-09-18T12:44:37.950 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.944+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666086933}) 2024-09-18T12:44:37.952 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.945+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.952 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.945+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281924224}) 2024-09-18T12:44:37.952 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.945+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501914865}) 2024-09-18T12:44:37.952 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.945+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.945+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281924224}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.945+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1719998709}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648379028}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=424538627}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648379028}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482752942}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.946+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684251058}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148455041}) 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.953 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684251058}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902991576}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386181329}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263715609}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386181329}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.947+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4231734273}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.948+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.948+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837156925}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.948+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3154035974}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.948+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.948+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837156925}) 2024-09-18T12:44:37.954 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.948+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428459240}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.949+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.949+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93179750}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.949+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012622717}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.949+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.949+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93179750}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.949+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477957717}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.950+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.950+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565067420}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.950+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043186225}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.950+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.950+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565067420}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.950+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464973193}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1837362982}) 2024-09-18T12:44:37.955 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917743584}) 2024-09-18T12:44:37.956 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.956 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1837362982}) 2024-09-18T12:44:37.956 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2654833274}) 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743949480}) 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603254128}) 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743949480}) 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.951+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=803842274}) 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.959 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472441212}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673094740}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472441212}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787630000}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626945920}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002989772}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626945920}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.952+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2361378582}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.953+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.953+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788449847}) 2024-09-18T12:44:37.960 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.953+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962236746}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.953+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.953+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788449847}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.953+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828371004}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.954+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.954+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1064178516}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.954+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055141023}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.954+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.954+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1064178516}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.954+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1037278179}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763881762}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2852264622}) 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.961 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763881762}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668728786}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738294669}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018364631}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738294669}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.955+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3222556872}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.956+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.956+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013565800}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.956+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2648238025}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.956+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.956+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013565800}) 2024-09-18T12:44:37.962 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.956+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1331581684}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821859232}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2434754262}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821859232}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=893077912}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132163726}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2411429875}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132163726}) 2024-09-18T12:44:37.963 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.957+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=284615013}) 2024-09-18T12:44:37.966 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185312531}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1288742255}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185312531}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3880504745}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679588999}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1544906098}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679588999}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.958+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179698088}) 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.959+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.967 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.959+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906057369}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.959+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=911651155}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.959+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.959+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2906057369}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.959+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575973440}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4026104772}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2508624385}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4026104772}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813467703}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233999094}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4239466189}) 2024-09-18T12:44:37.968 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233999094}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.960+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3167001638}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.961+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.961+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882670783}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.961+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534242952}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.961+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.961+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882670783}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.961+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284830308}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741917513}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268214693}) 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.969 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741917513}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2313824369}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832802284}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3118891420}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832802284}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.962+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3270821598}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.963+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.963+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745372029}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.963+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501610071}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.963+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.963+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745372029}) 2024-09-18T12:44:37.970 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.963+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213129936}) 2024-09-18T12:44:37.972 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.970+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.972 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.970+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273342728}) 2024-09-18T12:44:37.972 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.970+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1130107593}) 2024-09-18T12:44:37.972 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.970+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.972 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.970+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273342728}) 2024-09-18T12:44:37.972 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.970+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067349941}) 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575530147}) 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619849713}) 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575530147}) 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886424646}) 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.974 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235982244}) 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.971+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1543094785}) 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235982244}) 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179134369}) 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837139139}) 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812318240}) 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837139139}) 2024-09-18T12:44:37.975 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.972+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700867929}) 2024-09-18T12:44:37.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831333121}) 2024-09-18T12:44:37.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428639293}) 2024-09-18T12:44:37.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.979 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831333121}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172362391}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914708710}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=172319997}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914708710}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.973+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164075437}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.974+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.974+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023924026}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.974+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4262533573}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.974+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.974+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023924026}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.974+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3313635772}) 2024-09-18T12:44:37.980 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2357977905}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401926683}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2357977905}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304599471}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259396784}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2737608985}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259396784}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.975+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420738424}) 2024-09-18T12:44:37.981 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446265607}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1822761855}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446265607}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122091268}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.976+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341832023}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1153764963}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341832023}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946514344}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979457544}) 2024-09-18T12:44:37.982 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3440653752}) 2024-09-18T12:44:37.983 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.983 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979457544}) 2024-09-18T12:44:37.983 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.977+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3220810462}) 2024-09-18T12:44:37.989 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.978+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.989 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.978+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127117368}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.978+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4041442880}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.978+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.978+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127117368}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.978+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188561807}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264156551}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4254133324}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264156551}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2009512603}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715843173}) 2024-09-18T12:44:37.990 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3824049440}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715843173}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.979+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3325307698}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.980+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.980+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230219860}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.980+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445371827}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.980+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.980+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230219860}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.980+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258726929}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.981+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.981+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236345201}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.981+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083039539}) 2024-09-18T12:44:37.991 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.981+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.981+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236345201}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.981+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2623910365}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.981+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026178088}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2225632367}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026178088}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3751056616}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272474507}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935514434}) 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.992 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2272474507}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.982+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2589557232}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165703656}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1949932184}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165703656}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3927280538}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355848882}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3080267233}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355848882}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3452805238}) 2024-09-18T12:44:37.993 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.984+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.984+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3198174784}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.984+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519958838}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.984+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.984+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3198174784}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.984+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403517089}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99225523}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=252684927}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99225523}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3891051389}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705819694}) 2024-09-18T12:44:37.994 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025942809}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705819694}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.985+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2153694454}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.986+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.986+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229851511}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.986+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=884732235}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.986+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.986+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229851511}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.986+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4294465360}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551183067}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800953457}) 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.995 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551183067}) 2024-09-18T12:44:37.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247390687}) 2024-09-18T12:44:37.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558637318}) 2024-09-18T12:44:37.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872628192}) 2024-09-18T12:44:37.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:37.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558637318}) 2024-09-18T12:44:37.996 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.987+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=928905530}) 2024-09-18T12:44:38.010 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.988+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.010 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.988+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519251518}) 2024-09-18T12:44:38.010 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.988+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4159995446}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.988+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.988+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519251518}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.988+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1588473017}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960981969}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2156198717}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960981969}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437707385}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031131492}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837610160}) 2024-09-18T12:44:38.011 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031131492}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.989+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151086585}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.990+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.990+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728482976}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.990+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2049230569}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.990+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.990+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728482976}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.990+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661191276}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.991+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.991+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463056594}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.991+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255823625}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.991+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.991+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463056594}) 2024-09-18T12:44:38.012 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.991+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619172569}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523328982}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032742961}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523328982}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3208300348}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547069156}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1957641356}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547069156}) 2024-09-18T12:44:38.013 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.992+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3540153382}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.993+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.993+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634222009}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.993+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2958117816}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.993+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.993+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634222009}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.993+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3987505197}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.994+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.994+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126102225}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.994+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047798531}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.994+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.994+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126102225}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.994+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128002691}) 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.014 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241311461}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819795130}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241311461}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1394408877}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368100816}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762109737}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368100816}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.995+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1997613524}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.996+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.996+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948016475}) 2024-09-18T12:44:38.015 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.996+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290352526}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.996+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.996+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948016475}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.996+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2058417047}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.997+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.997+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775654954}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.997+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=159377423}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.997+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.997+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775654954}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.997+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1059693705}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242120741}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2409002956}) 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.016 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242120741}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3111252404}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163428561}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231732327}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163428561}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.998+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641329584}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.999+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.999+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260916801}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.999+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3871209323}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.999+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.999+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260916801}) 2024-09-18T12:44:38.017 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:37.999+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334323691}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.000+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.000+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207125798}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.000+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013225882}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.000+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.000+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207125798}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.000+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983442505}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.001+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.001+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093718993}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.001+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=934327931}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.001+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.001+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093718993}) 2024-09-18T12:44:38.018 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.001+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587606861}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912295161}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4233960070}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912295161}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3359174891}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806845365}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309221383}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806845365}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.002+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088833948}) 2024-09-18T12:44:38.019 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.003+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.003+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996351003}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.003+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593434874}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.003+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.003+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996351003}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.003+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526422686}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372845149}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826093797}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372845149}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.004+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021981200}) 2024-09-18T12:44:38.020 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430180337}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486164973}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430180337}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785752991}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791455405}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1267494703}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791455405}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.005+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1879166762}) 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.006+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.021 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.006+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482922690}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.006+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1625326852}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.006+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.006+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482922690}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.006+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1665917113}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.007+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.007+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799814401}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.007+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1141262087}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.007+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.007+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799814401}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.007+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830260840}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.008+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.008+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=3829203804,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2328392899,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=925889128}) 2024-09-18T12:44:38.022 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.008+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=21} crc {logm=2180093082,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=564524427,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=243587052}) 2024-09-18T12:44:38.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.008+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.008+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=22} crc {logm=3829203804,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=2328392899,mon_config_key=3655569570,monmap=148614106,osd_metadata=2440437218,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=925889128}) 2024-09-18T12:44:38.023 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.008+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=2,osdmap=23} crc {logm=635061474,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3099316058,mon_config_key=3655569570,monmap=148614106,osd_metadata=3354195179,osd_snap=2033388227,osdmap=372002093}) 2024-09-18T12:44:38.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.013+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.013+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=264663931}) 2024-09-18T12:44:38.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.013+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=1900868328}) 2024-09-18T12:44:38.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.014+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:44:38.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.014+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=264663931}) 2024-09-18T12:44:38.024 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:44:38.014+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=4203821145}) 2024-09-18T12:44:38.563 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:44:38.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.657+0000 7ffaf728b640 10 monclient: get_monmap_and_config 2024-09-18T12:44:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.657+0000 7ffaf728b640 10 monclient: build_initial_monmap 2024-09-18T12:44:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.657+0000 7ffaf728b640 10 monclient: monmap: 2024-09-18T12:44:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:38.659397+0000 2024-09-18T12:44:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:38.659397+0000 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:38.660 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 1 Processor -- start 2024-09-18T12:44:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 1 -- start start 2024-09-18T12:44:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 10 monclient: init 2024-09-18T12:44:38.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffaf0097570 0x7ffaf012e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf0097570 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffaf009a8e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.658+0000 7ffaf728b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ffaf0133010 0x7ffaf01353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf0133010 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf012dda0 con 0x7ffaf009a8e0 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf0097940 con 0x7ffaf0097570 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf0098b10 con 0x7ffaf0133010 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf728b640 10 monclient(hunting): authenticate will time out at 1905.981047s 2024-09-18T12:44:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffaf0097570 0x7ffaf012e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf4826640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffaf009a8e0 0x7ffaf009ad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51850/0 (socket says 172.21.15.116:51850) 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffaf0097570 0x7ffaf012e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58694/0 (socket says 172.21.15.116:58694) 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf4826640 1 -- 172.21.15.116:0/648755533 learned_addr learned my addr 172.21.15.116:0/648755533 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): get_auth_request con 0x7ffaf0097570 auth_method 0 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient(hunting): handle_auth_done global_id 194535 payload 274 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 1 -- 172.21.15.116:0/648755533 >> v2:172.21.15.144:3300/0 conn(0x7ffaf0133010 msgr2=0x7ffaf01353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5828640 1 --2- 172.21.15.116:0/648755533 >> v2:172.21.15.144:3300/0 conn(0x7ffaf0133010 0x7ffaf01353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 1 --2- 172.21.15.116:0/648755533 >> v2:172.21.15.144:3300/0 conn(0x7ffaf0133010 0x7ffaf01353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 1 -- 172.21.15.116:0/648755533 >> v1:172.21.15.116:6789/0 conn(0x7ffaf009a8e0 legacy=0x7ffaf009ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient: _finish_hunting 0 2024-09-18T12:44:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 1 monclient: found mon.noname-a 2024-09-18T12:44:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.659+0000 7ffaf5027640 1 -- 172.21.15.116:0/648755533 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf0098df0 con 0x7ffaf0097570 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf5027640 1 -- 172.21.15.116:0/648755533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0097570 msgr2=0x7ffaf012e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf5027640 1 --2- 172.21.15.116:0/648755533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0097570 0x7ffaf012e190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffaf009d8c0 tx=0x7ffaec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75eaa3a8a7944967 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 1 -- 172.21.15.116:0/648755533 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec032440 con 0x7ffaf0097570 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: dump: 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: _finish_auth 0 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 5 monclient: authenticate success, global_id 194535 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 1 -- 172.21.15.116:0/648755533 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec0325a0 con 0x7ffaf0097570 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 4 set_mon_vals no callback set 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 1 -- 172.21.15.116:0/648755533 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec032940 con 0x7ffaf0097570 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffae7fff640 10 monclient: dump: 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 10 monclient: shutdown 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 1 -- 172.21.15.116:0/648755533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0097570 msgr2=0x7ffaf012e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 1 --2- 172.21.15.116:0/648755533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0097570 0x7ffaf012e190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffaf009d8c0 tx=0x7ffaec0090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 1 -- 172.21.15.116:0/648755533 shutdown_connections 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 1 --2- 172.21.15.116:0/648755533 >> v2:172.21.15.144:3300/0 conn(0x7ffaf0133010 0x7ffaf01353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 1 --2- 172.21.15.116:0/648755533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0097570 0x7ffaf012e190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.662+0000 7ffaf728b640 1 -- 172.21.15.116:0/648755533 >> 172.21.15.116:0/648755533 conn(0x7ffaf00672b0 msgr2=0x7ffaf00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.663+0000 7ffaf728b640 1 -- 172.21.15.116:0/648755533 shutdown_connections 2024-09-18T12:44:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.663+0000 7ffaf728b640 1 -- 172.21.15.116:0/648755533 wait complete. 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.663+0000 7ffaf728b640 10 monclient: build_initial_monmap 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.663+0000 7ffaf728b640 10 monclient: monmap: 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.663+0000 7ffaf728b640 1 Processor -- start 2024-09-18T12:44:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.663+0000 7ffaf728b640 1 -- start start 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.663+0000 7ffaf728b640 10 monclient: init 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffaf0097570 0x7ffaf006f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): picked mon.noname-b con 0x7ffaf0097570 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffaf009a8e0 0x7ffaf01313d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf009a8e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 0x7ffaf0131910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf0133010 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf0098300 con 0x7ffaf0133010 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf009e300 con 0x7ffaf0097570 2024-09-18T12:44:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf006bfe0 con 0x7ffaf009a8e0 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf728b640 10 monclient(hunting): authenticate will time out at 1905.986484s 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 0x7ffaf0131910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffaf009a8e0 0x7ffaf01313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 0x7ffaf0131910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58696/0 (socket says 172.21.15.116:58696) 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 -- 172.21.15.116:0/3933731148 learned_addr learned my addr 172.21.15.116:0/3933731148 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): get_auth_request con 0x7ffaf0133010 auth_method 0 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 10 monclient(hunting): get_auth_request con 0x7ffaf009a8e0 auth_method 0 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5027640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffaf0097570 0x7ffaf006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5027640 10 monclient(hunting): get_auth_request con 0x7ffaf0097570 auth_method 0 2024-09-18T12:44:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5027640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5027640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5027640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient(hunting): handle_auth_done global_id 194538 payload 824 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 -- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffaf009a8e0 msgr2=0x7ffaf01313d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffaf009a8e0 0x7ffaf01313d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 -- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffaf0097570 msgr2=0x7ffaf006f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffaf0097570 0x7ffaf006f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient: _finish_hunting 0 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 monclient: found mon.noname-a 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5828640 1 -- 172.21.15.116:0/3933731148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec009390 con 0x7ffaf0133010 2024-09-18T12:44:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf5027640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffaf0097570 0x7ffaf006f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.664+0000 7ffaf4826640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffaf009a8e0 0x7ffaf01313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf5828640 1 -- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 msgr2=0x7ffaf0131910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf5828640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 0x7ffaf0131910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffae800cf00 tx=0x7ffae8007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 1 -- 172.21.15.116:0/3933731148 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae800e610 con 0x7ffaf0133010 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: dump: 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: _finish_auth 0 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf728b640 5 monclient: authenticate success, global_id 194538 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 1 -- 172.21.15.116:0/3933731148 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae800e790 con 0x7ffaf0133010 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf728b640 10 monclient: _renew_subs 2024-09-18T12:44:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 1 -- 172.21.15.116:0/3933731148 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae800eb50 con 0x7ffaf0133010 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf728b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf006b320 con 0x7ffaf0133010 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae57fa640 10 monclient: dump: 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:38.682 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffae77fe640 4 set_mon_vals no callback set 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf728b640 10 monclient: _renew_subs 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf728b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.667+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf006e7e0 con 0x7ffaf0133010 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffae57fa640 1 -- 172.21.15.116:0/3933731148 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7ffae8003020 con 0x7ffaf0133010 2024-09-18T12:44:38.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffaf728b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffaf728b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffaf728b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf004bf60 con 0x7ffaf0133010 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffae57fa640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffad40384e0 0x7ffad403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffae57fa640 1 -- 172.21.15.116:0/3933731148 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7ffae8012070 con 0x7ffaf0133010 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffaf5027640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffad40384e0 0x7ffad403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.668+0000 7ffaf5027640 10 monclient: get_auth_request con 0x7ffad40384e0 auth_method 0 2024-09-18T12:44:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.671+0000 7ffaf5027640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffad40384e0 0x7ffad403a990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ffaf006f930 tx=0x7ffaec032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.672+0000 7ffae57fa640 1 -- 172.21.15.116:0/3933731148 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7ffae807f030 con 0x7ffaf0133010 2024-09-18T12:44:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.672+0000 7ffae57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.672+0000 7ffae57fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:38.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.886+0000 7ffaf728b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.886+0000 7ffaf728b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.886+0000 7ffaf728b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.886+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffaf006b730 con 0x7ffaf0133010 2024-09-18T12:44:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.888+0000 7ffae57fa640 1 -- 172.21.15.116:0/3933731148 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7ffae8014b10 con 0x7ffaf0133010 2024-09-18T12:44:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.888+0000 7ffae57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.888+0000 7ffae57fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:38.889 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffad40384e0 msgr2=0x7ffad403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffad40384e0 0x7ffad403a990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ffaf006f930 tx=0x7ffaec032000 comp rx=0 tx=0).stop 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 10 monclient: shutdown 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 msgr2=0x7ffaf0131910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 0x7ffaf0131910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffae800cf00 tx=0x7ffae8007860 comp rx=0 tx=0).stop 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 shutdown_connections 2024-09-18T12:44:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7ffad40384e0 0x7ffad403a990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaf0133010 0x7ffaf0131910 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ffaf009a8e0 0x7ffaf01313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 --2- 172.21.15.116:0/3933731148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffaf0097570 0x7ffaf006f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 >> 172.21.15.116:0/3933731148 conn(0x7ffaf00672b0 msgr2=0x7ffaf012d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 shutdown_connections 2024-09-18T12:44:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.890+0000 7ffaf728b640 1 -- 172.21.15.116:0/3933731148 wait complete. 2024-09-18T12:44:38.901 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:44:38.901 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T12:44:38.901 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:44:38.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.994+0000 7f416d8e8640 10 monclient: get_monmap_and_config 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.994+0000 7f416d8e8640 10 monclient: build_initial_monmap 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.994+0000 7f416d8e8640 10 monclient: monmap: 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:38.994972+0000 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:38.994972+0000 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:38.996 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.994+0000 7f416d8e8640 1 Processor -- start 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.994+0000 7f416d8e8640 1 -- start start 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.994+0000 7f416d8e8640 10 monclient: init 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41680981c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 0x7f4168136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4168131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4168132170 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416809ca60 con 0x7f4168132170 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416809cca0 con 0x7f41680981c0 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4168131bd0 con 0x7f4168131da0 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f416d8e8640 10 monclient(hunting): authenticate will time out at 1906.316507s 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 0x7f4168136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41677fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4168132170 0x7f41681390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51858/0 (socket says 172.21.15.116:51858) 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41677fe640 1 -- 172.21.15.116:0/1816235224 learned_addr learned my addr 172.21.15.116:0/1816235224 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 0x7f4168136b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35976/0 (socket says 172.21.15.116:35976) 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): get_auth_request con 0x7f4168131da0 auth_method 0 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient(hunting): handle_auth_done global_id 224219 payload 274 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 1 -- 172.21.15.116:0/1816235224 >> v1:172.21.15.144:6789/0 conn(0x7f41680981c0 legacy=0x7f4168098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 1 -- 172.21.15.116:0/1816235224 >> v1:172.21.15.116:6789/0 conn(0x7f4168132170 legacy=0x7f41681390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 1 monclient: found mon.noname-b 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.995+0000 7f41667fc640 1 -- 172.21.15.116:0/1816235224 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416809e010 con 0x7f4168131da0 2024-09-18T12:44:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f41667fc640 1 -- 172.21.15.116:0/1816235224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 msgr2=0x7f4168136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f41667fc640 1 --2- 172.21.15.116:0/1816235224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 0x7f4168136b70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4150009880 tx=0x7f41500090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=436d35c3e015e0c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 1 -- 172.21.15.116:0/1816235224 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415002aea0 con 0x7f4168131da0 2024-09-18T12:44:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: dump: 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 1 -- 172.21.15.116:0/1816235224 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4150032440 con 0x7f4168131da0 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f416d8e8640 5 monclient: authenticate success, global_id 224219 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f416d8e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f416d8e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f416d8e8640 4 set_mon_vals no callback set 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 1 -- 172.21.15.116:0/1816235224 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41500327e0 con 0x7f4168131da0 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.998+0000 7f4165ffb640 10 monclient: dump: 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 10 monclient: shutdown 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 1 -- 172.21.15.116:0/1816235224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 msgr2=0x7f4168136b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 1 --2- 172.21.15.116:0/1816235224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 0x7f4168136b70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4150009880 tx=0x7f41500090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 1 -- 172.21.15.116:0/1816235224 shutdown_connections 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 1 --2- 172.21.15.116:0/1816235224 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4168131da0 0x7f4168136b70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 1 -- 172.21.15.116:0/1816235224 >> 172.21.15.116:0/1816235224 conn(0x7f41680a1e30 msgr2=0x7f41680a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 1 -- 172.21.15.116:0/1816235224 shutdown_connections 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:38.999+0000 7f416d8e8640 1 -- 172.21.15.116:0/1816235224 wait complete. 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient: build_initial_monmap 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient: monmap: 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 Processor -- start 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 -- start start 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient: init 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41680981c0 0x7f4168118490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): picked mon.noname-c con 0x7f41680981c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168131da0 0x7f4168138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): picked mon.noname-a con 0x7f4168131da0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4168132170 0x7f4168138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4168132170 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4168139e60 con 0x7f4168131da0 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4168098cb0 con 0x7f4168132170 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41680a0940 con 0x7f41680981c0 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.000+0000 7f416d8e8640 10 monclient(hunting): authenticate will time out at 1906.321983s 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41680981c0 0x7f4168118490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41680981c0 0x7f4168118490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35992/0 (socket says 172.21.15.116:35992) 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168131da0 0x7f4168138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 -- 172.21.15.116:0/597877741 learned_addr learned my addr 172.21.15.116:0/597877741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): get_auth_request con 0x7f41680981c0 auth_method 0 2024-09-18T12:44:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 10 monclient(hunting): get_auth_request con 0x7f4168131da0 auth_method 0 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient(hunting): handle_auth_done global_id 224222 payload 824 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41677fe640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4168132170 0x7f4168138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4168132170 msgr2=0x7f4168138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4168132170 0x7f4168138780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168131da0 msgr2=0x7f4168138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168131da0 0x7f4168138240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 monclient: found mon.noname-c 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f4166ffd640 1 -- 172.21.15.116:0/597877741 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4150009390 con 0x7f41680981c0 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41677fe640 10 monclient: get_auth_request con 0x7f4168132170 auth_method 0 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41677fe640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4168132170 0x7f4168138780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:44:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.001+0000 7f41667fc640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168131da0 0x7f4168138240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f4166ffd640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41680981c0 msgr2=0x7f4168118490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f4166ffd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41680981c0 0x7f4168118490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f416000c610 tx=0x7f4160007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160012840 con 0x7f41680981c0 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: dump: 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f416d8e8640 5 monclient: authenticate success, global_id 224222 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41600129e0 con 0x7f41680981c0 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f416d8e8640 10 monclient: _renew_subs 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f416d8e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41681195a0 con 0x7f41680981c0 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160012da0 con 0x7f41680981c0 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f415effd640 10 monclient: dump: 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f41657fa640 4 set_mon_vals no callback set 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f416d8e8640 10 monclient: _renew_subs 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f416d8e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.003+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4168131670 con 0x7f41680981c0 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f416d8e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f416d8e8640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 0x7f4168134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f416d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4168065f90 con 0x7f4168138cc0 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4160011590 con 0x7f41680981c0 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 0x7f4168134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient: get_auth_request con 0x7f4168138cc0 auth_method 0 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f415effd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4148038490 0x7f414803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 10 monclient(hunting): handle_auth_done global_id 194544 payload 824 2024-09-18T12:44:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f416004bd60 con 0x7f41680981c0 2024-09-18T12:44:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.004+0000 7f41667fc640 1 -- 172.21.15.116:0/597877741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4168119b90 con 0x7f4168138cc0 2024-09-18T12:44:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.005+0000 7f41677fe640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4148038490 0x7f414803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.005+0000 7f41677fe640 10 monclient: get_auth_request con 0x7f4148038490 auth_method 0 2024-09-18T12:44:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f41667fc640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 msgr2=0x7f4168134650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f41667fc640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 0x7f4168134650 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4150034000 tx=0x7f4150032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415002aea0 con 0x7f4168138cc0 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 10 monclient: dump: 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.022 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4150040070 con 0x7f4168138cc0 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 msgr2=0x7f4168134650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 0x7f4168134650 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4150034000 tx=0x7f4150032000 comp rx=0 tx=0).stop 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 msgr2=0x7f4168134650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f415effd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168138cc0 0x7f4168134650 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.007+0000 7f41677fe640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4148038490 0x7f414803a940 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4158005b90 tx=0x7f4158009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f416d8e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f416d8e8640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 0x7f4168134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f416d8e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4168119d00 con 0x7f4168134b90 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 0x7f4168134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient: get_auth_request con 0x7f4168134b90 auth_method 0 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 10 monclient(hunting): handle_auth_done global_id 194547 payload 824 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.013+0000 7f41667fc640 1 -- 172.21.15.116:0/597877741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f41681318f0 con 0x7f4168134b90 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.015+0000 7f41667fc640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 msgr2=0x7f4168134f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.015+0000 7f41667fc640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 0x7f4168134f90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4150009880 tx=0x7f415003e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150053420 con 0x7f4168134b90 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 10 monclient: dump: 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.025 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f4150032c50 con 0x7f4168134b90 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 msgr2=0x7f4168134f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 0x7f4168134f90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4150009880 tx=0x7f415003e070 comp rx=0 tx=0).stop 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 msgr2=0x7f4168134f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f415effd640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4168134b90 0x7f4168134f90 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4150009880 tx=0x7f415003e070 comp rx=0 tx=0).stop 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:39.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 188, 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:39.027 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.028 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:39.029 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:39.030 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.033 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.034 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:39.035 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.037 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:39.038 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4148038490 msgr2=0x7f414803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f4148038490 0x7f414803a940 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4158005b90 tx=0x7f4158009040 comp rx=0 tx=0).stop 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 10 monclient: shutdown 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41680981c0 msgr2=0x7f4168118490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 --2- 172.21.15.116:0/597877741 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f41680981c0 0x7f4168118490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f416000c610 tx=0x7f4160007ed0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f41677fe640 1 -- 172.21.15.116:0/597877741 reap_dead start 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f41677fe640 1 -- 172.21.15.116:0/597877741 reap_dead start 2024-09-18T12:44:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 shutdown_connections 2024-09-18T12:44:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 >> 172.21.15.116:0/597877741 conn(0x7f41680a1e30 msgr2=0x7f4168096f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 shutdown_connections 2024-09-18T12:44:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.016+0000 7f416d8e8640 1 -- 172.21.15.116:0/597877741 wait complete. 2024-09-18T12:44:39.041 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:44:39.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:44:39.041 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:44:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.133+0000 7fecc8094640 10 monclient: get_monmap_and_config 2024-09-18T12:44:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.133+0000 7fecc8094640 10 monclient: build_initial_monmap 2024-09-18T12:44:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.133+0000 7fecc8094640 10 monclient: monmap: 2024-09-18T12:44:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:39.134315+0000 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:39.134315+0000 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:39.135 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 1 Processor -- start 2024-09-18T12:44:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 1 -- start start 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient: init 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecc0096c60 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fecc0097030 0x7fecc0097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient(hunting): picked mon.noname-c con 0x7fecc0097030 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc0135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient(hunting): picked mon.noname-b con 0x7fecc0133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc009d2e0 con 0x7fecc0096c60 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc009d6b0 con 0x7fecc0097030 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc00992f0 con 0x7fecc0133b40 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.134+0000 7fecc8094640 10 monclient(hunting): authenticate will time out at 1906.455915s 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc0135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc0135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:35998/0 (socket says 172.21.15.116:35998) 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 -- 172.21.15.116:0/3313487738 learned_addr learned my addr 172.21.15.116:0/3313487738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): get_auth_request con 0x7fecc0133b40 auth_method 0 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc562f640 1 --2- 172.21.15.116:0/3313487738 >> v2:172.21.15.144:3300/0 conn(0x7fecc0097030 0x7fecc0097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient(hunting): handle_auth_done global_id 224225 payload 274 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 -- 172.21.15.116:0/3313487738 >> v2:172.21.15.144:3300/0 conn(0x7fecc0097030 msgr2=0x7fecc0097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 --2- 172.21.15.116:0/3313487738 >> v2:172.21.15.144:3300/0 conn(0x7fecc0097030 0x7fecc0097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 -- 172.21.15.116:0/3313487738 >> v1:172.21.15.144:6789/0 conn(0x7fecc0096c60 legacy=0x7fecc009a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 monclient: found mon.noname-b 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc6631640 1 -- 172.21.15.116:0/3313487738 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc00995d0 con 0x7fecc0133b40 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc562f640 10 monclient: get_auth_request con 0x7fecc0097030 auth_method 0 2024-09-18T12:44:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.135+0000 7fecc562f640 1 --2- 172.21.15.116:0/3313487738 >> v2:172.21.15.144:3300/0 conn(0x7fecc0097030 0x7fecc0097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:44:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc6631640 1 -- 172.21.15.116:0/3313487738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 msgr2=0x7fecc0135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc6631640 1 --2- 172.21.15.116:0/3313487738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc0135f20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fecb800d1a0 tx=0x7fecb800ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8701c130ec9970d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 1 -- 172.21.15.116:0/3313487738 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb800b3c0 con 0x7fecc0133b40 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: dump: 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 1 -- 172.21.15.116:0/3313487738 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb8009610 con 0x7fecc0133b40 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 5 monclient: authenticate success, global_id 224225 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 1 -- 172.21.15.116:0/3313487738 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb80099f0 con 0x7fecc0133b40 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 4 set_mon_vals no callback set 2024-09-18T12:44:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc4e2e640 10 monclient: dump: 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 10 monclient: shutdown 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 -- 172.21.15.116:0/3313487738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 msgr2=0x7fecc0135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 --2- 172.21.15.116:0/3313487738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc0135f20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fecb800d1a0 tx=0x7fecb800ba50 comp rx=0 tx=0).stop 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 -- 172.21.15.116:0/3313487738 shutdown_connections 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 --2- 172.21.15.116:0/3313487738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc0135f20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 --2- 172.21.15.116:0/3313487738 >> v2:172.21.15.144:3300/0 conn(0x7fecc0097030 0x7fecc0097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 -- 172.21.15.116:0/3313487738 >> 172.21.15.116:0/3313487738 conn(0x7fecc00672b0 msgr2=0x7fecc00676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 -- 172.21.15.116:0/3313487738 shutdown_connections 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.138+0000 7fecc8094640 1 -- 172.21.15.116:0/3313487738 wait complete. 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.139+0000 7fecc8094640 10 monclient: build_initial_monmap 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.139+0000 7fecc8094640 10 monclient: monmap: 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.139+0000 7fecc8094640 1 Processor -- start 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.139+0000 7fecc8094640 1 -- start start 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.139+0000 7fecc8094640 10 monclient: init 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0096c60 0x7fecc012e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): picked mon.noname-a con 0x7fecc0096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecc0097030 0x7fecc013d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): picked mon.noname-b con 0x7fecc0097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc013da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): picked mon.noname-c con 0x7fecc0133b40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0119c30 con 0x7fecc0096c60 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc006f750 con 0x7fecc0097030 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0119950 con 0x7fecc0133b40 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc8094640 10 monclient(hunting): authenticate will time out at 1906.461127s 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc013da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc013da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36000/0 (socket says 172.21.15.116:36000) 2024-09-18T12:44:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 -- 172.21.15.116:0/1227382657 learned_addr learned my addr 172.21.15.116:0/1227382657 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): get_auth_request con 0x7fecc0133b40 auth_method 0 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0096c60 0x7fecc012e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 10 monclient(hunting): get_auth_request con 0x7fecc0096c60 auth_method 0 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecc0097030 0x7fecc013d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 10 monclient(hunting): get_auth_request con 0x7fecc0097030 auth_method 0 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc562f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient(hunting): handle_auth_done global_id 224228 payload 824 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecc0097030 msgr2=0x7fecc013d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fecc0097030 0x7fecc013d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0096c60 msgr2=0x7fecc012e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0096c60 0x7fecc012e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 monclient: found mon.noname-c 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc6631640 1 -- 172.21.15.116:0/1227382657 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb800bcf0 con 0x7fecc0133b40 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.140+0000 7fecc5e30640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecc0096c60 0x7fecc012e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.142+0000 7fecc6631640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 msgr2=0x7fecc013da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.142+0000 7fecc6631640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc013da60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fecb800bcc0 tx=0x7fecb8039000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.142+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8034990 con 0x7fecc0133b40 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.142+0000 7fecb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.142+0000 7fecb5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.142+0000 7fecb5ffb640 10 monclient: dump: 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecc8094640 5 monclient: authenticate success, global_id 224228 2024-09-18T12:44:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb8034b10 con 0x7fecc0133b40 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecc8094640 10 monclient: _renew_subs 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecc8094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc0119380 con 0x7fecc0133b40 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb803f420 con 0x7fecc0133b40 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb5ffb640 10 monclient: dump: 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecc8094640 10 monclient: _renew_subs 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecb7fff640 4 set_mon_vals no callback set 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecc8094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.143+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc006f000 con 0x7fecc0133b40 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fecb8032070 con 0x7fecc0133b40 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc8094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc8094640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 0x7fecc00298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc8094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0065f20 con 0x7fecc0117c40 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 0x7fecc00298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecb5ffb640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feca4038490 0x7feca403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient: get_auth_request con 0x7fecc0117c40 auth_method 0 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fecb806f190 con 0x7fecc0133b40 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 10 monclient(hunting): handle_auth_done global_id 224231 payload 824 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc5e30640 1 -- 172.21.15.116:0/1227382657 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecc0119e70 con 0x7fecc0117c40 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc562f640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feca4038490 0x7feca403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.144+0000 7fecc562f640 10 monclient: get_auth_request con 0x7feca4038490 auth_method 0 2024-09-18T12:44:39.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecc5e30640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 msgr2=0x7fecc00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecc5e30640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 0x7fecc00298c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fecb000eb00 tx=0x7fecb0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb000ce80 con 0x7fecc0117c40 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 10 monclient: dump: 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fecb00123f0 con 0x7fecc0117c40 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 msgr2=0x7fecc00298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 0x7fecc00298c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fecb000eb00 tx=0x7fecb0009060 comp rx=0 tx=0).stop 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 msgr2=0x7fecc00298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.146+0000 7fecb5ffb640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0117c40 0x7fecc00298c0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fecb000eb00 tx=0x7fecb0009060 comp rx=0 tx=0).stop 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.147+0000 7fecc562f640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feca4038490 0x7feca403a940 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fecc0097dc0 tx=0x7fecbc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc8094640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc8094640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 0x7fecc0029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc8094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc004bf60 con 0x7fecc0118010 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 0x7fecc0029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 10 monclient: get_auth_request con 0x7fecc0118010 auth_method 0 2024-09-18T12:44:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.152+0000 7fecc5e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.153+0000 7fecc5e30640 10 monclient(hunting): handle_auth_done global_id 224234 payload 824 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.153+0000 7fecc5e30640 1 -- 172.21.15.116:0/1227382657 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fecc0067ba0 con 0x7fecc0118010 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc5e30640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 msgr2=0x7fecc0029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc5e30640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 0x7fecc0029e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fecb000eb00 tx=0x7fecb0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb001cd20 con 0x7fecc0118010 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 10 monclient: dump: 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fecb001f420 con 0x7fecc0118010 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 msgr2=0x7fecc0029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 0x7fecc0029e00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fecb000eb00 tx=0x7fecb0015000 comp rx=0 tx=0).stop 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 msgr2=0x7fecc0029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecb5ffb640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0118010 0x7fecc0029e00 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fecb000eb00 tx=0x7fecb0015000 comp rx=0 tx=0).stop 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 188, 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:44:39.166 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.167 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.168 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:39.169 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.170 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:39.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.174 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.175 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.176 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 3 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 6 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.177 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:39.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.178 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.178 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:39.178 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feca4038490 msgr2=0x7feca403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc8094640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7feca4038490 0x7feca403a940 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fecc0097dc0 tx=0x7fecbc008040 comp rx=0 tx=0).stop 2024-09-18T12:44:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc8094640 10 monclient: shutdown 2024-09-18T12:44:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc8094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 msgr2=0x7fecc013da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc8094640 1 --2- 172.21.15.116:0/1227382657 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fecc0133b40 0x7fecc013da60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fecb800bcc0 tx=0x7fecb8039000 comp rx=0 tx=0).stop 2024-09-18T12:44:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.155+0000 7fecc6631640 1 -- 172.21.15.116:0/1227382657 reap_dead start 2024-09-18T12:44:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.156+0000 7fecc6631640 1 -- 172.21.15.116:0/1227382657 reap_dead start 2024-09-18T12:44:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.156+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 shutdown_connections 2024-09-18T12:44:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.156+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 >> 172.21.15.116:0/1227382657 conn(0x7fecc00672b0 msgr2=0x7fecc0131d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.156+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 shutdown_connections 2024-09-18T12:44:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.156+0000 7fecc8094640 1 -- 172.21.15.116:0/1227382657 wait complete. 2024-09-18T12:44:39.180 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:44:39.180 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:44:39.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.273+0000 7f0799a20640 10 monclient: get_monmap_and_config 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.273+0000 7f0799a20640 10 monclient: build_initial_monmap 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.273+0000 7f0799a20640 10 monclient: monmap: 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:39.273873+0000 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:39.273873+0000 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:39.275 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.273+0000 7f0799a20640 1 Processor -- start 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.273+0000 7f0799a20640 1 -- start start 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.273+0000 7f0799a20640 10 monclient: init 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07941318f0 0x7f0794131f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient(hunting): picked mon.noname-c con 0x7f07941318f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f079409ea50 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 0x7f0794139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient(hunting): picked mon.noname-b con 0x7f0794136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0794131cc0 con 0x7f079409ea50 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079409f530 con 0x7f07941318f0 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079409fbf0 con 0x7f0794136c20 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0799a20640 10 monclient(hunting): authenticate will time out at 1906.595482s 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 0x7f0794139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 0x7f0794139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36028/0 (socket says 172.21.15.116:36028) 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 -- 172.21.15.116:0/987463985 learned_addr learned my addr 172.21.15.116:0/987463985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): get_auth_request con 0x7f0794136c20 auth_method 0 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 1 --2- 172.21.15.116:0/987463985 >> v2:172.21.15.144:3300/0 conn(0x7f07941318f0 0x7f0794131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 10 monclient(hunting): get_auth_request con 0x7f07941318f0 auth_method 0 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f0792ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient(hunting): handle_auth_done global_id 224237 payload 274 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 -- 172.21.15.116:0/987463985 >> v2:172.21.15.144:3300/0 conn(0x7f07941318f0 msgr2=0x7f0794131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 --2- 172.21.15.116:0/987463985 >> v2:172.21.15.144:3300/0 conn(0x7f07941318f0 0x7f0794131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 -- 172.21.15.116:0/987463985 >> v1:172.21.15.144:6789/0 conn(0x7f079409ea50 legacy=0x7f079409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 monclient: found mon.noname-b 2024-09-18T12:44:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.274+0000 7f07937fe640 1 -- 172.21.15.116:0/987463985 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079409fe30 con 0x7f0794136c20 2024-09-18T12:44:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f07937fe640 1 -- 172.21.15.116:0/987463985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 msgr2=0x7f0794139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f07937fe640 1 --2- 172.21.15.116:0/987463985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 0x7f0794139000 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f078c009910 tx=0x7f078c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9077985df58c84e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 1 -- 172.21.15.116:0/987463985 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c02ae90 con 0x7f0794136c20 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: dump: 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0799a20640 5 monclient: authenticate success, global_id 224237 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 1 -- 172.21.15.116:0/987463985 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c032440 con 0x7f0794136c20 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0799a20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0799a20640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0799a20640 4 set_mon_vals no callback set 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 1 -- 172.21.15.116:0/987463985 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c0327e0 con 0x7f0794136c20 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.277+0000 7f0791ffb640 10 monclient: dump: 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 10 monclient: shutdown 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 -- 172.21.15.116:0/987463985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 msgr2=0x7f0794139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 --2- 172.21.15.116:0/987463985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 0x7f0794139000 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f078c009910 tx=0x7f078c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 -- 172.21.15.116:0/987463985 shutdown_connections 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 --2- 172.21.15.116:0/987463985 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0794136c20 0x7f0794139000 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 --2- 172.21.15.116:0/987463985 >> v2:172.21.15.144:3300/0 conn(0x7f07941318f0 0x7f0794131f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 -- 172.21.15.116:0/987463985 >> 172.21.15.116:0/987463985 conn(0x7f0794067110 msgr2=0x7f0794067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 -- 172.21.15.116:0/987463985 shutdown_connections 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.278+0000 7f0799a20640 1 -- 172.21.15.116:0/987463985 wait complete. 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient: build_initial_monmap 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient: monmap: 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 Processor -- start 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 -- start start 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient: init 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f079409ea50 0x7f07941308d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): picked mon.noname-c con 0x7f079409ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07941318f0 0x7f0794130e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): picked mon.noname-a con 0x7f07941318f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0794136c20 0x7f07940996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): picked mon.noname-b con 0x7f0794136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079409b800 con 0x7f07941318f0 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079409bf10 con 0x7f0794136c20 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079409dd40 con 0x7f079409ea50 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.279+0000 7f0799a20640 10 monclient(hunting): authenticate will time out at 1906.600967s 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f079409ea50 0x7f07941308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f079409ea50 0x7f07941308d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:36034/0 (socket says 172.21.15.116:36034) 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 -- 172.21.15.116:0/2690108108 learned_addr learned my addr 172.21.15.116:0/2690108108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): get_auth_request con 0x7f079409ea50 auth_method 0 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07941318f0 0x7f0794130e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 10 monclient(hunting): get_auth_request con 0x7f07941318f0 auth_method 0 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient(hunting): handle_auth_done global_id 224240 payload 824 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0794136c20 msgr2=0x7f07940996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0794136c20 0x7f07940996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07941318f0 msgr2=0x7f0794130e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07941318f0 0x7f0794130e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 monclient: found mon.noname-c 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.280+0000 7f0792ffd640 1 -- 172.21.15.116:0/2690108108 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c009450 con 0x7f079409ea50 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0792ffd640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f079409ea50 msgr2=0x7f07941308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0792ffd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f079409ea50 0x7f07941308d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0784009f20 tx=0x7f078400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784014480 con 0x7f079409ea50 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: dump: 2024-09-18T12:44:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078400f040 con 0x7f079409ea50 2024-09-18T12:44:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0799a20640 5 monclient: authenticate success, global_id 224240 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07840147f0 con 0x7f079409ea50 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0799a20640 10 monclient: _renew_subs 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0799a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0794141eb0 con 0x7f079409ea50 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f077affd640 10 monclient: dump: 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f07917fa640 4 set_mon_vals no callback set 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0799a20640 10 monclient: _renew_subs 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0799a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.282+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079409b280 con 0x7f079409ea50 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f0799a20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f0799a20640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 0x7f07940298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f0799a20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0794065e00 con 0x7f079409a3a0 2024-09-18T12:44:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0784014b60 con 0x7f079409ea50 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f077affd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0780038490 0x7f078003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f078404cdf0 con 0x7f079409ea50 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f07927fc640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 0x7f07940298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f07927fc640 10 monclient: get_auth_request con 0x7f079409a3a0 auth_method 0 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f07927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f07927fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.283+0000 7f07927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.284+0000 7f07937fe640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0780038490 0x7f078003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.284+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.284+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.284+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.284+0000 7f07937fe640 10 monclient: get_auth_request con 0x7f0780038490 auth_method 0 2024-09-18T12:44:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.284+0000 7f07927fc640 10 monclient(hunting): handle_auth_done global_id 234136 payload 824 2024-09-18T12:44:39.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.284+0000 7f07927fc640 1 -- 172.21.15.116:0/2690108108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f079404bf60 con 0x7f079409a3a0 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.286+0000 7f07937fe640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0780038490 0x7f078003a940 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f079406f600 tx=0x7f078c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.286+0000 7f07927fc640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 msgr2=0x7f07940298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.286+0000 7f07927fc640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 0x7f07940298c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0788007fd0 tx=0x7f0788007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.286+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078800d7d0 con 0x7f079409a3a0 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.286+0000 7f077affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.286+0000 7f077affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.286+0000 7f077affd640 10 monclient: dump: 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.287+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f078800d930 con 0x7f079409a3a0 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.287+0000 7f077affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.287+0000 7f077affd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.287+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 msgr2=0x7f07940298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.287+0000 7f077affd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 0x7f07940298c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0788007fd0 tx=0x7f0788007e10 comp rx=0 tx=0).stop 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.287+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 msgr2=0x7f07940298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.287+0000 7f077affd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a3a0 0x7f07940298c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.292+0000 7f0799a20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.292+0000 7f0799a20640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 0x7f0794029e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.292+0000 7f0799a20640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.292+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079409e7e0 con 0x7f079409a770 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 0x7f0794029e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient: get_auth_request con 0x7f079409a770 auth_method 0 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 10 monclient(hunting): handle_auth_done global_id 234139 payload 824 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.293+0000 7f07927fc640 1 -- 172.21.15.116:0/2690108108 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0794135eb0 con 0x7f079409a770 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.295+0000 7f07927fc640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 msgr2=0x7f0794029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.295+0000 7f07927fc640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 0x7f0794029e00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0788007fd0 tx=0x7f0788019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.295+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788014e20 con 0x7f079409a770 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.295+0000 7f077affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.295+0000 7f077affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.295+0000 7f077affd640 10 monclient: dump: 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f078800d5c0 con 0x7f079409a770 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f077affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f077affd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 msgr2=0x7f0794029e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f077affd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 0x7f0794029e00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0788007fd0 tx=0x7f0788019000 comp rx=0 tx=0).stop 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f077affd640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 msgr2=0x7f0794029e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f077affd640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f079409a770 0x7f0794029e00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:44:39.304 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 188, 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 4, 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.305 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.306 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:39.308 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.309 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:39.311 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:39.312 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:39.313 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:39.314 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:39.315 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:39.315 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:39.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:39.315 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:39.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:39.315 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:39.315 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0780038490 msgr2=0x7f078003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0780038490 0x7f078003a940 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f079406f600 tx=0x7f078c032000 comp rx=0 tx=0).stop 2024-09-18T12:44:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 10 monclient: shutdown 2024-09-18T12:44:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f079409ea50 msgr2=0x7f07941308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 --2- 172.21.15.116:0/2690108108 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f079409ea50 0x7f07941308d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0784009f20 tx=0x7f078400b040 comp rx=0 tx=0).stop 2024-09-18T12:44:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f07937fe640 1 -- 172.21.15.116:0/2690108108 reap_dead start 2024-09-18T12:44:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f07937fe640 1 -- 172.21.15.116:0/2690108108 reap_dead start 2024-09-18T12:44:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 shutdown_connections 2024-09-18T12:44:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 >> 172.21.15.116:0/2690108108 conn(0x7f0794067110 msgr2=0x7f07940a3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 shutdown_connections 2024-09-18T12:44:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.296+0000 7f0799a20640 1 -- 172.21.15.116:0/2690108108 wait complete. 2024-09-18T12:44:39.317 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:44:39.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T12:44:39.318 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T12:44:39.318 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T12:44:39.318 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:44:39.318 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:44:39.318 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:44:39.330 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:44:39.330 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T12:44:39.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.460+0000 7fbe98289640 10 monclient: get_monmap_and_config 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.460+0000 7fbe98289640 10 monclient: build_initial_monmap 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.460+0000 7fbe98289640 10 monclient: monmap: 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:39.461307+0000 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:39.461307+0000 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:39.462 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 1 Processor -- start 2024-09-18T12:44:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 1 -- start start 2024-09-18T12:44:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient: init 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 0x7fbe9009af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient(hunting): picked mon.noname-b con 0x7fbe90098a40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe90098e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe90133c00 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90099ea0 con 0x7fbe90133c00 2024-09-18T12:44:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90099bc0 con 0x7fbe90098e10 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe900713a0 con 0x7fbe90098a40 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe98289640 10 monclient(hunting): authenticate will time out at 1906.782871s 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe96025640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 msgr2=0x7fbe9009af70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe96025640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 0x7fbe9009af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.461+0000 7fbe96826640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbe90133c00 0x7fbe90135fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51866/0 (socket says 172.21.15.116:51866) 2024-09-18T12:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.462+0000 7fbe96826640 1 -- 172.21.15.116:0/2167158739 learned_addr learned my addr 172.21.15.116:0/2167158739 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:39.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96826640 1 -- 172.21.15.116:0/2167158739 >> v1:172.21.15.116:6789/0 conn(0x7fbe90133c00 legacy=0x7fbe90135fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177555618 0 0) 0x7fbe90099ea0 con 0x7fbe90133c00 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): my global_id is 194556 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe7c0034e0 con 0x7fbe90133c00 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95824640 1 -- 172.21.15.116:0/2167158739 >> v1:172.21.15.144:6789/0 conn(0x7fbe90098e10 legacy=0x7fbe900991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3862471679 0 0) 0x7fbe7c0034e0 con 0x7fbe90133c00 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 msgr2=0x7fbe9009af70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 0x7fbe9009af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 >> v1:172.21.15.144:6789/0 conn(0x7fbe90098e10 legacy=0x7fbe900991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:44:39.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9009a180 con 0x7fbe90133c00 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fbe88001b20 con 0x7fbe90133c00 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe98289640 5 monclient: authenticate success, global_id 194556 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe98289640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: dump: 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.476 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 >> v1:172.21.15.116:6789/0 conn(0x7fbe90133c00 legacy=0x7fbe90135fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe7c003ad0 0x7fbe7c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): picked mon.a con 0x7fbe7c003ad0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:39.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe7c004780 0x7fbe7c004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): picked mon.b con 0x7fbe7c004780 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 0x7fbe7c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): picked mon.c con 0x7fbe90098a40 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c00bf10 con 0x7fbe7c003ad0 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c00c050 con 0x7fbe7c004780 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c00c190 con 0x7fbe90098a40 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95023640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96826640 1 -- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 msgr2=0x7fbe7c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe98289640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96826640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 0x7fbe7c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:39.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe7c003ad0 0x7fbe7c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): get_auth_request con 0x7fbe7c003ad0 auth_method 0 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95824640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe7c004780 0x7fbe7c004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95824640 10 monclient(hunting): get_auth_request con 0x7fbe7c004780 auth_method 0 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95824640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe95824640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient(hunting): handle_auth_done global_id 194556 payload 293 2024-09-18T12:44:39.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 1 -- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 msgr2=0x7fbe7c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe90098a40 0x7fbe7c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 1 -- 172.21.15.116:0/2167158739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe7c004780 msgr2=0x7fbe7c004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe7c004780 0x7fbe7c004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 1 monclient: found mon.a 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 1 -- 172.21.15.116:0/2167158739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c0034e0 con 0x7fbe7c003ad0 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.464+0000 7fbe96025640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.465+0000 7fbe95824640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe7c004780 0x7fbe7c004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe96025640 1 -- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe7c003ad0 msgr2=0x7fbe7c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe96025640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe7c003ad0 0x7fbe7c003ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0099c0 tx=0x7fbe8c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cccf87e966aec09 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c003b80 con 0x7fbe7c003ad0 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 10 monclient: dump: 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.481 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe98289640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c003d20 con 0x7fbe7c003ad0 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 1 -- 172.21.15.116:0/2167158739 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c02ab80 con 0x7fbe7c003ad0 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe98289640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe98289640 4 set_mon_vals no callback set 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.467+0000 7fbe95023640 10 monclient: dump: 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.482 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 10 monclient: shutdown 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 1 -- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe7c003ad0 msgr2=0x7fbe7c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 1 --2- 172.21.15.116:0/2167158739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe7c003ad0 0x7fbe7c003ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0099c0 tx=0x7fbe8c009060 comp rx=0 tx=0).stop 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe96826640 1 -- 172.21.15.116:0/2167158739 reap_dead start 2024-09-18T12:44:39.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 1 -- 172.21.15.116:0/2167158739 shutdown_connections 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 1 -- 172.21.15.116:0/2167158739 >> 172.21.15.116:0/2167158739 conn(0x7fbe900678a0 msgr2=0x7fbe900a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 1 -- 172.21.15.116:0/2167158739 shutdown_connections 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 1 -- 172.21.15.116:0/2167158739 wait complete. 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 10 monclient: build_initial_monmap 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.468+0000 7fbe98289640 10 monclient: monmap: 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:39.484 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 Processor -- start 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 -- start start 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient: init 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe7c003ad0 0x7fbe90126630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): picked mon.noname-c con 0x7fbe7c003ad0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 0x7fbe90126f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): picked mon.noname-a con 0x7fbe90126b70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe901430a0 0x7fbe90145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): picked mon.noname-b con 0x7fbe901430a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:39.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe96025640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe7c003ad0 msgr2=0x7fbe90126630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9009b8d0 con 0x7fbe90126b70 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90097c10 con 0x7fbe901430a0 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9009c6a0 con 0x7fbe7c003ad0 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe96025640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe7c003ad0 0x7fbe90126630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe98289640 10 monclient(hunting): authenticate will time out at 1906.790732s 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 0x7fbe90126f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 0x7fbe90126f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58768/0 (socket says 172.21.15.116:58768) 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 1 -- 172.21.15.116:0/1398876038 learned_addr learned my addr 172.21.15.116:0/1398876038 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 10 monclient(hunting): get_auth_request con 0x7fbe90126b70 auth_method 0 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:39.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.469+0000 7fbe95824640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 10 monclient(hunting): handle_auth_done global_id 194559 payload 824 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 1 -- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe7c003ad0 msgr2=0x7fbe90126630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe96826640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe901430a0 0x7fbe90145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe7c003ad0 0x7fbe90126630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 1 -- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe901430a0 msgr2=0x7fbe90145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe901430a0 0x7fbe90145480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 10 monclient: _finish_hunting 0 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 1 monclient: found mon.noname-a 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:39.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.470+0000 7fbe95824640 1 -- 172.21.15.116:0/1398876038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe901287e0 con 0x7fbe90126b70 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe95824640 1 -- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 msgr2=0x7fbe90126f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe95824640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 0x7fbe90126f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000e810 tx=0x7fbe80009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe95824640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 0x7fbe90126f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000e810 tx=0x7fbe80009040 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fbe800072e0 mon_map magic: 0 v1 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 1 -- 172.21.15.116:0/1398876038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe800072e0 con 0x7fbe90126b70 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 10 monclient: dump: 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.488 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 10 monclient: _finish_auth 0 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 1 -- 172.21.15.116:0/1398876038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe80007440 con 0x7fbe90126b70 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe98289640 5 monclient: authenticate success, global_id 194559 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe98289640 10 monclient: _renew_subs 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe98289640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe90127880 con 0x7fbe90126b70 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe867fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:39.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe94822640 4 set_mon_vals no callback set 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe98289640 10 monclient: _renew_subs 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe98289640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.472+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9009c180 con 0x7fbe90126b70 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe98289640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe98289640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe98289640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9004bf60 con 0x7fbe90126b70 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe867fc640 1 -- 172.21.15.116:0/1398876038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80005780 con 0x7fbe90126b70 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe867fc640 10 monclient: dump: 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:39.490 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe867fc640 1 -- 172.21.15.116:0/1398876038 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fbe80005b30 con 0x7fbe90126b70 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.473+0000 7fbe867fc640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe74038490 0x7fbe7403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.474+0000 7fbe96025640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe74038490 0x7fbe7403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.474+0000 7fbe96025640 10 monclient: get_auth_request con 0x7fbe74038490 auth_method 0 2024-09-18T12:44:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.476+0000 7fbe96025640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe74038490 0x7fbe7403a940 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c003940 tx=0x7fbe8c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:39.477+0000 7fbe867fc640 1 -- 172.21.15.116:0/1398876038 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fbe8004c300 con 0x7fbe90126b70 2024-09-18T12:44:40.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.300+0000 7fbe867fc640 1 -- 172.21.15.116:0/1398876038 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fbe8001ce30 con 0x7fbe90126b70 2024-09-18T12:44:40.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.300+0000 7fbe867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:40.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.300+0000 7fbe867fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:40.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.519+0000 7fbe98289640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.519+0000 7fbe98289640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.519+0000 7fbe98289640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.519+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe900638d0 con 0x7fbe90126b70 2024-09-18T12:44:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.520+0000 7fbe867fc640 1 -- 172.21.15.116:0/1398876038 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fbe8001c830 con 0x7fbe90126b70 2024-09-18T12:44:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.520+0000 7fbe867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.520+0000 7fbe867fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:40.521 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe74038490 msgr2=0x7fbe7403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe74038490 0x7fbe7403a940 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c003940 tx=0x7fbe8c030000 comp rx=0 tx=0).stop 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 10 monclient: shutdown 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 msgr2=0x7fbe90126f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 0x7fbe90126f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000e810 tx=0x7fbe80009040 comp rx=0 tx=0).stop 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 shutdown_connections 2024-09-18T12:44:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fbe74038490 0x7fbe7403a940 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbe901430a0 0x7fbe90145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90126b70 0x7fbe90126f40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 --2- 172.21.15.116:0/1398876038 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbe7c003ad0 0x7fbe90126630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 >> 172.21.15.116:0/1398876038 conn(0x7fbe900678a0 msgr2=0x7fbe90096240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 shutdown_connections 2024-09-18T12:44:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:40.522+0000 7fbe98289640 1 -- 172.21.15.116:0/1398876038 wait complete. 2024-09-18T12:44:43.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.636+0000 7f0041b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.637+0000 7f0041b9d640 10 monclient: build_initial_monmap 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.637+0000 7f0041b9d640 10 monclient: monmap: 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:43.637676+0000 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:43.637676+0000 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:43.638 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.637+0000 7f0041b9d640 1 Processor -- start 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.637+0000 7f0041b9d640 1 -- start start 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.637+0000 7f0041b9d640 10 monclient: init 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f003c09c9c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f003c09e340 0x7f003c09e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f003c09e340 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c134aa0 0x7f003c136e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f003c134aa0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c09bdd0 con 0x7f003c09c9c0 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c09ec90 con 0x7f003c09e340 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c09df40 con 0x7f003c134aa0 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f0041b9d640 10 monclient(hunting): authenticate will time out at 1910.959491s 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c134aa0 msgr2=0x7f003c136e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c134aa0 0x7f003c136e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f003c09c9c0 0x7f003c09d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51878/0 (socket says 172.21.15.116:51878) 2024-09-18T12:44:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003b7fe640 1 -- 172.21.15.116:0/124996428 learned_addr learned my addr 172.21.15.116:0/124996428 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 1 --2- 172.21.15.116:0/124996428 >> v2:172.21.15.144:3300/0 conn(0x7f003c09e340 0x7f003c09e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 10 monclient(hunting): get_auth_request con 0x7f003c09e340 auth_method 0 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.638+0000 7f003affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 10 monclient(hunting): handle_auth_done global_id 234145 payload 274 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 1 -- 172.21.15.116:0/124996428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c134aa0 msgr2=0x7f003c136e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 1 --2- 172.21.15.116:0/124996428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c134aa0 0x7f003c136e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 1 -- 172.21.15.116:0/124996428 >> v1:172.21.15.116:6789/0 conn(0x7f003c09c9c0 legacy=0x7f003c09d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 10 monclient: _finish_hunting 0 2024-09-18T12:44:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 1 monclient: found mon.noname-c 2024-09-18T12:44:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:44:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.639+0000 7f003affd640 1 -- 172.21.15.116:0/124996428 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c09a880 con 0x7f003c09e340 2024-09-18T12:44:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003affd640 1 -- 172.21.15.116:0/124996428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09e340 msgr2=0x7f003c09e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003affd640 1 --2- 172.21.15.116:0/124996428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09e340 0x7f003c09e750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0030009880 tx=0x7f00300090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=328d592af17d7e95 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 1 -- 172.21.15.116:0/124996428 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030032440 con 0x7f003c09e340 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: dump: 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 1 -- 172.21.15.116:0/124996428 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00300325a0 con 0x7f003c09e340 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 1 -- 172.21.15.116:0/124996428 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030032940 con 0x7f003c09e340 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f0041b9d640 5 monclient: authenticate success, global_id 234145 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f0041b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f0041b9d640 4 set_mon_vals no callback set 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.642+0000 7f003a7fc640 10 monclient: dump: 2024-09-18T12:44:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 10 monclient: shutdown 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 -- 172.21.15.116:0/124996428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09e340 msgr2=0x7f003c09e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 --2- 172.21.15.116:0/124996428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09e340 0x7f003c09e750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0030009880 tx=0x7f00300090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 -- 172.21.15.116:0/124996428 shutdown_connections 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 --2- 172.21.15.116:0/124996428 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c134aa0 0x7f003c136e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 --2- 172.21.15.116:0/124996428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09e340 0x7f003c09e750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 -- 172.21.15.116:0/124996428 >> 172.21.15.116:0/124996428 conn(0x7f003c0a2e40 msgr2=0x7f003c0a3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 -- 172.21.15.116:0/124996428 shutdown_connections 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 1 -- 172.21.15.116:0/124996428 wait complete. 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 10 monclient: build_initial_monmap 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.643+0000 7f0041b9d640 10 monclient: monmap: 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 Processor -- start 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 -- start start 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient: init 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09c9c0 0x7f003c068b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f003c09c9c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c09e340 0x7f003c069090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f003c09e340 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 0x7f003c0695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f003c134aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c099cc0 con 0x7f003c134aa0 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c136d20 con 0x7f003c09c9c0 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c11f580 con 0x7f003c09e340 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003affd640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c09e340 msgr2=0x7f003c069090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f0041b9d640 10 monclient(hunting): authenticate will time out at 1910.965726s 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c09e340 0x7f003c069090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 0x7f003c0695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09c9c0 0x7f003c068b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 0x7f003c0695d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58776/0 (socket says 172.21.15.116:58776) 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 1 -- 172.21.15.116:0/1018211474 learned_addr learned my addr 172.21.15.116:0/1018211474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:43.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 10 monclient(hunting): get_auth_request con 0x7f003c134aa0 auth_method 0 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003b7fe640 10 monclient(hunting): get_auth_request con 0x7f003c09c9c0 auth_method 0 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.644+0000 7f003b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:43.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 10 monclient(hunting): handle_auth_done global_id 194562 payload 824 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 1 -- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c09e340 msgr2=0x7f003c069090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c09e340 0x7f003c069090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 1 -- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09c9c0 msgr2=0x7f003c068b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09c9c0 0x7f003c068b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 1 monclient: found mon.noname-a 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003bfff640 1 -- 172.21.15.116:0/1018211474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0030009390 con 0x7f003c134aa0 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.645+0000 7f003b7fe640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09c9c0 0x7f003c068b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f003bfff640 1 -- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 msgr2=0x7f003c0695d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:43.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f003bfff640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 0x7f003c0695d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f003400bf80 tx=0x7f0034009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 1 -- 172.21.15.116:0/1018211474 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0034006150 con 0x7f003c134aa0 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: dump: 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:43.658 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: _finish_auth 0 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 1 -- 172.21.15.116:0/1018211474 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00340062f0 con 0x7f003c134aa0 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0041b9d640 5 monclient: authenticate success, global_id 194562 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 1 -- 172.21.15.116:0/1018211474 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00340064a0 con 0x7f003c134aa0 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0041b9d640 10 monclient: _renew_subs 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0041b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c11f890 con 0x7f003c134aa0 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0039ffb640 4 set_mon_vals no callback set 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f00237fe640 10 monclient: dump: 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0041b9d640 10 monclient: _renew_subs 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0041b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.647+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c11e9a0 con 0x7f003c134aa0 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.648+0000 7f00237fe640 1 -- 172.21.15.116:0/1018211474 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0034012020 con 0x7f003c134aa0 2024-09-18T12:44:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.648+0000 7f00237fe640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0028038440 0x7f002803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.648+0000 7f00237fe640 1 -- 172.21.15.116:0/1018211474 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f003404c530 con 0x7f003c134aa0 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.648+0000 7f0041b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.648+0000 7f0041b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.648+0000 7f0041b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.648+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c04bf60 con 0x7f003c134aa0 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.649+0000 7f003b7fe640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0028038440 0x7f002803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.649+0000 7f003b7fe640 10 monclient: get_auth_request con 0x7f0028038440 auth_method 0 2024-09-18T12:44:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.651+0000 7f003b7fe640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0028038440 0x7f002803a8f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f002c007fd0 tx=0x7f002c007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.653+0000 7f00237fe640 1 -- 172.21.15.116:0/1018211474 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f00340132e0 con 0x7f003c134aa0 2024-09-18T12:44:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.653+0000 7f00237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.653+0000 7f00237fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:43.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.871+0000 7f0041b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:43.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.871+0000 7f0041b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.871+0000 7f0041b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.871+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f003c0638d0 con 0x7f003c134aa0 2024-09-18T12:44:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.872+0000 7f00237fe640 1 -- 172.21.15.116:0/1018211474 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f00340072f0 con 0x7f003c134aa0 2024-09-18T12:44:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.872+0000 7f00237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.872+0000 7f00237fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:43.873 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0028038440 msgr2=0x7f002803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0028038440 0x7f002803a8f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f002c007fd0 tx=0x7f002c007f00 comp rx=0 tx=0).stop 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 10 monclient: shutdown 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 msgr2=0x7f003c0695d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 0x7f003c0695d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f003400bf80 tx=0x7f0034009730 comp rx=0 tx=0).stop 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 shutdown_connections 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0028038440 0x7f002803a8f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003c134aa0 0x7f003c0695d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f003c09e340 0x7f003c069090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 --2- 172.21.15.116:0/1018211474 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f003c09c9c0 0x7f003c068b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 >> 172.21.15.116:0/1018211474 conn(0x7f003c0a2e40 msgr2=0x7f003c0a1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 shutdown_connections 2024-09-18T12:44:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:43.874+0000 7f0041b9d640 1 -- 172.21.15.116:0/1018211474 wait complete. 2024-09-18T12:44:46.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.988+0000 7fad6883f640 10 monclient: get_monmap_and_config 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.988+0000 7fad6883f640 10 monclient: build_initial_monmap 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.988+0000 7fad6883f640 10 monclient: monmap: 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:46.989234+0000 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:46.989234+0000 2024-09-18T12:44:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:46.990 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 1 Processor -- start 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 1 -- start start 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient: init 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad640974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad6409a8a0 0x7fad6409acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient(hunting): picked mon.noname-b con 0x7fad6409a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad64133070 0x7fad64135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient(hunting): picked mon.noname-a con 0x7fad64133070 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6412b210 con 0x7fad640974f0 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad640978c0 con 0x7fad64133070 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6409ca20 con 0x7fad6409a8a0 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad6883f640 10 monclient(hunting): authenticate will time out at 1914.310905s 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad5bfff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad6409a8a0 msgr2=0x7fad6409acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.989+0000 7fad5bfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad6409a8a0 0x7fad6409acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad64133070 0x7fad64135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad64133070 0x7fad64135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43490/0 (socket says 172.21.15.116:43490) 2024-09-18T12:44:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 -- 172.21.15.116:0/3787104683 learned_addr learned my addr 172.21.15.116:0/3787104683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): get_auth_request con 0x7fad64133070 auth_method 0 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient(hunting): handle_auth_done global_id 194565 payload 274 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 -- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad6409a8a0 msgr2=0x7fad6409acb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 --2- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad6409a8a0 0x7fad6409acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 -- 172.21.15.116:0/3787104683 >> v1:172.21.15.144:6789/0 conn(0x7fad640974f0 legacy=0x7fad6412ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:44:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 monclient: found mon.noname-a 2024-09-18T12:44:46.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:46.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:46.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.990+0000 7fad62d9d640 1 -- 172.21.15.116:0/3787104683 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6409b1f0 con 0x7fad64133070 2024-09-18T12:44:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad62d9d640 1 -- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64133070 msgr2=0x7fad64135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad62d9d640 1 --2- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64133070 0x7fad64135450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c0d0 tx=0x7fad5c00a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42189197ad9be6b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 1 -- 172.21.15.116:0/3787104683 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c00a3c0 con 0x7fad64133070 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: dump: 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:47.002 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: _finish_auth 0 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 1 -- 172.21.15.116:0/3787104683 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c02e050 con 0x7fad64133070 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad6883f640 5 monclient: authenticate success, global_id 194565 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad6883f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad6883f640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 1 -- 172.21.15.116:0/3787104683 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c008950 con 0x7fad64133070 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad6883f640 4 set_mon_vals no callback set 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.993+0000 7fad61d9b640 10 monclient: dump: 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 10 monclient: shutdown 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 -- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64133070 msgr2=0x7fad64135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 --2- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64133070 0x7fad64135450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c0d0 tx=0x7fad5c00a950 comp rx=0 tx=0).stop 2024-09-18T12:44:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 -- 172.21.15.116:0/3787104683 shutdown_connections 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 --2- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad6409a8a0 0x7fad6409acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 --2- 172.21.15.116:0/3787104683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64133070 0x7fad64135450 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 -- 172.21.15.116:0/3787104683 >> 172.21.15.116:0/3787104683 conn(0x7fad640672b0 msgr2=0x7fad640676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 -- 172.21.15.116:0/3787104683 shutdown_connections 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 -- 172.21.15.116:0/3787104683 wait complete. 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 10 monclient: build_initial_monmap 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 10 monclient: monmap: 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 Processor -- start 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 1 -- start start 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.994+0000 7fad6883f640 10 monclient: init 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 0x7fad640a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): picked mon.noname-a con 0x7fad640974f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad6409a8a0 0x7fad6406a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): picked mon.noname-b con 0x7fad6409a8a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad64133070 0x7fad6406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): picked mon.noname-c con 0x7fad64133070 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6409e4a0 con 0x7fad640974f0 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6406d650 con 0x7fad6409a8a0 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6406f550 con 0x7fad64133070 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6883f640 10 monclient(hunting): authenticate will time out at 1914.316429s 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad62d9d640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad64133070 msgr2=0x7fad6406abb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad62d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad64133070 0x7fad6406abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 0x7fad640a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 0x7fad640a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43502/0 (socket says 172.21.15.116:43502) 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 -- 172.21.15.116:0/1970784857 learned_addr learned my addr 172.21.15.116:0/1970784857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): get_auth_request con 0x7fad640974f0 auth_method 0 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad5bfff640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad6409a8a0 0x7fad6406a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad5bfff640 10 monclient(hunting): get_auth_request con 0x7fad6409a8a0 auth_method 0 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient(hunting): handle_auth_done global_id 194568 payload 824 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 -- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad64133070 msgr2=0x7fad6406abb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad64133070 0x7fad6406abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 -- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad6409a8a0 msgr2=0x7fad6406a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad6409a8a0 0x7fad6406a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient: _finish_hunting 0 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 monclient: found mon.noname-a 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad6259c640 1 -- 172.21.15.116:0/1970784857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c00abf0 con 0x7fad640974f0 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.995+0000 7fad5bfff640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad6409a8a0 0x7fad6406a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6259c640 1 -- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 msgr2=0x7fad640a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6259c640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 0x7fad640a0500 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fad5400b4f0 tx=0x7fad540090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 1 -- 172.21.15.116:0/1970784857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5400f720 con 0x7fad640974f0 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: dump: 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 1 -- 172.21.15.116:0/1970784857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5400f8a0 con 0x7fad640974f0 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6883f640 5 monclient: authenticate success, global_id 194568 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6883f640 10 monclient: _renew_subs 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6883f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6406c3f0 con 0x7fad640974f0 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 1 -- 172.21.15.116:0/1970784857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5400fc60 con 0x7fad640974f0 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad5a7fc640 10 monclient: dump: 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6159a640 4 set_mon_vals no callback set 2024-09-18T12:44:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6883f640 10 monclient: _renew_subs 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6883f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.998+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6406b820 con 0x7fad640974f0 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad5a7fc640 1 -- 172.21.15.116:0/1970784857 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fad54011610 con 0x7fad640974f0 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad6883f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad6883f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad6883f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6404bf60 con 0x7fad640974f0 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad5a7fc640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad44038440 0x7fad4403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad5a7fc640 1 -- 172.21.15.116:0/1970784857 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fad5404beb0 con 0x7fad640974f0 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad5bfff640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad44038440 0x7fad4403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:46.999+0000 7fad5bfff640 10 monclient: get_auth_request con 0x7fad44038440 auth_method 0 2024-09-18T12:44:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.002+0000 7fad5bfff640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad44038440 0x7fad4403a8f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fad4c004fd0 tx=0x7fad4c004980 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.003+0000 7fad5a7fc640 1 -- 172.21.15.116:0/1970784857 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fad54017d20 con 0x7fad640974f0 2024-09-18T12:44:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.003+0000 7fad5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.003+0000 7fad5a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:47.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.215+0000 7fad6883f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:47.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.215+0000 7fad6883f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:47.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.215+0000 7fad6883f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:47.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.215+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad6406e000 con 0x7fad640974f0 2024-09-18T12:44:47.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.215+0000 7fad5a7fc640 1 -- 172.21.15.116:0/1970784857 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fad54017720 con 0x7fad640974f0 2024-09-18T12:44:47.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.215+0000 7fad5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:47.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.215+0000 7fad5a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:47.217 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":188,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:47.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad44038440 msgr2=0x7fad4403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad44038440 0x7fad4403a8f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fad4c004fd0 tx=0x7fad4c004980 comp rx=0 tx=0).stop 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 10 monclient: shutdown 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 msgr2=0x7fad640a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 0x7fad640a0500 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fad5400b4f0 tx=0x7fad540090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 shutdown_connections 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fad44038440 0x7fad4403a8f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fad64133070 0x7fad6406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad6409a8a0 0x7fad6406a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 --2- 172.21.15.116:0/1970784857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad640974f0 0x7fad640a0500 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:47.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 >> 172.21.15.116:0/1970784857 conn(0x7fad640672b0 msgr2=0x7fad64129490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:47.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 shutdown_connections 2024-09-18T12:44:47.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:47.217+0000 7fad6883f640 1 -- 172.21.15.116:0/1970784857 wait complete. 2024-09-18T12:44:50.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:44:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.331+0000 7f1c80094640 10 monclient: get_monmap_and_config 2024-09-18T12:44:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.331+0000 7f1c80094640 10 monclient: build_initial_monmap 2024-09-18T12:44:50.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.331+0000 7f1c80094640 10 monclient: monmap: 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:50.332475+0000 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:50.332475+0000 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:50.333 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:50.334 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:50.334 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.332+0000 7f1c80094640 1 Processor -- start 2024-09-18T12:44:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.332+0000 7f1c80094640 1 -- start start 2024-09-18T12:44:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.332+0000 7f1c80094640 10 monclient: init 2024-09-18T12:44:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T12:44:50.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient(hunting): picked mon.noname-a con 0x7f1c780981c0 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c78131da0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 0x7f1c78067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient(hunting): picked mon.noname-b con 0x7f1c78132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c78098ad0 con 0x7f1c78131da0 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c78131bd0 con 0x7f1c780981c0 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7809ca60 con 0x7f1c78132170 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c80094640 10 monclient(hunting): authenticate will time out at 1917.654174s 2024-09-18T12:44:50.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7e631640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 msgr2=0x7f1c78067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7e631640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 0x7f1c78067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43508/0 (socket says 172.21.15.116:43508) 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 learned_addr learned my addr 172.21.15.116:0/90771997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c780981c0 auth_method 0 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.333+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:50.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.335+0000 7f1c7d62f640 1 -- 172.21.15.116:0/90771997 >> v1:172.21.15.116:6789/0 conn(0x7f1c78131da0 legacy=0x7f1c78067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7e631640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 msgr2=0x7f1c78067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7e631640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 0x7f1c78067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:44:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c780981c0 auth_method 0 2024-09-18T12:44:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.534+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:44:50.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7e631640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 msgr2=0x7f1c78067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:50.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7e631640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 0x7f1c78067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:44:50.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c780981c0 auth_method 0 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:50.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:50.935+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:44:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7e631640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 msgr2=0x7f1c78067820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7e631640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 0x7f1c78067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c780981c0 auth_method 0 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:51.736+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:44:53.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 10 monclient(hunting): tick 2024-09-18T12:44:53.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 monclient(hunting): continuing hunt 2024-09-18T12:44:53.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:44:53.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 msgr2=0x7f1c78067820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:53.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c78132170 0x7f1c78067820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 msgr2=0x7f1c78098590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.116:3300/0 conn(0x7f1c780981c0 0x7f1c78098590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 -- 172.21.15.116:0/90771997 >> v1:172.21.15.116:6789/0 conn(0x7f1c78131da0 legacy=0x7f1c78067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 10 monclient: _add_conns ranks=[4,1] 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1c5c003720 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c5c006280 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 -- 172.21.15.116:0/90771997 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7809cca0 con 0x7f1c5c006280 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 1 -- 172.21.15.116:0/90771997 --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c78131bd0 con 0x7f1c5c003720 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c6b7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c5c003720 auth_method 0 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.333+0000 7f1c7de30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.334+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 msgr2=0x7f1c5c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.334+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 msgr2=0x7f1c5c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.334+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.334+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:53.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.336+0000 7f1c7d62f640 1 -- 172.21.15.116:0/90771997 >> v1:172.21.15.144:6789/0 conn(0x7f1c5c006280 legacy=0x7f1c5c008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:53.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c5c003720 auth_method 0 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 msgr2=0x7f1c5c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 msgr2=0x7f1c5c005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:53.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.535+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:44:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.575+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504757192 0 0) 0x7f1c7809cca0 con 0x7f1c5c006280 2024-09-18T12:44:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.575+0000 7f1c7ce2e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.575+0000 7f1c7ce2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.575+0000 7f1c7ce2e640 10 monclient(hunting): my global_id is 234154 2024-09-18T12:44:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.575+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c640034e0 con 0x7f1c5c006280 2024-09-18T12:44:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3531277319 0 0) 0x7f1c640034e0 con 0x7f1c5c006280 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 msgr2=0x7f1c5c005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 --2- 172.21.15.116:0/90771997 >> v2:172.21.15.144:3300/0 conn(0x7f1c5c003720 0x7f1c5c005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: _finish_hunting 0 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c00b020 con 0x7f1c5c006280 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: _finish_auth 0 2024-09-18T12:44:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f1c74001800 con 0x7f1c5c006280 2024-09-18T12:44:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c80094640 5 monclient: authenticate success, global_id 234154 2024-09-18T12:44:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c80094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:44:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: dump: 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 >> v1:172.21.15.144:6789/0 conn(0x7f1c5c006280 legacy=0x7f1c5c008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7e631640 1 -- 172.21.15.116:0/90771997 reap_dead start 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.580+0000 7f1c7ce2e640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c640039d0 0x7f1c64003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): picked mon.a con 0x7f1c640039d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c64004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): picked mon.c con 0x7f1c64004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c64005250 0x7f1c6400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): picked mon.b con 0x7f1c64005250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c78131bd0 con 0x7f1c640039d0 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6400c1c0 con 0x7f1c64005250 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6400c380 con 0x7f1c64004780 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7ce2e640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c80094640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7d62f640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 msgr2=0x7f1c64004bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7d62f640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c64004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c640039d0 0x7f1c64003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c640039d0 auth_method 0 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c64005250 0x7f1c6400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 10 monclient(hunting): get_auth_request con 0x7f1c64005250 auth_method 0 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7e631640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient(hunting): handle_auth_done global_id 234154 payload 293 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 msgr2=0x7f1c64004bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c64004bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c64005250 msgr2=0x7f1c6400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c64005250 0x7f1c6400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient: _finish_hunting 0 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 1 monclient: found mon.a 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640034e0 con 0x7f1c640039d0 2024-09-18T12:44:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 10 monclient: _finish_auth 0 2024-09-18T12:44:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.581+0000 7f1c7de30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7de30640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c640039d0 msgr2=0x7f1c64003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7de30640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c640039d0 0x7f1c64003dc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009650 tx=0x7f1c6c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6458f344867116d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c031660 con 0x7f1c640039d0 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 10 monclient: dump: 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c80094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c00f040 con 0x7f1c640039d0 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c80094640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 1 -- 172.21.15.116:0/90771997 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c031a50 con 0x7f1c640039d0 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c80094640 4 set_mon_vals no callback set 2024-09-18T12:44:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.584+0000 7f1c7ce2e640 10 monclient: dump: 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 10 monclient: shutdown 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 1 -- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c640039d0 msgr2=0x7f1c64003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c640039d0 0x7f1c64003dc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009650 tx=0x7f1c6c009040 comp rx=0 tx=0).stop 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 1 -- 172.21.15.116:0/90771997 shutdown_connections 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c64005250 0x7f1c6400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c64004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 1 --2- 172.21.15.116:0/90771997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c640039d0 0x7f1c64003dc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.585+0000 7f1c80094640 1 -- 172.21.15.116:0/90771997 >> 172.21.15.116:0/90771997 conn(0x7f1c780a1e30 msgr2=0x7f1c780a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.586+0000 7f1c80094640 1 -- 172.21.15.116:0/90771997 shutdown_connections 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.586+0000 7f1c80094640 1 -- 172.21.15.116:0/90771997 wait complete. 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.586+0000 7f1c80094640 10 monclient: build_initial_monmap 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.586+0000 7f1c80094640 10 monclient: monmap: 2024-09-18T12:44:53.601 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.586+0000 7f1c80094640 1 Processor -- start 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 1 -- start start 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient: init 2024-09-18T12:44:53.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c640039d0 0x7f1c78140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): picked mon.noname-b con 0x7f1c640039d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c781309e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): picked mon.noname-c con 0x7f1c64004780 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 0x7f1c7811a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): picked mon.noname-a con 0x7f1c64005250 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7809b9e0 con 0x7f1c64005250 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7811dde0 con 0x7f1c640039d0 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7811bfd0 con 0x7f1c64004780 2024-09-18T12:44:53.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c80094640 10 monclient(hunting): authenticate will time out at 1920.908527s 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7d62f640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 msgr2=0x7f1c781309e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7d62f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c781309e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 0x7f1c7811a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 0x7f1c7811a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43552/0 (socket says 172.21.15.116:43552) 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 1 -- 172.21.15.116:0/989983702 learned_addr learned my addr 172.21.15.116:0/989983702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): get_auth_request con 0x7f1c64005250 auth_method 0 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7de30640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c640039d0 0x7f1c78140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7de30640 10 monclient(hunting): get_auth_request con 0x7f1c640039d0 auth_method 0 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7de30640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7de30640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 10 monclient(hunting): handle_auth_done global_id 194571 payload 824 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 1 -- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 msgr2=0x7f1c781309e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c781309e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 1 -- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c640039d0 msgr2=0x7f1c78140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.587+0000 7f1c7e631640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c640039d0 0x7f1c78140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.588+0000 7f1c7e631640 10 monclient: _finish_hunting 0 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.588+0000 7f1c7e631640 1 monclient: found mon.noname-a 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.588+0000 7f1c7e631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.588+0000 7f1c7e631640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.588+0000 7f1c7e631640 1 -- 172.21.15.116:0/989983702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7811db00 con 0x7f1c64005250 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c7e631640 1 -- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 msgr2=0x7f1c7811a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c7e631640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 0x7f1c7811a530 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1c70000ea0 tx=0x7f1c70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 1 -- 172.21.15.116:0/989983702 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000eec0 con 0x7f1c64005250 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: dump: 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.607 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 1 -- 172.21.15.116:0/989983702 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c700054f0 con 0x7f1c64005250 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c80094640 5 monclient: authenticate success, global_id 194571 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c80094640 10 monclient: _renew_subs 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c80094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c78141af0 con 0x7f1c64005250 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 1 -- 172.21.15.116:0/989983702 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70005890 con 0x7f1c64005250 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:53.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c69ffb640 10 monclient: dump: 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c6bfff640 4 set_mon_vals no callback set 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c80094640 10 monclient: _renew_subs 2024-09-18T12:44:53.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c80094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.590+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c781413d0 con 0x7f1c64005250 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.591+0000 7f1c69ffb640 1 -- 172.21.15.116:0/989983702 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1c7000c040 con 0x7f1c64005250 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.591+0000 7f1c69ffb640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c5c038e50 0x7f1c5c03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.591+0000 7f1c69ffb640 1 -- 172.21.15.116:0/989983702 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1c7004c1c0 con 0x7f1c64005250 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.592+0000 7f1c7de30640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c5c038e50 0x7f1c5c03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.592+0000 7f1c80094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.592+0000 7f1c80094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.592+0000 7f1c80094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.592+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c7804bf60 con 0x7f1c64005250 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.592+0000 7f1c7de30640 10 monclient: get_auth_request con 0x7f1c5c038e50 auth_method 0 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.594+0000 7f1c7de30640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c5c038e50 0x7f1c5c03b300 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009450 tx=0x7f1c6c03a000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.596+0000 7f1c69ffb640 1 -- 172.21.15.116:0/989983702 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1c70010660 con 0x7f1c64005250 2024-09-18T12:44:53.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.596+0000 7f1c69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:53.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.596+0000 7f1c69ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:53.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.811+0000 7f1c80094640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:44:53.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.811+0000 7f1c80094640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:53.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.811+0000 7f1c80094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.811+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1c78098ee0 con 0x7f1c64005250 2024-09-18T12:44:53.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.812+0000 7f1c69ffb640 1 -- 172.21.15.116:0/989983702 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f1c70018ba0 con 0x7f1c64005250 2024-09-18T12:44:53.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.812+0000 7f1c69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:44:53.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.812+0000 7f1c69ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:53.813 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":192,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c5c038e50 msgr2=0x7f1c5c03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c5c038e50 0x7f1c5c03b300 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009450 tx=0x7f1c6c03a000 comp rx=0 tx=0).stop 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 10 monclient: shutdown 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 msgr2=0x7f1c7811a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 0x7f1c7811a530 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1c70000ea0 tx=0x7f1c70008040 comp rx=0 tx=0).stop 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 shutdown_connections 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1c5c038e50 0x7f1c5c03b300 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c64005250 0x7f1c7811a530 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1c64004780 0x7f1c781309e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 --2- 172.21.15.116:0/989983702 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1c640039d0 0x7f1c78140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 >> 172.21.15.116:0/989983702 conn(0x7f1c780a1e30 msgr2=0x7f1c78096f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:53.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 shutdown_connections 2024-09-18T12:44:53.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.814+0000 7f1c80094640 1 -- 172.21.15.116:0/989983702 wait complete. 2024-09-18T12:44:53.826 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T12:44:53.826 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T12:44:53.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:44:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.919+0000 7f818415d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.919+0000 7f818415d640 10 monclient: build_initial_monmap 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.919+0000 7f818415d640 10 monclient: monmap: 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:53.920169+0000 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:53.920169+0000 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:53.921 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 1 Processor -- start 2024-09-18T12:44:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 1 -- start start 2024-09-18T12:44:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient: init 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f817c0974f0 0x7f817c12ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient(hunting): picked mon.noname-c con 0x7f817c0974f0 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 0x7f817c09acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient(hunting): picked mon.noname-b con 0x7f817c09a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f817c12dd40 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c0978c0 con 0x7f817c12dd40 2024-09-18T12:44:53.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c09ca20 con 0x7f817c0974f0 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c09b1f0 con 0x7f817c09a8a0 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f818415d640 10 monclient(hunting): authenticate will time out at 1921.241746s 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f81816f8640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 msgr2=0x7f817c09acb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.920+0000 7f81816f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 0x7f817c09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f817c0974f0 0x7f817c12ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f81826fa640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f817c12dd40 0x7f817c12e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.116:45330/0 (socket says 172.21.15.116:45330) 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f817c0974f0 0x7f817c12ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.116:38660/0 (socket says 172.21.15.116:38660) 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f81826fa640 1 -- 172.21.15.116:0/3019579066 learned_addr learned my addr 172.21.15.116:0/3019579066 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:44:53.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): get_auth_request con 0x7f817c0974f0 auth_method 0 2024-09-18T12:44:53.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient(hunting): handle_auth_done global_id 234160 payload 274 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 1 -- 172.21.15.116:0/3019579066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 msgr2=0x7f817c09acb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 1 --2- 172.21.15.116:0/3019579066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 0x7f817c09acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 1 -- 172.21.15.116:0/3019579066 >> v1:172.21.15.144:6789/0 conn(0x7f817c12dd40 legacy=0x7f817c12e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient: _finish_hunting 0 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 1 monclient: found mon.noname-c 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:44:53.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.921+0000 7f8181ef9640 1 -- 172.21.15.116:0/3019579066 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c098ad0 con 0x7f817c0974f0 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8181ef9640 1 -- 172.21.15.116:0/3019579066 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 msgr2=0x7f817c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8181ef9640 1 --2- 172.21.15.116:0/3019579066 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 0x7f817c12ab00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8170009850 tx=0x7f81700090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54d1a2c7cae6c36a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 1 -- 172.21.15.116:0/3019579066 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170032440 con 0x7f817c0974f0 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: dump: 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.936 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: _finish_auth 0 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 1 -- 172.21.15.116:0/3019579066 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81700325a0 con 0x7f817c0974f0 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 5 monclient: authenticate success, global_id 234160 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 1 -- 172.21.15.116:0/3019579066 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170032940 con 0x7f817c0974f0 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 4 set_mon_vals no callback set 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f8180ef7640 10 monclient: dump: 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.938 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.939 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.939 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.939 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.939 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.939 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.939 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 10 monclient: shutdown 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 1 -- 172.21.15.116:0/3019579066 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 msgr2=0x7f817c12ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 1 --2- 172.21.15.116:0/3019579066 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 0x7f817c12ab00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8170009850 tx=0x7f81700090f0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.924+0000 7f818415d640 1 -- 172.21.15.116:0/3019579066 shutdown_connections 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 1 --2- 172.21.15.116:0/3019579066 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 0x7f817c09acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 1 --2- 172.21.15.116:0/3019579066 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 0x7f817c12ab00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 1 -- 172.21.15.116:0/3019579066 >> 172.21.15.116:0/3019579066 conn(0x7f817c0672b0 msgr2=0x7f817c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 1 -- 172.21.15.116:0/3019579066 shutdown_connections 2024-09-18T12:44:53.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 1 -- 172.21.15.116:0/3019579066 wait complete. 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 10 monclient: build_initial_monmap 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 10 monclient: monmap: 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 1 Processor -- start 2024-09-18T12:44:53.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 1 -- start start 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.925+0000 7f818415d640 10 monclient: init 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 0x7f817c13f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): picked mon.noname-b con 0x7f817c0974f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 0x7f817c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): picked mon.noname-c con 0x7f817c09a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c12dd40 0x7f817c13fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): picked mon.noname-a con 0x7f817c12dd40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c09e1a0 con 0x7f817c12dd40 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c12f700 con 0x7f817c0974f0 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c06efb0 con 0x7f817c09a8a0 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f818415d640 10 monclient(hunting): authenticate will time out at 1921.247289s 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81816f8640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 msgr2=0x7f817c13f950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81816f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 0x7f817c13f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c12dd40 0x7f817c13fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 0x7f817c13f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c12dd40 0x7f817c13fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43566/0 (socket says 172.21.15.116:43566) 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 -- 172.21.15.116:0/4275733659 learned_addr learned my addr 172.21.15.116:0/4275733659 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): get_auth_request con 0x7f817c12dd40 auth_method 0 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 10 monclient(hunting): get_auth_request con 0x7f817c0974f0 auth_method 0 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:53.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient(hunting): handle_auth_done global_id 194574 payload 824 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 msgr2=0x7f817c13f950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f817c09a8a0 0x7f817c13f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 msgr2=0x7f817c13f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f817c0974f0 0x7f817c13f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient: _finish_hunting 0 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 monclient: found mon.noname-a 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.926+0000 7f81826fa640 1 -- 172.21.15.116:0/4275733659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8170009390 con 0x7f817c12dd40 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.928+0000 7f81826fa640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c12dd40 msgr2=0x7f817c13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.928+0000 7f81826fa640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c12dd40 0x7f817c13fe90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f817400ccc0 tx=0x7f8174005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400ce80 con 0x7f817c12dd40 2024-09-18T12:44:53.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: dump: 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f818415d640 5 monclient: authenticate success, global_id 194574 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f818415d640 10 monclient: _renew_subs 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817400aa00 con 0x7f817c12dd40 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f818415d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c098300 con 0x7f817c12dd40 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400adc0 con 0x7f817c12dd40 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816dffb640 10 monclient: dump: 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f816ffff640 4 set_mon_vals no callback set 2024-09-18T12:44:53.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f818415d640 10 monclient: _renew_subs 2024-09-18T12:44:53.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f818415d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:53.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.929+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c12f4a0 con 0x7f817c12dd40 2024-09-18T12:44:53.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f8174007300 con 0x7f817c12dd40 2024-09-18T12:44:53.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f818415d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:53.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f818415d640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 0x7f817c069a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f818415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c065f20 con 0x7f817c0696a0 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f816dffb640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8160038440 0x7f816003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f817404c3b0 con 0x7f817c12dd40 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 0x7f817c069a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient: get_auth_request con 0x7f817c0696a0 auth_method 0 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f81816f8640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8160038440 0x7f816003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f81816f8640 10 monclient: get_auth_request con 0x7f8160038440 auth_method 0 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 10 monclient(hunting): handle_auth_done global_id 194577 payload 824 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.930+0000 7f8181ef9640 1 -- 172.21.15.116:0/4275733659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f817c1307a0 con 0x7f817c0696a0 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.932+0000 7f8181ef9640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 msgr2=0x7f817c069a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.932+0000 7f8181ef9640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 0x7f817c069a70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f817c065310 tx=0x7f8170032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.932+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817003d8f0 con 0x7f817c0696a0 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.932+0000 7f816dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.932+0000 7f816dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.932+0000 7f816dffb640 10 monclient: dump: 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f81816f8640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8160038440 0x7f816003a8f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8178006fd0 tx=0x7f8178006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8170041070 con 0x7f817c0696a0 2024-09-18T12:44:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f816dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:53.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f816dffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:53.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 msgr2=0x7f817c069a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f816dffb640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 0x7f817c069a70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f817c065310 tx=0x7f8170032000 comp rx=0 tx=0).stop 2024-09-18T12:44:53.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 msgr2=0x7f817c069a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:53.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.933+0000 7f816dffb640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c0696a0 0x7f817c069a70 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f817c065310 tx=0x7f8170032000 comp rx=0 tx=0).stop 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.938+0000 7f818415d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.938+0000 7f818415d640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 0x7f817c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.938+0000 7f818415d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.938+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c04bf60 con 0x7f817c069fb0 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.938+0000 7f8181ef9640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 0x7f817c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient: get_auth_request con 0x7f817c069fb0 auth_method 0 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 10 monclient(hunting): handle_auth_done global_id 194580 payload 824 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.939+0000 7f8181ef9640 1 -- 172.21.15.116:0/4275733659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f817c09e650 con 0x7f817c069fb0 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f8181ef9640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 msgr2=0x7f817c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f8181ef9640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 0x7f817c06d3d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8170009360 tx=0x7f817003f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817004c040 con 0x7f817c069fb0 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 10 monclient: dump: 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4739 (secure 0 0 0) 0x7f817003c810 con 0x7f817c069fb0 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 msgr2=0x7f817c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 0x7f817c06d3d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8170009360 tx=0x7f817003f070 comp rx=0 tx=0).stop 2024-09-18T12:44:53.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 msgr2=0x7f817c06d3d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.941+0000 7f816dffb640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c069fb0 0x7f817c06d3d0 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8170009360 tx=0x7f817003f070 comp rx=0 tx=0).stop 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 192, 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:44:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:53.971 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:53.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8160038440 msgr2=0x7f816003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f8160038440 0x7f816003a8f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8178006fd0 tx=0x7f8178006d30 comp rx=0 tx=0).stop 2024-09-18T12:44:53.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 10 monclient: shutdown 2024-09-18T12:44:53.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:53.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c12dd40 msgr2=0x7f817c13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:53.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 --2- 172.21.15.116:0/4275733659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f817c12dd40 0x7f817c13fe90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f817400ccc0 tx=0x7f8174005370 comp rx=0 tx=0).stop 2024-09-18T12:44:53.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f81826fa640 1 -- 172.21.15.116:0/4275733659 reap_dead start 2024-09-18T12:44:53.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f81826fa640 1 -- 172.21.15.116:0/4275733659 reap_dead start 2024-09-18T12:44:53.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 shutdown_connections 2024-09-18T12:44:53.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 >> 172.21.15.116:0/4275733659 conn(0x7f817c0672b0 msgr2=0x7f817c0a0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:53.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 shutdown_connections 2024-09-18T12:44:53.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:53.942+0000 7f818415d640 1 -- 172.21.15.116:0/4275733659 wait complete. 2024-09-18T12:44:53.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:44:53.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T12:44:53.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.066+0000 7f480bd5d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.066+0000 7f480bd5d640 10 monclient: build_initial_monmap 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.066+0000 7f480bd5d640 10 monclient: monmap: 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:44:54.066873+0000 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:44:54.066873+0000 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:54.068 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.066+0000 7f480bd5d640 1 Processor -- start 2024-09-18T12:44:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.066+0000 7f480bd5d640 1 -- start start 2024-09-18T12:44:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.066+0000 7f480bd5d640 10 monclient: init 2024-09-18T12:44:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T12:44:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4804131ab0 addr v1:172.21.15.144:6789/0 2024-09-18T12:44:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f480409e910 0x7f480409ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f480409e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f480409f250 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480409faf0 con 0x7f4804131ab0 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4804132590 con 0x7f480409f250 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480409fd30 con 0x7f480409e910 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480bd5d640 10 monclient(hunting): authenticate will time out at 1921.388468s 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43594/0 (socket says 172.21.15.116:43594) 2024-09-18T12:44:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 -- 172.21.15.116:0/1533646048 learned_addr learned my addr 172.21.15.116:0/1533646048 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): get_auth_request con 0x7f480409f250 auth_method 0 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f48092f8640 1 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.144:3300/0 conn(0x7f480409e910 0x7f480409ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f48092f8640 10 monclient(hunting): get_auth_request con 0x7f480409e910 auth_method 0 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f48092f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f48092f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f48092f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient(hunting): handle_auth_done global_id 194583 payload 274 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 -- 172.21.15.116:0/1533646048 >> v2:172.21.15.144:3300/0 conn(0x7f480409e910 msgr2=0x7f480409ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.144:3300/0 conn(0x7f480409e910 0x7f480409ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 -- 172.21.15.116:0/1533646048 >> v1:172.21.15.144:6789/0 conn(0x7f4804131ab0 legacy=0x7f4804131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient: _finish_hunting 0 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 monclient: found mon.noname-a 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.067+0000 7f480a2fa640 1 -- 172.21.15.116:0/1533646048 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48040a1110 con 0x7f480409f250 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.068+0000 7f480a2fa640 1 -- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 msgr2=0x7f4804138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.068+0000 7f480a2fa640 1 -- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 msgr2=0x7f4804138bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.068+0000 7f480a2fa640 1 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f480409bdb0 tx=0x7f47fc009840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.068+0000 7f480a2fa640 1 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:54.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f480a2fa640 1 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f480a2fa640 10 monclient: get_auth_request con 0x7f480409f250 auth_method 0 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f480a2fa640 0 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f480a2fa640 1 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient: ms_handle_reset current mon v2:172.21.15.116:3300/0 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 msgr2=0x7f4804138bc0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 1 --2- 172.21.15.116:0/1533646048 >> v2:172.21.15.116:3300/0 conn(0x7f480409f250 0x7f4804138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient: _add_conns ranks=[0,2] 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47f00034c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f00073e0 0x7f47f0009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient(hunting): picked mon.noname-b con 0x7f47f00073e0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f000a210 con 0x7f47f00034c0 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f000b420 con 0x7f47f00073e0 2024-09-18T12:44:54.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:54.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f4808af7640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.116:3300/0 2024-09-18T12:44:54.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f48092f8640 1 -- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f00073e0 msgr2=0x7f47f0009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:54.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.269+0000 7f48092f8640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f00073e0 0x7f47f0009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4809af9640 1 -- 172.21.15.116:0/1533646048 >> v1:172.21.15.116:6789/0 conn(0x7f47f00034c0 legacy=0x7f47f0005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139565031 0 0) 0x7f47f000a210 con 0x7f47f00034c0 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 --> v1:172.21.15.116:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f47f000a3a0 con 0x7f47f00034c0 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2358441485 0 0) 0x7f47f000a3a0 con 0x7f47f00034c0 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f00073e0 msgr2=0x7f47f0009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f00073e0 0x7f47f0009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: _finish_hunting 0 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f000b6a0 con 0x7f47f00034c0 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: _finish_auth 0 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f47f8002860 con 0x7f47f00034c0 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f480bd5d640 5 monclient: authenticate success, global_id 194583 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f480bd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: dump: 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.278 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 >> v1:172.21.15.116:6789/0 conn(0x7f47f00034c0 legacy=0x7f47f0005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f4808af7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.271+0000 7f480a2fa640 1 -- 172.21.15.116:0/1533646048 reap_dead start 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000abd0 0x7f47f0009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): picked mon.c con 0x7f47f000abd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000b8f0 0x7f47f000dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): picked mon.a con 0x7f47f000b8f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 0x7f47f00105f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): picked mon.b con 0x7f47f000e210 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f000b420 con 0x7f47f000b8f0 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0010d40 con 0x7f47f000e210 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4809af9640 1 -- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000abd0 msgr2=0x7f47f0009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0010f00 con 0x7f47f000abd0 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4809af9640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000abd0 0x7f47f0009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f4808af7640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f480bd5d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000b8f0 0x7f47f000dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): get_auth_request con 0x7f47f000b8f0 auth_method 0 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient(hunting): handle_auth_done global_id 194583 payload 293 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 1 -- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000abd0 msgr2=0x7f47f0009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000abd0 0x7f47f0009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 1 -- 172.21.15.116:0/1533646048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 msgr2=0x7f47f00105f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 0x7f47f00105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient: _finish_hunting 0 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 1 monclient: found mon.a 2024-09-18T12:44:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 1 -- 172.21.15.116:0/1533646048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f000a3a0 con 0x7f47f000b8f0 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 10 monclient: _finish_auth 0 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.272+0000 7f48092f8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f48092f8640 1 -- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000b8f0 msgr2=0x7f47f000dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f48092f8640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000b8f0 0x7f47f000dcd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4800006f90 tx=0x7f4800008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90489349585b8f4a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800018070 con 0x7f47f000b8f0 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 10 monclient: dump: 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480000d980 con 0x7f47f000b8f0 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f480bd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 1 -- 172.21.15.116:0/1533646048 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480000dd20 con 0x7f47f000b8f0 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f480bd5d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f480bd5d640 4 set_mon_vals no callback set 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:44:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.274+0000 7f4808af7640 10 monclient: dump: 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 10 monclient: shutdown 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 -- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000b8f0 msgr2=0x7f47f000dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000b8f0 0x7f47f000dcd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4800006f90 tx=0x7f4800008040 comp rx=0 tx=0).stop 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 -- 172.21.15.116:0/1533646048 shutdown_connections 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 0x7f47f00105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000b8f0 0x7f47f000dcd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 --2- 172.21.15.116:0/1533646048 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000abd0 0x7f47f0009810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 -- 172.21.15.116:0/1533646048 >> 172.21.15.116:0/1533646048 conn(0x7f4804067d30 msgr2=0x7f48040a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 -- 172.21.15.116:0/1533646048 shutdown_connections 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 1 -- 172.21.15.116:0/1533646048 wait complete. 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 10 monclient: build_initial_monmap 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.275+0000 7f480bd5d640 10 monclient: monmap: 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 Processor -- start 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 -- start start 2024-09-18T12:44:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient: init 2024-09-18T12:44:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:44:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 0x7f480409aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f47f000abd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000b8f0 0x7f4804130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f47f000b8f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 0x7f480411b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f47f000e210 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480411dcc0 con 0x7f47f000abd0 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480409bde0 con 0x7f47f000e210 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4804134820 con 0x7f47f000b8f0 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f480bd5d640 10 monclient(hunting): authenticate will time out at 1921.597786s 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f48092f8640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000b8f0 msgr2=0x7f4804130ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f48092f8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000b8f0 0x7f4804130ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 0x7f480409aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 0x7f480409aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43612/0 (socket says 172.21.15.116:43612) 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 1 -- 172.21.15.116:0/3599453551 learned_addr learned my addr 172.21.15.116:0/3599453551 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 10 monclient(hunting): get_auth_request con 0x7f47f000abd0 auth_method 0 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 msgr2=0x7f480409aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:44:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 msgr2=0x7f480409aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 0x7f480409aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.276+0000 7f4809af9640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 0x7f480409aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 0x7f480411b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): get_auth_request con 0x7f47f000e210 auth_method 0 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient(hunting): handle_auth_done global_id 234169 payload 824 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000b8f0 msgr2=0x7f4804130ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f47f000b8f0 0x7f4804130ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 msgr2=0x7f480409aae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f000abd0 0x7f480409aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient: _finish_hunting 0 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 1 monclient: found mon.noname-b 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.277+0000 7f480a2fa640 1 -- 172.21.15.116:0/3599453551 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480411cd60 con 0x7f47f000e210 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.279+0000 7f480a2fa640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 msgr2=0x7f480411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.279+0000 7f480a2fa640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 0x7f480411b6b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f480409bdb0 tx=0x7f47fc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc037070 con 0x7f47f000e210 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: dump: 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: _finish_auth 0 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47fc02d050 con 0x7f47f000e210 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f480bd5d640 5 monclient: authenticate success, global_id 234169 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0328c0 con 0x7f47f000e210 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f480bd5d640 10 monclient: _renew_subs 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f480bd5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4804141d30 con 0x7f47f000e210 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f5ffb640 10 monclient: dump: 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f47f7fff640 4 set_mon_vals no callback set 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f480bd5d640 10 monclient: _renew_subs 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f480bd5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.280+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480409b5d0 con 0x7f47f000e210 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f480bd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f480bd5d640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 0x7f480411bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f480bd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4804063740 con 0x7f48041310e0 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f47fc032cf0 con 0x7f47f000e210 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 0x7f480411bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient: get_auth_request con 0x7f48041310e0 auth_method 0 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f47f5ffb640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f47e8038440 0x7f47e803a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f47fc06c240 con 0x7f47f000e210 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 10 monclient(hunting): handle_auth_done global_id 234172 payload 824 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f4809af9640 1 -- 172.21.15.116:0/3599453551 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f480404bf60 con 0x7f48041310e0 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f48092f8640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f47e8038440 0x7f47e803a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.281+0000 7f48092f8640 10 monclient: get_auth_request con 0x7f47e8038440 auth_method 0 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.283+0000 7f4809af9640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 msgr2=0x7f480411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.283+0000 7f4809af9640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 0x7f480411bbf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47f80027b0 tx=0x7f47f80028d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800fe90 con 0x7f48041310e0 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 10 monclient: dump: 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f48092f8640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f47e8038440 0x7f47e803a8f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f47f000a1e0 tx=0x7f480001a000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f47f800dbb0 con 0x7f48041310e0 2024-09-18T12:44:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 msgr2=0x7f480411bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 0x7f480411bbf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47f80027b0 tx=0x7f47f80028d0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 msgr2=0x7f480411bbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.284+0000 7f47f5ffb640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48041310e0 0x7f480411bbf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f480bd5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f480bd5d640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 0x7f4804096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f480bd5d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041314b0 con 0x7f480411c130 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 0x7f4804096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient: get_auth_request con 0x7f480411c130 auth_method 0 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 10 monclient(hunting): handle_auth_done global_id 234175 payload 824 2024-09-18T12:44:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.290+0000 7f4809af9640 1 -- 172.21.15.116:0/3599453551 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f480411c530 con 0x7f480411c130 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.292+0000 7f4809af9640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 msgr2=0x7f4804096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.292+0000 7f4809af9640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 0x7f4804096a60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f47f80027b0 tx=0x7f47f8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f801dcc0 con 0x7f480411c130 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 10 monclient: dump: 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:44:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4567 (secure 0 0 0) 0x7f47f801de20 con 0x7f480411c130 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 msgr2=0x7f4804096a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 0x7f4804096a60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f47f80027b0 tx=0x7f47f8019000 comp rx=0 tx=0).stop 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 msgr2=0x7f4804096a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.293+0000 7f47f5ffb640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f480411c130 0x7f4804096a60 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f47f80027b0 tx=0x7f47f8019000 comp rx=0 tx=0).stop 2024-09-18T12:44:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 192, 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: 1 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 0, 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:44:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:44:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:44:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:44:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:44:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:44:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:44:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:44:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:44:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:44:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:44:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:44:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:44:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:44:54.317 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:44:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f47e8038440 msgr2=0x7f47e803a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f47e8038440 0x7f47e803a8f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f47f000a1e0 tx=0x7f480001a000 comp rx=0 tx=0).stop 2024-09-18T12:44:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 10 monclient: shutdown 2024-09-18T12:44:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 msgr2=0x7f480411b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 --2- 172.21.15.116:0/3599453551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47f000e210 0x7f480411b6b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f480409bdb0 tx=0x7f47fc009840 comp rx=0 tx=0).stop 2024-09-18T12:44:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480a2fa640 1 -- 172.21.15.116:0/3599453551 reap_dead start 2024-09-18T12:44:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480a2fa640 1 -- 172.21.15.116:0/3599453551 reap_dead start 2024-09-18T12:44:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 shutdown_connections 2024-09-18T12:44:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 >> 172.21.15.116:0/3599453551 conn(0x7f4804067d30 msgr2=0x7f48040a3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 shutdown_connections 2024-09-18T12:44:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:44:54.294+0000 7f480bd5d640 1 -- 172.21.15.116:0/3599453551 wait complete. 2024-09-18T12:44:54.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T12:44:54.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T12:44:57.430 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=110 2024-09-18T12:44:57.430 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 110 -ne 110 ]] 2024-09-18T12:44:57.430 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:44:57.430 INFO:tasks.workunit.client.0.smithi144.stderr:+ export CEPH_ARGS= 2024-09-18T12:44:57.430 INFO:tasks.workunit.client.0.smithi144.stderr:+ CEPH_ARGS= 2024-09-18T12:44:57.430 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph auth del client.bar 2024-09-18T12:44:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.503+0000 7f0e0eb1d640 10 monclient: get_monmap_and_config 2024-09-18T12:44:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.503+0000 7f0e0eb1d640 10 monclient: build_initial_monmap 2024-09-18T12:44:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.503+0000 7f0e0eb1d640 10 monclient: monmap: 2024-09-18T12:44:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:44:57.504 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:44:57.504356+0000 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:44:57.504356+0000 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:44:57.505 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:44:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 Processor -- start 2024-09-18T12:44:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 -- start start 2024-09-18T12:44:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient: init 2024-09-18T12:44:57.507 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e08099350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e08098f80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0e08099cb0 0x7f0e0809d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e08099cb0 addr v2:172.21.15.116:3300/0 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e0809df00 0x7f0e0809e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e0809df00 addr v2:172.21.15.144:3300/0 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e08069190 con 0x7f0e08099cb0 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0809f2d0 con 0x7f0e0809df00 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:57.508 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0809f510 con 0x7f0e08098f80 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0eb1d640 10 monclient(hunting): authenticate will time out at 1933.073724s 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e0809df00 0x7f0e0809e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0c8b9640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 msgr2=0x7f0e08099350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0c8b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e08099350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e0809df00 0x7f0e0809e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39480/0 (socket says 172.21.15.144:39480) 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/819538763 learned_addr learned my addr 172.21.15.144:0/819538763 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 10 monclient(hunting): get_auth_request con 0x7f0e0809df00 auth_method 0 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.504+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:57.509 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e07fff640 1 --2- 172.21.15.144:0/819538763 >> v2:172.21.15.116:3300/0 conn(0x7f0e08099cb0 0x7f0e0809d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e07fff640 10 monclient(hunting): get_auth_request con 0x7f0e08099cb0 auth_method 0 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_done global_id 234178 payload 274 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/819538763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 msgr2=0x7f0e08099350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 1 --2- 172.21.15.144:0/819538763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e08099350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/819538763 >> v2:172.21.15.116:3300/0 conn(0x7f0e08099cb0 msgr2=0x7f0e0809d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 1 --2- 172.21.15.144:0/819538763 >> v2:172.21.15.116:3300/0 conn(0x7f0e08099cb0 0x7f0e0809d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 10 monclient: _finish_hunting 0 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 1 monclient: found mon.noname-c 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:44:57.510 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.505+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/819538763 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08068810 con 0x7f0e0809df00 2024-09-18T12:44:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.507+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/819538763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 msgr2=0x7f0e0809e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.507+0000 7f0e0d0ba640 1 --2- 172.21.15.144:0/819538763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e0809e3a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e080699e0 tx=0x7f0e00009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4f4f3144734da4c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 1 -- 172.21.15.144:0/819538763 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00007610 con 0x7f0e0809df00 2024-09-18T12:44:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:57.516 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: dump: 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: _finish_auth 0 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:57.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 5 monclient: authenticate success, global_id 234178 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 1 -- 172.21.15.144:0/819538763 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0002d050 con 0x7f0e0809df00 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 10 monclient: get_monmap_and_config success 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 4 set_mon_vals no callback set 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 1 -- 172.21.15.144:0/819538763 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e000079e0 con 0x7f0e0809df00 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e077fe640 10 monclient: dump: 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:57.518 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 10 monclient: shutdown 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/819538763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 msgr2=0x7f0e0809e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/819538763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e0809e3a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e080699e0 tx=0x7f0e00009840 comp rx=0 tx=0).stop 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/819538763 shutdown_connections 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/819538763 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e0809e3a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/819538763 >> v2:172.21.15.116:3300/0 conn(0x7f0e08099cb0 0x7f0e0809d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/819538763 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e08099350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/819538763 >> 172.21.15.144:0/819538763 conn(0x7f0e08070fd0 msgr2=0x7f0e080a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:57.519 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/819538763 shutdown_connections 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.508+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/819538763 wait complete. 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.509+0000 7f0e0eb1d640 10 monclient: build_initial_monmap 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.509+0000 7f0e0eb1d640 10 monclient: monmap: 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:44:57.520 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.509+0000 7f0e0eb1d640 1 Processor -- start 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.509+0000 7f0e0eb1d640 1 -- start start 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.509+0000 7f0e0eb1d640 10 monclient: init 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient: _reopen_session rank -1 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e0813d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e08098f80 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08099cb0 0x7f0e0813ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e08099cb0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e08140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e0809df00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:57.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e080a2670 con 0x7f0e08099cb0 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08120310 con 0x7f0e0809df00 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08121d90 con 0x7f0e08098f80 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): _renew_subs 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0eb1d640 10 monclient(hunting): authenticate will time out at 1933.079174s 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0c8b9640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 msgr2=0x7f0e0813d640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0c8b9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e0813d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e08140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e08140b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39494/0 (socket says 172.21.15.144:39494) 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/3867871486 learned_addr learned my addr 172.21.15.144:0/3867871486 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): get_auth_request con 0x7f0e0809df00 auth_method 0 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:57.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08099cb0 0x7f0e0813ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 10 monclient(hunting): get_auth_request con 0x7f0e08099cb0 auth_method 0 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:44:57.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient(hunting): handle_auth_done global_id 234181 payload 824 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/3867871486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 msgr2=0x7f0e0813d640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e0813d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/3867871486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08099cb0 msgr2=0x7f0e0813ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08099cb0 0x7f0e0813ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient: _finish_hunting 0 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 monclient: found mon.noname-b 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.510+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/3867871486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e00009ae0 con 0x7f0e0809df00 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e0d0ba640 1 -- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 msgr2=0x7f0e08140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e0d0ba640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e08140b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e00009f50 tx=0x7f0e00036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:57.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00031c50 con 0x7f0e0809df00 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: dump: 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:57.525 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: _finish_auth 0 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e00031db0 con 0x7f0e0809df00 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e0eb1d640 5 monclient: authenticate success, global_id 234181 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e0eb1d640 10 monclient: _renew_subs 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e0eb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e08122070 con 0x7f0e0809df00 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0003c650 con 0x7f0e0809df00 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e04ff9640 10 monclient: dump: 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:44:57.526 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.513+0000 7f0e0eb1d640 10 monclient: _renew_subs 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.513+0000 7f0e0eb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.512+0000 7f0e06ffd640 4 set_mon_vals no callback set 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.513+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0813d230 con 0x7f0e0809df00 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.513+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0e00003020 con 0x7f0e0809df00 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e0eb1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e0eb1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:57.527 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e0eb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:44:57.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08066600 con 0x7f0e0809df00 2024-09-18T12:44:57.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e04ff9640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dec038440 0x7f0dec03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:44:57.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0e0006cfb0 con 0x7f0e0809df00 2024-09-18T12:44:57.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e0c8b9640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dec038440 0x7f0dec03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:44:57.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.514+0000 7f0e0c8b9640 10 monclient: get_auth_request con 0x7f0dec038440 auth_method 0 2024-09-18T12:44:57.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.516+0000 7f0e0c8b9640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dec038440 0x7f0dec03a8f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0038f0 tx=0x7f0dfc006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:44:57.528 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.518+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f0e0003a690 con 0x7f0e0809df00 2024-09-18T12:44:57.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.518+0000 7f0e04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:44:57.529 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.518+0000 7f0e04ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:44:57.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.734+0000 7f0e0eb1d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T12:44:57.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.734+0000 7f0e0eb1d640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T12:44:57.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.734+0000 7f0e0eb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:44:57.735 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.734+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f0e080696a0 con 0x7f0e0809df00 2024-09-18T12:44:57.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.738+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0e00040070 con 0x7f0e0809df00 2024-09-18T12:44:57.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.738+0000 7f0e04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T12:44:57.739 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.738+0000 7f0e04ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:44:57.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.889+0000 7f0e0eb1d640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T12:44:57.890 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.889+0000 7f0e0eb1d640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T12:44:57.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.889+0000 7f0e0eb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:44:57.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.889+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f0e08120e70 con 0x7f0e0809df00 2024-09-18T12:44:57.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.890+0000 7f0e04ff9640 1 -- 172.21.15.144:0/3867871486 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0e0003a690 con 0x7f0e0809df00 2024-09-18T12:44:57.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.890+0000 7f0e04ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T12:44:57.891 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.890+0000 7f0e04ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:44:57.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dec038440 msgr2=0x7f0dec03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:57.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dec038440 0x7f0dec03a8f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0038f0 tx=0x7f0dfc006d80 comp rx=0 tx=0).stop 2024-09-18T12:44:57.893 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 10 monclient: shutdown 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 msgr2=0x7f0e08140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e08140b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e00009f50 tx=0x7f0e00036000 comp rx=0 tx=0).stop 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 shutdown_connections 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0dec038440 0x7f0dec03a8f0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e0809df00 0x7f0e08140b90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08099cb0 0x7f0e0813ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 --2- 172.21.15.144:0/3867871486 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0e08098f80 0x7f0e0813d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 >> 172.21.15.144:0/3867871486 conn(0x7f0e08070fd0 msgr2=0x7f0e0809c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 shutdown_connections 2024-09-18T12:44:57.894 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:44:57.892+0000 7f0e0eb1d640 1 -- 172.21.15.144:0/3867871486 wait complete. 2024-09-18T12:44:57.902 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-18T12:44:57.902 INFO:tasks.workunit.client.0.smithi144.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T12:44:57.903 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-18T12:44:57.903 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=13 2024-09-18T12:44:57.903 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T12:44:57.903 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T12:45:00.975 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=13 2024-09-18T12:45:00.975 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T12:45:00.976 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:45:00.976 INFO:tasks.workunit.client.0.smithi144.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T12:45:00.977 INFO:tasks.workunit.client.0.smithi144.stderr:+ cat 2024-09-18T12:45:00.977 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph auth import -i - 2024-09-18T12:45:01.053 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.052+0000 7fcd35646640 10 monclient: get_monmap_and_config 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.052+0000 7fcd35646640 10 monclient: build_initial_monmap 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.052+0000 7fcd35646640 10 monclient: monmap: 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:45:01.053812+0000 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:45:01.053812+0000 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:01.054 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:01.055 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:01.055 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.053+0000 7fcd35646640 1 Processor -- start 2024-09-18T12:45:01.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.053+0000 7fcd35646640 1 -- start start 2024-09-18T12:45:01.057 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.053+0000 7fcd35646640 10 monclient: init 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.053+0000 7fcd35646640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.053+0000 7fcd35646640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.053+0000 7fcd35646640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd30098180 0x7fcd301321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): picked mon.noname-a con 0x7fcd30098180 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcd30098550 0x7fcd300674d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): picked mon.noname-c con 0x7fcd30098550 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098950 0x7fcd30067a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): picked mon.noname-b con 0x7fcd30098950 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd30096d30 con 0x7fcd30098180 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd300a1290 con 0x7fcd30098550 2024-09-18T12:45:01.058 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd300a1570 con 0x7fcd30098950 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd35646640 10 monclient(hunting): authenticate will time out at 1936.623051s 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2f7fe640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098950 msgr2=0x7fcd30067a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098950 0x7fcd30067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcd30098550 0x7fcd300674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fcd30098550 0x7fcd300674d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39522/0 (socket says 172.21.15.144:39522) 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 -- 172.21.15.144:0/854582531 learned_addr learned my addr 172.21.15.144:0/854582531 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): get_auth_request con 0x7fcd30098550 auth_method 0 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.059 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 1 --2- 172.21.15.144:0/854582531 >> v2:172.21.15.116:3300/0 conn(0x7fcd30098180 0x7fcd301321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 10 monclient(hunting): get_auth_request con 0x7fcd30098180 auth_method 0 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient(hunting): handle_auth_done global_id 234190 payload 274 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 -- 172.21.15.144:0/854582531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098950 msgr2=0x7fcd30067a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 --2- 172.21.15.144:0/854582531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098950 0x7fcd30067a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.060 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 -- 172.21.15.144:0/854582531 >> v2:172.21.15.116:3300/0 conn(0x7fcd30098180 msgr2=0x7fcd301321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:01.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 --2- 172.21.15.144:0/854582531 >> v2:172.21.15.116:3300/0 conn(0x7fcd30098180 0x7fcd301321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:45:01.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 monclient: found mon.noname-c 2024-09-18T12:45:01.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:01.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.061 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.054+0000 7fcd2e7fc640 1 -- 172.21.15.144:0/854582531 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3009fe60 con 0x7fcd30098550 2024-09-18T12:45:01.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2e7fc640 1 -- 172.21.15.144:0/854582531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098550 msgr2=0x7fcd300674d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2e7fc640 1 --2- 172.21.15.144:0/854582531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098550 0x7fcd300674d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcd300a2b40 tx=0x7fcd180091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aa660121741fc4f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.067 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 1 -- 172.21.15.144:0/854582531 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18032440 con 0x7fcd30098550 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: dump: 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.068 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 1 -- 172.21.15.144:0/854582531 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd180325a0 con 0x7fcd30098550 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd35646640 5 monclient: authenticate success, global_id 234190 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd35646640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd35646640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd35646640 4 set_mon_vals no callback set 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 1 -- 172.21.15.144:0/854582531 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18032940 con 0x7fcd30098550 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.057+0000 7fcd2dffb640 10 monclient: dump: 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.069 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 10 monclient: shutdown 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:01.070 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 -- 172.21.15.144:0/854582531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098550 msgr2=0x7fcd300674d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 --2- 172.21.15.144:0/854582531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098550 0x7fcd300674d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcd300a2b40 tx=0x7fcd180091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 -- 172.21.15.144:0/854582531 shutdown_connections 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 --2- 172.21.15.144:0/854582531 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098950 0x7fcd30067a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 --2- 172.21.15.144:0/854582531 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098550 0x7fcd300674d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 --2- 172.21.15.144:0/854582531 >> v2:172.21.15.116:3300/0 conn(0x7fcd30098180 0x7fcd301321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 -- 172.21.15.144:0/854582531 >> 172.21.15.144:0/854582531 conn(0x7fcd300a57e0 msgr2=0x7fcd300a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 -- 172.21.15.144:0/854582531 shutdown_connections 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.058+0000 7fcd35646640 1 -- 172.21.15.144:0/854582531 wait complete. 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient: build_initial_monmap 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient: monmap: 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:01.071 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 Processor -- start 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 -- start start 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient: init 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:01.072 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd30098180 0x7fcd3009b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): picked mon.noname-a con 0x7fcd30098180 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098550 0x7fcd30130b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): picked mon.noname-c con 0x7fcd30098550 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 0x7fcd30140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): picked mon.noname-b con 0x7fcd30098950 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3011df90 con 0x7fcd30098180 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.073 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3011c930 con 0x7fcd30098950 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3011dcb0 con 0x7fcd30098550 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.059+0000 7fcd35646640 10 monclient(hunting): authenticate will time out at 1936.628912s 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2e7fc640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098550 msgr2=0x7fcd30130b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 0x7fcd30140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2e7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098550 0x7fcd30130b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 0x7fcd30140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39526/0 (socket says 172.21.15.144:39526) 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 -- 172.21.15.144:0/1933756120 learned_addr learned my addr 172.21.15.144:0/1933756120 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): get_auth_request con 0x7fcd30098950 auth_method 0 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.074 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd30098180 0x7fcd3009b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 10 monclient(hunting): get_auth_request con 0x7fcd30098180 auth_method 0 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.075 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient(hunting): handle_auth_done global_id 234193 payload 824 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 -- 172.21.15.144:0/1933756120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098550 msgr2=0x7fcd30130b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098550 0x7fcd30130b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 -- 172.21.15.144:0/1933756120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd30098180 msgr2=0x7fcd3009b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd30098180 0x7fcd3009b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:01.076 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.060+0000 7fcd2f7fe640 1 -- 172.21.15.144:0/1933756120 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18009450 con 0x7fcd30098950 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.062+0000 7fcd2f7fe640 1 -- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 msgr2=0x7fcd30140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.062+0000 7fcd2f7fe640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 0x7fcd30140ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800bb10 tx=0x7fcd280097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.062+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800cbc0 con 0x7fcd30098950 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.062+0000 7fcd26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 10 monclient: dump: 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.077 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2800cd20 con 0x7fcd30098950 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd35646640 5 monclient: authenticate success, global_id 234193 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd35646640 10 monclient: _renew_subs 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd35646640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.078 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3011e270 con 0x7fcd30098950 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28006150 con 0x7fcd30098950 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 10 monclient: dump: 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.079 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd2d7fa640 4 set_mon_vals no callback set 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd35646640 10 monclient: _renew_subs 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd35646640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3011cf60 con 0x7fcd30098950 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.063+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fcd28012020 con 0x7fcd30098950 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd35646640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd35646640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd35646640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd30066600 con 0x7fcd30098950 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd26ffd640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcd10038440 0x7fcd1003a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.080 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fcd2804c240 con 0x7fcd30098950 2024-09-18T12:45:01.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd2effd640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcd10038440 0x7fcd1003a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.064+0000 7fcd2effd640 10 monclient: get_auth_request con 0x7fcd10038440 auth_method 0 2024-09-18T12:45:01.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.066+0000 7fcd2effd640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcd10038440 0x7fcd1003a8f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009d20 tx=0x7fcd200090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.068+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fcd2801dae0 con 0x7fcd30098950 2024-09-18T12:45:01.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.068+0000 7fcd26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.081 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.068+0000 7fcd26ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:01.282 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.281+0000 7fcd35646640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-18T12:45:01.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.281+0000 7fcd35646640 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-18T12:45:01.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.281+0000 7fcd35646640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.283 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.281+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7fcd301340e0 con 0x7fcd30098950 2024-09-18T12:45:01.284 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:01.283+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-18T12:45:01.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.286+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "auth import"}]=0 v35) v1 ==== 59+0+0 (secure 0 0 0) 0x7fcd2801b030 con 0x7fcd30098950 2024-09-18T12:45:01.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.286+0000 7fcd26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-18T12:45:01.287 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.286+0000 7fcd26ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:01.434 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.433+0000 7fcd35646640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-18T12:45:01.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.433+0000 7fcd35646640 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-18T12:45:01.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.433+0000 7fcd35646640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.435 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.433+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7fcd300008d0 con 0x7fcd30098950 2024-09-18T12:45:01.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.438+0000 7fcd26ffd640 1 -- 172.21.15.144:0/1933756120 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "auth import"}]=0 v36) v1 ==== 59+0+0 (secure 0 0 0) 0x7fcd2801b020 con 0x7fcd30098950 2024-09-18T12:45:01.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.438+0000 7fcd26ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-18T12:45:01.439 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.438+0000 7fcd26ffd640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:45:01.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcd10038440 msgr2=0x7fcd1003a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcd10038440 0x7fcd1003a8f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009d20 tx=0x7fcd200090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.441 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 10 monclient: shutdown 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 msgr2=0x7fcd30140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 0x7fcd30140ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800bb10 tx=0x7fcd280097d0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 shutdown_connections 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fcd10038440 0x7fcd1003a8f0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd30098950 0x7fcd30140ed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd30098550 0x7fcd30130b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 --2- 172.21.15.144:0/1933756120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd30098180 0x7fcd3009b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 >> 172.21.15.144:0/1933756120 conn(0x7fcd300a57e0 msgr2=0x7fcd3009eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 shutdown_connections 2024-09-18T12:45:01.442 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.440+0000 7fcd35646640 1 -- 172.21.15.144:0/1933756120 wait complete. 2024-09-18T12:45:01.451 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph health 2024-09-18T12:45:01.451 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep AUTH_BAD_CAP 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.516+0000 7eff46e8b640 10 monclient: get_monmap_and_config 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.516+0000 7eff46e8b640 10 monclient: build_initial_monmap 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.516+0000 7eff46e8b640 10 monclient: monmap: 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:45:01.517327+0000 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:45:01.517327+0000 2024-09-18T12:45:01.517 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:01.518 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.518 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:01.518 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:01.518 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:01.518 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:01.518 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:01.518 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 1 Processor -- start 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 1 -- start start 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient: init 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff401302d0 0x7eff4009ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient(hunting): picked mon.noname-b con 0x7eff401302d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff4009b760 0x7eff4009d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient(hunting): picked mon.noname-a con 0x7eff4009b760 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff4009d5b0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff40097850 con 0x7eff4009d5b0 2024-09-18T12:45:01.521 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff400980f0 con 0x7eff4009b760 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff400983b0 con 0x7eff401302d0 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff46e8b640 10 monclient(hunting): authenticate will time out at 1937.086706s 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff44c27640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff401302d0 msgr2=0x7eff4009ac70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff44c27640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff401302d0 0x7eff4009ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff37fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff4009b760 0x7eff4009d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff37fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff4009b760 0x7eff4009d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.144:58692/0 (socket says 172.21.15.144:58692) 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.517+0000 7eff45428640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7eff4009d5b0 0x7eff40136b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.144:43380/0 (socket says 172.21.15.144:43380) 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 1 -- 172.21.15.144:0/1509498235 learned_addr learned my addr 172.21.15.144:0/1509498235 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): get_auth_request con 0x7eff4009b760 auth_method 0 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.522 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient(hunting): handle_auth_done global_id 194607 payload 274 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 1 -- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff401302d0 msgr2=0x7eff4009ac70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 1 --2- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff401302d0 0x7eff4009ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 1 -- 172.21.15.144:0/1509498235 >> v1:172.21.15.116:6789/0 conn(0x7eff4009d5b0 legacy=0x7eff40136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 1 monclient: found mon.noname-a 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:01.523 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:01.524 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.518+0000 7eff37fff640 1 -- 172.21.15.144:0/1509498235 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4009e870 con 0x7eff4009b760 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff37fff640 1 -- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 msgr2=0x7eff4009d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff37fff640 1 --2- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 0x7eff4009d070 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7eff30009910 tx=0x7eff300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ce0a6412e2e325 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 1 -- 172.21.15.144:0/1509498235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30032440 con 0x7eff4009b760 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: dump: 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.530 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: _finish_auth 0 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 1 -- 172.21.15.144:0/1509498235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff300325a0 con 0x7eff4009b760 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 1 -- 172.21.15.144:0/1509498235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30032940 con 0x7eff4009b760 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:01.531 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff377fe640 10 monclient: dump: 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 5 monclient: authenticate success, global_id 194607 2024-09-18T12:45:01.532 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 4 set_mon_vals no callback set 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 10 monclient: shutdown 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 1 -- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 msgr2=0x7eff4009d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 1 --2- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 0x7eff4009d070 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7eff30009910 tx=0x7eff300091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 1 -- 172.21.15.144:0/1509498235 shutdown_connections 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 1 --2- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 0x7eff4009d070 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 1 --2- 172.21.15.144:0/1509498235 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff401302d0 0x7eff4009ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 1 -- 172.21.15.144:0/1509498235 >> 172.21.15.144:0/1509498235 conn(0x7eff400a20c0 msgr2=0x7eff4009e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.521+0000 7eff46e8b640 1 -- 172.21.15.144:0/1509498235 shutdown_connections 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.522+0000 7eff46e8b640 1 -- 172.21.15.144:0/1509498235 wait complete. 2024-09-18T12:45:01.533 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.522+0000 7eff46e8b640 10 monclient: build_initial_monmap 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.522+0000 7eff46e8b640 10 monclient: monmap: 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:01.534 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.522+0000 7eff46e8b640 1 Processor -- start 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.522+0000 7eff46e8b640 1 -- start start 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.522+0000 7eff46e8b640 10 monclient: init 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 0x7eff4006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): picked mon.noname-a con 0x7eff4009b760 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff4009d5b0 0x7eff40126c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.535 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): picked mon.noname-c con 0x7eff4009d5b0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 0x7eff4012ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): picked mon.noname-b con 0x7eff401302d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff40127e20 con 0x7eff4009b760 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4006f310 con 0x7eff401302d0 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff40128f90 con 0x7eff4009d5b0 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff46e8b640 10 monclient(hunting): authenticate will time out at 1937.092386s 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff37fff640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff4009d5b0 msgr2=0x7eff40126c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff37fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff4009d5b0 0x7eff40126c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:01.536 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 0x7eff4012ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 0x7eff4012ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39542/0 (socket says 172.21.15.144:39542) 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 -- 172.21.15.144:0/3551842955 learned_addr learned my addr 172.21.15.144:0/3551842955 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): get_auth_request con 0x7eff401302d0 auth_method 0 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 0x7eff4006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 10 monclient(hunting): get_auth_request con 0x7eff4009b760 auth_method 0 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.537 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff44c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient(hunting): handle_auth_done global_id 234196 payload 824 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 -- 172.21.15.144:0/3551842955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff4009d5b0 msgr2=0x7eff40126c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff4009d5b0 0x7eff40126c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 -- 172.21.15.144:0/3551842955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 msgr2=0x7eff4006e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 0x7eff4006e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient: _finish_hunting 0 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 monclient: found mon.noname-b 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:01.538 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.523+0000 7eff45428640 1 -- 172.21.15.144:0/3551842955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30009450 con 0x7eff401302d0 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff45428640 1 -- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 msgr2=0x7eff4012ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff45428640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 0x7eff4012ea90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c880 tx=0x7eff38004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3800ce70 con 0x7eff401302d0 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: dump: 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.539 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: _finish_auth 0 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff46e8b640 5 monclient: authenticate success, global_id 234196 2024-09-18T12:45:01.540 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3800a640 con 0x7eff401302d0 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff46e8b640 10 monclient: _renew_subs 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3800aa00 con 0x7eff401302d0 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff46e8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff40128c80 con 0x7eff401302d0 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff34ff9640 10 monclient: dump: 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.541 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff36ffd640 4 set_mon_vals no callback set 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff46e8b640 10 monclient: _renew_subs 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff46e8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.526+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff40127150 con 0x7eff401302d0 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7eff380070f0 con 0x7eff401302d0 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff34ff9640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff2c038440 0x7eff2c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7eff3804c050 con 0x7eff401302d0 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff46e8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff46e8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff46e8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.542 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff401290d0 con 0x7eff401302d0 2024-09-18T12:45:01.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff44c27640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff2c038440 0x7eff2c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.527+0000 7eff44c27640 10 monclient: get_auth_request con 0x7eff2c038440 auth_method 0 2024-09-18T12:45:01.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.529+0000 7eff44c27640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff2c038440 0x7eff2c03a8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0038f0 tx=0x7eff3c0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.529+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:45:01.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.531+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7eff38010d70 con 0x7eff401302d0 2024-09-18T12:45:01.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.531+0000 7eff34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.543 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.531+0000 7eff34ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:01.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.718+0000 7eff46e8b640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-18T12:45:01.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.718+0000 7eff46e8b640 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-18T12:45:01.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.718+0000 7eff46e8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.718+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7eff40066600 con 0x7eff401302d0 2024-09-18T12:45:01.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.719+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+126 (secure 0 0 0) 0x7eff3802a090 con 0x7eff401302d0 2024-09-18T12:45:01.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.719+0000 7eff34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-18T12:45:01.720 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.719+0000 7eff34ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:01.842 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.841+0000 7eff46e8b640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-18T12:45:01.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.841+0000 7eff46e8b640 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-18T12:45:01.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.841+0000 7eff46e8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.841+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7eff400008d0 con 0x7eff401302d0 2024-09-18T12:45:01.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.842+0000 7eff34ff9640 1 -- 172.21.15.144:0/3551842955 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+126 (secure 0 0 0) 0x7eff3802a020 con 0x7eff401302d0 2024-09-18T12:45:01.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.842+0000 7eff34ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-18T12:45:01.843 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.842+0000 7eff34ff9640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff2c038440 msgr2=0x7eff2c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff2c038440 0x7eff2c03a8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0038f0 tx=0x7eff3c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 10 monclient: shutdown 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 msgr2=0x7eff4012ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 0x7eff4012ea90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c880 tx=0x7eff38004ef0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 shutdown_connections 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7eff2c038440 0x7eff2c03a8f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7eff401302d0 0x7eff4012ea90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7eff4009d5b0 0x7eff40126c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 --2- 172.21.15.144:0/3551842955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff4009b760 0x7eff4006e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.845 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.843+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 >> 172.21.15.144:0/3551842955 conn(0x7eff400a20c0 msgr2=0x7eff401329c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:01.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.844+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 shutdown_connections 2024-09-18T12:45:01.846 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.844+0000 7eff46e8b640 1 -- 172.21.15.144:0/3551842955 wait complete. 2024-09-18T12:45:01.854 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph health detail 2024-09-18T12:45:01.854 INFO:tasks.workunit.client.0.smithi144.stderr:+ grep client.bad 2024-09-18T12:45:01.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.920+0000 7f4718e94640 10 monclient: get_monmap_and_config 2024-09-18T12:45:01.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.920+0000 7f4718e94640 10 monclient: build_initial_monmap 2024-09-18T12:45:01.921 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.920+0000 7f4718e94640 10 monclient: monmap: 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:45:01.921188+0000 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:45:01.921188+0000 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.920+0000 7f4718e94640 1 Processor -- start 2024-09-18T12:45:01.922 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.920+0000 7f4718e94640 1 -- start start 2024-09-18T12:45:01.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.920+0000 7f4718e94640 10 monclient: init 2024-09-18T12:45:01.923 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 0x7f4714097460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient(hunting): picked mon.noname-b con 0x7f4714131bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4714131f90 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4714132360 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47140a15e0 con 0x7f4714132360 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47140a1300 con 0x7f4714131f90 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47141327a0 con 0x7f4714131bc0 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4718e94640 10 monclient(hunting): authenticate will time out at 1937.490495s 2024-09-18T12:45:01.924 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4711d9b640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f4714131f90 0x7f47140674d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:47564/0 (socket says 172.21.15.144:47564) 2024-09-18T12:45:01.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f471259c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 msgr2=0x7f4714097460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:01.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f4711d9b640 1 -- 172.21.15.144:0/665465957 learned_addr learned my addr 172.21.15.144:0/665465957 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-09-18T12:45:01.925 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.921+0000 7f471259c640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 0x7f4714097460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:01.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.923+0000 7f4711d9b640 1 -- 172.21.15.144:0/665465957 >> v1:172.21.15.144:6789/0 conn(0x7f4714131f90 legacy=0x7f47140674d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.923+0000 7f4712d9d640 1 -- 172.21.15.144:0/665465957 >> v1:172.21.15.116:6789/0 conn(0x7f4714132360 legacy=0x7f4714067be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.932 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.923+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272541384 0 0) 0x7f47140a1300 con 0x7f4714131f90 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): my global_id is 234199 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46fc0034e0 con 0x7f4714131f90 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4108969023 0 0) 0x7f46fc0034e0 con 0x7f4714131f90 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 msgr2=0x7f4714097460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 0x7f4714097460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 >> v1:172.21.15.116:6789/0 conn(0x7f4714132360 legacy=0x7f4714067be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: _finish_hunting 0 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:45:01.933 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471409fed0 con 0x7f4714131f90 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: _finish_auth 0 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f47080017a0 con 0x7f4714131f90 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4718e94640 5 monclient: authenticate success, global_id 234199 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4718e94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: dump: 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.934 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 >> v1:172.21.15.144:6789/0 conn(0x7f4714131f90 legacy=0x7f47140674d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46fc003a50 0x7f46fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): picked mon.a con 0x7f46fc003a50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 0x7f46fc004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): picked mon.c con 0x7f4714131bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:01.935 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46fc004ec0 0x7f46fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): picked mon.b con 0x7f46fc004ec0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47141327a0 con 0x7f46fc003a50 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc005560 con 0x7f46fc004ec0 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc00bf10 con 0x7f4714131bc0 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471159a640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4718e94640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4711d9b640 1 -- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 msgr2=0x7f46fc004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4711d9b640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 0x7f46fc004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:01.936 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46fc004ec0 0x7f46fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): get_auth_request con 0x7f46fc004ec0 auth_method 0 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46fc003a50 0x7f46fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 10 monclient(hunting): get_auth_request con 0x7f46fc003a50 auth_method 0 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.937 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f471259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient(hunting): handle_auth_done global_id 234199 payload 293 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 1 -- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 msgr2=0x7f46fc004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4714131bc0 0x7f46fc004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 1 -- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46fc003a50 msgr2=0x7f46fc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46fc003a50 0x7f46fc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 1 monclient: found mon.b 2024-09-18T12:45:01.938 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 1 -- 172.21.15.144:0/665465957 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0034e0 con 0x7f46fc004ec0 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 10 monclient: _finish_auth 0 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.924+0000 7f4712d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f4712d9d640 1 -- 172.21.15.144:0/665465957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46fc004ec0 msgr2=0x7f46fc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f4712d9d640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46fc004ec0 0x7f46fc00b9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4704001bb0 tx=0x7f4704007c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f10c5745fe3dd5a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470402ad30 con 0x7f46fc004ec0 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 10 monclient: dump: 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.939 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47040070f0 con 0x7f46fc004ec0 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f4718e94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f4718e94640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:01.940 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 1 -- 172.21.15.144:0/665465957 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47040074b0 con 0x7f46fc004ec0 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f4718e94640 4 set_mon_vals no callback set 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.927+0000 7f471159a640 10 monclient: dump: 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.941 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 10 monclient: shutdown 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 1 -- 172.21.15.144:0/665465957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46fc004ec0 msgr2=0x7f46fc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 1 --2- 172.21.15.144:0/665465957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46fc004ec0 0x7f46fc00b9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4704001bb0 tx=0x7f4704007c00 comp rx=0 tx=0).stop 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4712d9d640 1 -- 172.21.15.144:0/665465957 reap_dead start 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 1 -- 172.21.15.144:0/665465957 shutdown_connections 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 1 -- 172.21.15.144:0/665465957 >> 172.21.15.144:0/665465957 conn(0x7f47140a51d0 msgr2=0x7f47140a55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 1 -- 172.21.15.144:0/665465957 shutdown_connections 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 1 -- 172.21.15.144:0/665465957 wait complete. 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 10 monclient: build_initial_monmap 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.928+0000 7f4718e94640 10 monclient: monmap: 2024-09-18T12:45:01.942 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 Processor -- start 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 -- start start 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient: init 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:01.943 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46fc004ec0 0x7f4714140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): picked mon.noname-c con 0x7f46fc004ec0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4714130b10 0x7f4714130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): picked mon.noname-a con 0x7f4714130b10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 0x7f471411da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): picked mon.noname-b con 0x7f471411d610 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714098e40 con 0x7f4714130b10 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714141cf0 con 0x7f471411d610 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4714120c60 con 0x7f46fc004ec0 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:01.944 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4718e94640 10 monclient(hunting): authenticate will time out at 1937.498658s 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f471259c640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46fc004ec0 msgr2=0x7f4714140e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 0x7f471411da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f471259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46fc004ec0 0x7f4714140e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 0x7f471411da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39560/0 (socket says 172.21.15.144:39560) 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 1 -- 172.21.15.144:0/3252860692 learned_addr learned my addr 172.21.15.144:0/3252860692 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 10 monclient(hunting): get_auth_request con 0x7f471411d610 auth_method 0 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4711d9b640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4714130b10 0x7f4714130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.945 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4712d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4711d9b640 10 monclient(hunting): get_auth_request con 0x7f4714130b10 auth_method 0 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4711d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4711d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4711d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4711d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.929+0000 7f4711d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4711d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 10 monclient(hunting): handle_auth_done global_id 234202 payload 824 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 1 -- 172.21.15.144:0/3252860692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46fc004ec0 msgr2=0x7f4714140e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46fc004ec0 0x7f4714140e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 1 -- 172.21.15.144:0/3252860692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4714130b10 msgr2=0x7f4714130f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4714130b10 0x7f4714130f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 1 monclient: found mon.noname-b 2024-09-18T12:45:01.946 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:01.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.947 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.930+0000 7f4712d9d640 1 -- 172.21.15.144:0/3252860692 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714120f40 con 0x7f471411d610 2024-09-18T12:45:01.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4712d9d640 1 -- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 msgr2=0x7f471411da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:01.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4712d9d640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 0x7f471411da80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4704001be0 tx=0x7f4704030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470403b070 con 0x7f471411d610 2024-09-18T12:45:01.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:01.949 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: dump: 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47040323d0 con 0x7f471411d610 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4718e94640 5 monclient: authenticate success, global_id 234202 2024-09-18T12:45:01.950 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4718e94640 10 monclient: _renew_subs 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704032750 con 0x7f471411d610 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4718e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47141413b0 con 0x7f471411d610 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f47027fc640 10 monclient: dump: 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:01.951 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4710d99640 4 set_mon_vals no callback set 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4718e94640 10 monclient: _renew_subs 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4718e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.932+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471411fd70 con 0x7f471411d610 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f4704030070 con 0x7f471411d610 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f4718e94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f4718e94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f4718e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471411ff90 con 0x7f471411d610 2024-09-18T12:45:01.952 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f47027fc640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46f4038440 0x7f46f403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:01.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f470406ae40 con 0x7f471411d610 2024-09-18T12:45:01.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f471259c640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46f4038440 0x7f46f403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:01.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.933+0000 7f471259c640 10 monclient: get_auth_request con 0x7f46f4038440 auth_method 0 2024-09-18T12:45:01.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.935+0000 7f471259c640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46f4038440 0x7f46f403a8f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f470c003c70 tx=0x7f470c0095d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:01.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.937+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f470402f8d0 con 0x7f471411d610 2024-09-18T12:45:01.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.937+0000 7f47027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:01.953 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:01.937+0000 7f47027fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:02.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.150+0000 7f4718e94640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-18T12:45:02.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.150+0000 7f4718e94640 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-18T12:45:02.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.150+0000 7f4718e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.150+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4714066600 con 0x7f471411d610 2024-09-18T12:45:02.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.150+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1208 (secure 0 0 0) 0x7f470402f280 con 0x7f471411d610 2024-09-18T12:45:02.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.150+0000 7f47027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-18T12:45:02.151 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.150+0000 7f47027fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:02.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.299+0000 7f4718e94640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-18T12:45:02.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.299+0000 7f4718e94640 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-18T12:45:02.300 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.299+0000 7f4718e94640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.299+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f47140682f0 con 0x7f471411d610 2024-09-18T12:45:02.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.299+0000 7f47027fc640 1 -- 172.21.15.144:0/3252860692 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1208 (secure 0 0 0) 0x7f470402f8d0 con 0x7f471411d610 2024-09-18T12:45:02.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.299+0000 7f47027fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-18T12:45:02.301 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.299+0000 7f47027fc640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:45:02.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46f4038440 msgr2=0x7f46f403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:02.304 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46f4038440 0x7f46f403a8f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f470c003c70 tx=0x7f470c0095d0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 10 monclient: shutdown 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 msgr2=0x7f471411da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 0x7f471411da80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4704001be0 tx=0x7f4704030000 comp rx=0 tx=0).stop 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 shutdown_connections 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46f4038440 0x7f46f403a8f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f471411d610 0x7f471411da80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4714130b10 0x7f4714130f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 --2- 172.21.15.144:0/3252860692 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46fc004ec0 0x7f4714140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 >> 172.21.15.144:0/3252860692 conn(0x7f47140a51d0 msgr2=0x7f471409b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 shutdown_connections 2024-09-18T12:45:02.305 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.301+0000 7f4718e94640 1 -- 172.21.15.144:0/3252860692 wait complete. 2024-09-18T12:45:02.311 INFO:tasks.workunit.client.0.smithi144.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-18T12:45:02.311 INFO:tasks.workunit.client.0.smithi144.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-18T12:45:02.311 INFO:tasks.workunit.client.0.smithi144.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-18T12:45:02.311 INFO:tasks.workunit.client.0.smithi144.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-18T12:45:02.311 INFO:tasks.workunit.client.0.smithi144.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-18T12:45:02.311 INFO:tasks.workunit.client.0.smithi144.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-18T12:45:02.311 INFO:tasks.workunit.client.0.smithi144.stderr:+ ceph auth rm client.bad 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.379+0000 7f6b4f35d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.379+0000 7f6b4f35d640 10 monclient: build_initial_monmap 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.379+0000 7f6b4f35d640 10 monclient: monmap: 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:45:02.380134+0000 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:45:02.380134+0000 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:02.380 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:02.381 INFO:tasks.workunit.client.0.smithi144.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:02.381 INFO:tasks.workunit.client.0.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:02.381 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:02.381 INFO:tasks.workunit.client.0.smithi144.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:02.381 INFO:tasks.workunit.client.0.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:02.381 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:02.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.379+0000 7f6b4f35d640 1 Processor -- start 2024-09-18T12:45:02.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.379+0000 7f6b4f35d640 1 -- start start 2024-09-18T12:45:02.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.379+0000 7f6b4f35d640 10 monclient: init 2024-09-18T12:45:02.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:02.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T12:45:02.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b48131bc0 0x7f6b48097460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:02.383 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): picked mon.noname-b con 0x7f6b48131bc0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b48131f90 0x7f6b48132360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): picked mon.noname-a con 0x7f6b48131f90 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b480674d0 0x7f6b48067910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): picked mon.noname-c con 0x7f6b480674d0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b480a15e0 con 0x7f6b48131f90 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b480a1300 con 0x7f6b480674d0 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4809fed0 con 0x7f6b48131bc0 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4f35d640 10 monclient(hunting): authenticate will time out at 1937.949437s 2024-09-18T12:45:02.384 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d0f9640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b48131bc0 msgr2=0x7f6b48097460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b480674d0 0x7f6b48067910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d0f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b48131bc0 0x7f6b48097460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b480674d0 0x7f6b48067910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39568/0 (socket says 172.21.15.144:39568) 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2726128156 learned_addr learned my addr 172.21.15.144:0/2726128156 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): get_auth_request con 0x7f6b480674d0 auth_method 0 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 1 --2- 172.21.15.144:0/2726128156 >> v2:172.21.15.116:3300/0 conn(0x7f6b48131f90 0x7f6b48132360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 10 monclient(hunting): get_auth_request con 0x7f6b48131f90 auth_method 0 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:02.385 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4c8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_done global_id 234205 payload 274 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2726128156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b48131bc0 msgr2=0x7f6b48097460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 --2- 172.21.15.144:0/2726128156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b48131bc0 0x7f6b48097460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2726128156 >> v2:172.21.15.116:3300/0 conn(0x7f6b48131f90 msgr2=0x7f6b48132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 --2- 172.21.15.144:0/2726128156 >> v2:172.21.15.116:3300/0 conn(0x7f6b48131f90 0x7f6b48132360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient: _finish_hunting 0 2024-09-18T12:45:02.386 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 monclient: found mon.noname-c 2024-09-18T12:45:02.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:02.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.387 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.380+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2726128156 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b480979a0 con 0x7f6b480674d0 2024-09-18T12:45:02.392 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2726128156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480674d0 msgr2=0x7f6b48067910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b4d8fa640 1 --2- 172.21.15.144:0/2726128156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480674d0 0x7f6b48067910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6b48132960 tx=0x7f6b40009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=439cb2ec3adbc82b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 1 -- 172.21.15.144:0/2726128156 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40007610 con 0x7f6b480674d0 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: dump: 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:02.393 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: _finish_auth 0 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b4f35d640 5 monclient: authenticate success, global_id 234205 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 1 -- 172.21.15.144:0/2726128156 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4002d050 con 0x7f6b480674d0 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b4f35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b4f35d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b4f35d640 4 set_mon_vals no callback set 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 1 -- 172.21.15.144:0/2726128156 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400079a0 con 0x7f6b480674d0 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:02.394 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.383+0000 7f6b3bfff640 10 monclient: dump: 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 10 monclient: shutdown 2024-09-18T12:45:02.395 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2726128156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480674d0 msgr2=0x7f6b48067910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2726128156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480674d0 0x7f6b48067910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6b48132960 tx=0x7f6b40009840 comp rx=0 tx=0).stop 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2726128156 shutdown_connections 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2726128156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480674d0 0x7f6b48067910 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2726128156 >> v2:172.21.15.116:3300/0 conn(0x7f6b48131f90 0x7f6b48132360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2726128156 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b48131bc0 0x7f6b48097460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2726128156 >> 172.21.15.144:0/2726128156 conn(0x7f6b480a51d0 msgr2=0x7f6b480a55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2726128156 shutdown_connections 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2726128156 wait complete. 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 10 monclient: build_initial_monmap 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.384+0000 7f6b4f35d640 10 monclient: monmap: 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 0 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:02.396 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 0.000000 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:created 0.000000 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 Processor -- start 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 -- start start 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient: init 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b480674d0 0x7f6b48130a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:02.397 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): picked mon.noname-c con 0x7f6b480674d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b48131bc0 0x7f6b4811d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): picked mon.noname-a con 0x7f6b48131bc0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 0x7f6b4811dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): picked mon.noname-b con 0x7f6b48131f90 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4811f840 con 0x7f6b48131bc0 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b48098e40 con 0x7f6b48131f90 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b480a3200 con 0x7f6b480674d0 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4f35d640 10 monclient(hunting): authenticate will time out at 1937.954853s 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4d0f9640 1 -- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b480674d0 msgr2=0x7f6b48130a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3301/0 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4d0f9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b480674d0 0x7f6b48130a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:02.398 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4d8fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 0x7f6b4811dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4d8fa640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 0x7f6b4811dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:39576/0 (socket says 172.21.15.144:39576) 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.385+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2894106290 learned_addr learned my addr 172.21.15.144:0/2894106290 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): get_auth_request con 0x7f6b48131f90 auth_method 0 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4c8f8640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b48131bc0 0x7f6b4811d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4c8f8640 10 monclient(hunting): get_auth_request con 0x7f6b48131bc0 auth_method 0 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4c8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4c8f8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:02.399 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4c8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient(hunting): handle_auth_done global_id 234208 payload 824 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b480674d0 msgr2=0x7f6b48130a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b480674d0 0x7f6b48130a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b48131bc0 msgr2=0x7f6b4811d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b48131bc0 0x7f6b4811d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient: _finish_hunting 0 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 1 monclient: found mon.noname-b 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2894106290 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40009ae0 con 0x7f6b48131f90 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.386+0000 7f6b4c8f8640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b48131bc0 0x7f6b4811d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4d8fa640 1 -- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 msgr2=0x7f6b4811dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4d8fa640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 0x7f6b4811dbc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6b4002abc0 tx=0x7f6b40036000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:02.400 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40007170 con 0x7f6b48131f90 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: dump: 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:02.401 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: _finish_auth 0 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b40032cc0 con 0x7f6b48131f90 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4f35d640 5 monclient: authenticate success, global_id 234208 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4003b680 con 0x7f6b48131f90 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4f35d640 10 monclient: _renew_subs 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4f35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4811fc10 con 0x7f6b48131f90 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b3b7fe640 4 set_mon_vals no callback set 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b397fa640 10 monclient: dump: 2024-09-18T12:45:02.402 INFO:tasks.workunit.client.0.smithi144.stderr:epoch 1 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:election_strategy: 1 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr: 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4f35d640 10 monclient: _renew_subs 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4f35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.388+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b481331b0 con 0x7f6b48131f90 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6b40003020 con 0x7f6b48131f90 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b4f35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:02.403 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b4f35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:02.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b4f35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b48066600 con 0x7f6b48131f90 2024-09-18T12:45:02.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b397fa640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b2c038440 0x7f6b2c03a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:02.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6b4006cc80 con 0x7f6b48131f90 2024-09-18T12:45:02.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b4d0f9640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b2c038440 0x7f6b2c03a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:02.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.389+0000 7f6b4d0f9640 10 monclient: get_auth_request con 0x7f6b2c038440 auth_method 0 2024-09-18T12:45:02.404 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.392+0000 7f6b4d0f9640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b2c038440 0x7f6b2c03a8f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0038f0 tx=0x7f6b3c006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:02.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.394+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6b4003b9f0 con 0x7f6b48131f90 2024-09-18T12:45:02.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.394+0000 7f6b397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:02.405 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.394+0000 7f6b397fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:02.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.608+0000 7f6b4f35d640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T12:45:02.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.608+0000 7f6b4f35d640 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T12:45:02.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.608+0000 7f6b4f35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.609 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.608+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7f6b48120fa0 con 0x7f6b48131f90 2024-09-18T12:45:02.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.613+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v37) v1 ==== 79+0+0 (secure 0 0 0) 0x7f6b4003f020 con 0x7f6b48131f90 2024-09-18T12:45:02.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.613+0000 7f6b397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T12:45:02.614 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.613+0000 7f6b397fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:02.766 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.765+0000 7f6b4f35d640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T12:45:02.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.765+0000 7f6b4f35d640 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T12:45:02.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.765+0000 7f6b4f35d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:02.804 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.765+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7f6b480008d0 con 0x7f6b48131f90 2024-09-18T12:45:02.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.766+0000 7f6b397fa640 1 -- 172.21.15.144:0/2894106290 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v37) v1 ==== 79+0+0 (secure 0 0 0) 0x7f6b40039300 con 0x7f6b48131f90 2024-09-18T12:45:02.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.767+0000 7f6b397fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T12:45:02.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.767+0000 7f6b397fa640 10 monclient: _finish_command 3 = system:0 2024-09-18T12:45:02.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b2c038440 msgr2=0x7f6b2c03a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:02.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b2c038440 0x7f6b2c03a8f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0038f0 tx=0x7f6b3c006d80 comp rx=0 tx=0).stop 2024-09-18T12:45:02.805 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 10 monclient: shutdown 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 msgr2=0x7f6b4811dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 0x7f6b4811dbc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6b4002abc0 tx=0x7f6b40036000 comp rx=0 tx=0).stop 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 shutdown_connections 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b2c038440 0x7f6b2c03a8f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b48131f90 0x7f6b4811dbc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b48131bc0 0x7f6b4811d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 --2- 172.21.15.144:0/2894106290 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b480674d0 0x7f6b48130a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.768+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 >> 172.21.15.144:0/2894106290 conn(0x7f6b480a51d0 msgr2=0x7f6b480a3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.769+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 shutdown_connections 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:2024-09-18T12:45:02.769+0000 7f6b4f35d640 1 -- 172.21.15.144:0/2894106290 wait complete. 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-18T12:45:02.806 INFO:tasks.workunit.client.0.smithi144.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-18T12:45:02.807 INFO:tasks.workunit.client.0.smithi144.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-18T12:45:02.807 INFO:tasks.workunit.client.0.smithi144.stderr:+ expected_ret=1 2024-09-18T12:45:02.807 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-18T12:45:02.807 INFO:tasks.workunit.client.0.smithi144.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-18T12:45:03.081 INFO:tasks.workunit.client.0.smithi144.stderr:+ ret=1 2024-09-18T12:45:03.081 INFO:tasks.workunit.client.0.smithi144.stderr:+ [[ 1 -ne 1 ]] 2024-09-18T12:45:03.081 INFO:tasks.workunit.client.0.smithi144.stderr:+ return 0 2024-09-18T12:45:03.081 INFO:tasks.workunit.client.0.smithi144.stderr:+ echo OK 2024-09-18T12:45:03.082 INFO:tasks.workunit.client.0.smithi144.stdout:OK 2024-09-18T12:45:03.082 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T12:45:03.082 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T12:45:03.106 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-18T12:45:03.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T12:45:03.405 DEBUG:teuthology.parallel:result is None 2024-09-18T12:45:03.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:45:03.430 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:45:03.430 DEBUG:teuthology.orchestra.run.smithi144:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T12:45:03.497 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T12:45:03.498 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-18T12:45:03.507 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-18T12:45:14.320 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T12:45:14.320 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T12:45:14.321 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T12:45:14.321 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T12:45:14.324 INFO:tasks.ceph.mon.c:Started 2024-09-18T12:45:14.324 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:45:14.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:45:14.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.427+0000 7fb283ed4640 10 monclient: get_monmap_and_config 2024-09-18T12:45:14.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.427+0000 7fb283ed4640 10 monclient: build_initial_monmap 2024-09-18T12:45:14.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.427+0000 7fb283ed4640 10 monclient: monmap: 2024-09-18T12:45:14.428 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:14.428 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:14.428 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:14.428205+0000 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:14.428205+0000 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:14.429 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 1 Processor -- start 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 1 -- start start 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient: init 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb27c131ab0 0x7fb27c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient(hunting): picked mon.noname-c con 0x7fb27c131ab0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:14.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27c09e910 0x7fb27c09ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient(hunting): picked mon.noname-a con 0x7fb27c09e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb27c09f250 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c09faf0 con 0x7fb27c09f250 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c09fd30 con 0x7fb27c09e910 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c0a1110 con 0x7fb27c131ab0 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.428+0000 7fb283ed4640 10 monclient(hunting): authenticate will time out at 1941.749992s 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27c09e910 0x7fb27c09ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27c09e910 0x7fb27c09ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56446/0 (socket says 172.21.15.116:56446) 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 -- 172.21.15.116:0/2322941883 learned_addr learned my addr 172.21.15.116:0/2322941883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:14.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb282471640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb27c09f250 0x7fb27c068ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34602/0 (socket says 172.21.15.116:34602) 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): get_auth_request con 0x7fb27c09e910 auth_method 0 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb281c70640 1 --2- 172.21.15.116:0/2322941883 >> v2:172.21.15.144:3300/0 conn(0x7fb27c131ab0 0x7fb27c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb281c70640 10 monclient(hunting): get_auth_request con 0x7fb27c131ab0 auth_method 0 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb281c70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb281c70640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb281c70640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:14.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient(hunting): handle_auth_done global_id 194628 payload 274 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 -- 172.21.15.116:0/2322941883 >> v2:172.21.15.144:3300/0 conn(0x7fb27c131ab0 msgr2=0x7fb27c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 --2- 172.21.15.116:0/2322941883 >> v2:172.21.15.144:3300/0 conn(0x7fb27c131ab0 0x7fb27c131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 -- 172.21.15.116:0/2322941883 >> v1:172.21.15.116:6789/0 conn(0x7fb27c09f250 legacy=0x7fb27c068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient: _finish_hunting 0 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 monclient: found mon.noname-a 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:14.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.429+0000 7fb28146f640 1 -- 172.21.15.116:0/2322941883 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c0a13f0 con 0x7fb27c09e910 2024-09-18T12:45:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb28146f640 1 -- 172.21.15.116:0/2322941883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 msgr2=0x7fb27c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb28146f640 1 --2- 172.21.15.116:0/2322941883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c09ed10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb27c076760 tx=0x7fb26c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4c85986a7e1580 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 1 -- 172.21.15.116:0/2322941883 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c032440 con 0x7fb27c09e910 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: dump: 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: _finish_auth 0 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 5 monclient: authenticate success, global_id 194628 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 1 -- 172.21.15.116:0/2322941883 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c0325a0 con 0x7fb27c09e910 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 1 -- 172.21.15.116:0/2322941883 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c032940 con 0x7fb27c09e910 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb280c6e640 10 monclient: dump: 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 4 set_mon_vals no callback set 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 10 monclient: shutdown 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 1 -- 172.21.15.116:0/2322941883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 msgr2=0x7fb27c09ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 1 --2- 172.21.15.116:0/2322941883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c09ed10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb27c076760 tx=0x7fb26c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 1 -- 172.21.15.116:0/2322941883 shutdown_connections 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 1 --2- 172.21.15.116:0/2322941883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c09ed10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 1 --2- 172.21.15.116:0/2322941883 >> v2:172.21.15.144:3300/0 conn(0x7fb27c131ab0 0x7fb27c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.432+0000 7fb283ed4640 1 -- 172.21.15.116:0/2322941883 >> 172.21.15.116:0/2322941883 conn(0x7fb27c067d30 msgr2=0x7fb27c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.433+0000 7fb283ed4640 1 -- 172.21.15.116:0/2322941883 shutdown_connections 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.433+0000 7fb283ed4640 1 -- 172.21.15.116:0/2322941883 wait complete. 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.433+0000 7fb283ed4640 10 monclient: build_initial_monmap 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.433+0000 7fb283ed4640 10 monclient: monmap: 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.433+0000 7fb283ed4640 1 Processor -- start 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.433+0000 7fb283ed4640 1 -- start start 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.433+0000 7fb283ed4640 10 monclient: init 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c11e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): picked mon.noname-a con 0x7fb27c09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 0x7fb27c11e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): picked mon.noname-c con 0x7fb27c09f250 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb27c131ab0 0x7fb27c11eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): picked mon.noname-b con 0x7fb27c131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c130af0 con 0x7fb27c09e910 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1214c0 con 0x7fb27c131ab0 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c120400 con 0x7fb27c09f250 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb283ed4640 10 monclient(hunting): authenticate will time out at 1941.755261s 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c11e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c11e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56462/0 (socket says 172.21.15.116:56462) 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 -- 172.21.15.116:0/3554761738 learned_addr learned my addr 172.21.15.116:0/3554761738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): get_auth_request con 0x7fb27c09e910 auth_method 0 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 0x7fb27c11e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 10 monclient(hunting): get_auth_request con 0x7fb27c09f250 auth_method 0 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb27c131ab0 0x7fb27c11eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 1 -- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 msgr2=0x7fb27c11e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 1 -- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 msgr2=0x7fb27c11e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 0x7fb27c11e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 10 monclient(hunting): get_auth_request con 0x7fb27c131ab0 auth_method 0 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb28146f640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 0x7fb27c11e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb282471640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient(hunting): handle_auth_done global_id 194631 payload 824 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 -- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 msgr2=0x7fb27c11e5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 0x7fb27c11e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 -- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb27c131ab0 msgr2=0x7fb27c11eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb27c131ab0 0x7fb27c11eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient: _finish_hunting 0 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 monclient: found mon.noname-a 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.434+0000 7fb281c70640 1 -- 172.21.15.116:0/3554761738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c009450 con 0x7fb27c09e910 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.436+0000 7fb281c70640 1 -- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 msgr2=0x7fb27c11e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.436+0000 7fb281c70640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c11e0a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb27800d890 tx=0x7fb278008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 1 -- 172.21.15.116:0/3554761738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27800de20 con 0x7fb27c09e910 2024-09-18T12:45:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: dump: 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 1 -- 172.21.15.116:0/3554761738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27800bae0 con 0x7fb27c09e910 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb283ed4640 5 monclient: authenticate success, global_id 194631 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb283ed4640 10 monclient: _renew_subs 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb283ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c09bde0 con 0x7fb27c09e910 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 1 -- 172.21.15.116:0/3554761738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27800be80 con 0x7fb27c09e910 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb271ffb640 10 monclient: dump: 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb273fff640 4 set_mon_vals no callback set 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb283ed4640 10 monclient: _renew_subs 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb283ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.437+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c131550 con 0x7fb27c09e910 2024-09-18T12:45:14.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb271ffb640 1 -- 172.21.15.116:0/3554761738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb278014640 con 0x7fb27c09e910 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb283ed4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb283ed4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb283ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb271ffb640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb260038490 0x7fb26003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c121930 con 0x7fb27c09e910 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb271ffb640 1 -- 172.21.15.116:0/3554761738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb27804c950 con 0x7fb27c09e910 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb28146f640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb260038490 0x7fb26003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.438+0000 7fb28146f640 10 monclient: get_auth_request con 0x7fb260038490 auth_method 0 2024-09-18T12:45:14.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.441+0000 7fb28146f640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb260038490 0x7fb26003a940 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb27c06a250 tx=0x7fb26c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:14.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.442+0000 7fb271ffb640 1 -- 172.21.15.116:0/3554761738 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb27801e460 con 0x7fb27c09e910 2024-09-18T12:45:14.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.442+0000 7fb271ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:14.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.442+0000 7fb271ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.662+0000 7fb283ed4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:45:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.662+0000 7fb283ed4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.662+0000 7fb283ed4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.662+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb27c04bf60 con 0x7fb27c09e910 2024-09-18T12:45:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.662+0000 7fb271ffb640 1 -- 172.21.15.116:0/3554761738 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fb27801dbb0 con 0x7fb27c09e910 2024-09-18T12:45:14.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.662+0000 7fb271ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:14.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.662+0000 7fb271ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:14.664 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":192,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb260038490 msgr2=0x7fb26003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb260038490 0x7fb26003a940 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb27c06a250 tx=0x7fb26c032000 comp rx=0 tx=0).stop 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 10 monclient: shutdown 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 msgr2=0x7fb27c11e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c11e0a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb27800d890 tx=0x7fb278008040 comp rx=0 tx=0).stop 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 shutdown_connections 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb260038490 0x7fb26003a940 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb27c131ab0 0x7fb27c11eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb27c09f250 0x7fb27c11e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 --2- 172.21.15.116:0/3554761738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c09e910 0x7fb27c11e0a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:14.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 >> 172.21.15.116:0/3554761738 conn(0x7fb27c067d30 msgr2=0x7fb27c09d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:14.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 shutdown_connections 2024-09-18T12:45:14.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:14.664+0000 7fb283ed4640 1 -- 172.21.15.116:0/3554761738 wait complete. 2024-09-18T12:45:17.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:45:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.779+0000 7f66d98d4640 10 monclient: get_monmap_and_config 2024-09-18T12:45:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.779+0000 7f66d98d4640 10 monclient: build_initial_monmap 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.779+0000 7f66d98d4640 10 monclient: monmap: 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:17.781605+0000 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:17.781605+0000 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:17.782 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:17.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.780+0000 7f66d98d4640 1 Processor -- start 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.780+0000 7f66d98d4640 1 -- start start 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.780+0000 7f66d98d4640 10 monclient: init 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66d406e6f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66d406eac0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66d4137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient(hunting): picked mon.noname-b con 0x7f66d4135ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d409d730 con 0x7f66d406e6f0 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d409c610 con 0x7f66d406eac0 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d409b090 con 0x7f66d4135ab0 2024-09-18T12:45:17.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d98d4640 10 monclient(hunting): authenticate will time out at 1945.103221s 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66d4137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d2ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f66d406e6f0 0x7f66d412fc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59564/0 (socket says 172.21.15.116:59564) 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d2ffd640 1 -- 172.21.15.116:0/974927349 learned_addr learned my addr 172.21.15.116:0/974927349 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 10 monclient(hunting): get_auth_request con 0x7f66d4135ab0 auth_method 0 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 1 -- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 msgr2=0x7f66d4137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 1 -- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 msgr2=0x7f66d4137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66d4137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.781+0000 7f66d37fe640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66d4137e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:17.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d2ffd640 1 -- 172.21.15.116:0/974927349 >> v1:172.21.15.116:6789/0 conn(0x7f66d406e6f0 legacy=0x7f66d412fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428802219 0 0) 0x7f66d409d730 con 0x7f66d406e6f0 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient(hunting): my global_id is 194634 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66bc0034e0 con 0x7f66d406e6f0 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d27fc640 1 -- 172.21.15.116:0/974927349 >> v1:172.21.15.144:6789/0 conn(0x7f66d406eac0 legacy=0x7f66d406eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 785348799 0 0) 0x7f66bc0034e0 con 0x7f66d406e6f0 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 msgr2=0x7f66d4137e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66d4137e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 >> v1:172.21.15.144:6789/0 conn(0x7f66d406eac0 legacy=0x7f66d406eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient: _finish_hunting 0 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 monclient: found mon.noname-a-legacy 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.116:6789/0 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d409c890 con 0x7f66d406e6f0 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d98d4640 5 monclient: authenticate success, global_id 194634 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7f66c4001b20 con 0x7f66d406e6f0 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d98d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient: dump: 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] but i'm connected to v1:172.21.15.116:6789/0, reconnecting 2024-09-18T12:45:17.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.783+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 >> v1:172.21.15.116:6789/0 conn(0x7f66d406e6f0 legacy=0x7f66d412fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66bc003ba0 0x7f66bc003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): picked mon.a con 0x7f66bc003ba0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66bc004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): picked mon.c con 0x7f66d4135ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc004ee0 0x7f66bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): picked mon.b con 0x7f66bc004ee0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d409b090 con 0x7f66bc003ba0 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc00bf10 con 0x7f66bc004ee0 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc00c050 con 0x7f66d4135ab0 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d98d4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66bc003ba0 0x7f66bc003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): get_auth_request con 0x7f66bc003ba0 auth_method 0 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66bc004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 10 monclient(hunting): get_auth_request con 0x7f66d4135ab0 auth_method 0 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d37fe640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc004ee0 0x7f66bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d37fe640 10 monclient(hunting): get_auth_request con 0x7f66bc004ee0 auth_method 0 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient(hunting): handle_auth_done global_id 194634 payload 293 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 1 -- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 msgr2=0x7f66bc004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66bc004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 1 -- 172.21.15.116:0/974927349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc004ee0 msgr2=0x7f66bc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc004ee0 0x7f66bc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient: _finish_hunting 0 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 1 monclient: found mon.a 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 1 -- 172.21.15.116:0/974927349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc0034e0 con 0x7f66bc003ba0 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d27fc640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d4135ab0 0x7f66bc004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.784+0000 7f66d37fe640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc004ee0 0x7f66bc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d2ffd640 1 -- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66bc003ba0 msgr2=0x7f66bc003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:17.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d2ffd640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66bc003ba0 0x7f66bc003fb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f66c4001850 tx=0x7f66c400a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6de44b07968e4e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c402b670 con 0x7f66bc003ba0 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 10 monclient: dump: 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:17.803 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c402b7f0 con 0x7f66bc003ba0 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 1 -- 172.21.15.116:0/974927349 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c402bbb0 con 0x7f66bc003ba0 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 4 set_mon_vals no callback set 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d1ffb640 10 monclient: dump: 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 10 monclient: shutdown 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 1 -- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66bc003ba0 msgr2=0x7f66bc003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 1 --2- 172.21.15.116:0/974927349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66bc003ba0 0x7f66bc003fb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f66c4001850 tx=0x7f66c400a240 comp rx=0 tx=0).stop 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d37fe640 1 -- 172.21.15.116:0/974927349 reap_dead start 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 1 -- 172.21.15.116:0/974927349 shutdown_connections 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 1 -- 172.21.15.116:0/974927349 >> 172.21.15.116:0/974927349 conn(0x7f66d40672b0 msgr2=0x7f66d40676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 1 -- 172.21.15.116:0/974927349 shutdown_connections 2024-09-18T12:45:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.787+0000 7f66d98d4640 1 -- 172.21.15.116:0/974927349 wait complete. 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.788+0000 7f66d98d4640 10 monclient: build_initial_monmap 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.788+0000 7f66d98d4640 10 monclient: monmap: 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.788+0000 7f66d98d4640 1 Processor -- start 2024-09-18T12:45:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.788+0000 7f66d98d4640 1 -- start start 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.788+0000 7f66d98d4640 10 monclient: init 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc003ba0 0x7f66d4117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): picked mon.noname-b con 0x7f66bc003ba0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 0x7f66d4117d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): picked mon.noname-c con 0x7f66d41179c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66d412de00 0x7f66d412e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): picked mon.noname-a con 0x7f66d412de00 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d409fcf0 con 0x7f66d412de00 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d4119b00 con 0x7f66bc003ba0 2024-09-18T12:45:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d41182d0 con 0x7f66d41179c0 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d98d4640 10 monclient(hunting): authenticate will time out at 1945.111431s 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 0x7f66d4117d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66d412de00 0x7f66d412e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 0x7f66d4117d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:58140/0 (socket says 172.21.15.116:58140) 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 1 -- 172.21.15.116:0/2318037016 learned_addr learned my addr 172.21.15.116:0/2318037016 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 10 monclient(hunting): get_auth_request con 0x7f66d41179c0 auth_method 0 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): get_auth_request con 0x7f66d412de00 auth_method 0 2024-09-18T12:45:17.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 1 -- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 msgr2=0x7f66d4117d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 1 -- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 msgr2=0x7f66d4117d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 0x7f66d4117d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d27fc640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 0x7f66d4117d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc003ba0 0x7f66d4117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 10 monclient(hunting): get_auth_request con 0x7f66bc003ba0 auth_method 0 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient(hunting): handle_auth_done global_id 194637 payload 824 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 1 -- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 msgr2=0x7f66d4117d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 0x7f66d4117d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 1 -- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc003ba0 msgr2=0x7f66d4117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc003ba0 0x7f66d4117480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 1 monclient: found mon.noname-a 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.789+0000 7f66d37fe640 1 -- 172.21.15.116:0/2318037016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4119820 con 0x7f66d412de00 2024-09-18T12:45:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d37fe640 1 -- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66d412de00 msgr2=0x7f66d412e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d37fe640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66d412de00 0x7f66d412e230 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f66d409afb0 tx=0x7f66cc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 1 -- 172.21.15.116:0/2318037016 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc00fca0 con 0x7f66d412de00 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: dump: 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 1 -- 172.21.15.116:0/2318037016 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66cc00fe40 con 0x7f66d412de00 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d98d4640 5 monclient: authenticate success, global_id 194637 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d98d4640 10 monclient: _renew_subs 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d98d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d4119de0 con 0x7f66d412de00 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 1 -- 172.21.15.116:0/2318037016 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc0146b0 con 0x7f66d412de00 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66baffd640 10 monclient: dump: 2024-09-18T12:45:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d98d4640 10 monclient: _renew_subs 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d98d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d4135620 con 0x7f66d412de00 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.792+0000 7f66d17fa640 4 set_mon_vals no callback set 2024-09-18T12:45:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66baffd640 1 -- 172.21.15.116:0/2318037016 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f66cc014aa0 con 0x7f66d412de00 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66d98d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66d98d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66d98d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d41350d0 con 0x7f66d412de00 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66baffd640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66c0038490 0x7f66c003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66baffd640 1 -- 172.21.15.116:0/2318037016 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f66cc04ba30 con 0x7f66d412de00 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.793+0000 7f66d2ffd640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66c0038490 0x7f66c003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.794+0000 7f66d2ffd640 10 monclient: get_auth_request con 0x7f66c0038490 auth_method 0 2024-09-18T12:45:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.796+0000 7f66d2ffd640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66c0038490 0x7f66c003a940 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f66c4001d00 tx=0x7f66c4031000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.797+0000 7f66baffd640 1 -- 172.21.15.116:0/2318037016 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f66cc01a2c0 con 0x7f66d412de00 2024-09-18T12:45:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.797+0000 7f66baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:17.797+0000 7f66baffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.011+0000 7f66d98d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:45:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.011+0000 7f66d98d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.011+0000 7f66d98d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.011+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f66d404bf60 con 0x7f66d412de00 2024-09-18T12:45:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.012+0000 7f66baffd640 1 -- 172.21.15.116:0/2318037016 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f66cc01cdd0 con 0x7f66d412de00 2024-09-18T12:45:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.012+0000 7f66baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.012+0000 7f66baffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:18.014 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":192,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.013+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66c0038490 msgr2=0x7f66c003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66c0038490 0x7f66c003a940 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f66c4001d00 tx=0x7f66c4031000 comp rx=0 tx=0).stop 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 10 monclient: shutdown 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66d412de00 msgr2=0x7f66d412e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66d412de00 0x7f66d412e230 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f66d409afb0 tx=0x7f66cc008040 comp rx=0 tx=0).stop 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 shutdown_connections 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f66c0038490 0x7f66c003a940 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66d412de00 0x7f66d412e230 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f66d41179c0 0x7f66d4117d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 --2- 172.21.15.116:0/2318037016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f66bc003ba0 0x7f66d4117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:18.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 >> 172.21.15.116:0/2318037016 conn(0x7f66d40672b0 msgr2=0x7f66d4096c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:18.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 shutdown_connections 2024-09-18T12:45:18.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:18.014+0000 7f66d98d4640 1 -- 172.21.15.116:0/2318037016 wait complete. 2024-09-18T12:45:21.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:45:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.128+0000 7f9e918e8640 10 monclient: get_monmap_and_config 2024-09-18T12:45:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.128+0000 7f9e918e8640 10 monclient: build_initial_monmap 2024-09-18T12:45:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.129+0000 7f9e918e8640 10 monclient: monmap: 2024-09-18T12:45:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:21.130652+0000 2024-09-18T12:45:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:21.130652+0000 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.129+0000 7f9e918e8640 1 Processor -- start 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.129+0000 7f9e918e8640 1 -- start start 2024-09-18T12:45:21.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.129+0000 7f9e918e8640 10 monclient: init 2024-09-18T12:45:21.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.129+0000 7f9e918e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:21.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T12:45:21.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e8c0981c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e8c131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient(hunting): picked mon.noname-c con 0x7f9e8c132170 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8c09ca60 con 0x7f9e8c0981c0 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c131bd0 con 0x7f9e8c132170 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c09cca0 con 0x7f9e8c131da0 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e918e8640 10 monclient(hunting): authenticate will time out at 1948.452104s 2024-09-18T12:45:21.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9e8c0981c0 0x7f9e8c098590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59566/0 (socket says 172.21.15.116:59566) 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 learned_addr learned my addr 172.21.15.116:0/3116209421 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request con 0x7f9e8c131da0 auth_method 0 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e8c132170 auth_method 0 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:21.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.130+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:21.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.132+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v1:172.21.15.116:6789/0 conn(0x7f9e8c0981c0 legacy=0x7f9e8c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request con 0x7f9e8c131da0 auth_method 0 2024-09-18T12:45:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e8c132170 auth_method 0 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:21.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.331+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:45:21.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:21.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request con 0x7f9e8c131da0 auth_method 0 2024-09-18T12:45:21.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:21.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:21.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:21.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e8c132170 auth_method 0 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:21.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:21.732+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request con 0x7f9e8c131da0 auth_method 0 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:22.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e8c132170 auth_method 0 2024-09-18T12:45:22.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:22.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:22.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:22.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:22.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:22.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:22.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:22.533+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:45:24.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 10 monclient(hunting): tick 2024-09-18T12:45:24.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 monclient(hunting): continuing hunt 2024-09-18T12:45:24.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-18T12:45:24.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 msgr2=0x7f9e8c136b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:24.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c131da0 0x7f9e8c136b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 msgr2=0x7f9e8c1390c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.144:3300/0 conn(0x7f9e8c132170 0x7f9e8c1390c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 -- 172.21.15.116:0/3116209421 >> v1:172.21.15.116:6789/0 conn(0x7f9e8c0981c0 legacy=0x7f9e8c098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 10 monclient: _add_conns ranks=[3,1] 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f9e78003700 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e78006240 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 -- 172.21.15.116:0/3116209421 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8c09e010 con 0x7f9e78006240 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 1 -- 172.21.15.116:0/3116209421 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c131bd0 con 0x7f9e78003700 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e88ff9640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:24.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 10 monclient(hunting): get_auth_request con 0x7f9e78003700 auth_method 0 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 msgr2=0x7f9e78005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 msgr2=0x7f9e78005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.130+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:24.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.133+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3116209421 >> v1:172.21.15.144:6789/0 conn(0x7f9e78006240 legacy=0x7f9e780086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 10 monclient(hunting): get_auth_request con 0x7f9e78003700 auth_method 0 2024-09-18T12:45:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 msgr2=0x7f9e78005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:24.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 msgr2=0x7f9e78005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:24.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:24.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.331+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:45:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 10 monclient(hunting): get_auth_request con 0x7f9e78003700 auth_method 0 2024-09-18T12:45:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 msgr2=0x7f9e78005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 msgr2=0x7f9e78005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:24.732+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:45:25.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> v2:172.21.15.116:3300/0 conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): get_auth_request con 0x7f9e78003700 auth_method 0 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e89ffb640 1 -- 172.21.15.116:0/3116209421 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069774842 0 0) 0x7f9e8c09e010 con 0x7f9e78006240 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e89ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:25.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e89ffb640 10 monclient(hunting): my global_id is 234229 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e89ffb640 1 -- 172.21.15.116:0/3116209421 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e740034e0 con 0x7f9e78006240 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient(hunting): handle_auth_done global_id 194640 payload 274 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> v1:172.21.15.144:6789/0 conn(0x7f9e78006240 legacy=0x7f9e780086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient: _finish_hunting 0 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 1 monclient: found mon.noname-a 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:25.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.533+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7800b000 con 0x7f9e78003700 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.536+0000 7f9e8affd640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e78003700 msgr2=0x7f9e78005b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.536+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e78003700 0x7f9e78005b50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c007fd0 tx=0x7f9e7c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd36f4ae714ef45 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.536+0000 7f9e89ffb640 1 -- 172.21.15.116:0/3116209421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c02f470 con 0x7f9e78003700 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.536+0000 7f9e89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.536+0000 7f9e89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.536+0000 7f9e89ffb640 10 monclient: dump: 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.539 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 5 monclient: authenticate success, global_id 194640 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 1 -- 172.21.15.116:0/3116209421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c02f5d0 con 0x7f9e78003700 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:25.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 4 set_mon_vals no callback set 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 1 -- 172.21.15.116:0/3116209421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c02f990 con 0x7f9e78003700 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e89ffb640 10 monclient: dump: 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:25.541 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 10 monclient: shutdown 2024-09-18T12:45:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 1 -- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e78003700 msgr2=0x7f9e78005b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e78003700 0x7f9e78005b50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c007fd0 tx=0x7f9e7c009040 comp rx=0 tx=0).stop 2024-09-18T12:45:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3116209421 reap_dead start 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 1 -- 172.21.15.116:0/3116209421 shutdown_connections 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3116209421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e78003700 0x7f9e78005b50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 1 -- 172.21.15.116:0/3116209421 >> 172.21.15.116:0/3116209421 conn(0x7f9e8c0a1e30 msgr2=0x7f9e8c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 1 -- 172.21.15.116:0/3116209421 shutdown_connections 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.537+0000 7f9e918e8640 1 -- 172.21.15.116:0/3116209421 wait complete. 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.538+0000 7f9e918e8640 10 monclient: build_initial_monmap 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.538+0000 7f9e918e8640 10 monclient: monmap: 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 Processor -- start 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 -- start start 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient: init 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e78003700 0x7f9e8c131340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e78003700 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:25.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 0x7f9e8c138210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): picked mon.noname-a con 0x7f9e8c137e40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 0x7f9e8c147920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): picked mon.noname-c con 0x7f9e8c138750 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c098cb0 con 0x7f9e8c137e40 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c09ba00 con 0x7f9e78003700 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c11a2b0 con 0x7f9e8c138750 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e918e8640 10 monclient(hunting): authenticate will time out at 1952.861847s 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.539+0000 7f9e8a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 0x7f9e8c138210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 0x7f9e8c147920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 0x7f9e8c138210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41750/0 (socket says 172.21.15.116:41750) 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3427998945 learned_addr learned my addr 172.21.15.116:0/3427998945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request con 0x7f9e8c137e40 auth_method 0 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e8c138750 auth_method 0 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e78003700 0x7f9e8c131340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 10 monclient(hunting): get_auth_request con 0x7f9e78003700 auth_method 0 2024-09-18T12:45:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 msgr2=0x7f9e8c147920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 1 -- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 msgr2=0x7f9e8c147920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 0x7f9e8c147920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8b7fe640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 0x7f9e8c147920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient(hunting): handle_auth_done global_id 194643 payload 824 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 msgr2=0x7f9e8c147920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 0x7f9e8c147920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e78003700 msgr2=0x7f9e8c131340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e78003700 0x7f9e8c131340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 monclient: found mon.noname-a 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.540+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3427998945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c00f040 con 0x7f9e8c137e40 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e8a7fc640 1 -- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 msgr2=0x7f9e8c138210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e8a7fc640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 0x7f9e8c138210 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000e010 tx=0x7f9e80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 1 -- 172.21.15.116:0/3427998945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e80011940 con 0x7f9e8c137e40 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: dump: 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 1 -- 172.21.15.116:0/3427998945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e80011aa0 con 0x7f9e8c137e40 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 1 -- 172.21.15.116:0/3427998945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e80011e60 con 0x7f9e8c137e40 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e72ffd640 10 monclient: dump: 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.543+0000 7f9e918e8640 5 monclient: authenticate success, global_id 194643 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.543+0000 7f9e918e8640 10 monclient: _renew_subs 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.543+0000 7f9e918e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.542+0000 7f9e897fa640 4 set_mon_vals no callback set 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.543+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c11a870 con 0x7f9e8c137e40 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.543+0000 7f9e918e8640 10 monclient: _renew_subs 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.543+0000 7f9e918e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:25.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.543+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8c118c30 con 0x7f9e8c137e40 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e72ffd640 1 -- 172.21.15.116:0/3427998945 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9e80016670 con 0x7f9e8c137e40 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e918e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e918e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e918e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8c11a440 con 0x7f9e8c137e40 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e72ffd640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e780392a0 0x7f9e7803b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e72ffd640 1 -- 172.21.15.116:0/3427998945 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f9e8004ca00 con 0x7f9e8c137e40 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e780392a0 0x7f9e7803b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:25.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.544+0000 7f9e8affd640 10 monclient: get_auth_request con 0x7f9e780392a0 auth_method 0 2024-09-18T12:45:25.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.547+0000 7f9e8affd640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e780392a0 0x7f9e7803b750 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9e78006210 tx=0x7f9e7c039000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:25.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.549+0000 7f9e72ffd640 1 -- 172.21.15.116:0/3427998945 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f9e80014b30 con 0x7f9e8c137e40 2024-09-18T12:45:25.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.549+0000 7f9e72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:25.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.549+0000 7f9e72ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.765+0000 7f9e918e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:45:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.765+0000 7f9e918e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.765+0000 7f9e918e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.765+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e8c04bf60 con 0x7f9e8c137e40 2024-09-18T12:45:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.766+0000 7f9e72ffd640 1 -- 172.21.15.116:0/3427998945 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9e80014350 con 0x7f9e8c137e40 2024-09-18T12:45:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.766+0000 7f9e72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.766+0000 7f9e72ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:25.768 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":196,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e780392a0 msgr2=0x7f9e7803b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e780392a0 0x7f9e7803b750 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9e78006210 tx=0x7f9e7c039000 comp rx=0 tx=0).stop 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 10 monclient: shutdown 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 msgr2=0x7f9e8c138210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 0x7f9e8c138210 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000e010 tx=0x7f9e80008040 comp rx=0 tx=0).stop 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 shutdown_connections 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9e780392a0 0x7f9e7803b750 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9e8c138750 0x7f9e8c147920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e8c137e40 0x7f9e8c138210 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 --2- 172.21.15.116:0/3427998945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e78003700 0x7f9e8c131340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 >> 172.21.15.116:0/3427998945 conn(0x7f9e8c0a1e30 msgr2=0x7f9e8c096f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 shutdown_connections 2024-09-18T12:45:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:25.768+0000 7f9e918e8640 1 -- 172.21.15.116:0/3427998945 wait complete. 2024-09-18T12:45:28.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:45:28.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.881+0000 7fc22bef8640 10 monclient: get_monmap_and_config 2024-09-18T12:45:28.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.881+0000 7fc22bef8640 10 monclient: build_initial_monmap 2024-09-18T12:45:28.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.881+0000 7fc22bef8640 10 monclient: monmap: 2024-09-18T12:45:28.886 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:28.883533+0000 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:28.883533+0000 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.882+0000 7fc22bef8640 1 Processor -- start 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.882+0000 7fc22bef8640 1 -- start start 2024-09-18T12:45:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.882+0000 7fc22bef8640 10 monclient: init 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc224096e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2240971e0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient(hunting): picked mon.noname-a con 0x7fc224134b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22409d4a0 con 0x7fc2240971e0 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc224099620 con 0x7fc224096e10 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc224099340 con 0x7fc224134b40 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22bef8640 10 monclient(hunting): authenticate will time out at 1956.205284s 2024-09-18T12:45:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41768/0 (socket says 172.21.15.116:41768) 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 learned_addr learned my addr 172.21.15.116:0/1640691242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 10 monclient(hunting): get_auth_request con 0x7fc224134b40 auth_method 0 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc229493640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc2240971e0 0x7fc2240975e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59154/0 (socket says 172.21.15.116:59154) 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.883+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.885+0000 7fc229493640 1 -- 172.21.15.116:0/1640691242 >> v1:172.21.15.116:6789/0 conn(0x7fc2240971e0 legacy=0x7fc2240975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:28.885+0000 7fc229c94640 1 -- 172.21.15.116:0/1640691242 >> v1:172.21.15.144:6789/0 conn(0x7fc224096e10 legacy=0x7fc22409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:29.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:29.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 10 monclient(hunting): get_auth_request con 0x7fc224134b40 auth_method 0 2024-09-18T12:45:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.084+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 10 monclient(hunting): get_auth_request con 0x7fc224134b40 auth_method 0 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:29.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:29.485+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:45:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 10 monclient(hunting): get_auth_request con 0x7fc224134b40 auth_method 0 2024-09-18T12:45:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T12:45:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:30.286+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:45:31.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102419004 0 0) 0x7fc224099620 con 0x7fc224096e10 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): my global_id is 234235 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2100034e0 con 0x7fc224096e10 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3701037613 0 0) 0x7fc2100034e0 con 0x7fc224096e10 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 msgr2=0x7fc224136f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 --2- 172.21.15.116:0/1640691242 >> v2:172.21.15.116:3300/0 conn(0x7fc224134b40 0x7fc224136f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 >> v1:172.21.15.116:6789/0 conn(0x7fc2240971e0 legacy=0x7fc2240975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: _finish_hunting 0 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:45:31.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc224099900 con 0x7fc224096e10 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: _finish_auth 0 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fc2200017f0 con 0x7fc224096e10 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22bef8640 5 monclient: authenticate success, global_id 234235 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22bef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: dump: 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:31.537 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:31.538 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.538 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.538 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:31.538 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:31.538 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:31.538 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:31.538 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 >> v1:172.21.15.144:6789/0 conn(0x7fc224096e10 legacy=0x7fc22409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc210003bc0 0x7fc210003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): picked mon.b con 0x7fc210003bc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc210004780 0x7fc210004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): picked mon.a con 0x7fc210004780 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 0x7fc21000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): picked mon.c con 0x7fc2100099c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc224099340 con 0x7fc210004780 2024-09-18T12:45:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21000c420 con 0x7fc210003bc0 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21000c5e0 con 0x7fc2100099c0 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc228c92640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22bef8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 0x7fc21000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22a495640 10 monclient(hunting): get_auth_request con 0x7fc2100099c0 auth_method 0 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22a495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22a495640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc22a495640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc229c94640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc210003bc0 0x7fc210003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc229c94640 10 monclient(hunting): get_auth_request con 0x7fc210003bc0 auth_method 0 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc229c94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc229c94640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.531+0000 7fc229c94640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:31.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc229c94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc229c94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc229c94640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 10 monclient(hunting): handle_auth_done global_id 234235 payload 293 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc229493640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc210004780 0x7fc210004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc210003bc0 msgr2=0x7fc210003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc210003bc0 0x7fc210003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc210004780 msgr2=0x7fc210004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc210004780 0x7fc210004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 10 monclient: _finish_hunting 0 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 1 monclient: found mon.c 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2100034e0 con 0x7fc2100099c0 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 10 monclient: _finish_auth 0 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc22a495640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc229493640 10 monclient: get_auth_request con 0x7fc210004780 auth_method 0 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc229c94640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc210003bc0 0x7fc210003fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.532+0000 7fc229493640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc210004780 0x7fc210004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.533+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 reap_dead start 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc22a495640 1 -- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 msgr2=0x7fc21000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 0x7fc21000bda0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009cd0 tx=0x7fc21c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e337e23947049a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc22a495640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 0x7fc21000bda0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009cd0 tx=0x7fc21c009040 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fc21c02bb40 mon_map magic: 0 v1 2024-09-18T12:45:31.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c02bb40 con 0x7fc2100099c0 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc228c92640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc228c92640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc228c92640 10 monclient: dump: 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:31.551 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc22bef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c02f070 con 0x7fc2100099c0 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc228c92640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc22bef8640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.535+0000 7fc22bef8640 4 set_mon_vals no callback set 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 10 monclient: shutdown 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 1 -- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 msgr2=0x7fc21000bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:31.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 0x7fc21000bda0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009cd0 tx=0x7fc21c009040 comp rx=0 tx=0).stop 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc228c92640 1 -- 172.21.15.116:0/1640691242 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7fc21c035400 con 0x7fc2100099c0 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc228c92640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 1 -- 172.21.15.116:0/1640691242 shutdown_connections 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 1 --2- 172.21.15.116:0/1640691242 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc2100099c0 0x7fc21000bda0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 1 -- 172.21.15.116:0/1640691242 >> 172.21.15.116:0/1640691242 conn(0x7fc2240672b0 msgr2=0x7fc2240676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 1 -- 172.21.15.116:0/1640691242 shutdown_connections 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.536+0000 7fc22bef8640 1 -- 172.21.15.116:0/1640691242 wait complete. 2024-09-18T12:45:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.537+0000 7fc22bef8640 10 monclient: build_initial_monmap 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.537+0000 7fc22bef8640 10 monclient: monmap: 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.537+0000 7fc22bef8640 1 Processor -- start 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.537+0000 7fc22bef8640 1 -- start start 2024-09-18T12:45:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.537+0000 7fc22bef8640 10 monclient: init 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 0x7fc224130890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): picked mon.noname-a con 0x7fc2100099c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 0x7fc22411b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): picked mon.noname-c con 0x7fc224130dd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc22411bdd0 0x7fc22411c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): picked mon.noname-b con 0x7fc22411bdd0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22411d8d0 con 0x7fc2100099c0 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22411e9c0 con 0x7fc22411bdd0 2024-09-18T12:45:31.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22409b050 con 0x7fc224130dd0 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22bef8640 10 monclient(hunting): authenticate will time out at 1958.860319s 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 0x7fc224130890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 0x7fc224130890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41824/0 (socket says 172.21.15.116:41824) 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 -- 172.21.15.116:0/4247563773 learned_addr learned my addr 172.21.15.116:0/4247563773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): get_auth_request con 0x7fc2100099c0 auth_method 0 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 0x7fc22411b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc22411bdd0 0x7fc22411c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 10 monclient(hunting): get_auth_request con 0x7fc224130dd0 auth_method 0 2024-09-18T12:45:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 10 monclient(hunting): get_auth_request con 0x7fc22411bdd0 auth_method 0 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 1 -- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 msgr2=0x7fc22411b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 1 -- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 msgr2=0x7fc22411b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 0x7fc22411b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229493640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 0x7fc22411b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient(hunting): handle_auth_done global_id 194646 payload 824 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 -- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 msgr2=0x7fc22411b890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 0x7fc22411b890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 -- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc22411bdd0 msgr2=0x7fc22411c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc22411bdd0 0x7fc22411c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient: _finish_hunting 0 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 monclient: found mon.noname-a 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc229c94640 1 -- 172.21.15.116:0/4247563773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22411d340 con 0x7fc2100099c0 2024-09-18T12:45:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.538+0000 7fc22a495640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc22411bdd0 0x7fc22411c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc229c94640 1 -- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 msgr2=0x7fc224130890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc229c94640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 0x7fc224130890 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc220006ac0 tx=0x7fc2200079e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 1 -- 172.21.15.116:0/4247563773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22000fb50 con 0x7fc2100099c0 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: dump: 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 1 -- 172.21.15.116:0/4247563773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22000fcb0 con 0x7fc2100099c0 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc22bef8640 5 monclient: authenticate success, global_id 194646 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 1 -- 172.21.15.116:0/4247563773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2200133f0 con 0x7fc2100099c0 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc22bef8640 10 monclient: _renew_subs 2024-09-18T12:45:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc22bef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22409fa30 con 0x7fc2100099c0 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc219ffb640 10 monclient: dump: 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:31.561 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc21bfff640 4 set_mon_vals no callback set 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc22bef8640 10 monclient: _renew_subs 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc22bef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.541+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc224141d60 con 0x7fc2100099c0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.542+0000 7fc219ffb640 1 -- 172.21.15.116:0/4247563773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fc220013820 con 0x7fc2100099c0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.542+0000 7fc219ffb640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc208038490 0x7fc20803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.542+0000 7fc219ffb640 1 -- 172.21.15.116:0/4247563773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fc22004c790 con 0x7fc2100099c0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.543+0000 7fc22bef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.543+0000 7fc229493640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc208038490 0x7fc20803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.543+0000 7fc22bef8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.543+0000 7fc22bef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.543+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22404bf60 con 0x7fc2100099c0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.543+0000 7fc229493640 10 monclient: get_auth_request con 0x7fc208038490 auth_method 0 2024-09-18T12:45:31.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.545+0000 7fc229493640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc208038490 0x7fc20803a940 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc224133a80 tx=0x7fc214009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.547+0000 7fc219ffb640 1 -- 172.21.15.116:0/4247563773 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fc22001c400 con 0x7fc2100099c0 2024-09-18T12:45:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.547+0000 7fc219ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.547+0000 7fc219ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:31.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.763+0000 7fc22bef8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:45:31.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.763+0000 7fc22bef8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:31.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.763+0000 7fc22bef8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:31.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.763+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc22411ec00 con 0x7fc2100099c0 2024-09-18T12:45:31.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.763+0000 7fc219ffb640 1 -- 172.21.15.116:0/4247563773 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fc220023770 con 0x7fc2100099c0 2024-09-18T12:45:31.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.763+0000 7fc219ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:31.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.763+0000 7fc219ffb640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:31.766 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":200,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:45:31.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc208038490 msgr2=0x7fc20803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:31.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc208038490 0x7fc20803a940 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc224133a80 tx=0x7fc214009040 comp rx=0 tx=0).stop 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 10 monclient: shutdown 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 msgr2=0x7fc224130890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 0x7fc224130890 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc220006ac0 tx=0x7fc2200079e0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 shutdown_connections 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fc208038490 0x7fc20803a940 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc22411bdd0 0x7fc22411c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc224130dd0 0x7fc22411b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 --2- 172.21.15.116:0/4247563773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2100099c0 0x7fc224130890 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.765+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 >> 172.21.15.116:0/4247563773 conn(0x7fc2240672b0 msgr2=0x7fc22409b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.766+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 shutdown_connections 2024-09-18T12:45:31.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:31.766+0000 7fc22bef8640 1 -- 172.21.15.116:0/4247563773 wait complete. 2024-09-18T12:45:34.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.880+0000 7fb3e2d9d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.880+0000 7fb3e2d9d640 10 monclient: build_initial_monmap 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.880+0000 7fb3e2d9d640 10 monclient: monmap: 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:34.881832+0000 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:34.881832+0000 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:34.882 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:34.883 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.880+0000 7fb3e2d9d640 1 Processor -- start 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.880+0000 7fb3e2d9d640 1 -- start start 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient: init 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3dc131ab0 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3dc09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3dc09f220 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc09faf0 con 0x7fb3dc09f220 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3dc09fd30 con 0x7fb3dc131ab0 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0a1110 con 0x7fb3dc09e910 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e2d9d640 10 monclient(hunting): authenticate will time out at 1962.203514s 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41834/0 (socket says 172.21.15.116:41834) 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 learned_addr learned my addr 172.21.15.116:0/2136810816 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request con 0x7fb3dc131ab0 auth_method 0 2024-09-18T12:45:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3dc09e910 auth_method 0 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.881+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:34.884+0000 7fb3e133a640 1 -- 172.21.15.116:0/2136810816 >> v1:172.21.15.144:6789/0 conn(0x7fb3dc09f220 legacy=0x7fb3dc068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:35.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request con 0x7fb3dc131ab0 auth_method 0 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3dc09e910 auth_method 0 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:35.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:45:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:45:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.083+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T12:45:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3dc09e910 auth_method 0 2024-09-18T12:45:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request con 0x7fb3dc131ab0 auth_method 0 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:45:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:35.484+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T12:45:36.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request con 0x7fb3dc131ab0 auth_method 0 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3dc09e910 auth_method 0 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:45:36.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:36.285+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229796155 0 0) 0x7fb3dc09faf0 con 0x7fb3dc09f220 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): my global_id is 234241 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c80034e0 con 0x7fb3dc09f220 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3647712420 0 0) 0x7fb3c80034e0 con 0x7fb3dc09f220 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3dc09ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3dc09ece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 msgr2=0x7fb3dc131e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 --2- 172.21.15.116:0/2136810816 >> v2:172.21.15.116:3300/0 conn(0x7fb3dc131ab0 0x7fb3dc131e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 monclient: found mon.noname-c-legacy 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.144:6789/0 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 --> v1:172.21.15.144:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc0a13f0 con 0x7fb3dc09f220 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: _finish_auth 0 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 <== mon.1 v1:172.21.15.144:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3217541656 0 0) 0x7fb3d4001b70 con 0x7fb3dc09f220 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e2d9d640 5 monclient: authenticate success, global_id 234241 2024-09-18T12:45:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: dump: 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] but i'm connected to v1:172.21.15.144:6789/0, reconnecting 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 >> v1:172.21.15.144:6789/0 conn(0x7fb3dc09f220 legacy=0x7fb3dc068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3c8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): picked mon.c con 0x7fb3dc09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c80041f0 0x7fb3c8004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): picked mon.b con 0x7fb3c80041f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3c8004de0 0x7fb3c8005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): picked mon.a con 0x7fb3c8004de0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc09fd30 con 0x7fb3c8004de0 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0a1110 con 0x7fb3c80041f0 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c800bd50 con 0x7fb3dc09e910 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3db7fe640 10 monclient(hunting): _finish_auth 0 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e2d9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3c8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c80041f0 0x7fb3c8004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3c80041f0 auth_method 0 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request con 0x7fb3dc09e910 auth_method 0 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3c8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T12:45:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3c8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T12:45:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3c8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.532+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3c8003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_done global_id 234241 payload 293 2024-09-18T12:45:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 msgr2=0x7fb3c8003b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3dc09e910 0x7fb3c8003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3c8004de0 msgr2=0x7fb3c8005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3c8004de0 0x7fb3c8005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 1 monclient: found mon.b 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c80034e0 con 0x7fb3c80041f0 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 10 monclient: _finish_auth 0 2024-09-18T12:45:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.533+0000 7fb3dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.536+0000 7fb3dbfff640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c80041f0 msgr2=0x7fb3c8004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.536+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c80041f0 0x7fb3c8004780 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4006ea0 tx=0x7fb3c4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d840b1521d73a603 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c402b7b0 con 0x7fb3c80041f0 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 10 monclient: dump: 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c402f070 con 0x7fb3c80041f0 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 1 -- 172.21.15.116:0/2136810816 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c402bb80 con 0x7fb3c80041f0 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3e2d9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3e2d9d640 4 set_mon_vals no callback set 2024-09-18T12:45:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3db7fe640 10 monclient: dump: 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3e2d9d640 10 monclient: shutdown 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3e2d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/2136810816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c80041f0 msgr2=0x7fb3c8004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3e2d9d640 1 --2- 172.21.15.116:0/2136810816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c80041f0 0x7fb3c8004780 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4006ea0 tx=0x7fb3c4008040 comp rx=0 tx=0).stop 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.539+0000 7fb3e133a640 1 -- 172.21.15.116:0/2136810816 reap_dead start 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.540+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/2136810816 shutdown_connections 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.540+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/2136810816 >> 172.21.15.116:0/2136810816 conn(0x7fb3dc067d30 msgr2=0x7fb3dc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.540+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/2136810816 shutdown_connections 2024-09-18T12:45:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.540+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/2136810816 wait complete. 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.540+0000 7fb3e2d9d640 10 monclient: build_initial_monmap 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.540+0000 7fb3e2d9d640 10 monclient: monmap: 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.541+0000 7fb3e2d9d640 1 Processor -- start 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.541+0000 7fb3e2d9d640 1 -- start start 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.541+0000 7fb3e2d9d640 10 monclient: init 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.541+0000 7fb3e2d9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.541+0000 7fb3e2d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.541+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 0x7fb3dc130270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3c80041f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3dc11f2f0 0x7fb3dc11cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb3dc11f2f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:37.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3dc11d170 0x7fb3dc11d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb3dc11d170 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc1208a0 con 0x7fb3dc11d170 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc09bde0 con 0x7fb3dc11f2f0 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc09dc00 con 0x7fb3c80041f0 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e2d9d640 10 monclient(hunting): authenticate will time out at 1964.864089s 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 0x7fb3dc130270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:37.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 0x7fb3dc130270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47736/0 (socket says 172.21.15.116:47736) 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3dc11d170 0x7fb3dc11d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 1 -- 172.21.15.116:0/971357177 learned_addr learned my addr 172.21.15.116:0/971357177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request con 0x7fb3c80041f0 auth_method 0 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): get_auth_request con 0x7fb3dc11d170 auth_method 0 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 1 -- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 msgr2=0x7fb3dc130270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 1 -- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 msgr2=0x7fb3dc130270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 0x7fb3dc130270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3dc11f2f0 0x7fb3dc11cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request con 0x7fb3dc11f2f0 auth_method 0 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 0x7fb3dc130270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient(hunting): handle_auth_done global_id 194649 payload 824 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 1 -- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 msgr2=0x7fb3dc130270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 0x7fb3dc130270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 1 -- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3dc11f2f0 msgr2=0x7fb3dc11cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3dc11f2f0 0x7fb3dc11cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient: _finish_hunting 0 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 1 monclient: found mon.noname-a 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.542+0000 7fb3e133a640 1 -- 172.21.15.116:0/971357177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc120250 con 0x7fb3dc11d170 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.544+0000 7fb3e133a640 1 -- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3dc11d170 msgr2=0x7fb3dc11d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.544+0000 7fb3e133a640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3dc11d170 0x7fb3dc11d610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0a5ec0 tx=0x7fb3d4001b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 1 -- 172.21.15.116:0/971357177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4005b00 con 0x7fb3dc11d170 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: dump: 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: _finish_auth 0 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 1 -- 172.21.15.116:0/971357177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d4005c80 con 0x7fb3dc11d170 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3e2d9d640 5 monclient: authenticate success, global_id 194649 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 1 -- 172.21.15.116:0/971357177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d40070d0 con 0x7fb3dc11d170 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3e2d9d640 10 monclient: _renew_subs 2024-09-18T12:45:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3e2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc11e720 con 0x7fb3dc11d170 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3d8ff9640 10 monclient: dump: 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3daffd640 4 set_mon_vals no callback set 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3e2d9d640 10 monclient: _renew_subs 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3e2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.545+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc12f7d0 con 0x7fb3dc11d170 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3d8ff9640 1 -- 172.21.15.116:0/971357177 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb3d4007480 con 0x7fb3dc11d170 2024-09-18T12:45:37.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3e2d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3e2d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3e2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc04bf60 con 0x7fb3dc11d170 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3d8ff9640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3bc038490 0x7fb3bc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3d8ff9640 1 -- 172.21.15.116:0/971357177 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb3d404b210 con 0x7fb3dc11d170 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3bc038490 0x7fb3bc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.546+0000 7fb3e0b39640 10 monclient: get_auth_request con 0x7fb3bc038490 auth_method 0 2024-09-18T12:45:37.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.549+0000 7fb3e0b39640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3bc038490 0x7fb3bc03a940 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006ed0 tx=0x7fb3cc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:37.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.552+0000 7fb3d8ff9640 1 -- 172.21.15.116:0/971357177 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb3d40192c0 con 0x7fb3dc11d170 2024-09-18T12:45:37.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.552+0000 7fb3d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:37.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.552+0000 7fb3d8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:37.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.769+0000 7fb3e2d9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:45:37.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.769+0000 7fb3e2d9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:37.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.769+0000 7fb3e2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:37.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.769+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb3dc066280 con 0x7fb3dc11d170 2024-09-18T12:45:37.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.769+0000 7fb3d8ff9640 1 -- 172.21.15.116:0/971357177 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fb3d4017020 con 0x7fb3dc11d170 2024-09-18T12:45:37.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.769+0000 7fb3d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:37.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.769+0000 7fb3d8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:37.772 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":204,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:45:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3bc038490 msgr2=0x7fb3bc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3bc038490 0x7fb3bc03a940 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc006ed0 tx=0x7fb3cc008040 comp rx=0 tx=0).stop 2024-09-18T12:45:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 10 monclient: shutdown 2024-09-18T12:45:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3dc11d170 msgr2=0x7fb3dc11d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3dc11d170 0x7fb3dc11d610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0a5ec0 tx=0x7fb3d4001b70 comp rx=0 tx=0).stop 2024-09-18T12:45:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 shutdown_connections 2024-09-18T12:45:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb3bc038490 0x7fb3bc03a940 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3dc11d170 0x7fb3dc11d610 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3dc11f2f0 0x7fb3dc11cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 --2- 172.21.15.116:0/971357177 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb3c80041f0 0x7fb3dc130270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 >> 172.21.15.116:0/971357177 conn(0x7fb3dc067d30 msgr2=0x7fb3dc1310a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 shutdown_connections 2024-09-18T12:45:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:37.771+0000 7fb3e2d9d640 1 -- 172.21.15.116:0/971357177 wait complete. 2024-09-18T12:45:40.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:45:40.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.888+0000 7f6ba159d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.888+0000 7f6ba159d640 10 monclient: build_initial_monmap 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.889+0000 7f6ba159d640 10 monclient: monmap: 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:40.889664+0000 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:40.889664+0000 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:40.891 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:40.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.889+0000 7f6ba159d640 1 Processor -- start 2024-09-18T12:45:40.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.889+0000 7f6ba159d640 1 -- start start 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.889+0000 7f6ba159d640 10 monclient: init 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b9c0981c0 0x7f6b9c098590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient(hunting): picked mon.noname-c con 0x7f6b9c0981c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b9c131da0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b9c132170 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c098ad0 con 0x7f6b9c132170 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c067f30 con 0x7f6b9c131da0 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:40.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c131bd0 con 0x7f6b9c0981c0 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6ba159d640 10 monclient(hunting): authenticate will time out at 1968.211220s 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6b9c132170 0x7f6b9c067820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56406/0 (socket says 172.21.15.116:56406) 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9b7fe640 1 -- 172.21.15.116:0/3967804419 learned_addr learned my addr 172.21.15.116:0/3967804419 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 1 --2- 172.21.15.116:0/3967804419 >> v2:172.21.15.144:3300/0 conn(0x7f6b9c0981c0 0x7f6b9c098590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 10 monclient(hunting): get_auth_request con 0x7f6b9c0981c0 auth_method 0 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:40.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.890+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.892+0000 7f6b9b7fe640 1 -- 172.21.15.116:0/3967804419 >> v1:172.21.15.116:6789/0 conn(0x7f6b9c132170 legacy=0x7f6b9c067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.892+0000 7f6b99ffb640 1 -- 172.21.15.116:0/3967804419 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996660173 0 0) 0x7f6b9c098ad0 con 0x7f6b9c132170 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.892+0000 7f6b99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.892+0000 7f6b99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.892+0000 7f6b99ffb640 10 monclient(hunting): my global_id is 194652 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.892+0000 7f6b99ffb640 1 -- 172.21.15.116:0/3967804419 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b840034e0 con 0x7f6b9c132170 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.892+0000 7f6b9a7fc640 1 -- 172.21.15.116:0/3967804419 >> v1:172.21.15.144:6789/0 conn(0x7f6b9c131da0 legacy=0x7f6b9c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 10 monclient(hunting): handle_auth_done global_id 234247 payload 274 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b99ffb640 1 -- 172.21.15.116:0/3967804419 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591171009 0 0) 0x7f6b840034e0 con 0x7f6b9c132170 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 1 -- 172.21.15.116:0/3967804419 >> v1:172.21.15.144:6789/0 conn(0x7f6b9c131da0 legacy=0x7f6b9c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:40.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 1 -- 172.21.15.116:0/3967804419 >> v1:172.21.15.116:6789/0 conn(0x7f6b9c132170 legacy=0x7f6b9c067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:40.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 10 monclient: _finish_hunting 0 2024-09-18T12:45:40.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 1 monclient: found mon.noname-c 2024-09-18T12:45:40.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:40.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-09-18T12:45:40.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b9affd640 1 -- 172.21.15.116:0/3967804419 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c09ca60 con 0x7f6b9c0981c0 2024-09-18T12:45:40.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.893+0000 7f6b99ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T12:45:40.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b9affd640 1 -- 172.21.15.116:0/3967804419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c0981c0 msgr2=0x7f6b9c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:40.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b9affd640 1 --2- 172.21.15.116:0/3967804419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c098590 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009950 tx=0x7f6b8c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a5aba3aab867327 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:40.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 1 -- 172.21.15.116:0/3967804419 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94001be0 con 0x7f6b9c0981c0 2024-09-18T12:45:40.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: dump: 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 1 -- 172.21.15.116:0/3967804419 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c02acd0 con 0x7f6b9c0981c0 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 1 -- 172.21.15.116:0/3967804419 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c032670 con 0x7f6b9c0981c0 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6b99ffb640 10 monclient: dump: 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6ba159d640 5 monclient: authenticate success, global_id 234247 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6ba159d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.896+0000 7f6ba159d640 4 set_mon_vals no callback set 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 10 monclient: shutdown 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 1 -- 172.21.15.116:0/3967804419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c0981c0 msgr2=0x7f6b9c098590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 1 --2- 172.21.15.116:0/3967804419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c098590 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009950 tx=0x7f6b8c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 1 -- 172.21.15.116:0/3967804419 shutdown_connections 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 1 --2- 172.21.15.116:0/3967804419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c098590 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 1 -- 172.21.15.116:0/3967804419 >> 172.21.15.116:0/3967804419 conn(0x7f6b9c0a1e30 msgr2=0x7f6b9c0a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 1 -- 172.21.15.116:0/3967804419 shutdown_connections 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 1 -- 172.21.15.116:0/3967804419 wait complete. 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 10 monclient: build_initial_monmap 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.897+0000 7f6ba159d640 10 monclient: monmap: 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:40.905 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 Processor -- start 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 -- start start 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient: init 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c11dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): picked mon.noname-a con 0x7f6b9c0981c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c131da0 0x7f6b9c11e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): picked mon.noname-b con 0x7f6b9c131da0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c132170 0x7f6b9c11e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): picked mon.noname-c con 0x7f6b9c132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c1201b0 con 0x7f6b9c0981c0 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c0a0770 con 0x7f6b9c131da0 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c11f7b0 con 0x7f6b9c132170 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6ba159d640 10 monclient(hunting): authenticate will time out at 1968.219625s 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c11dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c11dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41708/0 (socket says 172.21.15.116:41708) 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c132170 0x7f6b9c11e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 1 -- 172.21.15.116:0/2439829968 learned_addr learned my addr 172.21.15.116:0/2439829968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 10 monclient(hunting): get_auth_request con 0x7f6b9c0981c0 auth_method 0 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 10 monclient(hunting): get_auth_request con 0x7f6b9c132170 auth_method 0 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9a7fc640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c131da0 0x7f6b9c11e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9a7fc640 10 monclient(hunting): get_auth_request con 0x7f6b9c131da0 auth_method 0 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.898+0000 7f6b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 10 monclient(hunting): handle_auth_done global_id 194655 payload 824 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 1 -- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c132170 msgr2=0x7f6b9c11e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c132170 0x7f6b9c11e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:40.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 1 -- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c131da0 msgr2=0x7f6b9c11e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:40.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c131da0 0x7f6b9c11e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:40.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 10 monclient: _finish_hunting 0 2024-09-18T12:45:40.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 1 monclient: found mon.noname-a 2024-09-18T12:45:40.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:40.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:40.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9affd640 1 -- 172.21.15.116:0/2439829968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c009390 con 0x7f6b9c0981c0 2024-09-18T12:45:40.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.899+0000 7f6b9b7fe640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c132170 0x7f6b9c11e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:40.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.901+0000 7f6b9affd640 1 -- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 msgr2=0x7f6b9c11dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:40.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.901+0000 7f6b9affd640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c11dc20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009950 tx=0x7f6b8c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:40.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.901+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 DispatchQueue::entry inject delay of 0.002000 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c03b820 con 0x7f6b9c0981c0 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: dump: 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:40.913 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c03b980 con 0x7f6b9c0981c0 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6ba159d640 5 monclient: authenticate success, global_id 194655 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6ba159d640 10 monclient: _renew_subs 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c03bd40 con 0x7f6b9c0981c0 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6ba159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c098cb0 con 0x7f6b9c0981c0 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b82ffd640 10 monclient: dump: 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6b997fa640 4 set_mon_vals no callback set 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6ba159d640 10 monclient: _renew_subs 2024-09-18T12:45:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6ba159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.903+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c121270 con 0x7f6b9c0981c0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6b8c039040 con 0x7f6b9c0981c0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6ba159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6ba159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6ba159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9c1313c0 con 0x7f6b9c0981c0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6b82ffd640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b880384e0 0x7f6b8803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f6b8c02ae40 con 0x7f6b9c0981c0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.904+0000 7f6b9a7fc640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b880384e0 0x7f6b8803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.905+0000 7f6b9a7fc640 10 monclient: get_auth_request con 0x7f6b880384e0 auth_method 0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.907+0000 7f6b9a7fc640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b880384e0 0x7f6b8803a990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6b90007fd0 tx=0x7f6b90009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.909+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6b8c0433c0 con 0x7f6b9c0981c0 2024-09-18T12:45:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.909+0000 7f6b82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:40.909+0000 7f6b82ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:41.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.125+0000 7f6ba159d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:45:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.125+0000 7f6ba159d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.125+0000 7f6ba159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.125+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6b9c04bf60 con 0x7f6b9c0981c0 2024-09-18T12:45:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.125+0000 7f6b82ffd640 1 -- 172.21.15.116:0/2439829968 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f6b8c037620 con 0x7f6b9c0981c0 2024-09-18T12:45:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.125+0000 7f6b82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.125+0000 7f6b82ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:41.129 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:45:41.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b880384e0 msgr2=0x7f6b8803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b880384e0 0x7f6b8803a990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6b90007fd0 tx=0x7f6b90009040 comp rx=0 tx=0).stop 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 10 monclient: shutdown 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 msgr2=0x7f6b9c11dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c11dc20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009950 tx=0x7f6b8c032000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 shutdown_connections 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f6b880384e0 0x7f6b8803a990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6b9c132170 0x7f6b9c11e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b9c131da0 0x7f6b9c11e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 --2- 172.21.15.116:0/2439829968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0981c0 0x7f6b9c11dc20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.127+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 >> 172.21.15.116:0/2439829968 conn(0x7f6b9c0a1e30 msgr2=0x7f6b9c09bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.128+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 shutdown_connections 2024-09-18T12:45:41.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.128+0000 7f6ba159d640 1 -- 172.21.15.116:0/2439829968 wait complete. 2024-09-18T12:45:41.139 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:45:41.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T12:45:41.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.231+0000 7f927e69f640 10 monclient: get_monmap_and_config 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.231+0000 7f927e69f640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.231+0000 7f927e69f640 10 monclient: monmap: 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:41.231867+0000 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:41.231867+0000 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:41.232 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:41.233 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:41.233 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:41.233 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.231+0000 7f927e69f640 1 Processor -- start 2024-09-18T12:45:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.231+0000 7f927e69f640 1 -- start start 2024-09-18T12:45:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.231+0000 7f927e69f640 10 monclient: init 2024-09-18T12:45:41.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f927809a510 0x7f9278096c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): picked mon.noname-a con 0x7f927809a510 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f92780971a0 0x7f92780975a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): picked mon.noname-c con 0x7f92780971a0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927812f670 0x7f927812fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): picked mon.noname-b con 0x7f927812f670 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9278097ae0 con 0x7f927809a510 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927809cb00 con 0x7f92780971a0 2024-09-18T12:45:41.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927809a8e0 con 0x7f927812f670 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927e69f640 10 monclient(hunting): authenticate will time out at 1968.553435s 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927812f670 0x7f927812fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f927809a510 0x7f9278096c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f927809a510 0x7f9278096c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41724/0 (socket says 172.21.15.116:41724) 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927812f670 0x7f927812fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47748/0 (socket says 172.21.15.116:47748) 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 -- 172.21.15.116:0/3399938074 learned_addr learned my addr 172.21.15.116:0/3399938074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): get_auth_request con 0x7f927809a510 auth_method 0 2024-09-18T12:45:41.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 10 monclient(hunting): get_auth_request con 0x7f927812f670 auth_method 0 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f92777fe640 1 --2- 172.21.15.116:0/3399938074 >> v2:172.21.15.144:3300/0 conn(0x7f92780971a0 0x7f92780975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f92777fe640 10 monclient(hunting): get_auth_request con 0x7f92780971a0 auth_method 0 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f92777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f92777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f92777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient(hunting): handle_auth_done global_id 194658 payload 274 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 -- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927812f670 msgr2=0x7f927812fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 --2- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927812f670 0x7f927812fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 -- 172.21.15.116:0/3399938074 >> v2:172.21.15.144:3300/0 conn(0x7f92780971a0 msgr2=0x7f92780975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 --2- 172.21.15.116:0/3399938074 >> v2:172.21.15.144:3300/0 conn(0x7f92780971a0 0x7f92780975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 monclient: found mon.noname-a 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f9277fff640 1 -- 172.21.15.116:0/3399938074 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927806ed80 con 0x7f927809a510 2024-09-18T12:45:41.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.232+0000 7f927cc3c640 1 --2- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927812f670 0x7f927812fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9277fff640 1 -- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927809a510 msgr2=0x7f9278096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9277fff640 1 --2- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927809a510 0x7f9278096c60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9260009910 tx=0x7f92600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecde6d404a37199a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 1 -- 172.21.15.116:0/3399938074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926002aea0 con 0x7f927809a510 2024-09-18T12:45:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: dump: 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 1 -- 172.21.15.116:0/3399938074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9260032440 con 0x7f927809a510 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f927e69f640 5 monclient: authenticate success, global_id 194658 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f927e69f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 1 -- 172.21.15.116:0/3399938074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92600327e0 con 0x7f927809a510 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f927e69f640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f927e69f640 4 set_mon_vals no callback set 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.235+0000 7f9276ffd640 10 monclient: dump: 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.249 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 10 monclient: shutdown 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 -- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927809a510 msgr2=0x7f9278096c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 --2- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927809a510 0x7f9278096c60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9260009910 tx=0x7f92600091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 -- 172.21.15.116:0/3399938074 shutdown_connections 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 --2- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927812f670 0x7f927812fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 --2- 172.21.15.116:0/3399938074 >> v2:172.21.15.144:3300/0 conn(0x7f92780971a0 0x7f92780975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 --2- 172.21.15.116:0/3399938074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927809a510 0x7f9278096c60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 -- 172.21.15.116:0/3399938074 >> 172.21.15.116:0/3399938074 conn(0x7f92780672b0 msgr2=0x7f92780676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 -- 172.21.15.116:0/3399938074 shutdown_connections 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 1 -- 172.21.15.116:0/3399938074 wait complete. 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.236+0000 7f927e69f640 10 monclient: monmap: 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.251 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 Processor -- start 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 -- start start 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient: init 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92780971a0 0x7f9278132f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): picked mon.noname-b con 0x7f92780971a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927809a510 0x7f927812e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): picked mon.noname-c con 0x7f927809a510 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812f670 0x7f9278119300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): picked mon.noname-a con 0x7f927812f670 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927813fc30 con 0x7f927812f670 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927811cbc0 con 0x7f92780971a0 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927811c8e0 con 0x7f927809a510 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927e69f640 10 monclient(hunting): authenticate will time out at 1968.558867s 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927cc3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812f670 0x7f9278119300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f92777fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927809a510 0x7f927812e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927cc3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812f670 0x7f9278119300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41734/0 (socket says 172.21.15.116:41734) 2024-09-18T12:45:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927cc3c640 1 -- 172.21.15.116:0/1628319963 learned_addr learned my addr 172.21.15.116:0/1628319963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f92777fe640 10 monclient(hunting): get_auth_request con 0x7f927809a510 auth_method 0 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f92777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f92777fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f92777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927cc3c640 10 monclient(hunting): get_auth_request con 0x7f927812f670 auth_method 0 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927cc3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927cc3c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.237+0000 7f927cc3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92780971a0 0x7f9278132f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f927cc3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f927cc3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f927cc3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 10 monclient(hunting): get_auth_request con 0x7f92780971a0 auth_method 0 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 10 monclient(hunting): handle_auth_done global_id 244106 payload 824 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92780971a0 msgr2=0x7f9278132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f92780971a0 0x7f9278132f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812f670 msgr2=0x7f9278119300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812f670 0x7f9278119300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 1 monclient: found mon.noname-c 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f92777fe640 1 -- 172.21.15.116:0/1628319963 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9270009a20 con 0x7f927809a510 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.238+0000 7f927cc3c640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812f670 0x7f9278119300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f92777fe640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927809a510 msgr2=0x7f927812e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f92777fe640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927809a510 0x7f927812e7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f926800d890 tx=0x7f9268008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800b8d0 con 0x7f927809a510 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: dump: 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.256 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926800ba30 con 0x7f927809a510 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f927e69f640 5 monclient: authenticate success, global_id 244106 2024-09-18T12:45:41.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800bdf0 con 0x7f927809a510 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f927e69f640 10 monclient: _renew_subs 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f927e69f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9260009450 con 0x7f927809a510 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f926ffff640 10 monclient: dump: 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f92767fc640 4 set_mon_vals no callback set 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f927e69f640 10 monclient: _renew_subs 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f927e69f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.240+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927811b980 con 0x7f927809a510 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f927e69f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f927e69f640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 0x7f927811a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9278065f20 con 0x7f927812ecf0 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f9268014640 con 0x7f927809a510 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 0x7f927811a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient: get_auth_request con 0x7f927812ecf0 auth_method 0 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f926ffff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9258038490 0x7f925803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f926804c880 con 0x7f927809a510 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 10 monclient(hunting): handle_auth_done global_id 194664 payload 824 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f9277fff640 1 -- 172.21.15.116:0/1628319963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f927811cd80 con 0x7f927812ecf0 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f927cc3c640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9258038490 0x7f925803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.241+0000 7f927cc3c640 10 monclient: get_auth_request con 0x7f9258038490 auth_method 0 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f9277fff640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 msgr2=0x7f927811a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f9277fff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 0x7f927811a000 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9260034000 tx=0x7f9260032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926002aea0 con 0x7f927812ecf0 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 10 monclient: dump: 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9260031ab0 con 0x7f927812ecf0 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 msgr2=0x7f927811a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 0x7f927811a000 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9260034000 tx=0x7f9260032000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 msgr2=0x7f927811a000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f926ffff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927812ecf0 0x7f927811a000 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.244+0000 7f927cc3c640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9258038490 0x7f925803a940 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f927811d2c0 tx=0x7f927000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.249+0000 7f927e69f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:45:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.249+0000 7f927e69f640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 0x7f92780298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.249+0000 7f927e69f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.249+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927812f0c0 con 0x7f927811a540 2024-09-18T12:45:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 0x7f92780298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient: get_auth_request con 0x7f927811a540 auth_method 0 2024-09-18T12:45:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 10 monclient(hunting): handle_auth_done global_id 194667 payload 824 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.250+0000 7f9277fff640 1 -- 172.21.15.116:0/1628319963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f927811bba0 con 0x7f927811a540 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f9277fff640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 msgr2=0x7f92780298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f9277fff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 0x7f92780298c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f927811adf0 tx=0x7f926003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260046810 con 0x7f927811a540 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:41.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 10 monclient: dump: 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f92600318a0 con 0x7f927811a540 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:41.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 msgr2=0x7f92780298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 0x7f92780298c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f927811adf0 tx=0x7f926003d000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 msgr2=0x7f92780298c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.252+0000 7f926ffff640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f927811a540 0x7f92780298c0 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f927811adf0 tx=0x7f926003d000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "leader", 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 206, 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:45:41.267 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.268 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.269 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:45:41.270 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.271 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:45:41.273 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:45:41.276 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:45:41.277 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.278 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:45:41.279 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9258038490 msgr2=0x7f925803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f9258038490 0x7f925803a940 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f927811d2c0 tx=0x7f927000b040 comp rx=0 tx=0).stop 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 10 monclient: shutdown 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927809a510 msgr2=0x7f927812e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 --2- 172.21.15.116:0/1628319963 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f927809a510 0x7f927812e7b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f926800d890 tx=0x7f9268008040 comp rx=0 tx=0).stop 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927cc3c640 1 -- 172.21.15.116:0/1628319963 reap_dead start 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927cc3c640 1 -- 172.21.15.116:0/1628319963 reap_dead start 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 shutdown_connections 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 >> 172.21.15.116:0/1628319963 conn(0x7f92780672b0 msgr2=0x7f9278131eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 shutdown_connections 2024-09-18T12:45:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.253+0000 7f927e69f640 1 -- 172.21.15.116:0/1628319963 wait complete. 2024-09-18T12:45:41.281 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:45:41.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T12:45:41.282 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.374+0000 7fa3936d4640 10 monclient: get_monmap_and_config 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.374+0000 7fa3936d4640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.374+0000 7fa3936d4640 10 monclient: monmap: 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:41.374981+0000 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:41.374981+0000 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.374+0000 7fa3936d4640 1 Processor -- start 2024-09-18T12:45:41.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 1 -- start start 2024-09-18T12:45:41.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient: init 2024-09-18T12:45:41.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fa38c096c60 0x7fa38c09a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): picked mon.noname-c con 0x7fa38c096c60 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa38c097030 0x7fa38c097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): picked mon.noname-a con 0x7fa38c097030 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12e5f0 0x7fa38c12ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): picked mon.noname-b con 0x7fa38c12e5f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c09d2e0 con 0x7fa38c097030 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c09ac00 con 0x7fa38c096c60 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c09d6b0 con 0x7fa38c12e5f0 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa3936d4640 10 monclient(hunting): authenticate will time out at 1968.696546s 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa38c097030 0x7fa38c097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12e5f0 0x7fa38c12ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa38c097030 0x7fa38c097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41752/0 (socket says 172.21.15.116:41752) 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 1 -- 172.21.15.116:0/533390817 learned_addr learned my addr 172.21.15.116:0/533390817 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 10 monclient(hunting): get_auth_request con 0x7fa38c097030 auth_method 0 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): get_auth_request con 0x7fa38c12e5f0 auth_method 0 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient(hunting): handle_auth_done global_id 244109 payload 274 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 1 -- 172.21.15.116:0/533390817 >> v2:172.21.15.144:3300/0 conn(0x7fa38c096c60 msgr2=0x7fa38c09a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391470640 1 --2- 172.21.15.116:0/533390817 >> v2:172.21.15.144:3300/0 conn(0x7fa38c096c60 0x7fa38c09a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 1 --2- 172.21.15.116:0/533390817 >> v2:172.21.15.144:3300/0 conn(0x7fa38c096c60 0x7fa38c09a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 1 -- 172.21.15.116:0/533390817 >> v2:172.21.15.116:3300/0 conn(0x7fa38c097030 msgr2=0x7fa38c097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 1 --2- 172.21.15.116:0/533390817 >> v2:172.21.15.116:3300/0 conn(0x7fa38c097030 0x7fa38c097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 1 monclient: found mon.noname-b 2024-09-18T12:45:41.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391c71640 1 -- 172.21.15.116:0/533390817 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0992f0 con 0x7fa38c12e5f0 2024-09-18T12:45:41.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa391470640 10 monclient: get_auth_request con 0x7fa38c096c60 auth_method 0 2024-09-18T12:45:41.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.375+0000 7fa390c6f640 1 --2- 172.21.15.116:0/533390817 >> v2:172.21.15.116:3300/0 conn(0x7fa38c097030 0x7fa38c097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:41.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.376+0000 7fa391470640 1 --2- 172.21.15.116:0/533390817 >> v2:172.21.15.144:3300/0 conn(0x7fa38c096c60 0x7fa38c09a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:45:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa391c71640 1 -- 172.21.15.116:0/533390817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12e5f0 msgr2=0x7fa38c12ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa391c71640 1 --2- 172.21.15.116:0/533390817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12e5f0 0x7fa38c12ea30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0637c0 tx=0x7fa37c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef170831e5f48e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 1 -- 172.21.15.116:0/533390817 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c007760 con 0x7fa38c12e5f0 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: dump: 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 1 -- 172.21.15.116:0/533390817 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37c02d050 con 0x7fa38c12e5f0 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa3936d4640 5 monclient: authenticate success, global_id 244109 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa3936d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa3936d4640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 1 -- 172.21.15.116:0/533390817 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c007b30 con 0x7fa38c12e5f0 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa3936d4640 4 set_mon_vals no callback set 2024-09-18T12:45:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.378+0000 7fa38bfff640 10 monclient: dump: 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 10 monclient: shutdown 2024-09-18T12:45:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 -- 172.21.15.116:0/533390817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12e5f0 msgr2=0x7fa38c12ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 --2- 172.21.15.116:0/533390817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12e5f0 0x7fa38c12ea30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0637c0 tx=0x7fa37c009780 comp rx=0 tx=0).stop 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 -- 172.21.15.116:0/533390817 shutdown_connections 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 --2- 172.21.15.116:0/533390817 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12e5f0 0x7fa38c12ea30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 --2- 172.21.15.116:0/533390817 >> v2:172.21.15.116:3300/0 conn(0x7fa38c097030 0x7fa38c097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 --2- 172.21.15.116:0/533390817 >> v2:172.21.15.144:3300/0 conn(0x7fa38c096c60 0x7fa38c09a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 -- 172.21.15.116:0/533390817 >> 172.21.15.116:0/533390817 conn(0x7fa38c0672b0 msgr2=0x7fa38c0676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 -- 172.21.15.116:0/533390817 shutdown_connections 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.379+0000 7fa3936d4640 1 -- 172.21.15.116:0/533390817 wait complete. 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.380+0000 7fa3936d4640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.380+0000 7fa3936d4640 10 monclient: monmap: 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.380+0000 7fa3936d4640 1 Processor -- start 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.380+0000 7fa3936d4640 1 -- start start 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.380+0000 7fa3936d4640 10 monclient: init 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c096c60 0x7fa38c140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): picked mon.noname-c con 0x7fa38c096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c097030 0x7fa38c12d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): picked mon.noname-b con 0x7fa38c097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa38c12e5f0 0x7fa38c117530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): picked mon.noname-a con 0x7fa38c12e5f0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c09add0 con 0x7fa38c12e5f0 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0677b0 con 0x7fa38c097030 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c11ae20 con 0x7fa38c096c60 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa3936d4640 10 monclient(hunting): authenticate will time out at 1968.702150s 2024-09-18T12:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c096c60 0x7fa38c140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c096c60 0x7fa38c140de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47770/0 (socket says 172.21.15.116:47770) 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa38c12e5f0 0x7fa38c117530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 -- 172.21.15.116:0/1339914610 learned_addr learned my addr 172.21.15.116:0/1339914610 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): get_auth_request con 0x7fa38c096c60 auth_method 0 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 10 monclient(hunting): get_auth_request con 0x7fa38c12e5f0 auth_method 0 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c097030 0x7fa38c12d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 10 monclient(hunting): get_auth_request con 0x7fa38c097030 auth_method 0 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient(hunting): handle_auth_done global_id 244112 payload 824 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c097030 msgr2=0x7fa38c12d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa38c097030 0x7fa38c12d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa38c12e5f0 msgr2=0x7fa38c117530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa38c12e5f0 0x7fa38c117530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 monclient: found mon.noname-c 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391470640 1 -- 172.21.15.116:0/1339914610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa380009450 con 0x7fa38c096c60 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.381+0000 7fa391c71640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa38c12e5f0 0x7fa38c117530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa391470640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c096c60 msgr2=0x7fa38c140de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa391470640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c096c60 0x7fa38c140de0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa38400c880 tx=0x7fa384008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38400ce10 con 0x7fa38c096c60 2024-09-18T12:45:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: dump: 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa384007aa0 con 0x7fa38c096c60 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3936d4640 5 monclient: authenticate success, global_id 244112 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3936d4640 10 monclient: _renew_subs 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384007e60 con 0x7fa38c096c60 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3936d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c009a20 con 0x7fa38c096c60 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3897fa640 10 monclient: dump: 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa38b7fe640 4 set_mon_vals no callback set 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3936d4640 10 monclient: _renew_subs 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3936d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.383+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c119b50 con 0x7fa38c096c60 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fa384011600 con 0x7fa38c096c60 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa3936d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa3936d4640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 0x7fa38c117a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c065f20 con 0x7fa38c141320 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 0x7fa38c117a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 10 monclient: get_auth_request con 0x7fa38c141320 auth_method 0 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa3897fa640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3780384c0 0x7fa37803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fa38404c2b0 con 0x7fa38c096c60 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.384+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.385+0000 7fa390c6f640 10 monclient(hunting): handle_auth_done global_id 244115 payload 824 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.385+0000 7fa390c6f640 1 -- 172.21.15.116:0/1339914610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa38c04bf60 con 0x7fa38c141320 2024-09-18T12:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.385+0000 7fa391c71640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3780384c0 0x7fa37803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.385+0000 7fa391c71640 10 monclient: get_auth_request con 0x7fa3780384c0 auth_method 0 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa390c6f640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 msgr2=0x7fa38c117a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa390c6f640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 0x7fa38c117a70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa38c119220 tx=0x7fa3800091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38000fe50 con 0x7fa38c141320 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 10 monclient: dump: 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa380014650 con 0x7fa38c141320 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 msgr2=0x7fa38c117a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 0x7fa38c117a70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa38c119220 tx=0x7fa3800091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 msgr2=0x7fa38c117a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa3897fa640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c141320 0x7fa38c117a70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.387+0000 7fa391c71640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3780384c0 0x7fa37803a970 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa37c034000 tx=0x7fa37c037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa3936d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa3936d4640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 0x7fa38c117fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa3936d4640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c1416f0 con 0x7fa38c12ded0 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 0x7fa38c117fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient: get_auth_request con 0x7fa38c12ded0 auth_method 0 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 10 monclient(hunting): handle_auth_done global_id 244118 payload 824 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.393+0000 7fa390c6f640 1 -- 172.21.15.116:0/1339914610 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa38c11af60 con 0x7fa38c12ded0 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.395+0000 7fa390c6f640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 msgr2=0x7fa38c117fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.395+0000 7fa390c6f640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 0x7fa38c117fb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa38001c000 tx=0x7fa38001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.395+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa380025040 con 0x7fa38c12ded0 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 10 monclient: dump: 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 <== mon.2 v2:172.21.15.116:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fa380014440 con 0x7fa38c12ded0 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 msgr2=0x7fa38c117fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 0x7fa38c117fb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa38001c000 tx=0x7fa38001d000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 msgr2=0x7fa38c117fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3897fa640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c12ded0 0x7fa38c117fb0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa38001c000 tx=0x7fa38001d000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 206, 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:45:41.409 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 2, 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.410 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.411 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:45:41.412 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.413 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:45:41.415 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.419 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:45:41.420 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:41.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3780384c0 msgr2=0x7fa37803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fa3780384c0 0x7fa37803a970 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa37c034000 tx=0x7fa37c037000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 10 monclient: shutdown 2024-09-18T12:45:41.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c096c60 msgr2=0x7fa38c140de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 --2- 172.21.15.116:0/1339914610 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fa38c096c60 0x7fa38c140de0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa38400c880 tx=0x7fa384008040 comp rx=0 tx=0).stop 2024-09-18T12:45:41.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa391c71640 1 -- 172.21.15.116:0/1339914610 reap_dead start 2024-09-18T12:45:41.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa391c71640 1 -- 172.21.15.116:0/1339914610 reap_dead start 2024-09-18T12:45:41.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 shutdown_connections 2024-09-18T12:45:41.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 >> 172.21.15.116:0/1339914610 conn(0x7fa38c0672b0 msgr2=0x7fa38c09d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 shutdown_connections 2024-09-18T12:45:41.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.396+0000 7fa3936d4640 1 -- 172.21.15.116:0/1339914610 wait complete. 2024-09-18T12:45:41.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:45:41.423 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T12:45:41.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.516+0000 7f346e690640 10 monclient: get_monmap_and_config 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.516+0000 7f346e690640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.516+0000 7f346e690640 10 monclient: monmap: 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:41.516982+0000 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:41.516982+0000 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:41.518 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:41.518 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:41.518 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 1 Processor -- start 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 1 -- start start 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient: init 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3468131980 0x7f3468131d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient(hunting): picked mon.noname-c con 0x7f3468131980 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f3468132290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient(hunting): picked mon.noname-b con 0x7f346809fda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f346809e9c0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346809f510 con 0x7f346809e9c0 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34680a1180 con 0x7f3468131980 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34680a1460 con 0x7f346809fda0 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346e690640 10 monclient(hunting): authenticate will time out at 1968.838537s 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f3468132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346cc2d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f346809e9c0 0x7f346809ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56408/0 (socket says 172.21.15.116:56408) 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f346cc2d640 1 -- 172.21.15.116:0/4013042876 learned_addr learned my addr 172.21.15.116:0/4013042876 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): get_auth_request con 0x7f346809fda0 auth_method 0 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f3467fff640 1 --2- 172.21.15.116:0/4013042876 >> v2:172.21.15.144:3300/0 conn(0x7f3468131980 0x7f3468131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f3467fff640 10 monclient(hunting): get_auth_request con 0x7f3468131980 auth_method 0 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f3467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f3467fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f3467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient(hunting): handle_auth_done global_id 244121 payload 274 2024-09-18T12:45:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 1 -- 172.21.15.116:0/4013042876 >> v2:172.21.15.144:3300/0 conn(0x7f3468131980 msgr2=0x7f3468131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 1 --2- 172.21.15.116:0/4013042876 >> v2:172.21.15.144:3300/0 conn(0x7f3468131980 0x7f3468131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 1 -- 172.21.15.116:0/4013042876 >> v1:172.21.15.116:6789/0 conn(0x7f346809e9c0 legacy=0x7f346809ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 1 monclient: found mon.noname-b 2024-09-18T12:45:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.517+0000 7f34677fe640 1 -- 172.21.15.116:0/4013042876 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468139cd0 con 0x7f346809fda0 2024-09-18T12:45:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f34677fe640 1 -- 172.21.15.116:0/4013042876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 msgr2=0x7f3468132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f34677fe640 1 --2- 172.21.15.116:0/4013042876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f3468132290 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f345c009850 tx=0x7f345c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a124be55e20777b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 1 -- 172.21.15.116:0/4013042876 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c02add0 con 0x7f346809fda0 2024-09-18T12:45:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: dump: 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 1 -- 172.21.15.116:0/4013042876 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c032440 con 0x7f346809fda0 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f346e690640 5 monclient: authenticate success, global_id 244121 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f346e690640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 1 -- 172.21.15.116:0/4013042876 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0327e0 con 0x7f346809fda0 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f346e690640 4 set_mon_vals no callback set 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.520+0000 7f3466ffd640 10 monclient: dump: 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.531 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 10 monclient: shutdown 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 -- 172.21.15.116:0/4013042876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 msgr2=0x7f3468132290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 --2- 172.21.15.116:0/4013042876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f3468132290 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f345c009850 tx=0x7f345c0090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 -- 172.21.15.116:0/4013042876 shutdown_connections 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 --2- 172.21.15.116:0/4013042876 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f3468132290 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 --2- 172.21.15.116:0/4013042876 >> v2:172.21.15.144:3300/0 conn(0x7f3468131980 0x7f3468131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 -- 172.21.15.116:0/4013042876 >> 172.21.15.116:0/4013042876 conn(0x7f3468067bd0 msgr2=0x7f34680a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 -- 172.21.15.116:0/4013042876 shutdown_connections 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.521+0000 7f346e690640 1 -- 172.21.15.116:0/4013042876 wait complete. 2024-09-18T12:45:41.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient: monmap: 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 Processor -- start 2024-09-18T12:45:41.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 -- start start 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient: init 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346809e9c0 0x7f34681374f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): picked mon.noname-b con 0x7f346809e9c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f34681338e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): picked mon.noname-c con 0x7f346809fda0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3468131980 0x7f3468133e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): picked mon.noname-a con 0x7f3468131980 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34680a4220 con 0x7f3468131980 2024-09-18T12:45:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3468136900 con 0x7f346809e9c0 2024-09-18T12:45:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3468135750 con 0x7f346809fda0 2024-09-18T12:45:41.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.522+0000 7f346e690640 10 monclient(hunting): authenticate will time out at 1968.843976s 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f34681338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f34681338e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47804/0 (socket says 172.21.15.116:47804) 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 -- 172.21.15.116:0/601384526 learned_addr learned my addr 172.21.15.116:0/601384526 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): get_auth_request con 0x7f346809fda0 auth_method 0 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3468131980 0x7f3468133e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 10 monclient(hunting): get_auth_request con 0x7f3468131980 auth_method 0 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f346cc2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f3467fff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346809e9c0 0x7f34681374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient(hunting): handle_auth_done global_id 244124 payload 824 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346809e9c0 msgr2=0x7f34681374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346809e9c0 0x7f34681374f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3468131980 msgr2=0x7f3468133e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3468131980 0x7f3468133e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 monclient: found mon.noname-c 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f34677fe640 1 -- 172.21.15.116:0/601384526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c009390 con 0x7f346809fda0 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f3467fff640 10 monclient: get_auth_request con 0x7f346809e9c0 auth_method 0 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.523+0000 7f3467fff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f346809e9c0 0x7f34681374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f34677fe640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 msgr2=0x7f34681338e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f34677fe640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f34681338e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f345c009850 tx=0x7f345c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c031d40 con 0x7f346809fda0 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: dump: 2024-09-18T12:45:41.539 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f346e690640 5 monclient: authenticate success, global_id 244124 2024-09-18T12:45:41.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f346e690640 10 monclient: _renew_subs 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c03d440 con 0x7f346809fda0 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f346e690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3468136b70 con 0x7f346809fda0 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c031ea0 con 0x7f346809fda0 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f344ffff640 10 monclient: dump: 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.541 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f34667fc640 4 set_mon_vals no callback set 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f346e690640 10 monclient: _renew_subs 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f346e690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.525+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3468133460 con 0x7f346809fda0 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f346e690640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f346e690640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 0x7f3468141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f346e690640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3468065e00 con 0x7f3468141150 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f345c039040 con 0x7f346809fda0 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f344ffff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3454038490 0x7f345403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 0x7f3468141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f345c06c230 con 0x7f346809fda0 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 10 monclient: get_auth_request con 0x7f3468141150 auth_method 0 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f3467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f346cc2d640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3454038490 0x7f345403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.526+0000 7f346cc2d640 10 monclient: get_auth_request con 0x7f3454038490 auth_method 0 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.527+0000 7f3467fff640 10 monclient(hunting): handle_auth_done global_id 234268 payload 824 2024-09-18T12:45:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.527+0000 7f3467fff640 1 -- 172.21.15.116:0/601384526 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3468134b30 con 0x7f3468141150 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f3467fff640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 msgr2=0x7f3468141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f3467fff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 0x7f3468141520 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f345800e750 tx=0x7f3458008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f346cc2d640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3454038490 0x7f345403a940 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3460006d70 tx=0x7f34600052b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345800c7c0 con 0x7f3468141150 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 10 monclient: dump: 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f345800c920 con 0x7f3468141150 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 msgr2=0x7f3468141520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 0x7f3468141520 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f345800e750 tx=0x7f3458008040 comp rx=0 tx=0).stop 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 msgr2=0x7f3468141520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.529+0000 7f344ffff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141150 0x7f3468141520 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f346e690640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f346e690640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 0x7f3468098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f346e690640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346809f510 con 0x7f3468141a60 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 0x7f3468098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 10 monclient: get_auth_request con 0x7f3468141a60 auth_method 0 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.535+0000 7f3467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.536+0000 7f3467fff640 10 monclient(hunting): handle_auth_done global_id 234271 payload 824 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.536+0000 7f3467fff640 1 -- 172.21.15.116:0/601384526 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f346804bf60 con 0x7f3468141a60 2024-09-18T12:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.538+0000 7f3467fff640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 msgr2=0x7f3468098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.538+0000 7f3467fff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 0x7f3468098a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3458018000 tx=0x7f345801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.538+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458013c70 con 0x7f3468141a60 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.538+0000 7f344ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.538+0000 7f344ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.538+0000 7f344ffff640 10 monclient: dump: 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 <== mon.1 v2:172.21.15.144:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f345800c5b0 con 0x7f3468141a60 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f344ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f344ffff640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 msgr2=0x7f3468098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f344ffff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 0x7f3468098a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3458018000 tx=0x7f345801b000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f344ffff640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 msgr2=0x7f3468098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f344ffff640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3468141a60 0x7f3468098a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.550 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "peon", 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "election_epoch": 206, 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum": [ 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: 0, 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: 1, 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: 2 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_age": 3, 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "required_con": "2449958755906961412", 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "required_mon": [ 2024-09-18T12:45:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_con": "4540138320759226367", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "quorum_mon": [ 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.552 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "outside_quorum": [], 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "extra_probe_peers": [], 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_provider": [], 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "monmap": { 2024-09-18T12:45:41.553 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "fsid": "2f704457-11dd-41ce-9561-691d67058c00", 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "modified": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "created": "2024-09-18T12:24:46.763770Z", 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release": 17, 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min_mon_release_name": "quincy", 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "election_strategy": 1, 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "disallowed_leaders: ": "", 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false, 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "tiebreaker_mon": "", 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "removed_ranks: ": "", 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "features": { 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "persistent": [ 2024-09-18T12:45:41.554 INFO:teuthology.orchestra.run.smithi116.stdout: "kraken", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "luminous", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "mimic", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap-prune", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "nautilus", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "elector-pinging", 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy" 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "optional": [] 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "mons": [ 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.555 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 0, 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3300", 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789", 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.556 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6789/0", 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6789/0", 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 1, 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.557 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:3300", 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789", 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.144:6789/0", 2024-09-18T12:45:41.558 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.144:6789/0", 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "rank": 2, 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "c", 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addrs": { 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:3301", 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790", 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 0 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6790/0", 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "public_addr": "172.21.15.116:6790/0", 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "priority": 0, 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "weight": 0, 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_location": "{}" 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.560 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "feature_map": { 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": [ 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": [ 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:45:41.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "client": [ 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 2 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": [ 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "release": "luminous", 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 1 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-09-18T12:45:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-18T12:45:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-09-18T12:45:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "stretch_mode": false 2024-09-18T12:45:41.563 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:41.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3454038490 msgr2=0x7f345403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3454038490 0x7f345403a940 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3460006d70 tx=0x7f34600052b0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 10 monclient: shutdown 2024-09-18T12:45:41.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 msgr2=0x7f34681338e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 --2- 172.21.15.116:0/601384526 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f346809fda0 0x7f34681338e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f345c009850 tx=0x7f345c032000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346cc2d640 1 -- 172.21.15.116:0/601384526 reap_dead start 2024-09-18T12:45:41.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346cc2d640 1 -- 172.21.15.116:0/601384526 reap_dead start 2024-09-18T12:45:41.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 shutdown_connections 2024-09-18T12:45:41.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 >> 172.21.15.116:0/601384526 conn(0x7f3468067bd0 msgr2=0x7f346809d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 shutdown_connections 2024-09-18T12:45:41.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.539+0000 7f346e690640 1 -- 172.21.15.116:0/601384526 wait complete. 2024-09-18T12:45:41.565 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T12:45:41.565 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T12:45:41.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.659+0000 7f190519d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.659+0000 7f190519d640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.659+0000 7f190519d640 10 monclient: monmap: 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:41.660181+0000 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:41.660181+0000 2024-09-18T12:45:41.660 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:41.661 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:41.661 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 1 Processor -- start 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 1 -- start start 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient: init 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19000975f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f19000979c0 0x7f1900097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient(hunting): picked mon.noname-c con 0x7f19000979c0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:41.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient(hunting): picked mon.noname-b con 0x7f19001349f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190009d700 con 0x7f19000975f0 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190009d1b0 con 0x7f19000979c0 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000961e0 con 0x7f19001349f0 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f190519d640 10 monclient(hunting): authenticate will time out at 1968.981622s 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47808/0 (socket says 172.21.15.116:47808) 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 1 -- 172.21.15.116:0/773612968 learned_addr learned my addr 172.21.15.116:0/773612968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): get_auth_request con 0x7f19001349f0 auth_method 0 2024-09-18T12:45:41.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18fe59c640 1 --2- 172.21.15.116:0/773612968 >> v2:172.21.15.144:3300/0 conn(0x7f19000979c0 0x7f1900097dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.660+0000 7f18ff59e640 10 monclient(hunting): handle_auth_done global_id 244127 payload 274 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 1 -- 172.21.15.116:0/773612968 >> v2:172.21.15.144:3300/0 conn(0x7f19000979c0 msgr2=0x7f1900097dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 1 --2- 172.21.15.116:0/773612968 >> v2:172.21.15.144:3300/0 conn(0x7f19000979c0 0x7f1900097dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 1 -- 172.21.15.116:0/773612968 >> v1:172.21.15.144:6789/0 conn(0x7f19000975f0 legacy=0x7f1900095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 1 monclient: found mon.noname-b 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18ff59e640 1 -- 172.21.15.116:0/773612968 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190009d420 con 0x7f19001349f0 2024-09-18T12:45:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18fe59c640 10 monclient: get_auth_request con 0x7f19000979c0 auth_method 0 2024-09-18T12:45:41.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.661+0000 7f18fe59c640 1 --2- 172.21.15.116:0/773612968 >> v2:172.21.15.144:3300/0 conn(0x7f19000979c0 0x7f1900097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:45:41.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18ff59e640 1 -- 172.21.15.116:0/773612968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 msgr2=0x7f1900136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18ff59e640 1 --2- 172.21.15.116:0/773612968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900136dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f19000959e0 tx=0x7f18f80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ba10cfd0634eec7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18fdd9b640 1 -- 172.21.15.116:0/773612968 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8032440 con 0x7f19001349f0 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18fdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18fdd9b640 10 monclient: dump: 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.672 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18fdd9b640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f18fdd9b640 1 -- 172.21.15.116:0/773612968 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f80325a0 con 0x7f19001349f0 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.663+0000 7f190519d640 5 monclient: authenticate success, global_id 244127 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f18fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 4 set_mon_vals no callback set 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f18fdd9b640 1 -- 172.21.15.116:0/773612968 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8032940 con 0x7f19001349f0 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f18fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f18fdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f18fdd9b640 10 monclient: dump: 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 10 monclient: shutdown 2024-09-18T12:45:41.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 -- 172.21.15.116:0/773612968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 msgr2=0x7f1900136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 --2- 172.21.15.116:0/773612968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900136dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f19000959e0 tx=0x7f18f80091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 -- 172.21.15.116:0/773612968 shutdown_connections 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 --2- 172.21.15.116:0/773612968 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900136dd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 --2- 172.21.15.116:0/773612968 >> v2:172.21.15.144:3300/0 conn(0x7f19000979c0 0x7f1900097dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 -- 172.21.15.116:0/773612968 >> 172.21.15.116:0/773612968 conn(0x7f19000678a0 msgr2=0x7f19000a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 -- 172.21.15.116:0/773612968 shutdown_connections 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.664+0000 7f190519d640 1 -- 172.21.15.116:0/773612968 wait complete. 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient: build_initial_monmap 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient: monmap: 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:41.675 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 Processor -- start 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 -- start start 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient: init 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19000975f0 0x7f1900094a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): picked mon.noname-b con 0x7f19000975f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19000979c0 0x7f1900088800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): picked mon.noname-a con 0x7f19000979c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900088d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): picked mon.noname-c con 0x7f19001349f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19000983c0 con 0x7f19000979c0 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19000a1890 con 0x7f19000975f0 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190008c340 con 0x7f19001349f0 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.665+0000 7f190519d640 10 monclient(hunting): authenticate will time out at 1968.986992s 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19000979c0 0x7f1900088800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900088d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19000979c0 0x7f1900088800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41776/0 (socket says 172.21.15.116:41776) 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900088d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47814/0 (socket says 172.21.15.116:47814) 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 1 -- 172.21.15.116:0/299462369 learned_addr learned my addr 172.21.15.116:0/299462369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 10 monclient(hunting): get_auth_request con 0x7f19000979c0 auth_method 0 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): get_auth_request con 0x7f19001349f0 auth_method 0 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19000975f0 0x7f1900094a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 10 monclient(hunting): get_auth_request con 0x7f19000975f0 auth_method 0 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient(hunting): handle_auth_done global_id 244130 payload 824 2024-09-18T12:45:41.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 -- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19000975f0 msgr2=0x7f1900094a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19000975f0 0x7f1900094a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 -- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19000979c0 msgr2=0x7f1900088800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19000979c0 0x7f1900088800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient: _finish_hunting 0 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 monclient: found mon.noname-c 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18ff59e640 1 -- 172.21.15.116:0/299462369 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f8009450 con 0x7f19001349f0 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.666+0000 7f18fe59c640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19000979c0 0x7f1900088800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18ff59e640 1 -- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 msgr2=0x7f1900088d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18ff59e640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900088d40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18f8034000 tx=0x7f18f8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 1 -- 172.21.15.116:0/299462369 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8031d40 con 0x7f19001349f0 2024-09-18T12:45:41.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: dump: 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 1 -- 172.21.15.116:0/299462369 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f803d440 con 0x7f19001349f0 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f190519d640 5 monclient: authenticate success, global_id 244130 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 1 -- 172.21.15.116:0/299462369 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8031ea0 con 0x7f19001349f0 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18f6ffd640 10 monclient: dump: 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f190519d640 10 monclient: _renew_subs 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f190519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190008bd80 con 0x7f19001349f0 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f18fd59a640 4 set_mon_vals no callback set 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f190519d640 10 monclient: _renew_subs 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f190519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.668+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190008a950 con 0x7f19001349f0 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.669+0000 7f18f6ffd640 1 -- 172.21.15.116:0/299462369 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f18f8039040 con 0x7f19001349f0 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.669+0000 7f190519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.669+0000 7f190519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.669+0000 7f190519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.669+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190004bf60 con 0x7f19001349f0 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.669+0000 7f18f6ffd640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f18e0038490 0x7f18e003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.669+0000 7f18f6ffd640 1 -- 172.21.15.116:0/299462369 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f18f806c070 con 0x7f19001349f0 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.670+0000 7f18fed9d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f18e0038490 0x7f18e003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.670+0000 7f18fed9d640 10 monclient: get_auth_request con 0x7f18e0038490 auth_method 0 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.672+0000 7f18fed9d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f18e0038490 0x7f18e003a940 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f18f0007fd0 tx=0x7f18f0002630 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:41.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.674+0000 7f18f6ffd640 1 -- 172.21.15.116:0/299462369 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f18f8044950 con 0x7f19001349f0 2024-09-18T12:45:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.674+0000 7f18f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:41.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.674+0000 7f18f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.888+0000 7f190519d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T12:45:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.888+0000 7f190519d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T12:45:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.888+0000 7f190519d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.888+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f19000638d0 con 0x7f19001349f0 2024-09-18T12:45:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f18f6ffd640 1 -- 172.21.15.116:0/299462369 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f18f8037080 con 0x7f19001349f0 2024-09-18T12:45:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f18f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T12:45:41.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f18f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:41.890 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.890 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=4035872314,config=3140514009,health=193804072,kv=2187923472,logm=1554989795}) 2024-09-18T12:45:41.890 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=2205312884,config=3140514009,health=193804072,kv=3604100458,logm=404147814}) 2024-09-18T12:45:41.890 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.891 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=4035872314,config=3140514009,health=193804072,kv=2187923472,logm=1554989795}) 2024-09-18T12:45:41.891 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.889+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=7,logm=53} crc {auth=3649700256,config=3140514009,health=1809256884,kv=1892713131,logm=1333395487}) 2024-09-18T12:45:41.891 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.891+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.891 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.891+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115632003}) 2024-09-18T12:45:41.891 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.891+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3584948822}) 2024-09-18T12:45:41.891 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.891+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.892 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.891+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115632003}) 2024-09-18T12:45:41.892 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.891+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051411018}) 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f18e0038490 msgr2=0x7f18e003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f18e0038490 0x7f18e003a940 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f18f0007fd0 tx=0x7f18f0002630 comp rx=0 tx=0).stop 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 10 monclient: shutdown 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 msgr2=0x7f1900088d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900088d40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18f8034000 tx=0x7f18f8032000 comp rx=0 tx=0).stop 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 shutdown_connections 2024-09-18T12:45:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f18e0038490 0x7f18e003a940 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f19001349f0 0x7f1900088d40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19000979c0 0x7f1900088800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 --2- 172.21.15.116:0/299462369 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19000975f0 0x7f1900094a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 >> 172.21.15.116:0/299462369 conn(0x7f19000678a0 msgr2=0x7f1900136770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 shutdown_connections 2024-09-18T12:45:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f190519d640 1 -- 172.21.15.116:0/299462369 wait complete. 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137573015}) 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958007823}) 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137573015}) 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.892+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496787327}) 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.893+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.893+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397988083}) 2024-09-18T12:45:41.896 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.893+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170302679}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.893+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.893+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397988083}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.893+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3365276354}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.894+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.894+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917043564}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.894+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583583406}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.894+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.894+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917043564}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.894+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=786707161}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.895+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.895+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435064213}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.895+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262478874}) 2024-09-18T12:45:41.897 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.895+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.898 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.895+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435064213}) 2024-09-18T12:45:41.898 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.895+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900376092}) 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257450411}) 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1740523800}) 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257450411}) 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186387747}) 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935996788}) 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450308209}) 2024-09-18T12:45:41.899 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.900 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935996788}) 2024-09-18T12:45:41.900 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.896+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3569902984}) 2024-09-18T12:45:41.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T12:45:41.904 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.903+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.904 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.903+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537134589}) 2024-09-18T12:45:41.905 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.903+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848053924}) 2024-09-18T12:45:41.905 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.903+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.905 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.903+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537134589}) 2024-09-18T12:45:41.905 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.903+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984960902}) 2024-09-18T12:45:41.906 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.904+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.906 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.904+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815254218}) 2024-09-18T12:45:41.906 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.904+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616461934}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.904+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.904+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815254218}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.904+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1418222682}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.905+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.905+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652178742}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.905+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1410844943}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.905+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.905+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652178742}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.905+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4190217414}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380725946}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2296274655}) 2024-09-18T12:45:41.907 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.908 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380725946}) 2024-09-18T12:45:41.908 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495589277}) 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966828706}) 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3450095605}) 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966828706}) 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.906+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2726571143}) 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3533819392}) 2024-09-18T12:45:41.912 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683999458}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3533819392}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3889186776}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025671962}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1894587227}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025671962}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.907+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306653325}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.908+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.908+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186353302}) 2024-09-18T12:45:41.913 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.908+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2361099405}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.908+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.908+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186353302}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.908+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896370790}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320106492}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1999487}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320106492}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258138758}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870279585}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2164878181}) 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870279585}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.909+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474890934}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714282819}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1274086985}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714282819}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1054759518}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885115907}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2864158728}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885115907}) 2024-09-18T12:45:41.915 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.910+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1214575087}) 2024-09-18T12:45:41.916 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.911+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.916 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.911+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297836551}) 2024-09-18T12:45:41.916 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.911+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010634354}) 2024-09-18T12:45:41.916 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.911+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.916 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.911+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297836551}) 2024-09-18T12:45:41.916 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.911+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505245958}) 2024-09-18T12:45:41.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.911+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.921 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298145001}) 2024-09-18T12:45:41.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2421378670}) 2024-09-18T12:45:41.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298145001}) 2024-09-18T12:45:41.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1284457522}) 2024-09-18T12:45:41.922 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.923 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602344744}) 2024-09-18T12:45:41.923 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1912840220}) 2024-09-18T12:45:41.923 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.923 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602344744}) 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.912+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2128030816}) 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264849962}) 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2232574491}) 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264849962}) 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043055681}) 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909656392}) 2024-09-18T12:45:41.924 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3533342335}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909656392}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.913+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106933641}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108977163}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2645329852}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108977163}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3111790168}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668186044}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1369267122}) 2024-09-18T12:45:41.925 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668186044}) 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.914+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506538126}) 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.915+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.915+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240617252}) 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.915+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3130817675}) 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.915+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.915+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240617252}) 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.915+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=920310536}) 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397786275}) 2024-09-18T12:45:41.926 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1519583979}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397786275}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1677936002}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2474056827}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859839176}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2474056827}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.916+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358122983}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.917+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.917+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809873821}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.917+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3250471553}) 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.917+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.927 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.917+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809873821}) 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.917+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798540895}) 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599448675}) 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4003616017}) 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599448675}) 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3372279840}) 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589220037}) 2024-09-18T12:45:41.928 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899662934}) 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589220037}) 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.918+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756087955}) 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.919+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.919+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760323368}) 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.919+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=56449037}) 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.919+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.919+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760323368}) 2024-09-18T12:45:41.929 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.919+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2177633675}) 2024-09-18T12:45:41.984 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.984 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340082860}) 2024-09-18T12:45:41.985 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367337293}) 2024-09-18T12:45:41.985 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:41.985 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340082860}) 2024-09-18T12:45:41.985 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:41.983+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887981507}) 2024-09-18T12:45:42.164 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.163+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.164 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.163+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1590070272}) 2024-09-18T12:45:42.164 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.163+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2635370666}) 2024-09-18T12:45:42.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.163+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.163+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1590070272}) 2024-09-18T12:45:42.165 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.163+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094860969}) 2024-09-18T12:45:42.166 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.164+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.166 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.164+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911191734}) 2024-09-18T12:45:42.166 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.164+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808772459}) 2024-09-18T12:45:42.166 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.164+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.166 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.164+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911191734}) 2024-09-18T12:45:42.166 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.164+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3088984711}) 2024-09-18T12:45:42.167 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.165+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.167 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.165+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353435569}) 2024-09-18T12:45:42.167 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.165+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2924777009}) 2024-09-18T12:45:42.167 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.165+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.167 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.165+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353435569}) 2024-09-18T12:45:42.167 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.165+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498665722}) 2024-09-18T12:45:42.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.194+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.194+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752076419}) 2024-09-18T12:45:42.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.194+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487028257}) 2024-09-18T12:45:42.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.194+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.195 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.194+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752076419}) 2024-09-18T12:45:42.196 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.194+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237204583}) 2024-09-18T12:45:42.197 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.195+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.195+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118786096}) 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.195+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495610921}) 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.195+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.195+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118786096}) 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.195+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1724750536}) 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217820497}) 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1014930978}) 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.198 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217820497}) 2024-09-18T12:45:42.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133506485}) 2024-09-18T12:45:42.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025008147}) 2024-09-18T12:45:42.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2395243105}) 2024-09-18T12:45:42.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025008147}) 2024-09-18T12:45:42.199 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.196+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=240910312}) 2024-09-18T12:45:42.206 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.197+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.197+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981997714}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.197+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=643480727}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.197+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.197+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981997714}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.197+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820901234}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.198+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.198+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205374233}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.198+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4197330707}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.198+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.198+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205374233}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.198+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2558522722}) 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.198+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.207 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453769116}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501162866}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453769116}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1650778894}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973712868}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1711266052}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973712868}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.199+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492057128}) 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.200+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.208 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.200+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169531503}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.200+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3566110966}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.200+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.200+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169531503}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.200+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717284810}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376856880}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279068335}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376856880}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774680936}) 2024-09-18T12:45:42.209 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758790594}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659907921}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758790594}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.201+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866560544}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.202+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.202+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976630229}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.202+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851685930}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.202+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.202+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976630229}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.202+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162626682}) 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.210 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114364312}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3886933464}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114364312}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188094717}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344109788}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293690264}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344109788}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.203+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3854427380}) 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.204+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.211 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.204+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083804752}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.204+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2618062863}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.204+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.204+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083804752}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.204+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1427041080}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.205+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.205+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166436493}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.205+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125689586}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.205+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.205+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166436493}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.205+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2518208230}) 2024-09-18T12:45:42.212 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.205+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507452669}) 2024-09-18T12:45:42.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161341538}) 2024-09-18T12:45:42.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507452669}) 2024-09-18T12:45:42.213 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1106260048}) 2024-09-18T12:45:42.217 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.217 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740242585}) 2024-09-18T12:45:42.217 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2044372499}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740242585}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.206+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155180227}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.207+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.207+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194437813}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.207+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578973138}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.207+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.207+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194437813}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.207+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2605649535}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265206639}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125201516}) 2024-09-18T12:45:42.218 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265206639}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1031235242}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357590804}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3696646788}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357590804}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.208+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2632695017}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.209+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.209+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000660877}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.209+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1093400184}) 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.209+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.219 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.209+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000660877}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.209+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771088943}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299716221}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2574989123}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299716221}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133335322}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634488949}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1947492311}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634488949}) 2024-09-18T12:45:42.220 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.210+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3554901931}) 2024-09-18T12:45:42.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.211+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.211+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51456576}) 2024-09-18T12:45:42.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.211+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968928235}) 2024-09-18T12:45:42.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.211+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.211+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51456576}) 2024-09-18T12:45:42.221 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.211+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=457253201}) 2024-09-18T12:45:42.606 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.605+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.607 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.605+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492759655}) 2024-09-18T12:45:42.607 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.605+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2631580526}) 2024-09-18T12:45:42.607 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.605+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.607 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.605+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492759655}) 2024-09-18T12:45:42.607 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.605+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=478164499}) 2024-09-18T12:45:42.617 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.616+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.617 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.616+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090840611}) 2024-09-18T12:45:42.617 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.616+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074627946}) 2024-09-18T12:45:42.617 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.616+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.617 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.616+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090840611}) 2024-09-18T12:45:42.617 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.616+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399929052}) 2024-09-18T12:45:42.646 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.646 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800125204}) 2024-09-18T12:45:42.646 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264939375}) 2024-09-18T12:45:42.647 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.647 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800125204}) 2024-09-18T12:45:42.647 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3620293638}) 2024-09-18T12:45:42.647 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.647 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203645090}) 2024-09-18T12:45:42.648 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=768720758}) 2024-09-18T12:45:42.648 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.646+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.648 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203645090}) 2024-09-18T12:45:42.648 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818060198}) 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916378776}) 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725473105}) 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916378776}) 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.647+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3041615289}) 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263111927}) 2024-09-18T12:45:42.650 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2669700495}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263111927}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1478105408}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085762270}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719553454}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085762270}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.648+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3214068882}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.649+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.649+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2197408140}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.649+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2573819288}) 2024-09-18T12:45:42.651 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.649+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.652 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.649+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2197408140}) 2024-09-18T12:45:42.652 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.649+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392239}) 2024-09-18T12:45:42.659 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.659 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220322548}) 2024-09-18T12:45:42.659 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1304309265}) 2024-09-18T12:45:42.659 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.659 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220322548}) 2024-09-18T12:45:42.659 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=445703280}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093271866}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1636129815}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093271866}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.650+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509617977}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.651+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.651+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667421856}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.651+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3045563482}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.651+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.651+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667421856}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.651+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2435827766}) 2024-09-18T12:45:42.660 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.651+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852744912}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096048347}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852744912}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3053425511}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738950124}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675424538}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738950124}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.652+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121398269}) 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.661 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152326553}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483227763}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1152326553}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434253662}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818971230}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149507995}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818971230}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.653+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4058589967}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417633133}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4252254116}) 2024-09-18T12:45:42.662 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417633133}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.654+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1513079770}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405311116}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3210591719}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405311116}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3934808723}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961926825}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1510714340}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961926825}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.655+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407096688}) 2024-09-18T12:45:42.663 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945886474}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2363268615}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945886474}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913173172}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180153681}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912083762}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180153681}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.656+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=834988558}) 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.664 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848224884}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2075382321}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848224884}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.657+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508732339}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082743356}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640842395}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082743356}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159385695}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2744190380}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526642547}) 2024-09-18T12:45:42.665 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.666 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2744190380}) 2024-09-18T12:45:42.666 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.658+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163047595}) 2024-09-18T12:45:42.669 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.659+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.669 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.659+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689133}) 2024-09-18T12:45:42.669 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.659+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1765808270}) 2024-09-18T12:45:42.669 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.659+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.669 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.659+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689133}) 2024-09-18T12:45:42.669 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.659+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104344700}) 2024-09-18T12:45:42.669 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.659+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333169472}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064024328}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333169472}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2938359333}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145640858}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489368265}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145640858}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.660+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742146263}) 2024-09-18T12:45:42.670 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490118360}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1250352133}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490118360}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1572709293}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300601091}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902965870}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300601091}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.661+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926832359}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.662+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.662+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002866289}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.662+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1929537178}) 2024-09-18T12:45:42.671 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.662+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.662+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002866289}) 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.662+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756876351}) 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.663+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.663+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749964026}) 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.663+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2574220617}) 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.663+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.663+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749964026}) 2024-09-18T12:45:42.672 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.663+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329310297}) 2024-09-18T12:45:42.763 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108947612}) 2024-09-18T12:45:42.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835131063}) 2024-09-18T12:45:42.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108947612}) 2024-09-18T12:45:42.764 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3034578749}) 2024-09-18T12:45:42.767 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435169371}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2191283972}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435169371}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.763+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896846906}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.764+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.764+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533969212}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.764+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034736431}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.764+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.764+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533969212}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.764+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4244764154}) 2024-09-18T12:45:42.768 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239190107}) 2024-09-18T12:45:42.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4152584809}) 2024-09-18T12:45:42.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239190107}) 2024-09-18T12:45:42.769 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=285463773}) 2024-09-18T12:45:42.773 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417977574}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1092813720}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417977574}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.765+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1923130993}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.766+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.766+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029317431}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.766+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3765283553}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.766+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.766+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029317431}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.766+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466747948}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313815602}) 2024-09-18T12:45:42.774 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97432787}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313815602}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365707647}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134387200}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884103625}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134387200}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.767+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003027151}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.768+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.768+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300607505}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.768+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4048266074}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.768+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.768+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300607505}) 2024-09-18T12:45:42.775 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.768+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594732563}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.768+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042944110}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2139246073}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042944110}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300822054}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21437429}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011891028}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21437429}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.769+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=875632269}) 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.776 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197473467}) 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2596504828}) 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197473467}) 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.770+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1045045065}) 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198181829}) 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271433123}) 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198181829}) 2024-09-18T12:45:42.777 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.771+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2213513338}) 2024-09-18T12:45:42.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896112810}) 2024-09-18T12:45:42.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4283171934}) 2024-09-18T12:45:42.781 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896112810}) 2024-09-18T12:45:42.782 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.780+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1158710278}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1571140211}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3702998308}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1571140211}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.781+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317018899}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3417411259}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304718980}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3417411259}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4048996459}) 2024-09-18T12:45:42.784 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896414477}) 2024-09-18T12:45:42.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=932549126}) 2024-09-18T12:45:42.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896414477}) 2024-09-18T12:45:42.785 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.782+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3771663163}) 2024-09-18T12:45:42.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.817+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.818 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.817+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=4} crc {logm=1508268227,mdsmap=3585761405,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3456358934,mon_config_key=3655569570,monmap=1105712893,osd_metadata=3874183036}) 2024-09-18T12:45:42.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.817+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=4,monmap=3,osd_metadata=4} crc {logm=1963889260,mdsmap=1531151339,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=4234672039,mon_config_key=565504783,monmap=148614106,osd_metadata=3874183036}) 2024-09-18T12:45:42.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.817+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:42.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.817+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=4} crc {logm=1508268227,mdsmap=3585761405,mgr=3426601819,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=3456358934,mon_config_key=3655569570,monmap=1105712893,osd_metadata=3874183036}) 2024-09-18T12:45:42.819 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:42.817+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=4} crc {logm=4294831248,mdsmap=3140514009,mgr=175081845,mgr_command_descs=3545737492,mgr_metadata=3239289319,mgrstat=1254476593,mon_config_key=3655569570,monmap=148614106,osd_metadata=83811991}) 2024-09-18T12:45:42.905 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T12:45:42.905 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T12:45:43.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.001+0000 7f2a75f9d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.001+0000 7f2a75f9d640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.001+0000 7f2a75f9d640 10 monclient: monmap: 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:43.001823+0000 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:43.001823+0000 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:43.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:43.003 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:43.003 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:43.003 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.001+0000 7f2a75f9d640 1 Processor -- start 2024-09-18T12:45:43.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.001+0000 7f2a75f9d640 1 -- start start 2024-09-18T12:45:43.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.001+0000 7f2a75f9d640 10 monclient: init 2024-09-18T12:45:43.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:45:43.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a70096e10 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:43.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2a700971e0 0x7f2a700975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a700971e0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 0x7f2a70135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a70133b30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a70099620 con 0x7f2a70096e10 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a70099340 con 0x7f2a700971e0 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a70099900 con 0x7f2a70133b30 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a75f9d640 10 monclient(hunting): authenticate will time out at 1970.323366s 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 0x7f2a70135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 0x7f2a70135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47842/0 (socket says 172.21.15.116:47842) 2024-09-18T12:45:43.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 -- 172.21.15.116:0/900961077 learned_addr learned my addr 172.21.15.116:0/900961077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): get_auth_request con 0x7f2a70133b30 auth_method 0 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6effd640 1 --2- 172.21.15.116:0/900961077 >> v2:172.21.15.144:3300/0 conn(0x7f2a700971e0 0x7f2a700975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6effd640 10 monclient(hunting): get_auth_request con 0x7f2a700971e0 auth_method 0 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient(hunting): handle_auth_done global_id 244133 payload 274 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 -- 172.21.15.116:0/900961077 >> v2:172.21.15.144:3300/0 conn(0x7f2a700971e0 msgr2=0x7f2a700975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 --2- 172.21.15.116:0/900961077 >> v2:172.21.15.144:3300/0 conn(0x7f2a700971e0 0x7f2a700975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 -- 172.21.15.116:0/900961077 >> v1:172.21.15.144:6789/0 conn(0x7f2a70096e10 legacy=0x7f2a7009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 monclient: found mon.noname-b 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.002+0000 7f2a6ffff640 1 -- 172.21.15.116:0/900961077 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7009ac70 con 0x7f2a70133b30 2024-09-18T12:45:43.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6ffff640 1 -- 172.21.15.116:0/900961077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 msgr2=0x7f2a70135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6ffff640 1 --2- 172.21.15.116:0/900961077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 0x7f2a70135f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a7009b020 tx=0x7f2a6800ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6da1012594223e09 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 1 -- 172.21.15.116:0/900961077 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68030050 con 0x7f2a70133b30 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: dump: 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.015 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a75f9d640 5 monclient: authenticate success, global_id 244133 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 1 -- 172.21.15.116:0/900961077 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a68009610 con 0x7f2a70133b30 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a75f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a75f9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a75f9d640 4 set_mon_vals no callback set 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 1 -- 172.21.15.116:0/900961077 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a680099b0 con 0x7f2a70133b30 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:43.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.005+0000 7f2a6e7fc640 10 monclient: dump: 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 10 monclient: shutdown 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 -- 172.21.15.116:0/900961077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 msgr2=0x7f2a70135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/900961077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 0x7f2a70135f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a7009b020 tx=0x7f2a6800ba50 comp rx=0 tx=0).stop 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 -- 172.21.15.116:0/900961077 shutdown_connections 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/900961077 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70133b30 0x7f2a70135f10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/900961077 >> v2:172.21.15.144:3300/0 conn(0x7f2a700971e0 0x7f2a700975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 -- 172.21.15.116:0/900961077 >> 172.21.15.116:0/900961077 conn(0x7f2a700672b0 msgr2=0x7f2a700676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 -- 172.21.15.116:0/900961077 shutdown_connections 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 1 -- 172.21.15.116:0/900961077 wait complete. 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.006+0000 7f2a75f9d640 10 monclient: monmap: 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.018 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 Processor -- start 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 -- start start 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient: init 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 0x7f2a7012e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a70096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a700971e0 0x7f2a7006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a700971e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:43.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70133b30 0x7f2a7006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a70133b30 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7011dc70 con 0x7f2a70133b30 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7011d3a0 con 0x7f2a700971e0 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7009b050 con 0x7f2a70096e10 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a75f9d640 10 monclient(hunting): authenticate will time out at 1970.328734s 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 0x7f2a7012e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 0x7f2a7012e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47852/0 (socket says 172.21.15.116:47852) 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 1 -- 172.21.15.116:0/2252495497 learned_addr learned my addr 172.21.15.116:0/2252495497 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2a70096e10 auth_method 0 2024-09-18T12:45:43.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70133b30 0x7f2a7006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 10 monclient(hunting): get_auth_request con 0x7f2a70133b30 auth_method 0 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6effd640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a700971e0 0x7f2a7006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6effd640 10 monclient(hunting): get_auth_request con 0x7f2a700971e0 auth_method 0 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.007+0000 7f2a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 10 monclient(hunting): handle_auth_done global_id 244136 payload 824 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 1 -- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a700971e0 msgr2=0x7f2a7006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a700971e0 0x7f2a7006dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 1 -- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70133b30 msgr2=0x7f2a7006e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70133b30 0x7f2a7006e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6f7fe640 1 -- 172.21.15.116:0/2252495497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6800bcf0 con 0x7f2a70096e10 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6effd640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a700971e0 0x7f2a7006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.008+0000 7f2a6ffff640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70133b30 0x7f2a7006e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a6f7fe640 1 -- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 msgr2=0x7f2a7012e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a6f7fe640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 0x7f2a7012e4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c640 tx=0x7f2a60007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 1 -- 172.21.15.116:0/2252495497 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60012840 con 0x7f2a70096e10 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: dump: 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.023 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 1 -- 172.21.15.116:0/2252495497 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a600129a0 con 0x7f2a70096e10 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a75f9d640 5 monclient: authenticate success, global_id 244136 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a75f9d640 10 monclient: _renew_subs 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a75f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a70133100 con 0x7f2a70096e10 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 1 -- 172.21.15.116:0/2252495497 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60012d60 con 0x7f2a70096e10 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a677fe640 10 monclient: dump: 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.025 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a75f9d640 10 monclient: _renew_subs 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a6dffb640 4 set_mon_vals no callback set 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a75f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.010+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a701343c0 con 0x7f2a70096e10 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a677fe640 1 -- 172.21.15.116:0/2252495497 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f2a60011590 con 0x7f2a70096e10 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a75f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a75f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a75f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a677fe640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7011da20 con 0x7f2a70096e10 2024-09-18T12:45:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a677fe640 1 -- 172.21.15.116:0/2252495497 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f2a6004c560 con 0x7f2a70096e10 2024-09-18T12:45:43.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a6effd640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a6effd640 1 -- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 msgr2=0x7f2a5003a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:45:43.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a6effd640 1 -- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 msgr2=0x7f2a5003a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:43.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a6effd640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:43.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.011+0000 7f2a6effd640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:43.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.016+0000 7f2a677fe640 1 -- 172.21.15.116:0/2252495497 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f2a600108b0 con 0x7f2a70096e10 2024-09-18T12:45:43.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.016+0000 7f2a677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.016+0000 7f2a677fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:43.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.212+0000 7f2a6effd640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.212+0000 7f2a6effd640 10 monclient: get_auth_request con 0x7f2a500384e0 auth_method 0 2024-09-18T12:45:43.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.215+0000 7f2a6effd640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2a58006fd0 tx=0x7f2a58008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.231+0000 7f2a75f9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.232+0000 7f2a75f9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.232+0000 7f2a75f9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.232+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2a7004bf60 con 0x7f2a70096e10 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.232+0000 7f2a677fe640 1 -- 172.21.15.116:0/2252495497 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2a60016070 con 0x7f2a70096e10 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.232+0000 7f2a677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.232+0000 7f2a677fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:43.233 INFO:teuthology.orchestra.run.smithi116.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","modified":"2024-09-18T12:24:46.763770Z","created":"2024-09-18T12:24:46.763770Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3301","nonce":0},{"type":"v1","addr":"172.21.15.116:6790","nonce":0}]},"addr":"172.21.15.116:6790/0","public_addr":"172.21.15.116:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T12:45:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 msgr2=0x7f2a5003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2a58006fd0 tx=0x7f2a58008040 comp rx=0 tx=0).stop 2024-09-18T12:45:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 10 monclient: shutdown 2024-09-18T12:45:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 msgr2=0x7f2a7012e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 0x7f2a7012e4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c640 tx=0x7f2a60007ed0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 shutdown_connections 2024-09-18T12:45:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f2a500384e0 0x7f2a5003a990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70133b30 0x7f2a7006e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2a700971e0 0x7f2a7006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 --2- 172.21.15.116:0/2252495497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2a70096e10 0x7f2a7012e4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 >> 172.21.15.116:0/2252495497 conn(0x7f2a700672b0 msgr2=0x7f2a70131800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 shutdown_connections 2024-09-18T12:45:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.234+0000 7f2a75f9d640 1 -- 172.21.15.116:0/2252495497 wait complete. 2024-09-18T12:45:43.245 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T12:45:43.245 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T12:45:43.254 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-18T12:45:43.255 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:45:43.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.346+0000 7f74be234640 10 monclient: get_monmap_and_config 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.346+0000 7f74be234640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.346+0000 7f74be234640 10 monclient: monmap: 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:43.347540+0000 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:43.347540+0000 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:43.349 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.347+0000 7f74be234640 1 Processor -- start 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.347+0000 7f74be234640 1 -- start start 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.347+0000 7f74be234640 10 monclient: init 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 0x7f74b812f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient(hunting): picked mon.noname-b con 0x7f74b8067110 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74b809b010 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f74b809b400 0x7f74b809cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient(hunting): picked mon.noname-a con 0x7f74b809b400 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b8130540 con 0x7f74b809b010 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b8067ed0 con 0x7f74b809b400 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8099800 con 0x7f74b8067110 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74be234640 10 monclient(hunting): authenticate will time out at 1970.669178s 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 0x7f74b812f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f74b809b400 0x7f74b809cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 0x7f74b812f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47854/0 (socket says 172.21.15.116:47854) 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 -- 172.21.15.116:0/1064604451 learned_addr learned my addr 172.21.15.116:0/1064604451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 10 monclient(hunting): get_auth_request con 0x7f74b809b400 auth_method 0 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): get_auth_request con 0x7f74b8067110 auth_method 0 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient(hunting): handle_auth_done global_id 244139 payload 274 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 -- 172.21.15.116:0/1064604451 >> v2:172.21.15.116:3300/0 conn(0x7f74b809b400 msgr2=0x7f74b809cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 --2- 172.21.15.116:0/1064604451 >> v2:172.21.15.116:3300/0 conn(0x7f74b809b400 0x7f74b809cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 -- 172.21.15.116:0/1064604451 >> v1:172.21.15.144:6789/0 conn(0x7f74b809b010 legacy=0x7f74b809c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 monclient: found mon.noname-b 2024-09-18T12:45:43.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b77fe640 1 -- 172.21.15.116:0/1064604451 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b80674e0 con 0x7f74b8067110 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.348+0000 7f74b7fff640 1 --2- 172.21.15.116:0/1064604451 >> v2:172.21.15.116:3300/0 conn(0x7f74b809b400 0x7f74b809cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b77fe640 1 -- 172.21.15.116:0/1064604451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 msgr2=0x7f74b812f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b77fe640 1 --2- 172.21.15.116:0/1064604451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 0x7f74b812f670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009910 tx=0x7f74a80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1b4be323b9c71b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 1 -- 172.21.15.116:0/1064604451 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a802ae90 con 0x7f74b8067110 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: dump: 2024-09-18T12:45:43.363 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 1 -- 172.21.15.116:0/1064604451 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a8032440 con 0x7f74b8067110 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 1 -- 172.21.15.116:0/1064604451 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8032820 con 0x7f74b8067110 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74be234640 5 monclient: authenticate success, global_id 244139 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74be234640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74be234640 4 set_mon_vals no callback set 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.351+0000 7f74b67fc640 10 monclient: dump: 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.365 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 10 monclient: shutdown 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 -- 172.21.15.116:0/1064604451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 msgr2=0x7f74b812f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 --2- 172.21.15.116:0/1064604451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 0x7f74b812f670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009910 tx=0x7f74a80091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 -- 172.21.15.116:0/1064604451 shutdown_connections 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 --2- 172.21.15.116:0/1064604451 >> v2:172.21.15.116:3300/0 conn(0x7f74b809b400 0x7f74b809cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 --2- 172.21.15.116:0/1064604451 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b8067110 0x7f74b812f670 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 -- 172.21.15.116:0/1064604451 >> 172.21.15.116:0/1064604451 conn(0x7f74b80a1e30 msgr2=0x7f74b80a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 -- 172.21.15.116:0/1064604451 shutdown_connections 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 1 -- 172.21.15.116:0/1064604451 wait complete. 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.352+0000 7f74be234640 10 monclient: monmap: 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:43.367 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 Processor -- start 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 -- start start 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient: init 2024-09-18T12:45:43.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b8067110 0x7f74b812e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): picked mon.noname-b con 0x7f74b8067110 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74b809b010 0x7f74b813ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): picked mon.noname-a con 0x7f74b809b010 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 0x7f74b813f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): picked mon.noname-c con 0x7f74b809b400 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b811b570 con 0x7f74b809b010 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b811cd90 con 0x7f74b8067110 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b809e330 con 0x7f74b809b400 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74be234640 10 monclient(hunting): authenticate will time out at 1970.674730s 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74b809b010 0x7f74b813ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 0x7f74b813f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 0x7f74b813f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47856/0 (socket says 172.21.15.116:47856) 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 1 -- 172.21.15.116:0/1925139467 learned_addr learned my addr 172.21.15.116:0/1925139467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 10 monclient(hunting): get_auth_request con 0x7f74b809b400 auth_method 0 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 10 monclient(hunting): get_auth_request con 0x7f74b809b010 auth_method 0 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.353+0000 7f74b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b77fe640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b8067110 0x7f74b812e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b77fe640 10 monclient(hunting): get_auth_request con 0x7f74b8067110 auth_method 0 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 10 monclient(hunting): handle_auth_done global_id 244142 payload 824 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 1 -- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b8067110 msgr2=0x7f74b812e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b8067110 0x7f74b812e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 1 -- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74b809b010 msgr2=0x7f74b813ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74b809b010 0x7f74b813ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 1 monclient: found mon.noname-c 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b7fff640 1 -- 172.21.15.116:0/1925139467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0009a20 con 0x7f74b809b400 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.354+0000 7f74b6ffd640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74b809b010 0x7f74b813ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74b7fff640 1 -- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 msgr2=0x7f74b813f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74b7fff640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 0x7f74b813f260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c0f0 tx=0x7f74b000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 1 -- 172.21.15.116:0/1925139467 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b000ced0 con 0x7f74b809b400 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: dump: 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.373 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74be234640 5 monclient: authenticate success, global_id 244142 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74be234640 10 monclient: _renew_subs 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 1 -- 172.21.15.116:0/1925139467 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b000f040 con 0x7f74b809b400 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74be234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a8009450 con 0x7f74b809b400 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 1 -- 172.21.15.116:0/1925139467 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b00148e0 con 0x7f74b809b400 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f749f7fe640 10 monclient: dump: 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74be234640 10 monclient: _renew_subs 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74be234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b812eec0 con 0x7f74b809b400 2024-09-18T12:45:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.356+0000 7f74b5ffb640 4 set_mon_vals no callback set 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f749f7fe640 1 -- 172.21.15.116:0/1925139467 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f74b0003020 con 0x7f74b809b400 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f74be234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f74be234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f74be234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b804bf60 con 0x7f74b809b400 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f749f7fe640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74a4038490 0x7f74a403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f749f7fe640 1 -- 172.21.15.116:0/1925139467 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f74b004ce20 con 0x7f74b809b400 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f74b77fe640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74a4038490 0x7f74a403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.357+0000 7f74b77fe640 10 monclient: get_auth_request con 0x7f74a4038490 auth_method 0 2024-09-18T12:45:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.360+0000 7f74b77fe640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74a4038490 0x7f74a403a940 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f74a8034000 tx=0x7f74a8032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.361+0000 7f749f7fe640 1 -- 172.21.15.116:0/1925139467 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f74b0012df0 con 0x7f74b809b400 2024-09-18T12:45:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.361+0000 7f749f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.361+0000 7f749f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:43.385 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:43.384+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:43.385 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:43.384+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_snap=2,osdmap=50} crc {osd_metadata=1559247500,osd_snap=2033388227,osdmap=658690280}) 2024-09-18T12:45:43.385 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:43.384+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {osd_metadata=1559247500,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=2358447857}) 2024-09-18T12:45:43.385 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:43.384+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T12:45:43.386 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:43.384+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_snap=2,osdmap=50} crc {osd_metadata=1559247500,osd_snap=2033388227,osdmap=658690280}) 2024-09-18T12:45:43.386 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-09-18T12:45:43.384+0000 7f291cff1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {osd_metadata=1559247500,osd_pg_creating=2312870804,osd_snap=2033388227,osdmap=1448828345}) 2024-09-18T12:45:43.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.517+0000 7f74be234640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:45:43.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.517+0000 7f74be234640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:45:43.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.517+0000 7f74be234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.517+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b811c720 con 0x7f74b809b400 2024-09-18T12:45:43.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.518+0000 7f749f7fe640 1 -- 172.21.15.116:0/1925139467 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) v1 ==== 74+0+20424 (secure 0 0 0) 0x7f74b0007e60 con 0x7f74b809b400 2024-09-18T12:45:43.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.518+0000 7f749f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:45:43.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.518+0000 7f749f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:43.519 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:45:43.519 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":267,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:34:45.514528+0000","last_up_change":"2024-09-18T12:25:21.884529+0000","last_in_change":"2024-09-18T12:30:33.575141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":181,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T12:25:22.596367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T12:25:24.541264+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-18T12:30:56.176913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-18T12:30:56.676013+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":31578},{"type":"v1","addr":"172.21.15.116:6817","nonce":31578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":31578},{"type":"v1","addr":"172.21.15.116:6819","nonce":31578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":31578},{"type":"v1","addr":"172.21.15.116:6823","nonce":31578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":31578},{"type":"v1","addr":"172.21.15.116:6821","nonce":31578}]},"public_addr":"172.21.15.116:6817/31578","cluster_addr":"172.21.15.116:6819/31578","heartbeat_back_addr":"172.21.15.116:6823/31578","heartbeat_front_addr":"172.21.15.116:6821/31578","state":["exists","up"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":31579},{"type":"v1","addr":"172.21.15.116:6801","nonce":31579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":31579},{"type":"v1","addr":"172.21.15.116:6803","nonce":31579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":31579},{"type":"v1","addr":"172.21.15.116:6807","nonce":31579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":31579},{"type":"v1","addr":"172.21.15.116:6805","nonce":31579}]},"public_addr":"172.21.15.116:6801/31579","cluster_addr":"172.21.15.116:6803/31579","heartbeat_back_addr":"172.21.15.116:6807/31579","heartbeat_front_addr":"172.21.15.116:6805/31579","state":["exists","up"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":31580},{"type":"v1","addr":"172.21.15.116:6809","nonce":31580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":31580},{"type":"v1","addr":"172.21.15.116:6811","nonce":31580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":31580},{"type":"v1","addr":"172.21.15.116:6815","nonce":31580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":31580},{"type":"v1","addr":"172.21.15.116:6813","nonce":31580}]},"public_addr":"172.21.15.116:6809/31580","cluster_addr":"172.21.15.116:6811/31580","heartbeat_back_addr":"172.21.15.116:6815/31580","heartbeat_front_addr":"172.21.15.116:6813/31580","state":["exists","up"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":32112},{"type":"v1","addr":"172.21.15.144:6806","nonce":32112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":32112},{"type":"v1","addr":"172.21.15.144:6811","nonce":32112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":32112},{"type":"v1","addr":"172.21.15.144:6818","nonce":32112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":32112},{"type":"v1","addr":"172.21.15.144:6815","nonce":32112}]},"public_addr":"172.21.15.144:6806/32112","cluster_addr":"172.21.15.144:6811/32112","heartbeat_back_addr":"172.21.15.144:6818/32112","heartbeat_front_addr":"172.21.15.144:6815/32112","state":["exists","up"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":32113},{"type":"v1","addr":"172.21.15.144:6801","nonce":32113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":32113},{"type":"v1","addr":"172.21.15.144:6803","nonce":32113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":32113},{"type":"v1","addr":"172.21.15.144:6810","nonce":32113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6805","nonce":32113},{"type":"v1","addr":"172.21.15.144:6807","nonce":32113}]},"public_addr":"172.21.15.144:6801/32113","cluster_addr":"172.21.15.144:6803/32113","heartbeat_back_addr":"172.21.15.144:6810/32113","heartbeat_front_addr":"172.21.15.144:6807/32113","state":["exists","up"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":32114},{"type":"v1","addr":"172.21.15.144:6814","nonce":32114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":32114},{"type":"v1","addr":"172.21.15.144:6819","nonce":32114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":32114},{"type":"v1","addr":"172.21.15.144:6823","nonce":32114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":32114},{"type":"v1","addr":"172.21.15.144:6821","nonce":32114}]},"public_addr":"172.21.15.144:6814/32114","cluster_addr":"172.21.15.144:6819/32114","heartbeat_back_addr":"172.21.15.144:6823/32114","heartbeat_front_addr":"172.21.15.144:6821/32114","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"4b751cce-d39e-4f85-9cff-56f6325f3ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.664684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.608350+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.689049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.706015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.672781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.692341+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74a4038490 msgr2=0x7f74a403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74a4038490 0x7f74a403a940 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f74a8034000 tx=0x7f74a8032000 comp rx=0 tx=0).stop 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 10 monclient: shutdown 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 msgr2=0x7f74b813f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 0x7f74b813f260 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74b000c0f0 tx=0x7f74b000b040 comp rx=0 tx=0).stop 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 shutdown_connections 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f74a4038490 0x7f74a403a940 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f74b809b400 0x7f74b813f260 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74b809b010 0x7f74b813ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 --2- 172.21.15.116:0/1925139467 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f74b8067110 0x7f74b812e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 >> 172.21.15.116:0/1925139467 conn(0x7f74b80a1e30 msgr2=0x7f74b809cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:43.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 shutdown_connections 2024-09-18T12:45:43.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.520+0000 7f74be234640 1 -- 172.21.15.116:0/1925139467 wait complete. 2024-09-18T12:45:43.531 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T12:45:43.532 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:45:43.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.624+0000 7fe94e4c0640 10 monclient: get_monmap_and_config 2024-09-18T12:45:43.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.624+0000 7fe94e4c0640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.624+0000 7fe94e4c0640 10 monclient: monmap: 2024-09-18T12:45:43.625 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:43.625420+0000 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:43.625420+0000 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:43.626 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.625+0000 7fe94e4c0640 1 Processor -- start 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.625+0000 7fe94e4c0640 1 -- start start 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.625+0000 7fe94e4c0640 10 monclient: init 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9480975f0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9480979c0 0x7fe948097dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient(hunting): picked mon.noname-b con 0x7fe9480979c0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9481349f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94809d700 con 0x7fe9480975f0 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94809d1b0 con 0x7fe9481349f0 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480961e0 con 0x7fe9480979c0 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe94e4c0640 10 monclient(hunting): authenticate will time out at 1970.947446s 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9480979c0 0x7fe948097dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe947fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe9480975f0 0x7fe948095ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56424/0 (socket says 172.21.15.116:56424) 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe947fff640 1 -- 172.21.15.116:0/2661309402 learned_addr learned my addr 172.21.15.116:0/2661309402 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): get_auth_request con 0x7fe9480979c0 auth_method 0 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient(hunting): handle_auth_done global_id 244145 payload 274 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 1 -- 172.21.15.116:0/2661309402 >> v1:172.21.15.144:6789/0 conn(0x7fe9481349f0 legacy=0x7fe948136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 1 -- 172.21.15.116:0/2661309402 >> v1:172.21.15.116:6789/0 conn(0x7fe9480975f0 legacy=0x7fe948095ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 1 monclient: found mon.noname-b 2024-09-18T12:45:43.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.626+0000 7fe93ffff640 1 -- 172.21.15.116:0/2661309402 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94809d420 con 0x7fe9480979c0 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe93ffff640 1 -- 172.21.15.116:0/2661309402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9480979c0 msgr2=0x7fe948097dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe93ffff640 1 --2- 172.21.15.116:0/2661309402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9480979c0 0x7fe948097dc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe930009880 tx=0x7fe9300090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cdabc21258099f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 1 -- 172.21.15.116:0/2661309402 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93002aea0 con 0x7fe9480979c0 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: dump: 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.641 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 1 -- 172.21.15.116:0/2661309402 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe930032440 con 0x7fe9480979c0 2024-09-18T12:45:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe94e4c0640 5 monclient: authenticate success, global_id 244145 2024-09-18T12:45:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe94e4c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe94e4c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 1 -- 172.21.15.116:0/2661309402 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930032820 con 0x7fe9480979c0 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe94e4c0640 4 set_mon_vals no callback set 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.629+0000 7fe9477fe640 10 monclient: dump: 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 10 monclient: shutdown 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 1 -- 172.21.15.116:0/2661309402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9480979c0 msgr2=0x7fe948097dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/2661309402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9480979c0 0x7fe948097dc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe930009880 tx=0x7fe9300090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 1 -- 172.21.15.116:0/2661309402 shutdown_connections 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/2661309402 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9480979c0 0x7fe948097dc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 1 -- 172.21.15.116:0/2661309402 >> 172.21.15.116:0/2661309402 conn(0x7fe9480678a0 msgr2=0x7fe9480a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 1 -- 172.21.15.116:0/2661309402 shutdown_connections 2024-09-18T12:45:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.630+0000 7fe94e4c0640 1 -- 172.21.15.116:0/2661309402 wait complete. 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient: monmap: 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:43.644 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 Processor -- start 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 -- start start 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient: init 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9480975f0 0x7fe948087ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): picked mon.noname-b con 0x7fe9480975f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9480979c0 0x7fe948088530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): picked mon.noname-a con 0x7fe9480979c0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 0x7fe948088a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): picked mon.noname-c con 0x7fe9481349f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480983c0 con 0x7fe9480979c0 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94809f810 con 0x7fe9480975f0 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94808b5d0 con 0x7fe9481349f0 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94e4c0640 10 monclient(hunting): authenticate will time out at 1970.952893s 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe93ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9480979c0 0x7fe948088530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94ca5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 0x7fe948088a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94ca5d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 0x7fe948088a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47880/0 (socket says 172.21.15.116:47880) 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe93ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9480979c0 0x7fe948088530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41822/0 (socket says 172.21.15.116:41822) 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94ca5d640 1 -- 172.21.15.116:0/1246413890 learned_addr learned my addr 172.21.15.116:0/1246413890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.631+0000 7fe94ca5d640 10 monclient(hunting): get_auth_request con 0x7fe9481349f0 auth_method 0 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe93ffff640 10 monclient(hunting): get_auth_request con 0x7fe9480979c0 auth_method 0 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe93ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe93ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe947fff640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9480975f0 0x7fe948087ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe93ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe947fff640 10 monclient(hunting): get_auth_request con 0x7fe9480975f0 auth_method 0 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe947fff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient(hunting): handle_auth_done global_id 244148 payload 824 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 1 -- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9480975f0 msgr2=0x7fe948087ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9480975f0 0x7fe948087ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 1 -- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9480979c0 msgr2=0x7fe948088530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9480979c0 0x7fe948088530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 1 monclient: found mon.noname-c 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe94ca5d640 1 -- 172.21.15.116:0/1246413890 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930009390 con 0x7fe9481349f0 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.632+0000 7fe947fff640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9480975f0 0x7fe948087ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94ca5d640 1 -- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 msgr2=0x7fe948088a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94ca5d640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 0x7fe948088a70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c810 tx=0x7fe940002620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 1 -- 172.21.15.116:0/1246413890 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94000ce00 con 0x7fe9481349f0 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: dump: 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94e4c0640 5 monclient: authenticate success, global_id 244148 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94e4c0640 10 monclient: _renew_subs 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 1 -- 172.21.15.116:0/1246413890 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94000aa60 con 0x7fe9481349f0 2024-09-18T12:45:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94e4c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe948089c30 con 0x7fe9481349f0 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 1 -- 172.21.15.116:0/1246413890 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94000ae20 con 0x7fe9481349f0 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe944ff9640 10 monclient: dump: 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94e4c0640 10 monclient: _renew_subs 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe946ffd640 4 set_mon_vals no callback set 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94e4c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.634+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94808ab70 con 0x7fe9481349f0 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.635+0000 7fe944ff9640 1 -- 172.21.15.116:0/1246413890 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe940012630 con 0x7fe9481349f0 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.635+0000 7fe94e4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.635+0000 7fe94e4c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.635+0000 7fe94e4c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.635+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94804bf60 con 0x7fe9481349f0 2024-09-18T12:45:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.635+0000 7fe944ff9640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe928038490 0x7fe92803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.635+0000 7fe944ff9640 1 -- 172.21.15.116:0/1246413890 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe94004bfc0 con 0x7fe9481349f0 2024-09-18T12:45:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.636+0000 7fe947fff640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe928038490 0x7fe92803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.636+0000 7fe947fff640 10 monclient: get_auth_request con 0x7fe928038490 auth_method 0 2024-09-18T12:45:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.638+0000 7fe947fff640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe928038490 0x7fe92803a940 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe938007fd0 tx=0x7fe938009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.640+0000 7fe944ff9640 1 -- 172.21.15.116:0/1246413890 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fe9400188c0 con 0x7fe9481349f0 2024-09-18T12:45:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.640+0000 7fe944ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.640+0000 7fe944ff9640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.799+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe94808b340 con 0x7fe928038490 2024-09-18T12:45:43.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.803+0000 7fe944ff9640 1 -- 172.21.15.116:0/1246413890 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97874 (secure 0 0 0) 0x7fe94808b340 con 0x7fe928038490 2024-09-18T12:45:43.804 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:45:43.805 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe928038490 msgr2=0x7fe92803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe928038490 0x7fe92803a940 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe938007fd0 tx=0x7fe938009040 comp rx=0 tx=0).stop 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 10 monclient: shutdown 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 msgr2=0x7fe948088a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 0x7fe948088a70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c810 tx=0x7fe940002620 comp rx=0 tx=0).stop 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 shutdown_connections 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe928038490 0x7fe92803a940 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe9481349f0 0x7fe948088a70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9480979c0 0x7fe948088530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 --2- 172.21.15.116:0/1246413890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe9480975f0 0x7fe948087ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 >> 172.21.15.116:0/1246413890 conn(0x7fe9480678a0 msgr2=0x7fe948136dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:43.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 shutdown_connections 2024-09-18T12:45:43.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.805+0000 7fe94e4c0640 1 -- 172.21.15.116:0/1246413890 wait complete. 2024-09-18T12:45:43.816 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":872,"stamp":"2024-09-18T12:45:42.816617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1815804,"kb_used_data":10456,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564415236,"statfs":{"total":579820584960,"available":577961201664,"internally_reserved":0,"allocated":10706944,"data_stored":8512606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":175,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946900+0000","last_change":"2024-09-18T12:32:22.608257+0000","last_active":"2024-09-18T12:34:45.946900+0000","last_peered":"2024-09-18T12:34:45.946900+0000","last_clean":"2024-09-18T12:34:45.946900+0000","last_became_active":"2024-09-18T12:32:22.607844+0000","last_became_peered":"2024-09-18T12:32:22.607844+0000","last_unstale":"2024-09-18T12:34:45.946900+0000","last_undegraded":"2024-09-18T12:34:45.946900+0000","last_fullsized":"2024-09-18T12:34:45.946900+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:52.306141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947577+0000","last_change":"2024-09-18T12:32:22.602302+0000","last_active":"2024-09-18T12:34:42.947577+0000","last_peered":"2024-09-18T12:34:42.947577+0000","last_clean":"2024-09-18T12:34:42.947577+0000","last_became_active":"2024-09-18T12:32:22.601974+0000","last_became_peered":"2024-09-18T12:32:22.601974+0000","last_unstale":"2024-09-18T12:34:42.947577+0000","last_undegraded":"2024-09-18T12:34:42.947577+0000","last_fullsized":"2024-09-18T12:34:42.947577+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:05.401601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510608+0000","last_change":"2024-09-18T12:32:22.605364+0000","last_active":"2024-09-18T12:34:44.510608+0000","last_peered":"2024-09-18T12:34:44.510608+0000","last_clean":"2024-09-18T12:34:44.510608+0000","last_became_active":"2024-09-18T12:32:22.604956+0000","last_became_peered":"2024-09-18T12:32:22.604956+0000","last_unstale":"2024-09-18T12:34:44.510608+0000","last_undegraded":"2024-09-18T12:34:44.510608+0000","last_fullsized":"2024-09-18T12:34:44.510608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:21:23.773463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":83,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947446+0000","last_change":"2024-09-18T12:32:22.604750+0000","last_active":"2024-09-18T12:34:42.947446+0000","last_peered":"2024-09-18T12:34:42.947446+0000","last_clean":"2024-09-18T12:34:42.947446+0000","last_became_active":"2024-09-18T12:32:22.604332+0000","last_became_peered":"2024-09-18T12:32:22.604332+0000","last_unstale":"2024-09-18T12:34:42.947446+0000","last_undegraded":"2024-09-18T12:34:42.947446+0000","last_fullsized":"2024-09-18T12:34:42.947446+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:50:27.338272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":127,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947518+0000","last_change":"2024-09-18T12:32:23.804087+0000","last_active":"2024-09-18T12:34:42.947518+0000","last_peered":"2024-09-18T12:34:42.947518+0000","last_clean":"2024-09-18T12:34:42.947518+0000","last_became_active":"2024-09-18T12:32:23.803732+0000","last_became_peered":"2024-09-18T12:32:23.803732+0000","last_unstale":"2024-09-18T12:34:42.947518+0000","last_undegraded":"2024-09-18T12:34:42.947518+0000","last_fullsized":"2024-09-18T12:34:42.947518+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:19:27.867616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":151,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946811+0000","last_change":"2024-09-18T12:32:22.604980+0000","last_active":"2024-09-18T12:34:45.946811+0000","last_peered":"2024-09-18T12:34:45.946811+0000","last_clean":"2024-09-18T12:34:45.946811+0000","last_became_active":"2024-09-18T12:32:22.604577+0000","last_became_peered":"2024-09-18T12:32:22.604577+0000","last_unstale":"2024-09-18T12:34:45.946811+0000","last_undegraded":"2024-09-18T12:34:45.946811+0000","last_fullsized":"2024-09-18T12:34:45.946811+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:17:06.770342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144754+0000","last_change":"2024-09-18T12:32:22.608738+0000","last_active":"2024-09-18T12:34:43.144754+0000","last_peered":"2024-09-18T12:34:43.144754+0000","last_clean":"2024-09-18T12:34:43.144754+0000","last_became_active":"2024-09-18T12:32:22.608401+0000","last_became_peered":"2024-09-18T12:32:22.608401+0000","last_unstale":"2024-09-18T12:34:43.144754+0000","last_undegraded":"2024-09-18T12:34:43.144754+0000","last_fullsized":"2024-09-18T12:34:43.144754+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:59.037772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511904+0000","last_change":"2024-09-18T12:32:22.603874+0000","last_active":"2024-09-18T12:34:44.511904+0000","last_peered":"2024-09-18T12:34:44.511904+0000","last_clean":"2024-09-18T12:34:44.511904+0000","last_became_active":"2024-09-18T12:32:22.603533+0000","last_became_peered":"2024-09-18T12:32:22.603533+0000","last_unstale":"2024-09-18T12:34:44.511904+0000","last_undegraded":"2024-09-18T12:34:44.511904+0000","last_fullsized":"2024-09-18T12:34:44.511904+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:29.278327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947428+0000","last_change":"2024-09-18T12:32:22.607436+0000","last_active":"2024-09-18T12:34:42.947428+0000","last_peered":"2024-09-18T12:34:42.947428+0000","last_clean":"2024-09-18T12:34:42.947428+0000","last_became_active":"2024-09-18T12:32:22.607051+0000","last_became_peered":"2024-09-18T12:32:22.607051+0000","last_unstale":"2024-09-18T12:34:42.947428+0000","last_undegraded":"2024-09-18T12:34:42.947428+0000","last_fullsized":"2024-09-18T12:34:42.947428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:38.371976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144608+0000","last_change":"2024-09-18T12:32:22.604621+0000","last_active":"2024-09-18T12:34:43.144608+0000","last_peered":"2024-09-18T12:34:43.144608+0000","last_clean":"2024-09-18T12:34:43.144608+0000","last_became_active":"2024-09-18T12:32:22.604295+0000","last_became_peered":"2024-09-18T12:32:22.604295+0000","last_unstale":"2024-09-18T12:34:43.144608+0000","last_undegraded":"2024-09-18T12:34:43.144608+0000","last_fullsized":"2024-09-18T12:34:43.144608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:48:28.798147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":130,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144436+0000","last_change":"2024-09-18T12:32:22.606967+0000","last_active":"2024-09-18T12:34:43.144436+0000","last_peered":"2024-09-18T12:34:43.144436+0000","last_clean":"2024-09-18T12:34:43.144436+0000","last_became_active":"2024-09-18T12:32:22.606557+0000","last_became_peered":"2024-09-18T12:32:22.606557+0000","last_unstale":"2024-09-18T12:34:43.144436+0000","last_undegraded":"2024-09-18T12:34:43.144436+0000","last_fullsized":"2024-09-18T12:34:43.144436+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:13:49.746180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511862+0000","last_change":"2024-09-18T12:32:22.608034+0000","last_active":"2024-09-18T12:34:44.511862+0000","last_peered":"2024-09-18T12:34:44.511862+0000","last_clean":"2024-09-18T12:34:44.511862+0000","last_became_active":"2024-09-18T12:32:22.607698+0000","last_became_peered":"2024-09-18T12:32:22.607698+0000","last_unstale":"2024-09-18T12:34:44.511862+0000","last_undegraded":"2024-09-18T12:34:44.511862+0000","last_fullsized":"2024-09-18T12:34:44.511862+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:13.180633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":90,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511848+0000","last_change":"2024-09-18T12:32:22.606942+0000","last_active":"2024-09-18T12:34:44.511848+0000","last_peered":"2024-09-18T12:34:44.511848+0000","last_clean":"2024-09-18T12:34:44.511848+0000","last_became_active":"2024-09-18T12:32:22.606593+0000","last_became_peered":"2024-09-18T12:32:22.606593+0000","last_unstale":"2024-09-18T12:34:44.511848+0000","last_undegraded":"2024-09-18T12:34:44.511848+0000","last_fullsized":"2024-09-18T12:34:44.511848+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:53:56.502695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":182,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946736+0000","last_change":"2024-09-18T12:32:22.608147+0000","last_active":"2024-09-18T12:34:45.946736+0000","last_peered":"2024-09-18T12:34:45.946736+0000","last_clean":"2024-09-18T12:34:45.946736+0000","last_became_active":"2024-09-18T12:32:22.607839+0000","last_became_peered":"2024-09-18T12:32:22.607839+0000","last_unstale":"2024-09-18T12:34:45.946736+0000","last_undegraded":"2024-09-18T12:34:45.946736+0000","last_fullsized":"2024-09-18T12:34:45.946736+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:11:45.988633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510548+0000","last_change":"2024-09-18T12:32:23.599583+0000","last_active":"2024-09-18T12:34:44.510548+0000","last_peered":"2024-09-18T12:34:44.510548+0000","last_clean":"2024-09-18T12:34:44.510548+0000","last_became_active":"2024-09-18T12:32:23.599235+0000","last_became_peered":"2024-09-18T12:32:23.599235+0000","last_unstale":"2024-09-18T12:34:44.510548+0000","last_undegraded":"2024-09-18T12:34:44.510548+0000","last_fullsized":"2024-09-18T12:34:44.510548+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:33.154296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511941+0000","last_change":"2024-09-18T12:32:22.600178+0000","last_active":"2024-09-18T12:34:44.511941+0000","last_peered":"2024-09-18T12:34:44.511941+0000","last_clean":"2024-09-18T12:34:44.511941+0000","last_became_active":"2024-09-18T12:32:22.599859+0000","last_became_peered":"2024-09-18T12:32:22.599859+0000","last_unstale":"2024-09-18T12:34:44.511941+0000","last_undegraded":"2024-09-18T12:34:44.511941+0000","last_fullsized":"2024-09-18T12:34:44.511941+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:19:22.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":338,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144609+0000","last_change":"2024-09-18T12:32:22.605174+0000","last_active":"2024-09-18T12:34:43.144609+0000","last_peered":"2024-09-18T12:34:43.144609+0000","last_clean":"2024-09-18T12:34:43.144609+0000","last_became_active":"2024-09-18T12:32:22.604820+0000","last_became_peered":"2024-09-18T12:32:22.604820+0000","last_unstale":"2024-09-18T12:34:43.144609+0000","last_undegraded":"2024-09-18T12:34:43.144609+0000","last_fullsized":"2024-09-18T12:34:43.144609+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:41:49.881448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":266,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947516+0000","last_change":"2024-09-18T12:32:22.610862+0000","last_active":"2024-09-18T12:34:42.947516+0000","last_peered":"2024-09-18T12:34:42.947516+0000","last_clean":"2024-09-18T12:34:42.947516+0000","last_became_active":"2024-09-18T12:32:22.604471+0000","last_became_peered":"2024-09-18T12:32:22.604471+0000","last_unstale":"2024-09-18T12:34:42.947516+0000","last_undegraded":"2024-09-18T12:34:42.947516+0000","last_fullsized":"2024-09-18T12:34:42.947516+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:31.307834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144612+0000","last_change":"2024-09-18T12:32:22.608239+0000","last_active":"2024-09-18T12:34:43.144612+0000","last_peered":"2024-09-18T12:34:43.144612+0000","last_clean":"2024-09-18T12:34:43.144612+0000","last_became_active":"2024-09-18T12:32:22.607827+0000","last_became_peered":"2024-09-18T12:32:22.607827+0000","last_unstale":"2024-09-18T12:34:43.144612+0000","last_undegraded":"2024-09-18T12:34:43.144612+0000","last_fullsized":"2024-09-18T12:34:43.144612+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:40.035295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510540+0000","last_change":"2024-09-18T12:32:22.604079+0000","last_active":"2024-09-18T12:34:44.510540+0000","last_peered":"2024-09-18T12:34:44.510540+0000","last_clean":"2024-09-18T12:34:44.510540+0000","last_became_active":"2024-09-18T12:32:22.603727+0000","last_became_peered":"2024-09-18T12:32:22.603727+0000","last_unstale":"2024-09-18T12:34:44.510540+0000","last_undegraded":"2024-09-18T12:34:44.510540+0000","last_fullsized":"2024-09-18T12:34:44.510540+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:42:39.085495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":268,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947425+0000","last_change":"2024-09-18T12:32:22.615941+0000","last_active":"2024-09-18T12:34:42.947425+0000","last_peered":"2024-09-18T12:34:42.947425+0000","last_clean":"2024-09-18T12:34:42.947425+0000","last_became_active":"2024-09-18T12:32:22.608226+0000","last_became_peered":"2024-09-18T12:32:22.608226+0000","last_unstale":"2024-09-18T12:34:42.947425+0000","last_undegraded":"2024-09-18T12:34:42.947425+0000","last_fullsized":"2024-09-18T12:34:42.947425+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:13:36.171958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":264,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511965+0000","last_change":"2024-09-18T12:32:23.598518+0000","last_active":"2024-09-18T12:34:44.511965+0000","last_peered":"2024-09-18T12:34:44.511965+0000","last_clean":"2024-09-18T12:34:44.511965+0000","last_became_active":"2024-09-18T12:32:23.598179+0000","last_became_peered":"2024-09-18T12:32:23.598179+0000","last_unstale":"2024-09-18T12:34:44.511965+0000","last_undegraded":"2024-09-18T12:34:44.511965+0000","last_fullsized":"2024-09-18T12:34:44.511965+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:42.637189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144647+0000","last_change":"2024-09-18T12:32:22.605496+0000","last_active":"2024-09-18T12:34:43.144647+0000","last_peered":"2024-09-18T12:34:43.144647+0000","last_clean":"2024-09-18T12:34:43.144647+0000","last_became_active":"2024-09-18T12:32:22.605190+0000","last_became_peered":"2024-09-18T12:32:22.605190+0000","last_unstale":"2024-09-18T12:34:43.144647+0000","last_undegraded":"2024-09-18T12:34:43.144647+0000","last_fullsized":"2024-09-18T12:34:43.144647+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:50.861016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511899+0000","last_change":"2024-09-18T12:32:22.603956+0000","last_active":"2024-09-18T12:34:44.511899+0000","last_peered":"2024-09-18T12:34:44.511899+0000","last_clean":"2024-09-18T12:34:44.511899+0000","last_became_active":"2024-09-18T12:32:22.603556+0000","last_became_peered":"2024-09-18T12:32:22.603556+0000","last_unstale":"2024-09-18T12:34:44.511899+0000","last_undegraded":"2024-09-18T12:34:44.511899+0000","last_fullsized":"2024-09-18T12:34:44.511899+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:40:30.975910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144746+0000","last_change":"2024-09-18T12:32:22.601754+0000","last_active":"2024-09-18T12:34:43.144746+0000","last_peered":"2024-09-18T12:34:43.144746+0000","last_clean":"2024-09-18T12:34:43.144746+0000","last_became_active":"2024-09-18T12:32:22.601418+0000","last_became_peered":"2024-09-18T12:32:22.601418+0000","last_unstale":"2024-09-18T12:34:43.144746+0000","last_undegraded":"2024-09-18T12:34:43.144746+0000","last_fullsized":"2024-09-18T12:34:43.144746+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:18:06.635553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640501,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.45900000000000002},"min":{"1min":0.40000000000000002,"5min":0.32400000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.69099999999999995},"last":0.495},{"interface":"front","average":{"1min":0.499,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.20999999999999999},"max":{"1min":0.57999999999999996,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53900000000000003,"15min":0.51700000000000002},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.23999999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.48999999999999999,"5min":0.39900000000000002,"15min":0.27200000000000002},"max":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.80200000000000005},"last":0.504}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":1.304,"15min":1.304},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.25700000000000001},"max":{"1min":0.63300000000000001,"5min":0.72799999999999998,"15min":0.753},"last":0.46999999999999997}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.371,"15min":0.36199999999999999},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":0.51600000000000001,"5min":0.59599999999999997,"15min":0.60299999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.20100000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.64200000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":0.58899999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.223},"max":{"1min":0.61199999999999999,"5min":0.68200000000000005,"15min":0.73799999999999999},"last":0.371}]}]},{"osd":4,"up_from":11,"seq":47244640501,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298924,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072916,"statfs":{"total":96636764160,"available":96330665984,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.47699999999999998},"min":{"1min":0.37,"5min":0.27400000000000002,"15min":0.223},"max":{"1min":0.60199999999999998,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.378},{"interface":"front","average":{"1min":0.504,"5min":0.49199999999999999,"15min":0.47799999999999998},"min":{"1min":0.379,"5min":0.33100000000000002,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45400000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.25900000000000001},"max":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.753},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.71999999999999997,"15min":0.872},"last":0.46500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71499999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45100000000000001,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.443}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41499999999999998,"15min":0.38400000000000001},"min":{"1min":0.33600000000000002,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.59399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.307,"5min":0.255,"15min":0.249},"max":{"1min":0.53700000000000003,"5min":0.68899999999999995,"15min":0.70299999999999996},"last":0.28499999999999998}]},{"osd":5,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59499999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26100000000000001,"15min":0.223},"max":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.60999999999999999},"last":0.34399999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640504,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304012,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067828,"statfs":{"total":96636764160,"available":96325455872,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65300000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.34599999999999997,"15min":0.33100000000000002},"max":{"1min":0.68600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.5}]},{"osd":1,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51000000000000001,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.35599999999999998,"15min":0.26600000000000001},"max":{"1min":0.64500000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.68600000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.70099999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.503,"15min":0.5},"min":{"1min":0.37,"5min":0.33400000000000002,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.873,"15min":0.873},"last":0.48499999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36099999999999999,"15min":0.36499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.214},"max":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.624},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.36199999999999999,"15min":0.372},"min":{"1min":0.254,"5min":0.254,"15min":0.221},"max":{"1min":0.48499999999999999,"5min":0.58199999999999996,"15min":0.67200000000000004},"last":0.39100000000000001}]},{"osd":5,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36099999999999999,"15min":0.36799999999999999},"min":{"1min":0.26500000000000001,"5min":0.26100000000000001,"15min":0.20300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.64600000000000002,"15min":0.70599999999999996},"last":0.443}]}]},{"osd":2,"up_from":11,"seq":47244640503,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.315,"15min":0.32200000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.156},"max":{"1min":0.46800000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34399999999999997,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.628},"last":0.443}]},{"osd":1,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35299999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51700000000000002,"5min":0.60599999999999998,"15min":0.63900000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.33600000000000002,"15min":0.34200000000000003},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.17699999999999999},"max":{"1min":0.54200000000000004,"5min":0.625,"15min":0.625},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":0.77000000000000002,"15min":0.77300000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.308},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.72999999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75700000000000001,"5min":1.034,"15min":1.034},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.83099999999999996},"last":0.80000000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.40300000000000002,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":0.72999999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.746},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55100000000000005,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.78900000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640503,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":0.41999999999999998,"5min":0.48499999999999999,"15min":0.70699999999999996},"last":0.316},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33700000000000002,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.214},"max":{"1min":0.44700000000000001,"5min":0.629,"15min":0.80000000000000004},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34499999999999997,"15min":0.35899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":0.46800000000000003,"5min":0.52000000000000002,"15min":0.86599999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.26900000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48599999999999999,"5min":0.53600000000000003,"15min":0.82599999999999996},"last":0.375}]},{"osd":3,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.41699999999999998,"5min":0.377,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":0.84199999999999997,"15min":0.95699999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.54200000000000004},"min":{"1min":0.40699999999999997,"5min":0.375,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.83799999999999997,"15min":0.97899999999999998},"last":0.59499999999999997}]},{"osd":4,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.317},"max":{"1min":0.76400000000000001,"5min":0.82099999999999995,"15min":0.997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.34999999999999998},"max":{"1min":0.71399999999999997,"5min":0.85199999999999998,"15min":1.034},"last":0.53100000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":0.68200000000000005,"5min":0.82999999999999996,"15min":0.88300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.42199999999999999,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":0.625,"5min":0.81299999999999994,"15min":1.0960000000000001},"last":0.61399999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640502,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303608,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068232,"statfs":{"total":96636764160,"available":96325869568,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.30599999999999999,"15min":0.317},"min":{"1min":0.223,"5min":0.14799999999999999,"15min":0.13400000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.55500000000000005},"last":0.313},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.16400000000000001},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.64500000000000002},"last":0.375}]},{"osd":2,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.35499999999999998},"min":{"1min":0.21299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.48599999999999999,"5min":0.502,"15min":0.76800000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.36799999999999999},"min":{"1min":0.28499999999999998,"5min":0.221,"15min":0.16},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.73499999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.69499999999999995,"5min":1.177,"15min":1.177},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.53700000000000003},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.54100000000000004}]},{"osd":4,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":1.127,"15min":1.127},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.69999999999999996,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.54000000000000004},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":1.141,"15min":1.141},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34100000000000003},"max":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:45:43.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:45:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.909+0000 7f37488c0640 10 monclient: get_monmap_and_config 2024-09-18T12:45:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.909+0000 7f37488c0640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.909+0000 7f37488c0640 10 monclient: monmap: 2024-09-18T12:45:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:43.910320+0000 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:43.910320+0000 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:43.911 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 1 Processor -- start 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 1 -- start start 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient: init 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3744096c60 0x7f374409a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient(hunting): picked mon.noname-c con 0x7f3744096c60 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3744097030 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3744133b40 0x7f3744135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient(hunting): picked mon.noname-a con 0x7f3744133b40 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37440696a0 con 0x7f3744097030 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374409ac40 con 0x7f3744133b40 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374409d6b0 con 0x7f3744096c60 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f37488c0640 10 monclient(hunting): authenticate will time out at 1971.231870s 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.910+0000 7f3742d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3744133b40 0x7f3744135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3744133b40 0x7f3744135f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41830/0 (socket says 172.21.15.116:41830) 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 1 -- 172.21.15.116:0/2047499647 learned_addr learned my addr 172.21.15.116:0/2047499647 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): get_auth_request con 0x7f3744133b40 auth_method 0 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient(hunting): handle_auth_done global_id 194694 payload 274 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 1 -- 172.21.15.116:0/2047499647 >> v2:172.21.15.144:3300/0 conn(0x7f3744096c60 msgr2=0x7f374409a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f374259c640 1 --2- 172.21.15.116:0/2047499647 >> v2:172.21.15.144:3300/0 conn(0x7f3744096c60 0x7f374409a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 1 --2- 172.21.15.116:0/2047499647 >> v2:172.21.15.144:3300/0 conn(0x7f3744096c60 0x7f374409a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 1 -- 172.21.15.116:0/2047499647 >> v1:172.21.15.144:6789/0 conn(0x7f3744097030 legacy=0x7f3744097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 1 monclient: found mon.noname-a 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.911+0000 7f3742d9d640 1 -- 172.21.15.116:0/2047499647 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374409d2e0 con 0x7f3744133b40 2024-09-18T12:45:43.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f3742d9d640 1 -- 172.21.15.116:0/2047499647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 msgr2=0x7f3744135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f3742d9d640 1 --2- 172.21.15.116:0/2047499647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f3744135f20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f37440637c0 tx=0x7f373c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c490818a44126a01 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 1 -- 172.21.15.116:0/2047499647 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c007610 con 0x7f3744133b40 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: dump: 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 1 -- 172.21.15.116:0/2047499647 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c02d050 con 0x7f3744133b40 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 5 monclient: authenticate success, global_id 194694 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 1 -- 172.21.15.116:0/2047499647 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c0079a0 con 0x7f3744133b40 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 4 set_mon_vals no callback set 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f374159a640 10 monclient: dump: 2024-09-18T12:45:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 10 monclient: shutdown 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 -- 172.21.15.116:0/2047499647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 msgr2=0x7f3744135f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 --2- 172.21.15.116:0/2047499647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f3744135f20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f37440637c0 tx=0x7f373c009840 comp rx=0 tx=0).stop 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 -- 172.21.15.116:0/2047499647 shutdown_connections 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 --2- 172.21.15.116:0/2047499647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f3744135f20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 --2- 172.21.15.116:0/2047499647 >> v2:172.21.15.144:3300/0 conn(0x7f3744096c60 0x7f374409a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 -- 172.21.15.116:0/2047499647 >> 172.21.15.116:0/2047499647 conn(0x7f37440672b0 msgr2=0x7f37440676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 -- 172.21.15.116:0/2047499647 shutdown_connections 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.914+0000 7f37488c0640 1 -- 172.21.15.116:0/2047499647 wait complete. 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.915+0000 7f37488c0640 10 monclient: build_initial_monmap 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.915+0000 7f37488c0640 10 monclient: monmap: 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.915+0000 7f37488c0640 1 Processor -- start 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.915+0000 7f37488c0640 1 -- start start 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.915+0000 7f37488c0640 10 monclient: init 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 0x7f374409b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): picked mon.noname-c con 0x7f3744096c60 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744097030 0x7f37441322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): picked mon.noname-b con 0x7f3744097030 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f37441327e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): picked mon.noname-a con 0x7f3744133b40 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374406f500 con 0x7f3744133b40 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374409add0 con 0x7f3744097030 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3744135880 con 0x7f3744096c60 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f37488c0640 10 monclient(hunting): authenticate will time out at 1971.237090s 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 0x7f374409b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f37441327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 0x7f374409b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47886/0 (socket says 172.21.15.116:47886) 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 -- 172.21.15.116:0/2049646772 learned_addr learned my addr 172.21.15.116:0/2049646772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): get_auth_request con 0x7f3744096c60 auth_method 0 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 10 monclient(hunting): get_auth_request con 0x7f3744133b40 auth_method 0 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744097030 0x7f37441322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 10 monclient(hunting): get_auth_request con 0x7f3744097030 auth_method 0 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3741d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient(hunting): handle_auth_done global_id 244151 payload 824 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 -- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744097030 msgr2=0x7f37441322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744097030 0x7f37441322a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 -- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 msgr2=0x7f37441327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f37441327e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient: _finish_hunting 0 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 monclient: found mon.noname-c 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f374259c640 1 -- 172.21.15.116:0/2049646772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c009ae0 con 0x7f3744096c60 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.916+0000 7f3742d9d640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f37441327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f374259c640 1 -- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 msgr2=0x7f374409b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f374259c640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 0x7f374409b890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f373400c880 tx=0x7f3734008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 1 -- 172.21.15.116:0/2049646772 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373400ce10 con 0x7f3744096c60 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: dump: 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 1 -- 172.21.15.116:0/2049646772 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3734007a40 con 0x7f3744096c60 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37488c0640 5 monclient: authenticate success, global_id 244151 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37488c0640 10 monclient: _renew_subs 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37488c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374406a7e0 con 0x7f3744096c60 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 1 -- 172.21.15.116:0/2049646772 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734007e00 con 0x7f3744096c60 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37327fc640 10 monclient: dump: 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37488c0640 10 monclient: _renew_subs 2024-09-18T12:45:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37488c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374409bfd0 con 0x7f3744096c60 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.918+0000 7f3740d99640 4 set_mon_vals no callback set 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.919+0000 7f37327fc640 1 -- 172.21.15.116:0/2049646772 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3734011600 con 0x7f3744096c60 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.919+0000 7f37488c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.919+0000 7f37488c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.919+0000 7f37488c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.919+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374406a610 con 0x7f3744096c60 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.919+0000 7f37327fc640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3724038490 0x7f372403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.919+0000 7f37327fc640 1 -- 172.21.15.116:0/2049646772 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f373404c120 con 0x7f3744096c60 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.920+0000 7f3741d9b640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3724038490 0x7f372403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.920+0000 7f3741d9b640 10 monclient: get_auth_request con 0x7f3724038490 auth_method 0 2024-09-18T12:45:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.922+0000 7f3741d9b640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3724038490 0x7f372403a940 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3738006fd0 tx=0x7f3738009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.924+0000 7f37327fc640 1 -- 172.21.15.116:0/2049646772 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3734020ae0 con 0x7f3744096c60 2024-09-18T12:45:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.924+0000 7f37327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:43.924+0000 7f37327fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.077+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f374404bf60 con 0x7f3724038490 2024-09-18T12:45:44.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.080+0000 7f37327fc640 1 -- 172.21.15.116:0/2049646772 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97874 (secure 0 0 0) 0x7f374404bf60 con 0x7f3724038490 2024-09-18T12:45:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:45:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:45:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3724038490 msgr2=0x7f372403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3724038490 0x7f372403a940 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3738006fd0 tx=0x7f3738009060 comp rx=0 tx=0).stop 2024-09-18T12:45:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 10 monclient: shutdown 2024-09-18T12:45:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 msgr2=0x7f374409b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 0x7f374409b890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f373400c880 tx=0x7f3734008040 comp rx=0 tx=0).stop 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 shutdown_connections 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3724038490 0x7f372403a940 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3744133b40 0x7f37441327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3744097030 0x7f37441322a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 --2- 172.21.15.116:0/2049646772 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3744096c60 0x7f374409b890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 >> 172.21.15.116:0/2049646772 conn(0x7f37440672b0 msgr2=0x7f3744135cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 shutdown_connections 2024-09-18T12:45:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.082+0000 7f37488c0640 1 -- 172.21.15.116:0/2049646772 wait complete. 2024-09-18T12:45:44.093 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":872,"stamp":"2024-09-18T12:45:42.816617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1815804,"kb_used_data":10456,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564415236,"statfs":{"total":579820584960,"available":577961201664,"internally_reserved":0,"allocated":10706944,"data_stored":8512606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":175,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946900+0000","last_change":"2024-09-18T12:32:22.608257+0000","last_active":"2024-09-18T12:34:45.946900+0000","last_peered":"2024-09-18T12:34:45.946900+0000","last_clean":"2024-09-18T12:34:45.946900+0000","last_became_active":"2024-09-18T12:32:22.607844+0000","last_became_peered":"2024-09-18T12:32:22.607844+0000","last_unstale":"2024-09-18T12:34:45.946900+0000","last_undegraded":"2024-09-18T12:34:45.946900+0000","last_fullsized":"2024-09-18T12:34:45.946900+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:52.306141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947577+0000","last_change":"2024-09-18T12:32:22.602302+0000","last_active":"2024-09-18T12:34:42.947577+0000","last_peered":"2024-09-18T12:34:42.947577+0000","last_clean":"2024-09-18T12:34:42.947577+0000","last_became_active":"2024-09-18T12:32:22.601974+0000","last_became_peered":"2024-09-18T12:32:22.601974+0000","last_unstale":"2024-09-18T12:34:42.947577+0000","last_undegraded":"2024-09-18T12:34:42.947577+0000","last_fullsized":"2024-09-18T12:34:42.947577+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:05.401601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510608+0000","last_change":"2024-09-18T12:32:22.605364+0000","last_active":"2024-09-18T12:34:44.510608+0000","last_peered":"2024-09-18T12:34:44.510608+0000","last_clean":"2024-09-18T12:34:44.510608+0000","last_became_active":"2024-09-18T12:32:22.604956+0000","last_became_peered":"2024-09-18T12:32:22.604956+0000","last_unstale":"2024-09-18T12:34:44.510608+0000","last_undegraded":"2024-09-18T12:34:44.510608+0000","last_fullsized":"2024-09-18T12:34:44.510608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:21:23.773463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":83,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947446+0000","last_change":"2024-09-18T12:32:22.604750+0000","last_active":"2024-09-18T12:34:42.947446+0000","last_peered":"2024-09-18T12:34:42.947446+0000","last_clean":"2024-09-18T12:34:42.947446+0000","last_became_active":"2024-09-18T12:32:22.604332+0000","last_became_peered":"2024-09-18T12:32:22.604332+0000","last_unstale":"2024-09-18T12:34:42.947446+0000","last_undegraded":"2024-09-18T12:34:42.947446+0000","last_fullsized":"2024-09-18T12:34:42.947446+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:50:27.338272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":127,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947518+0000","last_change":"2024-09-18T12:32:23.804087+0000","last_active":"2024-09-18T12:34:42.947518+0000","last_peered":"2024-09-18T12:34:42.947518+0000","last_clean":"2024-09-18T12:34:42.947518+0000","last_became_active":"2024-09-18T12:32:23.803732+0000","last_became_peered":"2024-09-18T12:32:23.803732+0000","last_unstale":"2024-09-18T12:34:42.947518+0000","last_undegraded":"2024-09-18T12:34:42.947518+0000","last_fullsized":"2024-09-18T12:34:42.947518+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:19:27.867616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":151,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946811+0000","last_change":"2024-09-18T12:32:22.604980+0000","last_active":"2024-09-18T12:34:45.946811+0000","last_peered":"2024-09-18T12:34:45.946811+0000","last_clean":"2024-09-18T12:34:45.946811+0000","last_became_active":"2024-09-18T12:32:22.604577+0000","last_became_peered":"2024-09-18T12:32:22.604577+0000","last_unstale":"2024-09-18T12:34:45.946811+0000","last_undegraded":"2024-09-18T12:34:45.946811+0000","last_fullsized":"2024-09-18T12:34:45.946811+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:17:06.770342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144754+0000","last_change":"2024-09-18T12:32:22.608738+0000","last_active":"2024-09-18T12:34:43.144754+0000","last_peered":"2024-09-18T12:34:43.144754+0000","last_clean":"2024-09-18T12:34:43.144754+0000","last_became_active":"2024-09-18T12:32:22.608401+0000","last_became_peered":"2024-09-18T12:32:22.608401+0000","last_unstale":"2024-09-18T12:34:43.144754+0000","last_undegraded":"2024-09-18T12:34:43.144754+0000","last_fullsized":"2024-09-18T12:34:43.144754+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:59.037772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511904+0000","last_change":"2024-09-18T12:32:22.603874+0000","last_active":"2024-09-18T12:34:44.511904+0000","last_peered":"2024-09-18T12:34:44.511904+0000","last_clean":"2024-09-18T12:34:44.511904+0000","last_became_active":"2024-09-18T12:32:22.603533+0000","last_became_peered":"2024-09-18T12:32:22.603533+0000","last_unstale":"2024-09-18T12:34:44.511904+0000","last_undegraded":"2024-09-18T12:34:44.511904+0000","last_fullsized":"2024-09-18T12:34:44.511904+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:29.278327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947428+0000","last_change":"2024-09-18T12:32:22.607436+0000","last_active":"2024-09-18T12:34:42.947428+0000","last_peered":"2024-09-18T12:34:42.947428+0000","last_clean":"2024-09-18T12:34:42.947428+0000","last_became_active":"2024-09-18T12:32:22.607051+0000","last_became_peered":"2024-09-18T12:32:22.607051+0000","last_unstale":"2024-09-18T12:34:42.947428+0000","last_undegraded":"2024-09-18T12:34:42.947428+0000","last_fullsized":"2024-09-18T12:34:42.947428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:38.371976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144608+0000","last_change":"2024-09-18T12:32:22.604621+0000","last_active":"2024-09-18T12:34:43.144608+0000","last_peered":"2024-09-18T12:34:43.144608+0000","last_clean":"2024-09-18T12:34:43.144608+0000","last_became_active":"2024-09-18T12:32:22.604295+0000","last_became_peered":"2024-09-18T12:32:22.604295+0000","last_unstale":"2024-09-18T12:34:43.144608+0000","last_undegraded":"2024-09-18T12:34:43.144608+0000","last_fullsized":"2024-09-18T12:34:43.144608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:48:28.798147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":130,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144436+0000","last_change":"2024-09-18T12:32:22.606967+0000","last_active":"2024-09-18T12:34:43.144436+0000","last_peered":"2024-09-18T12:34:43.144436+0000","last_clean":"2024-09-18T12:34:43.144436+0000","last_became_active":"2024-09-18T12:32:22.606557+0000","last_became_peered":"2024-09-18T12:32:22.606557+0000","last_unstale":"2024-09-18T12:34:43.144436+0000","last_undegraded":"2024-09-18T12:34:43.144436+0000","last_fullsized":"2024-09-18T12:34:43.144436+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:13:49.746180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511862+0000","last_change":"2024-09-18T12:32:22.608034+0000","last_active":"2024-09-18T12:34:44.511862+0000","last_peered":"2024-09-18T12:34:44.511862+0000","last_clean":"2024-09-18T12:34:44.511862+0000","last_became_active":"2024-09-18T12:32:22.607698+0000","last_became_peered":"2024-09-18T12:32:22.607698+0000","last_unstale":"2024-09-18T12:34:44.511862+0000","last_undegraded":"2024-09-18T12:34:44.511862+0000","last_fullsized":"2024-09-18T12:34:44.511862+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:13.180633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":90,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511848+0000","last_change":"2024-09-18T12:32:22.606942+0000","last_active":"2024-09-18T12:34:44.511848+0000","last_peered":"2024-09-18T12:34:44.511848+0000","last_clean":"2024-09-18T12:34:44.511848+0000","last_became_active":"2024-09-18T12:32:22.606593+0000","last_became_peered":"2024-09-18T12:32:22.606593+0000","last_unstale":"2024-09-18T12:34:44.511848+0000","last_undegraded":"2024-09-18T12:34:44.511848+0000","last_fullsized":"2024-09-18T12:34:44.511848+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:53:56.502695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":182,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946736+0000","last_change":"2024-09-18T12:32:22.608147+0000","last_active":"2024-09-18T12:34:45.946736+0000","last_peered":"2024-09-18T12:34:45.946736+0000","last_clean":"2024-09-18T12:34:45.946736+0000","last_became_active":"2024-09-18T12:32:22.607839+0000","last_became_peered":"2024-09-18T12:32:22.607839+0000","last_unstale":"2024-09-18T12:34:45.946736+0000","last_undegraded":"2024-09-18T12:34:45.946736+0000","last_fullsized":"2024-09-18T12:34:45.946736+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:11:45.988633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510548+0000","last_change":"2024-09-18T12:32:23.599583+0000","last_active":"2024-09-18T12:34:44.510548+0000","last_peered":"2024-09-18T12:34:44.510548+0000","last_clean":"2024-09-18T12:34:44.510548+0000","last_became_active":"2024-09-18T12:32:23.599235+0000","last_became_peered":"2024-09-18T12:32:23.599235+0000","last_unstale":"2024-09-18T12:34:44.510548+0000","last_undegraded":"2024-09-18T12:34:44.510548+0000","last_fullsized":"2024-09-18T12:34:44.510548+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:33.154296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511941+0000","last_change":"2024-09-18T12:32:22.600178+0000","last_active":"2024-09-18T12:34:44.511941+0000","last_peered":"2024-09-18T12:34:44.511941+0000","last_clean":"2024-09-18T12:34:44.511941+0000","last_became_active":"2024-09-18T12:32:22.599859+0000","last_became_peered":"2024-09-18T12:32:22.599859+0000","last_unstale":"2024-09-18T12:34:44.511941+0000","last_undegraded":"2024-09-18T12:34:44.511941+0000","last_fullsized":"2024-09-18T12:34:44.511941+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:19:22.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":338,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144609+0000","last_change":"2024-09-18T12:32:22.605174+0000","last_active":"2024-09-18T12:34:43.144609+0000","last_peered":"2024-09-18T12:34:43.144609+0000","last_clean":"2024-09-18T12:34:43.144609+0000","last_became_active":"2024-09-18T12:32:22.604820+0000","last_became_peered":"2024-09-18T12:32:22.604820+0000","last_unstale":"2024-09-18T12:34:43.144609+0000","last_undegraded":"2024-09-18T12:34:43.144609+0000","last_fullsized":"2024-09-18T12:34:43.144609+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:41:49.881448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":266,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947516+0000","last_change":"2024-09-18T12:32:22.610862+0000","last_active":"2024-09-18T12:34:42.947516+0000","last_peered":"2024-09-18T12:34:42.947516+0000","last_clean":"2024-09-18T12:34:42.947516+0000","last_became_active":"2024-09-18T12:32:22.604471+0000","last_became_peered":"2024-09-18T12:32:22.604471+0000","last_unstale":"2024-09-18T12:34:42.947516+0000","last_undegraded":"2024-09-18T12:34:42.947516+0000","last_fullsized":"2024-09-18T12:34:42.947516+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:31.307834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144612+0000","last_change":"2024-09-18T12:32:22.608239+0000","last_active":"2024-09-18T12:34:43.144612+0000","last_peered":"2024-09-18T12:34:43.144612+0000","last_clean":"2024-09-18T12:34:43.144612+0000","last_became_active":"2024-09-18T12:32:22.607827+0000","last_became_peered":"2024-09-18T12:32:22.607827+0000","last_unstale":"2024-09-18T12:34:43.144612+0000","last_undegraded":"2024-09-18T12:34:43.144612+0000","last_fullsized":"2024-09-18T12:34:43.144612+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:40.035295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510540+0000","last_change":"2024-09-18T12:32:22.604079+0000","last_active":"2024-09-18T12:34:44.510540+0000","last_peered":"2024-09-18T12:34:44.510540+0000","last_clean":"2024-09-18T12:34:44.510540+0000","last_became_active":"2024-09-18T12:32:22.603727+0000","last_became_peered":"2024-09-18T12:32:22.603727+0000","last_unstale":"2024-09-18T12:34:44.510540+0000","last_undegraded":"2024-09-18T12:34:44.510540+0000","last_fullsized":"2024-09-18T12:34:44.510540+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:42:39.085495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":268,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947425+0000","last_change":"2024-09-18T12:32:22.615941+0000","last_active":"2024-09-18T12:34:42.947425+0000","last_peered":"2024-09-18T12:34:42.947425+0000","last_clean":"2024-09-18T12:34:42.947425+0000","last_became_active":"2024-09-18T12:32:22.608226+0000","last_became_peered":"2024-09-18T12:32:22.608226+0000","last_unstale":"2024-09-18T12:34:42.947425+0000","last_undegraded":"2024-09-18T12:34:42.947425+0000","last_fullsized":"2024-09-18T12:34:42.947425+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:13:36.171958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":264,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511965+0000","last_change":"2024-09-18T12:32:23.598518+0000","last_active":"2024-09-18T12:34:44.511965+0000","last_peered":"2024-09-18T12:34:44.511965+0000","last_clean":"2024-09-18T12:34:44.511965+0000","last_became_active":"2024-09-18T12:32:23.598179+0000","last_became_peered":"2024-09-18T12:32:23.598179+0000","last_unstale":"2024-09-18T12:34:44.511965+0000","last_undegraded":"2024-09-18T12:34:44.511965+0000","last_fullsized":"2024-09-18T12:34:44.511965+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:42.637189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144647+0000","last_change":"2024-09-18T12:32:22.605496+0000","last_active":"2024-09-18T12:34:43.144647+0000","last_peered":"2024-09-18T12:34:43.144647+0000","last_clean":"2024-09-18T12:34:43.144647+0000","last_became_active":"2024-09-18T12:32:22.605190+0000","last_became_peered":"2024-09-18T12:32:22.605190+0000","last_unstale":"2024-09-18T12:34:43.144647+0000","last_undegraded":"2024-09-18T12:34:43.144647+0000","last_fullsized":"2024-09-18T12:34:43.144647+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:50.861016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511899+0000","last_change":"2024-09-18T12:32:22.603956+0000","last_active":"2024-09-18T12:34:44.511899+0000","last_peered":"2024-09-18T12:34:44.511899+0000","last_clean":"2024-09-18T12:34:44.511899+0000","last_became_active":"2024-09-18T12:32:22.603556+0000","last_became_peered":"2024-09-18T12:32:22.603556+0000","last_unstale":"2024-09-18T12:34:44.511899+0000","last_undegraded":"2024-09-18T12:34:44.511899+0000","last_fullsized":"2024-09-18T12:34:44.511899+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:40:30.975910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144746+0000","last_change":"2024-09-18T12:32:22.601754+0000","last_active":"2024-09-18T12:34:43.144746+0000","last_peered":"2024-09-18T12:34:43.144746+0000","last_clean":"2024-09-18T12:34:43.144746+0000","last_became_active":"2024-09-18T12:32:22.601418+0000","last_became_peered":"2024-09-18T12:32:22.601418+0000","last_unstale":"2024-09-18T12:34:43.144746+0000","last_undegraded":"2024-09-18T12:34:43.144746+0000","last_fullsized":"2024-09-18T12:34:43.144746+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:18:06.635553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640501,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.45900000000000002},"min":{"1min":0.40000000000000002,"5min":0.32400000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.69099999999999995},"last":0.495},{"interface":"front","average":{"1min":0.499,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.20999999999999999},"max":{"1min":0.57999999999999996,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53900000000000003,"15min":0.51700000000000002},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.23999999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.48999999999999999,"5min":0.39900000000000002,"15min":0.27200000000000002},"max":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.80200000000000005},"last":0.504}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":1.304,"15min":1.304},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.25700000000000001},"max":{"1min":0.63300000000000001,"5min":0.72799999999999998,"15min":0.753},"last":0.46999999999999997}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.371,"15min":0.36199999999999999},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":0.51600000000000001,"5min":0.59599999999999997,"15min":0.60299999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.20100000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.64200000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":0.58899999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.223},"max":{"1min":0.61199999999999999,"5min":0.68200000000000005,"15min":0.73799999999999999},"last":0.371}]}]},{"osd":4,"up_from":11,"seq":47244640501,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298924,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072916,"statfs":{"total":96636764160,"available":96330665984,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.47699999999999998},"min":{"1min":0.37,"5min":0.27400000000000002,"15min":0.223},"max":{"1min":0.60199999999999998,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.378},{"interface":"front","average":{"1min":0.504,"5min":0.49199999999999999,"15min":0.47799999999999998},"min":{"1min":0.379,"5min":0.33100000000000002,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45400000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.25900000000000001},"max":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.753},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.71999999999999997,"15min":0.872},"last":0.46500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71499999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45100000000000001,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.443}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41499999999999998,"15min":0.38400000000000001},"min":{"1min":0.33600000000000002,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.59399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.307,"5min":0.255,"15min":0.249},"max":{"1min":0.53700000000000003,"5min":0.68899999999999995,"15min":0.70299999999999996},"last":0.28499999999999998}]},{"osd":5,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59499999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26100000000000001,"15min":0.223},"max":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.60999999999999999},"last":0.34399999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640504,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304012,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067828,"statfs":{"total":96636764160,"available":96325455872,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65300000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.34599999999999997,"15min":0.33100000000000002},"max":{"1min":0.68600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.5}]},{"osd":1,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51000000000000001,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.35599999999999998,"15min":0.26600000000000001},"max":{"1min":0.64500000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.68600000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.70099999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.503,"15min":0.5},"min":{"1min":0.37,"5min":0.33400000000000002,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.873,"15min":0.873},"last":0.48499999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36099999999999999,"15min":0.36499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.214},"max":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.624},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.36199999999999999,"15min":0.372},"min":{"1min":0.254,"5min":0.254,"15min":0.221},"max":{"1min":0.48499999999999999,"5min":0.58199999999999996,"15min":0.67200000000000004},"last":0.39100000000000001}]},{"osd":5,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36099999999999999,"15min":0.36799999999999999},"min":{"1min":0.26500000000000001,"5min":0.26100000000000001,"15min":0.20300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.64600000000000002,"15min":0.70599999999999996},"last":0.443}]}]},{"osd":2,"up_from":11,"seq":47244640503,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.315,"15min":0.32200000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.156},"max":{"1min":0.46800000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34399999999999997,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.628},"last":0.443}]},{"osd":1,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35299999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51700000000000002,"5min":0.60599999999999998,"15min":0.63900000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.33600000000000002,"15min":0.34200000000000003},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.17699999999999999},"max":{"1min":0.54200000000000004,"5min":0.625,"15min":0.625},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":0.77000000000000002,"15min":0.77300000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.308},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.72999999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75700000000000001,"5min":1.034,"15min":1.034},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.83099999999999996},"last":0.80000000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.40300000000000002,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":0.72999999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.746},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55100000000000005,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.78900000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640503,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":0.41999999999999998,"5min":0.48499999999999999,"15min":0.70699999999999996},"last":0.316},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33700000000000002,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.214},"max":{"1min":0.44700000000000001,"5min":0.629,"15min":0.80000000000000004},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34499999999999997,"15min":0.35899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":0.46800000000000003,"5min":0.52000000000000002,"15min":0.86599999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.26900000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48599999999999999,"5min":0.53600000000000003,"15min":0.82599999999999996},"last":0.375}]},{"osd":3,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.41699999999999998,"5min":0.377,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":0.84199999999999997,"15min":0.95699999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.54200000000000004},"min":{"1min":0.40699999999999997,"5min":0.375,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.83799999999999997,"15min":0.97899999999999998},"last":0.59499999999999997}]},{"osd":4,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.317},"max":{"1min":0.76400000000000001,"5min":0.82099999999999995,"15min":0.997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.34999999999999998},"max":{"1min":0.71399999999999997,"5min":0.85199999999999998,"15min":1.034},"last":0.53100000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":0.68200000000000005,"5min":0.82999999999999996,"15min":0.88300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.42199999999999999,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":0.625,"5min":0.81299999999999994,"15min":1.0960000000000001},"last":0.61399999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640502,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303608,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068232,"statfs":{"total":96636764160,"available":96325869568,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.30599999999999999,"15min":0.317},"min":{"1min":0.223,"5min":0.14799999999999999,"15min":0.13400000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.55500000000000005},"last":0.313},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.16400000000000001},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.64500000000000002},"last":0.375}]},{"osd":2,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.35499999999999998},"min":{"1min":0.21299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.48599999999999999,"5min":0.502,"15min":0.76800000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.36799999999999999},"min":{"1min":0.28499999999999998,"5min":0.221,"15min":0.16},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.73499999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.69499999999999995,"5min":1.177,"15min":1.177},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.53700000000000003},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.54100000000000004}]},{"osd":4,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":1.127,"15min":1.127},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.69999999999999996,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.54000000000000004},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":1.141,"15min":1.141},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34100000000000003},"max":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:45:44.095 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T12:45:44.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:45:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.185+0000 7f1f88b9d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.185+0000 7f1f88b9d640 10 monclient: build_initial_monmap 2024-09-18T12:45:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.185+0000 7f1f88b9d640 10 monclient: monmap: 2024-09-18T12:45:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:44.186315+0000 2024-09-18T12:45:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:44.186315+0000 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:44.187 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 1 Processor -- start 2024-09-18T12:45:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 1 -- start start 2024-09-18T12:45:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient: init 2024-09-18T12:45:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T12:45:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f1f84096e10 0x7f1f8409a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f1f84096e10 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f8409aaa0 0x7f1f840971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8409aaa0 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f84097720 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f84099620 con 0x7f1f84097720 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f84099340 con 0x7f1f8409aaa0 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f84099900 con 0x7f1f84096e10 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f88b9d640 10 monclient(hunting): authenticate will time out at 1971.507886s 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.186+0000 7f1f7bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f8409aaa0 0x7f1f840971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f8409aaa0 0x7f1f840971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41844/0 (socket says 172.21.15.116:41844) 2024-09-18T12:45:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2059528374 learned_addr learned my addr 172.21.15.116:0/2059528374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): get_auth_request con 0x7f1f8409aaa0 auth_method 0 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_done global_id 194700 payload 274 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f8259c640 1 --2- 172.21.15.116:0/2059528374 >> v2:172.21.15.144:3300/0 conn(0x7f1f84096e10 0x7f1f8409a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2059528374 >> v2:172.21.15.144:3300/0 conn(0x7f1f84096e10 msgr2=0x7f1f8409a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 1 --2- 172.21.15.116:0/2059528374 >> v2:172.21.15.144:3300/0 conn(0x7f1f84096e10 0x7f1f8409a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2059528374 >> v1:172.21.15.144:6789/0 conn(0x7f1f84097720 legacy=0x7f1f8412e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 1 monclient: found mon.noname-a 2024-09-18T12:45:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.187+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2059528374 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84097b20 con 0x7f1f8409aaa0 2024-09-18T12:45:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2059528374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409aaa0 msgr2=0x7f1f840971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f7bfff640 1 --2- 172.21.15.116:0/2059528374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409aaa0 0x7f1f840971e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1f84076be0 tx=0x7f1f6c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e3ca51c262da13 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 1 -- 172.21.15.116:0/2059528374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c032440 con 0x7f1f8409aaa0 2024-09-18T12:45:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: dump: 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: _finish_auth 0 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 1 -- 172.21.15.116:0/2059528374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c0325a0 con 0x7f1f8409aaa0 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 5 monclient: authenticate success, global_id 194700 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 1 -- 172.21.15.116:0/2059528374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c032940 con 0x7f1f8409aaa0 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 4 set_mon_vals no callback set 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f81d9b640 10 monclient: dump: 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.199 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 10 monclient: shutdown 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2059528374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409aaa0 msgr2=0x7f1f840971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2059528374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409aaa0 0x7f1f840971e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1f84076be0 tx=0x7f1f6c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2059528374 shutdown_connections 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2059528374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f8409aaa0 0x7f1f840971e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2059528374 >> v2:172.21.15.144:3300/0 conn(0x7f1f84096e10 0x7f1f8409a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2059528374 >> 172.21.15.116:0/2059528374 conn(0x7f1f840672b0 msgr2=0x7f1f840676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2059528374 shutdown_connections 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.190+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2059528374 wait complete. 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.191+0000 7f1f88b9d640 10 monclient: build_initial_monmap 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.191+0000 7f1f88b9d640 10 monclient: monmap: 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.201 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.191+0000 7f1f88b9d640 1 Processor -- start 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.191+0000 7f1f88b9d640 1 -- start start 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.191+0000 7f1f88b9d640 10 monclient: init 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.191+0000 7f1f88b9d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f84096e10 0x7f1f8412cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f1f84096e10 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 0x7f1f84118920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f84097720 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409aaa0 0x7f1f84118e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8409aaa0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8411c000 con 0x7f1f84097720 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8409b050 con 0x7f1f8409aaa0 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8412e740 con 0x7f1f84096e10 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f88b9d640 10 monclient(hunting): authenticate will time out at 1971.513078s 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 0x7f1f84118920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 0x7f1f84118920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41850/0 (socket says 172.21.15.116:41850) 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f84096e10 0x7f1f8412cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2206772207 learned_addr learned my addr 172.21.15.116:0/2206772207 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): get_auth_request con 0x7f1f84097720 auth_method 0 2024-09-18T12:45:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 10 monclient(hunting): get_auth_request con 0x7f1f84096e10 auth_method 0 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f82d9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409aaa0 0x7f1f84118e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f82d9d640 10 monclient(hunting): get_auth_request con 0x7f1f8409aaa0 auth_method 0 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f82d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient(hunting): handle_auth_done global_id 194703 payload 824 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f84096e10 msgr2=0x7f1f8412cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f84096e10 0x7f1f8412cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409aaa0 msgr2=0x7f1f84118e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409aaa0 0x7f1f84118e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 monclient: found mon.noname-a 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2206772207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c009450 con 0x7f1f84097720 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.192+0000 7f1f82d9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409aaa0 0x7f1f84118e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7bfff640 1 -- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 msgr2=0x7f1f84118920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7bfff640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 0x7f1f84118920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009210 tx=0x7f1f6c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 1 -- 172.21.15.116:0/2206772207 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c031b10 con 0x7f1f84097720 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: dump: 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 1 -- 172.21.15.116:0/2206772207 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c031c90 con 0x7f1f84097720 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f88b9d640 5 monclient: authenticate success, global_id 194703 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 1 -- 172.21.15.116:0/2206772207 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c03b650 con 0x7f1f84097720 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f88b9d640 10 monclient: _renew_subs 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f88b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8411cbc0 con 0x7f1f84097720 2024-09-18T12:45:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f7a7fc640 10 monclient: dump: 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f8159a640 4 set_mon_vals no callback set 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f88b9d640 10 monclient: _renew_subs 2024-09-18T12:45:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f88b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.194+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8411aad0 con 0x7f1f84097720 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.195+0000 7f1f88b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.195+0000 7f1f7a7fc640 1 -- 172.21.15.116:0/2206772207 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f1f6c039040 con 0x7f1f84097720 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.195+0000 7f1f88b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.195+0000 7f1f88b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.195+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8411c640 con 0x7f1f84097720 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.196+0000 7f1f7a7fc640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f64038490 0x7f1f6403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.196+0000 7f1f7a7fc640 1 -- 172.21.15.116:0/2206772207 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f1f6c06c100 con 0x7f1f84097720 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.196+0000 7f1f8259c640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f64038490 0x7f1f6403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.196+0000 7f1f8259c640 10 monclient: get_auth_request con 0x7f1f64038490 auth_method 0 2024-09-18T12:45:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.198+0000 7f1f8259c640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f64038490 0x7f1f6403a940 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1f74006fd0 tx=0x7f1f74008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.200+0000 7f1f7a7fc640 1 -- 172.21.15.116:0/2206772207 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f1f6c03ba00 con 0x7f1f84097720 2024-09-18T12:45:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.200+0000 7f1f7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.200+0000 7f1f7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.352+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f8404bf60 con 0x7f1f64038490 2024-09-18T12:45:44.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.356+0000 7f1f7a7fc640 1 -- 172.21.15.116:0/2206772207 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97874 (secure 0 0 0) 0x7f1f8404bf60 con 0x7f1f64038490 2024-09-18T12:45:44.357 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:45:44.357 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:45:44.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f64038490 msgr2=0x7f1f6403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f64038490 0x7f1f6403a940 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1f74006fd0 tx=0x7f1f74008040 comp rx=0 tx=0).stop 2024-09-18T12:45:44.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 10 monclient: shutdown 2024-09-18T12:45:44.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 msgr2=0x7f1f84118920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 0x7f1f84118920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009210 tx=0x7f1f6c032000 comp rx=0 tx=0).stop 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 shutdown_connections 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f1f64038490 0x7f1f6403a940 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f8409aaa0 0x7f1f84118e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f84097720 0x7f1f84118920 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 --2- 172.21.15.116:0/2206772207 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f1f84096e10 0x7f1f8412cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 >> 172.21.15.116:0/2206772207 conn(0x7f1f840672b0 msgr2=0x7f1f8412dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 shutdown_connections 2024-09-18T12:45:44.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.358+0000 7f1f88b9d640 1 -- 172.21.15.116:0/2206772207 wait complete. 2024-09-18T12:45:44.369 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":872,"stamp":"2024-09-18T12:45:42.816617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1815804,"kb_used_data":10456,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564415236,"statfs":{"total":579820584960,"available":577961201664,"internally_reserved":0,"allocated":10706944,"data_stored":8512606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":175,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946900+0000","last_change":"2024-09-18T12:32:22.608257+0000","last_active":"2024-09-18T12:34:45.946900+0000","last_peered":"2024-09-18T12:34:45.946900+0000","last_clean":"2024-09-18T12:34:45.946900+0000","last_became_active":"2024-09-18T12:32:22.607844+0000","last_became_peered":"2024-09-18T12:32:22.607844+0000","last_unstale":"2024-09-18T12:34:45.946900+0000","last_undegraded":"2024-09-18T12:34:45.946900+0000","last_fullsized":"2024-09-18T12:34:45.946900+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:52.306141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947577+0000","last_change":"2024-09-18T12:32:22.602302+0000","last_active":"2024-09-18T12:34:42.947577+0000","last_peered":"2024-09-18T12:34:42.947577+0000","last_clean":"2024-09-18T12:34:42.947577+0000","last_became_active":"2024-09-18T12:32:22.601974+0000","last_became_peered":"2024-09-18T12:32:22.601974+0000","last_unstale":"2024-09-18T12:34:42.947577+0000","last_undegraded":"2024-09-18T12:34:42.947577+0000","last_fullsized":"2024-09-18T12:34:42.947577+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:05.401601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510608+0000","last_change":"2024-09-18T12:32:22.605364+0000","last_active":"2024-09-18T12:34:44.510608+0000","last_peered":"2024-09-18T12:34:44.510608+0000","last_clean":"2024-09-18T12:34:44.510608+0000","last_became_active":"2024-09-18T12:32:22.604956+0000","last_became_peered":"2024-09-18T12:32:22.604956+0000","last_unstale":"2024-09-18T12:34:44.510608+0000","last_undegraded":"2024-09-18T12:34:44.510608+0000","last_fullsized":"2024-09-18T12:34:44.510608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:21:23.773463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":83,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947446+0000","last_change":"2024-09-18T12:32:22.604750+0000","last_active":"2024-09-18T12:34:42.947446+0000","last_peered":"2024-09-18T12:34:42.947446+0000","last_clean":"2024-09-18T12:34:42.947446+0000","last_became_active":"2024-09-18T12:32:22.604332+0000","last_became_peered":"2024-09-18T12:32:22.604332+0000","last_unstale":"2024-09-18T12:34:42.947446+0000","last_undegraded":"2024-09-18T12:34:42.947446+0000","last_fullsized":"2024-09-18T12:34:42.947446+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:50:27.338272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":127,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947518+0000","last_change":"2024-09-18T12:32:23.804087+0000","last_active":"2024-09-18T12:34:42.947518+0000","last_peered":"2024-09-18T12:34:42.947518+0000","last_clean":"2024-09-18T12:34:42.947518+0000","last_became_active":"2024-09-18T12:32:23.803732+0000","last_became_peered":"2024-09-18T12:32:23.803732+0000","last_unstale":"2024-09-18T12:34:42.947518+0000","last_undegraded":"2024-09-18T12:34:42.947518+0000","last_fullsized":"2024-09-18T12:34:42.947518+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:19:27.867616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":151,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946811+0000","last_change":"2024-09-18T12:32:22.604980+0000","last_active":"2024-09-18T12:34:45.946811+0000","last_peered":"2024-09-18T12:34:45.946811+0000","last_clean":"2024-09-18T12:34:45.946811+0000","last_became_active":"2024-09-18T12:32:22.604577+0000","last_became_peered":"2024-09-18T12:32:22.604577+0000","last_unstale":"2024-09-18T12:34:45.946811+0000","last_undegraded":"2024-09-18T12:34:45.946811+0000","last_fullsized":"2024-09-18T12:34:45.946811+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:17:06.770342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144754+0000","last_change":"2024-09-18T12:32:22.608738+0000","last_active":"2024-09-18T12:34:43.144754+0000","last_peered":"2024-09-18T12:34:43.144754+0000","last_clean":"2024-09-18T12:34:43.144754+0000","last_became_active":"2024-09-18T12:32:22.608401+0000","last_became_peered":"2024-09-18T12:32:22.608401+0000","last_unstale":"2024-09-18T12:34:43.144754+0000","last_undegraded":"2024-09-18T12:34:43.144754+0000","last_fullsized":"2024-09-18T12:34:43.144754+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:59.037772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511904+0000","last_change":"2024-09-18T12:32:22.603874+0000","last_active":"2024-09-18T12:34:44.511904+0000","last_peered":"2024-09-18T12:34:44.511904+0000","last_clean":"2024-09-18T12:34:44.511904+0000","last_became_active":"2024-09-18T12:32:22.603533+0000","last_became_peered":"2024-09-18T12:32:22.603533+0000","last_unstale":"2024-09-18T12:34:44.511904+0000","last_undegraded":"2024-09-18T12:34:44.511904+0000","last_fullsized":"2024-09-18T12:34:44.511904+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:29.278327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947428+0000","last_change":"2024-09-18T12:32:22.607436+0000","last_active":"2024-09-18T12:34:42.947428+0000","last_peered":"2024-09-18T12:34:42.947428+0000","last_clean":"2024-09-18T12:34:42.947428+0000","last_became_active":"2024-09-18T12:32:22.607051+0000","last_became_peered":"2024-09-18T12:32:22.607051+0000","last_unstale":"2024-09-18T12:34:42.947428+0000","last_undegraded":"2024-09-18T12:34:42.947428+0000","last_fullsized":"2024-09-18T12:34:42.947428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:38.371976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144608+0000","last_change":"2024-09-18T12:32:22.604621+0000","last_active":"2024-09-18T12:34:43.144608+0000","last_peered":"2024-09-18T12:34:43.144608+0000","last_clean":"2024-09-18T12:34:43.144608+0000","last_became_active":"2024-09-18T12:32:22.604295+0000","last_became_peered":"2024-09-18T12:32:22.604295+0000","last_unstale":"2024-09-18T12:34:43.144608+0000","last_undegraded":"2024-09-18T12:34:43.144608+0000","last_fullsized":"2024-09-18T12:34:43.144608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:48:28.798147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":130,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144436+0000","last_change":"2024-09-18T12:32:22.606967+0000","last_active":"2024-09-18T12:34:43.144436+0000","last_peered":"2024-09-18T12:34:43.144436+0000","last_clean":"2024-09-18T12:34:43.144436+0000","last_became_active":"2024-09-18T12:32:22.606557+0000","last_became_peered":"2024-09-18T12:32:22.606557+0000","last_unstale":"2024-09-18T12:34:43.144436+0000","last_undegraded":"2024-09-18T12:34:43.144436+0000","last_fullsized":"2024-09-18T12:34:43.144436+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:13:49.746180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511862+0000","last_change":"2024-09-18T12:32:22.608034+0000","last_active":"2024-09-18T12:34:44.511862+0000","last_peered":"2024-09-18T12:34:44.511862+0000","last_clean":"2024-09-18T12:34:44.511862+0000","last_became_active":"2024-09-18T12:32:22.607698+0000","last_became_peered":"2024-09-18T12:32:22.607698+0000","last_unstale":"2024-09-18T12:34:44.511862+0000","last_undegraded":"2024-09-18T12:34:44.511862+0000","last_fullsized":"2024-09-18T12:34:44.511862+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:13.180633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":90,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511848+0000","last_change":"2024-09-18T12:32:22.606942+0000","last_active":"2024-09-18T12:34:44.511848+0000","last_peered":"2024-09-18T12:34:44.511848+0000","last_clean":"2024-09-18T12:34:44.511848+0000","last_became_active":"2024-09-18T12:32:22.606593+0000","last_became_peered":"2024-09-18T12:32:22.606593+0000","last_unstale":"2024-09-18T12:34:44.511848+0000","last_undegraded":"2024-09-18T12:34:44.511848+0000","last_fullsized":"2024-09-18T12:34:44.511848+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:53:56.502695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":182,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946736+0000","last_change":"2024-09-18T12:32:22.608147+0000","last_active":"2024-09-18T12:34:45.946736+0000","last_peered":"2024-09-18T12:34:45.946736+0000","last_clean":"2024-09-18T12:34:45.946736+0000","last_became_active":"2024-09-18T12:32:22.607839+0000","last_became_peered":"2024-09-18T12:32:22.607839+0000","last_unstale":"2024-09-18T12:34:45.946736+0000","last_undegraded":"2024-09-18T12:34:45.946736+0000","last_fullsized":"2024-09-18T12:34:45.946736+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:11:45.988633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510548+0000","last_change":"2024-09-18T12:32:23.599583+0000","last_active":"2024-09-18T12:34:44.510548+0000","last_peered":"2024-09-18T12:34:44.510548+0000","last_clean":"2024-09-18T12:34:44.510548+0000","last_became_active":"2024-09-18T12:32:23.599235+0000","last_became_peered":"2024-09-18T12:32:23.599235+0000","last_unstale":"2024-09-18T12:34:44.510548+0000","last_undegraded":"2024-09-18T12:34:44.510548+0000","last_fullsized":"2024-09-18T12:34:44.510548+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:33.154296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511941+0000","last_change":"2024-09-18T12:32:22.600178+0000","last_active":"2024-09-18T12:34:44.511941+0000","last_peered":"2024-09-18T12:34:44.511941+0000","last_clean":"2024-09-18T12:34:44.511941+0000","last_became_active":"2024-09-18T12:32:22.599859+0000","last_became_peered":"2024-09-18T12:32:22.599859+0000","last_unstale":"2024-09-18T12:34:44.511941+0000","last_undegraded":"2024-09-18T12:34:44.511941+0000","last_fullsized":"2024-09-18T12:34:44.511941+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:19:22.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":338,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144609+0000","last_change":"2024-09-18T12:32:22.605174+0000","last_active":"2024-09-18T12:34:43.144609+0000","last_peered":"2024-09-18T12:34:43.144609+0000","last_clean":"2024-09-18T12:34:43.144609+0000","last_became_active":"2024-09-18T12:32:22.604820+0000","last_became_peered":"2024-09-18T12:32:22.604820+0000","last_unstale":"2024-09-18T12:34:43.144609+0000","last_undegraded":"2024-09-18T12:34:43.144609+0000","last_fullsized":"2024-09-18T12:34:43.144609+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:41:49.881448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":266,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947516+0000","last_change":"2024-09-18T12:32:22.610862+0000","last_active":"2024-09-18T12:34:42.947516+0000","last_peered":"2024-09-18T12:34:42.947516+0000","last_clean":"2024-09-18T12:34:42.947516+0000","last_became_active":"2024-09-18T12:32:22.604471+0000","last_became_peered":"2024-09-18T12:32:22.604471+0000","last_unstale":"2024-09-18T12:34:42.947516+0000","last_undegraded":"2024-09-18T12:34:42.947516+0000","last_fullsized":"2024-09-18T12:34:42.947516+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:31.307834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144612+0000","last_change":"2024-09-18T12:32:22.608239+0000","last_active":"2024-09-18T12:34:43.144612+0000","last_peered":"2024-09-18T12:34:43.144612+0000","last_clean":"2024-09-18T12:34:43.144612+0000","last_became_active":"2024-09-18T12:32:22.607827+0000","last_became_peered":"2024-09-18T12:32:22.607827+0000","last_unstale":"2024-09-18T12:34:43.144612+0000","last_undegraded":"2024-09-18T12:34:43.144612+0000","last_fullsized":"2024-09-18T12:34:43.144612+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:40.035295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510540+0000","last_change":"2024-09-18T12:32:22.604079+0000","last_active":"2024-09-18T12:34:44.510540+0000","last_peered":"2024-09-18T12:34:44.510540+0000","last_clean":"2024-09-18T12:34:44.510540+0000","last_became_active":"2024-09-18T12:32:22.603727+0000","last_became_peered":"2024-09-18T12:32:22.603727+0000","last_unstale":"2024-09-18T12:34:44.510540+0000","last_undegraded":"2024-09-18T12:34:44.510540+0000","last_fullsized":"2024-09-18T12:34:44.510540+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:42:39.085495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":268,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947425+0000","last_change":"2024-09-18T12:32:22.615941+0000","last_active":"2024-09-18T12:34:42.947425+0000","last_peered":"2024-09-18T12:34:42.947425+0000","last_clean":"2024-09-18T12:34:42.947425+0000","last_became_active":"2024-09-18T12:32:22.608226+0000","last_became_peered":"2024-09-18T12:32:22.608226+0000","last_unstale":"2024-09-18T12:34:42.947425+0000","last_undegraded":"2024-09-18T12:34:42.947425+0000","last_fullsized":"2024-09-18T12:34:42.947425+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:13:36.171958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":264,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511965+0000","last_change":"2024-09-18T12:32:23.598518+0000","last_active":"2024-09-18T12:34:44.511965+0000","last_peered":"2024-09-18T12:34:44.511965+0000","last_clean":"2024-09-18T12:34:44.511965+0000","last_became_active":"2024-09-18T12:32:23.598179+0000","last_became_peered":"2024-09-18T12:32:23.598179+0000","last_unstale":"2024-09-18T12:34:44.511965+0000","last_undegraded":"2024-09-18T12:34:44.511965+0000","last_fullsized":"2024-09-18T12:34:44.511965+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:42.637189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144647+0000","last_change":"2024-09-18T12:32:22.605496+0000","last_active":"2024-09-18T12:34:43.144647+0000","last_peered":"2024-09-18T12:34:43.144647+0000","last_clean":"2024-09-18T12:34:43.144647+0000","last_became_active":"2024-09-18T12:32:22.605190+0000","last_became_peered":"2024-09-18T12:32:22.605190+0000","last_unstale":"2024-09-18T12:34:43.144647+0000","last_undegraded":"2024-09-18T12:34:43.144647+0000","last_fullsized":"2024-09-18T12:34:43.144647+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:50.861016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511899+0000","last_change":"2024-09-18T12:32:22.603956+0000","last_active":"2024-09-18T12:34:44.511899+0000","last_peered":"2024-09-18T12:34:44.511899+0000","last_clean":"2024-09-18T12:34:44.511899+0000","last_became_active":"2024-09-18T12:32:22.603556+0000","last_became_peered":"2024-09-18T12:32:22.603556+0000","last_unstale":"2024-09-18T12:34:44.511899+0000","last_undegraded":"2024-09-18T12:34:44.511899+0000","last_fullsized":"2024-09-18T12:34:44.511899+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:40:30.975910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144746+0000","last_change":"2024-09-18T12:32:22.601754+0000","last_active":"2024-09-18T12:34:43.144746+0000","last_peered":"2024-09-18T12:34:43.144746+0000","last_clean":"2024-09-18T12:34:43.144746+0000","last_became_active":"2024-09-18T12:32:22.601418+0000","last_became_peered":"2024-09-18T12:32:22.601418+0000","last_unstale":"2024-09-18T12:34:43.144746+0000","last_undegraded":"2024-09-18T12:34:43.144746+0000","last_fullsized":"2024-09-18T12:34:43.144746+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:18:06.635553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640501,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.45900000000000002},"min":{"1min":0.40000000000000002,"5min":0.32400000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.69099999999999995},"last":0.495},{"interface":"front","average":{"1min":0.499,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.20999999999999999},"max":{"1min":0.57999999999999996,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53900000000000003,"15min":0.51700000000000002},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.23999999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.48999999999999999,"5min":0.39900000000000002,"15min":0.27200000000000002},"max":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.80200000000000005},"last":0.504}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":1.304,"15min":1.304},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.25700000000000001},"max":{"1min":0.63300000000000001,"5min":0.72799999999999998,"15min":0.753},"last":0.46999999999999997}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.371,"15min":0.36199999999999999},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":0.51600000000000001,"5min":0.59599999999999997,"15min":0.60299999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.20100000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.64200000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":0.58899999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.223},"max":{"1min":0.61199999999999999,"5min":0.68200000000000005,"15min":0.73799999999999999},"last":0.371}]}]},{"osd":4,"up_from":11,"seq":47244640501,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298924,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072916,"statfs":{"total":96636764160,"available":96330665984,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.47699999999999998},"min":{"1min":0.37,"5min":0.27400000000000002,"15min":0.223},"max":{"1min":0.60199999999999998,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.378},{"interface":"front","average":{"1min":0.504,"5min":0.49199999999999999,"15min":0.47799999999999998},"min":{"1min":0.379,"5min":0.33100000000000002,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45400000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.25900000000000001},"max":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.753},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.71999999999999997,"15min":0.872},"last":0.46500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71499999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45100000000000001,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.443}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41499999999999998,"15min":0.38400000000000001},"min":{"1min":0.33600000000000002,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.59399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.307,"5min":0.255,"15min":0.249},"max":{"1min":0.53700000000000003,"5min":0.68899999999999995,"15min":0.70299999999999996},"last":0.28499999999999998}]},{"osd":5,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59499999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26100000000000001,"15min":0.223},"max":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.60999999999999999},"last":0.34399999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640504,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304012,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067828,"statfs":{"total":96636764160,"available":96325455872,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65300000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.34599999999999997,"15min":0.33100000000000002},"max":{"1min":0.68600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.5}]},{"osd":1,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51000000000000001,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.35599999999999998,"15min":0.26600000000000001},"max":{"1min":0.64500000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.68600000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.70099999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.503,"15min":0.5},"min":{"1min":0.37,"5min":0.33400000000000002,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.873,"15min":0.873},"last":0.48499999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36099999999999999,"15min":0.36499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.214},"max":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.624},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.36199999999999999,"15min":0.372},"min":{"1min":0.254,"5min":0.254,"15min":0.221},"max":{"1min":0.48499999999999999,"5min":0.58199999999999996,"15min":0.67200000000000004},"last":0.39100000000000001}]},{"osd":5,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36099999999999999,"15min":0.36799999999999999},"min":{"1min":0.26500000000000001,"5min":0.26100000000000001,"15min":0.20300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.64600000000000002,"15min":0.70599999999999996},"last":0.443}]}]},{"osd":2,"up_from":11,"seq":47244640503,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.315,"15min":0.32200000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.156},"max":{"1min":0.46800000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34399999999999997,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.628},"last":0.443}]},{"osd":1,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35299999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51700000000000002,"5min":0.60599999999999998,"15min":0.63900000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.33600000000000002,"15min":0.34200000000000003},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.17699999999999999},"max":{"1min":0.54200000000000004,"5min":0.625,"15min":0.625},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":0.77000000000000002,"15min":0.77300000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.308},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.72999999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75700000000000001,"5min":1.034,"15min":1.034},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.83099999999999996},"last":0.80000000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.40300000000000002,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":0.72999999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.746},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55100000000000005,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.78900000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640503,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":0.41999999999999998,"5min":0.48499999999999999,"15min":0.70699999999999996},"last":0.316},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33700000000000002,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.214},"max":{"1min":0.44700000000000001,"5min":0.629,"15min":0.80000000000000004},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34499999999999997,"15min":0.35899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":0.46800000000000003,"5min":0.52000000000000002,"15min":0.86599999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.26900000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48599999999999999,"5min":0.53600000000000003,"15min":0.82599999999999996},"last":0.375}]},{"osd":3,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.41699999999999998,"5min":0.377,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":0.84199999999999997,"15min":0.95699999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.54200000000000004},"min":{"1min":0.40699999999999997,"5min":0.375,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.83799999999999997,"15min":0.97899999999999998},"last":0.59499999999999997}]},{"osd":4,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.317},"max":{"1min":0.76400000000000001,"5min":0.82099999999999995,"15min":0.997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.34999999999999998},"max":{"1min":0.71399999999999997,"5min":0.85199999999999998,"15min":1.034},"last":0.53100000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":0.68200000000000005,"5min":0.82999999999999996,"15min":0.88300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.42199999999999999,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":0.625,"5min":0.81299999999999994,"15min":1.0960000000000001},"last":0.61399999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640502,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303608,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068232,"statfs":{"total":96636764160,"available":96325869568,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.30599999999999999,"15min":0.317},"min":{"1min":0.223,"5min":0.14799999999999999,"15min":0.13400000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.55500000000000005},"last":0.313},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.16400000000000001},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.64500000000000002},"last":0.375}]},{"osd":2,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.35499999999999998},"min":{"1min":0.21299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.48599999999999999,"5min":0.502,"15min":0.76800000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.36799999999999999},"min":{"1min":0.28499999999999998,"5min":0.221,"15min":0.16},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.73499999999999999},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.69499999999999995,"5min":1.177,"15min":1.177},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.53700000000000003},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.54100000000000004}]},{"osd":4,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":1.127,"15min":1.127},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.69999999999999996,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.54000000000000004},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":1.141,"15min":1.141},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34100000000000003},"max":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:45:44.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:45:44.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.462+0000 7efcd362b640 10 monclient: get_monmap_and_config 2024-09-18T12:45:44.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.462+0000 7efcd362b640 10 monclient: build_initial_monmap 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.462+0000 7efcd362b640 10 monclient: monmap: 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:44.463369+0000 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:44.463369+0000 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:44.464 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 Processor -- start 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 -- start start 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient: init 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 0x7efccc131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): picked mon.noname-b con 0x7efccc131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efccc09e910 0x7efccc09ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): picked mon.noname-c con 0x7efccc09e910 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:44.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efccc09f220 0x7efccc068ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): picked mon.noname-a con 0x7efccc09f220 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efccc09faf0 con 0x7efccc09f220 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efccc09fd30 con 0x7efccc09e910 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efccc0a1110 con 0x7efccc131ab0 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.463+0000 7efcd362b640 10 monclient(hunting): authenticate will time out at 1971.784921s 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 0x7efccc131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efccc09f220 0x7efccc068ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 0x7efccc131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:47904/0 (socket says 172.21.15.116:47904) 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 -- 172.21.15.116:0/1756944888 learned_addr learned my addr 172.21.15.116:0/1756944888 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): get_auth_request con 0x7efccc131ab0 auth_method 0 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 10 monclient(hunting): get_auth_request con 0x7efccc09f220 auth_method 0 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient(hunting): handle_auth_done global_id 244157 payload 274 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd0bc6640 1 --2- 172.21.15.116:0/1756944888 >> v2:172.21.15.144:3300/0 conn(0x7efccc09e910 0x7efccc09ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 -- 172.21.15.116:0/1756944888 >> v2:172.21.15.144:3300/0 conn(0x7efccc09e910 msgr2=0x7efccc09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 --2- 172.21.15.116:0/1756944888 >> v2:172.21.15.144:3300/0 conn(0x7efccc09e910 0x7efccc09ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 -- 172.21.15.116:0/1756944888 >> v2:172.21.15.116:3300/0 conn(0x7efccc09f220 msgr2=0x7efccc068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 --2- 172.21.15.116:0/1756944888 >> v2:172.21.15.116:3300/0 conn(0x7efccc09f220 0x7efccc068ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient: _finish_hunting 0 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 monclient: found mon.noname-b 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd13c7640 1 -- 172.21.15.116:0/1756944888 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc0a13f0 con 0x7efccc131ab0 2024-09-18T12:45:44.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.464+0000 7efcd1bc8640 1 --2- 172.21.15.116:0/1756944888 >> v2:172.21.15.116:3300/0 conn(0x7efccc09f220 0x7efccc068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd13c7640 1 -- 172.21.15.116:0/1756944888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 msgr2=0x7efccc131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd13c7640 1 --2- 172.21.15.116:0/1756944888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 0x7efccc131e80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efccc0a5ec0 tx=0x7efcc80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e47816077dfff128 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 1 -- 172.21.15.116:0/1756944888 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8032440 con 0x7efccc131ab0 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: dump: 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: _finish_auth 0 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 1 -- 172.21.15.116:0/1756944888 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc80325a0 con 0x7efccc131ab0 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 1 -- 172.21.15.116:0/1756944888 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8032980 con 0x7efccc131ab0 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 5 monclient: authenticate success, global_id 244157 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 4 set_mon_vals no callback set 2024-09-18T12:45:44.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcc3fff640 10 monclient: dump: 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 10 monclient: shutdown 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 -- 172.21.15.116:0/1756944888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 msgr2=0x7efccc131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 --2- 172.21.15.116:0/1756944888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 0x7efccc131e80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efccc0a5ec0 tx=0x7efcc80091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 -- 172.21.15.116:0/1756944888 shutdown_connections 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 --2- 172.21.15.116:0/1756944888 >> v2:172.21.15.116:3300/0 conn(0x7efccc09f220 0x7efccc068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 --2- 172.21.15.116:0/1756944888 >> v2:172.21.15.144:3300/0 conn(0x7efccc09e910 0x7efccc09ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 --2- 172.21.15.116:0/1756944888 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc131ab0 0x7efccc131e80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 -- 172.21.15.116:0/1756944888 >> 172.21.15.116:0/1756944888 conn(0x7efccc067d30 msgr2=0x7efccc0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 -- 172.21.15.116:0/1756944888 shutdown_connections 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.467+0000 7efcd362b640 1 -- 172.21.15.116:0/1756944888 wait complete. 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.468+0000 7efcd362b640 10 monclient: build_initial_monmap 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.468+0000 7efcd362b640 10 monclient: monmap: 2024-09-18T12:45:44.481 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.468+0000 7efcd362b640 1 Processor -- start 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.468+0000 7efcd362b640 1 -- start start 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.468+0000 7efcd362b640 10 monclient: init 2024-09-18T12:45:44.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 0x7efccc09aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): picked mon.noname-c con 0x7efccc09e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efccc130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): picked mon.noname-b con 0x7efccc09f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efccc11b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): picked mon.noname-a con 0x7efccc131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efccc11dd20 con 0x7efccc131ab0 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efccc09bde0 con 0x7efccc09f220 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efccc11c770 con 0x7efccc09e910 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd362b640 10 monclient(hunting): authenticate will time out at 1971.790328s 2024-09-18T12:45:44.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efccc11b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 0x7efccc09aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efccc11b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41866/0 (socket says 172.21.15.116:41866) 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 1 -- 172.21.15.116:0/3955829680 learned_addr learned my addr 172.21.15.116:0/3955829680 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 10 monclient(hunting): get_auth_request con 0x7efccc131ab0 auth_method 0 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): get_auth_request con 0x7efccc09e910 auth_method 0 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efccc130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 10 monclient(hunting): get_auth_request con 0x7efccc09f220 auth_method 0 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd0bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient(hunting): handle_auth_done global_id 244160 payload 824 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 msgr2=0x7efccc130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efccc130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 msgr2=0x7efccc11b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efccc11b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient: _finish_hunting 0 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 1 monclient: found mon.noname-c 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc4009a20 con 0x7efccc09e910 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.469+0000 7efcd1bc8640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efccc11b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.471+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 msgr2=0x7efccc09aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.471+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 0x7efccc09aaa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efcc8031000 tx=0x7efcc8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:44.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 msgr2=0x7efccc09aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 msgr2=0x7efccc09aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 0x7efccc09aaa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efcc8031000 tx=0x7efcc8032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 0x7efccc09aaa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efcc8031000 tx=0x7efcc8032000 comp rx=0 tx=0).stop 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 msgr2=0x7efccc09aaa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efccc09e910 0x7efccc09aaa0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efcb0005470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient(hunting): picked mon.noname-a con 0x7efccc131ab0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb0005dd0 0x7efcb0008250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient(hunting): picked mon.noname-c con 0x7efcb0005dd0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:44.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efcb000ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient(hunting): picked mon.noname-b con 0x7efccc09f220 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb000b790 con 0x7efccc131ab0 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb000b900 con 0x7efccc09f220 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb000bac0 con 0x7efcb0005dd0 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcc17fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efcb0005470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb0005dd0 0x7efcb0008250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): get_auth_request con 0x7efccc131ab0 auth_method 0 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 10 monclient(hunting): get_auth_request con 0x7efcb0005dd0 auth_method 0 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd1bc8640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efcb000ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd1bc8640 10 monclient(hunting): get_auth_request con 0x7efccc09f220 auth_method 0 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd1bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:44.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd1bc8640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd1bc8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient(hunting): handle_auth_done global_id 244160 payload 293 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb0005dd0 msgr2=0x7efcb0008250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb0005dd0 0x7efcb0008250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 msgr2=0x7efcb000ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efcb000ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient: _finish_hunting 0 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 monclient: found mon.noname-a 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb000bd40 con 0x7efccc131ab0 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd1bc8640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efcb000ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.472+0000 7efcd0bc6640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb0005dd0 0x7efcb0008250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd13c7640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 msgr2=0x7efcb0005470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd13c7640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efcb0005470 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009420 tx=0x7efcc800b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8031ed0 con 0x7efccc131ab0 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: dump: 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: _finish_auth 0 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd362b640 5 monclient: authenticate success, global_id 244160 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc8044410 con 0x7efccc131ab0 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd362b640 10 monclient: _renew_subs 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd362b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc8009450 con 0x7efccc131ab0 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc80447f0 con 0x7efccc131ab0 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc17fa640 10 monclient: dump: 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd362b640 10 monclient: _renew_subs 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcc37fe640 4 set_mon_vals no callback set 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd362b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.474+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efccc1416c0 con 0x7efccc131ab0 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.475+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7efcc8044be0 con 0x7efccc131ab0 2024-09-18T12:45:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.475+0000 7efcd362b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.475+0000 7efcd362b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.475+0000 7efcd362b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.475+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efccc04bf60 con 0x7efccc131ab0 2024-09-18T12:45:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.475+0000 7efcc17fa640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efccc09e910 0x7efcb0042540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.476+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7efcc806dfe0 con 0x7efccc131ab0 2024-09-18T12:45:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.476+0000 7efcd0bc6640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efccc09e910 0x7efcb0042540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.476+0000 7efcd0bc6640 10 monclient: get_auth_request con 0x7efccc09e910 auth_method 0 2024-09-18T12:45:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.478+0000 7efcd0bc6640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efccc09e910 0x7efcb0042540 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7efcbc006f80 tx=0x7efcbc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.480+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7efcc8030340 con 0x7efccc131ab0 2024-09-18T12:45:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.480+0000 7efcc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.480+0000 7efcc17fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.634+0000 7efcd362b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:45:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.634+0000 7efcd362b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:45:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.634+0000 7efcd362b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.634+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efccc11d990 con 0x7efccc131ab0 2024-09-18T12:45:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.635+0000 7efcc17fa640 1 -- 172.21.15.116:0/3955829680 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) v1 ==== 74+0+20424 (secure 0 0 0) 0x7efcc804a070 con 0x7efccc131ab0 2024-09-18T12:45:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.635+0000 7efcc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T12:45:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.635+0000 7efcc17fa640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:45:44.637 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:45:44.637 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":267,"fsid":"2f704457-11dd-41ce-9561-691d67058c00","created":"2024-09-18T12:25:12.556239+0000","modified":"2024-09-18T12:34:45.514528+0000","last_up_change":"2024-09-18T12:25:21.884529+0000","last_in_change":"2024-09-18T12:30:33.575141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":181,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T12:25:22.596367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T12:25:24.541264+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-18T12:30:56.176913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-18T12:30:56.676013+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"158","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"d06ed7ed-3035-4edf-bcb6-d5e47db40d9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":31578},{"type":"v1","addr":"172.21.15.116:6817","nonce":31578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":31578},{"type":"v1","addr":"172.21.15.116:6819","nonce":31578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":31578},{"type":"v1","addr":"172.21.15.116:6823","nonce":31578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":31578},{"type":"v1","addr":"172.21.15.116:6821","nonce":31578}]},"public_addr":"172.21.15.116:6817/31578","cluster_addr":"172.21.15.116:6819/31578","heartbeat_back_addr":"172.21.15.116:6823/31578","heartbeat_front_addr":"172.21.15.116:6821/31578","state":["exists","up"]},{"osd":1,"uuid":"93e7e90b-2e87-4214-be44-eee121567700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":31579},{"type":"v1","addr":"172.21.15.116:6801","nonce":31579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":31579},{"type":"v1","addr":"172.21.15.116:6803","nonce":31579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":31579},{"type":"v1","addr":"172.21.15.116:6807","nonce":31579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":31579},{"type":"v1","addr":"172.21.15.116:6805","nonce":31579}]},"public_addr":"172.21.15.116:6801/31579","cluster_addr":"172.21.15.116:6803/31579","heartbeat_back_addr":"172.21.15.116:6807/31579","heartbeat_front_addr":"172.21.15.116:6805/31579","state":["exists","up"]},{"osd":2,"uuid":"34d020ff-733c-4528-a9ff-b1ef60f8f74d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":31580},{"type":"v1","addr":"172.21.15.116:6809","nonce":31580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":31580},{"type":"v1","addr":"172.21.15.116:6811","nonce":31580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":31580},{"type":"v1","addr":"172.21.15.116:6815","nonce":31580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":31580},{"type":"v1","addr":"172.21.15.116:6813","nonce":31580}]},"public_addr":"172.21.15.116:6809/31580","cluster_addr":"172.21.15.116:6811/31580","heartbeat_back_addr":"172.21.15.116:6815/31580","heartbeat_front_addr":"172.21.15.116:6813/31580","state":["exists","up"]},{"osd":3,"uuid":"26e02223-6416-401c-a325-01fad4cf7ce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":32112},{"type":"v1","addr":"172.21.15.144:6806","nonce":32112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6809","nonce":32112},{"type":"v1","addr":"172.21.15.144:6811","nonce":32112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":32112},{"type":"v1","addr":"172.21.15.144:6818","nonce":32112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6813","nonce":32112},{"type":"v1","addr":"172.21.15.144:6815","nonce":32112}]},"public_addr":"172.21.15.144:6806/32112","cluster_addr":"172.21.15.144:6811/32112","heartbeat_back_addr":"172.21.15.144:6818/32112","heartbeat_front_addr":"172.21.15.144:6815/32112","state":["exists","up"]},{"osd":4,"uuid":"9263cabb-9a65-4cec-a9ab-e25999ca2ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":32113},{"type":"v1","addr":"172.21.15.144:6801","nonce":32113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":32113},{"type":"v1","addr":"172.21.15.144:6803","nonce":32113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":32113},{"type":"v1","addr":"172.21.15.144:6810","nonce":32113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6805","nonce":32113},{"type":"v1","addr":"172.21.15.144:6807","nonce":32113}]},"public_addr":"172.21.15.144:6801/32113","cluster_addr":"172.21.15.144:6803/32113","heartbeat_back_addr":"172.21.15.144:6810/32113","heartbeat_front_addr":"172.21.15.144:6807/32113","state":["exists","up"]},{"osd":5,"uuid":"174ff7a9-21fc-4aa3-8801-81ebe54449b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":32114},{"type":"v1","addr":"172.21.15.144:6814","nonce":32114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":32114},{"type":"v1","addr":"172.21.15.144:6819","nonce":32114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":32114},{"type":"v1","addr":"172.21.15.144:6823","nonce":32114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":32114},{"type":"v1","addr":"172.21.15.144:6821","nonce":32114}]},"public_addr":"172.21.15.144:6814/32114","cluster_addr":"172.21.15.144:6819/32114","heartbeat_back_addr":"172.21.15.144:6823/32114","heartbeat_front_addr":"172.21.15.144:6821/32114","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"4b751cce-d39e-4f85-9cff-56f6325f3ab2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.664684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.608350+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.689049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.706015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.672781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T12:25:20.692341+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:45:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efccc09e910 msgr2=0x7efcb0042540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efccc09e910 0x7efcb0042540 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7efcbc006f80 tx=0x7efcbc008040 comp rx=0 tx=0).stop 2024-09-18T12:45:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 10 monclient: shutdown 2024-09-18T12:45:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 msgr2=0x7efcb0005470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efcb0005470 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009420 tx=0x7efcc800b050 comp rx=0 tx=0).stop 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 shutdown_connections 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7efccc09e910 0x7efcb0042540 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efccc09f220 0x7efcb000ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efcb0005dd0 0x7efcb0008250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 --2- 172.21.15.116:0/3955829680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efccc131ab0 0x7efcb0005470 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 >> 172.21.15.116:0/3955829680 conn(0x7efccc067d30 msgr2=0x7efccc0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 shutdown_connections 2024-09-18T12:45:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:44.637+0000 7efcd362b640 1 -- 172.21.15.116:0/3955829680 wait complete. 2024-09-18T12:45:45.649 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T12:45:45.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:45:45.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.757+0000 7f7990709640 10 monclient: get_monmap_and_config 2024-09-18T12:45:45.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.757+0000 7f7990709640 10 monclient: build_initial_monmap 2024-09-18T12:45:45.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.757+0000 7f7990709640 10 monclient: monmap: 2024-09-18T12:45:45.758 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:45.758440+0000 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:45.758440+0000 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:45.759 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 1 Processor -- start 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 1 -- start start 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 10 monclient: init 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f798806ec00 0x7f79880981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 10 monclient(hunting): picked mon.noname-a con 0x7f798806ec00 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.758+0000 7f7990709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f798809cd10 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7988131df0 0x7f7988138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 10 monclient(hunting): picked mon.noname-b con 0x7f7988131df0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7988132390 con 0x7f798809cd10 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798809cad0 con 0x7f798806ec00 2024-09-18T12:45:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7988098e30 con 0x7f7988131df0 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f7990709640 10 monclient(hunting): authenticate will time out at 1973.080033s 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f798806ec00 0x7f79880981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7988131df0 0x7f7988138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f798806ec00 0x7f79880981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47050/0 (socket says 172.21.15.116:47050) 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 -- 172.21.15.116:0/305995730 learned_addr learned my addr 172.21.15.116:0/305995730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): get_auth_request con 0x7f798806ec00 auth_method 0 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 10 monclient(hunting): get_auth_request con 0x7f7988131df0 auth_method 0 2024-09-18T12:45:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient(hunting): handle_auth_done global_id 194712 payload 274 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 -- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7988131df0 msgr2=0x7f7988138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 --2- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7988131df0 0x7f7988138cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 -- 172.21.15.116:0/305995730 >> v1:172.21.15.144:6789/0 conn(0x7f798809cd10 legacy=0x7f7988098720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:45.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient: _finish_hunting 0 2024-09-18T12:45:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 monclient: found mon.noname-a 2024-09-18T12:45:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798e4a5640 1 -- 172.21.15.116:0/305995730 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798806efd0 con 0x7f798806ec00 2024-09-18T12:45:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.759+0000 7f798eca6640 1 --2- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7988131df0 0x7f7988138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798e4a5640 1 -- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798806ec00 msgr2=0x7f79880981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798e4a5640 1 --2- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798806ec00 0x7f79880981e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7984009940 tx=0x7f79840091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bdbc768049c7c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 1 -- 172.21.15.116:0/305995730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798402ae70 con 0x7f798806ec00 2024-09-18T12:45:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: dump: 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: _finish_auth 0 2024-09-18T12:45:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f7990709640 5 monclient: authenticate success, global_id 194712 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f7990709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 1 -- 172.21.15.116:0/305995730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7984032440 con 0x7f798806ec00 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f7990709640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f7990709640 4 set_mon_vals no callback set 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 1 -- 172.21.15.116:0/305995730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79840327e0 con 0x7f798806ec00 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.762+0000 7f798d4a3640 10 monclient: dump: 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 10 monclient: shutdown 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 -- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798806ec00 msgr2=0x7f79880981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 --2- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798806ec00 0x7f79880981e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7984009940 tx=0x7f79840091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 -- 172.21.15.116:0/305995730 shutdown_connections 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 --2- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7988131df0 0x7f7988138cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 --2- 172.21.15.116:0/305995730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798806ec00 0x7f79880981e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 -- 172.21.15.116:0/305995730 >> 172.21.15.116:0/305995730 conn(0x7f79880a1d20 msgr2=0x7f79880a20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 -- 172.21.15.116:0/305995730 shutdown_connections 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 -- 172.21.15.116:0/305995730 wait complete. 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 10 monclient: build_initial_monmap 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 10 monclient: monmap: 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:45.775 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.763+0000 7f7990709640 1 Processor -- start 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 1 -- start start 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient: init 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f798806ec00 0x7f79881416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): picked mon.noname-c con 0x7f798806ec00 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798809cd10 0x7f7988141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): picked mon.noname-a con 0x7f798809cd10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7988131df0 0x7f7988142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): picked mon.noname-b con 0x7f7988131df0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798811a160 con 0x7f798809cd10 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798809d670 con 0x7f7988131df0 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798809fec0 con 0x7f798806ec00 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f7990709640 10 monclient(hunting): authenticate will time out at 1973.085484s 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f798806ec00 0x7f79881416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798809cd10 0x7f7988141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f798806ec00 0x7f79881416e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54510/0 (socket says 172.21.15.116:54510) 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798809cd10 0x7f7988141c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47058/0 (socket says 172.21.15.116:47058) 2024-09-18T12:45:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 1 -- 172.21.15.116:0/2195795368 learned_addr learned my addr 172.21.15.116:0/2195795368 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 10 monclient(hunting): get_auth_request con 0x7f798806ec00 auth_method 0 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): get_auth_request con 0x7f798809cd10 auth_method 0 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798eca6640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7988131df0 0x7f7988142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798eca6640 10 monclient(hunting): get_auth_request con 0x7f7988131df0 auth_method 0 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798eca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798eca6640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798eca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient(hunting): handle_auth_done global_id 194715 payload 824 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 -- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f798806ec00 msgr2=0x7f79881416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f798806ec00 0x7f79881416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 -- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7988131df0 msgr2=0x7f7988142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7988131df0 0x7f7988142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient: _finish_hunting 0 2024-09-18T12:45:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 monclient: found mon.noname-a 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798dca4640 1 -- 172.21.15.116:0/2195795368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980009a20 con 0x7f798809cd10 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.764+0000 7f798e4a5640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f798806ec00 0x7f79881416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f798dca4640 1 -- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798809cd10 msgr2=0x7f7988141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f798dca4640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798809cd10 0x7f7988141c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f797800c4e0 tx=0x7f7978007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780117a0 con 0x7f798809cd10 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: dump: 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f7990709640 5 monclient: authenticate success, global_id 194715 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978011900 con 0x7f798809cd10 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978011ca0 con 0x7f798809cd10 2024-09-18T12:45:45.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f7990709640 10 monclient: _renew_subs 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f7990709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7984009450 con 0x7f798809cd10 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f797e7fc640 10 monclient: dump: 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f798cca2640 4 set_mon_vals no callback set 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f7990709640 10 monclient: _renew_subs 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f7990709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.767+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7988118930 con 0x7f798809cd10 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f7990709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f798805ed70 con 0x7f798809cd10 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7978012630 con 0x7f798809cd10 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f797e7fc640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f796c038490 0x7f796c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f797804bfe0 con 0x7f798809cd10 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f797e7fc640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f796c041450 0x7f796c0438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 --> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f796c043f70 con 0x7f796c041450 2024-09-18T12:45:45.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=267) v2 ==== 24+0+0 (secure 0 0 0) 0x7f797804e570 con 0x7f798809cd10 2024-09-18T12:45:45.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f797e7fc640 10 monclient: handle_get_version_reply finishing 1 version 267 2024-09-18T12:45:45.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f798eca6640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f796c041450 0x7f796c0438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f798eca6640 10 monclient: get_auth_request con 0x7f796c041450 auth_method 0 2024-09-18T12:45:45.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f798e4a5640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f796c038490 0x7f796c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.768+0000 7f798e4a5640 10 monclient: get_auth_request con 0x7f796c038490 auth_method 0 2024-09-18T12:45:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.771+0000 7f798e4a5640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f796c038490 0x7f796c03a940 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7984009210 tx=0x7f7984032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.773+0000 7f798eca6640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f796c041450 0x7f796c0438a0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.773+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== osd.0 v2:172.21.15.116:6816/31578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f796c043f70 con 0x7f796c041450 2024-09-18T12:45:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 --> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f79880638d0 con 0x7f796c041450 2024-09-18T12:45:45.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f797e7fc640 1 -- 172.21.15.116:0/2195795368 <== osd.0 v2:172.21.15.116:6816/31578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f79880638d0 con 0x7f796c041450 2024-09-18T12:45:45.786 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:45.786 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:45:45.786 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:45.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f796c041450 msgr2=0x7f796c0438a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:6816/31578,v1:172.21.15.116:6817/31578] conn(0x7f796c041450 0x7f796c0438a0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f796c038490 msgr2=0x7f796c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f796c038490 0x7f796c03a940 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7984009210 tx=0x7f7984032000 comp rx=0 tx=0).stop 2024-09-18T12:45:45.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 10 monclient: shutdown 2024-09-18T12:45:45.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:45.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798809cd10 msgr2=0x7f7988141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 --2- 172.21.15.116:0/2195795368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798809cd10 0x7f7988141c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f797800c4e0 tx=0x7f7978007e10 comp rx=0 tx=0).stop 2024-09-18T12:45:45.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f798eca6640 1 -- 172.21.15.116:0/2195795368 reap_dead start 2024-09-18T12:45:45.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 shutdown_connections 2024-09-18T12:45:45.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 >> 172.21.15.116:0/2195795368 conn(0x7f79880a1d20 msgr2=0x7f7988067110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:45.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 shutdown_connections 2024-09-18T12:45:45.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.785+0000 7f7990709640 1 -- 172.21.15.116:0/2195795368 wait complete. 2024-09-18T12:45:45.797 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T12:45:45.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.890+0000 7f7e1599d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.890+0000 7f7e1599d640 10 monclient: build_initial_monmap 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.890+0000 7f7e1599d640 10 monclient: monmap: 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:45.891354+0000 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:45.891354+0000 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:45.895 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 1 Processor -- start 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 1 -- start start 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient: init 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e10131ab0 0x7f7e10131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e10131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e1009e910 0x7f7e1009ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e1009e910 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e1009f220 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:45.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1009faf0 con 0x7f7e1009f220 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1009fd30 con 0x7f7e1009e910 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100a1110 con 0x7f7e10131ab0 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e1599d640 10 monclient(hunting): authenticate will time out at 1973.212755s 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e10131ab0 0x7f7e10131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e1009e910 0x7f7e1009ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e1009e910 0x7f7e1009ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47070/0 (socket says 172.21.15.116:47070) 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e10131ab0 0x7f7e10131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54526/0 (socket says 172.21.15.116:54526) 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 1 -- 172.21.15.116:0/1170813102 learned_addr learned my addr 172.21.15.116:0/1170813102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e1009e910 auth_method 0 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0effd640 10 monclient(hunting): get_auth_request con 0x7f7e10131ab0 auth_method 0 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.891+0000 7f7e0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 10 monclient(hunting): handle_auth_done global_id 194718 payload 274 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 1 -- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e10131ab0 msgr2=0x7f7e10131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:45.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 1 --2- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e10131ab0 0x7f7e10131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 1 -- 172.21.15.116:0/1170813102 >> v1:172.21.15.144:6789/0 conn(0x7f7e1009f220 legacy=0x7f7e10068ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 10 monclient: _finish_hunting 0 2024-09-18T12:45:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 1 monclient: found mon.noname-a 2024-09-18T12:45:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0e7fc640 1 -- 172.21.15.116:0/1170813102 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100a13f0 con 0x7f7e1009e910 2024-09-18T12:45:45.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.892+0000 7f7e0effd640 1 --2- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e10131ab0 0x7f7e10131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.894+0000 7f7e0e7fc640 1 -- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009e910 msgr2=0x7f7e1009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.894+0000 7f7e0e7fc640 1 --2- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009e910 0x7f7e1009ece0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009910 tx=0x7f7e040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fc048ea9309dfa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 1 -- 172.21.15.116:0/1170813102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04032440 con 0x7f7e1009e910 2024-09-18T12:45:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: dump: 2024-09-18T12:45:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 5 monclient: authenticate success, global_id 194718 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 1 -- 172.21.15.116:0/1170813102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e040325a0 con 0x7f7e1009e910 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 1 -- 172.21.15.116:0/1170813102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04032980 con 0x7f7e1009e910 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 4 set_mon_vals no callback set 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e0dffb640 10 monclient: dump: 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 10 monclient: shutdown 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 -- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009e910 msgr2=0x7f7e1009ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 --2- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009e910 0x7f7e1009ece0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009910 tx=0x7f7e040091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 -- 172.21.15.116:0/1170813102 shutdown_connections 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 --2- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009e910 0x7f7e1009ece0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 --2- 172.21.15.116:0/1170813102 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e10131ab0 0x7f7e10131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 -- 172.21.15.116:0/1170813102 >> 172.21.15.116:0/1170813102 conn(0x7f7e10067d30 msgr2=0x7f7e100a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 -- 172.21.15.116:0/1170813102 shutdown_connections 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.895+0000 7f7e1599d640 1 -- 172.21.15.116:0/1170813102 wait complete. 2024-09-18T12:45:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 10 monclient: build_initial_monmap 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 10 monclient: monmap: 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 1 Processor -- start 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 1 -- start start 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 10 monclient: init 2024-09-18T12:45:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 0x7f7e1011de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e1009e910 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.896+0000 7f7e1599d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009f220 0x7f7e1011e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e1009f220 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e10131ab0 0x7f7e1011e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e10131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101205b0 con 0x7f7e1009f220 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1009bde0 con 0x7f7e10131ab0 2024-09-18T12:45:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100a4430 con 0x7f7e1009e910 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e1599d640 10 monclient(hunting): authenticate will time out at 1973.218035s 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 0x7f7e1011de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009f220 0x7f7e1011e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 0x7f7e1011de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54538/0 (socket says 172.21.15.116:54538) 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 -- 172.21.15.116:0/3555514927 learned_addr learned my addr 172.21.15.116:0/3555514927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): get_auth_request con 0x7f7e1009e910 auth_method 0 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 10 monclient(hunting): get_auth_request con 0x7f7e1009f220 auth_method 0 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient(hunting): handle_auth_done global_id 244172 payload 824 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0f7fe640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e10131ab0 0x7f7e1011e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 -- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e10131ab0 msgr2=0x7f7e1011e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e10131ab0 0x7f7e1011e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 -- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009f220 msgr2=0x7f7e1011e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009f220 0x7f7e1011e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient: _finish_hunting 0 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 monclient: found mon.noname-c 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0effd640 1 -- 172.21.15.116:0/3555514927 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00009390 con 0x7f7e1009e910 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0f7fe640 10 monclient: get_auth_request con 0x7f7e10131ab0 auth_method 0 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0e7fc640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009f220 0x7f7e1011e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.897+0000 7f7e0f7fe640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e10131ab0 0x7f7e1011e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e0effd640 1 -- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 msgr2=0x7f7e1011de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e0effd640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 0x7f7e1011de40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000ec80 tx=0x7f7e000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 1 -- 172.21.15.116:0/3555514927 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000eec0 con 0x7f7e1009e910 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: dump: 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 1 -- 172.21.15.116:0/3555514927 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0000cdd0 con 0x7f7e1009e910 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e1599d640 5 monclient: authenticate success, global_id 244172 2024-09-18T12:45:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e1599d640 10 monclient: _renew_subs 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e1599d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e04009450 con 0x7f7e1009e910 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 1 -- 172.21.15.116:0/3555514927 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00013680 con 0x7f7e1009e910 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7df6ffd640 10 monclient: dump: 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e1599d640 10 monclient: _renew_subs 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e1599d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e10131490 con 0x7f7e1009e910 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.899+0000 7f7e0d7fa640 4 set_mon_vals no callback set 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.900+0000 7f7df6ffd640 1 -- 172.21.15.116:0/3555514927 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f7e000139f0 con 0x7f7e1009e910 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.900+0000 7f7e1599d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.900+0000 7f7e1599d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.900+0000 7f7e1599d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.900+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10121920 con 0x7f7e1009e910 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.900+0000 7f7df6ffd640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7dfc038490 0x7f7dfc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.900+0000 7f7df6ffd640 1 -- 172.21.15.116:0/3555514927 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f7e0004c720 con 0x7f7e1009e910 2024-09-18T12:45:45.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.901+0000 7f7e0e7fc640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7dfc038490 0x7f7dfc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:45.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.901+0000 7f7e0e7fc640 10 monclient: get_auth_request con 0x7f7dfc038490 auth_method 0 2024-09-18T12:45:45.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.903+0000 7f7e0e7fc640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7dfc038490 0x7f7dfc03a940 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7e04034000 tx=0x7f7e04032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:45.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.905+0000 7f7df6ffd640 1 -- 172.21.15.116:0/3555514927 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f7e000236b0 con 0x7f7e1009e910 2024-09-18T12:45:45.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.905+0000 7f7df6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:45.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:45.905+0000 7f7df6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.060+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e1011f9f0 con 0x7f7dfc038490 2024-09-18T12:45:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.060+0000 7f7df6ffd640 1 -- 172.21.15.116:0/3555514927 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7e0403c440 con 0x7f7dfc038490 2024-09-18T12:45:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T12:45:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7dfc038490 msgr2=0x7f7dfc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7dfc038490 0x7f7dfc03a940 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7e04034000 tx=0x7f7e04032000 comp rx=0 tx=0).stop 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 10 monclient: shutdown 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 msgr2=0x7f7e1011de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 0x7f7e1011de40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000ec80 tx=0x7f7e000090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 shutdown_connections 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f7dfc038490 0x7f7dfc03a940 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e10131ab0 0x7f7e1011e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e1009f220 0x7f7e1011e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 --2- 172.21.15.116:0/3555514927 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7e1009e910 0x7f7e1011de40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 >> 172.21.15.116:0/3555514927 conn(0x7f7e10067d30 msgr2=0x7f7e1009b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 shutdown_connections 2024-09-18T12:45:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.062+0000 7f7e1599d640 1 -- 172.21.15.116:0/3555514927 wait complete. 2024-09-18T12:45:46.074 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T12:45:46.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.169+0000 7f043f79d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.169+0000 7f043f79d640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.169+0000 7f043f79d640 10 monclient: monmap: 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:46.170073+0000 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:46.170073+0000 2024-09-18T12:45:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:46.171 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.169+0000 7f043f79d640 1 Processor -- start 2024-09-18T12:45:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 1 -- start start 2024-09-18T12:45:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient: init 2024-09-18T12:45:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0438098740 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0438098b10 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f04381370c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient(hunting): picked mon.noname-b con 0x7f0438098f30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0438099a90 con 0x7f0438098b10 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0438099720 con 0x7f0438098740 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0438097b00 con 0x7f0438098f30 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043f79d640 10 monclient(hunting): authenticate will time out at 1973.491744s 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f04381370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043cd38640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0438098b10 0x7f0438133990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60820/0 (socket says 172.21.15.116:60820) 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043cd38640 1 -- 172.21.15.116:0/3962116892 learned_addr learned my addr 172.21.15.116:0/3962116892 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 10 monclient(hunting): get_auth_request con 0x7f0438098f30 auth_method 0 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.170+0000 7f043dd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 10 monclient(hunting): handle_auth_done global_id 244175 payload 274 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 1 -- 172.21.15.116:0/3962116892 >> v1:172.21.15.144:6789/0 conn(0x7f0438098740 legacy=0x7f043809a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 1 -- 172.21.15.116:0/3962116892 >> v1:172.21.15.116:6789/0 conn(0x7f0438098b10 legacy=0x7f0438133990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 1 monclient: found mon.noname-b 2024-09-18T12:45:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.171+0000 7f043dd3a640 1 -- 172.21.15.116:0/3962116892 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438099440 con 0x7f0438098f30 2024-09-18T12:45:46.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.173+0000 7f043dd3a640 1 -- 172.21.15.116:0/3962116892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 msgr2=0x7f04381370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.173+0000 7f043dd3a640 1 --2- 172.21.15.116:0/3962116892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f04381370c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04380998b0 tx=0x7f0430009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da9aadc257a43272 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 1 -- 172.21.15.116:0/3962116892 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430007840 con 0x7f0438098f30 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: dump: 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 5 monclient: authenticate success, global_id 244175 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 1 -- 172.21.15.116:0/3962116892 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043002d050 con 0x7f0438098f30 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 1 -- 172.21.15.116:0/3962116892 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430007bd0 con 0x7f0438098f30 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 4 set_mon_vals no callback set 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:46.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f042ffff640 10 monclient: dump: 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 10 monclient: shutdown 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 1 -- 172.21.15.116:0/3962116892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 msgr2=0x7f04381370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 1 --2- 172.21.15.116:0/3962116892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f04381370c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04380998b0 tx=0x7f0430009780 comp rx=0 tx=0).stop 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 1 -- 172.21.15.116:0/3962116892 shutdown_connections 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 1 --2- 172.21.15.116:0/3962116892 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f04381370c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 1 -- 172.21.15.116:0/3962116892 >> 172.21.15.116:0/3962116892 conn(0x7f04380a1150 msgr2=0x7f04380a1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 1 -- 172.21.15.116:0/3962116892 shutdown_connections 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.174+0000 7f043f79d640 1 -- 172.21.15.116:0/3962116892 wait complete. 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.175+0000 7f043f79d640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.175+0000 7f043f79d640 10 monclient: monmap: 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.175+0000 7f043f79d640 1 Processor -- start 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.175+0000 7f043f79d640 1 -- start start 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.175+0000 7f043f79d640 10 monclient: init 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0438098740 0x7f043809b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): picked mon.noname-b con 0x7f0438098740 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0438098b10 0x7f04381346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): picked mon.noname-a con 0x7f0438098b10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f0438134c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): picked mon.noname-c con 0x7f0438098f30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043809b1d0 con 0x7f0438098b10 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0438117a80 con 0x7f0438098740 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0438117d30 con 0x7f0438098f30 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043f79d640 10 monclient(hunting): authenticate will time out at 1973.497141s 2024-09-18T12:45:46.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f0438134c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0438098b10 0x7f04381346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f0438134c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54552/0 (socket says 172.21.15.116:54552) 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 -- 172.21.15.116:0/1474848407 learned_addr learned my addr 172.21.15.116:0/1474848407 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): get_auth_request con 0x7f0438098f30 auth_method 0 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0438098b10 0x7f04381346e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47092/0 (socket says 172.21.15.116:47092) 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 10 monclient(hunting): get_auth_request con 0x7f0438098b10 auth_method 0 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043d539640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0438098740 0x7f043809b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043d539640 10 monclient(hunting): get_auth_request con 0x7f0438098740 auth_method 0 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043d539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043d539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043d539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient(hunting): handle_auth_done global_id 244178 payload 824 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 -- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0438098740 msgr2=0x7f043809b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0438098740 0x7f043809b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 -- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0438098b10 msgr2=0x7f04381346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0438098b10 0x7f04381346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 monclient: found mon.noname-c 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043dd3a640 1 -- 172.21.15.116:0/1474848407 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0430009a20 con 0x7f0438098f30 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043d539640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0438098740 0x7f043809b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.176+0000 7f043cd38640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0438098b10 0x7f04381346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f043dd3a640 1 -- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 msgr2=0x7f0438134c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f043dd3a640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f0438134c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0430009de0 tx=0x7f0430037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430031cd0 con 0x7f0438098f30 2024-09-18T12:45:46.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: dump: 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0430040070 con 0x7f0438098f30 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430045640 con 0x7f0438098f30 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042d7fa640 10 monclient: dump: 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.192 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 5 monclient: authenticate success, global_id 244178 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 10 monclient: _renew_subs 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043809c2e0 con 0x7f0438098f30 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.178+0000 7f042f7fe640 4 set_mon_vals no callback set 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 10 monclient: _renew_subs 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0438135ab0 con 0x7f0438098f30 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f043805ed70 con 0x7f0438098f30 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.179+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0430042030 con 0x7f0438098f30 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f042d7fa640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f041c038490 0x7f041c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f043006dec0 con 0x7f0438098f30 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f042d7fa640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7f041c041450 0x7f041c0438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 --> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f041c043f70 con 0x7f041c041450 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=267) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0430070450 con 0x7f0438098f30 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f042d7fa640 10 monclient: handle_get_version_reply finishing 1 version 267 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f043cd38640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7f041c041450 0x7f041c0438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f043cd38640 10 monclient: get_auth_request con 0x7f041c041450 auth_method 0 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f043d539640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f041c038490 0x7f041c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.180+0000 7f043d539640 10 monclient: get_auth_request con 0x7f041c038490 auth_method 0 2024-09-18T12:45:46.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.183+0000 7f043d539640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f041c038490 0x7f041c03a940 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0434009970 tx=0x7f0434008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.184+0000 7f043cd38640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7f041c041450 0x7f041c0438a0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.185+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== osd.1 v2:172.21.15.116:6800/31579 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f041c043f70 con 0x7f041c041450 2024-09-18T12:45:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.196+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 --> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0438118a30 con 0x7f041c041450 2024-09-18T12:45:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.196+0000 7f042d7fa640 1 -- 172.21.15.116:0/1474848407 <== osd.1 v2:172.21.15.116:6800/31579 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0428007200 con 0x7f041c041450 2024-09-18T12:45:46.197 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:46.197 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:45:46.197 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7f041c041450 msgr2=0x7f041c0438a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:6800/31579,v1:172.21.15.116:6801/31579] conn(0x7f041c041450 0x7f041c0438a0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f041c038490 msgr2=0x7f041c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f041c038490 0x7f041c03a940 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0434009970 tx=0x7f0434008040 comp rx=0 tx=0).stop 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 10 monclient: shutdown 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 msgr2=0x7f0438134c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 --2- 172.21.15.116:0/1474848407 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0438098f30 0x7f0438134c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0430009de0 tx=0x7f0430037000 comp rx=0 tx=0).stop 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043dd3a640 1 -- 172.21.15.116:0/1474848407 reap_dead start 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 shutdown_connections 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 >> 172.21.15.116:0/1474848407 conn(0x7f04380a1150 msgr2=0x7f043809fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 shutdown_connections 2024-09-18T12:45:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.197+0000 7f043f79d640 1 -- 172.21.15.116:0/1474848407 wait complete. 2024-09-18T12:45:46.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T12:45:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.299+0000 7f46cc909640 10 monclient: get_monmap_and_config 2024-09-18T12:45:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.299+0000 7f46cc909640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.299+0000 7f46cc909640 10 monclient: monmap: 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:46.300390+0000 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:46.300390+0000 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:46.301 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 1 Processor -- start 2024-09-18T12:45:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 1 -- start start 2024-09-18T12:45:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient: init 2024-09-18T12:45:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46c80974f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c809a8a0 0x7f46c809acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient(hunting): picked mon.noname-b con 0x7f46c809a8a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46c8133070 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c812b210 con 0x7f46c8133070 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c80978c0 con 0x7f46c80974f0 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c809ca20 con 0x7f46c809a8a0 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.300+0000 7f46cc909640 10 monclient(hunting): authenticate will time out at 1973.621964s 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46c6d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f46c8133070 0x7f46c8135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60824/0 (socket says 172.21.15.116:60824) 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c809a8a0 0x7f46c809acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46c6d9d640 1 -- 172.21.15.116:0/3875259179 learned_addr learned my addr 172.21.15.116:0/3875259179 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): get_auth_request con 0x7f46c809a8a0 auth_method 0 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient(hunting): handle_auth_done global_id 244181 payload 274 2024-09-18T12:45:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 1 -- 172.21.15.116:0/3875259179 >> v1:172.21.15.144:6789/0 conn(0x7f46c80974f0 legacy=0x7f46c812ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 1 -- 172.21.15.116:0/3875259179 >> v1:172.21.15.116:6789/0 conn(0x7f46c8133070 legacy=0x7f46c8135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 1 monclient: found mon.noname-b 2024-09-18T12:45:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.301+0000 7f46bffff640 1 -- 172.21.15.116:0/3875259179 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c809b1f0 con 0x7f46c809a8a0 2024-09-18T12:45:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46bffff640 1 -- 172.21.15.116:0/3875259179 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c809a8a0 msgr2=0x7f46c809acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46bffff640 1 --2- 172.21.15.116:0/3875259179 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c809a8a0 0x7f46c809acb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009880 tx=0x7f46b00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc6f53728fe518c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 1 -- 172.21.15.116:0/3875259179 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0033440 con 0x7f46c809a8a0 2024-09-18T12:45:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: dump: 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 5 monclient: authenticate success, global_id 244181 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 1 -- 172.21.15.116:0/3875259179 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b00335a0 con 0x7f46c809a8a0 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 4 set_mon_vals no callback set 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 1 -- 172.21.15.116:0/3875259179 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0033940 con 0x7f46c809a8a0 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46c5d9b640 10 monclient: dump: 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.314 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 10 monclient: shutdown 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 1 -- 172.21.15.116:0/3875259179 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c809a8a0 msgr2=0x7f46c809acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 1 --2- 172.21.15.116:0/3875259179 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c809a8a0 0x7f46c809acb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009880 tx=0x7f46b00090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 1 -- 172.21.15.116:0/3875259179 shutdown_connections 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 1 --2- 172.21.15.116:0/3875259179 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c809a8a0 0x7f46c809acb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 1 -- 172.21.15.116:0/3875259179 >> 172.21.15.116:0/3875259179 conn(0x7f46c80672b0 msgr2=0x7f46c80676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 1 -- 172.21.15.116:0/3875259179 shutdown_connections 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.304+0000 7f46cc909640 1 -- 172.21.15.116:0/3875259179 wait complete. 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.305+0000 7f46cc909640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.305+0000 7f46cc909640 10 monclient: monmap: 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:46.316 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.305+0000 7f46cc909640 1 Processor -- start 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.305+0000 7f46cc909640 1 -- start start 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.305+0000 7f46cc909640 10 monclient: init 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 0x7f46c8131c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): picked mon.noname-c con 0x7f46c80974f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46c809a8a0 0x7f46c812db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): picked mon.noname-a con 0x7f46c809a8a0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46c8133070 0x7f46c812e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): picked mon.noname-b con 0x7f46c8133070 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81315f0 con 0x7f46c809a8a0 2024-09-18T12:45:46.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8098300 con 0x7f46c8133070 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c809a460 con 0x7f46c80974f0 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46cc909640 10 monclient(hunting): authenticate will time out at 1973.627086s 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 0x7f46c8131c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46c809a8a0 0x7f46c812db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46c809a8a0 0x7f46c812db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47100/0 (socket says 172.21.15.116:47100) 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 0x7f46c8131c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54580/0 (socket says 172.21.15.116:54580) 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 1 -- 172.21.15.116:0/1009942409 learned_addr learned my addr 172.21.15.116:0/1009942409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 10 monclient(hunting): get_auth_request con 0x7f46c809a8a0 auth_method 0 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): get_auth_request con 0x7f46c80974f0 auth_method 0 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c6d9d640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46c8133070 0x7f46c812e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c6d9d640 10 monclient(hunting): get_auth_request con 0x7f46c8133070 auth_method 0 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient(hunting): handle_auth_done global_id 244184 payload 824 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 -- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46c8133070 msgr2=0x7f46c812e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46c8133070 0x7f46c812e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 -- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46c809a8a0 msgr2=0x7f46c812db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46c809a8a0 0x7f46c812db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 monclient: found mon.noname-c 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c659c640 1 -- 172.21.15.116:0/1009942409 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b0009390 con 0x7f46c80974f0 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46c6d9d640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46c8133070 0x7f46c812e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.306+0000 7f46bffff640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46c809a8a0 0x7f46c812db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:46.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46c659c640 1 -- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 msgr2=0x7f46c8131c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46c659c640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 0x7f46c8131c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c540 tx=0x7f46b8007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 1 -- 172.21.15.116:0/1009942409 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b80117a0 con 0x7f46c80974f0 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: dump: 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.321 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 1 -- 172.21.15.116:0/1009942409 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b8011900 con 0x7f46c80974f0 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46cc909640 5 monclient: authenticate success, global_id 244184 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46cc909640 10 monclient: _renew_subs 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46cc909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c8130cc0 con 0x7f46c80974f0 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 1 -- 172.21.15.116:0/1009942409 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8011cc0 con 0x7f46c80974f0 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:46.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46be7fc640 10 monclient: dump: 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46cc909640 10 monclient: _renew_subs 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46cc909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46c559a640 4 set_mon_vals no callback set 2024-09-18T12:45:46.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.308+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c81309b0 con 0x7f46c80974f0 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.309+0000 7f46be7fc640 1 -- 172.21.15.116:0/1009942409 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f46b8012630 con 0x7f46c80974f0 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.309+0000 7f46cc909640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.309+0000 7f46cc909640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.309+0000 7f46cc909640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.309+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c804bf60 con 0x7f46c80974f0 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.309+0000 7f46be7fc640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46a8038490 0x7f46a803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.309+0000 7f46be7fc640 1 -- 172.21.15.116:0/1009942409 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f46b804bec0 con 0x7f46c80974f0 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.310+0000 7f46bffff640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46a8038490 0x7f46a803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.310+0000 7f46bffff640 10 monclient: get_auth_request con 0x7f46a8038490 auth_method 0 2024-09-18T12:45:46.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.312+0000 7f46bffff640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46a8038490 0x7f46a803a940 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f46b0036040 tx=0x7f46b0033000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.314+0000 7f46be7fc640 1 -- 172.21.15.116:0/1009942409 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f46b8011e60 con 0x7f46c80974f0 2024-09-18T12:45:46.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.314+0000 7f46be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:46.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.314+0000 7f46be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:46.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.468+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f46c8067ba0 con 0x7f46a8038490 2024-09-18T12:45:46.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.468+0000 7f46be7fc640 1 -- 172.21.15.116:0/1009942409 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46c8067ba0 con 0x7f46a8038490 2024-09-18T12:45:46.469 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.470+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46a8038490 msgr2=0x7f46a803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46a8038490 0x7f46a803a940 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f46b0036040 tx=0x7f46b0033000 comp rx=0 tx=0).stop 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 10 monclient: shutdown 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 msgr2=0x7f46c8131c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 0x7f46c8131c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c540 tx=0x7f46b8007e10 comp rx=0 tx=0).stop 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 shutdown_connections 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f46a8038490 0x7f46a803a940 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46c8133070 0x7f46c812e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46c809a8a0 0x7f46c812db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 --2- 172.21.15.116:0/1009942409 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f46c80974f0 0x7f46c8131c90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 >> 172.21.15.116:0/1009942409 conn(0x7f46c80672b0 msgr2=0x7f46c81297d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 shutdown_connections 2024-09-18T12:45:46.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.471+0000 7f46cc909640 1 -- 172.21.15.116:0/1009942409 wait complete. 2024-09-18T12:45:46.482 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T12:45:46.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:45:46.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.577+0000 7f0f93a94640 10 monclient: get_monmap_and_config 2024-09-18T12:45:46.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.577+0000 7f0f93a94640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.577+0000 7f0f93a94640 10 monclient: monmap: 2024-09-18T12:45:46.578 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:46.578362+0000 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:46.578362+0000 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:46.579 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 1 Processor -- start 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 1 -- start start 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 10 monclient: init 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f8c098740 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f8c098b10 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098f30 0x7f0f8c1370c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8c098f30 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c099a90 con 0x7f0f8c098740 2024-09-18T12:45:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.578+0000 7f0f93a94640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c099720 con 0x7f0f8c098b10 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f93a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f93a94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c097b00 con 0x7f0f8c098f30 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f93a94640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f93a94640 10 monclient(hunting): authenticate will time out at 1973.900014s 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f91830640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0f8c098740 0x7f0f8c09a950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60830/0 (socket says 172.21.15.116:60830) 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f91830640 1 -- 172.21.15.116:0/4011944528 learned_addr learned my addr 172.21.15.116:0/4011944528 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 1 --2- 172.21.15.116:0/4011944528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098f30 0x7f0f8c1370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): get_auth_request con 0x7f0f8c098f30 auth_method 0 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient(hunting): handle_auth_done global_id 244187 payload 274 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 1 -- 172.21.15.116:0/4011944528 >> v1:172.21.15.144:6789/0 conn(0x7f0f8c098b10 legacy=0x7f0f8c133990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 1 -- 172.21.15.116:0/4011944528 >> v1:172.21.15.116:6789/0 conn(0x7f0f8c098740 legacy=0x7f0f8c09a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 1 monclient: found mon.noname-b 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.579+0000 7f0f92031640 1 -- 172.21.15.116:0/4011944528 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c099440 con 0x7f0f8c098f30 2024-09-18T12:45:46.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f92031640 1 -- 172.21.15.116:0/4011944528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098f30 msgr2=0x7f0f8c1370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f92031640 1 --2- 172.21.15.116:0/4011944528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098f30 0x7f0f8c1370c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0998b0 tx=0x7f0f84009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b82620be3dc9bca6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 1 -- 172.21.15.116:0/4011944528 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84007840 con 0x7f0f8c098f30 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: dump: 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.591 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 5 monclient: authenticate success, global_id 244187 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 1 -- 172.21.15.116:0/4011944528 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8402d050 con 0x7f0f8c098f30 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 4 set_mon_vals no callback set 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 1 -- 172.21.15.116:0/4011944528 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84007bd0 con 0x7f0f8c098f30 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:46.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f9082e640 10 monclient: dump: 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.593 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 10 monclient: shutdown 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 1 -- 172.21.15.116:0/4011944528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098f30 msgr2=0x7f0f8c1370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 1 --2- 172.21.15.116:0/4011944528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098f30 0x7f0f8c1370c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0998b0 tx=0x7f0f84009780 comp rx=0 tx=0).stop 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 1 -- 172.21.15.116:0/4011944528 shutdown_connections 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 1 --2- 172.21.15.116:0/4011944528 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098f30 0x7f0f8c1370c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.582+0000 7f0f93a94640 1 -- 172.21.15.116:0/4011944528 >> 172.21.15.116:0/4011944528 conn(0x7f0f8c0a1150 msgr2=0x7f0f8c0a1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.583+0000 7f0f93a94640 1 -- 172.21.15.116:0/4011944528 shutdown_connections 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.583+0000 7f0f93a94640 1 -- 172.21.15.116:0/4011944528 wait complete. 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.583+0000 7f0f93a94640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.583+0000 7f0f93a94640 10 monclient: monmap: 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:46.594 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.583+0000 7f0f93a94640 1 Processor -- start 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.583+0000 7f0f93a94640 1 -- start start 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.583+0000 7f0f93a94640 10 monclient: init 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098740 0x7f0f8c135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): picked mon.noname-c con 0x7f0f8c098740 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f8c098b10 0x7f0f8c11d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): picked mon.noname-a con 0x7f0f8c098b10 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f8c098f30 0x7f0f8c11de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8c098f30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c09b180 con 0x7f0f8c098b10 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c1211e0 con 0x7f0f8c098f30 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c11f950 con 0x7f0f8c098740 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f93a94640 10 monclient(hunting): authenticate will time out at 1973.905272s 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098740 0x7f0f8c135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f8c098b10 0x7f0f8c11d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098740 0x7f0f8c135fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54598/0 (socket says 172.21.15.116:54598) 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 -- 172.21.15.116:0/1367572909 learned_addr learned my addr 172.21.15.116:0/1367572909 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): get_auth_request con 0x7f0f8c098740 auth_method 0 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 10 monclient(hunting): get_auth_request con 0x7f0f8c098b10 auth_method 0 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f8c098f30 0x7f0f8c11de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 10 monclient(hunting): get_auth_request con 0x7f0f8c098f30 auth_method 0 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f92031640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient(hunting): handle_auth_done global_id 244190 payload 824 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 -- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f8c098f30 msgr2=0x7f0f8c11de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f8c098f30 0x7f0f8c11de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 -- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f8c098b10 msgr2=0x7f0f8c11d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f8c098b10 0x7f0f8c11d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 monclient: found mon.noname-c 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f91830640 1 -- 172.21.15.116:0/1367572909 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f84009a20 con 0x7f0f8c098740 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.584+0000 7f0f9102f640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f8c098b10 0x7f0f8c11d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.586+0000 7f0f91830640 1 -- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098740 msgr2=0x7f0f8c135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.586+0000 7f0f91830640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098740 0x7f0f8c135fb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800b510 tx=0x7f0f880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.586+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88011840 con 0x7f0f8c098740 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.586+0000 7f0f81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.586+0000 7f0f81ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.586+0000 7f0f81ffb640 10 monclient: dump: 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.600 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 5 monclient: authenticate success, global_id 244190 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 10 monclient: _renew_subs 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f880119c0 con 0x7f0f8c098740 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c120ed0 con 0x7f0f8c098740 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88011d80 con 0x7f0f8c098740 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 10 monclient: dump: 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f83fff640 4 set_mon_vals no callback set 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 10 monclient: _renew_subs 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c1354f0 con 0x7f0f8c098740 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0f88010590 con 0x7f0f8c098740 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.587+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f8c05ed70 con 0x7f0f8c098740 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f81ffb640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f70038490 0x7f0f7003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f0f8804c280 con 0x7f0f8c098740 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f81ffb640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f0f70041450 0x7f0f700438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 --> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f70043f70 con 0x7f0f70041450 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=267) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f8804e7a0 con 0x7f0f8c098740 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f81ffb640 10 monclient: handle_get_version_reply finishing 1 version 267 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f92031640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f0f70041450 0x7f0f700438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f92031640 10 monclient: get_auth_request con 0x7f0f70041450 auth_method 0 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f9102f640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f70038490 0x7f0f7003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.588+0000 7f0f9102f640 10 monclient: get_auth_request con 0x7f0f70038490 auth_method 0 2024-09-18T12:45:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.591+0000 7f0f9102f640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f70038490 0x7f0f7003a940 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c11f840 tx=0x7f0f7c007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.592+0000 7f0f92031640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f0f70041450 0x7f0f700438a0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.593+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== osd.2 v2:172.21.15.116:6808/31580 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0f70043f70 con 0x7f0f70041450 2024-09-18T12:45:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.604+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 --> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0f8c121320 con 0x7f0f70041450 2024-09-18T12:45:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f81ffb640 1 -- 172.21.15.116:0/1367572909 <== osd.2 v2:172.21.15.116:6808/31580 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0f8c121320 con 0x7f0f70041450 2024-09-18T12:45:46.606 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:46.606 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:45:46.606 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f0f70041450 msgr2=0x7f0f700438a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:6808/31580,v1:172.21.15.116:6809/31580] conn(0x7f0f70041450 0x7f0f700438a0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f70038490 msgr2=0x7f0f7003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0f70038490 0x7f0f7003a940 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c11f840 tx=0x7f0f7c007e10 comp rx=0 tx=0).stop 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 10 monclient: shutdown 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098740 msgr2=0x7f0f8c135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 --2- 172.21.15.116:0/1367572909 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0f8c098740 0x7f0f8c135fb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800b510 tx=0x7f0f880090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f92031640 1 -- 172.21.15.116:0/1367572909 reap_dead start 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 shutdown_connections 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 >> 172.21.15.116:0/1367572909 conn(0x7f0f8c0a1150 msgr2=0x7f0f8c09fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 shutdown_connections 2024-09-18T12:45:46.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.605+0000 7f0f93a94640 1 -- 172.21.15.116:0/1367572909 wait complete. 2024-09-18T12:45:46.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T12:45:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.710+0000 7fd9f479d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.710+0000 7fd9f479d640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.710+0000 7fd9f479d640 10 monclient: monmap: 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:46.710834+0000 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:46.710834+0000 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:46.711 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:46.712 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:46.712 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:46.712 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.710+0000 7fd9f479d640 1 Processor -- start 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.710+0000 7fd9f479d640 1 -- start start 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.710+0000 7fd9f479d640 10 monclient: init 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9ec1318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fd9ec09ea50 0x7fd9ec09ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9ec09ea50 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9ec136c20 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec131cc0 con 0x7fd9ec1318f0 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ec09fbf0 con 0x7fd9ec136c20 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ec09fe30 con 0x7fd9ec09ea50 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f479d640 10 monclient(hunting): authenticate will time out at 1974.032428s 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47128/0 (socket says 172.21.15.116:47128) 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 -- 172.21.15.116:0/3449000485 learned_addr learned my addr 172.21.15.116:0/3449000485 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): get_auth_request con 0x7fd9ec136c20 auth_method 0 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient(hunting): handle_auth_done global_id 194733 payload 274 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f1d38640 1 --2- 172.21.15.116:0/3449000485 >> v2:172.21.15.144:3300/0 conn(0x7fd9ec09ea50 0x7fd9ec09ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 -- 172.21.15.116:0/3449000485 >> v2:172.21.15.144:3300/0 conn(0x7fd9ec09ea50 msgr2=0x7fd9ec09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 --2- 172.21.15.116:0/3449000485 >> v2:172.21.15.144:3300/0 conn(0x7fd9ec09ea50 0x7fd9ec09ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 -- 172.21.15.116:0/3449000485 >> v1:172.21.15.144:6789/0 conn(0x7fd9ec1318f0 legacy=0x7fd9ec131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 monclient: found mon.noname-a 2024-09-18T12:45:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.711+0000 7fd9f2d3a640 1 -- 172.21.15.116:0/3449000485 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec0a1210 con 0x7fd9ec136c20 2024-09-18T12:45:46.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f2d3a640 1 -- 172.21.15.116:0/3449000485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec136c20 msgr2=0x7fd9ec139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f2d3a640 1 --2- 172.21.15.116:0/3449000485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec136c20 0x7fd9ec139000 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec06f600 tx=0x7fd9e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1648960995eca22 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 1 -- 172.21.15.116:0/3449000485 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4007610 con 0x7fd9ec136c20 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: dump: 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 1 -- 172.21.15.116:0/3449000485 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e402d050 con 0x7fd9ec136c20 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f479d640 5 monclient: authenticate success, global_id 194733 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f479d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f479d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f479d640 4 set_mon_vals no callback set 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 1 -- 172.21.15.116:0/3449000485 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e40079a0 con 0x7fd9ec136c20 2024-09-18T12:45:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.714+0000 7fd9f1537640 10 monclient: dump: 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 10 monclient: shutdown 2024-09-18T12:45:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 -- 172.21.15.116:0/3449000485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec136c20 msgr2=0x7fd9ec139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 --2- 172.21.15.116:0/3449000485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec136c20 0x7fd9ec139000 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec06f600 tx=0x7fd9e4009840 comp rx=0 tx=0).stop 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 -- 172.21.15.116:0/3449000485 shutdown_connections 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 --2- 172.21.15.116:0/3449000485 >> v2:172.21.15.144:3300/0 conn(0x7fd9ec09ea50 0x7fd9ec09ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 --2- 172.21.15.116:0/3449000485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec136c20 0x7fd9ec139000 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 -- 172.21.15.116:0/3449000485 >> 172.21.15.116:0/3449000485 conn(0x7fd9ec067110 msgr2=0x7fd9ec067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 -- 172.21.15.116:0/3449000485 shutdown_connections 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 1 -- 172.21.15.116:0/3449000485 wait complete. 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.715+0000 7fd9f479d640 10 monclient: monmap: 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 Processor -- start 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 -- start start 2024-09-18T12:45:46.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient: init 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 0x7fd9ec141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9ec09ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9ec1318f0 0x7fd9ec1415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9ec1318f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9ec136c20 0x7fd9ec141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): picked mon.noname-b con 0x7fd9ec136c20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec11c730 con 0x7fd9ec09ea50 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec11b8e0 con 0x7fd9ec136c20 2024-09-18T12:45:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec09bf10 con 0x7fd9ec1318f0 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f479d640 10 monclient(hunting): authenticate will time out at 1974.037739s 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 0x7fd9ec141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9ec1318f0 0x7fd9ec1415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 0x7fd9ec141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47136/0 (socket says 172.21.15.116:47136) 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 1 -- 172.21.15.116:0/2974632150 learned_addr learned my addr 172.21.15.116:0/2974632150 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 10 monclient(hunting): get_auth_request con 0x7fd9ec09ea50 auth_method 0 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 10 monclient(hunting): get_auth_request con 0x7fd9ec1318f0 auth_method 0 2024-09-18T12:45:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f1d38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.716+0000 7fd9f2d3a640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9ec136c20 0x7fd9ec141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2d3a640 10 monclient(hunting): get_auth_request con 0x7fd9ec136c20 auth_method 0 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2d3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 10 monclient(hunting): handle_auth_done global_id 194736 payload 824 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 1 -- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9ec1318f0 msgr2=0x7fd9ec1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9ec1318f0 0x7fd9ec1415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 1 -- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9ec136c20 msgr2=0x7fd9ec141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9ec136c20 0x7fd9ec141b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 1 monclient: found mon.noname-a 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2539640 1 -- 172.21.15.116:0/2974632150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4009ae0 con 0x7fd9ec09ea50 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.717+0000 7fd9f2d3a640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9ec136c20 0x7fd9ec141b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f2539640 1 -- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 msgr2=0x7fd9ec141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:46.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f2539640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 0x7fd9ec141090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000b5e0 tx=0x7fd9e00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 1 -- 172.21.15.116:0/2974632150 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e000f7a0 con 0x7fd9ec09ea50 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: dump: 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.732 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 1 -- 172.21.15.116:0/2974632150 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e000f920 con 0x7fd9ec09ea50 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f479d640 5 monclient: authenticate success, global_id 194736 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f479d640 10 monclient: _renew_subs 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec09de90 con 0x7fd9ec09ea50 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 1 -- 172.21.15.116:0/2974632150 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e000fce0 con 0x7fd9ec09ea50 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9de7fc640 10 monclient: dump: 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f0d36640 4 set_mon_vals no callback set 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f479d640 10 monclient: _renew_subs 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.719+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec09b910 con 0x7fd9ec09ea50 2024-09-18T12:45:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9de7fc640 1 -- 172.21.15.116:0/2974632150 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd9e0010630 con 0x7fd9ec09ea50 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9de7fc640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9d0038490 0x7fd9d003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9de7fc640 1 -- 172.21.15.116:0/2974632150 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd9e004bf90 con 0x7fd9ec09ea50 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9f479d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9f479d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9f479d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ec11cba0 con 0x7fd9ec09ea50 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9f1d38640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9d0038490 0x7fd9d003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.720+0000 7fd9f1d38640 10 monclient: get_auth_request con 0x7fd9d0038490 auth_method 0 2024-09-18T12:45:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.723+0000 7fd9f1d38640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9d0038490 0x7fd9d003a940 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec09ac00 tx=0x7fd9e8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.724+0000 7fd9de7fc640 1 -- 172.21.15.116:0/2974632150 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd9e001a460 con 0x7fd9ec09ea50 2024-09-18T12:45:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.724+0000 7fd9de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.724+0000 7fd9de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.878+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9ec04bf60 con 0x7fd9d0038490 2024-09-18T12:45:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.878+0000 7fd9de7fc640 1 -- 172.21.15.116:0/2974632150 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd9ec04bf60 con 0x7fd9d0038490 2024-09-18T12:45:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T12:45:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9d0038490 msgr2=0x7fd9d003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9d0038490 0x7fd9d003a940 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec09ac00 tx=0x7fd9e8008040 comp rx=0 tx=0).stop 2024-09-18T12:45:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 10 monclient: shutdown 2024-09-18T12:45:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 msgr2=0x7fd9ec141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 0x7fd9ec141090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000b5e0 tx=0x7fd9e00090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 shutdown_connections 2024-09-18T12:45:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd9d0038490 0x7fd9d003a940 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9ec136c20 0x7fd9ec141b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd9ec1318f0 0x7fd9ec1415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 --2- 172.21.15.116:0/2974632150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9ec09ea50 0x7fd9ec141090 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 >> 172.21.15.116:0/2974632150 conn(0x7fd9ec067110 msgr2=0x7fd9ec0a3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 shutdown_connections 2024-09-18T12:45:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.880+0000 7fd9f479d640 1 -- 172.21.15.116:0/2974632150 wait complete. 2024-09-18T12:45:46.892 INFO:tasks.ceph:Scrubbing osd.3 2024-09-18T12:45:46.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.988+0000 7faa4784f640 10 monclient: get_monmap_and_config 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.988+0000 7faa4784f640 10 monclient: build_initial_monmap 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.988+0000 7faa4784f640 10 monclient: monmap: 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:46.989080+0000 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:46.989080+0000 2024-09-18T12:45:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:46.990 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.988+0000 7faa4784f640 1 Processor -- start 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 1 -- start start 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient: init 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa401319f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 0x7faa4009e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient(hunting): picked mon.noname-b con 0x7faa401324d0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faa4009ab70 0x7faa40138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient(hunting): picked mon.noname-c con 0x7faa4009ab70 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4006a2b0 con 0x7faa401319f0 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa400689b0 con 0x7faa4009ab70 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4009b1d0 con 0x7faa401324d0 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa4784f640 10 monclient(hunting): authenticate will time out at 1974.310748s 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 0x7faa4009e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 0x7faa4009e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54610/0 (socket says 172.21.15.116:54610) 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 1 -- 172.21.15.116:0/3743973086 learned_addr learned my addr 172.21.15.116:0/3743973086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 10 monclient(hunting): get_auth_request con 0x7faa401324d0 auth_method 0 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.989+0000 7faa44dea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 10 monclient(hunting): handle_auth_done global_id 244196 payload 274 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa45dec640 1 --2- 172.21.15.116:0/3743973086 >> v2:172.21.15.144:3300/0 conn(0x7faa4009ab70 0x7faa40138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 1 -- 172.21.15.116:0/3743973086 >> v2:172.21.15.144:3300/0 conn(0x7faa4009ab70 msgr2=0x7faa40138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 1 --2- 172.21.15.116:0/3743973086 >> v2:172.21.15.144:3300/0 conn(0x7faa4009ab70 0x7faa40138ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 1 -- 172.21.15.116:0/3743973086 >> v1:172.21.15.144:6789/0 conn(0x7faa401319f0 legacy=0x7faa40131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 10 monclient: _finish_hunting 0 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 1 monclient: found mon.noname-b 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.990+0000 7faa44dea640 1 -- 172.21.15.116:0/3743973086 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40069fd0 con 0x7faa401324d0 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.992+0000 7faa44dea640 1 -- 172.21.15.116:0/3743973086 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 msgr2=0x7faa4009e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.992+0000 7faa44dea640 1 --2- 172.21.15.116:0/3743973086 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 0x7faa4009e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faa34009910 tx=0x7faa340091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab5a6ffccf755171 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 1 -- 172.21.15.116:0/3743973086 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3402ae00 con 0x7faa401324d0 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: dump: 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 1 -- 172.21.15.116:0/3743973086 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3402f4b0 con 0x7faa401324d0 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 5 monclient: authenticate success, global_id 244196 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 1 -- 172.21.15.116:0/3743973086 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3402f890 con 0x7faa401324d0 2024-09-18T12:45:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 4 set_mon_vals no callback set 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa3ffff640 10 monclient: dump: 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.003 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 10 monclient: shutdown 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 -- 172.21.15.116:0/3743973086 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 msgr2=0x7faa4009e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 --2- 172.21.15.116:0/3743973086 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 0x7faa4009e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faa34009910 tx=0x7faa340091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 -- 172.21.15.116:0/3743973086 shutdown_connections 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 --2- 172.21.15.116:0/3743973086 >> v2:172.21.15.144:3300/0 conn(0x7faa4009ab70 0x7faa40138ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 --2- 172.21.15.116:0/3743973086 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401324d0 0x7faa4009e840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 -- 172.21.15.116:0/3743973086 >> 172.21.15.116:0/3743973086 conn(0x7faa400a2e40 msgr2=0x7faa400a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 -- 172.21.15.116:0/3743973086 shutdown_connections 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.993+0000 7faa4784f640 1 -- 172.21.15.116:0/3743973086 wait complete. 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.994+0000 7faa4784f640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.994+0000 7faa4784f640 10 monclient: monmap: 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.994+0000 7faa4784f640 1 Processor -- start 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.994+0000 7faa4784f640 1 -- start start 2024-09-18T12:45:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.994+0000 7faa4784f640 10 monclient: init 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4009ab70 0x7faa401329c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): picked mon.noname-a con 0x7faa4009ab70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401319f0 0x7faa40132f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): picked mon.noname-c con 0x7faa401319f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa401324d0 0x7faa40133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): picked mon.noname-b con 0x7faa401324d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40134300 con 0x7faa4009ab70 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40138800 con 0x7faa401324d0 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa40135e00 con 0x7faa401319f0 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa4784f640 10 monclient(hunting): authenticate will time out at 1974.316275s 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401319f0 0x7faa40132f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4009ab70 0x7faa401329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4009ab70 0x7faa401329c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47148/0 (socket says 172.21.15.116:47148) 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401319f0 0x7faa40132f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54620/0 (socket says 172.21.15.116:54620) 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 1 -- 172.21.15.116:0/3489455426 learned_addr learned my addr 172.21.15.116:0/3489455426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 10 monclient(hunting): get_auth_request con 0x7faa4009ab70 auth_method 0 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): get_auth_request con 0x7faa401319f0 auth_method 0 2024-09-18T12:45:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa401324d0 0x7faa40133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 10 monclient(hunting): get_auth_request con 0x7faa401324d0 auth_method 0 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa45dec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient(hunting): handle_auth_done global_id 244199 payload 824 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 -- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa401324d0 msgr2=0x7faa40133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa401324d0 0x7faa40133440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 -- 172.21.15.116:0/3489455426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4009ab70 msgr2=0x7faa401329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4009ab70 0x7faa401329c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 monclient: found mon.noname-c 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa44dea640 1 -- 172.21.15.116:0/3489455426 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa34009450 con 0x7faa401319f0 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.995+0000 7faa455eb640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4009ab70 0x7faa401329c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.997+0000 7faa44dea640 1 -- 172.21.15.116:0/3489455426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401319f0 msgr2=0x7faa40132f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.997+0000 7faa44dea640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401319f0 0x7faa40132f00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faa4009b360 tx=0x7faa34032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34032a30 con 0x7faa401319f0 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: dump: 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 5 monclient: authenticate success, global_id 244199 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa34032bb0 con 0x7faa401319f0 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 10 monclient: _renew_subs 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa40099cc0 con 0x7faa401319f0 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3403d440 con 0x7faa401319f0 2024-09-18T12:45:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 10 monclient: dump: 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.012 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 10 monclient: _renew_subs 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa40135b20 con 0x7faa401319f0 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3f7fe640 4 set_mon_vals no callback set 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faa4005ed70 con 0x7faa401319f0 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.998+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7faa34039040 con 0x7faa401319f0 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa3d7fa640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa2c038490 0x7faa2c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7faa3406c110 con 0x7faa401319f0 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa3d7fa640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7faa2c041450 0x7faa2c0438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 --> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa2c043f70 con 0x7faa2c041450 2024-09-18T12:45:47.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=267) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa3406e720 con 0x7faa401319f0 2024-09-18T12:45:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa3d7fa640 10 monclient: handle_get_version_reply finishing 1 version 267 2024-09-18T12:45:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa455eb640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa2c038490 0x7faa2c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa455eb640 10 monclient: get_auth_request con 0x7faa2c038490 auth_method 0 2024-09-18T12:45:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa45dec640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7faa2c041450 0x7faa2c0438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:46.999+0000 7faa45dec640 10 monclient: get_auth_request con 0x7faa2c041450 auth_method 0 2024-09-18T12:45:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.001+0000 7faa455eb640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa2c038490 0x7faa2c03a940 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7faa38007660 tx=0x7faa38009840 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.004+0000 7faa45dec640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7faa2c041450 0x7faa2c0438a0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.005+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== osd.3 v2:172.21.15.144:6804/32112 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7faa2c043f70 con 0x7faa2c041450 2024-09-18T12:45:47.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.016+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 --> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faa40134950 con 0x7faa2c041450 2024-09-18T12:45:47.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.016+0000 7faa3d7fa640 1 -- 172.21.15.116:0/3489455426 <== osd.3 v2:172.21.15.144:6804/32112 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7faa40134950 con 0x7faa2c041450 2024-09-18T12:45:47.017 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:47.017 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:45:47.017 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:47.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.016+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7faa2c041450 msgr2=0x7faa2c0438a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.016+0000 7faa4784f640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6804/32112,v1:172.21.15.144:6806/32112] conn(0x7faa2c041450 0x7faa2c0438a0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa2c038490 msgr2=0x7faa2c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7faa2c038490 0x7faa2c03a940 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7faa38007660 tx=0x7faa38009840 comp rx=0 tx=0).stop 2024-09-18T12:45:47.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 10 monclient: shutdown 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401319f0 msgr2=0x7faa40132f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 --2- 172.21.15.116:0/3489455426 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7faa401319f0 0x7faa40132f00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faa4009b360 tx=0x7faa34032000 comp rx=0 tx=0).stop 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa45dec640 1 -- 172.21.15.116:0/3489455426 reap_dead start 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 shutdown_connections 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 >> 172.21.15.116:0/3489455426 conn(0x7faa400a2e40 msgr2=0x7faa400a19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 shutdown_connections 2024-09-18T12:45:47.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.017+0000 7faa4784f640 1 -- 172.21.15.116:0/3489455426 wait complete. 2024-09-18T12:45:47.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-18T12:45:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.120+0000 7f57782a8640 10 monclient: get_monmap_and_config 2024-09-18T12:45:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.120+0000 7f57782a8640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.120+0000 7f57782a8640 10 monclient: monmap: 2024-09-18T12:45:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:47.120802+0000 2024-09-18T12:45:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:47.120802+0000 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.120+0000 7f57782a8640 1 Processor -- start 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.120+0000 7f57782a8640 1 -- start start 2024-09-18T12:45:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.120+0000 7f57782a8640 10 monclient: init 2024-09-18T12:45:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T12:45:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57701318f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f577009ea50 0x7f577009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient(hunting): picked mon.noname-b con 0x7f577009ea50 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5770136c20 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5770132640 con 0x7f5770136c20 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5770131d00 con 0x7f57701318f0 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577009fc60 con 0x7f577009ea50 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f57782a8640 10 monclient(hunting): authenticate will time out at 1974.442430s 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f577009ea50 0x7f577009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5776845640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5770136c20 0x7f5770139000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60840/0 (socket says 172.21.15.116:60840) 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5776845640 1 -- 172.21.15.116:0/3704042285 learned_addr learned my addr 172.21.15.116:0/3704042285 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): get_auth_request con 0x7f577009ea50 auth_method 0 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient(hunting): handle_auth_done global_id 244202 payload 274 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 1 -- 172.21.15.116:0/3704042285 >> v1:172.21.15.144:6789/0 conn(0x7f57701318f0 legacy=0x7f5770131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 1 -- 172.21.15.116:0/3704042285 >> v1:172.21.15.116:6789/0 conn(0x7f5770136c20 legacy=0x7f5770139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 1 monclient: found mon.noname-b 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.121+0000 7f5775843640 1 -- 172.21.15.116:0/3704042285 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577009fe30 con 0x7f577009ea50 2024-09-18T12:45:47.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775843640 1 -- 172.21.15.116:0/3704042285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f577009ea50 msgr2=0x7f577009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775843640 1 --2- 172.21.15.116:0/3704042285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f577009ea50 0x7f577009ee20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5760009880 tx=0x7f57600090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c6e07c705ce111 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 1 -- 172.21.15.116:0/3704042285 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576002aea0 con 0x7f577009ea50 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: dump: 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.133 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f57782a8640 5 monclient: authenticate success, global_id 244202 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 1 -- 172.21.15.116:0/3704042285 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5760032440 con 0x7f577009ea50 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f57782a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f57782a8640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 1 -- 172.21.15.116:0/3704042285 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57600327e0 con 0x7f577009ea50 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f57782a8640 4 set_mon_vals no callback set 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.124+0000 7f5775042640 10 monclient: dump: 2024-09-18T12:45:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 10 monclient: shutdown 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 1 -- 172.21.15.116:0/3704042285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f577009ea50 msgr2=0x7f577009ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 1 --2- 172.21.15.116:0/3704042285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f577009ea50 0x7f577009ee20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5760009880 tx=0x7f57600090f0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 1 -- 172.21.15.116:0/3704042285 shutdown_connections 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 1 --2- 172.21.15.116:0/3704042285 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f577009ea50 0x7f577009ee20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 1 -- 172.21.15.116:0/3704042285 >> 172.21.15.116:0/3704042285 conn(0x7f5770067110 msgr2=0x7f5770067510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 1 -- 172.21.15.116:0/3704042285 shutdown_connections 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 1 -- 172.21.15.116:0/3704042285 wait complete. 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.125+0000 7f57782a8640 10 monclient: monmap: 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:47.136 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 Processor -- start 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 -- start start 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient: init 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577009ea50 0x7f5770096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): picked mon.noname-a con 0x7f577009ea50 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57701318f0 0x7f5770096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): picked mon.noname-b con 0x7f57701318f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5770136c20 0x7f57700974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5770136c20 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5770099b10 con 0x7f577009ea50 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577009db90 con 0x7f57701318f0 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577009a0d0 con 0x7f5770136c20 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f57782a8640 10 monclient(hunting): authenticate will time out at 1974.447740s 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577009ea50 0x7f5770096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5770136c20 0x7f57700974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577009ea50 0x7f5770096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47156/0 (socket says 172.21.15.116:47156) 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 1 -- 172.21.15.116:0/197068002 learned_addr learned my addr 172.21.15.116:0/197068002 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 10 monclient(hunting): get_auth_request con 0x7f577009ea50 auth_method 0 2024-09-18T12:45:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 10 monclient(hunting): get_auth_request con 0x7f5770136c20 auth_method 0 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776044640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.126+0000 7f5776845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5775843640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57701318f0 0x7f5770096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 10 monclient(hunting): handle_auth_done global_id 244205 payload 824 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 1 -- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57701318f0 msgr2=0x7f5770096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57701318f0 0x7f5770096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 1 -- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577009ea50 msgr2=0x7f5770096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577009ea50 0x7f5770096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 1 monclient: found mon.noname-c 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776845640 1 -- 172.21.15.116:0/197068002 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5760009390 con 0x7f5770136c20 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5775843640 10 monclient: get_auth_request con 0x7f57701318f0 auth_method 0 2024-09-18T12:45:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5776044640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577009ea50 0x7f5770096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.127+0000 7f5775843640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57701318f0 0x7f5770096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:45:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f5776845640 1 -- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5770136c20 msgr2=0x7f57700974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f5776845640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5770136c20 0x7f57700974e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c460 tx=0x7f576c007ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 1 -- 172.21.15.116:0/197068002 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c005cb0 con 0x7f5770136c20 2024-09-18T12:45:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: dump: 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57782a8640 5 monclient: authenticate success, global_id 244205 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 1 -- 172.21.15.116:0/197068002 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c005e50 con 0x7f5770136c20 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57782a8640 10 monclient: _renew_subs 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 1 -- 172.21.15.116:0/197068002 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c007320 con 0x7f5770136c20 2024-09-18T12:45:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57782a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5770098b10 con 0x7f5770136c20 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57667fc640 10 monclient: dump: 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.144 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f5774841640 4 set_mon_vals no callback set 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57782a8640 10 monclient: _renew_subs 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57782a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.129+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5770097d00 con 0x7f5770136c20 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f57667fc640 1 -- 172.21.15.116:0/197068002 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f576c012400 con 0x7f5770136c20 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f57782a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f57782a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f57782a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577004bf60 con 0x7f5770136c20 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f57667fc640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5754038490 0x7f575403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f57667fc640 1 -- 172.21.15.116:0/197068002 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f576c04b630 con 0x7f5770136c20 2024-09-18T12:45:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f5776044640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5754038490 0x7f575403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.130+0000 7f5776044640 10 monclient: get_auth_request con 0x7f5754038490 auth_method 0 2024-09-18T12:45:47.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.133+0000 7f5776044640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5754038490 0x7f575403a940 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f576800f470 tx=0x7f57680026c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.134+0000 7f57667fc640 1 -- 172.21.15.116:0/197068002 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f576c007750 con 0x7f5770136c20 2024-09-18T12:45:47.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.134+0000 7f57667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:47.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.134+0000 7f57667fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.289+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5770099c50 con 0x7f5754038490 2024-09-18T12:45:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.289+0000 7f57667fc640 1 -- 172.21.15.116:0/197068002 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5770099c50 con 0x7f5754038490 2024-09-18T12:45:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 3 to deep-scrub 2024-09-18T12:45:47.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5754038490 msgr2=0x7f575403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5754038490 0x7f575403a940 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f576800f470 tx=0x7f57680026c0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 10 monclient: shutdown 2024-09-18T12:45:47.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5770136c20 msgr2=0x7f57700974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5770136c20 0x7f57700974e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c460 tx=0x7f576c007ee0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 shutdown_connections 2024-09-18T12:45:47.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f5754038490 0x7f575403a940 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f5770136c20 0x7f57700974e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57701318f0 0x7f5770096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 --2- 172.21.15.116:0/197068002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577009ea50 0x7f5770096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 >> 172.21.15.116:0/197068002 conn(0x7f5770067110 msgr2=0x7f57700a4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 shutdown_connections 2024-09-18T12:45:47.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.291+0000 7f57782a8640 1 -- 172.21.15.116:0/197068002 wait complete. 2024-09-18T12:45:47.302 INFO:tasks.ceph:Scrubbing osd.4 2024-09-18T12:45:47.303 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:45:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.398+0000 7f3c0339d640 10 monclient: get_monmap_and_config 2024-09-18T12:45:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.399+0000 7f3c0339d640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.399+0000 7f3c0339d640 10 monclient: monmap: 2024-09-18T12:45:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.425 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.426 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:47.399677+0000 2024-09-18T12:45:47.426 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:47.399677+0000 2024-09-18T12:45:47.426 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.426 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.426 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:47.426 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:47.427 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:47.427 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:47.427 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:47.427 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.399+0000 7f3c0339d640 1 Processor -- start 2024-09-18T12:45:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.399+0000 7f3c0339d640 1 -- start start 2024-09-18T12:45:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.399+0000 7f3c0339d640 10 monclient: init 2024-09-18T12:45:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T12:45:47.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bfc1319f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3bfc1324d0 0x7f3bfc09e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc1324d0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 0x7f3bfc138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc09ab70 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc06a2b0 con 0x7f3bfc1319f0 2024-09-18T12:45:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc0689b0 con 0x7f3bfc1324d0 2024-09-18T12:45:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc09b1d0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0339d640 10 monclient(hunting): authenticate will time out at 1974.721316s 2024-09-18T12:45:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 0x7f3bfc138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 0x7f3bfc138ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54644/0 (socket says 172.21.15.116:54644) 2024-09-18T12:45:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 -- 172.21.15.116:0/3336713855 learned_addr learned my addr 172.21.15.116:0/3336713855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): get_auth_request con 0x7f3bfc09ab70 auth_method 0 2024-09-18T12:45:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c00938640 1 --2- 172.21.15.116:0/3336713855 >> v2:172.21.15.144:3300/0 conn(0x7f3bfc1324d0 0x7f3bfc09e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c00938640 10 monclient(hunting): get_auth_request con 0x7f3bfc1324d0 auth_method 0 2024-09-18T12:45:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c00938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c00938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c00938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient(hunting): handle_auth_done global_id 244208 payload 274 2024-09-18T12:45:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 -- 172.21.15.116:0/3336713855 >> v2:172.21.15.144:3300/0 conn(0x7f3bfc1324d0 msgr2=0x7f3bfc09e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 --2- 172.21.15.116:0/3336713855 >> v2:172.21.15.144:3300/0 conn(0x7f3bfc1324d0 0x7f3bfc09e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 -- 172.21.15.116:0/3336713855 >> v1:172.21.15.144:6789/0 conn(0x7f3bfc1319f0 legacy=0x7f3bfc131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 monclient: found mon.noname-b 2024-09-18T12:45:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.400+0000 7f3c0193a640 1 -- 172.21.15.116:0/3336713855 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc069fd0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3c0193a640 1 -- 172.21.15.116:0/3336713855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 msgr2=0x7f3bfc138ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3c0193a640 1 --2- 172.21.15.116:0/3336713855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 0x7f3bfc138ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009f20 tx=0x7f3bf4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0758a8c27e272a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 1 -- 172.21.15.116:0/3336713855 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4007610 con 0x7f3bfc09ab70 2024-09-18T12:45:47.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:47.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: dump: 2024-09-18T12:45:47.436 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.436 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.436 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.437 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.437 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.437 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.437 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.437 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.438 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.438 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3c0339d640 5 monclient: authenticate success, global_id 244208 2024-09-18T12:45:47.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3c0339d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:47.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 1 -- 172.21.15.116:0/3336713855 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf402d050 con 0x7f3bfc09ab70 2024-09-18T12:45:47.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3c0339d640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:47.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3c0339d640 4 set_mon_vals no callback set 2024-09-18T12:45:47.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 1 -- 172.21.15.116:0/3336713855 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40079a0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.403+0000 7f3bf3fff640 10 monclient: dump: 2024-09-18T12:45:47.440 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.440 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.441 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.441 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.441 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.441 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.441 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.442 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.442 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.442 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 10 monclient: shutdown 2024-09-18T12:45:47.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 -- 172.21.15.116:0/3336713855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 msgr2=0x7f3bfc138ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 --2- 172.21.15.116:0/3336713855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 0x7f3bfc138ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009f20 tx=0x7f3bf4009840 comp rx=0 tx=0).stop 2024-09-18T12:45:47.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 -- 172.21.15.116:0/3336713855 shutdown_connections 2024-09-18T12:45:47.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 --2- 172.21.15.116:0/3336713855 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc09ab70 0x7f3bfc138ed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 --2- 172.21.15.116:0/3336713855 >> v2:172.21.15.144:3300/0 conn(0x7f3bfc1324d0 0x7f3bfc09e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 -- 172.21.15.116:0/3336713855 >> 172.21.15.116:0/3336713855 conn(0x7f3bfc0a2e40 msgr2=0x7f3bfc0a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 -- 172.21.15.116:0/3336713855 shutdown_connections 2024-09-18T12:45:47.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 1 -- 172.21.15.116:0/3336713855 wait complete. 2024-09-18T12:45:47.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.404+0000 7f3c0339d640 10 monclient: monmap: 2024-09-18T12:45:47.444 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.445 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:47.445 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:47.445 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:47.445 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.445 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.446 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:47.446 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:47.446 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:47.446 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 Processor -- start 2024-09-18T12:45:47.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 -- start start 2024-09-18T12:45:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient: init 2024-09-18T12:45:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3bfc1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc09ab70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 0x7f3bfc141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc1319f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 0x7f3bfc1329c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc1324d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc099cc0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0a23e0 con 0x7f3bfc1324d0 2024-09-18T12:45:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc134b20 con 0x7f3bfc1319f0 2024-09-18T12:45:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c0339d640 10 monclient(hunting): authenticate will time out at 1974.726786s 2024-09-18T12:45:47.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 0x7f3bfc141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c01139640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3bfc1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 0x7f3bfc141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54650/0 (socket says 172.21.15.116:54650) 2024-09-18T12:45:47.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c01139640 10 monclient(hunting): get_auth_request con 0x7f3bfc09ab70 auth_method 0 2024-09-18T12:45:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 learned_addr learned my addr 172.21.15.116:0/457608194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c01139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c01139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c01139640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 10 monclient(hunting): get_auth_request con 0x7f3bfc1319f0 auth_method 0 2024-09-18T12:45:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.405+0000 7f3c00938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c01139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c01139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c01139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 0x7f3bfc1329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 10 monclient(hunting): get_auth_request con 0x7f3bfc1324d0 auth_method 0 2024-09-18T12:45:47.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c0193a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 10 monclient(hunting): handle_auth_done global_id 244211 payload 824 2024-09-18T12:45:47.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 msgr2=0x7f3bfc1329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 0x7f3bfc1329c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 msgr2=0x7f3bfc1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3bfc1418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 1 monclient: found mon.noname-c 2024-09-18T12:45:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4009ae0 con 0x7f3bfc1319f0 2024-09-18T12:45:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.406+0000 7f3c01139640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3bfc1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 msgr2=0x7f3bfc141e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 0x7f3bfc141e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00d8c0 tx=0x7f3bec008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00de50 con 0x7f3bfc1319f0 2024-09-18T12:45:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 msgr2=0x7f3bfc141e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:45:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 msgr2=0x7f3bfc141e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 0x7f3bfc141e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00d8c0 tx=0x7f3bec008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 0x7f3bfc141e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00d8c0 tx=0x7f3bec008040 comp rx=0 tx=0).stop 2024-09-18T12:45:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient: dump: 2024-09-18T12:45:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.460 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 msgr2=0x7f3bfc141e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3bfc1319f0 0x7f3bfc141e30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3be00036a0 0x7f3be0005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient(hunting): picked mon.c con 0x7f3be00036a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3be00085a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient(hunting): picked mon.a con 0x7f3bfc09ab70 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 0x7f3be000af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient(hunting): picked mon.b con 0x7f3bfc1324d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be000b910 con 0x7f3bfc09ab70 2024-09-18T12:45:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be000ba50 con 0x7f3bfc1324d0 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be000bb90 con 0x7f3be00036a0 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3bf17fa640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3be00036a0 0x7f3be0005ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 10 monclient(hunting): get_auth_request con 0x7f3be00036a0 auth_method 0 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3be00085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): get_auth_request con 0x7f3bfc09ab70 auth_method 0 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c0193a640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 0x7f3be000af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c0193a640 10 monclient(hunting): get_auth_request con 0x7f3bfc1324d0 auth_method 0 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c0193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c0193a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c0193a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T12:45:47.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient(hunting): handle_auth_done global_id 244211 payload 293 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3be00036a0 msgr2=0x7f3be0005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3be00036a0 0x7f3be0005ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 msgr2=0x7f3be000af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3bfc1324d0 0x7f3be000af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 monclient: found mon.a 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f3be000bdc0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c00938640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c01139640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3be00036a0 0x7f3be0005ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:47.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.408+0000 7f3c0339d640 5 monclient: authenticate success, global_id 244211 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 10 monclient: _renew_subs 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc1344c0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 10 monclient: _renew_subs 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc136bc0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.409+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3bfc135b60 con 0x7f3bfc09ab70 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3c00938640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 msgr2=0x7f3be00085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=2) v1 -- 0x7f3be000c010 con 0x7f3bfc09ab70 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3c00938640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3be00085a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3bec018000 tx=0x7f3bec01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec029450 con 0x7f3bfc09ab70 2024-09-18T12:45:47.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 10 monclient: dump: 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be000c010 con 0x7f3bfc09ab70 2024-09-18T12:45:47.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.411+0000 7f3bf37fe640 4 set_mon_vals no callback set 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3bec0295d0 con 0x7f3bfc09ab70 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3bfc1319f0 0x7f3be0042bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== mon.0 v2:172.21.15.116:3300/0 4 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3bec04f190 con 0x7f3bfc09ab70 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f3be0049560 0x7f3be004b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3be004c080 con 0x7f3be0049560 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== mon.0 v2:172.21.15.116:3300/0 5 ==== mon_get_version_reply(handle=1 version=267) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bec04f430 con 0x7f3bfc09ab70 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 10 monclient: handle_get_version_reply finishing 1 version 267 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=2 version=267) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bec04f650 con 0x7f3bfc09ab70 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3bf17fa640 10 monclient: handle_get_version_reply finishing 2 version 267 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3c01139640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3bfc1319f0 0x7f3be0042bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3c01139640 10 monclient: get_auth_request con 0x7f3bfc1319f0 auth_method 0 2024-09-18T12:45:47.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3c0193a640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f3be0049560 0x7f3be004b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.412+0000 7f3c0193a640 10 monclient: get_auth_request con 0x7f3be0049560 auth_method 0 2024-09-18T12:45:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.415+0000 7f3c01139640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3bfc1319f0 0x7f3be0042bf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8006f20 tx=0x7f3bf8006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.417+0000 7f3c0193a640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f3be0049560 0x7f3be004b9b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.418+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== osd.4 v2:172.21.15.144:6800/32113 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3be004c080 con 0x7f3be0049560 2024-09-18T12:45:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.429+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 --> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3bfc05ed70 con 0x7f3be0049560 2024-09-18T12:45:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.429+0000 7f3bf17fa640 1 -- 172.21.15.116:0/457608194 <== osd.4 v2:172.21.15.144:6800/32113 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3bfc05ed70 con 0x7f3be0049560 2024-09-18T12:45:47.470 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:47.470 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:45:47.470 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f3be0049560 msgr2=0x7f3be004b9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6800/32113,v1:172.21.15.144:6801/32113] conn(0x7f3be0049560 0x7f3be004b9b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3bfc1319f0 msgr2=0x7f3be0042bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3bfc1319f0 0x7f3be0042bf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8006f20 tx=0x7f3bf8006f50 comp rx=0 tx=0).stop 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 10 monclient: shutdown 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 msgr2=0x7f3be00085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 --2- 172.21.15.116:0/457608194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bfc09ab70 0x7f3be00085a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3bec018000 tx=0x7f3bec01b000 comp rx=0 tx=0).stop 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0193a640 1 -- 172.21.15.116:0/457608194 reap_dead start 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 shutdown_connections 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 >> 172.21.15.116:0/457608194 conn(0x7f3bfc0a2e40 msgr2=0x7f3bfc138c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 shutdown_connections 2024-09-18T12:45:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.430+0000 7f3c0339d640 1 -- 172.21.15.116:0/457608194 wait complete. 2024-09-18T12:45:47.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.565+0000 7fb961846640 10 monclient: get_monmap_and_config 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.565+0000 7fb961846640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.565+0000 7fb961846640 10 monclient: monmap: 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:47.565951+0000 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:47.565951+0000 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.566 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.567 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:47.567 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:47.567 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:47.567 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:47.567 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:47.567 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.565+0000 7fb961846640 1 Processor -- start 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.565+0000 7fb961846640 1 -- start start 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.565+0000 7fb961846640 10 monclient: init 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c131ab0 0x7fb95c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient(hunting): picked mon.noname-b con 0x7fb95c131ab0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb95c09e910 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:47.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb95c136bb0 0x7fb95c138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c136bb0 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c09faf0 con 0x7fb95c09e910 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95c09fd30 con 0x7fb95c136bb0 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c0a1110 con 0x7fb95c131ab0 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb961846640 10 monclient(hunting): authenticate will time out at 1974.887453s 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c131ab0 0x7fb95c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95a7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb95c09e910 0x7fb95c09ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60852/0 (socket says 172.21.15.116:60852) 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95a7fc640 1 -- 172.21.15.116:0/4024839548 learned_addr learned my addr 172.21.15.116:0/4024839548 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:47.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): get_auth_request con 0x7fb95c131ab0 auth_method 0 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95b7fe640 1 --2- 172.21.15.116:0/4024839548 >> v2:172.21.15.144:3300/0 conn(0x7fb95c136bb0 0x7fb95c138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request con 0x7fb95c136bb0 auth_method 0 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient(hunting): handle_auth_done global_id 244214 payload 274 2024-09-18T12:45:47.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 1 -- 172.21.15.116:0/4024839548 >> v2:172.21.15.144:3300/0 conn(0x7fb95c136bb0 msgr2=0x7fb95c138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 1 --2- 172.21.15.116:0/4024839548 >> v2:172.21.15.144:3300/0 conn(0x7fb95c136bb0 0x7fb95c138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 1 -- 172.21.15.116:0/4024839548 >> v1:172.21.15.116:6789/0 conn(0x7fb95c09e910 legacy=0x7fb95c09ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 1 monclient: found mon.noname-b 2024-09-18T12:45:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.566+0000 7fb95affd640 1 -- 172.21.15.116:0/4024839548 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c0a13f0 con 0x7fb95c131ab0 2024-09-18T12:45:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb95affd640 1 -- 172.21.15.116:0/4024839548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c131ab0 msgr2=0x7fb95c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb95affd640 1 --2- 172.21.15.116:0/4024839548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c131ab0 0x7fb95c131e80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009e60 tx=0x7fb94c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfbe3eea0733a548 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 1 -- 172.21.15.116:0/4024839548 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c007760 con 0x7fb95c131ab0 2024-09-18T12:45:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:47.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: dump: 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 1 -- 172.21.15.116:0/4024839548 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c02d050 con 0x7fb95c131ab0 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb961846640 5 monclient: authenticate success, global_id 244214 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb961846640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb961846640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 1 -- 172.21.15.116:0/4024839548 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c007b30 con 0x7fb95c131ab0 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb961846640 4 set_mon_vals no callback set 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.569+0000 7fb959ffb640 10 monclient: dump: 2024-09-18T12:45:47.580 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 10 monclient: shutdown 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 -- 172.21.15.116:0/4024839548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c131ab0 msgr2=0x7fb95c131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 --2- 172.21.15.116:0/4024839548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c131ab0 0x7fb95c131e80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009e60 tx=0x7fb94c009780 comp rx=0 tx=0).stop 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 -- 172.21.15.116:0/4024839548 shutdown_connections 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 --2- 172.21.15.116:0/4024839548 >> v2:172.21.15.144:3300/0 conn(0x7fb95c136bb0 0x7fb95c138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 --2- 172.21.15.116:0/4024839548 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c131ab0 0x7fb95c131e80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 -- 172.21.15.116:0/4024839548 >> 172.21.15.116:0/4024839548 conn(0x7fb95c067d30 msgr2=0x7fb95c0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 -- 172.21.15.116:0/4024839548 shutdown_connections 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.570+0000 7fb961846640 1 -- 172.21.15.116:0/4024839548 wait complete. 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient: monmap: 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.582 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 Processor -- start 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 -- start start 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient: init 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 0x7fb95c12f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c09e910 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb95c131ab0 0x7fb95c12f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): picked mon.noname-b con 0x7fb95c131ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c136bb0 0x7fb95c12fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c136bb0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c1337f0 con 0x7fb95c09e910 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c09bde0 con 0x7fb95c131ab0 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c139140 con 0x7fb95c136bb0 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.571+0000 7fb961846640 10 monclient(hunting): authenticate will time out at 1974.892963s 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 0x7fb95c12f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 0x7fb95c12f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47196/0 (socket says 172.21.15.116:47196) 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 -- 172.21.15.116:0/344917336 learned_addr learned my addr 172.21.15.116:0/344917336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): get_auth_request con 0x7fb95c09e910 auth_method 0 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c136bb0 0x7fb95c12fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request con 0x7fb95c136bb0 auth_method 0 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb95c131ab0 0x7fb95c12f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 10 monclient(hunting): get_auth_request con 0x7fb95c131ab0 auth_method 0 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient(hunting): handle_auth_done global_id 194748 payload 824 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 -- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c136bb0 msgr2=0x7fb95c12fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c136bb0 0x7fb95c12fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 -- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb95c131ab0 msgr2=0x7fb95c12f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb95c131ab0 0x7fb95c12f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 monclient: found mon.noname-a 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95affd640 1 -- 172.21.15.116:0/344917336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c009a20 con 0x7fb95c09e910 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.572+0000 7fb95b7fe640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c136bb0 0x7fb95c12fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb95affd640 1 -- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 msgr2=0x7fb95c12f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb95affd640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 0x7fb95c12f240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009e60 tx=0x7fb94c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 1 -- 172.21.15.116:0/344917336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c031ac0 con 0x7fb95c09e910 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: dump: 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 1 -- 172.21.15.116:0/344917336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c040070 con 0x7fb95c09e910 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb961846640 5 monclient: authenticate success, global_id 194748 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb961846640 10 monclient: _renew_subs 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 1 -- 172.21.15.116:0/344917336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c031e30 con 0x7fb95c09e910 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb961846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c133510 con 0x7fb95c09e910 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb942ffd640 10 monclient: dump: 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.589 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb9597fa640 4 set_mon_vals no callback set 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb961846640 10 monclient: _renew_subs 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb961846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.574+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c130d60 con 0x7fb95c09e910 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.575+0000 7fb942ffd640 1 -- 172.21.15.116:0/344917336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fb94c003020 con 0x7fb95c09e910 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.575+0000 7fb961846640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.575+0000 7fb961846640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.575+0000 7fb961846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.575+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c04bf60 con 0x7fb95c09e910 2024-09-18T12:45:47.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.575+0000 7fb942ffd640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9480384c0 0x7fb94803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.576+0000 7fb942ffd640 1 -- 172.21.15.116:0/344917336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fb94c06ded0 con 0x7fb95c09e910 2024-09-18T12:45:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.576+0000 7fb95a7fc640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9480384c0 0x7fb94803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.576+0000 7fb95a7fc640 10 monclient: get_auth_request con 0x7fb9480384c0 auth_method 0 2024-09-18T12:45:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.578+0000 7fb95a7fc640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9480384c0 0x7fb94803a970 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb950007fd0 tx=0x7fb950007f00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.580+0000 7fb942ffd640 1 -- 172.21.15.116:0/344917336 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fb94c03ab80 con 0x7fb95c09e910 2024-09-18T12:45:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.580+0000 7fb942ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:47.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.580+0000 7fb942ffd640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:47.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.736+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb95c133960 con 0x7fb9480384c0 2024-09-18T12:45:47.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.737+0000 7fb942ffd640 1 -- 172.21.15.116:0/344917336 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb95c133960 con 0x7fb9480384c0 2024-09-18T12:45:47.738 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 4 to deep-scrub 2024-09-18T12:45:47.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9480384c0 msgr2=0x7fb94803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9480384c0 0x7fb94803a970 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb950007fd0 tx=0x7fb950007f00 comp rx=0 tx=0).stop 2024-09-18T12:45:47.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 10 monclient: shutdown 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 msgr2=0x7fb95c12f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 0x7fb95c12f240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009e60 tx=0x7fb94c037000 comp rx=0 tx=0).stop 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 shutdown_connections 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fb9480384c0 0x7fb94803a970 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fb95c136bb0 0x7fb95c12fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb95c131ab0 0x7fb95c12f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 --2- 172.21.15.116:0/344917336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c09e910 0x7fb95c12f240 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 >> 172.21.15.116:0/344917336 conn(0x7fb95c067d30 msgr2=0x7fb95c0a3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 shutdown_connections 2024-09-18T12:45:47.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.739+0000 7fb961846640 1 -- 172.21.15.116:0/344917336 wait complete. 2024-09-18T12:45:47.750 INFO:tasks.ceph:Scrubbing osd.5 2024-09-18T12:45:47.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:45:47.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.846+0000 7fe52fec9640 10 monclient: get_monmap_and_config 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.846+0000 7fe52fec9640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.846+0000 7fe52fec9640 10 monclient: monmap: 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:47.846900+0000 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:47.846900+0000 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:47.848 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:47.848 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:47.848 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.846+0000 7fe52fec9640 1 Processor -- start 2024-09-18T12:45:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.846+0000 7fe52fec9640 1 -- start start 2024-09-18T12:45:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.846+0000 7fe52fec9640 10 monclient: init 2024-09-18T12:45:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T12:45:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5281319f0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5281324d0 0x7fe52809e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient(hunting): picked mon.noname-a con 0x7fe5281324d0 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe52809ab70 0x7fe528138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient(hunting): picked mon.noname-c con 0x7fe52809ab70 addr v2:172.21.15.144:3300/0 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52806a2b0 con 0x7fe5281319f0 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5280689b0 con 0x7fe5281324d0 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52809b1d0 con 0x7fe52809ab70 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52fec9640 10 monclient(hunting): authenticate will time out at 1975.168478s 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5281324d0 0x7fe52809e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5281324d0 0x7fe52809e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47206/0 (socket says 172.21.15.116:47206) 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 -- 172.21.15.116:0/756769712 learned_addr learned my addr 172.21.15.116:0/756769712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): get_auth_request con 0x7fe5281324d0 auth_method 0 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52e466640 1 --2- 172.21.15.116:0/756769712 >> v2:172.21.15.144:3300/0 conn(0x7fe52809ab70 0x7fe528138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52e466640 10 monclient(hunting): get_auth_request con 0x7fe52809ab70 auth_method 0 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52e466640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52e466640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient(hunting): handle_auth_done global_id 194751 payload 274 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 -- 172.21.15.116:0/756769712 >> v2:172.21.15.144:3300/0 conn(0x7fe52809ab70 msgr2=0x7fe528138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 --2- 172.21.15.116:0/756769712 >> v2:172.21.15.144:3300/0 conn(0x7fe52809ab70 0x7fe528138ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 -- 172.21.15.116:0/756769712 >> v1:172.21.15.144:6789/0 conn(0x7fe5281319f0 legacy=0x7fe528131dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 monclient: found mon.noname-a 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.847+0000 7fe52d464640 1 -- 172.21.15.116:0/756769712 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528069fd0 con 0x7fe5281324d0 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52d464640 1 -- 172.21.15.116:0/756769712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 msgr2=0x7fe52809e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52d464640 1 --2- 172.21.15.116:0/756769712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe52809e840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe52809ed80 tx=0x7fe5180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3d0f0f125d63c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 1 -- 172.21.15.116:0/756769712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518032440 con 0x7fe5281324d0 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: dump: 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 1 -- 172.21.15.116:0/756769712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5180325a0 con 0x7fe5281324d0 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52fec9640 5 monclient: authenticate success, global_id 194751 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52fec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52fec9640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52fec9640 4 set_mon_vals no callback set 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 1 -- 172.21.15.116:0/756769712 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518032940 con 0x7fe5281324d0 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.850+0000 7fe52cc63640 10 monclient: dump: 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.861 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 10 monclient: shutdown 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 -- 172.21.15.116:0/756769712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 msgr2=0x7fe52809e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 --2- 172.21.15.116:0/756769712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe52809e840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe52809ed80 tx=0x7fe5180091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 -- 172.21.15.116:0/756769712 shutdown_connections 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 --2- 172.21.15.116:0/756769712 >> v2:172.21.15.144:3300/0 conn(0x7fe52809ab70 0x7fe528138ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 --2- 172.21.15.116:0/756769712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe52809e840 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 -- 172.21.15.116:0/756769712 >> 172.21.15.116:0/756769712 conn(0x7fe5280a2e40 msgr2=0x7fe5280a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 -- 172.21.15.116:0/756769712 shutdown_connections 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.851+0000 7fe52fec9640 1 -- 172.21.15.116:0/756769712 wait complete. 2024-09-18T12:45:47.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient: monmap: 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 Processor -- start 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 -- start start 2024-09-18T12:45:47.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient: init 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52809ab70 0x7fe528141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): picked mon.noname-b con 0x7fe52809ab70 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5281319f0 0x7fe528141c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): picked mon.noname-c con 0x7fe5281319f0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe528142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): picked mon.noname-a con 0x7fe5281324d0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281346b0 con 0x7fe5281324d0 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528136380 con 0x7fe52809ab70 2024-09-18T12:45:47.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52809c5a0 con 0x7fe5281319f0 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52fec9640 10 monclient(hunting): authenticate will time out at 1975.173880s 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52e466640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe528142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.852+0000 7fe52d464640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5281319f0 0x7fe528141c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe528142180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47220/0 (socket says 172.21.15.116:47220) 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 1 -- 172.21.15.116:0/1662077715 learned_addr learned my addr 172.21.15.116:0/1662077715 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): get_auth_request con 0x7fe5281319f0 auth_method 0 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 10 monclient(hunting): get_auth_request con 0x7fe5281324d0 auth_method 0 2024-09-18T12:45:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52809ab70 0x7fe528141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 10 monclient(hunting): get_auth_request con 0x7fe52809ab70 auth_method 0 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52dc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient(hunting): handle_auth_done global_id 244220 payload 824 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 1 -- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52809ab70 msgr2=0x7fe528141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe52809ab70 0x7fe528141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 1 -- 172.21.15.116:0/1662077715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 msgr2=0x7fe528142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe528142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 1 monclient: found mon.noname-c 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52d464640 1 -- 172.21.15.116:0/1662077715 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe518009450 con 0x7fe5281319f0 2024-09-18T12:45:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.853+0000 7fe52e466640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5281324d0 0x7fe528142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52d464640 1 -- 172.21.15.116:0/1662077715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5281319f0 msgr2=0x7fe528141c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52d464640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5281319f0 0x7fe528141c40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe518031000 tx=0x7fe518032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518031a40 con 0x7fe5281319f0 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: dump: 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52fec9640 5 monclient: authenticate success, global_id 244220 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe518031be0 con 0x7fe5281319f0 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52fec9640 10 monclient: _renew_subs 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52fec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe528099c40 con 0x7fe5281319f0 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51803d440 con 0x7fe5281319f0 2024-09-18T12:45:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51dffb640 10 monclient: dump: 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52fec9640 10 monclient: _renew_subs 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52fec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe528135d00 con 0x7fe5281319f0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.855+0000 7fe51ffff640 4 set_mon_vals no callback set 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.856+0000 7fe52fec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.856+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe528134b80 con 0x7fe5281319f0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.856+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fe518039040 con 0x7fe5281319f0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe51dffb640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe50c0384c0 0x7fe50c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fe51806baf0 con 0x7fe5281319f0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe51dffb640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fe50c041480 0x7fe50c0438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe50c043fa0 con 0x7fe50c041480 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_get_version_reply(handle=1 version=267) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5180702a0 con 0x7fe5281319f0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe51dffb640 10 monclient: handle_get_version_reply finishing 1 version 267 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe52dc65640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe50c0384c0 0x7fe50c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe52dc65640 10 monclient: get_auth_request con 0x7fe50c0384c0 auth_method 0 2024-09-18T12:45:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe52e466640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fe50c041480 0x7fe50c0438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.857+0000 7fe52e466640 10 monclient: get_auth_request con 0x7fe50c041480 auth_method 0 2024-09-18T12:45:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.860+0000 7fe52dc65640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe50c0384c0 0x7fe50c03a970 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe524007fd0 tx=0x7fe524007e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.862+0000 7fe52e466640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fe50c041480 0x7fe50c0438d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.863+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== osd.5 v2:172.21.15.144:6812/32114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fe50c043fa0 con 0x7fe50c041480 2024-09-18T12:45:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.874+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 --> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe52805ed70 con 0x7fe50c041480 2024-09-18T12:45:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe51dffb640 1 -- 172.21.15.116:0/1662077715 <== osd.5 v2:172.21.15.144:6812/32114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe52805ed70 con 0x7fe50c041480 2024-09-18T12:45:47.875 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-18T12:45:47.876 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:45:47.876 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-18T12:45:47.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fe50c041480 msgr2=0x7fe50c0438d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6812/32114,v1:172.21.15.144:6814/32114] conn(0x7fe50c041480 0x7fe50c0438d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe50c0384c0 msgr2=0x7fe50c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fe50c0384c0 0x7fe50c03a970 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe524007fd0 tx=0x7fe524007e10 comp rx=0 tx=0).stop 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 10 monclient: shutdown 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5281319f0 msgr2=0x7fe528141c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 --2- 172.21.15.116:0/1662077715 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe5281319f0 0x7fe528141c40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe518031000 tx=0x7fe518032000 comp rx=0 tx=0).stop 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52e466640 1 -- 172.21.15.116:0/1662077715 reap_dead start 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 shutdown_connections 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 >> 172.21.15.116:0/1662077715 conn(0x7fe5280a2e40 msgr2=0x7fe5280a1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 shutdown_connections 2024-09-18T12:45:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.875+0000 7fe52fec9640 1 -- 172.21.15.116:0/1662077715 wait complete. 2024-09-18T12:45:47.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.979+0000 7f3524e34640 10 monclient: get_monmap_and_config 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.979+0000 7f3524e34640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.979+0000 7f3524e34640 10 monclient: monmap: 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:47.979870+0000 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:47.979870+0000 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.980 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:47.981 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:47.981 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:47.981 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:47.981 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:47.981 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.979+0000 7f3524e34640 1 Processor -- start 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.979+0000 7f3524e34640 1 -- start start 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.979+0000 7f3524e34640 10 monclient: init 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352009c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient(hunting): picked mon.noname-b con 0x7f35200a1ef0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f352009abd0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3520136d00 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352006f010 con 0x7f352009abd0 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352009dcc0 con 0x7f3520136d00 2024-09-18T12:45:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352009d9e0 con 0x7f35200a1ef0 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f3524e34640 10 monclient(hunting): authenticate will time out at 1975.301460s 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351dd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f352009abd0 0x7f352009afa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60856/0 (socket says 172.21.15.116:60856) 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351dd9b640 1 -- 172.21.15.116:0/1797489803 learned_addr learned my addr 172.21.15.116:0/1797489803 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): get_auth_request con 0x7f35200a1ef0 auth_method 0 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient(hunting): handle_auth_done global_id 244223 payload 274 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 1 -- 172.21.15.116:0/1797489803 >> v1:172.21.15.144:6789/0 conn(0x7f3520136d00 legacy=0x7f35201390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 1 -- 172.21.15.116:0/1797489803 >> v1:172.21.15.116:6789/0 conn(0x7f352009abd0 legacy=0x7f352009afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 1 monclient: found mon.noname-b 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:45:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.980+0000 7f351e59c640 1 -- 172.21.15.116:0/1797489803 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35200a2310 con 0x7f35200a1ef0 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.983+0000 7f351e59c640 1 -- 172.21.15.116:0/1797489803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 msgr2=0x7f352009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.983+0000 7f351e59c640 1 --2- 172.21.15.116:0/1797489803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352009c6a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3518009e90 tx=0x7f3518009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12d557e11da3df02 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 1 -- 172.21.15.116:0/1797489803 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518007630 con 0x7f35200a1ef0 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: dump: 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: _finish_auth 0 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 5 monclient: authenticate success, global_id 244223 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 1 -- 172.21.15.116:0/1797489803 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351802d050 con 0x7f35200a1ef0 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:47.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 1 -- 172.21.15.116:0/1797489803 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35180079c0 con 0x7f35200a1ef0 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 4 set_mon_vals no callback set 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f351d59a640 10 monclient: dump: 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:47.993 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 10 monclient: shutdown 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 1 -- 172.21.15.116:0/1797489803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 msgr2=0x7f352009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 1 --2- 172.21.15.116:0/1797489803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352009c6a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3518009e90 tx=0x7f3518009780 comp rx=0 tx=0).stop 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 1 -- 172.21.15.116:0/1797489803 shutdown_connections 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 1 --2- 172.21.15.116:0/1797489803 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352009c6a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 1 -- 172.21.15.116:0/1797489803 >> 172.21.15.116:0/1797489803 conn(0x7f35200678a0 msgr2=0x7f35200a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 1 -- 172.21.15.116:0/1797489803 shutdown_connections 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.984+0000 7f3524e34640 1 -- 172.21.15.116:0/1797489803 wait complete. 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient: build_initial_monmap 2024-09-18T12:45:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient: monmap: 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 Processor -- start 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 -- start start 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient: init 2024-09-18T12:45:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 0x7f3520141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): picked mon.noname-a con 0x7f352009abd0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352012f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): picked mon.noname-c con 0x7f35200a1ef0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3520136d00 0x7f3520134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): picked mon.noname-b con 0x7f3520136d00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3520126ec0 con 0x7f352009abd0 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352012fc00 con 0x7f3520136d00 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35201284d0 con 0x7f35200a1ef0 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.985+0000 7f3524e34640 10 monclient(hunting): authenticate will time out at 1975.306995s 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 0x7f3520141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352012f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 0x7f3520141570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47236/0 (socket says 172.21.15.116:47236) 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 -- 172.21.15.116:0/963929289 learned_addr learned my addr 172.21.15.116:0/963929289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 10 monclient(hunting): get_auth_request con 0x7f35200a1ef0 auth_method 0 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): get_auth_request con 0x7f352009abd0 auth_method 0 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351ed9d640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3520136d00 0x7f3520134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351ed9d640 10 monclient(hunting): get_auth_request con 0x7f3520136d00 auth_method 0 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient(hunting): handle_auth_done global_id 194757 payload 824 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 -- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 msgr2=0x7f352012f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352012f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 -- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3520136d00 msgr2=0x7f3520134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3520136d00 0x7f3520134b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient: _finish_hunting 0 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 monclient: found mon.noname-a 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351e59c640 1 -- 172.21.15.116:0/963929289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3518009a20 con 0x7f352009abd0 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351dd9b640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352012f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.986+0000 7f351ed9d640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3520136d00 0x7f3520134b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f351e59c640 1 -- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 msgr2=0x7f3520141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f351e59c640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 0x7f3520141570 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3518009de0 tx=0x7f3518037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 1 -- 172.21.15.116:0/963929289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518031890 con 0x7f352009abd0 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: dump: 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: _finish_auth 0 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 1 -- 172.21.15.116:0/963929289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3518040070 con 0x7f352009abd0 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f3524e34640 5 monclient: authenticate success, global_id 194757 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f3524e34640 10 monclient: _renew_subs 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f3524e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3520142450 con 0x7f352009abd0 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 1 -- 172.21.15.116:0/963929289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518031c60 con 0x7f352009abd0 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f35167fc640 10 monclient: dump: 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f351cd99640 4 set_mon_vals no callback set 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f3524e34640 10 monclient: _renew_subs 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f3524e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.988+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35201266c0 con 0x7f352009abd0 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.989+0000 7f35167fc640 1 -- 172.21.15.116:0/963929289 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3518042030 con 0x7f352009abd0 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.989+0000 7f3524e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.989+0000 7f3524e34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.989+0000 7f3524e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.989+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352009b940 con 0x7f352009abd0 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.989+0000 7f35167fc640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3500038490 0x7f350003a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.989+0000 7f35167fc640 1 -- 172.21.15.116:0/963929289 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f351806d790 con 0x7f352009abd0 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.990+0000 7f351dd9b640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3500038490 0x7f350003a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.990+0000 7f351dd9b640 10 monclient: get_auth_request con 0x7f3500038490 auth_method 0 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.992+0000 7f351dd9b640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3500038490 0x7f350003a940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f35201303e0 tx=0x7f3508006f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.994+0000 7f35167fc640 1 -- 172.21.15.116:0/963929289 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f351804e9a0 con 0x7f352009abd0 2024-09-18T12:45:48.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.994+0000 7f35167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:48.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:47.994+0000 7f35167fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.152+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f35200723d0 con 0x7f3500038490 2024-09-18T12:45:48.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.152+0000 7f35167fc640 1 -- 172.21.15.116:0/963929289 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f35200723d0 con 0x7f3500038490 2024-09-18T12:45:48.153 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 5 to deep-scrub 2024-09-18T12:45:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3500038490 msgr2=0x7f350003a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3500038490 0x7f350003a940 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f35201303e0 tx=0x7f3508006f50 comp rx=0 tx=0).stop 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 10 monclient: shutdown 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 msgr2=0x7f3520141570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 0x7f3520141570 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3518009de0 tx=0x7f3518037000 comp rx=0 tx=0).stop 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 shutdown_connections 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3500038490 0x7f350003a940 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3520136d00 0x7f3520134b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f35200a1ef0 0x7f352012f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 --2- 172.21.15.116:0/963929289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f352009abd0 0x7f3520141570 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 >> 172.21.15.116:0/963929289 conn(0x7f35200678a0 msgr2=0x7f3520095430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 shutdown_connections 2024-09-18T12:45:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.154+0000 7f3524e34640 1 -- 172.21.15.116:0/963929289 wait complete. 2024-09-18T12:45:48.166 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:45:48.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.256+0000 7fd29bb00640 10 monclient: get_monmap_and_config 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.256+0000 7fd29bb00640 10 monclient: build_initial_monmap 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.256+0000 7fd29bb00640 10 monclient: monmap: 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:45:48.257440+0000 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:45:48.257440+0000 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:45:48.258 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:45:48.259 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:45:48.259 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:45:48.259 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 1 Processor -- start 2024-09-18T12:45:48.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 1 -- start start 2024-09-18T12:45:48.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 10 monclient: init 2024-09-18T12:45:48.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:48.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T12:45:48.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd29406e6f0 0x7fd29412fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:48.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 10 monclient(hunting): picked mon.noname-a con 0x7fd29406e6f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:45:48.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd29406eac0 addr v1:172.21.15.116:6789/0 2024-09-18T12:45:48.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.257+0000 7fd29bb00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd294134aa0 addr v1:172.21.15.144:6789/0 2024-09-18T12:45:48.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29bb00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29409c610 con 0x7fd29406eac0 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29bb00640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29409b090 con 0x7fd294134aa0 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29bb00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29bb00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd294130180 con 0x7fd29406e6f0 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29bb00640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29bb00640 10 monclient(hunting): authenticate will time out at 1975.579033s 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd29406e6f0 0x7fd29412fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd29406e6f0 0x7fd29412fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47238/0 (socket says 172.21.15.116:47238) 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 1 -- 172.21.15.116:0/678405949 learned_addr learned my addr 172.21.15.116:0/678405949 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29909b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd29406eac0 0x7fd29406eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60870/0 (socket says 172.21.15.116:60870) 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): get_auth_request con 0x7fd29406e6f0 auth_method 0 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:48.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient(hunting): handle_auth_done global_id 194760 payload 274 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 1 -- 172.21.15.116:0/678405949 >> v1:172.21.15.144:6789/0 conn(0x7fd294134aa0 legacy=0x7fd294136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 1 -- 172.21.15.116:0/678405949 >> v1:172.21.15.116:6789/0 conn(0x7fd29406eac0 legacy=0x7fd29406eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient: _finish_hunting 0 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 1 monclient: found mon.noname-a 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.258+0000 7fd29989c640 1 -- 172.21.15.116:0/678405949 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29409c890 con 0x7fd29406e6f0 2024-09-18T12:45:48.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29989c640 1 -- 172.21.15.116:0/678405949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd29406e6f0 msgr2=0x7fd29412fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:48.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29989c640 1 --2- 172.21.15.116:0/678405949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd29406e6f0 0x7fd29412fc40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd290009910 tx=0x7fd2900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a487ddb115cc519 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 1 -- 172.21.15.116:0/678405949 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290032440 con 0x7fd29406e6f0 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: dump: 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: _finish_auth 0 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 1 -- 172.21.15.116:0/678405949 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2900325a0 con 0x7fd29406e6f0 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 5 monclient: authenticate success, global_id 194760 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 10 monclient: get_monmap_and_config success 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 1 -- 172.21.15.116:0/678405949 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290032940 con 0x7fd29406e6f0 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 4 set_mon_vals no callback set 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29889a640 10 monclient: dump: 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 10 monclient: shutdown 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 1 -- 172.21.15.116:0/678405949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd29406e6f0 msgr2=0x7fd29412fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 1 --2- 172.21.15.116:0/678405949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd29406e6f0 0x7fd29412fc40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd290009910 tx=0x7fd2900091d0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 1 -- 172.21.15.116:0/678405949 shutdown_connections 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 1 --2- 172.21.15.116:0/678405949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd29406e6f0 0x7fd29412fc40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 1 -- 172.21.15.116:0/678405949 >> 172.21.15.116:0/678405949 conn(0x7fd2940672b0 msgr2=0x7fd2940676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 1 -- 172.21.15.116:0/678405949 shutdown_connections 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.261+0000 7fd29bb00640 1 -- 172.21.15.116:0/678405949 wait complete. 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.262+0000 7fd29bb00640 10 monclient: build_initial_monmap 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.262+0000 7fd29bb00640 10 monclient: monmap: 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.262+0000 7fd29bb00640 1 Processor -- start 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.262+0000 7fd29bb00640 1 -- start start 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.262+0000 7fd29bb00640 10 monclient: init 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient: _reopen_session rank -1 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd29406e6f0 0x7fd29412e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): picked mon.noname-b con 0x7fd29406e6f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 0x7fd29412ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): picked mon.noname-c con 0x7fd29406eac0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd294134aa0 0x7fd294098380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): picked mon.noname-a con 0x7fd294134aa0 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29406f6a0 con 0x7fd294134aa0 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29409fdb0 con 0x7fd29406e6f0 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): start opening mon connection 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29406c8d0 con 0x7fd29406eac0 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): _renew_subs 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29bb00640 10 monclient(hunting): authenticate will time out at 1975.584270s 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 0x7fd29412ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd294134aa0 0x7fd294098380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 0x7fd29412ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:54680/0 (socket says 172.21.15.116:54680) 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 1 -- 172.21.15.116:0/251511271 learned_addr learned my addr 172.21.15.116:0/251511271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 10 monclient(hunting): get_auth_request con 0x7fd29406eac0 auth_method 0 2024-09-18T12:45:48.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): get_auth_request con 0x7fd294134aa0 auth_method 0 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29989c640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd29406e6f0 0x7fd29412e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:48.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 1 -- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 msgr2=0x7fd29412ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 1 -- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 msgr2=0x7fd29412ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 0x7fd29412ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29989c640 10 monclient(hunting): get_auth_request con 0x7fd29406e6f0 auth_method 0 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29989c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29989c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29989c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29909b640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 0x7fd29412ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient(hunting): handle_auth_done global_id 194763 payload 824 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 1 -- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 msgr2=0x7fd29412ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 0x7fd29412ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 1 -- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd29406e6f0 msgr2=0x7fd29412e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd29406e6f0 0x7fd29412e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient: _finish_hunting 0 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 1 monclient: found mon.noname-a 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.263+0000 7fd29a09d640 1 -- 172.21.15.116:0/251511271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290009450 con 0x7fd294134aa0 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.265+0000 7fd29a09d640 1 -- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd294134aa0 msgr2=0x7fd294098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.265+0000 7fd29a09d640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd294134aa0 0x7fd294098380 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00b760 tx=0x7fd28c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 1 -- 172.21.15.116:0/251511271 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00f8a0 con 0x7fd294134aa0 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: dump: 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.277 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: _finish_auth 0 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 1 -- 172.21.15.116:0/251511271 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c00fa20 con 0x7fd294134aa0 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd29bb00640 5 monclient: authenticate success, global_id 194763 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd29bb00640 10 monclient: _renew_subs 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 1 -- 172.21.15.116:0/251511271 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00fde0 con 0x7fd294134aa0 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd29bb00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29406c5f0 con 0x7fd294134aa0 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd28bfff640 4 set_mon_vals no callback set 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 10 monclient: dump: 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:45:48.279 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd29bb00640 10 monclient: _renew_subs 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd29bb00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29406b690 con 0x7fd294134aa0 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.266+0000 7fd289ffb640 1 -- 172.21.15.116:0/251511271 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7fd28c00d620 con 0x7fd294134aa0 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd289ffb640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd278038490 0x7fd27803a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd29bb00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd29bb00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd29bb00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.116:3300/0 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd289ffb640 1 -- 172.21.15.116:0/251511271 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7fd28c04c730 con 0x7fd294134aa0 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29406ca90 con 0x7fd294134aa0 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd29989c640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd278038490 0x7fd27803a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:45:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.267+0000 7fd29989c640 10 monclient: get_auth_request con 0x7fd278038490 auth_method 0 2024-09-18T12:45:48.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.270+0000 7fd29989c640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd278038490 0x7fd27803a940 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd2940695f0 tx=0x7fd290032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:45:48.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.271+0000 7fd289ffb640 1 -- 172.21.15.116:0/251511271 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7fd28c007b20 con 0x7fd294134aa0 2024-09-18T12:45:48.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.271+0000 7fd289ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:45:48.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.271+0000 7fd289ffb640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:45:48.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.423+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd29404bf60 con 0x7fd278038490 2024-09-18T12:45:48.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.427+0000 7fd289ffb640 1 -- 172.21.15.116:0/251511271 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97932 (secure 0 0 0) 0x7fd29404bf60 con 0x7fd278038490 2024-09-18T12:45:48.428 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:45:48.428 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:45:48.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd278038490 msgr2=0x7fd27803a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:48.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd278038490 0x7fd27803a940 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd2940695f0 tx=0x7fd290032000 comp rx=0 tx=0).stop 2024-09-18T12:45:48.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 10 monclient: shutdown 2024-09-18T12:45:48.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd294134aa0 msgr2=0x7fd294098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd294134aa0 0x7fd294098380 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00b760 tx=0x7fd28c009780 comp rx=0 tx=0).stop 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 shutdown_connections 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7fd278038490 0x7fd27803a940 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd294134aa0 0x7fd294098380 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd29406eac0 0x7fd29412ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 --2- 172.21.15.116:0/251511271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd29406e6f0 0x7fd29412e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 >> 172.21.15.116:0/251511271 conn(0x7fd2940672b0 msgr2=0x7fd29409ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 shutdown_connections 2024-09-18T12:45:48.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:45:48.429+0000 7fd29bb00640 1 -- 172.21.15.116:0/251511271 wait complete. 2024-09-18T12:45:48.440 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":874,"stamp":"2024-09-18T12:45:46.817712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1815804,"kb_used_data":10456,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564415236,"statfs":{"total":579820584960,"available":577961201664,"internally_reserved":0,"allocated":10706944,"data_stored":8512606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003276"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":175,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946900+0000","last_change":"2024-09-18T12:32:22.608257+0000","last_active":"2024-09-18T12:34:45.946900+0000","last_peered":"2024-09-18T12:34:45.946900+0000","last_clean":"2024-09-18T12:34:45.946900+0000","last_became_active":"2024-09-18T12:32:22.607844+0000","last_became_peered":"2024-09-18T12:32:22.607844+0000","last_unstale":"2024-09-18T12:34:45.946900+0000","last_undegraded":"2024-09-18T12:34:45.946900+0000","last_fullsized":"2024-09-18T12:34:45.946900+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:24:52.306141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947577+0000","last_change":"2024-09-18T12:32:22.602302+0000","last_active":"2024-09-18T12:34:42.947577+0000","last_peered":"2024-09-18T12:34:42.947577+0000","last_clean":"2024-09-18T12:34:42.947577+0000","last_became_active":"2024-09-18T12:32:22.601974+0000","last_became_peered":"2024-09-18T12:32:22.601974+0000","last_unstale":"2024-09-18T12:34:42.947577+0000","last_undegraded":"2024-09-18T12:34:42.947577+0000","last_fullsized":"2024-09-18T12:34:42.947577+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:05.401601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510608+0000","last_change":"2024-09-18T12:32:22.605364+0000","last_active":"2024-09-18T12:34:44.510608+0000","last_peered":"2024-09-18T12:34:44.510608+0000","last_clean":"2024-09-18T12:34:44.510608+0000","last_became_active":"2024-09-18T12:32:22.604956+0000","last_became_peered":"2024-09-18T12:32:22.604956+0000","last_unstale":"2024-09-18T12:34:44.510608+0000","last_undegraded":"2024-09-18T12:34:44.510608+0000","last_fullsized":"2024-09-18T12:34:44.510608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:21:23.773463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":83,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947446+0000","last_change":"2024-09-18T12:32:22.604750+0000","last_active":"2024-09-18T12:34:42.947446+0000","last_peered":"2024-09-18T12:34:42.947446+0000","last_clean":"2024-09-18T12:34:42.947446+0000","last_became_active":"2024-09-18T12:32:22.604332+0000","last_became_peered":"2024-09-18T12:32:22.604332+0000","last_unstale":"2024-09-18T12:34:42.947446+0000","last_undegraded":"2024-09-18T12:34:42.947446+0000","last_fullsized":"2024-09-18T12:34:42.947446+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:50:27.338272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":127,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947518+0000","last_change":"2024-09-18T12:32:23.804087+0000","last_active":"2024-09-18T12:34:42.947518+0000","last_peered":"2024-09-18T12:34:42.947518+0000","last_clean":"2024-09-18T12:34:42.947518+0000","last_became_active":"2024-09-18T12:32:23.803732+0000","last_became_peered":"2024-09-18T12:32:23.803732+0000","last_unstale":"2024-09-18T12:34:42.947518+0000","last_undegraded":"2024-09-18T12:34:42.947518+0000","last_fullsized":"2024-09-18T12:34:42.947518+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:19:27.867616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":151,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946811+0000","last_change":"2024-09-18T12:32:22.604980+0000","last_active":"2024-09-18T12:34:45.946811+0000","last_peered":"2024-09-18T12:34:45.946811+0000","last_clean":"2024-09-18T12:34:45.946811+0000","last_became_active":"2024-09-18T12:32:22.604577+0000","last_became_peered":"2024-09-18T12:32:22.604577+0000","last_unstale":"2024-09-18T12:34:45.946811+0000","last_undegraded":"2024-09-18T12:34:45.946811+0000","last_fullsized":"2024-09-18T12:34:45.946811+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:17:06.770342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":85,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144754+0000","last_change":"2024-09-18T12:32:22.608738+0000","last_active":"2024-09-18T12:34:43.144754+0000","last_peered":"2024-09-18T12:34:43.144754+0000","last_clean":"2024-09-18T12:34:43.144754+0000","last_became_active":"2024-09-18T12:32:22.608401+0000","last_became_peered":"2024-09-18T12:32:22.608401+0000","last_unstale":"2024-09-18T12:34:43.144754+0000","last_undegraded":"2024-09-18T12:34:43.144754+0000","last_fullsized":"2024-09-18T12:34:43.144754+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:59.037772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511904+0000","last_change":"2024-09-18T12:32:22.603874+0000","last_active":"2024-09-18T12:34:44.511904+0000","last_peered":"2024-09-18T12:34:44.511904+0000","last_clean":"2024-09-18T12:34:44.511904+0000","last_became_active":"2024-09-18T12:32:22.603533+0000","last_became_peered":"2024-09-18T12:32:22.603533+0000","last_unstale":"2024-09-18T12:34:44.511904+0000","last_undegraded":"2024-09-18T12:34:44.511904+0000","last_fullsized":"2024-09-18T12:34:44.511904+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:29.278327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947428+0000","last_change":"2024-09-18T12:32:22.607436+0000","last_active":"2024-09-18T12:34:42.947428+0000","last_peered":"2024-09-18T12:34:42.947428+0000","last_clean":"2024-09-18T12:34:42.947428+0000","last_became_active":"2024-09-18T12:32:22.607051+0000","last_became_peered":"2024-09-18T12:32:22.607051+0000","last_unstale":"2024-09-18T12:34:42.947428+0000","last_undegraded":"2024-09-18T12:34:42.947428+0000","last_fullsized":"2024-09-18T12:34:42.947428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:38.371976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":110,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144608+0000","last_change":"2024-09-18T12:32:22.604621+0000","last_active":"2024-09-18T12:34:43.144608+0000","last_peered":"2024-09-18T12:34:43.144608+0000","last_clean":"2024-09-18T12:34:43.144608+0000","last_became_active":"2024-09-18T12:32:22.604295+0000","last_became_peered":"2024-09-18T12:32:22.604295+0000","last_unstale":"2024-09-18T12:34:43.144608+0000","last_undegraded":"2024-09-18T12:34:43.144608+0000","last_fullsized":"2024-09-18T12:34:43.144608+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:48:28.798147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":130,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144436+0000","last_change":"2024-09-18T12:32:22.606967+0000","last_active":"2024-09-18T12:34:43.144436+0000","last_peered":"2024-09-18T12:34:43.144436+0000","last_clean":"2024-09-18T12:34:43.144436+0000","last_became_active":"2024-09-18T12:32:22.606557+0000","last_became_peered":"2024-09-18T12:32:22.606557+0000","last_unstale":"2024-09-18T12:34:43.144436+0000","last_undegraded":"2024-09-18T12:34:43.144436+0000","last_fullsized":"2024-09-18T12:34:43.144436+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:13:49.746180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511862+0000","last_change":"2024-09-18T12:32:22.608034+0000","last_active":"2024-09-18T12:34:44.511862+0000","last_peered":"2024-09-18T12:34:44.511862+0000","last_clean":"2024-09-18T12:34:44.511862+0000","last_became_active":"2024-09-18T12:32:22.607698+0000","last_became_peered":"2024-09-18T12:32:22.607698+0000","last_unstale":"2024-09-18T12:34:44.511862+0000","last_undegraded":"2024-09-18T12:34:44.511862+0000","last_fullsized":"2024-09-18T12:34:44.511862+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:13.180633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":90,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511848+0000","last_change":"2024-09-18T12:32:22.606942+0000","last_active":"2024-09-18T12:34:44.511848+0000","last_peered":"2024-09-18T12:34:44.511848+0000","last_clean":"2024-09-18T12:34:44.511848+0000","last_became_active":"2024-09-18T12:32:22.606593+0000","last_became_peered":"2024-09-18T12:32:22.606593+0000","last_unstale":"2024-09-18T12:34:44.511848+0000","last_undegraded":"2024-09-18T12:34:44.511848+0000","last_fullsized":"2024-09-18T12:34:44.511848+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:53:56.502695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":182,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:45.946736+0000","last_change":"2024-09-18T12:32:22.608147+0000","last_active":"2024-09-18T12:34:45.946736+0000","last_peered":"2024-09-18T12:34:45.946736+0000","last_clean":"2024-09-18T12:34:45.946736+0000","last_became_active":"2024-09-18T12:32:22.607839+0000","last_became_peered":"2024-09-18T12:32:22.607839+0000","last_unstale":"2024-09-18T12:34:45.946736+0000","last_undegraded":"2024-09-18T12:34:45.946736+0000","last_fullsized":"2024-09-18T12:34:45.946736+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:11:45.988633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510548+0000","last_change":"2024-09-18T12:32:23.599583+0000","last_active":"2024-09-18T12:34:44.510548+0000","last_peered":"2024-09-18T12:34:44.510548+0000","last_clean":"2024-09-18T12:34:44.510548+0000","last_became_active":"2024-09-18T12:32:23.599235+0000","last_became_peered":"2024-09-18T12:32:23.599235+0000","last_unstale":"2024-09-18T12:34:44.510548+0000","last_undegraded":"2024-09-18T12:34:44.510548+0000","last_fullsized":"2024-09-18T12:34:44.510548+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:33.154296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511941+0000","last_change":"2024-09-18T12:32:22.600178+0000","last_active":"2024-09-18T12:34:44.511941+0000","last_peered":"2024-09-18T12:34:44.511941+0000","last_clean":"2024-09-18T12:34:44.511941+0000","last_became_active":"2024-09-18T12:32:22.599859+0000","last_became_peered":"2024-09-18T12:32:22.599859+0000","last_unstale":"2024-09-18T12:34:44.511941+0000","last_undegraded":"2024-09-18T12:34:44.511941+0000","last_fullsized":"2024-09-18T12:34:44.511941+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:19:22.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":338,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144609+0000","last_change":"2024-09-18T12:32:22.605174+0000","last_active":"2024-09-18T12:34:43.144609+0000","last_peered":"2024-09-18T12:34:43.144609+0000","last_clean":"2024-09-18T12:34:43.144609+0000","last_became_active":"2024-09-18T12:32:22.604820+0000","last_became_peered":"2024-09-18T12:32:22.604820+0000","last_unstale":"2024-09-18T12:34:43.144609+0000","last_undegraded":"2024-09-18T12:34:43.144609+0000","last_fullsized":"2024-09-18T12:34:43.144609+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:22.895270+0000","last_clean_scrub_stamp":"2024-09-18T12:25:22.895270+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:41:49.881448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":266,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947516+0000","last_change":"2024-09-18T12:32:22.610862+0000","last_active":"2024-09-18T12:34:42.947516+0000","last_peered":"2024-09-18T12:34:42.947516+0000","last_clean":"2024-09-18T12:34:42.947516+0000","last_became_active":"2024-09-18T12:32:22.604471+0000","last_became_peered":"2024-09-18T12:32:22.604471+0000","last_unstale":"2024-09-18T12:34:42.947516+0000","last_undegraded":"2024-09-18T12:34:42.947516+0000","last_fullsized":"2024-09-18T12:34:42.947516+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:31.307834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144612+0000","last_change":"2024-09-18T12:32:22.608239+0000","last_active":"2024-09-18T12:34:43.144612+0000","last_peered":"2024-09-18T12:34:43.144612+0000","last_clean":"2024-09-18T12:34:43.144612+0000","last_became_active":"2024-09-18T12:32:22.607827+0000","last_became_peered":"2024-09-18T12:32:22.607827+0000","last_unstale":"2024-09-18T12:34:43.144612+0000","last_undegraded":"2024-09-18T12:34:43.144612+0000","last_fullsized":"2024-09-18T12:34:43.144612+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:40.035295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.510540+0000","last_change":"2024-09-18T12:32:22.604079+0000","last_active":"2024-09-18T12:34:44.510540+0000","last_peered":"2024-09-18T12:34:44.510540+0000","last_clean":"2024-09-18T12:34:44.510540+0000","last_became_active":"2024-09-18T12:32:22.603727+0000","last_became_peered":"2024-09-18T12:32:22.603727+0000","last_unstale":"2024-09-18T12:34:44.510540+0000","last_undegraded":"2024-09-18T12:34:44.510540+0000","last_fullsized":"2024-09-18T12:34:44.510540+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:42:39.085495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":268,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:42.947425+0000","last_change":"2024-09-18T12:32:22.615941+0000","last_active":"2024-09-18T12:34:42.947425+0000","last_peered":"2024-09-18T12:34:42.947425+0000","last_clean":"2024-09-18T12:34:42.947425+0000","last_became_active":"2024-09-18T12:32:22.608226+0000","last_became_peered":"2024-09-18T12:32:22.608226+0000","last_unstale":"2024-09-18T12:34:42.947425+0000","last_undegraded":"2024-09-18T12:34:42.947425+0000","last_fullsized":"2024-09-18T12:34:42.947425+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:13:36.171958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":264,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511965+0000","last_change":"2024-09-18T12:32:23.598518+0000","last_active":"2024-09-18T12:34:44.511965+0000","last_peered":"2024-09-18T12:34:44.511965+0000","last_clean":"2024-09-18T12:34:44.511965+0000","last_became_active":"2024-09-18T12:32:23.598179+0000","last_became_peered":"2024-09-18T12:32:23.598179+0000","last_unstale":"2024-09-18T12:34:44.511965+0000","last_undegraded":"2024-09-18T12:34:44.511965+0000","last_fullsized":"2024-09-18T12:34:44.511965+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:42.637189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144647+0000","last_change":"2024-09-18T12:32:22.605496+0000","last_active":"2024-09-18T12:34:43.144647+0000","last_peered":"2024-09-18T12:34:43.144647+0000","last_clean":"2024-09-18T12:34:43.144647+0000","last_became_active":"2024-09-18T12:32:22.605190+0000","last_became_peered":"2024-09-18T12:32:22.605190+0000","last_unstale":"2024-09-18T12:34:43.144647+0000","last_undegraded":"2024-09-18T12:34:43.144647+0000","last_fullsized":"2024-09-18T12:34:43.144647+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:50.861016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T12:34:44.511899+0000","last_change":"2024-09-18T12:32:22.603956+0000","last_active":"2024-09-18T12:34:44.511899+0000","last_peered":"2024-09-18T12:34:44.511899+0000","last_clean":"2024-09-18T12:34:44.511899+0000","last_became_active":"2024-09-18T12:32:22.603556+0000","last_became_peered":"2024-09-18T12:32:22.603556+0000","last_unstale":"2024-09-18T12:34:44.511899+0000","last_undegraded":"2024-09-18T12:34:44.511899+0000","last_fullsized":"2024-09-18T12:34:44.511899+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:40:30.975910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T12:34:43.144746+0000","last_change":"2024-09-18T12:32:22.601754+0000","last_active":"2024-09-18T12:34:43.144746+0000","last_peered":"2024-09-18T12:34:43.144746+0000","last_clean":"2024-09-18T12:34:43.144746+0000","last_became_active":"2024-09-18T12:32:22.601418+0000","last_became_peered":"2024-09-18T12:32:22.601418+0000","last_unstale":"2024-09-18T12:34:43.144746+0000","last_undegraded":"2024-09-18T12:34:43.144746+0000","last_fullsized":"2024-09-18T12:34:43.144746+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:18:06.635553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640502,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.45900000000000002},"min":{"1min":0.40000000000000002,"5min":0.32400000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.69099999999999995},"last":0.373},{"interface":"front","average":{"1min":0.499,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.20999999999999999},"max":{"1min":0.57999999999999996,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53900000000000003,"15min":0.51700000000000002},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.23999999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.48999999999999999,"5min":0.39900000000000002,"15min":0.27200000000000002},"max":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.80200000000000005},"last":0.42899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":1.304,"15min":1.304},"last":0.439},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.25700000000000001},"max":{"1min":0.63300000000000001,"5min":0.72799999999999998,"15min":0.753},"last":0.39900000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.371,"15min":0.36199999999999999},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":0.51600000000000001,"5min":0.59599999999999997,"15min":0.60299999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.20100000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.64200000000000002},"last":0.497}]},{"osd":4,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":0.58899999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.223},"max":{"1min":0.61199999999999999,"5min":0.68200000000000005,"15min":0.73799999999999999},"last":0.60899999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640502,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298924,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072916,"statfs":{"total":96636764160,"available":96330665984,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.47699999999999998},"min":{"1min":0.37,"5min":0.27400000000000002,"15min":0.223},"max":{"1min":0.60199999999999998,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.498},{"interface":"front","average":{"1min":0.504,"5min":0.49199999999999999,"15min":0.47799999999999998},"min":{"1min":0.379,"5min":0.33100000000000002,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.434}]},{"osd":1,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.25900000000000001},"max":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.753},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.30399999999999999},"max":{"1min":0.63600000000000001,"5min":0.71999999999999997,"15min":0.872},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71499999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45100000000000001,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41499999999999998,"15min":0.38400000000000001},"min":{"1min":0.33600000000000002,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.59399999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.307,"5min":0.255,"15min":0.249},"max":{"1min":0.53700000000000003,"5min":0.68899999999999995,"15min":0.70299999999999996},"last":0.54400000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59499999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.39100000000000001,"15min":0.38300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26100000000000001,"15min":0.223},"max":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.60999999999999999},"last":0.59999999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640505,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304012,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067828,"statfs":{"total":96636764160,"available":96325455872,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.33700000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65300000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.376,"5min":0.34599999999999997,"15min":0.33100000000000002},"max":{"1min":0.68600000000000005,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.44500000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51000000000000001,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.35599999999999998,"15min":0.26600000000000001},"max":{"1min":0.64500000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.68600000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":0.70099999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.503,"15min":0.5},"min":{"1min":0.37,"5min":0.33400000000000002,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.873,"15min":0.873},"last":0.52300000000000002}]},{"osd":4,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36099999999999999,"15min":0.36499999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.214},"max":{"1min":0.55000000000000004,"5min":0.56799999999999995,"15min":0.624},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.36199999999999999,"15min":0.372},"min":{"1min":0.254,"5min":0.254,"15min":0.221},"max":{"1min":0.48499999999999999,"5min":0.58199999999999996,"15min":0.67200000000000004},"last":0.32300000000000001}]},{"osd":5,"last update":"Wed Sep 18 12:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36099999999999999,"15min":0.36799999999999999},"min":{"1min":0.26500000000000001,"5min":0.26100000000000001,"15min":0.20300000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.64600000000000002,"15min":0.70599999999999996},"last":0.36699999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.315,"15min":0.32200000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.156},"max":{"1min":0.46800000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.313},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34399999999999997,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.628},"last":0.32900000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35299999999999998,"15min":0.37},"min":{"1min":0.23599999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51700000000000002,"5min":0.60599999999999998,"15min":0.63900000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.33600000000000002,"15min":0.34200000000000003},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.17699999999999999},"max":{"1min":0.54200000000000004,"5min":0.625,"15min":0.625},"last":0.29399999999999998}]},{"osd":3,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":0.77000000000000002,"15min":0.77300000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.308},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53900000000000003}]},{"osd":4,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.56000000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75700000000000001,"5min":1.034,"15min":1.034},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.83099999999999996},"last":0.55300000000000005}]},{"osd":5,"last update":"Wed Sep 18 12:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.40300000000000002,"5min":0.378,"15min":0.28399999999999997},"max":{"1min":0.72999999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55100000000000005,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.32400000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.57899999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640504,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303088,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068752,"statfs":{"total":96636764160,"available":96326402048,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":0.41999999999999998,"5min":0.48499999999999999,"15min":0.70699999999999996},"last":0.437},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33700000000000002,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.214},"max":{"1min":0.44700000000000001,"5min":0.629,"15min":0.80000000000000004},"last":0.51700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34499999999999997,"15min":0.35899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":0.46800000000000003,"5min":0.52000000000000002,"15min":0.86599999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.26900000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48599999999999999,"5min":0.53600000000000003,"15min":0.82599999999999996},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.41699999999999998,"5min":0.377,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":0.84199999999999997,"15min":0.95699999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55300000000000005,"15min":0.54200000000000004},"min":{"1min":0.40699999999999997,"5min":0.375,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.83799999999999997,"15min":0.97899999999999998},"last":0.54700000000000004}]},{"osd":4,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.317},"max":{"1min":0.76400000000000001,"5min":0.82099999999999995,"15min":0.997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.34999999999999998},"max":{"1min":0.71399999999999997,"5min":0.85199999999999998,"15min":1.034},"last":0.53200000000000003}]},{"osd":5,"last update":"Wed Sep 18 12:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":0.68200000000000005,"5min":0.82999999999999996,"15min":0.88300000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58299999999999996},"min":{"1min":0.42199999999999999,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":0.625,"5min":0.81299999999999994,"15min":1.0960000000000001},"last":0.59499999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303608,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068232,"statfs":{"total":96636764160,"available":96325869568,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.30599999999999999,"15min":0.317},"min":{"1min":0.223,"5min":0.14799999999999999,"15min":0.13400000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.55500000000000005},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.16400000000000001},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.64500000000000002},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.35499999999999998},"min":{"1min":0.21299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.48599999999999999,"5min":0.502,"15min":0.76800000000000002},"last":0.246},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.36799999999999999},"min":{"1min":0.28499999999999998,"5min":0.221,"15min":0.16},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.73499999999999999},"last":0.373}]},{"osd":3,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.69499999999999995,"5min":1.177,"15min":1.177},"last":0.504},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.53700000000000003},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52500000000000002}]},{"osd":4,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":1.127,"15min":1.127},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.69999999999999996,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.58199999999999996}]},{"osd":5,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.54000000000000004},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":1.141,"15min":1.141},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34100000000000003},"max":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:45:48.447 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.447 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.447 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.447 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.447 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.447 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.448 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T12:25:22.895270+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=22, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.449 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.450 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:45:48.450 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T12:46:08.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:46:08.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.556+0000 7f026799d640 10 monclient: get_monmap_and_config 2024-09-18T12:46:08.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.556+0000 7f026799d640 10 monclient: build_initial_monmap 2024-09-18T12:46:08.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.556+0000 7f026799d640 10 monclient: monmap: 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:46:08.557047+0000 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:46:08.557047+0000 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:46:08.558 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.556+0000 7f026799d640 1 Processor -- start 2024-09-18T12:46:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 1 -- start start 2024-09-18T12:46:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient: init 2024-09-18T12:46:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient: _reopen_session rank -1 2024-09-18T12:46:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T12:46:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02600981c0 addr v1:172.21.15.144:6789/0 2024-09-18T12:46:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0260131da0 0x7f0260136b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient(hunting): picked mon.noname-c con 0x7f0260131da0 addr v2:172.21.15.144:3300/0 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 0x7f02601390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient(hunting): picked mon.noname-b con 0x7f0260132170 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026009ca60 con 0x7f02600981c0 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0260131bd0 con 0x7f0260131da0 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026009cca0 con 0x7f0260132170 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient(hunting): _renew_subs 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f026799d640 10 monclient(hunting): authenticate will time out at 1995.878664s 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 0x7f02601390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 0x7f02601390c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51690/0 (socket says 172.21.15.116:51690) 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 1 -- 172.21.15.116:0/88338553 learned_addr learned my addr 172.21.15.116:0/88338553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:46:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 10 monclient(hunting): get_auth_request con 0x7f0260132170 auth_method 0 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0264f38640 1 --2- 172.21.15.116:0/88338553 >> v2:172.21.15.144:3300/0 conn(0x7f0260131da0 0x7f0260136b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0265f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0264f38640 10 monclient(hunting): get_auth_request con 0x7f0260131da0 auth_method 0 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0264f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0264f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.557+0000 7f0264f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0264f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0264f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0264f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 10 monclient(hunting): handle_auth_done global_id 244232 payload 274 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 1 -- 172.21.15.116:0/88338553 >> v2:172.21.15.144:3300/0 conn(0x7f0260131da0 msgr2=0x7f0260136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 1 --2- 172.21.15.116:0/88338553 >> v2:172.21.15.144:3300/0 conn(0x7f0260131da0 0x7f0260136b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 1 -- 172.21.15.116:0/88338553 >> v1:172.21.15.144:6789/0 conn(0x7f02600981c0 legacy=0x7f0260098590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 10 monclient: _finish_hunting 0 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 1 monclient: found mon.noname-b 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.558+0000 7f0265f3a640 1 -- 172.21.15.116:0/88338553 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026009e010 con 0x7f0260132170 2024-09-18T12:46:08.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.560+0000 7f0265f3a640 1 -- 172.21.15.116:0/88338553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 msgr2=0x7f02601390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.560+0000 7f0265f3a640 1 --2- 172.21.15.116:0/88338553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 0x7f02601390c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0260132870 tx=0x7f0258009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618786ab0b181ab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 1 -- 172.21.15.116:0/88338553 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0258007610 con 0x7f0260132170 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: dump: 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:08.567 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: _finish_auth 0 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 1 -- 172.21.15.116:0/88338553 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025802d050 con 0x7f0260132170 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 5 monclient: authenticate success, global_id 244232 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 10 monclient: get_monmap_and_config success 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 1 -- 172.21.15.116:0/88338553 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02580079a0 con 0x7f0260132170 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:08.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 4 set_mon_vals no callback set 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f0257fff640 10 monclient: dump: 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:08.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 10 monclient: shutdown 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 1 -- 172.21.15.116:0/88338553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 msgr2=0x7f02601390c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 1 --2- 172.21.15.116:0/88338553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 0x7f02601390c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0260132870 tx=0x7f0258009840 comp rx=0 tx=0).stop 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 1 -- 172.21.15.116:0/88338553 shutdown_connections 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 1 --2- 172.21.15.116:0/88338553 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260132170 0x7f02601390c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 1 --2- 172.21.15.116:0/88338553 >> v2:172.21.15.144:3300/0 conn(0x7f0260131da0 0x7f0260136b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.561+0000 7f026799d640 1 -- 172.21.15.116:0/88338553 >> 172.21.15.116:0/88338553 conn(0x7f02600a1e30 msgr2=0x7f02600a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.562+0000 7f026799d640 1 -- 172.21.15.116:0/88338553 shutdown_connections 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.562+0000 7f026799d640 1 -- 172.21.15.116:0/88338553 wait complete. 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.562+0000 7f026799d640 10 monclient: build_initial_monmap 2024-09-18T12:46:08.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.562+0000 7f026799d640 10 monclient: monmap: 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.562+0000 7f026799d640 1 Processor -- start 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.562+0000 7f026799d640 1 -- start start 2024-09-18T12:46:08.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.562+0000 7f026799d640 10 monclient: init 2024-09-18T12:46:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient: _reopen_session rank -1 2024-09-18T12:46:08.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02600981c0 0x7f02601313e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): picked mon.noname-b con 0x7f02600981c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 0x7f0260137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): picked mon.noname-c con 0x7f0260131da0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0260132170 0x7f02601382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): picked mon.noname-a con 0x7f0260132170 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0260130940 con 0x7f0260132170 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0260118d30 con 0x7f02600981c0 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:08.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026009bc10 con 0x7f0260131da0 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): _renew_subs 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f026799d640 10 monclient(hunting): authenticate will time out at 1995.884431s 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 0x7f0260137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 0x7f0260137d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:51698/0 (socket says 172.21.15.116:51698) 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0260132170 0x7f02601382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 -- 172.21.15.116:0/2723387421 learned_addr learned my addr 172.21.15.116:0/2723387421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): get_auth_request con 0x7f0260131da0 auth_method 0 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 10 monclient(hunting): get_auth_request con 0x7f0260132170 auth_method 0 2024-09-18T12:46:08.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265739640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02600981c0 0x7f02601313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265739640 10 monclient(hunting): get_auth_request con 0x7f02600981c0 auth_method 0 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265739640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265739640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient(hunting): handle_auth_done global_id 244235 payload 824 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 -- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02600981c0 msgr2=0x7f02601313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02600981c0 0x7f02601313e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 -- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0260132170 msgr2=0x7f02601382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0260132170 0x7f02601382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient: _finish_hunting 0 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 monclient: found mon.noname-c 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:46:08.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0264f38640 1 -- 172.21.15.116:0/2723387421 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258009ae0 con 0x7f0260131da0 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.563+0000 7f0265739640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02600981c0 0x7f02601313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f0264f38640 1 -- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 msgr2=0x7f0260137d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f0264f38640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 0x7f0260137d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f025000ea20 tx=0x7f02500090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 1 -- 172.21.15.116:0/2723387421 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250009600 con 0x7f0260131da0 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: dump: 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.582 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: _finish_auth 0 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f026799d640 5 monclient: authenticate success, global_id 244235 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 1 -- 172.21.15.116:0/2723387421 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025000cc20 con 0x7f0260131da0 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f026799d640 10 monclient: _renew_subs 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f026799d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026011ab80 con 0x7f0260131da0 2024-09-18T12:46:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 1 -- 172.21.15.116:0/2723387421 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250013470 con 0x7f0260131da0 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02557fa640 10 monclient: dump: 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:08.584 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f026799d640 10 monclient: _renew_subs 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f026799d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026011a8a0 con 0x7f0260131da0 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.566+0000 7f02577fe640 4 set_mon_vals no callback set 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f02557fa640 1 -- 172.21.15.116:0/2723387421 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f0250013860 con 0x7f0260131da0 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f026799d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f026799d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f026799d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026004bf60 con 0x7f0260131da0 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f02557fa640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0244038490 0x7f024403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:08.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f02557fa640 1 -- 172.21.15.116:0/2723387421 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f025004c650 con 0x7f0260131da0 2024-09-18T12:46:08.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f0265739640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0244038490 0x7f024403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:08.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.567+0000 7f0265739640 10 monclient: get_auth_request con 0x7f0244038490 auth_method 0 2024-09-18T12:46:08.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.570+0000 7f0265739640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0244038490 0x7f024403a940 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f025c009970 tx=0x7f025c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:08.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.571+0000 7f02557fa640 1 -- 172.21.15.116:0/2723387421 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f02500236b0 con 0x7f0260131da0 2024-09-18T12:46:08.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.571+0000 7f02557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:08.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.571+0000 7f02557fa640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:46:08.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.724+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0260098ee0 con 0x7f0244038490 2024-09-18T12:46:08.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.728+0000 7f02557fa640 1 -- 172.21.15.116:0/2723387421 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98104 (secure 0 0 0) 0x7f0260098ee0 con 0x7f0244038490 2024-09-18T12:46:08.729 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:46:08.730 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0244038490 msgr2=0x7f024403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0244038490 0x7f024403a940 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f025c009970 tx=0x7f025c008040 comp rx=0 tx=0).stop 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 10 monclient: shutdown 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 msgr2=0x7f0260137d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 0x7f0260137d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f025000ea20 tx=0x7f02500090f0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 shutdown_connections 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f0244038490 0x7f024403a940 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0260132170 0x7f02601382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f0260131da0 0x7f0260137d60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 --2- 172.21.15.116:0/2723387421 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02600981c0 0x7f02601313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 >> 172.21.15.116:0/2723387421 conn(0x7f02600a1e30 msgr2=0x7f026009b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:46:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 shutdown_connections 2024-09-18T12:46:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:08.730+0000 7f026799d640 1 -- 172.21.15.116:0/2723387421 wait complete. 2024-09-18T12:46:08.741 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-09-18T12:46:06.822256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1815368,"kb_used_data":10456,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564415672,"statfs":{"total":579820584960,"available":577961648128,"internally_reserved":0,"allocated":10706944,"data_stored":8512606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002918"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":186,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:50.386562+0000","last_change":"2024-09-18T12:45:50.386527+0000","last_active":"2024-09-18T12:45:50.386562+0000","last_peered":"2024-09-18T12:45:50.386562+0000","last_clean":"2024-09-18T12:45:50.386562+0000","last_became_active":"2024-09-18T12:32:22.607844+0000","last_became_peered":"2024-09-18T12:32:22.607844+0000","last_unstale":"2024-09-18T12:45:50.386562+0000","last_undegraded":"2024-09-18T12:45:50.386562+0000","last_fullsized":"2024-09-18T12:45:50.386562+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:50.386445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:50.386445+0000","last_clean_scrub_stamp":"2024-09-18T12:45:50.386445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:54:37.397067+0000","scrub_duration":0.001865991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":95,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:59.048343+0000","last_change":"2024-09-18T12:45:59.048315+0000","last_active":"2024-09-18T12:45:59.048343+0000","last_peered":"2024-09-18T12:45:59.048343+0000","last_clean":"2024-09-18T12:45:59.048343+0000","last_became_active":"2024-09-18T12:32:22.601974+0000","last_became_peered":"2024-09-18T12:32:22.601974+0000","last_unstale":"2024-09-18T12:45:59.048343+0000","last_undegraded":"2024-09-18T12:45:59.048343+0000","last_fullsized":"2024-09-18T12:45:59.048343+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:59.048223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:59.048223+0000","last_clean_scrub_stamp":"2024-09-18T12:45:59.048223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:45:49.110046+0000","scrub_duration":0.001456457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":169,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:47.162252+0000","last_change":"2024-09-18T12:45:47.162218+0000","last_active":"2024-09-18T12:45:47.162252+0000","last_peered":"2024-09-18T12:45:47.162252+0000","last_clean":"2024-09-18T12:45:47.162252+0000","last_became_active":"2024-09-18T12:32:22.604956+0000","last_became_peered":"2024-09-18T12:32:22.604956+0000","last_unstale":"2024-09-18T12:45:47.162252+0000","last_undegraded":"2024-09-18T12:45:47.162252+0000","last_fullsized":"2024-09-18T12:45:47.162252+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:47.162148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:47.162148+0000","last_clean_scrub_stamp":"2024-09-18T12:45:47.162148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:54:07.071324+0000","scrub_duration":0.001412724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":84,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:34:46.149100+0000","last_change":"2024-09-18T12:32:22.604750+0000","last_active":"2024-09-18T12:34:46.149100+0000","last_peered":"2024-09-18T12:34:46.149100+0000","last_clean":"2024-09-18T12:34:46.149100+0000","last_became_active":"2024-09-18T12:32:22.604332+0000","last_became_peered":"2024-09-18T12:32:22.604332+0000","last_unstale":"2024-09-18T12:34:46.149100+0000","last_undegraded":"2024-09-18T12:34:46.149100+0000","last_fullsized":"2024-09-18T12:34:46.149100+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:56.227060+0000","last_clean_scrub_stamp":"2024-09-18T12:30:56.227060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":128,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:34:46.149194+0000","last_change":"2024-09-18T12:32:23.804087+0000","last_active":"2024-09-18T12:34:46.149194+0000","last_peered":"2024-09-18T12:34:46.149194+0000","last_clean":"2024-09-18T12:34:46.149194+0000","last_became_active":"2024-09-18T12:32:23.803732+0000","last_became_peered":"2024-09-18T12:32:23.803732+0000","last_unstale":"2024-09-18T12:34:46.149194+0000","last_undegraded":"2024-09-18T12:34:46.149194+0000","last_fullsized":"2024-09-18T12:34:46.149194+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:30:57.290984+0000","last_clean_scrub_stamp":"2024-09-18T12:30:57.290984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":162,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.348872+0000","last_change":"2024-09-18T12:45:48.348851+0000","last_active":"2024-09-18T12:45:48.348872+0000","last_peered":"2024-09-18T12:45:48.348872+0000","last_clean":"2024-09-18T12:45:48.348872+0000","last_became_active":"2024-09-18T12:32:22.604577+0000","last_became_peered":"2024-09-18T12:32:22.604577+0000","last_unstale":"2024-09-18T12:45:48.348872+0000","last_undegraded":"2024-09-18T12:45:48.348872+0000","last_fullsized":"2024-09-18T12:45:48.348872+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.348781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.348781+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.348781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:32:33.271576+0000","scrub_duration":0.001644026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":96,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:53.308507+0000","last_change":"2024-09-18T12:45:53.308469+0000","last_active":"2024-09-18T12:45:53.308507+0000","last_peered":"2024-09-18T12:45:53.308507+0000","last_clean":"2024-09-18T12:45:53.308507+0000","last_became_active":"2024-09-18T12:32:22.608401+0000","last_became_peered":"2024-09-18T12:32:22.608401+0000","last_unstale":"2024-09-18T12:45:53.308507+0000","last_undegraded":"2024-09-18T12:45:53.308507+0000","last_fullsized":"2024-09-18T12:45:53.308507+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:53.308379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:53.308379+0000","last_clean_scrub_stamp":"2024-09-18T12:45:53.308379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:28:05.578973+0000","scrub_duration":0.0019799219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:59.426629+0000","last_change":"2024-09-18T12:45:59.426585+0000","last_active":"2024-09-18T12:45:59.426629+0000","last_peered":"2024-09-18T12:45:59.426629+0000","last_clean":"2024-09-18T12:45:59.426629+0000","last_became_active":"2024-09-18T12:32:22.603533+0000","last_became_peered":"2024-09-18T12:32:22.603533+0000","last_unstale":"2024-09-18T12:45:59.426629+0000","last_undegraded":"2024-09-18T12:45:59.426629+0000","last_fullsized":"2024-09-18T12:45:59.426629+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:59.426489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:59.426489+0000","last_clean_scrub_stamp":"2024-09-18T12:45:59.426489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:50:50.132667+0000","scrub_duration":0.0016270550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":121,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:56.012671+0000","last_change":"2024-09-18T12:45:56.012629+0000","last_active":"2024-09-18T12:45:56.012671+0000","last_peered":"2024-09-18T12:45:56.012671+0000","last_clean":"2024-09-18T12:45:56.012671+0000","last_became_active":"2024-09-18T12:32:22.607051+0000","last_became_peered":"2024-09-18T12:32:22.607051+0000","last_unstale":"2024-09-18T12:45:56.012671+0000","last_undegraded":"2024-09-18T12:45:56.012671+0000","last_fullsized":"2024-09-18T12:45:56.012671+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:56.012535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:56.012535+0000","last_clean_scrub_stamp":"2024-09-18T12:45:56.012535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:32:06.110764+0000","scrub_duration":0.001727463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":121,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:51.322062+0000","last_change":"2024-09-18T12:45:51.322033+0000","last_active":"2024-09-18T12:45:51.322062+0000","last_peered":"2024-09-18T12:45:51.322062+0000","last_clean":"2024-09-18T12:45:51.322062+0000","last_became_active":"2024-09-18T12:32:22.604295+0000","last_became_peered":"2024-09-18T12:32:22.604295+0000","last_unstale":"2024-09-18T12:45:51.322062+0000","last_undegraded":"2024-09-18T12:45:51.322062+0000","last_fullsized":"2024-09-18T12:45:51.322062+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:51.321933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:51.321933+0000","last_clean_scrub_stamp":"2024-09-18T12:45:51.321933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:28:05.462547+0000","scrub_duration":0.0017206050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":141,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.078232+0000","last_change":"2024-09-18T12:45:48.078179+0000","last_active":"2024-09-18T12:45:48.078232+0000","last_peered":"2024-09-18T12:45:48.078232+0000","last_clean":"2024-09-18T12:45:48.078232+0000","last_became_active":"2024-09-18T12:32:22.606557+0000","last_became_peered":"2024-09-18T12:32:22.606557+0000","last_unstale":"2024-09-18T12:45:48.078232+0000","last_undegraded":"2024-09-18T12:45:48.078232+0000","last_fullsized":"2024-09-18T12:45:48.078232+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.078076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.078076+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.078076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:50:42.309857+0000","scrub_duration":0.001689879,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.478217+0000","last_change":"2024-09-18T12:45:48.478195+0000","last_active":"2024-09-18T12:45:48.478217+0000","last_peered":"2024-09-18T12:45:48.478217+0000","last_clean":"2024-09-18T12:45:48.478217+0000","last_became_active":"2024-09-18T12:32:22.607698+0000","last_became_peered":"2024-09-18T12:32:22.607698+0000","last_unstale":"2024-09-18T12:45:48.478217+0000","last_undegraded":"2024-09-18T12:45:48.478217+0000","last_fullsized":"2024-09-18T12:45:48.478217+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.478124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.478124+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.478124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:34:05.636171+0000","scrub_duration":0.001238491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":101,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:58.456787+0000","last_change":"2024-09-18T12:45:58.456753+0000","last_active":"2024-09-18T12:45:58.456787+0000","last_peered":"2024-09-18T12:45:58.456787+0000","last_clean":"2024-09-18T12:45:58.456787+0000","last_became_active":"2024-09-18T12:32:22.606593+0000","last_became_peered":"2024-09-18T12:32:22.606593+0000","last_unstale":"2024-09-18T12:45:58.456787+0000","last_undegraded":"2024-09-18T12:45:58.456787+0000","last_fullsized":"2024-09-18T12:45:58.456787+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:58.456663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:58.456663+0000","last_clean_scrub_stamp":"2024-09-18T12:45:58.456663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:15:43.618669+0000","scrub_duration":0.001914282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":193,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:47.316897+0000","last_change":"2024-09-18T12:45:47.316877+0000","last_active":"2024-09-18T12:45:47.316897+0000","last_peered":"2024-09-18T12:45:47.316897+0000","last_clean":"2024-09-18T12:45:47.316897+0000","last_became_active":"2024-09-18T12:32:22.607839+0000","last_became_peered":"2024-09-18T12:32:22.607839+0000","last_unstale":"2024-09-18T12:45:47.316897+0000","last_undegraded":"2024-09-18T12:45:47.316897+0000","last_fullsized":"2024-09-18T12:45:47.316897+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:47.316811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:47.316811+0000","last_clean_scrub_stamp":"2024-09-18T12:45:47.316811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:08:49.061260+0000","scrub_duration":0.0011678649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":98,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:49.138377+0000","last_change":"2024-09-18T12:45:49.138346+0000","last_active":"2024-09-18T12:45:49.138377+0000","last_peered":"2024-09-18T12:45:49.138377+0000","last_clean":"2024-09-18T12:45:49.138377+0000","last_became_active":"2024-09-18T12:32:23.599235+0000","last_became_peered":"2024-09-18T12:32:23.599235+0000","last_unstale":"2024-09-18T12:45:49.138377+0000","last_undegraded":"2024-09-18T12:45:49.138377+0000","last_fullsized":"2024-09-18T12:45:49.138377+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:49.138269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:49.138269+0000","last_clean_scrub_stamp":"2024-09-18T12:45:49.138269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:50:20.823661+0000","scrub_duration":0.001819174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:57.469596+0000","last_change":"2024-09-18T12:45:57.469554+0000","last_active":"2024-09-18T12:45:57.469596+0000","last_peered":"2024-09-18T12:45:57.469596+0000","last_clean":"2024-09-18T12:45:57.469596+0000","last_became_active":"2024-09-18T12:32:22.599859+0000","last_became_peered":"2024-09-18T12:32:22.599859+0000","last_unstale":"2024-09-18T12:45:57.469596+0000","last_undegraded":"2024-09-18T12:45:57.469596+0000","last_fullsized":"2024-09-18T12:45:57.469596+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:57.469460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:57.469460+0000","last_clean_scrub_stamp":"2024-09-18T12:45:57.469460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:01:36.232281+0000","scrub_duration":0.0019671229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"267'74","reported_seq":351,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:58.375793+0000","last_change":"2024-09-18T12:45:58.375751+0000","last_active":"2024-09-18T12:45:58.375793+0000","last_peered":"2024-09-18T12:45:58.375793+0000","last_clean":"2024-09-18T12:45:58.375793+0000","last_became_active":"2024-09-18T12:32:22.604820+0000","last_became_peered":"2024-09-18T12:32:22.604820+0000","last_unstale":"2024-09-18T12:45:58.375793+0000","last_undegraded":"2024-09-18T12:45:58.375793+0000","last_fullsized":"2024-09-18T12:45:58.375793+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'74","last_scrub_stamp":"2024-09-18T12:45:58.375649+0000","last_deep_scrub":"267'74","last_deep_scrub_stamp":"2024-09-18T12:45:58.375649+0000","last_clean_scrub_stamp":"2024-09-18T12:45:58.375649+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:34:10.370999+0000","scrub_duration":0.011783999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"267'2","reported_seq":278,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:50.070267+0000","last_change":"2024-09-18T12:45:50.070232+0000","last_active":"2024-09-18T12:45:50.070267+0000","last_peered":"2024-09-18T12:45:50.070267+0000","last_clean":"2024-09-18T12:45:50.070267+0000","last_became_active":"2024-09-18T12:32:22.604471+0000","last_became_peered":"2024-09-18T12:32:22.604471+0000","last_unstale":"2024-09-18T12:45:50.070267+0000","last_undegraded":"2024-09-18T12:45:50.070267+0000","last_fullsized":"2024-09-18T12:45:50.070267+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'2","last_scrub_stamp":"2024-09-18T12:45:50.070083+0000","last_deep_scrub":"267'2","last_deep_scrub_stamp":"2024-09-18T12:45:50.070083+0000","last_clean_scrub_stamp":"2024-09-18T12:45:50.070083+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:59:44.447590+0000","scrub_duration":0.0032318070000000002,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":300,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.296980+0000","last_change":"2024-09-18T12:45:48.296944+0000","last_active":"2024-09-18T12:45:48.296980+0000","last_peered":"2024-09-18T12:45:48.296980+0000","last_clean":"2024-09-18T12:45:48.296980+0000","last_became_active":"2024-09-18T12:32:22.607827+0000","last_became_peered":"2024-09-18T12:32:22.607827+0000","last_unstale":"2024-09-18T12:45:48.296980+0000","last_undegraded":"2024-09-18T12:45:48.296980+0000","last_fullsized":"2024-09-18T12:45:48.296980+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.296841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.296841+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.296841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:41:06.523245+0000","scrub_duration":0.0016207960000000001,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":98,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.141219+0000","last_change":"2024-09-18T12:45:48.141192+0000","last_active":"2024-09-18T12:45:48.141219+0000","last_peered":"2024-09-18T12:45:48.141219+0000","last_clean":"2024-09-18T12:45:48.141219+0000","last_became_active":"2024-09-18T12:32:22.603727+0000","last_became_peered":"2024-09-18T12:32:22.603727+0000","last_unstale":"2024-09-18T12:45:48.141219+0000","last_undegraded":"2024-09-18T12:45:48.141219+0000","last_fullsized":"2024-09-18T12:45:48.141219+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.141119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.141119+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.141119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:35:09.560138+0000","scrub_duration":0.0012581960000000001,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"267'3","reported_seq":280,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:51.065560+0000","last_change":"2024-09-18T12:45:51.065532+0000","last_active":"2024-09-18T12:45:51.065560+0000","last_peered":"2024-09-18T12:45:51.065560+0000","last_clean":"2024-09-18T12:45:51.065560+0000","last_became_active":"2024-09-18T12:32:22.608226+0000","last_became_peered":"2024-09-18T12:32:22.608226+0000","last_unstale":"2024-09-18T12:45:51.065560+0000","last_undegraded":"2024-09-18T12:45:51.065560+0000","last_fullsized":"2024-09-18T12:45:51.065560+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'3","last_scrub_stamp":"2024-09-18T12:45:51.065445+0000","last_deep_scrub":"267'3","last_deep_scrub_stamp":"2024-09-18T12:45:51.065445+0000","last_clean_scrub_stamp":"2024-09-18T12:45:51.065445+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:22:43.174515+0000","scrub_duration":0.003296944,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":265,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:34:46.149455+0000","last_change":"2024-09-18T12:32:23.598518+0000","last_active":"2024-09-18T12:34:46.149455+0000","last_peered":"2024-09-18T12:34:46.149455+0000","last_clean":"2024-09-18T12:34:46.149455+0000","last_became_active":"2024-09-18T12:32:23.598179+0000","last_became_peered":"2024-09-18T12:32:23.598179+0000","last_unstale":"2024-09-18T12:34:46.149455+0000","last_undegraded":"2024-09-18T12:34:46.149455+0000","last_fullsized":"2024-09-18T12:34:46.149455+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:25:24.910822+0000","last_clean_scrub_stamp":"2024-09-18T12:25:24.910822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:46:00.348007+0000","last_change":"2024-09-18T12:46:00.347979+0000","last_active":"2024-09-18T12:46:00.348007+0000","last_peered":"2024-09-18T12:46:00.348007+0000","last_clean":"2024-09-18T12:46:00.348007+0000","last_became_active":"2024-09-18T12:32:22.605190+0000","last_became_peered":"2024-09-18T12:32:22.605190+0000","last_unstale":"2024-09-18T12:46:00.348007+0000","last_undegraded":"2024-09-18T12:46:00.348007+0000","last_fullsized":"2024-09-18T12:46:00.348007+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:46:00.347873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:46:00.347873+0000","last_clean_scrub_stamp":"2024-09-18T12:46:00.347873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:04:45.509794+0000","scrub_duration":0.001858186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:52.462616+0000","last_change":"2024-09-18T12:45:52.462566+0000","last_active":"2024-09-18T12:45:52.462616+0000","last_peered":"2024-09-18T12:45:52.462616+0000","last_clean":"2024-09-18T12:45:52.462616+0000","last_became_active":"2024-09-18T12:32:22.603556+0000","last_became_peered":"2024-09-18T12:32:22.603556+0000","last_unstale":"2024-09-18T12:45:52.462616+0000","last_undegraded":"2024-09-18T12:45:52.462616+0000","last_fullsized":"2024-09-18T12:45:52.462616+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:52.462462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:52.462462+0000","last_clean_scrub_stamp":"2024-09-18T12:45:52.462462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:07:15.250417+0000","scrub_duration":0.0020054740000000001,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:50.280086+0000","last_change":"2024-09-18T12:45:50.280056+0000","last_active":"2024-09-18T12:45:50.280086+0000","last_peered":"2024-09-18T12:45:50.280086+0000","last_clean":"2024-09-18T12:45:50.280086+0000","last_became_active":"2024-09-18T12:32:22.601418+0000","last_became_peered":"2024-09-18T12:32:22.601418+0000","last_unstale":"2024-09-18T12:45:50.280086+0000","last_undegraded":"2024-09-18T12:45:50.280086+0000","last_fullsized":"2024-09-18T12:45:50.280086+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:50.279963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:50.279963+0000","last_clean_scrub_stamp":"2024-09-18T12:45:50.279963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:03:26.802247+0000","scrub_duration":0.0013355419999999999,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640506,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.19700000000000001},"max":{"1min":0.59099999999999997,"5min":0.65700000000000003,"15min":0.69099999999999995},"last":0.443},{"interface":"front","average":{"1min":0.5,"5min":0.50700000000000001,"15min":0.48399999999999999},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.51800000000000002},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.23999999999999999},"max":{"1min":0.65100000000000002,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55800000000000005,"15min":0.53800000000000003},"min":{"1min":0.42899999999999999,"5min":0.39900000000000002,"15min":0.27200000000000002},"max":{"1min":0.67700000000000005,"5min":0.73499999999999999,"15min":0.80200000000000005},"last":0.47799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.50600000000000001},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.29099999999999998},"max":{"1min":0.68700000000000006,"5min":1.304,"15min":1.304},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51500000000000001,"15min":0.52100000000000002},"min":{"1min":0.39900000000000002,"5min":0.376,"15min":0.25700000000000001},"max":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.753},"last":0.42599999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60299999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.379},"min":{"1min":0.30299999999999999,"5min":0.29199999999999998,"15min":0.20100000000000001},"max":{"1min":0.51000000000000001,"5min":0.59999999999999998,"15min":0.64200000000000002},"last":0.30499999999999999}]},{"osd":4,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.377,"15min":0.374},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":0.52800000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.28599999999999998,"15min":0.223},"max":{"1min":0.63600000000000001,"5min":0.68200000000000005,"15min":0.73799999999999999},"last":0.36499999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640506,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298920,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072920,"statfs":{"total":96636764160,"available":96330670080,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.378,"5min":0.27400000000000002,"15min":0.223},"max":{"1min":0.63,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.36699999999999999,"5min":0.33800000000000002,"15min":0.26400000000000001},"max":{"1min":0.60299999999999998,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.496}]},{"osd":1,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.25900000000000001},"max":{"1min":0.68700000000000006,"5min":0.71099999999999997,"15min":0.753},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53100000000000003},"min":{"1min":0.434,"5min":0.37,"15min":0.30499999999999999},"max":{"1min":0.71699999999999997,"5min":0.71999999999999997,"15min":0.872},"last":0.55500000000000005}]},{"osd":2,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.39400000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.53100000000000003},"min":{"1min":0.41499999999999998,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52600000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.42499999999999999,"15min":0.39000000000000001},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.67900000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40000000000000002,"15min":0.40899999999999997},"min":{"1min":0.28499999999999998,"5min":0.255,"15min":0.252},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.70299999999999996},"last":0.36399999999999999}]},{"osd":5,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.37,"15min":0.35899999999999999},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.21199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.38700000000000001},"min":{"1min":0.314,"5min":0.26300000000000001,"15min":0.22500000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303600,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068240,"statfs":{"total":96636764160,"available":96325877760,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.35199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.432},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.495,"15min":0.48899999999999999},"min":{"1min":0.40699999999999997,"5min":0.34699999999999998,"15min":0.33100000000000002},"max":{"1min":0.65100000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.47599999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.26600000000000001},"max":{"1min":0.63300000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.501},{"interface":"front","average":{"1min":0.499,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.36199999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.64300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60799999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.505,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33100000000000002},"max":{"1min":0.58499999999999996,"5min":0.873,"15min":0.873},"last":0.59999999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.35699999999999998,"15min":0.36399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.214},"max":{"1min":0.496,"5min":0.55000000000000004,"15min":0.624},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.372},"min":{"1min":0.29699999999999999,"5min":0.254,"15min":0.23100000000000001},"max":{"1min":0.53500000000000003,"5min":0.57499999999999996,"15min":0.67200000000000004},"last":0.61699999999999999}]},{"osd":5,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.28999999999999998,"5min":0.26100000000000001,"15min":0.20300000000000001},"max":{"1min":0.45000000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.437,"5min":0.43099999999999999,"15min":0.41099999999999998},"min":{"1min":0.36399999999999999,"5min":0.31900000000000001,"15min":0.23499999999999999},"max":{"1min":0.60599999999999998,"5min":0.621,"15min":0.70599999999999996},"last":0.58299999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640508,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303080,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068760,"statfs":{"total":96636764160,"available":96326410240,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.312,"15min":0.32200000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.156},"max":{"1min":0.46600000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.35799999999999998},"min":{"1min":0.22900000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.628},"last":0.33400000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.34899999999999998,"15min":0.37},"min":{"1min":0.27500000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.504,"5min":0.57899999999999996,"15min":0.63900000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.34200000000000003},"min":{"1min":0.22600000000000001,"5min":0.20499999999999999,"15min":0.17699999999999999},"max":{"1min":0.504,"5min":0.55800000000000005,"15min":0.625},"last":0.312}]},{"osd":3,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.54300000000000004},"min":{"1min":0.38700000000000001,"5min":0.36499999999999999,"15min":0.27000000000000002},"max":{"1min":0.74399999999999999,"5min":0.77000000000000002,"15min":0.77300000000000002},"last":0.373},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":0.73999999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.39900000000000002}]},{"osd":4,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75800000000000001,"5min":1.034,"15min":1.034},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.29399999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.41599999999999998}]},{"osd":5,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.28399999999999997},"max":{"1min":0.746,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.54300000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.32400000000000001},"max":{"1min":0.80000000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44400000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640508,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32200000000000001,"15min":0.32900000000000001},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":0.437,"5min":0.48499999999999999,"15min":0.70699999999999996},"last":0.377},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.214},"max":{"1min":0.51700000000000002,"5min":0.629,"15min":0.80000000000000004},"last":0.40999999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.35799999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.50800000000000001,"5min":0.52000000000000002,"15min":0.86599999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35999999999999999,"15min":0.36299999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.20499999999999999},"max":{"1min":0.47499999999999998,"5min":0.53600000000000003,"15min":0.82599999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.437,"5min":0.377,"15min":0.307},"max":{"1min":0.67400000000000004,"5min":0.84199999999999997,"15min":0.95699999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55500000000000005,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.83799999999999997,"15min":0.97899999999999998},"last":0.51100000000000001}]},{"osd":4,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.317},"max":{"1min":0.74099999999999999,"5min":0.82099999999999995,"15min":0.997},"last":0.495},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.34999999999999998},"max":{"1min":0.65100000000000002,"5min":0.85199999999999998,"15min":1.034},"last":0.52500000000000002}]},{"osd":5,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55700000000000005,"15min":0.57099999999999995},"min":{"1min":0.39400000000000002,"5min":0.38800000000000001,"15min":0.36699999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.88300000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56399999999999995,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":0.71399999999999997,"5min":0.81299999999999994,"15min":1.0960000000000001},"last":0.60799999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640507,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303600,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068240,"statfs":{"total":96636764160,"available":96325877760,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.30599999999999999,"15min":0.317},"min":{"1min":0.223,"5min":0.14799999999999999,"15min":0.13400000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.55500000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.34100000000000003},"min":{"1min":0.22500000000000001,"5min":0.18099999999999999,"15min":0.16400000000000001},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.64500000000000002},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.35499999999999998},"min":{"1min":0.21299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.48599999999999999,"5min":0.502,"15min":0.76800000000000002},"last":0.378},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.36799999999999999},"min":{"1min":0.28499999999999998,"5min":0.221,"15min":0.16},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.73499999999999999},"last":0.29899999999999999}]},{"osd":3,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55700000000000005,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.69499999999999995,"5min":1.177,"15min":1.177},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.53700000000000003},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47099999999999997}]},{"osd":4,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.35399999999999998,"15min":0.34399999999999997},"max":{"1min":0.87,"5min":1.127,"15min":1.127},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.69999999999999996,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Wed Sep 18 12:45:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.54000000000000004},"min":{"1min":0.41699999999999998,"5min":0.36099999999999999,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":1.141,"15min":1.141},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34100000000000003},"max":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.79000000000000004},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:46:08.744 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-18T12:30:56.227060+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=56, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:46:08.744 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-18T12:30:57.290984+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=30, tm_sec=57, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:46:08.744 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T12:25:24.910822+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=25, tm_sec=24, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=45, tm_sec=44, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:46:08.744 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T12:46:28.746 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:46:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.849+0000 7f6026446640 10 monclient: get_monmap_and_config 2024-09-18T12:46:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.849+0000 7f6026446640 10 monclient: build_initial_monmap 2024-09-18T12:46:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.849+0000 7f6026446640 10 monclient: monmap: 2024-09-18T12:46:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:46:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:46:28.851122+0000 2024-09-18T12:46:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:46:28.851122+0000 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:46:28.852 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 Processor -- start 2024-09-18T12:46:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 -- start start 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient: init 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient: _reopen_session rank -1 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6020096c60 0x7f602009a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): picked mon.noname-c con 0x7f6020096c60 addr v2:172.21.15.144:3300/0 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f6020097430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): picked mon.noname-b con 0x7f6020097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f602012e5f0 0x7f602012ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): picked mon.noname-a con 0x7f602012e5f0 addr v2:172.21.15.116:3300/0 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602009d2e0 con 0x7f602012e5f0 2024-09-18T12:46:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602009ac00 con 0x7f6020096c60 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602009d6b0 con 0x7f6020097030 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): _renew_subs 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.850+0000 7f6026446640 10 monclient(hunting): authenticate will time out at 2016.172881s 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f602012e5f0 0x7f602012ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f6020097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f602012e5f0 0x7f602012ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52586/0 (socket says 172.21.15.116:52586) 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f6020097430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41234/0 (socket says 172.21.15.116:41234) 2024-09-18T12:46:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 1 -- 172.21.15.116:0/2340933523 learned_addr learned my addr 172.21.15.116:0/2340933523 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 10 monclient(hunting): get_auth_request con 0x7f602012e5f0 auth_method 0 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): get_auth_request con 0x7f6020097030 auth_method 0 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 1 --2- 172.21.15.116:0/2340933523 >> v2:172.21.15.144:3300/0 conn(0x7f6020096c60 0x7f602009a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 10 monclient(hunting): get_auth_request con 0x7f6020096c60 auth_method 0 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient(hunting): handle_auth_done global_id 244238 payload 274 2024-09-18T12:46:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 -- 172.21.15.116:0/2340933523 >> v2:172.21.15.144:3300/0 conn(0x7f6020096c60 msgr2=0x7f602009a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 --2- 172.21.15.116:0/2340933523 >> v2:172.21.15.144:3300/0 conn(0x7f6020096c60 0x7f602009a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 -- 172.21.15.116:0/2340933523 >> v2:172.21.15.116:3300/0 conn(0x7f602012e5f0 msgr2=0x7f602012ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 --2- 172.21.15.116:0/2340933523 >> v2:172.21.15.116:3300/0 conn(0x7f602012e5f0 0x7f602012ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 monclient: found mon.noname-b 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f601f7fe640 1 -- 172.21.15.116:0/2340933523 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60200992f0 con 0x7f6020097030 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.851+0000 7f60249e3640 1 --2- 172.21.15.116:0/2340933523 >> v2:172.21.15.116:3300/0 conn(0x7f602012e5f0 0x7f602012ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601f7fe640 1 -- 172.21.15.116:0/2340933523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 msgr2=0x7f6020097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601f7fe640 1 --2- 172.21.15.116:0/2340933523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f6020097430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f60200637c0 tx=0x7f60140090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d398266185a26637 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 1 -- 172.21.15.116:0/2340933523 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014032440 con 0x7f6020097030 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: dump: 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: _finish_auth 0 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 1 -- 172.21.15.116:0/2340933523 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60140325a0 con 0x7f6020097030 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 5 monclient: authenticate success, global_id 244238 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 10 monclient: get_monmap_and_config success 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 1 -- 172.21.15.116:0/2340933523 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014032940 con 0x7f6020097030 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 4 set_mon_vals no callback set 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f601effd640 10 monclient: dump: 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 10 monclient: shutdown 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 1 -- 172.21.15.116:0/2340933523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 msgr2=0x7f6020097430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.854+0000 7f6026446640 1 --2- 172.21.15.116:0/2340933523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f6020097430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f60200637c0 tx=0x7f60140090f0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 -- 172.21.15.116:0/2340933523 shutdown_connections 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 --2- 172.21.15.116:0/2340933523 >> v2:172.21.15.116:3300/0 conn(0x7f602012e5f0 0x7f602012ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 --2- 172.21.15.116:0/2340933523 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f6020097430 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 --2- 172.21.15.116:0/2340933523 >> v2:172.21.15.144:3300/0 conn(0x7f6020096c60 0x7f602009a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 -- 172.21.15.116:0/2340933523 >> 172.21.15.116:0/2340933523 conn(0x7f60200672b0 msgr2=0x7f60200676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:46:28.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 -- 172.21.15.116:0/2340933523 shutdown_connections 2024-09-18T12:46:28.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 -- 172.21.15.116:0/2340933523 wait complete. 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 10 monclient: build_initial_monmap 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 10 monclient: monmap: 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:46:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.855+0000 7f6026446640 1 Processor -- start 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 1 -- start start 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient: init 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient: _reopen_session rank -1 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6020096c60 0x7f602013cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): picked mon.noname-a con 0x7f6020096c60 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f602012d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): picked mon.noname-c con 0x7f6020097030 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f602012e5f0 0x7f60201182f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): picked mon.noname-b con 0x7f602012e5f0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:46:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602011b910 con 0x7f6020096c60 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602011bed0 con 0x7f602012e5f0 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602013db00 con 0x7f6020097030 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): _renew_subs 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f6026446640 10 monclient(hunting): authenticate will time out at 2016.178619s 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6020096c60 0x7f602013cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f602012d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6020096c60 0x7f602013cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52596/0 (socket says 172.21.15.116:52596) 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 1 -- 172.21.15.116:0/3327503144 learned_addr learned my addr 172.21.15.116:0/3327503144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): get_auth_request con 0x7f6020097030 auth_method 0 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 10 monclient(hunting): get_auth_request con 0x7f6020096c60 auth_method 0 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f60249e3640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f602012e5f0 0x7f60201182f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f60249e3640 10 monclient(hunting): get_auth_request con 0x7f602012e5f0 auth_method 0 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f60249e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f60249e3640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f60249e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.856+0000 7f601f7fe640 10 monclient(hunting): handle_auth_done global_id 244241 payload 824 2024-09-18T12:46:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 1 -- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f602012e5f0 msgr2=0x7f60201182f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f602012e5f0 0x7f60201182f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 1 -- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6020096c60 msgr2=0x7f602013cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6020096c60 0x7f602013cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 10 monclient: _finish_hunting 0 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 1 monclient: found mon.noname-c 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601f7fe640 1 -- 172.21.15.116:0/3327503144 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6018009ae0 con 0x7f6020097030 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f601ffff640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6020096c60 0x7f602013cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.857+0000 7f60249e3640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f602012e5f0 0x7f60201182f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f601f7fe640 1 -- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 msgr2=0x7f602012d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f601f7fe640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f602012d7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6014031000 tx=0x7f6014035000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 1 -- 172.21.15.116:0/3327503144 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60140319c0 con 0x7f6020097030 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: dump: 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: _finish_auth 0 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 1 -- 172.21.15.116:0/3327503144 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6014031b40 con 0x7f6020097030 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6026446640 5 monclient: authenticate success, global_id 244241 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6026446640 10 monclient: _renew_subs 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6026446640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6014010810 con 0x7f6020097030 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 1 -- 172.21.15.116:0/3327503144 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601403e450 con 0x7f6020097030 2024-09-18T12:46:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6007fff640 10 monclient: dump: 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f601e7fc640 4 set_mon_vals no callback set 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6026446640 10 monclient: _renew_subs 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6026446640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.859+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602011ab50 con 0x7f6020097030 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f6007fff640 1 -- 172.21.15.116:0/3327503144 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f6014047450 con 0x7f6020097030 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f6026446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f6026446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f6026446640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6020119de0 con 0x7f6020097030 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f6007fff640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f600c038490 0x7f600c03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f6007fff640 1 -- 172.21.15.116:0/3327503144 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f601406c5b0 con 0x7f6020097030 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f601ffff640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f600c038490 0x7f600c03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.860+0000 7f601ffff640 10 monclient: get_auth_request con 0x7f600c038490 auth_method 0 2024-09-18T12:46:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.863+0000 7f601ffff640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f600c038490 0x7f600c03a940 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f60100114c0 tx=0x7f60100090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:28.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.864+0000 7f6007fff640 1 -- 172.21.15.116:0/3327503144 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f6014045b00 con 0x7f6020097030 2024-09-18T12:46:28.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.865+0000 7f6007fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:28.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:28.865+0000 7f6007fff640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:46:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.018+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 --> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f602004bf60 con 0x7f600c038490 2024-09-18T12:46:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.021+0000 7f6007fff640 1 -- 172.21.15.116:0/3327503144 <== mgr.4099 v2:172.21.15.144:6824/31687 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98296 (secure 0 0 0) 0x7f602004bf60 con 0x7f600c038490 2024-09-18T12:46:29.024 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-18T12:46:29.025 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-09-18T12:46:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f600c038490 msgr2=0x7f600c03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f600c038490 0x7f600c03a940 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f60100114c0 tx=0x7f60100090f0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 10 monclient: shutdown 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 msgr2=0x7f602012d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f602012d7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6014031000 tx=0x7f6014035000 comp rx=0 tx=0).stop 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 shutdown_connections 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f600c038490 0x7f600c03a940 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f602012e5f0 0x7f60201182f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f6020097030 0x7f602012d7e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 --2- 172.21.15.116:0/3327503144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6020096c60 0x7f602013cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 >> 172.21.15.116:0/3327503144 conn(0x7f60200672b0 msgr2=0x7f602012f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 shutdown_connections 2024-09-18T12:46:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.024+0000 7f6026446640 1 -- 172.21.15.116:0/3327503144 wait complete. 2024-09-18T12:46:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":895,"stamp":"2024-09-18T12:46:28.827464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1815368,"kb_used_data":10456,"kb_used_omap":9,"kb_used_meta":1804854,"kb_avail":564415672,"statfs":{"total":579820584960,"available":577961648128,"internally_reserved":0,"allocated":10706944,"data_stored":8512606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1848170512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002884"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":186,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:50.386562+0000","last_change":"2024-09-18T12:45:50.386527+0000","last_active":"2024-09-18T12:45:50.386562+0000","last_peered":"2024-09-18T12:45:50.386562+0000","last_clean":"2024-09-18T12:45:50.386562+0000","last_became_active":"2024-09-18T12:32:22.607844+0000","last_became_peered":"2024-09-18T12:32:22.607844+0000","last_unstale":"2024-09-18T12:45:50.386562+0000","last_undegraded":"2024-09-18T12:45:50.386562+0000","last_fullsized":"2024-09-18T12:45:50.386562+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:50.386445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:50.386445+0000","last_clean_scrub_stamp":"2024-09-18T12:45:50.386445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:54:37.397067+0000","scrub_duration":0.001865991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":95,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:59.048343+0000","last_change":"2024-09-18T12:45:59.048315+0000","last_active":"2024-09-18T12:45:59.048343+0000","last_peered":"2024-09-18T12:45:59.048343+0000","last_clean":"2024-09-18T12:45:59.048343+0000","last_became_active":"2024-09-18T12:32:22.601974+0000","last_became_peered":"2024-09-18T12:32:22.601974+0000","last_unstale":"2024-09-18T12:45:59.048343+0000","last_undegraded":"2024-09-18T12:45:59.048343+0000","last_fullsized":"2024-09-18T12:45:59.048343+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:59.048223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:59.048223+0000","last_clean_scrub_stamp":"2024-09-18T12:45:59.048223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:45:49.110046+0000","scrub_duration":0.001456457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":169,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:47.162252+0000","last_change":"2024-09-18T12:45:47.162218+0000","last_active":"2024-09-18T12:45:47.162252+0000","last_peered":"2024-09-18T12:45:47.162252+0000","last_clean":"2024-09-18T12:45:47.162252+0000","last_became_active":"2024-09-18T12:32:22.604956+0000","last_became_peered":"2024-09-18T12:32:22.604956+0000","last_unstale":"2024-09-18T12:45:47.162252+0000","last_undegraded":"2024-09-18T12:45:47.162252+0000","last_fullsized":"2024-09-18T12:45:47.162252+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:47.162148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:47.162148+0000","last_clean_scrub_stamp":"2024-09-18T12:45:47.162148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:54:07.071324+0000","scrub_duration":0.001412724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":94,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:46:03.175291+0000","last_change":"2024-09-18T12:46:03.175265+0000","last_active":"2024-09-18T12:46:03.175291+0000","last_peered":"2024-09-18T12:46:03.175291+0000","last_clean":"2024-09-18T12:46:03.175291+0000","last_became_active":"2024-09-18T12:32:22.604332+0000","last_became_peered":"2024-09-18T12:32:22.604332+0000","last_unstale":"2024-09-18T12:46:03.175291+0000","last_undegraded":"2024-09-18T12:46:03.175291+0000","last_fullsized":"2024-09-18T12:46:03.175291+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:46:03.175181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:46:03.175181+0000","last_clean_scrub_stamp":"2024-09-18T12:46:03.175181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:11:20.908182+0000","scrub_duration":0.0019839739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":138,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:46:05.229595+0000","last_change":"2024-09-18T12:46:05.229552+0000","last_active":"2024-09-18T12:46:05.229595+0000","last_peered":"2024-09-18T12:46:05.229595+0000","last_clean":"2024-09-18T12:46:05.229595+0000","last_became_active":"2024-09-18T12:32:23.803732+0000","last_became_peered":"2024-09-18T12:32:23.803732+0000","last_unstale":"2024-09-18T12:46:05.229595+0000","last_undegraded":"2024-09-18T12:46:05.229595+0000","last_fullsized":"2024-09-18T12:46:05.229595+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:46:05.229460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:46:05.229460+0000","last_clean_scrub_stamp":"2024-09-18T12:46:05.229460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:42:22.168596+0000","scrub_duration":0.001882407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":162,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.348872+0000","last_change":"2024-09-18T12:45:48.348851+0000","last_active":"2024-09-18T12:45:48.348872+0000","last_peered":"2024-09-18T12:45:48.348872+0000","last_clean":"2024-09-18T12:45:48.348872+0000","last_became_active":"2024-09-18T12:32:22.604577+0000","last_became_peered":"2024-09-18T12:32:22.604577+0000","last_unstale":"2024-09-18T12:45:48.348872+0000","last_undegraded":"2024-09-18T12:45:48.348872+0000","last_fullsized":"2024-09-18T12:45:48.348872+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.348781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.348781+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.348781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:32:33.271576+0000","scrub_duration":0.001644026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":96,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:53.308507+0000","last_change":"2024-09-18T12:45:53.308469+0000","last_active":"2024-09-18T12:45:53.308507+0000","last_peered":"2024-09-18T12:45:53.308507+0000","last_clean":"2024-09-18T12:45:53.308507+0000","last_became_active":"2024-09-18T12:32:22.608401+0000","last_became_peered":"2024-09-18T12:32:22.608401+0000","last_unstale":"2024-09-18T12:45:53.308507+0000","last_undegraded":"2024-09-18T12:45:53.308507+0000","last_fullsized":"2024-09-18T12:45:53.308507+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:53.308379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:53.308379+0000","last_clean_scrub_stamp":"2024-09-18T12:45:53.308379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:28:05.578973+0000","scrub_duration":0.0019799219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:59.426629+0000","last_change":"2024-09-18T12:45:59.426585+0000","last_active":"2024-09-18T12:45:59.426629+0000","last_peered":"2024-09-18T12:45:59.426629+0000","last_clean":"2024-09-18T12:45:59.426629+0000","last_became_active":"2024-09-18T12:32:22.603533+0000","last_became_peered":"2024-09-18T12:32:22.603533+0000","last_unstale":"2024-09-18T12:45:59.426629+0000","last_undegraded":"2024-09-18T12:45:59.426629+0000","last_fullsized":"2024-09-18T12:45:59.426629+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:59.426489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:59.426489+0000","last_clean_scrub_stamp":"2024-09-18T12:45:59.426489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:50:50.132667+0000","scrub_duration":0.0016270550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":121,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:56.012671+0000","last_change":"2024-09-18T12:45:56.012629+0000","last_active":"2024-09-18T12:45:56.012671+0000","last_peered":"2024-09-18T12:45:56.012671+0000","last_clean":"2024-09-18T12:45:56.012671+0000","last_became_active":"2024-09-18T12:32:22.607051+0000","last_became_peered":"2024-09-18T12:32:22.607051+0000","last_unstale":"2024-09-18T12:45:56.012671+0000","last_undegraded":"2024-09-18T12:45:56.012671+0000","last_fullsized":"2024-09-18T12:45:56.012671+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:56.012535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:56.012535+0000","last_clean_scrub_stamp":"2024-09-18T12:45:56.012535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:32:06.110764+0000","scrub_duration":0.001727463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":121,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:51.322062+0000","last_change":"2024-09-18T12:45:51.322033+0000","last_active":"2024-09-18T12:45:51.322062+0000","last_peered":"2024-09-18T12:45:51.322062+0000","last_clean":"2024-09-18T12:45:51.322062+0000","last_became_active":"2024-09-18T12:32:22.604295+0000","last_became_peered":"2024-09-18T12:32:22.604295+0000","last_unstale":"2024-09-18T12:45:51.322062+0000","last_undegraded":"2024-09-18T12:45:51.322062+0000","last_fullsized":"2024-09-18T12:45:51.322062+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:51.321933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:51.321933+0000","last_clean_scrub_stamp":"2024-09-18T12:45:51.321933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:28:05.462547+0000","scrub_duration":0.0017206050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":141,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.078232+0000","last_change":"2024-09-18T12:45:48.078179+0000","last_active":"2024-09-18T12:45:48.078232+0000","last_peered":"2024-09-18T12:45:48.078232+0000","last_clean":"2024-09-18T12:45:48.078232+0000","last_became_active":"2024-09-18T12:32:22.606557+0000","last_became_peered":"2024-09-18T12:32:22.606557+0000","last_unstale":"2024-09-18T12:45:48.078232+0000","last_undegraded":"2024-09-18T12:45:48.078232+0000","last_fullsized":"2024-09-18T12:45:48.078232+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.078076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.078076+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.078076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:50:42.309857+0000","scrub_duration":0.001689879,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.478217+0000","last_change":"2024-09-18T12:45:48.478195+0000","last_active":"2024-09-18T12:45:48.478217+0000","last_peered":"2024-09-18T12:45:48.478217+0000","last_clean":"2024-09-18T12:45:48.478217+0000","last_became_active":"2024-09-18T12:32:22.607698+0000","last_became_peered":"2024-09-18T12:32:22.607698+0000","last_unstale":"2024-09-18T12:45:48.478217+0000","last_undegraded":"2024-09-18T12:45:48.478217+0000","last_fullsized":"2024-09-18T12:45:48.478217+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.478124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.478124+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.478124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:34:05.636171+0000","scrub_duration":0.001238491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":101,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:58.456787+0000","last_change":"2024-09-18T12:45:58.456753+0000","last_active":"2024-09-18T12:45:58.456787+0000","last_peered":"2024-09-18T12:45:58.456787+0000","last_clean":"2024-09-18T12:45:58.456787+0000","last_became_active":"2024-09-18T12:32:22.606593+0000","last_became_peered":"2024-09-18T12:32:22.606593+0000","last_unstale":"2024-09-18T12:45:58.456787+0000","last_undegraded":"2024-09-18T12:45:58.456787+0000","last_fullsized":"2024-09-18T12:45:58.456787+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:58.456663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:58.456663+0000","last_clean_scrub_stamp":"2024-09-18T12:45:58.456663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:15:43.618669+0000","scrub_duration":0.001914282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":193,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:47.316897+0000","last_change":"2024-09-18T12:45:47.316877+0000","last_active":"2024-09-18T12:45:47.316897+0000","last_peered":"2024-09-18T12:45:47.316897+0000","last_clean":"2024-09-18T12:45:47.316897+0000","last_became_active":"2024-09-18T12:32:22.607839+0000","last_became_peered":"2024-09-18T12:32:22.607839+0000","last_unstale":"2024-09-18T12:45:47.316897+0000","last_undegraded":"2024-09-18T12:45:47.316897+0000","last_fullsized":"2024-09-18T12:45:47.316897+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:47.316811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:47.316811+0000","last_clean_scrub_stamp":"2024-09-18T12:45:47.316811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:08:49.061260+0000","scrub_duration":0.0011678649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":98,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:49.138377+0000","last_change":"2024-09-18T12:45:49.138346+0000","last_active":"2024-09-18T12:45:49.138377+0000","last_peered":"2024-09-18T12:45:49.138377+0000","last_clean":"2024-09-18T12:45:49.138377+0000","last_became_active":"2024-09-18T12:32:23.599235+0000","last_became_peered":"2024-09-18T12:32:23.599235+0000","last_unstale":"2024-09-18T12:45:49.138377+0000","last_undegraded":"2024-09-18T12:45:49.138377+0000","last_fullsized":"2024-09-18T12:45:49.138377+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:49.138269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:49.138269+0000","last_clean_scrub_stamp":"2024-09-18T12:45:49.138269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:50:20.823661+0000","scrub_duration":0.001819174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:57.469596+0000","last_change":"2024-09-18T12:45:57.469554+0000","last_active":"2024-09-18T12:45:57.469596+0000","last_peered":"2024-09-18T12:45:57.469596+0000","last_clean":"2024-09-18T12:45:57.469596+0000","last_became_active":"2024-09-18T12:32:22.599859+0000","last_became_peered":"2024-09-18T12:32:22.599859+0000","last_unstale":"2024-09-18T12:45:57.469596+0000","last_undegraded":"2024-09-18T12:45:57.469596+0000","last_fullsized":"2024-09-18T12:45:57.469596+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:57.469460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:57.469460+0000","last_clean_scrub_stamp":"2024-09-18T12:45:57.469460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:01:36.232281+0000","scrub_duration":0.0019671229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"267'74","reported_seq":351,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:58.375793+0000","last_change":"2024-09-18T12:45:58.375751+0000","last_active":"2024-09-18T12:45:58.375793+0000","last_peered":"2024-09-18T12:45:58.375793+0000","last_clean":"2024-09-18T12:45:58.375793+0000","last_became_active":"2024-09-18T12:32:22.604820+0000","last_became_peered":"2024-09-18T12:32:22.604820+0000","last_unstale":"2024-09-18T12:45:58.375793+0000","last_undegraded":"2024-09-18T12:45:58.375793+0000","last_fullsized":"2024-09-18T12:45:58.375793+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'74","last_scrub_stamp":"2024-09-18T12:45:58.375649+0000","last_deep_scrub":"267'74","last_deep_scrub_stamp":"2024-09-18T12:45:58.375649+0000","last_clean_scrub_stamp":"2024-09-18T12:45:58.375649+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:34:10.370999+0000","scrub_duration":0.011783999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"267'2","reported_seq":278,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:50.070267+0000","last_change":"2024-09-18T12:45:50.070232+0000","last_active":"2024-09-18T12:45:50.070267+0000","last_peered":"2024-09-18T12:45:50.070267+0000","last_clean":"2024-09-18T12:45:50.070267+0000","last_became_active":"2024-09-18T12:32:22.604471+0000","last_became_peered":"2024-09-18T12:32:22.604471+0000","last_unstale":"2024-09-18T12:45:50.070267+0000","last_undegraded":"2024-09-18T12:45:50.070267+0000","last_fullsized":"2024-09-18T12:45:50.070267+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'2","last_scrub_stamp":"2024-09-18T12:45:50.070083+0000","last_deep_scrub":"267'2","last_deep_scrub_stamp":"2024-09-18T12:45:50.070083+0000","last_clean_scrub_stamp":"2024-09-18T12:45:50.070083+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:59:44.447590+0000","scrub_duration":0.0032318070000000002,"objects_trimmed":0,"snaptrim_duration":0.000409233,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":300,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.296980+0000","last_change":"2024-09-18T12:45:48.296944+0000","last_active":"2024-09-18T12:45:48.296980+0000","last_peered":"2024-09-18T12:45:48.296980+0000","last_clean":"2024-09-18T12:45:48.296980+0000","last_became_active":"2024-09-18T12:32:22.607827+0000","last_became_peered":"2024-09-18T12:32:22.607827+0000","last_unstale":"2024-09-18T12:45:48.296980+0000","last_undegraded":"2024-09-18T12:45:48.296980+0000","last_fullsized":"2024-09-18T12:45:48.296980+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.296841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.296841+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.296841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:41:06.523245+0000","scrub_duration":0.0016207960000000001,"objects_trimmed":0,"snaptrim_duration":0.00050680600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":98,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:48.141219+0000","last_change":"2024-09-18T12:45:48.141192+0000","last_active":"2024-09-18T12:45:48.141219+0000","last_peered":"2024-09-18T12:45:48.141219+0000","last_clean":"2024-09-18T12:45:48.141219+0000","last_became_active":"2024-09-18T12:32:22.603727+0000","last_became_peered":"2024-09-18T12:32:22.603727+0000","last_unstale":"2024-09-18T12:45:48.141219+0000","last_undegraded":"2024-09-18T12:45:48.141219+0000","last_fullsized":"2024-09-18T12:45:48.141219+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:48.141119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:48.141119+0000","last_clean_scrub_stamp":"2024-09-18T12:45:48.141119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:35:09.560138+0000","scrub_duration":0.0012581960000000001,"objects_trimmed":0,"snaptrim_duration":0.00031917499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"267'3","reported_seq":280,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:51.065560+0000","last_change":"2024-09-18T12:45:51.065532+0000","last_active":"2024-09-18T12:45:51.065560+0000","last_peered":"2024-09-18T12:45:51.065560+0000","last_clean":"2024-09-18T12:45:51.065560+0000","last_became_active":"2024-09-18T12:32:22.608226+0000","last_became_peered":"2024-09-18T12:32:22.608226+0000","last_unstale":"2024-09-18T12:45:51.065560+0000","last_undegraded":"2024-09-18T12:45:51.065560+0000","last_fullsized":"2024-09-18T12:45:51.065560+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'3","last_scrub_stamp":"2024-09-18T12:45:51.065445+0000","last_deep_scrub":"267'3","last_deep_scrub_stamp":"2024-09-18T12:45:51.065445+0000","last_clean_scrub_stamp":"2024-09-18T12:45:51.065445+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:22:43.174515+0000","scrub_duration":0.003296944,"objects_trimmed":0,"snaptrim_duration":0.00083113200000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":275,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:46:04.441387+0000","last_change":"2024-09-18T12:46:04.441361+0000","last_active":"2024-09-18T12:46:04.441387+0000","last_peered":"2024-09-18T12:46:04.441387+0000","last_clean":"2024-09-18T12:46:04.441387+0000","last_became_active":"2024-09-18T12:32:23.598179+0000","last_became_peered":"2024-09-18T12:32:23.598179+0000","last_unstale":"2024-09-18T12:46:04.441387+0000","last_undegraded":"2024-09-18T12:46:04.441387+0000","last_fullsized":"2024-09-18T12:46:04.441387+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:46:04.441278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:46:04.441278+0000","last_clean_scrub_stamp":"2024-09-18T12:46:04.441278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:59:30.874329+0000","scrub_duration":0.001415342,"objects_trimmed":0,"snaptrim_duration":0.000409064,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:46:00.348007+0000","last_change":"2024-09-18T12:46:00.347979+0000","last_active":"2024-09-18T12:46:00.348007+0000","last_peered":"2024-09-18T12:46:00.348007+0000","last_clean":"2024-09-18T12:46:00.348007+0000","last_became_active":"2024-09-18T12:32:22.605190+0000","last_became_peered":"2024-09-18T12:32:22.605190+0000","last_unstale":"2024-09-18T12:46:00.348007+0000","last_undegraded":"2024-09-18T12:46:00.348007+0000","last_fullsized":"2024-09-18T12:46:00.348007+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:46:00.347873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:46:00.347873+0000","last_clean_scrub_stamp":"2024-09-18T12:46:00.347873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:04:45.509794+0000","scrub_duration":0.001858186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:52.462616+0000","last_change":"2024-09-18T12:45:52.462566+0000","last_active":"2024-09-18T12:45:52.462616+0000","last_peered":"2024-09-18T12:45:52.462616+0000","last_clean":"2024-09-18T12:45:52.462616+0000","last_became_active":"2024-09-18T12:32:22.603556+0000","last_became_peered":"2024-09-18T12:32:22.603556+0000","last_unstale":"2024-09-18T12:45:52.462616+0000","last_undegraded":"2024-09-18T12:45:52.462616+0000","last_fullsized":"2024-09-18T12:45:52.462616+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:52.462462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:52.462462+0000","last_clean_scrub_stamp":"2024-09-18T12:45:52.462462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:07:15.250417+0000","scrub_duration":0.0020054740000000001,"objects_trimmed":0,"snaptrim_duration":0.00049494800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-18T12:45:50.280086+0000","last_change":"2024-09-18T12:45:50.280056+0000","last_active":"2024-09-18T12:45:50.280086+0000","last_peered":"2024-09-18T12:45:50.280086+0000","last_clean":"2024-09-18T12:45:50.280086+0000","last_became_active":"2024-09-18T12:32:22.601418+0000","last_became_peered":"2024-09-18T12:32:22.601418+0000","last_unstale":"2024-09-18T12:45:50.280086+0000","last_undegraded":"2024-09-18T12:45:50.280086+0000","last_fullsized":"2024-09-18T12:45:50.280086+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:45:50.279963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:45:50.279963+0000","last_clean_scrub_stamp":"2024-09-18T12:45:50.279963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:03:26.802247+0000","scrub_duration":0.0013355419999999999,"objects_trimmed":0,"snaptrim_duration":0.00032647299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640511,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.19700000000000001},"max":{"1min":0.59099999999999997,"5min":0.65700000000000003,"15min":0.69099999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.50700000000000001,"15min":0.48399999999999999},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.51800000000000002},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.23999999999999999},"max":{"1min":0.65100000000000002,"5min":0.71499999999999997,"15min":0.76800000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55800000000000005,"15min":0.53800000000000003},"min":{"1min":0.42899999999999999,"5min":0.39900000000000002,"15min":0.27200000000000002},"max":{"1min":0.67700000000000005,"5min":0.73499999999999999,"15min":0.80200000000000005},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.50600000000000001},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.29099999999999998},"max":{"1min":0.68700000000000006,"5min":1.304,"15min":1.304},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51500000000000001,"15min":0.52100000000000002},"min":{"1min":0.39900000000000002,"5min":0.376,"15min":0.25700000000000001},"max":{"1min":0.61399999999999999,"5min":0.71899999999999997,"15min":0.753},"last":0.51000000000000001}]},{"osd":3,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60299999999999998},"last":0.314},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.379},"min":{"1min":0.30299999999999999,"5min":0.29199999999999998,"15min":0.20100000000000001},"max":{"1min":0.51000000000000001,"5min":0.59999999999999998,"15min":0.64200000000000002},"last":0.32700000000000001}]},{"osd":4,"last update":"Wed Sep 18 12:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.377,"15min":0.374},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":0.52800000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.28599999999999998,"15min":0.223},"max":{"1min":0.63600000000000001,"5min":0.68200000000000005,"15min":0.73799999999999999},"last":0.34699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640511,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298920,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072920,"statfs":{"total":96636764160,"available":96330670080,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.378,"5min":0.27400000000000002,"15min":0.223},"max":{"1min":0.63,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.36699999999999999,"5min":0.33800000000000002,"15min":0.26400000000000001},"max":{"1min":0.60299999999999998,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51900000000000002,"15min":0.51400000000000001},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.25900000000000001},"max":{"1min":0.68700000000000006,"5min":0.71099999999999997,"15min":0.753},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53100000000000003},"min":{"1min":0.434,"5min":0.37,"15min":0.30499999999999999},"max":{"1min":0.71699999999999997,"5min":0.71999999999999997,"15min":0.872},"last":0.503}]},{"osd":2,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.39400000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.53100000000000003},"min":{"1min":0.41499999999999998,"5min":0.29999999999999999,"15min":0.29499999999999998},"max":{"1min":0.70399999999999996,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52900000000000003}]},{"osd":3,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.42499999999999999,"15min":0.39000000000000001},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.189},"max":{"1min":0.60899999999999999,"5min":0.65700000000000003,"15min":0.67900000000000005},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40000000000000002,"15min":0.40899999999999997},"min":{"1min":0.28499999999999998,"5min":0.255,"15min":0.252},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.70299999999999996},"last":0.35299999999999998}]},{"osd":5,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.37,"15min":0.35899999999999999},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.21199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.38700000000000001},"min":{"1min":0.314,"5min":0.26300000000000001,"15min":0.22500000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.32900000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640514,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303600,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068240,"statfs":{"total":96636764160,"available":96325877760,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.35199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.495,"15min":0.48899999999999999},"min":{"1min":0.40699999999999997,"5min":0.34699999999999998,"15min":0.33100000000000002},"max":{"1min":0.65100000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.26600000000000001},"max":{"1min":0.63300000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.36199999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.64300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.60799999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.505,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33100000000000002},"max":{"1min":0.58499999999999996,"5min":0.873,"15min":0.873},"last":0.50800000000000001}]},{"osd":4,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.35699999999999998,"15min":0.36399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.214},"max":{"1min":0.496,"5min":0.55000000000000004,"15min":0.624},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.372},"min":{"1min":0.29699999999999999,"5min":0.254,"15min":0.23100000000000001},"max":{"1min":0.53500000000000003,"5min":0.57499999999999996,"15min":0.67200000000000004},"last":0.35699999999999998}]},{"osd":5,"last update":"Wed Sep 18 12:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36799999999999999},"min":{"1min":0.28999999999999998,"5min":0.26100000000000001,"15min":0.20300000000000001},"max":{"1min":0.45000000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.43099999999999999,"15min":0.41099999999999998},"min":{"1min":0.36399999999999999,"5min":0.31900000000000001,"15min":0.23499999999999999},"max":{"1min":0.60599999999999998,"5min":0.621,"15min":0.70599999999999996},"last":0.39700000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303080,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068760,"statfs":{"total":96636764160,"available":96326410240,"internally_reserved":0,"allocated":1605632,"data_stored":1242452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.312,"15min":0.32200000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.156},"max":{"1min":0.46600000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.35799999999999998},"min":{"1min":0.22900000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.628},"last":0.29999999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.34899999999999998,"15min":0.37},"min":{"1min":0.27500000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.504,"5min":0.57899999999999996,"15min":0.63900000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.34200000000000003},"min":{"1min":0.22600000000000001,"5min":0.20499999999999999,"15min":0.17699999999999999},"max":{"1min":0.504,"5min":0.55800000000000005,"15min":0.625},"last":0.27400000000000002}]},{"osd":3,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.54300000000000004},"min":{"1min":0.38700000000000001,"5min":0.36499999999999999,"15min":0.27000000000000002},"max":{"1min":0.74399999999999999,"5min":0.77000000000000002,"15min":0.77300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":0.73999999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75800000000000001,"5min":1.034,"15min":1.034},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.29399999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46999999999999997}]},{"osd":5,"last update":"Wed Sep 18 12:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.28399999999999997},"max":{"1min":0.746,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.54300000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.32400000000000001},"max":{"1min":0.80000000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56399999999999995}]}]},{"osd":1,"up_from":11,"seq":47244640513,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1609728,"data_stored":1242471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32200000000000001,"15min":0.32900000000000001},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.17499999999999999},"max":{"1min":0.437,"5min":0.48499999999999999,"15min":0.70699999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.214},"max":{"1min":0.51700000000000002,"5min":0.629,"15min":0.80000000000000004},"last":0.46500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.35799999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.50800000000000001,"5min":0.52000000000000002,"15min":0.86599999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35999999999999999,"15min":0.36299999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.20499999999999999},"max":{"1min":0.47499999999999998,"5min":0.53600000000000003,"15min":0.82599999999999996},"last":0.45000000000000001}]},{"osd":3,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55500000000000005,"15min":0.54500000000000004},"min":{"1min":0.437,"5min":0.377,"15min":0.307},"max":{"1min":0.67400000000000004,"5min":0.84199999999999997,"15min":0.95699999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55500000000000005,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.83799999999999997,"15min":0.97899999999999998},"last":0.65600000000000003}]},{"osd":4,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.317},"max":{"1min":0.74099999999999999,"5min":0.82099999999999995,"15min":0.997},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.34999999999999998},"max":{"1min":0.65100000000000002,"5min":0.85199999999999998,"15min":1.034},"last":0.79700000000000004}]},{"osd":5,"last update":"Wed Sep 18 12:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55700000000000005,"15min":0.57099999999999995},"min":{"1min":0.39400000000000002,"5min":0.38800000000000001,"15min":0.36699999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.88300000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56399999999999995,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":0.71399999999999997,"5min":0.81299999999999994,"15min":1.0960000000000001},"last":0.69799999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640512,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303600,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068240,"statfs":{"total":96636764160,"available":96325877760,"internally_reserved":0,"allocated":2138112,"data_stored":1771380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.316},"min":{"1min":0.23100000000000001,"5min":0.14799999999999999,"15min":0.13400000000000001},"max":{"1min":0.44,"5min":0.46000000000000002,"15min":0.55500000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33400000000000002,"15min":0.34100000000000003},"min":{"1min":0.254,"5min":0.18099999999999999,"15min":0.16400000000000001},"max":{"1min":0.434,"5min":0.51000000000000001,"15min":0.64500000000000002},"last":0.378}]},{"osd":2,"last update":"Wed Sep 18 12:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.32200000000000001,"15min":0.35499999999999998},"min":{"1min":0.246,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.46400000000000002,"5min":0.502,"15min":0.76800000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.379,"15min":0.36899999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.16},"max":{"1min":0.44900000000000001,"5min":0.61199999999999999,"15min":0.73499999999999999},"last":0.35499999999999998}]},{"osd":3,"last update":"Wed Sep 18 12:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55400000000000005,"15min":0.54500000000000004},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.30599999999999999},"max":{"1min":0.67600000000000005,"5min":1.177,"15min":1.177},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55200000000000005,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.33600000000000002,"15min":0.317},"max":{"1min":0.73999999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.51500000000000001}]},{"osd":4,"last update":"Wed Sep 18 12:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56100000000000005,"15min":0.54200000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":1.127,"15min":1.127},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.54200000000000004},"min":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33000000000000002},"max":{"1min":0.65200000000000002,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Wed Sep 18 12:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.42699999999999999,"5min":0.36099999999999999,"15min":0.31},"max":{"1min":0.73799999999999999,"5min":1.141,"15min":1.141},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.34100000000000003},"max":{"1min":0.65700000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:46:29.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T12:46:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.129+0000 7f3b48eef640 10 monclient: get_monmap_and_config 2024-09-18T12:46:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.129+0000 7f3b48eef640 10 monclient: build_initial_monmap 2024-09-18T12:46:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.129+0000 7f3b48eef640 10 monclient: monmap: 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:46:29.131117+0000 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:46:29.131117+0000 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:0: v1:172.21.15.116:6789/0 mon.noname-a-legacy 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-b 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:3: v2:172.21.15.116:3300/0 mon.noname-a 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.129+0000 7f3b48eef640 1 Processor -- start 2024-09-18T12:46:29.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 1 -- start start 2024-09-18T12:46:29.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient: init 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b440988b0 0x7f3b44099ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient(hunting): picked mon.noname-c con 0x7f3b440988b0 addr v2:172.21.15.144:3300/0 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient(hunting): picked mon.noname-b con 0x7f3b4409d2a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b4406f350 addr v1:172.21.15.144:6789/0 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b44099ac0 con 0x7f3b4406f350 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4409d060 con 0x7f3b440988b0 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b44099770 con 0x7f3b4409d2a0 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b48eef640 10 monclient(hunting): authenticate will time out at 2016.452715s 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41254/0 (socket says 172.21.15.116:41254) 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 1 -- 172.21.15.116:0/648175930 learned_addr learned my addr 172.21.15.116:0/648175930 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 10 monclient(hunting): get_auth_request con 0x7f3b4409d2a0 auth_method 0 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.130+0000 7f3b41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 10 monclient(hunting): handle_auth_done global_id 244244 payload 274 2024-09-18T12:46:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b4259c640 1 --2- 172.21.15.116:0/648175930 >> v2:172.21.15.144:3300/0 conn(0x7f3b440988b0 0x7f3b44099ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 1 -- 172.21.15.116:0/648175930 >> v2:172.21.15.144:3300/0 conn(0x7f3b440988b0 msgr2=0x7f3b44099ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 1 --2- 172.21.15.116:0/648175930 >> v2:172.21.15.144:3300/0 conn(0x7f3b440988b0 0x7f3b44099ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 1 -- 172.21.15.116:0/648175930 >> v1:172.21.15.144:6789/0 conn(0x7f3b4406f350 legacy=0x7f3b44096d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 10 monclient: _finish_hunting 0 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 1 monclient: found mon.noname-b 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.116:3301/0 2024-09-18T12:46:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.131+0000 7f3b41d9b640 1 -- 172.21.15.116:0/648175930 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b44097480 con 0x7f3b4409d2a0 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.133+0000 7f3b41d9b640 1 -- 172.21.15.116:0/648175930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 msgr2=0x7f3b4406ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.133+0000 7f3b41d9b640 1 --2- 172.21.15.116:0/648175930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406ee10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009910 tx=0x7f3b2c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37082da217b7d293 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 1 -- 172.21.15.116:0/648175930 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c032440 con 0x7f3b4409d2a0 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: dump: 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:29.143 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: _finish_auth 0 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 1 -- 172.21.15.116:0/648175930 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c0325a0 con 0x7f3b4409d2a0 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 5 monclient: authenticate success, global_id 244244 2024-09-18T12:46:29.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 10 monclient: get_monmap_and_config success 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 4 set_mon_vals no callback set 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 1 -- 172.21.15.116:0/648175930 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c032940 con 0x7f3b4409d2a0 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b4159a640 10 monclient: dump: 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 10 monclient: shutdown 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 -- 172.21.15.116:0/648175930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 msgr2=0x7f3b4406ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 --2- 172.21.15.116:0/648175930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406ee10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009910 tx=0x7f3b2c0091d0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 -- 172.21.15.116:0/648175930 shutdown_connections 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 --2- 172.21.15.116:0/648175930 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406ee10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 --2- 172.21.15.116:0/648175930 >> v2:172.21.15.144:3300/0 conn(0x7f3b440988b0 0x7f3b44099ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 -- 172.21.15.116:0/648175930 >> 172.21.15.116:0/648175930 conn(0x7f3b440a0f50 msgr2=0x7f3b440a1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 -- 172.21.15.116:0/648175930 shutdown_connections 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.134+0000 7f3b48eef640 1 -- 172.21.15.116:0/648175930 wait complete. 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient: build_initial_monmap 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient: monmap: 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 0 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 0.000000 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:created 0.000000 2024-09-18T12:46:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 0 (unknown) 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.noname-a 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.noname-c 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 Processor -- start 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 -- start start 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient: init 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient: _reopen_session rank -1 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4406f350 0x7f3b44130010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): picked mon.noname-a con 0x7f3b4406f350 addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b440988b0 0x7f3b4406bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): picked mon.noname-b con 0x7f3b440988b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): picked mon.noname-c con 0x7f3b4409d2a0 addr [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b44117f00 con 0x7f3b4406f350 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b44098da0 con 0x7f3b440988b0 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): start opening mon connection 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b441306d0 con 0x7f3b4409d2a0 2024-09-18T12:46:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): _renew_subs 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.135+0000 7f3b48eef640 10 monclient(hunting): authenticate will time out at 2016.457945s 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4406f350 0x7f3b44130010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:41258/0 (socket says 172.21.15.116:41258) 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4406f350 0x7f3b44130010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52612/0 (socket says 172.21.15.116:52612) 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 -- 172.21.15.116:0/4113514943 learned_addr learned my addr 172.21.15.116:0/4113514943 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): get_auth_request con 0x7f3b4409d2a0 auth_method 0 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 10 monclient(hunting): get_auth_request con 0x7f3b4406f350 auth_method 0 2024-09-18T12:46:29.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 10 monclient(hunting): _init_auth method 2 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b41d9b640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b440988b0 0x7f3b4406bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient(hunting): handle_auth_done global_id 244247 payload 824 2024-09-18T12:46:29.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 -- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b440988b0 msgr2=0x7f3b4406bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b440988b0 0x7f3b4406bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 -- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4406f350 msgr2=0x7f3b44130010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4406f350 0x7f3b44130010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient: _finish_hunting 0 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 monclient: found mon.noname-c 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.116:3301/0 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b42d9d640 1 -- 172.21.15.116:0/4113514943 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009450 con 0x7f3b4409d2a0 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b41d9b640 10 monclient: get_auth_request con 0x7f3b440988b0 auth_method 0 2024-09-18T12:46:29.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b4259c640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4406f350 0x7f3b44130010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.136+0000 7f3b41d9b640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b440988b0 0x7f3b4406bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b42d9d640 1 -- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 msgr2=0x7f3b4406c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b42d9d640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406c450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00b820 tx=0x7f3b3c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c00c770 con 0x7f3b4409d2a0 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: dump: 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.153 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:29.154 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: _finish_auth 0 2024-09-18T12:46:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T12:46:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c00c8f0 con 0x7f3b4409d2a0 2024-09-18T12:46:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b48eef640 5 monclient: authenticate success, global_id 244247 2024-09-18T12:46:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-18T12:46:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b48eef640 10 monclient: _renew_subs 2024-09-18T12:46:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c00ccb0 con 0x7f3b4409d2a0 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b48eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4406e200 con 0x7f3b4409d2a0 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b3a7fc640 10 monclient: dump: 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:epoch 1 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:fsid 2f704457-11dd-41ce-9561-691d67058c00 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:last_changed 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:created 2024-09-18T12:24:46.763770+0000 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:min_mon_release 17 (quincy) 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:election_strategy: 1 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-09-18T12:46:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b40d99640 4 set_mon_vals no callback set 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b48eef640 10 monclient: _renew_subs 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b48eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.138+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b44118850 con 0x7f3b4409d2a0 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.139+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 44681+0+0 (secure 0 0 0) 0x7f3b3c012070 con 0x7f3b4409d2a0 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.139+0000 7f3b48eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.139+0000 7f3b48eef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.139+0000 7f3b48eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.139+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b44118240 con 0x7f3b4409d2a0 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.139+0000 7f3b3a7fc640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b24038490 0x7f3b2403a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.139+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(267..267 src has 242..267) v4 ==== 7581+0+0 (secure 0 0 0) 0x7f3b3c04bbc0 con 0x7f3b4409d2a0 2024-09-18T12:46:29.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.140+0000 7f3b4259c640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b24038490 0x7f3b2403a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T12:46:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.140+0000 7f3b4259c640 10 monclient: get_auth_request con 0x7f3b24038490 auth_method 0 2024-09-18T12:46:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.142+0000 7f3b4259c640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b24038490 0x7f3b2403a940 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400afd0 tx=0x7f3b34008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T12:46:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.144+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176153 (secure 0 0 0) 0x7f3b44118240 con 0x7f3b4409d2a0 2024-09-18T12:46:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.144+0000 7f3b3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T12:46:29.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.144+0000 7f3b3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-18T12:46:29.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.299+0000 7f3b48eef640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T12:46:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.299+0000 7f3b48eef640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T12:46:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.299+0000 7f3b48eef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.116:3301/0 2024-09-18T12:46:29.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.299+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f3b44118240 con 0x7f3b4409d2a0 2024-09-18T12:46:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.304+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3b3c009980 con 0x7f3b4409d2a0 2024-09-18T12:46:29.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.304+0000 7f3b3a7fc640 10 monclient: handle_config config(1 keys) v1 2024-09-18T12:46:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.304+0000 7f3b40d99640 4 set_mon_vals no callback set 2024-09-18T12:46:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.304+0000 7f3b40d99640 10 set_mon_vals mon_health_to_clog = false 2024-09-18T12:46:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.304+0000 7f3b3a7fc640 1 -- 172.21.15.116:0/4113514943 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f3b44118240 con 0x7f3b4409d2a0 2024-09-18T12:46:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.304+0000 7f3b3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T12:46:29.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.304+0000 7f3b3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-18T12:46:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b24038490 msgr2=0x7f3b2403a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b24038490 0x7f3b2403a940 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400afd0 tx=0x7f3b34008040 comp rx=0 tx=0).stop 2024-09-18T12:46:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 10 monclient: shutdown 2024-09-18T12:46:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T12:46:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 msgr2=0x7f3b4406c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T12:46:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406c450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00b820 tx=0x7f3b3c009840 comp rx=0 tx=0).stop 2024-09-18T12:46:29.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 shutdown_connections 2024-09-18T12:46:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:6824/31687,v1:172.21.15.144:6825/31687] conn(0x7f3b24038490 0x7f3b2403a940 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f3b4409d2a0 0x7f3b4406c450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b440988b0 0x7f3b4406bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 --2- 172.21.15.116:0/4113514943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4406f350 0x7f3b44130010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T12:46:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 >> 172.21.15.116:0/4113514943 conn(0x7f3b440a0f50 msgr2=0x7f3b44135be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T12:46:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 shutdown_connections 2024-09-18T12:46:29.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-18T12:46:29.306+0000 7f3b48eef640 1 -- 172.21.15.116:0/4113514943 wait complete. 2024-09-18T12:46:29.318 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T12:46:29.318 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T12:46:29.319 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T12:46:29.319 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.350 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T12:46:29.350 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T12:46:29.350 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.415 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T12:46:29.415 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T12:46:29.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.485 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T12:46:29.485 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T12:46:29.485 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.518 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T12:46:29.518 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-18T12:46:29.518 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.585 INFO:tasks.ceph.osd.4:Stopped 2024-09-18T12:46:29.586 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-18T12:46:29.586 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.653 INFO:tasks.ceph.osd.5:Stopped 2024-09-18T12:46:29.654 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T12:46:29.654 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T12:46:29.654 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.722 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T12:46:29.722 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T12:46:29.722 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:46:29.722 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.736 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:46:29.737 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:46:29.737 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.788 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:46:29.788 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:46:29.788 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:46:29.802 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:46:29.802 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T12:46:29.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-18T12:46:29.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:46:29.864 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T12:46:30.049 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:46:30.049 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T12:46:30.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:46:30.140 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T12:46:30.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:46:30.196 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-18T12:46:30.312 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:46:30.312 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-18T12:46:30.382 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:46:30.383 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-18T12:46:30.440 INFO:tasks.ceph:Archiving mon data... 2024-09-18T12:46:30.442 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/data/mon.a.tgz 2024-09-18T12:46:30.442 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-09-18T12:46:30.458 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.u8qszcll5n 2024-09-18T12:46:30.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.u8qszcll5n 2024-09-18T12:46:30.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.u8qszcll5n 2024-09-18T12:46:30.729 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.u8qszcll5n is 812KB 2024-09-18T12:46:30.803 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.u8qszcll5n 2024-09-18T12:46:30.818 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/data/mon.c.tgz 2024-09-18T12:46:30.818 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-09-18T12:46:30.872 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.JZ2vC17A5j 2024-09-18T12:46:30.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.JZ2vC17A5j 2024-09-18T12:46:31.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.JZ2vC17A5j 2024-09-18T12:46:31.133 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.JZ2vC17A5j is 814KB 2024-09-18T12:46:31.235 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.JZ2vC17A5j 2024-09-18T12:46:31.250 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/data/mon.b.tgz 2024-09-18T12:46:31.251 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-09-18T12:46:31.269 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.IvMU39xZyQ 2024-09-18T12:46:31.269 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.IvMU39xZyQ 2024-09-18T12:46:31.467 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.IvMU39xZyQ 2024-09-18T12:46:31.550 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.IvMU39xZyQ is 812KB 2024-09-18T12:46:31.652 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.IvMU39xZyQ 2024-09-18T12:46:31.666 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T12:46:31.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T12:46:31.669 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T12:46:31.730 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T12:46:31.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T12:46:31.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T12:46:31.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T12:46:31.796 INFO:teuthology.orchestra.run.smithi144.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T12:46:31.801 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T12:46:31.802 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/remote/smithi116/crash 2024-09-18T12:46:31.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T12:46:31.833 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/remote/smithi144/crash 2024-09-18T12:46:31.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T12:46:31.865 INFO:tasks.ceph:Compressing logs... 2024-09-18T12:46:31.865 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T12:46:31.878 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T12:46:31.901 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-18T12:46:31.901 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51572.log 2024-09-18T12:46:31.901 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51404.log 2024-09-18T12:46:32.178 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42130.log 2024-09-18T12:46:32.178 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37466.log 2024-09-18T12:46:32.178 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42677.log 2024-09-18T12:46:32.178 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38092.log 2024-09-18T12:46:32.178 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42130.log.gz 2024-09-18T12:46:32.178 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43069.log 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37466.log.gz 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42995.log 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.43069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35533.log 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38092.log.gz 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42677.log.gz 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42995.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43069.log.gz 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35248.log 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42995.log.gz 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39180.log 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35533.log.gz 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36228.log 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35248.log.gz 2024-09-18T12:46:32.179 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35132.log 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39701.log 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39180.log.gz 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37943.log 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36228.log.gz 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36720.log 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35132.log.gz 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42204.log 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39701.log.gz 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35019.log 88.3% 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37943.log.gz 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35913.log 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36720.log.gz 2024-09-18T12:46:32.180 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42204.log.gz 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34995.log 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35019.log.gz 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41358.log 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35913.log.gz 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37442.log 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37028.log 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34995.log.gz 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38188.log 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35642.log 2024-09-18T12:46:32.181 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37442.log.gz 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35365.log 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37028.log.gz 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38188.log.gz 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41406.log 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35642.log.gz 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37416.log 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35365.log.gz 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40271.log 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41430.log 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41406.log.gz 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35663.log 2024-09-18T12:46:32.182 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37416.log.gz 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35663.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40271.log.gz 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41430.log.gz 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36424.log 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37654.log 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35663.log.gz 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40247.log 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41116.log 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36424.log.gz 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40745.log 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37654.log.gz 2024-09-18T12:46:32.183 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35621.log 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40247.log.gz 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36884.log 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41116.log.gz 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39878.log 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40745.log.gz 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37392.log 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35621.log.gz 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40151.log 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37392.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36884.log.gz 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39878.log.gz 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37200.log 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37392.log.gz 2024-09-18T12:46:32.184 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40151.log.gz 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42421.log 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz/var/log/ceph/ceph-client.admin.41478.log: 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37226.log 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37200.log.gz 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.foo.42421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37100.log 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.foo.42421.log.gz 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37226.log.gz 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38262.log 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gz 2024-09-18T12:46:32.185 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36790.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37100.log.gz 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38774.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39650.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38262.log.gz 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36790.log.gz 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43045.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38774.log.gz 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36033.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39650.log.gz 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.43045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43020.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36033.log: gzip -5 --verbose -- 89.5% /var/log/ceph/ceph-client.admin.40721.log 2024-09-18T12:46:32.186 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43045.log.gz 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.43020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42736.log 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36033.log.gz 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39328.log 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43020.log.gz 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.bar.42736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38310.log 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40721.log.gz 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36548.log 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39328.log: 88.2% -- replaced with /var/log/ceph/ceph-client.bar.42736.log.gz 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39524.log 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39328.log.gz 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36376.log 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-09-18T12:46:32.187 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38310.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36548.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36081.log 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39524.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37052.log 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36376.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38553.log 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37801.log 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36081.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35756.log 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37052.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35421.log 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38553.log.gz 2024-09-18T12:46:32.188 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35804.log 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37801.log.gz 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35756.log.gz 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32032.log 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35421.log.gz 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36279.log 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35804.log.gz 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.32032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40577.log 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35293.log 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32032.log.gz 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40577.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36279.log.gz 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40007.log 2024-09-18T12:46:32.189 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38064.log 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40577.log.gz 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41019.log 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35293.log.gz 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37174.log 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40007.log.gz 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41454.log 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38064.log.gz 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42373.log 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41019.log.gz 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39677.log 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37174.log.gz 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36836.log 2024-09-18T12:46:32.190 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41454.log.gz 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38164.log 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42373.log.gz 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38605.log 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39677.log.gz 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31912.log 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36836.log.gz 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37274.log 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38164.log.gz 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.31912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36400.log 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38605.log.gz 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36980.log 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31912.log.gz 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38016.log 2024-09-18T12:46:32.191 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37274.log.gz 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36400.log.gz 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42338.log 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36980.log.gz 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41091.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38016.log.gzgzip 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37729.log 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40175.log 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35156.log 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42338.log.gz 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37729.log.gz 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39205.log 2024-09-18T12:46:32.192 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40175.log.gz 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36620.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35156.log.gz 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36768.log 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39452.log 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39205.log.gz 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36303.log 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36768.log.gz 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39452.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35202.log 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39452.log.gz 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40769.log 2024-09-18T12:46:32.193 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36303.log.gz 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35985.log 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38238.log 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35202.log.gz 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40295.log 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40769.log.gz 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39279.log 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41993.log 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38238.log.gz 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39279.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40295.log.gz 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42081.log 2024-09-18T12:46:32.194 INFO:teuthology.orchestra.run.smithi144.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39279.log.gz 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41262.log 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41993.log.gz 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37679.log 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38654.log 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42081.log.gz 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41262.log.gz 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42934.log 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37679.log.gz 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34922.log 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38654.log.gz 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42934.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36860.log 2024-09-18T12:46:32.195 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34922.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42934.log.gz 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37825.log 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39080.log 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34922.log.gz 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36860.log: /var/log/ceph/ceph-client.admin.37825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36523.log 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37825.log.gz 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36860.log.gz 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35961.log 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39080.log.gz 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40553.log 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-09-18T12:46:32.196 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36523.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35961.log: /var/log/ceph/ceph-client.admin.40553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36448.log 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41574.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35961.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42958.log 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40553.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38868.log 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr: 90.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.bar.42958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36744.log 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36448.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36351.log 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr: 86.3%/var/log/ceph/ceph-client.admin.38868.log: -- replaced with /var/log/ceph/ceph-client.bar.42958.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42760.log 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38868.log.gz 2024-09-18T12:46:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35889.log 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36744.log.gz 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.bar.42760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36351.log.gz 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35889.log: 88.5% -- replaced with /var/log/ceph/ceph-client.bar.42760.log.gz 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38579.log 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:gzip/var/log/ceph/ceph-client.admin.40793.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39830.log 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35889.log.gz 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40055.log 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39830.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38579.log.gz 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37919.log 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35477.log 2024-09-18T12:46:32.198 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39830.log.gz 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37991.log 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40055.log.gz 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35684.log 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37919.log.gz 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38726.log 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35477.log.gz 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35555.log 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37991.log.gz 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38796.log 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35684.log.gz 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35447.log 2024-09-18T12:46:32.199 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38726.log.gz 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35555.log: /var/log/ceph/ceph-client.admin.38796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36057.log 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35555.log.gz 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37348.log 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38796.log.gz 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35447.log.gz 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36057.log.gz 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40392.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37348.log.gz 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38844.log 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38938.log 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2024-09-18T12:46:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38844.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38212.log 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41213.log 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38938.log: /var/log/ceph/ceph-client.admin.38212.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38844.log.gz 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39623.log 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38938.log.gz 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41334.log 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38212.log.gz 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38750.log 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41213.log.gz 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37584.log 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39623.log.gz 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39500.log 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41334.log.gz 2024-09-18T12:46:32.201 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38750.log.gz 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37704.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37584.log.gz 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37895.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39500.log.gz 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42535.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37895.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gzgzip 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37538.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.foo.42535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41945.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37895.log.gz 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35937.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2024-09-18T12:46:32.202 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37538.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35937.log: gzip -5 87.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41945.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr: /var/log/ceph/ceph-client.admin.37490.log 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.foo.42535.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35503.log 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35937.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42513.log 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40416.log 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37490.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35503.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36153.log 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.foo.42513.log: /var/log/ceph/ceph-client.admin.40416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36646.log 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.foo.42513.log.gz 2024-09-18T12:46:32.203 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39905.log 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40416.log.gz 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42804.log 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36153.log.gz 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39905.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36646.log.gz 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42491.log 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.bar.42804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35341.log 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39905.log.gz 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.foo.42491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35577.log 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.bar.42804.log.gz 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35341.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.foo.42491.log.gz 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37967.log 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42106.log 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35341.log.gz 2024-09-18T12:46:32.204 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37967.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35577.log.gz -5 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40440.log 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42106.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37967.log.gz 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42826.log 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35178.log 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.bar.42826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35044.log 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40440.log.gz 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37630.log 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.bar.42826.log.gz 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35178.log.gz 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35044.log: /var/log/ceph/ceph-client.admin.38358.log 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2024-09-18T12:46:32.205 INFO:teuthology.orchestra.run.smithi144.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35044.log.gz 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37630.log.gz 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42712.log 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38358.log.gz 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40127.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39932.log 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37753.log 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.42712.log.gz 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40031.log 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39932.log.gz 2024-09-18T12:46:32.206 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40320.log 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37753.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37753.log.gz 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40031.log.gz 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35828.log 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40320.log.gz 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35317.log 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35317.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35828.log.gz 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-09-18T12:46:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35317.log.gz 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37076.log 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36696.log 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37562.log 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36473.log 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37076.log.gz 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39806.log 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36696.log.gz 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37562.log.gz/var/log/ceph/ceph-client.admin.36473.log: 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42228.log 2024-09-18T12:46:32.208 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34970.log 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36473.log.gz 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38284.log 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39806.log.gz 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34970.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.39548.log 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42228.log.gz 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38963.log 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34970.log.gz 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40529.log 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38284.log.gz 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36908.log 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39548.log.gz 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40529.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38963.log.gz 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41752.log 2024-09-18T12:46:32.209 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39012.log 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40529.log.gz 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39983.log 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36908.log.gz 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39752.log 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41752.log.gz 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39012.log.gz 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39752.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39983.log.gzgzip 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37148.log 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39104.log 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39752.log.gz 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37148.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.35780.log -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz 2024-09-18T12:46:32.210 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36572.log 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37148.log.gz 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36672.log 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39104.log.gz 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40995.log 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35780.log.gz 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36572.log.gz 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36672.log: /var/log/ceph/ceph-client.admin.40995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34946.log 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36672.log.gz 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40995.log.gz 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37252.log 2024-09-18T12:46:32.211 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40223.log 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41286.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34946.log.gz 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41848.log 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38630.log 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37252.log.gz 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38890.log 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41848.log: /var/log/ceph/ceph-client.admin.38630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35088.log 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40223.log.gz 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38890.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41848.log.gz 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40817.log 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38630.log.gz 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41776.log 2024-09-18T12:46:32.212 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35088.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38890.log.gz 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35110.log 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35088.log.gz 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40817.log.gz 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37322.log 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41776.log.gz 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37004.log 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35110.log.gz 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37322.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37004.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37322.log.gz 2024-09-18T12:46:32.213 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38506.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37004.log.gz 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38506.log.gz -5 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42155.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37298.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41646.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37298.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36129.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31861.log 2024-09-18T12:46:32.214 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37298.log.gz 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41646.log.gz 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.31861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36932.log 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36129.log.gz 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41165.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31861.log.gz 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40971.log 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37606.log 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36932.log.gz 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36253.log 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40505.log 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40971.log.gz 2024-09-18T12:46:32.215 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36814.log 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37606.log.gz 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36253.log.gz 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40505.log: -- /var/log/ceph/ceph-client.admin.40481.log 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35599.log 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40505.log.gz 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40481.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36814.log.gz 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35599.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42971.log 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40625.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35599.log.gz 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42469.log 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39229.log 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2024-09-18T12:46:32.216 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.foo.42469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41067.log 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42971.log.gz 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39229.log: -- 88.8% -- replaced with /var/log/ceph/ceph-client.foo.42469.log.gz 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr: /var/log/ceph/ceph-client.admin.42179.log 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42397.log 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39229.log.gz 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42179.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41067.log.gz 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31821.log 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42397.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42179.log.gz 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42276.log 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.31821.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42397.log.gz 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38382.log 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.42276.log: gzip 88.3% -5 --verbose -- /var/log/ceph/ceph-client.bar.42782.log 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31821.log.gz 2024-09-18T12:46:32.217 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37847.log 88.4% 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42276.log.gz 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.bar.42782.log: -- /var/log/ceph/ceph-client.admin.38530.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38382.log.gz 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.bar.42782.log.gz 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37847.log.gzgzip 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38430.log 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38530.log: /var/log/ceph/ceph-client.admin.40697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40841.log 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31952.log 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40841.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38430.log.gz 2024-09-18T12:46:32.218 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31952.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40841.log.gz 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr: /var/log/ceph/ceph-mon.b.log 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31952.log.gz 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39404.log 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4%/var/log/ceph/ceph-client.admin.39404.log: -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37871.log 2024-09-18T12:46:32.219 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39304.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39404.log.gz 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36177.log 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36327.log 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37871.log.gz 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36202.log: /var/log/ceph/ceph-client.admin.36177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39572.log 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36105.log 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36177.log.gz 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39130.log 2024-09-18T12:46:32.220 INFO:teuthology.orchestra.run.smithi144.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36327.log.gz 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39572.log.gz 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:gzip/var/log/ceph/ceph-client.admin.36105.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39725.log 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36105.log.gz 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41969.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38914.log 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39130.log.gz 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39725.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38454.log 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38914.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42848.log 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39725.log.gz 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42445.log 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38914.log.gz 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-09-18T12:46:32.221 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.bar.42848.log: 89.1%/var/log/ceph/ceph-client.foo.42445.log: -- replaced with /var/log/ceph/ceph-client.admin.38454.log.gz 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38116.log 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41310.log: -- /var/log/ceph/ceph-client.bazar.42362.log 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.foo.42445.log.gz 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr: -- /var/log/ceph/ceph-client.admin.41873.log 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.bazar.42362.log: /var/log/ceph/ceph-client.admin.38116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31992.log 83.2% -- replaced with /var/log/ceph/ceph-client.bazar.42362.log.gz 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.bar.42848.log.gz 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41873.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38116.log.gz 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.31992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36498.log 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41873.log.gz 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37516.log 2024-09-18T12:46:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40079.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31992.log.gz 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.36498.log: -- /var/log/ceph/ceph-client.admin.40865.log 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37516.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38334.log 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36498.log.gz 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38040.log 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37516.log.gz 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40865.log: /var/log/ceph/ceph-client.admin.38334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38040.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40865.log.gzgzip 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38334.log.gz 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38702.log 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38040.log.gz 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41502.log: /var/log/ceph/ceph-client.admin.36956.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gz 2024-09-18T12:46:32.223 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35732.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36596.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39959.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38702.log.gz 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35708.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35732.log.gz 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39959.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36596.log.gz 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35066.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35708.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39959.log.gz 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35269.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35066.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35708.log.gz 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38406.log 2024-09-18T12:46:32.224 INFO:teuthology.orchestra.run.smithi144.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35066.log.gz 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35269.log.gz 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37370.log 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38406.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35391.log 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37370.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35224.log 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38406.log.gz 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37370.log.gz 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.35224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40889.log 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.35391.log.gz 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40649.log: /var/log/ceph/ceph-osd.3.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35224.log.gz 2024-09-18T12:46:32.225 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38678.log 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37124.log 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38678.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.41921.log 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38678.log.gz 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39779.log 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.37124.log: /var/log/ceph/ceph-client.admin.41921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38480.log 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.39779.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37124.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41921.log.gz 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39779.log.gz 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38140.log 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.38480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37777.log 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38480.log.gz/var/log/ceph/ceph-client.admin.38140.log: 2024-09-18T12:46:32.226 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40199.log 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41728.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38140.log.gz 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.40199.log: /var/log/ceph/ceph-client.admin.37777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42701.log 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40199.log.gz 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37777.log.gz 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.41728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36009.log 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.foo.42701.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41728.log.gz 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.foo.42701.log.gz 2024-09-18T12:46:32.227 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.36009.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36009.log.gz 2024-09-18T12:46:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36678.log 2024-09-18T12:46:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46047.log 2024-09-18T12:46:32.249 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51572.log.gz 2024-09-18T12:46:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35074.log 2024-09-18T12:46:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36678.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51404.log.gz 2024-09-18T12:46:32.249 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50043.log 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48665.log 89.7% 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36678.log.gz 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.46047.log.gz 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.50043.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38210.log 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35074.log.gz 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39240.log 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50043.log.gz 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48665.log.gz 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35398.log 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.38210.log.gz 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44871.log: /var/log/ceph/ceph-client.admin.35398.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-09-18T12:46:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46131.log 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36953.log 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35398.log.gz 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39854.log: /var/log/ceph/ceph-client.admin.46131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46695.log 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49270.log 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46131.log.gz 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38168.log 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49270.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.36953.log.gz 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34527.log 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46695.log.gz 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37771.log 2024-09-18T12:46:32.251 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49270.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34527.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38168.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48976.log 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36720.log 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34527.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37771.log: /var/log/ceph/ceph-client.admin.48976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41358.log 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37813.log 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37771.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.48976.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34956.log 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36720.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34956.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37813.log.gz 2024-09-18T12:46:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37981.log 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47680.log 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41694.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34956.log.gz 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37981.log.gz 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47680.log.gz 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35818.log 2024-09-18T12:46:32.253 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35692.log 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39282.log 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47031.log 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35818.log.gz 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39282.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35692.log.gz 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35650.log 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49749.log 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39282.log.gz 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36424.log 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47031.log.gz 2024-09-18T12:46:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49749.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35650.log.gz 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48159.log 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47260.log 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49749.log.gz 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51026.log 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36424.log.gz 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42171.log 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.48159.log.gz 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47890.log 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42171.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.47260.log.gz 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36787.log 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51026.log.gz 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36995.log 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47890.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42171.log.gz 2024-09-18T12:46:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47470.log 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36995.log: gzip -5 --verbose 89.3% -- -- replaced with /var/log/ceph/ceph-client.admin.47890.log.gz /var/log/ceph/ceph-client.admin.45165.log 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36787.log.gz 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48201.log 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36995.log.gz 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35188.log 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47470.log.gz 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45501.log 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48201.log.gz 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45501.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35188.log.gz 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49623.log 2024-09-18T12:46:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41610.log 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42549.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45501.log.gz 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48538.log 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41274.log 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49623.log.gz 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49144.log 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41610.log.gz 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41274.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48538.log.gz 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52035.log 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41274.log.gz 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50337.log 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49144.log.gz 2024-09-18T12:46:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.52035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34775.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50337.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52035.log.gz 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51698.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34775.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50337.log.gz 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42297.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34775.log.gz 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46528.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gz 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42297.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51698.log.gz 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51782.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49665.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46528.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42297.log.gz 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35048.log 2024-09-18T12:46:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49665.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46528.log.gz 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48623.log 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51782.log.gz 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48327.log 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35048.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49665.log.gz 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40711.log 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48327.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35048.log.gz 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39770.log 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48623.log.gz 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48327.log.gz 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44139.log 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40711.log.gz 2024-09-18T12:46:32.259 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37079.log 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.39770.log.gz 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37855.log 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45627.log 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37855.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44139.log.gz 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34780.log 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37079.log.gz 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34523.log 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37855.log.gz 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39938.log 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45627.log.gz 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34523.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.34780.log.gz -5 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2024-09-18T12:46:32.260 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45543.log 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39938.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34523.log.gz 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47302.log 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45543.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39938.log.gz 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36382.log 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47302.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45543.log.gz 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35944.log 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42255.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47302.log.gz 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36382.log.gz 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38946.log 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48411.log 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35944.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2024-09-18T12:46:32.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37121.log 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5%/var/log/ceph/ceph-client.admin.48411.log: -- replaced with /var/log/ceph/ceph-client.admin.35944.log.gz 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37121.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44517.log 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38946.log.gz 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48411.log.gz 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43576.log 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37121.log.gz 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51614.log 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44517.log: /var/log/ceph/ceph-client.admin.43576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42675.log 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51614.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44517.log.gz 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45459.log 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43576.log.gz 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42675.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.51614.log.gz -5 2024-09-18T12:46:32.262 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43660.log 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42675.log.gz 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43786.log 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45459.log.gz 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51152.log 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43660.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37290.log 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43786.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43660.log.gz 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34390.log 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51152.log: /var/log/ceph/ceph-client.admin.37290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48892.log 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.43786.log.gz 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.51152.log.gz 2024-09-18T12:46:32.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50899.log 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37290.log.gz 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42465.log 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48892.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34390.log.gz 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48892.log.gz 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50899.log.gz 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35356.log 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42465.log.gz 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42885.log 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46005.log 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35356.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-09-18T12:46:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42003.log 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35356.log.gz 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.46005.log: --verbose -- /var/log/ceph/ceph-client.admin.50942.log 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42885.log.gz 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36466.log 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46005.log.gz 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42003.log.gz 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47951.log 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.50942.log.gz 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38252.log 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36466.log.gz 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44955.log 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47951.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-09-18T12:46:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48117.log 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47951.log.gz 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44955.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38252.log.gz 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51909.log 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38778.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44955.log.gz 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49917.log 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48117.log.gz 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38023.log 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51909.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51909.log.gz 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36340.log 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49917.log.gz 2024-09-18T12:46:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48807.log 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38023.log.gz 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43970.log 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48807.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36340.log.gz 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.48807.log.gz --verbose 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.34190.log 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43970.log: /var/log/ceph/ceph-client.admin.38462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50001.log 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34190.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43970.log.gz 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50253.log 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50001.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34190.log.gz 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47159.log 2024-09-18T12:46:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37729.log 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50001.log.gz 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45333.log 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8%/var/log/ceph/ceph-client.admin.37729.log: -- replaced with /var/log/ceph/ceph-client.admin.50253.log.gz 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47159.log.gz 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39156.log 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.37729.log.gz 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.45333.log.gz 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43137.log 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35608.log 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39156.log.gz 2024-09-18T12:46:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43137.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43011.log 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50505.log 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35608.log: /var/log/ceph/ceph-client.admin.43011.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43137.log.gz 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52077.log 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35608.log.gz 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.52077.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.43011.log.gz 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50505.log.gz 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35272.log 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52077.log.gz 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45123.log: /var/log/ceph/ceph-client.admin.46653.log 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2024-09-18T12:46:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43702.log 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46215.log 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35272.log.gz 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40248.log 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46215.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43702.log.gz 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34306.log 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.46653.log.gz 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40248.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46215.log.gz 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35076.log 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.audit.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34306.log.gz 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 89.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40248.log.gz -- 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.43618.log 2024-09-18T12:46:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43240.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35076.log: /var/log/ceph/ceph-client.admin.43618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51530.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35076.log.gz 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43240.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43618.log.gz 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44307.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51530.log: gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43240.log.gz 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.40585.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44307.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51530.log.gz 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37432.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.40585.log: --verbose -- /var/log/ceph/ceph-client.admin.49833.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44307.log.gz 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40585.log.gz 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45417.log 2024-09-18T12:46:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49833.log.gz 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- 92.7% -- replaced with /var/log/ceph/ceph-client.admin.37432.log.gz 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.46863.log 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45417.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34227.log 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46863.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45417.log.gz 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44602.log 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46863.log.gz 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36214.log 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43053.log 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44602.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34227.log.gz 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49186.log 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43053.log: gzip -5 89.7% --verbose -- /var/log/ceph/ceph-client.admin.36508.log 2024-09-18T12:46:32.272 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44602.log.gz 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36214.log.gz 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35524.log 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 90.9% -- replaced with /var/log/ceph/ceph-client.admin.43053.log.gz 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.49186.log.gz 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35524.log: gzip -5 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36508.log.gz 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50984.log 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44013.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35524.log.gz 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51446.log 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44744.log 2024-09-18T12:46:32.273 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34528.log 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.50984.log.gz 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37645.log 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51446.log.gz 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46257.log 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37645.log: gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44744.log.gz 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34528.log.gz 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.45795.log 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46257.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37645.log.gz 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37561.log 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45795.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46257.log.gz 2024-09-18T12:46:32.274 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51068.log 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50211.log 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44663.log 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51068.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.45795.log.gz 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37561.log.gz 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37603.log 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51068.log.gz 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49354.log 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44663.log: /var/log/ceph/ceph-client.admin.37603.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50211.log.gz 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44349.log 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44663.log.gz 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44828.log 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37603.log.gz 2024-09-18T12:46:32.275 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44349.log: /var/log/ceph/ceph-client.admin.44828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39644.log 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49354.log.gz 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37477.log: gzip -5 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49018.log 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44828.log.gz 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39644.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51824.log 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45249.log 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49018.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39644.log.gz 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45249.log: gzip 90.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.49018.log.gz --verbose 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.36869.log 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51824.log.gz 2024-09-18T12:46:32.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36256.log 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.45249.log.gz 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50589.log 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36256.log: /var/log/ceph/ceph-client.admin.50631.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36869.log.gz 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41652.log 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36256.log.gz 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50589.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46821.log 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50589.log.gz 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42927.log 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz 2024-09-18T12:46:32.277 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48580.log 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39072.log: /var/log/ceph/ceph-client.admin.42927.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46821.log.gz 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50818.log 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42927.log.gz 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41919.log 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50818.log: /var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51278.log 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.48580.log.gz 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50818.log.gz 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41919.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47074.log 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41919.log.gz 2024-09-18T12:46:32.278 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.47074.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51278.log.gz 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51951.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42801.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.47074.log.gz 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51236.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51951.log.gz 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35440.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42801.log.gz/var/log/ceph/ceph-client.admin.51236.log: 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37519.log 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-09-18T12:46:32.279 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51236.log.gz 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44097.log 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35440.log.gz 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37519.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41484.log 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37519.log.gz/var/log/ceph/ceph-client.admin.44097.log: 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50715.log 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.41484.log: --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.41400.log 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50715.log: gzip -5 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41484.log.gz 2024-09-18T12:46:32.280 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47512.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46989.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50715.log.gz 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47512.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41400.log.gz 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47848.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50547.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47512.log.gz 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35860.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46989.log.gz 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45039.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47848.log.gz 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.35860.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50547.log.gz 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39686.log 2024-09-18T12:46:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39198.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35860.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45039.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52119.log 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49791.log 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39686.log: -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.52119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35986.log 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39686.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49791.log: gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.52119.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45207.log 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39896.log 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45207.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49791.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35986.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51993.log 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45207.log.gz 2024-09-18T12:46:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45081.log 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39896.log.gz 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39812.log 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50421.log 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51993.log.gz 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39812.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45081.log.gz 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44181.log 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46089.log 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50421.log.gz 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45963.log 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44181.log.gz 2024-09-18T12:46:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45963.log: gzip -5 88.7% --verbose -- /var/log/ceph/ceph-client.admin.46467.log 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46089.log.gz 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46737.log 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.45963.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47344.log 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8%/var/log/ceph/ceph-client.admin.46467.log: -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46737.log.gz 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49060.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46467.log.gz 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48032.log 2024-09-18T12:46:32.284 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38631.log 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40167.log: /var/log/ceph/ceph-client.admin.48032.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26959.log 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38631.log: gzip 89.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz --verbose 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.40753.log 88.1% 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48032.log.gz 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.tmp-client.admin.26959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37205.log 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38631.log.gz 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.26959.log.gz 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31577.log 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51740.log 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.31577.log: gzip -5 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37205.log.gz 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50673.log 2024-09-18T12:46:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51740.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31577.log.gz 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51656.log 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50673.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51740.log.gz 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47596.log 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49497.log 90.5% -- replaced with /var/log/ceph/ceph-client.admin.50673.log.gz 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47596.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51656.log.gz 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34997.log 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49497.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47596.log.gz 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48496.log 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34997.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49497.log.gz 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48496.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34997.log.gz 2024-09-18T12:46:32.286 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40375.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38862.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48496.log.gz 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38988.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31348.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43408.log/var/log/ceph/ceph-client.admin.38862.log: 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8%/var/log/ceph/ceph-client.admin.31348.log: -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38862.log.gz 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43366.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3%/var/log/ceph/ceph-client.admin.43408.log: /var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.31348.log.gz 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43324.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34532.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43408.log.gz 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34783.log 2024-09-18T12:46:32.287 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43366.log.gz 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37332.log 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34783.log: 88.4% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34532.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43324.log.gz 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38547.log 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6%/var/log/ceph/ceph-client.admin.37332.log: -- replaced with /var/log/ceph/ceph-client.admin.34783.log.gz 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34530.log 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38336.log 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37332.log.gz 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38547.log: /var/log/ceph/ceph-client.admin.34530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34784.log 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38336.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34530.log.gz 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38547.log.gz 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38336.log.gz 2024-09-18T12:46:32.288 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48453.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43282.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48453.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49581.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34784.log.gz 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.43282.log: --verbose -- /var/log/ceph/ceph-client.admin.37687.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38589.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48453.log.gz 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37687.log: -- replaced with /var/log/ceph/ceph-client.admin.43282.log.gz 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49581.log.gz 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48850.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37687.log.gz 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.38589.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48369.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38086.log 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38589.log.gz 2024-09-18T12:46:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40963.log 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.48850.log.gz 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45669.log 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48369.log.gz 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40963.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38086.log.gz 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42339.log 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40627.log 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8%/var/log/ceph/ceph-client.admin.45669.log: -- replaced with /var/log/ceph/ceph-client.admin.40963.log.gz 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41568.log 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8%/var/log/ceph/ceph-client.admin.40627.log: -- replaced with /var/log/ceph/ceph-client.admin.45669.log.gz 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42339.log.gz 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36634.log 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45753.log 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40627.log.gz 2024-09-18T12:46:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41568.log: /var/log/ceph/ceph-client.admin.36634.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8%/var/log/ceph/ceph-client.admin.45753.log: -- replaced with /var/log/ceph/ceph-client.admin.41568.log.gz 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37037.log 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45753.log.gz 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36634.log.gz 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41877.log 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8%/var/log/ceph/ceph-client.admin.41779.log: -- replaced with /var/log/ceph/ceph-client.admin.37037.log.gz 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49959.log 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41877.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47806.log 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41877.log.gz 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42759.log 2024-09-18T12:46:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47806.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49959.log.gz 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.47806.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34531.log 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45879.log 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42759.log: /var/log/ceph/ceph-client.admin.34531.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34531.log.gz 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42759.log.gz 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51362.log 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38735.log 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51362.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45879.log.gz 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31690.log 2024-09-18T12:46:32.292 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51362.log.gz 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46947.log 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.31690.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38735.log.gz 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31690.log.gz 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35734.log 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45837.log 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35734.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.46947.log.gz 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35734.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36550.log 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46905.log 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45837.log: /var/log/ceph/ceph-client.admin.36550.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36550.log.gz 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45837.log.gz 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34774.log 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38904.log 2024-09-18T12:46:32.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34774.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46905.log.gz 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42507.log 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34774.log.gz 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36130.log 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38904.log.gz 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40795.log 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36130.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42507.log.gz 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35902.log 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36130.log.gz 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45921.log 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40795.log: /var/log/ceph/ceph-client.admin.35902.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40795.log.gz 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40106.log 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35902.log.gz 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50127.log 2024-09-18T12:46:32.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40106.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45921.log.gz 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40106.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45585.log 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40669.log 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50127.log: /var/log/ceph/ceph-client.admin.45585.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45585.log.gz 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50127.log.gz 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43492.log 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42087.log 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43492.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40669.log.gz 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52159.log 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42087.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43492.log.gz 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51110.log 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.42087.log.gz 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.52159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50463.log 2024-09-18T12:46:32.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51110.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52159.log.gz 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47428.log 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.51110.log.gz 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49707.log 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47428.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.50463.log.gz 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47428.log.gz 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42633.log 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35776.log 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42633.log: /var/log/ceph/ceph-client.admin.49707.log: 88.2% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49707.log.gz 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42633.log.gz 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40022.log 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34782.log 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35776.log: 89.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42843.log 2024-09-18T12:46:32.296 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35776.log.gz 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34782.log: 90.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51866.log 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40022.log.gz 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 89.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.48075.log 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34782.log.gz 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42843.log: /var/log/ceph/ceph-client.admin.51866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38294.log 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48075.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42843.log.gz 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.51866.log.gz 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37163.log 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 90.1% -- replaced with /var/log/ceph/ceph-client.admin.48075.log.gz 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38294.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43828.log 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36048.log 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38294.log.gz 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43828.log: /var/log/ceph/ceph-client.admin.37163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42213.log 2024-09-18T12:46:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36048.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43828.log.gz 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37163.log.gz 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36048.log.gz 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45711.log 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49102.log 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44391.log 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42213.log.gz 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41147.log 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49102.log: /var/log/ceph/ceph-client.admin.44391.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45711.log.gz 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49102.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50169.log 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44391.log.gz 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39980.log 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41147.log: /var/log/ceph/ceph-client.admin.50169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34348.log 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39980.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41147.log.gz 2024-09-18T12:46:32.298 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50169.log.gz 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37248.log 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39980.log.gz 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43450.log/var/log/ceph/ceph-client.admin.34348.log: 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47638.log 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34348.log.gz 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 91.1% -- replaced with /var/log/ceph/ceph-client.admin.37248.log.gz 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40501.log 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43450.log.gz 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40921.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47638.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42969.log 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44913.log 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42969.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-09-18T12:46:32.299 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40501.log.gz 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38504.log 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42969.log.gz 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44055.log 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46609.log 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38504.log: /var/log/ceph/ceph-client.admin.44055.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44913.log.gz 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9%gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.36911.log 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44055.log.gz 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38504.log.gz 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47722.log 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51194.log 88.6% 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46609.log.gz 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36911.log: gzip/var/log/ceph/ceph-client.admin.47722.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45291.log 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51194.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47722.log.gz 2024-09-18T12:46:32.300 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51194.log.gz 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50379.log 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.36911.log.gz 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40459.log 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43744.log 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45291.log.gz 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6%/var/log/ceph/ceph-client.admin.40459.log: -- replaced with /var/log/ceph/ceph-client.admin.50379.log.gz 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48243.log 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36298.log 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48243.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40459.log.gz 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34936.log 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43744.log.gz 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48243.log.gz 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.36298.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44997.log 2024-09-18T12:46:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35314.log 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36298.log.gz 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44997.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34936.log.gz 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49539.log 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44997.log.gz 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44786.log 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35314.log.gz 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49539.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gzgzip -5 --verbose -- 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.51320.log 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35482.log 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51320.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49539.log.gz 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44786.log.gz 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49875.log 2024-09-18T12:46:32.302 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51320.log.gz 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42717.log 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51488.log 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49875.log: /var/log/ceph/ceph-client.admin.42717.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35482.log.gz 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48934.log 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42717.log.gz 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8%/var/log/ceph/ceph-client.admin.51488.log: -- replaced with /var/log/ceph/ceph-client.admin.49875.log.gz 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46341.log 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48934.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.51488.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39387.log 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43889.log 89.9% 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48934.log.gz 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46341.log: /var/log/ceph/ceph-client.admin.39387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7%/var/log/ceph/ceph-client.admin.43889.log: -- replaced with /var/log/ceph/ceph-client.admin.39387.log.gz 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46341.log.gz 2024-09-18T12:46:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44223.log 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43889.log.gz 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50295.log 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39469.log 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz/var/log/ceph/ceph-client.admin.44223.log: 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42381.log 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39469.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44223.log.gz 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50295.log.gz 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46173.log 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42381.log: /var/log/ceph/ceph-client.admin.40837.log 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% 92.2%/var/log/ceph/ceph-client.admin.46173.log: -- replaced with /var/log/ceph/ceph-client.admin.39469.log.gz 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42381.log.gz 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48285.log 2024-09-18T12:46:32.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40837.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46173.log.gz 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35208.log 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48285.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50085.log 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46299.log 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35208.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48285.log.gz 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46383.log 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46299.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35208.log.gz 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50085.log.gz 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46299.log.gz 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44265.log 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36592.log 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46383.log.gz 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41442.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44265.log.gz 2024-09-18T12:46:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35566.log 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38378.log 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35566.log: 88.9% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36592.log.gz 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35566.log.gz 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47117.log 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47554.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38378.log.gz 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34134.log 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.44560.log 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47117.log: /var/log/ceph/ceph-client.admin.34134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45375.log 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34134.log.gz 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44560.log: -- replaced with /var/log/ceph/ceph-client.admin.47117.log.gz 2024-09-18T12:46:32.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46779.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44560.log.gz 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37939.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48726.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.45375.log.gz 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37939.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.46779.log.gz 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47386.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46425.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47386.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37939.log.gz 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48726.log.gz 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37897.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47386.log.gz 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46425.log: /var/log/ceph/ceph-client.admin.44475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42129.log 2024-09-18T12:46:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37897.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46425.log.gz 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37897.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36172.log 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42129.log: /var/log/ceph/ceph-client.admin.40064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47764.log 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42129.log.gz/var/log/ceph/ceph-client.admin.36172.log: 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49312.log 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49228.log 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36172.log.gz 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50757.log 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47764.log.gz 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49228.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49455.log 2024-09-18T12:46:32.308 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49312.log.gz 2024-09-18T12:46:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38420.log 2024-09-18T12:46:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49455.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.49228.log.gz 2024-09-18T12:46:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-09-18T12:46:32.309 INFO:teuthology.orchestra.run.smithi116.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.50757.log.gz 2024-09-18T12:46:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.38420.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.49455.log.gz 2024-09-18T12:46:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41823.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38420.log.gz 2024-09-18T12:46:32.309 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-09-18T12:46:32.318 INFO:teuthology.orchestra.run.smithi144.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-18T12:46:32.433 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-18T12:46:32.673 INFO:teuthology.orchestra.run.smithi144.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-18T12:46:32.819 INFO:teuthology.orchestra.run.smithi116.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-18T12:46:32.891 INFO:teuthology.orchestra.run.smithi144.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-18T12:46:32.961 INFO:teuthology.orchestra.run.smithi116.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-18T12:46:32.993 INFO:teuthology.orchestra.run.smithi144.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-18T12:46:33.006 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-18T12:46:33.046 INFO:teuthology.orchestra.run.smithi144.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-18T12:46:33.048 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-18T12:46:33.048 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m1.127s 2024-09-18T12:46:33.048 INFO:teuthology.orchestra.run.smithi144.stderr:user 0m3.956s 2024-09-18T12:46:33.048 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m0.350s 2024-09-18T12:46:33.162 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-18T12:46:33.164 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-09-18T12:46:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m1.273s 2024-09-18T12:46:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m4.751s 2024-09-18T12:46:33.164 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m0.446s 2024-09-18T12:46:33.164 INFO:tasks.ceph:Archiving logs... 2024-09-18T12:46:33.165 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/remote/smithi116/log 2024-09-18T12:46:33.166 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T12:46:34.221 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/remote/smithi144/log 2024-09-18T12:46:34.222 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T12:46:35.138 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T12:46:35.148 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T12:46:35.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T12:46:35.152 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T12:46:35.232 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T12:46:35.242 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T12:46:35.242 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:46:35.245 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:46:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-09-18T12:46:35.262 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-09-18T12:46:35.263 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T12:46:35.263 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-09-18T12:46:35.263 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 +362us[ +355us] +/- 40ms 2024-09-18T12:46:35.263 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 -1902us[-1902us] +/- 58ms 2024-09-18T12:46:35.263 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +1754us[+1754us] +/- 67ms 2024-09-18T12:46:35.263 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:46:35.266 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T12:46:35.266 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-09-18T12:46:35.266 INFO:teuthology.orchestra.run.smithi144.stdout:^* hv01.front.sepia.ceph.com 3 6 377 21 +352us[ +350us] +/- 40ms 2024-09-18T12:46:35.266 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -1912us[-1912us] +/- 58ms 2024-09-18T12:46:35.267 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +1720us[+1720us] +/- 67ms 2024-09-18T12:46:35.267 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T12:46:35.267 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T12:46:35.277 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T12:46:35.277 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T12:46:35.286 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T12:46:35.378 DEBUG:teuthology.orchestra.run.smithi144:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T12:46:35.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T12:46:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:46:35.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T12:46:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:46:35.505 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T12:46:35.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T12:46:35.523 INFO:teuthology.task.internal:Duration was 1674.380082 seconds 2024-09-18T12:46:35.523 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T12:46:35.533 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T12:46:35.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T12:46:35.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T12:46:35.572 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T12:46:35.587 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T12:46:35.985 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T12:46:35.985 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-09-18T12:46:35.986 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T12:46:36.048 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2024-09-18T12:46:36.048 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T12:46:36.079 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T12:46:36.079 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T12:46:36.090 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T12:46:36.149 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T12:46:36.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T12:46:36.155 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T12:46:36.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T12:46:36.305 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T12:46:36.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T12:46:36.330 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T12:46:36.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T12:46:36.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:46:36.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:46:36.394 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-09-18T12:46:36.419 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2024-09-18T12:46:36.432 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:46:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:46:36.477 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:46:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:46:36.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T12:46:36.502 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T12:46:36.503 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/remote/smithi116 2024-09-18T12:46:36.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T12:46:36.552 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911038/remote/smithi144 2024-09-18T12:46:36.552 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T12:46:36.585 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T12:46:36.585 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T12:46:36.588 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T12:46:36.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T12:46:36.646 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T12:46:36.647 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T12:46:36.656 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T12:46:36.656 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T12:46:36.658 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T12:46:36.671 INFO:teuthology.orchestra.run.smithi116.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 12:46 /home/ubuntu/cephtest 2024-09-18T12:46:36.689 INFO:teuthology.orchestra.run.smithi144.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 12:46 /home/ubuntu/cephtest 2024-09-18T12:46:36.690 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T12:46:36.701 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T12:46:36.739 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/one workloads/rados_mon_workunits} duration: 1674.380081653595 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T12:46:36.739 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T12:46:36.825 INFO:teuthology.run:pass